Files
PaperClipAI/tests
DottaandPaperclip 4edd10804e fix(evals): batch task search result readback across renders (#18370)
## Thinking Path

> - Paperclip lets people manage AI agents and saved work.
> - Product journeys check user actions against saved state.
> - The task search journey compares the exact visible task identities.
> - A search can replace rows while the harness reads an old indexed
locator.
> - The attribute read can then time out after the correct results
appear.
> - This pull request reads the visible links together and preserves
strict result checks.

## Linked Issues or Issue Description

Refs #18355.

**What happened?**

An attached Cloud attempt passed the task search journey. A later
attempt on the same harness and app revision failed at search-result
readback. Its owned screenshot showed the correct target and completed
control. Authentication, collection, and cleanup passed. The failure
remains recorded and the journey has not been admitted to the daily
plan.

The reader called `all()`, then awaited `isVisible()` and
`getAttribute()` for each indexed locator. When a search replaced the
old collection between those reads, the next attribute read waited for
an index that no longer existed.

**Expected behavior**

Read the matching visible links together. Compare the complete identity
set against the owned target and controls. A render transition must not
consume the whole poll deadline on a stale row index.

**Steps to reproduce**

1. Render three task links in the task collection.
2. Let the reader observe the third link as visible.
3. Replace the collection with the two correct search matches before its
attribute read.
4. The old reader times out waiting for `nth(2)`, even though the two
correct rows are present.

## What Changed

- Read visible link attributes in one batch. Discard a detached snapshot
and retry within the existing poll deadline.
- Keep the exact identity comparison, duplicate detection, control
exclusion, saved-content checks, and zero-run checks.
- Add a real Chromium render-transition regression and invalid-result
cases. Extend the stateful flow test for a detached snapshot.
- Document the readback boundary. No app UI, target configuration, or
cleanup behavior changes.

## Verification

- Review follow-up: the existing journey wait helper is now exercised
with the real Playwright poll. Its first read evaluates the actual
reader callback on detached browser DOM nodes and returns no evidence.
The second read contains an extra owned control. Only the third, exact
pair succeeds. The test observes all three reads. The 30-second deadline
and grading behavior are unchanged.
- Rebased onto current master, including the merged project journey.
Credential-free discovery confirms exactly one `project-task-context`
and one `task-search-filter` entry.
- Latest rebased checks passed: all nine Chromium search regressions,
all 381 Product journey unit tests, runner typecheck, discovery, maps,
and diff checks. Exact head 183c3e1d19
passed all 58 CI checks, external Snyk status, and fresh Greptile 5/5
with no unresolved findings.

- Red: the real Chromium transition test failed on the old reader with
`locator.getAttribute: Timeout 150ms exceeded`, waiting for `nth(2)`.
The short timeout bounds only this regression fixture.
- Green: all nine credential-free Chromium search tests passed. They
cover the transition, duplicate/extra/unknown/wrong destinations,
visible-link scope, and status-label behavior.
- All 131 focused search/oracle/contract/campaign tests passed.
- Runner typecheck passed: `node node_modules/typescript/bin/tsc -p
tests/runner-e2e/tsconfig.json`.
- `node scripts/check-maps.mjs` and `git diff --check` passed.
- Fresh QA staging acceptance passed at this exact head: all five
checks, authentication/setup/journey/collection/cleanup phases, and zero
agent runs. The [measured QA
report](https://pages.paperclip.ing/evals-qa-smoke-41124961876-1/)
preserves current source and measurement provenance. Daily admission
follows merge and a normal TestOps job verification; the earlier failure
remains recorded.

## Risks

- The fix uses the existing Playwright visible-element predicate and
task-link selector. A future UI selector change can still fail the
journey.
- A detached snapshot returns no evidence. It cannot satisfy a nonempty
expected result set. The original poll deadline stays in place; there is
no added journey retry.
- Browser fixtures prove the readback mechanism. They do not replace the
required live Cloud pass.

## Model Used

OpenAI Codex, GPT-6 family. The host does not expose an exact model
version or context limit. Used repository tools, TypeScript checks, and
fresh local Chromium contexts.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-30 15:51:28 -05:00
..