Files
PaperClipAI/tests
DottaandPaperclip 8941db9ce6 fix(evals): accept editor escapes and cancel sensitive QA requests (#18244)
## Thinking Path

> - Paperclip manages AI agents that do work for users.
> - Product E2E journeys check real browser actions and saved results.
> - A rich editor can store plain underscores as escaped Markdown.
> - The QA reply check treated that valid saved message as a failure.
> - A private target adapter also needs to cancel credential-safe HTTP
requests before cleanup.
> - This PR fixes reply comparison and adds abort and timeout options to
the existing sensitive transport.

## Linked Issues or Issue Description

Refs #18222.

**What happened?**

The artifact revision and Stop journeys rejected the correct single user
reply because its underscore was stored as `\_`. Sensitive connection
setup also lacked a way to pass its lifecycle signal to the actual HTTP
request.

**Expected behavior**

The check accepts the editor's underscore escape and still rejects
changed, duplicate, missing, or foreign replies. A cancelled sensitive
request closes before setup cleanup proceeds.

**Steps to reproduce**

Run an artifact revision with an underscore in the reply. For the
transport case, abort a pending sensitive request or let its deadline
expire.

## What Changed

- Compare the fixture reply after undoing only underscore escaping. Keep
exact text, count, and author checks.
- Let `ConnectionApi.postSensitive` accept a bounded timeout and abort
signal. Check cancellation again after cookie loading and before
dispatch.
- Test the retained reply failure and both cancellation paths with an
actual loopback HTTP server. Document the adapter contract.

## Verification

- Focused Product E2E tests: 30 passed across two files.
- Product E2E TypeScript check passed. After rebase, a final check first
found four diagnostics from stale runner/plugin SDK declarations;
rebuilding those two workspace packages cleared all four, and the
unchanged E2E check passed.
- The first sandbox run failed because loopback `listen` returned
`EPERM`. The unchanged HTTP tests passed with loopback permission. The
test now reports listener errors directly.
- Both independent reviewers found a possible pre-arrival deadline-test
hang. The follow-up races arrival against request settlement and adds
one second of deadline headroom. All 30 affected tests passed after that
fix.
- Full current-head CI passed in run 40982914525. Greptile and Paperclip
Review both scored 5/5 at `f0526f5fbbf13de5ea43e868eb012069ca3d8907`;
all review threads are resolved. Live QA requalification follows the
merged public and private fixes; no live pass is claimed here.

## Risks

The text normalization applies only to plain fixture replies and only
removes underscore escapes. It does not remove arbitrary Markdown.
Sensitive calls retain their existing 30-second default and closed error
messages. API wrappers must forward the optional signal and await the
actual request.

## Model Used

OpenAI Codex, GPT-6 family. The serving model ID and context window are
not exposed by this session. Used reasoning, code editing, shell
execution, and test inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 20:14:56 -05:00
..