mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
bf18b348c5
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog drives real chat journeys in a browser and grades saved product state > - In full cohort 3, four cases acted on the page before the page was ready, or did not step through a card > - The product was correct in each case, so these failures hide real product failures > - This pull request makes the harness wait for the saved, rendered reply and for a settled form, and step through multi-question cards > - The benefit is fewer false failures, with the same product requirements ## Linked Issues or Issue Description Refs #21000 Refs #21008 **What happened?** - `companies.fast-responses`, `agents.choose-primary`, `budgets-costs.set-budget`: the server recorded the chat run as finished 0.2 to 1.3 seconds before it saved the reply comment. The harness read the reply links right after the run finished. It found no reply, so it found no links. - `budgets-costs.set-budget`: the budget card remounted after the harness typed the new amount. The remount reset the draft, so the Update budget button stayed disabled until the click timed out. - `questions-and-approvals.necessary-question`: the agent asked the required product-name question and one optional question. The card showed Next on its first page. The harness only clicked Submit answers. The grade also required exactly one question on the card. **Expected behavior** The harness reads a turn's links after its reply is saved and rendered. It saves the budget only on a settled form. It pages through a card, answers its one required question and skips optional ones. **Steps to reproduce** Run the three link cases when the reply is saved after the run finishes. Run set-budget when the budget card reloads after typing. Run necessary-question when the card has an optional second question. ## What Changed - `chat-rendered-links.ts`: new shared `waitForSavedTurnReplyRender`. It waits (bounded, 15 s) until each finished lead run's reply is saved, then uses the existing render wait for the latest reply. It returns `unsaved` when no reply arrives. The caller then fails with its own error. - `identity-support.ts`: new `waitForTurnReplyLinks` uses that wait for the source conversation. `companies.fast-responses` (v3), `agents.choose-primary` (v4) and `budgets-costs.set-budget` (v3) call it before they read reply links. - `budgets-costs.set-budget`: `settledInput` resolves when the same input node stays on the page for one second. The harness types and saves on that node. If the card still remounts and drops the amount, it types it again, at most three times. - `questions-and-approvals.necessary-question` (v2): the harness pages through a multi-question card. It answers the one required question and clicks Skip on each optional question. The grade now requires exactly one required question, and that question must ask for the product name as free text. Other questions must be optional. An optional necessary question, a second required question, or a required question about something else still fails. - Descriptors are updated. No scenario yaml changed, so the TestOps manifest does not change. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-rendered-links.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-catalog-fingerprint.test.ts tests/runner-e2e/chat-catalog-domains/identity.test.ts`: 100 tests pass. - New tests: a reply saved after the run finished is waited for and then rendered; a run that saves no reply returns `unsaved` within its bound. The necessary-question grade accepts the recorded card (required name plus optional follow-up) and rejects the broken cards above. - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - One paid run on a combined branch with #21008. The result is in a comment on this pull request. ## Risks - A turn whose run saves no reply now waits up to 15 seconds before the case continues. Only the three named cases call the wait. - `settledInput` marks the input node with a `data-e2e-settle-mark` attribute to detect a remount. It does not change product behavior. - #21000 changes the render-wait expiry in the same file. Whichever merges second rebases. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>