mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
42fd790e80
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog runs real chat journeys against real agents and grades the saved product state > - The `steering.approval-delivery` case checks that a person's answer to an agent's question stays separate from editable queued messages and reaches the agent once > - The task composer lists one pending wake at a time. It shows editable ordinary messages before a saved card response, by design since #19331 > - When the chat lead relays the person's answer as its own task comment, the composer lists that relay. The saved answer waits on its own wake and is not listed > - The oracle required the answer and the relay in one listing. The product never produces that shape, so the case timed out every time the lead relayed > - This pull request grades the case on the saved answer and on its delivery, and accepts either valid listing > - The benefit is that the case measures the promise in its expected outcome and still fails on a lost, replayed, or editable answer ## Linked Issues or Issue Description **What happened?** The paid chat eval `steering.approval-delivery` (OpenCode, DeepSeek flash) did the answer, edit, reorder, and discard steps. It then timed out while it waited for the answer to show in the queue. **Steps to reproduce** Run the case. Let the chat lead relay the answer as its own task comment. After the person's two messages are discarded, the queue lists only the relay. The answer is saved as `answered` and keeps its own pending wake. **Expected behavior** The case passes when the answer stays saved and unchanged, cannot be edited through the queue, and reaches exactly one later agent turn. Refs #19331, #20899 ## What Changed - `runtime-approval-queue.ts`: add `approvalResponseListing`. After the discards, accept the answer listed alone and locked, or only the chat lead's relay listed. Reject any person message or an editable answer. - Keep the edit attempt through the queue route for both listings. Require it to be refused and the saved answer to stay unchanged. Check the locked card in the UI when the answer is listed. - Add `classifyApprovalSuccessors`. Grade delivery on saved run state: exactly one later worker turn bound to the answer, with the question ID and Acorn in its native input. Allow at most one other turn that carries only the lead's relay. Reject any other turn. - Accept the Acorn document from the bound turn or the relay turn. Still require the earlier work to be kept, all runs succeeded, the queue empty, and the task done. - Oracle `approval-delivery.v6`. Run bound 5 with a minimum of 4, to allow the one relay turn. - Calibration tests: the retained relay shape passes, the original listed shape passes, and these fail: no bound turn, two bound turns, a person message left behind the relay, an unexpected worker turn, two relay turns, a recorded listing that disagrees with the queue, and a listed answer whose controls were not seen as locked. No YAML changed. The expected outcome already promises retention and once-only delivery, not a queue listing. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts` (46 passed) - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` (clean) - `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check` (no change) - One paid single-case run of this branch. The result is added below in a comment. ## Risks - Low risk. The change is in eval code only. No product behavior changes. - The grader now allows one extra worker turn for the lead's relay. A turn that carries any other input still fails the case. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>