mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
DottaandClaude Opus 5.5
4228bcf89c
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents that use OpenCode models reply in chat through the Runner's OpenCode driver. The driver settles one final response, and Paperclip saves it as the reply comment > - In chat-eval cohort 5 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`), four saved replies were the agent's reasoning, for example "Confirmed down. Now report honestly. ... I'll finish with paperclip_finish." > - The run events show the cause. The model wrote a reasoning part, then wrote the same text again as an ordinary text part, and ended the step without `paperclip_finish`. The driver took that copy as the final answer. This is not the #20999 path: no completion call happened > - A second, related problem is in the #20999/#21025 path. When the model writes an answer, calls `paperclip_finish`, and then writes its own new answer, the driver can save both texts, or a status note together with the answer > - This pull request drops the repeated reasoning, lets the existing empty-response correction ask for the real answer, and saves the post-completion message alone unless it is a short line that refers back > - The benefit is that people never see the agent's planning text or tool names as its reply, and the eval catalog now fails when they do ## Linked Issues or Issue Description Refs #20999 Refs #21025 **What happened?** The saved chat reply held the agent's reasoning, not an answer. Cohort 5 cases: `connection-setup.unreachable-mcp`, `plugins.configure-plugin`, `tasks.website-task`, `teams.install-team`. In each case the reply text is byte-identical to the reasoning part before it, and the run result evidence is `provider:opencode:agent-message` (no `paperclip_finish` call). Also, `recovery.explain-stall` saved its answer twice, and `status-cards.refresh-card` saved a "STATUS: ..." note above its answer. **Expected behavior** The saved reply is the model's answer to the person. It never holds reasoning, completion tool names or report field names. A short closing line such as "Answered above" still keeps the earlier answer. **Steps to reproduce** 1. Use an OpenCode agent with `openrouter/deepseek/deepseek-v4.1-flash-1017`. 2. Ask a chat request that the model finishes without a completion call after a long reasoning step. 3. Read the saved reply with `GET /api/issues/{id}/comments`. It holds the reasoning text. **Paperclip version or commit** master at98c18784c6(cohort 5) and3c058c37e6(base of this PR). ## What Changed - `OpenCodeServerDriver` records the text of each completed reasoning part in the run. A text part that still matches the start of a completed reasoning part does not stream as progress. A completed text part that repeats a reasoning part exactly is not a response candidate, and the driver emits the `opencode_reasoning_echo_discarded` diagnostic. With no answer left, the existing empty-response correction asks the model for a final answer with reasoning disabled. - After the completion call, the driver keeps the prose written between the last work tool and that call only when the later message is a closing reference: a short line with no link of its own that points back to it ("Answered above.", "Commands above are ready"), or one under half that prose's length. A later message that repeats the prose is saved once. Otherwise the later message is the reply. The completion tool description already names the post-call message as the final response. This replaces the `restatesEarlierReply` heuristic from #21025, which is removed. - Tests: a new `reasoning-echo` mode in `empty-response.test.ts` uses the recorded `tasks.website-task` text. Two new driver tests use the recorded `recovery.explain-stall` and `status-cards.refresh-card` texts. The #20999 "Answered above" test and the #21025 restatement test pass without change. After review: the reasoning-echo mode streams the copy with a trailing newline (it must stay held back), a new driver test keeps "The result is 42." when the later message is "Answered above.", and two more check that a STATUS note followed by a short complete answer saves only the answer, and a repeated short answer is saved once. - No eval keyword gate: a draft shared check that matched completion tool names in replies was removed after review, because chat catalog graders must not grade free-form replies by keyword. Replies stay saved for human review; the driver fix is what keeps reasoning out of them. ## Verification - `cd packages/paperclip-runner && npx vitest run src/drivers/opencode`: 17 files, 207 tests pass. - Red/green: with the driver from `origin/master`, the `reasoning-echo` test fails (the saved reply is the reasoning text), and both new post-completion tests fail (two texts saved). With this change, they pass. - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog tests/runner-e2e/chat-reply tests/runner-e2e/source-sdk`: 3903 tests pass. - Check calibration: the new unit tests cover recorded valid replies, reworded valid replies (`paperclip_runner` adapter type, `PAPERCLIP_API_KEY`, `paperclip_finish_task`, plain "summary"/"evidence"), and the four recorded broken replies. Over all 193 cohort-5 conversations the check flags exactly the four broken cases. - Over the 30 cohort-5 turns with prose before and after `paperclip_finish`, the new rule changes 3: `recovery.explain-stall` (duplicate removed), `status-cards.refresh-card` ("STATUS:" note removed) and `assistant-connections.consent-read-only` (shorter restatement kept, same link). The pointer-line turns (`tasks.website-without-project`, `cli-operations.choose-cli-context`) still keep both texts. - `npx tsc -p packages/paperclip-runner/tsconfig.json --noEmit` passes. - A paid chat-eval run on this change is linked in a comment below. ## Risks - A turn that repeats its reasoning as text now takes one more model request (the existing empty-response correction). If the corrected response is empty again, the turn fails with `opencode_empty_response`, as before. - A real answer that is exactly equal to a reasoning part is dropped and requested again. The correction recovers it. - When a long post-completion message drops something the earlier prose said, only the later message is saved. The completion tool contract names that message as the final response, and the earlier prose remains in the run transcript. - Saved run state: no persisted run, session, checkpoint or event shape changes. The reasoning texts are in-memory driver state per run. A run saved by the previous release resumes without change; only a newly settled final response differs. - The new catalog check can fail a case that passed before. In cohort 5 it fails only the four reasoning replies. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>