Commit Graph
4 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 3fc2198759 fix(runner): treat tool-call markup in OpenCode text as an unexecuted call (#21361)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner drives OpenCode and turns its event stream into
the run's progress, reply, and completion result
> - Some model hosts return a tool call as raw tool-call syntax inside
the assistant text, not as a structured tool call
> - The OpenCode driver streamed that syntax as progress and accepted it
as the final answer, so the run succeeded with markup as its reply and
the intended tool call never ran
> - This pull request treats leaked tool-call markup as an unexecuted,
malformed tool call
> - The benefit is that the person never sees raw call syntax as a
reply, and the agent gets one bounded correction to make the call
properly

## Linked Issues or Issue Description

**What happened?**
An agent run that uses an OpenRouter model can end with raw tool-call
markup as its reply. The model meant to call the completion tool. The
host put the call into the message text as DeepSeek DSML markup (for
example `<|DSML| invoke name="...">`). The runner accepted that text as
the final answer and settled the run as succeeded. The completion call
did not run, so follow-up automation repeated the work.

**Steps to reproduce**
1. Run an OpenCode agent with a DeepSeek model through an OpenRouter
host that leaks tool-call syntax into text.
2. Let the model end its turn with a completion call that the host
returns as text.
3. Look at the run reply.

**Expected behavior**
The runner does not show the markup and does not accept it as the reply.
It tells the model that the tool did not run and asks it once to call
the tool through the tool interface.

**Paperclip version or commit**
`master` at the time of this pull request.

**Additional context**
Today the markup streams as progress and becomes the run's final reply.
The run succeeds without its completion call.

## What Changed

- Added `tool-call-markup.ts` in the OpenCode driver. It detects leaked
tool-call markup in one linear scan: DSML and DeepSeek call-token
sentinels anywhere, and `<function_calls>` / `<tool_call>` wrappers only
at a line start with a call body. It ignores fenced code blocks.
- The driver holds markup (and a trailing fragment that can still become
markup) out of streamed progress text.
- A completed text part that contains markup is not a final-answer
candidate. The driver emits the `opencode_tool_call_markup_discarded`
diagnostic and records the position, so earlier text cannot become the
reply either.
- The turn then goes through the existing one-shot empty-response
correction. The correction text says the tool did not run and asks the
model to call tools only through the tool interface.
- Regression tests: detection cases (recorded DSML shape, other
families, prose and code-fence negatives, streaming monotonicity) and a
driver case that replays the recorded markup.

## Verification

- `cd packages/paperclip-runner && npx vitest run src/drivers/opencode/`
passes (20 files, 263 tests).
- Red/green: the new `tool-call-markup` case in `empty-response.test.ts`
fails on the unchanged driver (the markup is selected as the final
reply) and passes with this change.
- `npx tsc --noEmit -p packages/paperclip-runner` passes.
- Cost: the scan adds about 22 µs for each streamed update of a 20 KB
answer with `<` on every line (43 ms over 1,960 updates), and 0.05 ms
for each completed part. It adds no waits and does not touch boot or the
start of a turn.

## Risks

- Low risk. A real answer that puts DSML sentinels or a
`<function_calls>`/`<tool_call>` wrapper with a call body outside a code
fence is now treated as an unexecuted call. The correction then asks
once for a proper answer.
- If the model leaks markup again after the correction, the existing
empty-response failure path applies.
- No schema, API, or saved run-state change.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:00:29 -06:00
DottaandClaude Opus 5.5 4228bcf89c fix(opencode): never save the model's reasoning as its chat reply (#21051)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode models reply in chat through the Runner's
OpenCode driver. The driver settles one final response, and Paperclip
saves it as the reply comment
> - In chat-eval cohort 5 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`), four saved replies were
the agent's reasoning, for example "Confirmed down. Now report honestly.
... I'll finish with paperclip_finish."
> - The run events show the cause. The model wrote a reasoning part,
then wrote the same text again as an ordinary text part, and ended the
step without `paperclip_finish`. The driver took that copy as the final
answer. This is not the #20999 path: no completion call happened
> - A second, related problem is in the #20999/#21025 path. When the
model writes an answer, calls `paperclip_finish`, and then writes its
own new answer, the driver can save both texts, or a status note
together with the answer
> - This pull request drops the repeated reasoning, lets the existing
empty-response correction ask for the real answer, and saves the
post-completion message alone unless it is a short line that refers back
> - The benefit is that people never see the agent's planning text or
tool names as its reply, and the eval catalog now fails when they do

## Linked Issues or Issue Description

Refs #20999
Refs #21025

**What happened?**
The saved chat reply held the agent's reasoning, not an answer. Cohort 5
cases: `connection-setup.unreachable-mcp`, `plugins.configure-plugin`,
`tasks.website-task`, `teams.install-team`. In each case the reply text
is byte-identical to the reasoning part before it, and the run result
evidence is `provider:opencode:agent-message` (no `paperclip_finish`
call). Also, `recovery.explain-stall` saved its answer twice, and
`status-cards.refresh-card` saved a "STATUS: ..." note above its answer.

**Expected behavior**
The saved reply is the model's answer to the person. It never holds
reasoning, completion tool names or report field names. A short closing
line such as "Answered above" still keeps the earlier answer.

**Steps to reproduce**
1. Use an OpenCode agent with
`openrouter/deepseek/deepseek-v4.1-flash-1017`.
2. Ask a chat request that the model finishes without a completion call
after a long reasoning step.
3. Read the saved reply with `GET /api/issues/{id}/comments`. It holds
the reasoning text.

**Paperclip version or commit**
master at 98c18784c6 (cohort 5) and 3c058c37e6 (base of this PR).

## What Changed

- `OpenCodeServerDriver` records the text of each completed reasoning
part in the run. A text part that still matches the start of a completed
reasoning part does not stream as progress. A completed text part that
repeats a reasoning part exactly is not a response candidate, and the
driver emits the `opencode_reasoning_echo_discarded` diagnostic. With no
answer left, the existing empty-response correction asks the model for a
final answer with reasoning disabled.
- After the completion call, the driver keeps the prose written between
the last work tool and that call only when the later message is a
closing reference: a short line with no link of its own that points back
to it ("Answered above.", "Commands above are ready"), or one under half
that prose's length. A later message that repeats the prose is saved
once. Otherwise the later message is the reply. The completion tool
description already names the post-call message as the final response.
This replaces the `restatesEarlierReply` heuristic from #21025, which is
removed.
- Tests: a new `reasoning-echo` mode in `empty-response.test.ts` uses
the recorded `tasks.website-task` text. Two new driver tests use the
recorded `recovery.explain-stall` and `status-cards.refresh-card` texts.
The #20999 "Answered above" test and the #21025 restatement test pass
without change. After review: the reasoning-echo mode streams the copy
with a trailing newline (it must stay held back), a new driver test
keeps "The result is 42." when the later message is "Answered above.",
and two more check that a STATUS note followed by a short complete
answer saves only the answer, and a repeated short answer is saved once.
- No eval keyword gate: a draft shared check that matched completion
tool names in replies was removed after review, because chat catalog
graders must not grade free-form replies by keyword. Replies stay saved
for human review; the driver fix is what keeps reasoning out of them.

## Verification

- `cd packages/paperclip-runner && npx vitest run src/drivers/opencode`:
17 files, 207 tests pass.
- Red/green: with the driver from `origin/master`, the `reasoning-echo`
test fails (the saved reply is the reasoning text), and both new
post-completion tests fail (two texts saved). With this change, they
pass.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog tests/runner-e2e/chat-reply
tests/runner-e2e/source-sdk`: 3903 tests pass.
- Check calibration: the new unit tests cover recorded valid replies,
reworded valid replies (`paperclip_runner` adapter type,
`PAPERCLIP_API_KEY`, `paperclip_finish_task`, plain
"summary"/"evidence"), and the four recorded broken replies. Over all
193 cohort-5 conversations the check flags exactly the four broken
cases.
- Over the 30 cohort-5 turns with prose before and after
`paperclip_finish`, the new rule changes 3: `recovery.explain-stall`
(duplicate removed), `status-cards.refresh-card` ("STATUS:" note
removed) and `assistant-connections.consent-read-only` (shorter
restatement kept, same link). The pointer-line turns
(`tasks.website-without-project`, `cli-operations.choose-cli-context`)
still keep both texts.
- `npx tsc -p packages/paperclip-runner/tsconfig.json --noEmit` passes.
- A paid chat-eval run on this change is linked in a comment below.

## Risks

- A turn that repeats its reasoning as text now takes one more model
request (the existing empty-response correction). If the corrected
response is empty again, the turn fails with `opencode_empty_response`,
as before.
- A real answer that is exactly equal to a reasoning part is dropped and
requested again. The correction recovers it.
- When a long post-completion message drops something the earlier prose
said, only the later message is saved. The completion tool contract
names that message as the final response, and the earlier prose remains
in the run transcript.
- Saved run state: no persisted run, session, checkpoint or event shape
changes. The reasoning texts are in-memory driver state per run. A run
saved by the previous release resumes without change; only a newly
settled final response differs.
- The new catalog check can fail a case that passed before. In cohort 5
it fails only the four reasoning replies.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:06:40 -06:00
DottaandClaude Opus 5.5 951d1b0b1b fix(opencode): ask for the empty-response correction with reasoning off (#20890)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner drives OpenCode turns for agents that use OpenCode
models, such as DeepSeek through OpenRouter
> - Sometimes the model ends a turn inside its reasoning. It reasons
that it will reply and call `paperclip_finish`, and then it stops with
no answer and no tool call
> - The driver sends one correction prompt. The correction used the same
reasoning variant, and in the failed runs the model stopped inside its
reasoning again
> - The run then failed with `opencode_empty_response`, and the person
saw "The OpenCode run failed before it completed."
> - This pull request sends the correction prompt with the no-reasoning
variant, so the final answer comes in the answer channel
> - The benefit is fewer failed chat replies, with no extra prompts and
no change to the normal turn

## Linked Issues or Issue Description

**What happened**
A chat reply failed with `opencode_empty_response` in eval run
gha-43069861174 (`agent-chat.conference-room`, OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`). The first completion
and the one correction each held only a reasoning item (403 and 411
output tokens). Both reasoning items said the model would reply and call
`paperclip_finish`. No answer text and no tool call followed. In
retained eval evidence, 14 of 879 attempts needed this correction and 3
of those 14 failed. Each failed correction was another reasoning-only
completion.

**Expected behavior**
After an empty completion, the correction gets a final answer or a
completion report.

**Steps to reproduce**
1. Start an OpenCode turn with reasoning enabled on an OpenRouter model.
2. Let the provider return a completion that holds only reasoning.
3. See that the correction prompt uses `paperclip-default` (reasoning
on), so the model can end inside its reasoning again.

## What Changed

- `OpenCodeServerDriver` sends the empty-response correction with the
`paperclip-no-reasoning` variant. Normal turns keep their own reasoning
mode.
- `empty-response.test.ts` starts the `reply` and `empty-again` cases
with reasoning on. It checks that the first prompt uses
`paperclip-default` and the correction uses `paperclip-no-reasoning`.

## Verification

- `pnpm exec vitest run src/drivers/opencode/empty-response.test.ts
src/drivers/opencode/reasoning-mode.test.ts
src/drivers/opencode/interrupt-pending-tool-call.test.ts` (in
`packages/paperclip-runner`): 23 passed. Without the driver change, the
`reply` and `empty-again` cases fail (red/green).
- `node --test
tests/runner-e2e/native-completion-source-contract.test.mjs`: 29 passed.
- A paid eval run with this change and the related eval fixes is in
progress. The PR comments will have its result. One run cannot prove the
rate change, because the failure is rare.

## Risks

- Low risk. Only the correction prompt changes. A model with no
reasoning can give a shorter final answer, but the correction asks only
for the final answer and the completion report from work already done.
- Providers other than OpenRouter do not get a variant. Their behavior
does not change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), extended thinking, tool
use and code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:31:38 -06:00
DottaandPaperclip 66e8fab944 fix: complete native chat responses and improve API discovery (#18364)
## Thinking Path

> - Paperclip must perform work and report its outcome accurately.
> - A chat response and its delegated or operator-owned work are
separate stages.
> - Native chat contracts instead required literal completion of the
user's whole request.
> - Agents kept searching or waiting even when a truthful handoff should
finish the response.
> - OpenCode also has separate acknowledgement backlog and
empty-response failure paths.
> - This PR aligns chat completion with its current stage and makes
OpenCode completion truthful.

## Linked Issues or Issue Description

Refs #18098 for retained OpenCode evidence and #18377 for separate eval
setup, oracle and reporting repairs.

**What happened?** Real pinned OpenCode 1.18.34 reproduced an MCP
timeout behind 800 text fragments. Pilot
[41222196525](https://github.com/paperclipai/paperclip/actions/runs/41222196525)
separately exposed a fabricated successful reply after a reasoning-only
response. Later restart attempts timed out without an answer. Their
native criterion literally required restarting the instance, although
the chat instructions and authority required an operator handoff.
General authority guidance alone did not resolve it.

**Expected behavior:** A standing chat completes its current authorized
response. It performs and verifies permitted actions, verifies delegated
handoffs, or explains the specific unavailable action and operator path.
The worker's task remains separate. Ordinary execution tasks must still
complete their requested work. Empty provider responses must not become
invented success.

**Version and reproduction:** Actual pinned OpenCode and native Rust
Runner, with retained full-stack AWS attempts. Product baseline master
`51ea624344`. This PR adds no historical catalog to master.

## What Changed

- Build standing-chat completion contracts from the server-owned
conversation identity. Scope each current request to the response stage;
preserve request text, criterion IDs, order, human-response scope and
source revisions. Require verified authorized actions and truthful
handoffs. Ordinary task contracts stay strict. Existing admitted inputs
retain their saved contract during recovery; new contract content uses
the existing revision/hash mechanism.
- Guide prospective defaults to their owning resource and preserve
existing work unless backfill is requested. Guide unavailable
board/operator operations to the supported handoff. These are general
chat instructions with no eval identities or expected answers.
- Batch adjacent assistant text for at most 50 ms and 4,000 UTF-8 bytes.
Preserve text, metadata and order; flush before non-text frames and
shutdown.
- Give an empty OpenCode response one correction within its existing
turn, model, reasoning, context and deadline. Accept actual completed
answer parts with incomplete headers. Preserve actual active tool work,
bound idle evidence grace and fail truthfully if correction stays empty.
- Reject empty OpenCode terminals in Rust without actual prose or an
accepted semantic result. Never invent successful replies or evidence.
Refresh generated project-discovery contracts.
- Preserve accurate public API-discovery descriptions for existing
project defaults and managed instance restart operations, including
their field scope and authority limits.

## Verification

**Qualified: [pilot
41256592937](https://github.com/paperclipai/paperclip/actions/runs/41256592937)
passes 3/3, with all 35 independent checks passing.**

| Case | Checks | Full case duration | Evidence and cleanup |
| --- | --- | --- | --- |
| Instance restart | 9/9 | 140.1 s | complete |
| Project defaults | 11/11 | 204.2 s | complete |
| Project work | 15/15 | 204.0 s | complete |

All three are original-attempt results from source
`c51e079f7f2364305b131ffd549e263ff78c5335`, containing the exact product
files from `13c2b61f31d873a475ea0c3c026b19926ff42301`. Actual saved
conversation contracts were inspected, as were ordinary native worker
contracts where present. Each protected archive includes the actual API
state and final PNG; cleanup completes. Original queries,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, finite case clocks,
concurrency one and zero eval retries are preserved. Earlier passes do
not transfer into this qualification.

Both current heads pass all checks: product **62** (56 success, one
neutral, five skipped), catalog **56** (54 success, one neutral, one
skipped). Both have exact-head Greptile **5/5**, zero unresolved review
threads and clean mergeability. The three trusted private-evidence
protocol files remain byte-identical to the dispatch workflow. Raw
evidence stays protected.

- **2,044 API discovery tests pass.** Red controls showed that “restart
instance” selected unrelated drain/settings operations and “save project
defaults” selected repository replacement. Accurate public summaries now
rank the existing operations correctly; route registration preserves
descriptions. No endpoint, permission or field behavior changes.
- **112 real-database, authenticated HTTP, native Runner and OpenAPI
consumer tests pass locally**, using disposable task-owned Docker
PostgreSQL. It was removed afterward; shared IPC was untouched. These
cover persisted contract selection, admitted native input and the model
envelope together. The isolated CI suites also pass.
- **77 focused contract/input/context tests pass.** Red controls catch
the wrong conversation predicate; canonical identity requires both agent
and user IDs, including the actual heartbeat projection. Ordinary tasks,
agent-only issues and planner tasks remain strict. Coalesced requests
preserve criterion identities, source hashes and human-response scope.
- **144 OpenCode driver tests pass**; native core **441 pass, one
ignored**; packaging **56 pass, six skipped**. Server typecheck and
build pass.
- Real pinned OpenCode/native Runner: a delayed 25-second
acknowledgement completes within the original 30-second MCP deadline.
The text-fragment control changes 817 notifications / 54.1 seconds /
timeout to 45 / 19.0 seconds / accepted finish. Separate fast,
six-minute and real authenticated API probes complete 32 tool calls plus
finish. These do not reproduce every remaining historical MCP timeout,
and no common cause is claimed for all such failures.

Retained failures remain attributed to their original source. Earlier
runs exposed fabricated empty-response success, prospective-default
backfill, a literal whole-task chat criterion, fixture readiness and
oracle/reporting defects. The immediately preceding
[41250502608](https://github.com/paperclipai/paperclip/actions/runs/41250502608),
product `c0bf249807` / source `7b1de24ce1`, is **1/3**: work 15/15;
restart 4/9 with the repaired contract actually admitted but no answer;
defaults timed out before settlement, then exceeded the unchanged 256
MiB original-evidence allowance. Defaults domain behavior on that
attempt remains inconclusive. #18377 retains the full attempt ledger,
including missing archives and pre-paid cancellations. The fresh 3/3
establishes these selected outcomes; it does not establish full-catalog
reliability.

## Risks

The shared contract change applies to standing native chats across
providers. It changes model instructions and completion scope, while
permission, approval and review gates remain enforced by their existing
owners. Incorrect model claims remain possible; the product evals
independently require actual mutations, native work and delivery. The
correction adds at most one provider request within the existing
deadline; text batching can delay adjacent text by 50 ms. Three
qualified attempts do not establish full-catalog reliability.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, tool use, code
execution and live eval tooling. Exact hosted model revision and
context-window size are not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

- [x] Final same-source three-cell pilot passes 3/3 with required
evidence and cleanup

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-31 19:52:42 -05:00