Commit Graph
2 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 e601f9d2fc fix(runner): answer each OpenCode MCP tool call on its own connection (#20886)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Paperclip runner runs native OpenCode agents. OpenCode calls
Paperclip tools through a local MCP bridge in the OpenCode proxy.
OpenCode runs on Bun and uses Bun `fetch` for each MCP request.
> - In one chat eval, the model made two parallel tool calls. The bridge
produced and wrote both results in about 0.55 s. OpenCode received only
one of them.
> - The other call waited until Bun's default fetch timeout fired, about
five minutes later. The chat turn lost five minutes, and the eval missed
its deadline.
> - The most likely trigger is the reuse of an idle keep-alive socket.
We could not reproduce the loss locally, so this pull request is
hardening that is based on a hypothesis.
> - This pull request makes the bridge answer every request on its own
connection. It also records how each tool result left the bridge, so a
recurrence can be attributed from the run log.
> - The benefit is that a pooled socket can no longer strand a tool
result. If a call is stranded again, the run log shows whether the
bridge wrote the answer.

## Linked Issues or Issue Description

No public issue exists. Related: Refs #18751 (draft native client
acceptance diagnostic), Refs #19546 (abandoned queued calls), Refs
#20862 (transport event delivery; it does not touch this path).

**What happened**

In CI run `gha-42925998380` (chat-features, runner-opencode,
`cases.create-case`), the CEO chat run used OpenCode 1.18.34 with an
OpenRouter DeepSeek model. At 00:01:12.16 the model issued
`list_workflows` (MCP id 3) and `list_projects` (MCP id 4) in parallel.

The runner emitted the dynamic tool results for id 4 at 00:01:12.468 and
for id 3 at 00:01:12.714 (ok, HTTP 200). The bridge writes the HTTP
response directly after that event. There is no queue between them.

OpenCode completed `list_projects` at 00:01:13.113. It failed
`list_workflows` only at 00:06:01.761 with `The operation timed out.`

We read the bundled OpenCode source. Its MCP tool call has no timeout
that produces that message: the MCP request timeout gives `-32001
Request timed out`, and the configured value is 1 h for durable calls.
With the same binary in Bun mode (Bun 1.3.14), a fetch on a reused
keep-alive socket that never gets an answer fails after 298 s with
`TimeoutError: The operation timed out.` So Bun's default fetch timeout
ended the call, and the client never consumed the written answer.

The previous call (`list_agents`) answered at about 00:01:06.89, so its
socket was idle for about 5.27 s when the parallel pair started. Our
thesis is that call 3 reused that pooled socket, and call 4 opened a new
one.

**Expected behavior**

When the bridge has the result of a tool call, the MCP client gets it.
The client must not wait for a five-minute transport timeout.

**Steps to reproduce**

We did not reproduce it locally. This is the only occurrence in the
retained evidence. A macOS stress test (Bun 1.3.14 client, Node 26
server with the bridge's default keep-alive, one call, then an idle time
of 4.8–7.2 s, then two parallel calls, 8 pairs in parallel) completed
343 trials and about 1,029 requests with 0 lost results. The failing run
was on Linux CI with Node 24.

## What Changed

- `mcp-bridge.ts`: every bridge response sends `Connection: close`. The
MCP client cannot reuse a pooled keep-alive socket for a later request.
A loopback connect per request costs much less than the failure mode.
- `mcp-bridge.ts`: each `tools/call` records one closed delivery
diagnostic. This includes rejected calls (unsupported tool, invalid
input, duplicate identity), which record the outcome `error`
(`paperclip.opencode.tool-delivery-diagnostic/v1`) when its response
closes. The fields are the call id SHA-256, the socket request index,
the socket idle ms, the handler ms, the outcome (`result`, `error` or
`unanswered`) and `written` (`writableFinished`). It holds no tool
names, inputs or results. The limit is 256 records per bridge.
- `opencode-proxy-finish-diagnostics.ts`: the proxy forwards these
records to the run log as `warning` frames after session binding. It
validates the exact key set and value types and drops anything else.
- Tests: a new `mcp-bridge-delivery.test.ts` uses a pooling keep-alive
client with the incident's call shape (one call, then two parallel
calls). New forwarder cases cover the record validation. The existing
finish diagnostic assertion now filters by its own schema.

## Verification

- Red: with the `Connection: close` line removed,
`mcp-bridge-delivery.test.ts` fails with `expected 'keep-alive' to be
'close'`. The pooling client then reuses the socket.
- Green: `cd packages/paperclip-runner && pnpm exec vitest run
src/drivers/opencode/mcp-bridge-delivery.test.ts
src/drivers/opencode/mcp-bridge-diagnostics.test.ts
src/drivers/opencode/mcp-bridge.test.ts
src/cli/opencode-proxy-finish-diagnostics.test.ts
src/cli/opencode-app-server-proxy.test.ts
src/cli/opencode-proxy-abandoned-tool-call.test.ts` gives 6 files and 45
tests passed.
- The new test asserts that all three results arrive with the correct
JSON-RPC ids, that no socket is reused, and that each call records
`socketRequestIndex: 1`, `outcome: "result"`, `written: true` without
tool names or raw ids. A second test asserts that a client that leaves
before its answer records `outcome: "unanswered"`, `written: false`. A
third test asserts that a rejected call records `outcome: "error"`,
`written: true`.
- `tsc -p tsconfig.json --noEmit` for `packages/paperclip-runner`
reports no errors in the changed files. Seven existing errors in
`codex-runtime-adapter.ts` and `eval/workflow-harness.ts` come from this
worktree's linked dependencies. Required CI typecheck is the source of
truth.
- This does not prove the production fix. A recurrence on CI will show
socket reuse, idle time and `written` for the stranded call in the run
log.

## Risks

- Low risk. Each MCP request now opens a new loopback TCP connection.
That adds well under 1 ms per call, and there is no TLS.
- The `notifications/cancelled` handling and the abandoned-call abort do
not change. The abort still reads `writableFinished` on response close.
- One extra run-log warning for each tool call (at most 256 for each
bridge). The record holds only a hash, counters and timings.
- Previous-release story (upgrade-safe run state): no persisted run,
session, launch or pack state changes. A bridge from the previous
release keeps its behavior until the proxy restarts.
- If the thesis is wrong, the five-minute stall can recur. The new
records will then show `written: true` on a fresh socket and point the
next investigation at OpenCode or Bun.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:29:59 -06:00
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00