mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
test(opencode): pin Stop during a pending MCP tool call (#19550)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner runs OpenCode agents and settles their cost after each run > - An operator can stop a run while the agent waits inside a Paperclip MCP tool call > - Before #18610, OpenCode then kept calling the model after the Stop. The turn never closed, so the run cost never settled > - #18610 fixed the cause, but no test covers a Stop during a pending MCP tool call > - This pull request adds that regression test at the driver boundary > - The benefit is that a change to empty-answer recovery or interruption cannot bring back runaway model work after a Stop ## Linked Issues or Issue Description Refs #18610 (the fix that this test pins) Refs #19497 (accounting that waits on a terminal run receipt) **What happened?** A board operator stopped a native OpenCode run. The agent was blocked inside a gated `paperclip_call_assigned_tool` call. The runner acknowledged the interrupt, but OpenCode made 8 more model requests in the next 60 seconds. No `turn.cancelled` event arrived. The run stayed at `costAccountingPending: true` and `accountingReceiptReady: false`, so cost accounting never settled. **Expected behavior** A Stop aborts the provider. The turn closes with `turn.cancelled`. The provider makes no new model requests. Cost accounting then settles from the final receipt. **Steps to reproduce** 1. Start an OpenCode turn with a dynamic Paperclip tool whose handler does not return. 2. Let the provider call that tool through the MCP bridge. 3. Fail the pending call (runnerd does this first on Stop), then interrupt the turn. 4. Let OpenCode report the abort as a bare `session.idle`. 5. Before #18610, the driver sent a corrective "give a final answer" prompt. The provider then continued work. **Root cause** The driver sent `/session/:id/abort` correctly. OpenCode 1.18.34 aborts correctly, and sometimes reports the abort only as `session.idle`. Empty-answer recovery treated that idle as an incomplete answer and sent a new prompt. #18610 added `#interruptedTurnId`, so recovery now stops for an interrupted turn. The failing eval ran on `3b66c3c144`, which is earlier than #18610. ## What Changed - Add `packages/paperclip-runner/src/drivers/opencode/interrupt-pending-tool-call.test.ts`. It calls the real MCP bridge over HTTP, holds the dynamic tool call open, and interrupts the turn. It checks three abort reports: - `idle-after-settle`: runnerd fails the call, then the interrupt, then a bare idle. - `idle-before-settle`: an idle while the call is still pending, then the late failure, then an idle. - `aborted-error`: `MessageAbortedError`. - Each case asserts one prompt only, exactly one terminal event (`turn.cancelled`), and no empty-response correction diagnostic. - No product code changes. Master already has the fix. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/opencode/interrupt-pending-tool-call.test.ts`: 3 passed on this branch. - The same test at `3b66c3c144` (before #18610): `idle-after-settle` and `idle-before-settle` fail with 2 prompts (the corrective prompt). `aborted-error` passes. This matches the eval symptom. - `vitest run src/drivers/opencode/`: 14 files, 185 tests passed. - `tsc -p tsconfig.test.json --noEmit` for the runner: no new errors. A pre-existing error in `src/eval/workflow-harness.ts` comes from an unbuilt local eval-kernel package. - `server` `native-stop-accounting.integration.test.ts`: 5 passed. This confirms that settlement completes after a Stop when the terminal event and final receipt arrive. - Manual, not committed: the same flow with the real OpenCode 1.18.34 binary, a mock OpenAI-compatible model, and a gated tool. I tested it through the driver alone and through runnerd, the proxy, and the driver. The interrupt produced `turn.cancelled`, and the model received no more requests. I also tried different gate-release and model-latency timings. - The new test waits for the terminal event with a bounded `expect.poll`, not a fixed delay (review feedback). - `pnpm verify` exited 2 (broader verification required) because the new test file is outside the resolved test graph. CI supplies the broad coverage. ## Risks Low risk. This change adds one test file only. The test uses the existing fake OpenCode fixture and an injected `fetch`. It uses real timers with short waits. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, code execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
c650f36007
commit
86e9c9354d
1 file changed
+130
@@ -0,0 +1,130 @@
|
||||
import { chmod, copyFile, mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { expect, it } from "vitest";
|
||||
import type { PrpEvent } from "../../protocol/replay-contract.js";
|
||||
import { OpenCodeServerDriver } from "./opencode-server-driver.js";
|
||||
|
||||
// Operator Stop while the provider waits inside a Paperclip MCP tool call.
|
||||
// runnerd settles the pending call before it asks the driver to interrupt, and
|
||||
// OpenCode 1.18 can then report the abort as a bare idle. That idle closes the
|
||||
// stopped turn. It must not count as an empty answer that earns a corrective
|
||||
// prompt, because that prompt restarts model work after the run is cancelled.
|
||||
it.each(["idle-after-settle", "idle-before-settle", "aborted-error"] as const)(
|
||||
"closes an OpenCode turn interrupted during a pending MCP tool call without new model work: %s",
|
||||
async (abortReport) => {
|
||||
const root = await mkdtemp(join(tmpdir(), "opencode-interrupt-pending-tool-"));
|
||||
const command = join(root, "fake-opencode.mjs");
|
||||
await copyFile(resolve("test/fixtures/fake-opencode-server.mjs"), command);
|
||||
await chmod(command, 0o755);
|
||||
let stream: ReadableStreamDefaultController<Uint8Array> | undefined;
|
||||
let streamOpened!: () => void;
|
||||
const opened = new Promise<void>((done) => { streamOpened = done; });
|
||||
const prompts: unknown[] = [];
|
||||
const aborts: string[] = [];
|
||||
const observed: PrpEvent[] = [];
|
||||
let ownedSessionId = "";
|
||||
const emit = (value: unknown) =>
|
||||
stream!.enqueue(new TextEncoder().encode(`data: ${JSON.stringify(value)}\n\n`));
|
||||
let releaseTool!: (outcome: Error) => void;
|
||||
const toolReleased = new Promise<Error>((done) => { releaseTool = done; });
|
||||
let toolStarted!: () => void;
|
||||
const toolCalled = new Promise<void>((done) => { toolStarted = done; });
|
||||
const driver = new OpenCodeServerDriver({
|
||||
model: "openrouter/controlled-model",
|
||||
runtimeDirectory: root,
|
||||
command,
|
||||
environment: { PATH: process.env.PATH, OPENROUTER_API_KEY: "controlled-fixture-key" },
|
||||
dynamicTools: [{ name: "paperclip_call_assigned_tool", inputSchema: { type: "object" } }],
|
||||
dynamicToolHandler: async () => {
|
||||
toolStarted();
|
||||
// The controller's gated assigned tool never answers this call: runnerd
|
||||
// fails it with its stop courtesy result once the operator stops the run.
|
||||
throw await toolReleased;
|
||||
},
|
||||
fetch: async (input, init) => {
|
||||
const url = new URL(String(input));
|
||||
if (url.pathname === "/event")
|
||||
return new Response(new ReadableStream<Uint8Array>({
|
||||
start(controller) { stream = controller; streamOpened(); },
|
||||
}), { headers: { "Content-Type": "text/event-stream" } });
|
||||
const sessionID = decodeURIComponent(url.pathname.split("/")[2] ?? "");
|
||||
if (url.pathname.endsWith("/prompt_async")) {
|
||||
ownedSessionId = sessionID;
|
||||
prompts.push(JSON.parse(String(init?.body)));
|
||||
return new Response(null, { status: 204 });
|
||||
}
|
||||
if (url.pathname.endsWith("/abort")) {
|
||||
aborts.push(sessionID);
|
||||
if (abortReport === "aborted-error")
|
||||
queueMicrotask(() => emit({ type: "session.error", properties: { sessionID,
|
||||
error: { name: "MessageAbortedError", data: { message: "Aborted" } } } }));
|
||||
if (abortReport === "idle-before-settle")
|
||||
queueMicrotask(() => emit({ type: "session.idle", properties: { sessionID } }));
|
||||
return Response.json(true);
|
||||
}
|
||||
return fetch(input, init);
|
||||
},
|
||||
});
|
||||
const session = await driver.openSession({
|
||||
runId: "owned-run", normalizedSessionId: "owned-session", workingDirectory: root,
|
||||
});
|
||||
let collecting: Promise<void> | undefined;
|
||||
const settle = () => new Promise((done) => setTimeout(done, 50));
|
||||
try {
|
||||
await opened;
|
||||
collecting = (async () => { for await (const event of session.events()) observed.push(event); })();
|
||||
const turn = await session.startTurn({ message: { role: "user", text: "Read the website brief." } });
|
||||
expect(prompts).toHaveLength(1);
|
||||
|
||||
// OpenCode calls the Paperclip tool through the authenticated MCP bridge.
|
||||
const config = JSON.parse(await readFile(join(root, "owned-session", "config", "opencode", "opencode.json"), "utf8"));
|
||||
const bridge = config.mcp.paperclip as { url: string; headers: Record<string, string> };
|
||||
emit({ type: "message.updated", properties: { info: { id: "tool-message", sessionID: ownedSessionId, role: "assistant" } } });
|
||||
const toolResponse = fetch(bridge.url, {
|
||||
method: "POST",
|
||||
headers: { ...bridge.headers, "Content-Type": "application/json", Accept: "application/json, text/event-stream" },
|
||||
body: JSON.stringify({ jsonrpc: "2.0", id: "assigned-call", method: "tools/call",
|
||||
params: { name: "paperclip_call_assigned_tool", arguments: { name: "read_website_brief", arguments: {} } } }),
|
||||
});
|
||||
await toolCalled;
|
||||
|
||||
const terminalEvents = () => observed.filter((event) =>
|
||||
["turn.completed", "turn.cancelled", "turn.failed"].includes(event.eventType));
|
||||
if (abortReport === "idle-after-settle") {
|
||||
// runnerd answers the pending call before it interrupts the turn.
|
||||
releaseTool(new Error("Paperclip stopped the active provider turn"));
|
||||
await toolResponse;
|
||||
}
|
||||
await session.interrupt?.({ turnId: turn.turnId, reason: "Cancelled by a board operator" });
|
||||
expect(aborts).toEqual([ownedSessionId]);
|
||||
await settle();
|
||||
if (abortReport !== "idle-after-settle") {
|
||||
// The stopped provider abandons its tool call; the late courtesy result
|
||||
// must not reopen the turn either.
|
||||
releaseTool(new Error("Paperclip stopped the active provider turn"));
|
||||
await toolResponse;
|
||||
await settle();
|
||||
}
|
||||
if (abortReport !== "aborted-error")
|
||||
emit({ type: "session.idle", properties: { sessionID: ownedSessionId } });
|
||||
// Wait for the terminal frame with a bound rather than a fixed delay, then
|
||||
// give a corrective prompt (the regression) time to appear.
|
||||
await expect.poll(terminalEvents, { timeout: 5_000 }).toHaveLength(1);
|
||||
await settle();
|
||||
|
||||
expect(prompts).toHaveLength(1);
|
||||
expect(terminalEvents().map((event) => [event.eventType, event.turnId])).toEqual([
|
||||
["turn.cancelled", turn.turnId],
|
||||
]);
|
||||
expect(observed.some((event) => event.eventType === "harness.diagnostic"
|
||||
&& event.payload.code === "opencode_empty_response_correction")).toBe(false);
|
||||
} finally {
|
||||
releaseTool(new Error("test cleanup"));
|
||||
await session.close({ reason: "controlled test complete" });
|
||||
await collecting;
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
15_000,
|
||||
);
|
||||
Reference in new issue
Block a user