From 9fbfd93c9b87a4bd54b2e7add883a61c76360346 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 06:15:42 -0600 Subject: [PATCH] fix(runner): record an OpenCode call's serving host when the call ends (#21358) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - OpenCode runs that use OpenRouter record which host served each model call in the run log > - Operators use this host to compare quality and failures between OpenRouter hosts > - About one in three of the first model calls in a turn had no recorded host, while later calls had a host about 93% of the time > - The provider proxy cleared the recorded host when each request started, and this clear raced the usage report of the step before > - This pull request records a call's host when the call ends, not when the next call starts > - The benefit is that each step's usage names the host that served it, so host comparisons are complete ## Linked Issues or Issue Description **What happened?** In an evaluation cohort of OpenCode runs through OpenRouter, 128 of 356 first usage events in a run (36%) had no `usage.upstreamProvider`. For later usage events the rate was 168 of 2,362 (7%). Root cause: - OpenCode 1.18.34 sends a step's final token count only 14 to 20 ms before it sends the next model request. We measured this three times with the real OpenCode binary and a local fake provider. - The provider proxy cleared the recorded host when each request started. The driver reads OpenCode's event stream. When the next request started before the driver handled the token event, the clear won, and the step's usage had no host. - If the next call's first chunk had arrived first, the step's usage would show the next call's host. - The first call of a turn lost the race most often, because the driver is busiest at turn start. **Expected behavior** Each step's usage names the host that served that step's model call. A call whose host the proxy cannot read carries no host. A call never inherits another call's host. **Steps to reproduce** 1. Start an OpenCode session through the OpenRouter provider proxy. 2. Complete one model call that names a host. 3. Start a second model call and keep its response open. 4. Let OpenCode report the first step's usage. 5. See that the usage has no `upstreamProvider`. **Paperclip version or commit** origin/master ff971149c2. **Deployment mode** Any deployment that runs OpenCode with an OpenRouter model. **Agent adapter(s) involved** OpenCode (Paperclip Runner OpenCode server driver). ## What Changed - `opencode-server-driver.ts`: the provider proxy keeps the host it reads for each call and records it when that call's response ends. It no longer clears the recorded host when a request starts. A call that ends without a readable host (skipped, non-2xx, redirect, or failed) records no host. - `opencode-server-driver.test.ts`: a new red/green test completes one call. Then it starts a second call that has already sent its first chunk with another host, and keeps it open. The test checks that the turn's usage names the first call's host. Then it ends the second call and checks that the next turn names the second call's host. - `doc/run-log-events.md`: the "OpenRouter serving host" section now says the host is recorded when a call ends, and why. ## Verification - `cd packages/paperclip-runner && npx vitest run src/drivers/opencode/opencode-server-driver.test.ts src/drivers/opencode/upstream-served.test.ts` (91 and 8 passed). - Red/green: without the source change, the new test "keeps a finished call's host on its usage while OpenCode's next model call is already in flight" fails. The first step's usage names the second call's host. With the second call's first chunk held back, the old code instead gave a usage with no `upstreamProvider`, which is the symptom we saw in the cohort. With the change, the test passes. - The existing test "does not carry an earlier call's host onto a call the tap skips, or into another run" still passes. - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes. - Latency: the change adds no `await` on the request path. It removes one synchronous callback before the upstream request and adds one after the response ends. The tap still forwards each chunk before it reads it, so time to first token and boot time do not change. ## Risks - Low risk. The change only affects run-log fields (`upstreamProvider`, `providerRequestId`). These fields stay in the instance database and are not sent to Paperclip Telemetry or OpenTelemetry. - A small race remains if the driver handles a step's usage after the whole next call has ended. That window is now one full model call instead of about 15 ms. - If two model calls run at the same time in one session, the usage names the call that ended last. This was also true before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, code execution). It did the investigation, the change, the tests, and this description. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/run-log-events.md | 15 +++-- .../opencode/opencode-server-driver.test.ts | 58 +++++++++++++++++++ .../opencode/opencode-server-driver.ts | 55 ++++++++++-------- 3 files changed, 97 insertions(+), 31 deletions(-) diff --git a/doc/run-log-events.md b/doc/run-log-events.md index 8b1566c2b0f..5720fb6dad1 100644 --- a/doc/run-log-events.md +++ b/doc/run-log-events.md @@ -984,15 +984,18 @@ never affects the response or the turn. Each `item.completed` usage event (`payload.kind: "usage"`) carries two fields in `payload.usage`: -- `upstreamProvider`: the host that served the most recent model call when the - usage was reported, such as `Novita` or `DeepInfra`. +- `upstreamProvider`: the host that served the most recently finished model + call when the usage was reported, such as `Novita` or `DeepInfra`. - `providerRequestId`: that call's OpenRouter generation ID, such as `gen-1762981234-AbCdEfGh`. -The proxy clears the recorded host when each new model request starts, and a -session clears it when it attaches another run. A call whose host the proxy -could not read therefore carries neither field. It never inherits an earlier -call's or an earlier run's values. +The proxy records a call's host when that call's response ends, not when the +request starts. OpenCode reports a step's usage only a few milliseconds before +it sends the next model request, so a step's usage still names its own call's +host while the next call is in flight. A call whose host the proxy could not +read records no host when it ends, and a session clears the recorded host when +it attaches another run. Such a call therefore carries neither field. It never +inherits an earlier call's or an earlier run's values. The runner's internal `usage.reported` event carries the same values, using its existing `providerRequestId` field and an optional `upstreamProvider` field. diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts index 5dab58875b1..fe3dea0a1a5 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts @@ -1249,6 +1249,64 @@ describe("OpenCodeServerDriver", () => { } }); + it("keeps a finished call's host on its usage while OpenCode's next model call is already in flight", async () => { + await chmod(fixture, 0o755); + const model = "deepseek/deepseek-v4.1-flash-1017"; + let releaseNext: (() => void) | undefined; + let call = 0; + const upstream = createHttpServer((request, response) => { + void (async () => { + for await (const _chunk of request) void _chunk; + call += 1; + response.writeHead(200, { "content-type": "text/event-stream" }); + if (call === 1) { + response.end('data: {"id":"gen-first","provider":"Novita","choices":[{"delta":{"content":"Hi"}}]}\n\ndata: [DONE]\n\n'); + return; + } + response.write('data: {"id":"gen-next","provider":"Chutes","choices":[{"delta":{"content":"Hi"}}]}\n\n'); + await new Promise(resolve => { releaseNext = resolve; }); + response.end("data: [DONE]\n\n"); + })().catch(() => response.destroy()); + }); + await new Promise(resolve => upstream.listen(0, "127.0.0.1", resolve)); + const address = upstream.address(); + if (!address || typeof address === "string") throw new Error("Missing fixture address"); + upstreamTransport.endpoint = `http://127.0.0.1:${address.port}/chat/completions`; + upstreamTransport.urls = []; + const root = await mkdtemp(join(tmpdir(), "paperclip-openrouter-inflight-")); + roots.push(root); + let session: Awaited> | undefined; + try { + const driver = new OpenCodeServerDriver({ model: `openrouter/${model}`, runtimeDirectory: root, command: fixture, + environment: { PATH: process.env.PATH, OPENROUTER_API_KEY: "synthetic-inflight-key" } }); + session = await driver.openSession({ runId: "inflight", normalizedSessionId: "inflight", workingDirectory: root }); + const config = JSON.parse(await readFile(join(root, "inflight", "config", "opencode", "opencode.json"), "utf8")); + const { baseURL, apiKey } = config.provider.openrouter.options; + const send = () => fetch(`${baseURL}/chat/completions`, { method: "POST", + headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" }, + body: JSON.stringify({ model, stream: true, messages: [{ role: "user", content: "synthetic" }] }) }); + await (await send()).arrayBuffer(); + const next = await send(); + expect(next.status).toBe(200); + const reader = next.body!.getReader(); + expect(new TextDecoder().decode((await reader.read()).value)).toContain('"provider":"Chutes"'); + await session.startTurn({ message: { role: "user", text: "finish" } }); + const events = await collectTurnEvents(session.events()); + const usage = events.find(event => event.eventType === "item.completed" && event.payload.kind === "usage")?.payload.usage; + expect(usage).toMatchObject({ upstreamProvider: "Novita", providerRequestId: "gen-first" }); + releaseNext?.(); + while (!(await reader.read()).done); + await session.startTurn({ message: { role: "user", text: "finish" } }); + await collectTurnEvents(session.events()); + expect(await session.usage?.()).toMatchObject({ upstreamProvider: "Chutes", providerRequestId: "gen-next" }); + } finally { + releaseNext?.(); + await session?.close({ reason: "test" }); + upstreamTransport.endpoint = ""; + await new Promise(resolve => { upstream.close(() => resolve()); upstream.closeAllConnections(); }); + } + }); + it("does not carry an earlier call's host onto a call the tap skips, or into another run", async () => { await chmod(fixture, 0o755); const model = "deepseek/deepseek-v4.1-flash-1017"; diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts index 800fdc599d0..e296d581734 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts @@ -2537,32 +2537,37 @@ async function startOpenCodeProviderProxy(baseUrl: string, key: string, model: s const redacted = redactOpenCodeModelRequest(body, secretValues, payload); if (redacted) body = redacted; } - try { onServed?.(null); } catch { /* Recording never affects the request. */ } - const upstream = await new Promise((resolve, reject) => { - const outgoing = (upstreamUrl.protocol === "https:" ? requestHttps : requestHttp)(upstreamUrl, { - method: "POST", - agent, - headers: { ...(isOpenRouterUrl(baseUrl) ? OPENROUTER_ATTRIBUTION_HEADERS : {}), "Content-Type": "application/json", "Content-Length": body.length, ...(key ? { Authorization: `Bearer ${key}` } : {}) }, - signal: AbortSignal.any([controller.signal, AbortSignal.timeout(600_000)]), - }, resolve); - outgoing.once("error", reject); - outgoing.end(body); - }); - // Never forward authentication to a redirect destination. - const status = upstream.statusCode ?? 502; - if (status >= 300 && status < 400) { - upstream.destroy(); - throw new Error("Provider redirects are not supported"); + let served: UpstreamServed | null = null; + try { + const upstream = await new Promise((resolve, reject) => { + const outgoing = (upstreamUrl.protocol === "https:" ? requestHttps : requestHttp)(upstreamUrl, { + method: "POST", + agent, + headers: { ...(isOpenRouterUrl(baseUrl) ? OPENROUTER_ATTRIBUTION_HEADERS : {}), "Content-Type": "application/json", "Content-Length": body.length, ...(key ? { Authorization: `Bearer ${key}` } : {}) }, + signal: AbortSignal.any([controller.signal, AbortSignal.timeout(600_000)]), + }, resolve); + outgoing.once("error", reject); + outgoing.end(body); + }); + // Never forward authentication to a redirect destination. + const status = upstream.statusCode ?? 502; + if (status >= 300 && status < 400) { + upstream.destroy(); + throw new Error("Provider redirects are not supported"); + } + const headers: Record = {}; + for (const name of ["content-type", "content-encoding", "cache-control", "retry-after"]) { + const value = upstream.headers[name]; + if (typeof value === "string") headers[name] = value; + } + response.writeHead(status, headers); + if (onServed && status >= 200 && status < 300) { + await pipeline(upstream, createUpstreamServedTap({ contentType: headers["content-type"], contentEncoding: headers["content-encoding"], + onServed: value => { served = value; } }), response); + } else await pipeline(upstream, response); + } finally { + try { onServed?.(served); } catch { /* Recording never affects the response. */ } } - const headers: Record = {}; - for (const name of ["content-type", "content-encoding", "cache-control", "retry-after"]) { - const value = upstream.headers[name]; - if (typeof value === "string") headers[name] = value; - } - response.writeHead(status, headers); - if (onServed && status >= 200 && status < 300) { - await pipeline(upstream, createUpstreamServedTap({ contentType: headers["content-type"], contentEncoding: headers["content-encoding"], onServed }), response); - } else await pipeline(upstream, response); })().catch(() => { if (!response.headersSent) response.writeHead(502).end("Provider request failed"); else response.destroy();