From 055a0a3c54122476a01e29d1bef286eebbc0ee95 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 13 Nov 2026 14:39:55 -0800 Subject: [PATCH] refactor(acpx-sidecar): extract the runtime event step and the test process helper (#21419) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - The last part of that stack adds a second kind of turn to the sidecar. It needs two pieces of the code of a normal turn. > - This pull request moves those two pieces into functions, and moves a test helper to a file of its own. It changes no behavior. > - The benefit is that the later pull request shows only new behavior, not moved code. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 1. Its base branch is `refactor/remove-direct-gateway-path`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** Nothing for a user. The structure of the ACPX sidecar code. **Subsystem affected** Paperclip Runner: `src/cli/acpx-runtime-sidecar.ts` and its test. **Current behavior** - `pumpTurn` has the code that prepares each runtime event for the Runner, and the code that sends the terminal frame of a turn, inline. - The sidecar test file has the `SidecarProcess` helper that starts a sidecar process. **Proposed behavior** - `createRuntimeEventStep` has the event code. `emitTurnTerminal` has the terminal code. `pumpTurn` calls both. - `src/test-support/acpx-sidecar-process.ts` has the helper, so a second test file can use it. - No protocol, event or timing change. **Reason and benefit** A later pull request adds a turn that Claude starts by itself. It uses the same two functions, and its tests use the same helper. ## What Changed - `src/cli/acpx-runtime-sidecar.ts`: 74 lines added and 56 removed. The code moved with its comments, with no change. - `src/cli/acpx-runtime-sidecar.test.ts` and `src/test-support/acpx-sidecar-process.ts`: the helper moved, 138 lines added and 132 removed. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. - `npx vitest run src/cli/acpx-runtime-sidecar.test.ts`: 125 passed. The tests are the same as before, apart from the import of the helper. Not run: the full suite. CI supplies that coverage. ## Risks - **A refactor of the turn loop of each ACPX agent.** The moved code runs for Claude, Codex, Pi, Kimi, Cursor, Copilot and Grok. The existing sidecar suite covers it. No new test is necessary, because no behavior changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../src/cli/acpx-runtime-sidecar.test.ts | 139 +----------------- .../src/cli/acpx-runtime-sidecar.ts | 130 +++++++++------- .../src/test-support/acpx-sidecar-process.ts | 131 +++++++++++++++++ 3 files changed, 212 insertions(+), 188 deletions(-) create mode 100644 packages/paperclip-runner/src/test-support/acpx-sidecar-process.ts diff --git a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts index e2a6ee7da19..9f720600e5b 100644 --- a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts +++ b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts @@ -26,6 +26,7 @@ import { classifyClaudeStatusBanner } from "../drivers/acpx/claude-status-banner import { acpxGoalProjection } from "../drivers/acpx/session-goals.js"; import { afterEach, describe, expect, it, vi } from "vitest"; +import { initializeRequest, SidecarProcess } from "../test-support/acpx-sidecar-process.js"; describe("live sidecar pending request snapshots", () => { it("preserves explicit turn cost through the sidecar's closed usage projection", () => { @@ -115,6 +116,12 @@ import { const children = new Set(); +function startSidecar(): SidecarProcess { + const sidecar = new SidecarProcess(); + children.add(sidecar); + return sidecar; +} + afterEach(async () => { vi.unstubAllEnvs(); await Promise.all([...children].map((child) => child.close())); @@ -2174,138 +2181,6 @@ describe("qualified ACPX runtime sidecar", () => { }); }); -function initializeRequest( - id: number, - agent: string, - model = "gpt-5.6-sol", -) { - return { - protocolVersion: ACPX_SIDECAR_PROTOCOL_VERSION, - id, - command: "initialize", - params: { agent, model }, - }; -} - -function startSidecar(): SidecarProcess { - const sidecar = new SidecarProcess(); - children.add(sidecar); - return sidecar; -} - -class SidecarProcess { - readonly #child: ChildProcessWithoutNullStreams; - readonly #frames: Array> = []; - readonly #signals: Array<() => void> = []; - #stderr = ""; - #closed = false; - readonly #exit: Promise<{code:number|null;signal:NodeJS.Signals|null}>; - - constructor(options: {loader?:string;environment?:Record} = {}) { - this.#child = spawn( - options.loader ? process.execPath : fileURLToPath(new URL("../../node_modules/.bin/tsx", import.meta.url)), - [...(options.loader ? ["--import","tsx","--import",options.loader] : []), fileURLToPath(new URL("./acpx-runtime-sidecar.ts", import.meta.url))], - { stdio: ["pipe", "pipe", "pipe"], env: { PATH: process.env.PATH, LANG: "C.UTF-8", ...options.environment } }, - ); - this.#exit = new Promise(resolve => this.#child.once("exit", (code, signal) => resolve({code,signal}))); - let stdout = ""; - this.#child.stdout.setEncoding("utf8"); - this.#child.stdout.on("data", (chunk: string) => { - stdout += chunk; - for (;;) { - const newline = stdout.indexOf("\n"); - if (newline < 0) break; - const line = stdout.slice(0, newline); - stdout = stdout.slice(newline + 1); - if (!line.trim()) continue; - this.#frames.push(JSON.parse(line) as Record); - for (const signal of this.#signals.splice(0)) signal(); - } - }); - this.#child.stderr.setEncoding("utf8"); - this.#child.stderr.on("data", (chunk: string) => { - this.#stderr += chunk; - }); - } - - write(value: Record): void { - this.#child.stdin.write(`${JSON.stringify(value)}\n`); - } - - closeOutput(): void { - this.#child.stdout.destroy(); - } - - async shutdown(trigger: "stdin" | "SIGTERM" | "SIGINT") { - if (trigger === "stdin") this.#child.stdin.end(); - else this.#child.kill(trigger); - return this.exited(); - } - - async exited() { - let timer: ReturnType | undefined; - try { - return await Promise.race([ - this.#exit, - new Promise((_, reject) => { - timer = setTimeout(() => reject(new Error("Sidecar did not exit without forced cleanup")), 5000); - }), - ]); - } finally { clearTimeout(timer); } - } - - stderr(): string { - return this.#stderr; - } - - pending(predicate: (frame: Record) => boolean): number { - return this.#frames.filter(predicate).length; - } - - async next( - predicate: (frame: Record) => boolean, - ): Promise> { - const deadline = Date.now() + 5_000; - for (;;) { - const index = this.#frames.findIndex(predicate); - if (index >= 0) return this.#frames.splice(index, 1)[0]!; - const remaining = deadline - Date.now(); - if (remaining <= 0) { - throw new Error( - `Timed out waiting for sidecar frame. stderr=${JSON.stringify(this.#stderr)}`, - ); - } - await new Promise((resolve, reject) => { - const timer = setTimeout(() => { - const index = this.#signals.indexOf(signal); - if (index >= 0) this.#signals.splice(index, 1); - reject(new Error("Timed out waiting for sidecar output")); - }, remaining); - const signal = () => { - clearTimeout(timer); - resolve(); - }; - this.#signals.push(signal); - }); - } - } - - async close(): Promise { - if (this.#closed || this.#child.exitCode !== null || this.#child.signalCode !== null) return; - this.#closed = true; - this.#child.stdin.end(); - const exit = new Promise((resolve) => { - this.#child.once("exit", () => resolve()); - }); - const timeout = new Promise((resolve) => { - setTimeout(() => { - if (this.#child.exitCode === null) this.#child.kill("SIGKILL"); - resolve(); - }, 2_000).unref(); - }); - await Promise.race([exit, timeout]); - } -} type ToolCallResponseBody = { result: { content: Array<{ text: string }>; isError?: boolean } }; diff --git a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts index 3833b9c6108..f1cf1febc3b 100644 --- a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts +++ b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts @@ -714,6 +714,68 @@ async function dispatch( throw new Error("unreachable ACPX sidecar command"); } +function createRuntimeEventStep( + currentTurnId: string, + toolEvidence?: AcpxToolEvidence, +): { accept(event: AcpRuntimeEvent): void; settle(): void } { + // ACP tool updates are deltas. Preserve the opening event's identity and + // display metadata for later progress/completion frames before they cross + // the sidecar boundary, matching the in-process ACPX driver path. + const normalizeToolEvent = createAcpxToolEventNormalizer(); + const piMessages = initializedAgent === "pi" ? createPiMessageProjection() : null; + const kimiMessages = initializedAgent === "kimi" ? createKimiMessageProjection() : null; + const normalizeMessage = kimiMessages ? kimiMessages.normalize : piMessages ? piMessages.normalize : initializedAgent === "grok" + ? createGrokMessageNormalizer() : (event: AcpRuntimeEvent) => event; + // Claude subagent lineage reads the bounded `_meta` and `rawInput` before + // they are dropped at this boundary. Only display data leaves: a + // schema-checked delegation rich event and a child tool's parent id. + const trackSubagents = initializedAgent === "claude" + ? createClaudeSubagentLineageTracker((id) => stableProviderIdentity(id, "tool")) + : null; + const accept = (event: AcpRuntimeEvent): void => { + toolUseAdmission?.observe(currentTurnId, event); + toolEvidence?.tool(event); + // Claude hook lifecycle rides on session_info_update `_meta`; publish it + // as schema-validated display activity rather than a raw status event. + const hook = claudeHookCanonicalEvent(event); + if (hook) { + try { validateAcpxRichEvent(hook); emit("runtime.rich_event", { ...hook }, currentTurnId); } catch { /* display only */ } + return; + } + const sanitized = sanitizeRuntimeEvent( + normalizeMessage(normalizeToolEvent(withSidecarToolDetail(event, boundRuntimeEventForNormalization(event)))), + ); + const lineage = trackSubagents && event.type === "tool_call" && typeof sanitized.toolCallId === "string" + ? trackSubagents({ + type: "tool_call", + toolCallId: sanitized.toolCallId, + tag: event.tag, + status: event.status, + rawInput: event.rawInput, + rawOutput: event.rawOutput, + _meta: boundedAcpMeta((event as { _meta?: unknown })._meta), + }) + : null; + if (lineage?.paperclipParentToolCallId) sanitized.parentToolCallId = lineage.paperclipParentToolCallId; + emit("runtime.event", sanitized, currentTurnId); + if (lineage?.paperclipSubagent) { + // Rust names the tool call (and each child's parentItemId) with + // acpx_opaque_item_id; the delegation must use the same identity. + const delegationId = acpxOpaqueToolItemId(String(sanitized.toolCallId)); + for (const mapped of canonicalProviderEventsFromAcpxRuntimeEvent(lineage, `${currentTurnId}:subagent`, currentTurnId)) { + const rich = withDelegationIdentity(mapped, delegationId); + try { + validateAcpxRichEvent(rich); + emit("runtime.rich_event", { ...rich }, currentTurnId); + } catch { + // Display-only lineage must never fail the provider turn. + } + } + } + }; + return { accept, settle: () => { piMessages?.settle(); kimiMessages?.settle(); } }; +} + /** Turns whose native plan approval was declined after capturing the proposal. */ const planProposalTurns = new Set(); @@ -729,63 +791,10 @@ async function pumpTurn( ): Promise { let terminal: Record; try { - // ACP tool updates are deltas. Preserve the opening event's identity and - // display metadata for later progress/completion frames before they cross - // the sidecar boundary, matching the in-process ACPX driver path. - const normalizeToolEvent = createAcpxToolEventNormalizer(); - const piMessages = initializedAgent === "pi" ? createPiMessageProjection() : null; - const kimiMessages = initializedAgent === "kimi" ? createKimiMessageProjection() : null; - const normalizeMessage = kimiMessages ? kimiMessages.normalize : piMessages ? piMessages.normalize : initializedAgent === "grok" - ? createGrokMessageNormalizer() : (event: AcpRuntimeEvent) => event; - // Claude subagent lineage reads the bounded `_meta` and `rawInput` before - // they are dropped at this boundary. Only display data leaves: a - // schema-checked delegation rich event and a child tool's parent id. - const trackSubagents = initializedAgent === "claude" - ? createClaudeSubagentLineageTracker((id) => stableProviderIdentity(id, "tool")) - : null; - for await (const event of runtimeTurn.events) { - toolUseAdmission?.observe(currentTurnId, event); - toolEvidence?.tool(event); - // Claude hook lifecycle rides on session_info_update `_meta`; publish it - // as schema-validated display activity rather than a raw status event. - const hook = claudeHookCanonicalEvent(event); - if (hook) { - try { validateAcpxRichEvent(hook); emit("runtime.rich_event", { ...hook }, currentTurnId); } catch { /* display only */ } - continue; - } - const sanitized = sanitizeRuntimeEvent( - normalizeMessage(normalizeToolEvent(withSidecarToolDetail(event, boundRuntimeEventForNormalization(event)))), - ); - const lineage = trackSubagents && event.type === "tool_call" && typeof sanitized.toolCallId === "string" - ? trackSubagents({ - type: "tool_call", - toolCallId: sanitized.toolCallId, - tag: event.tag, - status: event.status, - rawInput: event.rawInput, - rawOutput: event.rawOutput, - _meta: boundedAcpMeta((event as { _meta?: unknown })._meta), - }) - : null; - if (lineage?.paperclipParentToolCallId) sanitized.parentToolCallId = lineage.paperclipParentToolCallId; - emit("runtime.event", sanitized, currentTurnId); - if (lineage?.paperclipSubagent) { - // Rust names the tool call (and each child's parentItemId) with - // acpx_opaque_item_id; the delegation must use the same identity. - const delegationId = acpxOpaqueToolItemId(String(sanitized.toolCallId)); - for (const mapped of canonicalProviderEventsFromAcpxRuntimeEvent(lineage, `${currentTurnId}:subagent`, currentTurnId)) { - const rich = withDelegationIdentity(mapped, delegationId); - try { - validateAcpxRichEvent(rich); - emit("runtime.rich_event", { ...rich }, currentTurnId); - } catch { - // Display-only lineage must never fail the provider turn. - } - } - } - } + const step = createRuntimeEventStep(currentTurnId, toolEvidence); + for await (const event of runtimeTurn.events) step.accept(event); const result = await runtimeTurn.result; - if (result.status === "completed") { piMessages?.settle(); kimiMessages?.settle(); } + if (result.status === "completed") step.settle(); await drainExtensions(); try { const usageAfter = await readSidecarHostStatusWithin(activeHost); @@ -836,6 +845,15 @@ async function pumpTurn( if (declinedPlan) { terminal = { status: "completed", stopReason: "end_turn" }; } + emitTurnTerminal(currentTurnId, terminal, activeHost, declinedPlan); +} + +function emitTurnTerminal( + currentTurnId: string, + terminal: Record, + activeHost: AcpxRuntimeHost, + declinedPlan: boolean, +): void { // A terminal frame is also runnerd's permission to recycle this provider. // Publish it only after no callback can inherit this turn's mutable binding. // runnerd can settle this turn offline using the same observation as session.goal.get. diff --git a/packages/paperclip-runner/src/test-support/acpx-sidecar-process.ts b/packages/paperclip-runner/src/test-support/acpx-sidecar-process.ts new file mode 100644 index 00000000000..36ba1a1f2db --- /dev/null +++ b/packages/paperclip-runner/src/test-support/acpx-sidecar-process.ts @@ -0,0 +1,131 @@ +import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process"; +import { fileURLToPath } from "node:url"; + +import { ACPX_SIDECAR_PROTOCOL_VERSION } from "../drivers/acpx/sidecar-protocol.js"; + +export function initializeRequest( + id: number, + agent: string, + model = "gpt-5.6-sol", +) { + return { + protocolVersion: ACPX_SIDECAR_PROTOCOL_VERSION, + id, + command: "initialize", + params: { agent, model }, + }; +} + +export class SidecarProcess { + readonly #child: ChildProcessWithoutNullStreams; + readonly #frames: Array> = []; + readonly #signals: Array<() => void> = []; + #stderr = ""; + #closed = false; + readonly #exit: Promise<{code:number|null;signal:NodeJS.Signals|null}>; + + constructor(options: {loader?:string;environment?:Record} = {}) { + this.#child = spawn( + options.loader ? process.execPath : fileURLToPath(new URL("../../node_modules/.bin/tsx", import.meta.url)), + [...(options.loader ? ["--import","tsx","--import",options.loader] : []), fileURLToPath(new URL("../cli/acpx-runtime-sidecar.ts", import.meta.url))], + { stdio: ["pipe", "pipe", "pipe"], env: { PATH: process.env.PATH, LANG: "C.UTF-8", ...options.environment } }, + ); + this.#exit = new Promise(resolve => this.#child.once("exit", (code, signal) => resolve({code,signal}))); + let stdout = ""; + this.#child.stdout.setEncoding("utf8"); + this.#child.stdout.on("data", (chunk: string) => { + stdout += chunk; + for (;;) { + const newline = stdout.indexOf("\n"); + if (newline < 0) break; + const line = stdout.slice(0, newline); + stdout = stdout.slice(newline + 1); + if (!line.trim()) continue; + this.#frames.push(JSON.parse(line) as Record); + for (const signal of this.#signals.splice(0)) signal(); + } + }); + this.#child.stderr.setEncoding("utf8"); + this.#child.stderr.on("data", (chunk: string) => { + this.#stderr += chunk; + }); + } + + write(value: Record): void { + this.#child.stdin.write(`${JSON.stringify(value)}\n`); + } + + closeOutput(): void { + this.#child.stdout.destroy(); + } + + async shutdown(trigger: "stdin" | "SIGTERM" | "SIGINT") { + if (trigger === "stdin") this.#child.stdin.end(); + else this.#child.kill(trigger); + return this.exited(); + } + + async exited() { + let timer: ReturnType | undefined; + try { + return await Promise.race([ + this.#exit, + new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error("Sidecar did not exit without forced cleanup")), 5000); + }), + ]); + } finally { clearTimeout(timer); } + } + + stderr(): string { + return this.#stderr; + } + + pending(predicate: (frame: Record) => boolean): number { + return this.#frames.filter(predicate).length; + } + + async next( + predicate: (frame: Record) => boolean, + ): Promise> { + const deadline = Date.now() + 5_000; + for (;;) { + const index = this.#frames.findIndex(predicate); + if (index >= 0) return this.#frames.splice(index, 1)[0]!; + const remaining = deadline - Date.now(); + if (remaining <= 0) { + throw new Error( + `Timed out waiting for sidecar frame. stderr=${JSON.stringify(this.#stderr)}`, + ); + } + await new Promise((resolve, reject) => { + const timer = setTimeout(() => { + const index = this.#signals.indexOf(signal); + if (index >= 0) this.#signals.splice(index, 1); + reject(new Error("Timed out waiting for sidecar output")); + }, remaining); + const signal = () => { + clearTimeout(timer); + resolve(); + }; + this.#signals.push(signal); + }); + } + } + + async close(): Promise { + if (this.#closed || this.#child.exitCode !== null || this.#child.signalCode !== null) return; + this.#closed = true; + this.#child.stdin.end(); + const exit = new Promise((resolve) => { + this.#child.once("exit", () => resolve()); + }); + const timeout = new Promise((resolve) => { + setTimeout(() => { + if (this.#child.exitCode === null) this.#child.kill("SIGKILL"); + resolve(); + }, 2_000).unref(); + }); + await Promise.race([exit, timeout]); + } +}