From e65a3095a711b7745654130f753af1eaae4e5936 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Thu, 12 Nov 2026 21:19:24 -0800 Subject: [PATCH] fix(runner): quiesce the Core-dialed route once the Runner commits its suspend (#21245) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Core runs agent sessions through the Paperclip Runner. With provider ingress, Core dials the Runner over a PRP WebSocket. > - When Core closes a session, it asks the Runner to suspend. The Runner suspends, reports the result, and exits. > - Core's ingress connection did not know that this exit was expected. It treated the drop as a transport loss and redialed the Runner. Each redial got a 404 until the close released the route. > - The release (park) path already quiesces the connection before the Runner goes away. The suspend path did not. > - This pull request makes the suspend close quiesce the route in the same way, once the Runner commits its suspend. > - The benefit is that a clean suspend no longer looks like a lost Runner. A drop before the suspend is committed still redials and recovers as before. ## Linked Issues or Issue Description **What happened?** After a provider-ingress session closed cleanly, Core redialed the Runner's ingress endpoint several times and got 404 each time. The Runner state was `suspended`, `runner.drain` and `runner.suspend` had both completed, and the Runner had exited with code 0. The redials followed the outbound reconnect backoff (about 250, 500, 1000 and 2000 ms) and stopped when the registration release closed the outbound. **Expected behavior** After Core commits to `runner.suspend` and the Runner reports it completed, the Runner's exit is expected. Core must not redial or classify the drop as a transport loss. The release path already works this way: it calls `outbound.quiesce()` before it releases the task. **Steps to reproduce** 1. Open a session through a Core-dialed route (plugin provider ingress or the `listen_ws` driver). 2. Close the session. Core drains, queues `runner.suspend`, and the Runner suspends and exits. 3. See that the outbound reconnect loop redials the gone Runner until the registration release closes it. **Paperclip version or commit** `master` at b4ca19f3a8. **Agent adapter(s) involved** Not adapter-specific (core bug). It affects any Core-dialed Runner route: plugin provider ingress and the `listen_ws` driver. **Additional context** The redials do not fail the session. The plugin outbound uses a 7-minute recovery grace, and `#failTransport` ignores failures after `#closeAndSettle` sets `#closed`. The new test confirms this: without the fix, the close still settles, but the route redials. This PR removes the misleading redial. It does not change how a run's own failure is classified. ## What Changed - `RunnerExecutionDriver.controlPlaneRegistration` results can now include an optional `quiesce()`. Core calls it when it has committed the Runner's exit. The route then stops liveness and reconnect, and it keeps the established receive/ACK path. - `DurablePrpControlPlane.onRunnerExitCommitted(listener)` is new. The authority calls its listeners after it saves a completed `runner.suspend` or `runner.shutdown` result and before it sends the acknowledgement. The Runner exits when it gets that acknowledgement, so a route quiesced here never sees the exit as a loss. - `awaitRunnerSuspensionBarrier` calls a new `onExitCommitted` callback once, when its suspend command is `completed`. It also calls the callback when it reuses a completed receipt and the durable state is still `suspended`. A rejected, failed or unanswered suspend never calls it. - `DurablePrpCodexTransport` registers each control-plane route through one `#registerControlPlane` helper. The helper records each route's `quiesce` until that route is released. The suspend close subscribes to `onRunnerExitCommitted` while its barrier runs and quiesces all of these routes from that notification. The barrier's own `onExitCommitted` covers a completed receipt that was acknowledged before this close. Quiescing clears the tracked routes, so it runs once per close. The helper copies property descriptors, so getters such as the `listen_ws` driver's lazy `failure` stay live. - The plugin task registration (`plugin-runner-task.ts`) and the `listen_ws` driver registration (`native-runner-execution-driver.ts`) implement `quiesce` with `outbound?.quiesce()`. This is the same call that the release path makes. ## Verification - New transport test with a real runnerd: `packages/paperclip-runner/src/live/runnerd-codex-transport-suspend-route.test.ts`. The test route acts like a Core-dialed provider ingress: on a socket drop that it was not told to expect, it records a redial into a 404. Before the fix the events were `connected, redial: 404, release`. After the fix they are `connected, quiesce after runner.suspend completed, release`. The close settles, and the durable Runner state is `suspended`. A second case delays the durable-state read by 1.5 s, so the Runner exits while the barrier still reads. That case fails without the authority hook and passes with it. - New barrier unit test in `runnerd-codex-transport.test.ts`. It shows that the exit is committed only after the retried suspend completes, never on a rejected or unanswered suspend, and on a reused receipt only when the state is `suspended`. - New server tests for the plugin task and `listen_ws` registrations. `quiesce` reaches the active outbound and does not close it. Release still closes it. - Existing tests pass. These include the outbound release-path quiesce test, the half-open and lease-expiry failure tests (a drop before the commit still fails after its grace), and the plugin release/resume facade tests. - `pnpm verify` exited 2 (the change touches files outside the resolved test graphs), so I ran focused suites. Locally: 8 runner transport files and the 4 control-plane test files, 667 tests (`runnerd-codex-transport*.test.ts` for suspension, release/resume, remote retention, settlement, warm attachment and attachment recovery, plus the new suspend-route test), and 5 server files, 148 tests (both plugin Runner integration tests, the plugin task, the execution driver and the outbound). Required CI supplies the broad coverage. - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. `quiesce` is optional. Registrations that do not supply it behave as before. - The quiesce happens only after the Runner reports a completed suspend. A socket drop before that point still redials and recovers. - The quiesced route keeps its receive/ACK path, so the Runner's last frames before exit are still acknowledged. - No persisted state or protocol change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../durable-prp-control-plane.ts | 23 ++++ .../control-plane/runner-execution-driver.ts | 4 + ...nerd-codex-transport-suspend-route.test.ts | 108 +++++++++++++++++ .../src/live/runnerd-codex-transport.test.ts | 65 +++++++++++ .../src/live/runnerd-codex-transport.ts | 109 ++++++++++++++---- .../native-runner-execution-driver.test.ts | 20 ++++ .../native-runner-execution-driver.ts | 1 + .../native-runtime/plugin-runner-task.test.ts | 19 +++ .../native-runtime/plugin-runner-task.ts | 2 + 9 files changed, 329 insertions(+), 22 deletions(-) create mode 100644 packages/paperclip-runner/src/live/runnerd-codex-transport-suspend-route.test.ts diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index d928f3d40f0..6f73fe23383 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -2117,6 +2117,7 @@ export class DurablePrpControlPlane { #beforeAuthenticatedConnection?: DurablePrpControlPlaneOptions["beforeAuthenticatedConnection"]; #onProtocolIntegrityError?: DurablePrpControlPlaneOptions["onProtocolIntegrityError"]; #onAuthenticatedConnection?: DurablePrpControlPlaneOptions["onAuthenticatedConnection"]; + readonly #runnerExitListeners = new Set<() => void>(); #protocolIntegrityError: NativeSessionProtocolIntegrityError | null = null; #eventDeliveryStall: { runId: string; ackedSourceSeq: number; since: number; failures: number } | null = null; #connectionLeaseTtlMs: number; @@ -2200,6 +2201,19 @@ export class DurablePrpControlPlane { }; } + /** + * Notify when the Runner reports a completed suspend or shutdown. The + * listener runs after the result is saved and before its acknowledgement; + * the Runner exits once acknowledged, so a dropped connection after this + * point is that exit, not a transport loss. + */ + onRunnerExitCommitted(listener: () => void): () => void { + this.#runnerExitListeners.add(listener); + return () => { + this.#runnerExitListeners.delete(listener); + }; + } + #readableStateChanged(): void { for (const listener of this.#readableStateListeners) { // A reader can never fail the commit that woke it. @@ -4215,6 +4229,15 @@ export class DurablePrpControlPlane { command.result = structuredClone(result); this.#store.save(); if (turnStart) crashPoint("runner.turn_start.after_controller_journal_before_db_checkpoint"); + if (this.#isTerminalLifecycleCommand(command) && status === "completed") { + for (const listener of this.#runnerExitListeners) { + try { + listener(); + } catch { + // An observer can never change the acknowledgement. + } + } + } this.#ackTerminalCommandResult(connection, command); this.#releaseServingLifecycle(connection, command); if (!this.#isTerminalLifecycleCommand(command) || connection.terminalLifecycleCommandId === null) { diff --git a/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts b/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts index 79165267471..18c7baa84cc 100644 --- a/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts +++ b/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts @@ -48,6 +48,10 @@ export interface RunnerExecutionDriver { | "runner_ingress_unavailable"; /** Persists only exact, independently verified suspended remote state. */ checkpoint?: (settlement: "settled" | "unsettled") => Promise | void; + /** Core committed the Runner's exit (its close suspend completed). Stop + * treating a dropped connection as transport loss, so a Core-dialed route + * neither redials nor fails; keep the established receive/ACK path. */ + quiesce?: () => void; release: (reason?: "detach") => Promise | void; }>; /** Register source and target checkpoint ownership before staging changes the stored run. */ diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport-suspend-route.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport-suspend-route.test.ts new file mode 100644 index 00000000000..4efc52d36d7 --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport-suspend-route.test.ts @@ -0,0 +1,108 @@ +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { createServer } from "node:http"; +import type { Socket } from "node:net"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { expect, it } from "vitest"; +import type { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js"; +import { createCapabilityRunnerdCodexTransport } from "./runnerd-codex-transport.js"; +import { defaultCapabilityRunnerdBinary, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js"; + +// A Core-dialed route (provider ingress) treats every socket drop it did not +// cause as a transport loss: it redials, and the endpoint answers 404 once the +// Runner process is gone. This route keeps that shape, so a drop it was not +// told to expect shows up as a redial into a 404. +function coreDialedRoute() { + const events: string[] = []; + let authority: DurablePrpControlPlane | null = null; + let quiesced = false; + let released = false; + let rejectFailure!: (error: Error) => void; + const failure = new Promise((_resolve, reject) => { rejectFailure = reject; }); + void failure.catch(() => undefined); + const server = createServer(); + server.on("upgrade", (request, socket: Socket, head) => { + if (!authority) { socket.destroy(); return; } + events.push("connected"); + socket.once("close", () => { + if (quiesced || released) return; + events.push("redial: 404"); + rejectFailure(new Error("runner_ingress_unavailable: Runner ingress connection failed.")); + }); + authority.handleUpgrade(request, socket, "/runner", head); + }); + return { + events, + async listen() { + await new Promise(resolveListen => server.listen(0, "127.0.0.1", resolveListen)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Missing route listener"); + return address.port; + }, + registration(port: number) { + return async (next: DurablePrpControlPlane) => { + authority = next; + return { + connectUrl: `ws://127.0.0.1:${port}/runner`, + failure, + quiesce: () => { + const suspend = next.store.state.commands.findLast(command => command.type === "runner.suspend"); + events.push(`quiesce after runner.suspend ${suspend?.status ?? "missing"}`); + quiesced = true; + }, + release: () => { + events.push("release"); + released = true; + if (authority === next) authority = null; + }, + }; + }; + }, + async close() { + server.closeAllConnections(); + if (server.listening) await new Promise(resolveClose => server.close(() => resolveClose())); + }, + }; +} + +it.each([ + { title: "", stateReadDelayMs: 0 }, + // A remote durable-state read can outlast the Runner: it exits as soon as + // Core acknowledges its suspend result, while the barrier is still reading. + { title: " while the barrier is still reading its durable state", stateReadDelayMs: 1_500 }, +])("quiesces a Core-dialed route once the Runner commits its suspend$title, so its exit is not a transport loss", async ({ stateReadDelayMs }) => { + const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-suspend-route-")); + const route = coreDialedRoute(); + const port = await route.listen(); + const bundle = createCapabilityRunnerdCodexTransport({ + runnerBinary: defaultCapabilityRunnerdBinary(), + codexCommand: fakeCodex, + codexArgs: fakeCodexArgs(stateDirectory), + stateDirectory, + controlPlaneRegistration: route.registration(port), + ...(stateReadDelayMs > 0 ? { + readRunnerState: async () => { + await new Promise(resolveRead => setTimeout(resolveRead, stateReadDelayMs)); + return JSON.parse(await readFile(join(stateDirectory, "runner", "runner-state.json"), "utf8")) as Record; + }, + } : {}), + }); + bundle.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] })); + try { + await bundle.transport.request("initialize", {}); + await bundle.transport.request("thread/start", { cwd: stateDirectory, dynamicTools: [] }); + // Before the suspend is committed, a drop is still an ordinary loss. + expect(route.events).toEqual(["connected"]); + + await expect(bundle.transport.close("native session execution complete")).resolves.toBeUndefined(); + + expect(route.events).toEqual(["connected", "quiesce after runner.suspend completed", "release"]); + const runnerState = JSON.parse(await readFile(join(stateDirectory, "runner", "runner-state.json"), "utf8")) as Record; + expect(runnerState.lifecycle).toBe("suspended"); + expect(bundle.evidence().runnerExited).toBe(true); + } finally { + await bundle.transport.close().catch(() => undefined); + await route.close(); + await rm(stateDirectory, { recursive: true, force: true }); + } +}, 30_000); diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts index cce59af2f7f..b6b7cdac739 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts @@ -2526,6 +2526,71 @@ it("does not accept process exit without durable suspension", async () => { ).resolves.toBe(false); }); +it("commits the Runner's exit only once its suspend command completes", async () => { + const commands: Array<{ commandId: string; type: string; status: string }> = []; + const order: string[] = []; + let lifecycle = "ready"; + let pumps = 0; + const exitCommitted = vi.fn(() => { order.push(`exit committed after ${commands.length} attempts`); }); + await expect( + runnerdRecoveryInternals.awaitRunnerSuspensionBarrier({ + commands: () => commands, + queueSuspend: commandId => { commands.push({ commandId, type: "runner.suspend", status: "pending" }); }, + readRunnerState: async () => { + order.push(`state ${lifecycle}`); + return { lifecycle }; + }, + runnerHasExited: async () => false, + pump: () => { + pumps += 1; + const pending = commands.find(command => command.status === "pending"); + if (!pending) return; + // The first attempt is rejected; the retry stays pending for one + // round before the Runner reports it completed. + if (commands.length === 1) pending.status = "failed"; + else if (pumps >= 4) { pending.status = "completed"; lifecycle = "suspended"; } + }, + onExitCommitted: exitCommitted, + deadline: Date.now() + 2_000, + pollIntervalMs: 0, + }), + ).resolves.toBe(true); + expect(exitCommitted).toHaveBeenCalledOnce(); + expect(order.indexOf("exit committed after 2 attempts")).toBeGreaterThan(order.indexOf("state ready")); + expect(order.at(-1)).toBe("state suspended"); + + // A reused completed receipt commits the exit only once the durable state + // proves the Runner is still suspended. + for (const [state, committed] of [["suspended", true], ["ready", false]] as const) { + const reusedCommitted = vi.fn(); + await runnerdRecoveryInternals.awaitRunnerSuspensionBarrier({ + commands: () => [{ commandId: "command_prior_suspend", type: "runner.suspend", status: "completed" }], + queueSuspend: () => undefined, + readRunnerState: async () => ({ lifecycle: state }), + runnerHasExited: async () => true, + pump: () => undefined, + onExitCommitted: reusedCommitted, + deadline: Date.now() + 20, + pollIntervalMs: 0, + }); + expect(reusedCommitted).toHaveBeenCalledTimes(committed ? 1 : 0); + } + + // A rejected or unanswered suspend never commits the exit. + const neverCommitted = vi.fn(); + await expect(runnerdRecoveryInternals.awaitRunnerSuspensionBarrier({ + commands: () => commands.slice(0, 0), + queueSuspend: () => undefined, + readRunnerState: async () => ({ lifecycle: "ready" }), + runnerHasExited: async () => false, + pump: () => undefined, + onExitCommitted: neverCommitted, + deadline: Date.now() + 20, + pollIntervalMs: 0, + })).resolves.toBe(false); + expect(neverCommitted).not.toHaveBeenCalled(); +}); + it("reserves a bounded suspension window after close preparation", () => { expect(runnerdRecoveryInternals.runnerCloseDeadlines(1_000, 10_000)).toEqual({ preparationDeadline: 8_500, diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index 20223a7f1bd..4c21575c630 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -1024,6 +1024,8 @@ function turnStartCommandResultValid(input: { ); } +type ControlPlaneRegistration = Awaited>>; + type CloseCheckpointRetry = { checkpoint: () => Promise | void; finish: (deadline: number) => Promise; @@ -1079,10 +1081,18 @@ async function awaitRunnerSuspensionBarrier(input: { readRunnerState: () => Promise>; runnerHasExited: () => Promise; pump: () => void; + /** Called once the Runner's exit is certain: its suspend command completed. */ + onExitCommitted?: () => void; deadline: number; pollIntervalMs?: number; }): Promise { if (Date.now() >= input.deadline) return false; + let exitCommitted = false; + const commitExit = () => { + if (exitCommitted) return; + exitCommitted = true; + input.onExitCommitted?.(); + }; const readRunnerState = async (): Promise | undefined> => { const remainingMs = input.deadline - Date.now(); if (remainingMs <= 0) return undefined; @@ -1112,8 +1122,10 @@ async function awaitRunnerSuspensionBarrier(input: { try { const state = await readRunnerState(); if (state === undefined) return false; - if (state.lifecycle === "suspended") + if (state.lifecycle === "suspended") { + commitExit(); return Date.now() < input.deadline; + } } catch { // The normal bounded barrier below handles unavailable state. } @@ -1156,6 +1168,9 @@ async function awaitRunnerSuspensionBarrier(input: { input.queueSuspend(commandId); continue; } + // The Runner exits right after it reports a completed suspend, so its exit + // is certain before this barrier reads the durable state. + if (command?.status === "completed") commitExit(); let lifecycle: unknown; try { const state = await readRunnerState(); @@ -4131,6 +4146,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ((settlement: "settled" | "unsettled") => Promise | void) | null = null; #controlPlaneRelease: ((reason?: "detach") => Promise | void) | null = null; + /** Quiesce hooks of the registered routes that have not been released yet. */ + readonly #controlPlaneQuiesces = new Set<() => void>(); #nextTraceDebugSequence = 1; #traceRehydrationSpoolOverflow = false; readonly #traceFrameIndex = new RunnerdTraceFrameIndex(); @@ -4720,6 +4737,42 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ); } + async #registerControlPlane( + core: DurablePrpControlPlane, + identity?: DurableRecoveryIdentity, + ): Promise { + const registration = await this.options.controlPlaneRegistration!(core, identity); + const quiesce = registration.quiesce; + if (!quiesce) return registration; + this.#controlPlaneQuiesces.add(quiesce); + // Copy descriptors so a registration's getters (such as a lazily created + // route's failure) stay live; only release changes. + const release = registration.release; + return Object.defineProperties({}, { + ...Object.getOwnPropertyDescriptors(registration), + release: { + enumerable: true, + value: (reason?: "detach") => { + this.#controlPlaneQuiesces.delete(quiesce); + return release.call(registration, reason); + }, + }, + }) as ControlPlaneRegistration; + } + + /** The Runner's exit is committed: its routes' drops are no longer transport loss. */ + #quiesceControlPlaneRoutes(): void { + const routes = [...this.#controlPlaneQuiesces]; + this.#controlPlaneQuiesces.clear(); + for (const quiesce of routes) { + try { + quiesce(); + } catch (error) { + this.#diagnostic(`control-plane route quiesce failed: ${String(error)}`); + } + } + } + #quietReleasedRunner(): void { this.#stopPump(); if (this.#adoptedRunnerMonitor !== null) clearInterval(this.#adoptedRunnerMonitor); @@ -4834,7 +4887,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { await this.#claimReleasedRunner(claimantRunId); const core = this.#core!; if (core.activeRunnerConnectionCount() !== 1) { - const registration = await this.options.controlPlaneRegistration?.(core, this.#releaseEpisode!.source); + const registration = this.options.controlPlaneRegistration + ? await this.#registerControlPlane(core, this.#releaseEpisode!.source) : undefined; try { this.#assertStartupOpen(); if (!registration?.checkpoint) throw new Error("native_runner_released_cleanup_unavailable"); @@ -4949,7 +5003,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { itemId: input.itemId, }; const registration = this.options.controlPlaneRegistration - ? await this.options.controlPlaneRegistration(core, desired) + ? await this.#registerControlPlane(core, desired) : null; const previousRelease = this.#controlPlaneRelease; let previousReleased = false; @@ -5635,21 +5689,32 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { // the next runnerd on this root as a stale terminal command. const lostLocalRunner = this.#handle !== null && this.#lostRunnerHandle === this.#handle && !this.options.runnerProcessLauncher && !adoptedRunner; - runnerSuspended = !lostLocalRunner && await awaitRunnerSuspensionBarrier({ - commands: () => this.#core?.store.state.commands ?? [], - queueSuspend: (commandId) => { - this.#core?.queueCommand("runner.suspend", {}, commandId, true); - }, - readRunnerState: async () => { - const state = await this.#readDurableRunnerState(); - lastSuspensionState = state; - assertSuspendedRunnerState(state, this.#core!.store.state.identity); - return state; - }, - runnerHasExited: () => this.#runnerHasExited(), - pump: () => this.#pumpEventsSafely(), - deadline: closeDeadline, - }); + // Quiesce before the authority acknowledges a completed suspend: the + // Runner exits on that acknowledgement, possibly while the barrier is + // still reading durable state, and a Core-dialed route would otherwise + // redial it. The release path quiesces its route the same way. + const stopObservingExit = this.#core.onRunnerExitCommitted(() => this.#quiesceControlPlaneRoutes()); + try { + runnerSuspended = !lostLocalRunner && await awaitRunnerSuspensionBarrier({ + commands: () => this.#core?.store.state.commands ?? [], + queueSuspend: (commandId) => { + this.#core?.queueCommand("runner.suspend", {}, commandId, true); + }, + readRunnerState: async () => { + const state = await this.#readDurableRunnerState(); + lastSuspensionState = state; + assertSuspendedRunnerState(state, this.#core!.store.state.identity); + return state; + }, + runnerHasExited: () => this.#runnerHasExited(), + pump: () => this.#pumpEventsSafely(), + // Also covers a completed receipt acknowledged before this close. + onExitCommitted: () => this.#quiesceControlPlaneRoutes(), + deadline: closeDeadline, + }); + } finally { + stopObservingExit(); + } runnerLost = !runnerSuspended && this.#handle !== null && this.#lostRunnerHandle === this.#handle; if (!runnerSuspended) { const command = [...this.#core.store.state.commands].reverse().find( @@ -6269,7 +6334,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#openingCommandId = opening?.commandId ?? null; } const registration = this.options.controlPlaneRegistration - ? await this.options.controlPlaneRegistration(core) + ? await this.#registerControlPlane(core) : null; this.#startupFailureCode = registration?.startupFailureCode ?? "runner_local_connect_failed"; @@ -7040,9 +7105,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { && this.options.warmTransitionRegistrationMode === "routed_same_endpoint"))) { throw new Error("native_runner_warm_transition_requires_exact_owned_endpoint"); } - oldTransitionRegistration = await this.options.controlPlaneRegistration(core, recoverySource); + oldTransitionRegistration = await this.#registerControlPlane(core, recoverySource); this.#assertStartupOpen(); - newTransitionRegistration = await this.options.controlPlaneRegistration(core, recoveryTarget); + newTransitionRegistration = await this.#registerControlPlane(core, recoveryTarget); const connection = newTransitionRegistration.connection ?? { mode: "connect", connectUrl: newTransitionRegistration.connectUrl }; if (durableRecoveryInternals.canonicalJson(connection) !== durableRecoveryInternals.canonicalJson(expectedRecoveryConnection)) { throw new Error("native_runner_warm_transition_registered_endpoint_mismatch"); @@ -7056,7 +7121,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ? recoveryIdentityMatches(warmRecovery?.runnerIdentity ?? queuedSourceAttachment!.sourceIdentity, recoverySource!) ? oldTransitionRegistration : newTransitionRegistration : this.options.controlPlaneRegistration - ? await this.options.controlPlaneRegistration(core, this.#retainsSourceInputs() ? identity : this.#sourceRecoveryIdentity ?? undefined) : null; + ? await this.#registerControlPlane(core, this.#retainsSourceInputs() ? identity : this.#sourceRecoveryIdentity ?? undefined) : null; this.#startupFailureCode = registration?.startupFailureCode ?? "runner_local_connect_failed"; if (registration === null) { diff --git a/server/src/services/native-runtime/native-runner-execution-driver.test.ts b/server/src/services/native-runtime/native-runner-execution-driver.test.ts index f7579b6e0e0..ff214cd2bc3 100644 --- a/server/src/services/native-runtime/native-runner-execution-driver.test.ts +++ b/server/src/services/native-runtime/native-runner-execution-driver.test.ts @@ -183,6 +183,26 @@ describe("shared Runner execution driver", () => { expect(transport.connect.mock.results[0].value.close).toHaveBeenCalledOnce(); }); + it("quiesces its ingress when the Runner's exit is committed, before release closes it", async () => { + const quiesce = vi.fn(); + const close = vi.fn(async () => {}); + transport.connect.mockReturnValue({ ready: Promise.resolve(), quiesce, close }); + const driver = createNativeRunnerExecutionDriver({ + target: { ...sandbox(), getRunnerIngressEndpoint: async () => endpoint() } as AdapterExecutionTarget, + binding, runnerIngressAuthorized: true, prepareRemote: async () => {}, processStarted: Promise.resolve(), + }); + const connection = await driver.controlPlaneRegistration!(authority); + connection.quiesce!(); + expect(quiesce).not.toHaveBeenCalled(); + await connection.activate!(); + await connection.ready!(); + connection.quiesce!(); + expect(quiesce).toHaveBeenCalledOnce(); + expect(close).not.toHaveBeenCalled(); + await connection.release(); + expect(close).toHaveBeenCalledOnce(); + }); + it("does not fall back to outbound WSS when sandbox ingress is unauthorized", async () => { const prepare = vi.fn(); const acquire = vi.fn(); diff --git a/server/src/services/native-runtime/native-runner-execution-driver.ts b/server/src/services/native-runtime/native-runner-execution-driver.ts index f7c6980e202..95eba9c5515 100644 --- a/server/src/services/native-runtime/native-runner-execution-driver.ts +++ b/server/src/services/native-runtime/native-runner-execution-driver.ts @@ -329,6 +329,7 @@ export function createNativeRunnerExecutionDriver(input: NativeRunnerExecutionDr }, startupFailureCode: "runner_ingress_unavailable" as const, checkpoint: input.checkpoint, + quiesce: () => { outbound?.quiesce(); }, release: async () => { if (outbound) await outbound.close(); else await transport.ingress.close(); diff --git a/server/src/services/native-runtime/plugin-runner-task.test.ts b/server/src/services/native-runtime/plugin-runner-task.test.ts index 6ee44595a4e..2b7f08efdf9 100644 --- a/server/src/services/native-runtime/plugin-runner-task.test.ts +++ b/server/src/services/native-runtime/plugin-runner-task.test.ts @@ -438,6 +438,25 @@ describe("plugin native Runner ownership", () => { expect(task.mock.calls.find(([op]) => op.kind === "stop")).toBeUndefined(); await registration.release(); await registration.checkpoint("settled"); await handle.completion; }); + it("quiesces the Runner's ingress when its exit is committed, before release closes it", async () => { + const { task } = provider(); + const quiesce = vi.fn(); + const close = vi.fn(async () => {}); + ingress.connect.mockReset().mockReturnValue({ ready: Promise.resolve(), failure: new Promise(() => {}), quiesce, close }); + const owner = createPluginRunnerTaskOwner({ taskId: "task", projectAuthCheckVersion: 1, task, projectIds: [], harness: "codex", pollMs: 1 }); + await prepare(owner); + const registration = await owner.registration({} as never, "run"); + registration.quiesce(); + const handle = owner.launcher(launchSpec()); + registration.activate(); await registration.ready(); + registration.quiesce(); + expect(quiesce).toHaveBeenCalledOnce(); + expect(close).not.toHaveBeenCalled(); + expect(task.mock.calls.find(([op]) => op.kind === "stop")).toBeUndefined(); + await registration.release(); await registration.checkpoint("settled"); await handle.completion; + expect(close).toHaveBeenCalledOnce(); + }); + it("surfaces a proven admission rejection without polling or stopping the task", async () => { const { task } = provider(); const perform = task.getMockImplementation()!; diff --git a/server/src/services/native-runtime/plugin-runner-task.ts b/server/src/services/native-runtime/plugin-runner-task.ts index 91a9b491c30..9c9a8d2a3e7 100644 --- a/server/src/services/native-runtime/plugin-runner-task.ts +++ b/server/src/services/native-runtime/plugin-runner-task.ts @@ -343,6 +343,8 @@ export function createPluginRunnerTaskOwner(input: { if (identity) await input.onRetired?.(identity); } }, + // The Runner is about to exit (its close suspend completed), as on release. + quiesce: () => { outbound?.quiesce(); }, release: async (reason?: "detach") => { registrationReleased = true; if (generation === registrationGeneration) released = true;