mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
fix(runner): quiesce the Core-dialed route once the Runner commits its suspend (#21245)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Core runs agent sessions through the Paperclip Runner. With provider
ingress, Core dials the Runner over a PRP WebSocket.
> - When Core closes a session, it asks the Runner to suspend. The
Runner suspends, reports the result, and exits.
> - Core's ingress connection did not know that this exit was expected.
It treated the drop as a transport loss and redialed the Runner. Each
redial got a 404 until the close released the route.
> - The release (park) path already quiesces the connection before the
Runner goes away. The suspend path did not.
> - This pull request makes the suspend close quiesce the route in the
same way, once the Runner commits its suspend.
> - The benefit is that a clean suspend no longer looks like a lost
Runner. A drop before the suspend is committed still redials and
recovers as before.
## Linked Issues or Issue Description
**What happened?**
After a provider-ingress session closed cleanly, Core redialed the
Runner's ingress endpoint several times and got 404 each time. The
Runner state was `suspended`, `runner.drain` and `runner.suspend` had
both completed, and the Runner had exited with code 0. The redials
followed the outbound reconnect backoff (about 250, 500, 1000 and 2000
ms) and stopped when the registration release closed the outbound.
**Expected behavior**
After Core commits to `runner.suspend` and the Runner reports it
completed, the Runner's exit is expected. Core must not redial or
classify the drop as a transport loss. The release path already works
this way: it calls `outbound.quiesce()` before it releases the task.
**Steps to reproduce**
1. Open a session through a Core-dialed route (plugin provider ingress
or the `listen_ws` driver).
2. Close the session. Core drains, queues `runner.suspend`, and the
Runner suspends and exits.
3. See that the outbound reconnect loop redials the gone Runner until
the registration release closes it.
**Paperclip version or commit**
`master` at b4ca19f3a8.
**Agent adapter(s) involved**
Not adapter-specific (core bug). It affects any Core-dialed Runner
route: plugin provider ingress and the `listen_ws` driver.
**Additional context**
The redials do not fail the session. The plugin outbound uses a 7-minute
recovery grace, and `#failTransport` ignores failures after
`#closeAndSettle` sets `#closed`. The new test confirms this: without
the fix, the close still settles, but the route redials. This PR removes
the misleading redial. It does not change how a run's own failure is
classified.
## What Changed
- `RunnerExecutionDriver.controlPlaneRegistration` results can now
include an optional `quiesce()`. Core calls it when it has committed the
Runner's exit. The route then stops liveness and reconnect, and it keeps
the established receive/ACK path.
- `DurablePrpControlPlane.onRunnerExitCommitted(listener)` is new. The
authority calls its listeners after it saves a completed
`runner.suspend` or `runner.shutdown` result and before it sends the
acknowledgement. The Runner exits when it gets that acknowledgement, so
a route quiesced here never sees the exit as a loss.
- `awaitRunnerSuspensionBarrier` calls a new `onExitCommitted` callback
once, when its suspend command is `completed`. It also calls the
callback when it reuses a completed receipt and the durable state is
still `suspended`. A rejected, failed or unanswered suspend never calls
it.
- `DurablePrpCodexTransport` registers each control-plane route through
one `#registerControlPlane` helper. The helper records each route's
`quiesce` until that route is released. The suspend close subscribes to
`onRunnerExitCommitted` while its barrier runs and quiesces all of these
routes from that notification. The barrier's own `onExitCommitted`
covers a completed receipt that was acknowledged before this close.
Quiescing clears the tracked routes, so it runs once per close. The
helper copies property descriptors, so getters such as the `listen_ws`
driver's lazy `failure` stay live.
- The plugin task registration (`plugin-runner-task.ts`) and the
`listen_ws` driver registration (`native-runner-execution-driver.ts`)
implement `quiesce` with `outbound?.quiesce()`. This is the same call
that the release path makes.
## Verification
- New transport test with a real runnerd:
`packages/paperclip-runner/src/live/runnerd-codex-transport-suspend-route.test.ts`.
The test route acts like a Core-dialed provider ingress: on a socket
drop that it was not told to expect, it records a redial into a 404.
Before the fix the events were `connected, redial: 404, release`. After
the fix they are `connected, quiesce after runner.suspend completed,
release`. The close settles, and the durable Runner state is
`suspended`. A second case delays the durable-state read by 1.5 s, so
the Runner exits while the barrier still reads. That case fails without
the authority hook and passes with it.
- New barrier unit test in `runnerd-codex-transport.test.ts`. It shows
that the exit is committed only after the retried suspend completes,
never on a rejected or unanswered suspend, and on a reused receipt only
when the state is `suspended`.
- New server tests for the plugin task and `listen_ws` registrations.
`quiesce` reaches the active outbound and does not close it. Release
still closes it.
- Existing tests pass. These include the outbound release-path quiesce
test, the half-open and lease-expiry failure tests (a drop before the
commit still fails after its grace), and the plugin release/resume
facade tests.
- `pnpm verify` exited 2 (the change touches files outside the resolved
test graphs), so I ran focused suites. Locally: 8 runner transport files
and the 4 control-plane test files, 667 tests
(`runnerd-codex-transport*.test.ts` for suspension, release/resume,
remote retention, settlement, warm attachment and attachment recovery,
plus the new suspend-route test), and 5 server files, 148 tests (both
plugin Runner integration tests, the plugin task, the execution driver
and the outbound). Required CI supplies the broad coverage.
- `pnpm --filter @paperclipai/server typecheck` passes.
## Risks
- Low risk. `quiesce` is optional. Registrations that do not supply it
behave as before.
- The quiesce happens only after the Runner reports a completed suspend.
A socket drop before that point still redials and recovers.
- The quiesced route keeps its receive/ACK path, so the Runner's last
frames before exit are still acknowledged.
- No persisted state or protocol change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
62d673cc3f
commit
e65a3095a7
9 files changed
+329
-22
No files matched your search
@@ -2117,6 +2117,7 @@ export class DurablePrpControlPlane {
|
||||
#beforeAuthenticatedConnection?: DurablePrpControlPlaneOptions["beforeAuthenticatedConnection"];
|
||||
#onProtocolIntegrityError?: DurablePrpControlPlaneOptions["onProtocolIntegrityError"];
|
||||
#onAuthenticatedConnection?: DurablePrpControlPlaneOptions["onAuthenticatedConnection"];
|
||||
readonly #runnerExitListeners = new Set<() => void>();
|
||||
#protocolIntegrityError: NativeSessionProtocolIntegrityError | null = null;
|
||||
#eventDeliveryStall: { runId: string; ackedSourceSeq: number; since: number; failures: number } | null = null;
|
||||
#connectionLeaseTtlMs: number;
|
||||
@@ -2200,6 +2201,19 @@ export class DurablePrpControlPlane {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Notify when the Runner reports a completed suspend or shutdown. The
|
||||
* listener runs after the result is saved and before its acknowledgement;
|
||||
* the Runner exits once acknowledged, so a dropped connection after this
|
||||
* point is that exit, not a transport loss.
|
||||
*/
|
||||
onRunnerExitCommitted(listener: () => void): () => void {
|
||||
this.#runnerExitListeners.add(listener);
|
||||
return () => {
|
||||
this.#runnerExitListeners.delete(listener);
|
||||
};
|
||||
}
|
||||
|
||||
#readableStateChanged(): void {
|
||||
for (const listener of this.#readableStateListeners) {
|
||||
// A reader can never fail the commit that woke it.
|
||||
@@ -4215,6 +4229,15 @@ export class DurablePrpControlPlane {
|
||||
command.result = structuredClone(result);
|
||||
this.#store.save();
|
||||
if (turnStart) crashPoint("runner.turn_start.after_controller_journal_before_db_checkpoint");
|
||||
if (this.#isTerminalLifecycleCommand(command) && status === "completed") {
|
||||
for (const listener of this.#runnerExitListeners) {
|
||||
try {
|
||||
listener();
|
||||
} catch {
|
||||
// An observer can never change the acknowledgement.
|
||||
}
|
||||
}
|
||||
}
|
||||
this.#ackTerminalCommandResult(connection, command);
|
||||
this.#releaseServingLifecycle(connection, command);
|
||||
if (!this.#isTerminalLifecycleCommand(command) || connection.terminalLifecycleCommandId === null) {
|
||||
|
||||
@@ -48,6 +48,10 @@ export interface RunnerExecutionDriver {
|
||||
| "runner_ingress_unavailable";
|
||||
/** Persists only exact, independently verified suspended remote state. */
|
||||
checkpoint?: (settlement: "settled" | "unsettled") => Promise<void> | void;
|
||||
/** Core committed the Runner's exit (its close suspend completed). Stop
|
||||
* treating a dropped connection as transport loss, so a Core-dialed route
|
||||
* neither redials nor fails; keep the established receive/ACK path. */
|
||||
quiesce?: () => void;
|
||||
release: (reason?: "detach") => Promise<void> | void;
|
||||
}>;
|
||||
/** Register source and target checkpoint ownership before staging changes the stored run. */
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
import { mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import { createServer } from "node:http";
|
||||
import type { Socket } from "node:net";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { expect, it } from "vitest";
|
||||
import type { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js";
|
||||
import { createCapabilityRunnerdCodexTransport } from "./runnerd-codex-transport.js";
|
||||
import { defaultCapabilityRunnerdBinary, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js";
|
||||
|
||||
// A Core-dialed route (provider ingress) treats every socket drop it did not
|
||||
// cause as a transport loss: it redials, and the endpoint answers 404 once the
|
||||
// Runner process is gone. This route keeps that shape, so a drop it was not
|
||||
// told to expect shows up as a redial into a 404.
|
||||
function coreDialedRoute() {
|
||||
const events: string[] = [];
|
||||
let authority: DurablePrpControlPlane | null = null;
|
||||
let quiesced = false;
|
||||
let released = false;
|
||||
let rejectFailure!: (error: Error) => void;
|
||||
const failure = new Promise<never>((_resolve, reject) => { rejectFailure = reject; });
|
||||
void failure.catch(() => undefined);
|
||||
const server = createServer();
|
||||
server.on("upgrade", (request, socket: Socket, head) => {
|
||||
if (!authority) { socket.destroy(); return; }
|
||||
events.push("connected");
|
||||
socket.once("close", () => {
|
||||
if (quiesced || released) return;
|
||||
events.push("redial: 404");
|
||||
rejectFailure(new Error("runner_ingress_unavailable: Runner ingress connection failed."));
|
||||
});
|
||||
authority.handleUpgrade(request, socket, "/runner", head);
|
||||
});
|
||||
return {
|
||||
events,
|
||||
async listen() {
|
||||
await new Promise<void>(resolveListen => server.listen(0, "127.0.0.1", resolveListen));
|
||||
const address = server.address();
|
||||
if (!address || typeof address === "string") throw new Error("Missing route listener");
|
||||
return address.port;
|
||||
},
|
||||
registration(port: number) {
|
||||
return async (next: DurablePrpControlPlane) => {
|
||||
authority = next;
|
||||
return {
|
||||
connectUrl: `ws://127.0.0.1:${port}/runner`,
|
||||
failure,
|
||||
quiesce: () => {
|
||||
const suspend = next.store.state.commands.findLast(command => command.type === "runner.suspend");
|
||||
events.push(`quiesce after runner.suspend ${suspend?.status ?? "missing"}`);
|
||||
quiesced = true;
|
||||
},
|
||||
release: () => {
|
||||
events.push("release");
|
||||
released = true;
|
||||
if (authority === next) authority = null;
|
||||
},
|
||||
};
|
||||
};
|
||||
},
|
||||
async close() {
|
||||
server.closeAllConnections();
|
||||
if (server.listening) await new Promise<void>(resolveClose => server.close(() => resolveClose()));
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
it.each([
|
||||
{ title: "", stateReadDelayMs: 0 },
|
||||
// A remote durable-state read can outlast the Runner: it exits as soon as
|
||||
// Core acknowledges its suspend result, while the barrier is still reading.
|
||||
{ title: " while the barrier is still reading its durable state", stateReadDelayMs: 1_500 },
|
||||
])("quiesces a Core-dialed route once the Runner commits its suspend$title, so its exit is not a transport loss", async ({ stateReadDelayMs }) => {
|
||||
const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-suspend-route-"));
|
||||
const route = coreDialedRoute();
|
||||
const port = await route.listen();
|
||||
const bundle = createCapabilityRunnerdCodexTransport({
|
||||
runnerBinary: defaultCapabilityRunnerdBinary(),
|
||||
codexCommand: fakeCodex,
|
||||
codexArgs: fakeCodexArgs(stateDirectory),
|
||||
stateDirectory,
|
||||
controlPlaneRegistration: route.registration(port),
|
||||
...(stateReadDelayMs > 0 ? {
|
||||
readRunnerState: async () => {
|
||||
await new Promise(resolveRead => setTimeout(resolveRead, stateReadDelayMs));
|
||||
return JSON.parse(await readFile(join(stateDirectory, "runner", "runner-state.json"), "utf8")) as Record<string, unknown>;
|
||||
},
|
||||
} : {}),
|
||||
});
|
||||
bundle.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] }));
|
||||
try {
|
||||
await bundle.transport.request("initialize", {});
|
||||
await bundle.transport.request("thread/start", { cwd: stateDirectory, dynamicTools: [] });
|
||||
// Before the suspend is committed, a drop is still an ordinary loss.
|
||||
expect(route.events).toEqual(["connected"]);
|
||||
|
||||
await expect(bundle.transport.close("native session execution complete")).resolves.toBeUndefined();
|
||||
|
||||
expect(route.events).toEqual(["connected", "quiesce after runner.suspend completed", "release"]);
|
||||
const runnerState = JSON.parse(await readFile(join(stateDirectory, "runner", "runner-state.json"), "utf8")) as Record<string, unknown>;
|
||||
expect(runnerState.lifecycle).toBe("suspended");
|
||||
expect(bundle.evidence().runnerExited).toBe(true);
|
||||
} finally {
|
||||
await bundle.transport.close().catch(() => undefined);
|
||||
await route.close();
|
||||
await rm(stateDirectory, { recursive: true, force: true });
|
||||
}
|
||||
}, 30_000);
|
||||
@@ -2526,6 +2526,71 @@ it("does not accept process exit without durable suspension", async () => {
|
||||
).resolves.toBe(false);
|
||||
});
|
||||
|
||||
it("commits the Runner's exit only once its suspend command completes", async () => {
|
||||
const commands: Array<{ commandId: string; type: string; status: string }> = [];
|
||||
const order: string[] = [];
|
||||
let lifecycle = "ready";
|
||||
let pumps = 0;
|
||||
const exitCommitted = vi.fn(() => { order.push(`exit committed after ${commands.length} attempts`); });
|
||||
await expect(
|
||||
runnerdRecoveryInternals.awaitRunnerSuspensionBarrier({
|
||||
commands: () => commands,
|
||||
queueSuspend: commandId => { commands.push({ commandId, type: "runner.suspend", status: "pending" }); },
|
||||
readRunnerState: async () => {
|
||||
order.push(`state ${lifecycle}`);
|
||||
return { lifecycle };
|
||||
},
|
||||
runnerHasExited: async () => false,
|
||||
pump: () => {
|
||||
pumps += 1;
|
||||
const pending = commands.find(command => command.status === "pending");
|
||||
if (!pending) return;
|
||||
// The first attempt is rejected; the retry stays pending for one
|
||||
// round before the Runner reports it completed.
|
||||
if (commands.length === 1) pending.status = "failed";
|
||||
else if (pumps >= 4) { pending.status = "completed"; lifecycle = "suspended"; }
|
||||
},
|
||||
onExitCommitted: exitCommitted,
|
||||
deadline: Date.now() + 2_000,
|
||||
pollIntervalMs: 0,
|
||||
}),
|
||||
).resolves.toBe(true);
|
||||
expect(exitCommitted).toHaveBeenCalledOnce();
|
||||
expect(order.indexOf("exit committed after 2 attempts")).toBeGreaterThan(order.indexOf("state ready"));
|
||||
expect(order.at(-1)).toBe("state suspended");
|
||||
|
||||
// A reused completed receipt commits the exit only once the durable state
|
||||
// proves the Runner is still suspended.
|
||||
for (const [state, committed] of [["suspended", true], ["ready", false]] as const) {
|
||||
const reusedCommitted = vi.fn();
|
||||
await runnerdRecoveryInternals.awaitRunnerSuspensionBarrier({
|
||||
commands: () => [{ commandId: "command_prior_suspend", type: "runner.suspend", status: "completed" }],
|
||||
queueSuspend: () => undefined,
|
||||
readRunnerState: async () => ({ lifecycle: state }),
|
||||
runnerHasExited: async () => true,
|
||||
pump: () => undefined,
|
||||
onExitCommitted: reusedCommitted,
|
||||
deadline: Date.now() + 20,
|
||||
pollIntervalMs: 0,
|
||||
});
|
||||
expect(reusedCommitted).toHaveBeenCalledTimes(committed ? 1 : 0);
|
||||
}
|
||||
|
||||
// A rejected or unanswered suspend never commits the exit.
|
||||
const neverCommitted = vi.fn();
|
||||
await expect(runnerdRecoveryInternals.awaitRunnerSuspensionBarrier({
|
||||
commands: () => commands.slice(0, 0),
|
||||
queueSuspend: () => undefined,
|
||||
readRunnerState: async () => ({ lifecycle: "ready" }),
|
||||
runnerHasExited: async () => false,
|
||||
pump: () => undefined,
|
||||
onExitCommitted: neverCommitted,
|
||||
deadline: Date.now() + 20,
|
||||
pollIntervalMs: 0,
|
||||
})).resolves.toBe(false);
|
||||
expect(neverCommitted).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("reserves a bounded suspension window after close preparation", () => {
|
||||
expect(runnerdRecoveryInternals.runnerCloseDeadlines(1_000, 10_000)).toEqual({
|
||||
preparationDeadline: 8_500,
|
||||
|
||||
@@ -1024,6 +1024,8 @@ function turnStartCommandResultValid(input: {
|
||||
);
|
||||
}
|
||||
|
||||
type ControlPlaneRegistration = Awaited<ReturnType<NonNullable<RunnerExecutionDriver["controlPlaneRegistration"]>>>;
|
||||
|
||||
type CloseCheckpointRetry = {
|
||||
checkpoint: () => Promise<void> | void;
|
||||
finish: (deadline: number) => Promise<void>;
|
||||
@@ -1079,10 +1081,18 @@ async function awaitRunnerSuspensionBarrier(input: {
|
||||
readRunnerState: () => Promise<Record<string, unknown>>;
|
||||
runnerHasExited: () => Promise<boolean>;
|
||||
pump: () => void;
|
||||
/** Called once the Runner's exit is certain: its suspend command completed. */
|
||||
onExitCommitted?: () => void;
|
||||
deadline: number;
|
||||
pollIntervalMs?: number;
|
||||
}): Promise<boolean> {
|
||||
if (Date.now() >= input.deadline) return false;
|
||||
let exitCommitted = false;
|
||||
const commitExit = () => {
|
||||
if (exitCommitted) return;
|
||||
exitCommitted = true;
|
||||
input.onExitCommitted?.();
|
||||
};
|
||||
const readRunnerState = async (): Promise<Record<string, unknown> | undefined> => {
|
||||
const remainingMs = input.deadline - Date.now();
|
||||
if (remainingMs <= 0) return undefined;
|
||||
@@ -1112,8 +1122,10 @@ async function awaitRunnerSuspensionBarrier(input: {
|
||||
try {
|
||||
const state = await readRunnerState();
|
||||
if (state === undefined) return false;
|
||||
if (state.lifecycle === "suspended")
|
||||
if (state.lifecycle === "suspended") {
|
||||
commitExit();
|
||||
return Date.now() < input.deadline;
|
||||
}
|
||||
} catch {
|
||||
// The normal bounded barrier below handles unavailable state.
|
||||
}
|
||||
@@ -1156,6 +1168,9 @@ async function awaitRunnerSuspensionBarrier(input: {
|
||||
input.queueSuspend(commandId);
|
||||
continue;
|
||||
}
|
||||
// The Runner exits right after it reports a completed suspend, so its exit
|
||||
// is certain before this barrier reads the durable state.
|
||||
if (command?.status === "completed") commitExit();
|
||||
let lifecycle: unknown;
|
||||
try {
|
||||
const state = await readRunnerState();
|
||||
@@ -4131,6 +4146,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
((settlement: "settled" | "unsettled") => Promise<void> | void) | null =
|
||||
null;
|
||||
#controlPlaneRelease: ((reason?: "detach") => Promise<void> | void) | null = null;
|
||||
/** Quiesce hooks of the registered routes that have not been released yet. */
|
||||
readonly #controlPlaneQuiesces = new Set<() => void>();
|
||||
#nextTraceDebugSequence = 1;
|
||||
#traceRehydrationSpoolOverflow = false;
|
||||
readonly #traceFrameIndex = new RunnerdTraceFrameIndex();
|
||||
@@ -4720,6 +4737,42 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
);
|
||||
}
|
||||
|
||||
async #registerControlPlane(
|
||||
core: DurablePrpControlPlane,
|
||||
identity?: DurableRecoveryIdentity,
|
||||
): Promise<ControlPlaneRegistration> {
|
||||
const registration = await this.options.controlPlaneRegistration!(core, identity);
|
||||
const quiesce = registration.quiesce;
|
||||
if (!quiesce) return registration;
|
||||
this.#controlPlaneQuiesces.add(quiesce);
|
||||
// Copy descriptors so a registration's getters (such as a lazily created
|
||||
// route's failure) stay live; only release changes.
|
||||
const release = registration.release;
|
||||
return Object.defineProperties({}, {
|
||||
...Object.getOwnPropertyDescriptors(registration),
|
||||
release: {
|
||||
enumerable: true,
|
||||
value: (reason?: "detach") => {
|
||||
this.#controlPlaneQuiesces.delete(quiesce);
|
||||
return release.call(registration, reason);
|
||||
},
|
||||
},
|
||||
}) as ControlPlaneRegistration;
|
||||
}
|
||||
|
||||
/** The Runner's exit is committed: its routes' drops are no longer transport loss. */
|
||||
#quiesceControlPlaneRoutes(): void {
|
||||
const routes = [...this.#controlPlaneQuiesces];
|
||||
this.#controlPlaneQuiesces.clear();
|
||||
for (const quiesce of routes) {
|
||||
try {
|
||||
quiesce();
|
||||
} catch (error) {
|
||||
this.#diagnostic(`control-plane route quiesce failed: ${String(error)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#quietReleasedRunner(): void {
|
||||
this.#stopPump();
|
||||
if (this.#adoptedRunnerMonitor !== null) clearInterval(this.#adoptedRunnerMonitor);
|
||||
@@ -4834,7 +4887,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
await this.#claimReleasedRunner(claimantRunId);
|
||||
const core = this.#core!;
|
||||
if (core.activeRunnerConnectionCount() !== 1) {
|
||||
const registration = await this.options.controlPlaneRegistration?.(core, this.#releaseEpisode!.source);
|
||||
const registration = this.options.controlPlaneRegistration
|
||||
? await this.#registerControlPlane(core, this.#releaseEpisode!.source) : undefined;
|
||||
try {
|
||||
this.#assertStartupOpen();
|
||||
if (!registration?.checkpoint) throw new Error("native_runner_released_cleanup_unavailable");
|
||||
@@ -4949,7 +5003,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
itemId: input.itemId,
|
||||
};
|
||||
const registration = this.options.controlPlaneRegistration
|
||||
? await this.options.controlPlaneRegistration(core, desired)
|
||||
? await this.#registerControlPlane(core, desired)
|
||||
: null;
|
||||
const previousRelease = this.#controlPlaneRelease;
|
||||
let previousReleased = false;
|
||||
@@ -5635,21 +5689,32 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
// the next runnerd on this root as a stale terminal command.
|
||||
const lostLocalRunner = this.#handle !== null && this.#lostRunnerHandle === this.#handle
|
||||
&& !this.options.runnerProcessLauncher && !adoptedRunner;
|
||||
runnerSuspended = !lostLocalRunner && await awaitRunnerSuspensionBarrier({
|
||||
commands: () => this.#core?.store.state.commands ?? [],
|
||||
queueSuspend: (commandId) => {
|
||||
this.#core?.queueCommand("runner.suspend", {}, commandId, true);
|
||||
},
|
||||
readRunnerState: async () => {
|
||||
const state = await this.#readDurableRunnerState();
|
||||
lastSuspensionState = state;
|
||||
assertSuspendedRunnerState(state, this.#core!.store.state.identity);
|
||||
return state;
|
||||
},
|
||||
runnerHasExited: () => this.#runnerHasExited(),
|
||||
pump: () => this.#pumpEventsSafely(),
|
||||
deadline: closeDeadline,
|
||||
});
|
||||
// Quiesce before the authority acknowledges a completed suspend: the
|
||||
// Runner exits on that acknowledgement, possibly while the barrier is
|
||||
// still reading durable state, and a Core-dialed route would otherwise
|
||||
// redial it. The release path quiesces its route the same way.
|
||||
const stopObservingExit = this.#core.onRunnerExitCommitted(() => this.#quiesceControlPlaneRoutes());
|
||||
try {
|
||||
runnerSuspended = !lostLocalRunner && await awaitRunnerSuspensionBarrier({
|
||||
commands: () => this.#core?.store.state.commands ?? [],
|
||||
queueSuspend: (commandId) => {
|
||||
this.#core?.queueCommand("runner.suspend", {}, commandId, true);
|
||||
},
|
||||
readRunnerState: async () => {
|
||||
const state = await this.#readDurableRunnerState();
|
||||
lastSuspensionState = state;
|
||||
assertSuspendedRunnerState(state, this.#core!.store.state.identity);
|
||||
return state;
|
||||
},
|
||||
runnerHasExited: () => this.#runnerHasExited(),
|
||||
pump: () => this.#pumpEventsSafely(),
|
||||
// Also covers a completed receipt acknowledged before this close.
|
||||
onExitCommitted: () => this.#quiesceControlPlaneRoutes(),
|
||||
deadline: closeDeadline,
|
||||
});
|
||||
} finally {
|
||||
stopObservingExit();
|
||||
}
|
||||
runnerLost = !runnerSuspended && this.#handle !== null && this.#lostRunnerHandle === this.#handle;
|
||||
if (!runnerSuspended) {
|
||||
const command = [...this.#core.store.state.commands].reverse().find(
|
||||
@@ -6269,7 +6334,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
this.#openingCommandId = opening?.commandId ?? null;
|
||||
}
|
||||
const registration = this.options.controlPlaneRegistration
|
||||
? await this.options.controlPlaneRegistration(core)
|
||||
? await this.#registerControlPlane(core)
|
||||
: null;
|
||||
this.#startupFailureCode =
|
||||
registration?.startupFailureCode ?? "runner_local_connect_failed";
|
||||
@@ -7040,9 +7105,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
&& this.options.warmTransitionRegistrationMode === "routed_same_endpoint"))) {
|
||||
throw new Error("native_runner_warm_transition_requires_exact_owned_endpoint");
|
||||
}
|
||||
oldTransitionRegistration = await this.options.controlPlaneRegistration(core, recoverySource);
|
||||
oldTransitionRegistration = await this.#registerControlPlane(core, recoverySource);
|
||||
this.#assertStartupOpen();
|
||||
newTransitionRegistration = await this.options.controlPlaneRegistration(core, recoveryTarget);
|
||||
newTransitionRegistration = await this.#registerControlPlane(core, recoveryTarget);
|
||||
const connection = newTransitionRegistration.connection ?? { mode: "connect", connectUrl: newTransitionRegistration.connectUrl };
|
||||
if (durableRecoveryInternals.canonicalJson(connection) !== durableRecoveryInternals.canonicalJson(expectedRecoveryConnection)) {
|
||||
throw new Error("native_runner_warm_transition_registered_endpoint_mismatch");
|
||||
@@ -7056,7 +7121,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
? recoveryIdentityMatches(warmRecovery?.runnerIdentity ?? queuedSourceAttachment!.sourceIdentity, recoverySource!)
|
||||
? oldTransitionRegistration : newTransitionRegistration
|
||||
: this.options.controlPlaneRegistration
|
||||
? await this.options.controlPlaneRegistration(core, this.#retainsSourceInputs() ? identity : this.#sourceRecoveryIdentity ?? undefined) : null;
|
||||
? await this.#registerControlPlane(core, this.#retainsSourceInputs() ? identity : this.#sourceRecoveryIdentity ?? undefined) : null;
|
||||
this.#startupFailureCode =
|
||||
registration?.startupFailureCode ?? "runner_local_connect_failed";
|
||||
if (registration === null) {
|
||||
|
||||
@@ -183,6 +183,26 @@ describe("shared Runner execution driver", () => {
|
||||
expect(transport.connect.mock.results[0].value.close).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("quiesces its ingress when the Runner's exit is committed, before release closes it", async () => {
|
||||
const quiesce = vi.fn();
|
||||
const close = vi.fn(async () => {});
|
||||
transport.connect.mockReturnValue({ ready: Promise.resolve(), quiesce, close });
|
||||
const driver = createNativeRunnerExecutionDriver({
|
||||
target: { ...sandbox(), getRunnerIngressEndpoint: async () => endpoint() } as AdapterExecutionTarget,
|
||||
binding, runnerIngressAuthorized: true, prepareRemote: async () => {}, processStarted: Promise.resolve(),
|
||||
});
|
||||
const connection = await driver.controlPlaneRegistration!(authority);
|
||||
connection.quiesce!();
|
||||
expect(quiesce).not.toHaveBeenCalled();
|
||||
await connection.activate!();
|
||||
await connection.ready!();
|
||||
connection.quiesce!();
|
||||
expect(quiesce).toHaveBeenCalledOnce();
|
||||
expect(close).not.toHaveBeenCalled();
|
||||
await connection.release();
|
||||
expect(close).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("does not fall back to outbound WSS when sandbox ingress is unauthorized", async () => {
|
||||
const prepare = vi.fn();
|
||||
const acquire = vi.fn();
|
||||
|
||||
@@ -329,6 +329,7 @@ export function createNativeRunnerExecutionDriver(input: NativeRunnerExecutionDr
|
||||
},
|
||||
startupFailureCode: "runner_ingress_unavailable" as const,
|
||||
checkpoint: input.checkpoint,
|
||||
quiesce: () => { outbound?.quiesce(); },
|
||||
release: async () => {
|
||||
if (outbound) await outbound.close();
|
||||
else await transport.ingress.close();
|
||||
|
||||
@@ -438,6 +438,25 @@ describe("plugin native Runner ownership", () => {
|
||||
expect(task.mock.calls.find(([op]) => op.kind === "stop")).toBeUndefined();
|
||||
await registration.release(); await registration.checkpoint("settled"); await handle.completion;
|
||||
});
|
||||
it("quiesces the Runner's ingress when its exit is committed, before release closes it", async () => {
|
||||
const { task } = provider();
|
||||
const quiesce = vi.fn();
|
||||
const close = vi.fn(async () => {});
|
||||
ingress.connect.mockReset().mockReturnValue({ ready: Promise.resolve(), failure: new Promise<never>(() => {}), quiesce, close });
|
||||
const owner = createPluginRunnerTaskOwner({ taskId: "task", projectAuthCheckVersion: 1, task, projectIds: [], harness: "codex", pollMs: 1 });
|
||||
await prepare(owner);
|
||||
const registration = await owner.registration({} as never, "run");
|
||||
registration.quiesce();
|
||||
const handle = owner.launcher(launchSpec());
|
||||
registration.activate(); await registration.ready();
|
||||
registration.quiesce();
|
||||
expect(quiesce).toHaveBeenCalledOnce();
|
||||
expect(close).not.toHaveBeenCalled();
|
||||
expect(task.mock.calls.find(([op]) => op.kind === "stop")).toBeUndefined();
|
||||
await registration.release(); await registration.checkpoint("settled"); await handle.completion;
|
||||
expect(close).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("surfaces a proven admission rejection without polling or stopping the task", async () => {
|
||||
const { task } = provider();
|
||||
const perform = task.getMockImplementation()!;
|
||||
|
||||
@@ -343,6 +343,8 @@ export function createPluginRunnerTaskOwner(input: {
|
||||
if (identity) await input.onRetired?.(identity);
|
||||
}
|
||||
},
|
||||
// The Runner is about to exit (its close suspend completed), as on release.
|
||||
quiesce: () => { outbound?.quiesce(); },
|
||||
release: async (reason?: "detach") => {
|
||||
registrationReleased = true;
|
||||
if (generation === registrationGeneration) released = true;
|
||||
|
||||
Reference in new issue
Block a user