fix(runner): answer a live Runner's commands while it replays its backlog after a reconnect (#21442)

**DO NOT MERGE until the coordinator confirms the Daytona validation.**

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox is driven by a Runner process. The
server talks to it over PRP and saves the controller state in a
checkpoint.
> - After a release swap, the incoming server reconnects to Runners that
kept working while no server was attached. Each Runner first replays the
events it built up.
> - The server answered commands one at a time and only after every
event queued ahead of them. It wrote one checkpoint and one
acknowledgement per event. A recovery read sent at reconnect therefore
timed out at 30 s behind the replay, although the Runner was alive and
had answered.
> - A tool call that the previous server started and never answered
blocked every later command, and a failed close was replayed forever.
> - This pull request lets read and barrier answers pass the replay,
extends a wait while the replay still commits, acknowledges a replayed
burst once per batch, delivers every pending command at attach, settles
an orphaned tool call at attach, and retries a close whose drain timed
out.
> - The benefit is that a live Runner is reached after a late reconnect
instead of being declared unanswered and then replaced, which lost its
turn.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

Authentication took under 1 s. The identity was recorded 4 to 7 minutes
later, because it is written only after the Runner answers `run.attach`,
`runner.drain` and the thread read. Those answers waited behind the
replayed event backlog. The failure codes on the first attempt were:

- `PRP command session.snapshot timed out` and
`codex_history_read_failed: thread/turns/list read failed` (all 10
`ingress` runs in the sample)
- `PRP command runner.drain timed out (runner connected; pending ahead 2
[semantic_tool.result] oldest 1023s)` on the retry

A separate class, `cleanup_quarantined` (23 failures in 3 hours, no swap
involved), showed "prior session cleanup remains incomplete" on every
retry, because a close that failed once kept replaying the same
rejection.

A live Daytona capture showed the stale `semantic_tool.result`: Codex
waited for the result of an exec tool call that the stopped server
owned. It was not an unacknowledged frame.

**Expected behavior**

A live Runner answers recovery commands within its deadline after a late
reconnect. A tool call left by an earlier controller is settled once
(with its durable receipt, or as interrupted with an unknown outcome,
never executed again). A close that failed only because its journal
drain timed out reaches its real outcome on the next close.

**Steps to reproduce**

1. Start a control plane with a commit callback that takes 50 ms per
event.
2. Connect a Runner that replays 1,000 events, then send
`session.snapshot`.
3. On master the snapshot times out at its 30 s deadline. The new test
`answers a command issued during a 1,000-event replay within its
deadline` does this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

All changes are controller-side TypeScript under
`packages/paperclip-runner/src`.

- `durable-prp-control-plane.ts`: a `command_result` for
`session.snapshot`, `session.goal.get`, `runner.drain` or
`semantic_tool.result` that reports completed or rejected is applied
when it arrives, not after the events queued ahead of it. Failed
results, lifecycle, attachment and turn commands keep their place.
- `durable-prp-control-plane.ts`: the welcome carries every consecutive
pending command (up to 32, within the existing byte bound). A lifecycle
command or `run.attach` that is not first is never batched.
- `durable-prp-control-plane.ts`: an ordinary event inside a replayed
burst is committed at once but checkpointed and acknowledged once per
batch (when the queue drains, when a critical event arrives, or after 64
events). An ACK is still sent only after the checkpoint that contains
its cursor. The owner fence in `flushCheckpoint` is unchanged. Critical
events keep their own checkpoint and ACK.
- `runnerd-codex-transport.ts`: `waitForRunnerCommand` also extends a
deadline while the controller keeps committing the Runner's events. A
Runner with no committed event for 30 s still times out at the original
deadline. A Runner that keeps emitting but never answers times out after
at most 5 more minutes.
- `runnerd-codex-transport.ts`: recovery of a semantic tool input that
an earlier generation dispatched and never answered now also runs when
the controller attaches, not only after startup completes.
`paperclip_finish` and `paperclip_block` still wait for startup.
- `runnerd-codex-transport.ts`: a close that failed only because the
controller journal drain timed out is no longer cached. The next close
drains again with a fresh bound and finishes as settled or as a typed
`NativeSessionCloseUnrecoverableError`.

## Verification

Red on master, green on this branch:

- `durable-prp-control-plane.test.ts`
- `answers a command issued during a 1,000-event replay within its
deadline`
- `keeps a failed command result behind the events the Runner sent
before it`
- `delivers every command pending at attach so an earlier generation's
result cannot hold the recovery probe`
  - `never delivers a lifecycle command ahead of its turn at attach`
- `acknowledges a replayed burst once per batch, never ahead of its
durable checkpoint`
- `still acknowledges a lifecycle event inside a burst before the next
frame`
- `answers a snapshot from a Runner whose provider stays wedged until
its whole replay is acknowledged` (models the Daytona capture: the Codex
Runner stops reading provider output while it holds unacknowledged
events)
- `runnerd-codex-transport-reattach-carried-commands.test.ts`
- `keeps waiting past the deadline while the controller is still
committing the Runner's replayed events`
- `times out a Runner that keeps replaying but never answers once the
replay extension is spent`
- `does not extend a deadline for a Runner whose replay has stopped
advancing`
- `runnerd-codex-transport-orphaned-tool-at-attach.test.ts`: `settles an
earlier generation's in-flight tool call at attach even when that attach
never finishes startup`
- `runnerd-detach-durability.test.ts`: new case `close_timeout` in
`joins an output flush before controller retirement permits a successor
writer`

Measured answer latency for a snapshot sent during a replay (50 ms
commit callback): 100 / 1,000 / 10,000 events took 13.95 s / more than
900 s / did not finish on master, and 26 ms / 26 ms / 170 ms with this
branch.

Local runs on this branch: `tsc --noEmit` is clean for
`packages/paperclip-runner` and `server`.
`durable-prp-control-plane.test.ts`: 227 of 227 pass.
`runnerd-codex-transport-reattach-carried-commands`, `-adoption`,
`-orphaned-tool-at-attach` and `runnerd-detach-durability`: all pass.
`runnerd-codex-transport-settlement` and `-deadlines`: the run on this
shared machine (load 50 to 96) had 8 tests fail. A rerun of those two
files alone (load 40 to 72) still had 7 failures: one timing assertion
(`expected 7419 to be less than 3500`) and six settlement cases that
ended with `native_cleanup_maintenance_unproven` after timeouts. The set
of failing cases changed between the two runs. The CI Runner support
suite ran both files on this head and passed, so I rely on CI for them.

Runner content ID: this PR changes only controller-side TypeScript under
`packages/paperclip-runner/src`. There is no Rust, `protocol/` or
`generated/` change. The Runner content ID changes because its hash
covers that source, but existing Runner images work with this
controller. The Daytona cells below ran this controller against an image
built for an earlier content ID.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR and
three others (the reattach-first PR (`fix/reattach-live-runs-first`),
the adopt-live-Runner PR (`fix/adopt-live-runner-on-retry`), and #21362,
which is merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755; 480 s delay injected on the first
identity write per run): one Claude run reproduced the staging chain:
authentication 419 ms, identity +423.5 s, attempt 1 `native session
provider recovery timed out after 900000ms`, then the retry failed with
`runner_harness_state_mismatch: resumed_not_suspended`.
- G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished
hands-off. That chain did not recur. All 6 runs that crossed swap 2 were
adopted on the same Runner process, with no retry and no 900 s timeout.
- Caveat: G5c ran before the reattach-phase timing fix in the
reattach-first PR (`fix/reattach-live-runs-first`). The command fixes in
this PR were present in G5c except the test-only fix for the
wedged-provider test.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection): 0
failed runs at either swap; 11 of 11 survivors at both. Swap 2 was slow
(see the reattach-first PR (`fix/reattach-live-runs-first`)).

## Risks

- Not proven: the Runner-side stall. A Codex Runner stops reading
provider output while no controller is attached (bounded 256-line
channel, provider stdout pipe full). This PR makes the controller
consume the replay quickly, so the stall clears in about 2 to 6 s once
the controller drives it. The durable fix is in the Runner (keep
draining provider output to the durable outbox while disconnected, save
once per ACK batch, run commands ahead of ACKs). That needs a new Runner
image and is not in this PR.
- Not proven: why a connected Runner with an empty outbox sometimes did
not answer for more than 30 s. The G5 capture points to the server
stalling past its own deadline (a `session.open` answered and
acknowledged 8 minutes before the server reported it timed out), not to
the Runner.
- Not done: cleanup takeover for a same-run retry when
`remoteCleanupScope` is absent, and for a predecessor run that shares
the warm sandbox. This PR only stops the replayed close failure.
- Time to reach a wedged Codex Runner still grows with its backlog:
about backlog x per-event commit / database concurrency (40 runs x 1,000
events x 20 ms / 8 is about 100 s).
- Behavior shift: a crash during a batched burst replays at most the
unacknowledged batch (up to 64 events). The Runner still holds those
events, and commit order is unchanged.
- A Runner that keeps emitting but never answers now fails after at most
5 more minutes instead of 30 s.
- The predecessor-retirement PR (`fix/retire-predecessor-session`)
builds on the close path changed here (`retry a close whose controller
journal drain timed out`).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code. The fixes were written by Claude agents in the
same session family.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 12:34:31 -06:00
1 parent 89e6575418
commit d9b977baf2
6 files changed
+578 -21

No files matched your search

@@ -24,7 +24,7 @@ import { describe, expect, it, vi } from "vitest";
import { configuredEnvironmentProjection } from "../configured-environment.js";
import { NativeSessionProtocolIntegrityError } from "../contracts/native-session-backend.js";
import { createCapabilityRunnerdCodexTransport, createCapabilityRunnerdProviderEnvironment } from "../live/runnerd-codex-transport.js";
import { createCapabilityRunnerdCodexTransport, createCapabilityRunnerdProviderEnvironment, waitForRunnerCommand } from "../live/runnerd-codex-transport.js";
import { ACPX_CREDENTIAL_BINDING_ENV, createAcpxSidecarHostEnvironment } from "../drivers/acpx/environment.js";
import { validatePrpEvent } from "../protocol/replay-contract.js";
import { digestPaperclipSemanticContent } from "../semantic-tools/receipts.js";
@@ -1863,6 +1863,245 @@ it("answers lease renewal while an earlier event commit is still in flight", asy
}
});
function replayBacklogCore(root: string, commitMs: number) {
let stopped = false;
const commits = vi.fn(async () => {
if (stopped) throw new Error("replay_test_stopped");
await new Promise<void>(resolveCommit => setTimeout(resolveCommit, commitMs));
});
const core = new DurablePrpControlPlane({ stateDirectory: root, identity,
expectedRunnerVersion, expectedRunnerDigest, onCommittedEvent: commits });
return { core, commits, stop: () => { stopped = true; } };
}
function commandResultFrame(client: AuthenticatedClient, command: { commandId: string; type: string; controllerSeq: number },
result: Record<string, unknown> = { status: "completed" }, status = "completed"): Record<string, unknown> {
return { protocol: "paperclip.runner", version: client.welcome.version, kind: "command_result",
payload: { commandId: command.commandId, commandType: command.type, controllerSeq: command.controllerSeq, status, result } };
}
async function receiveCommand(client: AuthenticatedClient): Promise<Record<string, unknown>> {
for (;;) {
const frame = await receiveSecure(client);
if (frame === null) throw new Error("connection closed before the command arrived");
if (frame.kind === "command") return frame;
}
}
it("answers a command issued during a 1,000-event replay within its deadline", async () => {
const root = mkdtempSync(resolve(tmpdir(), "runner-command-under-replay-"));
const { core, commits, stop } = replayBacklogCore(root, 50);
try {
await core.start();
const client = (await authenticate(core, core.issueBootstrapTicket()))!;
for (let seq = 1; seq <= 1_000; seq++) sendSecure(client, ordinaryOutput(seq));
await vi.waitFor(() => expect(commits).toHaveBeenCalled());
const snapshot = core.queueCommand("session.snapshot", {}, "snapshot-during-replay", true);
expect((await receiveCommand(client)).payload).toMatchObject({ commandId: snapshot.commandId });
sendSecure(client, commandResultFrame(client, snapshot));
await vi.waitFor(() => expect(core.commandOutcome(snapshot.commandId)?.status).toBe("completed"),
{ timeout: 30_000, interval: 10 });
expect(core.store.state.ackedSourceSeq).toBeLessThan(1_000);
client.socket.destroy();
} finally {
stop();
await core.stop();
await core.drainPendingConnectionProcessing();
rmSync(root, { recursive: true, force: true });
}
}, 60_000);
it("keeps a failed command result behind the events the Runner sent before it", async () => {
const root = mkdtempSync(resolve(tmpdir(), "runner-failed-result-order-"));
const { core, stop } = replayBacklogCore(root, 20);
try {
await core.start();
const client = (await authenticate(core, core.issueBootstrapTicket()))!;
const snapshot = core.queueCommand("session.snapshot", {}, "failed-after-evidence", true);
await receiveCommand(client);
for (let seq = 1; seq <= 20; seq++) sendSecure(client, ordinaryOutput(seq));
sendSecure(client, commandResultFrame(client, snapshot, { code: "command_execution_failed" }, "failed"));
await vi.waitFor(() => expect(core.commandOutcome(snapshot.commandId)?.status).toBe("failed"), { timeout: 10_000 });
expect(core.store.state.ackedSourceSeq).toBe(20);
client.socket.destroy();
} finally {
stop();
await core.stop();
await core.drainPendingConnectionProcessing();
rmSync(root, { recursive: true, force: true });
}
}, 30_000);
function staleToolResult(core: DurablePrpControlPlane) {
return core.queueCommand("semantic_tool.result", { callId: "call-earlier-generation", operationId: "get_task_context",
sourceEventId: "tool-input-earlier-generation", sourceEventType: "semantic_tool.input", inputDigest: "digest",
correlation: { ...identity }, result: { ok: true }, isError: false }, "command_tool_earlier_generation");
}
it("delivers every command pending at attach so an earlier generation's result cannot hold the recovery probe", async () => {
const root = mkdtempSync(resolve(tmpdir(), "runner-stale-pending-at-attach-"));
const { core, stop } = replayBacklogCore(root, 50);
try {
const stale = staleToolResult(core);
const probe = core.queueCommand("runner.drain", {}, "command_resume_probe_attach");
await core.start();
const client = (await authenticate(core, core.issueBootstrapTicket()))!;
const pending = (client.welcome.payload as { pendingCommands: Array<{ commandId: string }> }).pendingCommands;
expect(pending.map(command => command.commandId)).toEqual([stale.commandId, probe.commandId]);
sendSecure(client, commandResultFrame(client, stale));
sendSecure(client, commandResultFrame(client, probe, { status: "completed", retainedEventsDrained: false }));
for (let seq = 1; seq <= 1_000; seq++) sendSecure(client, ordinaryOutput(seq));
await vi.waitFor(() => expect(core.commandOutcome(probe.commandId)?.status).toBe("completed"), { timeout: 5_000 });
expect(core.commandOutcome(stale.commandId)?.status).toBe("completed");
expect(core.store.state.commandDeliveryCounts).toEqual({ [stale.commandId]: 1, [probe.commandId]: 1 });
client.socket.destroy();
} finally {
stop();
await core.stop();
await core.drainPendingConnectionProcessing();
rmSync(root, { recursive: true, force: true });
}
}, 30_000);
it("never delivers a lifecycle command ahead of its turn at attach", async () => {
const root = mkdtempSync(resolve(tmpdir(), "runner-attach-lifecycle-fence-"));
const { core, stop } = replayBacklogCore(root, 1);
try {
const stale = staleToolResult(core);
const drain = core.queueCommand("runner.drain", {}, "command_drain_before_suspend");
core.queueCommand("runner.suspend", {}, "command_suspend_after_drain");
core.queueCommand("session.snapshot", {}, "command_snapshot_after_suspend");
await core.start();
const client = (await authenticate(core, core.issueBootstrapTicket()))!;
const pending = (client.welcome.payload as { pendingCommands: Array<{ commandId: string }> }).pendingCommands;
expect(pending.map(command => command.commandId)).toEqual([stale.commandId, drain.commandId]);
sendSecure(client, commandResultFrame(client, stale));
sendSecure(client, commandResultFrame(client, drain, { status: "completed", retainedEventsDrained: true }));
expect((await receiveCommand(client)).payload).toMatchObject({ commandId: "command_suspend_after_drain" });
client.socket.destroy();
} finally {
stop();
await core.stop();
await core.drainPendingConnectionProcessing();
rmSync(root, { recursive: true, force: true });
}
}, 30_000);
function completedItem(sourceSeq: number): Record<string, unknown> {
const envelope = ordinaryOutput(sourceSeq);
Object.assign(envelope.payload as Record<string, unknown>, {
sourceEventId: `completed-${sourceSeq}`, eventType: "item.completed", priority: 1,
payload: { item: { id: `item-${sourceSeq}` } },
});
return envelope;
}
it("acknowledges a replayed burst once per batch, never ahead of its durable checkpoint", async () => {
const root = mkdtempSync(resolve(tmpdir(), "runner-batched-ack-"));
const { core, stop } = replayBacklogCore(root, 5);
let persistedCursor = 0;
const persisted: number[] = [];
core.setCheckpointPersistence(async saved => {
await new Promise<void>(resolveWrite => setTimeout(resolveWrite, 5));
persistedCursor = JSON.parse(saved.records.get("cursor")!).ackedSourceSeq;
persisted.push(persistedCursor);
});
try {
await core.start();
const client = (await authenticate(core, core.issueBootstrapTicket()))!;
const before = persisted.length;
for (let seq = 1; seq <= 300; seq++) sendSecure(client, completedItem(seq));
const acks: number[] = [];
while (acks.at(-1) !== 300) {
const frame = await receiveSecure(client);
if (frame === null) throw new Error("connection closed before the burst was acknowledged");
if (frame.kind !== "ack") continue;
const acked = (frame.payload as { ackedSourceSeq: number }).ackedSourceSeq;
expect(acked).toBeLessThanOrEqual(persistedCursor);
acks.push(acked);
}
expect(acks).toEqual([...acks].sort((left, right) => left - right));
expect(acks.length).toBeLessThan(300 / 16);
expect(persisted.length - before).toBeLessThan(300 / 16);
expect(core.store.state.committedEvents.map(event => event.sourceSeq)).toEqual(Array.from({ length: 300 }, (_, index) => index + 1));
client.socket.destroy();
} finally {
stop();
await core.stop();
await core.drainPendingConnectionProcessing();
rmSync(root, { recursive: true, force: true });
}
}, 60_000);
it("still acknowledges a lifecycle event inside a burst before the next frame", async () => {
const root = mkdtempSync(resolve(tmpdir(), "runner-batched-ack-lifecycle-"));
const { core, stop } = replayBacklogCore(root, 5);
try {
await core.start();
const client = (await authenticate(core, core.issueBootstrapTicket()))!;
const turn = completedItem(11);
Object.assign(turn.payload as Record<string, unknown>, { sourceEventId: "turn-11", eventType: "turn.completed", priority: 0,
payload: { turnId: identity.turnId } });
for (let seq = 1; seq <= 20; seq++) sendSecure(client, seq === 11 ? turn : completedItem(seq));
const acks: number[] = [];
while (acks.at(-1) !== 20) {
const frame = await receiveSecure(client);
if (frame === null) throw new Error("connection closed before the burst was acknowledged");
if (frame.kind === "ack") acks.push((frame.payload as { ackedSourceSeq: number }).ackedSourceSeq);
}
expect(acks).toContain(11);
client.socket.destroy();
} finally {
stop();
await core.stop();
await core.drainPendingConnectionProcessing();
rmSync(root, { recursive: true, force: true });
}
}, 60_000);
it("answers a snapshot from a Runner whose provider stays wedged until its whole replay is acknowledged", async () => {
const root = mkdtempSync(resolve(tmpdir(), "runner-wedged-provider-replay-"));
const { core, stop } = replayBacklogCore(root, 20);
try {
await core.start();
const client = (await authenticate(core, core.issueBootstrapTicket()))!;
const backlog = 250;
for (let seq = 1; seq <= backlog; seq++) sendSecure(client, completedItem(seq));
const snapshot = core.queueCommand("session.snapshot", {}, "snapshot-behind-wedged-provider", true);
const runner = (async () => {
let delivered = false;
let acked = 0;
while (!delivered || acked < backlog) {
const frame = await receiveSecure(client, null);
if (frame === null) return;
if (frame.kind === "command") delivered = true;
if (frame.kind === "ack") acked = (frame.payload as { ackedSourceSeq: number }).ackedSourceSeq;
}
sendSecure(client, commandResultFrame(client, snapshot));
})();
await waitForRunnerCommand({
type: "session.snapshot",
deadline: Date.now() + 2_000,
abortOnClose: false,
isClosed: () => false,
throwIfFailed: () => undefined,
command: () => core.getCommand(snapshot.commandId),
runnerHasExited: async () => false,
failureCode: () => "provider_transport_failed",
answeredProgressAt: () => core.queuedCommandResultProgressAt(snapshot.commandId),
replayProgressAt: () => core.committedEventProgressAt?.() ?? null,
});
expect(core.store.state.ackedSourceSeq).toBe(backlog);
await runner;
client.socket.destroy();
} finally {
stop();
await core.stop();
await core.drainPendingConnectionProcessing();
rmSync(root, { recursive: true, force: true });
}
}, 60_000);
it("keeps a staged tool-progress write when a lease renewal lands mid-write", async () => {
const root = mkdtempSync(resolve(tmpdir(), "runner-lease-renewal-staged-write-"));
let release!: () => void;
@@ -2690,13 +2929,13 @@ class ServerFrameReader {
socket.once("close", () => this.#publish(null));
}
next(): Promise<Record<string, unknown> | null> {
next(timeoutMs: number | null = 2_000): Promise<Record<string, unknown> | null> {
const queued = this.#frames.shift();
if (queued !== undefined) return Promise.resolve(queued);
return new Promise((resolveFrame, rejectFrame) => {
const timer = setTimeout(
const timer = timeoutMs === null ? undefined : setTimeout(
() => rejectFrame(new Error("Server WebSocket frame timed out.")),
2_000,
timeoutMs,
);
this.#waiters.push((value) => {
clearTimeout(timer);
@@ -3131,8 +3370,9 @@ function sendSecure(
async function receiveSecure(
client: AuthenticatedClient,
frameTimeoutMs: number | null = 2_000,
): Promise<Record<string, unknown> | null> {
const frame = await client.reader.next();
const frame = await client.reader.next(frameTimeoutMs);
if (frame === null) return null;
const counter = BigInt(frame.counter as number);
expect(counter).toBe(client.receiveCounter);
@@ -4619,11 +4859,6 @@ describe("DurablePrpControlPlane", () => {
{ callId: "call-1" },
"command-tool-1",
);
controlPlane.queueCommand(
"turn.interrupt",
{ turnId: "turn-1" },
"command-interrupt-1",
);
const client = await authenticate(
controlPlane,
controlPlane.issueBootstrapTicket(),
@@ -4633,6 +4868,11 @@ describe("DurablePrpControlPlane", () => {
expect.objectContaining({ commandId: "command-tool-1" }),
],
});
controlPlane.queueCommand(
"turn.interrupt",
{ turnId: "turn-1" },
"command-interrupt-1",
);
// Exactly what runnerd replays after it is killed between journaling a
// command and confirming its effect. Its durable contract promotes such a
@@ -65,6 +65,8 @@ const maxFrameBytes = 1024 * 1024;
// Secure frames hex-encode ciphertext. Reserve envelope/tag space as well.
const maxCommandBytes = Math.floor((maxFrameBytes - 4 * 1024) / 2);
const maxCommands = 500;
const maxCommandsAtAttach = 32;
const maxEventsPerAck = 64;
class CommandJournalLimitError extends Error {
constructor(readonly code: "command_payload_too_large" | "command_journal_full") {
super(`Durable PRP command journal bound exceeded: ${code}.`);
@@ -83,6 +85,13 @@ const runnerDigestPattern = /^sha256:[0-9a-f]{64}$/;
export function isRunnerRecoveryMaintenanceCommand(type: string): boolean {
return ["session.snapshot", "session.goal.get", "runner.drain", "runner.suspend", "runner.shutdown", "turn.stop"].includes(type);
}
const orderIndependentResultTypes = new Set(["session.snapshot", "session.goal.get", "runner.drain", "semantic_tool.result"]);
function answersAheadOfEvents(command: DurableRecoveryCoreCommand | undefined, result: unknown): boolean {
if (command === undefined || command.status !== "pending" || !orderIndependentResultTypes.has(command.type)) return false;
if (command.type === "session.snapshot" && command.payload.quiesceForWarmAttach === true) return false;
const status = isRecord(result) ? result.status : undefined;
return status === "completed" || status === "rejected";
}
/** The Runner journaled this command before its effect, then died. It sends
* that `indeterminate` entry as `failed` with `execution_indeterminate`, never
@@ -2033,6 +2042,9 @@ class AuthorityConnection {
warmTransitionVersion: 1 | null = null;
identity: DurableRecoveryIdentity | null = null;
replayOnly = false;
readonly deliveredAtAttach = new Set<string>();
queuedInboundFrames = 0;
unacknowledgedEvents = 0;
activationReceipt: DurableWarmRunTransition | null = null;
readonly wire: PrpWireConnection;
#closed = false;
@@ -2100,6 +2112,7 @@ export class DurablePrpControlPlane {
#semanticCallbacksRetired = false;
#queuedCommandResults = new Map<string, { count: number; queuedAt: number }>();
#inboundFrameSettledAt = 0;
#eventCommittedAt: number | null = null;
#semanticResultPersistenceFailed = false;
#semanticResultFailures: Array<{
callId: string;
@@ -2708,6 +2721,10 @@ export class DurablePrpControlPlane {
return queued === undefined ? null : Math.max(queued.queuedAt, this.#inboundFrameSettledAt);
}
committedEventProgressAt(): number | null {
return this.#eventCommittedAt;
}
withdrawUndeliveredTerminalCommands(): string[] {
if (this.#store.state.warmTransition) return [];
const commands = this.#store.state.commands;
@@ -3139,6 +3156,7 @@ export class DurablePrpControlPlane {
}
return;
}
if (decrypted.kind === "command_result" && this.#answerOnArrival(connection, decrypted)) return;
frame = { decrypted };
} else {
undecryptedFrames += 1;
@@ -3151,12 +3169,15 @@ export class DurablePrpControlPlane {
const queued = this.#queuedCommandResults.get(queuedResult);
this.#queuedCommandResults.set(queuedResult, { count: (queued?.count ?? 0) + 1, queuedAt: queued?.queuedAt ?? Date.now() });
}
connection.queuedInboundFrames += 1;
processing = processing
.then(() => "raw" in frame
? this.#handleJson(connection, frame.raw).finally(() => { undecryptedFrames -= 1; })
: this.#handleDecryptedFrame(connection, frame.decrypted))
.then(() => connection.queuedInboundFrames === 1 ? this.#acknowledgeDeferredEvents(connection) : undefined)
.catch(() => connection.close())
.finally(() => {
connection.queuedInboundFrames -= 1;
this.#inboundFrameSettledAt = Date.now();
if (queuedResult === null) return;
const queued = this.#queuedCommandResults.get(queuedResult);
@@ -3237,6 +3258,31 @@ export class DurablePrpControlPlane {
if (envelope.kind === "lease_renew") this.#renewLease(connection, envelope);
}
#answerOnArrival(
connection: AuthorityConnection,
envelope: Record<string, unknown>,
): boolean {
const result = envelope.payload;
const commandId = isRecord(result) ? result.commandId : undefined;
if (
typeof commandId !== "string" ||
connection.replayOnly ||
this.#store.state.warmTransition !== undefined ||
!answersAheadOfEvents(this.#store.state.commands.find(command => command.commandId === commandId), result)
) return false;
try {
this.#store.assertWritable();
if (!this.#admitEnvelopeVersion(connection, envelope) || !this.#admitAuthenticatedFrame(connection)
|| connection.lease === null) return true;
this.#commandResult(connection, envelope);
} catch {
connection.close();
return true;
}
void this.flushCheckpoint().catch(() => connection.close());
return true;
}
#admitEnvelopeVersion(
connection: AuthorityConnection,
envelope: Record<string, unknown>,
@@ -3982,12 +4028,13 @@ export class DurablePrpControlPlane {
this.#store.state.lastLeaseId = lease.leaseId;
this.#store.state.lastLeaseExpiresAt = lease.expiresAt;
const pending = connection.replayOnly ? [] : this.#nextPendingCommand();
const pending = connection.replayOnly ? [] : this.#pendingCommandsAtAttach();
const [pendingCommand] = pending;
connection.terminalLifecycleCommandId =
pendingCommand && this.#isTerminalLifecycleCommand(pendingCommand)
? pendingCommand.commandId
: null;
for (const command of pending.slice(1)) connection.deliveredAtAttach.add(command.commandId);
for (const command of pending) {
this.#store.state.commandDeliveryCounts[command.commandId] =
(this.#store.state.commandDeliveryCounts[command.commandId] ?? 0) + 1;
@@ -4065,6 +4112,24 @@ export class DurablePrpControlPlane {
return command === undefined ? [] : [command];
}
#pendingCommandsAtAttach(): DurableRecoveryCoreCommand[] {
const [head] = this.#nextPendingCommand();
if (head === undefined) return [];
const batch = [head];
if (this.#isTerminalLifecycleCommand(head) || head.type === "run.attach") return batch;
const commands = this.#store.state.commands;
let bytes = Buffer.byteLength(JSON.stringify(this.#wireCommand(head)));
for (let index = commands.indexOf(head) + 1; index < commands.length && batch.length < maxCommandsAtAttach; index += 1) {
const command = commands[index]!;
if (command.status !== "pending" || command.controllerSeq !== batch.at(-1)!.controllerSeq + 1
|| this.#isTerminalLifecycleCommand(command) || command.type === "run.attach") break;
bytes += Buffer.byteLength(JSON.stringify(this.#wireCommand(command)));
if (bytes > maxCommandBytes) break;
batch.push(command);
}
return batch;
}
#controlEnvelope(
connection: AuthorityConnection,
envelopeId: string,
@@ -4102,7 +4167,7 @@ export class DurablePrpControlPlane {
)
return;
const [command] = this.#nextPendingCommand();
if (command === undefined) return;
if (command === undefined || connection.deliveredAtAttach.has(command.commandId)) return;
if (this.#isTerminalLifecycleCommand(command)) {
connection.terminalLifecycleCommandId = command.commandId;
}
@@ -4330,6 +4395,24 @@ export class DurablePrpControlPlane {
);
}
async #acknowledgeDeferredEvents(connection: AuthorityConnection): Promise<void> {
if (connection.unacknowledgedEvents === 0) return;
connection.unacknowledgedEvents = 0;
const identity = this.#identity;
if (this.#eventOwner !== connection || this.#protocolIntegrityError !== null) return;
try {
await this.flushCheckpoint();
} catch (error) {
if (this.#eventOwner === connection && this.#identity === identity) {
this.#eventCommitFailed(connection, `checkpoint_flush_failed:${safeErrorCauseCode(error)}`);
}
return;
}
if (this.#eventOwner !== connection || this.#identity !== identity || this.#protocolIntegrityError !== null) return;
this.#sendJson(connection, this.#controlEnvelope(connection, `ack_${this.#store.state.ackedSourceSeq}`, "ack",
{ ackedSourceSeq: this.#store.state.ackedSourceSeq }));
}
#eventCommitFailed(connection: AuthorityConnection, cause: string, sourceSeq?: number, now = Date.now()): void {
const acked = this.#store.state.ackedSourceSeq;
const runId = this.#identity.runId;
@@ -4546,7 +4629,8 @@ export class DurablePrpControlPlane {
};
const isCurrentOwner = () => this.#eventOwner === connection &&
this.#identity === identity && this.#protocolIntegrityError === null;
if (priority === 0 || isSemanticInput || eventType.startsWith("turn.") || eventType.startsWith("run.")) {
const critical = priority === 0 || isSemanticInput || eventType.startsWith("turn.") || eventType.startsWith("run.");
if (critical) {
// Critical lifecycle suffixes and tool admission must finish in this
// turn before readers react or a replacement owner adopts their cursor.
const candidate = buildCandidate(this.#store.state);
@@ -4575,6 +4659,12 @@ export class DurablePrpControlPlane {
}
}
if (!isCurrentOwner()) return;
this.#eventCommittedAt = Date.now();
if (!critical && connection.queuedInboundFrames > 1 && connection.unacknowledgedEvents + 1 < maxEventsPerAck) {
connection.unacknowledgedEvents += 1;
return;
}
connection.unacknowledgedEvents = 0;
const liveCall = !this.#retainsSourceInputs() && isSemanticInput && this.#onSemanticToolInput && semantic !== undefined
&& typeof semantic.callId === "string" && typeof semantic.operationId === "string"
@@ -0,0 +1,122 @@
import { mkdtemp, readFile, rm } from "node:fs/promises";
import { createServer } from "node:http";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { expect, it, vi } from "vitest";
import { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js";
import { recoverToolCallback, TOOL_CALLBACK_INTERRUPTED_CODE } from "../control-plane/orphaned-tool-callback.js";
import { createCapabilityRunnerdCodexTransport } from "./runnerd-codex-transport.js";
import { defaultCapabilityRunnerdBinary, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js";
const taskTool = { name: "get_task_context", description: "Read current task", inputSchema: { type: "object", properties: {}, additionalProperties: false } };
type SavedCommand = { commandId: string; type: string; status: string; payload: Record<string, unknown> };
it("settles an earlier generation's in-flight tool call at attach even when that attach never finishes startup", async () => {
const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-orphaned-tool-at-attach-"));
const controlPlaneStatePath = join(stateDirectory, "control-plane", "control-plane-state.json");
const server = createServer();
let authority: DurablePrpControlPlane | null = null;
let admitConnections = true;
server.on("upgrade", (request, socket, head) => {
if (!authority || !admitConnections) {
socket.destroy();
return;
}
authority.handleUpgrade(request, socket, "/runner", head);
});
await new Promise<void>(resolveListen => server.listen(0, "127.0.0.1", resolveListen));
const address = server.address();
if (!address || typeof address === "string") throw new Error("Expected a test listener");
const registration = vi.fn(async (next: DurablePrpControlPlane) => {
authority = next;
return {
connectUrl: `ws://127.0.0.1:${address.port}/runner`,
release: async () => { if (authority === next) authority = null; },
};
});
const identity = {
runnerInstanceId: "runner-orphaned-tool",
environmentLeaseId: "lease-orphaned-tool",
runId: "run-orphaned-tool",
normalizedSessionId: "session-orphaned-tool",
turnId: "turn-orphaned-tool",
itemId: "item-orphaned-tool",
};
const options = {
runnerBinary: defaultCapabilityRunnerdBinary(),
codexCommand: fakeCodex,
codexArgs: fakeCodexArgs(stateDirectory, "--emit-tool-call"),
stateDirectory,
prpIdentity: identity,
lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 },
controlPlaneRegistration: registration,
};
const recovered = vi.fn(async (call: { operationId: string; callId: string; input: unknown; admission?: "this_controller" | "earlier_controller" }) =>
recoverToolCallback({ runId: identity.runId, call: { tool: call.operationId, callId: call.callId, arguments: call.input },
admission: call.admission, recover: async () => null }));
const first = createCapabilityRunnerdCodexTransport(options);
first.transport.setServerRequestHandler(() => new Promise<Record<string, unknown>>(() => undefined));
const generations: Array<ReturnType<typeof createCapabilityRunnerdCodexTransport>> = [];
let runnerPid: number | null = null;
const savedToolResults = async () => (JSON.parse(await readFile(controlPlaneStatePath, "utf8")) as { commands: SavedCommand[] })
.commands.filter(command => command.type === "semantic_tool.result");
try {
const opened = (await first.transport.request("thread/start", { cwd: tmpdir(), dynamicTools: [taskTool] })).thread as { id: string; sessionId: string };
runnerPid = first.evidence().runnerPid;
const runnerStartedAt = first.transport.processInfo!().startedAt;
await first.transport.request("turn/start", { input: [{ type: "text", text: "Read current task" }] });
await vi.waitFor(async () => {
const saved = JSON.parse(await readFile(controlPlaneStatePath, "utf8")) as { committedEvents: Array<{ eventType: string }> };
expect(saved.committedEvents.some(event => event.eventType === "semantic_tool.input")).toBe(true);
}, { timeout: 10_000 });
await first.detachControllerForRestart();
expect(await savedToolResults()).toEqual([]);
const adopt = (graceMs?: number) => {
const generation = createCapabilityRunnerdCodexTransport({
...options,
...(graceMs === undefined ? {} : { runnerReconnectGraceMs: graceMs }),
recoverSemanticToolInput: recovered,
resumeDynamicTools: [taskTool],
resumeProviderSession: { driverSessionId: opened.id, providerSessionId: opened.sessionId },
runnerProcessLauncher: () => { throw new Error("duplicate runner spawn attempted"); },
adoptExistingRunner: {
pid: runnerPid!, processGroupId: runnerPid, startedAt: runnerStartedAt, signal: () => true,
isAlive: () => { try { process.kill(runnerPid!, 0); return true; } catch { return false; } },
},
});
generation.transport.setServerRequestHandler(vi.fn(async () => ({ success: true, contentItems: [] })));
generations.push(generation);
return generation;
};
admitConnections = false;
const stalled = adopt(1_500);
await expect(stalled.transport.request("thread/read", {})).rejects.toThrow();
await stalled.transport.close().catch(() => undefined);
const settled = await savedToolResults();
expect(settled).toHaveLength(1);
expect(JSON.stringify(settled[0]!.payload)).toContain(TOOL_CALLBACK_INTERRUPTED_CODE);
admitConnections = true;
const next = adopt();
await expect(next.transport.request("thread/read", {})).resolves.toEqual(
expect.objectContaining({ thread: expect.objectContaining({ id: opened.id }) }));
await vi.waitFor(async () => {
const results = await savedToolResults();
expect(results).toHaveLength(1);
expect(results[0]).toMatchObject({ commandId: settled[0]!.commandId, status: "completed" });
}, { timeout: 10_000 });
expect(recovered).toHaveBeenCalledTimes(1);
} finally {
for (const generation of generations.reverse()) await generation.transport.close().catch(() => undefined);
await first.transport.close().catch(() => undefined);
if (runnerPid) {
try { process.kill(-runnerPid, "SIGKILL"); } catch { runnerPid = null; }
}
server.closeAllConnections();
if (server.listening) await new Promise<void>(resolveClose => server.close(() => resolveClose()));
await rm(stateDirectory, { recursive: true, force: true });
}
}, 120_000);
@@ -4,7 +4,7 @@ import { tmpdir } from "node:os";
import { join } from "node:path";
import { expect, it, vi } from "vitest";
import { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js";
import { createCapabilityRunnerdCodexTransport, RunnerStallLedger, waitForRunnerCommand } from "./runnerd-codex-transport.js";
import { createCapabilityRunnerdCodexTransport, RUNNER_REPLAY_PROGRESS_MAX_EXTENSION_MS, RunnerStallLedger, waitForRunnerCommand } from "./runnerd-codex-transport.js";
import { defaultCapabilityRunnerdBinary, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js";
type Carried = "late-tool-result" | "close-path-suspend" | "suspend-then-tool-result" | "event-backlog";
@@ -222,6 +222,73 @@ it("times out when the runner's queued answer stops advancing", async () => {
})).rejects.toThrow("provider_transport_failed: PRP command session.snapshot timed out");
});
it("keeps waiting past the deadline while the controller is still committing the Runner's replayed events", async () => {
let replayedAt = Date.now();
const command = pendingCommand();
const replaying = setInterval(() => { replayedAt = Date.now(); }, 5);
setTimeout(() => { command.status = "completed"; }, 150);
try {
await expect(waitForRunnerCommand({
type: "runner.drain",
deadline: Date.now() + 20,
abortOnClose: false,
isClosed: () => false,
throwIfFailed: () => undefined,
command: () => command,
runnerHasExited: async () => false,
failureCode: () => "runner_ingress_unavailable",
answeredProgressAt: () => null,
replayProgressAt: () => replayedAt,
})).resolves.toBeUndefined();
} finally {
clearInterval(replaying);
}
});
it("times out a Runner that keeps replaying but never answers once the replay extension is spent", async () => {
vi.useFakeTimers();
try {
const deadline = Date.now() + 30_000;
const waited = waitForRunnerCommand({
type: "session.snapshot",
deadline,
abortOnClose: false,
isClosed: () => false,
throwIfFailed: () => undefined,
command: () => pendingCommand(),
runnerHasExited: async () => false,
failureCode: () => "provider_transport_failed",
answeredProgressAt: () => null,
replayProgressAt: () => Date.now(),
});
const outcome = waited.then(() => "completed", (error: unknown) => String(error));
await vi.advanceTimersByTimeAsync(30_000 + RUNNER_REPLAY_PROGRESS_MAX_EXTENSION_MS - 1_000);
let settled = false;
void outcome.then(() => { settled = true; });
await vi.advanceTimersByTimeAsync(0);
expect(settled).toBe(false);
await vi.advanceTimersByTimeAsync(2_000);
await expect(outcome).resolves.toContain("provider_transport_failed: PRP command session.snapshot timed out");
} finally {
vi.useRealTimers();
}
});
it("does not extend a deadline for a Runner whose replay has stopped advancing", async () => {
await expect(waitForRunnerCommand({
type: "runner.drain",
deadline: Date.now() + 20,
abortOnClose: false,
isClosed: () => false,
throwIfFailed: () => undefined,
command: () => pendingCommand(),
runnerHasExited: async () => false,
failureCode: () => "runner_ingress_unavailable",
answeredProgressAt: () => null,
replayProgressAt: () => Date.now() - 31_000,
})).rejects.toThrow("runner_ingress_unavailable: PRP command runner.drain timed out");
});
it("leaves later barriers that share an admission deadline their time after an answered barrier ran past it", async () => {
const clock = new RunnerStallLedger();
const deadline = clock.now() + 20;
@@ -4690,8 +4690,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
this.#recoverPendingSemanticInputsWhenReady();
}
#recoverPendingSemanticInputsWhenReady(): void {
if (!this.#retainsSourceInputs() && this.options.recoveryPurpose !== "stop_cleanup" && this.#startupComplete && this.#handlerInstalled && this.#sourceRecoveryIdentity === null && this.#pendingWarmRecoveryCompletion === null) {
#recoverPendingSemanticInputsWhenReady(atAttach = false): void {
if (!this.#retainsSourceInputs() && this.options.recoveryPurpose !== "stop_cleanup"
&& (atAttach || (this.#startupComplete && this.#handlerInstalled))
&& this.#sourceRecoveryIdentity === null && this.#pendingWarmRecoveryCompletion === null) {
void Promise.resolve().then(() => this.#core?.recoverPendingSemanticInputs())
.catch(error => this.#failTransport(error instanceof Error ? error : new Error(String(error))));
}
@@ -5879,6 +5881,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
let checkpointFailed = false;
let releaseFailed = false;
let processStopFailed = false;
let retirementFailed = false;
try {
await releaseRunnerProcessOwnership({
runnerSettled,
@@ -5920,7 +5923,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
throw error;
}
}),
retireController: () => this.#stopAndDrainAuthority(closeDeadline),
retireController: async () => {
try {
await this.#stopAndDrainAuthority(closeDeadline);
} catch (error) {
retirementFailed = true;
throw error;
}
},
});
} catch (error) {
if (checkpoint && checkpointFailed && !releaseFailed && !processStopFailed) {
@@ -5928,6 +5938,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
checkpoint: () => checkpoint(runnerSettled ? "settled" : "unsettled"),
finish: finishClose,
};
} else if (retirementFailed && !releaseFailed && !processStopFailed) {
this.#pendingCloseCheckpointRetry = {
checkpoint: async () => {
await this.#stopAndDrainAuthority(Date.now() + runnerCloseGraceMs(this.options));
await checkpoint?.(runnerSettled ? "settled" : "unsettled");
},
finish: finishClose,
};
}
throw error;
} finally {
@@ -7386,6 +7404,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
}
this.#assertStartupOpen();
this.#startPump();
this.#recoverPendingSemanticInputsWhenReady(true);
if (adoptedRunner) {
await this.#awaitAdoptedRunnerConnection(
adoptedRunner,
@@ -7469,7 +7488,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
if (this.#retainsSourceInputs() || (this.#authorityRunId() !== undefined && call.correlation.runId !== this.#authorityRunId())) return null;
if (this.#sourceRecoveryIdentity) throw new NativeRunnerSourceReconciliationRequiredError("semantic");
if (call.operationId === "paperclip_finish" || call.operationId === "paperclip_block") {
return this.#handleSemanticToolInput(call);
return this.#startupComplete && this.#handlerInstalled ? this.#handleSemanticToolInput(call) : null;
}
const outcome = await this.options.recoverSemanticToolInput?.(call);
return outcome ? this.#bindSemanticToolOutcome(call, outcome) : null;
@@ -8081,6 +8100,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
}
return progressAt;
},
replayProgressAt: () => this.#core?.committedEventProgressAt() ?? null,
describeStall: () => this.#core === null ? null : describeRunnerCommandStall({
commands: this.#core.store.state.commands,
commandId: commandId ?? this.#core.store.state.commands.find(candidate => candidate.type === type)?.commandId,
@@ -9727,6 +9747,7 @@ export class RunnerStallLedger {
/** Shared polling path keeps startup cancellation separate from owned cleanup commands. */
const RUNNER_ANSWERED_RESULT_STALL_MS = 30_000;
export const RUNNER_REPLAY_PROGRESS_MAX_EXTENSION_MS = 300_000;
export async function waitForRunnerCommand(input: {
type: string;
@@ -9742,12 +9763,19 @@ export async function waitForRunnerCommand(input: {
/** Bounded, content-free evidence of what the command waited behind. */
describeStall?: () => string | null;
answeredProgressAt?: () => number | null;
replayProgressAt?: () => number | null;
}): Promise<void> {
const now = () => input.clock ? input.clock.now() : Date.now();
const answeredAndAdvancing = () => {
const progressAt = input.answeredProgressAt?.() ?? null;
return progressAt !== null && Date.now() - progressAt < RUNNER_ANSWERED_RESULT_STALL_MS;
};
const replayExtensionEndsAt = Date.now() + Math.max(0, input.deadline - now()) + RUNNER_REPLAY_PROGRESS_MAX_EXTENSION_MS;
const replayAdvancing = () => {
const progressAt = input.replayProgressAt?.() ?? null;
return progressAt !== null && Date.now() < replayExtensionEndsAt
&& Date.now() - progressAt < RUNNER_ANSWERED_RESULT_STALL_MS;
};
let extended = false;
const settled = () => {
const command = input.command();
@@ -9768,7 +9796,7 @@ export async function waitForRunnerCommand(input: {
input.clock?.credit(Math.max(0, now() - input.deadline) + RUNNER_POST_STOP_GRACE_MS);
};
let observed = false;
while (now() < input.deadline || answeredAndAdvancing()) {
while (now() < input.deadline || answeredAndAdvancing() || replayAdvancing()) {
observed = true;
if (now() >= input.deadline) extended = true;
input.throwIfFailed();
@@ -4,6 +4,7 @@ import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { expect, it, vi } from "vitest";
import { DurablePrpControlPlane, type RunnerProcessResult } from "../control-plane/durable-prp-control-plane.js";
import { NativeSessionCloseUnrecoverableError } from "../contracts/native-session-backend.js";
import { createCapabilityRunnerdCodexTransport, defaultCapabilityRunnerdBinary, readRunnerdArtifactBinding, runnerdRecoveryInternals } from "./runnerd-codex-transport.js";
it.each(["resolved", "rejected"] as const)("keeps the checkpoint behind controller retirement (%s)", async outcome => {
@@ -32,7 +33,7 @@ it.each(["resolved", "rejected"] as const)("keeps the checkpoint behind controll
}
});
it.each(["drain", "timeout", "external_close"] as const)("joins an output flush before controller retirement permits a successor writer (%s)", async mode => {
it.each(["drain", "timeout", "external_close", "close_timeout"] as const)("joins an output flush before controller retirement permits a successor writer (%s)", async mode => {
const root = await fs.mkdtemp(join(tmpdir(), "runnerd-detach-write-"));
await fs.mkdir(join(root, "diagnostics"), { mode: 0o700 });
const binary = defaultCapabilityRunnerdBinary();
@@ -64,7 +65,7 @@ it.each(["drain", "timeout", "external_close"] as const)("joins an output flush
codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"),
codexArgs: ["--state-file", join(root, "provider-state.json"), "--split-event-burst", "--split-event-prefix-count", "1"],
stateDirectory: root,
closeGraceMs: mode === "timeout" ? 200 : 1_500,
closeGraceMs: mode === "timeout" || mode === "close_timeout" ? 200 : 1_500,
...(mode === "external_close" ? { controlPlaneRegistration: async (authority: DurablePrpControlPlane) => {
await authority.start();
const stop = authority.stop.bind(authority);
@@ -105,6 +106,15 @@ it.each(["drain", "timeout", "external_close"] as const)("joins an output flush
expect(closeSteps.indexOf("release")).toBeLessThan(closeSteps.indexOf("stop"));
return;
}
if (mode === "close_timeout") {
await expect(bundle.transport.close()).rejects.toThrow("journal drain timed out");
release();
await core.drainPendingConnectionProcessing();
const retried = await bundle.transport.close().then(() => null, (error: Error) => error);
expect(retried).toBeInstanceOf(NativeSessionCloseUnrecoverableError);
expect(retried?.message ?? "").not.toContain("journal drain timed out");
return;
}
let settled = false;
detached = bundle.detachControllerForRestart().then(() => { settled = true; });
await vi.waitFor(() => expect(core.activeRunnerConnectionCount()).toBe(0));