fix(runner): keep reattached runners answering after a restart (#21161)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute in a remote Runner that the server controls over
the durable PRP channel
> - After a server restart, the next server reattaches to the live
Runner and sends recovery barriers (`runner.drain`, `session.snapshot`)
> - These barriers timed out while the Runner stayed connected, and each
same-run retry failed the same way
> - After three attempts the run stopped with
`native_session_retry_exhausted` and needed a manual Retry
> - This pull request finds the two causes and fixes both in the
controller and the transport
> - The benefit is that restart recovery finishes on its own, without a
manual Retry

## Linked Issues or Issue Description

**What happened?**

After a server restart, a run reattached to its live remote Runner. The
Runner authenticated. Then a recovery command got no recorded answer for
30 s or more, although the Runner stayed connected. Errors on staging:

- `PRP command session.snapshot timed out (runner connected; pending
ahead 0; last runner event seq 869 55s ago)`
- `PRP command runner.drain timed out (runner connected; pending ahead 1
[semantic_tool.result] oldest 170s ...)`
- `runnerd exited while waiting for runner.drain`

Each reattach failed the same way. After three attempts the run ended
with "Workspace repair required" (`native_session_retry_exhausted`).

Root causes, proven with the real Runner and the fake Codex harness:

1. The Runner answered in about 40 ms. At reconnect it also sends the
event backlog that it collected while the server was down. The
controller handles inbound frames of a connection in order, and it
commits each event (accounting write and checkpoint flush) before the
next frame. So the answer waited behind the whole backlog. With 400
events at 100 ms each, the answer was handled after 44 s, and the 30 s
barrier had already failed. The `semantic_tool.result` case is the same:
its answer waited behind the backlog, and the next command is sent only
after that answer is handled. The Runner code was not the cause.
2. When a Runner did not answer during close, the close path queued
`runner.drain` and `runner.suspend`, kept the Runner alive for
reattachment, and saved them in the journal. The next attempt delivered
that old `runner.suspend` to the reattached Runner, and the Runner
suspended and exited.

**Expected behavior**

A reattach to a live Runner completes recovery. A Runner answer that is
queued behind earlier frames is not a timeout. A reattach never delivers
a terminal command from an earlier attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Runner with a turn that streams output.
2. Restart the server. Let the Runner collect a few hundred events while
it is disconnected.
3. Let the next server reattach while event commits are slow, as on a
loaded database.
4. See `session.snapshot timed out (runner connected; pending ahead 0)`.
5. Alternatively, let a close path leave `runner.drain` and
`runner.suspend` pending, then reattach. See `runnerd exited while
waiting for runner.drain`.

**Paperclip version or commit**

`master` at 3fb35fcf13.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `DurablePrpControlPlane` records each Runner `command_result` frame
when it arrives, before the ordered frame chain handles it.
`queuedCommandResultProgressAt(commandId)` returns the latest time that
the chain made progress while that answer is queued.
- `waitForRunnerCommand` keeps waiting past its deadline while the
answer to the command, or to a pending command ahead of it, is queued
and the chain advanced in the last 30 s. When the chain stops for 30 s,
the barrier fails as before. A wait that starts after its deadline still
rejects a late ACK. Frame order does not change.
- When a barrier completes after such an extension, it credits the time
past the deadline (plus the 5 s post-stop grace) to the admission clock
(`RunnerStallLedger.credit`). Later barriers that share a cold admission
deadline (Pi, OpenCode, ACPX) keep their time.
`#waitForProviderIdentityWork` checks the ready state once before it
rejects an expired deadline.
- `DurablePrpControlPlane.withdrawUndeliveredTerminalCommands()` runs
only when the transport adopts a live Runner. The controller sends only
the first pending command, so no command after it reached the Runner,
and a live, serving Runner cannot have executed a suspend. The method
removes every pending `runner.suspend` and `runner.shutdown` from that
pending suffix. It keeps every other pending command, for example a
`semantic_tool.result` that a still-running tool callback queued after
the close, and renumbers the kept commands so the Runner's command
sequence stays contiguous. Pending drains stay, because they have no
effect.
- A generated command id now skips an id that a renumbered command
already holds.
- `doc/execution-semantics.md` describes both rules.

### Changed existing test

`runnerd-codex-transport-remote-retention.test.ts` asserted that the
reattaching controller delivers the failed close's `runner.suspend`, so
the adopted Runner suspends and exits, and a later attempt relaunches it
(#20923). On staging that path failed the attempt with `runnerd exited
while waiting for runner.drain` (run 3e28c2b0) and did not recover. This
PR requires that a reattach does not inherit that suspend. The test now
asserts that the adopted Runner stays alive, answers `thread/read`,
keeps a `ready` journal, gets no suspend, and no duplicate Runner is
launched. The retention part of the test (the close keeps the frozen
Runner alive) is unchanged.

## Verification

- Red before the fix, green after (real `paperclip-runnerd` and fake
Codex):
- `... past an event backlog ahead of its recovery snapshot result`
failed with the staging error `session.snapshot timed out (runner
connected; pending ahead 0; ...)`. Debug timing showed the Runner
answered in 40 ms and the controller handled the answer 44 s later.
- `... past a runner.suspend left by an earlier attempt's close path`
failed with the staging error `runnerd exited while waiting for
runner.drain`.
- `... past a runner.suspend with a tool result queued after it` failed
the same way with tail-only withdrawal (the Greptile case), and passes
now. The tool result is delivered once and all sequence numbers stay
contiguous.
- Unit tests for `waitForRunnerCommand`: it waits while a queued answer
advances, it times out when progress stops, and a later barrier that
shares the admission deadline still completes after an extended one.
- `... past a tool result its predecessor queued after the restart`
guards delivery of a `semantic_tool.result` queued between controller
generations.
- The existing `shares Pi's absolute cold deadline across recovery
barriers and rejects a late ACK` passes unchanged.
- `cd packages/paperclip-runner && pnpm exec vitest run
src/live/runnerd-codex-transport.test.ts
src/live/runnerd-codex-transport-remote-retention.test.ts
src/live/runnerd-codex-transport-reattach-carried-commands.test.ts`
passes (320 tests).
- Related suites pass: adoption, carried suspend, stale controller
journal, stall, deadlines,
`src/backends/harness-driver-backend.test.ts`, and all
`src/control-plane` tests (330 tests).
- `tsc --noEmit -p tsconfig.test.json` passes in
`packages/paperclip-runner`.
- `pnpm verify` asked for broader coverage (exit 2). Required CI
supplies it.

## Risks

- A barrier can now wait longer than its deadline when the Runner has
answered and the controller still commits a large backlog. The wait ends
30 s after progress stops. This trades a slower recovery for no failed
run.
- The withdrawal applies only to a live adopted Runner. If such a Runner
had somehow journaled a withdrawn suspend, the next command gets a
`controller_sequence_mismatch` rejection, and the Runner executes
nothing.
- Renumbering changes the `controllerSeq` of pending commands that never
reached the Runner. Their command ids do not change.
- Not in this PR: a fallback that stops a connected Runner that never
answers and resumes it from its journal. With these fixes, the observed
staging failures no longer reach that state.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool
use (shell, file editing, browser for read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 12:50:43 -06:00
1 parent a924d2e2c7
commit ef6d347f79
5 files changed
+404 -14

No files matched your search

+16
View File
@@ -1772,6 +1772,22 @@ in-flight turn stay for the next same-run reattachment. An abort (a Stop or the
recovery deadline), any identity or authority refusal, any other recovery
failure, and a protocol integrity fault still stop it.
A failed close can leave its `runner.suspend` (or `runner.shutdown`) pending in
the saved journal of a Runner it kept alive. The controller sends one command at
a time, so neither that command nor any command queued after it reached the
Runner. A reattachment that adopts the live Runner withdraws those terminal
commands before it sends anything. It keeps every other pending command, such
as a tool result that arrived after the close, and renumbers it so the Runner's
command sequence stays contiguous. The adopted Runner continues without exiting.
At reconnect, a Runner also sends the events it collected while it was
disconnected. The controller commits a connection's frames in order, so a
recovery command's answer can wait behind that backlog. A recovery barrier does
not time out while the Runner's answer to it, or to a command ahead of it, is
queued and the controller committed a frame in the last 30 s. The time spent
past the deadline is credited to the admission clock, so later barriers that
share the deadline keep their time.
A remote native run can lose its original runner or sandbox before reattachment.
This happens after a server restart, and for a same-run retry after a turn ends
without a semantic result. The reattachment then fails with
@@ -2098,6 +2098,8 @@ export class DurablePrpControlPlane {
* in-flight and failed dispatches remain. */
#liveDispatchedSemanticCalls = new Set<string>();
#semanticCallbacksRetired = false;
#queuedCommandResults = new Map<string, { count: number; queuedAt: number }>();
#inboundFrameSettledAt = 0;
#semanticResultPersistenceFailed = false;
#semanticResultFailures: Array<{
callId: string;
@@ -2675,6 +2677,27 @@ export class DurablePrpControlPlane {
* source for a later run.attach. Repeating the same write is idempotent;
* changing an established seed fails closed.
*/
queuedCommandResultProgressAt(commandId: string): number | null {
const queued = this.#queuedCommandResults.get(commandId);
return queued === undefined ? null : Math.max(queued.queuedAt, this.#inboundFrameSettledAt);
}
withdrawUndeliveredTerminalCommands(): string[] {
if (this.#store.state.warmTransition) return [];
const commands = this.#store.state.commands;
const firstPending = commands.findIndex((command) => command.status === "pending");
if (firstPending < 0) return [];
const suffix = commands.slice(firstPending);
const kept = suffix.filter((command) => !this.#isTerminalLifecycleCommand(command));
if (kept.length === suffix.length) return [];
const withdrawn = suffix.filter((command) => this.#isTerminalLifecycleCommand(command));
for (const command of withdrawn) delete this.#store.state.commandDeliveryCounts[command.commandId];
kept.forEach((command, index) => { command.controllerSeq = firstPending + index + 1; });
commands.splice(firstPending, suffix.length, ...kept);
this.#store.save();
return withdrawn.map((command) => command.type);
}
persistRunAttachTemplate(runAttachTemplate: Record<string, unknown>): void {
if (!isRecord(runAttachTemplate.provider)) {
throw new Error("Durable PRP run attachment template is invalid.");
@@ -2950,14 +2973,18 @@ export class DurablePrpControlPlane {
}
}
const controllerSeq = this.#store.state.commands.length + 1;
const sequenceId = `command_prp_${controllerSeq.toString().padStart(8, "0")}`;
let generatedId = sequenceId;
for (let reuse = 1; this.#store.state.commands.some((candidate) => candidate.commandId === generatedId); reuse += 1) {
generatedId = `${sequenceId}_${reuse}`;
}
const command: DurableRecoveryCoreCommand = {
schema: type === "external_provider.operation" || type === "runner.attachment.download"
? "paperclip.prp.command.v3"
: type.startsWith("session.goal.")
? "paperclip.prp.command.v2"
: "paperclip.prp.command.v1",
commandId:
commandId ?? `command_prp_${controllerSeq.toString().padStart(8, "0")}`,
commandId: commandId ?? generatedId,
controllerSeq,
type,
issuedAt: new Date().toISOString(),
@@ -3091,11 +3118,25 @@ export class DurablePrpControlPlane {
undecryptedFrames += 1;
frame = { raw: value };
}
const answeredCommandId = "decrypted" in frame && frame.decrypted.kind === "command_result"
? (frame.decrypted.payload as Record<string, unknown> | undefined)?.commandId : undefined;
const queuedResult = typeof answeredCommandId === "string" ? answeredCommandId : null;
if (queuedResult !== null) {
const queued = this.#queuedCommandResults.get(queuedResult);
this.#queuedCommandResults.set(queuedResult, { count: (queued?.count ?? 0) + 1, queuedAt: queued?.queuedAt ?? Date.now() });
}
processing = processing
.then(() => "raw" in frame
? this.#handleJson(connection, frame.raw).finally(() => { undecryptedFrames -= 1; })
: this.#handleDecryptedFrame(connection, frame.decrypted))
.catch(() => connection.close());
.catch(() => connection.close())
.finally(() => {
this.#inboundFrameSettledAt = Date.now();
if (queuedResult === null) return;
const queued = this.#queuedCommandResults.get(queuedResult);
if (queued !== undefined && queued.count > 1) queued.count -= 1;
else this.#queuedCommandResults.delete(queuedResult);
});
const tail = processing;
this.#connectionProcessing.set(connection, tail);
const release = () => {
@@ -0,0 +1,279 @@
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { createServer } from "node:http";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { expect, it, vi } from "vitest";
import { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js";
import { createCapabilityRunnerdCodexTransport, RunnerStallLedger, waitForRunnerCommand } from "./runnerd-codex-transport.js";
import { defaultCapabilityRunnerdBinary, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js";
type Carried = "late-tool-result" | "close-path-suspend" | "suspend-then-tool-result" | "event-backlog";
const BACKLOG = 400;
const taskTool = { name: "get_task_context", description: "Read current task", inputSchema: { type: "object", properties: {}, additionalProperties: false } };
async function reattachWithCarriedCommands(carried: Carried) {
const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-reattach-carried-"));
const controlPlaneStatePath = join(stateDirectory, "control-plane", "control-plane-state.json");
const server = createServer();
let authority: DurablePrpControlPlane | null = null;
server.on("upgrade", (request, socket, head) => {
if (!authority) {
socket.destroy();
return;
}
authority.handleUpgrade(request, socket, "/runner", head);
});
await new Promise<void>(resolveListen => server.listen(0, "127.0.0.1", resolveListen));
const address = server.address();
if (!address || typeof address === "string") throw new Error("Expected a reattachment test listener");
const registration = vi.fn(async (next: DurablePrpControlPlane) => {
authority = next;
return {
connectUrl: `ws://127.0.0.1:${address.port}/runner`,
release: async () => {
if (authority === next) authority = null;
},
};
});
const identity = {
runnerInstanceId: "runner-reattach-carried",
environmentLeaseId: "lease-reattach-carried",
runId: "run-reattach-carried",
normalizedSessionId: "session-reattach-carried",
turnId: "turn-reattach-carried",
itemId: "item-reattach-carried",
};
const toolCall = carried === "late-tool-result" || carried === "suspend-then-tool-result";
const options = {
runnerBinary: defaultCapabilityRunnerdBinary(),
codexCommand: fakeCodex,
codexArgs: fakeCodexArgs(stateDirectory, ...(toolCall ? ["--emit-tool-call"] : carried === "event-backlog"
? ["--hold-turn", "--held-output", "--held-output-count", String(BACKLOG), "--held-output-release-file", join(stateDirectory, "release-output")] : [])),
stateDirectory,
prpIdentity: identity,
lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 },
controlPlaneRegistration: registration,
};
const first = createCapabilityRunnerdCodexTransport(options);
let finishTool!: (value: Record<string, unknown>) => void;
const lateTool = vi.fn(() => new Promise<Record<string, unknown>>(resolveTool => { finishTool = resolveTool; }));
first.transport.setServerRequestHandler(lateTool);
let adopted: ReturnType<typeof createCapabilityRunnerdCodexTransport> | null = null;
let runnerPid: number | null = null;
try {
const opened = (await first.transport.request("thread/start", {
cwd: tmpdir(),
dynamicTools: toolCall ? [taskTool] : [],
})).thread as { id: string; sessionId: string };
runnerPid = first.evidence().runnerPid;
const runnerStartedAt = first.transport.processInfo!().startedAt;
if (carried === "event-backlog") {
await first.transport.request("turn/start", { input: [{ type: "text", text: "Keep working" }] });
}
if (toolCall) {
await first.transport.request("turn/start", { input: [{ type: "text", text: "Read current task" }] });
await vi.waitFor(() => expect(lateTool).toHaveBeenCalledOnce(), { timeout: 5_000 });
}
await first.detachControllerForRestart();
expect(() => process.kill(runnerPid!, 0)).not.toThrow();
if (toolCall) {
finishTool({ success: true, contentItems: [{ type: "inputText", text: "task context" }] });
await vi.waitFor(async () => {
const saved = JSON.parse(await readFile(controlPlaneStatePath, "utf8")) as { commands: Array<{ type: string; status: string }> };
expect(saved.commands.at(-1)).toMatchObject({ type: "semantic_tool.result", status: "pending" });
}, { timeout: 5_000 });
if (carried === "suspend-then-tool-result") {
const saved = JSON.parse(await readFile(controlPlaneStatePath, "utf8")) as { commands: Array<Record<string, unknown>> };
const result = saved.commands.pop()!;
for (const type of ["runner.drain", "runner.suspend"]) {
const controllerSeq = saved.commands.length + 1;
saved.commands.push({
schema: "paperclip.prp.command.v1",
commandId: `command_prp_${controllerSeq.toString().padStart(8, "0")}`,
controllerSeq,
type,
issuedAt: new Date().toISOString(),
payload: {},
status: "pending",
result: null,
});
}
saved.commands.push({ ...result, controllerSeq: saved.commands.length + 1 });
await writeFile(controlPlaneStatePath, JSON.stringify(saved), { mode: 0o600 });
}
} else if (carried === "event-backlog") {
await writeFile(join(stateDirectory, "release-output"), "release");
await new Promise(resolveWait => setTimeout(resolveWait, 3_000));
} else {
const saved = JSON.parse(await readFile(controlPlaneStatePath, "utf8")) as { commands: Array<Record<string, unknown>> };
for (const type of ["runner.drain", "runner.suspend"]) {
const controllerSeq = saved.commands.length + 1;
saved.commands.push({
schema: "paperclip.prp.command.v1",
commandId: `command_prp_${controllerSeq.toString().padStart(8, "0")}`,
controllerSeq,
type,
issuedAt: new Date().toISOString(),
payload: {},
status: "pending",
result: null,
});
}
await writeFile(controlPlaneStatePath, JSON.stringify(saved), { mode: 0o600 });
}
const redispatched = vi.fn(async () => ({ success: true, contentItems: [] }));
adopted = createCapabilityRunnerdCodexTransport({
...options,
onAccountingEvent: async () => {
if (carried === "event-backlog") await new Promise(resolveWait => setTimeout(resolveWait, 100));
},
resumeDynamicTools: toolCall ? [taskTool] : [],
resumeProviderSession: { driverSessionId: opened.id, providerSessionId: opened.sessionId },
runnerProcessLauncher: () => {
throw new Error("duplicate runner spawn attempted");
},
adoptExistingRunner: {
pid: runnerPid!,
processGroupId: runnerPid,
startedAt: runnerStartedAt,
signal: () => true,
isAlive: () => {
try {
process.kill(runnerPid!, 0);
return true;
} catch {
return false;
}
},
},
});
adopted.transport.setServerRequestHandler(redispatched);
await expect(adopted.transport.request("thread/read", {})).resolves.toEqual(
expect.objectContaining({ thread: expect.objectContaining({ id: opened.id }) }),
);
expect(() => process.kill(runnerPid!, 0)).not.toThrow();
const commands = authority!.store.state.commands;
expect(commands.map(command => command.controllerSeq)).toEqual(commands.map((_command, index) => index + 1));
expect(commands.some(command => command.type === "runner.suspend")).toBe(false);
if (toolCall) {
expect(commands.filter(command => command.type === "semantic_tool.result")).toMatchObject([{ status: "completed" }]);
expect(redispatched).not.toHaveBeenCalled();
}
} finally {
await adopted?.transport.close().catch(() => undefined);
await first.transport.close().catch(() => undefined);
if (runnerPid) {
try { process.kill(-runnerPid, "SIGKILL"); } catch { /* already exited */ }
}
server.closeAllConnections();
if (server.listening) await new Promise<void>(resolveClose => server.close(() => resolveClose()));
await rm(stateDirectory, { recursive: true, force: true });
}
}
it.each([
["a tool result its predecessor queued after the restart", "late-tool-result"],
["a runner.suspend left by an earlier attempt's close path", "close-path-suspend"],
["a runner.suspend with a tool result queued after it", "suspend-then-tool-result"],
["an event backlog ahead of its recovery snapshot result", "event-backlog"],
] as const)("reattaches a live runner past %s", async (_label, carried) => {
await reattachWithCarriedCommands(carried);
}, 120_000);
function pendingCommand() {
return { status: "pending" };
}
it("keeps waiting past the deadline while the runner's answer is queued behind frames the controller is still committing", async () => {
let progressAt = Date.now();
const command = pendingCommand();
const advancing = setInterval(() => { progressAt = Date.now(); }, 5);
setTimeout(() => { command.status = "completed"; }, 150);
try {
await expect(waitForRunnerCommand({
type: "session.snapshot",
deadline: Date.now() + 20,
abortOnClose: false,
isClosed: () => false,
throwIfFailed: () => undefined,
command: () => command,
runnerHasExited: async () => false,
failureCode: () => "provider_transport_failed",
answeredProgressAt: () => progressAt,
})).resolves.toBeUndefined();
} finally {
clearInterval(advancing);
}
});
it("times out when the runner's queued answer stops advancing", async () => {
await expect(waitForRunnerCommand({
type: "session.snapshot",
deadline: Date.now() + 20,
abortOnClose: false,
isClosed: () => false,
throwIfFailed: () => undefined,
command: () => pendingCommand(),
runnerHasExited: async () => false,
failureCode: () => "provider_transport_failed",
answeredProgressAt: () => Date.now() - 31_000,
})).rejects.toThrow("provider_transport_failed: PRP command session.snapshot timed out");
});
it("leaves later barriers that share an admission deadline their time after an answered barrier ran past it", async () => {
const clock = new RunnerStallLedger();
const deadline = clock.now() + 20;
let progressAt = Date.now();
const drain = pendingCommand();
const snapshot = pendingCommand();
const advancing = setInterval(() => { progressAt = Date.now(); }, 5);
setTimeout(() => { drain.status = "completed"; }, 150);
const wait = (type: string, command: { status: string }, answered: () => number | null) => waitForRunnerCommand({
type,
deadline,
clock,
abortOnClose: false,
isClosed: () => false,
throwIfFailed: () => undefined,
command: () => command,
runnerHasExited: async () => false,
failureCode: () => "provider_transport_failed",
answeredProgressAt: answered,
});
try {
await expect(wait("runner.drain", drain, () => progressAt)).resolves.toBeUndefined();
} finally {
clearInterval(advancing);
}
setTimeout(() => { snapshot.status = "completed"; }, 50);
await expect(wait("session.snapshot", snapshot, () => null)).resolves.toBeUndefined();
});
it("completes a command whose answer arrives during the last sleep before its deadline", async () => {
vi.useFakeTimers();
try {
const command = pendingCommand();
const deadline = Date.now() + 25;
const waited = waitForRunnerCommand({
type: "runner.drain",
deadline,
abortOnClose: false,
isClosed: () => false,
throwIfFailed: () => undefined,
command: () => command,
runnerHasExited: async () => false,
failureCode: () => "provider_transport_failed",
answeredProgressAt: () => null,
});
const outcome = waited.then(() => "completed", (error: unknown) => String(error));
await vi.advanceTimersByTimeAsync(20);
command.status = "completed";
await vi.advanceTimersByTimeAsync(10);
expect(Date.now()).toBeGreaterThanOrEqual(deadline);
await expect(outcome).resolves.toBe("completed");
} finally {
vi.useRealTimers();
}
});
@@ -95,7 +95,7 @@ it.each([
expect(result.runnerState).toBe("gone");
}, 30_000);
it("keeps a retained remote Runner for the reattaching controller, which settles it into a journal the next attempt can relaunch", async () => {
it("keeps a retained remote Runner for the reattaching controller, which continues it without the failed close's suspend", async () => {
const root = await mkdtemp(join(tmpdir(), "runnerd-remote-retention-adopt-"));
const server = createServer();
let authority: DurablePrpControlPlane | null = null;
@@ -172,9 +172,12 @@ it("keeps a retained remote Runner for the reattaching controller, which settles
},
});
adopted.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] }));
await adopted.transport.request("thread/read", {}).catch(() => undefined);
await vi.waitFor(() => expect(processState(adoptedPid)).toBe("gone"), { timeout: 10_000, interval: 50 });
expect(await journal()).toMatchObject({ lifecycle: "suspended", outbox: [], pendingTerminalDelivery: null });
await expect(adopted.transport.request("thread/read", {})).resolves.toEqual(
expect.objectContaining({ thread: expect.objectContaining({ id: thread.id }) }));
expect(processState(adoptedPid)).toBe("alive");
expect(await journal()).toMatchObject({ lifecycle: "ready", pendingTerminalDelivery: null });
expect(adopted.evidence().diagnostics.join("\n")).toContain("before reattaching the live runner");
expect(authority!.store.state.commands.some(command => command.type === "runner.suspend" && command.status === "pending")).toBe(false);
expect(duplicateLaunch).not.toHaveBeenCalled();
expect(launch).toHaveBeenCalledTimes(1);
} finally {
@@ -6822,6 +6822,12 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
});
this.#core = core;
this.#followCommittedEvents(core);
if (this.options.adoptExistingRunner) {
const withdrawn = core.withdrawUndeliveredTerminalCommands();
if (withdrawn.length > 0) {
this.#diagnostic(`withdrew ${withdrawn.join(", ")} queued by an earlier attempt before reattaching the live runner`);
}
}
this.#carriedCommandCount = core.store.state.commands.length;
// Reusing resources still requires current environment authorization. The
// optional hook avoids re-preparation after controller storage is lost.
@@ -7914,7 +7920,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
expectedEventType?: "harness.ready" | "session.started" | "session.resumed",
deadline = this.#waitNow() + 30_000,
): Promise<void> {
while (this.#waitNow() < deadline) {
for (;;) {
this.#throwIfFailed();
if (this.#closed) throw new Error("runnerd transport closed during provider startup");
this.#pumpEvents();
@@ -7929,6 +7935,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
return;
if (await this.#runnerHasExited())
throw new Error("runnerd exited before provider startup");
if (this.#waitNow() >= deadline) break;
await new Promise((resolveWait) => setTimeout(resolveWait, 10));
}
throw new Error("runnerd did not report its provider identity");
@@ -7962,6 +7969,20 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
: this.#core?.getCommand(commandId),
runnerHasExited: () => this.#runnerHasExited(),
failureCode: () => this.#startupComplete ? "provider_transport_failed" : this.#startupFailureCode,
answeredProgressAt: () => {
const core = this.#core;
if (core === null) return null;
const commands = core.store.state.commands;
const target = commandId === undefined ? commands.find(candidate => candidate.type === type) : core.getCommand(commandId);
if (target === undefined) return null;
let progressAt: number | null = null;
for (const command of commands) {
if (command.controllerSeq > target.controllerSeq || command.status !== "pending") continue;
const at = core.queuedCommandResultProgressAt(command.commandId);
if (at !== null && (progressAt === null || at > progressAt)) progressAt = at;
}
return progressAt;
},
describeStall: () => this.#core === null ? null : describeRunnerCommandStall({
commands: this.#core.store.state.commands,
commandId: commandId ?? this.#core.store.state.commands.find(candidate => candidate.type === type)?.commandId,
@@ -9583,6 +9604,11 @@ export class RunnerStallLedger {
return this.#creditedMs;
}
credit(ms: number): void {
this.now();
this.#creditedMs += Math.max(0, ms);
}
now(): number {
const at = this.wallNow();
if (at - this.#last > RUNNER_STALL_THRESHOLD_MS) {
@@ -9602,6 +9628,8 @@ export class RunnerStallLedger {
}
/** Shared polling path keeps startup cancellation separate from owned cleanup commands. */
const RUNNER_ANSWERED_RESULT_STALL_MS = 30_000;
export async function waitForRunnerCommand(input: {
type: string;
deadline: number;
@@ -9615,15 +9643,17 @@ export async function waitForRunnerCommand(input: {
failureCode: () => string;
/** Bounded, content-free evidence of what the command waited behind. */
describeStall?: () => string | null;
answeredProgressAt?: () => number | null;
}): Promise<void> {
const now = () => input.clock ? input.clock.now() : Date.now();
while (now() < input.deadline) {
input.throwIfFailed();
if (input.abortOnClose && input.isClosed()) {
throw new Error(`runnerd transport closed while waiting for ${input.type}`);
}
const answeredAndAdvancing = () => {
const progressAt = input.answeredProgressAt?.() ?? null;
return progressAt !== null && Date.now() - progressAt < RUNNER_ANSWERED_RESULT_STALL_MS;
};
let extended = false;
const settled = () => {
const command = input.command();
if (command?.status === "completed") return;
if (command?.status === "completed") return true;
if (command !== undefined && command.status !== "pending") {
const detail = record(record(command.result).result);
const reportedResult = readProviderFailure(detail.providerFailure)
@@ -9634,11 +9664,32 @@ export async function waitForRunnerCommand(input: {
Object.defineProperty(error, "runnerCommandResult", { value: structuredClone(command.result), enumerable: false });
throw error;
}
return false;
};
const completeExtended = () => {
input.clock?.credit(Math.max(0, now() - input.deadline) + RUNNER_POST_STOP_GRACE_MS);
};
let observed = false;
while (now() < input.deadline || answeredAndAdvancing()) {
observed = true;
if (now() >= input.deadline) extended = true;
input.throwIfFailed();
if (input.abortOnClose && input.isClosed()) {
throw new Error(`runnerd transport closed while waiting for ${input.type}`);
}
if (settled()) {
if (extended) completeExtended();
return;
}
if (await input.runnerHasExited()) {
throw new Error(`runnerd exited while waiting for ${input.type}`);
}
await new Promise((resolveWait) => setTimeout(resolveWait, 10));
}
if (observed && settled()) {
if (extended) completeExtended();
return;
}
let stall: string | null = null;
try { stall = input.describeStall?.() ?? null; } catch { stall = null; }
throw new Error(`${input.failureCode()}: PRP command ${input.type} timed out${stall ? ` (${stall})` : ""}`);