mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs execute in a remote Runner that the server controls over the durable PRP channel > - When a run is interrupted, a same-run retry reattaches the live Runner and reads its session (`session.snapshot`) before it continues > - On staging, the reattached Runner authenticated and restored its session, and then did not answer the read > - The recovery kept that Runner, so each later attempt adopted the same Runner and failed the same way until the attempt budget ended the run > - This pull request makes the next attempt replace that Runner with a fresh Runner over its saved journal, and makes a failed Codex history read say what failed > - The benefit is that an interrupted Codex run continues without a manual Retry, and its provider turn is not sent again ## Linked Issues or Issue Description Refs #21161, Refs #21053, Refs #20853 **What happened?** A Codex agent on Daytona ran recovery runs (wake `issue_recovery_action_restored`). In one hour, 11 of about 35 finished runs failed. All failed the same way: - `native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: runnerd exited while waiting for session.snapshot` (8 runs) - `native_session_recovery_failed: HarnessReconciliationError: codex_history_read_failed: thread/turns/list requires supported paginated history; Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 338 76s ago)` (3 runs) The run log of each run shows one sequence: 1. Attempt 1 fails with `runnerd did not report the provider turn identity`. The close leaves the Runner alive and unsettled for the same-run reattachment. 2. Attempt 2 adopts that Runner (same PID). The Runner authenticates. The log shows `runnerd restored its durable PRP session and provider thread`. Then the driver's recovery read (`thread/read` or `thread/turns/list`, which both send `session.snapshot`) gets no answer. The Runner still answers WebSocket pings, but it sends no command result and almost no events (`pending ahead 0`, last event 76 s to 391 s earlier). The recovery treats this as transient and detaches. The Runner stays alive. 3. Attempt 3 adopts the same Runner again and fails the same way. In some runs a protocol fault (`thread_binding_mismatch`) closes the transport during the read, so the message is `runnerd exited while waiting for session.snapshot`. 4. The provider attempt budget ends the run. #21161 did not help. Its wait extension applies only while a queued answer is being committed. Here the Runner sent no answer at all. The `paginated history` text was also wrong. `codex-history.ts` called every failed history read unsupported, also a transport timeout. **Expected behavior** A Runner that left a recovery read unanswered is not adopted again. The next attempt starts a fresh Runner from the saved journal, resumes the provider session from the saved checkpoint, and does not send the provider turn again. A failed history read reports the real failure. A Codex build that refuses paginated history falls back to one complete thread read. **Steps to reproduce** 1. Start a Codex run in a remote (Daytona) Runner. 2. Let the first attempt fail at turn start, so that the close keeps the Runner alive. 3. Let the reattached Runner stop answering commands after it authenticates (for example, stop the Runner process after its first answer). 4. See `PRP command session.snapshot timed out (runner connected; pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3, with the same Runner PID. **Paperclip version or commit** Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on `master` after `53608bd1d6`. **Deployment mode** Hosted (staging), remote Daytona sandbox Runner. **Agent adapter(s) involved** Codex (native Runner, `paperclip-runnerd`, `codex_app_server`). ## What Changed - `nativeRecoveryCommandUnanswered` (executor) finds a `native_session_recovery_failed` whose Runner did not answer a recovery command (`PRP command <type> timed out` or `runnerd exited while waiting for <type>`). Identity, authority, integrity and incompatibility refusals do not match. - When such a failure ends a `reattach_remote_runner` attempt on a sandbox target, the failure detail records `adoptedRunnerUnanswered: true`. - `claimNativeRestartRecoveries` copies it to the next claim as `replaceUnansweredRunner: true`, in the same way as `exitedBeforeSessionOpen`. - `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps all identity checks. When the old Runner is still alive, it returns no adopted process, and the attempt launches a fresh Runner from the saved journal. The existing launch fence stops the old `paperclip-runnerd` on that state folder and its whole lineage, and fails the launch closed when it cannot prove them gone. A Runner that already exited mid-turn keeps the existing fail-closed path, because its provider or tool processes can survive where the fence cannot find them. The task then continues in a fresh session from the saved work. The attempt budget is unchanged. - `codex-history.ts`: a history read that fails without a provider answer is now `codex_history_read_failed: <method> read failed; <error>`. A provider refusal of the method (`-32601` or `-32600`) on the first page falls back to one `thread/read` with `includeTurns: true`. That fallback fails closed when the thread differs, or when turns, items or turn identities are missing. A refusal on a later page still fails closed. - `doc/execution-semantics.md` describes both rules. The launch-fence code comment now names the replacement case. ### Changed existing test `codex-history.test.ts` "does not turn an interrupted page or unsupported method into idle history" used a plain `Error("method not found")` on page 2 and expected the `supported paginated history` text. A plain error is now a failed read, not an unsupported method. The test now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2 and still expects a fail-closed rejection. A new test covers the failed-read message. ## Verification - Red before the fix, green after: - `server/src/services/native-runtime/native-session-executor.test.ts` › "a same-run reattachment whose adopted Runner never answered its recovery read": the replacement and staging-message tests failed on the old code (the Runner was adopted again, the three staging messages were not recognized). Two further tests ("never resumes beside an exited unanswered Runner's lineage") failed on the first revision of this PR, which relaunched beside an exited mid-turn Runner, and pass now. All 15 pass. - `native-runner-restart-recovery.integration.test.ts` › "asks the same-run reattachment to replace a Runner that left its recovery read unanswered" (embedded Postgres): the claim had no `replaceUnansweredRunner` on the old code. Now passes. - `codex-history.test.ts`: 7 new tests failed on the old code. All 16 pass. - Real `paperclip-runnerd` and fake Codex: `runnerd-codex-transport-remote-retention.test.ts` › "continues the same provider thread from a fresh Runner over the journal of a retained Runner that stopped answering, without replaying its turn". It runs the staging sequence: turn started, close keeps the stopped Runner, a second controller adopts it, the Runner stops answering and the controller detaches, the Runner and provider are killed (the launch fence), and a fresh Runner over the same journal returns the same thread. The fake Codex call log shows one `thread/start` and one `turn/start`. - Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus the remote-retention, adoption and cold-restore transport suites (442 tests). Two suites timed out under parallel load and pass alone. Server: `native-session-executor`, `native-restart-recovery`, `native-runner-restart-recovery.integration`, `runner-lost-mid-turn-continuation` (1714 passed, 21 skipped). - `pnpm typecheck` in `server` (all layers) and `tsc` for `paperclip-runner` source and tests pass. - Not run locally: the container-gated remote Runner suites (`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the full suite. ## Risks - Medium. The replacement relaunches a Runner over a journal that is not settled, but only while the old Runner is alive. Safety comes from three facts: the launch fence stops the old Runner and its lineage with proof or fails the launch; the Runner restores its own durable journal (as local `resume_dead_runner` does); and the real-runnerd test shows that the turn is not sent again. An exited Runner is never relaunched beside its possible survivors. - If the previous close left a `runner.suspend` pending, a fresh Runner can execute it and exit. The attempt then fails as it does today. The usual detach path after an unanswered read sends no suspend. - The replacement uses the existing attempt budget. It does not add attempts. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run events and logs), extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
9b613562b7
commit
497ccba8f1
8 files changed
+389
-45
No files matched your search
@@ -1782,6 +1782,35 @@ in-flight turn stay for the next same-run reattachment. An abort (a Stop or the
|
||||
recovery deadline), any identity or authority refusal, any other recovery
|
||||
failure, and a protocol integrity fault still stop it.
|
||||
|
||||
A kept Runner can answer its WebSocket pings but no recovery command. On
|
||||
staging, a reattached Codex Runner authenticated and restored its session, then
|
||||
left `session.snapshot` unanswered (`pending ahead 0; last runner event seq 338
|
||||
76s ago`), and each later attempt adopted the same Runner and failed the same
|
||||
way until the attempt budget ended the run. When a same-run reattachment fails
|
||||
because its Runner left a recovery command unanswered (`PRP command <type> timed
|
||||
out`, or `runnerd exited while waiting for <type>`), and the failure is not an
|
||||
identity, authority or integrity refusal, the run records
|
||||
`adoptedRunnerUnanswered` on its failure and the next claim carries
|
||||
`replaceUnansweredRunner`. That attempt does not adopt the Runner again. After
|
||||
the usual identity checks it launches a fresh Runner from the Runner's saved
|
||||
journal and resumes the provider session from the saved checkpoint. This applies
|
||||
only while the old Runner is alive: the launch fence stops that
|
||||
`paperclip-runnerd` on the state folder and its whole process lineage, and fails
|
||||
the launch closed when it cannot prove them gone. A Runner that already exited
|
||||
mid-turn can leave provider or tool processes that the fence cannot find, so it
|
||||
keeps the fail-closed path below, and the task continues in a fresh session from
|
||||
the saved work. The fresh Runner
|
||||
restores the provider thread; its in-flight turn is reconciled from history and
|
||||
never submitted again. The provider attempt budget still bounds these attempts.
|
||||
|
||||
A paginated Codex history read (`thread/turns/list`, `thread/items/list`) that
|
||||
fails without an answer is reported as `codex_history_read_failed: <method> read
|
||||
failed`, with the underlying error. Only a provider answer that refuses the method
|
||||
on its first page means the build lacks paginated history. Then the driver reads
|
||||
the complete thread once (`thread/read` with `includeTurns: true`) and uses its
|
||||
turns and items, failing closed when that read names another thread or omits
|
||||
turns, items or turn identities. A refusal on a later page still fails closed.
|
||||
|
||||
A failed close can leave its `runner.suspend` (or `runner.shutdown`) pending in
|
||||
the saved journal of a Runner it kept alive. The controller sends one command at
|
||||
a time, so neither that command nor any command queued after it reached the
|
||||
@@ -5896,7 +5925,8 @@ lifecycle `suspended`, an empty outbox and no pending terminal delivery, provide
|
||||
cleanup or warm transition, the retry launches a Runner from that state instead
|
||||
of adopting one. This matches a settled staged source. Any other exit stays
|
||||
fail-closed for reconciliation, and the provider attempt budget still bounds the
|
||||
retries.
|
||||
retries. A live Runner that left the previous attempt's recovery read unanswered
|
||||
is replaced, not adopted (see above).
|
||||
The final admitted provider attempt may use this path; attempts above the limit
|
||||
cannot. Process verification and the final state read share one recovery
|
||||
deadline. A timeout, command failure, transport failure or invalid saved JSON on
|
||||
|
||||
@@ -1,10 +1,13 @@
|
||||
import { describe, it, expect, vi } from "vitest";
|
||||
import { CODEX_METHOD_NOT_FOUND, CodexRpcError } from "./app-server-transport.js";
|
||||
import {
|
||||
readCodexThreadState,
|
||||
readCodexTurnMetadata,
|
||||
readCodexTurnItems,
|
||||
} from "./codex-history.js";
|
||||
|
||||
const unsupported = (method: string) => new CodexRpcError(`${method} is not supported`, CODEX_METHOD_NOT_FOUND);
|
||||
|
||||
describe("Codex paginated history", () => {
|
||||
it("reads state without hydrating history", async () => {
|
||||
const request = vi.fn().mockResolvedValue({ thread: { id: "thread" } });
|
||||
@@ -79,14 +82,67 @@ describe("Codex paginated history", () => {
|
||||
"codex_history_incomplete",
|
||||
);
|
||||
});
|
||||
it("does not turn an interrupted page or unsupported method into idle history", async () => {
|
||||
it("does not turn an unsupported later page into idle history", async () => {
|
||||
const request = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce({ data: [], nextCursor: "next" })
|
||||
.mockRejectedValueOnce(new Error("method not found"));
|
||||
.mockRejectedValueOnce(unsupported("thread/turns/list"));
|
||||
await expect(readCodexTurnMetadata({ request }, "thread")).rejects.toThrow(
|
||||
"supported paginated history",
|
||||
);
|
||||
expect(request.mock.calls.map(call => call[0])).toEqual(["thread/turns/list", "thread/turns/list"]);
|
||||
});
|
||||
it("names a failed history read as a failed read, not as unsupported history", async () => {
|
||||
const timeout = new Error("provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0)");
|
||||
const request = vi.fn().mockRejectedValueOnce(timeout);
|
||||
await expect(readCodexTurnMetadata({ request }, "thread")).rejects.toThrow(
|
||||
`codex_history_read_failed: thread/turns/list read failed; Error: ${timeout.message}`,
|
||||
);
|
||||
expect(request).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
it("reads complete turn history from the thread when paginated history is unsupported", async () => {
|
||||
const request = vi.fn(async (method: string) => {
|
||||
if (method === "thread/turns/list") throw unsupported(method);
|
||||
return { thread: { id: "thread", turns: [
|
||||
{ id: "one", status: "completed", items: [{ id: "answer", type: "agentMessage", text: "done" }] },
|
||||
{ id: "two", status: "inProgress", items: [] },
|
||||
] } };
|
||||
});
|
||||
expect(await readCodexTurnMetadata({ request }, "thread")).toEqual([
|
||||
{ id: "one", status: "completed", items: [], itemsView: "notLoaded" },
|
||||
{ id: "two", status: "inProgress", items: [], itemsView: "notLoaded" },
|
||||
]);
|
||||
expect(request).toHaveBeenLastCalledWith("thread/read", { threadId: "thread", includeTurns: true });
|
||||
});
|
||||
it("reads a turn's items from the thread when paginated items are unsupported", async () => {
|
||||
const answer = { id: "answer", type: "agentMessage", text: "done" };
|
||||
const request = vi.fn(async (method: string) => {
|
||||
if (method === "thread/items/list") throw unsupported(method);
|
||||
return { thread: { id: "thread", turns: [
|
||||
{ id: "one", status: "completed", items: [answer] },
|
||||
{ id: "two", status: "inProgress", items: [{ id: "other" }] },
|
||||
] } };
|
||||
});
|
||||
expect(await readCodexTurnItems({ request }, "thread", "one")).toEqual([answer]);
|
||||
});
|
||||
it.each([
|
||||
["another thread", { thread: { id: "other", turns: [] } }, "different driver session"],
|
||||
["no turns", { thread: { id: "thread" } }, "supported paginated history"],
|
||||
["a turn without a status", { thread: { id: "thread", turns: [{ id: "one" }] } }, "codex_history_incomplete"],
|
||||
["a turn without an id", { thread: { id: "thread", turns: [{ status: "completed" }] } }, "codex_history_incomplete"],
|
||||
])("fails closed when the unpaginated thread read returns %s", async (_case, thread, message) => {
|
||||
const request = vi.fn(async (method: string) => {
|
||||
if (method === "thread/turns/list") throw unsupported(method);
|
||||
return thread;
|
||||
});
|
||||
await expect(readCodexTurnMetadata({ request }, "thread")).rejects.toThrow(message);
|
||||
});
|
||||
it("fails closed when the unpaginated thread read omits the requested turn's items", async () => {
|
||||
const request = vi.fn(async (method: string) => {
|
||||
if (method === "thread/items/list") throw unsupported(method);
|
||||
return { thread: { id: "thread", turns: [{ id: "one", status: "completed" }] } };
|
||||
});
|
||||
await expect(readCodexTurnItems({ request }, "thread", "one")).rejects.toThrow("codex_history_incomplete");
|
||||
});
|
||||
it("rejects another thread or turn", async () => {
|
||||
await expect(
|
||||
|
||||
@@ -1,9 +1,13 @@
|
||||
import type { CodexAppServerTransport } from "./app-server-transport.js";
|
||||
import { isCodexMethodUnavailable, type CodexAppServerTransport } from "./app-server-transport.js";
|
||||
import { HarnessReconciliationError } from "../../contracts/harness-driver.js";
|
||||
import { record, text } from "./codex-driver-values.js";
|
||||
|
||||
type Requester = Pick<CodexAppServerTransport, "request">;
|
||||
|
||||
const TURN_STATUSES = ["inProgress", "completed", "failed", "interrupted", "cancelled"];
|
||||
|
||||
class CodexPaginatedHistoryUnsupported extends HarnessReconciliationError {}
|
||||
|
||||
/** A partial or unsupported provider read is never evidence that work is idle. */
|
||||
async function pages(
|
||||
transport: Requester,
|
||||
@@ -23,9 +27,14 @@ async function pages(
|
||||
...(cursor ? { cursor } : {}),
|
||||
});
|
||||
} catch (error) {
|
||||
throw new HarnessReconciliationError(
|
||||
`codex_history_read_failed: ${method} requires supported paginated history; ${String(error)}`,
|
||||
);
|
||||
if (!isCodexMethodUnavailable(error))
|
||||
throw new HarnessReconciliationError(
|
||||
`codex_history_read_failed: ${method} read failed; ${String(error)}`,
|
||||
);
|
||||
const unsupported = `codex_history_read_failed: ${method} requires supported paginated history; ${String(error)}`;
|
||||
throw page === 0
|
||||
? new CodexPaginatedHistoryUnsupported(unsupported)
|
||||
: new HarnessReconciliationError(unsupported);
|
||||
}
|
||||
if (!Array.isArray(response.data))
|
||||
throw new HarnessReconciliationError(
|
||||
@@ -71,31 +80,57 @@ export async function readCodexThreadState(
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
export function readCodexTurnMetadata(
|
||||
function turnIdentity(value: Record<string, unknown>): string {
|
||||
if (!TURN_STATUSES.includes(text(value.status))) {
|
||||
throw new HarnessReconciliationError(
|
||||
"codex_history_incomplete: invalid turn status",
|
||||
);
|
||||
}
|
||||
return text(value.id);
|
||||
}
|
||||
|
||||
async function readCompleteThreadTurns(
|
||||
transport: Requester,
|
||||
threadId: string,
|
||||
method: string,
|
||||
cause: unknown,
|
||||
): Promise<Record<string, unknown>[]> {
|
||||
if (!(cause instanceof CodexPaginatedHistoryUnsupported)) throw cause;
|
||||
const snapshot = await transport.request("thread/read", { threadId, includeTurns: true });
|
||||
const thread = record(snapshot.thread);
|
||||
if (text(thread.id) !== threadId)
|
||||
throw new HarnessReconciliationError(
|
||||
"thread/read returned a different driver session",
|
||||
);
|
||||
if (!Array.isArray(thread.turns))
|
||||
throw new HarnessReconciliationError(
|
||||
`codex_history_read_failed: ${method} requires supported paginated history; thread/read omitted turns`,
|
||||
);
|
||||
const turns = thread.turns.map(record);
|
||||
for (const turn of turns) {
|
||||
if (!turnIdentity(turn))
|
||||
throw new HarnessReconciliationError(
|
||||
"codex_history_incomplete: thread/read omitted an identity",
|
||||
);
|
||||
}
|
||||
return turns;
|
||||
}
|
||||
|
||||
export async function readCodexTurnMetadata(
|
||||
transport: Requester,
|
||||
threadId: string,
|
||||
): Promise<Record<string, unknown>[]> {
|
||||
return pages(
|
||||
transport,
|
||||
"thread/turns/list",
|
||||
{ threadId, sortDirection: "asc", itemsView: "notLoaded" },
|
||||
(value) => {
|
||||
if (
|
||||
![
|
||||
"inProgress",
|
||||
"completed",
|
||||
"failed",
|
||||
"interrupted",
|
||||
"cancelled",
|
||||
].includes(text(value.status))
|
||||
) {
|
||||
throw new HarnessReconciliationError(
|
||||
"codex_history_incomplete: invalid turn status",
|
||||
);
|
||||
}
|
||||
return text(value.id);
|
||||
},
|
||||
);
|
||||
try {
|
||||
return await pages(
|
||||
transport,
|
||||
"thread/turns/list",
|
||||
{ threadId, sortDirection: "asc", itemsView: "notLoaded" },
|
||||
turnIdentity,
|
||||
);
|
||||
} catch (error) {
|
||||
const turns = await readCompleteThreadTurns(transport, threadId, "thread/turns/list", error);
|
||||
return turns.map((turn) => ({ ...turn, items: [], itemsView: "notLoaded" }));
|
||||
}
|
||||
}
|
||||
|
||||
export async function readCodexTurnItems(
|
||||
@@ -103,17 +138,28 @@ export async function readCodexTurnItems(
|
||||
threadId: string,
|
||||
turnId: string,
|
||||
): Promise<Record<string, unknown>[]> {
|
||||
const entries = await pages(
|
||||
transport,
|
||||
"thread/items/list",
|
||||
{ threadId, turnId, sortDirection: "asc" },
|
||||
(value) => {
|
||||
if (text(value.turnId) !== turnId)
|
||||
throw new HarnessReconciliationError(
|
||||
"thread/items/list returned a different turn",
|
||||
);
|
||||
return text(record(value.item).id);
|
||||
},
|
||||
);
|
||||
let entries: Record<string, unknown>[];
|
||||
try {
|
||||
entries = await pages(
|
||||
transport,
|
||||
"thread/items/list",
|
||||
{ threadId, turnId, sortDirection: "asc" },
|
||||
(value) => {
|
||||
if (text(value.turnId) !== turnId)
|
||||
throw new HarnessReconciliationError(
|
||||
"thread/items/list returned a different turn",
|
||||
);
|
||||
return text(record(value.item).id);
|
||||
},
|
||||
);
|
||||
} catch (error) {
|
||||
const turns = await readCompleteThreadTurns(transport, threadId, "thread/items/list", error);
|
||||
const items = turns.find((turn) => text(turn.id) === turnId)?.items;
|
||||
if (!Array.isArray(items))
|
||||
throw new HarnessReconciliationError(
|
||||
"codex_history_incomplete: thread/read omitted the turn's items",
|
||||
);
|
||||
return items.map(record);
|
||||
}
|
||||
return entries.map((entry) => record(entry.item));
|
||||
}
|
||||
@@ -189,3 +189,111 @@ it("keeps a retained remote Runner for the reattaching controller, which continu
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
it("continues the same provider thread from a fresh Runner over the journal of a retained Runner that stopped answering, without replaying its turn", async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), "runnerd-remote-retention-replace-"));
|
||||
const callLog = join(root, "provider-calls.log");
|
||||
const server = createServer();
|
||||
let authority: DurablePrpControlPlane | null = null;
|
||||
server.on("upgrade", (request, socket, head) => {
|
||||
if (!authority) { socket.destroy(); return; }
|
||||
authority.handleUpgrade(request, socket, "/runner", head);
|
||||
});
|
||||
await new Promise<void>(resolveListen => server.listen(0, "127.0.0.1", resolveListen));
|
||||
const address = server.address();
|
||||
if (!address || typeof address === "string") throw new Error("Missing replacement test listener");
|
||||
const registration = async (next: DurablePrpControlPlane) => {
|
||||
authority = next;
|
||||
return { connectUrl: `ws://127.0.0.1:${address.port}/runner`, release: () => { if (authority === next) authority = null; } };
|
||||
};
|
||||
const children: ReturnType<typeof spawn>[] = [];
|
||||
const launch = vi.fn((spec: durableControlPlane.RunnerProcessLaunchSpec) => {
|
||||
const child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: spec.environment, stdio: "ignore", detached: true });
|
||||
children.push(child);
|
||||
return {
|
||||
child,
|
||||
completion: new Promise<durableControlPlane.RunnerProcessResult>((resolveExit, rejectExit) => {
|
||||
child.once("error", rejectExit);
|
||||
child.once("exit", (code, exitSignal) => resolveExit({ code, signal: exitSignal, stdout: "", stderr: "" }));
|
||||
}),
|
||||
isExitConfirmed: () => child.exitCode !== null || child.signalCode !== null,
|
||||
};
|
||||
});
|
||||
const shared = {
|
||||
runnerBinary: defaultCapabilityRunnerdBinary(),
|
||||
codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"),
|
||||
codexArgs: ["--state-file", join(root, "provider-state.json"), "--hold-first-durable-turn", "--call-log", callLog],
|
||||
stateDirectory: root,
|
||||
prpIdentity: {
|
||||
runnerInstanceId: "runner-replaced", environmentLeaseId: "lease-replaced", runId: "run-replaced",
|
||||
normalizedSessionId: "session-replaced", turnId: "turn-replaced", itemId: "item-replaced",
|
||||
},
|
||||
runnerReconnectGraceMs: 5_000,
|
||||
runnerProcessReconnectGraceMs: null,
|
||||
controlPlaneRegistration: registration,
|
||||
runnerProcessLauncher: launch,
|
||||
};
|
||||
const first = createCapabilityRunnerdCodexTransport({ ...shared, closeGraceMs: 600 });
|
||||
let adopted: ReturnType<typeof createCapabilityRunnerdCodexTransport> | undefined;
|
||||
let replacement: ReturnType<typeof createCapabilityRunnerdCodexTransport> | undefined;
|
||||
const providerPids: number[] = [];
|
||||
try {
|
||||
const opened = await first.transport.request("thread/start", { cwd: root, dynamicTools: codexSemanticToolSpecs() });
|
||||
await first.transport.request("turn/start", { input: [{ type: "text", text: "Keep this task active." }] });
|
||||
for await (const notification of first.transport.notifications()) {
|
||||
if (notification.method === "turn/started") break;
|
||||
}
|
||||
const wedged = children[0]!.pid!;
|
||||
providerPids.push(first.evidence().codexPid!);
|
||||
signal(wedged, "SIGSTOP");
|
||||
await expect(first.transport.close("native session execution complete", { retainUnsettledRemoteRunner: true }))
|
||||
.rejects.toBeInstanceOf(NativeSessionCloseUnrecoverableError);
|
||||
expect(processState(wedged)).toBe("alive");
|
||||
signal(wedged, "SIGCONT");
|
||||
const thread = opened.thread as Record<string, unknown>;
|
||||
const resume = {
|
||||
resumeDynamicTools: codexSemanticToolSpecs(),
|
||||
resumeProviderSession: { driverSessionId: String(thread.id), providerSessionId: String(thread.sessionId) },
|
||||
};
|
||||
adopted = createCapabilityRunnerdCodexTransport({
|
||||
...shared, ...resume,
|
||||
runnerProcessLauncher: vi.fn(() => { throw new Error("Must adopt the existing runner"); }),
|
||||
adoptExistingRunner: {
|
||||
pid: wedged, processGroupId: wedged, startedAt: new Date().toISOString(),
|
||||
isAlive: () => processState(wedged) === "alive",
|
||||
signal: name => { try { process.kill(-wedged, name); return true; } catch { return false; } },
|
||||
},
|
||||
});
|
||||
adopted.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] }));
|
||||
await expect(adopted.transport.request("thread/read", {})).resolves.toEqual(
|
||||
expect.objectContaining({ thread: expect.objectContaining({ id: thread.id }) }));
|
||||
signal(wedged, "SIGSTOP");
|
||||
await adopted.transport.detachControllerForRestart!();
|
||||
expect(processState(wedged)).toBe("alive");
|
||||
const journal = JSON.parse(await readFile(join(root, "runner", "runner-state.json"), "utf8")) as Record<string, unknown>;
|
||||
expect(journal.lifecycle).not.toBe("suspended");
|
||||
signal(-wedged, "SIGKILL");
|
||||
for (const pid of providerPids) signal(pid, "SIGKILL");
|
||||
await vi.waitFor(() => expect(processState(wedged)).toBe("gone"), { timeout: 5_000 });
|
||||
replacement = createCapabilityRunnerdCodexTransport({
|
||||
...shared, ...resume,
|
||||
});
|
||||
replacement.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] }));
|
||||
await expect(replacement.transport.request("thread/read", {})).resolves.toEqual(
|
||||
expect.objectContaining({ thread: expect.objectContaining({ id: thread.id }) }));
|
||||
if (replacement.evidence().codexPid) providerPids.push(replacement.evidence().codexPid!);
|
||||
expect(launch).toHaveBeenCalledTimes(2);
|
||||
expect(replacement.evidence().runnerPid).toBe(children[1]!.pid);
|
||||
const calls = (await readFile(callLog, "utf8")).trim().split("\n");
|
||||
expect(calls.filter(call => call === "thread/start")).toHaveLength(1);
|
||||
expect(calls.filter(call => call === "turn/start")).toHaveLength(1);
|
||||
} finally {
|
||||
await replacement?.transport.close().catch(() => undefined);
|
||||
await adopted?.transport.close().catch(() => undefined);
|
||||
for (const child of children) { signal(child.pid, "SIGCONT"); signal(child.pid ? -child.pid : null, "SIGKILL"); }
|
||||
for (const pid of providerPids) signal(pid, "SIGKILL");
|
||||
server.closeAllConnections();
|
||||
if (server.listening) await new Promise<void>(resolveClose => server.close(() => resolveClose()));
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
}, 60_000);
|
||||
@@ -103,6 +103,7 @@ export type NativeRestartRecoveryClaim =
|
||||
/** Discovery evidence only; executor revalidates the encrypted header. */
|
||||
stagedAttachment?: Pick<NativeStagedAttachmentRecovery, "sourceIdentity" | "targetIdentity" | "templateDigest">;
|
||||
exitedBeforeSessionOpen?: true;
|
||||
replaceUnansweredRunner?: true;
|
||||
}
|
||||
| {
|
||||
kind: "resume_dead_runner";
|
||||
@@ -2121,6 +2122,8 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
},
|
||||
...(!pluginTask && row.coordinator.failureDetail?.runnerExitedBeforeSessionOpen === true
|
||||
? { exitedBeforeSessionOpen: true as const } : {}),
|
||||
...(!pluginTask && row.coordinator.failureDetail?.adoptedRunnerUnanswered === true
|
||||
? { replaceUnansweredRunner: true as const } : {}),
|
||||
} satisfies NativeRestartRecoveryClaim;
|
||||
}
|
||||
if (claimKind === "reattach_existing_runner") {
|
||||
|
||||
@@ -1573,6 +1573,39 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
|
||||
expect(run?.processPid).toBe(remotePid);
|
||||
});
|
||||
|
||||
it.each([true, false])("asks the same-run reattachment to replace a Runner that left its recovery read unanswered: %s", async (unanswered) => {
|
||||
const fixture = await seedRun(`remote-unanswered-${unanswered}`);
|
||||
await fixture.db.update(heartbeatRuns).set({
|
||||
processPid: null,
|
||||
processStartedAt: new Date("2020-01-01T00:00:00Z"),
|
||||
runnerProfileJson: {
|
||||
nativeWorkspaceSync: {
|
||||
schema: "paperclip.native-workspace-sync/v1", state: "prepared",
|
||||
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64),
|
||||
workspaceId: "workspace", leaseId: "lease", providerLeaseId: "sandbox",
|
||||
remoteCwd: "/workspace",
|
||||
},
|
||||
sessionCheckpoint: {
|
||||
identity: { companyId, agentId, issueId: fixture.issueId, runId: fixture.runId,
|
||||
sessionId: fixture.native.normalizedSessionId },
|
||||
providerSessionId: "provider-session",
|
||||
process: { providerPid: process.pid },
|
||||
},
|
||||
},
|
||||
}).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
await fixture.db.update(nativeRunFinalizations).set({ attempt: 2, failureDetail: {
|
||||
message: "native_session_recovery_failed: HarnessReconciliationError: codex_history_read_failed: thread/turns/list requires supported paginated history; Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 337 111s ago)",
|
||||
recoveryMode: "exact_checkpoint_resume", checkpointExists: true,
|
||||
...(unanswered ? { adoptedRunnerUnanswered: true } : {}),
|
||||
} }).where(eq(nativeRunFinalizations.runId, fixture.runId));
|
||||
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
|
||||
restartKind: "graceful", runIds: [fixture.runId] });
|
||||
expect(claim).toMatchObject({ kind: "reattach_remote_runner", runId: fixture.runId, providerAttempt: 2,
|
||||
remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } });
|
||||
expect(claim?.kind === "reattach_remote_runner" ? claim.replaceUnansweredRunner : "not remote")
|
||||
.toBe(unanswered ? true : undefined);
|
||||
});
|
||||
|
||||
it("keeps a restart recovery's attempt credit while the agent directory is busy, then resumes at attempt 3", async () => {
|
||||
const fixture = await seedRun("remote-busy-directory");
|
||||
await fixture.db.update(heartbeatRuns).set({
|
||||
|
||||
@@ -476,6 +476,7 @@ import {
|
||||
verifyRemoteRunnerReattachment,
|
||||
stopStaleRemoteRunnerBeforeBootstrap,
|
||||
verifyRemoteRunnerOrSettledExit,
|
||||
nativeRecoveryCommandUnanswered,
|
||||
readRemoteProviderPackManifest,
|
||||
verifyRemoteProviderPackManifest,
|
||||
prepareRemoteProviderPackOnTarget,
|
||||
@@ -1141,6 +1142,54 @@ describe("remote controller restart adoption", () => {
|
||||
target: exitedRunner(state) as never, identity, runId: "run", normalizedSessionId: "session", providerAttempt }).catch(caught => caught);
|
||||
expect(readRemoteRunnerRecoveryError(error)).toMatchObject({ stage: "process_verification", reason: "process_not_running" });
|
||||
});
|
||||
describe("a same-run reattachment whose adopted Runner never answered its recovery read", () => {
|
||||
const unanswered = { ...claim, replaceUnansweredRunner: true as const };
|
||||
const midTurn = { lifecycle: "turn_active", outbox: [{ id: 338 }], pendingTerminalDelivery: null, pendingProviderCleanup: null };
|
||||
it.each([2, 3])("launches a fresh Runner instead of adopting the live one again on attempt %i", async providerAttempt => {
|
||||
const { target, execute } = fixture(midTurn);
|
||||
await expect(verifyRemoteRunnerOrSettledExit({ claim: unanswered, target: target as never, identity,
|
||||
runId: "run", normalizedSessionId: "session", providerAttempt })).resolves.toBeUndefined();
|
||||
expect(execute.mock.calls.some(([request]) => /kill -/.test(request.args.join(" ")))).toBe(false);
|
||||
});
|
||||
it("adopts the live Runner when the earlier attempt did not report an unanswered read", async () => {
|
||||
const { target } = fixture(midTurn);
|
||||
const adopted = await verifyRemoteRunnerOrSettledExit({ claim, target: target as never, identity,
|
||||
runId: "run", normalizedSessionId: "session", providerAttempt: 2 });
|
||||
expect(adopted?.pid).toBe(123);
|
||||
});
|
||||
it.each([
|
||||
["a mid-turn exit whose provider and tools may survive it", midTurn, 3],
|
||||
["a pending provider cleanup", { ...midTurn, pendingProviderCleanup: { pid: 7 } }, 3],
|
||||
["an exhausted attempt budget", midTurn, 4],
|
||||
] as const)("never resumes beside an exited unanswered Runner's lineage: %s", async (_case, state, providerAttempt) => {
|
||||
const target = exitedRunner(state);
|
||||
const error = await verifyRemoteRunnerOrSettledExit({ claim: unanswered, target: target as never,
|
||||
identity, runId: "run", normalizedSessionId: "session", providerAttempt }).catch(caught => caught);
|
||||
expect(readRemoteRunnerRecoveryError(error)).toMatchObject({ stage: "process_verification", reason: "process_not_running" });
|
||||
});
|
||||
it("keeps the live Runner's identity checks before replacing it", async () => {
|
||||
const { target } = fixture({ ...midTurn, runnerInstanceId: "different" });
|
||||
await expect(verifyRemoteRunnerOrSettledExit({ claim: unanswered, target: target as never, identity,
|
||||
runId: "run", normalizedSessionId: "session", providerAttempt: 2 })).rejects.toThrow("runner_remote_recovery_identity_mismatch");
|
||||
});
|
||||
it.each([
|
||||
"native_session_recovery_failed: HarnessReconciliationError: codex_history_read_failed: thread/turns/list requires supported paginated history; Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 338 76s ago)",
|
||||
"native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: runnerd exited while waiting for session.snapshot",
|
||||
"native_session_recovery_failed: Error: runner_ingress_unavailable: PRP command runner.drain timed out (runner connected; pending ahead 0; last runner event seq 93 391s ago)",
|
||||
])("recognizes the staging recovery read that got no answer: %s", message => {
|
||||
expect(nativeRecoveryCommandUnanswered(new Error(message))).toBe(true);
|
||||
expect(nativeRecoveryCommandUnanswered(new Error("wrapped", { cause: new Error(message) }))).toBe(true);
|
||||
});
|
||||
it.each([
|
||||
"runnerd did not report the provider turn identity",
|
||||
"provider_transport_failed: PRP command session.snapshot timed out",
|
||||
"native_session_recovery_failed: provider resumed a different session",
|
||||
"native_session_recovery_failed: provider_transport_failed: PRP command session.snapshot timed out; native_adopted_provider_identity_mismatch",
|
||||
"native_session_recovery_failed: restart interruption history is incomplete",
|
||||
])("does not replace a Runner for another failure: %s", message => {
|
||||
expect(nativeRecoveryCommandUnanswered(new Error(message))).toBe(false);
|
||||
});
|
||||
});
|
||||
it.each([
|
||||
["an unsettled suspension", { ...settledSuspension, outbox: [{ id: 1 }] }, 2],
|
||||
["a pending provider cleanup", { ...settledSuspension, pendingProviderCleanup: { pid: 7 } }, 2],
|
||||
|
||||
@@ -8860,6 +8860,16 @@ export function nativeRunnerExitedBeforeSessionOpen(error: unknown): boolean {
|
||||
return nativeRunnerExitConfirmed(error) && runnerExitBeforeSessionOpenMessage(error);
|
||||
}
|
||||
|
||||
export function nativeRecoveryCommandUnanswered(error: unknown, depth = 0): boolean {
|
||||
if (!error || typeof error !== "object" || depth > 5) return false;
|
||||
const value = error as Record<string, unknown>;
|
||||
const message = typeof value.message === "string" ? value.message : "";
|
||||
if (message.startsWith("native_session_recovery_failed:")
|
||||
&& /PRP command [\w.]+ timed out|runnerd exited while waiting for [\w.]+/.test(message)
|
||||
&& !/mismatch|different|identity|authority|integrity|incompatible/i.test(message)) return true;
|
||||
return nativeRecoveryCommandUnanswered(value.cause, depth + 1);
|
||||
}
|
||||
|
||||
/** Same provider, category and message (numbers ignored): a retry would meet the same fault. */
|
||||
export function nativeStartupFailureSignature(failure: unknown): string | null {
|
||||
const value = record(failure);
|
||||
@@ -13054,6 +13064,9 @@ async function executePaperclipNativeSessionAttempt(
|
||||
? { runnerExitConfirmed: true } : {}),
|
||||
...(input.runnerExecutionTarget?.kind === "remote" && nativeRunnerExitedBeforeSessionOpen(error)
|
||||
? { runnerExitedBeforeSessionOpen: true } : {}),
|
||||
...(input.runnerExecutionTarget?.kind === "remote" && input.restartRecovery?.kind === "reattach_remote_runner"
|
||||
&& input.restartRecovery.remote.kind !== "plugin_task" && nativeRecoveryCommandUnanswered(error)
|
||||
? { adoptedRunnerUnanswered: true } : {}),
|
||||
...(providerCooldownAt ? { providerCooldownAt: providerCooldownAt.toISOString() } : {}),
|
||||
...(providerLimit?.plan.resetAt ? { providerLimitResetAt: providerLimit.plan.resetAt.toISOString() } : {}),
|
||||
recoveryOwner: recoveryProjection.recoveryOwner,
|
||||
@@ -14987,7 +15000,8 @@ const REMOTE_RUNNER_PROCESS_STDERR_TRIM_SCRIPT =
|
||||
* A launch therefore stops a `paperclip-runnerd` that is still serving this
|
||||
* exact `--state-dir`. A launch only happens after the server chose not to
|
||||
* adopt a Runner: a same-run reattachment proves the marker's Runner gone
|
||||
* first, and a transient reattachment failure never reaches a launch.
|
||||
* first, or replaces a Runner that left the previous attempt's recovery read
|
||||
* unanswered. Any other transient reattachment failure never reaches a launch.
|
||||
*
|
||||
* The stop runs in node ($1 the runnerd binary, $2 the state directory, then
|
||||
* candidate PIDs):
|
||||
@@ -15270,16 +15284,20 @@ async function readRemoteRunnerExitEvidence(runner: CommandManagedRuntimeRunner,
|
||||
|
||||
export async function verifyRemoteRunnerOrSettledExit(
|
||||
input: Parameters<typeof verifyRemoteRunnerReattachment>[0] & { providerAttempt: number;
|
||||
claim: { exitedBeforeSessionOpen?: true } },
|
||||
claim: { exitedBeforeSessionOpen?: true; replaceUnansweredRunner?: true } },
|
||||
) {
|
||||
const recoveryDeadline = Math.min(input.recoveryDeadline ?? Number.POSITIVE_INFINITY,
|
||||
Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS);
|
||||
const stall = input.stall ?? remoteRunnerPluginStall(input.recoveryDeadline);
|
||||
const target = input.target;
|
||||
const replaceUnanswered = input.claim.replaceUnansweredRunner === true
|
||||
&& input.providerAttempt <= NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT
|
||||
&& target?.kind === "remote" && target.transport === "sandbox" && Boolean(target.runner);
|
||||
let verified: Awaited<ReturnType<typeof verifyRemoteRunnerReattachment>>;
|
||||
try {
|
||||
return await verifyRemoteRunnerReattachment({ ...input, recoveryDeadline, stall });
|
||||
verified = await verifyRemoteRunnerReattachment({ ...input, recoveryDeadline, stall });
|
||||
} catch (error) {
|
||||
const diagnostic = readRemoteRunnerRecoveryError(error);
|
||||
const target = input.target;
|
||||
const exitedBeforeSessionOpen = input.claim.exitedBeforeSessionOpen === true;
|
||||
if (input.providerAttempt > NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT || diagnostic?.stage !== "process_verification" ||
|
||||
diagnostic.reason !== "process_not_running" ||
|
||||
@@ -15292,6 +15310,7 @@ export async function verifyRemoteRunnerOrSettledExit(
|
||||
: state.lifecycle !== "suspended" || !describeExitedRemoteRunner(state).runnerSettled) throw error;
|
||||
return undefined;
|
||||
}
|
||||
return replaceUnanswered ? undefined : verified;
|
||||
}
|
||||
|
||||
function runnerStateAllowsFreshStart(state: Record<string, unknown>): boolean {
|
||||
|
||||
Reference in new issue
Block a user