fix(runner): finalize a durable proposed result when restore loses its Codex turn (#19658)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs survive control-plane restarts: runnerd restores
the provider conversation, and the control plane continues the task from
its checkpoint
> - A hard kill can land after the agent proposed its result
(`run.result.proposed`) but before the provider turn ended
> - The restored runner then proves that the turn ended with the
accepted result. It records `session.reconciled` and `run.terminal`, but
no provider turn terminal
> - The Codex recovery read did not see that turn as ended, so recovery
failed with "persisted active turn … is missing from thread/read".
Retries ran out, and the Retry re-ran finished work
> - This pull request lets recovery settle that turn with its durable
result
> - The benefit is that a task that already finished its work continues
to finalization after a restart, instead of failing and redoing the work

## Linked Issues or Issue Description

Refs #19525, #19591, #19539, #19501

**What happened?**
In a restart storm, a native Codex task (B1) proposed a complete
`paperclip.run_result.v1` with passing tests. The control plane accepted
the completion tool call. About 9 s later, a K1 (SIGKILL and restart of
the control plane) killed the controller, runnerd, and the Codex
app-server. After startup, `resume_dead_runner` resumed the same run two
times. Each resume spent about 25 s in `runner.session.startup` and
failed with `native_session_recovery_failed: HarnessReconciliationError:
persisted active turn <id> is missing from thread/read`. Then
`native_session_retry_exhausted`, replay was blocked
(`preserve_without_replay_v1`), and the run ended
`native_session_interrupted`. The task needed a Retry, which re-ran the
finished work.

**Steps to reproduce**
1. Start a native Codex task through runnerd.
2. Let the agent call `paperclip_finish`, and let Paperclip accept the
result.
3. Kill the controller, runnerd, and Codex before the provider turn
ends.
4. Let restart recovery resume the run.

Durable evidence from the kept instance:
- The run events have `run.result.proposed` for the turn, then a
`completion.tool_accepted` span, then the kill.
- The saved checkpoint has `activeTurnId` set to that turn and
`semanticResult: null`.
- The Codex rollout on disk does contain the turn. The provider did
persist it.
- The runnerd journal for the first resume has `session.reconciled
{previousProviderTurnId: <turn>, activeProviderTurnId: null}` and then
`run.terminal {turnTerminalState: completed, runTerminalState:
succeeded, reportedWorkDisposition: done}`. It has no `turn.*` event for
that turn. The runnerd Codex provider state shows
`completedTurnAuthoritative: true` and `completedProviderTurnId:
<turn>`.

**Expected behavior**
Recovery finalizes the accepted result. It does not re-run the task.

**Actual behavior**
The runnerd Codex transport builds the recovery `thread/read` reply from
runnerd's journal. It looked only for
`turn.completed|failed|interrupted|cancelled` events that name the turn.
Runnerd's "turn ended with an accepted result" branch
(`provider_backend.rs`, `recovered_turn_ended_with_result`) emits only
the reconciliation and the run terminal. So the reply omitted the turn,
and `CodexHarnessSession.reconcile()` threw at
`codex-harness-session.ts:796`. The second resume read the same journal
and failed the same way.

## What Changed

- `runnerd-codex-transport.ts`: a new `runnerdRecoveredTurnTerminal`
helper selects the terminal for a restored turn. A provider `turn.*`
terminal that names the turn wins, as before, with its cause. A
`run.terminal` that follows a `session.reconciled` that names the turn
as the ended previous turn (no active turn, no later `turn.started`) now
also settles that turn, with the run terminal's `turnTerminalState`. The
`thread/read` wait loop uses this helper.
- `codex-harness-session.ts`: `reconcile()` now settles a checkpointed
active turn that provider history no longer contains when the session
holds the semantic result that this exact turn proposed (`resultTurnId`
equals the turn). Recovery restores that result from the durable
`run.result.proposed` event. Without such a result, a missing turn still
fails closed.
- `codex-harness-session.ts`: the same rule applies when the restored
runner reports that turn `failed` with `provider_turn_lost_on_restore`.
This happens when the runner cannot prove the accepted result itself.
Later recoveries keep the settled outcome. Without a held result, a
restore-lost turn keeps its existing failure and same-conversation
continuation.
- `fake-codex-app-server`: a test-only `--hold-after-terminal-result`
flag keeps the turn open after an accepted completion report.
- `doc/execution-semantics.md`: one additive paragraph on this recovery
rule.

## Verification

- Red before the fix, real `paperclip-runnerd` and fake Codex: `pnpm
--filter @paperclipai/paperclip-runner exec vitest run
src/live/runnerd-restart-continuation.test.ts -t "durable proposed
result"` failed with `native_session_recovery_failed:
HarnessReconciliationError: persisted active turn provider-turn-1 is
missing from thread/read` at `native-session-runtime.ts:2267`. This is
the same stack as the incident.
- Green after the fix: the run finalizes on `provider-turn-1` with the
proposed result and `runTerminalState: succeeded`. It has no
`session.failed` or `turn.failed`, and Codex received only one
`turn/start` (no re-run). With the transport fix alone, the test passes
in about 5 s. With only the session fix, it also passes (about 30 s,
through the 5 s read wait).
- Driver red/green:
`src/drivers/codex/codex-app-server-driver.recovery.test.ts -t "missing
persisted turn"`. The settle case failed before the fix (`recovered:
false`) and passes after it. The guard case (the held result belongs to
another turn) still fails closed. The existing "fails recovery
explicitly when the persisted active turn is missing or replaced" test
still passes.
- Driver red/green for the restore-lost case: `-t "restore-lost turn"`.
The held=true case failed before the fix and passes after it. The
held=false case still maps `turn.failed` with the lost cause.
- Transport unit test: `runnerd-codex-transport.test.ts -t "reconciled
result terminal"` covers the run-terminal rule and its negative cases.
- Suites: `src/drivers/codex/`, `src/native-session-runtime.test.ts`,
`src/backends/`, `src/live/runnerd-restart-continuation.test.ts`, and
`src/live/runnerd-codex-transport.test.ts` pass. Some load-sensitive
tests time out at host load 25–30 and pass when rerun alone. The two
`proves local suspension after an event backlog … (1024 suffix deltas)`
cases also time out on unmodified master.
- `pnpm --filter @paperclipai/paperclip-runner typecheck:typescript`
passes.
- Live chaos replay with real Codex: K1 0–3 s after
`run.result.proposed`, 3/3 PASS (R1). Each case resumed the same run,
which ended `succeeded`. No Retry was needed and no work was re-run.
Details are in the comment below.

## Risks

- The transport rule accepts a run terminal only directly after
runnerd's own reconciliation that names the ended turn with no active
turn. Runnerd emits that pair only after it restored the conversation
and proved the turn ended. A provider turn terminal still wins. Other
run terminals do not settle a turn.
- The session rules use only a result that Paperclip already holds for
the exact turn. The control plane accepted that result before the kill.
A later provider terminal for that turn cannot change the outcome. The
finalizer can already interrupt a provider after an accepted result.
- A missing turn without a durable result still fails closed. Its
outcome is unknown, and the existing Retry stays available.
- No profile, schema, pack contract, or database changes. The Rust
change touches only the fake Codex test binary.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-06 18:02:40 -06:00
1 parent 961455b9a0
commit f4b19e4231
7 files changed
+327 -51

No files matched your search

+9
View File
@@ -2499,6 +2499,15 @@ results, intentional stops, and historical unmarked failures keep their existing
terminal behavior. This continuation uses the already charged restart attempt and
does not reset the provider-attempt budget.
A restart can also land after the agent proposed its result but before the
provider turn ended. When the restored runner proves that turn ended with the
result Paperclip accepted, it records a reconciliation and the run terminal,
which settle that turn. When provider history no longer contains the checkpointed
turn, or the runner reports it lost on restore, but Paperclip durably holds the
result that exact turn proposed, recovery also settles the turn with that result. Either way, recovery finalizes the
accepted result instead of failing and rerunning finished work. A missing turn
without such a result still fails closed, because its outcome is unknown.
Provider execution and control-plane finalization have different clocks. A healthy provider can think or execute a long tool without output. Once execution settles, recovery and finalization control steps have a 60-second deadline, checked on startup and every 15 seconds. With a healthy database and scheduler, an abandoned transition must be repaired or surfaced within 90 seconds. Terminal persistence must not wait on provider cleanup or publication; a late finalizer cannot change a reassigned or closed task or release another run's locks. In particular, a done report from a run whose task now belongs to another agent, a user or nobody is kept as an accepted result but records `completion_owner_superseded` and preserves the task status for the current owner. Historical ambiguous runs are never automatically replayed after an upgrade.
When that deadline fails a native run with a prepared workspace, the same
@@ -734,6 +734,11 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
let exit_after_tool_call_completion = args
.iter()
.any(|value| value == "--exit-after-tool-call-completion");
// Keep the turn open after an accepted completion report, so a test can
// kill the runner between the durable proposal and the provider terminal.
let hold_after_terminal_result = args
.iter()
.any(|value| value == "--hold-after-terminal-result");
let emit_tool_call_on_resume = args
.iter()
.any(|value| value == "--emit-tool-call-on-resume");
@@ -1112,6 +1117,9 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
"terminal-result:rejected"
},
)?;
if success && hold_after_terminal_result {
continue;
}
finish_turn(
&state_path,
&mut state,
@@ -1224,6 +1224,110 @@ describe("Codex app-server Codex driver", () => {
}
});
it("settles a missing persisted turn with the durable result it already proposed", async () => {
// Provider history lost the turn (for example, it was killed before the
// turn terminal was flushed), but Paperclip durably holds the result that
// turn proposed. That result is the turn's outcome: finalize it instead of
// failing recovery and re-running finished work.
const first = new FakeCodexTransport();
const second = new FakeCodexTransport();
second.readResponse = {
thread: { id: "thread-1", sessionId: "provider-session-1", cwd: WORKSPACE, turns: [] },
};
const driver = makeDriver([first, second]);
const original = await driver.openSession({
runId: "run-missing-proposed-turn",
normalizedSessionId: "normalized-missing-proposed-turn",
workingDirectory: WORKSPACE,
});
await original.startTurn({ message: { role: "user", text: "Complete." } });
expect(
await first.invoke({
id: "call-before-kill",
method: "item/tool/call",
params: { threadId: "thread-1", turnId: "turn-1", callId: "call-before-kill", tool: "paperclip_finish", arguments: result },
}),
).toMatchObject({ success: true });
const snapshot = await original.snapshot();
expect(snapshot).toMatchObject({ activeTurnId: "turn-1", semanticResult: { turnId: "turn-1", result } });
await original.close({ reason: "killed before the turn terminal" });
const recovery = await driver.recoverSession?.(snapshot);
expect(recovery).toMatchObject({ recovered: true });
const events = await collectUntilTerminal(recovery!.session!.events());
expect(events.filter((event) => event.eventType === "turn.completed")).toHaveLength(1);
expect(events.filter((event) => event.eventType === "run.result.proposed")).toHaveLength(0);
expect(await recovery!.session!.snapshot()).toMatchObject({
activeTurnId: null,
semanticResult: { turnId: "turn-1", result },
terminalTurns: [expect.objectContaining({ turnId: "turn-1" })],
});
});
it.each([true, false])("settles a restore-lost turn from its durable result only when one is held (held=%s)", async (held) => {
// The runner can report the checkpointed turn lost on restore when it
// cannot prove the accepted result itself, while Paperclip already holds it.
const lost = { code: "provider_turn_lost_on_restore", recoverable: true };
const first = new FakeCodexTransport();
const second = new FakeCodexTransport();
second.readResponse = {
thread: { id: "thread-1", sessionId: "provider-session-1", cwd: WORKSPACE,
turns: [{ id: "turn-1", status: "failed", error: lost, items: [] }] },
};
const driver = makeDriver([first, second]);
const original = await driver.openSession({
runId: `run-lost-proposed-${held}`,
normalizedSessionId: `normalized-lost-proposed-${held}`,
workingDirectory: WORKSPACE,
});
await original.startTurn({ message: { role: "user", text: "Complete." } });
if (held) {
await first.invoke({
id: "call-before-kill",
method: "item/tool/call",
params: { threadId: "thread-1", turnId: "turn-1", callId: "call-before-kill", tool: "paperclip_finish", arguments: result },
});
}
const snapshot = await original.snapshot();
await original.close({ reason: "killed before the turn terminal" });
const recovery = await driver.recoverSession?.(snapshot);
expect(recovery).toMatchObject({ recovered: true });
const events = await collectUntilTerminal(recovery!.session!.events());
const terminal = events.find((event) => event.eventType.startsWith("turn."));
expect(terminal?.eventType).toBe(held ? "turn.completed" : "turn.failed");
if (!held) expect(terminal?.payload).toMatchObject({ error: lost });
});
it("does not settle a missing persisted turn with a result another turn proposed", async () => {
const first = new FakeCodexTransport();
const second = new FakeCodexTransport();
second.readResponse = {
thread: { id: "thread-1", sessionId: "provider-session-1", cwd: WORKSPACE, turns: [] },
};
const driver = makeDriver([first, second]);
const original = await driver.openSession({
runId: "run-missing-other-turn",
normalizedSessionId: "normalized-missing-other-turn",
workingDirectory: WORKSPACE,
});
await original.startTurn({ message: { role: "user", text: "Complete." } });
await first.invoke({
id: "call-before-kill",
method: "item/tool/call",
params: { threadId: "thread-1", turnId: "turn-1", callId: "call-before-kill", tool: "paperclip_finish", arguments: result },
});
const snapshot = await original.snapshot();
await original.close({ reason: "killed before the turn terminal" });
const recovery = await driver.recoverSession?.({
...snapshot,
semanticResult: { ...snapshot.semanticResult!, turnId: "turn-older" },
});
expect(recovery).toMatchObject({
recovered: false,
reason: expect.stringContaining("persisted active turn turn-1 is missing"),
});
});
it("reconciles a turn completed during transport loss without replacing either session identity", async () => {
const first = new FakeCodexTransport(
"driver-thread-loss",
@@ -1,7 +1,7 @@
import { readCodexThreadState, readCodexTurnMetadata, readCodexTurnItems } from "./codex-history.js";
import { codexRunUsage, codexUsageBaselineForNextRun, observeCodexUsage } from "./codex-usage-baseline.js";
import { randomUUID } from "node:crypto";
import { NativeProviderTerminalFailure } from "../../contracts/native-session-backend.js";
import { isNativeRestartInterruption, NativeProviderTerminalFailure } from "../../contracts/native-session-backend.js";
import type {
HarnessGoalOperation,
@@ -791,29 +791,45 @@ export class CodexHarnessSession
const expectedTurn = turns.find(
(turn) => text(turn.id) === expectedTurnId,
);
if (expectedTurn === undefined) {
const holdsTurnResult = this.result !== null && this.resultTurnId === expectedTurnId;
if (holdsTurnResult && (expectedTurn === undefined || (
text(expectedTurn.status) === "failed" && isNativeRestartInterruption(expectedTurn.error)))) {
// Provider history can lose a turn whose process was killed before its
// terminal was flushed, or the restored runner can report it lost on
// restore. Paperclip already holds the semantic result this exact turn
// proposed, so that durable result is the turn's outcome: settle it
// rather than fail recovery and rerun finished work. Without such a
// result the outcome stays unknown and recovery fails closed.
reconciledTerminalTurnId = expectedTurnId;
mapTerminalTurn(this, { id: expectedTurnId, status: "completed", items: [] }, expectedTurnId, true);
} else if (expectedTurn === undefined) {
throw new HarnessReconciliationError(
`persisted active turn ${expectedTurnId} is missing from thread/read`,
);
}
const status = text(expectedTurn.status);
if (status === "inProgress") {
this.activeTurnId = expectedTurnId;
} else if (
["completed", "failed", "interrupted", "cancelled"].includes(status)
) {
reconciledTerminalTurnId = expectedTurnId;
mapTerminalTurn(this, expectedTurn, expectedTurnId, true);
} else {
throw new HarnessReconciliationError(
`persisted active turn ${expectedTurnId} has unreconcilable status ${status || "missing"}`,
);
const status = text(expectedTurn.status);
if (status === "inProgress") {
this.activeTurnId = expectedTurnId;
} else if (
["completed", "failed", "interrupted", "cancelled"].includes(status)
) {
reconciledTerminalTurnId = expectedTurnId;
mapTerminalTurn(this, expectedTurn, expectedTurnId, true);
} else {
throw new HarnessReconciliationError(
`persisted active turn ${expectedTurnId} has unreconcilable status ${status || "missing"}`,
);
}
}
}
for (const [turnId, fingerprint] of this.terminalTurns) {
const observed = turns.find((turn) => text(turn.id) === turnId);
if (observed === undefined) continue;
// A turn settled from its durable result keeps that outcome when the
// restored runner reports the same turn lost on restore.
if (this.result !== null && this.resultTurnId === turnId &&
text(observed.status) === "failed" && isNativeRestartInterruption(observed.error)) continue;
if (
!["completed", "failed", "interrupted", "cancelled"].includes(
text(observed.status),
@@ -1505,6 +1505,26 @@ it("does not hold a pending turn start behind another turn's restore-lost termin
params: { turn: { id: "provider-turn-raw", status: "failed" } } }] })).toBe("provider-turn-raw");
});
it("settles a restored turn from the runner's reconciled result terminal", () => {
const { runnerdRecoveredTurnTerminal: recovered } = runnerdRecoveryInternals;
const event = (eventType: string, payload: Record<string, unknown>) => ({ eventType, envelope: { payload: { payload } } });
const reconciled = event("session.reconciled", { previousProviderTurnId: "turn-1", activeProviderTurnId: null });
const runTerminal = event("run.terminal", { turnTerminalState: "completed", runTerminalState: "succeeded" });
// The runner recovered turn-1's accepted result: no provider turn terminal,
// only the reconciliation and the run terminal.
expect(recovered([event("turn.started", { providerTurnId: "turn-1" }), reconciled, runTerminal], "turn-1"))
.toEqual({ status: "completed", error: null });
// An explicit provider terminal keeps its cause.
const lost = { code: "provider_turn_lost_on_restore", recoverable: true };
expect(recovered([reconciled, event("turn.failed", { providerTurnId: "turn-1", error: lost }), runTerminal], "turn-1"))
.toEqual({ status: "failed", error: lost });
// A run terminal settles only the turn the runner reconciled as ended.
expect(recovered([event("session.reconciled", { previousProviderTurnId: "turn-0", activeProviderTurnId: null }), runTerminal], "turn-1")).toBeNull();
expect(recovered([event("session.reconciled", { previousProviderTurnId: "turn-1", activeProviderTurnId: "turn-1" }), runTerminal], "turn-1")).toBeNull();
expect(recovered([reconciled, event("turn.started", { providerTurnId: "turn-2" }), runTerminal], "turn-1")).toBeNull();
expect(recovered([runTerminal], "turn-1")).toBeNull();
});
it("requires ACPX command results to preserve the requested turn identity", () => {
expect(
runnerdRecoveryInternals.turnStartCommandResultValid({
@@ -788,6 +788,52 @@ function turnStartResponseReady(input: {
);
}
type RunnerdRecoveredTurnTerminal = {
status: "completed" | "failed" | "interrupted" | "cancelled";
error: unknown;
};
const RUNNERD_TURN_TERMINAL_STATUSES = new Set(["completed", "failed", "interrupted", "cancelled"]);
/**
* The terminal the runner recorded for a provider turn that is no longer
* active after restore. A provider terminal names the turn explicitly. When the
* runner restores a conversation whose turn already ended with the semantic
* result Paperclip accepted, it records `session.reconciled` (naming the ended
* turn as previous, with no active turn) followed by the run terminal, and no
* provider turn terminal. That run terminal settles the reconciled turn.
*/
function runnerdRecoveredTurnTerminal(
events: readonly Pick<DurableRecoveryCommittedEvent, "eventType" | "envelope">[],
turnId: string,
): RunnerdRecoveredTurnTerminal | null {
let terminal: RunnerdRecoveredTurnTerminal | null = null;
let reconciledEndedTurn = false;
for (const event of events) {
const payload = record(record(event.envelope.payload).payload);
const providerTurnId = payload.providerTurnId ?? payload.turnId ?? record(payload.turn).id;
if (event.eventType === "turn.started") {
reconciledEndedTurn = false;
} else if (event.eventType === "session.reconciled") {
reconciledEndedTurn = payload.previousProviderTurnId === turnId && payload.activeProviderTurnId == null;
} else if (event.eventType.startsWith("turn.") && RUNNERD_TURN_TERMINAL_STATUSES.has(event.eventType.slice("turn.".length))) {
if (providerTurnId !== turnId) continue;
// Reconciliation must retain the cause, including the runner's explicit
// process-loss marker, rather than inventing error:null.
terminal = { status: event.eventType.slice("turn.".length) as RunnerdRecoveredTurnTerminal["status"],
error: payload.error ?? record(payload.turn).error ?? null };
reconciledEndedTurn = false;
} else if (event.eventType === "run.terminal" && reconciledEndedTurn) {
const status = String(payload.turnTerminalState);
if (RUNNERD_TURN_TERMINAL_STATUSES.has(status)) {
terminal = { status: status as RunnerdRecoveredTurnTerminal["status"], error: null };
}
reconciledEndedTurn = false;
}
}
return terminal;
}
/** The provider turn a durable terminal names, when it names one explicitly. */
function runnerdTerminalProviderTurnId(eventType: string, payload: unknown): string | null {
const named = (value: unknown) => typeof value === "string" && value.length > 0 ? value : null;
@@ -4130,48 +4176,19 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
: [{ id: activeProviderTurnId, status: "inProgress" }];
if (activeProviderTurnId === null && this.#turnId.length > 0) {
const recoveryDeadline = Date.now() + 5_000;
let terminal = (this.#core?.store.state.committedEvents ?? []).find(
() => false,
);
let terminal: RunnerdRecoveredTurnTerminal | null = null;
while (Date.now() < recoveryDeadline) {
this.#throwIfFailed();
this.#pumpEvents();
terminal = [...(this.#core?.store.state.committedEvents ?? [])]
.reverse()
.find((event) => {
if (
event.eventType !== "turn.completed" &&
event.eventType !== "turn.failed" &&
event.eventType !== "turn.interrupted" &&
event.eventType !== "turn.cancelled"
)
return false;
const payload = record(record(event.envelope.payload).payload);
const providerTurnId =
payload.providerTurnId ??
payload.turnId ??
record(payload.turn).id;
return providerTurnId === this.#turnId;
});
if (terminal !== undefined) break;
terminal = runnerdRecoveredTurnTerminal(
this.#core?.store.state.committedEvents ?? [],
this.#turnId,
);
if (terminal !== null) break;
await new Promise((resolveWait) => setTimeout(resolveWait, 10));
}
if (terminal !== undefined) {
const payload = record(record(terminal.envelope.payload).payload);
recoveredTurns.push({
id: this.#turnId,
// Reconciliation must retain the cause, including the runner's
// explicit process-loss marker, rather than inventing error:null.
error: payload.error ?? record(payload.turn).error ?? null,
status:
terminal.eventType === "turn.completed"
? "completed"
: terminal.eventType === "turn.interrupted"
? "interrupted"
: terminal.eventType === "turn.cancelled"
? "cancelled"
: "failed",
});
if (terminal !== null) {
recoveredTurns.push({ id: this.#turnId, ...terminal });
}
}
if (restoreRuntimeRequests && activeProviderTurnId !== null) {
@@ -8360,6 +8377,7 @@ export const runnerdRecoveryInternals = Object.freeze({
rotatedRunAttachPayload,
localRunnerStateArchive,
rotateAuthorityEpoch,
runnerdRecoveredTurnTerminal,
runnerdTerminalProviderTurnId,
terminalMayBelongToPendingTurnStart,
turnStartCommandResultValid,
@@ -208,3 +208,104 @@ it("starts a new turn when the restored runner reports a lost turn the checkpoin
await rm(root, { recursive: true, force: true });
}
}, 40_000);
it("finalizes the durable proposed result when the runner and provider die before the turn terminal", async () => {
// The provider proposed a complete result and Paperclip durably recorded it.
// Both processes were then killed before the provider turn ended. The
// restored runner recovers that turn's authoritative result: recovery must
// finalize it instead of failing reconciliation or re-running the task.
const root = await mkdtemp(join(tmpdir(), "runner-restart-proposed-"));
const identity = { runId: "run-proposed", sessionId: "session-proposed", companyId: "company-proposed", issueId: "issue-proposed", agentId: "agent-proposed" };
const execution: NativeExecutionInputV1 = {
schema: "paperclip.native-execution-input.v1",
binding: { runId: identity.runId, companyId: identity.companyId, issueId: identity.issueId, agentId: identity.agentId, executionWorkspaceId: "workspace-proposed" },
session: { normalizedSessionId: identity.sessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
task: { identifier: "TEST-3", title: "Finish checks", description: null, prompt: "Run the checks and report the result.", workMode: "standard" },
workspace: { cwd: root, repoUrl: null, repoRef: null, branchName: null },
provider: { kind: "codex", model: null },
completionContract: { id: "contract-proposed", sha256: "digest", schemaVersion: "paperclip.completion-contract.v1", contract: {
revision: "1", objective: "Finish checks", criteria: [{ id: "checks", requirement: "Check the outcome" }],
} },
interactionResponses: [], credentialBindings: [],
};
const report = { schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: "The checks pass.",
completionClaim: { contractRevision: "1", objectiveSatisfied: true, criteria: [{ criterionId: "checks", status: "satisfied", evidenceRefs: [] }], remainingWork: [] },
evidence: [], verification: [], attentionRequests: [], artifacts: [] };
await writeFile(join(root, "report.json"), JSON.stringify(report));
const options = {
runnerBinary: defaultCapabilityRunnerdBinary(),
codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"),
codexArgs: ["--state-file", join(root, "fake-state.json"), "--call-log", join(root, "calls.log"), "--durable-turn-ids",
"--terminal-result-file", join(root, "report.json"), "--hold-after-terminal-result"],
stateDirectory: root,
prpIdentity: { runnerInstanceId: "runner-proposed", environmentLeaseId: "lease-proposed", runId: identity.runId, normalizedSessionId: identity.sessionId, turnId: "turn-proposed", itemId: "item-proposed" },
};
const first = createRunnerdCodexTransport(options);
let restored: ReturnType<typeof createRunnerdCodexTransport> | undefined;
const alive = (pid: number | null) => {
if (!pid) return false;
try { process.kill(pid, 0); return true; } catch { return false; }
};
const kill = (pid: number | null) => {
if (!pid) return;
try { process.kill(pid, "SIGKILL"); } catch (error) { if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; }
};
try {
const backend = createRunnerdNativeSessionBackend(execution, { runnerInstanceId: "runner-proposed", transportFactory: () => first.transport });
const session = await backend.openSession({ identity, workingDirectory: root });
await session.startTurn({ message: { role: "user", text: execution.task.prompt } });
// The controller's last checkpoint names the active turn but not the result.
const checkpoint = await session.snapshot();
expect(checkpoint.activeTurnId).toBe("provider-turn-1");
expect(checkpoint.semanticResult ?? null).toBeNull();
// The control plane durably appended the proposal before the kill.
const durableEvents: PrpEvent[] = [];
const initialEvents = session.events()[Symbol.asyncIterator]();
while (!durableEvents.some(event => event.eventType === "run.result.proposed")) {
const event = await initialEvents.next();
if (event.done) throw new Error("initial event history ended before the proposal");
durableEvents.push(event.value);
}
await vi.waitFor(async () => expect(await readFile(join(root, "calls.log"), "utf8")).toContain("terminal-result:accepted"));
const owner = first.evidence();
await first.detachControllerForRestart();
kill(owner.runnerPid);
kill(owner.providerPid);
await vi.waitFor(() => { expect(alive(owner.runnerPid)).toBe(false); expect(alive(owner.providerPid)).toBe(false); });
const providerState = JSON.parse(await readFile(join(root, "fake-state.json"), "utf8"));
providerState.activeTurnId = null;
await writeFile(join(root, "fake-state.json"), JSON.stringify(providerState));
restored = createRunnerdCodexTransport({ ...options, resumeDynamicTools: [], resumeActiveTurnId: checkpoint.activeTurnId });
const recovering = createRunnerdNativeSessionBackend(execution, { runnerInstanceId: "runner-proposed", transportFactory: () => restored!.transport });
const events: PrpEvent[] = [];
const completion = await executeNativeSession({
input: execution, backend: recovering, persistedSession: checkpoint, resumeInterruptedTurn: true,
runnerInstanceId: "runner-proposed", controlPlaneInstanceId: "controller-proposed", timeoutMs: 15_000,
controlPlane: {
async openRun() {},
async checkpointSession() {},
async appendEvent(event) { if (!("sourceSeq" in event)) throw new Error("Expected a durable PRP event"); events.push(event); durableEvents.push(event); return { cursor: durableEvents.length, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" }; },
async replayEvents(request) {
const source = durableEvents.filter(event => event.sourceInstanceId === request.sourceInstanceId);
return { events: source.filter(event => event.sourceSeq > request.afterSourceSeq), highestContiguousSourceSeq: source.at(-1)?.sourceSeq ?? 0 };
},
async completeRun() {},
},
});
expect(completion).toMatchObject({ providerSessionId: checkpoint.providerSessionId, turnId: "provider-turn-1",
result: { reportedWorkDisposition: "done", summary: "The checks pass." }, terminal: { runTerminalState: "succeeded" } });
expect(events.find(event => event.eventType === "session.failed")).toBeUndefined();
expect(events.filter(event => event.eventType === "turn.failed")).toHaveLength(0);
// The finished work is not re-run.
expect((await readFile(join(root, "calls.log"), "utf8")).match(/^turn\/start$/gm)).toHaveLength(1);
} finally {
await restored?.transport.close().catch(() => undefined);
for (const bundle of [first, restored]) {
if (!bundle) continue;
const evidence = bundle.evidence();
kill(evidence.runnerPid);
kill(evidence.providerPid);
}
await rm(root, { recursive: true, force: true });
}
}, 40_000);