fix(runner): keep a restart continuation when a stale earlier continuation is reported lost (#20118)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a restart loses a native run's active turn, the runner reports
it as `provider_turn_lost_on_restore` and the controller starts one
same-conversation continuation. A later restart that loses the
continuation is answered the same way
> - With back-to-back restarts, the controller's checkpoint can miss a
continuation that the runner already accepted. On the next restore the
controller starts a fresh continuation, and the restored runner then
reports the unrecorded earlier continuation as lost
> - The event consumer treated that stale lost terminal as the run's
terminal, so the run failed while its new continuation was already
running
> - Recovery then refused to replay the failed run, and the task needed
a Retry. In an M1 parent/subtask task, the parent's Retry succeeded but
the parent stayed `blocked` on that subtask
> - This pull request makes the consumer skip a restart-lost terminal
for any turn other than the continuation it started
> - The benefit is that back-to-back restarts no longer turn a
recoverable lost turn into a failed run that needs Retry

## Linked Issues or Issue Description

Refs #20107

**What happened?**
Restart-chaos K14 wave w-20261108T213600-s9803 (local, an upgrade
between builds plus repeated restarts) at 070365404e. In task t02
(claude M1), the codex subtask CHA-7 failed with
`native_provider_terminal_failed: The runner restored the conversation
after process loss, but the previous turn is no longer active.` The
sequence:
1. Restore 1 lost turn `01a1bd17…`, and the controller started
continuation `01a1bd18-15ec…` (runner command 14). A restart followed
within a second, before the controller's checkpoint recorded that
continuation.
2. Restore 2 saw the checkpointed interruption of `01a1bd17…` and
started a fresh continuation, `01a1bd18-4cc4…` (command 24). The
restored runner also reported the unrecorded continuation
`01a1bd18-15ec…` as `provider_turn_lost_on_restore`.
3. Because a continuation was already started, the consumer returned
that stale terminal as the run's terminal. The run failed, the new
continuation was stopped (command 25), and automatic recovery refused to
replay it (`uncertain_provider_action`, `preserve_without_replay_v1`).
4. The parent CHA-2 asked CHA-7 to resume by comment, but an agent
comment cannot authorize a blocked replay, so the wake stayed
`deferred_issue_execution`. The parent's own Retry succeeded, but CHA-2
stayed `blocked` on CHA-7 (`R3:retry_did_not_complete`).

**Expected behavior**
A restart-lost terminal for an older, unrecorded continuation is
superseded by the continuation this restore started. The run continues
on that continuation and needs no Retry.

**Steps to reproduce**
1. Checkpoint a native session whose last terminal is a restart-lost
turn and whose active turn is null, so recovery starts a continuation.
2. Have the restored session report another turn (the unrecorded earlier
continuation) as `provider_turn_lost_on_restore`, either before or after
the new continuation starts.
3. Before this change, the run ends on that stale terminal.

**Paperclip version or commit**
070365404e (wave); branch rebased on current master.

## What Changed

- `native-session-runtime.ts`:
  - Records the turn id of the restart continuation it starts.
- New `supersededByRestartContinuation`, used by the consumer's
`resumeInterruptedTurn`, keeps consuming past a
`provider_turn_lost_on_restore` `turn.failed` when all of these hold:
    - the run was recovered from a checkpointed interruption;
- the failed turn is neither the checkpointed interrupted turn nor the
started continuation;
    - a continuation was actually started.
- The check first waits for the startup decision, because the consumer
starts before the continuation does.
- `native-session-runtime.test.ts`: a red/green test, with the stale
lost terminal arriving both after and before the continuation starts.

## Verification

- Red/green:
  - On master, both test cases end the run on the stale terminal.
- With the change, both resolve on the continuation's result. The
continuation is started once, and the stale `turn.failed` is still
appended to the journal.
- `vitest run` over the restart, recovery, continuation and terminal
test files in `packages/paperclip-runner`, including all of
`native-session-runtime.test.ts`: 9 files, 380 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck:typescript`
(`packages/paperclip-runner`): passed.

## Risks

- Low risk. The skip applies only to restart-lost failures of turns
other than the checkpointed interrupted turn and the started
continuation, and only after a continuation actually started. Real
provider failures, a loss of the continuation itself, and runs without a
checkpointed interruption keep their current handling.
- If startup fails, the startup decision still settles. The consumer
then falls back to the existing behavior and the run fails as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-08 16:44:13 -06:00
1 parent 412d49e48d
commit b43cccbb08
2 files changed
+74

No files matched your search

@@ -586,6 +586,62 @@ describe("executeNativeSession recovery", () => {
expect(checkpoints.some(snapshot => snapshot.terminalTurns?.some(turn => turn.turnId === "turn-recovery"))).toBe(true);
});
it.each([
{ label: "after the continuation starts", staleFirst: false },
{ label: "before the continuation starts", staleFirst: true },
])("keeps the restart continuation when the restored runner reports an earlier unrecorded continuation lost $label", async ({ staleFirst }) => {
let staleAppended!: () => void;
const staleTerminal = new Promise<void>(resolve => { staleAppended = resolve; });
const failure = { code: "provider_turn_lost_on_restore", recoverable: true };
const interrupted: PersistedNativeSession = {
backendKind: "runner", driverKind: "codex_app_server", sessionId: "thread-recovery", identity,
providerSessionId: "provider-recovery", cursor: "0", activeTurnId: null,
terminalTurns: [{ turnId: "turn-recovery", fingerprint: JSON.stringify({ terminalState: "failed", error: failure, result: null }) }],
semanticResult: null, pendingRuntimeRequests: [], lineage: [],
};
const startTurn = vi.fn<NativeSession["startTurn"]>(async () => {
if (staleFirst) await staleTerminal;
return { turnId: "turn-continuation" };
});
const session: NativeSession = {
identity: () => identity,
async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; },
async *events() {
yield { ...runnerEvent(1, "turn.failed", { status: "failed", error: failure }), turnId: "turn-unrecorded-continuation" };
yield { ...runnerEvent(2, "turn.completed", { status: "completed" }), turnId: "turn-continuation" };
},
startTurn,
async result() { return startTurn.mock.calls.length ? { result, terminal, turnId: "turn-continuation" } : null; },
async snapshot() { return structuredClone(interrupted); },
async close() {},
};
const backend: NativeSessionBackend = {
async descriptor() { return { kind: "runner", name: "restart", version: "1", capabilities: await session.capabilities() }; },
async openSession() { throw new Error("must preserve the provider session"); },
async recoverSession() { return { recovered: true, session }; },
};
const events: PrpEvent[] = [];
const port: ControlPlanePort = {
async openRun() {},
async checkpointSession() {},
async appendEvent(event: PrpEvent) {
events.push(event);
if (event.eventType === "turn.failed") staleAppended();
return { cursor: event.sourceSeq, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" };
},
async replayEvents() { return { events: [], highestContiguousSourceSeq: 0 }; },
async completeRun() {},
};
await expect(executeNativeSession({
input, backend, controlPlane: port, persistedSession: interrupted,
runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery",
resumeInterruptedTurn: true,
})).resolves.toMatchObject({ result, turnId: "turn-continuation", terminal: { runTerminalState: "succeeded" } });
expect(startTurn).toHaveBeenCalledOnce();
expect((startTurn.mock.calls[0]![0] as Parameters<NativeSession["startTurn"]>[0]).continuation).toBe(true);
expect(events.filter(event => event.eventType === "turn.failed").map(event => event.turnId)).toEqual(["turn-unrecorded-continuation"]);
});
it.each(["acpx_runtime", "opencode_server"] as const)("continues a restart-lost %s turn once in the same conversation", async (driverKind) => {
const qualified = resolveQualifiedAcpxProfile("claude", "claude-sonnet-5");
const acpxInput: NativeExecutionInputV1 = (driverKind === "acpx_runtime" ? {
@@ -2704,6 +2704,9 @@ export async function executeNativeSession(
};
let restartContinuationStarted = false;
let borrowedContinuationTurnId: string | null = null;
let restartContinuationTurnId: string | null = null;
let settleStartupDecision: () => void = () => undefined;
const startupDecision = new Promise<void>(resolve => { settleStartupDecision = resolve; });
let continuationDispositionRequested = false;
// The consumer may already be running when a continuation starts, and the
// continuation can complete before its start checkpoint settles. A
@@ -2718,6 +2721,7 @@ export async function executeNativeSession(
const steeredMessages = await options.getUnreadSteeredMessages?.() ?? [];
signal?.throwIfAborted();
const started = await session.startTurn(restartContinuation(steeredMessages));
restartContinuationTurnId = started.turnId;
if (started.borrowedDispositionMarker === true) borrowedContinuationTurnId = started.turnId;
await checkpoint(signal);
})();
@@ -2740,8 +2744,20 @@ export async function executeNativeSession(
await checkpoint(signal);
return true;
};
const supersededByRestartContinuation = async (event: PrpEvent, signal: AbortSignal) => {
if (!recovered || !checkpointedInterruption || event.eventType !== "turn.failed" ||
!isNativeRestartInterruption(event.payload.error) || !event.turnId ||
event.turnId === checkpointedInterruption) return false;
await startupDecision;
signal.throwIfAborted();
if (!restartContinuationStarted) return false;
await continuationSetup?.catch(() => undefined);
signal.throwIfAborted();
return restartContinuationTurnId !== null && event.turnId !== restartContinuationTurnId;
};
const resumeInterruptedTurn = async (event: PrpEvent, signal: AbortSignal) => {
if (await requestContinuationDisposition(event, signal)) return true;
if (await supersededByRestartContinuation(event, signal)) return true;
if (restartContinuationStarted || !recovered || !options.resumeInterruptedTurn ||
// A compaction-only run never continues task work after a restart.
// Runnerd records a restart-lost turn only for these providers, after
@@ -3044,7 +3060,9 @@ export async function executeNativeSession(
});
await checkpoint();
}
settleStartupDecision();
} catch (error) {
settleStartupDecision();
// Consumption starts before provider launch so eager events cannot be
// lost. If launch or its checkpoint fails, abort any in-flight append
// before joining cleanup so the failed turn cannot commit late or