mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
fix(runner): keep a restart continuation when a stale earlier continuation is reported lost (#20118)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the runner reports it as `provider_turn_lost_on_restore` and the controller starts one same-conversation continuation. A later restart that loses the continuation is answered the same way > - With back-to-back restarts, the controller's checkpoint can miss a continuation that the runner already accepted. On the next restore the controller starts a fresh continuation, and the restored runner then reports the unrecorded earlier continuation as lost > - The event consumer treated that stale lost terminal as the run's terminal, so the run failed while its new continuation was already running > - Recovery then refused to replay the failed run, and the task needed a Retry. In an M1 parent/subtask task, the parent's Retry succeeded but the parent stayed `blocked` on that subtask > - This pull request makes the consumer skip a restart-lost terminal for any turn other than the continuation it started > - The benefit is that back-to-back restarts no longer turn a recoverable lost turn into a failed run that needs Retry ## Linked Issues or Issue Description Refs #20107 **What happened?** Restart-chaos K14 wave w-20261108T213600-s9803 (local, an upgrade between builds plus repeated restarts) at070365404e. In task t02 (claude M1), the codex subtask CHA-7 failed with `native_provider_terminal_failed: The runner restored the conversation after process loss, but the previous turn is no longer active.` The sequence: 1. Restore 1 lost turn `01a1bd17…`, and the controller started continuation `01a1bd18-15ec…` (runner command 14). A restart followed within a second, before the controller's checkpoint recorded that continuation. 2. Restore 2 saw the checkpointed interruption of `01a1bd17…` and started a fresh continuation, `01a1bd18-4cc4…` (command 24). The restored runner also reported the unrecorded continuation `01a1bd18-15ec…` as `provider_turn_lost_on_restore`. 3. Because a continuation was already started, the consumer returned that stale terminal as the run's terminal. The run failed, the new continuation was stopped (command 25), and automatic recovery refused to replay it (`uncertain_provider_action`, `preserve_without_replay_v1`). 4. The parent CHA-2 asked CHA-7 to resume by comment, but an agent comment cannot authorize a blocked replay, so the wake stayed `deferred_issue_execution`. The parent's own Retry succeeded, but CHA-2 stayed `blocked` on CHA-7 (`R3:retry_did_not_complete`). **Expected behavior** A restart-lost terminal for an older, unrecorded continuation is superseded by the continuation this restore started. The run continues on that continuation and needs no Retry. **Steps to reproduce** 1. Checkpoint a native session whose last terminal is a restart-lost turn and whose active turn is null, so recovery starts a continuation. 2. Have the restored session report another turn (the unrecorded earlier continuation) as `provider_turn_lost_on_restore`, either before or after the new continuation starts. 3. Before this change, the run ends on that stale terminal. **Paperclip version or commit**070365404e(wave); branch rebased on current master. ## What Changed - `native-session-runtime.ts`: - Records the turn id of the restart continuation it starts. - New `supersededByRestartContinuation`, used by the consumer's `resumeInterruptedTurn`, keeps consuming past a `provider_turn_lost_on_restore` `turn.failed` when all of these hold: - the run was recovered from a checkpointed interruption; - the failed turn is neither the checkpointed interrupted turn nor the started continuation; - a continuation was actually started. - The check first waits for the startup decision, because the consumer starts before the continuation does. - `native-session-runtime.test.ts`: a red/green test, with the stale lost terminal arriving both after and before the continuation starts. ## Verification - Red/green: - On master, both test cases end the run on the stale terminal. - With the change, both resolve on the continuation's result. The continuation is started once, and the stale `turn.failed` is still appended to the journal. - `vitest run` over the restart, recovery, continuation and terminal test files in `packages/paperclip-runner`, including all of `native-session-runtime.test.ts`: 9 files, 380 tests passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck:typescript` (`packages/paperclip-runner`): passed. ## Risks - Low risk. The skip applies only to restart-lost failures of turns other than the checkpointed interrupted turn and the started continuation, and only after a continuation actually started. Real provider failures, a loss of the continuation itself, and runs without a checkpointed interruption keep their current handling. - If startup fails, the startup decision still settles. The consumer then falls back to the existing behavior and the run fails as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
412d49e48d
commit
b43cccbb08
2 files changed
+74
No files matched your search
@@ -586,6 +586,62 @@ describe("executeNativeSession recovery", () => {
|
||||
expect(checkpoints.some(snapshot => snapshot.terminalTurns?.some(turn => turn.turnId === "turn-recovery"))).toBe(true);
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ label: "after the continuation starts", staleFirst: false },
|
||||
{ label: "before the continuation starts", staleFirst: true },
|
||||
])("keeps the restart continuation when the restored runner reports an earlier unrecorded continuation lost $label", async ({ staleFirst }) => {
|
||||
let staleAppended!: () => void;
|
||||
const staleTerminal = new Promise<void>(resolve => { staleAppended = resolve; });
|
||||
const failure = { code: "provider_turn_lost_on_restore", recoverable: true };
|
||||
const interrupted: PersistedNativeSession = {
|
||||
backendKind: "runner", driverKind: "codex_app_server", sessionId: "thread-recovery", identity,
|
||||
providerSessionId: "provider-recovery", cursor: "0", activeTurnId: null,
|
||||
terminalTurns: [{ turnId: "turn-recovery", fingerprint: JSON.stringify({ terminalState: "failed", error: failure, result: null }) }],
|
||||
semanticResult: null, pendingRuntimeRequests: [], lineage: [],
|
||||
};
|
||||
const startTurn = vi.fn<NativeSession["startTurn"]>(async () => {
|
||||
if (staleFirst) await staleTerminal;
|
||||
return { turnId: "turn-continuation" };
|
||||
});
|
||||
const session: NativeSession = {
|
||||
identity: () => identity,
|
||||
async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; },
|
||||
async *events() {
|
||||
yield { ...runnerEvent(1, "turn.failed", { status: "failed", error: failure }), turnId: "turn-unrecorded-continuation" };
|
||||
yield { ...runnerEvent(2, "turn.completed", { status: "completed" }), turnId: "turn-continuation" };
|
||||
},
|
||||
startTurn,
|
||||
async result() { return startTurn.mock.calls.length ? { result, terminal, turnId: "turn-continuation" } : null; },
|
||||
async snapshot() { return structuredClone(interrupted); },
|
||||
async close() {},
|
||||
};
|
||||
const backend: NativeSessionBackend = {
|
||||
async descriptor() { return { kind: "runner", name: "restart", version: "1", capabilities: await session.capabilities() }; },
|
||||
async openSession() { throw new Error("must preserve the provider session"); },
|
||||
async recoverSession() { return { recovered: true, session }; },
|
||||
};
|
||||
const events: PrpEvent[] = [];
|
||||
const port: ControlPlanePort = {
|
||||
async openRun() {},
|
||||
async checkpointSession() {},
|
||||
async appendEvent(event: PrpEvent) {
|
||||
events.push(event);
|
||||
if (event.eventType === "turn.failed") staleAppended();
|
||||
return { cursor: event.sourceSeq, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" };
|
||||
},
|
||||
async replayEvents() { return { events: [], highestContiguousSourceSeq: 0 }; },
|
||||
async completeRun() {},
|
||||
};
|
||||
await expect(executeNativeSession({
|
||||
input, backend, controlPlane: port, persistedSession: interrupted,
|
||||
runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery",
|
||||
resumeInterruptedTurn: true,
|
||||
})).resolves.toMatchObject({ result, turnId: "turn-continuation", terminal: { runTerminalState: "succeeded" } });
|
||||
expect(startTurn).toHaveBeenCalledOnce();
|
||||
expect((startTurn.mock.calls[0]![0] as Parameters<NativeSession["startTurn"]>[0]).continuation).toBe(true);
|
||||
expect(events.filter(event => event.eventType === "turn.failed").map(event => event.turnId)).toEqual(["turn-unrecorded-continuation"]);
|
||||
});
|
||||
|
||||
it.each(["acpx_runtime", "opencode_server"] as const)("continues a restart-lost %s turn once in the same conversation", async (driverKind) => {
|
||||
const qualified = resolveQualifiedAcpxProfile("claude", "claude-sonnet-5");
|
||||
const acpxInput: NativeExecutionInputV1 = (driverKind === "acpx_runtime" ? {
|
||||
|
||||
@@ -2704,6 +2704,9 @@ export async function executeNativeSession(
|
||||
};
|
||||
let restartContinuationStarted = false;
|
||||
let borrowedContinuationTurnId: string | null = null;
|
||||
let restartContinuationTurnId: string | null = null;
|
||||
let settleStartupDecision: () => void = () => undefined;
|
||||
const startupDecision = new Promise<void>(resolve => { settleStartupDecision = resolve; });
|
||||
let continuationDispositionRequested = false;
|
||||
// The consumer may already be running when a continuation starts, and the
|
||||
// continuation can complete before its start checkpoint settles. A
|
||||
@@ -2718,6 +2721,7 @@ export async function executeNativeSession(
|
||||
const steeredMessages = await options.getUnreadSteeredMessages?.() ?? [];
|
||||
signal?.throwIfAborted();
|
||||
const started = await session.startTurn(restartContinuation(steeredMessages));
|
||||
restartContinuationTurnId = started.turnId;
|
||||
if (started.borrowedDispositionMarker === true) borrowedContinuationTurnId = started.turnId;
|
||||
await checkpoint(signal);
|
||||
})();
|
||||
@@ -2740,8 +2744,20 @@ export async function executeNativeSession(
|
||||
await checkpoint(signal);
|
||||
return true;
|
||||
};
|
||||
const supersededByRestartContinuation = async (event: PrpEvent, signal: AbortSignal) => {
|
||||
if (!recovered || !checkpointedInterruption || event.eventType !== "turn.failed" ||
|
||||
!isNativeRestartInterruption(event.payload.error) || !event.turnId ||
|
||||
event.turnId === checkpointedInterruption) return false;
|
||||
await startupDecision;
|
||||
signal.throwIfAborted();
|
||||
if (!restartContinuationStarted) return false;
|
||||
await continuationSetup?.catch(() => undefined);
|
||||
signal.throwIfAborted();
|
||||
return restartContinuationTurnId !== null && event.turnId !== restartContinuationTurnId;
|
||||
};
|
||||
const resumeInterruptedTurn = async (event: PrpEvent, signal: AbortSignal) => {
|
||||
if (await requestContinuationDisposition(event, signal)) return true;
|
||||
if (await supersededByRestartContinuation(event, signal)) return true;
|
||||
if (restartContinuationStarted || !recovered || !options.resumeInterruptedTurn ||
|
||||
// A compaction-only run never continues task work after a restart.
|
||||
// Runnerd records a restart-lost turn only for these providers, after
|
||||
@@ -3044,7 +3060,9 @@ export async function executeNativeSession(
|
||||
});
|
||||
await checkpoint();
|
||||
}
|
||||
settleStartupDecision();
|
||||
} catch (error) {
|
||||
settleStartupDecision();
|
||||
// Consumption starts before provider launch so eager events cannot be
|
||||
// lost. If launch or its checkpoint fails, abort any in-flight append
|
||||
// before joining cleanup so the failed turn cannot commit late or
|
||||
|
||||
Reference in new issue
Block a user