fix(runner): prove ownership right before starting a turn in a recovered session (#21260)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner runs native agent sessions (Codex, Claude, and
others) and keeps them alive across server swaps. A new server
generation recovers the session and continues it.
> - Only the generation that owns the run may submit a turn into the
provider session. The Runner proves ownership with a fenced checkpoint
(controller boot, generation and lease, judged by the database clock).
> - On a recovered session, the startup turn and the restart
continuation took that checkpoint right after recovery, then prepared
the turn's input (attachments, queued input, steered messages), then
submitted.
> - A generation that lost the run during input preparation still
submitted a turn into the shared provider session.
> - This pull request repeats the fenced checkpoint right before
`turn.start` on those two recovered-session paths.
> - The benefit is that a server generation that no longer owns a run
cannot add a turn to its provider session. New sessions are unchanged,
so nothing is added before their first turn.

## Linked Issues or Issue Description

**What happened?**
On staging, a Codex run crossed three server swaps. After its turn
finished without a result, a turn that no generation could account for
appeared in the same provider thread. Every path that submits a turn
into an existing session is preceded by the fenced checkpoint, but the
two startup paths took it before input preparation, not right before the
submit. A generation that lost the run in that window still submitted.

**Expected behavior**
A generation submits a turn into a recovered session only if it still
owns the run at the moment of the submit.

**Steps to reproduce**
1. Recover a native session on a new server generation.
2. While it prepares the startup turn's input, fence the generation
(another generation takes the lease).
3. Before this change, the turn is still submitted. After it, the
checkpoint fails and no turn starts.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
- The restart continuation calls the fenced `checkpoint` right before
`session.startTurn`, after attachments and steered messages are
prepared.
- The startup turn on a recovered session (`recovered === true`) calls
`checkpoint` right before `session.startTurn`. A new session's normal
path is unchanged.
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: new
tests "proves it still owns a recovered session right before starting a
turn in it" (startup turn) and "proves it still owns a recovered session
right before a restart continuation" at startup and when the lost turn's
terminal arrives (`resumeInterruptedTurn`).

Turn starts into an existing session, and how each proves ownership
right before the submit:
- Startup turn on a recovered session: new checkpoint, covered by the
first test.
- Restart continuation (`startRestartContinuation`), at startup and via
`resumeInterruptedTurn`: new checkpoint, covered by the second test
(both cases).
- Continuation disposition (`requestContinuationDisposition`): already
persists the fenced checkpoint immediately before `startTurn`, with only
a snapshot read between them and no input preparation. No change.
- Compaction turn: started right after the fenced recovery checkpoint
with an empty message; there is no input preparation between them (the
only possible await is the disposition-replay read, which applies only
when a disposition turn was submitted, and then no compaction turn
starts). No change.
- Runner-only change. No change to `protocol/`, `generated/`, Rust
shapes or the ACPX patch, so the provider-pack `contractDigest` and
interface revisions are unchanged and older image packs stay compatible.
The Runner build and image content for this source change are produced
by CI (`runner-artifacts`); nothing is deployed by this PR.

## Verification

- Red-to-green: "proves it still owns a recovered session right before
starting a turn in it" fails without the startup-turn checkpoint (the
turn is submitted after the generation is fenced) and passes with it.
Both restart-continuation cases fail with the checkpoint removed from
`startRestartContinuation` and pass with it.
- Local: `cd packages/paperclip-runner && pnpm exec vitest run
src/native-session-runtime.test.ts`: 290 passed.
- Local: `node scripts/typecheck-test-programs.mjs
packages/paperclip-runner/tsconfig.test.json`: passed (the first CI run
failed on the test's attachment prompt returning `undefined`; it now
returns a string).
- **Not run locally:** one test case that needs the Rust
`paperclip-runnerd` binary, and
`src/live/runnerd-restart-continuation.test.ts` (10 cases). The Rust
binary was not built on this machine. CI must cover them.

## Risks

- One extra checkpoint round trip before `turn.start` on a recovered
session and on a restart continuation. These are recovery paths; a new
session's first turn gets no added wait, so boot and time-to-first-token
for new runs are unchanged.
- If the checkpoint fails (the generation lost the run), the turn is not
submitted. That is the intended behaviour; the owning generation
continues the run.
- No migration. No schema change. No pack contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 21:08:19 -06:00
1 parent db802d1230
commit a298ca7fbc
2 files changed
+106

No files matched your search

@@ -8896,6 +8896,105 @@ describe("executeNativeSession recovery", () => {
).toEqual(["run.result.accepted", "run.terminal"]);
});
it("proves it still owns a recovered session right before starting a turn in it", async () => {
// Staging run cd2bf273: a turn nobody can attribute appeared in the same
// provider thread across server swaps. The recovery checkpoint is fenced
// (controller boot, generation, lease), but preparing the turn's input can
// take a while and another generation may take the run over meanwhile.
// The fenced checkpoint is repeated right before the submit, so the
// replaced server submits nothing.
const checkpoint: PersistedNativeSession = {
backendKind: "mock", sessionId: "driver-recovery", identity, providerSessionId: "provider-recovery", cursor: "1",
activeTurnId: null, terminalTurns: [], pendingRuntimeRequests: [], lineage: [],
};
const startTurn = vi.fn(async () => ({ turnId: "late-turn" }));
const session: NativeSession = {
identity: () => identity,
async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; },
events: () => (async function* () {})(),
startTurn,
async result() { return null; },
async snapshot() { return structuredClone(checkpoint); },
async close() {},
};
const backend: NativeSessionBackend = {
async descriptor() {
return { kind: "mock", name: "recovery-backend", version: "1",
capabilities: { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true } };
},
async openSession() { throw new Error("must recover the provider session"); },
async recoverSession() { return { recovered: true, session }; },
};
let takenOver = false;
const checkpointSession = vi.fn(async () => { if (takenOver) throw new Error("native_runner_checkpoint_owner_lost"); });
const port: ControlPlanePort = {
async openRun() {},
async loadSessionCheckpoint() { return structuredClone(checkpoint); },
checkpointSession,
async appendEvent() { return { cursor: 1, highestContiguousSourceSeq: 0, disposition: "committed" }; },
async replayEvents() { return { events: [], highestContiguousSourceSeq: 0 }; },
completeRun: vi.fn(async () => undefined),
};
await executeNativeSession({ input, backend, controlPlane: port, runnerInstanceId: "runner-recovery",
controlPlaneInstanceId: "control-recovery",
// Another server generation claims the run while this one prepares the turn.
prepareWakeAttachmentPrompt: async () => { takenOver = true; return ""; } }).catch(() => undefined);
expect(takenOver).toBe(true);
expect(startTurn).not.toHaveBeenCalled();
});
it.each([
{ label: "at startup", interruptedAtStartup: true },
{ label: "when the lost turn's terminal arrives", interruptedAtStartup: false },
])("proves it still owns a recovered session right before a restart continuation $label", async ({ interruptedAtStartup }) => {
// Same takeover window as the startup turn, on the restart-continuation
// path: steered messages and attachments are prepared before the submit.
const failure = { code: "provider_turn_lost_on_restore", recoverable: true };
const checkpoint: PersistedNativeSession = {
backendKind: "runner", driverKind: "acpx_runtime", sessionId: "record-recovery", identity,
providerSessionId: "provider-recovery", cursor: "0", activeTurnId: "turn-recovery",
terminalTurns: [], semanticResult: null, pendingRuntimeRequests: [], lineage: [],
};
const interrupted: PersistedNativeSession = {
...checkpoint, activeTurnId: null,
terminalTurns: [{ turnId: "turn-recovery", fingerprint: JSON.stringify({ terminalState: "failed", error: failure, result: null }) }],
};
const startTurn = vi.fn<NativeSession["startTurn"]>(async () => ({ turnId: "turn-continuation" }));
const session: NativeSession = {
identity: () => identity,
async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; },
async *events() {
if (!interruptedAtStartup) yield runnerEvent(1, "turn.failed", { status: "failed", error: failure });
},
startTurn,
async result() { return null; },
async snapshot() { return structuredClone(interrupted); },
async close() {},
};
const backend: NativeSessionBackend = {
async descriptor() { return { kind: "runner", name: "restart", version: "1", capabilities: await session.capabilities() }; },
async openSession() { throw new Error("must preserve the provider session"); },
async recoverSession() { return { recovered: true, session }; },
};
let takenOver = false;
const port: ControlPlanePort = {
async openRun() {},
async checkpointSession() { if (takenOver) throw new Error("native_runner_checkpoint_owner_lost"); },
async appendEvent(event: PrpEvent) { return { cursor: event.sourceSeq, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" }; },
async replayEvents() { return { events: [], highestContiguousSourceSeq: 0 }; },
async completeRun() {},
};
const prepareWakeAttachmentPrompt = vi.fn(async () => { takenOver = true; return ""; });
await executeNativeSession({
input, backend, controlPlane: port, persistedSession: interruptedAtStartup ? interrupted : checkpoint,
runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery", resumeInterruptedTurn: true,
// Another server generation claims the run while this one prepares the continuation.
prepareWakeAttachmentPrompt,
}).catch(() => undefined);
expect(prepareWakeAttachmentPrompt).toHaveBeenCalledOnce();
expect(startTurn).not.toHaveBeenCalled();
});
it("resolves a checkpointed result-less disposition without resubmitting when its terminal event is missing", async () => {
const workProposal: PrpEvent = {
...runnerEvent(1, "run.result.proposed", result),
@@ -2867,6 +2867,9 @@ export async function executeNativeSession(
signal?.throwIfAborted();
const attachmentPrompt = await options.prepareWakeAttachmentPrompt?.();
signal?.throwIfAborted();
// The fenced checkpoint proves this controller still owns the run
// right before the submit; input preparation above may have taken a while.
await checkpoint(signal);
const started = await session.startTurn(restartContinuation(steeredMessages, attachmentPrompt));
restartContinuationTurnId = started.turnId;
if (started.borrowedDispositionMarker === true) borrowedContinuationTurnId = started.turnId;
@@ -3208,6 +3211,10 @@ export async function executeNativeSession(
modelEnvelope.events = `${modelEnvelope.events}\n\n${supplemental}`;
}
}
// A recovered session is shared provider state: prove ownership with
// the fenced checkpoint right before the submit. A new session's
// normal path adds nothing before turn.start.
if (recovered) await checkpoint();
await session.startTurn({
message: { role: "user", text: JSON.stringify(modelEnvelope) },
...(recovered && modelEnvelope.schema === "paperclip.native-continuation.v1"