fix(native): keep a remote Runner alive when a failed turn start or event stream closes the run (#21023)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 stopped a failed close from force-stopping an unreachable
remote Runner. It asks for retention only when the session was not
cancelled
> - The runtime cancels the session itself when a turn start or the
event stream fails ("Native session event consumption failed."). That
cleanup sets the same flag as a user Stop
> - So after a failed turn start, the close still sent `SIGKILL` to the
Runner, and the same-run reattachment failed with
`runner_remote_process_identity_unavailable`. The task waited for Retry
> - This pull request separates failure cleanup from Stop. Failure
cleanup still revokes the turn's output, but the close keeps the Runner
for the same-run reattachment
> - The benefit is that these runs continue without Retry. Stop and
protocol integrity faults still stop the Runner

## Linked Issues or Issue Description

Refs #20923, #20853

**What happened?**
After the staging redeploy, five remote native runs failed with error
`runner_remote_process_identity_unavailable` and error code
`native_continuation_requires_reconciliation`. In four of them, the
first attempt started after the deploy. Its `turn/start` failed with
`runnerd did not report the provider turn identity`. The run log then
showed `runner did not prove durable suspension before checkpoint` and
`runner shutdown failed: Error: Durable recovery runner timed out.` The
diagnostic `remote runner left alive unsettled for same-run
reattachment` from #20923 did not appear. One second later the Runner
exited (`runner_remote_process_exited lifecycle=ready`). The same-run
reattachment 20–40 s later found `process_verification` /
`process_not_running` with `runnerSettled: false`.

**What did you expect to happen?**
The failed close keeps the Runner alive, as #20923 intended. The
same-run reattachment adopts it, and the run continues without Retry.

**Steps to reproduce**
1. Start a remote native Codex run.
2. Make `turn/start` fail (for example the Runner does not report the
provider turn identity in time), or let the event stream fail or time
out.
3. The runtime cancels the session as cleanup. `HarnessNativeSession`
treats that as a Stop, so `close` asks for `retainUnsettledRemoteRunner:
false`, and the transport kills the Runner.

**Root cause**
`packages/paperclip-runner/src/native-session-runtime.ts`
`attemptOptionalSessionCancellation` calls `session.cancel`.
`HarnessNativeSession.cancel`
(`packages/paperclip-runner/src/backends/harness-driver-backend.ts`)
sets `#explicitlyCancelled`, and `close` used
`retainUnsettledRemoteRunner: !this.#explicitlyCancelled`.

The fifth run is different. A server shutdown detached its controller
and left the Runner alive. Then the sandbox environment was acquired
fresh (45.9 s) before the reattachment, and the Runner was gone in the
middle of a turn. This pull request does not change that path. See
Risks.

## What Changed

- `NativeSession.cancel` accepts an optional `failureCleanup` field.
- `attemptOptionalSessionCancellation` passes `failureCleanup: true`,
except for a `NativeSessionProtocolIntegrityError`.
- `HarnessNativeSession` tracks a Stop separately from output
revocation. `cancel` with `failureCleanup` revokes the turn's output
only. `cancel` without it, and `revokeTurnPublication`, record a Stop.
`close` asks for retention when no Stop was recorded and no protocol
integrity fault occurred.
- Tests: three runtime cases with the real Codex backend (turn start
fails, event stream times out, Stop then failure) and a new backend
case.
- Docs: `doc/execution-semantics.md` describes the rule.

## Verification

- Red, on `master` with only the new tests: `cd
packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts -t "remote Runner retention after a
failed run"` fails 2 of 3 (`expected false to be true`): the close asked
to kill the Runner after the turn start failed and after the event
stream timed out. The Stop case passed.
- Green with the fix: the same command passes 3 of 3.
`src/backends/harness-driver-backend.test.ts -t "retain an unsettled"`
passes 3 of 3, including the failure-cleanup case. That case also checks
that the turn's result stays unpublished.
- `npx vitest run src/native-session-runtime.test.ts src/backends
src/drivers/codex
src/live/runnerd-codex-transport-remote-retention.test.ts
--maxWorkers=4`: 31 files, 814 tests pass.
`codex-protocol-integrity.test.ts` confirms that an integrity fault
still stops the Runner.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` is
clean.
- `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global
CI input. CI supplies that coverage.
- Not run: a live Daytona run. The change is in the controller library
that runs on the server, so it needs no new Runner image.

## Risks

- Low to medium. After a failed turn start or event stream, an
unreachable remote Runner now stays alive instead of being killed. This
is the state #20923 already accepted for other failed closes. The
session stays quarantined, and the same-run reattachment or a Stop
cleanup owns the Runner.
- Stop is unchanged. A server Stop calls `cancel` without
`failureCleanup`, so the close still stops the Runner, also when the
stopped run later fails. A governed wait uses `revokeTurnPublication`,
which also records a Stop.
- A protocol integrity fault needs an operator and is not resumed on the
same run, so it still stops the Runner.
- Previous-release story: this change adds and changes no persisted
state. A Runner that an older release already killed keeps its current
named refusal and Retry. A deploy during a run changes nothing, because
the decision is made in the closing process.
- Not covered: a Runner that dies with its sandbox while the controller
is detached for a restart, in the middle of a turn. Its journal is a
settled `ready` cut with an active turn. #20853 resumes only idle
`ready` cuts. This needs a separate fix.
- No wait is added before `turn.start`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/execution-semantics.md

@@ -1745,6 +1745,11 @@ this, the close stopped the Runner mid-turn, so the reattachment always found it
 gone with an unsettled journal (`runner_remote_process_identity_unavailable`) and
 the task waited for Retry. A Stop, a cancelled turn and a `stop_cleanup` recovery
 still stop the Runner, and a Runner that settled or exited is handled as before.
+The runtime's own cleanup after a failed turn start or a failed event stream
+(for example `runnerd did not report the provider turn identity` or the
+execution timeout) revokes the turn's output but is not a Stop, so it keeps the
+Runner for the same-run reattachment too. A protocol integrity fault, which
+needs an operator, still stops the Runner.
 
 A remote native run can lose its original runner or sandbox before reattachment.
 This happens after a server restart, and for a same-run retry after a turn ends

packages/paperclip-runner/src/backends/harness-driver-backend.test.ts

@@ -125,17 +125,20 @@ describe("HarnessDriverBackend", () => {
   });
 
   it.each([
-    { cancelled: false, retainUnsettledRemoteRunner: true },
+    { cancelled: false, failureCleanup: false, retainUnsettledRemoteRunner: true },
-    { cancelled: true, retainUnsettledRemoteRunner: false },
+    { cancelled: true, failureCleanup: false, retainUnsettledRemoteRunner: false },
-  ])("asks to retain an unsettled remote Runner on close only when the run was not cancelled (cancelled=$cancelled)",
+    { cancelled: true, failureCleanup: true, retainUnsettledRemoteRunner: true },
-    async ({ cancelled, retainUnsettledRemoteRunner }) => {
+  ])("asks to retain an unsettled remote Runner on close only when the run was not stopped (cancelled=$cancelled, failureCleanup=$failureCleanup)",
+    async ({ cancelled, failureCleanup, retainUnsettledRemoteRunner }) => {
       const close = vi.fn(async () => {});
       const session = Object.assign(new FakeHarnessSession(), { close });
       const backend = new HarnessDriverBackend({ ...driver, openSession: async () => session });
       const opened = await backend.openSession({ identity: {
         runId: "run-1", sessionId: "session-1", companyId: "company-1", issueId: "issue-1", agentId: "agent-1",
       } });
-      if (cancelled) await cancelSession(opened, { reason: "Stop requested", signal: new AbortController().signal }).cleanup;
+      if (cancelled) await cancelSession(opened, { reason: failureCleanup ? "Native session event consumption failed." : "Stop requested",
+        signal: new AbortController().signal, ...(failureCleanup ? { failureCleanup } : {}) }).cleanup;
+      if (failureCleanup) expect(await opened.result()).toBeNull();
       await opened.close({ reason: "native session execution complete" });
       expect(close).toHaveBeenCalledExactlyOnceWith({ reason: "native session execution complete", retainUnsettledRemoteRunner });
     });

packages/paperclip-runner/src/backends/harness-driver-backend.ts

@@ -404,6 +404,7 @@ class HarnessNativeSession implements NativeSession {
   readonly #session: HarnessSession;
   #terminal: PrpTerminalState | null = null;
   #explicitlyCancelled = false;
+  #stopRequested = false;
   #protocolIntegrityFailure: NativeSessionProtocolIntegrityError | null = null;
 
   #assertProtocolIntegrity(): void {
@@ -525,6 +526,7 @@ class HarnessNativeSession implements NativeSession {
     this.#input = { ...this.#input, identity: structuredClone(input.identity) };
     this.#terminal = null;
     this.#explicitlyCancelled = false;
+    this.#stopRequested = false;
   }
 
   async release(): Promise<DurableRecoveryIdentity | false> {
@@ -780,6 +782,7 @@ class HarnessNativeSession implements NativeSession {
 
   revokeTurnPublication() {
     this.#explicitlyCancelled = true;
+    this.#stopRequested = true;
   }
 
   settleCancelledTurn(input: { turnId: string; reason: string }) {
@@ -788,7 +791,7 @@ class HarnessNativeSession implements NativeSession {
     return this.#withProtocolIntegrity(() => this.#session.settleCancelledTurn?.(input) ?? null);
   }
 
-  cancel(input: { reason: string; signal: AbortSignal }) {
+  cancel(input: { reason: string; signal: AbortSignal; failureCleanup?: boolean }) {
     if (input.signal.aborted) {
       throw (
         input.signal.reason ?? new Error("native session cancellation aborted")
@@ -797,7 +800,8 @@ class HarnessNativeSession implements NativeSession {
     // This flag is the adapter's synchronous publication boundary. Provider
     // interruption happens afterward as passive cleanup, so a slow or broken
     // transport cannot synthesize or publish new accepted output for the turn.
-    this.revokeTurnPublication();
+    if (input.failureCleanup === true) this.#explicitlyCancelled = true;
+    else this.revokeTurnPublication();
     const interrupt = this.#session.interrupt;
     return {
       ...(this.terminalDrainTimeoutMs === undefined ? {} : { terminalDrainTimeoutMs: this.terminalDrainTimeoutMs }),
@@ -922,7 +926,7 @@ class HarnessNativeSession implements NativeSession {
   }
 
   close(input: { reason: string }) {
-    return this.#session.close({ ...input, retainUnsettledRemoteRunner: !this.#explicitlyCancelled });
+    return this.#session.close({ ...input, retainUnsettledRemoteRunner: !this.#stopRequested && this.#protocolIntegrityFailure === null });
   }
 }
 

packages/paperclip-runner/src/contracts/native-session-backend.ts

@@ -270,6 +270,7 @@ export interface NativeSession {
   cancel?(input: {
     reason: string;
     signal: AbortSignal;
+    failureCleanup?: boolean;
   }): NativeSessionCancellation;
   settleCancelledTurn?(input: { turnId: string; reason: string }): Promise<CancelledTurnSettlement | null>;
   resolveRuntimeRequest?(input: {

packages/paperclip-runner/src/native-session-runtime.test.ts

@@ -5961,6 +5961,7 @@ describe("executeNativeSession recovery", () => {
     expect(cancel).toHaveBeenCalledWith({
       reason: "Native session event consumption failed.",
       signal: expect.any(AbortSignal),
+      failureCleanup: true,
     });
     expect(providerResult).not.toHaveBeenCalled();
     expect(events.map((event) => event.eventType)).toEqual([
@@ -10973,3 +10974,62 @@ describe("first-turn queued input", () => {
     expect((JSON.parse(sent[0]!) as { task: { prompt: string } }).task.prompt).toBe(input.task.prompt);
   });
 });
+
+describe("remote Runner retention after a failed run", () => {
+  const port = (): ControlPlanePort => ({
+    async openRun() {}, async checkpointSession() {}, async completeRun() {},
+    async appendEvent(event: PrpEvent) {
+      return { cursor: event.sourceSeq, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" };
+    },
+    async replayEvents() { return { events: [], highestContiguousSourceSeq: 0 }; },
+  });
+  const startCodexRun = (transport: FakeCodexTransport, options: { timeoutMs?: number;
+    onSession?: (session: NativeSession | null) => void } = {}) => {
+    const close = vi.spyOn(transport, "close");
+    const executionInput = { ...input, workspace: { ...input.workspace, cwd: WORKSPACE } };
+    const backend = createCodexNativeSessionBackend(executionInput, {
+      runnerInstanceId: "runner-recovery", transportFactory: () => transport,
+    });
+    const execution = executeNativeSession({ input: executionInput, backend, controlPlane: port(),
+      runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery",
+      timeoutMs: options.timeoutMs ?? 5_000, requireSessionCloseBeforeReturn: true, onSession: options.onSession });
+    void execution.catch(() => undefined);
+    return { execution, close };
+  };
+  const retainedOn = (close: { mock: { calls: unknown[][] } }): Array<boolean | undefined> =>
+    close.mock.calls.map((call: unknown[]) =>
+      (call[1] as { retainUnsettledRemoteRunner?: boolean } | undefined)?.retainUnsettledRemoteRunner);
+
+  it("keeps the Runner for same-run reattachment when the turn start fails", async () => {
+    const transport = new FakeCodexTransport();
+    transport.rejectMethods.set("turn/start", new Error("runnerd did not report the provider turn identity"));
+    const { execution, close } = startCodexRun(transport);
+    await expect(execution).rejects.toThrow("runnerd did not report the provider turn identity");
+    expect(retainedOn(close)).not.toHaveLength(0);
+    expect(retainedOn(close).every(retained => retained === true)).toBe(true);
+  });
+
+  it("keeps the Runner for same-run reattachment when event consumption fails", async () => {
+    const transport = new FakeCodexTransport();
+    const { execution, close } = startCodexRun(transport, { timeoutMs: 100 });
+    await vi.waitFor(() => expect(transport.calls.some(call => call.method === "turn/start")).toBe(true));
+    transport.push("turn/started", { threadId: "thread-1", turn: { id: "turn-1", status: "inProgress" } });
+    await expect(execution).rejects.toThrow("native session timed out after 100ms");
+    expect(retainedOn(close)).not.toHaveLength(0);
+    expect(retainedOn(close).every(retained => retained === true)).toBe(true);
+  });
+
+  it("still stops the Runner after a Stop, even when the stopped run then fails", async () => {
+    const transport = new FakeCodexTransport();
+    let published: NativeSession | null = null;
+    const { execution, close } = startCodexRun(transport, { timeoutMs: 300,
+      onSession: session => { published ??= session; } });
+    await vi.waitFor(() => expect(transport.calls.some(call => call.method === "turn/start")).toBe(true));
+    transport.push("turn/started", { threadId: "thread-1", turn: { id: "turn-1", status: "inProgress" } });
+    await vi.waitFor(() => expect(published).not.toBeNull());
+    void published!.cancel!({ reason: "Stop requested", signal: new AbortController().signal }).cleanup.catch(() => undefined);
+    await expect(execution).rejects.toThrow();
+    expect(retainedOn(close)).not.toHaveLength(0);
+    expect(retainedOn(close).every(retained => retained === false)).toBe(true);
+  });
+});

packages/paperclip-runner/src/native-session-runtime.ts

@@ -482,6 +482,7 @@ function terminalFromEvent(
 async function attemptOptionalSessionCancellation(
   session: NativeSession,
   reason: string,
+  failure: unknown,
 ): Promise<{ settlement: Promise<PromiseSettledResult<void>[]> } | null> {
   if (session.cancel === undefined) return null;
   const cancellationAbort = new AbortController();
@@ -493,6 +494,7 @@ async function attemptOptionalSessionCancellation(
     cleanup = session.cancel({
       reason,
       signal: cancellationAbort.signal,
+      ...(failure instanceof NativeSessionProtocolIntegrityError ? {} : { failureCleanup: true }),
     }).cleanup;
   } catch {
     return null;
@@ -1629,6 +1631,7 @@ async function consumeTurn(
       const deferredCancellation = await attemptOptionalSessionCancellation(
         session,
         "Native session event consumption failed.",
+        error,
       );
       deferredSessionCancellationSettlement =
         deferredCancellation?.settlement ?? null;