fix(runner): stop capability live-session tests from failing on unhandled turn-timeout rejections (#12676)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner package manages live sessions and durable
command recovery
> - Capability live-session tests can fail when a turn-timeout rejection
has no handler
> - A resumed session can also stall when the durable control plane
rejects an indeterminate command result
> - These failures make valid tests fail or hide the turn that stalled
> - This pull request captures timeout rejections early and accepts
indeterminate recovered commands
> - The benefit is stable tests and clearer timeout failures after a
runner restart

## Linked Issues or Issue Description

**What happened?**

Capability live-session tests failed intermittently on loaded CI hosts.
A timer could reject a turn promise before the test attached its
assertion. A resumed session could also stall after a runner restart
because the durable control plane rejected the indeterminate command
status.

**Expected behavior**

The test must handle a timeout rejection at promise creation. The
durable control plane must accept an indeterminate recovered command and
allow the session to continue. A configured timeout must persist in the
checkpoint and identify the stalled turn.

**Steps to reproduce**

1. Run the capability live-session test file on a loaded host.
2. Create a turn promise with a timeout and delay before attaching its
assertion.
3. Resume a session after a runner restart with a journaled but
unconfirmed command.
4. Observe the unhandled rejection or the stalled resumed session.

**Paperclip version or commit**

Commit `ede642e57e22ea3fb0a73590fca8bcc994f1a47f` on `master`.

**Deployment mode**

Built from source.

**Installation method**

Built from source.

**Agent adapter(s) involved**

Not adapter-specific. The tests use the Paperclip Runner package.

**Database mode**

Not database-related.

**Additional context**

Pull request #12646 also updates durable recovery for indeterminate
command results. If it lands first, this pull request must retain the
compatible behavior without duplicate edits.

## What Changed

- Add a helper that captures a turn rejection before any await step.
- Update three live-session test sites to assert the captured rejection
value.
- Add a helper test that waits past the turn timeout before it asserts.
- Accept indeterminate as a terminal recovered-command status.
- Add tests for acceptance, duplicate absorption, and reload from
persisted state.
- Add an optional turnTimeoutMs value to resume and pin its checkpoint
behavior.

## Verification

- `npx vitest run src/live/live-session.test.ts` from
`packages/paperclip-runner`: 19 passed, 1 skipped.
- `npx vitest run src/control-plane/durable-prp-control-plane.test.ts`
from `packages/paperclip-runner`: 5 passed.
- The live-session file passed 10 of 10 runs with 30 competing workers
on a 32-core host.
- TypeScript reported five pre-existing errors in
`src/eval/workflow-harness.ts`.
- CI must run `pnpm --filter @paperclipai/paperclip-runner check:all`.

## Risks

The durable control plane now accepts one additional terminal recovery
status. The change affects only recovered command handling and
capability live-session tests. The main risk is overlap with pull
request #12646 if that pull request lands first.

## Model Used

OpenAI Codex, GPT-5, with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

packages/paperclip-runner/src/live/live-session.test.ts

@@ -22,6 +22,7 @@ import {
 } from "./live-session.js";
 import { DurableCapabilityLiveSessionStore } from "./durable-live-session-store.js";
 import { defaultCapabilityRunnerdBinary } from "./runnerd-codex-transport.js";
+import { captureTurnRejection } from "../../test/capture-turn-rejection.js";
 
 class AsyncNotifications implements AsyncIterable<CodexRpcNotification> {
   #values: CodexRpcNotification[] = [];
@@ -781,7 +782,7 @@ describe("Capability live runnerd and Codex session", () => {
       turnTimeoutMs: 500,
     });
     state.holdAfterTool = true;
-    const killedTurn = first.sendMessage("Apply idempotent progress once.");
+    const killedTurn = captureTurnRejection(first.sendMessage("Apply idempotent progress once."));
     await vi.waitFor(async () => {
       expect((await firstStore.load(binding.sessionId))?.mockState).toContain(
         "Progress persisted through the live Codex tool loop.",
@@ -799,7 +800,7 @@ describe("Capability live runnerd and Codex session", () => {
       costNanodollars: 1_500,
     })).resolves.toBe("committed");
     await state.transports[0]?.close();
-    await expect(killedTurn).rejects.toThrow("timed out");
+    await expect(killedTurn).resolves.toMatchObject({ message: expect.stringContaining("timed out") });
 
     // A new service models worker termination: it owns no in-memory session.
     const resumedService = new CapabilityLiveSessionService({
@@ -864,6 +865,46 @@ describe("Capability live runnerd and Codex session", () => {
     });
   });
 
+  it("persists a resumed turnTimeoutMs override so a later resume that omits it keeps the value", async () => {
+    const state = providerState();
+    const store = new InMemoryCapabilityLiveSessionStore();
+    const sessionId = "session-turn-timeout-persists";
+    const firstService = new CapabilityLiveSessionService({
+      store,
+      transportFactory: fakeTransportFactory(state),
+    });
+    await firstService.create({
+      runId: "run-turn-timeout-persists",
+      sessionId,
+      attemptId: "attempt-first",
+    });
+
+    const secondService = new CapabilityLiveSessionService({
+      store,
+      transportFactory: fakeTransportFactory(state),
+    });
+    const secondResume = await secondService.resume({
+      sessionId,
+      attemptId: "attempt-second",
+      resumeOf: "attempt-first",
+      turnTimeoutMs: 5_000,
+    });
+    expect(secondResume.snapshot().config.turnTimeoutMs).toBe(5_000);
+    expect((await store.load(sessionId))?.config.turnTimeoutMs).toBe(5_000);
+
+    const thirdService = new CapabilityLiveSessionService({
+      store,
+      transportFactory: fakeTransportFactory(state),
+    });
+    const thirdResume = await thirdService.resume({
+      sessionId,
+      attemptId: "attempt-third",
+      resumeOf: "attempt-second",
+    });
+    expect(thirdResume.snapshot().config.turnTimeoutMs).toBe(5_000);
+    expect((await store.load(sessionId))?.config.turnTimeoutMs).toBe(5_000);
+  });
+
   it("replays only a durable duplicate after restore reconciles the provider turn as interrupted", async () => {
     const state = providerState();
     const store = new InMemoryCapabilityLiveSessionStore();
@@ -878,10 +919,7 @@ describe("Capability live runnerd and Codex session", () => {
       turnTimeoutMs: 50,
     });
     state.holdAfterTool = true;
-    const killedTurn = first.sendMessage("Apply idempotent progress once.").then(
-      () => null,
-      (error: unknown) => error,
-    );
+    const killedTurn = captureTurnRejection(first.sendMessage("Apply idempotent progress once."));
     await vi.waitFor(async () => {
       expect((await store.load(first.id))?.mockState).toContain("progress-governed-once");
     });
@@ -1037,6 +1075,20 @@ describe("Capability live runnerd and Codex session", () => {
     await expect(store.load(binding.sessionId)).rejects.toThrow("capability_live_checkpoint_corrupt");
   });
 
+  it("still asserts a turn-timeout rejection after a delay longer than the turn timeout", async () => {
+    const state = providerState();
+    const service = new CapabilityLiveSessionService({ transportFactory: fakeTransportFactory(state) });
+    const session = await service.create({ turnTimeoutMs: 50 });
+    state.holdAfterTool = true;
+    const heldTurn = captureTurnRejection(session.sendMessage("Apply idempotent progress once."));
+    // This delay outlasts turnTimeoutMs, so the rejection settles before this
+    // test attaches its own assertion. Vitest fails a file on an unhandled
+    // rejection, so this test would fail the file on its own without a
+    // handler already attached at the moment the turn promise was created.
+    await new Promise((resolve) => setTimeout(resolve, 300));
+    await expect(heldTurn).resolves.toMatchObject({ message: expect.stringContaining("timed out") });
+  });
+
   it.skipIf(process.platform === "win32" || !existsSync(defaultCapabilityRunnerdBinary()))(
     "terminates real runnerd after a durable receipt and resumes its exact provider thread",
     async () => {
@@ -1063,7 +1115,7 @@ describe("Capability live runnerd and Codex session", () => {
         attemptId: "attempt-real-killed",
         turnTimeoutMs: 2_000,
       });
-      const killedTurn = first.sendMessage("Apply the governed idempotent effect.");
+      const killedTurn = captureTurnRejection(first.sendMessage("Apply the governed idempotent effect."));
       await vi.waitFor(async () => {
         const checkpoint = await store.load(binding.sessionId);
         expect(checkpoint?.mockState).toContain("One durable governed effect.");
@@ -1084,7 +1136,7 @@ describe("Capability live runnerd and Codex session", () => {
       const runnerPid = killedCheckpoint?.process?.runnerPid;
       expect(runnerPid).toBeTypeOf("number");
       process.kill(runnerPid!, "SIGKILL");
-      await expect(killedTurn).rejects.toThrow();
+      await expect(killedTurn).resolves.toBeInstanceOf(Error);
 
       const resumedService = new CapabilityLiveSessionService({
         store: new DurableCapabilityLiveSessionStore({ directory, binding }),
@@ -1094,6 +1146,9 @@ describe("Capability live runnerd and Codex session", () => {
         sessionId: binding.sessionId,
         attemptId: "attempt-real-resumed",
         resumeOf: "attempt-real-killed",
+        // Smaller than this test's own timeout, so a stalled turn reports
+        // which turn stalled instead of surfacing only as a bare test timeout.
+        turnTimeoutMs: 10_000,
       });
       expect(resumed.snapshot().providerThreadId).toBe("thread-durable-runnerd");
       const reconciled = await resumed.reconcileActiveTurn();

packages/paperclip-runner/src/live/live-session.ts

@@ -316,6 +316,13 @@ export interface ResumeCapabilityLiveSessionInput {
   sessionId: string;
   attemptId: string;
   resumeOf: string;
+  // Sets the turn timeout of the session from this resume onward. The
+  // value persists into the saved checkpoint, so a later resume() call
+  // that omits this option keeps the value set here.
+  // A caller with a bounded recovery window should pass a value smaller
+  // than that window, so a stall inside the resumed turn reports which
+  // turn stalled instead of surfacing only as the caller's own timeout.
+  turnTimeoutMs?: number;
 }
 
 export interface CapabilityLiveTurnResult {
@@ -932,6 +939,9 @@ export class CapabilityLiveSessionService {
       status: "starting",
       attempts,
       currentAttemptId: input.attemptId,
+      ...(input.turnTimeoutMs === undefined ? {} : {
+        config: { ...snapshot.config, turnTimeoutMs: input.turnTimeoutMs },
+      }),
     };
     await this.#store.save(prepared);
     try {

packages/paperclip-runner/test/capture-turn-rejection.ts

@@ -0,0 +1,11 @@
+// A turn-timeout timer can reject its promise before a test reaches its own
+// assertion. Vitest treats a rejection with no attached handler as an
+// unhandled rejection and fails the whole file, even if every assertion
+// later passes. Call this at the same moment the promise is created, before
+// any `await`, so a handler is always in place before the timer can fire.
+export function captureTurnRejection<T>(promise: Promise<T>): Promise<unknown> {
+  return promise.then(
+    () => null,
+    (error: unknown) => error,
+  );
+}