test(native): cover a turn.start timeout that loses the remote Runner, end to end (#21041)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the same run
reattaches to that Runner
> - On staging, 31 tasks that a person retried are blocked. Their last
run failed with `runner_remote_process_identity_unavailable` /
`native_continuation_requires_reconciliation`
> - In each run, the `turn.start` PRP command timed out. The close then
killed the Runner. The same-run reattachment found the Runner gone
(`process_not_running`, `providerTurnIssued: false`), and the task
waited for a Retry
> - #21023 and #21039 fix this sequence. Neither one is deployed yet. No
test reproduces this exact sequence, so a later change could break it
without a signal
> - This pull request adds two tests. One reproduces the turn start
timeout with a real `paperclip-runnerd`. The other reproduces the
staging failure record of a person's Retry
> - The benefit is that a regression of this failure class fails CI
before it reaches staging

## Linked Issues or Issue Description

Refs #21023, #21039, #20923, #20853

**What happened?**
A person retried a blocked task that runs Codex on Daytona. The Retry
started a fresh provider session and submitted a turn. The `turn.start`
PRP command timed out (`provider_transport_failed: PRP command
turn.start timed out`). The run closed. The run log shows `runner did
not prove durable suspension before checkpoint`, `runner shutdown
failed: Error: Durable recovery runner timed out.`, and then
`runner_remote_process_exited lifecycle=ready`. The log does not show
`remote runner left alive unsettled for same-run reattachment`. The
same-run retry tried to reattach 50 s later. It recorded
`native.remote_runner_recovery_failed` with `stage:
process_verification`, `reason: process_not_running`, `runnerSettled:
false`, `providerTurnIssued: false`, and an empty Runner stderr. The run
failed with `runner_remote_process_identity_unavailable`, and the task
waited for a person.

**Expected behavior**
A turn start timeout does not kill the Runner, so the same-run
reattachment adopts it. If the Runner is gone anyway, the run continues
once by itself in a fresh provider session and does not replay the turn.

**Steps to reproduce**
1. Start a remote native Codex run on a release before #21023.
2. Make the provider not answer `turn/start`, so the `turn.start` PRP
command times out.
3. The runtime cancels the session as failure cleanup. The close treats
that as a Stop and kills `paperclip-runnerd`.
4. The same-run reattachment finds `process_not_running`. Before #21039,
the completion hook refuses an automatic Retry with
`provider_turn_unsettled`.

**Paperclip version or commit**
Staging runs 8cefbe9977. That build has neither #21023 nor #21039. This
pull request is based on master after both.

**Deployment mode**
Hosted staging cloud workspace. Native runtime with a remote Daytona
sandbox.

**Agent adapter(s) involved**
Codex (native `codex_app_server` driver through `paperclip-runnerd`).

## What Changed

-
`packages/paperclip-runner/src/live/runnerd-turn-start-timeout-retention.test.ts`
(new): runs `executeNativeSession` over a real `paperclip-runnerd` with
a launcher-owned (remote-shaped) process. The fake Codex never answers
`turn/start`, so the `turn.start` PRP command times out. The test checks
that the run fails with that exact error, that the Runner process is
still alive after the close, and that its journal is `ready` for the
same-run reattachment.
- `server/src/services/runner-lost-mid-turn-continuation.test.ts`: a new
case with the staging failure record. A person's Retry (`task_retry`, a
user intent with a non-hook key, `forceFreshSession`) has only
`turn.submitted`. Its refusal has `runnerSettled: false` and
`providerTurnIssued: false`. The completion hook requests one automatic
Retry. The test checks one unknown-outcome event with `replayed: false`,
an idempotent second call, saved-work recovery first, and one queued
fresh-session successor. The fixture gets small optional inputs for the
wake, the events, and the refusal payload. Other cases are unchanged.
- No production code changes.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-turn-start-timeout-retention.test.ts
src/live/runnerd-codex-transport-remote-retention.test.ts`: 6 of 6 pass.
The new test passed 4 runs in a row.
- Red without #21023: with `native-session-runtime.ts`,
`harness-driver-backend.ts`, and `native-session-backend.ts` from before
#21023, the new runner test fails with `expected 'gone' to be 'alive'`.
That is the staging failure.
- `cd server && npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts`: 9 of 9 pass.
- Red without #21039: with `restart-interrupted-turn.ts` from before
#21039, the new server case fails with `{ requested: false, reason:
"provider_turn_unsettled" }`.
- `npx tsc --noEmit` passes for `server` and
`packages/paperclip-runner`.
- `pnpm verify` exits 2 because the new runner test file is outside the
resolved test graphs. CI supplies that coverage.
- Staging evidence: read-only GET of the failed run's events and log.
- Not run: a live Daytona run. #21023 and #21039 are not deployed yet.

## Risks

- Low. Tests only. No production code changes.
- The runner test starts a real `paperclip-runnerd` and the fake Codex.
It kills both processes and removes its temporary directory in
`finally`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/runner-lost-mid-turn-continuation.test.ts

@@ -31,12 +31,18 @@ const stagingRefusal = (reason: "process_not_running" | "process_identity_change
 describe("a remote Runner that died in the middle of a provider turn", () => {
   useWorkspaceRecoveryFixture(targets);
 
-  async function runnerLostMidTurn(input: { wake: "user" | "system_retry" | "system_monitor";
-    refusal?: "process_not_running" | "process_identity_changed" | null }) {
+  async function runnerLostMidTurn(input: { wake: "user" | "user_retry" | "system_retry" | "system_monitor";
+    refusal?: "process_not_running" | "process_identity_changed" | null;
+    events?: string[]; refusalPayload?: Record<string, unknown> }) {
     const f = await seed("native");
     const original = await run(f.runId);
-    const system = input.wake !== "user";
-    const reason = input.wake === "user" ? "issue_assigned" : input.wake === "system_retry" ? "transient_failure_retry" : "issue_monitor_due";
+    const system = input.wake !== "user" && input.wake !== "user_retry";
+    const reason = input.wake === "user" ? "issue_assigned" : input.wake === "user_retry" ? "task_retry"
+      : input.wake === "system_retry" ? "transient_failure_retry" : "issue_monitor_due";
+    const [retryIntent] = input.wake === "user_retry" ? await db.insert(agentWakeupRequests).values({ companyId: f.companyId,
+      agentId: original.agentId, source: "on_demand", triggerDetail: "manual", reason: "task_retry", status: "completed",
+      idempotencyKey: `task-retry:${randomUUID()}`, requestedByActorType: "user", requestedByActorId: "local-board",
+      payload: { issueId: f.issueId } }).returning() : [];
     const [predecessor] = input.wake === "system_retry" ? await db.insert(heartbeatRuns).values({ companyId: f.companyId,
       agentId: original.agentId, issueId: f.issueId, status: "failed", createdAt: new Date(Date.now() - 60 * 60_000),
       finishedAt: new Date(Date.now() - 50 * 60_000),
@@ -48,23 +54,24 @@ describe("a remote Runner that died in the middle of a provider turn", () => {
     await db.update(heartbeatRuns).set({ errorCode: "native_continuation_requires_reconciliation",
       error: "runner_remote_process_identity_unavailable", wakeupRequestId: wake.id, responsibleUserId: "local-board",
       retryOfRunId: predecessor?.id ?? null, startedAt: new Date(Date.now() - 45 * 60_000),
-      contextSnapshot: { issueId: f.issueId, wakeReason: reason, ...(predecessor ? { retryOfRunId: predecessor.id,
+      contextSnapshot: { issueId: f.issueId, wakeReason: reason, ...(retryIntent ? { taskRetryRequestId: retryIntent.id,
+        forceFreshSession: true } : {}), ...(predecessor ? { retryOfRunId: predecessor.id,
         retryReason: "transient_failure", scheduledRetryAttempt: 1 } : {}) },
     }).where(eq(heartbeatRuns.id, f.runId));
-    for (const eventType of ["turn.submitted", "turn.started", "tool.execution.started", "turn.accepted", "tool.execution.completed"])
+    for (const eventType of input.events ?? ["turn.submitted", "turn.started", "tool.execution.started", "turn.accepted", "tool.execution.completed"])
       await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId, agentId: original.agentId,
         eventType, stream: "system", level: "info", message: eventType,
         payload: { prpEvent: { eventType, turnId: TURN_ID, payload: {} } } });
     if (input.refusal !== null) await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId,
       agentId: original.agentId, eventType: "native.remote_runner_recovery_failed", stream: "system", level: "error",
-      message: "Could not verify the existing remote runner.", payload: stagingRefusal(input.refusal) });
+      message: "Could not verify the existing remote runner.", payload: input.refusalPayload ?? stagingRefusal(input.refusal) });
     const claim = { kind: "reattach_remote_runner" as const, runId: f.runId, leaseOwner: `boot:2:${randomUUID()}`,
       controllerGeneration: 2, providerAttempt: 1, restartKind: "hard" as const, recoveryRequestId: null,
       remote: { providerLeaseId: f.providerLeaseId, remoteCwd: f.remote } };
     const service = heartbeatService(db, { environmentRuntime: f.environmentRuntime });
     // Keep the agent's only slot busy so a successor stays observable as queued.
     await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: original.agentId, status: "running" });
-    return { ...f, agentId: original.agentId, claim, service };
+    return { ...f, agentId: original.agentId, claim, service, retryIntent };
   }
   const queued = async (companyId: string) => db.select().from(heartbeatRuns)
     .where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.status, "queued")));
@@ -101,6 +108,28 @@ describe("a remote Runner that died in the middle of a provider turn", () => {
       idempotencyKey: `task-retry:restart-interrupted-turn:${f.runId}`, requestedByActorType: "user", requestedByActorId: "local-board" })]);
   }, 30_000);
 
+  it("continues a person's Retry whose turn start timed out and whose Runner was gone at the same-run reattachment", async () => {
+    const f = await runnerLostMidTurn({ wake: "user_retry", events: ["turn.submitted"], refusalPayload: {
+      remoteRunnerRecovery: { stage: "process_verification", reason: "process_not_running", runnerSettled: false,
+        runnerLifecycle: "ready", runnerDiagnostics: ["WebSocket peer closed the connection"], providerTurnIssued: false,
+        runnerExitEvidence: "== runnerd.process-stderr.log" } } });
+    expect(await requestInterruptedNativeTurnRetry(db, f.runId)).toEqual({ requested: true, requestId: expect.any(String) });
+    expect(await requestInterruptedNativeTurnRetry(db, f.runId)).toMatchObject({ requested: true });
+    expect(await unknownOutcomeEvents(f.runId)).toEqual([expect.objectContaining({ payload: expect.objectContaining({
+      lastTurnEvent: "turn.submitted", replayed: false }) })]);
+    await vi.waitFor(async () => {
+      await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, current => f.service.resumeRemoteStopComments(current)) ?? []);
+      expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
+    }, { timeout: 10_000, interval: 10 });
+    await vi.waitFor(async () => expect(await queued(f.companyId)).toHaveLength(1), { timeout: 10_000, interval: 50 });
+    const [successor] = await queued(f.companyId);
+    expect(successor.contextSnapshot).toMatchObject({ previousRunId: f.runId, wakeReason: "task_retry", forceFreshSession: true });
+    const retries = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),
+      eq(agentWakeupRequests.reason, "task_retry")));
+    expect(retries.filter(retry => retry.id !== f.retryIntent!.id && retry.idempotencyKey?.startsWith("task-retry:"))).toEqual([
+      expect.objectContaining({ idempotencyKey: `task-retry:restart-interrupted-turn:${f.runId}`, requestedByActorId: "local-board" })]);
+  }, 30_000);
+
   it.each([
     { title: "inherits the hook's Retry intent", inherit: true },
     { title: "reaches the hook's Retry run through its retry chain", inherit: false },