fix(native): roll back an undelivered warm attach rotation once its runner is proven dead (#20146)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A follow-up turn can attach to the previous turn's warm local
runner. The transport first rotates authority: it archives the previous
epoch (suspended runner state and settled controller) under
`authority-epochs/` and writes a new control plane that holds only the
`run.attach` command
> - A hard kill in that window, before the runner connects, leaves the
root with the new control plane and no `runner/runner-state.json`.
Restart recovery proves the warm runner dead and classifies the run
`resume_dead_runner`
> - The executor saw a root with the run's own identity and no runner
state, called its authority indeterminate, and quarantined it. The run
failed with `runner_state_identity_mismatch: authority_indeterminate`
and needed a Retry
> - The attach was never delivered, and the runner that would have
received it is proven dead, so it cannot have taken effect. The root's
real authority is still the archived previous epoch
> - This pull request rolls back such an undelivered rotation, so the
follow-up continues from the previous turn
> - The benefit is that a restart in the first second of a warm
follow-up no longer needs a Retry

## Linked Issues or Issue Description

Refs #20140

**What happened?**
Local restart-chaos K13 regression storm w-20261108T235957-s9401 (master
4420a5adcf), task t22 (codex C1):
1. Follow-up run `7b1b1650` started at 00:05:56 as a warm attach to the
previous run's runner (pid 11436).
2. The transport archived the previous epoch (`313ee831`: runner
`suspended`, every command completed) to `authority-epochs/epoch-2c2a…`.
It wrote a new control plane with `run.attach` pending,
`connectionCount` 0, no events, no delivery counts and
`bootstrapHistory.adopted` false.
3. A K1 at 00:05:57 killed the server before the runner connected.
Recovery stopped pid 11436 and claimed `resume_dead_runner`.
4. The executor found `runner/` non-empty (provider state) but without
`runner-state.json`, returned `authority_indeterminate`, and quarantined
the root. A Retry ran fresh.

**Expected behavior**
When a `resume_dead_runner` claim proves the local runner gone and the
run's own root holds only an undelivered `run.attach` over an archived
suspended epoch, recovery restores that epoch and continues. If delivery
is uncertain (any connection, event, delivery count or other command) or
there is no dead-runner claim, it still fails closed.

**Steps to reproduce**
1. Start a warm follow-up and hard-kill the control plane after the
authority rotation, before the runner connects.
2. Restart. Before this change, the follow-up fails with
`runner_state_identity_mismatch: authority_indeterminate`.

**Paperclip version or commit**
4420a5adcf (wave). Branch rebased on current master.

## What Changed

- `native-session-executor.ts`: the new
`rollBackUndeliveredAttachRotation` runs only for a `resume_dead_runner`
claim on this run with a local runner. It acts only when all of these
hold:
  - the active control plane is this run's own exact identity;
  - its only command is a pending `run.attach`;
- `connectionCount`, `ackedSourceSeq` and the delivery counts are 0,
there are no committed events, and it was never adopted and issued no
credentials;
  - it has no warm, staged or completed transition;
  - there is no `runner/runner-state.json`;
- the latest archived epoch is named by its identity's digest, shares
the runner instance, lease and session, and holds a `suspended` runner
state for that identity.

It moves the undelivered control plane to `continuity-breaks/` as
evidence, then puts the epoch's control plane and runner state back. The
executor then sees the previous owner and takes the normal
suspended-previous-owner path.
- `native-recovered-owner-followup.test.ts`: a red/green test of the
undelivered rotation. A possibly delivered attach (`connectionCount` 1)
and a missing dead-runner claim both still fail closed.

## Verification

- Red/green: on master, the undelivered case fails with exactly the t22
error. With the change, the follow-up is admitted: the previous control
plane is restored, `authority-epochs` is gone, and the undelivered
control plane is kept under `continuity-breaks`. The two fail-closed
cases fail on both.
- `vitest run` of `native-recovered-owner-followup.test.ts`,
`native-session-executor.test.ts`,
`native-runner-restart-recovery.integration.test.ts` and
`native-session-root-entries.test.ts`: 4 files, 1508 tests passed (14
skipped); after rebasing onto #20140,
`native-recovered-owner-followup.test.ts` again: 65 passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed.

## Risks

- Medium-low risk. The rollback changes durable state, but only for a
closed shape that proves the attach never reached any runner, after
recovery proved the runner process dead.
- A crash in the middle of the rollback leaves a root that still fails
closed, as before this change.
- The test exercises admission up to tool authority, not a full
real-runnerd turn. A chaos rerun of the K1 warm-follow-up window is the
remaining check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/native-runtime/native-recovered-owner-followup.test.ts

@@ -1,5 +1,5 @@
 import { spawn } from "node:child_process";
-import { randomUUID } from "node:crypto";
+import { createHash, randomUUID } from "node:crypto";
 import { existsSync } from "node:fs";
 import { cp, mkdir, mkdtemp, readFile, readdir, rename, rm, writeFile } from "node:fs/promises";
 import { tmpdir } from "node:os";
@@ -193,8 +193,9 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
         sandboxLeaseAcquisition: { outcome, providerLeaseId } } as never,
     }).then(() => "constructed", (error: Error) => error.message);
     // A local Runner keeps its controller root on this host between turns.
-    const startLocal = () => createRunnerdBackend({
+    const startLocal = (restartRecovery?: Parameters<typeof createRunnerdBackend>[0]["restartRecovery"]) => createRunnerdBackend({
       db, execution, runnerInstanceId: next.runnerInstanceId, onLog: async (_stream, chunk) => { logs.push(chunk); },
+      ...(restartRecovery ? { restartRecovery } : {}),
     }).then(() => "constructed", (error: Error) => error.message);
     return { runId, logs, start, startLocal, native: next, environmentId, leaseId, providerLeaseId, remoteExecute, selectedRunId: selected?.id ?? null };
   }
@@ -829,6 +830,45 @@ describe.skipIf(!recoveryPostgresSupport.supported)("follow-up after a recovered
         await rm(root, { recursive: true, force: true });
       }
     }, 60_000);
+  it.each([
+    { label: "never delivered to a dead runner", connectionCount: 0, claim: true, admitted: true },
+    { label: "possibly delivered", connectionCount: 1, claim: true, admitted: false },
+    { label: "without a dead-runner claim", connectionCount: 0, claim: false, admitted: false },
+  ])("resumes from the previous turn when a warm attach rotation was $label", async ({ connectionCount, claim, admitted }) => {
+    const prior = await seedRecoveredPriorRun();
+    const followUp = await admitFollowUp(prior);
+    const root = await restoreHeldRoot(prior, followUp);
+    const priorIdentity = JSON.parse(await readFile(join(root, "control-plane", "control-plane-state.json"), "utf8")).identity;
+    const epoch = join(root, "authority-epochs",
+      `epoch-${createHash("sha256").update(JSON.stringify(priorIdentity)).digest("hex").slice(0, 24)}`);
+    await mkdir(epoch, { recursive: true, mode: 0o700 });
+    await rename(join(root, "control-plane"), join(epoch, "control-plane"));
+    await writeFile(join(epoch, "runner-state.json"), JSON.stringify({ schema: "paperclip.runner.durable.state.v1", ...priorIdentity,
+      lifecycle: "suspended", outbox: [], pendingTerminalDelivery: null, pendingProviderCleanup: null }));
+    await mkdir(join(root, "runner"), { recursive: true, mode: 0o700 });
+    await writeFile(join(root, "runner", "codex-provider-state.json"), "{}");
+    const identity = { ...priorIdentity, runId: followUp.runId, turnId: `turn-${followUp.runId}`, itemId: `item-${followUp.runId}` };
+    await mkdir(join(root, "control-plane"), { mode: 0o700 });
+    const undelivered = JSON.stringify({ schema: "paperclip.runner.durable.control-plane-state.v2", identity,
+      bootstrapHistory: { schema: "paperclip.runner.bootstrap-history.v1", originIdentity: identity, credentialsIssued: 0, adopted: false },
+      runAttachTemplate: {}, tickets: {}, leases: {},
+      commands: [{ commandId: "attach", controllerSeq: 1, type: "run.attach", status: "pending", payload: {} }],
+      committedEvents: [], ackedSourceSeq: 0, connectionCount, commandDeliveryCounts: {} });
+    await writeFile(join(root, "control-plane", "control-plane-state.json"), undelivered);
+    const outcome = await followUp.startLocal(claim ? { kind: "resume_dead_runner", runId: followUp.runId, leaseOwner: "successor",
+      controllerGeneration: 2, providerAttempt: 1, restartKind: "hard", recoveryRequestId: null } as never : undefined);
+    if (!admitted) {
+      expect(outcome).toBe("runner_state_identity_mismatch: authority_indeterminate");
+      return;
+    }
+    expect(outcome).toBe(ADMITTED);
+    expect(JSON.parse(await readFile(join(root, "control-plane", "control-plane-state.json"), "utf8")).identity).toEqual(priorIdentity);
+    expect(existsSync(join(root, "authority-epochs"))).toBe(false);
+    const [archive] = await readdir(join(root, "continuity-breaks"));
+    expect(await readFile(join(root, "continuity-breaks", archive!, "control-plane", "control-plane-state.json"), "utf8")).toBe(undelivered);
+    expect(await quarantined()).toEqual([]);
+    expect(followUp.logs.join("")).toContain("stopped before its runner received the session");
+  }, 60_000);
 
   it("retires a finished local owner whose Runner stopped before it proved suspension", async () => {
     const { prior, followUp, root } = await seedUnsuspendedLocalOwner(await stoppedRunnerProcess());

server/src/services/native-runtime/native-session-executor.ts

@@ -5655,6 +5655,63 @@ async function retireFinishedPerTurnRunnerdOwner(input: Parameters<typeof inspec
   return { retired: true };
 }
 
+function undeliveredAttachRotationSource(root: string, execution: NativeExecutionInput): string | null {
+  try {
+    const control = readRunnerdControlPlaneState(root);
+    const identity = record(control.identity);
+    const commands = Array.isArray(control.commands) ? control.commands.map(record) : [];
+    const history = record(control.bootstrapHistory);
+    if (!RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(control.schema)) ||
+        !durableIdentityMatchesExecution(identity, execution) ||
+        control.connectionCount !== 0 || control.ackedSourceSeq !== 0 ||
+        !Array.isArray(control.committedEvents) || control.committedEvents.length !== 0 ||
+        Object.keys(record(control.commandDeliveryCounts)).length !== 0 ||
+        control.warmTransition != null || control.completedWarmTransition != null || control.stagedRunAttachment != null ||
+        history.adopted === true || history.credentialsIssued !== 0 ||
+        commands.length !== 1 || commands[0]!.type !== "run.attach" || commands[0]!.status !== "pending" ||
+        existsSync(resolve(root, "runner", "runner-state.json"))) return null;
+    const archives = resolve(root, "authority-epochs");
+    if (!isSafeNativeStateDirectory(archives)) return null;
+    const candidates = readdirSync(archives).map(name => resolve(archives, name)).filter(epoch => {
+      const state = resolve(epoch, "control-plane", "control-plane-state.json");
+      return isSafeNativeStateDirectory(epoch) && existsSync(state) && existsSync(resolve(epoch, "runner-state.json"));
+    }).sort((left, right) => lstatSync(resolve(right, "control-plane", "control-plane-state.json")).mtimeMs
+      - lstatSync(resolve(left, "control-plane", "control-plane-state.json")).mtimeMs);
+    const latest = candidates[0];
+    if (!latest) return null;
+    const prior = record(readRunnerdControlPlaneState(latest).identity);
+    const runner = record(JSON.parse(readBoundedNativeFile(resolve(latest, "runner-state.json"),
+      NATIVE_RUNNER_STATE_MAX_BYTES, "runner_state_too_large").toString("utf8")));
+    const digest = createHash("sha256").update(JSON.stringify(prior)).digest("hex").slice(0, 24);
+    if (basename(latest) !== `epoch-${digest}` || prior.runId === identity.runId ||
+        ["runnerInstanceId", "environmentLeaseId", "normalizedSessionId"].some(key => prior[key] !== identity[key]) ||
+        runner.schema !== RUNNERD_STATE_SCHEMA || runner.lifecycle !== "suspended" ||
+        ["runnerInstanceId", "environmentLeaseId", "runId", "normalizedSessionId"].some(key => runner[key] !== prior[key])) return null;
+    return latest;
+  } catch {
+    return null;
+  }
+}
+
+async function rollBackUndeliveredAttachRotation(root: string, execution: NativeExecutionInput,
+  onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>): Promise<void> {
+  if (!existsSync(root) || !isSafeNativeStateDirectory(root)) return;
+  const epoch = undeliveredAttachRotationSource(root, execution);
+  if (!epoch) return;
+  const archivedAt = Date.now();
+  const breakRoot = resolve(root, "continuity-breaks", `${archivedAt}-${randomUUID()}`);
+  mkdirSync(breakRoot, { recursive: true, mode: 0o700 });
+  noteNativeRunnerEvidence(archivedAt);
+  renameSync(resolve(root, "control-plane"), resolve(breakRoot, "control-plane"));
+  mkdirSync(resolve(root, "runner"), { recursive: true, mode: 0o700 });
+  renameSync(resolve(epoch, "runner-state.json"), resolve(root, "runner", "runner-state.json"));
+  renameSync(resolve(epoch, "control-plane"), resolve(root, "control-plane"));
+  rmdirSync(epoch);
+  if (readdirSync(resolve(root, "authority-epochs")).length === 0) rmdirSync(resolve(root, "authority-epochs"));
+  await onLog?.("stdout",
+    `[paperclip-runner] Run ${execution.binding.runId} stopped before its runner received the session; resuming from the previous turn.\n`);
+}
+
 async function migrateRunnerdStateRootForExecution(input: {
   db: Db;
   execution: NativeExecutionInput;
@@ -5710,6 +5767,10 @@ async function migrateRunnerdStateRootForExecution(input: {
       return verified;
     }
     await assertCleanupActivationCommitted(input.db, scoped, input.execution);
+    if (input.restartRecovery?.kind === "resume_dead_runner" && input.restartRecovery.runId === input.execution.binding.runId
+      && (!input.runnerExecutionTarget || input.runnerExecutionTarget.kind === "local")) {
+      await rollBackUndeliveredAttachRotation(scoped, input.execution, input.onLog);
+    }
     const identity = readRunnerdDurableIdentity(scoped);
     if (!identity && input.restartRecovery?.kind === "bootstrap_incomplete" &&
         (!input.runnerExecutionTarget || input.runnerExecutionTarget.kind === "local") &&