fix(native): take over a remote runner at once when its old controller is proven gone (#20240)

## Thinking Path

- Restart recovery should be fast and fault-proof; "restarts are product
failures". Remote (Daytona) runs must recover as quickly as local ones
once it is safe.
- Phase 3's Daytona K13 storm (`w-20261109T100435-s9503`) showed Codex
`restart_recovery_control_plane` p50 34.9 s, p95 61.1 s. That's
Paperclip's own share, before any provider load.
- The run logs show why. After a hard restart, each slow run sat in
"Native restart recovery is waiting for safe ownership evidence" with
reason `remote_controller_lease_live` for about 51 s, until the dead
controller's 60 s remote-controller lease expired.
- On the remote path, `evaluateNativeControllerTakeover` deliberately
skips every process proof and waits for the lease. That's correct when
the previous controller might live in another container or on another
host. When it recorded this exact process space, the shared prover (pid,
process start, boot id) can prove it gone, which is the same proof the
local path already trusts.

## Linked Issues or Issue Description

**What happened?**
After a hard control-plane restart, a remote (Daytona) native run could
not be recovered until the dead controller's 60 s remote lease expired.
Recovery took about 35–60 s, even though the old controller process was
provably gone.

**Steps to reproduce**
1. Run a Codex task in a Daytona environment.
2. Hard-kill the control plane (SIGKILL) while the run is active, then
restart it.
3. The run logs `native.recovery.transition` "waiting for safe ownership
evidence" (`remote_controller_lease_live`) and resumes only when the
lease expires.

**Expected behavior**
When the previous controller ran in this same process space and is
proven gone, recovery takes over the remote runner immediately. It still
waits for the lease whenever the previous controller cannot be proven
gone.

## What Changed

- `server/src/services/native-runtime/native-restart-recovery.ts`,
`evaluateNativeControllerTakeover` with `remoteLeaseFenced`:
  - An expired or released lease is allowed, as before.
- A live lease whose holder recorded this exact process space
(`controllerHostHash`, neither missing nor unknown) and a controller PID
goes through the shared owner-gone prover, with
`requireStoppedController`. A positive proof allows the takeover, with
reason `remote_lease_<proof reason>`, for example
`remote_lease_controller_process_dead`.
- The one-release null-host fallback is never accepted for a remote
runner.
- A holder in another process space is never probed (the existing "must
not probe another container's PID" test still holds). It waits for the
lease, as before.
- `native-restart-recovery.test.ts`: a new test, red on master, covers:
  - takeover when this-process-space controller is dead;
  - waiting while it is alive;
  - waiting without a probe when the process space differs.

## Verification

- The 7 server test files that cover the takeover decision pass (351
tests): `native-restart-recovery`, `native-controller-host-recovery`,
`native-remote-retry-admission`, `native-recovered-owner-followup`,
`native-runner-restart-diskless.integration`, `heartbeat/recovery` and
`controller-instance-lease`.
- Evidence: kept-instance DB copy from the Daytona K13 wave. In run
`ba194ae7`, the `awaiting_evidence` transition
`remote_controller_lease_live` lasted from 04:26:06.8 to 04:26:57.9,
then `bootstrap_incomplete` restart.
- A Daytona K13 rerun will measure the before and after.

## Risks

- **Takeover soundness** rests on the shared prover, which the local
path already uses for the same controllers: same process space, PID dead
or definitely restarted, and boot-id proof where needed. If the old
controller is alive, the decision still waits for the lease.
- **Multi-host or containerized controllers** record a different process
space, so they are unaffected and still fenced by the lease.
- **The remote runner** keeps its own reconnect grace (7 min). The new
controller reattaches through the existing durable-authority path; only
the wait before that changes.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/native-runtime/native-restart-recovery.test.ts

@@ -70,6 +70,26 @@ describe("native restart recovery classification", () => {
     expect(result.allowed).toBe(!live);
     expect(probe).not.toHaveBeenCalled();
   });
+  it("takes over a live remote lease at once when its controller is proven gone in this process space", async () => {
+    const now = new Date();
+    const startedAt = new Date(now.getTime() - 60_000);
+    const owner = { leaseOwner: "old-controller", leaseExpiresAt: new Date(now.getTime() + 50_000),
+      controllerPid: 424242, controllerProcessStartedAt: startedAt,
+      controllerHostHash: currentProcessSpaceHash(), controllerBootId: currentControllerBootId() };
+    // A hard restart: the previous controller's PID is gone.
+    await expect(evaluateNativeControllerTakeover({ now, remoteLeaseFenced: true, owner, isProcessAlive: () => false }))
+      .resolves.toMatchObject({ allowed: true, reason: "remote_lease_controller_process_dead" });
+    // The previous controller is still running: keep waiting for its lease.
+    await expect(evaluateNativeControllerTakeover({ now, remoteLeaseFenced: true, owner,
+      isProcessAlive: () => true, readProcessStartedAt: async () => startedAt }))
+      .resolves.toEqual({ allowed: false, reason: "remote_controller_lease_live" });
+    // Another process space (a different container or host): never probed, keep waiting.
+    const probe = vi.fn(() => false);
+    await expect(evaluateNativeControllerTakeover({ now, remoteLeaseFenced: true,
+      owner: { ...owner, controllerHostHash: "sha256:another-process-space" }, isProcessAlive: probe }))
+      .resolves.toEqual({ allowed: false, reason: "remote_controller_lease_live" });
+    expect(probe).not.toHaveBeenCalled();
+  });
   it("does not reopen a failed checkpoint or reset an exhausted provider budget", () => {
     const evidence = {
       runnerPidAlive: false,

server/src/services/native-runtime/native-restart-recovery.ts

@@ -419,9 +419,24 @@ export async function evaluateNativeControllerTakeover(input: {
 }): Promise<NativeControllerTakeoverDecision> {
   const { owner, now } = input;
   if (input.remoteLeaseFenced) {
-    return owner.leaseExpiresAt && owner.leaseExpiresAt > now
+    if (!owner.leaseExpiresAt || owner.leaseExpiresAt <= now) {
-      ? { allowed: false, reason: "remote_controller_lease_live" }
+      return { allowed: true, reason: "remote_controller_lease_released_or_expired" };
-      : { allowed: true, reason: "remote_controller_lease_released_or_expired" };
+    }
+    // The lease fences a second live controller from driving the same remote
+    // runner. When the lease holder recorded this exact process space, the
+    // shared prover can show it is gone (a hard restart), so the takeover need
+    // not wait out the lease. A controller in another process space (another
+    // container or host) is never probed here: its lease must expire. The
+    // one-release null-host fallback is never enough for a remote runner.
+    const sameProcessSpace = Object.prototype.hasOwnProperty.call(owner, "controllerHostHash")
+      && typeof owner.controllerHostHash === "string"
+      && owner.controllerHostHash !== UNKNOWN_PROCESS_SPACE
+      && owner.controllerHostHash === currentProcessProbe().processSpace;
+    if (sameProcessSpace && owner.controllerPid) {
+      const local = await evaluateNativeControllerTakeover({ ...input, remoteLeaseFenced: false, requireStoppedController: true });
+      if (local.allowed && !local.legacyHostFallback) return { allowed: true, reason: `remote_lease_${local.reason}` };
+    }
+    return { allowed: false, reason: "remote_controller_lease_live" };
   }
   if ((!owner.leaseOwner || !owner.leaseExpiresAt) && !input.requireStoppedController) {
     return { allowed: true, reason: "unowned" };