fix(native): restart a drained Daytona run whose runner opened its session (#19711)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. A graceful shutdown (every rolling deploy) suspends live native
runs "without cancelling provider work". The next boot then decides how
each run continues.
> - #19583, #19616 and #19684 let restart recovery start a remote
bootstrap again when durable evidence proves that no provider work
exists.
> - A chaos replay found another window. The drain suspended a remote
run just after its sandbox runner opened the provider session, before
the controller sent a turn and before a session snapshot existed.
> - Recovery accepted only an unfinished `session.open` in the
controller checkpoint, so it blocked the task as ambiguous. A Retry
recovered it, but automatic continuation was lost.
> - A related replay showed that the same-run bootstrap reuses the
sandbox. The interrupted runner can still hold the runner ingress port
there (`runner_ingress_bind_conflict`).
> - This pull request proves the "session opened, no turn" case and
stops the interrupted runner before relaunch.
> - The benefit is that a deploy during Daytona startup continues
automatically, without a port conflict.

## Linked Issues or Issue Description

**What happened?**

A chaos replay (ACPX Claude on Daytona) sent SIGTERM while a remote run
was starting. The drain recorded `restart_suspended`. The dying
controller still launched the runner and finished `session.open`. On the
next boot, restart recovery returned
`ambiguous_remote_recovery_evidence` with disposition `blocked`.

The decrypted controller checkpoint of that run showed:
- `run.prepare` completed and `session.open` completed (provider session
opened).
- Committed events `session.started`, `session.capabilities.updated` and
`session.goal.snapshot` (`ackedSourceSeq: 3`).
- No turn command. No run-profile session snapshot and no projected
provider events.

On the next run in that replay, a same-run bootstrap relaunched the
runner in the same sandbox (one lease for all attempts). It failed with
`runner_ingress_bind_conflict: failed to bind listener: Address already
in use`, because the interrupted runner from 51 s earlier was still
listening there. The bounded same-run retry recovered it on its last
attempt.

**Expected behavior**

When the durable controller record proves that no turn was ever sent,
recovery starts the bootstrap again. It first stops the interrupted
runner in the reused sandbox.

**Steps to reproduce**

1. Start a native run on a Daytona environment.
2. Send SIGTERM to the control plane while the runner opens its provider
session.
3. Restart the control plane after the controller lease expires.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.

Root cause: in
`server/src/services/native-runtime/native-restart-recovery.ts`, the
remote checkpoint proof (`remoteBootstrapProvenIncomplete`, #19583)
required an unfinished `session.open` with no committed events, and no
provider events at all. A completed `session.open` with only
session-open events had no rule, so it fell through to
`ambiguous_remote_recovery_evidence`.

Refs #19583, #19616, #19684, #19524

## What Changed

- `native-bootstrap-proof.ts`: new `runnerdStateHasNoProviderTurn`
proof. The state must hold only `run.prepare` and `session.open`
(pending or completed) and no warm, staged or adopted state. Every
committed event must be a session-open event, with contiguous sequence
numbers from 1, so no event is hidden. Only the controller issues turn
commands, and it writes each one to Postgres before sending it.
- `native-runner-checkpoint.ts`:
`nativeRunnerCheckpointRemoteBootstrapProof` replaces
`nativeRunnerCheckpointProvesIncompleteBootstrap`. It also reads a
bounded number of `event:N` parts and returns which proof holds. It
still rebuilds the state in memory.
- `native-restart-recovery.ts`: a remote run can now claim
`bootstrap_incomplete` with reason `remote_session_opened_without_turn`.
Projected provider events may only be session-open events
(`harness.ready`, `session.started`, `session.resumed`,
`session.updated`), and only together with the stronger session-opened
proof. A turn or provider event still blocks.
- `native-session-executor.ts`:
- For a remote sandbox target, the executor accepts the same proof on
the restored record.
- Before the same-run bootstrap relaunches,
`stopStaleRemoteRunnerBeforeBootstrap` stops the interrupted runner in
the reused sandbox. It uses the verified identity marker and process
check from reattachment, which signals only a process whose identity
matches this run: SIGTERM, a wait of up to 10 s, then SIGKILL.
- The stop is best effort. If no prior runner can be verified, nothing
is stopped, and the bounded same-run retry still handles any conflict.
- `doc/run-log-events.md`: document the reason and the stop.

## Verification

- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "drained remote"`. Before the fix, the opened-session case (with a
projected `session.started`) returned `blocked` /
`ambiguous_remote_recovery_evidence`.
- These keep the hold: a `turn.start` command, a committed
`turn.started`, an acknowledged event that is missing from the record,
and a projected `turn.started`. The executor test restores the
checkpoint and checks that the local proof is false and the remote proof
is true.
- Regression: `native-runner-restart-recovery.integration`,
`native-session-executor`, `native-restart-recovery`,
`native-bootstrap-retry.integration` and `heartbeat/recovery`: 1460
passed, 14 skipped.
- Stale runner stop: controlled sandbox-runner tests
(`native-session-executor.test.ts`, "stale runner stop before a same-run
remote bootstrap") run the real reattachment identity and process
scripts against a fake sandbox runner:
  - a verified runner that exits on SIGTERM gets SIGTERM only;
  - a runner that ignores SIGTERM gets SIGTERM, then SIGKILL;
  - a runner with another identity gets no signal;
  - no live prior runner means no signal.
- Live Daytona proof (wave `w-20261107T021837-s29`, ACPX Claude,
`--faults K2 --triggers time,result`, 4 tasks, concurrency 1). The image
was built from current master
(`paperclip-daytona-runner@sha256:71f711ab…`), and its content id
matches this branch.
- All 4 tasks crossed a graceful restart, and all passed R1. Two of the
K2s fired on the `result` trigger, during a running remote turn.
- The K2 during a turn reattached to the live remote runner
(`reattach_remote_runner`, generation 2), as designed.
- Task t01 failed R6 with an orphan Daytona sandbox that no lease
references. The graceful restart came during sandbox creation. This is a
separate signature, outside this change.
- The timed faults did not land in the exact "session opened, no turn"
window. The integration tests cover that window with the evidence from
the saved replay database.

## Risks

- The proof relies on two facts: only the controller issues turn
commands, and the controller writes each command before sending it
(`#sendJson` waits for the checkpoint flush). A turn always leaves a
`turn.start` command in the record.
- The provider session opened in the old sandbox runner is not resumed.
The new bootstrap opens a fresh session. No turn ran in the old one, so
no work is lost.
- The runner stop sends signals inside the run's own sandbox. It signals
only after the existing reattachment identity and process checks pass.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/native-runtime/native-session-executor.ts

@@ -90,7 +90,7 @@ import { createNativeProviderFailureObservation } from "./native-provider-failur
 import { nativeStartupRetryExhaustedAction, nativeStartupRetryWait, planNativeStartupRetry } from "./native-startup-retry.js";
 import { admitUnclaimedNativeBootstrapRetry } from "./native-bootstrap-retry.js";
 import { RUNNERD_CONTROL_PLANE_STATE_SCHEMAS, runnerdStateIsIncompleteBootstrap,
-  runnerdStateIsOpenedSessionBeforeTurn, runnerdStateIsUnconnectedBootstrap } from "./native-bootstrap-proof.js";
+  runnerdStateHasNoProviderTurn, runnerdStateIsOpenedSessionBeforeTurn, runnerdStateIsUnconnectedBootstrap } from "./native-bootstrap-proof.js";
 import { createNativeDeliverableTextObservation } from "./native-deliverable-text.js";
 import { createNativeGitHubAccess, type NativeGitHubAccess } from "./native-github-access.js";
 import { resolveGitHubOperationCredentials } from "../github-operation-credentials.js";
@@ -5659,7 +5659,14 @@ async function migrateRunnerdStateRootForExecution(input: {
       // A local runner's process ids belong to this host, so a fresh provider
       // session it opened before any turn can be proven gone here.
       const localProcesses = !input.runnerExecutionTarget || input.runnerExecutionTarget.kind === "local";
-      if (runnerdStateProvesIncompleteBootstrap(scoped, { localProcesses })) {
+      const target = input.runnerExecutionTarget;
+      const remoteSandbox = target?.kind === "remote" && target.transport === "sandbox" && !target.runnerTask;
+      // A remote controller record restored from Postgres is complete once its
+      // lease expired. A runner that opened its session but got no turn there
+      // holds no provider work (RC-29d).
+      if (runnerdStateProvesIncompleteBootstrap(scoped, { localProcesses })
+        || (remoteSandbox && runnerdStateProvesNoRemoteProviderTurn(scoped))) {
+        if (remoteSandbox) await stopStaleRemoteRunnerBeforeBootstrap({ ...input, identity });
         quarantineRunnerdStateRoot(scoped, "identity_indeterminate");
         return;
       }
@@ -6178,6 +6185,74 @@ export function runnerdStateProvesIncompleteBootstrap(
   }
 }
 
+const STALE_REMOTE_RUNNER_STOP_WAIT_MS = 10_000;
+
+/**
+ * The same-run bootstrap reuses the run's sandbox. A runner from the
+ * interrupted attempt can still be alive there (its reconnect grace is
+ * minutes) and still hold the runner ingress port, so the new runner would fail
+ * with runner_ingress_bind_conflict. Stop that exact runner first, using the
+ * same verified identity marker and process check as reattachment, which only
+ * signals a process whose identity matches this run. Best effort: if no prior
+ * runner can be verified there is nothing to stop, and a remaining conflict is
+ * still caught by the bounded same-run bootstrap retry.
+ */
+export async function stopStaleRemoteRunnerBeforeBootstrap(input: {
+  execution: NativeExecutionInput;
+  runnerExecutionTarget?: AdapterExecutionTarget | null;
+  identity: Record<string, unknown>;
+  onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
+}, options: {
+  verify?: typeof verifyRemoteRunnerReattachment;
+  waitMs?: number;
+  pollMs?: number;
+} = {}): Promise<"no_prior_runner" | "stopped" | "killed" | "stop_failed"> {
+  const target = input.runnerExecutionTarget;
+  if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner) return "no_prior_runner";
+  const providerLeaseId = target.sandboxLeaseAcquisition?.providerLeaseId;
+  if (!providerLeaseId) return "no_prior_runner";
+  const runId = input.execution.binding.runId;
+  const verify = options.verify ?? verifyRemoteRunnerReattachment;
+  let runner: Awaited<ReturnType<typeof verifyRemoteRunnerReattachment>>;
+  try {
+    runner = await verify({
+      claim: { runId, remote: { providerLeaseId, remoteCwd: target.remoteCwd } },
+      target, identity: input.identity, runId, normalizedSessionId: nativeSessionKey(input.execution),
+    });
+  } catch {
+    // No runner with this run's exact identity and generation is alive here.
+    return "no_prior_runner";
+  }
+  try {
+    let outcome: "stopped" | "killed" = "stopped";
+    if (await runner.signal("SIGTERM")) {
+      const deadline = Date.now() + (options.waitMs ?? STALE_REMOTE_RUNNER_STOP_WAIT_MS);
+      while (Date.now() < deadline && await runner.isAlive()) {
+        await new Promise(done => setTimeout(done, options.pollMs ?? 500));
+      }
+      if (await runner.isAlive()) {
+        await runner.signal("SIGKILL");
+        outcome = "killed";
+      }
+    }
+    await input.onLog?.("stdout", `[paperclip-runner] Stopped the interrupted runner ${runner.pid} before restarting the bootstrap.\n`);
+    return outcome;
+  } catch (error) {
+    await input.onLog?.("stderr", `[paperclip-runner] Could not stop the interrupted runner before restarting the bootstrap: ${
+      error instanceof Error ? error.message : String(error)}\n`);
+    return "stop_failed";
+  }
+}
+
+/** The remote controller record proves its runner was never sent a turn. */
+export function runnerdStateProvesNoRemoteProviderTurn(root: string): boolean {
+  try {
+    return runnerdStateHasNoProviderTurn(readRunnerdControlPlaneState(root));
+  } catch {
+    return false;
+  }
+}
+
 function runnerdStateProvesUnconnectedBootstrap(root: string): boolean {
   try {
     return runnerdStateIsUnconnectedBootstrap(readRunnerdControlPlaneState(root));