fix(runner): keep a connected remote Runner through a stalled plugin supervision probe (#20940)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
monitors the process with a health probe that goes through the
environment plugin (`environmentExecute`).
> - A stall of the Daytona plugin worker makes that probe RPC time out
after 40 seconds. The monitor treated the timeout as "process
supervision failed".
> - The transport then failed closed with
`runner_process_termination_unconfirmed`. The run ended
`notification_transport_failed`, `recoverable: false`, and the close
path suspended a healthy Runner. The task rested blocked with Retry.
> - Nothing was wrong with the sandbox, the Runner or the provider. The
Runner held its own authenticated connection to the controller during
the full stall.
> - This pull request makes a failed or timed-out probe "unknown". While
the Runner stays connected on its own channel, the monitor keeps the
same runner and turn and probes again, for a bounded time.
> - The benefit is that a short plugin stall no longer kills a healthy
run. The fail-closed cases stay the same.

## Linked Issues or Issue Description

Refs #20909 (chaos cell CR3, which reproduces this failure)

**What happened?**
A 45-second stall of the Daytona plugin worker killed a healthy run.
Chaos cell CR3 reproduced it 2 of 2 times (OpenCode and Claude,
Daytona). A staging run had the same failure with no deploy: a plugin
`environmentExecute` RPC stalled for more than 40 seconds. The run
failed with "Notification transport failed closed:
runner_process_termination_unconfirmed: runnerd process supervision
failed: RPC call \"environmentExecute\" timed out after 40000ms". The
close path then sent `runner.suspend`, and the healthy Runner suspended
and exited. The task rested blocked with Retry.

**Expected behavior**
A probe that times out or fails in transport is not proof that the
process stopped. While the Runner stays connected on its own channel,
the run continues. If supervision cannot come back within a bound, the
run rests with a working Retry. A probe that positively reports the
process gone still starts the existing recovery.

**Steps to reproduce**
1. Start a remote (Daytona) run with a T1 workload.
2. Send SIGSTOP to the server's Daytona plugin worker for 45 seconds,
then SIGCONT.
3. The run fails `notification_transport_failed` about 40 seconds into
the stall.

**Paperclip version or commit**
master at 7cd5558e41.

## What Changed

- `createRemoteRunnerProcessLauncher`
(`server/src/services/native-runtime/native-session-executor.ts`): when
the health probe throws (RPC timeout or transport error) and the Runner
holds an authenticated connection, the monitor waits one poll interval
and probes again. It does not reject the process completion. The unknown
period has a bound: the remote runner reconnect grace (7 minutes). When
the Runner is not connected, the monitor keeps the old behavior: it
rejects after the 30-second probe budget, without exit proof.
- Probe results that positively answer (exit 45 "process gone", or
identity and generation failures) keep their exact current handling.
- `RunnerProcessLaunchSpec`
(`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`)
has a new optional `isRunnerConnected` callback. `spawnRunner` passes it
to the process owner and keeps it on restart.
- The runnerd transport and the external runnerd driver supply
`isRunnerConnected` from the control plane's count of authenticated
Runner connections.
- `doc/execution-semantics.md` describes the unknown-probe rule and its
bound.

## Verification

- Red, on master: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "plugin
RPC timeouts until"` (in `server/`). The completion rejected after the
first stalled probe with `{ stage: "process_verification", reason:
"probe_transport_failed" }`. The transport turns that into
`runner_process_termination_unconfirmed`, which is the staging error.
- Red, on master: `npx vitest run
src/live/runnerd-codex-transport.test.ts -t "live evidence of the
runner"` (in `packages/paperclip-runner/`). The launch spec had no
`isRunnerConnected` (`expected undefined to be true`).
- Green: both tests pass. New tests also prove:
- three 40-second plugin RPC timeouts with the Runner connected leave
the same runner supervised, and a later exit-45 probe still confirms
exit;
- with the Runner disconnected, the same timeout rejects without exit
proof (`probe_transport_failed`, `isExitConfirmed() === false`);
- with the Runner connected and probes that never answer, supervision
rejects without exit proof after the 7-minute bound.
- `server/src/services/native-runtime/native-session-executor.test.ts`,
`remote-runner-recovery-probe.test.ts`,
`packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts`,
`src/control-plane/durable-prp-control-plane.test.ts` and
`src/drivers/external` pass locally.
- `tsc --noEmit` passes for `server` and `packages/paperclip-runner`.
- Chaos: CR3 results are in a follow-up comment on this PR.

## Risks

- While the Runner stays connected and the plugin cannot answer, the
controller waits up to 7 minutes before it fails closed. Before this
change it failed after about 40 seconds. During that time the Runner's
own connection is the liveness evidence, and the controller never
reports an exit.
- When the Runner is not connected, the timing is the same as before.
Sandbox loss still fails closed after the 30-second probe budget.
- The new spec field is optional. A process owner that does not read it
keeps the old behavior. State saved by an older version is not changed.
- No wait is added before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/execution-semantics.md

@@ -4814,7 +4814,13 @@ configuration from the same company, task, and bot binding.
 An active remote runner's health probe is an observation, not an exit receipt.
 Timed-out or rejected read-only probes retry within the existing 30-second
 budget. A successful probe keeps the same runner and turn; no provider work is
-replayed. Monitoring checks the saved process generation when available.
+replayed. A probe that times out or fails in transport is unknown, not an exit.
+For example, a stalled environment plugin RPC times out after 40 seconds. While
+the Runner holds its own authenticated connection to the controller, the monitor
+keeps the same runner and turn and probes again. The bound is the remote runner
+reconnect grace (7 minutes). When the Runner has no connection and the 30-second
+budget is spent, or when the bound is spent, the monitor reports
+`runner_process_termination_unconfirmed` and the run rests with Retry. Monitoring checks the saved process generation when available.
 Only a matching ownership marker and confirmed process exit permit automatic
 replacement. Missing or conflicting identity evidence, failed reads, and an
 exhausted probe budget retain the existing ownership and recovery safeguards.

packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts

@@ -646,6 +646,8 @@ export interface RunnerProcessLaunchSpec {
   args: readonly string[];
   cwd: string;
   environment: NodeJS.ProcessEnv;
+  /** True while the launched Runner holds an authenticated connection to its control plane. */
+  isRunnerConnected?: () => boolean;
 }
 
 function domainDigest(domain: string, parts: readonly Buffer[]): Buffer {
@@ -4732,6 +4734,7 @@ export function spawnRunner(options: {
   };
   environment?: NodeJS.ProcessEnv;
   processLauncher?: (spec: RunnerProcessLaunchSpec) => RunnerProcessHandle;
+  isRunnerConnected?: () => boolean;
   diagnosticsDirectory?: string;
 }): RunnerProcessHandle {
   for (const value of [options.maxRuntimeMs, options.maxLifetimeMs]) {
@@ -4881,6 +4884,7 @@ export function spawnRunner(options: {
   if (options.processLauncher !== undefined) {
     return withRestart(
       options.processLauncher({ command, args, cwd: packageRoot, environment,
+        ...(options.isRunnerConnected ? { isRunnerConnected: options.isRunnerConnected } : {}),
         nativeRunner: {
           identity: options.identity, ticket: options.ticket, connection, stateDirectory: options.stateDirectory,
           runnerVersion: options.runnerVersion, runnerDigest: options.runnerDigest,

packages/paperclip-runner/src/drivers/external/runnerd-external-driver.ts

@@ -232,7 +232,7 @@ class RunnerdExternalSession implements HarnessSession {
       const ticket = core.issueBootstrapTicket(60_000);
       await core.flushCheckpoint();
       this.#process = spawnRunner({
-        processLauncher: o.runnerProcessLauncher, connectUrl: registration?.connectUrl ?? (registration?.connection ? undefined : core.connectUrl),
+        processLauncher: o.runnerProcessLauncher, isRunnerConnected: () => core.activeRunnerConnectionCount() > 0, connectUrl: registration?.connectUrl ?? (registration?.connection ? undefined : core.connectUrl),
         connection: registration?.connection, stateDirectory: runnerState, identity: o.identity,
         ticket, runnerBinaryPath: binary,
         runnerVersion: artifact.version, runnerDigest: artifact.digest,

packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts

@@ -372,6 +372,51 @@ it.each(["resolved", "rejected"] as const)(
   15_000,
 );
 
+it("gives the process owner live evidence of the runner's own authenticated connection", async () => {
+  const directory = await mkdtemp(join(tmpdir(), "runnerd-connection-evidence-"));
+  const specs: durableControlPlane.RunnerProcessLaunchSpec[] = [];
+  let child: ReturnType<typeof spawn> | undefined;
+  let exited: Promise<void> | undefined;
+  const bundle = createCapabilityRunnerdCodexTransport({
+    runnerBinary: defaultCapabilityRunnerdBinary(),
+    codexCommand: fakeCodex,
+    codexArgs: fakeCodexArgs(directory),
+    stateDirectory: directory,
+    lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 },
+    runnerReconnectGraceMs: 60_000,
+    runnerProcessLauncher: spec => {
+      specs.push(spec);
+      const ownedChild = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: spec.environment, stdio: "ignore" });
+      child = ownedChild;
+      exited = new Promise(resolveExit => ownedChild.once("exit", () => resolveExit()));
+      return {
+        child: ownedChild,
+        completion: new Promise<durableControlPlane.RunnerProcessResult>(() => {}),
+        isExitConfirmed: () => false,
+      };
+    },
+  });
+  try {
+    await bundle.transport.request("thread/start", { cwd: directory });
+    expect(specs).toHaveLength(1);
+    expect(specs[0]!.isRunnerConnected?.()).toBe(true);
+    child!.kill("SIGKILL");
+    await exited;
+    await vi.waitFor(() => expect(specs[0]!.isRunnerConnected?.()).toBe(false), { timeout: 5_000 });
+  } finally {
+    const providerPid = bundle.evidence().codexPid;
+    if (providerPid) {
+      try { process.kill(providerPid, "SIGKILL"); } catch (error) {
+        if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error;
+      }
+    }
+    await bundle.transport.detachControllerForRestart!();
+    child?.kill("SIGKILL");
+    await exited;
+    await rm(directory, { recursive: true, force: true });
+  }
+}, 15_000);
+
 it.each([
   "transport_reconnect_grace_exceeded",
   "transport reconnect grace exceeded; durable state is preserved",

packages/paperclip-runner/src/live/runnerd-codex-transport.ts

@@ -6258,6 +6258,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
       environment: prepared.environment,
       diagnosticsDirectory: resolve(this.#root, "diagnostics"),
       processLauncher: this.options.runnerProcessLauncher,
+      isRunnerConnected: () => core.activeRunnerConnectionCount() > 0,
     });
     this.#handle = handle;
     this.#watchRunner(handle);
@@ -7132,6 +7133,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
           ),
           diagnosticsDirectory: resolve(this.#root, "diagnostics"),
           processLauncher: this.options.runnerProcessLauncher,
+          isRunnerConnected: () => core.activeRunnerConnectionCount() > 0,
         });
     if (handle) {
       this.#handle = handle;

server/src/services/native-runtime/native-session-executor.test.ts

@@ -83,6 +83,7 @@ import { DatabaseSync } from "node:sqlite";
 import { nativeSha256 } from "./canonical.js";
 import { resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js";
 import { REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT, RemoteRunnerRecoveryError, readRemoteRunnerRecoveryError } from "./remote-runner-recovery-diagnostics.js";
+import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "./native-runner-recovery-contract.js";
 import { agentDirectoryWorkingCopyService } from "../agent-directory-working-copies.js";
 import { probeAgentDirectory } from "../agent-directory-probe.js";
 import { AGENT_FILES_CONTRACT } from "../agent-file-store.js";
@@ -1608,6 +1609,97 @@ fs.readFileSync = function(path, ...args) {
     } finally { vi.useRealTimers(); }
   });
 
+  it("keeps supervising a connected runner through plugin RPC timeouts until the probe answers", async () => {
+    vi.useFakeTimers();
+    let nonce = "";
+    let stalledProbes = 0;
+    let connected = true;
+    const ok = { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" };
+    const execute = vi.fn(async (input: { args?: string[] }) => {
+      const label = input.args?.[2];
+      if (label === "paperclip-runner-launch") { nonce = input.args![4]!; return ok; }
+      if (label === "paperclip-runner-process-identity") return { ...ok, stdout: `${nonce}\n4321\n2026-09-06T00:00:00.000Z\nrunner-remote\n` };
+      if (label === "paperclip-runner-monitor" && stalledProbes < 3) {
+        stalledProbes++;
+        await new Promise(resolve => setTimeout(resolve, 40_000));
+        throw new Error('RPC call "environmentExecute" timed out after 40000ms');
+      }
+      if (label === "paperclip-runner-monitor") return { ...ok, exitCode: connected ? 0 : 45 };
+      return { ...ok, exitCode: 1 };
+    });
+    const handle = createRemoteRunnerProcessLauncher({
+      target: { kind: "remote", transport: "sandbox", environmentId: "environment", leaseId: "lease", remoteCwd: "/workspace" },
+      runner: { execute } as never, remoteBinary: "/runtime/runnerd", processIdentityPath: "/runtime/identity",
+      stateDirectory: "/runtime", diagnosticsDirectory: "/runtime/diagnostics", runnerInstanceId: "runner-remote",
+    })({ command: "/controller/runnerd", args: [], cwd: "/controller", environment: {}, isRunnerConnected: () => connected });
+    let settled = false;
+    const outcome = handle.completion.then(() => { settled = true; }, error => { settled = true; return error as Error; });
+    try {
+      await vi.advanceTimersByTimeAsync(130_000);
+      expect(settled ? readRemoteRunnerRecoveryError(await outcome) : "supervising").toBe("supervising");
+      expect(stalledProbes).toBe(3);
+      expect(handle.isExitConfirmed?.()).toBe(false);
+      connected = false;
+      await vi.advanceTimersByTimeAsync(5_000);
+      expect(await outcome).toBeUndefined();
+      expect(handle.isExitConfirmed?.()).toBe(true);
+      expect(execute.mock.calls.filter(([x]) => x.args?.[2] === "paperclip-runner-launch")).toHaveLength(1);
+    } finally { vi.useRealTimers(); }
+  });
+
+  it("fails closed without exit proof when the runner is also disconnected through plugin RPC timeouts", async () => {
+    vi.useFakeTimers();
+    let nonce = "";
+    const ok = { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" };
+    const execute = vi.fn(async (input: { args?: string[] }) => {
+      const label = input.args?.[2];
+      if (label === "paperclip-runner-launch") { nonce = input.args![4]!; return ok; }
+      if (label === "paperclip-runner-process-identity") return { ...ok, stdout: `${nonce}\n4321\n2026-09-06T00:00:00.000Z\nrunner-remote\n` };
+      await new Promise(resolve => setTimeout(resolve, 40_000));
+      throw new Error('RPC call "environmentExecute" timed out after 40000ms');
+    });
+    const handle = createRemoteRunnerProcessLauncher({
+      target: { kind: "remote", transport: "sandbox", environmentId: "environment", leaseId: "lease", remoteCwd: "/workspace" },
+      runner: { execute } as never, remoteBinary: "/runtime/runnerd", processIdentityPath: "/runtime/identity",
+      stateDirectory: "/runtime", diagnosticsDirectory: "/runtime/diagnostics", runnerInstanceId: "runner-remote",
+    })({ command: "/controller/runnerd", args: [], cwd: "/controller", environment: {}, isRunnerConnected: () => false });
+    const failure = handle.completion.catch(readRemoteRunnerRecoveryError);
+    try {
+      await vi.advanceTimersByTimeAsync(45_000);
+      expect(await failure).toEqual({ stage: "process_verification", reason: "probe_transport_failed" });
+      expect(handle.isExitConfirmed?.()).toBe(false);
+      expect(execute.mock.calls.some(([x]) => x.args?.[2] === "paperclip-runner-diagnostics")).toBe(false);
+    } finally { vi.useRealTimers(); }
+  });
+
+  it("bounds supervision without exit proof when a connected runner's probes never answer", async () => {
+    vi.useFakeTimers();
+    let nonce = "";
+    const ok = { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" };
+    const execute = vi.fn(async (input: { args?: string[] }) => {
+      const label = input.args?.[2];
+      if (label === "paperclip-runner-launch") { nonce = input.args![4]!; return ok; }
+      if (label === "paperclip-runner-process-identity") return { ...ok, stdout: `${nonce}\n4321\n2026-09-06T00:00:00.000Z\nrunner-remote\n` };
+      await new Promise(resolve => setTimeout(resolve, 40_000));
+      throw new Error('RPC call "environmentExecute" timed out after 40000ms');
+    });
+    const handle = createRemoteRunnerProcessLauncher({
+      target: { kind: "remote", transport: "sandbox", environmentId: "environment", leaseId: "lease", remoteCwd: "/workspace" },
+      runner: { execute } as never, remoteBinary: "/runtime/runnerd", processIdentityPath: "/runtime/identity",
+      stateDirectory: "/runtime", diagnosticsDirectory: "/runtime/diagnostics", runnerInstanceId: "runner-remote",
+    })({ command: "/controller/runnerd", args: [], cwd: "/controller", environment: {}, isRunnerConnected: () => true });
+    let settled = false;
+    const failure = handle.completion.catch(error => { settled = true; return readRemoteRunnerRecoveryError(error); });
+    try {
+      await vi.advanceTimersByTimeAsync(REMOTE_RUNNER_RECONNECT_GRACE_MS - 60_000);
+      expect(settled).toBe(false);
+      await vi.advanceTimersByTimeAsync(180_000);
+      expect(settled).toBe(true);
+      expect(await failure).toEqual({ stage: "process_verification", reason: "probe_transport_failed" });
+      expect(handle.isExitConfirmed?.()).toBe(false);
+    } finally { vi.useRealTimers(); }
+  });
+
   it.each(["delivered", "sandbox_missing", "logging_failed"] as const)(
     "detaches runnerd and contains asynchronous signal failures (%s)", async (signalOutcome) => {
     let launchNonce = "";

server/src/services/native-runtime/native-session-executor.ts

@@ -15168,27 +15168,42 @@ export function createRemoteRunnerProcessLauncher(input: {
         attributes: { identitySource: "remote_marker", detached: true },
       });
 
+      let supervisionUnknownSince: number | null = null;
       while (true) {
-        const observed = await readRemoteRunnerRecoveryProbe({
-          runner,
-          stage: "process_verification",
-          deadline: Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS,
-          readOnly: true,
-          request: {
-            command: "sh",
-            args: [
-              "-c",
-              REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT,
-              "paperclip-runner-monitor",
-              input.processIdentityPath,
-              identityNonce,
-              input.runnerInstanceId,
-              String(identity.pid),
-              ...(identity.processStartFingerprint ? [identity.processStartFingerprint] : []),
-            ],
-            bypassSession: true,
-          },
-        });
+        let observed: Awaited<ReturnType<typeof readRemoteRunnerRecoveryProbe>>;
+        try {
+          observed = await readRemoteRunnerRecoveryProbe({
+            runner,
+            stage: "process_verification",
+            deadline: Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS,
+            readOnly: true,
+            request: {
+              command: "sh",
+              args: [
+                "-c",
+                REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT,
+                "paperclip-runner-monitor",
+                input.processIdentityPath,
+                identityNonce,
+                input.runnerInstanceId,
+                String(identity.pid),
+                ...(identity.processStartFingerprint ? [identity.processStartFingerprint] : []),
+              ],
+              bypassSession: true,
+            },
+          });
+        } catch (error) {
+          supervisionUnknownSince ??= Date.now();
+          if (
+            spec.isRunnerConnected?.() !== true ||
+            Date.now() - supervisionUnknownSince >= REMOTE_RUNNER_RECONNECT_GRACE_MS
+          ) throw error;
+          await new Promise<void>((resolve) =>
+            setTimeout(resolve, REMOTE_RUNNER_PROCESS_POLL_MS),
+          );
+          continue;
+        }
+        supervisionUnknownSince = null;
         if (observed.exitCode === 0 && !observed.timedOut) {
           await new Promise<void>((resolve) =>
             setTimeout(resolve, REMOTE_RUNNER_PROCESS_POLL_MS),