From 48d23d15309300ef54d0b64ab9323745f831f91f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 15:45:25 -0600 Subject: [PATCH] fix(runner): keep a connected remote Runner through a stalled plugin supervision probe (#20940) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote run starts `paperclip-runnerd` in a sandbox. The controller monitors the process with a health probe that goes through the environment plugin (`environmentExecute`). > - A stall of the Daytona plugin worker makes that probe RPC time out after 40 seconds. The monitor treated the timeout as "process supervision failed". > - The transport then failed closed with `runner_process_termination_unconfirmed`. The run ended `notification_transport_failed`, `recoverable: false`, and the close path suspended a healthy Runner. The task rested blocked with Retry. > - Nothing was wrong with the sandbox, the Runner or the provider. The Runner held its own authenticated connection to the controller during the full stall. > - This pull request makes a failed or timed-out probe "unknown". While the Runner stays connected on its own channel, the monitor keeps the same runner and turn and probes again, for a bounded time. > - The benefit is that a short plugin stall no longer kills a healthy run. The fail-closed cases stay the same. ## Linked Issues or Issue Description Refs #20909 (chaos cell CR3, which reproduces this failure) **What happened?** A 45-second stall of the Daytona plugin worker killed a healthy run. Chaos cell CR3 reproduced it 2 of 2 times (OpenCode and Claude, Daytona). A staging run had the same failure with no deploy: a plugin `environmentExecute` RPC stalled for more than 40 seconds. The run failed with "Notification transport failed closed: runner_process_termination_unconfirmed: runnerd process supervision failed: RPC call \"environmentExecute\" timed out after 40000ms". The close path then sent `runner.suspend`, and the healthy Runner suspended and exited. The task rested blocked with Retry. **Expected behavior** A probe that times out or fails in transport is not proof that the process stopped. While the Runner stays connected on its own channel, the run continues. If supervision cannot come back within a bound, the run rests with a working Retry. A probe that positively reports the process gone still starts the existing recovery. **Steps to reproduce** 1. Start a remote (Daytona) run with a T1 workload. 2. Send SIGSTOP to the server's Daytona plugin worker for 45 seconds, then SIGCONT. 3. The run fails `notification_transport_failed` about 40 seconds into the stall. **Paperclip version or commit** master at 7cd5558e41. ## What Changed - `createRemoteRunnerProcessLauncher` (`server/src/services/native-runtime/native-session-executor.ts`): when the health probe throws (RPC timeout or transport error) and the Runner holds an authenticated connection, the monitor waits one poll interval and probes again. It does not reject the process completion. The unknown period has a bound: the remote runner reconnect grace (7 minutes). When the Runner is not connected, the monitor keeps the old behavior: it rejects after the 30-second probe budget, without exit proof. - Probe results that positively answer (exit 45 "process gone", or identity and generation failures) keep their exact current handling. - `RunnerProcessLaunchSpec` (`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`) has a new optional `isRunnerConnected` callback. `spawnRunner` passes it to the process owner and keeps it on restart. - The runnerd transport and the external runnerd driver supply `isRunnerConnected` from the control plane's count of authenticated Runner connections. - `doc/execution-semantics.md` describes the unknown-probe rule and its bound. ## Verification - Red, on master: `npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "plugin RPC timeouts until"` (in `server/`). The completion rejected after the first stalled probe with `{ stage: "process_verification", reason: "probe_transport_failed" }`. The transport turns that into `runner_process_termination_unconfirmed`, which is the staging error. - Red, on master: `npx vitest run src/live/runnerd-codex-transport.test.ts -t "live evidence of the runner"` (in `packages/paperclip-runner/`). The launch spec had no `isRunnerConnected` (`expected undefined to be true`). - Green: both tests pass. New tests also prove: - three 40-second plugin RPC timeouts with the Runner connected leave the same runner supervised, and a later exit-45 probe still confirms exit; - with the Runner disconnected, the same timeout rejects without exit proof (`probe_transport_failed`, `isExitConfirmed() === false`); - with the Runner connected and probes that never answer, supervision rejects without exit proof after the 7-minute bound. - `server/src/services/native-runtime/native-session-executor.test.ts`, `remote-runner-recovery-probe.test.ts`, `packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts`, `src/control-plane/durable-prp-control-plane.test.ts` and `src/drivers/external` pass locally. - `tsc --noEmit` passes for `server` and `packages/paperclip-runner`. - Chaos: CR3 results are in a follow-up comment on this PR. ## Risks - While the Runner stays connected and the plugin cannot answer, the controller waits up to 7 minutes before it fails closed. Before this change it failed after about 40 seconds. During that time the Runner's own connection is the liveness evidence, and the controller never reports an exit. - When the Runner is not connected, the timing is the same as before. Sandbox loss still fails closed after the 30-second probe budget. - The new spec field is optional. A process owner that does not read it keeps the old behavior. State saved by an older version is not changed. - No wait is added before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, code search). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 8 +- .../durable-prp-control-plane.ts | 4 + .../external/runnerd-external-driver.ts | 2 +- .../src/live/runnerd-codex-transport.test.ts | 45 +++++++++ .../src/live/runnerd-codex-transport.ts | 2 + .../native-session-executor.test.ts | 92 +++++++++++++++++++ .../native-runtime/native-session-executor.ts | 55 +++++++---- 7 files changed, 186 insertions(+), 22 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 6eac6f41144..65ebcda93af 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -4814,7 +4814,13 @@ configuration from the same company, task, and bot binding. An active remote runner's health probe is an observation, not an exit receipt. Timed-out or rejected read-only probes retry within the existing 30-second budget. A successful probe keeps the same runner and turn; no provider work is -replayed. Monitoring checks the saved process generation when available. +replayed. A probe that times out or fails in transport is unknown, not an exit. +For example, a stalled environment plugin RPC times out after 40 seconds. While +the Runner holds its own authenticated connection to the controller, the monitor +keeps the same runner and turn and probes again. The bound is the remote runner +reconnect grace (7 minutes). When the Runner has no connection and the 30-second +budget is spent, or when the bound is spent, the monitor reports +`runner_process_termination_unconfirmed` and the run rests with Retry. Monitoring checks the saved process generation when available. Only a matching ownership marker and confirmed process exit permit automatic replacement. Missing or conflicting identity evidence, failed reads, and an exhausted probe budget retain the existing ownership and recovery safeguards. diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index 3fc4438b0b2..3a737b70f1b 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -646,6 +646,8 @@ export interface RunnerProcessLaunchSpec { args: readonly string[]; cwd: string; environment: NodeJS.ProcessEnv; + /** True while the launched Runner holds an authenticated connection to its control plane. */ + isRunnerConnected?: () => boolean; } function domainDigest(domain: string, parts: readonly Buffer[]): Buffer { @@ -4732,6 +4734,7 @@ export function spawnRunner(options: { }; environment?: NodeJS.ProcessEnv; processLauncher?: (spec: RunnerProcessLaunchSpec) => RunnerProcessHandle; + isRunnerConnected?: () => boolean; diagnosticsDirectory?: string; }): RunnerProcessHandle { for (const value of [options.maxRuntimeMs, options.maxLifetimeMs]) { @@ -4881,6 +4884,7 @@ export function spawnRunner(options: { if (options.processLauncher !== undefined) { return withRestart( options.processLauncher({ command, args, cwd: packageRoot, environment, + ...(options.isRunnerConnected ? { isRunnerConnected: options.isRunnerConnected } : {}), nativeRunner: { identity: options.identity, ticket: options.ticket, connection, stateDirectory: options.stateDirectory, runnerVersion: options.runnerVersion, runnerDigest: options.runnerDigest, diff --git a/packages/paperclip-runner/src/drivers/external/runnerd-external-driver.ts b/packages/paperclip-runner/src/drivers/external/runnerd-external-driver.ts index 3daecd15d60..bf361d3211d 100644 --- a/packages/paperclip-runner/src/drivers/external/runnerd-external-driver.ts +++ b/packages/paperclip-runner/src/drivers/external/runnerd-external-driver.ts @@ -232,7 +232,7 @@ class RunnerdExternalSession implements HarnessSession { const ticket = core.issueBootstrapTicket(60_000); await core.flushCheckpoint(); this.#process = spawnRunner({ - processLauncher: o.runnerProcessLauncher, connectUrl: registration?.connectUrl ?? (registration?.connection ? undefined : core.connectUrl), + processLauncher: o.runnerProcessLauncher, isRunnerConnected: () => core.activeRunnerConnectionCount() > 0, connectUrl: registration?.connectUrl ?? (registration?.connection ? undefined : core.connectUrl), connection: registration?.connection, stateDirectory: runnerState, identity: o.identity, ticket, runnerBinaryPath: binary, runnerVersion: artifact.version, runnerDigest: artifact.digest, diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts index a9ce3e931f0..ee70d048216 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts @@ -372,6 +372,51 @@ it.each(["resolved", "rejected"] as const)( 15_000, ); +it("gives the process owner live evidence of the runner's own authenticated connection", async () => { + const directory = await mkdtemp(join(tmpdir(), "runnerd-connection-evidence-")); + const specs: durableControlPlane.RunnerProcessLaunchSpec[] = []; + let child: ReturnType | undefined; + let exited: Promise | undefined; + const bundle = createCapabilityRunnerdCodexTransport({ + runnerBinary: defaultCapabilityRunnerdBinary(), + codexCommand: fakeCodex, + codexArgs: fakeCodexArgs(directory), + stateDirectory: directory, + lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 }, + runnerReconnectGraceMs: 60_000, + runnerProcessLauncher: spec => { + specs.push(spec); + const ownedChild = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: spec.environment, stdio: "ignore" }); + child = ownedChild; + exited = new Promise(resolveExit => ownedChild.once("exit", () => resolveExit())); + return { + child: ownedChild, + completion: new Promise(() => {}), + isExitConfirmed: () => false, + }; + }, + }); + try { + await bundle.transport.request("thread/start", { cwd: directory }); + expect(specs).toHaveLength(1); + expect(specs[0]!.isRunnerConnected?.()).toBe(true); + child!.kill("SIGKILL"); + await exited; + await vi.waitFor(() => expect(specs[0]!.isRunnerConnected?.()).toBe(false), { timeout: 5_000 }); + } finally { + const providerPid = bundle.evidence().codexPid; + if (providerPid) { + try { process.kill(providerPid, "SIGKILL"); } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; + } + } + await bundle.transport.detachControllerForRestart!(); + child?.kill("SIGKILL"); + await exited; + await rm(directory, { recursive: true, force: true }); + } +}, 15_000); + it.each([ "transport_reconnect_grace_exceeded", "transport reconnect grace exceeded; durable state is preserved", diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index b55c9229682..2d33e6a8182 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -6258,6 +6258,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { environment: prepared.environment, diagnosticsDirectory: resolve(this.#root, "diagnostics"), processLauncher: this.options.runnerProcessLauncher, + isRunnerConnected: () => core.activeRunnerConnectionCount() > 0, }); this.#handle = handle; this.#watchRunner(handle); @@ -7132,6 +7133,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ), diagnosticsDirectory: resolve(this.#root, "diagnostics"), processLauncher: this.options.runnerProcessLauncher, + isRunnerConnected: () => core.activeRunnerConnectionCount() > 0, }); if (handle) { this.#handle = handle; diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 21d662aa422..26a0cbd008f 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -83,6 +83,7 @@ import { DatabaseSync } from "node:sqlite"; import { nativeSha256 } from "./canonical.js"; import { resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js"; import { REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT, RemoteRunnerRecoveryError, readRemoteRunnerRecoveryError } from "./remote-runner-recovery-diagnostics.js"; +import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "./native-runner-recovery-contract.js"; import { agentDirectoryWorkingCopyService } from "../agent-directory-working-copies.js"; import { probeAgentDirectory } from "../agent-directory-probe.js"; import { AGENT_FILES_CONTRACT } from "../agent-file-store.js"; @@ -1608,6 +1609,97 @@ fs.readFileSync = function(path, ...args) { } finally { vi.useRealTimers(); } }); + it("keeps supervising a connected runner through plugin RPC timeouts until the probe answers", async () => { + vi.useFakeTimers(); + let nonce = ""; + let stalledProbes = 0; + let connected = true; + const ok = { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" }; + const execute = vi.fn(async (input: { args?: string[] }) => { + const label = input.args?.[2]; + if (label === "paperclip-runner-launch") { nonce = input.args![4]!; return ok; } + if (label === "paperclip-runner-process-identity") return { ...ok, stdout: `${nonce}\n4321\n2026-09-06T00:00:00.000Z\nrunner-remote\n` }; + if (label === "paperclip-runner-monitor" && stalledProbes < 3) { + stalledProbes++; + await new Promise(resolve => setTimeout(resolve, 40_000)); + throw new Error('RPC call "environmentExecute" timed out after 40000ms'); + } + if (label === "paperclip-runner-monitor") return { ...ok, exitCode: connected ? 0 : 45 }; + return { ...ok, exitCode: 1 }; + }); + const handle = createRemoteRunnerProcessLauncher({ + target: { kind: "remote", transport: "sandbox", environmentId: "environment", leaseId: "lease", remoteCwd: "/workspace" }, + runner: { execute } as never, remoteBinary: "/runtime/runnerd", processIdentityPath: "/runtime/identity", + stateDirectory: "/runtime", diagnosticsDirectory: "/runtime/diagnostics", runnerInstanceId: "runner-remote", + })({ command: "/controller/runnerd", args: [], cwd: "/controller", environment: {}, isRunnerConnected: () => connected }); + let settled = false; + const outcome = handle.completion.then(() => { settled = true; }, error => { settled = true; return error as Error; }); + try { + await vi.advanceTimersByTimeAsync(130_000); + expect(settled ? readRemoteRunnerRecoveryError(await outcome) : "supervising").toBe("supervising"); + expect(stalledProbes).toBe(3); + expect(handle.isExitConfirmed?.()).toBe(false); + connected = false; + await vi.advanceTimersByTimeAsync(5_000); + expect(await outcome).toBeUndefined(); + expect(handle.isExitConfirmed?.()).toBe(true); + expect(execute.mock.calls.filter(([x]) => x.args?.[2] === "paperclip-runner-launch")).toHaveLength(1); + } finally { vi.useRealTimers(); } + }); + + it("fails closed without exit proof when the runner is also disconnected through plugin RPC timeouts", async () => { + vi.useFakeTimers(); + let nonce = ""; + const ok = { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" }; + const execute = vi.fn(async (input: { args?: string[] }) => { + const label = input.args?.[2]; + if (label === "paperclip-runner-launch") { nonce = input.args![4]!; return ok; } + if (label === "paperclip-runner-process-identity") return { ...ok, stdout: `${nonce}\n4321\n2026-09-06T00:00:00.000Z\nrunner-remote\n` }; + await new Promise(resolve => setTimeout(resolve, 40_000)); + throw new Error('RPC call "environmentExecute" timed out after 40000ms'); + }); + const handle = createRemoteRunnerProcessLauncher({ + target: { kind: "remote", transport: "sandbox", environmentId: "environment", leaseId: "lease", remoteCwd: "/workspace" }, + runner: { execute } as never, remoteBinary: "/runtime/runnerd", processIdentityPath: "/runtime/identity", + stateDirectory: "/runtime", diagnosticsDirectory: "/runtime/diagnostics", runnerInstanceId: "runner-remote", + })({ command: "/controller/runnerd", args: [], cwd: "/controller", environment: {}, isRunnerConnected: () => false }); + const failure = handle.completion.catch(readRemoteRunnerRecoveryError); + try { + await vi.advanceTimersByTimeAsync(45_000); + expect(await failure).toEqual({ stage: "process_verification", reason: "probe_transport_failed" }); + expect(handle.isExitConfirmed?.()).toBe(false); + expect(execute.mock.calls.some(([x]) => x.args?.[2] === "paperclip-runner-diagnostics")).toBe(false); + } finally { vi.useRealTimers(); } + }); + + it("bounds supervision without exit proof when a connected runner's probes never answer", async () => { + vi.useFakeTimers(); + let nonce = ""; + const ok = { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" }; + const execute = vi.fn(async (input: { args?: string[] }) => { + const label = input.args?.[2]; + if (label === "paperclip-runner-launch") { nonce = input.args![4]!; return ok; } + if (label === "paperclip-runner-process-identity") return { ...ok, stdout: `${nonce}\n4321\n2026-09-06T00:00:00.000Z\nrunner-remote\n` }; + await new Promise(resolve => setTimeout(resolve, 40_000)); + throw new Error('RPC call "environmentExecute" timed out after 40000ms'); + }); + const handle = createRemoteRunnerProcessLauncher({ + target: { kind: "remote", transport: "sandbox", environmentId: "environment", leaseId: "lease", remoteCwd: "/workspace" }, + runner: { execute } as never, remoteBinary: "/runtime/runnerd", processIdentityPath: "/runtime/identity", + stateDirectory: "/runtime", diagnosticsDirectory: "/runtime/diagnostics", runnerInstanceId: "runner-remote", + })({ command: "/controller/runnerd", args: [], cwd: "/controller", environment: {}, isRunnerConnected: () => true }); + let settled = false; + const failure = handle.completion.catch(error => { settled = true; return readRemoteRunnerRecoveryError(error); }); + try { + await vi.advanceTimersByTimeAsync(REMOTE_RUNNER_RECONNECT_GRACE_MS - 60_000); + expect(settled).toBe(false); + await vi.advanceTimersByTimeAsync(180_000); + expect(settled).toBe(true); + expect(await failure).toEqual({ stage: "process_verification", reason: "probe_transport_failed" }); + expect(handle.isExitConfirmed?.()).toBe(false); + } finally { vi.useRealTimers(); } + }); + it.each(["delivered", "sandbox_missing", "logging_failed"] as const)( "detaches runnerd and contains asynchronous signal failures (%s)", async (signalOutcome) => { let launchNonce = ""; diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 18fc09d6954..c4a1b48c36f 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -15168,27 +15168,42 @@ export function createRemoteRunnerProcessLauncher(input: { attributes: { identitySource: "remote_marker", detached: true }, }); + let supervisionUnknownSince: number | null = null; while (true) { - const observed = await readRemoteRunnerRecoveryProbe({ - runner, - stage: "process_verification", - deadline: Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS, - readOnly: true, - request: { - command: "sh", - args: [ - "-c", - REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT, - "paperclip-runner-monitor", - input.processIdentityPath, - identityNonce, - input.runnerInstanceId, - String(identity.pid), - ...(identity.processStartFingerprint ? [identity.processStartFingerprint] : []), - ], - bypassSession: true, - }, - }); + let observed: Awaited>; + try { + observed = await readRemoteRunnerRecoveryProbe({ + runner, + stage: "process_verification", + deadline: Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS, + readOnly: true, + request: { + command: "sh", + args: [ + "-c", + REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT, + "paperclip-runner-monitor", + input.processIdentityPath, + identityNonce, + input.runnerInstanceId, + String(identity.pid), + ...(identity.processStartFingerprint ? [identity.processStartFingerprint] : []), + ], + bypassSession: true, + }, + }); + } catch (error) { + supervisionUnknownSince ??= Date.now(); + if ( + spec.isRunnerConnected?.() !== true || + Date.now() - supervisionUnknownSince >= REMOTE_RUNNER_RECONNECT_GRACE_MS + ) throw error; + await new Promise((resolve) => + setTimeout(resolve, REMOTE_RUNNER_PROCESS_POLL_MS), + ); + continue; + } + supervisionUnknownSince = null; if (observed.exitCode === 0 && !observed.timedOut) { await new Promise((resolve) => setTimeout(resolve, REMOTE_RUNNER_PROCESS_POLL_MS),