From 757a20d2f108c162333bc944c9c4cc8d7de84e9b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 16:04:13 -0600 Subject: [PATCH] fix(runner): keep remote recovery and launch through a short plugin RPC stall (#20945) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote run starts `paperclip-runnerd` in a sandbox. The controller reads and controls it through the environment plugin (`environmentExecute`). > - #20940 made a stalled supervision probe "unknown" while the Runner stays connected. Other callers still turn the same plugin stall into a terminal failure. > - During restart reattachment, the saved-state read, the identity-marker read and the process check share a 30-second budget. One plugin RPC that stalls for 40 seconds uses all of it, so a short stall fails a reattachment to a healthy Runner. > - At launch, a stalled launch call or identity read fails the run with `runner_process_termination_unconfirmed` or `runner_remote_process_identity_unavailable`, although the runner may be healthy. > - This pull request treats a plugin RPC timeout as "unknown" and retries the read-only call within a bounded stall budget. A stalled launch call is not repeated: the controller reads the launch nonce's identity marker instead. > - The benefit is that a short plugin stall during recovery or launch no longer fails a healthy run. Every positive answer from the sandbox keeps its meaning, and the bound still fails closed. ## Linked Issues or Issue Description Refs #20940, #20909 **What happened?** A plugin `environmentExecute` RPC stall of more than 40 seconds makes a remote recovery read fail. Plugin RPCs time out at the requested timeout plus a 30-second overhead, so a 10-second probe can take 40 seconds. The reattachment read budget is 30 seconds. One stalled read therefore failed reattachment with `probe_transport_failed` or `probe_timeout`. At launch, a stalled launch call rejected the run with exit unconfirmed. A stalled identity read (20-second wait, 2-second reads that can take 32 seconds) failed the run with `runner_remote_process_identity_unavailable`. **Expected behavior** A plugin RPC timeout is a stall of the plugin, not an answer from the sandbox. Read-only recovery reads retry within a bound. A launch call that stalls is not repeated. The launch nonce's marker decides whether the one runner was launched. At the bound, the run fails closed and Retry works. **Steps to reproduce** 1. Start a remote (Daytona) run. 2. Restart the controller so that the run reattaches, or start a new run. 3. While the controller reads the sandbox, stop the plugin worker for 45 seconds. 4. The reattachment or launch fails. **Paperclip version or commit** master at 48d23d1530. ## What Changed - `remote-runner-recovery-probe.ts`: adds `isPluginRpcTimeout` (an error, or its cause chain, with `PLUGIN_RPC_ERROR_CODES.TIMEOUT`). It also adds a per-recovery `RemoteRunnerPluginStall` with a 3-minute budget (`REMOTE_RUNNER_PLUGIN_STALL_BUDGET_MS`). The caller's recovery deadline caps it. After the first plugin RPC timeout in a recovery, its read-only probes use the stall deadline instead of the 30-second deadline. Signals are never retried. Other transport failures keep the 30-second budget. - `verifyRemoteRunnerReattachment` and `verifyRemoteRunnerOrSettledExit` share one stall budget across the saved-state read, the identity-marker read, the process check and the post-exit settled-state read (`readRemoteRunnerState`). - `waitForRemoteRunnerProcessIdentity`: a plugin RPC timeout extends the 20-second identity wait to the stall budget. - Launch: a launch call that fails with a plugin RPC timeout is not repeated. The controller goes on to the identity-marker wait for the same nonce. A found marker adopts the one launched runner. A missing marker uses the existing nonce cleanup, and the result stays unconfirmed when the cleanup fails. - `doc/execution-semantics.md` describes these rules. ## Verification - Red on master: 7 new tests in `server/src/services/native-runtime/native-session-executor.test.ts` failed. Each one stalls a plugin RPC for 40 seconds with a `JsonRpcCallError` `TIMEOUT`: - reattachment through a stall in the saved-state, identity-marker and process-check reads (the reattachment failed with the stall error); - the bound test (it settled before the stall budget); - the post-exit settled-state read (it failed with the stall error); - launch-call and identity-read stalls (they failed with the stall error or `runner_remote_process_identity_unavailable`). - Green: all 7 pass, and a control test proves that a non-stall transport error keeps the 30-second bound. `native-session-executor.test.ts` and `remote-runner-recovery-probe.test.ts` pass (1400 tests). `tsc --noEmit` passes for `server`. - The tests also assert that one launch call happens, no cleanup or signal runs on the success path, and the bound fails closed without exit proof. ## Risks - During a real plugin stall, recovery and launch can now wait up to 3 minutes instead of about 30 seconds before they fail closed. The longer wait applies only after a plugin RPC timeout. A sandbox that answers, or a non-timeout transport error, keeps the old timing. - No wait is added before `turn.start` on the normal path. - State saved by an older version is not changed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, code search). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 14 +- .../native-session-executor.test.ts | 135 ++++++++++++++++++ .../native-runtime/native-session-executor.ts | 36 +++-- .../remote-runner-recovery-probe.ts | 32 ++++- 4 files changed, 199 insertions(+), 18 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index e8938daa137..7cceed149b4 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -4841,7 +4841,19 @@ the Runner holds its own authenticated connection to the controller, the monitor keeps the same runner and turn and probes again. The bound is the remote runner reconnect grace (7 minutes). When the Runner has no connection and the 30-second budget is spent, or when the bound is spent, the monitor reports -`runner_process_termination_unconfirmed` and the run rests with Retry. Monitoring checks the saved process generation when available. +`runner_process_termination_unconfirmed` and the run rests with Retry. + +A plugin RPC timeout (`PLUGIN_RPC_ERROR_CODES.TIMEOUT`) is a stall of the +environment plugin, not an answer from the sandbox. When a read-only recovery +read during reattachment times out this way, the recovery's probe budget grows +from 30 seconds to 3 minutes, capped by the caller's recovery deadline. This +covers the saved runner state, the identity marker and the process check, and +the settled-state read after a confirmed exit. A signal is never retried. Other +transport failures keep the 30-second budget. At the launch of a new remote +runner, a launch call that stalls this way is not retried. The controller reads +the identity marker for the launch nonce instead, so the one launched runner is +adopted or cleaned up by its nonce. The identity read uses the same 3-minute +stall budget. Each answer that the sandbox gives keeps its current meaning. Monitoring checks the saved process generation when available. Only a matching ownership marker and confirmed process exit permit automatic replacement. Missing or conflicting identity evidence, failed reads, and an exhausted probe budget retain the existing ownership and recovery safeguards. diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 26a0cbd008f..c9c71a3bcb3 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -84,6 +84,7 @@ import { nativeSha256 } from "./canonical.js"; import { resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js"; import { REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT, RemoteRunnerRecoveryError, readRemoteRunnerRecoveryError } from "./remote-runner-recovery-diagnostics.js"; import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "./native-runner-recovery-contract.js"; +import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES } from "@paperclipai/plugin-sdk"; import { agentDirectoryWorkingCopyService } from "../agent-directory-working-copies.js"; import { probeAgentDirectory } from "../agent-directory-probe.js"; import { AGENT_FILES_CONTRACT } from "../agent-file-store.js"; @@ -862,6 +863,96 @@ describe("remote controller restart adoption", () => { expect(probes).toBe(2); expect(execute.mock.calls.every(([request]) => !request.args.join(" ").match(/nohup|mkdir|rm -|mv -|kill -"/))).toBe(true); }); + const pluginStall = () => new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.TIMEOUT, + message: 'RPC call "environmentExecute" timed out after 40000ms' }); + it.each(["runner_state", "identity_marker", "process_verification"] as const)( + "reattaches the same runner through a plugin RPC stall during the %s read", async stage => { + vi.useFakeTimers(); + onTestFinished(() => { vi.useRealTimers(); }); + const { target, execute } = fixture(); + const original = execute.getMockImplementation()!; + const operation = stage === "identity_marker" ? "paperclip-runner-recovery-identity" : "paperclip-runner-recovery-check"; + let stalls = 0; + execute.mockImplementation(async request => { + if ((stage === "runner_state" ? request.args[1].includes("base64") : request.args[2] === operation) && stalls < 2) { + stalls++; + await new Promise(resolveWait => setTimeout(resolveWait, 40_000)); + throw pluginStall(); + } + return original(request); + }); + const outcome = verifyRemoteRunnerReattachment({ + claim, target: target as never, identity, runId: "run", normalizedSessionId: "session", + }).catch(error => error as Error); + await vi.advanceTimersByTimeAsync(90_000); + const adopted = await outcome; + expect(adopted).not.toBeInstanceOf(Error); + expect((adopted as { pid: number }).pid).toBe(123); + expect(stalls).toBe(2); + expect(execute.mock.calls.every(([request]) => !request.args.join(" ").match(/nohup|mkdir|rm -|mv -|kill -"/))).toBe(true); + }); + it("fails reattachment closed when the plugin stall outlasts its bound", async () => { + vi.useFakeTimers(); + onTestFinished(() => { vi.useRealTimers(); }); + const { target, execute } = fixture(); + const original = execute.getMockImplementation()!; + execute.mockImplementation(async request => { + if (request.args[2] !== "paperclip-runner-recovery-check") return original(request); + await new Promise(resolveWait => setTimeout(resolveWait, 40_000)); + throw pluginStall(); + }); + let settled = false; + const outcome = verifyRemoteRunnerReattachment({ + claim, target: target as never, identity, runId: "run", normalizedSessionId: "session", + }).catch(error => { settled = true; return error as Error; }); + await vi.advanceTimersByTimeAsync(150_000); + expect(settled).toBe(false); + await vi.advanceTimersByTimeAsync(80_000); + expect(readRemoteRunnerRecoveryError(await outcome)).toEqual({ stage: "process_verification", reason: "probe_transport_failed" }); + }); + it("keeps the 30-second bound for a transport failure that is not a plugin stall", async () => { + vi.useFakeTimers(); + onTestFinished(() => { vi.useRealTimers(); }); + const { target, execute } = fixture(); + const original = execute.getMockImplementation()!; + execute.mockImplementation(async request => { + if (request.args[2] !== "paperclip-runner-recovery-check") return original(request); + throw new Error("Sandbox with ID test-deleted-sandbox not found"); + }); + const startedAt = Date.now(); + const outcome = verifyRemoteRunnerReattachment({ + claim, target: target as never, identity, runId: "run", normalizedSessionId: "session", + }).catch(error => error as Error); + await vi.advanceTimersByTimeAsync(30_000); + expect(readRemoteRunnerRecoveryError(await outcome)).toEqual({ stage: "process_verification", reason: "probe_transport_failed" }); + expect(Date.now() - startedAt).toBeLessThanOrEqual(30_000); + }); + it("reads the settled state of an exited runner through a plugin RPC stall", async () => { + vi.useFakeTimers(); + onTestFinished(() => { vi.useRealTimers(); }); + const settled = { lifecycle: "suspended", outbox: [], pendingTerminalDelivery: null, pendingProviderCleanup: null }; + const { target, execute } = fixture(settled); + const original = execute.getMockImplementation()!; + let exited = false; + let stalls = 0; + execute.mockImplementation(async request => { + if (request.args[2] === "paperclip-runner-recovery-check") { + exited = true; + return { stdout: "", stderr: "", exitCode: 45, timedOut: false }; + } + if (exited && request.args[1].includes("base64") && stalls === 0) { + stalls++; + await new Promise(resolveWait => setTimeout(resolveWait, 40_000)); + throw pluginStall(); + } + return original(request); + }); + const outcome = verifyRemoteRunnerOrSettledExit({ claim, target: target as never, identity, + runId: "run", normalizedSessionId: "session", providerAttempt: 2 }).catch(error => error as Error); + await vi.advanceTimersByTimeAsync(60_000); + expect(await outcome).toBeUndefined(); + expect(stalls).toBe(1); + }); it("rejects a marker that cannot prove its process generation", async () => { const { target, execute } = fixture(); const original = execute.getMockImplementation()!; @@ -1672,6 +1763,50 @@ fs.readFileSync = function(path, ...args) { } finally { vi.useRealTimers(); } }); + it.each(["launch", "identity"] as const)("supervises the one launched runner through a plugin RPC stall on the %s call", async stalled => { + vi.useFakeTimers(); + let nonce = ""; + let markerWritten = false; + let stalls = 0; + const ok = { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" }; + const stall = async () => { + stalls++; + await new Promise(resolve => setTimeout(resolve, 40_000)); + throw new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.TIMEOUT, message: 'RPC call "environmentExecute" timed out after 40000ms' }); + }; + const execute = vi.fn(async (input: { args?: string[] }) => { + const label = input.args?.[2]; + if (label === "paperclip-runner-launch") { + nonce = input.args![4]!; + markerWritten = true; + if (stalled === "launch") return stall(); + return ok; + } + if (label === "paperclip-runner-process-identity") { + if (stalled === "identity" && stalls === 0) return stall(); + return markerWritten ? { ...ok, stdout: `${nonce}\n4321\n2026-09-06T00:00:00.000Z\nrunner-remote\n` } : { ...ok, exitCode: 1 }; + } + if (label === "paperclip-runner-monitor") return { ...ok, exitCode: 45 }; + return { ...ok, exitCode: 1 }; + }); + const onSpawn = vi.fn(async () => undefined); + const handle = createRemoteRunnerProcessLauncher({ + target: { kind: "remote", transport: "sandbox", environmentId: "environment", leaseId: "lease", remoteCwd: "/workspace" }, + runner: { execute } as never, remoteBinary: "/runtime/runnerd", processIdentityPath: "/runtime/identity", + stateDirectory: "/runtime", diagnosticsDirectory: "/runtime/diagnostics", runnerInstanceId: "runner-remote", onSpawn, + })({ command: "/controller/runnerd", args: [], cwd: "/controller", environment: {} }); + const outcome = handle.completion.catch(error => error as Error); + try { + await vi.advanceTimersByTimeAsync(60_000); + expect(await outcome).toMatchObject({ code: null }); + expect(stalls).toBe(1); + expect(onSpawn).toHaveBeenCalledExactlyOnceWith({ pid: 4321, processGroupId: null, startedAt: "2026-09-06T00:00:00.000Z" }); + expect(handle.isExitConfirmed?.()).toBe(true); + expect(execute.mock.calls.filter(([x]) => x.args?.[2] === "paperclip-runner-launch")).toHaveLength(1); + expect(execute.mock.calls.some(([x]) => x.args?.[2] === "paperclip-runner-identity-failure-cleanup")).toBe(false); + } finally { vi.useRealTimers(); } + }); + it("bounds supervision without exit proof when a connected runner's probes never answer", async () => { vi.useFakeTimers(); let nonce = ""; diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index c4a1b48c36f..db648cb26ed 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -282,7 +282,7 @@ import { REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT, RemoteRunnerRecoveryError, attachRemoteRunnerRecoveryDiagnostic, boundRemoteRunnerExitEvidence, describeExitedRemoteRunner, describeExitedRemoteRunnerReason, readRemoteRunnerRecoveryError, remoteRecoveryProbeReason, } from "./remote-runner-recovery-diagnostics.js"; -import { readRemoteRunnerRecoveryProbe, REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS } from "./remote-runner-recovery-probe.js"; +import { isPluginRpcTimeout, readRemoteRunnerRecoveryProbe, REMOTE_RUNNER_PLUGIN_STALL_BUDGET_MS, REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS, remoteRunnerPluginStall, type RemoteRunnerPluginStall } from "./remote-runner-recovery-probe.js"; import { describeOwnFailedAttachment, inspectRemoteWarmSessionRecovery, isSettledRemoteWarmControlState, summarizePostTerminalEvents, type RemoteWarmSessionRecovery, unsettledControllerCommands } from "./remote-warm-session-recovery.js"; import { assertNativeWarmIngressRetirement, nativeWarmIngressAllocation, nativeWarmIngressIdentity, type NativeWarmIngressClaim, type NativeWarmIngressOwner } from "./native-warm-ingress-ownership.js"; import { resolveNativeStagedAttachmentRecovery, type NativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js"; @@ -14506,6 +14506,7 @@ export async function readRemoteRunnerState(input: { runner: CommandManagedRuntimeRunner; stateDirectory: string; recoveryDeadline?: number; + stall?: RemoteRunnerPluginStall; }): Promise> { const statePath = posix.join(input.stateDirectory, "runner-state.json"); const escapedPath = statePath.replaceAll("'", "'\\''"); @@ -14519,7 +14520,7 @@ export async function readRemoteRunnerState(input: { ? await input.runner.execute(request) : await readRemoteRunnerRecoveryProbe({ runner: input.runner, request, stage: "runner_state", - deadline: input.recoveryDeadline, readOnly: true, + deadline: input.recoveryDeadline, readOnly: true, stall: input.stall, }); if (result.exitCode !== 0 || result.timedOut) { const error = new Error( @@ -14678,6 +14679,7 @@ export async function verifyRemoteRunnerReattachment(input: { normalizedSessionId: string; requireIdle?: boolean; recoveryDeadline?: number; + stall?: RemoteRunnerPluginStall; }) { const { claim, target, identity } = input; if (claim.remote.kind === "plugin_task") throw remoteRecoveryLeaseMismatch("runner_verification", "plugin_task"); @@ -14694,8 +14696,9 @@ export async function verifyRemoteRunnerReattachment(input: { const runner = target.runner; const probeDeadline = Math.min(input.recoveryDeadline ?? Number.POSITIVE_INFINITY, Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS); + const stall = input.stall ?? remoteRunnerPluginStall(input.recoveryDeadline); const stateDirectory = remoteRunnerStateDirectory(target.remoteCwd, input.normalizedSessionId); - const runnerState = await readRemoteRunnerState({ runner, stateDirectory, recoveryDeadline: probeDeadline }); + const runnerState = await readRemoteRunnerState({ runner, stateDirectory, recoveryDeadline: probeDeadline, stall }); if (input.requireIdle && (runnerState.lifecycle !== "ready" || !Array.isArray(runnerState.outbox) || runnerState.outbox.length !== 0 || runnerState.pendingTerminalDelivery !== null)) { throw new Error("runner_state_identity_mismatch: remote_warm_session_not_idle"); @@ -14711,7 +14714,7 @@ export async function verifyRemoteRunnerReattachment(input: { } const identityPath = posix.join(stateDirectory, "runner-process.identity"); const marker = await readRemoteRunnerRecoveryProbe({ - runner, stage: "identity_marker", deadline: probeDeadline, readOnly: true, + runner, stage: "identity_marker", deadline: probeDeadline, readOnly: true, stall, request: { command: "sh", args: [ @@ -14739,9 +14742,10 @@ export async function verifyRemoteRunnerReattachment(input: { if (!processIdentity) throw new RemoteRunnerRecoveryError({ stage: "identity_marker", reason: "marker_invalid" }); if (!processIdentity.processStartFingerprint) throw new RemoteRunnerRecoveryError({ stage: "identity_marker", reason: "fingerprint_missing" }); const expectedStartFingerprint = processIdentity.processStartFingerprint; - const check = async (signal?: NodeJS.Signals, deadline = Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS) => { + const check = async (signal?: NodeJS.Signals, deadline = Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS, + verificationStall?: RemoteRunnerPluginStall) => { const result = await readRemoteRunnerRecoveryProbe({ - runner, stage: "process_verification", deadline, readOnly: !signal, + runner, stage: "process_verification", deadline, readOnly: !signal, stall: verificationStall, request: { command: "sh", args: [ @@ -14766,7 +14770,7 @@ export async function verifyRemoteRunnerReattachment(input: { } return { alive, reason }; }; - const verified = await check(undefined, probeDeadline); + const verified = await check(undefined, probeDeadline, stall); if (!verified.alive) { throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: verified.reason, ...(verified.reason === "process_not_running" ? { @@ -14812,8 +14816,9 @@ export async function verifyRemoteRunnerOrSettledExit( ) { const recoveryDeadline = Math.min(input.recoveryDeadline ?? Number.POSITIVE_INFINITY, Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS); + const stall = input.stall ?? remoteRunnerPluginStall(input.recoveryDeadline); try { - return await verifyRemoteRunnerReattachment({ ...input, recoveryDeadline }); + return await verifyRemoteRunnerReattachment({ ...input, recoveryDeadline, stall }); } catch (error) { const diagnostic = readRemoteRunnerRecoveryError(error); const target = input.target; @@ -14823,7 +14828,7 @@ export async function verifyRemoteRunnerOrSettledExit( (!exitedBeforeSessionOpen && (diagnostic.runnerLifecycle !== "suspended" || diagnostic.runnerSettled !== true)) || target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner) throw error; const state = await readRemoteRunnerState({ runner: target.runner, - stateDirectory: remoteRunnerStateDirectory(target.remoteCwd, input.normalizedSessionId), recoveryDeadline }); + stateDirectory: remoteRunnerStateDirectory(target.remoteCwd, input.normalizedSessionId), recoveryDeadline, stall }); if (REMOTE_RUNNER_IDENTITY_FIELDS.some(field => state[field] !== input.identity[field])) throw error; if (exitedBeforeSessionOpen ? !runnerStateAllowsFreshStart(state) : state.lifecycle !== "suspended" || !describeExitedRemoteRunner(state).runnerSettled) throw error; @@ -14906,7 +14911,8 @@ async function waitForRemoteRunnerProcessIdentity(input: { runnerInstanceId: string; requireProcessStartFingerprint?: boolean; }): Promise<{ pid: number; startedAt: string; processStartFingerprint?: string }> { - const deadline = Date.now() + REMOTE_RUNNER_PROCESS_IDENTITY_WAIT_MS; + let deadline = Date.now() + REMOTE_RUNNER_PROCESS_IDENTITY_WAIT_MS; + const stallDeadline = Date.now() + REMOTE_RUNNER_PLUGIN_STALL_BUDGET_MS; while (Date.now() < deadline) { const result = await input.runner .execute({ @@ -14920,7 +14926,10 @@ async function waitForRemoteRunnerProcessIdentity(input: { bypassSession: true, timeoutMs: 2_000, }) - .catch(() => null); + .catch((error: unknown) => { + if (isPluginRpcTimeout(error)) deadline = Math.max(deadline, stallDeadline); + return null; + }); const identity = result && result.exitCode === 0 && !result.timedOut ? parseRemoteRunnerProcessIdentity(result.stdout, input) @@ -15113,8 +15122,11 @@ export function createRemoteRunnerProcessLauncher(input: { timeoutMs: 20_000, bypassSession: true, onLog: input.onLog, + }).catch((error: unknown) => { + if (isPluginRpcTimeout(error)) return null; + throw error; }); - if (launchResult.exitCode !== 0 || launchResult.timedOut) { + if (launchResult && (launchResult.exitCode !== 0 || launchResult.timedOut)) { throw new Error( launchResult.timedOut ? "runner_remote_process_launch_timed_out" diff --git a/server/src/services/native-runtime/remote-runner-recovery-probe.ts b/server/src/services/native-runtime/remote-runner-recovery-probe.ts index 77486aa4496..a5724bde041 100644 --- a/server/src/services/native-runtime/remote-runner-recovery-probe.ts +++ b/server/src/services/native-runtime/remote-runner-recovery-probe.ts @@ -1,4 +1,5 @@ import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; +import { PLUGIN_RPC_ERROR_CODES } from "@paperclipai/plugin-sdk"; import { attachRemoteRunnerRecoveryDiagnostic, RemoteRunnerRecoveryError, @@ -6,6 +7,25 @@ import { } from "./remote-runner-recovery-diagnostics.js"; export const REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS = 30_000; +export const REMOTE_RUNNER_PLUGIN_STALL_BUDGET_MS = 180_000; + +export function isPluginRpcTimeout(error: unknown): boolean { + let current: unknown = error; + for (let depth = 0; depth < 5 && current instanceof Error; depth++) { + if ((current as { code?: unknown }).code === PLUGIN_RPC_ERROR_CODES.TIMEOUT) return true; + current = current.cause; + } + return false; +} + +export interface RemoteRunnerPluginStall { + deadline: number; + observed: boolean; +} + +export function remoteRunnerPluginStall(cap = Number.POSITIVE_INFINITY): RemoteRunnerPluginStall { + return { deadline: Math.min(cap, Date.now() + REMOTE_RUNNER_PLUGIN_STALL_BUDGET_MS), observed: false }; +} export async function readRemoteRunnerRecoveryProbe(input: { runner: CommandManagedRuntimeRunner; @@ -13,15 +33,16 @@ export async function readRemoteRunnerRecoveryProbe(input: { stage: RemoteRunnerRecoveryDiagnostic["stage"]; deadline: number; readOnly: boolean; + stall?: RemoteRunnerPluginStall; }) { let lastError: unknown = new RemoteRunnerRecoveryError({ stage: input.stage, reason: "probe_timeout", }); - // A few fast gateway failures can exhaust an attempt count in under a - // second. Read-only observations share the caller's deadline instead; - // signals still have one attempt because their outcome may be unknown. + const deadline = () => input.readOnly && input.stall?.observed + ? Math.max(input.deadline, input.stall.deadline) + : input.deadline; for (let attempt = 0; ; attempt++) { - const remaining = input.deadline - Date.now(); + const remaining = deadline() - Date.now(); if (!Number.isFinite(remaining) || remaining <= 0) break; try { const result = await input.runner.execute({ @@ -31,12 +52,13 @@ export async function readRemoteRunnerRecoveryProbe(input: { if (!result.timedOut) return result; lastError = new RemoteRunnerRecoveryError({ stage: input.stage, reason: "probe_timeout" }); } catch (error) { + if (input.readOnly && input.stall && isPluginRpcTimeout(error)) input.stall.observed = true; lastError = attachRemoteRunnerRecoveryDiagnostic(error, { stage: input.stage, reason: "probe_transport_failed", }); } const delayMs = Math.min(250 * 2 ** Math.min(attempt, 3), 2_000); - if (!input.readOnly || input.deadline - Date.now() <= delayMs) break; + if (!input.readOnly || deadline() - Date.now() <= delayMs) break; await new Promise(resolve => setTimeout(resolve, delayMs)); } throw lastError;