fix(server): retry setup when the plugin worker was not running (#19249)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote agent runs call a sandbox provider plugin, such as Daytona, through a plugin worker process. > - Sometimes that worker is not running. It can be stopped, restarting after a crash, or retired by the host. > - Then the host refuses the call before it sends anything to the worker. > - Setup recorded this refusal as a generic `setup_failed`. It scheduled no retry, so the task became stranded and needed a person. > - This pull request types the refusal and gives it its own transient setup code. The bounded bootstrap retry then runs. > - The benefit is that the task continues on its own when the worker is available again. Users do not see a blocked task. ## Linked Issues or Issue Description Refs #19150 (removed one trigger for the worker retirement) Refs #19245 (restarts a worker after a fail-closed retirement) **What happened?** A plugin worker stopped. Each run that then called `environmentExecute` during setup failed with `Cannot call "environmentExecute" — worker for "<plugin>" is stopped`. The run got `setup_failed`, no retry was scheduled, and the task moved to board recovery. On one staging instance, 15 runs failed this way in 12 minutes. **Expected behavior** No provider work started, so the failure is safe to retry. The run must retry within the bootstrap budget, as it does for a reusable sandbox resume failure or a preparation timeout. **Steps to reproduce** 1. Stop the sandbox provider plugin worker, or let the host retire it. 2. Wake an agent whose run needs that provider during setup. 3. The run fails with `setup_failed`. No retry is scheduled. **Paperclip version or commit** master at 76e43852. **Deployment mode** Authenticated cloud deployment (staging). ## What Changed - Add `PluginWorkerUnavailableError` with the code `plugin_worker_unavailable` (`server/src/services/plugin-worker-unavailable-error.ts`). - The plugin worker manager now rejects with this error when the worker is stopped, crashed, restarting, not running, or not registered. The message text does not change. - Setup failure handling maps this error to `plugin_worker_unavailable`, but only before provider dispatch and only for a legacy run with no native session or runner instance. These are the same guards as for the sibling transient codes. - `isTransientWorkspacePreparationCode` includes the new code, so setup schedules the bounded bootstrap retry. - Legacy execution recovery and the recovery service treat the code like the other bootstrap-only codes. The stranded-task notice has a title and a next action for it. ## Verification - Red: the new `run-completion` test failed on the unchanged code. A fresh refusal got `setup_failed` and no retry. - Green: the same test passes. A fresh refusal gets `plugin_worker_unavailable`, `providerWorkStarted: false` and a bounded retry. A started, untrusted (message-only) or native run keeps `setup_failed` and gets no retry. - New worker manager test: a stopped handle and a missing worker both reject with the typed error. The call is not sent, and `onWorkerSettled` runs once. - `pnpm --filter @paperclipai/server exec tsc --noEmit -p .` passes. - Related suites pass: run-completion, plugin-worker-manager, legacy-execution-recovery, heartbeat-retry-scheduling and recovery (17 files, 439 tests). ## Risks - Low risk. The retry uses the existing bounded bootstrap budget. When the budget is exhausted, the existing board recovery action takes over. - The retry applies only before provider dispatch. A worker that stopped after work started keeps the old behavior. - A wrapper that drops the error `cause` keeps the old `setup_failed` behavior. It does not cause a wrong retry. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 12:33:05 +01:00 · 2026-11-05 09:05:14 -06:00
1 parent e37ab23dbc
commit cb5d7e440c
@@ -1,5 +1,6 @@
 import { readNativeWorkspaceMigration } from "./native-runtime/native-workspace-migration.js";
 import { ReusableSandboxResumeError } from "./reusable-sandbox-resume-error.js";
+import { PluginWorkerUnavailableError } from "./plugin-worker-unavailable-error.js";
 import { HttpError } from "../errors.js";
 import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js";
 import { createOwnedSandbox, recoverSandboxCreations, recoverableTaskCreation } from "./sandbox-creation-operations.js";
@@ -1638,9 +1639,10 @@ function createSandboxEnvironmentDriver(
          );
        }
        if (pluginProvider.state === "worker_unavailable") {
-          throw new Error(
-            `Sandbox provider "${parsed.config.provider}" is installed via plugin "${pluginProvider.resolved.plugin.pluginKey}", but its worker is not running.`,
-          );
+          // No request reached the worker, so setup may retry this refusal
+          // within its bootstrap budget once the worker runs again.
+          throw new PluginWorkerUnavailableError({ pluginId: pluginProvider.resolved.plugin.id, workerStatus: null,
+            message: `Sandbox provider "${parsed.config.provider}" is installed via plugin "${pluginProvider.resolved.plugin.pluginKey}", but its worker is not running.` });
        }

        const workerConfig = {
@@ -23,6 +23,7 @@ import {
 import { WORKSPACE_GIT_SCAN_ERROR_CODES } from "../workspace-git-operation-scheduler.js";
 import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "../remote-workspace-transport.js";
 import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "../reusable-sandbox-resume-error.js";
+import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js";
 import { CONTROLLER_LEASE_LOST_FAILURE_CODE } from "../legacy-controller-lease.js";
 import { randomUUID } from "node:crypto";
 import {
@@ -135,7 +136,7 @@ const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_WAKE_REASON = "transient_failure_retry";
 export function isTransientWorkspacePreparationCode(code: string | null | undefined): boolean {
  return code === WORKSPACE_GIT_SCAN_ERROR_CODES.timeout || code === WORKSPACE_GIT_SCAN_ERROR_CODES.saturated
    || code === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || code === REUSABLE_SANDBOX_RESUME_FAILED_CODE
-    || code === REMOTE_WORKSPACE_RESTORE_FAILED_CODE;
+    || code === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || code === PLUGIN_WORKER_UNAVAILABLE_CODE;
 }

 export const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS =
@@ -341,7 +342,8 @@ function isRetryableInteractionContinuationInfrastructureFailure(
  if (
    run.errorCode === WORKSPACE_VALIDATION_FAILURE_CODE ||
    run.errorCode === "process_lost" ||
-    run.errorCode === CONTROLLER_LEASE_LOST_FAILURE_CODE
+    run.errorCode === CONTROLLER_LEASE_LOST_FAILURE_CODE ||
+    run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE
  ) {
    return true;
  }
@@ -29,6 +29,7 @@ import { ProjectAccessDeniedError } from "../project-access-denial.js";
 import { RemoteWorkspaceOperationError, RemoteWorkspaceOperationTimeoutError, RemoteWorkspacePreparationTimeoutError, RemoteWorkspaceRestoreError } from "../remote-workspace-transport.js";
 import { EnvironmentRunError } from "../environment-run-orchestrator.js";
 import { ReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js";
+import { PluginWorkerUnavailableError } from "../plugin-worker-unavailable-error.js";
 import { createHeartbeatLifecycle, type HeartbeatLifecycleDependencies } from "./run-lifecycle.js";
 import { createHeartbeatRunPreparation, ConfigurationIncompleteFailure } from "./run-preparation.js";
 import { createHeartbeatRunState, getAdapterSessionCodec } from "./run-state.js";
@@ -949,6 +950,35 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)(
    } else expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
  });

+  it.each(["fresh", "started", "untrusted", "retained"] as const)("routes a %s stopped plugin worker refusal through bounded bootstrap recovery", async scenario => {
+    const f = await fixture({ runtimeMode: scenario === "retained" ? "native" : "legacy" });
+    const refusal = new PluginWorkerUnavailableError({ pluginId: "plugin-1", workerStatus: "stopped",
+      message: 'Cannot call "environmentExecute" — worker for "plugin-1" is stopped' });
+    const error = scenario === "untrusted" ? new Error(refusal.message)
+      : new EnvironmentRunError("workspace_realization_failed", refusal.message, { cause: refusal });
+    f.deps.scheduleBoundedRetryForRun.mockResolvedValueOnce({ outcome: "not_scheduled", reason: "Fixture scheduler boundary", issueId: null });
+    await f.completion.failRunSetup(f.setup(error, scenario === "started"));
+    const failed = await read(f.run.id);
+    expect(failed).toMatchObject({ status: "failed", errorCode: scenario === "fresh" ? "plugin_worker_unavailable" : "setup_failed" });
+    if (scenario === "fresh") {
+      expect(failed.resultJson).toMatchObject({ executionRecovery: { kind: "bootstrap", providerWorkStarted: false } });
+      expect(f.deps.scheduleBoundedRetryForRun).toHaveBeenCalledWith(expect.objectContaining({ id: f.run.id }), f.agent);
+    } else expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
+  });
+
+  it("routes a lease acquisition refused by a stopped provider worker through bounded bootstrap recovery", async () => {
+    const f = await fixture();
+    const refusal = new PluginWorkerUnavailableError({ pluginId: "plugin-1", workerStatus: null,
+      message: 'Sandbox provider "daytona" is installed via plugin "paperclip.daytona-sandbox-provider", but its worker is not running.' });
+    const wrapped = new EnvironmentRunError("lease_acquire_failed",
+      `Failed to acquire lease for environment "Paperclip Computer" (sandbox): ${refusal.message}`, { cause: refusal });
+    f.deps.scheduleBoundedRetryForRun.mockResolvedValueOnce({ outcome: "not_scheduled", reason: "Fixture scheduler boundary", issueId: null });
+    await f.completion.failRunSetup(f.setup(wrapped));
+    expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "plugin_worker_unavailable",
+      resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } });
+    expect(f.deps.scheduleBoundedRetryForRun).toHaveBeenCalledWith(expect.objectContaining({ id: f.run.id }), f.agent);
+  });
+
  it.each(["retried", "exhausted", "refused", "started", "in progress"] as const)("names a %s saved workspace restore failure and leaves recovery", async scenario => {
    const f = await fixture();
    const [issue] = await db.insert(issues).values({ companyId: f.run.companyId, title: "Parent task", status: scenario === "in progress" ? "in_progress" : "blocked",
@@ -12,6 +12,7 @@ import { readTransientRecoveryContractFromRun, readHeartbeatRunErrorFamily, isTr
 import { RemoteWorkspacePreparationTimeoutError, REMOTE_WORKSPACE_RESTORE_FAILED_CODE, findRemoteWorkspaceRestoreError } from "../remote-workspace-transport.js";
 import { escalateRemoteWorkspaceRestoreFailure } from "../remote-workspace-restore-recovery.js";
 import { findReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js";
+import { findPluginWorkerUnavailableError, PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js";
 import { type RunSessionOutcome, resolveNextSessionState, normalizeUsageTotals, resolveCacheAdjustedCostUsd, resolveLedgerBiller, resolveLedgerCostStatus, normalizeLedgerBillingType } from "./run-state.js";
 import { readGitConnectionFailure } from "../git-connection-failure.js";
 import { CONFIGURATION_INCOMPLETE_FAILURE_CODE, ConfigurationIncompleteFailure } from "./run-preparation.js";
@@ -40,7 +41,7 @@ import { resolveChatRunPresentationAuthorizationReason } from "../chat-run-publi
 import { hasChatRunOwnedProviderInteraction } from "../chat-interaction-arbitration.js";
 import { readAcceptedTaskContinuationSummary } from "../heartbeat-run-summary.js";
 import { SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON } from "../recovery/stranded-notice.js";
-import { MAX_TURN_CONTINUATION_RETRY_REASON } from "../../modules/run-dispatch/index.js";
+import { MAX_TURN_CONTINUATION_RETRY_REASON, isResolvedInteractionContinuationWakeContext } from "../../modules/run-dispatch/index.js";
 import { redactCurrentUserText } from "../../log-redaction.js";
 import type { createHeartbeatLifecycle } from "./run-lifecycle.js";
 import type { createHeartbeatRunPreparation } from "./run-preparation.js";
@@ -1370,11 +1371,17 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
    // on exhaustion, the board gets a named recovery action.
    const remoteWorkspaceRestoreFailure = !providerDispatchStarted && run.runtimeMode === "legacy"
      && run.nativeSessionId === null && run.runnerInstanceId === null ? findRemoteWorkspaceRestoreError(outerErr) : null;
+    // A plugin worker that was not running refused the call before sending it,
+    // so no provider work or plugin side effect started. Retry within the
+    // bootstrap budget; the worker is restarted or replaced meanwhile.
+    const pluginWorkerUnavailableFailure = !providerDispatchStarted && run.runtimeMode === "legacy"
+      && run.nativeSessionId === null && run.runnerInstanceId === null ? findPluginWorkerUnavailableError(outerErr) : null;
    const setupFailureErrorCode =
      (controllerLeaseLost ? CONTROLLER_LEASE_LOST_FAILURE_CODE : null) ??
      remoteWorkspacePreparationTimeout?.code ??
      reusableSandboxResumeFailure?.code ??
      remoteWorkspaceRestoreFailure?.code ??
+      pluginWorkerUnavailableFailure?.code ??
      workspaceGitScanFailure?.code ??
      workspaceValidationSetupFailure?.code ??
      configurationIncompleteSetupFailure?.code ??
@@ -1532,7 +1539,11 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
            });
          }
        } else if (livenessRun.errorCode !== PROJECT_ACCESS_DENIED) {
-          await (isTransientWorkspacePreparationCode(livenessRun.errorCode)
+          // An accepted interaction keeps its own continuation retry, which
+          // carries the interaction and escalates on exhaustion.
+          const interactionOwnsRetry = livenessRun.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE
+            && isResolvedInteractionContinuationWakeContext(livenessRun.contextSnapshot);
+          await (isTransientWorkspacePreparationCode(livenessRun.errorCode) && !interactionOwnsRetry
            ? scheduleBoundedRetryForRun(livenessRun, failedAgent)
            : scheduleInteractionContinuationInfrastructureRetryIfEligible(livenessRun, failedAgent)
          ).catch((retryError) => {
@@ -10,7 +10,7 @@ const stopped = {
  },
 };

-it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed"])("does not invent unknown provider actions after exhausted %s bootstrap retries", (errorCode) => {
+it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable"])("does not invent unknown provider actions after exhausted %s bootstrap retries", (errorCode) => {
  const run = { runtimeMode: "legacy", status: "failed", errorCode, scheduledRetryAttempt: 2,
    resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } };
  expect(legacyExecutionNeedsReconciliation(run)).toBe(false);
@@ -8,6 +8,7 @@ import { isPreDispatchReviewWaitVerified } from "./pre-dispatch-review-wait.js";
 import { isPreDispatchExecutionWaitVerified } from "./pre-dispatch-execution-wait.js";
 import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "./remote-workspace-transport.js";
 import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "./reusable-sandbox-resume-error.js";
+import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "./plugin-worker-unavailable-error.js";
 import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
 import { normalizeMaxTurnStopReason } from "./heartbeat-stop-metadata.js";
 import { claimedAdapterType, hasConversationContinuationPolicy } from "./conversation-continuation.js";
@@ -78,7 +79,7 @@ export function legacyExecutionNeedsReconciliation(
  // that the bootstrap evidence proves never started. Keep unknown outcomes held.
  if ((run.errorCode === "workspace_git_scan_timeout" || run.errorCode === "workspace_git_scan_saturated"
      || run.errorCode === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || run.errorCode === REUSABLE_SANDBOX_RESUME_FAILED_CODE
-      || run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE) &&
+      || run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE) &&
      evidence?.kind === "bootstrap" && evidence.providerWorkStarted === false) return false;
  if (executionFailureRetryCount(run) >= 2) return true;
  return !(
@@ -1,5 +1,6 @@
 import { pluginIdleManifestGeneration, type IdlePluginCatalogEntry } from "./plugin-idle-catalog.js";
 import { createHash, randomUUID } from "node:crypto";
+import { PluginWorkerUnavailableError } from "./plugin-worker-unavailable-error.js";
 /**
 * PluginWorkerManager — spawns and manages out-of-process plugin worker child
 * processes, routes JSON-RPC 2.0 calls over stdio, and handles lifecycle
@@ -3590,9 +3591,10 @@ export function createPluginWorkerHandle(
      if (!childProcess?.stdin?.writable) {
        onWorkerSettled?.();
        reject(
-          new Error(
-            `Cannot call "${method}" — worker for "${pluginId}" is not running`,
-          ),
+          // The status can still read "running" while the child's stdin is
+          // closed, so no status is reported for this refusal.
+          new PluginWorkerUnavailableError({ pluginId, workerStatus: null,
+            message: `Cannot call "${method}" — worker for "${pluginId}" is not running` }),
        );
        return;
      }
@@ -3833,9 +3835,8 @@ export function createPluginWorkerHandle(
      if (status !== "running" && status !== "starting") {
        onWorkerSettled?.();
        return Promise.reject(
-          new Error(
-            `Cannot call "${method}" — worker for "${pluginId}" is ${status}`,
-          ),
+          new PluginWorkerUnavailableError({ pluginId, workerStatus: status,
+            message: `Cannot call "${method}" — worker for "${pluginId}" is ${status}` }),
        );
      }
      return callInternal(method, params, timeoutMs, executeLogSink, onWorkerSettled, startupContextId, observeResult);
@@ -4226,7 +4227,8 @@ export function createPluginWorkerManager(
      if (!handle) {
        onWorkerSettled?.();
        return Promise.reject(
-          new Error(`No worker registered for plugin "${pluginId}"`),
+          new PluginWorkerUnavailableError({ pluginId, workerStatus: null,
+            message: `No worker registered for plugin "${pluginId}"` }),
        );
      }
      let finishInspection: (() => void) | undefined;
@@ -4256,7 +4258,8 @@ export function createPluginWorkerManager(
      const handle = workers.get(pluginId);
      if (!handle) {
        return Promise.reject(
-          new Error(`No worker registered for plugin "${pluginId}"`),
+          new PluginWorkerUnavailableError({ pluginId, workerStatus: null,
+            message: `No worker registered for plugin "${pluginId}"` }),
        );
      }
      return handle.openLoginPtySession(input);
@@ -0,0 +1,30 @@
+/**
+ * Setup failure code for a call that reached a plugin worker which was not
+ * running (stopped, crashed, restarting, or not registered). No request was
+ * sent to the worker, so the run is retried with the bounded bootstrap budget
+ * once the worker is available again.
+ */
+export const PLUGIN_WORKER_UNAVAILABLE_CODE = "plugin_worker_unavailable";
+
+/** A host call was refused because its plugin worker was not running. */
+export class PluginWorkerUnavailableError extends Error {
+  readonly code = PLUGIN_WORKER_UNAVAILABLE_CODE;
+  readonly pluginId: string;
+  /** The worker status at refusal, or null when no worker was registered. */
+  readonly workerStatus: string | null;
+
+  constructor(input: { pluginId: string; workerStatus: string | null; message: string }) {
+    super(input.message);
+    this.name = "PluginWorkerUnavailableError";
+    this.pluginId = input.pluginId;
+    this.workerStatus = input.workerStatus;
+  }
+}
+
+/** Find a worker refusal through the orchestrator's error wrappers. */
+export function findPluginWorkerUnavailableError(error: unknown): PluginWorkerUnavailableError | null {
+  for (let cause = error, depth = 0; cause && depth < 4; cause = (cause as { cause?: unknown }).cause, depth++) {
+    if (cause instanceof PluginWorkerUnavailableError) return cause;
+  }
+  return null;
+}
@@ -5,7 +5,7 @@ const run = (errorCode: string | null) =>
  ({ errorCode } as unknown as Parameters<typeof classifyContinuationFailure>[0]);

 describe("pause durability: continuation retry classification", () => {
-  it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "workspace_git_scan_failed", "workspace_git_scan_output_limit", "workspace_git_scan_cancelled", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed"])("does not grant %s another recovery budget", (code) => {
+  it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "workspace_git_scan_failed", "workspace_git_scan_output_limit", "workspace_git_scan_cancelled", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable"])("does not grant %s another recovery budget", (code) => {
    expect(classifyContinuationFailure(run(code))).toMatchObject({ kind: "non_retryable", maxAttempts: 0 });
  });
  it("agent_paused is retryable so work resumes (Option A: Resume Continues Work)", () => {
@@ -9,6 +9,7 @@ import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js"
 import { boundRunIssueId } from "../run-task-binding.js";
 import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "../remote-workspace-transport.js";
 import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "../reusable-sandbox-resume-error.js";
+import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js";
 import { recoveryRunContextColumn, recoveryRunVersionColumn } from "./run-context-read.js";
 import { recoveryRunResultColumn } from "./run-result-read.js";
 import { readLegacyDispositionSource } from "./legacy-disposition-read.js";
@@ -559,6 +560,7 @@ const NON_RETRYABLE_CONTINUATION_ERROR_CODES = new Set<string>([
  REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE,
  REUSABLE_SANDBOX_RESUME_FAILED_CODE,
  REMOTE_WORKSPACE_RESTORE_FAILED_CODE,
+  PLUGIN_WORKER_UNAVAILABLE_CODE,
  "low_trust_isolation_unavailable",
  "low_trust_requires_isolated_workspace",
  "low_trust_boundary_mismatch",
@@ -136,6 +136,17 @@ describe("buildStrandedRecoveryEscalationNotice", () => {
    expect(notice.body).not.toContain("retried continuation");
    expect(JSON.stringify(notice.metadata)).toContain("repository access and server load");
  });
+  it("names an environment plugin that was not running after its bounded retries", () => {
+    const notice = buildStrandedRecoveryEscalationNotice({
+      seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "in_progress" }),
+      recoveryActionId: "worker-recovery",
+      recoveryOwner: null,
+      sourceRun: { id: "failed-run", status: "failed", errorCode: "plugin_worker_unavailable" },
+    });
+    expect(notice.presentation.title).toBe("Environment plugin was not running");
+    expect(notice.body).toContain("before the agent started");
+    expect(JSON.stringify(notice.metadata)).toContain("worker status");
+  });
  const actionId = "6a2f8e64-6f5e-4b58-b7fd-111111111111";
  const owner = { id: "9b1c2d3e-4f50-4a61-8b72-222222222222", name: "CTO" };
  const sourceRun = {
@@ -56,6 +56,7 @@ const WORKSPACE_SCAN_NOTICES: Record<string, { title: string; nextAction: string
  workspace_git_scan_timeout: { title: "Workspace scan timed out", nextAction: "Check repository access and server load, then retry the task." },
  remote_workspace_prepare_timeout: { title: "Workspace setup timed out", nextAction: "Inspect the failed run and check repository access and sandbox load before retrying the task." },
  remote_workspace_restore_failed: { title: "Saved workspace could not be restored", nextAction: REMOTE_WORKSPACE_RESTORE_NEXT_ACTION },
+  plugin_worker_unavailable: { title: "Environment plugin was not running", nextAction: "Check the environment plugin's worker status, then retry the task once it is running." },
  reusable_sandbox_resume_failed: { title: "Saved sandbox could not be resumed", nextAction: "Inspect the failed run and the sandbox provider; the saved sandbox was kept. Retry the task once the provider can start it." },
  workspace_git_scan_saturated: { title: "Workspace scan queue is full", nextAction: "Check server load and the workspace scan queue, then retry the task." },
  workspace_git_scan_output_limit: { title: "Workspace scan exceeded its limit", nextAction: "Check the repository size and workspace scan output limit before retrying the task." },