fix(server): retry setup when the plugin worker was not running (#19249)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote agent runs call a sandbox provider plugin, such as Daytona,
through a plugin worker process.
> - Sometimes that worker is not running. It can be stopped, restarting
after a crash, or retired by the host.
> - Then the host refuses the call before it sends anything to the
worker.
> - Setup recorded this refusal as a generic `setup_failed`. It
scheduled no retry, so the task became stranded and needed a person.
> - This pull request types the refusal and gives it its own transient
setup code. The bounded bootstrap retry then runs.
> - The benefit is that the task continues on its own when the worker is
available again. Users do not see a blocked task.

## Linked Issues or Issue Description

Refs #19150 (removed one trigger for the worker retirement)
Refs #19245 (restarts a worker after a fail-closed retirement)

**What happened?**
A plugin worker stopped. Each run that then called `environmentExecute`
during setup failed with `Cannot call "environmentExecute" — worker for
"<plugin>" is stopped`. The run got `setup_failed`, no retry was
scheduled, and the task moved to board recovery. On one staging
instance, 15 runs failed this way in 12 minutes.

**Expected behavior**
No provider work started, so the failure is safe to retry. The run must
retry within the bootstrap budget, as it does for a reusable sandbox
resume failure or a preparation timeout.

**Steps to reproduce**
1. Stop the sandbox provider plugin worker, or let the host retire it.
2. Wake an agent whose run needs that provider during setup.
3. The run fails with `setup_failed`. No retry is scheduled.

**Paperclip version or commit**
master at 76e43852.

**Deployment mode**
Authenticated cloud deployment (staging).

## What Changed

- Add `PluginWorkerUnavailableError` with the code
`plugin_worker_unavailable`
(`server/src/services/plugin-worker-unavailable-error.ts`).
- The plugin worker manager now rejects with this error when the worker
is stopped, crashed, restarting, not running, or not registered. The
message text does not change.
- Setup failure handling maps this error to `plugin_worker_unavailable`,
but only before provider dispatch and only for a legacy run with no
native session or runner instance. These are the same guards as for the
sibling transient codes.
- `isTransientWorkspacePreparationCode` includes the new code, so setup
schedules the bounded bootstrap retry.
- Legacy execution recovery and the recovery service treat the code like
the other bootstrap-only codes. The stranded-task notice has a title and
a next action for it.

## Verification

- Red: the new `run-completion` test failed on the unchanged code. A
fresh refusal got `setup_failed` and no retry.
- Green: the same test passes. A fresh refusal gets
`plugin_worker_unavailable`, `providerWorkStarted: false` and a bounded
retry. A started, untrusted (message-only) or native run keeps
`setup_failed` and gets no retry.
- New worker manager test: a stopped handle and a missing worker both
reject with the typed error. The call is not sent, and `onWorkerSettled`
runs once.
- `pnpm --filter @paperclipai/server exec tsc --noEmit -p .` passes.
- Related suites pass: run-completion, plugin-worker-manager,
legacy-execution-recovery, heartbeat-retry-scheduling and recovery (17
files, 439 tests).

## Risks

- Low risk. The retry uses the existing bounded bootstrap budget. When
the budget is exhausted, the existing board recovery action takes over.
- The retry applies only before provider dispatch. A worker that stopped
after work started keeps the old behavior.
- A wrapper that drops the error `cause` keeps the old `setup_failed`
behavior. It does not cause a wrong retry.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/__tests__/environment-runtime.test.ts

@@ -1,5 +1,6 @@
 import { JsonRpcCallError, environmentResumeErrorData } from "@paperclipai/plugin-sdk";
 import { collectRunFailureDiagnostics } from "../services/run-failure-diagnostics.js";
+import { PluginWorkerUnavailableError } from "../services/plugin-worker-unavailable-error.js";
 import { createSandboxOrphanCleanupSpool } from "../services/sandbox-orphan-cleanup-spool.js";
 import { idleWorkSnapshot } from "../services/task-admission.js";
 import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
@@ -1084,8 +1085,9 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
       pluginWorkerManager: offlineManager,
       pluginWorkerReadyTimeoutMs: 0,
     });
-    await expect(runtimeWithStoppedWorker.acquireRunLease(input))
-      .rejects.toThrow("its worker is not running");
+    const refusal = await runtimeWithStoppedWorker.acquireRunLease(input).catch(error => error);
+    expect(refusal).toBeInstanceOf(PluginWorkerUnavailableError);
+    expect((refusal as Error).message).toContain("its worker is not running");
     expect(offlineManager.call).not.toHaveBeenCalled();
   });
 

server/src/__tests__/heartbeat-process-recovery.test.ts

@@ -5594,7 +5594,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
     const failedRun = runs?.find((row) => row.id === runId);
     const retryRun = runs?.find((row) => row.id !== runId);
     expect(failedRun?.status).toBe("failed");
-    expect(failedRun?.errorCode).toBe("setup_failed");
+    expect(failedRun?.errorCode).toBe("plugin_worker_unavailable");
     expect(failedRun?.error).toContain("worker is not running");
     expect(retryRun).toMatchObject({
       status: "scheduled_retry",

server/src/__tests__/plugin-worker-manager.test.ts

@@ -27,6 +27,7 @@ vi.mock("../middleware/logger.js", () => {
 });
 
 import { logger } from "../middleware/logger.js";
+import { PluginWorkerUnavailableError, PLUGIN_WORKER_UNAVAILABLE_CODE } from "../services/plugin-worker-unavailable-error.js";
 import { withRunExecutionTiming } from "../services/execution-preparation-timing.js";
 import {
   appendStderrExcerpt,
@@ -80,6 +81,24 @@ describe("worker configuration completion", () => {
   });
 });
 
+describe("plugin worker unavailable refusals", () => {
+  it("refuses calls to a stopped or missing worker with a typed, unsent error", async () => {
+    const handle = createPluginWorkerHandle("test.plugin", {
+      entrypointPath: DELAYED_WORKER_ENTRYPOINT, manifest: TEST_MANIFEST, config: {},
+      instanceInfo: { instanceId: "instance-1", hostVersion: "1.0.0" }, apiVersion: 1, hostHandlers: {},
+    });
+    const settled = vi.fn();
+    const stopped = await handle.call("environmentExecute", {} as never, undefined, undefined, settled).catch(error => error);
+    expect(stopped).toBeInstanceOf(PluginWorkerUnavailableError);
+    expect(stopped).toMatchObject({ code: PLUGIN_WORKER_UNAVAILABLE_CODE, pluginId: "test.plugin", workerStatus: "stopped",
+      message: 'Cannot call "environmentExecute" — worker for "test.plugin" is stopped' });
+    expect(settled).toHaveBeenCalledTimes(1);
+    const missing = await createPluginWorkerManager().call("missing.plugin", "environmentExecute", {} as never).catch(error => error);
+    expect(missing).toBeInstanceOf(PluginWorkerUnavailableError);
+    expect(missing).toMatchObject({ workerStatus: null, message: 'No worker registered for plugin "missing.plugin"' });
+  });
+});
+
 describe("resolveRpcCallTimeoutMs", () => {
   const MAX_RPC_TIMEOUT_MS = 15 * 60 * 1_000;
   const MAX_NODE_TIMER_TIMEOUT_MS = 2_147_483_647;