From 8c761abae39730f4d97b148c63a7f0e059a9f05b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 22:49:23 -0600 Subject: [PATCH] fix(environments): wait for a re-activating sandbox provider before refusing a lease resume (#21020) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a controller restart, a run whose remote Runner kept running reattaches. First it resumes its exact sandbox lease through the sandbox provider plugin (`resumeRunLease`). > - #20982 made a plugin whose worker `initialize` timed out retry its activation, so the plugin comes back seconds later instead of staying `error`. > - Chaos cell CR6 (#20983) shows the gap that remains. The reattaching run asks for its lease during those seconds and gets `PluginWorkerUnavailableError`. The setup retry covers that error only for legacy runs, so the native restart-recovery run fails and needs Retry. > - This pull request makes the exact-lease resume wait, for a bounded time, for the worker of a plugin that is `ready` (re-activating) or recovering from activation timeouts. > - The benefit is that a restarted run rides through a short plugin re-activation instead of failing into Retry. ## Linked Issues or Issue Description Refs #20982, #20983 **What happened?** CR6 after #20982 (`w-20261112T041333-s9531`) passed in part. The plugin re-activated 5 s after its `initialize` timeout, but both tasks finished with `R3:retry_needed`. Run `a6b3f0f6` resumed its lease at 22:15:35, inside that window. It failed with `setup_failed` / `plugin_worker_unavailable` and then `agent_stop_unconfirmed`, because `run-completion.ts` retries `PluginWorkerUnavailableError` only for legacy runs (`runtimeMode === "legacy" && nativeSessionId === null && runnerInstanceId === null`). **Expected behavior** The resume waits for the plugin's worker while the plugin re-activates. It resumes the same lease when the worker comes back. It still refuses, as `PluginWorkerUnavailableError`, after the bound. **Steps to reproduce** 1. Run chaos cell CR6: a K1 restart, then a stall of the replacement's plugin worker during activation. 2. The reattaching run fails and needs Retry, although the plugin recovers within seconds. **Paperclip version or commit** master at f1aef0cb58. ## What Changed - `environment-runtime.ts`, `resumeRunLease`: the worker is not running, but the plugin is `ready` or its activation-timeout recovery is in flight. In that case the resume polls `isRunning`, every `pluginWorkerReadyPollMs`, for up to `pluginWorkerResumeWaitMs` (default 60 s). That bound covers the first two activation retries (5 s, then 15 s). When the worker runs, the normal exact resume goes on. After the bound it refuses with `PluginWorkerUnavailableError`, as before. - Nothing changes when the worker is running: no wait, no added latency on the normal path. A plugin that is `disabled`, `uninstalled`, or `error` for a reason other than a timeout is not waited on. - `doc/plugins/PLUGIN_SPEC.md` §12 describes the wait. ## Verification - Red on master: `npx vitest run src/__tests__/environment-runtime.test.ts -t "re-activating plugin's worker comes back"` (in `server/`) fails with `PluginWorkerUnavailableError`. - Green: the test passes. The worker comes up on the fourth `isRunning` check, and the lease is resumed with one `environmentResumeLease` call. - The existing refusal test, now with a 50 ms bound, still refuses with no provider call. - `environment-runtime.test.ts` passes (205 tests), and `tsc --noEmit` passes for `server`. ## Risks - A restart-recovery run whose provider plugin never comes back now waits up to 60 s more before it fails. It fails the same way as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, code search). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- doc/plugins/PLUGIN_SPEC.md | 4 +++- .../src/__tests__/environment-runtime.test.ts | 19 ++++++++++++++++++- server/src/services/environment-runtime.ts | 9 ++++++++- 3 files changed, 29 insertions(+), 3 deletions(-) diff --git a/doc/plugins/PLUGIN_SPEC.md b/doc/plugins/PLUGIN_SPEC.md index 715a0476d5a..e6d96c59d44 100644 --- a/doc/plugins/PLUGIN_SPEC.md +++ b/doc/plugins/PLUGIN_SPEC.md @@ -538,7 +538,9 @@ expected current status, so a disable or uninstall during a retry wins. Server shutdown cancels pending retries and waits for running ones. Runs that need the plugin see a worker that is not running, which setup retries within its bounded budget. After the last retry, the plugin becomes `error` with that same prefix. A run that -needs the plugin then re-activates it on demand, at most once a minute. Any +needs the plugin then re-activates it on demand, at most once a minute. A run that resumes its exact sandbox lease after a +restart waits up to 60 seconds for a re-activating sandbox provider's worker +before it reports the worker unavailable. Any other activation failure, such as a bad manifest, a worker that exits during `initialize`, or a version mismatch, makes the plugin `error` at once. The next server boot retries it once. diff --git a/server/src/__tests__/environment-runtime.test.ts b/server/src/__tests__/environment-runtime.test.ts index 509646dc29b..57fae63cda3 100644 --- a/server/src/__tests__/environment-runtime.test.ts +++ b/server/src/__tests__/environment-runtime.test.ts @@ -4036,13 +4036,30 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { const call = vi.fn(); const requestActivationTimeoutRecovery = vi.fn(() => true); const workerManager = { isRunning: () => false, call, getWorker: () => undefined, requestActivationTimeoutRecovery } as unknown as PluginWorkerManager; - const error = await environmentRuntimeService(db, { pluginWorkerManager: workerManager }) + const error = await environmentRuntimeService(db, { pluginWorkerManager: workerManager, pluginWorkerResumeWaitMs: 50, pluginWorkerReadyPollMs: 5 }) .resumeRunLease({ environment: seeded.environment, lease }).catch(caught => caught); expect(error).toBeInstanceOf(PluginWorkerUnavailableError); expect(call).not.toHaveBeenCalled(); expect(requestActivationTimeoutRecovery).toHaveBeenCalledTimes(status === "error" ? 1 : 0); }); + it("resumes the exact lease once a re-activating plugin's worker comes back within the wait", async () => { + const seeded = await seedReusablePluginSandboxLease("paperclip_runner"); + await environmentService(db).releaseLease(seeded.reusableLease.id, "released", { cleanupStatus: "success" }); + const lease = (await environmentService(db).getLeaseById(seeded.reusableLease.id))!; + let checks = 0; + const call = vi.fn(async (_id: string, method: string) => { + if (method !== "environmentResumeLease") throw new Error("Only exact resume is permitted"); + return { providerLeaseId: lease.providerLeaseId, metadata: { remoteCwd: "/workspace" } }; + }); + const workerManager = { isRunning: () => ++checks > 3, call, + getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] }), + } as unknown as PluginWorkerManager; + await expect(environmentRuntimeService(db, { pluginWorkerManager: workerManager, pluginWorkerResumeWaitMs: 5_000, pluginWorkerReadyPollMs: 5 }) + .resumeRunLease({ environment: seeded.environment, lease })).resolves.toMatchObject({ providerLeaseId: lease.providerLeaseId }); + expect(call).toHaveBeenCalledOnce(); + }); + it.each([ { timeoutMs: 300_000, rpcTimeoutMs: 330_000 }, { timeoutMs: 1234, rpcTimeoutMs: 31_234 }, diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts index a567bcffcc8..62294857264 100644 --- a/server/src/services/environment-runtime.ts +++ b/server/src/services/environment-runtime.ts @@ -782,6 +782,7 @@ export type ProviderResourceDisposition = | "destroy"; const DEFAULT_PLUGIN_SANDBOX_WORKER_READY_TIMEOUT_MS = 5_000; +const DEFAULT_PLUGIN_SANDBOX_WORKER_RESUME_WAIT_MS = 60_000; const DEFAULT_PLUGIN_SANDBOX_WORKER_READY_POLL_MS = 100; const DEFAULT_DEFERRED_ORPHAN_CLEANUP_BUFFER_LIMIT = 256; @@ -1338,6 +1339,7 @@ function createSandboxEnvironmentDriver( readOnlyInspection?: boolean; pluginWorkerReadyTimeoutMs?: number; pluginWorkerReadyPollMs?: number; + pluginWorkerResumeWaitMs?: number; deferredOrphanCleanupBufferLimit?: number; orphanCleanupSpool?: SandboxOrphanCleanupSpool; orphanCleanupSpoolDir?: string; @@ -1346,6 +1348,7 @@ function createSandboxEnvironmentDriver( const pluginWorkerManager = options.pluginWorkerManager; const pluginWorkerReadyTimeoutMs = options.pluginWorkerReadyTimeoutMs ?? DEFAULT_PLUGIN_SANDBOX_WORKER_READY_TIMEOUT_MS; const pluginWorkerReadyPollMs = options.pluginWorkerReadyPollMs ?? DEFAULT_PLUGIN_SANDBOX_WORKER_READY_POLL_MS; + const pluginWorkerResumeWaitMs = options.pluginWorkerResumeWaitMs ?? DEFAULT_PLUGIN_SANDBOX_WORKER_RESUME_WAIT_MS; // The buffer holds at least one orphan, so a zero or negative override never // drops every buffered record. const deferredOrphanCleanupBufferLimit = Math.max( @@ -2702,7 +2705,9 @@ function createSandboxEnvironmentDriver( && !pluginWorkerManager.isRunning(pluginId)) { const plugin = await resolvePluginSandboxProviderDriverById({ db, pluginId, driverKey: providerKey }); if (plugin && (plugin.plugin.status === "ready" || pluginWorkerManager.requestActivationTimeoutRecovery?.(plugin.plugin))) { - throw new PluginWorkerUnavailableError({ pluginId, workerStatus: pluginWorkerManager.getWorker(pluginId)?.status ?? null, + const deadline = Date.now() + Math.max(0, pluginWorkerResumeWaitMs); + while (!pluginWorkerManager.isRunning(pluginId) && Date.now() < deadline) await delay(Math.max(1, pluginWorkerReadyPollMs)); + if (!pluginWorkerManager.isRunning(pluginId)) throw new PluginWorkerUnavailableError({ pluginId, workerStatus: pluginWorkerManager.getWorker(pluginId)?.status ?? null, message: `Sandbox provider "${providerKey}" is installed via plugin "${plugin.plugin.pluginKey}", but its worker is not running.` }); } } @@ -3786,6 +3791,7 @@ export function environmentRuntimeService( readOnlyInspection?: boolean; pluginWorkerReadyTimeoutMs?: number; pluginWorkerReadyPollMs?: number; + pluginWorkerResumeWaitMs?: number; deferredOrphanCleanupBufferLimit?: number; orphanCleanupSpool?: SandboxOrphanCleanupSpool; orphanCleanupSpoolDir?: string; @@ -3803,6 +3809,7 @@ export function environmentRuntimeService( readOnlyInspection: options.readOnlyInspection, pluginWorkerReadyTimeoutMs: options.pluginWorkerReadyTimeoutMs, pluginWorkerReadyPollMs: options.pluginWorkerReadyPollMs, + pluginWorkerResumeWaitMs: options.pluginWorkerResumeWaitMs, deferredOrphanCleanupBufferLimit: options.deferredOrphanCleanupBufferLimit, orphanCleanupSpool: options.orphanCleanupSpool, orphanCleanupSpoolDir: options.orphanCleanupSpoolDir,