From 38944cff1a195d22cc2980b05d982e772b00672a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 10 Nov 2026 13:51:10 -0600 Subject: [PATCH] fix(environments): retry and safely replace a reusable sandbox that will not resume (#20626) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs on a reusable (warm) cloud sandbox resumes the same sandbox on the next wake > - When a follow-up arrives while the provider is auto-stopping or archiving the idle sandbox, the resume can fail > - Each failed attempt waited the provider's whole five-minute start deadline and then failed with `reusable_sandbox_resume_failed`, even when the workspace was saved and nothing lived only on the sandbox > - Two such runs in a row made a user wait for many minutes and recorded two failed runs > - This pull request bounds the resume, retries a state change inside the run, and replaces the sandbox from the saved snapshot when that is safe > - The benefit is that recovery works automatically and the first turn is not delayed by minutes ## Linked Issues or Issue Description **What happened?** Two consecutive runs failed before provider dispatch with: `Reusable sandbox lease "" could not be resumed (cause=provider_error); the lease was preserved and no replacement was created.` Each run took about 5.5 minutes (320s and 330s) in preparation. The third run resumed the same sandbox from `stopped` in 4.7 seconds after the provider settled. The previous run had ended with a clean final save and a verified harness backup. **Steps to reproduce** 1. Let a task's reusable sandbox go idle so that the provider auto-stops or archives it. 2. Send a follow-up while the provider still reports the sandbox as changing state. 3. The Daytona worker waits in `ensureSandboxStarted` until its 300s start deadline and throws "is still at its start deadline". The host does not classify that as transient and it has no safe fallback, so the run fails. **Expected behavior** The run retries briefly. If the sandbox still cannot start and nothing lives only on it, the run continues on a replacement sandbox that restores the saved workspace. **Root cause** `server/src/services/environment-runtime.ts` (reusable resume block, about line 1820): the resume deadline was the provider's full start deadline (300s plus RPC margin), the transient classifier did not recognize a state transition, and the catch always threw `ReusableSandboxResumeError`. #20212 waits out a state change inside the same 300s start deadline, so it covers the start but not the time budget or the fallback. ## What Changed - Add `server/src/services/reusable-sandbox-resume-replacement.ts`: - `REUSABLE_SANDBOX_RESUME_BUDGET_MS` (90s): the total resume time for one run, retries included. - `sandboxStateTransitionFailure`: "state change in progress" and "is still at its start deadline" are transient. - `retireUnresumableReusableSandboxLease`: replace only when the lease is `released`/`retained` with no live owner, the task workspace is clean with a committed saved snapshot and no other writer, and a `paperclip_runner` session has a verified lease-bound harness backup. Access refusals (401/403) and an unavailable worker stay strict. After the replacement lease (which carries the harness backup) is recorded, the old lease moves to `pending_cleanup` (reason `resume_failed`) in one guarded write, so the existing cleanup sweep destroys the old sandbox and no provider delete runs before the first turn. A failed creation or a restart before that write leaves the old lease `retained` for the next attempt. - `environment-runtime.ts`: cap the resume deadline at the budget, treat a state transition as transient, and call the replacement policy before failing. A replacement records `sandboxLeaseAcquisition: { outcome: "replacement", reason: "resume_failed" }`; workspace preparation then restores the saved snapshot. - `remote-workspace-transport.ts`: add the closed failure label `sandbox_state_transition`, so a strict failure names the cause instead of `provider_error`. - `doc/execution-semantics.md`: document the budget, the in-run retry, the replacement conditions, and the strict cases. - Update the two slow-start tests: the resume RPC timeout is now 90s instead of 330s/150s. A 61s start still resumes the same sandbox. ## Verification - Red before the fix (new file `server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`, staging shape: lease created by the real acquisition path, then retained after a clean save with a backup stamp, 300s provider deadline, worker refusal with no status): 3 tests failed with `could not be resumed (cause=provider_error); the lease was preserved and no replacement was created.` - Green after the fix: 10/10 pass, including a case where the replacement creation fails and the next run replaces the sandbox with the backup carried forward. The safe case replaces the sandbox, keeps every resume RPC at or below 90s, makes no synchronous destroy call, and moves the old lease to `pending_cleanup`. A single state-change refusal resumes the same sandbox inside the run. Six strict cases keep the lease and fail with `cause=sandbox_state_transition`: unsaved work, no saved version, an uncommitted saved version, a live owner, no verified backup, and a 403. - `pnpm exec vitest run src/__tests__/environment-runtime.test.ts` (197 pass), `heartbeat-retry-scheduling`, `legacy-execution-recovery`, `remote-workspace-transport` (251 pass). - `tsc -p server/tsconfig.json --noEmit` passes. - `pnpm verify` exits 2 (a doc change is a global CI input). Required CI supplies the broad coverage. - Not run: a live Daytona run on staging. The provider message is kept off the run record, so the exact transition state on staging (stopping or archiving) is inferred from timing, not observed. ## Risks - A strict resume now gives up after 90s instead of 300s. The provider start continues in the background, and the bounded bootstrap retry resumes the same sandbox. A stopped sandbox resumes in seconds and an archived one in about 25s on Daytona, so 90s leaves headroom. - A replacement leaves the old sandbox to the cleanup sweep. If the old start finishes later, the sweep still destroys it. The guarded write only claims a `released`/`retained` lease with no live owner. - Upgrade safety: no new persisted fields are required. Leases saved by older releases are read with the same rule; one without a verified backup stays strict. The replacement reason `resume_failed` already exists in the acquisition outcome. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 35 ++- ...untime-reusable-resume-replacement.test.ts | 290 ++++++++++++++++++ .../src/__tests__/environment-runtime.test.ts | 6 +- server/src/services/environment-runtime.ts | 48 ++- .../services/remote-workspace-transport.ts | 2 + .../services/reusable-sandbox-resume-error.ts | 5 +- .../reusable-sandbox-resume-replacement.ts | 143 +++++++++ 7 files changed, 507 insertions(+), 22 deletions(-) create mode 100644 server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts create mode 100644 server/src/services/reusable-sandbox-resume-replacement.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index fcb3d4d6886..abe00a968bd 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1380,14 +1380,33 @@ identity, unknown failures, active leases, and incomplete cleanup still require their normal reconciliation and termination proof. Unknown historical setup errors remain outside this policy, and generic recovery cannot renew its exhausted budget. -A preserved reusable sandbox that the provider refuses to resume before provider -dispatch uses the same bounded bootstrap budget (`reusable_sandbox_resume_failed`). -A common cause is a follow-up that arrives while the provider is auto-stopping an -idle sandbox. The retry inherits the source wake, so a reply is delivered once, -and it resumes the same recorded lease. A refusal is not proof that the sandbox is -lost, so the retry never destroys it or creates a replacement. Only an explicit -provider not-found or a workspace identity mismatch with a verified backup permits -replacement. The run records the refusal by closed labels only (cause label, HTTP +A preserved reusable sandbox is resumed within a bounded budget of 90 seconds per +run (or the provider's configured start deadline when that is shorter), not the +provider's whole start deadline. A stopped Daytona sandbox resumes in seconds and +an archived one in under a minute. Inside that budget the run retries transient +refusals with jittered backoff: HTTP 429 and 5xx, closed connections, timeouts, +and a sandbox that is still changing state (for example "state change in +progress", or still stopping or archiving at the provider's start deadline). A +common cause is a follow-up that arrives while the provider is auto-stopping or +archiving an idle sandbox. + +When the resume still fails, the run replaces the sandbox only when nothing lives +solely on it: the lease is `released` or `retained` and no live run holds it, the +task workspace is clean with a committed saved snapshot, and a native runner +session has a verified, lease-bound harness backup. The run then moves the old +lease to `pending_cleanup` (failure reason `resume_failed`) in one guarded write, +so the cleanup sweep destroys the old sandbox later and no provider delete delays +the first turn. It creates a new sandbox, records the acquisition as +`replacement` with reason `resume_failed`, and workspace preparation restores the +saved snapshot into it. Leases saved by older releases have the same fields and +follow the same rule; one without a verified backup stays strict. + +Otherwise (unsaved work, a live owner, a workspace with no committed saved +version, an access refusal, or an unavailable worker) the refusal keeps the strict +behavior: the run fails with `reusable_sandbox_resume_failed` and uses the bounded +bootstrap budget. The retry inherits the source wake, so a reply is delivered once, +and it resumes the same recorded lease without destroying it. The run records the +refusal by closed labels only (cause label such as `sandbox_state_transition`, HTTP status, errno-style code), never by provider text. On exhaustion, the task gets a specific recovery notice, and generic recovery does not grant another budget. diff --git a/server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts b/server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts new file mode 100644 index 00000000000..d4a802a8b29 --- /dev/null +++ b/server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts @@ -0,0 +1,290 @@ +import { JsonRpcCallError, environmentResumeErrorData } from "@paperclipai/plugin-sdk"; +import { randomUUID } from "node:crypto"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { eq } from "drizzle-orm"; +import { + agents, + companies, + createDb, + environmentLeases, + environments, + executionWorkspaceSnapshots, + executionWorkspaces, + heartbeatRuns, + plugins, + projects, +} from "@paperclipai/db"; +import type { Environment } from "@paperclipai/shared"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { environmentRuntimeService } from "../services/environment-runtime.ts"; +import { environmentService } from "../services/environments.ts"; +import type { PluginWorkerManager } from "../services/plugin-worker-manager.ts"; + +// Durable harness-backup verification has its own tests. Here it is a boundary: +// a lease-bound stamp verifies, a missing stamp (an older release's lease) does not. +vi.mock("../services/native-runtime/native-harness-backup-storage.js", () => ({ + verifyRecoverableNativeHarnessBackup: vi.fn(async (input: { stamp: unknown; providerLeaseId: string }) => + Boolean(input.stamp && (input.stamp as { authorizedProviderLeaseId?: unknown }).authorizedProviderLeaseId === input.providerLeaseId)), +})); + +const externalTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ENVIRONMENT_RUNTIME_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalTestDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; + +// The staging failure: Daytona kept the idle sandbox in a state transition past +// its five-minute start deadline, and the worker reported it with no status. +const STAGING_REFUSAL = "Daytona sandbox 0f6c1e52-3d1a-4b7e-9c2a-5e8f7a1b2c3d is still stopping at its start deadline"; + +describeEmbeddedPostgres("reusable sandbox resume failure replacement", () => { + let stopDb: (() => Promise) | null = null; + let db!: ReturnType; + let connectionString: string; + + beforeAll(async () => { + if (externalTestDatabaseUrl) connectionString = externalTestDatabaseUrl; + else { + const started = await startEmbeddedPostgresTestDatabase("environment-runtime-resume-replacement"); + stopDb = started.cleanup; + connectionString = started.connectionString; + } + }); + beforeEach(() => { db = createDb(connectionString); }); + afterEach(async () => { + vi.restoreAllMocks(); + await db.delete(executionWorkspaceSnapshots); + await db.delete(environmentLeases); + await db.delete(heartbeatRuns); + await db.delete(agents); + await db.delete(environments); + await db.delete(executionWorkspaces); + await db.delete(plugins); + await db.delete(projects); + await db.delete(companies); + await db.$client.end({ timeout: 1 }); + }); + afterAll(async () => { await stopDb?.(); }); + + type Shape = { + leaseStatus?: "retained" | "released"; + previousRunStatus?: "succeeded" | "running"; + dirty?: boolean; + snapshot?: "committed" | "pending" | null; + backupStamp?: boolean; + }; + + async function seed(shape: Shape = {}) { + const companyId = randomUUID(); + const agentId = randomUUID(); + const environmentId = randomUUID(); + const pluginId = randomUUID(); + const previousRunId = randomUUID(); + const runId = randomUUID(); + const projectId = randomUUID(); + const executionWorkspaceId = randomUUID(); + const providerLeaseId = "0f6c1e52-3d1a-4b7e-9c2a-5e8f7a1b2c3d"; + // The staging environment's provider settings, including its 300s start deadline. + const providerConfig = { provider: "fake-plugin", image: "fake:test", timeoutMs: 300_000, reuseLease: true }; + await db.insert(companies).values({ id: companyId, name: "Acme", status: "active" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Coder", role: "engineer", status: "active", + adapterType: "paperclip_runner", adapterConfig: {}, runtimeConfig: {}, permissions: {} }); + await db.insert(environments).values({ id: environmentId, name: "Paperclip Sandbox", driver: "sandbox", status: "active", + config: providerConfig }); + await db.insert(plugins).values({ + id: pluginId, pluginKey: "acme.reusable-sandbox-provider", packageName: "@acme/reusable-sandbox-provider", + version: "1.0.0", apiVersion: 1, categories: ["automation"], status: "ready", installOrder: 1, updatedAt: new Date(), + manifestJson: { + id: "acme.reusable-sandbox-provider", apiVersion: 1, version: "1.0.0", displayName: "Reusable Sandbox Provider", + description: "Test provider", author: "Paperclip", categories: ["automation"], + capabilities: ["environment.drivers.register"], entrypoints: { worker: "dist/worker.js" }, + environmentDrivers: [{ driverKey: "fake-plugin", kind: "sandbox_provider", displayName: "Fake Plugin", + supportsReusableLeases: true, + configSchema: { type: "object", properties: { image: { type: "string" }, timeoutMs: { type: "number" }, reuseLease: { type: "boolean" } } } }], + }, + } as never); + await db.insert(projects).values({ id: projectId, companyId, name: "Project", status: "active" }); + await db.insert(executionWorkspaces).values({ id: executionWorkspaceId, companyId, projectId, mode: "isolated_workspace", + strategyType: "project_primary", name: "Task workspace", status: "active", providerType: "cloud_sandbox", + cwd: `/workspaces/${executionWorkspaceId}` }); + await db.insert(heartbeatRuns).values([ + { id: previousRunId, companyId, agentId, invocationSource: "assignment", status: "running" }, + { id: runId, companyId, agentId, invocationSource: "assignment", status: "running" }, + ]); + const environment: Environment = { id: environmentId, name: "Paperclip Sandbox", description: null, driver: "sandbox", + status: "active", config: providerConfig, envVars: {}, metadata: null, createdAt: new Date(), updatedAt: new Date() }; + // The previous run creates the sandbox through the real acquisition path, + // so the preserved lease carries exactly the identity production records. + const priming = environmentRuntimeService(db, { pluginWorkerManager: { + isRunning: () => true, + call: async (_pluginId: string, method: string) => { + if (method !== "environmentAcquireLease") throw new Error(`Unexpected plugin method: ${method}`); + return { providerLeaseId, metadata: { ...providerConfig, sandboxId: providerLeaseId, remoteCwd: "/home/daytona/paperclip-workspace" } }; + }, + getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] }), + } as unknown as PluginWorkerManager }); + const created = await priming.acquireRunLease({ companyId, environment, issueId: null, agentId, adapterType: "paperclip_runner", + heartbeatRunId: previousRunId, persistedExecutionWorkspace: { id: executionWorkspaceId, mode: "isolated_workspace" } }); + // The run then ended after a clean final save and left its lease-bound + // harness backup stamp. A lease saved by an older release has no + // acquisition outcome and may predate the backup stamp. + const { sandboxLeaseAcquisition: _outcome, ...metadata } = created.lease.metadata ?? {}; + const lease = (await environmentService(db).updateLeaseMetadata(created.lease.id, { + ...(shape.leaseStatus === "released" ? metadata : created.lease.metadata), + remoteWorkspaceId: executionWorkspaceId, + ...(shape.backupStamp === false ? {} : { nativeHarnessBackup: { + schema: "paperclip.native-harness-backup-stamp.v2", sourceProviderLeaseId: providerLeaseId, + authorizedProviderLeaseId: providerLeaseId, + } }), + }))!; + await db.update(environmentLeases).set({ status: shape.leaseStatus ?? "retained" }).where(eq(environmentLeases.id, lease.id)); + await db.update(heartbeatRuns).set({ status: shape.previousRunStatus ?? "succeeded" }).where(eq(heartbeatRuns.id, previousRunId)); + const snapshotId = randomUUID(); + if (shape.snapshot !== null) { + await db.insert(executionWorkspaceSnapshots).values({ id: snapshotId, companyId, executionWorkspaceId, + sourceOperationId: randomUUID(), sourceRunId: previousRunId, generation: 1, epoch: 1, status: shape.snapshot ?? "committed" }); + } + // This run already holds the workspace writer slot (reserved before the + // environment is acquired); the saved version is the previous run's save. + await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: { + schema: "paperclip.remote-workspace.v1", epoch: 2, snapshotId: shape.snapshot === null ? null : snapshotId, + source: { kind: "empty" }, writerRunId: runId, dirty: shape.dirty ?? false, + location: { environmentId, leaseId: lease.id, providerLeaseId, remoteCwd: "/home/daytona/paperclip-workspace/.paperclip-workspaces/w" }, + } } }).where(eq(executionWorkspaces.id, executionWorkspaceId)); + return { companyId, agentId, environment, pluginId, runId, executionWorkspaceId, lease, providerLeaseId }; + } + + function worker(seeded: Awaited>, resume: (attempt: number) => Promise, + options: { acquireFails?: boolean } = {}) { + let attempts = 0; + const call = vi.fn(async (_pluginId: string, method: string, _params: unknown, _timeoutMs?: number) => { + if (method === "environmentResumeLease") return await resume(++attempts); + if (method === "environmentAcquireLease") { + if (options.acquireFails) throw new JsonRpcCallError({ code: -32000, message: "create refused" }); + return { providerLeaseId: "replacement-sandbox", metadata: { provider: "fake-plugin", image: "fake:test", + timeoutMs: 300_000, reuseLease: true, remoteCwd: "/home/daytona/paperclip-workspace" } }; + } + throw new Error(`Unexpected plugin method: ${method}`); + }); + const manager = { isRunning: vi.fn((id: string) => id === seeded.pluginId), call, + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), + } as unknown as PluginWorkerManager; + return { call, runtime: environmentRuntimeService(db, { pluginWorkerManager: manager }) }; + } + + function acquire(seeded: Awaited>, runtime: ReturnType, + runId = seeded.runId) { + return runtime.acquireRunLease({ companyId: seeded.companyId, environment: seeded.environment, issueId: null, + agentId: seeded.agentId, adapterType: "paperclip_runner", heartbeatRunId: runId, + persistedExecutionWorkspace: { id: seeded.executionWorkspaceId, mode: "isolated_workspace" } }); + } + + // Each refused resume consumes 40s of the run's clock, so the in-run retry + // budget is exercised without real waiting. + function stepClock() { + let clock = Date.now(); + vi.spyOn(Date, "now").mockImplementation(() => clock); + return () => { clock += 40_000; }; + } + + it("replaces a clean, saved sandbox that keeps refusing to resume, within a bounded budget (staging shape)", async () => { + const seeded = await seed(); + const step = stepClock(); + const { call, runtime } = worker(seeded, async () => { + step(); + throw new JsonRpcCallError({ code: -32000, message: STAGING_REFUSAL }); + }); + + const acquired = await acquire(seeded, runtime); + + expect(acquired.lease.providerLeaseId).toBe("replacement-sandbox"); + expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "replacement", reason: "resume_failed" }); + const resumes = call.mock.calls.filter(([, method]) => method === "environmentResumeLease"); + // Retried inside the run, then stopped well short of the five-minute deadline. + expect(resumes.length).toBeGreaterThan(1); + for (const [, , , timeoutMs] of resumes) expect(timeoutMs).toBeLessThanOrEqual(90_000); + // The old sandbox is handed to the cleanup sweep; no synchronous destroy + // delays the first turn. + expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentDestroyLease", expect.anything(), expect.anything()); + await expect(environmentService(db).getLeaseById(seeded.lease.id)).resolves.toMatchObject({ + status: "pending_cleanup", cleanupStatus: "failed", failureReason: "resume_failed", + }); + }); + + it("keeps the preserved lease when the replacement cannot be created, so the next attempt replaces it", async () => { + const seeded = await seed(); + const refuse = async () => { throw new JsonRpcCallError({ code: -32000, message: STAGING_REFUSAL }); }; + const step = stepClock(); + const failing = worker(seeded, async () => { step(); return await refuse(); }, { acquireFails: true }); + await expect(acquire(seeded, failing.runtime)).rejects.toThrow(); + await expect(environmentService(db).getLeaseById(seeded.lease.id)).resolves.toMatchObject({ + status: "retained", providerLeaseId: seeded.providerLeaseId, + }); + + // The bounded retry runs as a new run that reserves the same clean workspace. + const retryRunId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: retryRunId, companyId: seeded.companyId, agentId: seeded.agentId, + invocationSource: "assignment", status: "running" }); + await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, seeded.runId)); + const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, seeded.executionWorkspaceId)); + await db.update(executionWorkspaces).set({ metadata: { ...workspace!.metadata, + remoteWorkspace: { ...(workspace!.metadata!.remoteWorkspace as Record), writerRunId: retryRunId } } }) + .where(eq(executionWorkspaces.id, seeded.executionWorkspaceId)); + const retry = worker(seeded, async () => { step(); return await refuse(); }); + const acquired = await acquire(seeded, retry.runtime, retryRunId); + expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "replacement", reason: "resume_failed" }); + // The replacement carries the verified harness backup, so the session restores + // without waiting for the old sandbox to be deleted. + expect(acquired.lease.metadata?.nativeHarnessBackup).toMatchObject({ authorizedProviderLeaseId: seeded.providerLeaseId }); + await expect(environmentService(db).getLeaseById(seeded.lease.id)).resolves.toMatchObject({ status: "pending_cleanup" }); + }); + + it("replaces a released lease saved by an older release in the same way", async () => { + const seeded = await seed({ leaseStatus: "released" }); + const step = stepClock(); + const { runtime } = worker(seeded, async () => { + step(); + throw new JsonRpcCallError({ code: -32000, message: "Sandbox state change in progress", + data: environmentResumeErrorData(Object.assign(new Error("private"), { statusCode: 409 })) }); + }); + const acquired = await acquire(seeded, runtime); + expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "replacement", reason: "resume_failed" }); + }); + + it("retries a sandbox state change inside the run and resumes the same sandbox", async () => { + const seeded = await seed(); + const { call, runtime } = worker(seeded, async (attempt) => { + if (attempt === 1) throw new JsonRpcCallError({ code: -32000, message: "Sandbox state change in progress" }); + return { providerLeaseId: seeded.providerLeaseId, metadata: { remoteCwd: "/home/daytona/paperclip-workspace" } }; + }); + const acquired = await acquire(seeded, runtime); + expect(acquired.lease.providerLeaseId).toBe(seeded.providerLeaseId); + expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "resumed" }); + expect(call.mock.calls.filter(([, method]) => method === "environmentResumeLease")).toHaveLength(2); + expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentAcquireLease", expect.anything(), expect.anything()); + }); + + it.each([ + ["unsaved work on the sandbox", { dirty: true }, undefined], + ["no saved workspace version", { snapshot: null }, undefined], + ["a saved version that never committed", { snapshot: "pending" }, undefined], + ["a live owner", { previousRunStatus: "running" }, undefined], + ["no verified harness backup (older lease)", { backupStamp: false }, undefined], + ["a provider access refusal", {}, 403], + ] as Array<[string, Shape, number | undefined]>)("keeps the sandbox and fails strictly with %s", async (_label, shape, status) => { + const seeded = await seed(shape); + const step = stepClock(); + const { call, runtime } = worker(seeded, async () => { + step(); + throw new JsonRpcCallError({ code: -32000, message: STAGING_REFUSAL, + ...(status ? { data: environmentResumeErrorData(Object.assign(new Error("private"), { statusCode: status })) } : {}) }); + }); + await expect(acquire(seeded, runtime)).rejects.toThrow( + /could not be resumed \(cause=sandbox_state_transition( status=403)?\); the lease was preserved and no replacement was created\.$/, + ); + expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentAcquireLease", expect.anything(), expect.anything()); + expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentDestroyLease", expect.anything(), expect.anything()); + await expect(environmentService(db).getLeaseById(seeded.lease.id)).resolves.toMatchObject({ + status: shape.leaseStatus ?? "retained", providerLeaseId: seeded.providerLeaseId, + }); + }); +}); diff --git a/server/src/__tests__/environment-runtime.test.ts b/server/src/__tests__/environment-runtime.test.ts index 99639d300a7..77fd0c8c0bb 100644 --- a/server/src/__tests__/environment-runtime.test.ts +++ b/server/src/__tests__/environment-runtime.test.ts @@ -3817,8 +3817,10 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }); it.each([ - { label: "declared provider startup budget", timeoutMs: null, defaultAcquireTimeoutMs: 300_000, expectedRpcTimeoutMs: 330_000 }, - { label: "explicit startup budget", timeoutMs: 120_000, defaultAcquireTimeoutMs: 300_000, expectedRpcTimeoutMs: 150_000 }, + // A run's resume budget is capped at 90s rather than the provider's whole + // start deadline; a 61s start still fits. + { label: "declared provider startup budget", timeoutMs: null, defaultAcquireTimeoutMs: 300_000, expectedRpcTimeoutMs: 90_000 }, + { label: "explicit startup budget", timeoutMs: 120_000, defaultAcquireTimeoutMs: 300_000, expectedRpcTimeoutMs: 90_000 }, ])("resumes the same sandbox after a slow start within its $label", async (options) => { const seeded = await seedReusablePluginSandboxLease("paperclip_runner", options); let signalResumeStarted!: () => void; diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts index fa4b8e0cf29..1d082cd7f33 100644 --- a/server/src/services/environment-runtime.ts +++ b/server/src/services/environment-runtime.ts @@ -3,6 +3,12 @@ import { ENVIRONMENT_LEASE_CONTROLLER_KEY } from "./environment-lease-controller import { RetainedSandboxAbsentError, resumeReplyConfirmsAbsence } from "./retained-sandbox-absent-error.js"; import { readNativeWorkspaceMigration } from "./native-runtime/native-workspace-migration.js"; import { ReusableSandboxResumeError } from "./reusable-sandbox-resume-error.js"; +import { + REUSABLE_SANDBOX_RESUME_BUDGET_MS, + retireReplacedReusableSandboxLease, + reusableSandboxReplacementIsSafe, + sandboxStateTransitionFailure, +} from "./reusable-sandbox-resume-replacement.js"; import { PluginWorkerUnavailableError } from "./plugin-worker-unavailable-error.js"; import { HttpError } from "../errors.js"; import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js"; @@ -793,6 +799,8 @@ function transientSandboxResumeFailure(error: unknown): boolean { ? candidate.statusCode : null; if (status === 429 || (status !== null && status >= 500)) return true; + // A sandbox still stopping, archiving, or starting can be started shortly. + if (sandboxStateTransitionFailure(error)) return true; const code = typeof candidate.code === "string" ? candidate.code.toUpperCase() : ""; if (["ETIMEDOUT", "ECONNRESET", "ECONNREFUSED", "EAI_AGAIN", "ENETUNREACH"].includes(code)) { return true; @@ -1822,11 +1830,15 @@ function createSandboxEnvironmentDriver( }); } try { - const configuredResumeTimeoutMs = + // One run spends a bounded budget resuming, retries included; the + // provider's own start deadline (five minutes on Daytona) is not it. + const configuredResumeTimeoutMs = Math.min( resolvePluginSandboxRpcTimeoutMs( workerConfig, pluginProvider.resolved.driver.defaultAcquireTimeoutMs, - ) ?? 60_000; + ) ?? 60_000, + REUSABLE_SANDBOX_RESUME_BUDGET_MS, + ); const resumeDeadline = Date.now() + configuredResumeTimeoutMs; let retryDelayMs = 250; let resumed: PluginEnvironmentLease; @@ -1878,13 +1890,21 @@ function createSandboxEnvironmentDriver( : "expired"; } } catch (error) { - throw new ReusableSandboxResumeError({ - provider: parsed.config.provider, - providerLeaseId: reusableLease.providerLeaseId, - cause: error, - }); + // A refusal is not proof the sandbox is lost. Replace it only when + // nothing lives solely on it; otherwise preserve it for the retry. + if (!await reusableSandboxReplacementIsSafe({ + db, lease: reusableLease, companyId: input.companyId, heartbeatRunId: input.heartbeatRunId, + executionWorkspaceId: input.executionWorkspaceId, adapterType: input.adapterType, cause: error, + })) { + throw new ReusableSandboxResumeError({ + provider: parsed.config.provider, + providerLeaseId: reusableLease.providerLeaseId, + cause: error, + }); + } + replacementReason = "resume_failed"; } - if (!providerLease) { + if (!providerLease && replacementReason !== "resume_failed") { if ( input.adapterType === "paperclip_runner" && !await verifyRecoverableNativeHarnessBackup({ db, companyId: reusableLease.companyId, @@ -2015,7 +2035,7 @@ function createSandboxEnvironmentDriver( : {}), ...(reusableScope ? { reusableSandboxLease: reusableScope } : {}), }; - return await environmentsSvc.acquireLease({ + const recordedLease = await environmentsSvc.acquireLease({ creationOperationId: providerLease ? undefined : acquiredLease.metadata?.creationOperationId as string | undefined, companyId: input.companyId, environmentId: input.environment.id, @@ -2042,7 +2062,15 @@ function createSandboxEnvironmentDriver( ? reusableLease?.id : null, }); - + // Retire the unresumable sandbox only once its replacement, which + // carries the harness backup, is recorded. A failed creation or a + // restart before this point leaves it eligible for the next attempt. + if (replacementReason === "resume_failed" && reusableLease && input.heartbeatRunId) { + await retireReplacedReusableSandboxLease({ + db, lease: reusableLease, companyId: input.companyId, heartbeatRunId: input.heartbeatRunId, + }); + } + return recordedLease; } // Built-in sandbox provider path. Same guard as the plugin-backed path: diff --git a/server/src/services/remote-workspace-transport.ts b/server/src/services/remote-workspace-transport.ts index 6958c0a55d9..75248a2cf38 100644 --- a/server/src/services/remote-workspace-transport.ts +++ b/server/src/services/remote-workspace-transport.ts @@ -151,6 +151,8 @@ export function findRemoteWorkspaceRestoreError(error: unknown): RemoteWorkspace const TRANSPORT_FAILURE_LABELS: Array<[RegExp, string]> = [ [/daytona_sandbox_not_found/i, "daytona_sandbox_not_found"], [/sandbox lease \S+ is no longer active/i, "sandbox_lease_inactive"], + // The provider was still stopping, archiving, or starting the sandbox. + [/state change in progress|\bis still [a-z_]+ at its start deadline\b/i, "sandbox_state_transition"], [/\b(?:EACCES|EPERM|permission denied)\b/i, "permission_denied"], [/\b(?:not (?:started|running)|is stopped|state[^A-Za-z]{1,4}stopped|stopped sandbox)\b/i, "sandbox_not_running"], [/\b(?:timed? ?out|timeout|ETIMEDOUT)\b/i, "transport_timeout"], diff --git a/server/src/services/reusable-sandbox-resume-error.ts b/server/src/services/reusable-sandbox-resume-error.ts index f7f507ef5e3..1ef63ed2746 100644 --- a/server/src/services/reusable-sandbox-resume-error.ts +++ b/server/src/services/reusable-sandbox-resume-error.ts @@ -3,8 +3,9 @@ import { sandboxProviderFailureLabel } from "./remote-workspace-transport.js"; /** * Setup failure code for a preserved reusable sandbox that could not be - * resumed before provider work began. The failure is retried with the bounded - * bootstrap budget; it never authorizes a replacement sandbox. + * resumed before provider work began, where replacing it was not safe (see + * `reusable-sandbox-resume-replacement.ts`). The failure is retried with the + * bounded bootstrap budget; it never authorizes a replacement sandbox. */ export const REUSABLE_SANDBOX_RESUME_FAILED_CODE = "reusable_sandbox_resume_failed"; diff --git a/server/src/services/reusable-sandbox-resume-replacement.ts b/server/src/services/reusable-sandbox-resume-replacement.ts new file mode 100644 index 00000000000..466a9cfc34c --- /dev/null +++ b/server/src/services/reusable-sandbox-resume-replacement.ts @@ -0,0 +1,143 @@ +import { and, eq, inArray, sql } from "drizzle-orm"; +import { readEnvironmentResumeErrorDiagnostic } from "@paperclipai/plugin-sdk"; +import type { Db } from "@paperclipai/db"; +import { environmentLeases, executionWorkspaces, executionWorkspaceSnapshots, heartbeatRuns } from "@paperclipai/db"; +import { readRemoteWorkspaceAuthority } from "@paperclipai/shared"; +import type { EnvironmentLease } from "@paperclipai/shared"; +import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js"; +import { verifyRecoverableNativeHarnessBackup } from "./native-runtime/native-harness-backup-storage.js"; +import { findPluginWorkerUnavailableError } from "./plugin-worker-unavailable-error.js"; +import { sandboxProviderFailureLabel } from "./remote-workspace-transport.js"; + +/** + * Total time one run spends resuming a preserved reusable sandbox, including + * its in-run retries. A stopped Daytona sandbox resumes in seconds and an + * archived one in under a minute, so this leaves headroom while keeping one + * attempt from spending the provider's whole five-minute start deadline. + */ +export const REUSABLE_SANDBOX_RESUME_BUDGET_MS = 90_000; + +/** Failure reason recorded on a preserved lease retired in favor of a replacement. */ +export const REUSABLE_SANDBOX_RESUME_REPLACED_REASON = "resume_failed"; + +/** + * A provider refused to start a sandbox because it is still changing state, + * for example auto-stopping or archiving after an idle period. Waiting can + * resolve it, so it is retried like a 5xx. + */ +export function sandboxStateTransitionFailure(error: unknown): boolean { + for (let cause = error, depth = 0; cause && depth < 4; cause = (cause as { cause?: unknown }).cause, depth++) { + const message = cause instanceof Error ? cause.message : typeof cause === "string" ? cause : ""; + if (/state change in progress|\bis still [a-z_]+ at its start deadline\b/i.test(message)) return true; + } + return false; +} + +/** + * Decide whether a preserved reusable sandbox that could not be resumed may be + * replaced. + * + * Replacement is safe only when nothing lives solely on that sandbox: + * - its last run has ended (released or retained, and no live run holds it); + * - the task workspace is clean and its saved version is a committed snapshot, + * so a new sandbox restores exactly the same files; + * - a native runner session has a verified, lease-bound harness backup. + * Anything else (unsaved work, a live owner, a legacy workspace with no saved + * version, an access refusal) keeps the strict preserve-and-retry behavior. + * + * This only reads. The caller creates and records the replacement first, then + * calls `retireReplacedReusableSandboxLease`, so a failed creation or a restart + * in between leaves the preserved lease eligible for the next attempt. + */ +export async function reusableSandboxReplacementIsSafe(input: { + db: Db; + lease: EnvironmentLease; + companyId: string; + heartbeatRunId: string | null; + executionWorkspaceId: string | null; + adapterType: string | null | undefined; + cause: unknown; +}): Promise { + const { db, lease } = input; + if (!input.heartbeatRunId || !lease.providerLeaseId || lease.companyId !== input.companyId) return false; + if (lease.status !== "released" && lease.status !== "retained") return false; + if (!input.executionWorkspaceId || lease.executionWorkspaceId !== input.executionWorkspaceId) return false; + // A replacement cannot succeed where the resume was refused for access, and + // an unavailable worker cannot create one either. + if (findPluginWorkerUnavailableError(input.cause)) return false; + const failure = sandboxProviderFailureLabel(input.cause); + const status = failure.status ?? readEnvironmentResumeErrorDiagnostic(input.cause)?.httpStatus ?? null; + if (failure.label === "provider_access_denied" || failure.label === "permission_denied" + || status === 401 || status === 403) return false; + // A run that still holds the sandbox may have work on it. + if (lease.heartbeatRunId && lease.heartbeatRunId !== input.heartbeatRunId) { + const [owner] = await db.select({ status: heartbeatRuns.status }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, lease.heartbeatRunId))); + if (owner && ["queued", "scheduled_retry", "running"].includes(owner.status)) return false; + } + + const [workspace] = await db.select({ metadata: executionWorkspaces.metadata }).from(executionWorkspaces) + .where(and(eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.executionWorkspaceId))); + let authority: ReturnType; + try { + authority = readRemoteWorkspaceAuthority(workspace?.metadata); + } catch { + return false; + } + if (!authority || authority.dirty || !authority.snapshotId) return false; + if (authority.writerRunId !== null && authority.writerRunId !== input.heartbeatRunId) return false; + if (authority.location && authority.location.providerLeaseId !== lease.providerLeaseId) return false; + const [snapshot] = await db.select({ id: executionWorkspaceSnapshots.id }).from(executionWorkspaceSnapshots).where(and( + eq(executionWorkspaceSnapshots.companyId, input.companyId), + eq(executionWorkspaceSnapshots.executionWorkspaceId, input.executionWorkspaceId), + eq(executionWorkspaceSnapshots.id, authority.snapshotId), + eq(executionWorkspaceSnapshots.status, "committed"), + )); + if (!snapshot) return false; + + const scope = lease.metadata?.reusableSandboxLease as { adapterType?: unknown } | undefined; + if ((input.adapterType === "paperclip_runner" || scope?.adapterType === "paperclip_runner") + && !await verifyRecoverableNativeHarnessBackup({ db, companyId: lease.companyId, + stamp: lease.metadata?.nativeHarnessBackup, providerLeaseId: lease.providerLeaseId })) return false; + + return true; +} + +/** + * Retire a preserved lease after its replacement lease is recorded. The old + * sandbox is not destroyed here: a provider that cannot start it may also be + * slow to delete it, and that must not delay this run's first turn. The + * `pending_cleanup` row hands the destroy to the existing cleanup sweep. The + * write is guarded, so a lease that another run resumed meanwhile is untouched. + */ +export async function retireReplacedReusableSandboxLease(input: { + db: Db; + lease: EnvironmentLease; + companyId: string; + heartbeatRunId: string; +}): Promise { + const { db, lease } = input; + if (!lease.providerLeaseId) return false; + const now = new Date(); + const [claimed] = await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ + status: "pending_cleanup", + cleanupStatus: "failed", + failureReason: REUSABLE_SANDBOX_RESUME_REPLACED_REASON, + releasedAt: now, + lastUsedAt: now, + updatedAt: now, + }).where(and( + eq(environmentLeases.id, lease.id), + eq(environmentLeases.companyId, input.companyId), + eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), + inArray(environmentLeases.status, ["released", "retained"]), + sql`coalesce(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') != 'true'`, + sql`NOT EXISTS ( + SELECT 1 FROM ${heartbeatRuns} + WHERE ${heartbeatRuns.id} = ${environmentLeases.heartbeatRunId} + AND ${heartbeatRuns.id} <> ${input.heartbeatRunId} + AND ${heartbeatRuns.status} IN ('queued', 'scheduled_retry', 'running') + )`, + )).returning({ id: environmentLeases.id }), { companyId: input.companyId, runId: lease.heartbeatRunId ?? undefined }); + return Boolean(claimed); +}