From 86814c6550f52e99ede150a883bdc773cbb540e3 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 4 Nov 2026 09:04:13 -0600 Subject: [PATCH] fix(recovery): ready retained sandboxes before preparation (#18934) ## Thinking Path > - Paperclip manages work performed by AI agents. > - Restart recovery must preserve each run's workspace and execution owner. > - A retained remote lease can refer to an archived sandbox. > - The reattachment path returned that lease without making it ready. > - A short project-authentication command then had to start the sandbox. > - This PR resumes and verifies the exact retained allocation before preparation. > - Recovery keeps the recorded workspace and preserves saved work if resume fails. ## Linked Issues or Issue Description **What happened?** A native run resumed after a controller restart. Its saved sandbox had been archived. Reattachment returned its lease without a provider lifecycle resume. The next project-authentication probe tried to start that sandbox with its 15-second command budget. The start failed before the probe could inspect project settings. The heartbeat reported a configuration conflict and stopped work. **What did you expect to happen?** Make the exact retained sandbox ready through the existing bounded lifecycle operation. Verify its allocation before running short preparation commands. Keep the validated lease and recorded working directory. Never acquire a replacement in this path. **Steps to reproduce** Hold the provider's exact-lease resume pending and call remote reattachment for an active ephemeral lease. Before this change, acquisition returns immediately. The added tests hold acquisition until resume completes and reject missing, changed or expired allocation evidence. Provider compatibility cases retain the recorded workspace when resume omits directory metadata or returns the allocation root. **Deployment mode** Native Runner in a sandbox with a retained workspace and managed AI connection. The observed provider was Daytona. Refs #18910, which restores selection of retries using current remote workspace storage. This change repairs the next preparation boundary. Related #17421 and #18559 change project-file inspection; they do not make retained sandbox leases ready. ## What Changed - Resume the exact validated sandbox lease before returning it to preparation. - Require the receipt to confirm the original provider allocation without reporting expiration. - Keep the original database lease and recorded directory. Resume metadata can describe the allocation root. - Pass the existing typed resume-directory input through the runtime facade. - Preserve retained E2B sandboxes when resume timeout setup or workspace inspection fails. Keep new-allocation cleanup and explicit release/destroy behavior. - Preserve plugin-task reattachment and lease identity checks. - Update the Stop-accounting fixture to confirm exact-allocation resume before cleanup, with its budget, ownership and no-paid-work assertions intact. - Document the ordering and add readiness, receipt, failure and ownership regressions. ## Verification - Red: seven initial retained-lease cases failed before the readiness change. Two receipt-compatibility cases and four E2B preservation cases failed before their fixes. - Green: all four runtime, workspace-preparation, heartbeat-recovery and orchestrator suites pass: **442 tests**. - The Stop-accounting CI failure reproduced locally. All **five** integration cases pass with the exact-allocation resume fixture. - All **22 E2B tests**, the standalone plugin typecheck and its build pass. - **13 Daytona safety cases** pass, including recorded directories, allocation sentinel rejection, saved-file reads and no replacement creation. The SDK boundary is simulated; the saved-file case uses real files and a POSIX shell. - These focused checks cover **482 passing cases**. - Full recursive typecheck, final server typecheck and full workspace build pass on macOS arm64 with Node 24.22.2 and PostgreSQL 18.1. - Greptile is **5/5** on head `32f915a475275680bcf2a2aa43ae0c72f4e4b1db`. Both original review threads are resolved. - All current-head CI gates pass. The check rollup has 54 successful checks, three skipped checks and one neutral check. - The separate Paperclip Review check remains queued. Its public response says work is paused and the request is saved. The review task is waiting for saved-file recovery, so no Paperclip assessment of this head is claimed. The branch-required gate has passed. - A read-only deployed trace showed an archived sandbox start fail at the short probe's 15-second deadline. Actual provider reattachment with this revision has not been exercised. Live recovery remains unverified until rollout. ## Risks Recovery now makes one bounded lifecycle call before short preparation commands. A missing resume capability or a missing, changed or expired allocation receipt holds recovery. The existing runtime and provider retain allocation and sentinel verification. The path does not acquire, release, replace or reseed the lease. Warm providers can return their existing allocation. E2B retains the existing sandbox if resume setup fails; explicit lifecycle cleanup still owns its removal. No background polling, credential-policy change, migration, workflow change or lockfile change is added. ## Model Used OpenAI GPT-6 through Codex, with reasoning, source inspection, tool use and code execution. The exact backend snapshot and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 9 +++ .../sandbox-providers/e2b/src/plugin.test.ts | 17 +++-- .../sandbox-providers/e2b/src/plugin.ts | 19 ++--- .../environment-run-orchestrator.test.ts | 73 ++++++++++++++++++- .../services/environment-run-orchestrator.ts | 20 ++++- server/src/services/environment-runtime.ts | 2 +- ...native-stop-accounting.integration.test.ts | 7 ++ 7 files changed, 125 insertions(+), 22 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index fc3720f0edc..625129be469 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1767,6 +1767,15 @@ agent, workspace, retained session, and termination of prior execution. The checkpoint must match the exact Runner, lease, run, session, turn, and item. A missing restart claim does not remove these checks or authorize a fresh session. +Reattachment readies the exact retained sandbox lease through the provider's +bounded lifecycle resume before running short preparation commands such as the +managed project-authentication check. The resume receipt must confirm the same +provider allocation without reporting expiration. Missing or mismatched receipts +hold recovery; this path never acquires a replacement sandbox. The validated +lease retains its recorded working directory, since optional provider metadata +can describe the allocation root. A resume setup failure preserves the retained +sandbox and saved work. Plugin task reattachment remains with its task owner. + After a remote sandbox session passes provider recovery and durable run admission, its local suspension receipt is removed. This applies to built-in sandbox runners and plugin tasks. The running session then owns authority; diff --git a/packages/plugins/sandbox-providers/e2b/src/plugin.test.ts b/packages/plugins/sandbox-providers/e2b/src/plugin.test.ts index 48c095ba586..851807e98b1 100644 --- a/packages/plugins/sandbox-providers/e2b/src/plugin.test.ts +++ b/packages/plugins/sandbox-providers/e2b/src/plugin.test.ts @@ -240,10 +240,13 @@ describe("E2B sandbox provider plugin", () => { expect(mockCreate).toHaveBeenCalledWith("base", expect.objectContaining({ apiKey: "host-key" })); }); - it("kills the sandbox if resume setup fails after reconnect", async () => { + it.each(([true, false] as const).flatMap(reuseLease => + (["timeout", "workspace"] as const).map(phase => ({ reuseLease, phase }))))( + "preserves the retained sandbox after resume $phase failure (reuse=$reuseLease)", async ({ reuseLease, phase }) => { const sandbox = createMockSandbox(); - const failure = new Error("set-timeout failed"); - sandbox.setTimeout.mockRejectedValueOnce(failure); + const failure = new Error("retained sandbox readiness failed"); + if (phase === "timeout") sandbox.setTimeout.mockRejectedValueOnce(failure); + else sandbox.commands.run.mockRejectedValueOnce(failure); mockConnect.mockResolvedValue(sandbox); await expect(plugin.definition.onEnvironmentResumeLease?.({ @@ -256,11 +259,13 @@ describe("E2B sandbox provider plugin", () => { template: "base", apiKey: "resolved-key", timeoutMs: 300000, - reuseLease: false, + reuseLease, }, - })).rejects.toThrow("set-timeout failed"); + })).rejects.toBe(failure); - expect(sandbox.kill).toHaveBeenCalled(); + expect(sandbox.kill).not.toHaveBeenCalled(); + expect(sandbox.files.remove).not.toHaveBeenCalled(); + expect(mockCreate).not.toHaveBeenCalled(); }); it("executes commands through a connected sandbox when stdin is provided", async () => { diff --git a/packages/plugins/sandbox-providers/e2b/src/plugin.ts b/packages/plugins/sandbox-providers/e2b/src/plugin.ts index 83bd930687c..21acbc88f55 100644 --- a/packages/plugins/sandbox-providers/e2b/src/plugin.ts +++ b/packages/plugins/sandbox-providers/e2b/src/plugin.ts @@ -310,18 +310,15 @@ const plugin = definePlugin({ const config = parseDriverConfig(params.config); try { const sandbox = await connectSandbox(config, params.providerLeaseId); - try { - await sandbox.setTimeout(config.timeoutMs); - const remoteCwd = await resolveSandboxWorkingDirectory(sandbox); + // A reconnect failure must preserve the retained allocation and its work. + // Explicit release or destroy owns cleanup of an existing sandbox. + await sandbox.setTimeout(config.timeoutMs); + const remoteCwd = await resolveSandboxWorkingDirectory(sandbox); - return { - providerLeaseId: sandbox.sandboxId, - metadata: leaseMetadata({ config, sandbox, remoteCwd, resumedLease: true }), - }; - } catch (error) { - await sandbox.kill().catch(() => undefined); - throw error; - } + return { + providerLeaseId: sandbox.sandboxId, + metadata: leaseMetadata({ config, sandbox, remoteCwd, resumedLease: true }), + }; } catch (error) { if (error instanceof SandboxNotFoundError) { return { providerLeaseId: null, metadata: { expired: true } }; diff --git a/server/src/__tests__/environment-run-orchestrator.test.ts b/server/src/__tests__/environment-run-orchestrator.test.ts index c33d65a1ca9..88743ab99e1 100644 --- a/server/src/__tests__/environment-run-orchestrator.test.ts +++ b/server/src/__tests__/environment-run-orchestrator.test.ts @@ -192,6 +192,10 @@ function makeRealizeInput(overrides: { function makeMockRuntime(overrides: Partial = {}): EnvironmentRuntimeService { return { acquireRunLease: vi.fn(), + resumeRunLease: vi.fn(async ({ lease, requestedCwd }: Parameters[0]) => ({ + providerLeaseId: lease.providerLeaseId, + metadata: { ...lease.metadata, remoteCwd: requestedCwd ?? lease.metadata?.remoteCwd }, + })), releaseRunLeases: vi.fn(), execute: vi.fn().mockResolvedValue({ exitCode: 0, @@ -906,17 +910,82 @@ describe("live remote runner lease reattachment", () => { mockGetEnvironment.mockResolvedValue({ ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false } }); mockGetLease.mockResolvedValue(originalLease()); }); - it("reattaches the original active ephemeral lease without acquiring or resuming a sandbox", async () => { + it("readies the original active ephemeral lease without acquiring a replacement", async () => { const savedLease = originalLease(); mockGetLease.mockResolvedValue(savedLease); const runtime = makeMockRuntime(); const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input); expect(mockGetLease).toHaveBeenCalledWith("lease-1"); expect(runtime.acquireRunLease).not.toHaveBeenCalled(); + expect(runtime.resumeRunLease).toHaveBeenCalledWith({ + environment: result.environment, lease: savedLease, requestedCwd: "/remote/workspace", + }); expect(result.lease).toEqual(savedLease); expect(result.leaseContext.executionWorkspaceId).toBe("ew-1"); expect(result.environment.config.reuseLease).toBe(false); }); + it("does not return the retained sandbox to preparation while its lifecycle resume is pending", async () => { + let ready = false; + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const savedLease = originalLease(); + const before = structuredClone(savedLease); + mockGetLease.mockResolvedValue(savedLease); + const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => { + await gate; + ready = true; + return { providerLeaseId: "original-sandbox", metadata: { remoteCwd: "/remote/workspace" } }; + }) }); + const pending = environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input); + try { + const state = await Promise.race([ + pending.then(() => "returned"), + new Promise(resolve => setImmediate(() => resolve("pending"))), + ]); + expect(state).toBe("pending"); + expect(ready).toBe(false); + } finally { release(); } + expect((await pending).lease).toEqual(before); + expect(ready).toBe(true); + expect(runtime.acquireRunLease).not.toHaveBeenCalled(); + expect(runtime.execute).not.toHaveBeenCalled(); + expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled(); + }); + it.each([ + null, + { providerLeaseId: "replacement", metadata: { remoteCwd: "/remote/workspace" } }, + { providerLeaseId: "original-sandbox", metadata: { expired: true } }, + ])("holds recovery when the retained lease resume cannot confirm its original allocation (%j)", async receipt => { + const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => receipt) }); + await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input)) + .rejects.toThrow("native_remote_recovery_lease_mismatch"); + expect(runtime.acquireRunLease).not.toHaveBeenCalled(); + expect(runtime.execute).not.toHaveBeenCalled(); + expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled(); + }); + it.each([undefined, { remoteCwd: "/allocation/root" }])( + "keeps the recorded workspace when resume returns allocation metadata (%j)", async metadata => { + const savedLease = originalLease(); + mockGetLease.mockResolvedValue(savedLease); + const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => ({ + providerLeaseId: "original-sandbox", metadata, + })) }); + const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input); + expect(result.lease).toBe(savedLease); + expect(result.lease.metadata?.remoteCwd).toBe("/remote/workspace"); + expect(runtime.resumeRunLease).toHaveBeenCalledOnce(); + expect(runtime.acquireRunLease).not.toHaveBeenCalled(); + expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled(); + }, + ); + it("preserves a retained sandbox startup failure without acquiring a replacement", async () => { + const error = new Error("retained sandbox startup timed out"); + const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => { throw error; }) }); + await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input)) + .rejects.toBe(error); + expect(runtime.acquireRunLease).not.toHaveBeenCalled(); + expect(runtime.execute).not.toHaveBeenCalled(); + }); it.each(["active", "foreign-company", "revoked", "undispatched", "stopped"])("inspects the original plugin task lease (%s)", async state => { mockGetEnvironment.mockResolvedValue(makeEnvironment("plugin")); const lease = originalLease(); @@ -932,6 +1001,7 @@ describe("live remote runner lease reattachment", () => { if (state === "active") await expect(acquired).resolves.toMatchObject({ lease }); else await expect(acquired).rejects.toThrow("native_remote_recovery_lease_mismatch"); expect(runtime.acquireRunLease).not.toHaveBeenCalled(); + expect(runtime.resumeRunLease).not.toHaveBeenCalled(); }); it.each([ { companyId: "foreign" }, { environmentId: "foreign" }, { heartbeatRunId: "another-run" }, @@ -946,6 +1016,7 @@ describe("live remote runner lease reattachment", () => { await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input)) .rejects.toThrow("native_remote_recovery_lease_mismatch"); expect(runtime.acquireRunLease).not.toHaveBeenCalled(); + expect(runtime.resumeRunLease).not.toHaveBeenCalled(); }); it("never replaces a missing original lease", async () => { mockGetLease.mockResolvedValue(null); diff --git a/server/src/services/environment-run-orchestrator.ts b/server/src/services/environment-run-orchestrator.ts index 943d01f2e58..957531683f2 100644 --- a/server/src/services/environment-run-orchestrator.ts +++ b/server/src/services/environment-run-orchestrator.ts @@ -327,9 +327,8 @@ export function environmentRunOrchestrator( }; let leaseRecord: EnvironmentRuntimeLeaseRecord; if (input.reattachRemoteLease) { - // Reattachment is inspection of an already running sandbox, never a new - // acquisition or a lifecycle resume. Those paths can create a replacement - // when the admitted per-turn policy deliberately disables reusable leases. + // Reattachment keeps the original lease, including ephemeral leases. + // Never reacquire: acquisition can replace a non-reusable sandbox. const expected = input.reattachRemoteLease; const lease = await environmentsSvc.getLeaseById(expected.leaseId); if (!lease || @@ -347,6 +346,21 @@ export function environmentRunOrchestrator( || lease.providerLeaseId !== expected.providerLeaseId || lease.metadata?.remoteCwd !== expected.remoteCwd)) { throw new Error("native_remote_recovery_lease_mismatch"); } + if (expected.kind !== "plugin_task") { + // The provider may have archived this retained sandbox while its + // controller was offline. Ready the exact lease under its lifecycle + // budget before short preparation commands (such as project-auth + // checks) can run. This API cannot acquire a replacement. + const resumed = await environmentRuntime.resumeRunLease({ + environment, lease, requestedCwd: expected.remoteCwd, + }); + if (!resumed || resumed.providerLeaseId !== expected.providerLeaseId || + resumed.metadata?.expired === true) { + throw new Error("native_remote_recovery_lease_mismatch"); + } + // Resume confirms allocation readiness. Its optional metadata may + // describe the allocation root; keep the validated lease's workspace. + } leaseRecord = { environment, lease, leaseContext: buildEnvironmentLeaseContext(input) }; } else { leaseRecord = await acquireLease(acquisitionInput); diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts index 77c6dff225d..31983fa5ac7 100644 --- a/server/src/services/environment-runtime.ts +++ b/server/src/services/environment-runtime.ts @@ -4307,7 +4307,7 @@ export function environmentRuntimeService( return { destroyed, failed, skippedLiveRun }; }, - async resumeRunLease(input: EnvironmentDriverLeaseInput): Promise { + async resumeRunLease(input: EnvironmentDriverResumeInput): Promise { if (await legacyShutdownWorkspaceResourceProtected(db, input.lease)) return null; const driver = requireDriverKey(getLeaseDriverKey(input.lease, input.environment)); if (!driver.resumeRunLease) { diff --git a/server/src/services/native-runtime/native-stop-accounting.integration.test.ts b/server/src/services/native-runtime/native-stop-accounting.integration.test.ts index d2ef4378083..33ea4d80323 100644 --- a/server/src/services/native-runtime/native-stop-accounting.integration.test.ts +++ b/server/src/services/native-runtime/native-stop-accounting.integration.test.ts @@ -222,7 +222,9 @@ describe("native Stop accounting after the last usage receipt", () => { const sync = vi.spyOn(workspaceSync, "prepareNativeWorkspaceSync").mockResolvedValue(null); const stage = vi.spyOn(attachmentStaging, "stageNativeRunnerWakeAttachments").mockRejectedValue(new Error("Stop cleanup must not stage attachments")); const acquire = vi.fn(async () => { throw new Error("Stop cleanup cannot acquire a new sandbox"); }); + const resume = vi.fn(async () => ({ providerLeaseId: "existing-sandbox" })); const runtime = environmentRuntimeService(db, { drivers: [{ driver: "sandbox", acquireRunLease: acquire, + resumeRunLease: resume, resolveCapabilities: async () => classifyEnvironmentCapabilities({}), realizeWorkspace: async () => ({ cwd: home.home }), retryPendingSandboxTeardown: async ({ lease }) => { @@ -244,6 +246,11 @@ describe("native Stop accounting after the last usage receipt", () => { const holds = await db.select().from(budgetReservations).where(eq(budgetReservations.runId, runId)); expect(await readClaimedNativeStopRecovery(db, current, owner.leaseOwner ?? undefined)).toBe(true); expect(holds).toMatchObject([{ id: hold.id, state: "held" }]); + expect(resume).toHaveBeenCalledOnce(); + expect(resume).toHaveBeenCalledWith(expect.objectContaining({ + lease: expect.objectContaining({ id: native.environmentLeaseId, providerLeaseId: "existing-sandbox" }), + requestedCwd: "/workspace", + })); reachedSelection = true; } }); try {