fix(recovery): ready retained sandboxes before preparation (#18934)

## Thinking Path

> - Paperclip manages work performed by AI agents.
> - Restart recovery must preserve each run's workspace and execution
owner.
> - A retained remote lease can refer to an archived sandbox.
> - The reattachment path returned that lease without making it ready.
> - A short project-authentication command then had to start the
sandbox.
> - This PR resumes and verifies the exact retained allocation before
preparation.
> - Recovery keeps the recorded workspace and preserves saved work if
resume fails.

## Linked Issues or Issue Description

**What happened?**

A native run resumed after a controller restart. Its saved sandbox had
been archived. Reattachment returned its lease without a provider
lifecycle resume. The next project-authentication probe tried to start
that sandbox with its 15-second command budget. The start failed before
the probe could inspect project settings. The heartbeat reported a
configuration conflict and stopped work.

**What did you expect to happen?**

Make the exact retained sandbox ready through the existing bounded
lifecycle operation. Verify its allocation before running short
preparation commands. Keep the validated lease and recorded working
directory. Never acquire a replacement in this path.

**Steps to reproduce**

Hold the provider's exact-lease resume pending and call remote
reattachment for an active ephemeral lease. Before this change,
acquisition returns immediately. The added tests hold acquisition until
resume completes and reject missing, changed or expired allocation
evidence. Provider compatibility cases retain the recorded workspace
when resume omits directory metadata or returns the allocation root.

**Deployment mode**

Native Runner in a sandbox with a retained workspace and managed AI
connection. The observed provider was Daytona.

Refs #18910, which restores selection of retries using current remote
workspace storage. This change repairs the next preparation boundary.
Related #17421 and #18559 change project-file inspection; they do not
make retained sandbox leases ready.

## What Changed

- Resume the exact validated sandbox lease before returning it to
preparation.
- Require the receipt to confirm the original provider allocation
without reporting expiration.
- Keep the original database lease and recorded directory. Resume
metadata can describe the allocation root.
- Pass the existing typed resume-directory input through the runtime
facade.
- Preserve retained E2B sandboxes when resume timeout setup or workspace
inspection fails. Keep new-allocation cleanup and explicit
release/destroy behavior.
- Preserve plugin-task reattachment and lease identity checks.
- Update the Stop-accounting fixture to confirm exact-allocation resume
before cleanup, with its budget, ownership and no-paid-work assertions
intact.
- Document the ordering and add readiness, receipt, failure and
ownership regressions.

## Verification

- Red: seven initial retained-lease cases failed before the readiness
change. Two receipt-compatibility cases and four E2B preservation cases
failed before their fixes.
- Green: all four runtime, workspace-preparation, heartbeat-recovery and
orchestrator suites pass: **442 tests**.
- The Stop-accounting CI failure reproduced locally. All **five**
integration cases pass with the exact-allocation resume fixture.
- All **22 E2B tests**, the standalone plugin typecheck and its build
pass.
- **13 Daytona safety cases** pass, including recorded directories,
allocation sentinel rejection, saved-file reads and no replacement
creation. The SDK boundary is simulated; the saved-file case uses real
files and a POSIX shell.
- These focused checks cover **482 passing cases**.
- Full recursive typecheck, final server typecheck and full workspace
build pass on macOS arm64 with Node 24.22.2 and PostgreSQL 18.1.
- Greptile is **5/5** on head
`32f915a475275680bcf2a2aa43ae0c72f4e4b1db`. Both original review threads
are resolved.
- All current-head CI gates pass. The check rollup has 54 successful
checks, three skipped checks and one neutral check.
- The separate Paperclip Review check remains queued. Its public
response says work is paused and the request is saved. The review task
is waiting for saved-file recovery, so no Paperclip assessment of this
head is claimed. The branch-required gate has passed.
- A read-only deployed trace showed an archived sandbox start fail at
the short probe's 15-second deadline. Actual provider reattachment with
this revision has not been exercised. Live recovery remains unverified
until rollout.

## Risks

Recovery now makes one bounded lifecycle call before short preparation
commands. A missing resume capability or a missing, changed or expired
allocation receipt holds recovery. The existing runtime and provider
retain allocation and sentinel verification. The path does not acquire,
release, replace or reseed the lease. Warm providers can return their
existing allocation. E2B retains the existing sandbox if resume setup
fails; explicit lifecycle cleanup still owns its removal. No background
polling, credential-policy change, migration, workflow change or
lockfile change is added.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, source inspection, tool use
and code execution. The exact backend snapshot and context window were
not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/execution-semantics.md

@@ -1767,6 +1767,15 @@ agent, workspace, retained session, and termination of prior execution. The
 checkpoint must match the exact Runner, lease, run, session, turn, and item.
 A missing restart claim does not remove these checks or authorize a fresh session.
 
+Reattachment readies the exact retained sandbox lease through the provider's
+bounded lifecycle resume before running short preparation commands such as the
+managed project-authentication check. The resume receipt must confirm the same
+provider allocation without reporting expiration. Missing or mismatched receipts
+hold recovery; this path never acquires a replacement sandbox. The validated
+lease retains its recorded working directory, since optional provider metadata
+can describe the allocation root. A resume setup failure preserves the retained
+sandbox and saved work. Plugin task reattachment remains with its task owner.
+
 After a remote sandbox session passes provider recovery and durable run admission,
 its local suspension receipt is removed. This applies to built-in sandbox runners
 and plugin tasks. The running session then owns authority;

packages/plugins/sandbox-providers/e2b/src/plugin.test.ts

@@ -240,10 +240,13 @@ describe("E2B sandbox provider plugin", () => {
     expect(mockCreate).toHaveBeenCalledWith("base", expect.objectContaining({ apiKey: "host-key" }));
   });
 
-  it("kills the sandbox if resume setup fails after reconnect", async () => {
+  it.each(([true, false] as const).flatMap(reuseLease =>
+    (["timeout", "workspace"] as const).map(phase => ({ reuseLease, phase }))))(
+    "preserves the retained sandbox after resume $phase failure (reuse=$reuseLease)", async ({ reuseLease, phase }) => {
     const sandbox = createMockSandbox();
-    const failure = new Error("set-timeout failed");
-    sandbox.setTimeout.mockRejectedValueOnce(failure);
+    const failure = new Error("retained sandbox readiness failed");
+    if (phase === "timeout") sandbox.setTimeout.mockRejectedValueOnce(failure);
+    else sandbox.commands.run.mockRejectedValueOnce(failure);
     mockConnect.mockResolvedValue(sandbox);
 
     await expect(plugin.definition.onEnvironmentResumeLease?.({
@@ -256,11 +259,13 @@ describe("E2B sandbox provider plugin", () => {
         template: "base",
         apiKey: "resolved-key",
         timeoutMs: 300000,
-        reuseLease: false,
+        reuseLease,
       },
-    })).rejects.toThrow("set-timeout failed");
+    })).rejects.toBe(failure);
 
-    expect(sandbox.kill).toHaveBeenCalled();
+    expect(sandbox.kill).not.toHaveBeenCalled();
+    expect(sandbox.files.remove).not.toHaveBeenCalled();
+    expect(mockCreate).not.toHaveBeenCalled();
   });
 
   it("executes commands through a connected sandbox when stdin is provided", async () => {

packages/plugins/sandbox-providers/e2b/src/plugin.ts

@@ -310,18 +310,15 @@ const plugin = definePlugin({
     const config = parseDriverConfig(params.config);
     try {
       const sandbox = await connectSandbox(config, params.providerLeaseId);
-      try {
-        await sandbox.setTimeout(config.timeoutMs);
-        const remoteCwd = await resolveSandboxWorkingDirectory(sandbox);
+      // A reconnect failure must preserve the retained allocation and its work.
+      // Explicit release or destroy owns cleanup of an existing sandbox.
+      await sandbox.setTimeout(config.timeoutMs);
+      const remoteCwd = await resolveSandboxWorkingDirectory(sandbox);
 
-        return {
-          providerLeaseId: sandbox.sandboxId,
-          metadata: leaseMetadata({ config, sandbox, remoteCwd, resumedLease: true }),
-        };
-      } catch (error) {
-        await sandbox.kill().catch(() => undefined);
-        throw error;
-      }
+      return {
+        providerLeaseId: sandbox.sandboxId,
+        metadata: leaseMetadata({ config, sandbox, remoteCwd, resumedLease: true }),
+      };
     } catch (error) {
       if (error instanceof SandboxNotFoundError) {
         return { providerLeaseId: null, metadata: { expired: true } };

server/src/__tests__/environment-run-orchestrator.test.ts

@@ -192,6 +192,10 @@ function makeRealizeInput(overrides: {
 function makeMockRuntime(overrides: Partial<EnvironmentRuntimeService> = {}): EnvironmentRuntimeService {
   return {
     acquireRunLease: vi.fn(),
+    resumeRunLease: vi.fn(async ({ lease, requestedCwd }: Parameters<EnvironmentRuntimeService["resumeRunLease"]>[0]) => ({
+      providerLeaseId: lease.providerLeaseId,
+      metadata: { ...lease.metadata, remoteCwd: requestedCwd ?? lease.metadata?.remoteCwd },
+    })),
     releaseRunLeases: vi.fn(),
     execute: vi.fn().mockResolvedValue({
       exitCode: 0,
@@ -906,17 +910,82 @@ describe("live remote runner lease reattachment", () => {
     mockGetEnvironment.mockResolvedValue({ ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false } });
     mockGetLease.mockResolvedValue(originalLease());
   });
-  it("reattaches the original active ephemeral lease without acquiring or resuming a sandbox", async () => {
+  it("readies the original active ephemeral lease without acquiring a replacement", async () => {
     const savedLease = originalLease();
     mockGetLease.mockResolvedValue(savedLease);
     const runtime = makeMockRuntime();
     const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
     expect(mockGetLease).toHaveBeenCalledWith("lease-1");
     expect(runtime.acquireRunLease).not.toHaveBeenCalled();
+    expect(runtime.resumeRunLease).toHaveBeenCalledWith({
+      environment: result.environment, lease: savedLease, requestedCwd: "/remote/workspace",
+    });
     expect(result.lease).toEqual(savedLease);
     expect(result.leaseContext.executionWorkspaceId).toBe("ew-1");
     expect(result.environment.config.reuseLease).toBe(false);
   });
+  it("does not return the retained sandbox to preparation while its lifecycle resume is pending", async () => {
+    let ready = false;
+    let release!: () => void;
+    const gate = new Promise<void>(resolve => { release = resolve; });
+    const savedLease = originalLease();
+    const before = structuredClone(savedLease);
+    mockGetLease.mockResolvedValue(savedLease);
+    const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => {
+      await gate;
+      ready = true;
+      return { providerLeaseId: "original-sandbox", metadata: { remoteCwd: "/remote/workspace" } };
+    }) });
+    const pending = environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
+    try {
+      const state = await Promise.race([
+        pending.then(() => "returned"),
+        new Promise<string>(resolve => setImmediate(() => resolve("pending"))),
+      ]);
+      expect(state).toBe("pending");
+      expect(ready).toBe(false);
+    } finally { release(); }
+    expect((await pending).lease).toEqual(before);
+    expect(ready).toBe(true);
+    expect(runtime.acquireRunLease).not.toHaveBeenCalled();
+    expect(runtime.execute).not.toHaveBeenCalled();
+    expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
+  });
+  it.each([
+    null,
+    { providerLeaseId: "replacement", metadata: { remoteCwd: "/remote/workspace" } },
+    { providerLeaseId: "original-sandbox", metadata: { expired: true } },
+  ])("holds recovery when the retained lease resume cannot confirm its original allocation (%j)", async receipt => {
+    const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => receipt) });
+    await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
+      .rejects.toThrow("native_remote_recovery_lease_mismatch");
+    expect(runtime.acquireRunLease).not.toHaveBeenCalled();
+    expect(runtime.execute).not.toHaveBeenCalled();
+    expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
+  });
+  it.each([undefined, { remoteCwd: "/allocation/root" }])(
+    "keeps the recorded workspace when resume returns allocation metadata (%j)", async metadata => {
+      const savedLease = originalLease();
+      mockGetLease.mockResolvedValue(savedLease);
+      const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => ({
+        providerLeaseId: "original-sandbox", metadata,
+      })) });
+      const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
+      expect(result.lease).toBe(savedLease);
+      expect(result.lease.metadata?.remoteCwd).toBe("/remote/workspace");
+      expect(runtime.resumeRunLease).toHaveBeenCalledOnce();
+      expect(runtime.acquireRunLease).not.toHaveBeenCalled();
+      expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
+    },
+  );
+  it("preserves a retained sandbox startup failure without acquiring a replacement", async () => {
+    const error = new Error("retained sandbox startup timed out");
+    const runtime = makeMockRuntime({ resumeRunLease: vi.fn(async () => { throw error; }) });
+    await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
+      .rejects.toBe(error);
+    expect(runtime.acquireRunLease).not.toHaveBeenCalled();
+    expect(runtime.execute).not.toHaveBeenCalled();
+  });
   it.each(["active", "foreign-company", "revoked", "undispatched", "stopped"])("inspects the original plugin task lease (%s)", async state => {
     mockGetEnvironment.mockResolvedValue(makeEnvironment("plugin"));
     const lease = originalLease();
@@ -932,6 +1001,7 @@ describe("live remote runner lease reattachment", () => {
     if (state === "active") await expect(acquired).resolves.toMatchObject({ lease });
     else await expect(acquired).rejects.toThrow("native_remote_recovery_lease_mismatch");
     expect(runtime.acquireRunLease).not.toHaveBeenCalled();
+    expect(runtime.resumeRunLease).not.toHaveBeenCalled();
   });
   it.each([
     { companyId: "foreign" }, { environmentId: "foreign" }, { heartbeatRunId: "another-run" },
@@ -946,6 +1016,7 @@ describe("live remote runner lease reattachment", () => {
     await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
       .rejects.toThrow("native_remote_recovery_lease_mismatch");
     expect(runtime.acquireRunLease).not.toHaveBeenCalled();
+    expect(runtime.resumeRunLease).not.toHaveBeenCalled();
   });
   it("never replaces a missing original lease", async () => {
     mockGetLease.mockResolvedValue(null);

server/src/services/environment-run-orchestrator.ts

@@ -327,9 +327,8 @@ export function environmentRunOrchestrator(
     };
     let leaseRecord: EnvironmentRuntimeLeaseRecord;
     if (input.reattachRemoteLease) {
-      // Reattachment is inspection of an already running sandbox, never a new
-      // acquisition or a lifecycle resume. Those paths can create a replacement
-      // when the admitted per-turn policy deliberately disables reusable leases.
+      // Reattachment keeps the original lease, including ephemeral leases.
+      // Never reacquire: acquisition can replace a non-reusable sandbox.
       const expected = input.reattachRemoteLease;
       const lease = await environmentsSvc.getLeaseById(expected.leaseId);
       if (!lease ||
@@ -347,6 +346,21 @@ export function environmentRunOrchestrator(
               || lease.providerLeaseId !== expected.providerLeaseId || lease.metadata?.remoteCwd !== expected.remoteCwd)) {
         throw new Error("native_remote_recovery_lease_mismatch");
       }
+      if (expected.kind !== "plugin_task") {
+        // The provider may have archived this retained sandbox while its
+        // controller was offline. Ready the exact lease under its lifecycle
+        // budget before short preparation commands (such as project-auth
+        // checks) can run. This API cannot acquire a replacement.
+        const resumed = await environmentRuntime.resumeRunLease({
+          environment, lease, requestedCwd: expected.remoteCwd,
+        });
+        if (!resumed || resumed.providerLeaseId !== expected.providerLeaseId ||
+            resumed.metadata?.expired === true) {
+          throw new Error("native_remote_recovery_lease_mismatch");
+        }
+        // Resume confirms allocation readiness. Its optional metadata may
+        // describe the allocation root; keep the validated lease's workspace.
+      }
       leaseRecord = { environment, lease, leaseContext: buildEnvironmentLeaseContext(input) };
     } else {
       leaseRecord = await acquireLease(acquisitionInput);

server/src/services/environment-runtime.ts

@@ -4307,7 +4307,7 @@ export function environmentRuntimeService(
       return { destroyed, failed, skippedLiveRun };
     },
 
-    async resumeRunLease(input: EnvironmentDriverLeaseInput): Promise<PluginEnvironmentLease | EnvironmentLease | null> {
+    async resumeRunLease(input: EnvironmentDriverResumeInput): Promise<PluginEnvironmentLease | EnvironmentLease | null> {
       if (await legacyShutdownWorkspaceResourceProtected(db, input.lease)) return null;
       const driver = requireDriverKey(getLeaseDriverKey(input.lease, input.environment));
       if (!driver.resumeRunLease) {

server/src/services/native-runtime/native-stop-accounting.integration.test.ts

@@ -222,7 +222,9 @@ describe("native Stop accounting after the last usage receipt", () => {
       const sync = vi.spyOn(workspaceSync, "prepareNativeWorkspaceSync").mockResolvedValue(null);
       const stage = vi.spyOn(attachmentStaging, "stageNativeRunnerWakeAttachments").mockRejectedValue(new Error("Stop cleanup must not stage attachments"));
       const acquire = vi.fn(async () => { throw new Error("Stop cleanup cannot acquire a new sandbox"); });
+      const resume = vi.fn(async () => ({ providerLeaseId: "existing-sandbox" }));
       const runtime = environmentRuntimeService(db, { drivers: [{ driver: "sandbox", acquireRunLease: acquire,
+        resumeRunLease: resume,
         resolveCapabilities: async () => classifyEnvironmentCapabilities({}),
         realizeWorkspace: async () => ({ cwd: home.home }),
         retryPendingSandboxTeardown: async ({ lease }) => {
@@ -244,6 +246,11 @@ describe("native Stop accounting after the last usage receipt", () => {
           const holds = await db.select().from(budgetReservations).where(eq(budgetReservations.runId, runId));
           expect(await readClaimedNativeStopRecovery(db, current, owner.leaseOwner ?? undefined)).toBe(true);
           expect(holds).toMatchObject([{ id: hold.id, state: "held" }]);
+          expect(resume).toHaveBeenCalledOnce();
+          expect(resume).toHaveBeenCalledWith(expect.objectContaining({
+            lease: expect.objectContaining({ id: native.environmentLeaseId, providerLeaseId: "existing-sandbox" }),
+            requestedCwd: "/workspace",
+          }));
           reachedSelection = true;
         } });
       try {