fix(environments): resume a warm sandbox that its finished predecessor is still releasing (#20953)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm Paperclip Runner session keeps its Daytona sandbox between
runs. The next run of the same task resumes that sandbox through a
`reuse_by_environment` lease.
> - Acquisition only resumes a lease that is `released`, `retained`, or
held by the same run. A finished run releases its lease only after it
saves its instruction and workspace files, which takes several seconds.
> - A board wake that arrives in that window does not see the warm
lease. It creates a fresh sandbox. The session's durable backup cannot
be restored there, because its source lease is still held. The run fails
with `native_session_recovery_failed: runner_harness_state_mismatch:
backup_without_reusable_lease` and the task needs Retry, which starts a
new provider session.
> - This pull request makes acquisition wait, for a short bounded time,
for a lease that a recently finished predecessor is still releasing, and
then resume it.
> - The benefit is that the follow-up continues the same warm session in
the same sandbox, with no Retry, no restore and no lost provider
context.

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261111T205259-s6301` (Codex, Daytona warm,
volume-free server, `--ephemeral-server` from #20930), task t01. The
board re-wake run `e7129ec7` started at 20:57:44. Its predecessor
`5cf8c306` reported `run.terminal` at 20:57:38 but saved its instruction
copy until 20:57:51, so its warm lease was still `active`. At 20:57:53
the new run recorded `startup reuse: environment (fresh)`. It then
failed with `runner_harness_state_mismatch:
backup_without_reusable_lease` (error code
`native_session_interrupted`). Recording the `workspace_finalize`
operation then failed with a Postgres deadlock, and the run settled
`native_finalization_missing`. Retry run `79aa18ec` worked with a fresh
provider session. The K1 restart in that wave was at 20:55:32 and is not
the cause: six warm runs after it reused the sandbox normally.

**Root cause**
`reusableLeaseCanBeResumed` in
`server/src/services/environment-runtime.ts` excludes an `active` lease
held by another run. That rule is correct for a running predecessor. For
a predecessor that has already ended, the `active` state is a short
release tail. The new run creates a sandbox, and
`retiredSourceAuthorizesNativeBackupRestore`
(`native-retired-backup-source.ts`) correctly refuses to restore the
backup from a source lease that is not yet `released` or `retained`
(`idleWarmSourceRetiresBackup`). The executor's last branch
(`native-session-executor.ts`, host preparation, created allocation)
then throws `backup_without_reusable_lease`.

**How often**
Seen in 2 of 361 retained wave result directories on this machine
(`s6301` today, `s5201` earlier). It needs a board wake within a few
seconds of the predecessor's terminal event, so it is timing dependent.

## What Changed

- `server/src/services/environment-runtime.ts`: add
`waitForSettlingReusableLease`. Before the plugin and built-in sandbox
paths select a reusable lease, acquisition polls (250 ms) for up to 20 s
while a matching `reuse_by_environment` lease is `active` under a
different run that is `succeeded`, `failed`, `cancelled` or `timed_out`
and finished in the last two minutes. It then continues with the normal
selection, which resumes the released lease.
- A running predecessor causes no wait (unchanged: the new run gets a
new sandbox). A predecessor that ended more than two minutes ago and
never released causes no wait.
- `server/src/__tests__/environment-runtime.test.ts`: two red/green
cases (succeeded and failed predecessor release the lease 400 ms after
the successor starts acquiring) and one case for an old unreleased lease
(no wait, new sandbox, old lease untouched).
- `doc/execution-semantics.md`: describe the wait next to the existing
warm-reuse rules.

## Verification

- Red on master: `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/environment-runtime.test.ts -t predecessor` fails both
cases with `expected 'warm-2' to be 'warm-1'`: the successor created a
second sandbox instead of resuming the first.
- Green: the same command passes (3 tests).
`environment-runtime.test.ts`, `plugin-warm-tasks.integration.test.ts`
and `environment-run-orchestrator.test.ts` pass (288 tests). Server `tsc
--noEmit` passes.
- Not verified in a chaos wave. The window is a few seconds wide and
appeared in 2 of 361 waves, so one wave cannot show the change.

## Risks

- A successor can start up to 20 s later when its predecessor's release
is slow. Creating a sandbox and restoring a backup takes longer, and
failing needs a Retry. The two-minute bound keeps a stuck,
never-released lease from delaying every later run.
- If a caller held a lock that the predecessor's release needs, the wait
ends at 20 s and acquisition behaves as before.
- Compared with #20863 (fresh-sandbox fallback) and #20811 (unsaved
provider state marker): this change does not alter either. It reduces
how often a warm continuation reaches the fresh-sandbox branch. The
backup restore proof is unchanged, so an unproven backup still fails
closed with a working Retry.
- The `workspace_finalize` deadlock in the same run is a separate defect
and is not fixed here.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, test runs). Default context window,
extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/execution-semantics.md

@@ -3338,6 +3338,14 @@ run, it has a confirmed stop-and-retain receipt with state `stopped`, and no
 unsaved writer or pending native workspace export is bound to it. The backup
 stamp, single-owner, and later-provider-work checks above still apply.
 
+A run can also start while its terminal predecessor still holds the warm lease
+as `active`, because the predecessor releases the lease only after it saves its
+instruction and workspace files. Before it selects a lease, acquisition waits up
+to 20 seconds for that release when the matching lease belongs to a run that
+ended in the last two minutes. It then resumes the warm sandbox. A predecessor
+that is still running, or that ended earlier and never released its lease,
+causes no wait; that run gets a new sandbox as before.
+
 Reuse also skips a warm sandbox while its previous run's workspace export is
 still saving. A run that starts in that window gets a new sandbox, waits for
 the export, and realizes the saved files there. By the time it prepares the

server/src/__tests__/environment-runtime.test.ts

@@ -4922,6 +4922,56 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
     });
   });
 
+  async function seedSettlingWarmLease(predecessorFinishedAt: Date, predecessorStatus: "succeeded" | "failed" = "succeeded", stopReplyDelayMs = 0) {
+    const seeded = await seedReusablePluginSandboxLease("paperclip_runner");
+    await environmentService(db).releaseLease(seeded.reusableLease.id, "expired");
+    const warmEnvironment = resolveRunnerEnvironmentForRun(seeded.environment, "paperclip_runner", { lifecycleMode: "warm" }, undefined, "selected-account:generation");
+    let created = 0;
+    const call = vi.fn(async (_pluginId: string, method: string, input: any) => {
+      if (method === "environmentResumeLease") return { providerLeaseId: input.providerLeaseId, metadata: { remoteCwd: "/workspace" } };
+      if (method === "environmentAcquireLease") return { providerLeaseId: `warm-${++created}`, metadata: { remoteCwd: "/workspace" } };
+      if (method === "environmentStopLease") await new Promise(resolve => setTimeout(resolve, stopReplyDelayMs));
+      if (method === "environmentDestroyLease" || method === "environmentStopLease" || method === "environmentReleaseLease") return {
+        providerLeaseId: input.providerLeaseId, state: method === "environmentDestroyLease" ? "destroyed" : "stopped",
+      };
+      throw new Error(`Unexpected plugin method: ${method}`);
+    });
+    const runtime = environmentRuntimeService(db, { pluginWorkerManager: {
+      isRunning: () => true, call,
+      getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentStopLease", "environmentDestroyLease"] }),
+    } as unknown as PluginWorkerManager });
+    const input = { companyId: seeded.companyId, agentId: seeded.agentId, issueId: null, adapterType: "paperclip_runner",
+      environment: warmEnvironment, persistedExecutionWorkspace: { id: seeded.executionWorkspaceId, mode: "shared_workspace" as const } };
+    const predecessorRunId = randomUUID();
+    await db.insert(heartbeatRuns).values({ id: predecessorRunId, companyId: seeded.companyId, agentId: seeded.agentId, status: "running" });
+    const predecessor = await runtime.acquireRunLease({ ...input, heartbeatRunId: predecessorRunId });
+    await db.update(heartbeatRuns).set({ status: predecessorStatus, finishedAt: predecessorFinishedAt }).where(eq(heartbeatRuns.id, predecessorRunId));
+    const nextRunId = randomUUID();
+    await db.insert(heartbeatRuns).values({ id: nextRunId, companyId: seeded.companyId, agentId: seeded.agentId, status: "running" });
+    return { runtime, call, input, predecessor, predecessorRunId, nextRunId };
+  }
+
+  it.each([["succeeded", 0], ["failed", 0], ["failed", 1_500]] as const)("resumes a warm lease that its %s predecessor is still releasing (stop reply after %i ms) instead of creating a fresh sandbox", async (predecessorStatus, stopReplyDelayMs) => {
+    const f = await seedSettlingWarmLease(new Date(), predecessorStatus, stopReplyDelayMs);
+    const released = new Promise<void>(resolve => setTimeout(() => {
+      void f.runtime.releaseRunLeases(f.predecessorRunId, "released", undefined, "stop_and_retain").then(() => resolve());
+    }, 400));
+    const next = await f.runtime.acquireRunLease({ ...f.input, heartbeatRunId: f.nextRunId });
+    await released;
+    expect(next.lease.providerLeaseId).toBe(f.predecessor.lease.providerLeaseId);
+    expect(next.lease.heartbeatRunId).toBe(f.nextRunId);
+    expect(f.call.mock.calls.filter(call => call[1] === "environmentAcquireLease")).toHaveLength(1);
+  });
+
+  it("does not wait for a warm lease that a predecessor finished long ago and never released", async () => {
+    const f = await seedSettlingWarmLease(new Date(Date.now() - 10 * 60_000));
+    const startedAt = Date.now();
+    const next = await f.runtime.acquireRunLease({ ...f.input, heartbeatRunId: f.nextRunId });
+    expect(Date.now() - startedAt).toBeLessThan(5_000);
+    expect(next.lease.providerLeaseId).not.toBe(f.predecessor.lease.providerLeaseId);
+    await expect(environmentService(db).getLeaseById(f.predecessor.lease.id)).resolves.toMatchObject({ status: "active" });
+  });
+
   it("does not retain or resume plugin-backed sandbox leases unless the provider opts in", async () => {
     const pluginId = randomUUID();
     const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment();

server/src/services/environment-runtime.ts

@@ -24,7 +24,7 @@ import { beginIdleTrackedWork } from "./task-admission.js";
 import { historicalProviderCredentials } from "./historical-provider-credentials.js";
 import { ownedWorkspaceRecoveryLease, type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js";
 import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
-import { hasStopOnlyCleanup, prepareSandboxStopAndRetain, readStopOnlyCleanup, settleStopOnlyCleanup, stopOnlyCleanupKey } from "./sandbox-stop-and-retain.js";
+import { hasStopOnlyCleanup, prepareSandboxStopAndRetain, readStopOnlyCleanup, SANDBOX_STOP_AND_RETAIN_KEY, settleStopOnlyCleanup, stopOnlyCleanupKey } from "./sandbox-stop-and-retain.js";
 import { JsonRpcCallError, readEnvironmentAcquisitionDiagnostic, readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk";
 import { preserveEnvironmentSyncOutErrorDiagnostic } from "./environment-sync-out-error.js";
 import { remoteTerminationReceipt } from "./remote-execution-termination.js";
@@ -1100,6 +1100,37 @@ function reusableSandboxLeaseScopeMatches(input: {
   });
 }
 
+const SETTLING_REUSABLE_LEASE_WAIT_MS = 20_000;
+const SETTLING_REUSABLE_LEASE_RECENT_MS = 2 * 60_000;
+
+async function waitForSettlingReusableLease(db: Db, input: {
+  companyId: string; environmentId: string; executionWorkspaceId: string | null; issueId: string | null;
+  agentId: string; heartbeatRunId: string;
+}) {
+  const deadline = Date.now() + SETTLING_REUSABLE_LEASE_WAIT_MS;
+  for (;;) {
+    const [settling] = await db.select({ id: environmentLeases.id }).from(environmentLeases)
+      .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId),
+        eq(heartbeatRuns.companyId, environmentLeases.companyId)))
+      .where(and(
+        eq(environmentLeases.companyId, input.companyId),
+        eq(environmentLeases.environmentId, input.environmentId),
+        eq(environmentLeases.leasePolicy, "reuse_by_environment"),
+        or(eq(environmentLeases.status, "active"), and(eq(environmentLeases.status, "pending_cleanup"),
+          sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) ? ${SANDBOX_STOP_AND_RETAIN_KEY}`)),
+        ne(environmentLeases.heartbeatRunId, input.heartbeatRunId),
+        input.executionWorkspaceId
+          ? eq(environmentLeases.executionWorkspaceId, input.executionWorkspaceId)
+          : and(sql`${environmentLeases.executionWorkspaceId} is null`, eq(environmentLeases.issueId, input.issueId!)),
+        sql`${environmentLeases.metadata}->>'agentId' = ${input.agentId}`,
+        inArray(heartbeatRuns.status, ["succeeded", "failed", "cancelled", "timed_out"]),
+        sql`${heartbeatRuns.finishedAt} > now() - make_interval(secs => ${SETTLING_REUSABLE_LEASE_RECENT_MS / 1000})`,
+      )).limit(1);
+    if (!settling || Date.now() >= deadline) return;
+    await new Promise(resolve => setTimeout(resolve, 250));
+  }
+}
+
 function reusableLeaseCanBeResumed(input: {
   lease: Pick<EnvironmentLease, "status" | "heartbeatRunId">;
   heartbeatRunId: string | null;
@@ -1748,6 +1779,12 @@ function createSandboxEnvironmentDriver(
         // We also filter out leases whose policy is not reuse_by_environment
         // and whose status is not reusable so non-reusable, cleanup-pending,
         // or terminal rows cannot be matched.
+        if (supportsReusableLeases && parsed.config.reuseLease && input.heartbeatRunId !== null
+          && (input.executionWorkspaceId !== null || input.issueId !== null) && input.agentId !== null) {
+          await waitForSettlingReusableLease(db, { companyId: input.companyId, environmentId: input.environment.id,
+            executionWorkspaceId: input.executionWorkspaceId, issueId: input.issueId, agentId: input.agentId,
+            heartbeatRunId: input.heartbeatRunId });
+        }
         const reusableCandidateLeases =
           supportsReusableLeases &&
           parsed.config.reuseLease &&
@@ -2119,6 +2156,12 @@ function createSandboxEnvironmentDriver(
               providerConfig: providerConfigForLease,
             })
           : null;
+      if (supportsReusableLeases && parsed.config.reuseLease && input.heartbeatRunId !== null
+        && (input.executionWorkspaceId !== null || input.issueId !== null) && input.agentId !== null) {
+        await waitForSettlingReusableLease(db, { companyId: input.companyId, environmentId: input.environment.id,
+          executionWorkspaceId: input.executionWorkspaceId, issueId: input.issueId, agentId: input.agentId,
+          heartbeatRunId: input.heartbeatRunId });
+      }
       const reusableCandidateLeases =
         supportsReusableLeases &&
         parsed.config.reuseLease &&