fix(environments): keep resuming an idle sandbox whose resume timed out (#21113)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on the next wake
> - When the resume call times out, the run failed after 90 seconds with
`reusable_sandbox_resume_failed (cause=transport_timeout)`
> - The provider keeps starting the sandbox after the host call times
out, so the sandbox is usually ready a little later
> - The run gave up after one call, and the task then waited behind a
Retry button
> - This pull request keeps the same run retrying an idle sandbox whose
resume only timed out, up to the provider's start deadline
> - The benefit is that recovery works automatically and a sandbox with
unsaved work is never destroyed

## Linked Issues or Issue Description

Refs #20626, #20828, #21020

**What happened?**
Three Codex runs on Daytona failed before provider dispatch with:
`Reusable sandbox lease "<id>" could not be resumed
(cause=transport_timeout); the lease was preserved and no replacement
was created.` Each run had `executionRecovery: { kind: "bootstrap",
providerWorkStarted: false }`. Each run took 100 to 122 seconds: one
resume call used the whole 90-second budget, so no in-run retry was
possible. The preserved leases were idle. Their last runs had ended
about two hours before, with a confirmed stop-and-retain. The workspaces
were not safe to replace, so the run failed strictly. The automatic
bootstrap retry was then refused ("The automatic retry could not
revalidate the original user continuation."), and all three tasks ended
`blocked` behind a Retry.

**Expected behavior**
A resume that only timed out, before any provider work, on a sandbox
that no live run holds, continues in the same run. The run keeps the
same sandbox and does not create a replacement or destroy the sandbox.

**Steps to reproduce**
1. Let a task's reusable Daytona sandbox stop and stay idle for some
hours (it auto-archives after 60 minutes).
2. Retry the task. The workspace has unsaved work, so replacement is not
safe.
3. The `environmentResumeLease` call takes longer than 90 seconds. The
host call times out (`RPC call "environmentResumeLease" timed out after
90000ms`), the 90-second budget is spent, and the run fails with
`reusable_sandbox_resume_failed`.

**Paperclip version or commit**
Staging at `849f109f` (current `master` has the same code path).

**Deployment mode**
Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases
on.

**Agent adapter(s) involved**
`codex_local` (Codex) on a Daytona sandbox environment. The change is
adapter-independent.

**Root cause**
`server/src/services/environment-runtime.ts`, reusable resume loop: the
first `environmentResumeLease` call got the whole 90-second budget as
its RPC timeout. When that call timed out, the loop saw no time left and
threw. The 90-second budget assumes that the bounded bootstrap retry
resumes the same sandbox later. That retry is not always admitted. A
timeout is different from a refusal: the provider is still starting the
sandbox.

## What Changed

- `reusable-sandbox-resume-replacement.ts`: add
`reusableSandboxLeaseIsIdle` (lease is `released` or `retained` and no
other live run holds it). `reusableSandboxReplacementIsSafe` now uses
it, with the same rules as before. Add
`REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS` (330 seconds, the Daytona start
deadline plus the call margin).
- `environment-runtime.ts`: when the 90-second budget ends on a
`transport_timeout` and the preserved lease is idle, extend the deadline
one time, up to the provider start deadline (at most 330 seconds from
the first attempt). The run keeps retrying the same sandbox with
backoff. Each call still has at most 90 seconds. Refusals, a live owner,
and other failures keep the current behavior. The run never replaces or
destroys the sandbox because of this extension.
- `doc/execution-semantics.md`: document the idle-timeout extension.

## Verification

- Red before the fix, with the new tests in
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`
(staging shape: released lease, finished owner run, workspace with
unsaved work, 300-second provider deadline, host RPC timeout): 2 tests
failed with the staging error `could not be resumed
(cause=transport_timeout); the lease was preserved and no replacement
was created.`
- Green after the fix: 16/16 pass. The timed-out idle sandbox resumes in
the same run with no create or destroy call. If every call times out,
the run stops at or before 330 seconds, makes more than one call, and
keeps the lease `released`. A sandbox that a live run still holds gets
one call and fails at 90 seconds, as before.
- `pnpm exec vitest run src/__tests__/environment-runtime.test.ts
src/services/environment-runtime-reusable-lease-probe.test.ts
src/services/remote-workspace-transport.test.ts`: 378 pass.
- `tsc -p server/tsconfig.json --noEmit` passes.
- `pnpm verify` exits 2 because a doc is a global CI input. Required CI
supplies the broad coverage.
- Not run: a live Daytona resume on staging. No later run attempted the
three preserved leases, so it is inferred, not observed, that the
sandboxes were ready after the host call timed out.

## Risks

- A run can now wait up to 330 seconds in preparation before its first
turn, but only when the resume timed out on an idle sandbox. Before this
change, that run failed and the task waited for a person.
- A provider that is down (not slow) now takes up to 330 seconds to fail
instead of 90 seconds. The failure is the same
`reusable_sandbox_resume_failed`, and the lease is still preserved.
- No persisted fields change. Leases saved by older releases follow the
same rule.
- The refused bootstrap retry admission for task Retry continuations is
a separate gap. This pull request does not change it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 08:28:32 -06:00
1 parent 704b3f4d63
commit ef2b53b675
4 files changed
+137 -23

No files matched your search

+9
View File
@@ -1444,6 +1444,15 @@ progress", or still stopping or archiving at the provider's start deadline). A
common cause is a follow-up that arrives while the provider is auto-stopping or
archiving an idle sandbox.
When a resume call only timed out and the preserved lease is idle (`released` or
`retained`, with no live run holding it), the provider is usually still starting
the sandbox. The run then keeps retrying the same sandbox up to the provider's
start deadline (at most 330 seconds from the first attempt) instead of failing
at 90 seconds. Each call still has at most 90 seconds. A retry that finds the
stopped sandbox still settling the earlier, timed-out call is retried too. This applies whether or
not the sandbox could be replaced, so a sandbox with unsaved work resumes in
the same run instead of waiting behind a Retry.
When the resume still fails, the run replaces the sandbox only when nothing lives
solely on it: the lease is `released` or `retained` and no live run holds it, the
task workspace is clean with a committed saved snapshot, and a native runner
@@ -153,11 +153,11 @@ describeEmbeddedPostgres("reusable sandbox resume failure replacement", () => {
return { companyId, agentId, environment, pluginId, runId, executionWorkspaceId, lease, providerLeaseId };
}
function worker(seeded: Awaited<ReturnType<typeof seed>>, resume: (attempt: number) => Promise<unknown>,
function worker(seeded: Awaited<ReturnType<typeof seed>>, resume: (attempt: number, timeoutMs: number) => Promise<unknown>,
options: { acquireFails?: boolean } = {}) {
let attempts = 0;
const call = vi.fn(async (_pluginId: string, method: string, _params: unknown, _timeoutMs?: number) => {
if (method === "environmentResumeLease") return await resume(++attempts);
const call = vi.fn(async (_pluginId: string, method: string, _params: unknown, timeoutMs?: number) => {
if (method === "environmentResumeLease") return await resume(++attempts, timeoutMs ?? 0);
if (method === "environmentAcquireLease") {
if (options.acquireFails) throw new JsonRpcCallError({ code: -32000, message: "create refused" });
return { providerLeaseId: "replacement-sandbox", metadata: { provider: "fake-plugin", image: "fake:test",
@@ -324,4 +324,84 @@ describeEmbeddedPostgres("reusable sandbox resume failure replacement", () => {
status: shape.leaseStatus ?? "retained", providerLeaseId: seeded.providerLeaseId,
});
});
function rpcTimeoutClock() {
let clock = Date.now();
const startedAt = clock;
vi.spyOn(Date, "now").mockImplementation(() => clock);
return {
elapsed: () => clock - startedAt,
timeOut: (timeoutMs: number) => {
clock += timeoutMs;
return new JsonRpcCallError({ code: -32003,
message: `RPC call "environmentResumeLease" timed out after ${timeoutMs}ms` });
},
};
}
it("keeps waiting for an idle sandbox whose resume timed out and resumes it in the same run (staging shape)", async () => {
const seeded = await seed({ leaseStatus: "released", dirty: true });
const clock = rpcTimeoutClock();
const { call, runtime } = worker(seeded, async (attempt, timeoutMs) => {
if (attempt === 1) throw clock.timeOut(timeoutMs);
return { providerLeaseId: seeded.providerLeaseId, metadata: { remoteCwd: "/home/daytona/paperclip-workspace" } };
});
const acquired = await acquire(seeded, runtime);
expect(acquired.lease.providerLeaseId).toBe(seeded.providerLeaseId);
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "resumed" });
const resumes = call.mock.calls.filter(([, method]) => method === "environmentResumeLease");
expect(resumes).toHaveLength(2);
for (const [, , , timeoutMs] of resumes) expect(timeoutMs).toBeLessThanOrEqual(90_000);
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentAcquireLease", expect.anything(), expect.anything());
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentDestroyLease", expect.anything(), expect.anything());
});
it("waits out the in-flight resume when a retry finds the stopped sandbox still settling", async () => {
const seeded = await seed({ leaseStatus: "released", dirty: true });
const clock = rpcTimeoutClock();
const { call, runtime } = worker(seeded, async (attempt, timeoutMs) => {
if (attempt === 1) throw clock.timeOut(timeoutMs);
if (attempt === 2) throw new JsonRpcCallError({ code: -32000,
message: "The stopped Daytona sandbox is still settling cancelled work. Retry shortly." });
return { providerLeaseId: seeded.providerLeaseId, metadata: { remoteCwd: "/home/daytona/paperclip-workspace" } };
});
const acquired = await acquire(seeded, runtime);
expect(acquired.lease.providerLeaseId).toBe(seeded.providerLeaseId);
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "resumed" });
expect(call.mock.calls.filter(([, method]) => method === "environmentResumeLease")).toHaveLength(3);
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentAcquireLease", expect.anything(), expect.anything());
});
it("stops waiting at the provider start deadline and keeps the sandbox when every resume times out", async () => {
const seeded = await seed({ leaseStatus: "released", dirty: true });
const clock = rpcTimeoutClock();
const { call, runtime } = worker(seeded, async (_attempt, timeoutMs) => { throw clock.timeOut(timeoutMs); });
await expect(acquire(seeded, runtime)).rejects.toThrow(
/could not be resumed \(cause=transport_timeout\); the lease was preserved and no replacement was created\.$/,
);
expect(call.mock.calls.filter(([, method]) => method === "environmentResumeLease").length).toBeGreaterThan(1);
expect(clock.elapsed()).toBeLessThanOrEqual(330_000);
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentAcquireLease", expect.anything(), expect.anything());
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentDestroyLease", expect.anything(), expect.anything());
await expect(environmentService(db).getLeaseById(seeded.lease.id)).resolves.toMatchObject({
status: "released", providerLeaseId: seeded.providerLeaseId,
});
});
it("does not extend the wait for a sandbox that a live run still holds", async () => {
const seeded = await seed({ previousRunStatus: "running", dirty: true });
const clock = rpcTimeoutClock();
const { call, runtime } = worker(seeded, async (_attempt, timeoutMs) => { throw clock.timeOut(timeoutMs); });
await expect(acquire(seeded, runtime)).rejects.toThrow(/could not be resumed \(cause=transport_timeout\)/);
expect(call.mock.calls.filter(([, method]) => method === "environmentResumeLease")).toHaveLength(1);
expect(clock.elapsed()).toBeLessThanOrEqual(90_000);
});
});
+24 -11
View File
@@ -3,10 +3,13 @@ import { ENVIRONMENT_LEASE_CONTROLLER_KEY } from "./environment-lease-controller
import { RetainedSandboxAbsentError, resumeReplyConfirmsAbsence } from "./retained-sandbox-absent-error.js";
import { readNativeWorkspaceMigration } from "./native-runtime/native-workspace-migration.js";
import { ReusableSandboxResumeError } from "./reusable-sandbox-resume-error.js";
import { sandboxProviderFailureLabel } from "./remote-workspace-transport.js";
import { NATIVE_PREDECESSOR_RETIREMENT_UNPROVEN_CODE, readReplaceBeforeReuseMarker } from "./native-runtime/predecessor-retirement.js";
import {
REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS,
REUSABLE_SANDBOX_RESUME_BUDGET_MS,
retireReplacedReusableSandboxLease,
reusableSandboxLeaseIsIdle,
reusableSandboxReplacementIsSafe,
sandboxStateTransitionFailure,
} from "./reusable-sandbox-resume-replacement.js";
@@ -1890,14 +1893,17 @@ function createSandboxEnvironmentDriver(
else try {
// One run spends a bounded budget resuming, retries included; the
// provider's own start deadline (five minutes on Daytona) is not it.
const configuredResumeTimeoutMs = Math.min(
resolvePluginSandboxRpcTimeoutMs(
workerConfig,
pluginProvider.resolved.driver.defaultAcquireTimeoutMs,
) ?? 60_000,
REUSABLE_SANDBOX_RESUME_BUDGET_MS,
const providerStartTimeoutMs = resolvePluginSandboxRpcTimeoutMs(
workerConfig,
pluginProvider.resolved.driver.defaultAcquireTimeoutMs,
) ?? 60_000;
const configuredResumeTimeoutMs = Math.min(providerStartTimeoutMs, REUSABLE_SANDBOX_RESUME_BUDGET_MS);
const resumeStartedAt = Date.now();
let resumeDeadline = resumeStartedAt + configuredResumeTimeoutMs;
const idleResumeDeadline = resumeStartedAt + Math.max(
configuredResumeTimeoutMs,
Math.min(providerStartTimeoutMs, REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS),
);
const resumeDeadline = Date.now() + configuredResumeTimeoutMs;
let retryDelayMs = 250;
let resumed: PluginEnvironmentLease;
while (true) {
@@ -1921,10 +1927,17 @@ function createSandboxEnvironmentDriver(
);
break;
} catch (error) {
if (
!transientSandboxResumeFailure(error) ||
Date.now() + retryDelayMs * 1.25 >= resumeDeadline
) throw error;
if (!transientSandboxResumeFailure(error)) throw error;
const retryAt = Date.now() + retryDelayMs * 1.25;
if (retryAt >= resumeDeadline) {
const waitForIdleStart = resumeDeadline < idleResumeDeadline && retryAt < idleResumeDeadline
&& sandboxProviderFailureLabel(error).label === "transport_timeout"
&& await reusableSandboxLeaseIsIdle({
db, lease: reusableLease, companyId: input.companyId, heartbeatRunId: input.heartbeatRunId,
});
if (!waitForIdleStart) throw error;
resumeDeadline = idleResumeDeadline;
}
const jitteredDelayMs = Math.max(
1,
Math.round(retryDelayMs * (0.75 + Math.random() * 0.5)),
@@ -17,6 +17,8 @@ import { sandboxProviderFailureLabel } from "./remote-workspace-transport.js";
*/
export const REUSABLE_SANDBOX_RESUME_BUDGET_MS = 90_000;
export const REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS = 330_000;
/** Failure reason recorded on a preserved lease retired in favor of a replacement. */
export const REUSABLE_SANDBOX_RESUME_REPLACED_REASON = "resume_failed";
@@ -28,11 +30,28 @@ export const REUSABLE_SANDBOX_RESUME_REPLACED_REASON = "resume_failed";
export function sandboxStateTransitionFailure(error: unknown): boolean {
for (let cause = error, depth = 0; cause && depth < 4; cause = (cause as { cause?: unknown }).cause, depth++) {
const message = cause instanceof Error ? cause.message : typeof cause === "string" ? cause : "";
if (/state change in progress|\bis still [a-z_]+ at its start deadline\b/i.test(message)) return true;
if (/state change in progress|\bis still [a-z_]+ at its start deadline\b|\bis still settling cancelled work\b/i.test(message)) return true;
}
return false;
}
export async function reusableSandboxLeaseIsIdle(input: {
db: Db;
lease: EnvironmentLease;
companyId: string;
heartbeatRunId: string | null;
}): Promise<boolean> {
const { db, lease } = input;
if (!input.heartbeatRunId || !lease.providerLeaseId || lease.companyId !== input.companyId) return false;
if (lease.status !== "released" && lease.status !== "retained") return false;
if (lease.heartbeatRunId && lease.heartbeatRunId !== input.heartbeatRunId) {
const [owner] = await db.select({ status: heartbeatRuns.status }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, lease.heartbeatRunId)));
if (owner && ["queued", "scheduled_retry", "running"].includes(owner.status)) return false;
}
return true;
}
/**
* Decide whether a preserved reusable sandbox that could not be resumed may be
* replaced.
@@ -59,8 +78,7 @@ export async function reusableSandboxReplacementIsSafe(input: {
cause: unknown;
}): Promise<boolean> {
const { db, lease } = input;
if (!input.heartbeatRunId || !lease.providerLeaseId || lease.companyId !== input.companyId) return false;
if (lease.status !== "released" && lease.status !== "retained") return false;
if (!lease.providerLeaseId || !await reusableSandboxLeaseIsIdle(input)) return false;
if (!input.executionWorkspaceId || lease.executionWorkspaceId !== input.executionWorkspaceId) return false;
// A replacement cannot succeed where the resume was refused for access, and
// an unavailable worker cannot create one either.
@@ -69,12 +87,6 @@ export async function reusableSandboxReplacementIsSafe(input: {
const status = failure.status ?? readEnvironmentResumeErrorDiagnostic(input.cause)?.httpStatus ?? null;
if (failure.label === "provider_access_denied" || failure.label === "permission_denied"
|| status === 401 || status === 403) return false;
// A run that still holds the sandbox may have work on it.
if (lease.heartbeatRunId && lease.heartbeatRunId !== input.heartbeatRunId) {
const [owner] = await db.select({ status: heartbeatRuns.status }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, lease.heartbeatRunId)));
if (owner && ["queued", "scheduled_retry", "running"].includes(owner.status)) return false;
}
const [workspace] = await db.select({ metadata: executionWorkspaces.metadata }).from(executionWorkspaces)
.where(and(eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.executionWorkspaceId)));