fix(daytona): keep the provider auto-stop strictly after the warm runner idle window (#19199)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote agent keeps its sandbox running after a turn. The host
keeps a warm runner for an idle window (`runnerIdleTimeoutMs`, or the
agent's `idleTimeoutMs`)
> - The Daytona provider also stops an idle sandbox after
`autoStopInterval` minutes. The default is 15 minutes
> - On staging, the warm idle window was 900,000 ms (15 minutes). The
two windows were equal
> - A follow-up turn at the end of the warm window resumed a sandbox
that Daytona was stopping. Daytona refused the resume in about 4
seconds, and the run failed at setup
> - #19195 added a bounded retry for that refusal. It did not remove the
cause
> - This pull request makes the two windows non-overlapping by
construction. Daytona derives its auto-stop for a warm reusable sandbox
from the same warm idle value that the host uses
> - The benefit is that a follow-up inside the warm window always finds
a running sandbox, and idle sandboxes still stop

## Linked Issues or Issue Description

Refs #19195 (bounded retry for a refused resume; this change removes the
common cause).

**What happened?**

On a staging soak, three follow-up runs failed at setup with a refused
sandbox resume. Each resume began 15 minutes 8 to 25 seconds after the
previous turn released the sandbox. The warm idle timeout was 900,000
ms. Daytona's default `autoStopInterval` is 15 minutes. Nothing in the
code kept these two values apart.

**Expected behavior**

A reusable sandbox that keeps a warm runner must not auto-stop inside
the warm idle window or at its edge. The provider backstop must fire
after the host's own idle handling, with a margin. Idle sandboxes must
still stop.

**Steps to reproduce**

1. Configure a Daytona sandbox environment with `reuseLease` and a warm
runner with `runnerIdleTimeoutMs: 900000` (or an agent with
`lifecycleMode: warm`, `idleTimeoutMs: 900000`).
2. Run a turn. Do not set `autoStopInterval`.
3. Before this change: the sandbox is created with `autoStopInterval:
15`, equal to the warm window. A follow-up near minute 15 races the
provider stop.

**Paperclip version or commit**

`c656b95b6b` (current `master`).

## What Changed

- Daytona plugin (`plugin.ts`): add `effectiveAutoStopIntervalMinutes`.
For a reusable lease that keeps a warm runner, the effective auto-stop
is at least `ceil(warm idle / 1 min) + 5` minutes. The plugin applies
the floor only when the host forwards `warmRunnerIdleTimeoutMs`, or when
the environment itself selects `runnerLifecycleMode: warm`. An explicit
`0` (disabled), a longer configured value, `inherit` with a per-turn
agent, per-turn, and non-reusable sandboxes keep the configured value.
- Daytona plugin: use the effective value at sandbox creation. On
resume, raise a shorter auto-stop on an existing sandbox. This update is
best effort and bounded to 5 seconds; the #19195 retry still covers a
refusal. The stale-handle refresh threshold keeps the configured
interval, so a cached handle is not trusted for longer before the
provider confirms the new interval. The warm window is not written to
the saved normalized config.
- Host (`runner-environment-lifecycle.ts`): add
`resolveWarmRunnerIdleTimeoutMsForRun`. It returns the run's effective
warm idle window (environment `runnerIdleTimeoutMs`, else the agent
`idleTimeoutMs`) for a warm `paperclip_runner` sandbox run. It shares
the lifecycle-mode resolution with `resolveRunnerEnvironmentForRun`.
- Host (`environment-run-orchestrator.ts`, `environment-runtime.ts`):
pass that value as `warmRunnerIdleTimeoutMs` to the sandbox-provider
driver on acquisition and on remote recovery resume. The driver adds it
to the provider worker config only. Lease identity, reuse matching, and
lease metadata are unchanged.
- Daytona manifest: describe the warm floor on `autoStopInterval`.
- `doc/execution-semantics.md`: document the contract.

## Verification

- Red (before the fix, same tests, with the floor disabled): 5 failures
in `plugin.test.ts` "warm idle window versus Daytona auto-stop". The
staging case failed with `expected 900000 to be greater than or equal to
1200000`. The resume case failed because `setAutostopInterval` was not
called.
- Red (host): the new `environment-runtime.test.ts` case first failed
because the provider worker config had no `runnerIdleTimeoutMs`. The
service wrapper dropped the field. The fix forwards it.
- Green: `npx vitest run --config vitest.config.ts` in
`packages/plugins/sandbox-providers/daytona`: 563 passed. This includes
a setter that never settles (resume completes after the 5-second bound)
and `inherit` with no forwarded window (explicit `autoStopInterval: 2`
stays 2).
- Green: `npx vitest run
src/services/runner-environment-lifecycle.test.ts
src/__tests__/environment-runtime.test.ts
src/__tests__/environment-run-orchestrator.test.ts` in `server/`: 242
passed. This includes recovery resume forwarding the admitted warm
window.
- The regression tests assert the relationship `autoStop >= warm idle +
margin` for several warm windows (125 s, 300 s default, 900 s, 1,800 s)
and for an explicit `autoStopInterval: 15`. They are not constant
checks.
- `node scripts/typecheck-low-memory.mjs` passes. `pnpm typecheck` in
the Daytona package passes.

## Risks

- A warm reusable Daytona sandbox now stays running up to the warm
window plus 5 minutes after its last activity, instead of 15 minutes.
With a 15-minute window this is 20 minutes. Cost is bounded and idle
sandboxes still stop.
- The resume-time `setAutostopInterval` call is best effort. If it
fails, the old interval stays and the #19195 retry still applies.
- Other providers are unchanged. They receive one extra host-owned
config key (`warmRunnerIdleTimeoutMs`) only for warm `paperclip_runner`
runs, and they ignore unknown keys.
- No lease identity change, so existing retained sandboxes are still
reused.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-05 01:10:19 -06:00
1 parent c656b95b6b
commit 037c58bb22
10 files changed
+288 -12

No files matched your search

@@ -924,6 +924,19 @@ describe("live remote runner lease reattachment", () => {
expect(result.leaseContext.executionWorkspaceId).toBe("ew-1");
expect(result.environment.config.reuseLease).toBe(false);
});
it("forwards the admitted warm idle window to the recovery resume", async () => {
const savedLease = originalLease();
mockGetLease.mockResolvedValue(savedLease);
const runtime = makeMockRuntime();
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun({
...input, admittedLifecycleMode: "warm", adapterConfig: { lifecycleMode: "warm", idleTimeoutMs: 1_800_000 },
});
expect(runtime.resumeRunLease).toHaveBeenCalledWith({
environment: result.environment, lease: savedLease, requestedCwd: "/remote/workspace",
warmRunnerIdleTimeoutMs: 1_800_000,
});
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
});
it("does not return the retained sandbox to preparation while its lifecycle resume is pending", async () => {
let ready = false;
let release!: () => void;
@@ -391,6 +391,29 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
expect(acquired.lease.metadata?.snapshot).toBe(config.snapshot);
});
it("forwards the run's warm idle window to the provider worker without changing lease identity", async () => {
const pluginId = randomUUID();
const { companyId, environment: baseEnvironment, runId } = await seedEnvironment();
const config = { provider: "daytona", image: "node:20", timeoutMs: 1234 };
await insertSandboxProviderPlugin(pluginId, { environmentDrivers: [{ driverKey: "daytona", kind: "sandbox_provider", displayName: "Daytona", configSchema: { type: "object" } }] });
await environmentService(db).update(baseEnvironment.id, { driver: "sandbox", config });
const environment: Environment = { ...baseEnvironment, driver: "sandbox", config };
const worker = createFreshPluginLeaseWorker(pluginId);
vi.mocked(worker.call).mockImplementation(async (_id, method) => {
if (method !== "environmentAcquireLease") throw new Error(`Unexpected ${method}`);
return { providerLeaseId: "warm-lease", metadata: { provider: "daytona", image: "node:20", remoteCwd: "/workspace" } };
});
const service = environmentRuntimeService(db, { pluginWorkerManager: worker });
const acquired = await service.acquireRunLease({ companyId,
environment: resolveRunnerEnvironmentForRun(environment, "paperclip_runner", { lifecycleMode: "warm", idleTimeoutMs: 900_000 }),
configuredEnvironment: environment, issueId: null, heartbeatRunId: runId, persistedExecutionWorkspace: null,
adapterType: "paperclip_runner", warmRunnerIdleTimeoutMs: 900_000 });
expect(worker.call).toHaveBeenCalledWith(pluginId, "environmentAcquireLease", expect.objectContaining({
config: expect.objectContaining({ reuseLease: true, warmRunnerIdleTimeoutMs: 900_000 }),
}), expect.any(Number));
expect(acquired.lease.metadata).not.toHaveProperty("warmRunnerIdleTimeoutMs");
});
it.each([["paperclip_runner", false, false], ["paperclip_runner", true, false], ["paperclip_runner", true, undefined], ["claude_local", false, false], [null, false, false]] as const)("pins qualified snapshots only for native execution (%s, warm=%s, savedReuse=%s)", async (adapterType, warm, savedReuse) => {
const pluginId = randomUUID();
const { companyId, environment: baseEnvironment, runId } = await seedEnvironment();
@@ -29,7 +29,7 @@ import type {
ExecutionWorkspaceConfig,
IssueExecutionWorkspaceSettings,
} from "@paperclipai/shared";
import { resolveRunnerEnvironmentForRun } from "./runner-environment-lifecycle.js";
import { resolveRunnerEnvironmentForRun, resolveWarmRunnerIdleTimeoutMsForRun } from "./runner-environment-lifecycle.js";
import { environmentService } from "./environments.js";
import {
environmentRuntimeService,
@@ -217,6 +217,7 @@ export function environmentRunOrchestrator(
executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
runnerTaskProjectIds?: readonly string[];
adapterType: string | null;
warmRunnerIdleTimeoutMs?: number;
}): Promise<EnvironmentRuntimeLeaseRecord> {
try {
return await environmentRuntime.acquireRunLease(input);
@@ -313,6 +314,9 @@ export function environmentRunOrchestrator(
}
// Step 2: Acquire lease
const warmRunnerIdleTimeoutMs = resolveWarmRunnerIdleTimeoutMsForRun(
selectedEnvironment, input.adapterType, input.adapterConfig, input.admittedLifecycleMode,
);
const acquisitionInput = {
companyId: input.companyId,
environment,
@@ -324,6 +328,7 @@ export function environmentRunOrchestrator(
executionWorkspaceSettings: input.executionWorkspaceSettings,
runnerTaskProjectIds: input.runnerTaskProjectIds,
adapterType: input.adapterType ?? null,
warmRunnerIdleTimeoutMs,
};
let leaseRecord: EnvironmentRuntimeLeaseRecord;
if (input.reattachRemoteLease) {
@@ -353,6 +358,7 @@ export function environmentRunOrchestrator(
// checks) can run. This API cannot acquire a replacement.
const resumed = await environmentRuntime.resumeRunLease({
environment, lease, requestedCwd: expected.remoteCwd,
...(warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs } : {}),
});
if (!resumed || resumed.providerLeaseId !== expected.providerLeaseId ||
resumed.metadata?.expired === true) {
+24 -4
View File
@@ -490,6 +490,13 @@ export interface EnvironmentDriverAcquireInput {
* behavior.
*/
requestedExpiresAt?: Date | null;
/**
* The run's warm runner idle window (ms) when the run keeps a warm runner on
* a reusable sandbox. The sandbox-provider driver forwards it to the provider
* worker only (never to lease identity or metadata), so the provider can keep
* its idle auto-stop strictly after the host's warm idle window.
*/
warmRunnerIdleTimeoutMs?: number;
/**
* Re-check the environment company binding inside the lease insert
* transaction. The login acquire paths set this so a managed reconciliation
@@ -545,6 +552,8 @@ export interface EnvironmentDriverLeaseInput {
export interface EnvironmentDriverResumeInput extends EnvironmentDriverLeaseInput {
requestedCwd?: string;
/** The run's warm runner idle window, forwarded to the provider worker only. */
warmRunnerIdleTimeoutMs?: number;
}
export interface EnvironmentDriverRealizeWorkspaceInput extends EnvironmentDriverLeaseInput {
@@ -1634,7 +1643,11 @@ function createSandboxEnvironmentDriver(
);
}
const workerConfig = stripSandboxProviderEnvelope(parsed.config);
const workerConfig = {
...stripSandboxProviderEnvelope(parsed.config),
// Provider-only view of the run's warm idle window (see the input doc).
...(input.warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs: input.warmRunnerIdleTimeoutMs } : {}),
};
const storedConfig = pinnedSnapshotId ? { ...storedParsed.config, snapshot: pinnedSnapshotId } : storedParsed.config;
if (pinnedSnapshotId) delete storedConfig.image;
const providerConfigForLease = sandboxConfigForLeaseMetadata(storedConfig);
@@ -2564,9 +2577,13 @@ function createSandboxEnvironmentDriver(
}
// This is a lifecycle resume, never acquisition: no replacement, host
// seed, lease mutation, or provider turn is allowed at this boundary.
const config = stripSandboxProviderEnvelope(await resolvePluginSandboxRuntimeConfig({
environment: input.environment, lease: input.lease, provider: providerKey,
}) as SandboxEnvironmentConfig);
const config = {
...stripSandboxProviderEnvelope(await resolvePluginSandboxRuntimeConfig({
environment: input.environment, lease: input.lease, provider: providerKey,
}) as SandboxEnvironmentConfig),
// Provider-only view of the run's warm idle window (see the input doc).
...(input.warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs: input.warmRunnerIdleTimeoutMs } : {}),
};
const resumed = await pluginWorkerManager.call(pluginId, "environmentResumeLease", {
driverKey: providerKey, companyId: input.lease.companyId, environmentId: input.environment.id,
issueId: input.lease.issueId, config, providerLeaseId: input.lease.providerLeaseId,
@@ -3800,6 +3817,8 @@ export function environmentRuntimeService(
* provider expiry only.
*/
requestedExpiresAt?: Date | null;
/** The run's warm runner idle window, forwarded to the provider worker only. */
warmRunnerIdleTimeoutMs?: number;
/**
* Re-check the environment company binding inside the lease insert
* transaction. The login acquire paths set this to reject a foreign-company
@@ -3831,6 +3850,7 @@ export function environmentRuntimeService(
applyCustomImageTemplate: input.applyCustomImageTemplate ?? false,
requestedExpiresAt: input.requestedExpiresAt ?? null,
assertCompanyBinding: input.assertCompanyBinding,
...(input.warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs: input.warmRunnerIdleTimeoutMs } : {}),
});
return {
@@ -1,5 +1,5 @@
import {expect,it} from "vitest";
import {resolveRunnerEnvironmentForRun} from "./runner-environment-lifecycle.js";
import {resolveRunnerEnvironmentForRun,resolveWarmRunnerIdleTimeoutMsForRun} from "./runner-environment-lifecycle.js";
it.each([undefined,"codex","acpx","opencode"])("defaults plugin Runner execution to warm without provider gates: %s",provider=>{
const environment={driver:"plugin" as const,config:{pluginKey:"fixture",driverKey:"tasks",driverConfig:{}}};
@@ -18,3 +18,15 @@ it("preserves explicit per-turn settings and the admitted recovery policy",()=>{
expect(resolveRunnerEnvironmentForRun({...environment,config:{...environment.config,runnerLifecycleMode:"per_turn"}},"paperclip_runner",{lifecycleMode:"warm"},undefined,"account:generation")).toMatchObject({config:{runnerLifecycleMode:"per_turn"}});
expect(resolveRunnerEnvironmentForRun(environment,"claude_local",{})).toBe(environment);
});
it("forwards the run's warm idle window for a warm sandbox-provider run only",()=>{
const sandbox={driver:"sandbox" as const,config:{provider:"daytona",reuseLease:true}};
// Agent-level warm with an inherited environment lifecycle: the provider must see the agent's window.
expect(resolveWarmRunnerIdleTimeoutMsForRun(sandbox,"paperclip_runner",{lifecycleMode:"warm",idleTimeoutMs:900_000})).toBe(900_000);
// The environment's own window wins over the agent's.
expect(resolveWarmRunnerIdleTimeoutMsForRun({...sandbox,config:{...sandbox.config,runnerLifecycleMode:"warm",runnerIdleTimeoutMs:600_000}},"paperclip_runner",{idleTimeoutMs:900_000})).toBe(600_000);
expect(resolveWarmRunnerIdleTimeoutMsForRun(sandbox,"paperclip_runner",{},"warm")).toBe(300_000);
expect(resolveWarmRunnerIdleTimeoutMsForRun(sandbox,"paperclip_runner",{lifecycleMode:"per_turn",idleTimeoutMs:900_000})).toBeUndefined();
expect(resolveWarmRunnerIdleTimeoutMsForRun({...sandbox,config:{...sandbox.config,runnerLifecycleMode:"per_turn"}},"paperclip_runner",{lifecycleMode:"warm"})).toBeUndefined();
expect(resolveWarmRunnerIdleTimeoutMsForRun(sandbox,"claude_local",{lifecycleMode:"warm"})).toBeUndefined();
expect(resolveWarmRunnerIdleTimeoutMsForRun({driver:"plugin" as const,config:{}},"paperclip_runner",{lifecycleMode:"warm"})).toBeUndefined();
});
@@ -1,6 +1,16 @@
import { resolvePaperclipRunnerIdleTimeoutMs } from "@paperclipai/adapter-utils";
import type { Environment } from "@paperclipai/shared";
function resolveRunnerLifecycleModeForRun(
config: Record<string, unknown>,
adapterConfig: Record<string, unknown>,
admittedLifecycleMode?: "warm" | "per_turn",
): unknown {
return admittedLifecycleMode ?? (config.runnerLifecycleMode === "warm" || config.runnerLifecycleMode === "per_turn"
? config.runnerLifecycleMode
: adapterConfig.lifecycleMode);
}
/** Resolve runner retention before acquiring a sandbox, not after it is ephemeral.
* This is a run-scoped view; never persist it over a shared environment's config.
* The driver still verifies provider support and chooses the actual lease policy.
@@ -14,9 +24,7 @@ export function resolveRunnerEnvironmentForRun<T extends Pick<Environment, "driv
): T {
if (adapterType !== "paperclip_runner" || !["sandbox", "plugin"].includes(environment.driver)) return environment;
const config = environment.config ?? {};
const lifecycleMode = admittedLifecycleMode ?? (config.runnerLifecycleMode === "warm" || config.runnerLifecycleMode === "per_turn"
? config.runnerLifecycleMode
: adapterConfig.lifecycleMode);
const lifecycleMode = resolveRunnerLifecycleModeForRun(config, adapterConfig, admittedLifecycleMode);
// Warm tasks need a managed credential identity; recovery keeps its admitted policy.
if (environment.driver === "plugin") return { ...environment, config: { ...config,
runnerLifecycleMode: admittedLifecycleMode ?? (managedAiCredentialIdentity && lifecycleMode !== "per_turn" ? "warm" : "per_turn"),
@@ -31,3 +39,22 @@ export function resolveRunnerEnvironmentForRun<T extends Pick<Environment, "driv
},
};
}
/**
* The warm runner idle window (ms) a sandbox-provider lease must outlive for
* this run, or undefined when the run keeps no warm runner on the sandbox. The
* host forwards it to the provider so a provider idle auto-stop (for example
* Daytona's `autoStopInterval`) is derived from the same value the host uses
* for its warm idle timer, instead of an independent default that can match it.
*/
export function resolveWarmRunnerIdleTimeoutMsForRun(
environment: Pick<Environment, "driver" | "config">,
adapterType: string,
adapterConfig: Record<string, unknown> = {},
admittedLifecycleMode?: "warm" | "per_turn",
): number | undefined {
if (adapterType !== "paperclip_runner" || environment.driver !== "sandbox") return undefined;
const config = environment.config ?? {};
if (resolveRunnerLifecycleModeForRun(config, adapterConfig, admittedLifecycleMode) !== "warm") return undefined;
return resolvePaperclipRunnerIdleTimeoutMs(config.runnerIdleTimeoutMs ?? adapterConfig.idleTimeoutMs);
}