fix(daytona): keep the provider auto-stop strictly after the warm runner idle window (#19199)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote agent keeps its sandbox running after a turn. The host
keeps a warm runner for an idle window (`runnerIdleTimeoutMs`, or the
agent's `idleTimeoutMs`)
> - The Daytona provider also stops an idle sandbox after
`autoStopInterval` minutes. The default is 15 minutes
> - On staging, the warm idle window was 900,000 ms (15 minutes). The
two windows were equal
> - A follow-up turn at the end of the warm window resumed a sandbox
that Daytona was stopping. Daytona refused the resume in about 4
seconds, and the run failed at setup
> - #19195 added a bounded retry for that refusal. It did not remove the
cause
> - This pull request makes the two windows non-overlapping by
construction. Daytona derives its auto-stop for a warm reusable sandbox
from the same warm idle value that the host uses
> - The benefit is that a follow-up inside the warm window always finds
a running sandbox, and idle sandboxes still stop

## Linked Issues or Issue Description

Refs #19195 (bounded retry for a refused resume; this change removes the
common cause).

**What happened?**

On a staging soak, three follow-up runs failed at setup with a refused
sandbox resume. Each resume began 15 minutes 8 to 25 seconds after the
previous turn released the sandbox. The warm idle timeout was 900,000
ms. Daytona's default `autoStopInterval` is 15 minutes. Nothing in the
code kept these two values apart.

**Expected behavior**

A reusable sandbox that keeps a warm runner must not auto-stop inside
the warm idle window or at its edge. The provider backstop must fire
after the host's own idle handling, with a margin. Idle sandboxes must
still stop.

**Steps to reproduce**

1. Configure a Daytona sandbox environment with `reuseLease` and a warm
runner with `runnerIdleTimeoutMs: 900000` (or an agent with
`lifecycleMode: warm`, `idleTimeoutMs: 900000`).
2. Run a turn. Do not set `autoStopInterval`.
3. Before this change: the sandbox is created with `autoStopInterval:
15`, equal to the warm window. A follow-up near minute 15 races the
provider stop.

**Paperclip version or commit**

`c656b95b6b` (current `master`).

## What Changed

- Daytona plugin (`plugin.ts`): add `effectiveAutoStopIntervalMinutes`.
For a reusable lease that keeps a warm runner, the effective auto-stop
is at least `ceil(warm idle / 1 min) + 5` minutes. The plugin applies
the floor only when the host forwards `warmRunnerIdleTimeoutMs`, or when
the environment itself selects `runnerLifecycleMode: warm`. An explicit
`0` (disabled), a longer configured value, `inherit` with a per-turn
agent, per-turn, and non-reusable sandboxes keep the configured value.
- Daytona plugin: use the effective value at sandbox creation. On
resume, raise a shorter auto-stop on an existing sandbox. This update is
best effort and bounded to 5 seconds; the #19195 retry still covers a
refusal. The stale-handle refresh threshold keeps the configured
interval, so a cached handle is not trusted for longer before the
provider confirms the new interval. The warm window is not written to
the saved normalized config.
- Host (`runner-environment-lifecycle.ts`): add
`resolveWarmRunnerIdleTimeoutMsForRun`. It returns the run's effective
warm idle window (environment `runnerIdleTimeoutMs`, else the agent
`idleTimeoutMs`) for a warm `paperclip_runner` sandbox run. It shares
the lifecycle-mode resolution with `resolveRunnerEnvironmentForRun`.
- Host (`environment-run-orchestrator.ts`, `environment-runtime.ts`):
pass that value as `warmRunnerIdleTimeoutMs` to the sandbox-provider
driver on acquisition and on remote recovery resume. The driver adds it
to the provider worker config only. Lease identity, reuse matching, and
lease metadata are unchanged.
- Daytona manifest: describe the warm floor on `autoStopInterval`.
- `doc/execution-semantics.md`: document the contract.

## Verification

- Red (before the fix, same tests, with the floor disabled): 5 failures
in `plugin.test.ts` "warm idle window versus Daytona auto-stop". The
staging case failed with `expected 900000 to be greater than or equal to
1200000`. The resume case failed because `setAutostopInterval` was not
called.
- Red (host): the new `environment-runtime.test.ts` case first failed
because the provider worker config had no `runnerIdleTimeoutMs`. The
service wrapper dropped the field. The fix forwards it.
- Green: `npx vitest run --config vitest.config.ts` in
`packages/plugins/sandbox-providers/daytona`: 563 passed. This includes
a setter that never settles (resume completes after the 5-second bound)
and `inherit` with no forwarded window (explicit `autoStopInterval: 2`
stays 2).
- Green: `npx vitest run
src/services/runner-environment-lifecycle.test.ts
src/__tests__/environment-runtime.test.ts
src/__tests__/environment-run-orchestrator.test.ts` in `server/`: 242
passed. This includes recovery resume forwarding the admitted warm
window.
- The regression tests assert the relationship `autoStop >= warm idle +
margin` for several warm windows (125 s, 300 s default, 900 s, 1,800 s)
and for an explicit `autoStopInterval: 15`. They are not constant
checks.
- `node scripts/typecheck-low-memory.mjs` passes. `pnpm typecheck` in
the Daytona package passes.

## Risks

- A warm reusable Daytona sandbox now stays running up to the warm
window plus 5 minutes after its last activity, instead of 15 minutes.
With a 15-minute window this is 20 minutes. Cost is bounded and idle
sandboxes still stop.
- The resume-time `setAutostopInterval` call is best effort. If it
fails, the old interval stays and the #19195 retry still applies.
- Other providers are unchanged. They receive one extra host-owned
config key (`warmRunnerIdleTimeoutMs`) only for warm `paperclip_runner`
runs, and they ignore unknown keys.
- No lease identity change, so existing retained sandboxes are still
reused.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-05 01:10:19 -06:00
1 parent c656b95b6b
commit 037c58bb22
10 files changed
+288 -12

No files matched your search

+12
View File
@@ -1022,6 +1022,18 @@ replacement. The run records the refusal by closed labels only (cause label, HTT
status, errno-style code), never by provider text. On exhaustion, the task gets a
specific recovery notice, and generic recovery does not grant another budget.
The provider's own idle auto-stop must not coincide with the warm runner idle
window. The host forwards the run's effective warm idle window (environment
`runnerIdleTimeoutMs`, else the agent's `idleTimeoutMs`) to the sandbox provider
worker as `warmRunnerIdleTimeoutMs`, on acquisition and on recovery resume. It is
not part of lease identity or metadata. Daytona derives its effective
`autoStopInterval` for a warm reusable sandbox as at least the warm window plus
5 minutes, applies it at creation, and raises an older sandbox's shorter interval
on resume (a bounded, best-effort update). Its cached-handle freshness keeps using
the configured interval until the provider confirms the new one. An explicit `0`
(auto-stop disabled), a longer configured interval, per-turn, and non-reusable
sandboxes are unchanged, and idle sandboxes still auto-stop.
When an admitted continuation reuses a sandbox after verified cleanup, an
unpublished preparation uses a directory bound to its writer epoch and lease.
Earlier unpublished directories and their receipts stay intact. Repeating the
@@ -160,7 +160,7 @@ const manifest: PaperclipPluginManifestV1 = {
autoStopInterval: {
type: "number",
description:
"Daytona auto-stop interval in minutes. `0` disables auto-stop. Defaults to 15 when unset.",
"Daytona auto-stop interval in minutes. `0` disables auto-stop. Defaults to 15 when unset. A reusable sandbox that keeps a warm runner never auto-stops before the warm idle window plus 5 minutes.",
default: 15,
},
autoArchiveInterval: {
@@ -35,6 +35,7 @@ import plugin, {
__resetDaytonaSandboxHandleCacheForTest,
__getDaytonaWritableDirsForTest,
__setDaytonaPluginContextForTest,
WARM_IDLE_AUTO_STOP_MARGIN_MINUTES,
} from "./plugin.js";
import type { PluginContext } from "@paperclipai/plugin-sdk";
import { environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic, PluginEnvironmentStopError, environmentStopErrorData, readEnvironmentStopFailure } from "@paperclipai/plugin-sdk";
@@ -353,6 +354,8 @@ function createMockSandbox(overrides: {
autoDestroyAt: overrides.autoDestroyAt ?? undefined,
updatedAt: overrides.updatedAt,
autoDeleteInterval: -1,
autoStopInterval: 15,
setAutostopInterval: vi.fn().mockResolvedValue(undefined),
getWorkDir: vi.fn().mockResolvedValue(overrides.workDir ?? "/home/daytona"),
getUserHomeDir: vi.fn().mockResolvedValue("/home/daytona"),
start: vi.fn().mockResolvedValue(undefined),
@@ -953,6 +956,101 @@ describe("Daytona sandbox provider plugin", () => {
});
});
describe("warm idle window versus Daytona auto-stop", () => {
// The host keeps a warm runner on a reusable sandbox for runnerIdleTimeoutMs.
// Daytona must never auto-stop that sandbox inside (or at the edge of) the
// warm window, or a follow-up at the window's end races the provider stop.
const acquire = async (config: Record<string, unknown>) => {
process.env.DAYTONA_API_KEY = "host-key";
mockCreate.mockResolvedValue(createMockSandbox());
await plugin.definition.onEnvironmentAcquireLease?.({
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", runId: "run-1",
config: { image: "node:20", timeoutMs: 300000, ...config },
});
const [createParams] = mockCreate.mock.calls.at(-1) as [Record<string, unknown>];
return createParams.autoStopInterval as number;
};
const resume = async (config: Record<string, unknown>) => {
process.env.DAYTONA_API_KEY = "host-key";
const sandbox = createMockSandbox({ id: "sandbox-warm", state: "started" });
sandbox.process.executeCommand.mockImplementation(async (command: string) => ({ exitCode: 0,
result: command.startsWith("cat ") ? JSON.stringify({ token: "sentinel-token" }) : "bash" }));
mockGet.mockResolvedValue(sandbox);
await plugin.definition.onEnvironmentResumeLease!({
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", providerLeaseId: sandbox.id,
config,
leaseMetadata: { workspaceSentinel: {
path: "/home/daytona/paperclip-workspace/.paperclip-runtime/reusable-sandbox-lease.json",
token: "sentinel-token", result: "written" } },
});
return sandbox;
};
const marginMs = WARM_IDLE_AUTO_STOP_MARGIN_MINUTES * 60_000;
it.each([
[{ runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 900_000 }, 900_000],
[{ runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 900_000, autoStopInterval: 15 }, 900_000],
// Agent-level warm on an inherit environment: the host forwards the window.
[{ runnerLifecycleMode: "inherit", warmRunnerIdleTimeoutMs: 1_800_000 }, 1_800_000],
[{ warmRunnerIdleTimeoutMs: 900_000, autoStopInterval: 15 }, 900_000],
[{ runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 125_000, autoStopInterval: 2 }, 125_000],
[{ runnerLifecycleMode: "warm" }, 300_000],
])("auto-stops a warm reusable sandbox only after the warm idle window plus a margin (%o)", async (config, warmIdleMs) => {
expect(marginMs).toBeGreaterThan(0);
const autoStopMs = (await acquire({ reuseLease: true, ...config })) * 60_000;
expect(autoStopMs).toBeGreaterThanOrEqual(warmIdleMs + marginMs);
});
it("keeps a longer configured auto-stop and an explicit 0 (disabled)", async () => {
expect(await acquire({ reuseLease: true, runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 300_000, autoStopInterval: 60 })).toBe(60);
expect(await acquire({ reuseLease: true, runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 900_000, autoStopInterval: 0 })).toBe(0);
});
it("leaves per-turn and non-reusable sandboxes on the configured auto-stop", async () => {
expect(await acquire({ reuseLease: false, runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 900_000 })).toBe(15);
expect(await acquire({ reuseLease: true, runnerLifecycleMode: "per_turn", runnerIdleTimeoutMs: 900_000 })).toBe(15);
// Inherit with a per-turn agent: the host forwards no warm window.
expect(await acquire({ reuseLease: true, runnerLifecycleMode: "inherit", autoStopInterval: 2 })).toBe(2);
expect(await acquire({ reuseLease: true, autoStopInterval: 2 })).toBe(2);
});
it("raises the auto-stop of an existing warm sandbox on resume", async () => {
const sandbox = await resume({ reuseLease: true, runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 900_000 });
expect(sandbox.setAutostopInterval).toHaveBeenCalledTimes(1);
const [minutes] = sandbox.setAutostopInterval.mock.calls[0] as [number];
expect(minutes * 60_000).toBeGreaterThanOrEqual(900_000 + marginMs);
});
it("bounds a resume-time auto-stop update that never settles", async () => {
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
try {
process.env.DAYTONA_API_KEY = "host-key";
const sandbox = createMockSandbox({ id: "sandbox-slow", state: "started" });
sandbox.setAutostopInterval.mockReturnValue(new Promise<void>(() => {}));
sandbox.process.executeCommand.mockImplementation(async (command: string) => ({ exitCode: 0,
result: command.startsWith("cat ") ? JSON.stringify({ token: "sentinel-token" }) : "bash" }));
mockGet.mockResolvedValue(sandbox);
const pending = plugin.definition.onEnvironmentResumeLease!({
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", providerLeaseId: sandbox.id,
config: { reuseLease: true, warmRunnerIdleTimeoutMs: 900_000 },
leaseMetadata: { workspaceSentinel: {
path: "/home/daytona/paperclip-workspace/.paperclip-runtime/reusable-sandbox-lease.json",
token: "sentinel-token", result: "written" } },
});
await vi.advanceTimersByTimeAsync(6_000);
await expect(pending).resolves.toMatchObject({ providerLeaseId: sandbox.id });
expect(sandbox.setAutostopInterval).toHaveBeenCalledTimes(1);
} finally {
vi.useRealTimers();
}
});
it("does not change auto-stop on resume of a per-turn reusable sandbox", async () => {
const sandbox = await resume({ reuseLease: true, runnerLifecycleMode: "per_turn" });
expect(sandbox.setAutostopInterval).not.toHaveBeenCalled();
});
});
describe("fresh acquisition deadline", () => {
const params = {
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", runId: "run-1",
@@ -194,6 +194,12 @@ interface DaytonaDriverConfig {
ttlMinutes: number | null;
reuseLease: boolean;
archiveOnRelease: boolean;
/**
* Host-owned warm runner idle window (ms) for a reusable warm lease, or null
* when the host keeps no warm runner on this sandbox. Never persisted in the
* normalized provider config; it only raises the effective auto-stop.
*/
warmIdleTimeoutMs: number | null;
}
type WorkspaceSentinelResult = {
@@ -238,6 +244,46 @@ const WORKSPACE_SENTINEL_RELATIVE_PATH = ".paperclip-runtime/reusable-sandbox-le
// - autoArchive: archive *stopped* sandboxes so they leave the disk quota.
// - autoDelete: backstop reaper for sandboxes nobody resumes.
const DEFAULT_AUTO_STOP_INTERVAL_MINUTES = 15;
// The host keeps a warm paperclip_runner on a reusable sandbox for
// `runnerIdleTimeoutMs` after a turn (host default 300000 ms). If Daytona's
// idle auto-stop fires at the same moment, a follow-up that arrives at the end
// of the warm window resumes a sandbox the provider is stopping and the resume
// is refused. Derive the provider auto-stop from the warm window so the
// provider backstop always fires strictly after the host's own idle handling:
// effective auto-stop >= ceil(warm idle) + this margin. Idle sandboxes still
// stop; only the backstop moves later by the margin.
export const DEFAULT_WARM_RUNNER_IDLE_TIMEOUT_MS = 300_000;
export const WARM_IDLE_AUTO_STOP_MARGIN_MINUTES = 5;
/**
* Effective Daytona auto-stop (minutes) for a sandbox. An explicit 0 (disabled)
* is preserved. A warm reusable lease never auto-stops before the host's warm
* idle window plus {@link WARM_IDLE_AUTO_STOP_MARGIN_MINUTES}.
*/
export function effectiveAutoStopIntervalMinutes(
config: Pick<DaytonaDriverConfig, "autoStopInterval" | "warmIdleTimeoutMs">,
): number | null {
const configured = config.autoStopInterval;
if (configured === 0 || config.warmIdleTimeoutMs == null) return configured;
const floor = Math.ceil(config.warmIdleTimeoutMs / 60_000) + WARM_IDLE_AUTO_STOP_MARGIN_MINUTES;
return Math.max(configured ?? DEFAULT_AUTO_STOP_INTERVAL_MINUTES, floor);
}
function parseWarmIdleTimeoutMs(raw: Record<string, unknown>): number | null {
// Only a reusable lease can host a warm runner.
if (raw.reuseLease !== true) return null;
// The host forwards the run's resolved warm window (environment or agent
// setting) only when the run actually keeps a warm runner.
const forwarded = parseOptionalInteger(raw.warmRunnerIdleTimeoutMs);
if (forwarded != null && forwarded > 0) return forwarded;
// An environment that itself selects a warm runner (no host forwarding, for
// example a legacy warm lifecycle) uses its own window. Inherit and per_turn
// apply no floor: the host forwards the window when the run is warm.
if (raw.runnerLifecycleMode !== "warm") return null;
const configured = parseOptionalInteger(raw.runnerIdleTimeoutMs);
return configured != null && configured > 0 ? configured : DEFAULT_WARM_RUNNER_IDLE_TIMEOUT_MS;
}
const DEFAULT_AUTO_ARCHIVE_INTERVAL_MINUTES = 60;
const DEFAULT_AUTO_DELETE_INTERVAL_MINUTES = 7 * 24 * 60; // 7 days
@@ -261,6 +307,10 @@ const GIT_NETWORK_TIMEOUT_MS = 120_000;
// less disables the extra bound.
const DEFAULT_LIVENESS_TIMEOUT_MS = 30_000;
// Bound on the optional resume-time auto-stop update, so a slow provider
// write cannot consume the host's resume budget.
const AUTO_STOP_UPDATE_TIMEOUT_MS = 5_000;
// Extra margin added to the SDK start/recover timeout when the plugin wraps
// those lifecycle calls in its own per-call bound. The SDK call already carries
// a `timeoutSeconds` deadline; the wrapper is a backstop for a connection-level
@@ -319,6 +369,7 @@ function parseDriverConfig(raw: Record<string, unknown>): DaytonaDriverConfig {
ttlMinutes: raw.ttlMinutes == null || raw.ttlMinutes === "" ? null : Number(raw.ttlMinutes),
reuseLease: raw.reuseLease === true,
archiveOnRelease: raw.archiveOnRelease === true,
warmIdleTimeoutMs: parseWarmIdleTimeoutMs(raw),
};
}
@@ -361,7 +412,7 @@ function buildCreateParams(
const base: CreateSandboxBaseParams = {
labels,
language: config.language ?? undefined,
autoStopInterval: config.autoStopInterval ?? undefined,
autoStopInterval: effectiveAutoStopIntervalMinutes(config) ?? undefined,
autoArchiveInterval: config.autoArchiveInterval ?? undefined,
autoDeleteInterval: config.autoDeleteInterval ?? undefined,
...(config.ttlMinutes != null ? { ttlMinutes: config.ttlMinutes } : {}),
@@ -2667,9 +2718,11 @@ const plugin = definePlugin(trackDaytonaHooks({
return { ok: false, errors };
}
// The warm idle window is host-owned; keep it out of the saved provider config.
const { warmIdleTimeoutMs: _warmIdleTimeoutMs, ...normalizedConfig } = config;
return {
ok: true,
normalizedConfig: { ...config },
normalizedConfig,
};
},
@@ -2949,6 +3002,18 @@ const plugin = definePlugin(trackDaytonaHooks({
}
const resumedFromState = sandbox.state ?? null;
await ensureSandboxStarted(sandbox, toTimeoutSeconds(config.timeoutMs), scope);
// A sandbox created before the warm floor (or under a shorter warm
// window) keeps its old auto-stop. Raise it on resume so the provider
// backstop cannot race the host's warm idle window. Best effort: the
// host's bounded resume retry still covers a refused resume.
const autoStopInterval = effectiveAutoStopIntervalMinutes(config);
const currentAutoStop = sandbox.autoStopInterval;
if (autoStopInterval != null && autoStopInterval > 0 && config.warmIdleTimeoutMs != null
&& typeof currentAutoStop === "number" && currentAutoStop > 0 && currentAutoStop < autoStopInterval) {
await withLivenessTimeout("sandbox.setAutostopInterval",
Math.max(1, Math.min(config.livenessTimeoutMs > 0 ? config.livenessTimeoutMs : AUTO_STOP_UPDATE_TIMEOUT_MS, AUTO_STOP_UPDATE_TIMEOUT_MS)),
() => Promise.resolve().then(() => sandbox.setAutostopInterval(autoStopInterval))).catch(() => undefined);
}
try {
return await traceLeaseStep("lease.verify", async () => {
const remoteCwd = await traceLeaseStep("lease.workspace", () => resolveSandboxWorkingDirectory(sandbox, false));
@@ -924,6 +924,19 @@ describe("live remote runner lease reattachment", () => {
expect(result.leaseContext.executionWorkspaceId).toBe("ew-1");
expect(result.environment.config.reuseLease).toBe(false);
});
it("forwards the admitted warm idle window to the recovery resume", async () => {
const savedLease = originalLease();
mockGetLease.mockResolvedValue(savedLease);
const runtime = makeMockRuntime();
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun({
...input, admittedLifecycleMode: "warm", adapterConfig: { lifecycleMode: "warm", idleTimeoutMs: 1_800_000 },
});
expect(runtime.resumeRunLease).toHaveBeenCalledWith({
environment: result.environment, lease: savedLease, requestedCwd: "/remote/workspace",
warmRunnerIdleTimeoutMs: 1_800_000,
});
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
});
it("does not return the retained sandbox to preparation while its lifecycle resume is pending", async () => {
let ready = false;
let release!: () => void;
@@ -391,6 +391,29 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
expect(acquired.lease.metadata?.snapshot).toBe(config.snapshot);
});
it("forwards the run's warm idle window to the provider worker without changing lease identity", async () => {
const pluginId = randomUUID();
const { companyId, environment: baseEnvironment, runId } = await seedEnvironment();
const config = { provider: "daytona", image: "node:20", timeoutMs: 1234 };
await insertSandboxProviderPlugin(pluginId, { environmentDrivers: [{ driverKey: "daytona", kind: "sandbox_provider", displayName: "Daytona", configSchema: { type: "object" } }] });
await environmentService(db).update(baseEnvironment.id, { driver: "sandbox", config });
const environment: Environment = { ...baseEnvironment, driver: "sandbox", config };
const worker = createFreshPluginLeaseWorker(pluginId);
vi.mocked(worker.call).mockImplementation(async (_id, method) => {
if (method !== "environmentAcquireLease") throw new Error(`Unexpected ${method}`);
return { providerLeaseId: "warm-lease", metadata: { provider: "daytona", image: "node:20", remoteCwd: "/workspace" } };
});
const service = environmentRuntimeService(db, { pluginWorkerManager: worker });
const acquired = await service.acquireRunLease({ companyId,
environment: resolveRunnerEnvironmentForRun(environment, "paperclip_runner", { lifecycleMode: "warm", idleTimeoutMs: 900_000 }),
configuredEnvironment: environment, issueId: null, heartbeatRunId: runId, persistedExecutionWorkspace: null,
adapterType: "paperclip_runner", warmRunnerIdleTimeoutMs: 900_000 });
expect(worker.call).toHaveBeenCalledWith(pluginId, "environmentAcquireLease", expect.objectContaining({
config: expect.objectContaining({ reuseLease: true, warmRunnerIdleTimeoutMs: 900_000 }),
}), expect.any(Number));
expect(acquired.lease.metadata).not.toHaveProperty("warmRunnerIdleTimeoutMs");
});
it.each([["paperclip_runner", false, false], ["paperclip_runner", true, false], ["paperclip_runner", true, undefined], ["claude_local", false, false], [null, false, false]] as const)("pins qualified snapshots only for native execution (%s, warm=%s, savedReuse=%s)", async (adapterType, warm, savedReuse) => {
const pluginId = randomUUID();
const { companyId, environment: baseEnvironment, runId } = await seedEnvironment();
@@ -29,7 +29,7 @@ import type {
ExecutionWorkspaceConfig,
IssueExecutionWorkspaceSettings,
} from "@paperclipai/shared";
import { resolveRunnerEnvironmentForRun } from "./runner-environment-lifecycle.js";
import { resolveRunnerEnvironmentForRun, resolveWarmRunnerIdleTimeoutMsForRun } from "./runner-environment-lifecycle.js";
import { environmentService } from "./environments.js";
import {
environmentRuntimeService,
@@ -217,6 +217,7 @@ export function environmentRunOrchestrator(
executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
runnerTaskProjectIds?: readonly string[];
adapterType: string | null;
warmRunnerIdleTimeoutMs?: number;
}): Promise<EnvironmentRuntimeLeaseRecord> {
try {
return await environmentRuntime.acquireRunLease(input);
@@ -313,6 +314,9 @@ export function environmentRunOrchestrator(
}
// Step 2: Acquire lease
const warmRunnerIdleTimeoutMs = resolveWarmRunnerIdleTimeoutMsForRun(
selectedEnvironment, input.adapterType, input.adapterConfig, input.admittedLifecycleMode,
);
const acquisitionInput = {
companyId: input.companyId,
environment,
@@ -324,6 +328,7 @@ export function environmentRunOrchestrator(
executionWorkspaceSettings: input.executionWorkspaceSettings,
runnerTaskProjectIds: input.runnerTaskProjectIds,
adapterType: input.adapterType ?? null,
warmRunnerIdleTimeoutMs,
};
let leaseRecord: EnvironmentRuntimeLeaseRecord;
if (input.reattachRemoteLease) {
@@ -353,6 +358,7 @@ export function environmentRunOrchestrator(
// checks) can run. This API cannot acquire a replacement.
const resumed = await environmentRuntime.resumeRunLease({
environment, lease, requestedCwd: expected.remoteCwd,
...(warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs } : {}),
});
if (!resumed || resumed.providerLeaseId !== expected.providerLeaseId ||
resumed.metadata?.expired === true) {
+24 -4
View File
@@ -490,6 +490,13 @@ export interface EnvironmentDriverAcquireInput {
* behavior.
*/
requestedExpiresAt?: Date | null;
/**
* The run's warm runner idle window (ms) when the run keeps a warm runner on
* a reusable sandbox. The sandbox-provider driver forwards it to the provider
* worker only (never to lease identity or metadata), so the provider can keep
* its idle auto-stop strictly after the host's warm idle window.
*/
warmRunnerIdleTimeoutMs?: number;
/**
* Re-check the environment company binding inside the lease insert
* transaction. The login acquire paths set this so a managed reconciliation
@@ -545,6 +552,8 @@ export interface EnvironmentDriverLeaseInput {
export interface EnvironmentDriverResumeInput extends EnvironmentDriverLeaseInput {
requestedCwd?: string;
/** The run's warm runner idle window, forwarded to the provider worker only. */
warmRunnerIdleTimeoutMs?: number;
}
export interface EnvironmentDriverRealizeWorkspaceInput extends EnvironmentDriverLeaseInput {
@@ -1634,7 +1643,11 @@ function createSandboxEnvironmentDriver(
);
}
const workerConfig = stripSandboxProviderEnvelope(parsed.config);
const workerConfig = {
...stripSandboxProviderEnvelope(parsed.config),
// Provider-only view of the run's warm idle window (see the input doc).
...(input.warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs: input.warmRunnerIdleTimeoutMs } : {}),
};
const storedConfig = pinnedSnapshotId ? { ...storedParsed.config, snapshot: pinnedSnapshotId } : storedParsed.config;
if (pinnedSnapshotId) delete storedConfig.image;
const providerConfigForLease = sandboxConfigForLeaseMetadata(storedConfig);
@@ -2564,9 +2577,13 @@ function createSandboxEnvironmentDriver(
}
// This is a lifecycle resume, never acquisition: no replacement, host
// seed, lease mutation, or provider turn is allowed at this boundary.
const config = stripSandboxProviderEnvelope(await resolvePluginSandboxRuntimeConfig({
environment: input.environment, lease: input.lease, provider: providerKey,
}) as SandboxEnvironmentConfig);
const config = {
...stripSandboxProviderEnvelope(await resolvePluginSandboxRuntimeConfig({
environment: input.environment, lease: input.lease, provider: providerKey,
}) as SandboxEnvironmentConfig),
// Provider-only view of the run's warm idle window (see the input doc).
...(input.warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs: input.warmRunnerIdleTimeoutMs } : {}),
};
const resumed = await pluginWorkerManager.call(pluginId, "environmentResumeLease", {
driverKey: providerKey, companyId: input.lease.companyId, environmentId: input.environment.id,
issueId: input.lease.issueId, config, providerLeaseId: input.lease.providerLeaseId,
@@ -3800,6 +3817,8 @@ export function environmentRuntimeService(
* provider expiry only.
*/
requestedExpiresAt?: Date | null;
/** The run's warm runner idle window, forwarded to the provider worker only. */
warmRunnerIdleTimeoutMs?: number;
/**
* Re-check the environment company binding inside the lease insert
* transaction. The login acquire paths set this to reject a foreign-company
@@ -3831,6 +3850,7 @@ export function environmentRuntimeService(
applyCustomImageTemplate: input.applyCustomImageTemplate ?? false,
requestedExpiresAt: input.requestedExpiresAt ?? null,
assertCompanyBinding: input.assertCompanyBinding,
...(input.warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs: input.warmRunnerIdleTimeoutMs } : {}),
});
return {
@@ -1,5 +1,5 @@
import {expect,it} from "vitest";
import {resolveRunnerEnvironmentForRun} from "./runner-environment-lifecycle.js";
import {resolveRunnerEnvironmentForRun,resolveWarmRunnerIdleTimeoutMsForRun} from "./runner-environment-lifecycle.js";
it.each([undefined,"codex","acpx","opencode"])("defaults plugin Runner execution to warm without provider gates: %s",provider=>{
const environment={driver:"plugin" as const,config:{pluginKey:"fixture",driverKey:"tasks",driverConfig:{}}};
@@ -18,3 +18,15 @@ it("preserves explicit per-turn settings and the admitted recovery policy",()=>{
expect(resolveRunnerEnvironmentForRun({...environment,config:{...environment.config,runnerLifecycleMode:"per_turn"}},"paperclip_runner",{lifecycleMode:"warm"},undefined,"account:generation")).toMatchObject({config:{runnerLifecycleMode:"per_turn"}});
expect(resolveRunnerEnvironmentForRun(environment,"claude_local",{})).toBe(environment);
});
it("forwards the run's warm idle window for a warm sandbox-provider run only",()=>{
const sandbox={driver:"sandbox" as const,config:{provider:"daytona",reuseLease:true}};
// Agent-level warm with an inherited environment lifecycle: the provider must see the agent's window.
expect(resolveWarmRunnerIdleTimeoutMsForRun(sandbox,"paperclip_runner",{lifecycleMode:"warm",idleTimeoutMs:900_000})).toBe(900_000);
// The environment's own window wins over the agent's.
expect(resolveWarmRunnerIdleTimeoutMsForRun({...sandbox,config:{...sandbox.config,runnerLifecycleMode:"warm",runnerIdleTimeoutMs:600_000}},"paperclip_runner",{idleTimeoutMs:900_000})).toBe(600_000);
expect(resolveWarmRunnerIdleTimeoutMsForRun(sandbox,"paperclip_runner",{},"warm")).toBe(300_000);
expect(resolveWarmRunnerIdleTimeoutMsForRun(sandbox,"paperclip_runner",{lifecycleMode:"per_turn",idleTimeoutMs:900_000})).toBeUndefined();
expect(resolveWarmRunnerIdleTimeoutMsForRun({...sandbox,config:{...sandbox.config,runnerLifecycleMode:"per_turn"}},"paperclip_runner",{lifecycleMode:"warm"})).toBeUndefined();
expect(resolveWarmRunnerIdleTimeoutMsForRun(sandbox,"claude_local",{lifecycleMode:"warm"})).toBeUndefined();
expect(resolveWarmRunnerIdleTimeoutMsForRun({driver:"plugin" as const,config:{}},"paperclip_runner",{lifecycleMode:"warm"})).toBeUndefined();
});
@@ -1,6 +1,16 @@
import { resolvePaperclipRunnerIdleTimeoutMs } from "@paperclipai/adapter-utils";
import type { Environment } from "@paperclipai/shared";
function resolveRunnerLifecycleModeForRun(
config: Record<string, unknown>,
adapterConfig: Record<string, unknown>,
admittedLifecycleMode?: "warm" | "per_turn",
): unknown {
return admittedLifecycleMode ?? (config.runnerLifecycleMode === "warm" || config.runnerLifecycleMode === "per_turn"
? config.runnerLifecycleMode
: adapterConfig.lifecycleMode);
}
/** Resolve runner retention before acquiring a sandbox, not after it is ephemeral.
* This is a run-scoped view; never persist it over a shared environment's config.
* The driver still verifies provider support and chooses the actual lease policy.
@@ -14,9 +24,7 @@ export function resolveRunnerEnvironmentForRun<T extends Pick<Environment, "driv
): T {
if (adapterType !== "paperclip_runner" || !["sandbox", "plugin"].includes(environment.driver)) return environment;
const config = environment.config ?? {};
const lifecycleMode = admittedLifecycleMode ?? (config.runnerLifecycleMode === "warm" || config.runnerLifecycleMode === "per_turn"
? config.runnerLifecycleMode
: adapterConfig.lifecycleMode);
const lifecycleMode = resolveRunnerLifecycleModeForRun(config, adapterConfig, admittedLifecycleMode);
// Warm tasks need a managed credential identity; recovery keeps its admitted policy.
if (environment.driver === "plugin") return { ...environment, config: { ...config,
runnerLifecycleMode: admittedLifecycleMode ?? (managedAiCredentialIdentity && lifecycleMode !== "per_turn" ? "warm" : "per_turn"),
@@ -31,3 +39,22 @@ export function resolveRunnerEnvironmentForRun<T extends Pick<Environment, "driv
},
};
}
/**
* The warm runner idle window (ms) a sandbox-provider lease must outlive for
* this run, or undefined when the run keeps no warm runner on the sandbox. The
* host forwards it to the provider so a provider idle auto-stop (for example
* Daytona's `autoStopInterval`) is derived from the same value the host uses
* for its warm idle timer, instead of an independent default that can match it.
*/
export function resolveWarmRunnerIdleTimeoutMsForRun(
environment: Pick<Environment, "driver" | "config">,
adapterType: string,
adapterConfig: Record<string, unknown> = {},
admittedLifecycleMode?: "warm" | "per_turn",
): number | undefined {
if (adapterType !== "paperclip_runner" || environment.driver !== "sandbox") return undefined;
const config = environment.config ?? {};
if (resolveRunnerLifecycleModeForRun(config, adapterConfig, admittedLifecycleMode) !== "warm") return undefined;
return resolvePaperclipRunnerIdleTimeoutMs(config.runnerIdleTimeoutMs ?? adapterConfig.idleTimeoutMs);
}