mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
fix(daytona): keep the provider auto-stop strictly after the warm runner idle window (#19199)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm remote agent keeps its sandbox running after a turn. The host keeps a warm runner for an idle window (`runnerIdleTimeoutMs`, or the agent's `idleTimeoutMs`) > - The Daytona provider also stops an idle sandbox after `autoStopInterval` minutes. The default is 15 minutes > - On staging, the warm idle window was 900,000 ms (15 minutes). The two windows were equal > - A follow-up turn at the end of the warm window resumed a sandbox that Daytona was stopping. Daytona refused the resume in about 4 seconds, and the run failed at setup > - #19195 added a bounded retry for that refusal. It did not remove the cause > - This pull request makes the two windows non-overlapping by construction. Daytona derives its auto-stop for a warm reusable sandbox from the same warm idle value that the host uses > - The benefit is that a follow-up inside the warm window always finds a running sandbox, and idle sandboxes still stop ## Linked Issues or Issue Description Refs #19195 (bounded retry for a refused resume; this change removes the common cause). **What happened?** On a staging soak, three follow-up runs failed at setup with a refused sandbox resume. Each resume began 15 minutes 8 to 25 seconds after the previous turn released the sandbox. The warm idle timeout was 900,000 ms. Daytona's default `autoStopInterval` is 15 minutes. Nothing in the code kept these two values apart. **Expected behavior** A reusable sandbox that keeps a warm runner must not auto-stop inside the warm idle window or at its edge. The provider backstop must fire after the host's own idle handling, with a margin. Idle sandboxes must still stop. **Steps to reproduce** 1. Configure a Daytona sandbox environment with `reuseLease` and a warm runner with `runnerIdleTimeoutMs: 900000` (or an agent with `lifecycleMode: warm`, `idleTimeoutMs: 900000`). 2. Run a turn. Do not set `autoStopInterval`. 3. Before this change: the sandbox is created with `autoStopInterval: 15`, equal to the warm window. A follow-up near minute 15 races the provider stop. **Paperclip version or commit** `c656b95b6b` (current `master`). ## What Changed - Daytona plugin (`plugin.ts`): add `effectiveAutoStopIntervalMinutes`. For a reusable lease that keeps a warm runner, the effective auto-stop is at least `ceil(warm idle / 1 min) + 5` minutes. The plugin applies the floor only when the host forwards `warmRunnerIdleTimeoutMs`, or when the environment itself selects `runnerLifecycleMode: warm`. An explicit `0` (disabled), a longer configured value, `inherit` with a per-turn agent, per-turn, and non-reusable sandboxes keep the configured value. - Daytona plugin: use the effective value at sandbox creation. On resume, raise a shorter auto-stop on an existing sandbox. This update is best effort and bounded to 5 seconds; the #19195 retry still covers a refusal. The stale-handle refresh threshold keeps the configured interval, so a cached handle is not trusted for longer before the provider confirms the new interval. The warm window is not written to the saved normalized config. - Host (`runner-environment-lifecycle.ts`): add `resolveWarmRunnerIdleTimeoutMsForRun`. It returns the run's effective warm idle window (environment `runnerIdleTimeoutMs`, else the agent `idleTimeoutMs`) for a warm `paperclip_runner` sandbox run. It shares the lifecycle-mode resolution with `resolveRunnerEnvironmentForRun`. - Host (`environment-run-orchestrator.ts`, `environment-runtime.ts`): pass that value as `warmRunnerIdleTimeoutMs` to the sandbox-provider driver on acquisition and on remote recovery resume. The driver adds it to the provider worker config only. Lease identity, reuse matching, and lease metadata are unchanged. - Daytona manifest: describe the warm floor on `autoStopInterval`. - `doc/execution-semantics.md`: document the contract. ## Verification - Red (before the fix, same tests, with the floor disabled): 5 failures in `plugin.test.ts` "warm idle window versus Daytona auto-stop". The staging case failed with `expected 900000 to be greater than or equal to 1200000`. The resume case failed because `setAutostopInterval` was not called. - Red (host): the new `environment-runtime.test.ts` case first failed because the provider worker config had no `runnerIdleTimeoutMs`. The service wrapper dropped the field. The fix forwards it. - Green: `npx vitest run --config vitest.config.ts` in `packages/plugins/sandbox-providers/daytona`: 563 passed. This includes a setter that never settles (resume completes after the 5-second bound) and `inherit` with no forwarded window (explicit `autoStopInterval: 2` stays 2). - Green: `npx vitest run src/services/runner-environment-lifecycle.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/environment-run-orchestrator.test.ts` in `server/`: 242 passed. This includes recovery resume forwarding the admitted warm window. - The regression tests assert the relationship `autoStop >= warm idle + margin` for several warm windows (125 s, 300 s default, 900 s, 1,800 s) and for an explicit `autoStopInterval: 15`. They are not constant checks. - `node scripts/typecheck-low-memory.mjs` passes. `pnpm typecheck` in the Daytona package passes. ## Risks - A warm reusable Daytona sandbox now stays running up to the warm window plus 5 minutes after its last activity, instead of 15 minutes. With a 15-minute window this is 20 minutes. Cost is bounded and idle sandboxes still stop. - The resume-time `setAutostopInterval` call is best effort. If it fails, the old interval stays and the #19195 retry still applies. - Other providers are unchanged. They receive one extra host-owned config key (`warmRunnerIdleTimeoutMs`) only for warm `paperclip_runner` runs, and they ignore unknown keys. - No lease identity change, so existing retained sandboxes are still reused. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c656b95b6b
commit
037c58bb22
10 files changed
+288
-12
No files matched your search
@@ -1022,6 +1022,18 @@ replacement. The run records the refusal by closed labels only (cause label, HTT
|
||||
status, errno-style code), never by provider text. On exhaustion, the task gets a
|
||||
specific recovery notice, and generic recovery does not grant another budget.
|
||||
|
||||
The provider's own idle auto-stop must not coincide with the warm runner idle
|
||||
window. The host forwards the run's effective warm idle window (environment
|
||||
`runnerIdleTimeoutMs`, else the agent's `idleTimeoutMs`) to the sandbox provider
|
||||
worker as `warmRunnerIdleTimeoutMs`, on acquisition and on recovery resume. It is
|
||||
not part of lease identity or metadata. Daytona derives its effective
|
||||
`autoStopInterval` for a warm reusable sandbox as at least the warm window plus
|
||||
5 minutes, applies it at creation, and raises an older sandbox's shorter interval
|
||||
on resume (a bounded, best-effort update). Its cached-handle freshness keeps using
|
||||
the configured interval until the provider confirms the new one. An explicit `0`
|
||||
(auto-stop disabled), a longer configured interval, per-turn, and non-reusable
|
||||
sandboxes are unchanged, and idle sandboxes still auto-stop.
|
||||
|
||||
When an admitted continuation reuses a sandbox after verified cleanup, an
|
||||
unpublished preparation uses a directory bound to its writer epoch and lease.
|
||||
Earlier unpublished directories and their receipts stay intact. Repeating the
|
||||
|
||||
@@ -160,7 +160,7 @@ const manifest: PaperclipPluginManifestV1 = {
|
||||
autoStopInterval: {
|
||||
type: "number",
|
||||
description:
|
||||
"Daytona auto-stop interval in minutes. `0` disables auto-stop. Defaults to 15 when unset.",
|
||||
"Daytona auto-stop interval in minutes. `0` disables auto-stop. Defaults to 15 when unset. A reusable sandbox that keeps a warm runner never auto-stops before the warm idle window plus 5 minutes.",
|
||||
default: 15,
|
||||
},
|
||||
autoArchiveInterval: {
|
||||
|
||||
@@ -35,6 +35,7 @@ import plugin, {
|
||||
__resetDaytonaSandboxHandleCacheForTest,
|
||||
__getDaytonaWritableDirsForTest,
|
||||
__setDaytonaPluginContextForTest,
|
||||
WARM_IDLE_AUTO_STOP_MARGIN_MINUTES,
|
||||
} from "./plugin.js";
|
||||
import type { PluginContext } from "@paperclipai/plugin-sdk";
|
||||
import { environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic, PluginEnvironmentStopError, environmentStopErrorData, readEnvironmentStopFailure } from "@paperclipai/plugin-sdk";
|
||||
@@ -353,6 +354,8 @@ function createMockSandbox(overrides: {
|
||||
autoDestroyAt: overrides.autoDestroyAt ?? undefined,
|
||||
updatedAt: overrides.updatedAt,
|
||||
autoDeleteInterval: -1,
|
||||
autoStopInterval: 15,
|
||||
setAutostopInterval: vi.fn().mockResolvedValue(undefined),
|
||||
getWorkDir: vi.fn().mockResolvedValue(overrides.workDir ?? "/home/daytona"),
|
||||
getUserHomeDir: vi.fn().mockResolvedValue("/home/daytona"),
|
||||
start: vi.fn().mockResolvedValue(undefined),
|
||||
@@ -953,6 +956,101 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("warm idle window versus Daytona auto-stop", () => {
|
||||
// The host keeps a warm runner on a reusable sandbox for runnerIdleTimeoutMs.
|
||||
// Daytona must never auto-stop that sandbox inside (or at the edge of) the
|
||||
// warm window, or a follow-up at the window's end races the provider stop.
|
||||
const acquire = async (config: Record<string, unknown>) => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
mockCreate.mockResolvedValue(createMockSandbox());
|
||||
await plugin.definition.onEnvironmentAcquireLease?.({
|
||||
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", runId: "run-1",
|
||||
config: { image: "node:20", timeoutMs: 300000, ...config },
|
||||
});
|
||||
const [createParams] = mockCreate.mock.calls.at(-1) as [Record<string, unknown>];
|
||||
return createParams.autoStopInterval as number;
|
||||
};
|
||||
const resume = async (config: Record<string, unknown>) => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox({ id: "sandbox-warm", state: "started" });
|
||||
sandbox.process.executeCommand.mockImplementation(async (command: string) => ({ exitCode: 0,
|
||||
result: command.startsWith("cat ") ? JSON.stringify({ token: "sentinel-token" }) : "bash" }));
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
await plugin.definition.onEnvironmentResumeLease!({
|
||||
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", providerLeaseId: sandbox.id,
|
||||
config,
|
||||
leaseMetadata: { workspaceSentinel: {
|
||||
path: "/home/daytona/paperclip-workspace/.paperclip-runtime/reusable-sandbox-lease.json",
|
||||
token: "sentinel-token", result: "written" } },
|
||||
});
|
||||
return sandbox;
|
||||
};
|
||||
const marginMs = WARM_IDLE_AUTO_STOP_MARGIN_MINUTES * 60_000;
|
||||
|
||||
it.each([
|
||||
[{ runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 900_000 }, 900_000],
|
||||
[{ runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 900_000, autoStopInterval: 15 }, 900_000],
|
||||
// Agent-level warm on an inherit environment: the host forwards the window.
|
||||
[{ runnerLifecycleMode: "inherit", warmRunnerIdleTimeoutMs: 1_800_000 }, 1_800_000],
|
||||
[{ warmRunnerIdleTimeoutMs: 900_000, autoStopInterval: 15 }, 900_000],
|
||||
[{ runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 125_000, autoStopInterval: 2 }, 125_000],
|
||||
[{ runnerLifecycleMode: "warm" }, 300_000],
|
||||
])("auto-stops a warm reusable sandbox only after the warm idle window plus a margin (%o)", async (config, warmIdleMs) => {
|
||||
expect(marginMs).toBeGreaterThan(0);
|
||||
const autoStopMs = (await acquire({ reuseLease: true, ...config })) * 60_000;
|
||||
expect(autoStopMs).toBeGreaterThanOrEqual(warmIdleMs + marginMs);
|
||||
});
|
||||
|
||||
it("keeps a longer configured auto-stop and an explicit 0 (disabled)", async () => {
|
||||
expect(await acquire({ reuseLease: true, runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 300_000, autoStopInterval: 60 })).toBe(60);
|
||||
expect(await acquire({ reuseLease: true, runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 900_000, autoStopInterval: 0 })).toBe(0);
|
||||
});
|
||||
|
||||
it("leaves per-turn and non-reusable sandboxes on the configured auto-stop", async () => {
|
||||
expect(await acquire({ reuseLease: false, runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 900_000 })).toBe(15);
|
||||
expect(await acquire({ reuseLease: true, runnerLifecycleMode: "per_turn", runnerIdleTimeoutMs: 900_000 })).toBe(15);
|
||||
// Inherit with a per-turn agent: the host forwards no warm window.
|
||||
expect(await acquire({ reuseLease: true, runnerLifecycleMode: "inherit", autoStopInterval: 2 })).toBe(2);
|
||||
expect(await acquire({ reuseLease: true, autoStopInterval: 2 })).toBe(2);
|
||||
});
|
||||
|
||||
it("raises the auto-stop of an existing warm sandbox on resume", async () => {
|
||||
const sandbox = await resume({ reuseLease: true, runnerLifecycleMode: "warm", runnerIdleTimeoutMs: 900_000 });
|
||||
expect(sandbox.setAutostopInterval).toHaveBeenCalledTimes(1);
|
||||
const [minutes] = sandbox.setAutostopInterval.mock.calls[0] as [number];
|
||||
expect(minutes * 60_000).toBeGreaterThanOrEqual(900_000 + marginMs);
|
||||
});
|
||||
|
||||
it("bounds a resume-time auto-stop update that never settles", async () => {
|
||||
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
|
||||
try {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox({ id: "sandbox-slow", state: "started" });
|
||||
sandbox.setAutostopInterval.mockReturnValue(new Promise<void>(() => {}));
|
||||
sandbox.process.executeCommand.mockImplementation(async (command: string) => ({ exitCode: 0,
|
||||
result: command.startsWith("cat ") ? JSON.stringify({ token: "sentinel-token" }) : "bash" }));
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
const pending = plugin.definition.onEnvironmentResumeLease!({
|
||||
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", providerLeaseId: sandbox.id,
|
||||
config: { reuseLease: true, warmRunnerIdleTimeoutMs: 900_000 },
|
||||
leaseMetadata: { workspaceSentinel: {
|
||||
path: "/home/daytona/paperclip-workspace/.paperclip-runtime/reusable-sandbox-lease.json",
|
||||
token: "sentinel-token", result: "written" } },
|
||||
});
|
||||
await vi.advanceTimersByTimeAsync(6_000);
|
||||
await expect(pending).resolves.toMatchObject({ providerLeaseId: sandbox.id });
|
||||
expect(sandbox.setAutostopInterval).toHaveBeenCalledTimes(1);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it("does not change auto-stop on resume of a per-turn reusable sandbox", async () => {
|
||||
const sandbox = await resume({ reuseLease: true, runnerLifecycleMode: "per_turn" });
|
||||
expect(sandbox.setAutostopInterval).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe("fresh acquisition deadline", () => {
|
||||
const params = {
|
||||
driverKey: "daytona", companyId: "company-1", environmentId: "env-1", runId: "run-1",
|
||||
|
||||
@@ -194,6 +194,12 @@ interface DaytonaDriverConfig {
|
||||
ttlMinutes: number | null;
|
||||
reuseLease: boolean;
|
||||
archiveOnRelease: boolean;
|
||||
/**
|
||||
* Host-owned warm runner idle window (ms) for a reusable warm lease, or null
|
||||
* when the host keeps no warm runner on this sandbox. Never persisted in the
|
||||
* normalized provider config; it only raises the effective auto-stop.
|
||||
*/
|
||||
warmIdleTimeoutMs: number | null;
|
||||
}
|
||||
|
||||
type WorkspaceSentinelResult = {
|
||||
@@ -238,6 +244,46 @@ const WORKSPACE_SENTINEL_RELATIVE_PATH = ".paperclip-runtime/reusable-sandbox-le
|
||||
// - autoArchive: archive *stopped* sandboxes so they leave the disk quota.
|
||||
// - autoDelete: backstop reaper for sandboxes nobody resumes.
|
||||
const DEFAULT_AUTO_STOP_INTERVAL_MINUTES = 15;
|
||||
|
||||
// The host keeps a warm paperclip_runner on a reusable sandbox for
|
||||
// `runnerIdleTimeoutMs` after a turn (host default 300000 ms). If Daytona's
|
||||
// idle auto-stop fires at the same moment, a follow-up that arrives at the end
|
||||
// of the warm window resumes a sandbox the provider is stopping and the resume
|
||||
// is refused. Derive the provider auto-stop from the warm window so the
|
||||
// provider backstop always fires strictly after the host's own idle handling:
|
||||
// effective auto-stop >= ceil(warm idle) + this margin. Idle sandboxes still
|
||||
// stop; only the backstop moves later by the margin.
|
||||
export const DEFAULT_WARM_RUNNER_IDLE_TIMEOUT_MS = 300_000;
|
||||
export const WARM_IDLE_AUTO_STOP_MARGIN_MINUTES = 5;
|
||||
|
||||
/**
|
||||
* Effective Daytona auto-stop (minutes) for a sandbox. An explicit 0 (disabled)
|
||||
* is preserved. A warm reusable lease never auto-stops before the host's warm
|
||||
* idle window plus {@link WARM_IDLE_AUTO_STOP_MARGIN_MINUTES}.
|
||||
*/
|
||||
export function effectiveAutoStopIntervalMinutes(
|
||||
config: Pick<DaytonaDriverConfig, "autoStopInterval" | "warmIdleTimeoutMs">,
|
||||
): number | null {
|
||||
const configured = config.autoStopInterval;
|
||||
if (configured === 0 || config.warmIdleTimeoutMs == null) return configured;
|
||||
const floor = Math.ceil(config.warmIdleTimeoutMs / 60_000) + WARM_IDLE_AUTO_STOP_MARGIN_MINUTES;
|
||||
return Math.max(configured ?? DEFAULT_AUTO_STOP_INTERVAL_MINUTES, floor);
|
||||
}
|
||||
|
||||
function parseWarmIdleTimeoutMs(raw: Record<string, unknown>): number | null {
|
||||
// Only a reusable lease can host a warm runner.
|
||||
if (raw.reuseLease !== true) return null;
|
||||
// The host forwards the run's resolved warm window (environment or agent
|
||||
// setting) only when the run actually keeps a warm runner.
|
||||
const forwarded = parseOptionalInteger(raw.warmRunnerIdleTimeoutMs);
|
||||
if (forwarded != null && forwarded > 0) return forwarded;
|
||||
// An environment that itself selects a warm runner (no host forwarding, for
|
||||
// example a legacy warm lifecycle) uses its own window. Inherit and per_turn
|
||||
// apply no floor: the host forwards the window when the run is warm.
|
||||
if (raw.runnerLifecycleMode !== "warm") return null;
|
||||
const configured = parseOptionalInteger(raw.runnerIdleTimeoutMs);
|
||||
return configured != null && configured > 0 ? configured : DEFAULT_WARM_RUNNER_IDLE_TIMEOUT_MS;
|
||||
}
|
||||
const DEFAULT_AUTO_ARCHIVE_INTERVAL_MINUTES = 60;
|
||||
const DEFAULT_AUTO_DELETE_INTERVAL_MINUTES = 7 * 24 * 60; // 7 days
|
||||
|
||||
@@ -261,6 +307,10 @@ const GIT_NETWORK_TIMEOUT_MS = 120_000;
|
||||
// less disables the extra bound.
|
||||
const DEFAULT_LIVENESS_TIMEOUT_MS = 30_000;
|
||||
|
||||
// Bound on the optional resume-time auto-stop update, so a slow provider
|
||||
// write cannot consume the host's resume budget.
|
||||
const AUTO_STOP_UPDATE_TIMEOUT_MS = 5_000;
|
||||
|
||||
// Extra margin added to the SDK start/recover timeout when the plugin wraps
|
||||
// those lifecycle calls in its own per-call bound. The SDK call already carries
|
||||
// a `timeoutSeconds` deadline; the wrapper is a backstop for a connection-level
|
||||
@@ -319,6 +369,7 @@ function parseDriverConfig(raw: Record<string, unknown>): DaytonaDriverConfig {
|
||||
ttlMinutes: raw.ttlMinutes == null || raw.ttlMinutes === "" ? null : Number(raw.ttlMinutes),
|
||||
reuseLease: raw.reuseLease === true,
|
||||
archiveOnRelease: raw.archiveOnRelease === true,
|
||||
warmIdleTimeoutMs: parseWarmIdleTimeoutMs(raw),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -361,7 +412,7 @@ function buildCreateParams(
|
||||
const base: CreateSandboxBaseParams = {
|
||||
labels,
|
||||
language: config.language ?? undefined,
|
||||
autoStopInterval: config.autoStopInterval ?? undefined,
|
||||
autoStopInterval: effectiveAutoStopIntervalMinutes(config) ?? undefined,
|
||||
autoArchiveInterval: config.autoArchiveInterval ?? undefined,
|
||||
autoDeleteInterval: config.autoDeleteInterval ?? undefined,
|
||||
...(config.ttlMinutes != null ? { ttlMinutes: config.ttlMinutes } : {}),
|
||||
@@ -2667,9 +2718,11 @@ const plugin = definePlugin(trackDaytonaHooks({
|
||||
return { ok: false, errors };
|
||||
}
|
||||
|
||||
// The warm idle window is host-owned; keep it out of the saved provider config.
|
||||
const { warmIdleTimeoutMs: _warmIdleTimeoutMs, ...normalizedConfig } = config;
|
||||
return {
|
||||
ok: true,
|
||||
normalizedConfig: { ...config },
|
||||
normalizedConfig,
|
||||
};
|
||||
},
|
||||
|
||||
@@ -2949,6 +3002,18 @@ const plugin = definePlugin(trackDaytonaHooks({
|
||||
}
|
||||
const resumedFromState = sandbox.state ?? null;
|
||||
await ensureSandboxStarted(sandbox, toTimeoutSeconds(config.timeoutMs), scope);
|
||||
// A sandbox created before the warm floor (or under a shorter warm
|
||||
// window) keeps its old auto-stop. Raise it on resume so the provider
|
||||
// backstop cannot race the host's warm idle window. Best effort: the
|
||||
// host's bounded resume retry still covers a refused resume.
|
||||
const autoStopInterval = effectiveAutoStopIntervalMinutes(config);
|
||||
const currentAutoStop = sandbox.autoStopInterval;
|
||||
if (autoStopInterval != null && autoStopInterval > 0 && config.warmIdleTimeoutMs != null
|
||||
&& typeof currentAutoStop === "number" && currentAutoStop > 0 && currentAutoStop < autoStopInterval) {
|
||||
await withLivenessTimeout("sandbox.setAutostopInterval",
|
||||
Math.max(1, Math.min(config.livenessTimeoutMs > 0 ? config.livenessTimeoutMs : AUTO_STOP_UPDATE_TIMEOUT_MS, AUTO_STOP_UPDATE_TIMEOUT_MS)),
|
||||
() => Promise.resolve().then(() => sandbox.setAutostopInterval(autoStopInterval))).catch(() => undefined);
|
||||
}
|
||||
try {
|
||||
return await traceLeaseStep("lease.verify", async () => {
|
||||
const remoteCwd = await traceLeaseStep("lease.workspace", () => resolveSandboxWorkingDirectory(sandbox, false));
|
||||
|
||||
@@ -924,6 +924,19 @@ describe("live remote runner lease reattachment", () => {
|
||||
expect(result.leaseContext.executionWorkspaceId).toBe("ew-1");
|
||||
expect(result.environment.config.reuseLease).toBe(false);
|
||||
});
|
||||
it("forwards the admitted warm idle window to the recovery resume", async () => {
|
||||
const savedLease = originalLease();
|
||||
mockGetLease.mockResolvedValue(savedLease);
|
||||
const runtime = makeMockRuntime();
|
||||
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun({
|
||||
...input, admittedLifecycleMode: "warm", adapterConfig: { lifecycleMode: "warm", idleTimeoutMs: 1_800_000 },
|
||||
});
|
||||
expect(runtime.resumeRunLease).toHaveBeenCalledWith({
|
||||
environment: result.environment, lease: savedLease, requestedCwd: "/remote/workspace",
|
||||
warmRunnerIdleTimeoutMs: 1_800_000,
|
||||
});
|
||||
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
|
||||
});
|
||||
it("does not return the retained sandbox to preparation while its lifecycle resume is pending", async () => {
|
||||
let ready = false;
|
||||
let release!: () => void;
|
||||
|
||||
@@ -391,6 +391,29 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
expect(acquired.lease.metadata?.snapshot).toBe(config.snapshot);
|
||||
});
|
||||
|
||||
it("forwards the run's warm idle window to the provider worker without changing lease identity", async () => {
|
||||
const pluginId = randomUUID();
|
||||
const { companyId, environment: baseEnvironment, runId } = await seedEnvironment();
|
||||
const config = { provider: "daytona", image: "node:20", timeoutMs: 1234 };
|
||||
await insertSandboxProviderPlugin(pluginId, { environmentDrivers: [{ driverKey: "daytona", kind: "sandbox_provider", displayName: "Daytona", configSchema: { type: "object" } }] });
|
||||
await environmentService(db).update(baseEnvironment.id, { driver: "sandbox", config });
|
||||
const environment: Environment = { ...baseEnvironment, driver: "sandbox", config };
|
||||
const worker = createFreshPluginLeaseWorker(pluginId);
|
||||
vi.mocked(worker.call).mockImplementation(async (_id, method) => {
|
||||
if (method !== "environmentAcquireLease") throw new Error(`Unexpected ${method}`);
|
||||
return { providerLeaseId: "warm-lease", metadata: { provider: "daytona", image: "node:20", remoteCwd: "/workspace" } };
|
||||
});
|
||||
const service = environmentRuntimeService(db, { pluginWorkerManager: worker });
|
||||
const acquired = await service.acquireRunLease({ companyId,
|
||||
environment: resolveRunnerEnvironmentForRun(environment, "paperclip_runner", { lifecycleMode: "warm", idleTimeoutMs: 900_000 }),
|
||||
configuredEnvironment: environment, issueId: null, heartbeatRunId: runId, persistedExecutionWorkspace: null,
|
||||
adapterType: "paperclip_runner", warmRunnerIdleTimeoutMs: 900_000 });
|
||||
expect(worker.call).toHaveBeenCalledWith(pluginId, "environmentAcquireLease", expect.objectContaining({
|
||||
config: expect.objectContaining({ reuseLease: true, warmRunnerIdleTimeoutMs: 900_000 }),
|
||||
}), expect.any(Number));
|
||||
expect(acquired.lease.metadata).not.toHaveProperty("warmRunnerIdleTimeoutMs");
|
||||
});
|
||||
|
||||
it.each([["paperclip_runner", false, false], ["paperclip_runner", true, false], ["paperclip_runner", true, undefined], ["claude_local", false, false], [null, false, false]] as const)("pins qualified snapshots only for native execution (%s, warm=%s, savedReuse=%s)", async (adapterType, warm, savedReuse) => {
|
||||
const pluginId = randomUUID();
|
||||
const { companyId, environment: baseEnvironment, runId } = await seedEnvironment();
|
||||
|
||||
@@ -29,7 +29,7 @@ import type {
|
||||
ExecutionWorkspaceConfig,
|
||||
IssueExecutionWorkspaceSettings,
|
||||
} from "@paperclipai/shared";
|
||||
import { resolveRunnerEnvironmentForRun } from "./runner-environment-lifecycle.js";
|
||||
import { resolveRunnerEnvironmentForRun, resolveWarmRunnerIdleTimeoutMsForRun } from "./runner-environment-lifecycle.js";
|
||||
import { environmentService } from "./environments.js";
|
||||
import {
|
||||
environmentRuntimeService,
|
||||
@@ -217,6 +217,7 @@ export function environmentRunOrchestrator(
|
||||
executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
|
||||
runnerTaskProjectIds?: readonly string[];
|
||||
adapterType: string | null;
|
||||
warmRunnerIdleTimeoutMs?: number;
|
||||
}): Promise<EnvironmentRuntimeLeaseRecord> {
|
||||
try {
|
||||
return await environmentRuntime.acquireRunLease(input);
|
||||
@@ -313,6 +314,9 @@ export function environmentRunOrchestrator(
|
||||
}
|
||||
|
||||
// Step 2: Acquire lease
|
||||
const warmRunnerIdleTimeoutMs = resolveWarmRunnerIdleTimeoutMsForRun(
|
||||
selectedEnvironment, input.adapterType, input.adapterConfig, input.admittedLifecycleMode,
|
||||
);
|
||||
const acquisitionInput = {
|
||||
companyId: input.companyId,
|
||||
environment,
|
||||
@@ -324,6 +328,7 @@ export function environmentRunOrchestrator(
|
||||
executionWorkspaceSettings: input.executionWorkspaceSettings,
|
||||
runnerTaskProjectIds: input.runnerTaskProjectIds,
|
||||
adapterType: input.adapterType ?? null,
|
||||
warmRunnerIdleTimeoutMs,
|
||||
};
|
||||
let leaseRecord: EnvironmentRuntimeLeaseRecord;
|
||||
if (input.reattachRemoteLease) {
|
||||
@@ -353,6 +358,7 @@ export function environmentRunOrchestrator(
|
||||
// checks) can run. This API cannot acquire a replacement.
|
||||
const resumed = await environmentRuntime.resumeRunLease({
|
||||
environment, lease, requestedCwd: expected.remoteCwd,
|
||||
...(warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs } : {}),
|
||||
});
|
||||
if (!resumed || resumed.providerLeaseId !== expected.providerLeaseId ||
|
||||
resumed.metadata?.expired === true) {
|
||||
|
||||
@@ -490,6 +490,13 @@ export interface EnvironmentDriverAcquireInput {
|
||||
* behavior.
|
||||
*/
|
||||
requestedExpiresAt?: Date | null;
|
||||
/**
|
||||
* The run's warm runner idle window (ms) when the run keeps a warm runner on
|
||||
* a reusable sandbox. The sandbox-provider driver forwards it to the provider
|
||||
* worker only (never to lease identity or metadata), so the provider can keep
|
||||
* its idle auto-stop strictly after the host's warm idle window.
|
||||
*/
|
||||
warmRunnerIdleTimeoutMs?: number;
|
||||
/**
|
||||
* Re-check the environment company binding inside the lease insert
|
||||
* transaction. The login acquire paths set this so a managed reconciliation
|
||||
@@ -545,6 +552,8 @@ export interface EnvironmentDriverLeaseInput {
|
||||
|
||||
export interface EnvironmentDriverResumeInput extends EnvironmentDriverLeaseInput {
|
||||
requestedCwd?: string;
|
||||
/** The run's warm runner idle window, forwarded to the provider worker only. */
|
||||
warmRunnerIdleTimeoutMs?: number;
|
||||
}
|
||||
|
||||
export interface EnvironmentDriverRealizeWorkspaceInput extends EnvironmentDriverLeaseInput {
|
||||
@@ -1634,7 +1643,11 @@ function createSandboxEnvironmentDriver(
|
||||
);
|
||||
}
|
||||
|
||||
const workerConfig = stripSandboxProviderEnvelope(parsed.config);
|
||||
const workerConfig = {
|
||||
...stripSandboxProviderEnvelope(parsed.config),
|
||||
// Provider-only view of the run's warm idle window (see the input doc).
|
||||
...(input.warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs: input.warmRunnerIdleTimeoutMs } : {}),
|
||||
};
|
||||
const storedConfig = pinnedSnapshotId ? { ...storedParsed.config, snapshot: pinnedSnapshotId } : storedParsed.config;
|
||||
if (pinnedSnapshotId) delete storedConfig.image;
|
||||
const providerConfigForLease = sandboxConfigForLeaseMetadata(storedConfig);
|
||||
@@ -2564,9 +2577,13 @@ function createSandboxEnvironmentDriver(
|
||||
}
|
||||
// This is a lifecycle resume, never acquisition: no replacement, host
|
||||
// seed, lease mutation, or provider turn is allowed at this boundary.
|
||||
const config = stripSandboxProviderEnvelope(await resolvePluginSandboxRuntimeConfig({
|
||||
environment: input.environment, lease: input.lease, provider: providerKey,
|
||||
}) as SandboxEnvironmentConfig);
|
||||
const config = {
|
||||
...stripSandboxProviderEnvelope(await resolvePluginSandboxRuntimeConfig({
|
||||
environment: input.environment, lease: input.lease, provider: providerKey,
|
||||
}) as SandboxEnvironmentConfig),
|
||||
// Provider-only view of the run's warm idle window (see the input doc).
|
||||
...(input.warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs: input.warmRunnerIdleTimeoutMs } : {}),
|
||||
};
|
||||
const resumed = await pluginWorkerManager.call(pluginId, "environmentResumeLease", {
|
||||
driverKey: providerKey, companyId: input.lease.companyId, environmentId: input.environment.id,
|
||||
issueId: input.lease.issueId, config, providerLeaseId: input.lease.providerLeaseId,
|
||||
@@ -3800,6 +3817,8 @@ export function environmentRuntimeService(
|
||||
* provider expiry only.
|
||||
*/
|
||||
requestedExpiresAt?: Date | null;
|
||||
/** The run's warm runner idle window, forwarded to the provider worker only. */
|
||||
warmRunnerIdleTimeoutMs?: number;
|
||||
/**
|
||||
* Re-check the environment company binding inside the lease insert
|
||||
* transaction. The login acquire paths set this to reject a foreign-company
|
||||
@@ -3831,6 +3850,7 @@ export function environmentRuntimeService(
|
||||
applyCustomImageTemplate: input.applyCustomImageTemplate ?? false,
|
||||
requestedExpiresAt: input.requestedExpiresAt ?? null,
|
||||
assertCompanyBinding: input.assertCompanyBinding,
|
||||
...(input.warmRunnerIdleTimeoutMs !== undefined ? { warmRunnerIdleTimeoutMs: input.warmRunnerIdleTimeoutMs } : {}),
|
||||
});
|
||||
|
||||
return {
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import {expect,it} from "vitest";
|
||||
import {resolveRunnerEnvironmentForRun} from "./runner-environment-lifecycle.js";
|
||||
import {resolveRunnerEnvironmentForRun,resolveWarmRunnerIdleTimeoutMsForRun} from "./runner-environment-lifecycle.js";
|
||||
|
||||
it.each([undefined,"codex","acpx","opencode"])("defaults plugin Runner execution to warm without provider gates: %s",provider=>{
|
||||
const environment={driver:"plugin" as const,config:{pluginKey:"fixture",driverKey:"tasks",driverConfig:{}}};
|
||||
@@ -18,3 +18,15 @@ it("preserves explicit per-turn settings and the admitted recovery policy",()=>{
|
||||
expect(resolveRunnerEnvironmentForRun({...environment,config:{...environment.config,runnerLifecycleMode:"per_turn"}},"paperclip_runner",{lifecycleMode:"warm"},undefined,"account:generation")).toMatchObject({config:{runnerLifecycleMode:"per_turn"}});
|
||||
expect(resolveRunnerEnvironmentForRun(environment,"claude_local",{})).toBe(environment);
|
||||
});
|
||||
it("forwards the run's warm idle window for a warm sandbox-provider run only",()=>{
|
||||
const sandbox={driver:"sandbox" as const,config:{provider:"daytona",reuseLease:true}};
|
||||
// Agent-level warm with an inherited environment lifecycle: the provider must see the agent's window.
|
||||
expect(resolveWarmRunnerIdleTimeoutMsForRun(sandbox,"paperclip_runner",{lifecycleMode:"warm",idleTimeoutMs:900_000})).toBe(900_000);
|
||||
// The environment's own window wins over the agent's.
|
||||
expect(resolveWarmRunnerIdleTimeoutMsForRun({...sandbox,config:{...sandbox.config,runnerLifecycleMode:"warm",runnerIdleTimeoutMs:600_000}},"paperclip_runner",{idleTimeoutMs:900_000})).toBe(600_000);
|
||||
expect(resolveWarmRunnerIdleTimeoutMsForRun(sandbox,"paperclip_runner",{},"warm")).toBe(300_000);
|
||||
expect(resolveWarmRunnerIdleTimeoutMsForRun(sandbox,"paperclip_runner",{lifecycleMode:"per_turn",idleTimeoutMs:900_000})).toBeUndefined();
|
||||
expect(resolveWarmRunnerIdleTimeoutMsForRun({...sandbox,config:{...sandbox.config,runnerLifecycleMode:"per_turn"}},"paperclip_runner",{lifecycleMode:"warm"})).toBeUndefined();
|
||||
expect(resolveWarmRunnerIdleTimeoutMsForRun(sandbox,"claude_local",{lifecycleMode:"warm"})).toBeUndefined();
|
||||
expect(resolveWarmRunnerIdleTimeoutMsForRun({driver:"plugin" as const,config:{}},"paperclip_runner",{lifecycleMode:"warm"})).toBeUndefined();
|
||||
});
|
||||
@@ -1,6 +1,16 @@
|
||||
import { resolvePaperclipRunnerIdleTimeoutMs } from "@paperclipai/adapter-utils";
|
||||
import type { Environment } from "@paperclipai/shared";
|
||||
|
||||
function resolveRunnerLifecycleModeForRun(
|
||||
config: Record<string, unknown>,
|
||||
adapterConfig: Record<string, unknown>,
|
||||
admittedLifecycleMode?: "warm" | "per_turn",
|
||||
): unknown {
|
||||
return admittedLifecycleMode ?? (config.runnerLifecycleMode === "warm" || config.runnerLifecycleMode === "per_turn"
|
||||
? config.runnerLifecycleMode
|
||||
: adapterConfig.lifecycleMode);
|
||||
}
|
||||
|
||||
/** Resolve runner retention before acquiring a sandbox, not after it is ephemeral.
|
||||
* This is a run-scoped view; never persist it over a shared environment's config.
|
||||
* The driver still verifies provider support and chooses the actual lease policy.
|
||||
@@ -14,9 +24,7 @@ export function resolveRunnerEnvironmentForRun<T extends Pick<Environment, "driv
|
||||
): T {
|
||||
if (adapterType !== "paperclip_runner" || !["sandbox", "plugin"].includes(environment.driver)) return environment;
|
||||
const config = environment.config ?? {};
|
||||
const lifecycleMode = admittedLifecycleMode ?? (config.runnerLifecycleMode === "warm" || config.runnerLifecycleMode === "per_turn"
|
||||
? config.runnerLifecycleMode
|
||||
: adapterConfig.lifecycleMode);
|
||||
const lifecycleMode = resolveRunnerLifecycleModeForRun(config, adapterConfig, admittedLifecycleMode);
|
||||
// Warm tasks need a managed credential identity; recovery keeps its admitted policy.
|
||||
if (environment.driver === "plugin") return { ...environment, config: { ...config,
|
||||
runnerLifecycleMode: admittedLifecycleMode ?? (managedAiCredentialIdentity && lifecycleMode !== "per_turn" ? "warm" : "per_turn"),
|
||||
@@ -31,3 +39,22 @@ export function resolveRunnerEnvironmentForRun<T extends Pick<Environment, "driv
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The warm runner idle window (ms) a sandbox-provider lease must outlive for
|
||||
* this run, or undefined when the run keeps no warm runner on the sandbox. The
|
||||
* host forwards it to the provider so a provider idle auto-stop (for example
|
||||
* Daytona's `autoStopInterval`) is derived from the same value the host uses
|
||||
* for its warm idle timer, instead of an independent default that can match it.
|
||||
*/
|
||||
export function resolveWarmRunnerIdleTimeoutMsForRun(
|
||||
environment: Pick<Environment, "driver" | "config">,
|
||||
adapterType: string,
|
||||
adapterConfig: Record<string, unknown> = {},
|
||||
admittedLifecycleMode?: "warm" | "per_turn",
|
||||
): number | undefined {
|
||||
if (adapterType !== "paperclip_runner" || environment.driver !== "sandbox") return undefined;
|
||||
const config = environment.config ?? {};
|
||||
if (resolveRunnerLifecycleModeForRun(config, adapterConfig, admittedLifecycleMode) !== "warm") return undefined;
|
||||
return resolvePaperclipRunnerIdleTimeoutMs(config.runnerIdleTimeoutMs ?? adapterConfig.idleTimeoutMs);
|
||||
}
|
||||
Reference in new issue
Block a user