mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs on a reusable cloud sandbox resumes the same sandbox on the next wake > - When the resume call times out, the run failed after 90 seconds with `reusable_sandbox_resume_failed (cause=transport_timeout)` > - The provider keeps starting the sandbox after the host call times out, so the sandbox is usually ready a little later > - The run gave up after one call, and the task then waited behind a Retry button > - This pull request keeps the same run retrying an idle sandbox whose resume only timed out, up to the provider's start deadline > - The benefit is that recovery works automatically and a sandbox with unsaved work is never destroyed ## Linked Issues or Issue Description Refs #20626, #20828, #21020 **What happened?** Three Codex runs on Daytona failed before provider dispatch with: `Reusable sandbox lease "<id>" could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` Each run had `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. Each run took 100 to 122 seconds: one resume call used the whole 90-second budget, so no in-run retry was possible. The preserved leases were idle. Their last runs had ended about two hours before, with a confirmed stop-and-retain. The workspaces were not safe to replace, so the run failed strictly. The automatic bootstrap retry was then refused ("The automatic retry could not revalidate the original user continuation."), and all three tasks ended `blocked` behind a Retry. **Expected behavior** A resume that only timed out, before any provider work, on a sandbox that no live run holds, continues in the same run. The run keeps the same sandbox and does not create a replacement or destroy the sandbox. **Steps to reproduce** 1. Let a task's reusable Daytona sandbox stop and stay idle for some hours (it auto-archives after 60 minutes). 2. Retry the task. The workspace has unsaved work, so replacement is not safe. 3. The `environmentResumeLease` call takes longer than 90 seconds. The host call times out (`RPC call "environmentResumeLease" timed out after 90000ms`), the 90-second budget is spent, and the run fails with `reusable_sandbox_resume_failed`. **Paperclip version or commit** Staging at `849f109f` (current `master` has the same code path). **Deployment mode** Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases on. **Agent adapter(s) involved** `codex_local` (Codex) on a Daytona sandbox environment. The change is adapter-independent. **Root cause** `server/src/services/environment-runtime.ts`, reusable resume loop: the first `environmentResumeLease` call got the whole 90-second budget as its RPC timeout. When that call timed out, the loop saw no time left and threw. The 90-second budget assumes that the bounded bootstrap retry resumes the same sandbox later. That retry is not always admitted. A timeout is different from a refusal: the provider is still starting the sandbox. ## What Changed - `reusable-sandbox-resume-replacement.ts`: add `reusableSandboxLeaseIsIdle` (lease is `released` or `retained` and no other live run holds it). `reusableSandboxReplacementIsSafe` now uses it, with the same rules as before. Add `REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS` (330 seconds, the Daytona start deadline plus the call margin). - `environment-runtime.ts`: when the 90-second budget ends on a `transport_timeout` and the preserved lease is idle, extend the deadline one time, up to the provider start deadline (at most 330 seconds from the first attempt). The run keeps retrying the same sandbox with backoff. Each call still has at most 90 seconds. Refusals, a live owner, and other failures keep the current behavior. The run never replaces or destroys the sandbox because of this extension. - `doc/execution-semantics.md`: document the idle-timeout extension. ## Verification - Red before the fix, with the new tests in `server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts` (staging shape: released lease, finished owner run, workspace with unsaved work, 300-second provider deadline, host RPC timeout): 2 tests failed with the staging error `could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` - Green after the fix: 16/16 pass. The timed-out idle sandbox resumes in the same run with no create or destroy call. If every call times out, the run stops at or before 330 seconds, makes more than one call, and keeps the lease `released`. A sandbox that a live run still holds gets one call and fails at 90 seconds, as before. - `pnpm exec vitest run src/__tests__/environment-runtime.test.ts src/services/environment-runtime-reusable-lease-probe.test.ts src/services/remote-workspace-transport.test.ts`: 378 pass. - `tsc -p server/tsconfig.json --noEmit` passes. - `pnpm verify` exits 2 because a doc is a global CI input. Required CI supplies the broad coverage. - Not run: a live Daytona resume on staging. No later run attempted the three preserved leases, so it is inferred, not observed, that the sandboxes were ready after the host call timed out. ## Risks - A run can now wait up to 330 seconds in preparation before its first turn, but only when the resume timed out on an idle sandbox. Before this change, that run failed and the task waited for a person. - A provider that is down (not slow) now takes up to 330 seconds to fail instead of 90 seconds. The failure is the same `reusable_sandbox_resume_failed`, and the lease is still preserved. - No persisted fields change. Leases saved by older releases follow the same rule. - The refused bootstrap retry admission for task Retry continuations is a separate gap. This pull request does not change it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
408 lines
24 KiB
TypeScript
408 lines
24 KiB
TypeScript
import { JsonRpcCallError, environmentResumeErrorData } from "@paperclipai/plugin-sdk";
|
|
import { randomUUID } from "node:crypto";
|
|
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { eq } from "drizzle-orm";
|
|
import {
|
|
agents,
|
|
companies,
|
|
createDb,
|
|
environmentLeases,
|
|
environments,
|
|
executionWorkspaceSnapshots,
|
|
executionWorkspaces,
|
|
heartbeatRuns,
|
|
plugins,
|
|
projects,
|
|
} from "@paperclipai/db";
|
|
import type { Environment } from "@paperclipai/shared";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
|
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
|
import { environmentRuntimeService } from "../services/environment-runtime.ts";
|
|
import { environmentService } from "../services/environments.ts";
|
|
import type { PluginWorkerManager } from "../services/plugin-worker-manager.ts";
|
|
|
|
// Durable harness-backup verification has its own tests. Here it is a boundary:
|
|
// a lease-bound stamp verifies, a missing stamp (an older release's lease) does not.
|
|
vi.mock("../services/native-runtime/native-harness-backup-storage.js", () => ({
|
|
verifyRecoverableNativeHarnessBackup: vi.fn(async (input: { stamp: unknown; providerLeaseId: string }) =>
|
|
Boolean(input.stamp && (input.stamp as { authorizedProviderLeaseId?: unknown }).authorizedProviderLeaseId === input.providerLeaseId)),
|
|
}));
|
|
|
|
const externalTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ENVIRONMENT_RUNTIME_TEST_DATABASE_URL");
|
|
const embeddedPostgresSupport = externalTestDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
|
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
|
|
|
// The staging failure: Daytona kept the idle sandbox in a state transition past
|
|
// its five-minute start deadline, and the worker reported it with no status.
|
|
const STAGING_REFUSAL = "Daytona sandbox 0f6c1e52-3d1a-4b7e-9c2a-5e8f7a1b2c3d is still stopping at its start deadline";
|
|
|
|
describeEmbeddedPostgres("reusable sandbox resume failure replacement", () => {
|
|
let stopDb: (() => Promise<void>) | null = null;
|
|
let db!: ReturnType<typeof createDb>;
|
|
let connectionString: string;
|
|
|
|
beforeAll(async () => {
|
|
if (externalTestDatabaseUrl) connectionString = externalTestDatabaseUrl;
|
|
else {
|
|
const started = await startEmbeddedPostgresTestDatabase("environment-runtime-resume-replacement");
|
|
stopDb = started.cleanup;
|
|
connectionString = started.connectionString;
|
|
}
|
|
});
|
|
beforeEach(() => { db = createDb(connectionString); });
|
|
afterEach(async () => {
|
|
vi.restoreAllMocks();
|
|
await db.delete(executionWorkspaceSnapshots);
|
|
await db.delete(environmentLeases);
|
|
await db.delete(heartbeatRuns);
|
|
await db.delete(agents);
|
|
await db.delete(environments);
|
|
await db.delete(executionWorkspaces);
|
|
await db.delete(plugins);
|
|
await db.delete(projects);
|
|
await db.delete(companies);
|
|
await db.$client.end({ timeout: 1 });
|
|
});
|
|
afterAll(async () => { await stopDb?.(); });
|
|
|
|
type Shape = {
|
|
leaseStatus?: "retained" | "released";
|
|
previousRunStatus?: "succeeded" | "running";
|
|
dirty?: boolean;
|
|
snapshot?: "committed" | "pending" | null;
|
|
backupStamp?: boolean;
|
|
};
|
|
|
|
async function seed(shape: Shape = {}) {
|
|
const companyId = randomUUID();
|
|
const agentId = randomUUID();
|
|
const environmentId = randomUUID();
|
|
const pluginId = randomUUID();
|
|
const previousRunId = randomUUID();
|
|
const runId = randomUUID();
|
|
const projectId = randomUUID();
|
|
const executionWorkspaceId = randomUUID();
|
|
const providerLeaseId = "0f6c1e52-3d1a-4b7e-9c2a-5e8f7a1b2c3d";
|
|
// The staging environment's provider settings, including its 300s start deadline.
|
|
const providerConfig = { provider: "fake-plugin", image: "fake:test", timeoutMs: 300_000, reuseLease: true };
|
|
await db.insert(companies).values({ id: companyId, name: "Acme", status: "active" });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Coder", role: "engineer", status: "active",
|
|
adapterType: "paperclip_runner", adapterConfig: {}, runtimeConfig: {}, permissions: {} });
|
|
await db.insert(environments).values({ id: environmentId, name: "Paperclip Sandbox", driver: "sandbox", status: "active",
|
|
config: providerConfig });
|
|
await db.insert(plugins).values({
|
|
id: pluginId, pluginKey: "acme.reusable-sandbox-provider", packageName: "@acme/reusable-sandbox-provider",
|
|
version: "1.0.0", apiVersion: 1, categories: ["automation"], status: "ready", installOrder: 1, updatedAt: new Date(),
|
|
manifestJson: {
|
|
id: "acme.reusable-sandbox-provider", apiVersion: 1, version: "1.0.0", displayName: "Reusable Sandbox Provider",
|
|
description: "Test provider", author: "Paperclip", categories: ["automation"],
|
|
capabilities: ["environment.drivers.register"], entrypoints: { worker: "dist/worker.js" },
|
|
environmentDrivers: [{ driverKey: "fake-plugin", kind: "sandbox_provider", displayName: "Fake Plugin",
|
|
supportsReusableLeases: true,
|
|
configSchema: { type: "object", properties: { image: { type: "string" }, timeoutMs: { type: "number" }, reuseLease: { type: "boolean" } } } }],
|
|
},
|
|
} as never);
|
|
await db.insert(projects).values({ id: projectId, companyId, name: "Project", status: "active" });
|
|
await db.insert(executionWorkspaces).values({ id: executionWorkspaceId, companyId, projectId, mode: "isolated_workspace",
|
|
strategyType: "project_primary", name: "Task workspace", status: "active", providerType: "cloud_sandbox",
|
|
cwd: `/workspaces/${executionWorkspaceId}` });
|
|
await db.insert(heartbeatRuns).values([
|
|
{ id: previousRunId, companyId, agentId, invocationSource: "assignment", status: "running" },
|
|
{ id: runId, companyId, agentId, invocationSource: "assignment", status: "running" },
|
|
]);
|
|
const environment: Environment = { id: environmentId, name: "Paperclip Sandbox", description: null, driver: "sandbox",
|
|
status: "active", config: providerConfig, envVars: {}, metadata: null, createdAt: new Date(), updatedAt: new Date() };
|
|
// The previous run creates the sandbox through the real acquisition path,
|
|
// so the preserved lease carries exactly the identity production records.
|
|
const priming = environmentRuntimeService(db, { pluginWorkerManager: {
|
|
isRunning: () => true,
|
|
call: async (_pluginId: string, method: string) => {
|
|
if (method !== "environmentAcquireLease") throw new Error(`Unexpected plugin method: ${method}`);
|
|
return { providerLeaseId, metadata: { ...providerConfig, sandboxId: providerLeaseId, remoteCwd: "/home/daytona/paperclip-workspace" } };
|
|
},
|
|
getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] }),
|
|
} as unknown as PluginWorkerManager });
|
|
const created = await priming.acquireRunLease({ companyId, environment, issueId: null, agentId, adapterType: "paperclip_runner",
|
|
heartbeatRunId: previousRunId, persistedExecutionWorkspace: { id: executionWorkspaceId, mode: "isolated_workspace" } });
|
|
// The run then ended after a clean final save and left its lease-bound
|
|
// harness backup stamp. A lease saved by an older release has no
|
|
// acquisition outcome and may predate the backup stamp.
|
|
const { sandboxLeaseAcquisition: _outcome, ...metadata } = created.lease.metadata ?? {};
|
|
const lease = (await environmentService(db).updateLeaseMetadata(created.lease.id, {
|
|
...(shape.leaseStatus === "released" ? metadata : created.lease.metadata),
|
|
remoteWorkspaceId: executionWorkspaceId,
|
|
...(shape.backupStamp === false ? {} : { nativeHarnessBackup: {
|
|
schema: "paperclip.native-harness-backup-stamp.v2", sourceProviderLeaseId: providerLeaseId,
|
|
authorizedProviderLeaseId: providerLeaseId,
|
|
} }),
|
|
}))!;
|
|
await db.update(environmentLeases).set({ status: shape.leaseStatus ?? "retained" }).where(eq(environmentLeases.id, lease.id));
|
|
await db.update(heartbeatRuns).set({ status: shape.previousRunStatus ?? "succeeded" }).where(eq(heartbeatRuns.id, previousRunId));
|
|
const snapshotId = randomUUID();
|
|
if (shape.snapshot !== null) {
|
|
await db.insert(executionWorkspaceSnapshots).values({ id: snapshotId, companyId, executionWorkspaceId,
|
|
sourceOperationId: randomUUID(), sourceRunId: previousRunId, generation: 1, epoch: 1, status: shape.snapshot ?? "committed" });
|
|
}
|
|
// This run already holds the workspace writer slot (reserved before the
|
|
// environment is acquired); the saved version is the previous run's save.
|
|
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: {
|
|
schema: "paperclip.remote-workspace.v1", epoch: 2, snapshotId: shape.snapshot === null ? null : snapshotId,
|
|
source: { kind: "empty" }, writerRunId: runId, dirty: shape.dirty ?? false,
|
|
location: { environmentId, leaseId: lease.id, providerLeaseId, remoteCwd: "/home/daytona/paperclip-workspace/.paperclip-workspaces/w" },
|
|
} } }).where(eq(executionWorkspaces.id, executionWorkspaceId));
|
|
return { companyId, agentId, environment, pluginId, runId, executionWorkspaceId, lease, providerLeaseId };
|
|
}
|
|
|
|
function worker(seeded: Awaited<ReturnType<typeof seed>>, resume: (attempt: number, timeoutMs: number) => Promise<unknown>,
|
|
options: { acquireFails?: boolean } = {}) {
|
|
let attempts = 0;
|
|
const call = vi.fn(async (_pluginId: string, method: string, _params: unknown, timeoutMs?: number) => {
|
|
if (method === "environmentResumeLease") return await resume(++attempts, timeoutMs ?? 0);
|
|
if (method === "environmentAcquireLease") {
|
|
if (options.acquireFails) throw new JsonRpcCallError({ code: -32000, message: "create refused" });
|
|
return { providerLeaseId: "replacement-sandbox", metadata: { provider: "fake-plugin", image: "fake:test",
|
|
timeoutMs: 300_000, reuseLease: true, remoteCwd: "/home/daytona/paperclip-workspace" } };
|
|
}
|
|
throw new Error(`Unexpected plugin method: ${method}`);
|
|
});
|
|
const manager = { isRunning: vi.fn((id: string) => id === seeded.pluginId), call,
|
|
getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })),
|
|
} as unknown as PluginWorkerManager;
|
|
return { call, runtime: environmentRuntimeService(db, { pluginWorkerManager: manager }) };
|
|
}
|
|
|
|
function acquire(seeded: Awaited<ReturnType<typeof seed>>, runtime: ReturnType<typeof environmentRuntimeService>,
|
|
runId = seeded.runId) {
|
|
return runtime.acquireRunLease({ companyId: seeded.companyId, environment: seeded.environment, issueId: null,
|
|
agentId: seeded.agentId, adapterType: "paperclip_runner", heartbeatRunId: runId,
|
|
persistedExecutionWorkspace: { id: seeded.executionWorkspaceId, mode: "isolated_workspace" } });
|
|
}
|
|
|
|
// Each refused resume consumes 40s of the run's clock, so the in-run retry
|
|
// budget is exercised without real waiting.
|
|
function stepClock() {
|
|
let clock = Date.now();
|
|
vi.spyOn(Date, "now").mockImplementation(() => clock);
|
|
return () => { clock += 40_000; };
|
|
}
|
|
|
|
it("replaces a clean, saved sandbox that keeps refusing to resume, within a bounded budget (staging shape)", async () => {
|
|
const seeded = await seed();
|
|
const step = stepClock();
|
|
const { call, runtime } = worker(seeded, async () => {
|
|
step();
|
|
throw new JsonRpcCallError({ code: -32000, message: STAGING_REFUSAL });
|
|
});
|
|
|
|
const acquired = await acquire(seeded, runtime);
|
|
|
|
expect(acquired.lease.providerLeaseId).toBe("replacement-sandbox");
|
|
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "replacement", reason: "resume_failed" });
|
|
const resumes = call.mock.calls.filter(([, method]) => method === "environmentResumeLease");
|
|
// Retried inside the run, then stopped well short of the five-minute deadline.
|
|
expect(resumes.length).toBeGreaterThan(1);
|
|
for (const [, , , timeoutMs] of resumes) expect(timeoutMs).toBeLessThanOrEqual(90_000);
|
|
// The old sandbox is handed to the cleanup sweep; no synchronous destroy
|
|
// delays the first turn.
|
|
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentDestroyLease", expect.anything(), expect.anything());
|
|
await expect(environmentService(db).getLeaseById(seeded.lease.id)).resolves.toMatchObject({
|
|
status: "pending_cleanup", cleanupStatus: "failed", failureReason: "resume_failed",
|
|
});
|
|
});
|
|
|
|
it("keeps the preserved lease when the replacement cannot be created, so the next attempt replaces it", async () => {
|
|
const seeded = await seed();
|
|
const refuse = async () => { throw new JsonRpcCallError({ code: -32000, message: STAGING_REFUSAL }); };
|
|
const step = stepClock();
|
|
const failing = worker(seeded, async () => { step(); return await refuse(); }, { acquireFails: true });
|
|
await expect(acquire(seeded, failing.runtime)).rejects.toThrow();
|
|
await expect(environmentService(db).getLeaseById(seeded.lease.id)).resolves.toMatchObject({
|
|
status: "retained", providerLeaseId: seeded.providerLeaseId,
|
|
});
|
|
|
|
// The bounded retry runs as a new run that reserves the same clean workspace.
|
|
const retryRunId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: retryRunId, companyId: seeded.companyId, agentId: seeded.agentId,
|
|
invocationSource: "assignment", status: "running" });
|
|
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, seeded.runId));
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
|
|
await db.update(executionWorkspaces).set({ metadata: { ...workspace!.metadata,
|
|
remoteWorkspace: { ...(workspace!.metadata!.remoteWorkspace as Record<string, unknown>), writerRunId: retryRunId } } })
|
|
.where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
|
|
const retry = worker(seeded, async () => { step(); return await refuse(); });
|
|
const acquired = await acquire(seeded, retry.runtime, retryRunId);
|
|
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "replacement", reason: "resume_failed" });
|
|
// The replacement carries the verified harness backup, so the session restores
|
|
// without waiting for the old sandbox to be deleted.
|
|
expect(acquired.lease.metadata?.nativeHarnessBackup).toMatchObject({ authorizedProviderLeaseId: seeded.providerLeaseId });
|
|
await expect(environmentService(db).getLeaseById(seeded.lease.id)).resolves.toMatchObject({ status: "pending_cleanup" });
|
|
});
|
|
|
|
it("replaces a released lease saved by an older release in the same way", async () => {
|
|
const seeded = await seed({ leaseStatus: "released" });
|
|
const step = stepClock();
|
|
const { runtime } = worker(seeded, async () => {
|
|
step();
|
|
throw new JsonRpcCallError({ code: -32000, message: "Sandbox state change in progress",
|
|
data: environmentResumeErrorData(Object.assign(new Error("private"), { statusCode: 409 })) });
|
|
});
|
|
const acquired = await acquire(seeded, runtime);
|
|
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "replacement", reason: "resume_failed" });
|
|
});
|
|
|
|
// Staging: a run could not retire its predecessor's warm
|
|
// Runner on this sandbox, did no provider work, and asked for a fresh one.
|
|
async function markReplaceBeforeReuse(seeded: Awaited<ReturnType<typeof seed>>) {
|
|
await environmentService(db).updateLeaseMetadata(seeded.lease.id, { ...seeded.lease.metadata,
|
|
replaceBeforeReuse: { schema: "paperclip.replace-before-reuse.v1", runId: randomUUID(), predecessorRunId: randomUUID(),
|
|
recordedAt: new Date().toISOString() } });
|
|
}
|
|
|
|
it("replaces a resumable sandbox whose previous Runner could not be retired, without resuming it", async () => {
|
|
const seeded = await seed();
|
|
await markReplaceBeforeReuse(seeded);
|
|
const { call, runtime } = worker(seeded, async () => ({ providerLeaseId: seeded.providerLeaseId,
|
|
metadata: { remoteCwd: "/home/daytona/paperclip-workspace" } }));
|
|
const acquired = await acquire(seeded, runtime);
|
|
expect(acquired.lease.providerLeaseId).toBe("replacement-sandbox");
|
|
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "replacement", reason: "resume_failed",
|
|
cause: "native_predecessor_retirement_unproven" });
|
|
// The new sandbox restores the saved session from the verified backup.
|
|
expect(acquired.lease.metadata?.nativeHarnessBackup).toMatchObject({ authorizedProviderLeaseId: seeded.providerLeaseId });
|
|
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentResumeLease", expect.anything(), expect.anything());
|
|
// The old sandbox, with the Runner on it, is kept for the cleanup sweep.
|
|
await expect(environmentService(db).getLeaseById(seeded.lease.id)).resolves.toMatchObject({ status: "pending_cleanup" });
|
|
});
|
|
|
|
it.each([
|
|
["unsaved work on the sandbox", { dirty: true }],
|
|
["no verified harness backup", { backupStamp: false }],
|
|
] as Array<[string, Shape]>)("resumes a sandbox marked for replacement as before when it holds %s", async (_label, shape) => {
|
|
const seeded = await seed(shape);
|
|
await markReplaceBeforeReuse(seeded);
|
|
const { runtime } = worker(seeded, async () => ({ providerLeaseId: seeded.providerLeaseId,
|
|
metadata: { remoteCwd: "/home/daytona/paperclip-workspace" } }));
|
|
const acquired = await acquire(seeded, runtime);
|
|
expect(acquired.lease.providerLeaseId).toBe(seeded.providerLeaseId);
|
|
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "resumed" });
|
|
});
|
|
|
|
it("retries a sandbox state change inside the run and resumes the same sandbox", async () => {
|
|
const seeded = await seed();
|
|
const { call, runtime } = worker(seeded, async (attempt) => {
|
|
if (attempt === 1) throw new JsonRpcCallError({ code: -32000, message: "Sandbox state change in progress" });
|
|
return { providerLeaseId: seeded.providerLeaseId, metadata: { remoteCwd: "/home/daytona/paperclip-workspace" } };
|
|
});
|
|
const acquired = await acquire(seeded, runtime);
|
|
expect(acquired.lease.providerLeaseId).toBe(seeded.providerLeaseId);
|
|
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "resumed" });
|
|
expect(call.mock.calls.filter(([, method]) => method === "environmentResumeLease")).toHaveLength(2);
|
|
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentAcquireLease", expect.anything(), expect.anything());
|
|
});
|
|
|
|
it.each([
|
|
["unsaved work on the sandbox", { dirty: true }, undefined],
|
|
["no saved workspace version", { snapshot: null }, undefined],
|
|
["a saved version that never committed", { snapshot: "pending" }, undefined],
|
|
["a live owner", { previousRunStatus: "running" }, undefined],
|
|
["no verified harness backup (older lease)", { backupStamp: false }, undefined],
|
|
["a provider access refusal", {}, 403],
|
|
] as Array<[string, Shape, number | undefined]>)("keeps the sandbox and fails strictly with %s", async (_label, shape, status) => {
|
|
const seeded = await seed(shape);
|
|
const step = stepClock();
|
|
const { call, runtime } = worker(seeded, async () => {
|
|
step();
|
|
throw new JsonRpcCallError({ code: -32000, message: STAGING_REFUSAL,
|
|
...(status ? { data: environmentResumeErrorData(Object.assign(new Error("private"), { statusCode: status })) } : {}) });
|
|
});
|
|
await expect(acquire(seeded, runtime)).rejects.toThrow(
|
|
/could not be resumed \(cause=sandbox_state_transition( status=403)?\); the lease was preserved and no replacement was created\.$/,
|
|
);
|
|
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentAcquireLease", expect.anything(), expect.anything());
|
|
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentDestroyLease", expect.anything(), expect.anything());
|
|
await expect(environmentService(db).getLeaseById(seeded.lease.id)).resolves.toMatchObject({
|
|
status: shape.leaseStatus ?? "retained", providerLeaseId: seeded.providerLeaseId,
|
|
});
|
|
});
|
|
|
|
function rpcTimeoutClock() {
|
|
let clock = Date.now();
|
|
const startedAt = clock;
|
|
vi.spyOn(Date, "now").mockImplementation(() => clock);
|
|
return {
|
|
elapsed: () => clock - startedAt,
|
|
timeOut: (timeoutMs: number) => {
|
|
clock += timeoutMs;
|
|
return new JsonRpcCallError({ code: -32003,
|
|
message: `RPC call "environmentResumeLease" timed out after ${timeoutMs}ms` });
|
|
},
|
|
};
|
|
}
|
|
|
|
it("keeps waiting for an idle sandbox whose resume timed out and resumes it in the same run (staging shape)", async () => {
|
|
const seeded = await seed({ leaseStatus: "released", dirty: true });
|
|
const clock = rpcTimeoutClock();
|
|
const { call, runtime } = worker(seeded, async (attempt, timeoutMs) => {
|
|
if (attempt === 1) throw clock.timeOut(timeoutMs);
|
|
return { providerLeaseId: seeded.providerLeaseId, metadata: { remoteCwd: "/home/daytona/paperclip-workspace" } };
|
|
});
|
|
|
|
const acquired = await acquire(seeded, runtime);
|
|
|
|
expect(acquired.lease.providerLeaseId).toBe(seeded.providerLeaseId);
|
|
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "resumed" });
|
|
const resumes = call.mock.calls.filter(([, method]) => method === "environmentResumeLease");
|
|
expect(resumes).toHaveLength(2);
|
|
for (const [, , , timeoutMs] of resumes) expect(timeoutMs).toBeLessThanOrEqual(90_000);
|
|
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentAcquireLease", expect.anything(), expect.anything());
|
|
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentDestroyLease", expect.anything(), expect.anything());
|
|
});
|
|
|
|
it("waits out the in-flight resume when a retry finds the stopped sandbox still settling", async () => {
|
|
const seeded = await seed({ leaseStatus: "released", dirty: true });
|
|
const clock = rpcTimeoutClock();
|
|
const { call, runtime } = worker(seeded, async (attempt, timeoutMs) => {
|
|
if (attempt === 1) throw clock.timeOut(timeoutMs);
|
|
if (attempt === 2) throw new JsonRpcCallError({ code: -32000,
|
|
message: "The stopped Daytona sandbox is still settling cancelled work. Retry shortly." });
|
|
return { providerLeaseId: seeded.providerLeaseId, metadata: { remoteCwd: "/home/daytona/paperclip-workspace" } };
|
|
});
|
|
|
|
const acquired = await acquire(seeded, runtime);
|
|
|
|
expect(acquired.lease.providerLeaseId).toBe(seeded.providerLeaseId);
|
|
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "resumed" });
|
|
expect(call.mock.calls.filter(([, method]) => method === "environmentResumeLease")).toHaveLength(3);
|
|
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentAcquireLease", expect.anything(), expect.anything());
|
|
});
|
|
|
|
it("stops waiting at the provider start deadline and keeps the sandbox when every resume times out", async () => {
|
|
const seeded = await seed({ leaseStatus: "released", dirty: true });
|
|
const clock = rpcTimeoutClock();
|
|
const { call, runtime } = worker(seeded, async (_attempt, timeoutMs) => { throw clock.timeOut(timeoutMs); });
|
|
|
|
await expect(acquire(seeded, runtime)).rejects.toThrow(
|
|
/could not be resumed \(cause=transport_timeout\); the lease was preserved and no replacement was created\.$/,
|
|
);
|
|
|
|
expect(call.mock.calls.filter(([, method]) => method === "environmentResumeLease").length).toBeGreaterThan(1);
|
|
expect(clock.elapsed()).toBeLessThanOrEqual(330_000);
|
|
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentAcquireLease", expect.anything(), expect.anything());
|
|
expect(call).not.toHaveBeenCalledWith(seeded.pluginId, "environmentDestroyLease", expect.anything(), expect.anything());
|
|
await expect(environmentService(db).getLeaseById(seeded.lease.id)).resolves.toMatchObject({
|
|
status: "released", providerLeaseId: seeded.providerLeaseId,
|
|
});
|
|
});
|
|
|
|
it("does not extend the wait for a sandbox that a live run still holds", async () => {
|
|
const seeded = await seed({ previousRunStatus: "running", dirty: true });
|
|
const clock = rpcTimeoutClock();
|
|
const { call, runtime } = worker(seeded, async (_attempt, timeoutMs) => { throw clock.timeOut(timeoutMs); });
|
|
|
|
await expect(acquire(seeded, runtime)).rejects.toThrow(/could not be resumed \(cause=transport_timeout\)/);
|
|
|
|
expect(call.mock.calls.filter(([, method]) => method === "environmentResumeLease")).toHaveLength(1);
|
|
expect(clock.elapsed()).toBeLessThanOrEqual(90_000);
|
|
});
|
|
});
|