mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path - #20065 made idle warm sandboxes owned records on the shared retained-orphan kind. A warm sandbox no turn can resume is stopped if running, then destroyed. - One branch of that predicate decides that a task "moved" off a workspace. As merged, it was true whenever no task was bound to the lease's workspace. That includes a task whose binding is null: finished, or between bindings. - A null binding is not proof of a move. The warm sandbox may still be resumable, and for a remote workspace it can hold the files. Phase 2's review flagged this as blocking, so this follow-up requires positive evidence. ## Linked Issues or Issue Description Refs #20048, follow-up to #20065 **What happened?** `taskMovedToAnotherWorkspace()` listed a warm lease as abandoned when its task had no bound workspace. The pending-export check was then the only thing preventing the destroy of a sandbox that could still be resumed. **Steps to reproduce** 1. Create a warm (`reuse_by_environment`) lease, released with a confirmed stop, bound to workspace W. 2. Clear the task's `execution_workspace_id` (task finished, or between bindings). 3. Run an ownership reconciliation pass: on master the lease is settled and queued for destruction. **Expected behavior** Only positive evidence counts as a move: the lease's own task is bound to a different, non-null workspace, and no task is bound to W. An unbound task's warm sandbox is kept; a done or cancelled task's sandbox goes through the existing retention paths. ## What Changed - `server/src/services/environment-retained-orphan-settlement.ts`: - `taskMovedToAnotherWorkspace()` now requires that the lease's own task is bound to a non-null workspace different from the lease's, and that no task in the company is bound to the lease's workspace. - `newestRowForSandbox()` breaks `created_at` ties by id, so two rows written in the same millisecond can't both count as newest. - `server/src/__tests__/environment-retained-orphan-sandbox.test.ts`, two keep cases: - the task has no bound workspace, for both a released and a retained warm lease; - two rows share the newest `created_at`. ## Verification - `cd server && npx vitest run src/__tests__/environment-retained-orphan-sandbox.test.ts`: 32/32 pass. - With master's settlement file, the null-binding case fails, because the lease is settled. The tie case also passes on master, since the settle's existing shared-resource hold protects it; it stays as a regression test for the listing. - The full chain (retained, stop confirmed, listed, handed to destruction) is covered in one test from #20065. ## Risks - **Fewer sandboxes cleaned.** A warm sandbox whose task finished unbound now waits for the existing retention paths instead of this branch. That is the conservative direction, so files are never lost. - **Scope.** No change to the holds or the settle contract. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
453 lines
27 KiB
TypeScript
453 lines
27 KiB
TypeScript
import { spawn } from "node:child_process";
|
|
import { randomUUID } from "node:crypto";
|
|
import { eq, ne } from "drizzle-orm";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import {
|
|
agents,
|
|
companies,
|
|
createDb,
|
|
environmentLeases,
|
|
environments,
|
|
executionWorkspaces,
|
|
heartbeatRuns,
|
|
issueRecoveryActions,
|
|
issues,
|
|
} from "@paperclipai/db";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
|
import { currentProcessSpaceHash } from "../services/controller-process-identity.js";
|
|
import { nextEnvironmentCleanupAt } from "../services/environment-cleanup-state.js";
|
|
import { ENVIRONMENT_LEASE_CONTROLLER_KEY } from "../services/environment-lease-controller.js";
|
|
import { hasPendingRunCleanup } from "../services/execution-blocker.js";
|
|
import { heartbeatService, type HeartbeatEnvironmentRuntime } from "../services/heartbeat.js";
|
|
import { hasRecoveredCleanupStopReceipt } from "../services/remote-execution-termination.js";
|
|
import { taskChatRetryExecutionStopped } from "../services/chat-channels/task-retry-authority.js";
|
|
import { createOwnershipReportState, runOwnershipReconciliationPass } from "../services/recovery/ownership-invariant-report.js";
|
|
import { ORPHANED_SANDBOX_RECOVERY_KEY, settleStopOnlyCleanup } from "../services/sandbox-stop-and-retain.js";
|
|
import { releasableRetainedOrphanCondition } from "../services/environment-retained-orphan-settlement.js";
|
|
|
|
const support = await getEmbeddedPostgresTestSupport();
|
|
const suite = support.supported ? describe : describe.skip;
|
|
const KIND = "environment_retained_orphan_sandbox";
|
|
|
|
async function exitedPid(): Promise<number> {
|
|
const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { stdio: "ignore" });
|
|
const pid = child.pid!;
|
|
await new Promise<void>(resolve => { child.once("exit", () => resolve()); child.kill("SIGKILL"); });
|
|
return pid;
|
|
}
|
|
|
|
suite("retained orphan sandbox on the shared ownership path", () => {
|
|
let db!: ReturnType<typeof createDb>;
|
|
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
|
let companyId = "";
|
|
let agentId = "";
|
|
let environmentId = "";
|
|
let issueId = "";
|
|
|
|
beforeAll(async () => {
|
|
temporary = await startEmbeddedPostgresTestDatabase("paperclip-retained-orphan-sandbox-");
|
|
db = createDb(temporary.connectionString);
|
|
companyId = randomUUID(); agentId = randomUUID(); environmentId = randomUUID(); issueId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Paperclip", issuePrefix: "ROS",
|
|
requireBoardApprovalForNewAgents: false });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Coder", role: "engineer", status: "idle",
|
|
adapterType: "codex_local", adapterConfig: {}, runtimeConfig: {}, permissions: {} });
|
|
await db.insert(environments).values({ id: environmentId, name: "Fake Sandbox", driver: "sandbox",
|
|
status: "active", config: { provider: "fake", image: "ubuntu:24.04" } });
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Orphaned work" });
|
|
}, 30_000);
|
|
|
|
afterEach(async () => {
|
|
await db.delete(issueRecoveryActions);
|
|
await db.delete(environmentLeases);
|
|
await db.delete(heartbeatRuns);
|
|
await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.id, issueId));
|
|
await db.delete(issues).where(ne(issues.id, issueId));
|
|
await db.delete(executionWorkspaces);
|
|
});
|
|
|
|
afterAll(async () => {
|
|
await temporary?.cleanup();
|
|
});
|
|
|
|
async function run(status: string) {
|
|
const id = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id, companyId, agentId, issueId, invocationSource: "on_demand", status, nextEventSeq: 1 });
|
|
return id;
|
|
}
|
|
|
|
async function row(id: string) {
|
|
const [found] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, id));
|
|
return found!;
|
|
}
|
|
|
|
/** The end state of an orphaned run lease: the ownership settle stopped the
|
|
* sandbox and kept it, and the stop was confirmed. */
|
|
async function retainedOrphan(runId: string, extra: { expiresAt?: Date; metadata?: Record<string, unknown> } = {}) {
|
|
const id = randomUUID(), providerLeaseId = `sandbox://fake/${id}`, now = new Date();
|
|
await db.insert(environmentLeases).values({ id, companyId, environmentId, issueId, heartbeatRunId: runId,
|
|
status: "released", cleanupStatus: "success", leasePolicy: "ephemeral", provider: "fake", providerLeaseId,
|
|
acquiredAt: new Date(now.getTime() - 60_000), releasedAt: now, expiresAt: extra.expiresAt ?? null,
|
|
metadata: {
|
|
[ORPHANED_SANDBOX_RECOVERY_KEY]: { recoveredOrphanedRunLease: true, ownerProof: "pid_dead", runId },
|
|
remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId, runId, leaseId: id,
|
|
provider: "fake", providerLeaseId, state: "stopped", confirmedAt: now.toISOString() },
|
|
sandboxStopAndRetainReceipt: { schema: "paperclip.sandbox-stop-and-retain-receipt.v1", requestId: randomUUID(),
|
|
companyId, runId, leaseId: id, provider: "fake", providerLeaseId, builtinProvider: "fake",
|
|
method: "builtin.stopLease", confirmedAt: now.toISOString() },
|
|
...extra.metadata,
|
|
} });
|
|
return id;
|
|
}
|
|
|
|
async function pass(record = true) {
|
|
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record });
|
|
return { report, kind: report.kinds.find(entry => entry.kind === KIND)! };
|
|
}
|
|
|
|
it("marks the lease when the orphaned-run-lease settle stops and keeps its sandbox", async () => {
|
|
const runId = await run("failed");
|
|
const id = randomUUID();
|
|
await db.insert(environmentLeases).values({ id, companyId, environmentId, heartbeatRunId: runId, status: "active",
|
|
provider: "fake", providerLeaseId: `sandbox://fake/${id}`, metadata: { [ENVIRONMENT_LEASE_CONTROLLER_KEY]: {
|
|
bootId: "earlier-boot", pid: await exitedPid(), hostHash: currentProcessSpaceHash(), processStartedAt: null } } });
|
|
await pass();
|
|
expect((await row(id)).metadata?.[ORPHANED_SANDBOX_RECOVERY_KEY]).toMatchObject({
|
|
recoveredOrphanedRunLease: true, ownerProof: "pid_dead", runId });
|
|
});
|
|
|
|
it("keeps the orphan marker for a stop intent saved before the top-level marker existed", async () => {
|
|
const runId = await run("failed");
|
|
const id = randomUUID(), providerLeaseId = `sandbox://fake/${id}`, attemptId = randomUUID();
|
|
const intent = { schema: "paperclip.sandbox-stop-and-retain.v1", requestId: attemptId, companyId, runId, leaseId: id,
|
|
provider: "fake", providerLeaseId, builtinProvider: "fake", recoveredOrphanedRunLease: true, ownerProof: "pid_dead" };
|
|
await db.insert(environmentLeases).values({ id, companyId, environmentId, issueId, heartbeatRunId: runId,
|
|
status: "pending_cleanup", cleanupStatus: "failed", failureReason: "sandbox_stop_pending", leasePolicy: "ephemeral",
|
|
provider: "fake", providerLeaseId, metadata: { sandboxStopAndRetain: intent, pendingCleanupAttemptId: attemptId,
|
|
pendingCleanupInFlight: true } });
|
|
const settled = await settleStopOnlyCleanup(db, await row(id), { attemptId, receipt: { providerLeaseId, state: "stopped" } });
|
|
expect(settled).toMatchObject({ status: "released", cleanupStatus: "success" });
|
|
expect(settled?.metadata?.[ORPHANED_SANDBOX_RECOVERY_KEY]).toMatchObject({ recoveredOrphanedRunLease: true,
|
|
ownerProof: "pid_dead", runId, fromStopIntent: true });
|
|
const { kind } = await pass();
|
|
expect(kind).toMatchObject({ settled: 1 });
|
|
expect((await row(id))).toMatchObject({ status: "pending_cleanup", failureReason: "retained_orphan_sandbox_released" });
|
|
});
|
|
|
|
it("does not mark an ordinary stop-and-retain as an orphan", async () => {
|
|
const runId = await run("failed");
|
|
const id = randomUUID(), providerLeaseId = `sandbox://fake/${id}`, attemptId = randomUUID();
|
|
const intent = { schema: "paperclip.sandbox-stop-and-retain.v1", requestId: attemptId, companyId, runId, leaseId: id,
|
|
provider: "fake", providerLeaseId, builtinProvider: "fake" };
|
|
await db.insert(environmentLeases).values({ id, companyId, environmentId, issueId, heartbeatRunId: runId,
|
|
status: "pending_cleanup", cleanupStatus: "failed", leasePolicy: "ephemeral", provider: "fake", providerLeaseId,
|
|
metadata: { sandboxStopAndRetain: intent, pendingCleanupAttemptId: attemptId, pendingCleanupInFlight: true } });
|
|
const settled = await settleStopOnlyCleanup(db, await row(id), { attemptId, receipt: { providerLeaseId, state: "stopped" } });
|
|
expect(settled?.metadata?.[ORPHANED_SANDBOX_RECOVERY_KEY]).toBeUndefined();
|
|
expect((await pass()).kind).toMatchObject({ settled: 0 });
|
|
});
|
|
|
|
it("destroys a retained orphan that nothing references, through the cleanup sweep", async () => {
|
|
const leaseId = await retainedOrphan(await run("failed"));
|
|
const { kind } = await pass();
|
|
expect(kind).toMatchObject({ inProgress: 1, settled: 1 });
|
|
const released = await row(leaseId);
|
|
expect(released).toMatchObject({ status: "pending_cleanup", leasePolicy: "ephemeral",
|
|
failureReason: "retained_orphan_sandbox_released" });
|
|
expect(released.metadata?.remoteExecutionTermination).toBeUndefined();
|
|
expect(released.metadata?.workspaceRecoveryCleanup).toMatchObject({ runId: released.heartbeatRunId,
|
|
reason: "retained_orphan_sandbox_released", ownerProof: "owner_run_terminal", stopReceipt: { state: "stopped" } });
|
|
// Admission and Retry still see a stopped sandbox while the destroy waits.
|
|
expect(hasRecoveredCleanupStopReceipt(released)).toBe(true);
|
|
expect(await hasPendingRunCleanup(db, companyId, released.heartbeatRunId!)).toBe(false);
|
|
const [failedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, released.heartbeatRunId!));
|
|
expect(await taskChatRetryExecutionStopped(db, failedRun!)).toBe(true);
|
|
// Due now, and the sweep destroys it from the recorded lease.
|
|
expect(await nextEnvironmentCleanupAt(db)).toBeLessThanOrEqual(Date.now());
|
|
const teardown = vi.fn(async ({ lease }: { lease: { providerLeaseId: string | null } }) =>
|
|
({ providerLeaseId: lease.providerLeaseId, state: "destroyed" }));
|
|
await heartbeatService(db, { environmentRuntime: { retryPendingSandboxTeardown: teardown } as unknown as HeartbeatEnvironmentRuntime })
|
|
.sweepPendingCleanupLeases({ backoffMs: 5 * 60_000 });
|
|
expect(teardown).toHaveBeenCalledOnce();
|
|
expect((await row(leaseId)).status).toBe("expired");
|
|
});
|
|
|
|
it.each([
|
|
["an open recovery action on the issue", async (_runId: string) => {
|
|
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
|
|
cause: "fixture", fingerprint: randomUUID(), nextAction: "Recover the workspace." });
|
|
return {};
|
|
}],
|
|
["a queued run on the issue", async (_runId: string) => { await run("queued"); return {}; }],
|
|
["a scheduled retry on the issue", async (_runId: string) => { await run("scheduled_retry"); return {}; }],
|
|
["a stop-and-retain intent", async () => ({ metadata: { sandboxStopAndRetain: { requestId: "pending" } } })],
|
|
["a native export resume", async () => ({ metadata: { nativeWorkspaceExportResume: { requestId: "pending" } } })],
|
|
["a workspace restore recovery", async () => ({ metadata: { workspaceRestoreRecovery: { runId: "fixture" } } })],
|
|
["an in-flight cleanup claim", async () => ({ metadata: { pendingCleanupInFlight: true, pendingCleanupAttemptId: "a" } })],
|
|
["a pending legacy shutdown fence", async () => ({ metadata: { legacyShutdownWorkspaceSettlement: { state: "pending" } } })],
|
|
] as const)("never destroys a sandbox recovery still wants: %s", async (_label, arrange) => {
|
|
const runId = await run("failed");
|
|
const extra = await arrange(runId);
|
|
const leaseId = await retainedOrphan(runId, extra);
|
|
const before = await row(leaseId);
|
|
const { kind } = await pass();
|
|
expect(kind).toMatchObject({ settled: 0 });
|
|
expect(await row(leaseId)).toEqual(before);
|
|
});
|
|
|
|
it("never destroys a sandbox another live lease holds, or one whose run is still active", async () => {
|
|
const live = await retainedOrphan(await run("running"));
|
|
const sharedRun = await run("failed");
|
|
const shared = await retainedOrphan(sharedRun);
|
|
const sharedRow = await row(shared);
|
|
await db.insert(environmentLeases).values({ companyId, environmentId, heartbeatRunId: await run("running"),
|
|
status: "active", provider: "fake", providerLeaseId: sharedRow.providerLeaseId });
|
|
await pass();
|
|
expect((await row(live)).status).toBe("released");
|
|
expect((await row(shared)).status).toBe("released");
|
|
});
|
|
|
|
it("keeps a softly referenced orphan only until its retention window ends, and wakes then", async () => {
|
|
const runId = await run("failed");
|
|
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
|
|
cause: "fixture", fingerprint: randomUUID(), nextAction: "Recover the workspace." });
|
|
const expiresAt = new Date(Date.now() + 60 * 60_000);
|
|
const leaseId = await retainedOrphan(runId, { expiresAt });
|
|
const first = await pass();
|
|
expect(first.kind).toMatchObject({ inProgress: 0 });
|
|
expect(first.report.nextDueAt?.getTime()).toBeLessThanOrEqual(expiresAt.getTime());
|
|
expect((await row(leaseId)).status).toBe("released");
|
|
// Past the window no recovery path accepts the stopped sandbox as a source.
|
|
await db.update(environmentLeases).set({ expiresAt: new Date(Date.now() - 1_000) }).where(eq(environmentLeases.id, leaseId));
|
|
const second = await pass();
|
|
expect(second.kind).toMatchObject({ settled: 1 });
|
|
expect((await row(leaseId)).status).toBe("pending_cleanup");
|
|
});
|
|
|
|
it("never changes anything on a read-only pass", async () => {
|
|
const leaseId = await retainedOrphan(await run("failed"));
|
|
const before = await row(leaseId);
|
|
const { kind } = await pass(false);
|
|
expect(kind).toMatchObject({ inProgress: 1, settled: 0, verdicts: { gone: 1 } });
|
|
expect(await row(leaseId)).toEqual(before);
|
|
});
|
|
|
|
it("leaves a reusable sandbox to warm reuse", async () => {
|
|
const leaseId = await retainedOrphan(await run("failed"));
|
|
await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment" }).where(eq(environmentLeases.id, leaseId));
|
|
const before = await row(leaseId);
|
|
const { kind } = await pass();
|
|
expect(kind).toMatchObject({ inProgress: 0 });
|
|
expect(await row(leaseId)).toEqual(before);
|
|
});
|
|
|
|
it("keeps a sandbox with a pending native workspace export even past its retention window", async () => {
|
|
const runId = await run("failed");
|
|
const leaseId = await retainedOrphan(runId, { expiresAt: new Date(Date.now() - 1_000) });
|
|
const lease = await row(leaseId);
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "native", runnerProfileJson: { nativeWorkspaceSync: {
|
|
schema: "paperclip.native-workspace-sync/v3", state: "prepared", descriptorSha256: "a".repeat(64),
|
|
baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId,
|
|
providerLeaseId: lease.providerLeaseId, remoteCwd: "/workspace", resourceDisposition: "destroy" } } })
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
const { kind } = await pass();
|
|
// Not listed, so the settle watch does not poll it while the export waits.
|
|
expect(kind).toMatchObject({ inProgress: 0 });
|
|
expect(await row(leaseId)).toEqual(lease);
|
|
});
|
|
|
|
it("releases a workspace-less native run's orphan whose descriptor stays prepared", async () => {
|
|
const runId = await run("failed");
|
|
const leaseId = await retainedOrphan(runId);
|
|
const lease = await row(leaseId);
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "native", runnerProfileJson: {
|
|
nativeExecutionInput: { workspace: { access: "none", cwd: null } },
|
|
nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: "prepared", descriptorSha256: "a".repeat(64),
|
|
baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId,
|
|
providerLeaseId: lease.providerLeaseId, remoteCwd: "/workspace", resourceDisposition: "destroy" } } })
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
// Listed (no export obligation is visible to the listing); the settle
|
|
// decides with the exact export check.
|
|
const { kind } = await pass(false);
|
|
expect(kind).toMatchObject({ inProgress: 1 });
|
|
});
|
|
|
|
it("releases an orphan whose run's prepared descriptor names another sandbox", async () => {
|
|
const runId = await run("failed");
|
|
const leaseId = await retainedOrphan(runId);
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "native", runnerProfileJson: {
|
|
nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: "prepared", descriptorSha256: "a".repeat(64),
|
|
baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId: randomUUID(),
|
|
providerLeaseId: "sandbox://fake/other", remoteCwd: "/workspace", resourceDisposition: "destroy" } } })
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
const { kind } = await pass();
|
|
expect(kind).toMatchObject({ settled: 1 });
|
|
expect((await row(leaseId)).status).toBe("pending_cleanup");
|
|
});
|
|
|
|
it("does not hide an orphan behind a prepared descriptor the settle would reject", async () => {
|
|
const runId = await run("failed");
|
|
const leaseId = await retainedOrphan(runId);
|
|
const lease = await row(leaseId);
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "native", runnerProfileJson: {
|
|
nativeWorkspaceSync: { state: "prepared", leaseId, providerLeaseId: lease.providerLeaseId } } })
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
const { kind } = await pass();
|
|
expect(kind).toMatchObject({ settled: 1 });
|
|
expect((await row(leaseId)).status).toBe("pending_cleanup");
|
|
});
|
|
|
|
async function workspace() {
|
|
const [created] = await db.insert(executionWorkspaces).values({ companyId, mode: "isolated_workspace",
|
|
strategyType: "git_worktree", name: "Task workspace" }).returning({ id: executionWorkspaces.id });
|
|
return created!.id;
|
|
}
|
|
|
|
/** A warm sandbox the previous release left behind: a reuse_by_environment
|
|
* lease the plugin stopped and kept for the next turn on its workspace. This
|
|
* is the exact row shape releases before this change wrote; it carries no
|
|
* new marker. */
|
|
async function warmLease(runId: string, input: { status?: string; workspaceId: string | null; receipt?: boolean;
|
|
providerLeaseId?: string; createdAt?: Date }) {
|
|
const id = randomUUID(), providerLeaseId = input.providerLeaseId ?? `sandbox://fake/${id}`;
|
|
const now = input.createdAt ?? new Date();
|
|
await db.insert(environmentLeases).values({ id, companyId, environmentId, issueId, heartbeatRunId: runId,
|
|
executionWorkspaceId: input.workspaceId, status: input.status ?? "released", cleanupStatus: "success",
|
|
leasePolicy: "reuse_by_environment", provider: "fake", providerLeaseId, acquiredAt: now, releasedAt: now,
|
|
createdAt: now, updatedAt: now,
|
|
metadata: { agentId, reuseLease: true, ...(input.receipt === false ? {} : { remoteExecutionTermination: {
|
|
schema: "paperclip.remote-termination.v1", companyId, runId, leaseId: id, provider: "fake", providerLeaseId,
|
|
state: "stopped", confirmedAt: now.toISOString() } }) } });
|
|
return id;
|
|
}
|
|
|
|
it("destroys a task's warm sandbox once the task moves to another workspace", async () => {
|
|
const [oldWorkspace, newWorkspace] = [await workspace(), await workspace()];
|
|
const leaseId = await warmLease(await run("failed"), { workspaceId: oldWorkspace });
|
|
// A Retry kept a fresh workspace: no acquire can match the old lineage again.
|
|
await db.update(issues).set({ executionWorkspaceId: newWorkspace }).where(eq(issues.id, issueId));
|
|
const { kind } = await pass();
|
|
expect(kind).toMatchObject({ settled: 1 });
|
|
const released = await row(leaseId);
|
|
expect(released).toMatchObject({ status: "pending_cleanup", failureReason: "retained_orphan_sandbox_released" });
|
|
// The successor is never blocked on this destroy.
|
|
expect(hasRecoveredCleanupStopReceipt(released)).toBe(true);
|
|
expect(await hasPendingRunCleanup(db, companyId, released.heartbeatRunId!)).toBe(false);
|
|
});
|
|
|
|
it("destroys a warm sandbox behind a failed lease, which no turn can resume", async () => {
|
|
const current = await workspace();
|
|
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
|
|
const leaseId = await warmLease(await run("failed"), { status: "failed", workspaceId: current });
|
|
const { kind } = await pass();
|
|
expect(kind).toMatchObject({ settled: 1 });
|
|
expect((await row(leaseId)).status).toBe("pending_cleanup");
|
|
});
|
|
|
|
it.each([
|
|
["the task can still resume it on its own workspace", async () => {
|
|
const current = await workspace();
|
|
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
|
|
return [await warmLease(await run("succeeded"), { workspaceId: current })];
|
|
}],
|
|
["it has no confirmed stop receipt", async () => {
|
|
await db.update(issues).set({ executionWorkspaceId: await workspace() }).where(eq(issues.id, issueId));
|
|
return [await warmLease(await run("failed"), { status: "failed", workspaceId: await workspace(), receipt: false })];
|
|
}],
|
|
["a newer lease on the same sandbox handed it on", async () => {
|
|
const current = await workspace();
|
|
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
|
|
const providerLeaseId = `sandbox://fake/${randomUUID()}`;
|
|
const older = await warmLease(await run("failed"), { status: "failed", workspaceId: current, providerLeaseId,
|
|
createdAt: new Date(Date.now() - 60_000) });
|
|
const newer = await warmLease(await run("succeeded"), { workspaceId: current, providerLeaseId });
|
|
return [older, newer];
|
|
}],
|
|
["its task has no bound workspace (finished, or between bindings)", async () => {
|
|
await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.id, issueId));
|
|
return [await warmLease(await run("succeeded"), { workspaceId: await workspace() }),
|
|
await warmLease(await run("succeeded"), { status: "retained", workspaceId: await workspace(), receipt: false })];
|
|
}],
|
|
["another task still shares its workspace", async () => {
|
|
const [shared, fresh] = [await workspace(), await workspace()];
|
|
await db.update(issues).set({ executionWorkspaceId: fresh }).where(eq(issues.id, issueId));
|
|
await db.insert(issues).values({ companyId, title: "Sibling on the shared workspace", executionWorkspaceId: shared });
|
|
return [await warmLease(await run("failed"), { workspaceId: shared })];
|
|
}],
|
|
["the task still has a run in flight", async () => {
|
|
const [oldWorkspace, newWorkspace] = [await workspace(), await workspace()];
|
|
await db.update(issues).set({ executionWorkspaceId: newWorkspace }).where(eq(issues.id, issueId));
|
|
const leaseId = await warmLease(await run("failed"), { workspaceId: oldWorkspace });
|
|
await run("running");
|
|
return [leaseId];
|
|
}],
|
|
] as const)("keeps a warm sandbox while %s", async (_label, arrange) => {
|
|
const ids = await arrange();
|
|
const before = await Promise.all(ids.map(row));
|
|
const { kind } = await pass();
|
|
expect(kind).toMatchObject({ settled: 0 });
|
|
expect(await Promise.all(ids.map(row))).toEqual(before);
|
|
});
|
|
|
|
it("lists only one newest row when two rows for a sandbox share a created_at", async () => {
|
|
const current = await workspace();
|
|
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
|
|
const providerLeaseId = `sandbox://fake/${randomUUID()}`, createdAt = new Date();
|
|
const [lower, higher] = ["00000000-0000-4000-8000-000000000001", "ffffffff-ffff-4fff-bfff-ffffffffffff"];
|
|
for (const id of [lower, higher]) {
|
|
const runId = await run("failed");
|
|
await db.insert(environmentLeases).values({ id, companyId, environmentId, issueId, heartbeatRunId: runId,
|
|
executionWorkspaceId: current, status: "failed", cleanupStatus: "success", leasePolicy: "reuse_by_environment",
|
|
provider: "fake", providerLeaseId, acquiredAt: createdAt, releasedAt: createdAt, createdAt, updatedAt: createdAt,
|
|
metadata: { agentId, remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId, runId,
|
|
leaseId: id, provider: "fake", providerLeaseId, state: "stopped", confirmedAt: createdAt.toISOString() } } });
|
|
}
|
|
const listed = await db.select({ id: environmentLeases.id }).from(environmentLeases)
|
|
.where(releasableRetainedOrphanCondition());
|
|
expect(listed.map(row => row.id)).toEqual([higher]);
|
|
});
|
|
|
|
it("stops, then destroys, a running warm sandbox once its task moves to another workspace", async () => {
|
|
const [oldWorkspace, newWorkspace] = [await workspace(), await workspace()];
|
|
const runId = await run("succeeded");
|
|
// A keep-running release: retained, no stop, no receipt. Its idle timer
|
|
// lived in the process that kept it.
|
|
const leaseId = await warmLease(runId, { status: "retained", workspaceId: oldWorkspace, receipt: false });
|
|
await db.update(environmentLeases).set({ releasedAt: null }).where(eq(environmentLeases.id, leaseId));
|
|
await db.update(issues).set({ executionWorkspaceId: newWorkspace }).where(eq(issues.id, issueId));
|
|
|
|
expect((await pass()).kind).toMatchObject({ settled: 1 });
|
|
const stopping = await row(leaseId);
|
|
expect(stopping).toMatchObject({ status: "pending_cleanup", failureReason: "sandbox_stop_pending" });
|
|
const intent = stopping.metadata?.sandboxStopAndRetain as { requestId: string; recoveredOrphanedRunLease?: boolean };
|
|
expect(intent).toMatchObject({ recoveredOrphanedRunLease: true, supersededWarmSandbox: true });
|
|
// The cleanup sweep's first stop is due at once.
|
|
expect(await nextEnvironmentCleanupAt(db)).toBeLessThanOrEqual(Date.now());
|
|
|
|
// The provider confirms the stop; the next pass hands it to destruction.
|
|
const stopped = await settleStopOnlyCleanup(db, stopping, { attemptId: intent.requestId,
|
|
receipt: { providerLeaseId: stopping.providerLeaseId, state: "stopped" } });
|
|
expect(stopped).toMatchObject({ status: "released", cleanupStatus: "success" });
|
|
expect((await pass()).kind).toMatchObject({ settled: 1 });
|
|
expect(await row(leaseId)).toMatchObject({ status: "pending_cleanup", failureReason: "retained_orphan_sandbox_released" });
|
|
});
|
|
|
|
it("keeps a running warm sandbox its task can still resume after a restart", async () => {
|
|
const current = await workspace();
|
|
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
|
|
const leaseId = await warmLease(await run("succeeded"), { status: "retained", workspaceId: current, receipt: false });
|
|
const before = await row(leaseId);
|
|
expect((await pass()).kind).toMatchObject({ settled: 0 });
|
|
expect(await row(leaseId)).toEqual(before);
|
|
});
|
|
|
|
it("leaves an ordinary retained sandbox (no orphan marker) alone", async () => {
|
|
const leaseId = await retainedOrphan(await run("failed"));
|
|
const { [ORPHANED_SANDBOX_RECOVERY_KEY]: _marker, ...metadata } = (await row(leaseId)).metadata!;
|
|
await db.update(environmentLeases).set({ metadata }).where(eq(environmentLeases.id, leaseId));
|
|
const { kind } = await pass();
|
|
expect(kind).toMatchObject({ inProgress: 0 });
|
|
expect((await row(leaseId)).status).toBe("released");
|
|
});
|
|
});
|