Files
PaperClipAI/server/src/__tests__/environment-retained-orphan-sandbox.test.ts
T
DottaandClaude Opus 5.5 0a64c9b4ab fix(recovery): treat a warm sandbox as abandoned only on positive evidence its task moved (#20070)
## Thinking Path

- #20065 made idle warm sandboxes owned records on the shared
retained-orphan kind. A warm sandbox no turn can resume is stopped if
running, then destroyed.
- One branch of that predicate decides that a task "moved" off a
workspace. As merged, it was true whenever no task was bound to the
lease's workspace. That includes a task whose binding is null: finished,
or between bindings.
- A null binding is not proof of a move. The warm sandbox may still be
resumable, and for a remote workspace it can hold the files. Phase 2's
review flagged this as blocking, so this follow-up requires positive
evidence.

## Linked Issues or Issue Description

Refs #20048, follow-up to #20065

**What happened?**
`taskMovedToAnotherWorkspace()` listed a warm lease as abandoned when
its task had no bound workspace. The pending-export check was then the
only thing preventing the destroy of a sandbox that could still be
resumed.

**Steps to reproduce**
1. Create a warm (`reuse_by_environment`) lease, released with a
confirmed stop, bound to workspace W.
2. Clear the task's `execution_workspace_id` (task finished, or between
bindings).
3. Run an ownership reconciliation pass: on master the lease is settled
and queued for destruction.

**Expected behavior**
Only positive evidence counts as a move: the lease's own task is bound
to a different, non-null workspace, and no task is bound to W. An
unbound task's warm sandbox is kept; a done or cancelled task's sandbox
goes through the existing retention paths.

## What Changed

- `server/src/services/environment-retained-orphan-settlement.ts`:
- `taskMovedToAnotherWorkspace()` now requires that the lease's own task
is bound to a non-null workspace different from the lease's, and that no
task in the company is bound to the lease's workspace.
- `newestRowForSandbox()` breaks `created_at` ties by id, so two rows
written in the same millisecond can't both count as newest.
- `server/src/__tests__/environment-retained-orphan-sandbox.test.ts`,
two keep cases:
- the task has no bound workspace, for both a released and a retained
warm lease;
  - two rows share the newest `created_at`.

## Verification

- `cd server && npx vitest run
src/__tests__/environment-retained-orphan-sandbox.test.ts`: 32/32 pass.
- With master's settlement file, the null-binding case fails, because
the lease is settled. The tie case also passes on master, since the
settle's existing shared-resource hold protects it; it stays as a
regression test for the listing.
- The full chain (retained, stop confirmed, listed, handed to
destruction) is covered in one test from #20065.

## Risks

- **Fewer sandboxes cleaned.** A warm sandbox whose task finished
unbound now waits for the existing retention paths instead of this
branch. That is the conservative direction, so files are never lost.
- **Scope.** No change to the holds or the settle contract.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 10:56:32 -06:00

453 lines
27 KiB
TypeScript

import { spawn } from "node:child_process";
import { randomUUID } from "node:crypto";
import { eq, ne } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
agents,
companies,
createDb,
environmentLeases,
environments,
executionWorkspaces,
heartbeatRuns,
issueRecoveryActions,
issues,
} from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { currentProcessSpaceHash } from "../services/controller-process-identity.js";
import { nextEnvironmentCleanupAt } from "../services/environment-cleanup-state.js";
import { ENVIRONMENT_LEASE_CONTROLLER_KEY } from "../services/environment-lease-controller.js";
import { hasPendingRunCleanup } from "../services/execution-blocker.js";
import { heartbeatService, type HeartbeatEnvironmentRuntime } from "../services/heartbeat.js";
import { hasRecoveredCleanupStopReceipt } from "../services/remote-execution-termination.js";
import { taskChatRetryExecutionStopped } from "../services/chat-channels/task-retry-authority.js";
import { createOwnershipReportState, runOwnershipReconciliationPass } from "../services/recovery/ownership-invariant-report.js";
import { ORPHANED_SANDBOX_RECOVERY_KEY, settleStopOnlyCleanup } from "../services/sandbox-stop-and-retain.js";
import { releasableRetainedOrphanCondition } from "../services/environment-retained-orphan-settlement.js";
const support = await getEmbeddedPostgresTestSupport();
const suite = support.supported ? describe : describe.skip;
const KIND = "environment_retained_orphan_sandbox";
async function exitedPid(): Promise<number> {
const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { stdio: "ignore" });
const pid = child.pid!;
await new Promise<void>(resolve => { child.once("exit", () => resolve()); child.kill("SIGKILL"); });
return pid;
}
suite("retained orphan sandbox on the shared ownership path", () => {
let db!: ReturnType<typeof createDb>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
let companyId = "";
let agentId = "";
let environmentId = "";
let issueId = "";
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("paperclip-retained-orphan-sandbox-");
db = createDb(temporary.connectionString);
companyId = randomUUID(); agentId = randomUUID(); environmentId = randomUUID(); issueId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Paperclip", issuePrefix: "ROS",
requireBoardApprovalForNewAgents: false });
await db.insert(agents).values({ id: agentId, companyId, name: "Coder", role: "engineer", status: "idle",
adapterType: "codex_local", adapterConfig: {}, runtimeConfig: {}, permissions: {} });
await db.insert(environments).values({ id: environmentId, name: "Fake Sandbox", driver: "sandbox",
status: "active", config: { provider: "fake", image: "ubuntu:24.04" } });
await db.insert(issues).values({ id: issueId, companyId, title: "Orphaned work" });
}, 30_000);
afterEach(async () => {
await db.delete(issueRecoveryActions);
await db.delete(environmentLeases);
await db.delete(heartbeatRuns);
await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.id, issueId));
await db.delete(issues).where(ne(issues.id, issueId));
await db.delete(executionWorkspaces);
});
afterAll(async () => {
await temporary?.cleanup();
});
async function run(status: string) {
const id = randomUUID();
await db.insert(heartbeatRuns).values({ id, companyId, agentId, issueId, invocationSource: "on_demand", status, nextEventSeq: 1 });
return id;
}
async function row(id: string) {
const [found] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, id));
return found!;
}
/** The end state of an orphaned run lease: the ownership settle stopped the
* sandbox and kept it, and the stop was confirmed. */
async function retainedOrphan(runId: string, extra: { expiresAt?: Date; metadata?: Record<string, unknown> } = {}) {
const id = randomUUID(), providerLeaseId = `sandbox://fake/${id}`, now = new Date();
await db.insert(environmentLeases).values({ id, companyId, environmentId, issueId, heartbeatRunId: runId,
status: "released", cleanupStatus: "success", leasePolicy: "ephemeral", provider: "fake", providerLeaseId,
acquiredAt: new Date(now.getTime() - 60_000), releasedAt: now, expiresAt: extra.expiresAt ?? null,
metadata: {
[ORPHANED_SANDBOX_RECOVERY_KEY]: { recoveredOrphanedRunLease: true, ownerProof: "pid_dead", runId },
remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId, runId, leaseId: id,
provider: "fake", providerLeaseId, state: "stopped", confirmedAt: now.toISOString() },
sandboxStopAndRetainReceipt: { schema: "paperclip.sandbox-stop-and-retain-receipt.v1", requestId: randomUUID(),
companyId, runId, leaseId: id, provider: "fake", providerLeaseId, builtinProvider: "fake",
method: "builtin.stopLease", confirmedAt: now.toISOString() },
...extra.metadata,
} });
return id;
}
async function pass(record = true) {
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record });
return { report, kind: report.kinds.find(entry => entry.kind === KIND)! };
}
it("marks the lease when the orphaned-run-lease settle stops and keeps its sandbox", async () => {
const runId = await run("failed");
const id = randomUUID();
await db.insert(environmentLeases).values({ id, companyId, environmentId, heartbeatRunId: runId, status: "active",
provider: "fake", providerLeaseId: `sandbox://fake/${id}`, metadata: { [ENVIRONMENT_LEASE_CONTROLLER_KEY]: {
bootId: "earlier-boot", pid: await exitedPid(), hostHash: currentProcessSpaceHash(), processStartedAt: null } } });
await pass();
expect((await row(id)).metadata?.[ORPHANED_SANDBOX_RECOVERY_KEY]).toMatchObject({
recoveredOrphanedRunLease: true, ownerProof: "pid_dead", runId });
});
it("keeps the orphan marker for a stop intent saved before the top-level marker existed", async () => {
const runId = await run("failed");
const id = randomUUID(), providerLeaseId = `sandbox://fake/${id}`, attemptId = randomUUID();
const intent = { schema: "paperclip.sandbox-stop-and-retain.v1", requestId: attemptId, companyId, runId, leaseId: id,
provider: "fake", providerLeaseId, builtinProvider: "fake", recoveredOrphanedRunLease: true, ownerProof: "pid_dead" };
await db.insert(environmentLeases).values({ id, companyId, environmentId, issueId, heartbeatRunId: runId,
status: "pending_cleanup", cleanupStatus: "failed", failureReason: "sandbox_stop_pending", leasePolicy: "ephemeral",
provider: "fake", providerLeaseId, metadata: { sandboxStopAndRetain: intent, pendingCleanupAttemptId: attemptId,
pendingCleanupInFlight: true } });
const settled = await settleStopOnlyCleanup(db, await row(id), { attemptId, receipt: { providerLeaseId, state: "stopped" } });
expect(settled).toMatchObject({ status: "released", cleanupStatus: "success" });
expect(settled?.metadata?.[ORPHANED_SANDBOX_RECOVERY_KEY]).toMatchObject({ recoveredOrphanedRunLease: true,
ownerProof: "pid_dead", runId, fromStopIntent: true });
const { kind } = await pass();
expect(kind).toMatchObject({ settled: 1 });
expect((await row(id))).toMatchObject({ status: "pending_cleanup", failureReason: "retained_orphan_sandbox_released" });
});
it("does not mark an ordinary stop-and-retain as an orphan", async () => {
const runId = await run("failed");
const id = randomUUID(), providerLeaseId = `sandbox://fake/${id}`, attemptId = randomUUID();
const intent = { schema: "paperclip.sandbox-stop-and-retain.v1", requestId: attemptId, companyId, runId, leaseId: id,
provider: "fake", providerLeaseId, builtinProvider: "fake" };
await db.insert(environmentLeases).values({ id, companyId, environmentId, issueId, heartbeatRunId: runId,
status: "pending_cleanup", cleanupStatus: "failed", leasePolicy: "ephemeral", provider: "fake", providerLeaseId,
metadata: { sandboxStopAndRetain: intent, pendingCleanupAttemptId: attemptId, pendingCleanupInFlight: true } });
const settled = await settleStopOnlyCleanup(db, await row(id), { attemptId, receipt: { providerLeaseId, state: "stopped" } });
expect(settled?.metadata?.[ORPHANED_SANDBOX_RECOVERY_KEY]).toBeUndefined();
expect((await pass()).kind).toMatchObject({ settled: 0 });
});
it("destroys a retained orphan that nothing references, through the cleanup sweep", async () => {
const leaseId = await retainedOrphan(await run("failed"));
const { kind } = await pass();
expect(kind).toMatchObject({ inProgress: 1, settled: 1 });
const released = await row(leaseId);
expect(released).toMatchObject({ status: "pending_cleanup", leasePolicy: "ephemeral",
failureReason: "retained_orphan_sandbox_released" });
expect(released.metadata?.remoteExecutionTermination).toBeUndefined();
expect(released.metadata?.workspaceRecoveryCleanup).toMatchObject({ runId: released.heartbeatRunId,
reason: "retained_orphan_sandbox_released", ownerProof: "owner_run_terminal", stopReceipt: { state: "stopped" } });
// Admission and Retry still see a stopped sandbox while the destroy waits.
expect(hasRecoveredCleanupStopReceipt(released)).toBe(true);
expect(await hasPendingRunCleanup(db, companyId, released.heartbeatRunId!)).toBe(false);
const [failedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, released.heartbeatRunId!));
expect(await taskChatRetryExecutionStopped(db, failedRun!)).toBe(true);
// Due now, and the sweep destroys it from the recorded lease.
expect(await nextEnvironmentCleanupAt(db)).toBeLessThanOrEqual(Date.now());
const teardown = vi.fn(async ({ lease }: { lease: { providerLeaseId: string | null } }) =>
({ providerLeaseId: lease.providerLeaseId, state: "destroyed" }));
await heartbeatService(db, { environmentRuntime: { retryPendingSandboxTeardown: teardown } as unknown as HeartbeatEnvironmentRuntime })
.sweepPendingCleanupLeases({ backoffMs: 5 * 60_000 });
expect(teardown).toHaveBeenCalledOnce();
expect((await row(leaseId)).status).toBe("expired");
});
it.each([
["an open recovery action on the issue", async (_runId: string) => {
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
cause: "fixture", fingerprint: randomUUID(), nextAction: "Recover the workspace." });
return {};
}],
["a queued run on the issue", async (_runId: string) => { await run("queued"); return {}; }],
["a scheduled retry on the issue", async (_runId: string) => { await run("scheduled_retry"); return {}; }],
["a stop-and-retain intent", async () => ({ metadata: { sandboxStopAndRetain: { requestId: "pending" } } })],
["a native export resume", async () => ({ metadata: { nativeWorkspaceExportResume: { requestId: "pending" } } })],
["a workspace restore recovery", async () => ({ metadata: { workspaceRestoreRecovery: { runId: "fixture" } } })],
["an in-flight cleanup claim", async () => ({ metadata: { pendingCleanupInFlight: true, pendingCleanupAttemptId: "a" } })],
["a pending legacy shutdown fence", async () => ({ metadata: { legacyShutdownWorkspaceSettlement: { state: "pending" } } })],
] as const)("never destroys a sandbox recovery still wants: %s", async (_label, arrange) => {
const runId = await run("failed");
const extra = await arrange(runId);
const leaseId = await retainedOrphan(runId, extra);
const before = await row(leaseId);
const { kind } = await pass();
expect(kind).toMatchObject({ settled: 0 });
expect(await row(leaseId)).toEqual(before);
});
it("never destroys a sandbox another live lease holds, or one whose run is still active", async () => {
const live = await retainedOrphan(await run("running"));
const sharedRun = await run("failed");
const shared = await retainedOrphan(sharedRun);
const sharedRow = await row(shared);
await db.insert(environmentLeases).values({ companyId, environmentId, heartbeatRunId: await run("running"),
status: "active", provider: "fake", providerLeaseId: sharedRow.providerLeaseId });
await pass();
expect((await row(live)).status).toBe("released");
expect((await row(shared)).status).toBe("released");
});
it("keeps a softly referenced orphan only until its retention window ends, and wakes then", async () => {
const runId = await run("failed");
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
cause: "fixture", fingerprint: randomUUID(), nextAction: "Recover the workspace." });
const expiresAt = new Date(Date.now() + 60 * 60_000);
const leaseId = await retainedOrphan(runId, { expiresAt });
const first = await pass();
expect(first.kind).toMatchObject({ inProgress: 0 });
expect(first.report.nextDueAt?.getTime()).toBeLessThanOrEqual(expiresAt.getTime());
expect((await row(leaseId)).status).toBe("released");
// Past the window no recovery path accepts the stopped sandbox as a source.
await db.update(environmentLeases).set({ expiresAt: new Date(Date.now() - 1_000) }).where(eq(environmentLeases.id, leaseId));
const second = await pass();
expect(second.kind).toMatchObject({ settled: 1 });
expect((await row(leaseId)).status).toBe("pending_cleanup");
});
it("never changes anything on a read-only pass", async () => {
const leaseId = await retainedOrphan(await run("failed"));
const before = await row(leaseId);
const { kind } = await pass(false);
expect(kind).toMatchObject({ inProgress: 1, settled: 0, verdicts: { gone: 1 } });
expect(await row(leaseId)).toEqual(before);
});
it("leaves a reusable sandbox to warm reuse", async () => {
const leaseId = await retainedOrphan(await run("failed"));
await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment" }).where(eq(environmentLeases.id, leaseId));
const before = await row(leaseId);
const { kind } = await pass();
expect(kind).toMatchObject({ inProgress: 0 });
expect(await row(leaseId)).toEqual(before);
});
it("keeps a sandbox with a pending native workspace export even past its retention window", async () => {
const runId = await run("failed");
const leaseId = await retainedOrphan(runId, { expiresAt: new Date(Date.now() - 1_000) });
const lease = await row(leaseId);
await db.update(heartbeatRuns).set({ runtimeMode: "native", runnerProfileJson: { nativeWorkspaceSync: {
schema: "paperclip.native-workspace-sync/v3", state: "prepared", descriptorSha256: "a".repeat(64),
baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId,
providerLeaseId: lease.providerLeaseId, remoteCwd: "/workspace", resourceDisposition: "destroy" } } })
.where(eq(heartbeatRuns.id, runId));
const { kind } = await pass();
// Not listed, so the settle watch does not poll it while the export waits.
expect(kind).toMatchObject({ inProgress: 0 });
expect(await row(leaseId)).toEqual(lease);
});
it("releases a workspace-less native run's orphan whose descriptor stays prepared", async () => {
const runId = await run("failed");
const leaseId = await retainedOrphan(runId);
const lease = await row(leaseId);
await db.update(heartbeatRuns).set({ runtimeMode: "native", runnerProfileJson: {
nativeExecutionInput: { workspace: { access: "none", cwd: null } },
nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: "prepared", descriptorSha256: "a".repeat(64),
baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId,
providerLeaseId: lease.providerLeaseId, remoteCwd: "/workspace", resourceDisposition: "destroy" } } })
.where(eq(heartbeatRuns.id, runId));
// Listed (no export obligation is visible to the listing); the settle
// decides with the exact export check.
const { kind } = await pass(false);
expect(kind).toMatchObject({ inProgress: 1 });
});
it("releases an orphan whose run's prepared descriptor names another sandbox", async () => {
const runId = await run("failed");
const leaseId = await retainedOrphan(runId);
await db.update(heartbeatRuns).set({ runtimeMode: "native", runnerProfileJson: {
nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: "prepared", descriptorSha256: "a".repeat(64),
baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId: randomUUID(),
providerLeaseId: "sandbox://fake/other", remoteCwd: "/workspace", resourceDisposition: "destroy" } } })
.where(eq(heartbeatRuns.id, runId));
const { kind } = await pass();
expect(kind).toMatchObject({ settled: 1 });
expect((await row(leaseId)).status).toBe("pending_cleanup");
});
it("does not hide an orphan behind a prepared descriptor the settle would reject", async () => {
const runId = await run("failed");
const leaseId = await retainedOrphan(runId);
const lease = await row(leaseId);
await db.update(heartbeatRuns).set({ runtimeMode: "native", runnerProfileJson: {
nativeWorkspaceSync: { state: "prepared", leaseId, providerLeaseId: lease.providerLeaseId } } })
.where(eq(heartbeatRuns.id, runId));
const { kind } = await pass();
expect(kind).toMatchObject({ settled: 1 });
expect((await row(leaseId)).status).toBe("pending_cleanup");
});
async function workspace() {
const [created] = await db.insert(executionWorkspaces).values({ companyId, mode: "isolated_workspace",
strategyType: "git_worktree", name: "Task workspace" }).returning({ id: executionWorkspaces.id });
return created!.id;
}
/** A warm sandbox the previous release left behind: a reuse_by_environment
* lease the plugin stopped and kept for the next turn on its workspace. This
* is the exact row shape releases before this change wrote; it carries no
* new marker. */
async function warmLease(runId: string, input: { status?: string; workspaceId: string | null; receipt?: boolean;
providerLeaseId?: string; createdAt?: Date }) {
const id = randomUUID(), providerLeaseId = input.providerLeaseId ?? `sandbox://fake/${id}`;
const now = input.createdAt ?? new Date();
await db.insert(environmentLeases).values({ id, companyId, environmentId, issueId, heartbeatRunId: runId,
executionWorkspaceId: input.workspaceId, status: input.status ?? "released", cleanupStatus: "success",
leasePolicy: "reuse_by_environment", provider: "fake", providerLeaseId, acquiredAt: now, releasedAt: now,
createdAt: now, updatedAt: now,
metadata: { agentId, reuseLease: true, ...(input.receipt === false ? {} : { remoteExecutionTermination: {
schema: "paperclip.remote-termination.v1", companyId, runId, leaseId: id, provider: "fake", providerLeaseId,
state: "stopped", confirmedAt: now.toISOString() } }) } });
return id;
}
it("destroys a task's warm sandbox once the task moves to another workspace", async () => {
const [oldWorkspace, newWorkspace] = [await workspace(), await workspace()];
const leaseId = await warmLease(await run("failed"), { workspaceId: oldWorkspace });
// A Retry kept a fresh workspace: no acquire can match the old lineage again.
await db.update(issues).set({ executionWorkspaceId: newWorkspace }).where(eq(issues.id, issueId));
const { kind } = await pass();
expect(kind).toMatchObject({ settled: 1 });
const released = await row(leaseId);
expect(released).toMatchObject({ status: "pending_cleanup", failureReason: "retained_orphan_sandbox_released" });
// The successor is never blocked on this destroy.
expect(hasRecoveredCleanupStopReceipt(released)).toBe(true);
expect(await hasPendingRunCleanup(db, companyId, released.heartbeatRunId!)).toBe(false);
});
it("destroys a warm sandbox behind a failed lease, which no turn can resume", async () => {
const current = await workspace();
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
const leaseId = await warmLease(await run("failed"), { status: "failed", workspaceId: current });
const { kind } = await pass();
expect(kind).toMatchObject({ settled: 1 });
expect((await row(leaseId)).status).toBe("pending_cleanup");
});
it.each([
["the task can still resume it on its own workspace", async () => {
const current = await workspace();
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
return [await warmLease(await run("succeeded"), { workspaceId: current })];
}],
["it has no confirmed stop receipt", async () => {
await db.update(issues).set({ executionWorkspaceId: await workspace() }).where(eq(issues.id, issueId));
return [await warmLease(await run("failed"), { status: "failed", workspaceId: await workspace(), receipt: false })];
}],
["a newer lease on the same sandbox handed it on", async () => {
const current = await workspace();
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
const providerLeaseId = `sandbox://fake/${randomUUID()}`;
const older = await warmLease(await run("failed"), { status: "failed", workspaceId: current, providerLeaseId,
createdAt: new Date(Date.now() - 60_000) });
const newer = await warmLease(await run("succeeded"), { workspaceId: current, providerLeaseId });
return [older, newer];
}],
["its task has no bound workspace (finished, or between bindings)", async () => {
await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.id, issueId));
return [await warmLease(await run("succeeded"), { workspaceId: await workspace() }),
await warmLease(await run("succeeded"), { status: "retained", workspaceId: await workspace(), receipt: false })];
}],
["another task still shares its workspace", async () => {
const [shared, fresh] = [await workspace(), await workspace()];
await db.update(issues).set({ executionWorkspaceId: fresh }).where(eq(issues.id, issueId));
await db.insert(issues).values({ companyId, title: "Sibling on the shared workspace", executionWorkspaceId: shared });
return [await warmLease(await run("failed"), { workspaceId: shared })];
}],
["the task still has a run in flight", async () => {
const [oldWorkspace, newWorkspace] = [await workspace(), await workspace()];
await db.update(issues).set({ executionWorkspaceId: newWorkspace }).where(eq(issues.id, issueId));
const leaseId = await warmLease(await run("failed"), { workspaceId: oldWorkspace });
await run("running");
return [leaseId];
}],
] as const)("keeps a warm sandbox while %s", async (_label, arrange) => {
const ids = await arrange();
const before = await Promise.all(ids.map(row));
const { kind } = await pass();
expect(kind).toMatchObject({ settled: 0 });
expect(await Promise.all(ids.map(row))).toEqual(before);
});
it("lists only one newest row when two rows for a sandbox share a created_at", async () => {
const current = await workspace();
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
const providerLeaseId = `sandbox://fake/${randomUUID()}`, createdAt = new Date();
const [lower, higher] = ["00000000-0000-4000-8000-000000000001", "ffffffff-ffff-4fff-bfff-ffffffffffff"];
for (const id of [lower, higher]) {
const runId = await run("failed");
await db.insert(environmentLeases).values({ id, companyId, environmentId, issueId, heartbeatRunId: runId,
executionWorkspaceId: current, status: "failed", cleanupStatus: "success", leasePolicy: "reuse_by_environment",
provider: "fake", providerLeaseId, acquiredAt: createdAt, releasedAt: createdAt, createdAt, updatedAt: createdAt,
metadata: { agentId, remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId, runId,
leaseId: id, provider: "fake", providerLeaseId, state: "stopped", confirmedAt: createdAt.toISOString() } } });
}
const listed = await db.select({ id: environmentLeases.id }).from(environmentLeases)
.where(releasableRetainedOrphanCondition());
expect(listed.map(row => row.id)).toEqual([higher]);
});
it("stops, then destroys, a running warm sandbox once its task moves to another workspace", async () => {
const [oldWorkspace, newWorkspace] = [await workspace(), await workspace()];
const runId = await run("succeeded");
// A keep-running release: retained, no stop, no receipt. Its idle timer
// lived in the process that kept it.
const leaseId = await warmLease(runId, { status: "retained", workspaceId: oldWorkspace, receipt: false });
await db.update(environmentLeases).set({ releasedAt: null }).where(eq(environmentLeases.id, leaseId));
await db.update(issues).set({ executionWorkspaceId: newWorkspace }).where(eq(issues.id, issueId));
expect((await pass()).kind).toMatchObject({ settled: 1 });
const stopping = await row(leaseId);
expect(stopping).toMatchObject({ status: "pending_cleanup", failureReason: "sandbox_stop_pending" });
const intent = stopping.metadata?.sandboxStopAndRetain as { requestId: string; recoveredOrphanedRunLease?: boolean };
expect(intent).toMatchObject({ recoveredOrphanedRunLease: true, supersededWarmSandbox: true });
// The cleanup sweep's first stop is due at once.
expect(await nextEnvironmentCleanupAt(db)).toBeLessThanOrEqual(Date.now());
// The provider confirms the stop; the next pass hands it to destruction.
const stopped = await settleStopOnlyCleanup(db, stopping, { attemptId: intent.requestId,
receipt: { providerLeaseId: stopping.providerLeaseId, state: "stopped" } });
expect(stopped).toMatchObject({ status: "released", cleanupStatus: "success" });
expect((await pass()).kind).toMatchObject({ settled: 1 });
expect(await row(leaseId)).toMatchObject({ status: "pending_cleanup", failureReason: "retained_orphan_sandbox_released" });
});
it("keeps a running warm sandbox its task can still resume after a restart", async () => {
const current = await workspace();
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
const leaseId = await warmLease(await run("succeeded"), { status: "retained", workspaceId: current, receipt: false });
const before = await row(leaseId);
expect((await pass()).kind).toMatchObject({ settled: 0 });
expect(await row(leaseId)).toEqual(before);
});
it("leaves an ordinary retained sandbox (no orphan marker) alone", async () => {
const leaseId = await retainedOrphan(await run("failed"));
const { [ORPHANED_SANDBOX_RECOVERY_KEY]: _marker, ...metadata } = (await row(leaseId)).metadata!;
await db.update(environmentLeases).set({ metadata }).where(eq(environmentLeases.id, leaseId));
const { kind } = await pass();
expect(kind).toMatchObject({ inProgress: 0 });
expect((await row(leaseId)).status).toBe("released");
});
});