mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
fix(recovery): own abandoned warm sandboxes on the retained-orphan path (#20065)
## Thinking Path - Paperclip runs agent tasks in remote sandboxes (Daytona). With a warm lifecycle, a sandbox outlives its turn so the next turn on the same workspace resumes it instead of provisioning and uploading again. - #20048 makes warm the default on remote sandboxes. Its merge condition: an idle warm sandbox must be an owned record, adopted by its successor or destroyed or suspended once its owner is proven gone, with zero orphans in a Daytona chaos wave. - That wave (warm environment, K1/K2/K4 faults) left sandboxes alive behind dead lease rows. Reading the wave's database shows three warm-specific shapes that nothing ever tears down: 1. **A Retry moved the task to a new workspace.** `retainedIssueWorkspaceForRetry` reuses only local leases, so a remote task always gets a fresh workspace. Reuse matches only the exact workspace, so the old sandbox is unreachable: `released` and stopped (wave t05), or still `retained` and running. 2. **A failed warm lease.** The plugin stopped the sandbox for reuse, but reuse never resumes a `failed` row (wave t12). 3. **A warm sandbox kept running (`retained`).** Its idle timer lived only in the process that kept it. If that task moves on, nothing stops the sandbox, even after a restart. - The shared ownership path already has a kind for exactly this lifecycle: `environment_retained_orphan_sandbox` destroys a stopped, retained sandbox once nothing references it, but it skipped warm leases on purpose. This PR extends that kind instead of adding a new sweep. ## Linked Issues or Issue Description Refs #20048 **What happened?** In Daytona warm wave w-20261108T142906-s9501: - t05: run 8758207a's sandbox was resumed by c99b3a2e and released stopped in workspace 35a678f0. The Retry kept a fresh workspace (8ac860c3) and provisioned a new sandbox, and the old one stayed forever. - t12: run f2bf2f05's lease on sandbox 6d4c91fb was released as `failed` with the sandbox stopped, and the next run on the same workspace provisioned a new one. Both sandboxes were still alive when the wave ended. **Steps to reproduce** 1. Use a sandbox environment with warm reuse (`reuseLease: true`). 2. Complete a turn so the warm lease is kept, retained or released and stopped. 3. Make the next run fail so the Retry keeps a fresh workspace (always the case for a remote task), or so the lease is released as `failed`. 4. The next run provisions a new sandbox; the old one is never resumed or destroyed. **Expected behavior** A warm sandbox that no turn can resume is an owned record. Once its owning run is terminal and nothing references it, the ownership reconciler stops it if it is still running, then destroys it. A sandbox its task can still resume on the same workspace is left for reuse. ## What Changed - `server/src/services/environment-retained-orphan-settlement.ts`: the listing (`retainedOrphanSandboxCondition`) now has two warm branches, each on the newest lease row for its sandbox: - `abandonedWarmSandboxCondition`: a `reuse_by_environment` lease with a confirmed `stopped` receipt for this lease and run, either `failed`, or `released` while the task's bound workspace differs from the lease's. The settle hands it to the cleanup sweep through the existing stopped-cleanup contract, so the old run's successors are never blocked on the destroy. - `supersededRunningWarmSandboxCondition`: a `retained` (still running) warm lease whose task moved workspace. The settle queues a stop-only intent. When the provider confirms the stop, the row becomes `released` with a receipt, and the first branch destroys it on a later pass. - The settle re-checks the same predicate under the row lock. Every existing hold still applies: the run still active, open runs or recovery actions on the issue, pending exports, stop or restore intents, or another live lease on the same sandbox. - `server/src/services/sandbox-stop-and-retain.ts`: new `prepareSupersededWarmSandboxStop`, which prepares the existing stop-only intent from `retained`. The intent carries the recovery marker, so the sweep dispatches its first attempt at once. - `server/src/__tests__/environment-retained-orphan-sandbox.test.ts`, new tests: - a superseded released lease is destroyed; - a failed warm lease is destroyed; - a running superseded lease is stopped, then destroyed; - keep cases: the task can still resume it (released, and retained after a restart), no stop receipt, a newer row on the same sandbox, a run in flight on the task. The positive tests fail on master and pass with this change. **Previous-release state:** the conditions read existing columns only (status, lease policy, workspace binding, stop receipt). Rows the previous release saved are judged as-is. The tests build them in exactly the shape that release writes (no new marker), so old-saved state is handled by the new code without migration. Rows without a confirmed stop are never listed. ## Verification - `cd server && npx vitest run src/__tests__/environment-retained-orphan-sandbox.test.ts`: 29/29 pass. With the two service files reverted to master, the 3 positive tests fail and the other 26 pass. - Neighbouring suites `environment-run-lease-ownership`, `heartbeat-pending-cleanup-sweep`, `ownership-reconciliation-report`, `heartbeat-orphaned-active-lease-sweep`, `environment-pending-cleanup-queue-owner` and `retained-native-session-cleanup-work`: 150/150 pass. - The wave's lease rows were read from the run's kept database copy to confirm the shapes above. - Not yet covered: the same Daytona warm wave, rerun with this fix. #20048 merges only at ≥11/12 with zero orphans. ## Risks - **Destroying a sandbox someone can still use.** Only the newest row for a sandbox is judged. `released` and `retained` rows qualify only when the task's bound workspace differs from the lease's, which is the exact rule reuse matches on. All existing holds still apply, and a stop must be provider-confirmed before any destroy. - **Unsaved work.** Pending native exports and stop or restore intents hold the sandbox. A stopped sandbox is destroyed only after its owning run is terminal and nothing on the task is still running. - **Blocking a successor.** The destroy uses the stopped-cleanup contract (`workspaceRecoveryCleanup` with the stop receipt), which `hasPendingRunCleanup` treats as resource cleanup. This is covered by a test. - **Out of scope.** A `retain_on_failure` lease kept for workspace restore (wave t12, a095e0b5) stays with restore recovery, which owns its own cleanup. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0746d1203d
commit
3448c7f9ba
3 files changed
+210
-12
No files matched your search
@@ -1,6 +1,6 @@
|
||||
import { spawn } from "node:child_process";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { eq, ne } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
agents,
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
createDb,
|
||||
environmentLeases,
|
||||
environments,
|
||||
executionWorkspaces,
|
||||
heartbeatRuns,
|
||||
issueRecoveryActions,
|
||||
issues,
|
||||
@@ -59,6 +60,9 @@ suite("retained orphan sandbox on the shared ownership path", () => {
|
||||
await db.delete(issueRecoveryActions);
|
||||
await db.delete(environmentLeases);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.id, issueId));
|
||||
await db.delete(issues).where(ne(issues.id, issueId));
|
||||
await db.delete(executionWorkspaces);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
@@ -292,6 +296,127 @@ suite("retained orphan sandbox on the shared ownership path", () => {
|
||||
expect((await row(leaseId)).status).toBe("pending_cleanup");
|
||||
});
|
||||
|
||||
async function workspace() {
|
||||
const [created] = await db.insert(executionWorkspaces).values({ companyId, mode: "isolated_workspace",
|
||||
strategyType: "git_worktree", name: "Task workspace" }).returning({ id: executionWorkspaces.id });
|
||||
return created!.id;
|
||||
}
|
||||
|
||||
/** A warm sandbox the previous release left behind: a reuse_by_environment
|
||||
* lease the plugin stopped and kept for the next turn on its workspace. This
|
||||
* is the exact row shape releases before this change wrote; it carries no
|
||||
* new marker. */
|
||||
async function warmLease(runId: string, input: { status?: string; workspaceId: string | null; receipt?: boolean;
|
||||
providerLeaseId?: string; createdAt?: Date }) {
|
||||
const id = randomUUID(), providerLeaseId = input.providerLeaseId ?? `sandbox://fake/${id}`;
|
||||
const now = input.createdAt ?? new Date();
|
||||
await db.insert(environmentLeases).values({ id, companyId, environmentId, issueId, heartbeatRunId: runId,
|
||||
executionWorkspaceId: input.workspaceId, status: input.status ?? "released", cleanupStatus: "success",
|
||||
leasePolicy: "reuse_by_environment", provider: "fake", providerLeaseId, acquiredAt: now, releasedAt: now,
|
||||
createdAt: now, updatedAt: now,
|
||||
metadata: { agentId, reuseLease: true, ...(input.receipt === false ? {} : { remoteExecutionTermination: {
|
||||
schema: "paperclip.remote-termination.v1", companyId, runId, leaseId: id, provider: "fake", providerLeaseId,
|
||||
state: "stopped", confirmedAt: now.toISOString() } }) } });
|
||||
return id;
|
||||
}
|
||||
|
||||
it("destroys a task's warm sandbox once the task moves to another workspace", async () => {
|
||||
const [oldWorkspace, newWorkspace] = [await workspace(), await workspace()];
|
||||
const leaseId = await warmLease(await run("failed"), { workspaceId: oldWorkspace });
|
||||
// A Retry kept a fresh workspace: no acquire can match the old lineage again.
|
||||
await db.update(issues).set({ executionWorkspaceId: newWorkspace }).where(eq(issues.id, issueId));
|
||||
const { kind } = await pass();
|
||||
expect(kind).toMatchObject({ settled: 1 });
|
||||
const released = await row(leaseId);
|
||||
expect(released).toMatchObject({ status: "pending_cleanup", failureReason: "retained_orphan_sandbox_released" });
|
||||
// The successor is never blocked on this destroy.
|
||||
expect(hasRecoveredCleanupStopReceipt(released)).toBe(true);
|
||||
expect(await hasPendingRunCleanup(db, companyId, released.heartbeatRunId!)).toBe(false);
|
||||
});
|
||||
|
||||
it("destroys a warm sandbox behind a failed lease, which no turn can resume", async () => {
|
||||
const current = await workspace();
|
||||
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
|
||||
const leaseId = await warmLease(await run("failed"), { status: "failed", workspaceId: current });
|
||||
const { kind } = await pass();
|
||||
expect(kind).toMatchObject({ settled: 1 });
|
||||
expect((await row(leaseId)).status).toBe("pending_cleanup");
|
||||
});
|
||||
|
||||
it.each([
|
||||
["the task can still resume it on its own workspace", async () => {
|
||||
const current = await workspace();
|
||||
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
|
||||
return [await warmLease(await run("succeeded"), { workspaceId: current })];
|
||||
}],
|
||||
["it has no confirmed stop receipt", async () => {
|
||||
await db.update(issues).set({ executionWorkspaceId: await workspace() }).where(eq(issues.id, issueId));
|
||||
return [await warmLease(await run("failed"), { status: "failed", workspaceId: await workspace(), receipt: false })];
|
||||
}],
|
||||
["a newer lease on the same sandbox handed it on", async () => {
|
||||
const current = await workspace();
|
||||
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
|
||||
const providerLeaseId = `sandbox://fake/${randomUUID()}`;
|
||||
const older = await warmLease(await run("failed"), { status: "failed", workspaceId: current, providerLeaseId,
|
||||
createdAt: new Date(Date.now() - 60_000) });
|
||||
const newer = await warmLease(await run("succeeded"), { workspaceId: current, providerLeaseId });
|
||||
return [older, newer];
|
||||
}],
|
||||
["another task still shares its workspace", async () => {
|
||||
const [shared, fresh] = [await workspace(), await workspace()];
|
||||
await db.update(issues).set({ executionWorkspaceId: fresh }).where(eq(issues.id, issueId));
|
||||
await db.insert(issues).values({ companyId, title: "Sibling on the shared workspace", executionWorkspaceId: shared });
|
||||
return [await warmLease(await run("failed"), { workspaceId: shared })];
|
||||
}],
|
||||
["the task still has a run in flight", async () => {
|
||||
const [oldWorkspace, newWorkspace] = [await workspace(), await workspace()];
|
||||
await db.update(issues).set({ executionWorkspaceId: newWorkspace }).where(eq(issues.id, issueId));
|
||||
const leaseId = await warmLease(await run("failed"), { workspaceId: oldWorkspace });
|
||||
await run("running");
|
||||
return [leaseId];
|
||||
}],
|
||||
] as const)("keeps a warm sandbox while %s", async (_label, arrange) => {
|
||||
const ids = await arrange();
|
||||
const before = await Promise.all(ids.map(row));
|
||||
const { kind } = await pass();
|
||||
expect(kind).toMatchObject({ settled: 0 });
|
||||
expect(await Promise.all(ids.map(row))).toEqual(before);
|
||||
});
|
||||
|
||||
it("stops, then destroys, a running warm sandbox once its task moves to another workspace", async () => {
|
||||
const [oldWorkspace, newWorkspace] = [await workspace(), await workspace()];
|
||||
const runId = await run("succeeded");
|
||||
// A keep-running release: retained, no stop, no receipt. Its idle timer
|
||||
// lived in the process that kept it.
|
||||
const leaseId = await warmLease(runId, { status: "retained", workspaceId: oldWorkspace, receipt: false });
|
||||
await db.update(environmentLeases).set({ releasedAt: null }).where(eq(environmentLeases.id, leaseId));
|
||||
await db.update(issues).set({ executionWorkspaceId: newWorkspace }).where(eq(issues.id, issueId));
|
||||
|
||||
expect((await pass()).kind).toMatchObject({ settled: 1 });
|
||||
const stopping = await row(leaseId);
|
||||
expect(stopping).toMatchObject({ status: "pending_cleanup", failureReason: "sandbox_stop_pending" });
|
||||
const intent = stopping.metadata?.sandboxStopAndRetain as { requestId: string; recoveredOrphanedRunLease?: boolean };
|
||||
expect(intent).toMatchObject({ recoveredOrphanedRunLease: true, supersededWarmSandbox: true });
|
||||
// The cleanup sweep's first stop is due at once.
|
||||
expect(await nextEnvironmentCleanupAt(db)).toBeLessThanOrEqual(Date.now());
|
||||
|
||||
// The provider confirms the stop; the next pass hands it to destruction.
|
||||
const stopped = await settleStopOnlyCleanup(db, stopping, { attemptId: intent.requestId,
|
||||
receipt: { providerLeaseId: stopping.providerLeaseId, state: "stopped" } });
|
||||
expect(stopped).toMatchObject({ status: "released", cleanupStatus: "success" });
|
||||
expect((await pass()).kind).toMatchObject({ settled: 1 });
|
||||
expect(await row(leaseId)).toMatchObject({ status: "pending_cleanup", failureReason: "retained_orphan_sandbox_released" });
|
||||
});
|
||||
|
||||
it("keeps a running warm sandbox its task can still resume after a restart", async () => {
|
||||
const current = await workspace();
|
||||
await db.update(issues).set({ executionWorkspaceId: current }).where(eq(issues.id, issueId));
|
||||
const leaseId = await warmLease(await run("succeeded"), { status: "retained", workspaceId: current, receipt: false });
|
||||
const before = await row(leaseId);
|
||||
expect((await pass()).kind).toMatchObject({ settled: 0 });
|
||||
expect(await row(leaseId)).toEqual(before);
|
||||
});
|
||||
|
||||
it("leaves an ordinary retained sandbox (no orphan marker) alone", async () => {
|
||||
const leaseId = await retainedOrphan(await run("failed"));
|
||||
const { [ORPHANED_SANDBOX_RECOVERY_KEY]: _marker, ...metadata } = (await row(leaseId)).metadata!;
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
import { and, eq, inArray, ne, notInArray, sql } from "drizzle-orm";
|
||||
import { and, eq, inArray, ne, notInArray, or, sql } from "drizzle-orm";
|
||||
import { environmentLeases, heartbeatRuns, issueRecoveryActions, type Db } from "@paperclipai/db";
|
||||
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { nativeWorkspaceExportPendingForLease } from "./native-runtime/native-workspace-finalization-state.js";
|
||||
import type { OwnerProof } from "./owner-proof.js";
|
||||
import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
|
||||
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import { ORPHANED_SANDBOX_RECOVERY_KEY } from "./sandbox-stop-and-retain.js";
|
||||
import { ORPHANED_SANDBOX_RECOVERY_KEY, prepareSupersededWarmSandboxStop } from "./sandbox-stop-and-retain.js";
|
||||
|
||||
/** Failure reason on a retained orphan sandbox released for destruction. */
|
||||
export const RETAINED_ORPHAN_RELEASED_REASON = "retained_orphan_sandbox_released";
|
||||
@@ -17,8 +17,64 @@ const OPEN_RUN_STATUSES = ["queued", "scheduled_retry", "running"];
|
||||
type LeaseRow = typeof environmentLeases.$inferSelect;
|
||||
|
||||
/** A retained orphan: the ownership settle stopped and kept its sandbox, and
|
||||
* the stop is confirmed. */
|
||||
* the stop is confirmed. Or an abandoned warm sandbox (see below). */
|
||||
export function retainedOrphanSandboxCondition() {
|
||||
return or(orphanedRunSandboxCondition(), abandonedWarmSandboxCondition(), supersededRunningWarmSandboxCondition());
|
||||
}
|
||||
|
||||
/** The newest lease row for its sandbox: an older row was handed on. */
|
||||
function newestRowForSandbox() {
|
||||
return sql`not exists (select 1 from environment_leases newer where newer.id <> ${environmentLeases.id}
|
||||
and newer.provider = ${environmentLeases.provider} and newer.provider_lease_id = ${environmentLeases.providerLeaseId}
|
||||
and newer.created_at > ${environmentLeases.createdAt})`;
|
||||
}
|
||||
|
||||
/** No task is bound to this lease's workspace any more (its own task moved to
|
||||
* a fresh one, and no other task shares it), so reuse, which matches the exact
|
||||
* workspace and agent, can never pick the sandbox again. */
|
||||
function taskMovedToAnotherWorkspace() {
|
||||
return and(sql`${environmentLeases.executionWorkspaceId} is not null`,
|
||||
sql`not exists (select 1 from issues bound where bound.company_id = ${environmentLeases.companyId}
|
||||
and bound.execution_workspace_id = ${environmentLeases.executionWorkspaceId})`);
|
||||
}
|
||||
|
||||
/**
|
||||
* A warm sandbox kept running for reuse (`retained` after a keep-running
|
||||
* release) whose task moved to another workspace. Its idle timer lived in the
|
||||
* process that kept it, so after a restart nothing else would ever stop it.
|
||||
* The settle stops it through the stop-only contract; once the provider
|
||||
* confirms the stop, it is an abandoned warm sandbox and is destroyed.
|
||||
*/
|
||||
export function supersededRunningWarmSandboxCondition() {
|
||||
return and(eq(environmentLeases.status, "retained"), eq(environmentLeases.leasePolicy, "reuse_by_environment"),
|
||||
eq(environmentLeases.cleanupStatus, "success"), sql`${environmentLeases.heartbeatRunId} is not null`,
|
||||
sql`${environmentLeases.provider} is not null`, sql`${environmentLeases.providerLeaseId} is not null`,
|
||||
newestRowForSandbox(), taskMovedToAnotherWorkspace());
|
||||
}
|
||||
|
||||
/**
|
||||
* A warm sandbox no turn can resume: a reuse_by_environment lease whose
|
||||
* sandbox the provider confirmed stopped and kept, on the newest row for that
|
||||
* sandbox, where either
|
||||
* - the row is `failed`, which reuse never resumes; or
|
||||
* - the row is `released` (stopped, kept for reuse) but bound to a workspace the
|
||||
* task no longer runs in: a Retry on a remote task keeps a fresh workspace,
|
||||
* and reuse matches only the same workspace.
|
||||
* Releases before this change wrote exactly these rows, so the condition reads
|
||||
* existing columns only. A row without a confirmed stop is never listed.
|
||||
*/
|
||||
export function abandonedWarmSandboxCondition() {
|
||||
return and(eq(environmentLeases.leasePolicy, "reuse_by_environment"), eq(environmentLeases.cleanupStatus, "success"),
|
||||
sql`${environmentLeases.heartbeatRunId} is not null`, sql`${environmentLeases.provider} is not null`,
|
||||
sql`${environmentLeases.providerLeaseId} is not null`,
|
||||
sql`${environmentLeases.metadata}#>>'{remoteExecutionTermination,state}' = 'stopped'`,
|
||||
sql`${environmentLeases.metadata}#>>'{remoteExecutionTermination,leaseId}' = ${environmentLeases.id}::text`,
|
||||
sql`${environmentLeases.metadata}#>>'{remoteExecutionTermination,runId}' = ${environmentLeases.heartbeatRunId}::text`,
|
||||
newestRowForSandbox(),
|
||||
or(eq(environmentLeases.status, "failed"), and(eq(environmentLeases.status, "released"), taskMovedToAnotherWorkspace())));
|
||||
}
|
||||
|
||||
function orphanedRunSandboxCondition() {
|
||||
return and(eq(environmentLeases.status, "released"), eq(environmentLeases.cleanupStatus, "success"),
|
||||
// An ephemeral lease only: a reusable lease stays for warm reuse, and a
|
||||
// retain-on-failure lease belongs to workspace restore recovery.
|
||||
@@ -127,15 +183,23 @@ export async function settleRetainedOrphanSandbox(db: Db, record: InProgressReco
|
||||
const runId = record.runId;
|
||||
return db.transaction(async transaction => {
|
||||
const tx = transaction as unknown as Db;
|
||||
// Re-judge the row under its lock with the same listing predicate.
|
||||
const [lease] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.id, record.recordId),
|
||||
eq(environmentLeases.heartbeatRunId, runId), eq(environmentLeases.status, "released"),
|
||||
eq(environmentLeases.cleanupStatus, "success"))).for("update");
|
||||
if (!lease || !hasRemoteTerminationReceipt(lease) || lease.leasePolicy !== "ephemeral" || !lease.metadata?.[ORPHANED_SANDBOX_RECOVERY_KEY]
|
||||
|| !lease.provider || !lease.providerLeaseId) {
|
||||
return { settled: false, reason: "no_longer_a_retained_orphan" };
|
||||
}
|
||||
eq(environmentLeases.heartbeatRunId, runId), retainedOrphanSandboxCondition())).for("update");
|
||||
if (!lease || !lease.provider || !lease.providerLeaseId) return { settled: false, reason: "no_longer_a_retained_orphan" };
|
||||
const held = await heldBy(tx, lease, runId, context.now);
|
||||
if (held) return { settled: false, reason: held };
|
||||
if (lease.status === "retained") {
|
||||
// Still running: stop it first. The confirmed stop makes it destroyable.
|
||||
const stopping = await prepareSupersededWarmSandboxStop(tx, lease,
|
||||
{ recoveredOrphanedRunLease: true, supersededWarmSandbox: true, ownerProof: context.proof.reason });
|
||||
return stopping ? { settled: true, reason: context.proof.reason } : { settled: false, reason: "lease_changed_since_proof" };
|
||||
}
|
||||
if (!hasRemoteTerminationReceipt(lease)
|
||||
|| (lease.leasePolicy === "ephemeral" ? !lease.metadata?.[ORPHANED_SANDBOX_RECOVERY_KEY] : lease.leasePolicy !== "reuse_by_environment")) {
|
||||
return { settled: false, reason: "no_longer_a_retained_orphan" };
|
||||
}
|
||||
const status = lease.status;
|
||||
const now = context.now;
|
||||
// The destroy is queued through the recognized stopped-cleanup contract
|
||||
// (`workspaceRecoveryCleanup` with the verified stop receipt), so admission
|
||||
@@ -149,7 +213,7 @@ export async function settleRetainedOrphanSandbox(db: Db, record: InProgressReco
|
||||
pendingCleanupInFlight: false, pendingCleanupRetryAfterMs: 0, pendingCleanupRetryAttempts: 0,
|
||||
pendingCleanupLeaseExpiresAtMs: 0 })}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.heartbeatRunId, runId),
|
||||
eq(environmentLeases.status, "released"), eq(environmentLeases.cleanupStatus, "success"),
|
||||
eq(environmentLeases.status, status), eq(environmentLeases.cleanupStatus, "success"),
|
||||
sql`${environmentLeases.metadata} = ${JSON.stringify(lease.metadata)}::jsonb`))
|
||||
.returning({ id: environmentLeases.id }), { companyId: lease.companyId, runId, leaseId: lease.id });
|
||||
return released.length ? { settled: true, reason: context.proof.reason } : { settled: false, reason: "lease_changed_since_proof" };
|
||||
|
||||
@@ -152,7 +152,16 @@ function stopOnlyAttempt(lease: Lease, attemptId: string) {
|
||||
sql`${environmentLeases.metadata}->>'pluginId' is not distinct from ${readStopOnlyCleanup(lease)?.pluginId ?? null}`);
|
||||
}
|
||||
|
||||
async function prepareStopAndRetain(db: Db, lease: Lease, fromStatus: "active" | "released", recovery: Record<string, unknown> = {}) {
|
||||
/** A warm sandbox kept running for reuse whose task moved to another workspace:
|
||||
* stop it through the stop-only contract so the provider confirms the stop
|
||||
* before anything destroys it. The ownership settle verifies the move. */
|
||||
export async function prepareSupersededWarmSandboxStop(db: Db, lease: typeof environmentLeases.$inferSelect, recovery: Record<string, unknown>) {
|
||||
if (lease.status !== "retained" || lease.leasePolicy !== "reuse_by_environment" || !lease.heartbeatRunId
|
||||
|| !lease.providerLeaseId || lease.metadata?.pendingCleanupInFlight === true) return null;
|
||||
return prepareStopAndRetain(db, lease, "retained", recovery);
|
||||
}
|
||||
|
||||
async function prepareStopAndRetain(db: Db, lease: Lease, fromStatus: "active" | "released" | "retained", recovery: Record<string, unknown> = {}) {
|
||||
const requestId = randomUUID(), now = new Date();
|
||||
const intent = { schema: "paperclip.sandbox-stop-and-retain.v1", requestId,
|
||||
companyId: lease.companyId, runId: lease.heartbeatRunId, leaseId: lease.id,
|
||||
|
||||
Reference in new issue
Block a user