mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 12:04:41 +01:00
feat: environment delete with agent reassignment and consented sandbox destroy (#12053)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environments define where agent runs execute: local, SSH, or provider sandboxes > - Operators can create and edit environments, but the UI has no way to delete one > - The server already exposes `DELETE /environments/:id` and a delete-blast-radius preflight, but no UI consumes them, and a delete blocked by reusable sandbox leases gives the operator no path forward > - This pull request adds the delete flow to the environment configuration page: a preflight-driven modal that reassigns dependent agents, names the workspaces that hold blocking sandbox leases, and can destroy those sandboxes with explicit consent > - The benefit is that operators can retire stale environments from the UI without database surgery, and dependent agents move to a chosen replacement instead of silently falling back ## Linked Issues or Issue Description Refs #8554 Refs #11124 **Subsystem affected** Environments (server routes, environment runtime service, and the environment settings UI). **Problem or motivation** The environment configuration page has no delete control. The server delete endpoint exists, but nothing in the UI calls it. When reusable sandbox leases block a delete, the 409 error names no owner, so the operator cannot find the blocking workspace. Agents that use the environment as their default lose it silently through the FK `on delete set null`. **Proposed solution** Add a delete button with a confirmation modal on the environment edit page. The modal reads the delete-blast-radius preflight. It offers a dropdown to reassign dependent agents to another environment before the delete. It lists each workspace that holds a blocking reusable sandbox lease, with a link. When those leases are the only blocker, the confirm button destroys the sandboxes inline (`?destroyReusableSandboxLeases=true`) and then deletes. A failed teardown falls back to `pending_cleanup` for the sweep, so no sandbox is orphaned. ## What Changed - `ui/src/pages/CompanyEnvironments.tsx`: delete button on the edit page header, confirmation modal with agent reassignment select, lease-holder list, impact notes, and a consent-labeled destroy-and-delete action - `ui/src/api/environments.ts`: `deleteBlastRadius` and `remove` client methods; `remove` takes an optional `destroyReusableSandboxLeases` flag - `server/src/routes/environments.ts`: `DELETE /environments/:id` accepts `?destroyReusableSandboxLeases=true`; it destroys the environment's reusable sandbox leases first, but only when those leases are the sole delete blocker, then re-checks the blast radius before it deletes - `server/src/services/environment-runtime.ts`: new `destroyReusableSandboxLeasesForEnvironment` — destroys every reusable sandbox lease an environment still owns while the environment config (provider credentials) is still available - `server/src/services/environments.ts`: the delete blast radius now returns `reusableSandboxLeaseHolders` (lease id, workspace, issue) so clients can name what blocks a delete - `packages/shared/src/types/environment.ts`: `EnvironmentDeleteReusableLeaseHolder` type on the blast radius - Tests: route gating for the consent flag (destroy runs, mixed-blocker rejection, surviving-lease rejection), runtime destroy scoped to an environment, blast-radius holder join, and UI tests for the reassignment flow, holder links, and the consent button ## Verification - `npx vitest run server/src/__tests__/environment-routes.test.ts server/src/__tests__/environment-service.test.ts server/src/__tests__/environment-runtime.test.ts ui/src/pages/CompanyEnvironments.test.tsx` - Manual: open Settings → Environments → edit an environment. The trash icon opens the modal. With agents on the environment, pick a reassignment target and confirm; agents move and the environment deletes. With reusable sandbox leases, the modal names the holding workspaces and the confirm button reads "Destroy N sandboxes and delete". ## Risks - The consented path destroys provider sandboxes. It runs only when reusable leases are the sole blocker, so a delete that would still be rejected never destroys anything. A failed teardown routes to `pending_cleanup` and the delete stays blocked until the sweep resolves it. - Agent reassignment issues one PATCH per agent from the client. A mid-sequence failure leaves some agents reassigned; the reassignments are valid on their own and the UI refreshes to the actual state. - Hard blockers (managed local, instance default, pending cleanup) keep the existing 409 behavior and disable the confirm button. ## Model Used - Claude (Anthropic) — Fable 5, model id `claude-fable-5`, extended thinking, agentic tool use via Claude Code CLI. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Devin Foley authored and GitHub committed
1 parent
627eef7cbd
commit
c62bb4b16b
12 files changed
+1107
-9
No files matched your search
@@ -26,6 +26,9 @@ const mockProjectService = vi.hoisted(() => ({
|
||||
clearExecutionWorkspaceEnvironmentSelection: vi.fn(),
|
||||
}));
|
||||
|
||||
const mockEnvironmentRuntimeService = vi.hoisted(() => ({
|
||||
destroyReusableSandboxLeasesForEnvironment: vi.fn(async () => ({ destroyed: 0, failed: 0, skippedLiveRun: 0 })),
|
||||
}));
|
||||
const mockInstanceSettingsService = vi.hoisted(() => ({
|
||||
listCompanyIds: vi.fn(),
|
||||
getGeneral: vi.fn(),
|
||||
@@ -105,6 +108,10 @@ vi.mock("../services/environments.js", () => ({
|
||||
environmentService: () => mockEnvironmentService,
|
||||
}));
|
||||
|
||||
vi.mock("../services/environment-runtime.js", () => ({
|
||||
environmentRuntimeService: () => mockEnvironmentRuntimeService,
|
||||
}));
|
||||
|
||||
vi.mock("../services/execution-workspaces.js", () => ({
|
||||
executionWorkspaceService: () => mockExecutionWorkspaceService,
|
||||
}));
|
||||
@@ -153,6 +160,14 @@ function createDeleteBlastRadius(overrides: Partial<{
|
||||
activeCustomImageSetupSessionCount: number;
|
||||
pendingCleanupLeaseCount: number;
|
||||
reusableSandboxLeaseCount: number;
|
||||
reusableSandboxLeaseHolders: Array<{
|
||||
leaseId: string;
|
||||
executionWorkspaceId: string | null;
|
||||
executionWorkspaceName: string | null;
|
||||
issueId: string | null;
|
||||
issueIdentifier: string | null;
|
||||
issueTitle: string | null;
|
||||
}>;
|
||||
}> = {}) {
|
||||
const staticReferences = {
|
||||
isManagedLocal: overrides.isManagedLocal ?? false,
|
||||
@@ -172,6 +187,16 @@ function createDeleteBlastRadius(overrides: Partial<{
|
||||
};
|
||||
const pendingCleanupLeaseCount = overrides.pendingCleanupLeaseCount ?? 0;
|
||||
const reusableSandboxLeaseCount = overrides.reusableSandboxLeaseCount ?? 0;
|
||||
const reusableSandboxLeaseHolders =
|
||||
overrides.reusableSandboxLeaseHolders
|
||||
?? Array.from({ length: reusableSandboxLeaseCount }, (_, index) => ({
|
||||
leaseId: `lease-${index + 1}`,
|
||||
executionWorkspaceId: null,
|
||||
executionWorkspaceName: null,
|
||||
issueId: null,
|
||||
issueIdentifier: null,
|
||||
issueTitle: null,
|
||||
}));
|
||||
const deleteBlockedReasons = [
|
||||
...(staticReferences.isManagedLocal ? ["managed_local" as const] : []),
|
||||
...(staticReferences.isInstanceDefault ? ["instance_default" as const] : []),
|
||||
@@ -184,6 +209,7 @@ function createDeleteBlastRadius(overrides: Partial<{
|
||||
deleteBlockedReasons,
|
||||
pendingCleanupLeaseCount,
|
||||
reusableSandboxLeaseCount,
|
||||
reusableSandboxLeaseHolders,
|
||||
staticReferences,
|
||||
activeRuntimeUse,
|
||||
};
|
||||
@@ -252,6 +278,8 @@ describe("environment routes", () => {
|
||||
mockIssueService.clearExecutionWorkspaceEnvironmentSelection.mockReset();
|
||||
mockProjectService.getById.mockReset();
|
||||
mockProjectService.clearExecutionWorkspaceEnvironmentSelection.mockReset();
|
||||
mockEnvironmentRuntimeService.destroyReusableSandboxLeasesForEnvironment.mockReset();
|
||||
mockEnvironmentRuntimeService.destroyReusableSandboxLeasesForEnvironment.mockResolvedValue({ destroyed: 0, failed: 0, skippedLiveRun: 0 });
|
||||
mockInstanceSettingsService.listCompanyIds.mockReset();
|
||||
mockInstanceSettingsService.getGeneral.mockReset();
|
||||
mockInstanceSettingsService.getGeneral.mockResolvedValue({ executionMode: "any" });
|
||||
@@ -1091,6 +1119,7 @@ describe("environment routes", () => {
|
||||
deleteBlockedReasons: [],
|
||||
pendingCleanupLeaseCount: 0,
|
||||
reusableSandboxLeaseCount: 0,
|
||||
reusableSandboxLeaseHolders: [],
|
||||
staticReferences: {
|
||||
isManagedLocal: false,
|
||||
isInstanceDefault: false,
|
||||
@@ -1601,6 +1630,89 @@ describe("environment routes", () => {
|
||||
);
|
||||
expect(res.body.details).toEqual({ deleteBlockedReasons: ["reusable_sandbox_lease"] });
|
||||
expect(mockEnvironmentService.removeIfDeletable).not.toHaveBeenCalled();
|
||||
expect(mockEnvironmentRuntimeService.destroyReusableSandboxLeasesForEnvironment).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("destroys reusable sandbox leases and deletes with explicit consent", async () => {
|
||||
const environment = createEnvironment();
|
||||
mockEnvironmentService.getById.mockResolvedValue(environment);
|
||||
mockEnvironmentService.getDeleteBlastRadius
|
||||
.mockResolvedValueOnce(createDeleteBlastRadius({ reusableSandboxLeaseCount: 2 }))
|
||||
.mockResolvedValueOnce(createDeleteBlastRadius());
|
||||
mockEnvironmentRuntimeService.destroyReusableSandboxLeasesForEnvironment.mockResolvedValue({
|
||||
destroyed: 2,
|
||||
failed: 0,
|
||||
skippedLiveRun: 0,
|
||||
});
|
||||
mockEnvironmentService.removeIfDeletable.mockResolvedValue(environment);
|
||||
mockInstanceSettingsService.listCompanyIds.mockResolvedValue(["company-1"]);
|
||||
const app = createApp({
|
||||
type: "board",
|
||||
userId: "admin-1",
|
||||
source: "local_implicit",
|
||||
});
|
||||
|
||||
const res = await request(app).delete("/api/environments/env-1?destroyReusableSandboxLeases=true");
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(mockEnvironmentRuntimeService.destroyReusableSandboxLeasesForEnvironment)
|
||||
.toHaveBeenCalledExactlyOnceWith({
|
||||
environmentId: "env-1",
|
||||
failureReason: "environment_deleted",
|
||||
});
|
||||
expect(mockEnvironmentService.removeIfDeletable).toHaveBeenCalledWith("env-1");
|
||||
expect(res.body.destroyedReusableSandboxLeaseCount).toBe(2);
|
||||
});
|
||||
|
||||
it("keeps rejecting a consented delete when a non-lease blocker remains", async () => {
|
||||
const environment = createEnvironment();
|
||||
mockEnvironmentService.getById.mockResolvedValue(environment);
|
||||
mockEnvironmentService.getDeleteBlastRadius.mockResolvedValue(createDeleteBlastRadius({
|
||||
isInstanceDefault: true,
|
||||
reusableSandboxLeaseCount: 1,
|
||||
}));
|
||||
const app = createApp({
|
||||
type: "board",
|
||||
userId: "admin-1",
|
||||
source: "local_implicit",
|
||||
});
|
||||
|
||||
const res = await request(app).delete("/api/environments/env-1?destroyReusableSandboxLeases=true");
|
||||
|
||||
// Destroying provider sandboxes and then rejecting on the other gate would
|
||||
// be an irreversible action with nothing gained, so the destroy must not run.
|
||||
expect(res.status).toBe(409);
|
||||
expect(mockEnvironmentRuntimeService.destroyReusableSandboxLeasesForEnvironment).not.toHaveBeenCalled();
|
||||
expect(mockEnvironmentService.removeIfDeletable).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("keeps rejecting when leases survive the consented destroy", async () => {
|
||||
const environment = createEnvironment();
|
||||
mockEnvironmentService.getById.mockResolvedValue(environment);
|
||||
mockEnvironmentService.getDeleteBlastRadius
|
||||
.mockResolvedValueOnce(createDeleteBlastRadius({ reusableSandboxLeaseCount: 1 }))
|
||||
.mockResolvedValueOnce(createDeleteBlastRadius({ pendingCleanupLeaseCount: 1 }));
|
||||
mockEnvironmentRuntimeService.destroyReusableSandboxLeasesForEnvironment.mockResolvedValue({
|
||||
destroyed: 1,
|
||||
failed: 1,
|
||||
skippedLiveRun: 0,
|
||||
});
|
||||
const app = createApp({
|
||||
type: "board",
|
||||
userId: "admin-1",
|
||||
source: "local_implicit",
|
||||
});
|
||||
|
||||
const res = await request(app).delete("/api/environments/env-1?destroyReusableSandboxLeases=true");
|
||||
|
||||
expect(res.status).toBe(409);
|
||||
// The rejection names what the consented destroy already did: provider
|
||||
// destruction is not transactional with the delete guard.
|
||||
expect(res.body.details).toEqual({
|
||||
deleteBlockedReasons: ["pending_sandbox_cleanup"],
|
||||
destroyedReusableSandboxLeaseCount: 1,
|
||||
});
|
||||
expect(mockEnvironmentService.removeIfDeletable).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("rejects a driver or provider config change while a sandbox cleanup is pending", async () => {
|
||||
|
||||
@@ -5639,6 +5639,70 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("destroys reusable plugin-backed sandbox leases scoped to an environment", async () => {
|
||||
const { pluginId, runId, reusableLease } = await seedReusablePluginSandboxLease();
|
||||
// The holding run is finished, so the reservation is stale and destroyable.
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, runId));
|
||||
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === pluginId),
|
||||
call: vi.fn(async (_pluginId: string, method: string) => {
|
||||
if (method === "environmentDestroyLease") {
|
||||
return undefined;
|
||||
}
|
||||
throw new Error(`Unexpected plugin method: ${method}`);
|
||||
}),
|
||||
getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })),
|
||||
} as unknown as PluginWorkerManager;
|
||||
const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager });
|
||||
|
||||
const result = await runtimeWithPlugin.destroyReusableSandboxLeasesForEnvironment({
|
||||
environmentId: reusableLease.environmentId!,
|
||||
failureReason: "environment_deleted",
|
||||
});
|
||||
|
||||
expect(result).toEqual({ destroyed: 1, failed: 0, skippedLiveRun: 0 });
|
||||
expect(workerManager.call).toHaveBeenCalledWith(
|
||||
pluginId,
|
||||
"environmentDestroyLease",
|
||||
expect.objectContaining({
|
||||
driverKey: "fake-plugin",
|
||||
providerLeaseId: "reusable-plugin-lease",
|
||||
}),
|
||||
31234,
|
||||
);
|
||||
await expect(environmentService(db).getLeaseById(reusableLease.id)).resolves.toMatchObject({
|
||||
status: "expired",
|
||||
failureReason: "environment_deleted",
|
||||
cleanupStatus: "success",
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps a reusable lease held by an in-flight run out of the environment-scoped destroy", async () => {
|
||||
const { pluginId, reusableLease } = await seedReusablePluginSandboxLease();
|
||||
// seedEnvironment leaves the holding run in `running` status.
|
||||
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === pluginId),
|
||||
call: vi.fn(async () => undefined),
|
||||
getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })),
|
||||
} as unknown as PluginWorkerManager;
|
||||
const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager });
|
||||
|
||||
const result = await runtimeWithPlugin.destroyReusableSandboxLeasesForEnvironment({
|
||||
environmentId: reusableLease.environmentId!,
|
||||
failureReason: "environment_deleted",
|
||||
});
|
||||
|
||||
expect(result).toEqual({ destroyed: 0, failed: 0, skippedLiveRun: 1 });
|
||||
expect(workerManager.call).not.toHaveBeenCalled();
|
||||
// The lease keeps its reusable status, so the delete guard still blocks.
|
||||
await expect(environmentService(db).getLeaseById(reusableLease.id)).resolves.toMatchObject({
|
||||
status: "active",
|
||||
leasePolicy: "reuse_by_environment",
|
||||
});
|
||||
});
|
||||
|
||||
it("sweeps reusable cleanup with the configuration recorded on the lease", async () => {
|
||||
const { pluginId, environment, reusableLease } = await seedReusablePluginSandboxLease();
|
||||
const environmentsSvc = environmentService(db);
|
||||
|
||||
@@ -399,6 +399,7 @@ describeEmbeddedPostgres("environmentService leases", () => {
|
||||
deleteBlockedReasons: ["instance_default"],
|
||||
pendingCleanupLeaseCount: 0,
|
||||
reusableSandboxLeaseCount: 0,
|
||||
reusableSandboxLeaseHolders: [],
|
||||
staticReferences: {
|
||||
isManagedLocal: false,
|
||||
isInstanceDefault: true,
|
||||
@@ -559,9 +560,47 @@ describeEmbeddedPostgres("environmentService leases", () => {
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
});
|
||||
const projectId = randomUUID();
|
||||
const issueId = randomUUID();
|
||||
const workspaceId = randomUUID();
|
||||
await db.insert(projects).values({
|
||||
id: projectId,
|
||||
companyId,
|
||||
name: "Project",
|
||||
status: "in_progress",
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
});
|
||||
await db.insert(issues).values({
|
||||
id: issueId,
|
||||
companyId,
|
||||
projectId,
|
||||
identifier: "ACME-7",
|
||||
title: "Reusable lease holder",
|
||||
status: "todo",
|
||||
priority: "medium",
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
});
|
||||
await db.insert(executionWorkspaces).values({
|
||||
id: workspaceId,
|
||||
companyId,
|
||||
projectId,
|
||||
sourceIssueId: issueId,
|
||||
mode: "isolated_workspace",
|
||||
strategyType: "git_worktree",
|
||||
name: "ACME-7-reusable-lease-holder",
|
||||
status: "active",
|
||||
providerType: "git_worktree",
|
||||
metadata: {},
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
});
|
||||
const lease = await svc.acquireLease({
|
||||
companyId,
|
||||
environmentId,
|
||||
executionWorkspaceId: workspaceId,
|
||||
issueId,
|
||||
leasePolicy: "reuse_by_environment",
|
||||
provider: "fake",
|
||||
providerLeaseId: "sandbox-reusable-1",
|
||||
@@ -577,6 +616,16 @@ describeEmbeddedPostgres("environmentService leases", () => {
|
||||
expect(activeImpact?.canDelete).toBe(false);
|
||||
expect(activeImpact?.deleteBlockedReasons).toContain("reusable_sandbox_lease");
|
||||
expect(activeImpact?.reusableSandboxLeaseCount).toBe(1);
|
||||
expect(activeImpact?.reusableSandboxLeaseHolders).toEqual([
|
||||
{
|
||||
leaseId: lease.id,
|
||||
executionWorkspaceId: workspaceId,
|
||||
executionWorkspaceName: "ACME-7-reusable-lease-holder",
|
||||
issueId,
|
||||
issueIdentifier: "ACME-7",
|
||||
issueTitle: "Reusable lease holder",
|
||||
},
|
||||
]);
|
||||
expect(await svc.removeIfDeletable(environmentId)).toBeNull();
|
||||
|
||||
// A released reusable lease still owns a provider sandbox that may be
|
||||
|
||||
@@ -57,6 +57,7 @@ import { getConfiguredSecretProvider } from "../secrets/configured-provider.js";
|
||||
import { assertBoardOrgAccess, getActorInfo } from "./authz.js";
|
||||
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
|
||||
import { environmentService } from "../services/environments.js";
|
||||
import { environmentRuntimeService } from "../services/environment-runtime.js";
|
||||
import { executionWorkspaceService } from "../services/execution-workspaces.js";
|
||||
|
||||
function isPlainRecord(value: unknown): value is Record<string, unknown> {
|
||||
@@ -329,6 +330,9 @@ export function environmentRoutes(
|
||||
) {
|
||||
const router = Router();
|
||||
const svc = environmentService(db);
|
||||
const environmentRuntime = environmentRuntimeService(db, {
|
||||
pluginWorkerManager: options.pluginWorkerManager,
|
||||
});
|
||||
const customImages = environmentCustomImageService(db, {
|
||||
pluginWorkerManager: options.pluginWorkerManager,
|
||||
});
|
||||
@@ -597,6 +601,10 @@ export function environmentRoutes(
|
||||
actor: ReturnType<typeof getActorInfo>;
|
||||
environment: { id: string; driver: string };
|
||||
impact: EnvironmentDeleteBlastRadius;
|
||||
// Sandboxes a consented delete destroyed before this rejection. Provider
|
||||
// destruction is not transactional with the delete guard, so a rejection
|
||||
// after a partial destroy must say what already happened.
|
||||
destroyedReusableSandboxLeaseCount?: number;
|
||||
}): never {
|
||||
const message =
|
||||
environmentDeleteBlockMessage(input.impact)
|
||||
@@ -606,6 +614,7 @@ export function environmentRoutes(
|
||||
environmentId: input.environment.id,
|
||||
environmentDriver: input.environment.driver,
|
||||
deleteBlockedReasons: input.impact.deleteBlockedReasons,
|
||||
destroyedReusableSandboxLeaseCount: input.destroyedReusableSandboxLeaseCount ?? 0,
|
||||
actorType: input.actor.actorType,
|
||||
actorId: input.actor.actorId,
|
||||
agentId: input.actor.agentId,
|
||||
@@ -613,7 +622,12 @@ export function environmentRoutes(
|
||||
},
|
||||
"environment delete rejected by guard",
|
||||
);
|
||||
throw conflict(message, { deleteBlockedReasons: input.impact.deleteBlockedReasons });
|
||||
throw conflict(message, {
|
||||
deleteBlockedReasons: input.impact.deleteBlockedReasons,
|
||||
...(input.destroyedReusableSandboxLeaseCount
|
||||
? { destroyedReusableSandboxLeaseCount: input.destroyedReusableSandboxLeaseCount }
|
||||
: {}),
|
||||
});
|
||||
}
|
||||
|
||||
function setupSessionActivityDetails(session: {
|
||||
@@ -1252,13 +1266,52 @@ export function environmentRoutes(
|
||||
}
|
||||
await assertPlatformProvisionedEnvironmentWritable(existing);
|
||||
const actor = getActorInfo(req);
|
||||
const impact = await svc.getDeleteBlastRadius(existing.id);
|
||||
let impact = await svc.getDeleteBlastRadius(existing.id);
|
||||
if (!impact) {
|
||||
res.status(404).json({ error: "Environment not found" });
|
||||
return;
|
||||
}
|
||||
// With explicit consent, destroy the environment's reusable sandbox leases
|
||||
// so the delete can proceed — but only while those leases are the sole
|
||||
// blocker. Destroying provider resources and then rejecting on another
|
||||
// gate would be an irreversible action with nothing gained. The destroy
|
||||
// must run before the delete: the driver needs the environment config to
|
||||
// reach the provider. A lease whose teardown fails lands in
|
||||
// `pending_cleanup`, and the re-fetched blast radius rejects below until
|
||||
// the cleanup sweep resolves it — no sandbox is ever orphaned silently.
|
||||
// A lease held by an in-flight run is skipped, keeps blocking, and the
|
||||
// re-check rejects the delete without touching that run's sandbox.
|
||||
//
|
||||
// The gate above and the destroy are not one atomic step: provider calls
|
||||
// cannot join a database transaction, so a blocker that lands in the
|
||||
// window between them (a new instance default, a fresh pending cleanup)
|
||||
// rejects the delete only after some sandboxes are already gone. Those
|
||||
// sandboxes belonged to the environment the operator consented to
|
||||
// destroy; the rejection reports the count so nothing is silent.
|
||||
let destroyedReusableSandboxLeaseCount = 0;
|
||||
if (
|
||||
req.query.destroyReusableSandboxLeases === "true"
|
||||
&& impact.reusableSandboxLeaseCount > 0
|
||||
&& impact.deleteBlockedReasons.every((reason) => reason === "reusable_sandbox_lease")
|
||||
) {
|
||||
const destroyResult = await environmentRuntime.destroyReusableSandboxLeasesForEnvironment({
|
||||
environmentId: existing.id,
|
||||
failureReason: "environment_deleted",
|
||||
});
|
||||
destroyedReusableSandboxLeaseCount = destroyResult.destroyed;
|
||||
impact = await svc.getDeleteBlastRadius(existing.id);
|
||||
if (!impact) {
|
||||
res.status(404).json({ error: "Environment not found" });
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (!impact.canDelete) {
|
||||
rejectEnvironmentDelete({ actor, environment: existing, impact });
|
||||
rejectEnvironmentDelete({
|
||||
actor,
|
||||
environment: existing,
|
||||
impact,
|
||||
destroyedReusableSandboxLeaseCount,
|
||||
});
|
||||
}
|
||||
|
||||
const removed = await svc.removeIfDeletable(existing.id);
|
||||
@@ -1301,9 +1354,15 @@ export function environmentRoutes(
|
||||
name: removed.name,
|
||||
driver: removed.driver,
|
||||
status: removed.status,
|
||||
...(destroyedReusableSandboxLeaseCount > 0
|
||||
? { destroyedReusableSandboxLeaseCount }
|
||||
: {}),
|
||||
},
|
||||
});
|
||||
res.json(presentEnvironmentForRead(req, removed));
|
||||
res.json({
|
||||
...presentEnvironmentForRead(req, removed),
|
||||
destroyedReusableSandboxLeaseCount,
|
||||
});
|
||||
});
|
||||
|
||||
router.post("/environments/:id/probe", async (req, res) => {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { and, eq, inArray } from "drizzle-orm";
|
||||
import { and, eq, inArray, sql } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { companySecrets, companySecretVersions, environmentLeases } from "@paperclipai/db";
|
||||
import { companySecrets, companySecretVersions, environmentLeases, heartbeatRuns } from "@paperclipai/db";
|
||||
import type {
|
||||
Environment,
|
||||
EnvironmentLease,
|
||||
@@ -3405,6 +3405,131 @@ export function environmentRuntimeService(
|
||||
return destroyed;
|
||||
},
|
||||
|
||||
/**
|
||||
* Destroy every reusable sandbox lease still owned by one environment, so a
|
||||
* consented environment delete can proceed. This must run while the
|
||||
* environment row still exists: the driver resolves provider credentials
|
||||
* from the environment config, and after the delete the normal destroy path
|
||||
* has no context left. A per-lease failure is contained — the driver routes
|
||||
* a failed teardown to `pending_cleanup` for the sweep, and an unexpected
|
||||
* throw leaves the lease in place — so the caller re-checks the blast
|
||||
* radius instead of trusting these counts for the delete decision.
|
||||
*/
|
||||
async destroyReusableSandboxLeasesForEnvironment(input: {
|
||||
environmentId: string;
|
||||
failureReason?: string;
|
||||
}): Promise<{ destroyed: number; failed: number; skippedLiveRun: number }> {
|
||||
const environment = await environmentsSvc.getById(input.environmentId);
|
||||
if (!environment) return { destroyed: 0, failed: 0, skippedLiveRun: 0 };
|
||||
const leaseRows = await db
|
||||
.select()
|
||||
.from(environmentLeases)
|
||||
.where(
|
||||
and(
|
||||
eq(environmentLeases.environmentId, input.environmentId),
|
||||
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
|
||||
inArray(environmentLeases.status, ["active", "released", "retained"]),
|
||||
),
|
||||
);
|
||||
|
||||
// A lease whose holding run is still in flight keeps its sandbox: the
|
||||
// consented delete must not tear a live run's environment out from under
|
||||
// it. The skipped lease keeps blocking the delete, so the caller's
|
||||
// blast-radius re-check rejects and the operator retries after the run
|
||||
// finishes. A lease pointing at a finished run — or at no run — is a
|
||||
// stale reservation and destroys normally.
|
||||
const holdingRunIds = leaseRows
|
||||
.map((row) => row.heartbeatRunId)
|
||||
.filter((runId): runId is string => Boolean(runId));
|
||||
const liveRunIds = new Set<string>();
|
||||
if (holdingRunIds.length > 0) {
|
||||
const liveRuns = await db
|
||||
.select({ id: heartbeatRuns.id })
|
||||
.from(heartbeatRuns)
|
||||
.where(
|
||||
and(
|
||||
inArray(heartbeatRuns.id, holdingRunIds),
|
||||
inArray(heartbeatRuns.status, ["queued", "scheduled_retry", "running"]),
|
||||
),
|
||||
);
|
||||
for (const run of liveRuns) liveRunIds.add(run.id);
|
||||
}
|
||||
|
||||
let destroyed = 0;
|
||||
let failed = 0;
|
||||
let skippedLiveRun = 0;
|
||||
const failureReason = input.failureReason ?? "environment_delete_requested";
|
||||
const now = new Date();
|
||||
for (const leaseRow of leaseRows) {
|
||||
if (leaseRow.heartbeatRunId && liveRunIds.has(leaseRow.heartbeatRunId)) {
|
||||
skippedLiveRun += 1;
|
||||
continue;
|
||||
}
|
||||
// Claim the row BEFORE the provider call, mirroring the inline-teardown
|
||||
// invariant used elsewhere in this file: no provider destroy without a
|
||||
// durable `pending_cleanup` reference already on disk. The claim is one
|
||||
// conditional UPDATE, so it is the fence against a racing resume: a
|
||||
// resume that re-activates the lease first makes the status predicate
|
||||
// (or the run-liveness predicate) fail and the claim loses — the live
|
||||
// run keeps its sandbox. A claim that wins parks the lease where the
|
||||
// cleanup sweep owns it, so a crash or thrown destroy after this point
|
||||
// is recovered by the sweep's idempotent teardown, and a double write
|
||||
// failure cannot strand the lease in a reusable status.
|
||||
const claimedRow = await db
|
||||
.update(environmentLeases)
|
||||
.set({
|
||||
status: "pending_cleanup",
|
||||
failureReason,
|
||||
cleanupStatus: "failed",
|
||||
releasedAt: now,
|
||||
lastUsedAt: now,
|
||||
updatedAt: now,
|
||||
})
|
||||
.where(
|
||||
and(
|
||||
eq(environmentLeases.id, leaseRow.id),
|
||||
inArray(environmentLeases.status, ["active", "released", "retained"]),
|
||||
sql`NOT EXISTS (
|
||||
SELECT 1 FROM ${heartbeatRuns}
|
||||
WHERE ${heartbeatRuns.id} = ${environmentLeases.heartbeatRunId}
|
||||
AND ${heartbeatRuns.status} IN ('queued', 'scheduled_retry', 'running')
|
||||
)`,
|
||||
),
|
||||
)
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!claimedRow) {
|
||||
// Lost to a racing resume or a concurrent terminal transition — the
|
||||
// lease is no longer ours to destroy.
|
||||
skippedLiveRun += 1;
|
||||
continue;
|
||||
}
|
||||
const leaseSnapshot = toEnvironmentLeaseSnapshot(claimedRow);
|
||||
try {
|
||||
const driver = getDriver(getLeaseDriverKey(leaseSnapshot, environment));
|
||||
if (!driver?.destroyRunLease) {
|
||||
// No driver available: the claim already parked the lease for the
|
||||
// sweep, which retries once the driver's plugin is back.
|
||||
failed += 1;
|
||||
continue;
|
||||
}
|
||||
const lease = await driver.destroyRunLease({
|
||||
environment,
|
||||
lease: leaseSnapshot,
|
||||
failureReason,
|
||||
});
|
||||
if (lease && lease.status !== "pending_cleanup") destroyed += 1;
|
||||
else failed += 1;
|
||||
} catch {
|
||||
// The claim above already parked the lease in `pending_cleanup`, so
|
||||
// the sweep owns the retry; its teardown is idempotent, so a destroy
|
||||
// that reached the provider before the throw resolves as success.
|
||||
failed += 1;
|
||||
}
|
||||
}
|
||||
return { destroyed, failed, skippedLiveRun };
|
||||
},
|
||||
|
||||
async resumeRunLease(input: EnvironmentDriverLeaseInput): Promise<PluginEnvironmentLease | EnvironmentLease | null> {
|
||||
const driver = requireDriverKey(getLeaseDriverKey(input.lease, input.environment));
|
||||
if (!driver.resumeRunLease) {
|
||||
|
||||
@@ -1230,8 +1230,17 @@ export function environmentService(db: Db) {
|
||||
),
|
||||
),
|
||||
db
|
||||
.select({ count: sql<number>`count(*)::int` })
|
||||
.select({
|
||||
leaseId: environmentLeases.id,
|
||||
executionWorkspaceId: environmentLeases.executionWorkspaceId,
|
||||
executionWorkspaceName: executionWorkspaces.name,
|
||||
issueId: environmentLeases.issueId,
|
||||
issueIdentifier: issues.identifier,
|
||||
issueTitle: issues.title,
|
||||
})
|
||||
.from(environmentLeases)
|
||||
.leftJoin(executionWorkspaces, eq(environmentLeases.executionWorkspaceId, executionWorkspaces.id))
|
||||
.leftJoin(issues, eq(environmentLeases.issueId, issues.id))
|
||||
.where(
|
||||
and(
|
||||
eq(environmentLeases.environmentId, id),
|
||||
@@ -1253,7 +1262,15 @@ export function environmentService(db: Db) {
|
||||
const isManagedLocal = environment.driver === "local";
|
||||
const isInstanceDefault = countFromRows(instanceDefaultRows) > 0;
|
||||
const pendingCleanupLeaseCount = countFromRows(pendingCleanupLeaseRows);
|
||||
const reusableSandboxLeaseCount = countFromRows(reusableSandboxLeaseRows);
|
||||
const reusableSandboxLeaseHolders = reusableSandboxLeaseRows.map((row) => ({
|
||||
leaseId: row.leaseId,
|
||||
executionWorkspaceId: row.executionWorkspaceId,
|
||||
executionWorkspaceName: row.executionWorkspaceName,
|
||||
issueId: row.issueId,
|
||||
issueIdentifier: row.issueIdentifier,
|
||||
issueTitle: row.issueTitle,
|
||||
}));
|
||||
const reusableSandboxLeaseCount = reusableSandboxLeaseHolders.length;
|
||||
const deleteBlockedReasons: EnvironmentDeleteBlockedReason[] = [];
|
||||
if (isManagedLocal) deleteBlockedReasons.push("managed_local");
|
||||
if (isInstanceDefault) deleteBlockedReasons.push("instance_default");
|
||||
@@ -1273,6 +1290,7 @@ export function environmentService(db: Db) {
|
||||
deleteBlockedReasons,
|
||||
pendingCleanupLeaseCount,
|
||||
reusableSandboxLeaseCount,
|
||||
reusableSandboxLeaseHolders,
|
||||
staticReferences: {
|
||||
isManagedLocal,
|
||||
isInstanceDefault,
|
||||
|
||||
Reference in new issue
Block a user