From 20ccf03676c2b70e3865e632780cc95f9aaaa051 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 07:23:35 -0600 Subject: [PATCH] fix(workspaces): settle closed remote workspaces after one confirmed destroy (#21616) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server cleans up closed execution workspaces in a background sweep (`terminal-workspaces`). For a remote workspace, cleanup destroys the sandbox lease after it confirms that the saved snapshot is committed. > - A closed remote workspace stays a sweep candidate while `metadata.remoteWorkspace.location` is an object. A confirmed destroy did not clear that location. > - The cleanup write goes through the lifecycle gateway, and the gateway published a `workspace:` hint for the same workspace. That hint started a new inspection, which cleaned the workspace again and published again. > - On a hosted instance with no runs, this loop kept three statements at about 15–24 per second each and a dedicated lock connection per inspection. Every pass also asked the sandbox provider to destroy a sandbox that was already destroyed. > - This pull request clears the location after a confirmed destroy, skips the provider call when the stored lease already has a destroy receipt, and stops the sweep from hinting itself for its own writes. > - The benefit is that an idle instance with closed remote workspaces settles: each workspace is cleaned once, and the provider is called at most once per lease. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** With no runs, the terminal workspace sweep never became idle. On the hosted instance these statements ran all the time: `workspaceHasActiveRun` (22.8/s), the task tree read (22.6/s) and the snapshot persistence read in `inspectRemoteWorkspacePersistence` (14.8/s). `SET application_name` for the `paperclip-workspace-finalization-lock` connections ran at 9.6/s. **Steps to reproduce** 1. Create closed (`archived`) remote workspaces that have a committed snapshot, a clean authority and a `location`. 2. Start the terminal workspace worker with no other activity. 3. Watch the sweeps. Each cleanup reports `archived`, the gateway publishes `workspace:`, and the next targeted inspection cleans the same workspace again. Locally, a fixture with 140 workspaces (40 of them in this state) ran 568 sweeps, 136,888 statements and 23,880 lease destroy calls in 30 idle minutes. The destroys always hit the same 40 workspaces. **Expected behavior** A closed remote workspace is cleaned once. After the destroy is confirmed it leaves the candidate set, and an idle instance issues no statements between real deadlines. ## What Changed - `server/src/services/remote-workspace-lifecycle.ts` (`cleanupRemoteWorkspaceAllocation`): - Read the stored lease first. When it already carries a destroy receipt for the same environment, lease and provider lease, do not call `destroyLease` again. - After a confirmed destroy, set `metadata.remoteWorkspace.location` to `null`. One fixed-text `jsonb_set` update does this, and only when the stored location is still the location that was destroyed. - `server/src/services/execution-workspaces.ts`: the gateway takes `sweepOwned`. The sweep's own cleanup, cleanup-failure and reopen-flag writes set it, and then the gateway does not publish a terminal workspace hint for that workspace. All other gateway callers still publish. - `server/src/__tests__/terminal-workspace-idle.test.ts`: two new tests on real PostgreSQL. ## Verification - `settles after one cleanup of each closed remote workspace and inspects nothing on its own writes`: - Fixture: 40 workspaces (open trees, a stranded `running` holder, cooldown, done, closed remote with a committed snapshot, closed remote without a snapshot). - Runs the real coordinator and worker on a fake clock for 2 minutes, then 30 idle minutes. - Asserts: each closed remote workspace is destroyed exactly once; no targeted inspection in the idle window; fewer than 5,000 statements in 30 minutes; every cleaned workspace has `location: null`. - Before the change it fails: 850 destroy calls instead of 10 in the warm-up alone. After the change it passes. - `clears the location of an already destroyed lease without calling the provider again`: the stored lease has a destroy receipt. Before the change, the sweep calls `destroyLease`. After the change the provider mock is never called, the location is cleared, and the next inspection finds no candidate. - `npx vitest run src/__tests__/terminal-workspace-idle.test.ts src/services/remote-workspace-lifecycle.test.ts src/__tests__/terminal-workspace-work.test.ts src/__tests__/execution-workspaces-service.test.ts src/__tests__/workspace-delivery-cleanup.integration.test.ts` (in `server/`). Local fixture, 30 idle minutes, 140 workspaces (40 closed remote with a committed snapshot): | | Before | After | | --- | --- | --- | | Sweeps | 568 (all targeted, continuous) | 20 (one deadline rotation every 5 minutes) | | Statements | 136,888 | 1,371 | | Lease destroy calls | 23,880 | 40 (once per workspace) | | `workspaceHasActiveRun` reads | 45,440 | 260 | | Snapshot persistence reads | 22,720 | 40 | The remaining rotation every 5 minutes comes from deadlines of workspaces that cannot make progress (a stranded active-run holder, cleanup backoff). A separate pull request handles it. ## Risks - **Who reads `remoteWorkspace.location`.** For a closed, clean workspace with a committed snapshot and a destroyed lease, none of these readers needs the location: - the live run binding checks (`remote-workspace-lifecycle`, `fresh-continuation-fallback`, `workspace-recovery-fallback`, `workspace-recovery-source`) need a current writer or a dirty run reference; - the abandonment and hold paths (`remote-workspace-epoch-abandonment`, `remote-workspace-abandoned-files-hold`, `remote-workspace-unchanged-proof`) need a dirty or held allocation; - `reusable-sandbox-resume-replacement` treats `null` as no conflicting sandbox; - `remote-workspace-files` reads the saved snapshot when the location is `null`. With a destroyed location it would fail as unavailable. - The lease-release path (`clearRemoteWorkspaceLocationForReleasedLease`) already writes the same `null` for the same state. This change makes the sweep stop depending on that side effect. - **Provider calls.** Before this change every loop pass re-claimed the lease and called the driver's `destroyRunLease` again, through `destroyClaimedSandboxLease` in `server/src/services/environment-runtime.ts`. After it, a lease with a stored destroy receipt causes no provider call. - **Existing backlog after deploy.** The startup rotation and the next deadline rotation visit each closed remote workspace that still has a location once, 50 per page with 1 second between pages. A lease that already has a destroy receipt costs no provider call, and its location is cleared. Only leases without a receipt get one destroy call. The workspaces then leave the candidate set. - **No self-hint.** The sweep already handles the workspace it writes, and an inspection's deadline bookkeeping is unchanged. Other writers still publish hints. The git restore point and task directory paths depend on task or explicit hints, not on these writes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../__tests__/terminal-workspace-idle.test.ts | 153 ++++++++++++++++++ server/src/services/execution-workspaces.ts | 14 +- .../services/remote-workspace-lifecycle.ts | 19 ++- 3 files changed, 176 insertions(+), 10 deletions(-) create mode 100644 server/src/__tests__/terminal-workspace-idle.test.ts diff --git a/server/src/__tests__/terminal-workspace-idle.test.ts b/server/src/__tests__/terminal-workspace-idle.test.ts new file mode 100644 index 00000000000..c1bef7fc449 --- /dev/null +++ b/server/src/__tests__/terminal-workspace-idle.test.ts @@ -0,0 +1,153 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { + agents, companies, createDb, environmentLeases, environments, executionWorkspaceSnapshots, executionWorkspaces, + heartbeatRuns, issues, projects, type Db, +} from "@paperclipai/db"; +import { readRemoteWorkspaceAuthority } from "@paperclipai/shared"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { executionWorkspaceService } from "../services/execution-workspaces.js"; +import { remoteTerminationReceipt } from "../services/remote-execution-termination.js"; +import { registerTerminalWorkspaceWork, TERMINAL_WORKSPACE_RETRY_MS } from "../services/terminal-workspace-work.js"; +import { meterDatabaseStatements } from "./helpers/database-statement-meter.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; + +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_TERMINAL_WORKSPACE_IDLE_TEST_DATABASE_URL"); +const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport(); +const DAY = 86_400_000; +const realSetTimeout = globalThis.setTimeout; +const realSleep = (ms: number) => new Promise(resolve => realSetTimeout(resolve, ms)); + +(support.supported ? describe : describe.skip)("terminal workspace sweep at idle", () => { + let temporary: Awaited> | undefined; + let db: Db; + let meter: ReturnType; + beforeAll(async () => { + if (!external) temporary = await startEmbeddedPostgresTestDatabase("terminal-workspace-idle-"); + const url = external ?? temporary!.connectionString; + db = createDb(url); + db.$client.options.onnotice = () => {}; + meter = meterDatabaseStatements(db, url); + }, 120_000); + afterEach(() => { vi.useRealTimers(); }); + afterAll(async () => { meter?.uninstall(); await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + + async function seedCompany() { + const companyId = randomUUID(), projectId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Idle", issuePrefix: `I${companyId.slice(0, 6)}` }); + await db.insert(projects).values({ id: projectId, companyId, name: "Project", status: "in_progress" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Agent", role: "engineer", status: "idle", adapterType: "process" }); + await db.insert(environments).values({ id: environmentId, name: `Sandbox ${environmentId}`, driver: "sandbox" }); + return { companyId, projectId, agentId, environmentId }; + } + + async function seedWorkspace(f: Awaited>, kind: "open" | "held" | "cooldown" | "done" | "remote" | "remote_unsaved") { + const old = new Date(Date.now() - 10 * DAY); + const id = randomUUID(), issueId = randomUUID(), snapshotId = randomUUID(); + const location = { environmentId: f.environmentId, leaseId: randomUUID(), providerLeaseId: `sandbox-${randomUUID()}`, remoteCwd: "/workspace" }; + const remote = kind === "remote" || kind === "remote_unsaved"; + let runId: string | null = null; + if (kind === "held") { + runId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId, status: "running" }); + } + await db.insert(executionWorkspaces).values({ id, companyId: f.companyId, projectId: f.projectId, + mode: remote ? "isolated_workspace" : "shared_workspace", strategyType: "project_primary", name: `${kind} ${id}`, + status: remote ? "archived" : "active", closedAt: remote ? old : null, providerType: remote ? "remote" : "local_fs", + cwd: `/missing/${id}`, createdAt: old, updatedAt: old, + metadata: remote ? { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId, + source: { kind: "empty" }, location, writerRunId: null, dirty: false } } : null }); + await db.insert(issues).values({ id: issueId, companyId: f.companyId, projectId: f.projectId, title: kind, + status: kind === "open" ? "in_progress" : "done", executionWorkspaceId: id, executionRunId: runId, + completedAt: kind === "open" ? null : kind === "cooldown" ? new Date(Date.now() - 3_600_000) : old, updatedAt: old }); + await db.update(executionWorkspaces).set({ sourceIssueId: issueId, updatedAt: old }).where(eq(executionWorkspaces.id, id)); + if (kind === "remote") await db.insert(executionWorkspaceSnapshots).values({ id: snapshotId, companyId: f.companyId, + executionWorkspaceId: id, sourceOperationId: randomUUID(), generation: 1, epoch: 1, status: "committed" }); + return { id, location }; + } + + function destroyedLease(input: { companyId: string; leaseId: string; providerLeaseId: string; environmentId: string }) { + const lease = { id: input.leaseId, companyId: input.companyId, heartbeatRunId: randomUUID(), provider: "daytona", + providerLeaseId: input.providerLeaseId, environmentId: input.environmentId }; + return { ...lease, status: "released", cleanupStatus: "success", releasedAt: new Date(), + metadata: { remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: input.providerLeaseId, state: "destroyed" }) } }; + } + + it("settles after one cleanup of each closed remote workspace and inspects nothing on its own writes", async () => { + const f = await seedCompany(); + const remote: string[] = []; + for (const kind of ["open", "open", "held", "cooldown", "done", "remote", "remote", "remote_unsaved"] as const) { + for (let index = 0; index < 5; index++) { + const workspace = await seedWorkspace(f, kind); + if (kind === "remote") remote.push(workspace.id); + } + } + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); + const destroyCalls: string[] = []; + const service = executionWorkspaceService(db, { + destroyRemoteWorkspaceLease: async input => { destroyCalls.push(input.workspaceId); return destroyedLease(input) as never; }, + workspaceReaperCooldownDays: 1, workspaceCompactionDelayMs: 3_600_000, + terminalWorkspaceInspectionBackoffMs: TERMINAL_WORKSPACE_RETRY_MS, + }); + const sweeps: Array<{ targeted: boolean; checked: number }> = []; + let inFlight = 0; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: () => {} }); + const worker = registerTerminalWorkspaceWork(coordinator, { ...service, + sweepTerminalWorkspaces: async (limit, signal, options) => { + inFlight += 1; + try { + const result = await service.sweepTerminalWorkspaces(limit, signal, options); + sweeps.push({ targeted: Boolean(options?.targets), checked: result.checked }); + return result; + } finally { inFlight -= 1; } + } }, () => true); + const idle = async (seconds: number) => { + for (let second = 0; second < seconds; second += 10) { + await vi.advanceTimersByTimeAsync(10_000); + do { await realSleep(10); await vi.advanceTimersByTimeAsync(1); } while (inFlight > 0); + } + }; + try { + await idle(120); + expect(new Set(destroyCalls)).toEqual(new Set(remote)); + expect(destroyCalls.length).toBe(remote.length); + const before = { statements: meter.read().statements, sweeps: sweeps.length, destroys: destroyCalls.length }; + await idle(30 * 60); + vi.useRealTimers(); + const window = sweeps.slice(before.sweeps); + expect(destroyCalls.length).toBe(before.destroys); + expect(window.filter(sweep => sweep.targeted)).toEqual([]); + expect(meter.read().statements - before.statements).toBeLessThan(5_000); + for (const id of remote) { + const [row] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, id)); + expect(readRemoteWorkspaceAuthority(row!.metadata)?.location).toBeNull(); + } + } finally { + vi.useRealTimers(); + await worker.stop(); await coordinator.stop(); + } + }, 600_000); + + it("clears the location of an already destroyed lease without calling the provider again", async () => { + const f = await seedCompany(); + const workspace = await seedWorkspace(f, "remote"); + const runId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId, status: "succeeded" }); + const lease = { id: workspace.location.leaseId, companyId: f.companyId, heartbeatRunId: runId, provider: "daytona", + providerLeaseId: workspace.location.providerLeaseId }; + await db.insert(environmentLeases).values({ ...lease, environmentId: f.environmentId, executionWorkspaceId: workspace.id, + status: "released", cleanupStatus: "success", releasedAt: new Date(), + metadata: { remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }) } }); + const destroy = vi.fn(); + const service = executionWorkspaceService(db, { destroyRemoteWorkspaceLease: destroy as never, workspaceReaperCooldownDays: 1 }); + expect(await service.sweepTerminalWorkspaces(50, undefined, { targets: { workspaceIds: [workspace.id], issueIds: [] } })) + .toMatchObject({ checked: 1, archived: 1 }); + expect(destroy).not.toHaveBeenCalled(); + const [row] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace.id)); + expect(readRemoteWorkspaceAuthority(row!.metadata)?.location).toBeNull(); + expect(await service.sweepTerminalWorkspaces(50, undefined, { targets: { workspaceIds: [workspace.id], issueIds: [] } })) + .toMatchObject({ checked: 0 }); + }); +}); diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index bb411786a65..6ec8deb869e 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -2285,6 +2285,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic isWriteTarget: (fresh: FenceableWorkspaceRow) => boolean; onSkip: () => T; skipLog?: { event: string; message: string }; + sweepOwned?: boolean; write: (context: { tx: DbTransaction; fresh: FenceableWorkspaceRow }) => Promise; }): Promise { return withExecutionWorkspaceLifecycleLock(db, input.workspaceId, async (tx) => { @@ -2317,7 +2318,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic } return input.onSkip(); } - await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${input.workspaceId}`)); + if (!input.sweepOwned) await measureRunPhase("workspace.notification", () => notifyTerminalWorkspaceWork(tx, `workspace:${input.workspaceId}`)); return input.write({ tx, fresh }); }); } @@ -2337,11 +2338,12 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic // snapshot. async function clearReopenPendingConsumptionUnderLock( workspaceId: string, - options: { expectedGeneration: number; requireStaleSinceBefore?: Date }, + options: { expectedGeneration: number; requireStaleSinceBefore?: Date; sweepOwned?: boolean }, ): Promise { return fenceLifecycleGenerationWrite({ workspaceId, expectedGeneration: options.expectedGeneration, + sweepOwned: options.sweepOwned, isWriteTarget: (fresh) => { if (!metadataHasReopenPendingConsumption(fresh.metadata)) return false; if (options.requireStaleSinceBefore) { @@ -2622,6 +2624,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic return fenceLifecycleGenerationWrite<{ cleaned: boolean; warnings: string[]; skippedReopened?: boolean }>({ workspaceId: workspace.id, expectedGeneration: capturedGeneration, + sweepOwned: true, isWriteTarget: (fresh) => isClosedExecutionWorkspaceStatus(fresh.status), skipLog: { event: "execution_workspace.cleanup_skipped", @@ -2753,6 +2756,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic return fenceLifecycleGenerationWrite({ workspaceId: input.workspaceId, expectedGeneration: input.capturedGeneration, + sweepOwned: true, isWriteTarget: (fresh) => isClosedExecutionWorkspaceStatus(fresh.status), skipLog: { event: "execution_workspace.cleanup_failed_write_skipped", @@ -3798,7 +3802,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic const expirationTree = usingIssueIds ? issueTree.filter(issue => usingIssueIds.has(issue.id)) : issueTree; if (!sourceIssue || expirationTree.some(issue => !TERMINAL_ISSUE_STATUSES.has(issue.status))) { if (reopenPending) { - await clearReopenPendingConsumptionUnderLock(workspace.id, { + await clearReopenPendingConsumptionUnderLock(workspace.id, { sweepOwned: true, expectedGeneration: readExecutionWorkspaceLifecycleGeneration( workspace.metadata as Record | null, ), @@ -4055,7 +4059,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic // a later terminal cycle can archive the workspace again. Pass the // generation from this snapshot, so the clear skips a newer reopen that // raised the generation and installed its own fence after this read. - await clearReopenPendingConsumptionUnderLock(workspace.id, { + await clearReopenPendingConsumptionUnderLock(workspace.id, { sweepOwned: true, expectedGeneration: readExecutionWorkspaceLifecycleGeneration( workspace.metadata as Record | null, ), @@ -4127,7 +4131,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic // the rebuilt worktree. Pass this snapshot's generation and re-confirm // staleness against the fresh row under the lock, so a newer reopen that // raised the generation or refreshed the timestamp keeps its live fence. - const cleared = await clearReopenPendingConsumptionUnderLock(workspace.id, { + const cleared = await clearReopenPendingConsumptionUnderLock(workspace.id, { sweepOwned: true, expectedGeneration: readExecutionWorkspaceLifecycleGeneration( workspace.metadata as Record | null, ), diff --git a/server/src/services/remote-workspace-lifecycle.ts b/server/src/services/remote-workspace-lifecycle.ts index b4d24873fe0..6ae4bbe562b 100644 --- a/server/src/services/remote-workspace-lifecycle.ts +++ b/server/src/services/remote-workspace-lifecycle.ts @@ -87,12 +87,21 @@ export async function cleanupRemoteWorkspaceAllocation(input: { if (!persistence.ready || !persistence.authority) throw new Error(persistence.reason ?? "remote_workspace_saved_state_required"); const location = persistence.authority.location; if (location) { - const released = await input.destroyLease({ companyId: workspace.companyId, workspaceId: workspace.id, - leaseId: location.leaseId, environmentId: location.environmentId, providerLeaseId: location.providerLeaseId }); - if (!released || !hasRemoteTerminationReceipt(released) - || (released.metadata?.remoteExecutionTermination as { state?: unknown } | undefined)?.state !== "destroyed") { - throw new Error("remote_workspace_cleanup_unconfirmed"); + const destroyed = (lease: Parameters[0] | null | undefined) => Boolean(lease && hasRemoteTerminationReceipt(lease) + && (lease.metadata?.remoteExecutionTermination as { state?: unknown } | undefined)?.state === "destroyed"); + const [stored] = await input.db.select().from(environmentLeases).where(and( + eq(environmentLeases.companyId, workspace.companyId), eq(environmentLeases.id, location.leaseId), + eq(environmentLeases.environmentId, location.environmentId), eq(environmentLeases.providerLeaseId, location.providerLeaseId), + )); + if (!destroyed(stored)) { + const released = await input.destroyLease({ companyId: workspace.companyId, workspaceId: workspace.id, + leaseId: location.leaseId, environmentId: location.environmentId, providerLeaseId: location.providerLeaseId }); + if (!destroyed(released)) throw new Error("remote_workspace_cleanup_unconfirmed"); } + await input.db.update(executionWorkspaces) + .set({ metadata: sql`jsonb_set(${executionWorkspaces.metadata}, '{remoteWorkspace,location}', 'null'::jsonb)`, updatedAt: new Date() }) + .where(and(eq(executionWorkspaces.companyId, workspace.companyId), eq(executionWorkspaces.id, workspace.id), + sql`${executionWorkspaces.metadata}->'remoteWorkspace'->'location' = ${JSON.stringify(location)}::jsonb`)); } return { cleaned: true, warnings: [] as string[] }; }