mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server cleans up closed execution workspaces in a background sweep (`terminal-workspaces`). For a remote workspace, cleanup destroys the sandbox lease after it confirms that the saved snapshot is committed. > - A closed remote workspace stays a sweep candidate while `metadata.remoteWorkspace.location` is an object. A confirmed destroy did not clear that location. > - The cleanup write goes through the lifecycle gateway, and the gateway published a `workspace:<id>` hint for the same workspace. That hint started a new inspection, which cleaned the workspace again and published again. > - On a hosted instance with no runs, this loop kept three statements at about 15–24 per second each and a dedicated lock connection per inspection. Every pass also asked the sandbox provider to destroy a sandbox that was already destroyed. > - This pull request clears the location after a confirmed destroy, skips the provider call when the stored lease already has a destroy receipt, and stops the sweep from hinting itself for its own writes. > - The benefit is that an idle instance with closed remote workspaces settles: each workspace is cleaned once, and the provider is called at most once per lease. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** With no runs, the terminal workspace sweep never became idle. On the hosted instance these statements ran all the time: `workspaceHasActiveRun` (22.8/s), the task tree read (22.6/s) and the snapshot persistence read in `inspectRemoteWorkspacePersistence` (14.8/s). `SET application_name` for the `paperclip-workspace-finalization-lock` connections ran at 9.6/s. **Steps to reproduce** 1. Create closed (`archived`) remote workspaces that have a committed snapshot, a clean authority and a `location`. 2. Start the terminal workspace worker with no other activity. 3. Watch the sweeps. Each cleanup reports `archived`, the gateway publishes `workspace:<id>`, and the next targeted inspection cleans the same workspace again. Locally, a fixture with 140 workspaces (40 of them in this state) ran 568 sweeps, 136,888 statements and 23,880 lease destroy calls in 30 idle minutes. The destroys always hit the same 40 workspaces. **Expected behavior** A closed remote workspace is cleaned once. After the destroy is confirmed it leaves the candidate set, and an idle instance issues no statements between real deadlines. ## What Changed - `server/src/services/remote-workspace-lifecycle.ts` (`cleanupRemoteWorkspaceAllocation`): - Read the stored lease first. When it already carries a destroy receipt for the same environment, lease and provider lease, do not call `destroyLease` again. - After a confirmed destroy, set `metadata.remoteWorkspace.location` to `null`. One fixed-text `jsonb_set` update does this, and only when the stored location is still the location that was destroyed. - `server/src/services/execution-workspaces.ts`: the gateway takes `sweepOwned`. The sweep's own cleanup, cleanup-failure and reopen-flag writes set it, and then the gateway does not publish a terminal workspace hint for that workspace. All other gateway callers still publish. - `server/src/__tests__/terminal-workspace-idle.test.ts`: two new tests on real PostgreSQL. ## Verification - `settles after one cleanup of each closed remote workspace and inspects nothing on its own writes`: - Fixture: 40 workspaces (open trees, a stranded `running` holder, cooldown, done, closed remote with a committed snapshot, closed remote without a snapshot). - Runs the real coordinator and worker on a fake clock for 2 minutes, then 30 idle minutes. - Asserts: each closed remote workspace is destroyed exactly once; no targeted inspection in the idle window; fewer than 5,000 statements in 30 minutes; every cleaned workspace has `location: null`. - Before the change it fails: 850 destroy calls instead of 10 in the warm-up alone. After the change it passes. - `clears the location of an already destroyed lease without calling the provider again`: the stored lease has a destroy receipt. Before the change, the sweep calls `destroyLease`. After the change the provider mock is never called, the location is cleared, and the next inspection finds no candidate. - `npx vitest run src/__tests__/terminal-workspace-idle.test.ts src/services/remote-workspace-lifecycle.test.ts src/__tests__/terminal-workspace-work.test.ts src/__tests__/execution-workspaces-service.test.ts src/__tests__/workspace-delivery-cleanup.integration.test.ts` (in `server/`). Local fixture, 30 idle minutes, 140 workspaces (40 closed remote with a committed snapshot): | | Before | After | | --- | --- | --- | | Sweeps | 568 (all targeted, continuous) | 20 (one deadline rotation every 5 minutes) | | Statements | 136,888 | 1,371 | | Lease destroy calls | 23,880 | 40 (once per workspace) | | `workspaceHasActiveRun` reads | 45,440 | 260 | | Snapshot persistence reads | 22,720 | 40 | The remaining rotation every 5 minutes comes from deadlines of workspaces that cannot make progress (a stranded active-run holder, cleanup backoff). A separate pull request handles it. ## Risks - **Who reads `remoteWorkspace.location`.** For a closed, clean workspace with a committed snapshot and a destroyed lease, none of these readers needs the location: - the live run binding checks (`remote-workspace-lifecycle`, `fresh-continuation-fallback`, `workspace-recovery-fallback`, `workspace-recovery-source`) need a current writer or a dirty run reference; - the abandonment and hold paths (`remote-workspace-epoch-abandonment`, `remote-workspace-abandoned-files-hold`, `remote-workspace-unchanged-proof`) need a dirty or held allocation; - `reusable-sandbox-resume-replacement` treats `null` as no conflicting sandbox; - `remote-workspace-files` reads the saved snapshot when the location is `null`. With a destroyed location it would fail as unavailable. - The lease-release path (`clearRemoteWorkspaceLocationForReleasedLease`) already writes the same `null` for the same state. This change makes the sweep stop depending on that side effect. - **Provider calls.** Before this change every loop pass re-claimed the lease and called the driver's `destroyRunLease` again, through `destroyClaimedSandboxLease` in `server/src/services/environment-runtime.ts`. After it, a lease with a stored destroy receipt causes no provider call. - **Existing backlog after deploy.** The startup rotation and the next deadline rotation visit each closed remote workspace that still has a location once, 50 per page with 1 second between pages. A lease that already has a destroy receipt costs no provider call, and its location is cleared. Only leases without a receipt get one destroy call. The workspaces then leave the candidate set. - **No self-hint.** The sweep already handles the workspace it writes, and an inspection's deadline bookkeeping is unchanged. Other writers still publish hints. The git restore point and task directory paths depend on task or explicit hints, not on these writes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
108 lines
7.8 KiB
TypeScript
108 lines
7.8 KiB
TypeScript
import { and, eq, sql } from "drizzle-orm";
|
|
import { environmentLeases, heartbeatRuns, executionWorkspaces, executionWorkspaceSnapshots, type Db } from "@paperclipai/db";
|
|
import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type EnvironmentLease } from "@paperclipai/shared";
|
|
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
|
|
|
type Workspace = Pick<typeof executionWorkspaces.$inferSelect, "id" | "companyId" | "metadata">;
|
|
type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
|
export type DestroyRemoteWorkspaceLease = (input: {
|
|
companyId: string; workspaceId: string; leaseId: string; environmentId: string; providerLeaseId: string;
|
|
}) => Promise<EnvironmentLease | null>;
|
|
|
|
export type WorkspaceSourceLease = Pick<typeof environmentLeases.$inferSelect,
|
|
"id" | "companyId" | "heartbeatRunId" | "provider" | "providerLeaseId">;
|
|
|
|
export async function preparedRemoteWorkspaceSource(db: Db, lease: WorkspaceSourceLease) {
|
|
if (!lease.heartbeatRunId || !lease.provider || lease.provider === "local" || !lease.providerLeaseId) return null;
|
|
const [bound] = await db.select({ lease: environmentLeases, run: {
|
|
id: heartbeatRuns.id, runtimeMode: heartbeatRuns.runtimeMode,
|
|
controllerBootId: heartbeatRuns.controllerBootId,
|
|
issueId: sql<string | null>`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text,
|
|
${heartbeatRuns.contextSnapshot}->>'issueId')`,
|
|
reference: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'`,
|
|
legacyReference: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'`,
|
|
} }).from(environmentLeases)
|
|
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId),
|
|
eq(heartbeatRuns.companyId, environmentLeases.companyId)))
|
|
.where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId),
|
|
eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId), eq(environmentLeases.provider, lease.provider),
|
|
eq(environmentLeases.providerLeaseId, lease.providerLeaseId),
|
|
sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'state' = 'prepared'`)).limit(1);
|
|
if (!bound) return null;
|
|
const parsed = remoteWorkspaceRunReferenceSchema.safeParse(bound.run.reference);
|
|
if (!parsed.success || parsed.data.state !== "prepared" || bound.run.legacyReference != null) return null;
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, lease.companyId), eq(executionWorkspaces.id, parsed.data.workspaceId))).limit(1);
|
|
return matchPreparedRemoteWorkspaceSource(bound, workspace);
|
|
}
|
|
|
|
type PreparedSourceBinding = {
|
|
lease: Pick<typeof environmentLeases.$inferSelect, "id" | "companyId" | "heartbeatRunId" | "provider" | "providerLeaseId"
|
|
| "issueId" | "executionWorkspaceId" | "environmentId" | "metadata">;
|
|
run: { id: string; runtimeMode: string; controllerBootId: string | null; issueId: string | null;
|
|
reference: unknown; legacyReference: unknown };
|
|
};
|
|
|
|
/** Shared binding proof for single-source inspection and batched allocation fencing. */
|
|
export function matchPreparedRemoteWorkspaceSource<T extends PreparedSourceBinding>(bound: T,
|
|
workspace: Pick<typeof executionWorkspaces.$inferSelect, "id" | "companyId" | "sourceIssueId" | "mode" | "metadata"> | null | undefined) {
|
|
const parsed = remoteWorkspaceRunReferenceSchema.safeParse(bound.run.reference);
|
|
if (!parsed.success || parsed.data.state !== "prepared" || bound.run.legacyReference != null) return null;
|
|
const reference = parsed.data;
|
|
const issueId = bound.run.issueId;
|
|
if (typeof issueId !== "string" || bound.lease.issueId !== issueId || reference.leaseId !== bound.lease.id
|
|
|| reference.providerLeaseId !== bound.lease.providerLeaseId || reference.remoteCwd !== bound.lease.metadata?.remoteCwd
|
|
|| bound.lease.executionWorkspaceId && bound.lease.executionWorkspaceId !== reference.workspaceId) return null;
|
|
const authority = readRemoteWorkspaceAuthority(workspace?.metadata);
|
|
if (!workspace || workspace.companyId !== bound.lease.companyId || workspace.id !== reference.workspaceId
|
|
|| workspace.sourceIssueId !== issueId && workspace.mode !== "shared_workspace"
|
|
|| !authority || !authority.dirty || authority.writerRunId !== bound.run.id || authority.epoch !== reference.epoch
|
|
|| authority.snapshotId !== reference.snapshotId || !authority.location
|
|
|| authority.location.environmentId !== bound.lease.environmentId || authority.location.leaseId !== bound.lease.id
|
|
|| authority.location.providerLeaseId !== bound.lease.providerLeaseId || authority.location.remoteCwd !== reference.remoteCwd) return null;
|
|
return { ...bound, workspace, reference, issueId };
|
|
}
|
|
|
|
export async function inspectRemoteWorkspacePersistence(db: Db | Transaction, workspace: Workspace) {
|
|
const authority = readRemoteWorkspaceAuthority(workspace.metadata);
|
|
if (!authority || authority.dirty || authority.writerRunId || !authority.snapshotId) {
|
|
return { ready: false, reason: "The workspace still has unsaved work or an unfinished writer.", authority };
|
|
}
|
|
const [snapshot] = await db.select({ id: executionWorkspaceSnapshots.id }).from(executionWorkspaceSnapshots).where(and(
|
|
eq(executionWorkspaceSnapshots.companyId, workspace.companyId), eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id),
|
|
eq(executionWorkspaceSnapshots.id, authority.snapshotId), eq(executionWorkspaceSnapshots.status, "committed"),
|
|
));
|
|
return { ready: Boolean(snapshot), reason: snapshot ? null : "The saved workspace is unavailable. Its remote files were retained.", authority };
|
|
}
|
|
|
|
/** Called within the workspace lifecycle fence; the saved workspace remains durable. */
|
|
export async function cleanupRemoteWorkspaceAllocation(input: {
|
|
db: Db; workspace: Workspace; destroyLease: DestroyRemoteWorkspaceLease;
|
|
}) {
|
|
const [workspace] = await input.db.select().from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, input.workspace.companyId), eq(executionWorkspaces.id, input.workspace.id),
|
|
));
|
|
if (!workspace || !["archived", "cleanup_failed"].includes(workspace.status)) throw new Error("remote_workspace_cleanup_binding_changed");
|
|
const persistence = await inspectRemoteWorkspacePersistence(input.db, workspace);
|
|
if (!persistence.ready || !persistence.authority) throw new Error(persistence.reason ?? "remote_workspace_saved_state_required");
|
|
const location = persistence.authority.location;
|
|
if (location) {
|
|
const destroyed = (lease: Parameters<typeof hasRemoteTerminationReceipt>[0] | null | undefined) => Boolean(lease && hasRemoteTerminationReceipt(lease)
|
|
&& (lease.metadata?.remoteExecutionTermination as { state?: unknown } | undefined)?.state === "destroyed");
|
|
const [stored] = await input.db.select().from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, workspace.companyId), eq(environmentLeases.id, location.leaseId),
|
|
eq(environmentLeases.environmentId, location.environmentId), eq(environmentLeases.providerLeaseId, location.providerLeaseId),
|
|
));
|
|
if (!destroyed(stored)) {
|
|
const released = await input.destroyLease({ companyId: workspace.companyId, workspaceId: workspace.id,
|
|
leaseId: location.leaseId, environmentId: location.environmentId, providerLeaseId: location.providerLeaseId });
|
|
if (!destroyed(released)) throw new Error("remote_workspace_cleanup_unconfirmed");
|
|
}
|
|
await input.db.update(executionWorkspaces)
|
|
.set({ metadata: sql`jsonb_set(${executionWorkspaces.metadata}, '{remoteWorkspace,location}', 'null'::jsonb)`, updatedAt: new Date() })
|
|
.where(and(eq(executionWorkspaces.companyId, workspace.companyId), eq(executionWorkspaces.id, workspace.id),
|
|
sql`${executionWorkspaces.metadata}->'remoteWorkspace'->'location' = ${JSON.stringify(location)}::jsonb`));
|
|
}
|
|
return { cleaned: true, warnings: [] as string[] };
|
|
}
|