mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core owns run leases and delegates remote execution to environment plugins. > - A retry must verify its old checkout on the placement that will run the task. > - A completed task must still release its provider resources. > - This change uses the existing lease for workspace verification and always calls the release hook after settlement. > - Saved settlement proof lets cleanup retry without repeating execution work. ## Linked Issues or Issue Description Refs #21237. That PR records stop proof before lease release. This PR keeps provider cleanup after settlement and changes workspace verification sequencing. Both touch task completion; preserve both changes if the merge order requires conflict resolution. **What happened?** Workspace verification ran before a portable task had a lease, so a provider could not verify the checkout on its selected placement. Core also skipped the provider release hook after successful task completion. This could leave resources allocated after the run ended. **Expected behavior** Verify the old checkout through the selected run lease. A confirmed refusal can use the authorized fresh-workspace fallback. A transport failure must remain retryable. Call the provider release hook after settlement and retry failed cleanup without changing the run result. **Steps to reproduce** 1. Retry a task whose prior checkout uses plugin-managed storage and portable task placement. 2. Verify that the plugin receives the same task lease for verification and preparation. 3. Complete a task, fail the provider release once, and reconstruct the cleanup controller. 4. Confirm that cleanup calls only the release hook and retains the completed run outcome. ## What Changed - Acquire the logical run lease before provider workspace verification and reuse it for preparation and execution. - Preserve the existing session and workspace choices. Recompute workspace-dependent inputs only after a definite refusal; acquire credentials and managed runtime inputs once. - Save the exact refusal and lease before cleanup. Use a workspace-scoped creation key only for the resulting fresh attempt. - Replay interrupted verification on the same unlaunched lease. Keep transport errors retryable and reject replay after execution starts. - Always call the provider release hook after task settlement. Pass refreshed settlement proof and route failed releases through the existing cleanup reaper. - Add database regressions for call order, restart, retained writers, refusal, transport errors, and settlement-only cleanup replay. `plugin-workspaces.ts` grows from 997 to 1,001 lines. The four added lines extend the existing workspace transaction and lease callbacks. Keeping the checks there preserves the ownership assertion, provider result validation, and binding update in one operation. Extracting only the verification branch would split that transaction without removing a separate responsibility. ## Verification - Server TypeScript check passed: `tsc -p server/tsconfig.json --noEmit`. The complete `server/scripts/typecheck-layers.mjs` check also passed, including all test-tree layers. - The first CI typecheck caught a readonly array in a test fixture that the narrower source check did not include. Commit8d9ec19e7acorrects that fixture; all layered checks pass afterward. - Added a public heartbeat refusal regression in7c598a4da3. All ten session-preparation tests pass; independent execution of the refusal case also passes. It checks successful completion, the saved fresh workspace, a new conversation, and one credential acquisition. - Nine database/service suites passed 335 tests: plugin warm tasks, plugin runner tasks, startup cleanup recovery, credential writeback, plugin workspaces, session preparation, plugin environments, retry workspace continuity, and heartbeat workspace preparation. - After the final restart change, both targeted refusal/restart tests passed. The primary agent independently reran those cases and the final server typecheck. After integrating current master, the primary agent's independent rerun passed all 237 tests across the warm-task, plugin-workspace, session-preparation, and heartbeat-workspace suites. - `pnpm verify` returned exit 2 and requested broad CI coverage because the plugin execution contract documentation changed. No full local Core-suite pass is claimed. - Previous-revision CI was green on8d9ec19e7a: 59 checks passed and 3 were skipped. The first run failed the unchanged legacy-continuation browser assertion and credential-cleanup server shard. Each passed on one targeted retry with no code changes. The credential-cleanup integration suite also passed all 21 tests locally on Node 22. The browser failure concerned an existing continuation metadata marker; the server failure cascaded after a cleanup error left a test spy installed. The successful retries do not establish the cause of the first failures. - Local database fixtures use simulated provider workers. No live provider or deployed environment was changed. - Additional verification for59c7a1dae8: the layered server typecheck passed. Independent execution passed all 265 tests across durable task retries, workspace preparation, and public heartbeat preparation. The new shutdown regression preserves the fresh Retry choice, waits for parent cleanup, and prevents the successor from inheriting the parent lease reference. Same-run recovery retains its receipt. Required CI is green on59c7a1dae8: 58 checks passed and 3 were skipped. Standard review scored 5/5 with no unresolved comments. The separate Superagent Security Scan is neutral/inconclusive because its workflow failed; a direct rerun request returned HTTP 404. That scan is not claimed as passed. ## Risks - Providers must use the supplied task lease for verification and return a definite `binding_mismatch` only when the checkout is unconfirmed. Transport errors retain the retry decision. - A refused checkout waits for confirmed release before a fresh allocation. Failed deletion remains owned by the existing cleanup reaper. - Task settlement and physical resource release stay separate. Retained handoff leases do not enter this release path. - No migration or SDK schema change is required. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) implemented the changes and tests. The primary Codex agent independently reviewed the connected Core and plugin paths and ran checks. A separate GPT-6.1 Sol agent reviewed lifecycle and restart boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue using the bug template labels - [x] I have not referenced internal or private project details - [x] My branch name describes the change - [x] I have added relevant tests and documentation - [x] I have considered and documented risks - [x] All required CI gates are green - [x] Greptile is 5/5 with no unresolved comments - [x] I will address reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1002 lines
43 KiB
TypeScript
1002 lines
43 KiB
TypeScript
import { isDeepStrictEqual } from "node:util";
|
|
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
|
|
import { assertProjectStorageEnvironment, effectiveProjectWorkspacePolicy, requireWorkspaceProviderWorker, type WorkspaceWorkers } from "./project-workspace-provider.js";
|
|
import { releaseProviderRecoveryTransfers } from "./provider-fresh-workspace.js";
|
|
import { retireAbandonedPluginWorkspaceReservation } from "./plugin-workspace-reservations.js";
|
|
import { randomUUID, createHash } from "node:crypto";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import {
|
|
environmentLeases,
|
|
executionWorkspaces,
|
|
heartbeatRuns,
|
|
issues,
|
|
projects,
|
|
projectWorkspaces,
|
|
type Db,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
readWorkspaceProviderBinding,
|
|
type WorkspaceProviderBinding,
|
|
type ExecutionWorkspace,
|
|
} from "@paperclipai/shared";
|
|
import {
|
|
pluginWorkspaceParamsSchema,
|
|
workspaceProviderOperationSchema,
|
|
type PluginWorkspaceOperation,
|
|
type PluginWorkspaceRecoveryPullRequest,
|
|
type PluginWorkspaceParams,
|
|
type PluginFreshWorkspace,
|
|
} from "@paperclipai/plugin-sdk";
|
|
import { callWorkspaceProvider } from "./plugin-workspace-preparation.js";
|
|
import { withPluginWorkspaceReplay } from "./plugin-workspace-replay-context.js";
|
|
import { pluginRegistryService } from "./plugin-registry.js";
|
|
import { notifyTerminalWorkspaceWork } from "./terminal-workspace-work.js";
|
|
import { withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifecycle-lock.js";
|
|
import {
|
|
assertTaskWorkspaceExecutionAccess,
|
|
assertTaskWorkspaceSourceProjectAccess,
|
|
} from "./task-workspace-access.js";
|
|
import { forbidden, conflict } from "../errors.js";
|
|
|
|
export { callWorkspaceProvider } from "./plugin-workspace-preparation.js";
|
|
export type { WorkspaceWorkers } from "./project-workspace-provider.js";
|
|
type Workspace = typeof executionWorkspaces.$inferSelect;
|
|
type TaskPlacement = NonNullable<PluginWorkspaceParams["taskLease"]>;
|
|
const preparationPlacementSchema = pluginWorkspaceParamsSchema.pick({
|
|
generation: true, runId: true, agentId: true, taskLease: true,
|
|
});
|
|
|
|
function readPreparationPlacement(workspace: Workspace) {
|
|
const saved = workspace.metadata?.workspaceProviderPreparationPlacement;
|
|
if (saved === undefined) return undefined;
|
|
const placement = preparationPlacementSchema.parse(saved);
|
|
if (!placement.runId || !placement.agentId ||
|
|
placement.generation !== (workspace.metadata?.lifecycleGeneration ?? 0))
|
|
throw conflict("Workspace preparation placement changed");
|
|
return placement;
|
|
}
|
|
type EnsureTaskWorkspace = (taskId: string, assertOwned: () => void) => Promise<void> | undefined;
|
|
|
|
export async function pluginWorkspaceSupportsMergeAncestry(db: Db, workers: WorkspaceWorkers | undefined, workspace: Workspace) {
|
|
const pluginId = readWorkspaceProviderBinding(workspace.metadata)?.pluginId;
|
|
if (!pluginId) return true;
|
|
const selected = await requireWorkspaceProviderWorker(db, workers, workspace.companyId, pluginId);
|
|
return selected.manifestJson.workspaceProvider?.supportsMergeAncestry !== false;
|
|
}
|
|
|
|
export async function preparePluginWorkspace(
|
|
db: Db,
|
|
workers: WorkspaceWorkers | undefined,
|
|
input: {
|
|
companyId: string;
|
|
projectId: string | null;
|
|
projectWorkspaceId: string | null;
|
|
issueId: string | null;
|
|
runId: string;
|
|
agentId: string;
|
|
responsibleUserId: string | null;
|
|
existingWorkspaceId: string | null;
|
|
mode: string;
|
|
strategy: string;
|
|
selectedProviderPluginId?: string | null;
|
|
environment?: Pick<import("@paperclipai/shared").Environment, "driver" | "config" | "name"> | null;
|
|
branchMode?: "create" | "existing";
|
|
branchName?: string | null;
|
|
baseRef?: string | null;
|
|
fresh?: PluginFreshWorkspace;
|
|
expectedBindingRevision?: number;
|
|
verifyExisting?: true;
|
|
assertOwned?: () => void;
|
|
acquireTaskLease?: (workspace: Pick<ExecutionWorkspace, "id" | "mode">) => Promise<{ leaseId: string; taskLease: TaskPlacement }>;
|
|
onTaskPrepared?: (ensure: EnsureTaskWorkspace) => void;
|
|
},
|
|
) {
|
|
const [previous] = input.existingWorkspaceId
|
|
? await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(
|
|
and(
|
|
eq(executionWorkspaces.id, input.existingWorkspaceId),
|
|
eq(executionWorkspaces.companyId, input.companyId),
|
|
),
|
|
)
|
|
: [];
|
|
const [reservation] = input.fresh ? await db.select().from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, input.companyId),
|
|
eq(executionWorkspaces.sourceIssueId, input.issueId!),
|
|
sql`${executionWorkspaces.metadata}#>>'{workspaceProviderFresh,attemptId}' = ${input.fresh.attemptId}`,
|
|
)).limit(1) : [];
|
|
const existing = input.fresh ? reservation : previous;
|
|
if (input.verifyExisting && (!existing || existing.metadata?.workspaceProviderRemoved === true ||
|
|
existing.metadata?.workspaceProviderRestoration)) return null;
|
|
const prior = existing ?? previous;
|
|
if (
|
|
existing?.metadata?.workspaceProviderRetired === true ||
|
|
existing?.metadata?.workspaceProviderRemoval
|
|
)
|
|
throw conflict("Workspace storage is retired or awaiting cleanup");
|
|
if (!existing && (!input.projectId || !input.projectWorkspaceId)) return null;
|
|
const saved = readWorkspaceProviderBinding(existing?.metadata);
|
|
const priorBinding = readWorkspaceProviderBinding(prior?.metadata);
|
|
const mode = input.fresh ? "isolated_workspace" : existing?.mode ?? input.mode,
|
|
strategy = existing?.strategyType ?? input.strategy;
|
|
const projectId = priorBinding?.projectId ?? input.projectId;
|
|
const [project] = projectId
|
|
? await db
|
|
.select()
|
|
.from(projects)
|
|
.where(
|
|
and(
|
|
eq(projects.id, projectId),
|
|
eq(projects.companyId, input.companyId),
|
|
),
|
|
)
|
|
: [];
|
|
if (prior && prior.providerType !== "plugin" && !priorBinding && !prior.metadata?.workspaceProviderOwner) return null;
|
|
const owner = priorBinding?.pluginId ?? prior?.metadata?.workspaceProviderOwner ??
|
|
(!existing && input.selectedProviderPluginId !== undefined ? input.selectedProviderPluginId :
|
|
!existing && project ? (await effectiveProjectWorkspacePolicy(db, input.companyId, project.executionWorkspacePolicy, false))?.storageProviderPluginId : null);
|
|
const pluginId = typeof owner === "string" ? owner : null;
|
|
if (!pluginId) {
|
|
if (existing?.providerType === "plugin")
|
|
throw conflict("Workspace storage ownership is missing");
|
|
return null;
|
|
}
|
|
if (
|
|
!input.issueId ||
|
|
!project ||
|
|
project.archivedAt ||
|
|
!input.projectWorkspaceId
|
|
)
|
|
throw conflict(
|
|
"Managed storage requires a task and an available project source",
|
|
);
|
|
if (
|
|
!["shared_workspace", "isolated_workspace", "operator_branch"].includes(
|
|
input.mode,
|
|
) ||
|
|
!["project_primary", "git_worktree"].includes(strategy)
|
|
)
|
|
throw conflict("Unsupported managed workspace strategy");
|
|
if (existing && existing.providerType !== "plugin")
|
|
throw conflict(
|
|
"An existing local workspace cannot change storage providers",
|
|
);
|
|
const selectedProvider = await requireWorkspaceProviderWorker(db, workers, input.companyId, pluginId, true);
|
|
if (input.environment) assertProjectStorageEnvironment(selectedProvider, input.environment);
|
|
const declaration = selectedProvider.manifestJson.workspaceProvider!;
|
|
if (input.verifyExisting && declaration.workspaceVerificationVersion !== 1) return null;
|
|
if (input.fresh && declaration.freshWorkspaceVersion !== 1)
|
|
throw conflict("A fresh attempt is unavailable for this workspace storage provider. The existing workspace is retained.", { reason: "fresh_workspace_storage_unsupported" });
|
|
if (
|
|
!declaration.modes.includes(mode as "isolated_workspace") ||
|
|
!declaration.strategies.includes(strategy as "git_worktree")
|
|
) {
|
|
throw conflict(
|
|
"The storage plugin does not support this workspace mode or strategy",
|
|
);
|
|
}
|
|
const actor = {
|
|
type: "agent" as const,
|
|
source: "agent_jwt" as const,
|
|
companyId: input.companyId,
|
|
agentId: input.agentId,
|
|
runId: input.runId,
|
|
onBehalfOfUserId:
|
|
input.responsibleUserId === "local-board"
|
|
? null
|
|
: input.responsibleUserId,
|
|
};
|
|
const authorize = async (workspaceId = existing?.id) => {
|
|
await assertTaskWorkspaceExecutionAccess(db, actor, input.companyId, {
|
|
issueId: input.issueId!,
|
|
...(workspaceId
|
|
? { workspaceId }
|
|
: { projectWorkspaceId: input.projectWorkspaceId }),
|
|
});
|
|
await assertTaskWorkspaceSourceProjectAccess(
|
|
db,
|
|
actor,
|
|
input.companyId,
|
|
project.id,
|
|
);
|
|
const [run] = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, input.runId),
|
|
eq(heartbeatRuns.companyId, input.companyId),
|
|
),
|
|
);
|
|
if (
|
|
!run ||
|
|
run.agentId !== input.agentId ||
|
|
![run.issueId, run.nativeIssueId].includes(input.issueId!) ||
|
|
run.status !== "running"
|
|
)
|
|
throw forbidden("Workspace preparation no longer has an active run");
|
|
};
|
|
await authorize();
|
|
const [source] = await db
|
|
.select()
|
|
.from(projectWorkspaces)
|
|
.where(
|
|
and(
|
|
eq(
|
|
projectWorkspaces.id,
|
|
saved?.projectWorkspaceId ?? input.projectWorkspaceId,
|
|
),
|
|
eq(projectWorkspaces.projectId, project.id),
|
|
eq(projectWorkspaces.companyId, input.companyId),
|
|
),
|
|
);
|
|
if (!source) throw conflict("Workspace repository source is unavailable");
|
|
const sourceInput = (prior?.metadata?.workspaceProviderSource as
|
|
| Extract<PluginWorkspaceOperation, { kind: "prepare" }>["source"]
|
|
| undefined) ?? {
|
|
id: source.id,
|
|
revision:
|
|
saved?.sourceRevision ??
|
|
createHash("sha256")
|
|
.update(
|
|
JSON.stringify([
|
|
source.id,
|
|
source.repoUrl,
|
|
source.repoRef,
|
|
source.defaultRef,
|
|
]),
|
|
)
|
|
.digest("hex"),
|
|
repoUrl: existing?.repoUrl ?? source.repoUrl,
|
|
repoRef: existing?.baseRef ?? source.repoRef,
|
|
defaultRef: source.defaultRef,
|
|
};
|
|
const branchName = input.fresh ? null : existing
|
|
? existing.branchName
|
|
: (input.branchName ??
|
|
(strategy === "git_worktree" ? `workspace-${randomUUID()}` : null));
|
|
// Reserve and bind before calling the provider. A crash must not orphan a created checkout.
|
|
const workspace =
|
|
existing ??
|
|
(await db.transaction(async (tx) => {
|
|
const [currentProject] = await tx
|
|
.select()
|
|
.from(projects)
|
|
.where(eq(projects.id, project.id))
|
|
.for("update");
|
|
const [currentSource] = await tx
|
|
.select()
|
|
.from(projectWorkspaces)
|
|
.where(eq(projectWorkspaces.id, source.id));
|
|
if (
|
|
!currentProject ||
|
|
currentProject.archivedAt ||
|
|
(await effectiveProjectWorkspacePolicy(tx, input.companyId, currentProject.executionWorkspacePolicy, false))?.storageProviderPluginId !==
|
|
pluginId ||
|
|
!currentSource ||
|
|
currentSource.repoUrl !== source.repoUrl ||
|
|
currentSource.repoRef !== source.repoRef ||
|
|
currentSource.defaultRef !== source.defaultRef
|
|
) {
|
|
throw conflict(
|
|
"Project storage configuration changed; retry preparation",
|
|
);
|
|
}
|
|
const [task] = await tx
|
|
.select()
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.id, input.issueId!),
|
|
eq(issues.companyId, input.companyId),
|
|
),
|
|
)
|
|
.for("update");
|
|
if (!task || (input.fresh ? task.executionWorkspaceId !== input.fresh.previousWorkspaceId || task.workspaceBindingRevision !== input.expectedBindingRevision : Boolean(task.executionWorkspaceId)))
|
|
throw conflict("Task workspace binding changed; retry preparation");
|
|
const id = randomUUID();
|
|
await notifyTerminalWorkspaceWork(tx, `workspace:${id}`);
|
|
const [row] = await tx
|
|
.insert(executionWorkspaces)
|
|
.values({
|
|
id,
|
|
companyId: input.companyId,
|
|
projectId: project.id,
|
|
projectWorkspaceId: source.id,
|
|
sourceIssueId: input.issueId,
|
|
mode,
|
|
strategyType: strategy,
|
|
name: branchName ?? source.name,
|
|
providerType: "plugin",
|
|
providerRef: null,
|
|
cwd: null,
|
|
branchName,
|
|
repoUrl: source.repoUrl,
|
|
baseRef: input.baseRef ?? source.repoRef,
|
|
status: "idle",
|
|
metadata: {
|
|
workspaceProviderOwner: pluginId,
|
|
workspaceProviderReservationRunId: input.runId,
|
|
workspaceProviderSource: sourceInput,
|
|
workspaceProviderBranchMode: input.fresh ? "create" : input.branchMode ?? "create",
|
|
...(input.fresh ? { workspaceProviderFresh: input.fresh } : {}),
|
|
},
|
|
})
|
|
.returning();
|
|
await notifyStatusCardIssueChange(tx, task.companyId);
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionWorkspaceId: id,
|
|
executionWorkspacePreference: "reuse_existing",
|
|
workspaceBindingRevision: task.workspaceBindingRevision + 1,
|
|
workspacePendingSelection:
|
|
task.workspacePendingSelection?.expectedBindingRevision ===
|
|
task.workspaceBindingRevision
|
|
? {
|
|
...task.workspacePendingSelection,
|
|
expectedBindingRevision: task.workspaceBindingRevision + 1,
|
|
}
|
|
: task.workspacePendingSelection,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(issues.id, task.id));
|
|
return row!;
|
|
}));
|
|
const driverKey = input.environment?.driver === "plugin" ? input.environment.config?.driverKey : undefined;
|
|
const placement = selectedProvider.manifestJson.environmentDrivers?.some(driver =>
|
|
driver.driverKey === driverKey && driver.sessionHandoffVersion === 1)
|
|
? await input.acquireTaskLease?.({ id: workspace.id, mode: mode as ExecutionWorkspace["mode"] }) : undefined;
|
|
const prepare = (workspace: Workspace, taskLease?: TaskPlacement, replacement?: { leaseId: string; assertOwned: () => void }) => withExecutionWorkspaceLifecycleLock(db, workspace.id, async (tx) => {
|
|
(replacement?.assertOwned ?? input.assertOwned)?.();
|
|
const [current] = await tx
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, workspace.id));
|
|
if (
|
|
!current ||
|
|
!["active", "idle", "in_review"].includes(current.status) ||
|
|
current.metadata?.workspaceProviderRetired === true ||
|
|
current.metadata?.workspaceProviderRemoval ||
|
|
current.providerRef !== workspace.providerRef ||
|
|
(current.metadata?.lifecycleGeneration ?? 0) !==
|
|
(workspace.metadata?.lifecycleGeneration ?? 0)
|
|
) {
|
|
throw conflict("Workspace lifecycle changed; reopen before preparing");
|
|
}
|
|
if (taskLease) await authorize(replacement ? workspace.id : undefined);
|
|
const replacementScope = replacement && taskLease ? and(
|
|
eq(environmentLeases.id, replacement.leaseId),
|
|
eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.heartbeatRunId, input.runId),
|
|
eq(environmentLeases.executionWorkspaceId, workspace.id),
|
|
eq(environmentLeases.environmentId, taskLease.environmentId),
|
|
eq(environmentLeases.providerLeaseId, taskLease.providerLeaseId),
|
|
eq(environmentLeases.status, "active"),
|
|
sql`${environmentLeases.metadata}->>'pluginId' = ${pluginId} and ${environmentLeases.metadata}->>'driverKey' = ${taskLease.driverKey}`,
|
|
sql`${environmentLeases.releasedAt} is null and (${environmentLeases.expiresAt} is null or ${environmentLeases.expiresAt} > clock_timestamp())`,
|
|
sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
|
|
) : undefined;
|
|
if (replacementScope && !(await tx.select({ id: environmentLeases.id }).from(environmentLeases).where(replacementScope)).length)
|
|
throw conflict("Workspace Launcher placement changed");
|
|
if (!current.providerRef && !input.verifyExisting) {
|
|
const generation = current.metadata?.lifecycleGeneration ?? 0;
|
|
const intent = { generation, runId: input.runId, agentId: input.agentId, ...(taskLease ? { taskLease } : {}) };
|
|
const previous = readPreparationPlacement(current);
|
|
if (!isDeepStrictEqual(previous, intent)) {
|
|
// The request must survive a lost reply or rollback of the lifecycle transaction.
|
|
const [recorded] = await db.update(executionWorkspaces).set({
|
|
metadata: sql`coalesce(${executionWorkspaces.metadata}, '{}'::jsonb) || ${JSON.stringify({ workspaceProviderPreparationPlacement: intent })}::jsonb`,
|
|
}).where(and(eq(executionWorkspaces.id, current.id), eq(executionWorkspaces.companyId, current.companyId),
|
|
sql`${executionWorkspaces.providerRef} is null
|
|
and coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${generation}
|
|
and coalesce(${executionWorkspaces.metadata}->'workspaceProviderPreparationPlacement', 'null'::jsonb)
|
|
= ${JSON.stringify(previous ?? null)}::jsonb`)).returning({ id: executionWorkspaces.id });
|
|
if (!recorded) throw conflict("Workspace preparation placement changed");
|
|
}
|
|
}
|
|
const result = await callWorkspaceProvider(tx as unknown as Db, workers, {
|
|
workspace,
|
|
pluginId,
|
|
runId: input.runId,
|
|
agentId: input.agentId,
|
|
taskLease,
|
|
assertOwned: replacement?.assertOwned ?? input.assertOwned,
|
|
operation: {
|
|
kind: "prepare",
|
|
source: sourceInput,
|
|
mode: workspace.mode as "isolated_workspace",
|
|
strategy: workspace.strategyType as "git_worktree",
|
|
branchName: workspace.metadata?.workspaceProviderFresh ? null : workspace.branchName,
|
|
branchMode:
|
|
(workspace.metadata?.workspaceProviderBranchMode as
|
|
| "create"
|
|
| "existing"
|
|
| undefined) ??
|
|
input.branchMode ??
|
|
"create",
|
|
baseRef: workspace.baseRef,
|
|
existing: readWorkspaceProviderBinding(workspace.metadata),
|
|
...(input.verifyExisting ? { verifyExisting: true } : {}),
|
|
...(workspace.metadata?.workspaceProviderFresh ? { fresh: workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace } : {}),
|
|
},
|
|
});
|
|
if (result.kind !== "prepared")
|
|
throw conflict("Workspace preparation is incomplete");
|
|
if (replacement && result.branchName !== workspace.branchName)
|
|
throw conflict("Workspace Launcher prepared a different branch");
|
|
const savedBase = workspace.metadata?.workspaceProviderBaseSha;
|
|
if (input.verifyExisting && (result.created || !result.baseSha ||
|
|
(saved && result.binding.workspaceRef !== saved.workspaceRef) ||
|
|
(workspace.branchName !== null && result.branchName !== workspace.branchName) ||
|
|
(typeof savedBase === "string" && result.baseSha !== savedBase))) {
|
|
return null;
|
|
}
|
|
(replacement?.assertOwned ?? input.assertOwned)?.();
|
|
await authorize();
|
|
const [currentProject] = await tx
|
|
.select()
|
|
.from(projects)
|
|
.where(eq(projects.id, project.id));
|
|
if (!currentProject || currentProject.archivedAt)
|
|
throw conflict("Project was archived during workspace preparation");
|
|
const [task] = await tx
|
|
.select()
|
|
.from(issues)
|
|
.where(eq(issues.id, input.issueId!));
|
|
if (task?.executionWorkspaceId !== workspace.id)
|
|
throw conflict("Task workspace binding changed during preparation");
|
|
if (replacementScope) {
|
|
const [currentLease] = await tx.update(environmentLeases).set({
|
|
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ workspaceProvider: result.binding })}::jsonb`,
|
|
}).where(replacementScope).returning({ id: environmentLeases.id });
|
|
if (!currentLease) throw conflict("Workspace Launcher placement changed during preparation");
|
|
}
|
|
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
|
|
const { workspaceProviderPreparationPlacement: _placement, ...metadata } = current.metadata ?? {};
|
|
const [updated] = await tx
|
|
.update(executionWorkspaces)
|
|
.set({
|
|
providerRef: result.binding.workspaceRef,
|
|
branchName: result.branchName,
|
|
status: "active",
|
|
lastUsedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
metadata: {
|
|
...metadata,
|
|
workspaceProvider: result.binding,
|
|
workspaceProviderBaseSha: result.baseSha,
|
|
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
|
|
workspaceProviderSource: sourceInput,
|
|
workspaceProviderRemoved: false,
|
|
createdByRuntime: result.branchCreatedByRuntime,
|
|
},
|
|
})
|
|
.where(
|
|
and(
|
|
eq(executionWorkspaces.id, workspace.id),
|
|
eq(executionWorkspaces.companyId, input.companyId),
|
|
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${workspace.metadata?.lifecycleGeneration ?? 0}`,
|
|
sql`${executionWorkspaces.providerRef} IS NOT DISTINCT FROM ${workspace.providerRef}`,
|
|
),
|
|
)
|
|
.returning();
|
|
if (!updated)
|
|
throw conflict("Workspace changed before preparation could be recorded");
|
|
return updated!;
|
|
});
|
|
const prepared = await prepare(workspace, placement?.taskLease);
|
|
if (prepared && placement && input.onTaskPrepared) {
|
|
let preparedTaskId = placement.taskLease.providerLeaseId;
|
|
let preparedWorkspace = prepared;
|
|
input.onTaskPrepared((taskId, assertOwned) => {
|
|
if (taskId === preparedTaskId) return;
|
|
return prepare(preparedWorkspace, { ...placement.taskLease, providerLeaseId: taskId }, { leaseId: placement.leaseId, assertOwned }).then(updated => {
|
|
if (!updated) throw conflict("Workspace preparation is incomplete");
|
|
preparedWorkspace = updated;
|
|
preparedTaskId = taskId;
|
|
});
|
|
});
|
|
}
|
|
return prepared;
|
|
}
|
|
|
|
/** Recover a lost prepare result even when the originating run has already stopped. */
|
|
async function reconcilePluginWorkspaceReservation(
|
|
db: Db,
|
|
workers: WorkspaceWorkers | undefined,
|
|
snapshot: Workspace,
|
|
): Promise<Workspace> {
|
|
const [workspace] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(
|
|
and(
|
|
eq(executionWorkspaces.id, snapshot.id),
|
|
eq(executionWorkspaces.companyId, snapshot.companyId),
|
|
),
|
|
);
|
|
if (!workspace) throw conflict("Workspace is unavailable");
|
|
if (readWorkspaceProviderBinding(workspace.metadata) || workspace.metadata?.workspaceProviderRemoval ||
|
|
workspace.metadata?.workspaceProviderRemoved === true) return workspace;
|
|
const owner = workspace.metadata?.workspaceProviderOwner;
|
|
const retired = typeof owner === "string" ? await retireAbandonedPluginWorkspaceReservation(db, workspace) : null;
|
|
if (retired) return retired;
|
|
if (typeof owner !== "string" || !workspace.projectId || !workspace.metadata?.workspaceProviderSource)
|
|
throw conflict("Workspace storage reservation is unavailable");
|
|
const operation = workspaceProviderOperationSchema.parse({
|
|
kind: "prepare",
|
|
source: workspace.metadata.workspaceProviderSource,
|
|
mode: workspace.mode,
|
|
strategy: workspace.strategyType,
|
|
branchName: workspace.metadata.workspaceProviderFresh ? null : workspace.branchName,
|
|
branchMode: workspace.metadata.workspaceProviderBranchMode ?? "create",
|
|
baseRef: workspace.baseRef,
|
|
existing: null,
|
|
...(workspace.metadata?.workspaceProviderFresh ? { fresh: workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace } : {}),
|
|
});
|
|
if (operation.kind !== "prepare") throw conflict("Workspace preparation is unavailable");
|
|
const result = await withPluginWorkspaceReplay({
|
|
pluginId: owner, companyId: workspace.companyId, projectId: workspace.projectId,
|
|
projectWorkspaceId: operation.source.id, executionWorkspaceId: workspace.id,
|
|
}, () => callWorkspaceProvider(db, workers, {
|
|
workspace,
|
|
pluginId: owner,
|
|
...readPreparationPlacement(workspace),
|
|
operation,
|
|
}));
|
|
if (result.kind !== "prepared")
|
|
throw conflict("Workspace reconciliation is incomplete");
|
|
const { workspaceProviderPreparationPlacement: _placement, ...metadata } = workspace.metadata ?? {};
|
|
const [updated] = await db.transaction(async (tx) => {
|
|
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
|
|
return tx
|
|
.update(executionWorkspaces)
|
|
.set({
|
|
providerRef: result.binding.workspaceRef,
|
|
branchName: result.branchName,
|
|
updatedAt: new Date(),
|
|
metadata: {
|
|
...metadata,
|
|
workspaceProvider: result.binding,
|
|
workspaceProviderBaseSha: result.baseSha,
|
|
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
|
|
createdByRuntime: result.branchCreatedByRuntime,
|
|
},
|
|
})
|
|
.where(
|
|
and(
|
|
eq(executionWorkspaces.id, workspace.id),
|
|
eq(executionWorkspaces.companyId, workspace.companyId),
|
|
sql`${executionWorkspaces.providerRef} IS NULL`,
|
|
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${workspace.metadata?.lifecycleGeneration ?? 0}`,
|
|
),
|
|
)
|
|
.returning();
|
|
});
|
|
if (!updated)
|
|
throw conflict("Workspace changed during reconciliation; retry inspection");
|
|
return updated;
|
|
}
|
|
|
|
async function inspectBinding(
|
|
db: Db,
|
|
workers: WorkspaceWorkers | undefined,
|
|
workspace: Workspace,
|
|
recoveryPullRequest?: PluginWorkspaceRecoveryPullRequest,
|
|
) {
|
|
const binding = readWorkspaceProviderBinding(workspace.metadata);
|
|
if (!binding) throw conflict("Workspace storage binding is unavailable");
|
|
const result = await callWorkspaceProvider(db, workers, {
|
|
workspace,
|
|
operation: { kind: "inspect", binding, ...(recoveryPullRequest ? { recoveryPullRequest } : {}) },
|
|
});
|
|
if (result.kind !== "inspection")
|
|
throw conflict("Workspace inspection is incomplete");
|
|
return result;
|
|
}
|
|
|
|
/** Reconcile durable lifecycle intent under the same lock for either read path. */
|
|
async function withRetainedWorkspace<T>(db: Db, workers: WorkspaceWorkers | undefined,
|
|
workspace: Workspace, operation: (workspace: Workspace) => Promise<T>) {
|
|
return withExecutionWorkspaceLifecycleLock(db, workspace.id, async () => {
|
|
workspace = await reconcilePluginWorkspaceReservation(db, workers, workspace);
|
|
if (workspace.metadata?.workspaceProviderRestoration)
|
|
workspace = await finishPendingRestoration(db, workers, workspace);
|
|
if (workspace.metadata?.workspaceProviderRemoval)
|
|
workspace = await finishPendingRemoval(db, workers, workspace);
|
|
return operation(workspace);
|
|
});
|
|
}
|
|
|
|
export async function inspectPluginWorkspace(db: Db, workers: WorkspaceWorkers | undefined, workspace: Workspace, recoveryPullRequest?: PluginWorkspaceRecoveryPullRequest) {
|
|
return withRetainedWorkspace(db, workers, workspace, (current) =>
|
|
current.metadata?.workspaceProviderRemoved === true
|
|
? Promise.resolve(removedInspection()) : inspectBinding(db, workers, current, recoveryPullRequest));
|
|
}
|
|
|
|
type PluginWorkspaceFinalizationPlan = { kind: "inspect" } | { kind: "finalize"; taskLease?: TaskPlacement };
|
|
|
|
export async function resolvePluginWorkspaceFinalization(db: Db, workspace: Workspace,
|
|
context: { runId: string; agentId: string; taskLease?: TaskPlacement | null }): Promise<PluginWorkspaceFinalizationPlan | undefined> {
|
|
if (workspace.providerType !== "plugin") return undefined;
|
|
if (context.taskLease) return { kind: "finalize", taskLease: context.taskLease };
|
|
const binding = readWorkspaceProviderBinding(workspace.metadata);
|
|
if (!binding) throw conflict("Workspace storage ownership is missing");
|
|
const selected = await pluginRegistryService(db).getById(binding.pluginId);
|
|
if (!selected) throw conflict("The workspace storage plugin is unavailable");
|
|
// Existing providers continue to use their established inspection contract.
|
|
if (selected.manifestJson.workspaceProvider?.finalizationVersion !== 1) return { kind: "inspect" };
|
|
const portableDrivers = new Set(selected.manifestJson.environmentDrivers?.filter(driver =>
|
|
driver.sessionHandoffVersion === 1).map(driver => driver.driverKey));
|
|
let taskLease: TaskPlacement | undefined;
|
|
if (context.taskLease === undefined && portableDrivers.size > 0) {
|
|
const leases = await db.select().from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, workspace.companyId),
|
|
eq(environmentLeases.executionWorkspaceId, workspace.id),
|
|
eq(environmentLeases.heartbeatRunId, context.runId),
|
|
eq(environmentLeases.status, "active"),
|
|
));
|
|
const placements = leases.flatMap(lease => {
|
|
const driverKey = lease.metadata?.driverKey;
|
|
return lease.metadata?.pluginId === binding.pluginId && lease.metadata?.agentId === context.agentId &&
|
|
lease.metadata?.runnerTaskLease === true && lease.environmentId && lease.providerLeaseId &&
|
|
typeof driverKey === "string" && portableDrivers.has(driverKey)
|
|
? [{ driverKey, environmentId: lease.environmentId, providerLeaseId: lease.providerLeaseId }] : [];
|
|
});
|
|
if (placements.length > 1) throw conflict("Workspace Launcher placement is ambiguous");
|
|
taskLease = placements[0];
|
|
}
|
|
return { kind: "finalize", taskLease };
|
|
}
|
|
|
|
/** Completion does not need the deletion-safety proof supplied by inspect. */
|
|
export async function finalizePluginWorkspace(db: Db, workers: WorkspaceWorkers | undefined,
|
|
workspace: Workspace, context: { runId: string; agentId: string; taskLease?: TaskPlacement | null;
|
|
plan?: PluginWorkspaceFinalizationPlan }) {
|
|
return withRetainedWorkspace(db, workers, workspace, async (current) => {
|
|
if (current.metadata?.workspaceProviderRemoved === true)
|
|
throw conflict("Workspace was removed before finalization");
|
|
const binding = readWorkspaceProviderBinding(current.metadata);
|
|
if (!binding) throw conflict("Workspace storage ownership is missing");
|
|
if (context.plan && !isDeepStrictEqual(binding, readWorkspaceProviderBinding(workspace.metadata)))
|
|
throw conflict("Workspace storage ownership changed");
|
|
const plan = context.plan ?? await resolvePluginWorkspaceFinalization(db, current, context);
|
|
if (!plan) throw conflict("The workspace storage plugin is unavailable");
|
|
if (plan.kind === "inspect") return inspectBinding(db, workers, current);
|
|
const taskLease = plan.taskLease;
|
|
const result = await callWorkspaceProvider(db, workers, {
|
|
workspace: current, runId: context.runId, agentId: context.agentId, taskLease, operation: { kind: "finalize", binding },
|
|
});
|
|
if (result.kind !== "finalized") throw conflict("Workspace finalization is incomplete");
|
|
return result;
|
|
});
|
|
}
|
|
|
|
const removedInspection = () => ({
|
|
kind: "inspection" as const,
|
|
headSha: null,
|
|
branchName: null,
|
|
hasDirtyTrackedFiles: false,
|
|
hasUntrackedFiles: false,
|
|
hasLocalOnlyData: false,
|
|
isMergedIntoBase: null,
|
|
remotelyRecoverable: true,
|
|
});
|
|
|
|
async function finishPendingRestoration(
|
|
db: Db,
|
|
workers: WorkspaceWorkers | undefined,
|
|
workspace: Workspace,
|
|
): Promise<Workspace> {
|
|
const pending = workspace.metadata?.workspaceProviderRestoration as {
|
|
generation?: unknown;
|
|
operation?: unknown;
|
|
};
|
|
const operation = workspaceProviderOperationSchema.safeParse(
|
|
pending?.operation,
|
|
);
|
|
const generation =
|
|
(workspace.metadata?.lifecycleGeneration as number | undefined) ?? 0;
|
|
if (
|
|
!operation.success ||
|
|
operation.data.kind !== "prepare" ||
|
|
pending.generation !== generation + 1 ||
|
|
JSON.stringify(operation.data.existing) !==
|
|
JSON.stringify(readWorkspaceProviderBinding(workspace.metadata)) ||
|
|
!["archived", "cleanup_failed"].includes(workspace.status)
|
|
)
|
|
throw conflict("Workspace restoration intent is unavailable or stale");
|
|
const result = await callWorkspaceProvider(db, workers, {
|
|
workspace,
|
|
operation: operation.data,
|
|
advanceGeneration: true,
|
|
});
|
|
if (result.kind !== "prepared")
|
|
throw conflict("Workspace restoration is incomplete");
|
|
const [updated] = await db.transaction(async (tx) => {
|
|
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
|
|
return tx
|
|
.update(executionWorkspaces)
|
|
.set({
|
|
providerRef: result.binding.workspaceRef,
|
|
branchName: result.branchName,
|
|
updatedAt: new Date(),
|
|
metadata: {
|
|
...workspace.metadata,
|
|
lifecycleGeneration: generation + 1,
|
|
workspaceProvider: result.binding,
|
|
workspaceProviderBaseSha: result.baseSha,
|
|
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
|
|
workspaceProviderRemoved: false,
|
|
workspaceProviderRestoration: null,
|
|
},
|
|
})
|
|
.where(
|
|
and(
|
|
eq(executionWorkspaces.id, workspace.id),
|
|
eq(executionWorkspaces.companyId, workspace.companyId),
|
|
eq(executionWorkspaces.status, workspace.status),
|
|
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${generation}`,
|
|
sql`${executionWorkspaces.metadata}->'workspaceProviderRestoration' = ${JSON.stringify(pending)}::jsonb`,
|
|
),
|
|
)
|
|
.returning();
|
|
});
|
|
if (!updated)
|
|
throw conflict("Workspace changed during restoration reconciliation");
|
|
return updated;
|
|
}
|
|
|
|
/** Replay the recorded delete before trying to inspect a checkout that may already be gone. */
|
|
async function finishPendingRemoval(
|
|
db: Db,
|
|
workers: WorkspaceWorkers | undefined,
|
|
workspace: Workspace,
|
|
): Promise<Workspace> {
|
|
if (workspace.metadata?.workspaceProviderRemoved === true) return workspace;
|
|
const pending = workspace.metadata?.workspaceProviderRemoval as
|
|
| { generation?: unknown; operation?: unknown; operationId?: unknown }
|
|
| undefined;
|
|
const parsed = workspaceProviderOperationSchema.safeParse(pending?.operation);
|
|
const binding = readWorkspaceProviderBinding(workspace.metadata);
|
|
if (
|
|
!parsed.success ||
|
|
parsed.data.kind !== "remove" ||
|
|
pending?.generation !== (workspace.metadata?.lifecycleGeneration ?? 0) ||
|
|
JSON.stringify(parsed.data.binding) !== JSON.stringify(binding)
|
|
)
|
|
throw conflict("Workspace removal intent is unavailable or stale");
|
|
if (pending.operationId === undefined && parsed.data.recoveryPullRequest)
|
|
throw conflict("Workspace removal identity is ambiguous; reconciliation requires the original request");
|
|
const operationId = createHash("sha256").update(JSON.stringify([workspace.id, pending.generation, parsed.data])).digest("hex");
|
|
if (pending.operationId !== undefined && pending.operationId !== operationId)
|
|
throw conflict("Workspace removal identity does not match its recorded payload");
|
|
const owner = workspace.metadata?.workspaceProviderOwner;
|
|
const result = await callWorkspaceProvider(db, workers, {
|
|
workspace,
|
|
pluginId: typeof owner === "string" ? owner : undefined,
|
|
operation: parsed.data,
|
|
removalOperationId: operationId,
|
|
});
|
|
if (result.kind !== "removed")
|
|
throw conflict("Workspace removal is incomplete");
|
|
if (workspace.metadata?.workspaceProviderFresh)
|
|
await releaseProviderRecoveryTransfers(workspace.companyId, workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace);
|
|
const [updated] = await db.transaction(async (tx) => {
|
|
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
|
|
return tx
|
|
.update(executionWorkspaces)
|
|
.set({
|
|
...(workspace.status === "cleanup_failed"
|
|
? { status: "archived", cleanupReason: null }
|
|
: {}),
|
|
metadata: {
|
|
...workspace.metadata,
|
|
workspaceProviderRemoval: null,
|
|
workspaceProviderRemoved: true,
|
|
},
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
and(
|
|
eq(executionWorkspaces.id, workspace.id),
|
|
eq(executionWorkspaces.companyId, workspace.companyId),
|
|
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${pending!.generation}`,
|
|
),
|
|
)
|
|
.returning();
|
|
});
|
|
if (!updated)
|
|
throw conflict("Workspace changed during removal reconciliation");
|
|
return updated;
|
|
}
|
|
|
|
export async function removePluginWorkspace(
|
|
db: Db,
|
|
workers: WorkspaceWorkers | undefined,
|
|
workspace: Workspace,
|
|
expectedHeadSha: string | null,
|
|
recoveryPullRequest?: PluginWorkspaceRecoveryPullRequest,
|
|
beforeRemove?: () => Promise<void>,
|
|
) {
|
|
workspace = await reconcilePluginWorkspaceReservation(db, workers, workspace);
|
|
if (workspace.metadata?.workspaceProviderRestoration)
|
|
throw conflict("Workspace restoration must be reconciled before removal");
|
|
if (workspace.metadata?.workspaceProviderRemoved === true) {
|
|
if (workspace.metadata?.workspaceProviderFresh)
|
|
await releaseProviderRecoveryTransfers(workspace.companyId, workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace);
|
|
return;
|
|
}
|
|
if (workspace.metadata?.workspaceProviderRemoval) {
|
|
await finishPendingRemoval(db, workers, workspace);
|
|
return;
|
|
}
|
|
const binding = readWorkspaceProviderBinding(workspace.metadata);
|
|
if (!binding) throw conflict("Workspace storage binding is unavailable");
|
|
const inspection = await inspectBinding(db, workers, workspace, recoveryPullRequest);
|
|
if (
|
|
inspection.hasDirtyTrackedFiles ||
|
|
inspection.hasUntrackedFiles ||
|
|
inspection.hasLocalOnlyData ||
|
|
inspection.headSha !== expectedHeadSha ||
|
|
(inspection.remotelyRecoverable !== true &&
|
|
inspection.isMergedIntoBase !== true)
|
|
) {
|
|
throw conflict(
|
|
"Workspace has local changes or cannot be recovered remotely",
|
|
);
|
|
}
|
|
await beforeRemove?.();
|
|
const generation = workspace.metadata?.lifecycleGeneration ?? 0;
|
|
const selectedProvider = await requireWorkspaceProviderWorker(db, workers, workspace.companyId, binding.pluginId);
|
|
const operation = workspaceProviderOperationSchema.parse({ kind: "remove", binding, expectedHeadSha,
|
|
...(recoveryPullRequest && selectedProvider.manifestJson.workspaceProvider?.recoveryPullRequestVersion === 1 ? { recoveryPullRequest } : {}) });
|
|
const operationId = createHash("sha256").update(JSON.stringify([workspace.id, generation, operation])).digest("hex");
|
|
const [pending] = await db.transaction(async (tx) => {
|
|
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
|
|
return tx
|
|
.update(executionWorkspaces)
|
|
.set({
|
|
metadata: {
|
|
...workspace.metadata,
|
|
workspaceProviderRemoval: { generation, operation, operationId },
|
|
},
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
and(
|
|
eq(executionWorkspaces.id, workspace.id),
|
|
eq(executionWorkspaces.companyId, workspace.companyId),
|
|
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${generation}`,
|
|
eq(executionWorkspaces.providerRef, binding.workspaceRef),
|
|
),
|
|
)
|
|
.returning();
|
|
});
|
|
if (!pending)
|
|
throw conflict("Workspace changed before removal could be recorded");
|
|
await finishPendingRemoval(db, workers, pending);
|
|
}
|
|
|
|
/** An authorized storage edit atomically retires closed, removed dependencies under the reopen fence. */
|
|
export async function retireRemovedPluginWorkspaceDependents(
|
|
db: Db,
|
|
projectId: string,
|
|
sourceId?: string,
|
|
) {
|
|
const rows = await db
|
|
.select({ id: executionWorkspaces.id })
|
|
.from(executionWorkspaces)
|
|
.where(
|
|
and(
|
|
eq(executionWorkspaces.projectId, projectId),
|
|
eq(executionWorkspaces.providerType, "plugin"),
|
|
sourceId
|
|
? eq(executionWorkspaces.projectWorkspaceId, sourceId)
|
|
: undefined,
|
|
),
|
|
);
|
|
for (const { id } of rows.sort((a, b) => a.id.localeCompare(b.id))) {
|
|
const acquired = await db.execute(
|
|
sql`select pg_try_advisory_xact_lock(hashtextextended(${`execution_workspace_lifecycle:${id}`}, 0)) as acquired`,
|
|
);
|
|
if (!acquired[0]?.acquired)
|
|
throw conflict("Workspace lifecycle is busy; retry the source edit");
|
|
const [row] = await db
|
|
.select()
|
|
.from(executionWorkspaces)
|
|
.where(eq(executionWorkspaces.id, id));
|
|
if (!row || row.metadata?.workspaceProviderRetired === true) continue;
|
|
if (
|
|
row.metadata?.workspaceProviderRemoved !== true ||
|
|
row.metadata?.workspaceProviderRestoration ||
|
|
!["archived", "cleanup_failed"].includes(row.status)
|
|
) {
|
|
throw conflict(
|
|
"Retire managed execution workspaces before changing their storage owner or repository source",
|
|
);
|
|
}
|
|
await db
|
|
.update(executionWorkspaces)
|
|
.set({
|
|
metadata: { ...row.metadata, workspaceProviderRetired: true },
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(executionWorkspaces.id, row.id));
|
|
}
|
|
}
|
|
|
|
/** Called inside Core's existing lifecycle fence after task authorization. */
|
|
export async function restorePluginWorkspace(
|
|
db: Db,
|
|
workers: WorkspaceWorkers | undefined,
|
|
workspace: Workspace,
|
|
) {
|
|
workspace = await reconcilePluginWorkspaceReservation(db, workers, workspace);
|
|
if (workspace.metadata?.workspaceProviderRetired === true)
|
|
throw conflict("Workspace storage was retired");
|
|
if (workspace.metadata?.workspaceProviderRemoval)
|
|
workspace = await finishPendingRemoval(db, workers, workspace);
|
|
const binding = readWorkspaceProviderBinding(workspace.metadata);
|
|
const source = workspace.metadata?.workspaceProviderSource;
|
|
if (!binding || !source)
|
|
throw conflict("Workspace restoration recipe is unavailable");
|
|
const operation = workspaceProviderOperationSchema.parse({
|
|
kind: "prepare",
|
|
source,
|
|
mode: workspace.mode,
|
|
strategy: workspace.strategyType,
|
|
branchName: workspace.metadata?.workspaceProviderFresh ? null : workspace.branchName,
|
|
branchMode: workspace.metadata?.workspaceProviderFresh ? "create" : "existing",
|
|
baseRef: workspace.baseRef,
|
|
existing: binding,
|
|
...(workspace.metadata?.workspaceProviderFresh ? { fresh: workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace } : {}),
|
|
});
|
|
const [pending] = await db.transaction(async (tx) => {
|
|
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
|
|
return tx
|
|
.update(executionWorkspaces)
|
|
.set({
|
|
metadata: {
|
|
...workspace.metadata,
|
|
workspaceProviderRemoved: false,
|
|
workspaceProviderRestoration: {
|
|
generation:
|
|
((workspace.metadata?.lifecycleGeneration as number) ?? 0) + 1,
|
|
operation,
|
|
},
|
|
},
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
and(
|
|
eq(executionWorkspaces.id, workspace.id),
|
|
eq(executionWorkspaces.companyId, workspace.companyId),
|
|
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${workspace.metadata?.lifecycleGeneration ?? 0}`,
|
|
),
|
|
)
|
|
.returning();
|
|
});
|
|
if (!pending)
|
|
throw conflict("Workspace changed before restoration could be recorded");
|
|
workspace = pending;
|
|
const result = await callWorkspaceProvider(db, workers, {
|
|
workspace,
|
|
operation,
|
|
advanceGeneration: true,
|
|
});
|
|
if (result.kind !== "prepared")
|
|
throw conflict("Workspace restoration is incomplete");
|
|
return {
|
|
...workspace.metadata,
|
|
workspaceProvider: result.binding,
|
|
workspaceProviderRemoved: false,
|
|
workspaceProviderRestoration: null,
|
|
workspaceProviderBaseSha: result.baseSha,
|
|
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
|
|
};
|
|
}
|