Files
PaperClipAI/server/src/services/plugin-workspaces.ts
T
Nicky LeachandPaperclip 7f69ef8b44 Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path

> - Paperclip lets plugins own managed Git workspaces.
> - A portable Runner task can move to a different physical allocation.
> - The checkout must be present on the allocation that launches the
Runner.
> - Workspace preparation and finalization need exact placement when
they retry.
> - This change checks the placement at Runner admission and records it
before finalization.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Workspace preparation and finalization for portable Runner tasks.

**Current behavior**
A task replacement can change the allocation after Git preparation. A
lost finalization reply can leave retry without the original placement.

**Proposed behavior**
Prepare the checkout on the actual allocation before admission. Record
finalization placement so an exact retry can use it.

Portable ownership and controller recovery are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066).

## What Changed

- Reserve the managed workspace before acquiring the portable task
lease.
- Pass the task placement to the workspace provider. Reuse the described
task while placement stays the same and prepare again before admitting a
replacement.
- Record the initial preparation's run, agent, generation, and optional
task placement before dispatch. Inspection and retries on the same
Launcher replay the same request; VM and sandbox preparation use the
same recovery path.
- Keep provider request identity separate from Git credential authority:
reservation reconciliation uses the existing project-consent path in a
host-only scope bound to the plugin and workspace. Foreground
preparation still requires an active run; consent, repository, and
revocation checks are unchanged.
- Acquire the current Launcher before retrying preparation. If placement
changes, use ordinary preparation on the replacement so the provider can
settle the prior writer and recover the retained checkout. No additional
provider discovery call is added.
- Select finalization placement before publishing the operation record,
then dispatch and recover using that saved choice. Retain the Launcher
during bounded finalization retries.
- Preserve interrupted pre-upgrade finalization requests without
placement, including their original operation IDs and provider routing.
No migration or new durable mode flag is required.

## Verification

- Final revision `74ea3e6712`: server typecheck and exact SDK source-pin
verification (538 inputs) passed.
- All 185 tests passed across the full plugin worker manager, project
Git authorization, and plugin workspace integration suites (including 56
workspace tests). Independent verification reran the three lost-reply
cases: inspection, same-Launcher retry, and replacement after deleting
the original Launcher. The tests assert the correct request identity,
durable intent, and clearing that intent after success.
- Before the final retry-accounting fix, all 323 tests passed across
workspace admission, runtime execution, native workspace finalization,
plugin workspace integration, and terminal cleanup retry. Regressions
exercise crashes after operation insertion but before dispatch in both
finalization paths, exact replay after lease release, and retry after
ambiguous placement clears without publishing an incomplete operation.
Pre-upgrade interrupted requests retain their operation IDs despite an
available portable lease.
- Final retry-accounting fix: all 84 native workspace finalizer,
failure, and supersession tests passed. A real missing-plugin regression
first failed on the prior commit, then verified healthy-run progress,
three-attempt exhaustion, terminal cleanup, and absence of a misleading
workspace operation. Independent verification also passed the legacy and
crash-before-dispatch cases.
- Worker-process tests cover both preparation RPC methods: the provider
still receives the original run and agent, maintenance receives
project-scoped credential authority, and foreground or mismatched-scope
requests remain denied. Database tests confirm a stopped run cannot
authorize foreground Git access while valid project consent can
authorize maintenance.
- Existing metadata protection coverage proves callers cannot overwrite
the private preparation intent or read it through workspace responses.
- Two existing Launcher Git tests passed: retry preserves dirty files
and the original base commit without fetching, and retry resumes the
saved checkout intent while rejecting a changed pending branch.
- Implementation was delegated to GPT-6.1 Sol and independently
reviewed. The GitHub stack's automatic rebase preserved the original
workspace-placement patch.

- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706)
passed on `74ea3e6712`, including server and workspace tests, Runner,
end-to-end tests, typecheck, build, and canary dry run. Ordinary
Greptile and
[Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388)
both scored `74ea3e6712` 5/5, with no unresolved findings.

## Risks

- Live Daytona end-to-end qualification is pending. The checkout record
persists on the mounted volume; replacement preparation relies on the
provider confirming the previous writer is settled before using that
retained checkout. Local tests cover Core request routing and Launcher's
durable checkout recovery separately.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 12:12:41 -08:00

998 lines
43 KiB
TypeScript

import { isDeepStrictEqual } from "node:util";
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
import { assertProjectStorageEnvironment, effectiveProjectWorkspacePolicy, requireWorkspaceProviderWorker, type WorkspaceWorkers } from "./project-workspace-provider.js";
import { releaseProviderRecoveryTransfers } from "./provider-fresh-workspace.js";
import { retireAbandonedPluginWorkspaceReservation } from "./plugin-workspace-reservations.js";
import { randomUUID, createHash } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import {
environmentLeases,
executionWorkspaces,
heartbeatRuns,
issues,
projects,
projectWorkspaces,
type Db,
} from "@paperclipai/db";
import {
readWorkspaceProviderBinding,
type WorkspaceProviderBinding,
type ExecutionWorkspace,
} from "@paperclipai/shared";
import {
pluginWorkspaceParamsSchema,
workspaceProviderOperationSchema,
type PluginWorkspaceOperation,
type PluginWorkspaceRecoveryPullRequest,
type PluginWorkspaceParams,
type PluginFreshWorkspace,
} from "@paperclipai/plugin-sdk";
import { callWorkspaceProvider } from "./plugin-workspace-preparation.js";
import { withPluginWorkspaceReplay } from "./plugin-workspace-replay-context.js";
import { pluginRegistryService } from "./plugin-registry.js";
import { notifyTerminalWorkspaceWork } from "./terminal-workspace-work.js";
import { withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifecycle-lock.js";
import {
assertTaskWorkspaceExecutionAccess,
assertTaskWorkspaceSourceProjectAccess,
} from "./task-workspace-access.js";
import { forbidden, conflict } from "../errors.js";
export { callWorkspaceProvider } from "./plugin-workspace-preparation.js";
export type { WorkspaceWorkers } from "./project-workspace-provider.js";
type Workspace = typeof executionWorkspaces.$inferSelect;
type TaskPlacement = NonNullable<PluginWorkspaceParams["taskLease"]>;
const preparationPlacementSchema = pluginWorkspaceParamsSchema.pick({
generation: true, runId: true, agentId: true, taskLease: true,
});
function readPreparationPlacement(workspace: Workspace) {
const saved = workspace.metadata?.workspaceProviderPreparationPlacement;
if (saved === undefined) return undefined;
const placement = preparationPlacementSchema.parse(saved);
if (!placement.runId || !placement.agentId ||
placement.generation !== (workspace.metadata?.lifecycleGeneration ?? 0))
throw conflict("Workspace preparation placement changed");
return placement;
}
type EnsureTaskWorkspace = (taskId: string, assertOwned: () => void) => Promise<void> | undefined;
export async function pluginWorkspaceSupportsMergeAncestry(db: Db, workers: WorkspaceWorkers | undefined, workspace: Workspace) {
const pluginId = readWorkspaceProviderBinding(workspace.metadata)?.pluginId;
if (!pluginId) return true;
const selected = await requireWorkspaceProviderWorker(db, workers, workspace.companyId, pluginId);
return selected.manifestJson.workspaceProvider?.supportsMergeAncestry !== false;
}
export async function preparePluginWorkspace(
db: Db,
workers: WorkspaceWorkers | undefined,
input: {
companyId: string;
projectId: string | null;
projectWorkspaceId: string | null;
issueId: string | null;
runId: string;
agentId: string;
responsibleUserId: string | null;
existingWorkspaceId: string | null;
mode: string;
strategy: string;
selectedProviderPluginId?: string | null;
environment?: Pick<import("@paperclipai/shared").Environment, "driver" | "config" | "name"> | null;
branchMode?: "create" | "existing";
branchName?: string | null;
baseRef?: string | null;
fresh?: PluginFreshWorkspace;
expectedBindingRevision?: number;
verifyExisting?: true;
acquireTaskLease?: (workspace: Pick<ExecutionWorkspace, "id" | "mode">) => Promise<{ leaseId: string; taskLease: TaskPlacement }>;
onTaskPrepared?: (ensure: EnsureTaskWorkspace) => void;
},
) {
const [previous] = input.existingWorkspaceId
? await db
.select()
.from(executionWorkspaces)
.where(
and(
eq(executionWorkspaces.id, input.existingWorkspaceId),
eq(executionWorkspaces.companyId, input.companyId),
),
)
: [];
const [reservation] = input.fresh ? await db.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, input.companyId),
eq(executionWorkspaces.sourceIssueId, input.issueId!),
sql`${executionWorkspaces.metadata}#>>'{workspaceProviderFresh,attemptId}' = ${input.fresh.attemptId}`,
)).limit(1) : [];
const existing = input.fresh ? reservation : previous;
if (input.verifyExisting && (!existing || existing.metadata?.workspaceProviderRemoved === true ||
existing.metadata?.workspaceProviderRestoration)) return null;
const prior = existing ?? previous;
if (
existing?.metadata?.workspaceProviderRetired === true ||
existing?.metadata?.workspaceProviderRemoval
)
throw conflict("Workspace storage is retired or awaiting cleanup");
if (!existing && (!input.projectId || !input.projectWorkspaceId)) return null;
const saved = readWorkspaceProviderBinding(existing?.metadata);
const priorBinding = readWorkspaceProviderBinding(prior?.metadata);
const mode = input.fresh ? "isolated_workspace" : existing?.mode ?? input.mode,
strategy = existing?.strategyType ?? input.strategy;
const projectId = priorBinding?.projectId ?? input.projectId;
const [project] = projectId
? await db
.select()
.from(projects)
.where(
and(
eq(projects.id, projectId),
eq(projects.companyId, input.companyId),
),
)
: [];
if (prior && prior.providerType !== "plugin" && !priorBinding && !prior.metadata?.workspaceProviderOwner) return null;
const owner = priorBinding?.pluginId ?? prior?.metadata?.workspaceProviderOwner ??
(!existing && input.selectedProviderPluginId !== undefined ? input.selectedProviderPluginId :
!existing && project ? (await effectiveProjectWorkspacePolicy(db, input.companyId, project.executionWorkspacePolicy, false))?.storageProviderPluginId : null);
const pluginId = typeof owner === "string" ? owner : null;
if (!pluginId) {
if (existing?.providerType === "plugin")
throw conflict("Workspace storage ownership is missing");
return null;
}
if (
!input.issueId ||
!project ||
project.archivedAt ||
!input.projectWorkspaceId
)
throw conflict(
"Managed storage requires a task and an available project source",
);
if (
!["shared_workspace", "isolated_workspace", "operator_branch"].includes(
input.mode,
) ||
!["project_primary", "git_worktree"].includes(strategy)
)
throw conflict("Unsupported managed workspace strategy");
if (existing && existing.providerType !== "plugin")
throw conflict(
"An existing local workspace cannot change storage providers",
);
const selectedProvider = await requireWorkspaceProviderWorker(db, workers, input.companyId, pluginId, true);
if (input.environment) assertProjectStorageEnvironment(selectedProvider, input.environment);
const declaration = selectedProvider.manifestJson.workspaceProvider!;
if (input.fresh && declaration.freshWorkspaceVersion !== 1)
throw conflict("A fresh attempt is unavailable for this workspace storage provider. The existing workspace is retained.", { reason: "fresh_workspace_storage_unsupported" });
if (
!declaration.modes.includes(mode as "isolated_workspace") ||
!declaration.strategies.includes(strategy as "git_worktree")
) {
throw conflict(
"The storage plugin does not support this workspace mode or strategy",
);
}
const actor = {
type: "agent" as const,
source: "agent_jwt" as const,
companyId: input.companyId,
agentId: input.agentId,
runId: input.runId,
onBehalfOfUserId:
input.responsibleUserId === "local-board"
? null
: input.responsibleUserId,
};
const authorize = async (workspaceId = existing?.id) => {
await assertTaskWorkspaceExecutionAccess(db, actor, input.companyId, {
issueId: input.issueId!,
...(workspaceId
? { workspaceId }
: { projectWorkspaceId: input.projectWorkspaceId }),
});
await assertTaskWorkspaceSourceProjectAccess(
db,
actor,
input.companyId,
project.id,
);
const [run] = await db
.select()
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.id, input.runId),
eq(heartbeatRuns.companyId, input.companyId),
),
);
if (
!run ||
run.agentId !== input.agentId ||
![run.issueId, run.nativeIssueId].includes(input.issueId!) ||
run.status !== "running"
)
throw forbidden("Workspace preparation no longer has an active run");
};
await authorize();
const [source] = await db
.select()
.from(projectWorkspaces)
.where(
and(
eq(
projectWorkspaces.id,
saved?.projectWorkspaceId ?? input.projectWorkspaceId,
),
eq(projectWorkspaces.projectId, project.id),
eq(projectWorkspaces.companyId, input.companyId),
),
);
if (!source) throw conflict("Workspace repository source is unavailable");
const sourceInput = (prior?.metadata?.workspaceProviderSource as
| Extract<PluginWorkspaceOperation, { kind: "prepare" }>["source"]
| undefined) ?? {
id: source.id,
revision:
saved?.sourceRevision ??
createHash("sha256")
.update(
JSON.stringify([
source.id,
source.repoUrl,
source.repoRef,
source.defaultRef,
]),
)
.digest("hex"),
repoUrl: existing?.repoUrl ?? source.repoUrl,
repoRef: existing?.baseRef ?? source.repoRef,
defaultRef: source.defaultRef,
};
const branchName = input.fresh ? null : existing
? existing.branchName
: (input.branchName ??
(strategy === "git_worktree" ? `workspace-${randomUUID()}` : null));
// Reserve and bind before calling the provider. A crash must not orphan a created checkout.
const workspace =
existing ??
(await db.transaction(async (tx) => {
const [currentProject] = await tx
.select()
.from(projects)
.where(eq(projects.id, project.id))
.for("update");
const [currentSource] = await tx
.select()
.from(projectWorkspaces)
.where(eq(projectWorkspaces.id, source.id));
if (
!currentProject ||
currentProject.archivedAt ||
(await effectiveProjectWorkspacePolicy(tx, input.companyId, currentProject.executionWorkspacePolicy, false))?.storageProviderPluginId !==
pluginId ||
!currentSource ||
currentSource.repoUrl !== source.repoUrl ||
currentSource.repoRef !== source.repoRef ||
currentSource.defaultRef !== source.defaultRef
) {
throw conflict(
"Project storage configuration changed; retry preparation",
);
}
const [task] = await tx
.select()
.from(issues)
.where(
and(
eq(issues.id, input.issueId!),
eq(issues.companyId, input.companyId),
),
)
.for("update");
if (!task || (input.fresh ? task.executionWorkspaceId !== input.fresh.previousWorkspaceId || task.workspaceBindingRevision !== input.expectedBindingRevision : Boolean(task.executionWorkspaceId)))
throw conflict("Task workspace binding changed; retry preparation");
const id = randomUUID();
await notifyTerminalWorkspaceWork(tx, `workspace:${id}`);
const [row] = await tx
.insert(executionWorkspaces)
.values({
id,
companyId: input.companyId,
projectId: project.id,
projectWorkspaceId: source.id,
sourceIssueId: input.issueId,
mode,
strategyType: strategy,
name: branchName ?? source.name,
providerType: "plugin",
providerRef: null,
cwd: null,
branchName,
repoUrl: source.repoUrl,
baseRef: input.baseRef ?? source.repoRef,
status: "idle",
metadata: {
workspaceProviderOwner: pluginId,
workspaceProviderReservationRunId: input.runId,
workspaceProviderSource: sourceInput,
workspaceProviderBranchMode: input.fresh ? "create" : input.branchMode ?? "create",
...(input.fresh ? { workspaceProviderFresh: input.fresh } : {}),
},
})
.returning();
await notifyStatusCardIssueChange(tx, task.companyId);
await tx
.update(issues)
.set({
executionWorkspaceId: id,
executionWorkspacePreference: "reuse_existing",
workspaceBindingRevision: task.workspaceBindingRevision + 1,
workspacePendingSelection:
task.workspacePendingSelection?.expectedBindingRevision ===
task.workspaceBindingRevision
? {
...task.workspacePendingSelection,
expectedBindingRevision: task.workspaceBindingRevision + 1,
}
: task.workspacePendingSelection,
updatedAt: new Date(),
})
.where(eq(issues.id, task.id));
return row!;
}));
const driverKey = input.environment?.driver === "plugin" ? input.environment.config?.driverKey : undefined;
const placement = selectedProvider.manifestJson.environmentDrivers?.some(driver =>
driver.driverKey === driverKey && driver.sessionHandoffVersion === 1)
? await input.acquireTaskLease?.({ id: workspace.id, mode: mode as ExecutionWorkspace["mode"] }) : undefined;
const prepare = (workspace: Workspace, taskLease?: TaskPlacement, replacement?: { leaseId: string; assertOwned: () => void }) => withExecutionWorkspaceLifecycleLock(db, workspace.id, async (tx) => {
replacement?.assertOwned();
const [current] = await tx
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, workspace.id));
if (
!current ||
!["active", "idle", "in_review"].includes(current.status) ||
current.metadata?.workspaceProviderRetired === true ||
current.metadata?.workspaceProviderRemoval ||
current.providerRef !== workspace.providerRef ||
(current.metadata?.lifecycleGeneration ?? 0) !==
(workspace.metadata?.lifecycleGeneration ?? 0)
) {
throw conflict("Workspace lifecycle changed; reopen before preparing");
}
if (taskLease) await authorize(replacement ? workspace.id : undefined);
const replacementScope = replacement && taskLease ? and(
eq(environmentLeases.id, replacement.leaseId),
eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.heartbeatRunId, input.runId),
eq(environmentLeases.executionWorkspaceId, workspace.id),
eq(environmentLeases.environmentId, taskLease.environmentId),
eq(environmentLeases.providerLeaseId, taskLease.providerLeaseId),
eq(environmentLeases.status, "active"),
sql`${environmentLeases.metadata}->>'pluginId' = ${pluginId} and ${environmentLeases.metadata}->>'driverKey' = ${taskLease.driverKey}`,
sql`${environmentLeases.releasedAt} is null and (${environmentLeases.expiresAt} is null or ${environmentLeases.expiresAt} > clock_timestamp())`,
sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
) : undefined;
if (replacementScope && !(await tx.select({ id: environmentLeases.id }).from(environmentLeases).where(replacementScope)).length)
throw conflict("Workspace Launcher placement changed");
if (!current.providerRef && !input.verifyExisting) {
const generation = current.metadata?.lifecycleGeneration ?? 0;
const intent = { generation, runId: input.runId, agentId: input.agentId, ...(taskLease ? { taskLease } : {}) };
const previous = readPreparationPlacement(current);
if (!isDeepStrictEqual(previous, intent)) {
// The request must survive a lost reply or rollback of the lifecycle transaction.
const [recorded] = await db.update(executionWorkspaces).set({
metadata: sql`coalesce(${executionWorkspaces.metadata}, '{}'::jsonb) || ${JSON.stringify({ workspaceProviderPreparationPlacement: intent })}::jsonb`,
}).where(and(eq(executionWorkspaces.id, current.id), eq(executionWorkspaces.companyId, current.companyId),
sql`${executionWorkspaces.providerRef} is null
and coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${generation}
and coalesce(${executionWorkspaces.metadata}->'workspaceProviderPreparationPlacement', 'null'::jsonb)
= ${JSON.stringify(previous ?? null)}::jsonb`)).returning({ id: executionWorkspaces.id });
if (!recorded) throw conflict("Workspace preparation placement changed");
}
}
const result = await callWorkspaceProvider(tx as unknown as Db, workers, {
workspace,
pluginId,
runId: input.runId,
agentId: input.agentId,
taskLease,
assertOwned: replacement?.assertOwned,
operation: {
kind: "prepare",
source: sourceInput,
mode: workspace.mode as "isolated_workspace",
strategy: workspace.strategyType as "git_worktree",
branchName: workspace.metadata?.workspaceProviderFresh ? null : workspace.branchName,
branchMode:
(workspace.metadata?.workspaceProviderBranchMode as
| "create"
| "existing"
| undefined) ??
input.branchMode ??
"create",
baseRef: workspace.baseRef,
existing: readWorkspaceProviderBinding(workspace.metadata),
...(input.verifyExisting ? { verifyExisting: true } : {}),
...(workspace.metadata?.workspaceProviderFresh ? { fresh: workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace } : {}),
},
});
if (result.kind !== "prepared")
throw conflict("Workspace preparation is incomplete");
if (replacement && result.branchName !== workspace.branchName)
throw conflict("Workspace Launcher prepared a different branch");
const savedBase = workspace.metadata?.workspaceProviderBaseSha;
if (input.verifyExisting && (result.created || !result.baseSha ||
(saved && result.binding.workspaceRef !== saved.workspaceRef) ||
(workspace.branchName !== null && result.branchName !== workspace.branchName) ||
(typeof savedBase === "string" && result.baseSha !== savedBase))) {
throw conflict("The existing checkout could not be verified");
}
await authorize();
const [currentProject] = await tx
.select()
.from(projects)
.where(eq(projects.id, project.id));
if (!currentProject || currentProject.archivedAt)
throw conflict("Project was archived during workspace preparation");
const [task] = await tx
.select()
.from(issues)
.where(eq(issues.id, input.issueId!));
if (task?.executionWorkspaceId !== workspace.id)
throw conflict("Task workspace binding changed during preparation");
if (replacementScope) {
const [currentLease] = await tx.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ workspaceProvider: result.binding })}::jsonb`,
}).where(replacementScope).returning({ id: environmentLeases.id });
if (!currentLease) throw conflict("Workspace Launcher placement changed during preparation");
}
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
const { workspaceProviderPreparationPlacement: _placement, ...metadata } = current.metadata ?? {};
const [updated] = await tx
.update(executionWorkspaces)
.set({
providerRef: result.binding.workspaceRef,
branchName: result.branchName,
status: "active",
lastUsedAt: new Date(),
updatedAt: new Date(),
metadata: {
...metadata,
workspaceProvider: result.binding,
workspaceProviderBaseSha: result.baseSha,
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
workspaceProviderSource: sourceInput,
workspaceProviderRemoved: false,
createdByRuntime: result.branchCreatedByRuntime,
},
})
.where(
and(
eq(executionWorkspaces.id, workspace.id),
eq(executionWorkspaces.companyId, input.companyId),
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${workspace.metadata?.lifecycleGeneration ?? 0}`,
sql`${executionWorkspaces.providerRef} IS NOT DISTINCT FROM ${workspace.providerRef}`,
),
)
.returning();
if (!updated)
throw conflict("Workspace changed before preparation could be recorded");
return updated!;
});
const prepared = await prepare(workspace, placement?.taskLease);
if (placement && input.onTaskPrepared) {
let preparedTaskId = placement.taskLease.providerLeaseId;
let preparedWorkspace = prepared;
input.onTaskPrepared((taskId, assertOwned) => {
if (taskId === preparedTaskId) return;
return prepare(preparedWorkspace, { ...placement.taskLease, providerLeaseId: taskId }, { leaseId: placement.leaseId, assertOwned }).then(updated => {
preparedWorkspace = updated;
preparedTaskId = taskId;
});
});
}
return prepared;
}
/** Recover a lost prepare result even when the originating run has already stopped. */
async function reconcilePluginWorkspaceReservation(
db: Db,
workers: WorkspaceWorkers | undefined,
snapshot: Workspace,
): Promise<Workspace> {
const [workspace] = await db
.select()
.from(executionWorkspaces)
.where(
and(
eq(executionWorkspaces.id, snapshot.id),
eq(executionWorkspaces.companyId, snapshot.companyId),
),
);
if (!workspace) throw conflict("Workspace is unavailable");
if (readWorkspaceProviderBinding(workspace.metadata) || workspace.metadata?.workspaceProviderRemoval ||
workspace.metadata?.workspaceProviderRemoved === true) return workspace;
const owner = workspace.metadata?.workspaceProviderOwner;
const retired = typeof owner === "string" ? await retireAbandonedPluginWorkspaceReservation(db, workspace) : null;
if (retired) return retired;
if (typeof owner !== "string" || !workspace.projectId || !workspace.metadata?.workspaceProviderSource)
throw conflict("Workspace storage reservation is unavailable");
const operation = workspaceProviderOperationSchema.parse({
kind: "prepare",
source: workspace.metadata.workspaceProviderSource,
mode: workspace.mode,
strategy: workspace.strategyType,
branchName: workspace.metadata.workspaceProviderFresh ? null : workspace.branchName,
branchMode: workspace.metadata.workspaceProviderBranchMode ?? "create",
baseRef: workspace.baseRef,
existing: null,
...(workspace.metadata?.workspaceProviderFresh ? { fresh: workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace } : {}),
});
if (operation.kind !== "prepare") throw conflict("Workspace preparation is unavailable");
const result = await withPluginWorkspaceReplay({
pluginId: owner, companyId: workspace.companyId, projectId: workspace.projectId,
projectWorkspaceId: operation.source.id, executionWorkspaceId: workspace.id,
}, () => callWorkspaceProvider(db, workers, {
workspace,
pluginId: owner,
...readPreparationPlacement(workspace),
operation,
}));
if (result.kind !== "prepared")
throw conflict("Workspace reconciliation is incomplete");
const { workspaceProviderPreparationPlacement: _placement, ...metadata } = workspace.metadata ?? {};
const [updated] = await db.transaction(async (tx) => {
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
return tx
.update(executionWorkspaces)
.set({
providerRef: result.binding.workspaceRef,
branchName: result.branchName,
updatedAt: new Date(),
metadata: {
...metadata,
workspaceProvider: result.binding,
workspaceProviderBaseSha: result.baseSha,
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
createdByRuntime: result.branchCreatedByRuntime,
},
})
.where(
and(
eq(executionWorkspaces.id, workspace.id),
eq(executionWorkspaces.companyId, workspace.companyId),
sql`${executionWorkspaces.providerRef} IS NULL`,
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${workspace.metadata?.lifecycleGeneration ?? 0}`,
),
)
.returning();
});
if (!updated)
throw conflict("Workspace changed during reconciliation; retry inspection");
return updated;
}
async function inspectBinding(
db: Db,
workers: WorkspaceWorkers | undefined,
workspace: Workspace,
recoveryPullRequest?: PluginWorkspaceRecoveryPullRequest,
) {
const binding = readWorkspaceProviderBinding(workspace.metadata);
if (!binding) throw conflict("Workspace storage binding is unavailable");
const result = await callWorkspaceProvider(db, workers, {
workspace,
operation: { kind: "inspect", binding, ...(recoveryPullRequest ? { recoveryPullRequest } : {}) },
});
if (result.kind !== "inspection")
throw conflict("Workspace inspection is incomplete");
return result;
}
/** Reconcile durable lifecycle intent under the same lock for either read path. */
async function withRetainedWorkspace<T>(db: Db, workers: WorkspaceWorkers | undefined,
workspace: Workspace, operation: (workspace: Workspace) => Promise<T>) {
return withExecutionWorkspaceLifecycleLock(db, workspace.id, async () => {
workspace = await reconcilePluginWorkspaceReservation(db, workers, workspace);
if (workspace.metadata?.workspaceProviderRestoration)
workspace = await finishPendingRestoration(db, workers, workspace);
if (workspace.metadata?.workspaceProviderRemoval)
workspace = await finishPendingRemoval(db, workers, workspace);
return operation(workspace);
});
}
export async function inspectPluginWorkspace(db: Db, workers: WorkspaceWorkers | undefined, workspace: Workspace, recoveryPullRequest?: PluginWorkspaceRecoveryPullRequest) {
return withRetainedWorkspace(db, workers, workspace, (current) =>
current.metadata?.workspaceProviderRemoved === true
? Promise.resolve(removedInspection()) : inspectBinding(db, workers, current, recoveryPullRequest));
}
type PluginWorkspaceFinalizationPlan = { kind: "inspect" } | { kind: "finalize"; taskLease?: TaskPlacement };
export async function resolvePluginWorkspaceFinalization(db: Db, workspace: Workspace,
context: { runId: string; agentId: string; taskLease?: TaskPlacement | null }): Promise<PluginWorkspaceFinalizationPlan | undefined> {
if (workspace.providerType !== "plugin") return undefined;
if (context.taskLease) return { kind: "finalize", taskLease: context.taskLease };
const binding = readWorkspaceProviderBinding(workspace.metadata);
if (!binding) throw conflict("Workspace storage ownership is missing");
const selected = await pluginRegistryService(db).getById(binding.pluginId);
if (!selected) throw conflict("The workspace storage plugin is unavailable");
// Existing providers continue to use their established inspection contract.
if (selected.manifestJson.workspaceProvider?.finalizationVersion !== 1) return { kind: "inspect" };
const portableDrivers = new Set(selected.manifestJson.environmentDrivers?.filter(driver =>
driver.sessionHandoffVersion === 1).map(driver => driver.driverKey));
let taskLease: TaskPlacement | undefined;
if (context.taskLease === undefined && portableDrivers.size > 0) {
const leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, workspace.companyId),
eq(environmentLeases.executionWorkspaceId, workspace.id),
eq(environmentLeases.heartbeatRunId, context.runId),
eq(environmentLeases.status, "active"),
));
const placements = leases.flatMap(lease => {
const driverKey = lease.metadata?.driverKey;
return lease.metadata?.pluginId === binding.pluginId && lease.metadata?.agentId === context.agentId &&
lease.metadata?.runnerTaskLease === true && lease.environmentId && lease.providerLeaseId &&
typeof driverKey === "string" && portableDrivers.has(driverKey)
? [{ driverKey, environmentId: lease.environmentId, providerLeaseId: lease.providerLeaseId }] : [];
});
if (placements.length > 1) throw conflict("Workspace Launcher placement is ambiguous");
taskLease = placements[0];
}
return { kind: "finalize", taskLease };
}
/** Completion does not need the deletion-safety proof supplied by inspect. */
export async function finalizePluginWorkspace(db: Db, workers: WorkspaceWorkers | undefined,
workspace: Workspace, context: { runId: string; agentId: string; taskLease?: TaskPlacement | null;
plan?: PluginWorkspaceFinalizationPlan }) {
return withRetainedWorkspace(db, workers, workspace, async (current) => {
if (current.metadata?.workspaceProviderRemoved === true)
throw conflict("Workspace was removed before finalization");
const binding = readWorkspaceProviderBinding(current.metadata);
if (!binding) throw conflict("Workspace storage ownership is missing");
if (context.plan && !isDeepStrictEqual(binding, readWorkspaceProviderBinding(workspace.metadata)))
throw conflict("Workspace storage ownership changed");
const plan = context.plan ?? await resolvePluginWorkspaceFinalization(db, current, context);
if (!plan) throw conflict("The workspace storage plugin is unavailable");
if (plan.kind === "inspect") return inspectBinding(db, workers, current);
const taskLease = plan.taskLease;
const result = await callWorkspaceProvider(db, workers, {
workspace: current, runId: context.runId, agentId: context.agentId, taskLease, operation: { kind: "finalize", binding },
});
if (result.kind !== "finalized") throw conflict("Workspace finalization is incomplete");
return result;
});
}
const removedInspection = () => ({
kind: "inspection" as const,
headSha: null,
branchName: null,
hasDirtyTrackedFiles: false,
hasUntrackedFiles: false,
hasLocalOnlyData: false,
isMergedIntoBase: null,
remotelyRecoverable: true,
});
async function finishPendingRestoration(
db: Db,
workers: WorkspaceWorkers | undefined,
workspace: Workspace,
): Promise<Workspace> {
const pending = workspace.metadata?.workspaceProviderRestoration as {
generation?: unknown;
operation?: unknown;
};
const operation = workspaceProviderOperationSchema.safeParse(
pending?.operation,
);
const generation =
(workspace.metadata?.lifecycleGeneration as number | undefined) ?? 0;
if (
!operation.success ||
operation.data.kind !== "prepare" ||
pending.generation !== generation + 1 ||
JSON.stringify(operation.data.existing) !==
JSON.stringify(readWorkspaceProviderBinding(workspace.metadata)) ||
!["archived", "cleanup_failed"].includes(workspace.status)
)
throw conflict("Workspace restoration intent is unavailable or stale");
const result = await callWorkspaceProvider(db, workers, {
workspace,
operation: operation.data,
advanceGeneration: true,
});
if (result.kind !== "prepared")
throw conflict("Workspace restoration is incomplete");
const [updated] = await db.transaction(async (tx) => {
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
return tx
.update(executionWorkspaces)
.set({
providerRef: result.binding.workspaceRef,
branchName: result.branchName,
updatedAt: new Date(),
metadata: {
...workspace.metadata,
lifecycleGeneration: generation + 1,
workspaceProvider: result.binding,
workspaceProviderBaseSha: result.baseSha,
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
workspaceProviderRemoved: false,
workspaceProviderRestoration: null,
},
})
.where(
and(
eq(executionWorkspaces.id, workspace.id),
eq(executionWorkspaces.companyId, workspace.companyId),
eq(executionWorkspaces.status, workspace.status),
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${generation}`,
sql`${executionWorkspaces.metadata}->'workspaceProviderRestoration' = ${JSON.stringify(pending)}::jsonb`,
),
)
.returning();
});
if (!updated)
throw conflict("Workspace changed during restoration reconciliation");
return updated;
}
/** Replay the recorded delete before trying to inspect a checkout that may already be gone. */
async function finishPendingRemoval(
db: Db,
workers: WorkspaceWorkers | undefined,
workspace: Workspace,
): Promise<Workspace> {
if (workspace.metadata?.workspaceProviderRemoved === true) return workspace;
const pending = workspace.metadata?.workspaceProviderRemoval as
| { generation?: unknown; operation?: unknown; operationId?: unknown }
| undefined;
const parsed = workspaceProviderOperationSchema.safeParse(pending?.operation);
const binding = readWorkspaceProviderBinding(workspace.metadata);
if (
!parsed.success ||
parsed.data.kind !== "remove" ||
pending?.generation !== (workspace.metadata?.lifecycleGeneration ?? 0) ||
JSON.stringify(parsed.data.binding) !== JSON.stringify(binding)
)
throw conflict("Workspace removal intent is unavailable or stale");
if (pending.operationId === undefined && parsed.data.recoveryPullRequest)
throw conflict("Workspace removal identity is ambiguous; reconciliation requires the original request");
const operationId = createHash("sha256").update(JSON.stringify([workspace.id, pending.generation, parsed.data])).digest("hex");
if (pending.operationId !== undefined && pending.operationId !== operationId)
throw conflict("Workspace removal identity does not match its recorded payload");
const owner = workspace.metadata?.workspaceProviderOwner;
const result = await callWorkspaceProvider(db, workers, {
workspace,
pluginId: typeof owner === "string" ? owner : undefined,
operation: parsed.data,
removalOperationId: operationId,
});
if (result.kind !== "removed")
throw conflict("Workspace removal is incomplete");
if (workspace.metadata?.workspaceProviderFresh)
await releaseProviderRecoveryTransfers(workspace.companyId, workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace);
const [updated] = await db.transaction(async (tx) => {
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
return tx
.update(executionWorkspaces)
.set({
...(workspace.status === "cleanup_failed"
? { status: "archived", cleanupReason: null }
: {}),
metadata: {
...workspace.metadata,
workspaceProviderRemoval: null,
workspaceProviderRemoved: true,
},
updatedAt: new Date(),
})
.where(
and(
eq(executionWorkspaces.id, workspace.id),
eq(executionWorkspaces.companyId, workspace.companyId),
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${pending!.generation}`,
),
)
.returning();
});
if (!updated)
throw conflict("Workspace changed during removal reconciliation");
return updated;
}
export async function removePluginWorkspace(
db: Db,
workers: WorkspaceWorkers | undefined,
workspace: Workspace,
expectedHeadSha: string | null,
recoveryPullRequest?: PluginWorkspaceRecoveryPullRequest,
beforeRemove?: () => Promise<void>,
) {
workspace = await reconcilePluginWorkspaceReservation(db, workers, workspace);
if (workspace.metadata?.workspaceProviderRestoration)
throw conflict("Workspace restoration must be reconciled before removal");
if (workspace.metadata?.workspaceProviderRemoved === true) {
if (workspace.metadata?.workspaceProviderFresh)
await releaseProviderRecoveryTransfers(workspace.companyId, workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace);
return;
}
if (workspace.metadata?.workspaceProviderRemoval) {
await finishPendingRemoval(db, workers, workspace);
return;
}
const binding = readWorkspaceProviderBinding(workspace.metadata);
if (!binding) throw conflict("Workspace storage binding is unavailable");
const inspection = await inspectBinding(db, workers, workspace, recoveryPullRequest);
if (
inspection.hasDirtyTrackedFiles ||
inspection.hasUntrackedFiles ||
inspection.hasLocalOnlyData ||
inspection.headSha !== expectedHeadSha ||
(inspection.remotelyRecoverable !== true &&
inspection.isMergedIntoBase !== true)
) {
throw conflict(
"Workspace has local changes or cannot be recovered remotely",
);
}
await beforeRemove?.();
const generation = workspace.metadata?.lifecycleGeneration ?? 0;
const selectedProvider = await requireWorkspaceProviderWorker(db, workers, workspace.companyId, binding.pluginId);
const operation = workspaceProviderOperationSchema.parse({ kind: "remove", binding, expectedHeadSha,
...(recoveryPullRequest && selectedProvider.manifestJson.workspaceProvider?.recoveryPullRequestVersion === 1 ? { recoveryPullRequest } : {}) });
const operationId = createHash("sha256").update(JSON.stringify([workspace.id, generation, operation])).digest("hex");
const [pending] = await db.transaction(async (tx) => {
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
return tx
.update(executionWorkspaces)
.set({
metadata: {
...workspace.metadata,
workspaceProviderRemoval: { generation, operation, operationId },
},
updatedAt: new Date(),
})
.where(
and(
eq(executionWorkspaces.id, workspace.id),
eq(executionWorkspaces.companyId, workspace.companyId),
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${generation}`,
eq(executionWorkspaces.providerRef, binding.workspaceRef),
),
)
.returning();
});
if (!pending)
throw conflict("Workspace changed before removal could be recorded");
await finishPendingRemoval(db, workers, pending);
}
/** An authorized storage edit atomically retires closed, removed dependencies under the reopen fence. */
export async function retireRemovedPluginWorkspaceDependents(
db: Db,
projectId: string,
sourceId?: string,
) {
const rows = await db
.select({ id: executionWorkspaces.id })
.from(executionWorkspaces)
.where(
and(
eq(executionWorkspaces.projectId, projectId),
eq(executionWorkspaces.providerType, "plugin"),
sourceId
? eq(executionWorkspaces.projectWorkspaceId, sourceId)
: undefined,
),
);
for (const { id } of rows.sort((a, b) => a.id.localeCompare(b.id))) {
const acquired = await db.execute(
sql`select pg_try_advisory_xact_lock(hashtextextended(${`execution_workspace_lifecycle:${id}`}, 0)) as acquired`,
);
if (!acquired[0]?.acquired)
throw conflict("Workspace lifecycle is busy; retry the source edit");
const [row] = await db
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, id));
if (!row || row.metadata?.workspaceProviderRetired === true) continue;
if (
row.metadata?.workspaceProviderRemoved !== true ||
row.metadata?.workspaceProviderRestoration ||
!["archived", "cleanup_failed"].includes(row.status)
) {
throw conflict(
"Retire managed execution workspaces before changing their storage owner or repository source",
);
}
await db
.update(executionWorkspaces)
.set({
metadata: { ...row.metadata, workspaceProviderRetired: true },
updatedAt: new Date(),
})
.where(eq(executionWorkspaces.id, row.id));
}
}
/** Called inside Core's existing lifecycle fence after task authorization. */
export async function restorePluginWorkspace(
db: Db,
workers: WorkspaceWorkers | undefined,
workspace: Workspace,
) {
workspace = await reconcilePluginWorkspaceReservation(db, workers, workspace);
if (workspace.metadata?.workspaceProviderRetired === true)
throw conflict("Workspace storage was retired");
if (workspace.metadata?.workspaceProviderRemoval)
workspace = await finishPendingRemoval(db, workers, workspace);
const binding = readWorkspaceProviderBinding(workspace.metadata);
const source = workspace.metadata?.workspaceProviderSource;
if (!binding || !source)
throw conflict("Workspace restoration recipe is unavailable");
const operation = workspaceProviderOperationSchema.parse({
kind: "prepare",
source,
mode: workspace.mode,
strategy: workspace.strategyType,
branchName: workspace.metadata?.workspaceProviderFresh ? null : workspace.branchName,
branchMode: workspace.metadata?.workspaceProviderFresh ? "create" : "existing",
baseRef: workspace.baseRef,
existing: binding,
...(workspace.metadata?.workspaceProviderFresh ? { fresh: workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace } : {}),
});
const [pending] = await db.transaction(async (tx) => {
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
return tx
.update(executionWorkspaces)
.set({
metadata: {
...workspace.metadata,
workspaceProviderRemoved: false,
workspaceProviderRestoration: {
generation:
((workspace.metadata?.lifecycleGeneration as number) ?? 0) + 1,
operation,
},
},
updatedAt: new Date(),
})
.where(
and(
eq(executionWorkspaces.id, workspace.id),
eq(executionWorkspaces.companyId, workspace.companyId),
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${workspace.metadata?.lifecycleGeneration ?? 0}`,
),
)
.returning();
});
if (!pending)
throw conflict("Workspace changed before restoration could be recorded");
workspace = pending;
const result = await callWorkspaceProvider(db, workers, {
workspace,
operation,
advanceGeneration: true,
});
if (result.kind !== "prepared")
throw conflict("Workspace restoration is incomplete");
return {
...workspace.metadata,
workspaceProvider: result.binding,
workspaceProviderRemoved: false,
workspaceProviderRestoration: null,
workspaceProviderBaseSha: result.baseSha,
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
};
}