Files
PaperClipAI/server/src/services/plugin-workspaces.ts
T
Nicky LeachandClaude Opus 5.5 1140984b82 feat(plugins): tell an opted-in workspace provider the run's environment (#21512)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin can be a project workspace provider and also an environment
driver for the agent runs that use those workspaces
> - Core asks the workspace provider to prepare the project workspace
before the run acquires its environment lease
> - The provider knows the run, agent, project, and execution workspace,
but not the environment that the run selected
> - A provider whose compute exists per run (for example, a sandbox that
it creates from the environment's driver config) cannot start or find
the run's compute for the workspace work
> - This pull request lets a provider opt in to receive the run's
selected environment, when that environment is one of the provider's own
drivers
> - The benefit is that such a provider can prepare the workspace on the
run's own compute, with no change to the order of operations in Core

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Plugin workspace preparation for runs whose environment is a driver of
the same plugin.

**Subsystem affected**
Plugin SDK workspace provider params, the plugin manifest workspace
provider declaration, and `callWorkspaceProvider` /
`preparePluginWorkspace` in the server.

**Current behavior**
`prepare` params carry `runId`, `agentId`, `projectId`, and
`executionWorkspaceId`. They carry `taskLease` only when Core acquired
the lease first. They never name the run's environment.

**Proposed behavior**
A provider that declares `workspaceProvider.workspaceEnvironmentVersion:
1` receives `environment: { id, driverKey, config }` when the run's
selected environment is a plugin environment of that same provider.
`config` is the environment's `driverConfig`, the value that the
driver's acquire receives.

## What Changed

- SDK: optional strict `environment` field on
`pluginWorkspaceParamsSchema`.
- Manifest: optional `workspaceProvider.workspaceEnvironmentVersion: 1`,
next to the sibling version fields. Core sends the new field only to
providers that declare it, because providers parse the params with the
strict schema.
- Server: `callWorkspaceProvider` adds `environment` only when the
provider opted in, the environment driver is `plugin`, its `pluginKey`
is this provider, and it names a `driverKey`. The field is not part of
the operation ID.
- Server: `preparePluginWorkspace` passes the run's environment to its
prepare call. This covers normal, verify-existing, fresh, and
replacement prepares.
- Calls that do not have the run's environment do not send it: finalize,
inspect, remove, reservation reconciliation, and the retry-candidate
verification in `fresh-workspace-continuation.ts`.
- Lease acquisition, the `sessionHandoffVersion` gate, and the order of
operations do not change.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` describes the
capability and its exceptions.

## Verification

- `packages/plugins/sdk/tests/workspace-provider.test.ts`: 11 passed.
The schema accepts the field and rejects malformed values.
- `packages/shared/src/validators/plugin.test.ts`: 65 passed. The
manifest accepts only version 1.
- `server`: `plugin-workspaces.integration`,
`plugin-fresh-workspaces.integration`, `project-environments`, and
`heartbeat/workspace-preparation`: 226 passed, against embedded
Postgres. The new integration test checks the exact shape, an unchanged
operation ID, verification prepares, and omission for an undeclared
provider, another plugin's environment, non-plugin drivers, a missing
`driverKey`, and no environment. It fails when the server stops passing
the environment.
- `pnpm typecheck` passes in `packages/shared`, `packages/plugins/sdk`,
and `server`.
- The full suite was not run locally. CI runs it.

## Risks

- Low risk. Providers that do not declare the capability get the same
params as before.
- `config` is the stored `driverConfig`, in the same form that the
driver's acquire gets. Secret references in it reach the provider in
that same form.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:43:10 -08:00

1003 lines
43 KiB
TypeScript

import { isDeepStrictEqual } from "node:util";
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
import { assertProjectStorageEnvironment, effectiveProjectWorkspacePolicy, requireWorkspaceProviderWorker, type WorkspaceWorkers } from "./project-workspace-provider.js";
import { releaseProviderRecoveryTransfers } from "./provider-fresh-workspace.js";
import { retireAbandonedPluginWorkspaceReservation } from "./plugin-workspace-reservations.js";
import { randomUUID, createHash } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import {
environmentLeases,
executionWorkspaces,
heartbeatRuns,
issues,
projects,
projectWorkspaces,
type Db,
} from "@paperclipai/db";
import {
readWorkspaceProviderBinding,
type WorkspaceProviderBinding,
type ExecutionWorkspace,
} from "@paperclipai/shared";
import {
pluginWorkspaceParamsSchema,
workspaceProviderOperationSchema,
type PluginWorkspaceOperation,
type PluginWorkspaceRecoveryPullRequest,
type PluginWorkspaceParams,
type PluginFreshWorkspace,
} from "@paperclipai/plugin-sdk";
import { callWorkspaceProvider } from "./plugin-workspace-preparation.js";
import { withPluginWorkspaceReplay } from "./plugin-workspace-replay-context.js";
import { pluginRegistryService } from "./plugin-registry.js";
import { notifyTerminalWorkspaceWork } from "./terminal-workspace-work.js";
import { withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifecycle-lock.js";
import {
assertTaskWorkspaceExecutionAccess,
assertTaskWorkspaceSourceProjectAccess,
} from "./task-workspace-access.js";
import { forbidden, conflict } from "../errors.js";
export { callWorkspaceProvider } from "./plugin-workspace-preparation.js";
export type { WorkspaceWorkers } from "./project-workspace-provider.js";
type Workspace = typeof executionWorkspaces.$inferSelect;
type TaskPlacement = NonNullable<PluginWorkspaceParams["taskLease"]>;
const preparationPlacementSchema = pluginWorkspaceParamsSchema.pick({
generation: true, runId: true, agentId: true, taskLease: true,
});
function readPreparationPlacement(workspace: Workspace) {
const saved = workspace.metadata?.workspaceProviderPreparationPlacement;
if (saved === undefined) return undefined;
const placement = preparationPlacementSchema.parse(saved);
if (!placement.runId || !placement.agentId ||
placement.generation !== (workspace.metadata?.lifecycleGeneration ?? 0))
throw conflict("Workspace preparation placement changed");
return placement;
}
type EnsureTaskWorkspace = (taskId: string, assertOwned: () => void) => Promise<void> | undefined;
export async function pluginWorkspaceSupportsMergeAncestry(db: Db, workers: WorkspaceWorkers | undefined, workspace: Workspace) {
const pluginId = readWorkspaceProviderBinding(workspace.metadata)?.pluginId;
if (!pluginId) return true;
const selected = await requireWorkspaceProviderWorker(db, workers, workspace.companyId, pluginId);
return selected.manifestJson.workspaceProvider?.supportsMergeAncestry !== false;
}
export async function preparePluginWorkspace(
db: Db,
workers: WorkspaceWorkers | undefined,
input: {
companyId: string;
projectId: string | null;
projectWorkspaceId: string | null;
issueId: string | null;
runId: string;
agentId: string;
responsibleUserId: string | null;
existingWorkspaceId: string | null;
mode: string;
strategy: string;
selectedProviderPluginId?: string | null;
environment?: Pick<import("@paperclipai/shared").Environment, "id" | "driver" | "config" | "name"> | null;
branchMode?: "create" | "existing";
branchName?: string | null;
baseRef?: string | null;
fresh?: PluginFreshWorkspace;
expectedBindingRevision?: number;
verifyExisting?: true;
assertOwned?: () => void;
acquireTaskLease?: (workspace: Pick<ExecutionWorkspace, "id" | "mode">) => Promise<{ leaseId: string; taskLease: TaskPlacement }>;
onTaskPrepared?: (ensure: EnsureTaskWorkspace) => void;
},
) {
const [previous] = input.existingWorkspaceId
? await db
.select()
.from(executionWorkspaces)
.where(
and(
eq(executionWorkspaces.id, input.existingWorkspaceId),
eq(executionWorkspaces.companyId, input.companyId),
),
)
: [];
const [reservation] = input.fresh ? await db.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, input.companyId),
eq(executionWorkspaces.sourceIssueId, input.issueId!),
sql`${executionWorkspaces.metadata}#>>'{workspaceProviderFresh,attemptId}' = ${input.fresh.attemptId}`,
)).limit(1) : [];
const existing = input.fresh ? reservation : previous;
if (input.verifyExisting && (!existing || existing.metadata?.workspaceProviderRemoved === true ||
existing.metadata?.workspaceProviderRestoration)) return null;
const prior = existing ?? previous;
if (
existing?.metadata?.workspaceProviderRetired === true ||
existing?.metadata?.workspaceProviderRemoval
)
throw conflict("Workspace storage is retired or awaiting cleanup");
if (!existing && (!input.projectId || !input.projectWorkspaceId)) return null;
const saved = readWorkspaceProviderBinding(existing?.metadata);
const priorBinding = readWorkspaceProviderBinding(prior?.metadata);
const mode = input.fresh ? "isolated_workspace" : existing?.mode ?? input.mode,
strategy = existing?.strategyType ?? input.strategy;
const projectId = priorBinding?.projectId ?? input.projectId;
const [project] = projectId
? await db
.select()
.from(projects)
.where(
and(
eq(projects.id, projectId),
eq(projects.companyId, input.companyId),
),
)
: [];
if (prior && prior.providerType !== "plugin" && !priorBinding && !prior.metadata?.workspaceProviderOwner) return null;
const owner = priorBinding?.pluginId ?? prior?.metadata?.workspaceProviderOwner ??
(!existing && input.selectedProviderPluginId !== undefined ? input.selectedProviderPluginId :
!existing && project ? (await effectiveProjectWorkspacePolicy(db, input.companyId, project.executionWorkspacePolicy, false))?.storageProviderPluginId : null);
const pluginId = typeof owner === "string" ? owner : null;
if (!pluginId) {
if (existing?.providerType === "plugin")
throw conflict("Workspace storage ownership is missing");
return null;
}
if (
!input.issueId ||
!project ||
project.archivedAt ||
!input.projectWorkspaceId
)
throw conflict(
"Managed storage requires a task and an available project source",
);
if (
!["shared_workspace", "isolated_workspace", "operator_branch"].includes(
input.mode,
) ||
!["project_primary", "git_worktree"].includes(strategy)
)
throw conflict("Unsupported managed workspace strategy");
if (existing && existing.providerType !== "plugin")
throw conflict(
"An existing local workspace cannot change storage providers",
);
const selectedProvider = await requireWorkspaceProviderWorker(db, workers, input.companyId, pluginId, true);
if (input.environment) assertProjectStorageEnvironment(selectedProvider, input.environment);
const declaration = selectedProvider.manifestJson.workspaceProvider!;
if (input.verifyExisting && declaration.workspaceVerificationVersion !== 1) return null;
if (input.fresh && declaration.freshWorkspaceVersion !== 1)
throw conflict("A fresh attempt is unavailable for this workspace storage provider. The existing workspace is retained.", { reason: "fresh_workspace_storage_unsupported" });
if (
!declaration.modes.includes(mode as "isolated_workspace") ||
!declaration.strategies.includes(strategy as "git_worktree")
) {
throw conflict(
"The storage plugin does not support this workspace mode or strategy",
);
}
const actor = {
type: "agent" as const,
source: "agent_jwt" as const,
companyId: input.companyId,
agentId: input.agentId,
runId: input.runId,
onBehalfOfUserId:
input.responsibleUserId === "local-board"
? null
: input.responsibleUserId,
};
const authorize = async (workspaceId = existing?.id) => {
await assertTaskWorkspaceExecutionAccess(db, actor, input.companyId, {
issueId: input.issueId!,
...(workspaceId
? { workspaceId }
: { projectWorkspaceId: input.projectWorkspaceId }),
});
await assertTaskWorkspaceSourceProjectAccess(
db,
actor,
input.companyId,
project.id,
);
const [run] = await db
.select()
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.id, input.runId),
eq(heartbeatRuns.companyId, input.companyId),
),
);
if (
!run ||
run.agentId !== input.agentId ||
![run.issueId, run.nativeIssueId].includes(input.issueId!) ||
run.status !== "running"
)
throw forbidden("Workspace preparation no longer has an active run");
};
await authorize();
const [source] = await db
.select()
.from(projectWorkspaces)
.where(
and(
eq(
projectWorkspaces.id,
saved?.projectWorkspaceId ?? input.projectWorkspaceId,
),
eq(projectWorkspaces.projectId, project.id),
eq(projectWorkspaces.companyId, input.companyId),
),
);
if (!source) throw conflict("Workspace repository source is unavailable");
const sourceInput = (prior?.metadata?.workspaceProviderSource as
| Extract<PluginWorkspaceOperation, { kind: "prepare" }>["source"]
| undefined) ?? {
id: source.id,
revision:
saved?.sourceRevision ??
createHash("sha256")
.update(
JSON.stringify([
source.id,
source.repoUrl,
source.repoRef,
source.defaultRef,
]),
)
.digest("hex"),
repoUrl: existing?.repoUrl ?? source.repoUrl,
repoRef: existing?.baseRef ?? source.repoRef,
defaultRef: source.defaultRef,
};
const branchName = input.fresh ? null : existing
? existing.branchName
: (input.branchName ??
(strategy === "git_worktree" ? `workspace-${randomUUID()}` : null));
// Reserve and bind before calling the provider. A crash must not orphan a created checkout.
const workspace =
existing ??
(await db.transaction(async (tx) => {
const [currentProject] = await tx
.select()
.from(projects)
.where(eq(projects.id, project.id))
.for("update");
const [currentSource] = await tx
.select()
.from(projectWorkspaces)
.where(eq(projectWorkspaces.id, source.id));
if (
!currentProject ||
currentProject.archivedAt ||
(await effectiveProjectWorkspacePolicy(tx, input.companyId, currentProject.executionWorkspacePolicy, false))?.storageProviderPluginId !==
pluginId ||
!currentSource ||
currentSource.repoUrl !== source.repoUrl ||
currentSource.repoRef !== source.repoRef ||
currentSource.defaultRef !== source.defaultRef
) {
throw conflict(
"Project storage configuration changed; retry preparation",
);
}
const [task] = await tx
.select()
.from(issues)
.where(
and(
eq(issues.id, input.issueId!),
eq(issues.companyId, input.companyId),
),
)
.for("update");
if (!task || (input.fresh ? task.executionWorkspaceId !== input.fresh.previousWorkspaceId || task.workspaceBindingRevision !== input.expectedBindingRevision : Boolean(task.executionWorkspaceId)))
throw conflict("Task workspace binding changed; retry preparation");
const id = randomUUID();
await notifyTerminalWorkspaceWork(tx, `workspace:${id}`);
const [row] = await tx
.insert(executionWorkspaces)
.values({
id,
companyId: input.companyId,
projectId: project.id,
projectWorkspaceId: source.id,
sourceIssueId: input.issueId,
mode,
strategyType: strategy,
name: branchName ?? source.name,
providerType: "plugin",
providerRef: null,
cwd: null,
branchName,
repoUrl: source.repoUrl,
baseRef: input.baseRef ?? source.repoRef,
status: "idle",
metadata: {
workspaceProviderOwner: pluginId,
workspaceProviderReservationRunId: input.runId,
workspaceProviderSource: sourceInput,
workspaceProviderBranchMode: input.fresh ? "create" : input.branchMode ?? "create",
...(input.fresh ? { workspaceProviderFresh: input.fresh } : {}),
},
})
.returning();
await notifyStatusCardIssueChange(tx, task.companyId);
await tx
.update(issues)
.set({
executionWorkspaceId: id,
executionWorkspacePreference: "reuse_existing",
workspaceBindingRevision: task.workspaceBindingRevision + 1,
workspacePendingSelection:
task.workspacePendingSelection?.expectedBindingRevision ===
task.workspaceBindingRevision
? {
...task.workspacePendingSelection,
expectedBindingRevision: task.workspaceBindingRevision + 1,
}
: task.workspacePendingSelection,
updatedAt: new Date(),
})
.where(eq(issues.id, task.id));
return row!;
}));
const driverKey = input.environment?.driver === "plugin" ? input.environment.config?.driverKey : undefined;
const placement = selectedProvider.manifestJson.environmentDrivers?.some(driver =>
driver.driverKey === driverKey && driver.sessionHandoffVersion === 1)
? await input.acquireTaskLease?.({ id: workspace.id, mode: mode as ExecutionWorkspace["mode"] }) : undefined;
const prepare = (workspace: Workspace, taskLease?: TaskPlacement, replacement?: { leaseId: string; assertOwned: () => void }) => withExecutionWorkspaceLifecycleLock(db, workspace.id, async (tx) => {
(replacement?.assertOwned ?? input.assertOwned)?.();
const [current] = await tx
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, workspace.id));
if (
!current ||
!["active", "idle", "in_review"].includes(current.status) ||
current.metadata?.workspaceProviderRetired === true ||
current.metadata?.workspaceProviderRemoval ||
current.providerRef !== workspace.providerRef ||
(current.metadata?.lifecycleGeneration ?? 0) !==
(workspace.metadata?.lifecycleGeneration ?? 0)
) {
throw conflict("Workspace lifecycle changed; reopen before preparing");
}
if (taskLease) await authorize(replacement ? workspace.id : undefined);
const replacementScope = replacement && taskLease ? and(
eq(environmentLeases.id, replacement.leaseId),
eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.heartbeatRunId, input.runId),
eq(environmentLeases.executionWorkspaceId, workspace.id),
eq(environmentLeases.environmentId, taskLease.environmentId),
eq(environmentLeases.providerLeaseId, taskLease.providerLeaseId),
eq(environmentLeases.status, "active"),
sql`${environmentLeases.metadata}->>'pluginId' = ${pluginId} and ${environmentLeases.metadata}->>'driverKey' = ${taskLease.driverKey}`,
sql`${environmentLeases.releasedAt} is null and (${environmentLeases.expiresAt} is null or ${environmentLeases.expiresAt} > clock_timestamp())`,
sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
) : undefined;
if (replacementScope && !(await tx.select({ id: environmentLeases.id }).from(environmentLeases).where(replacementScope)).length)
throw conflict("Workspace Launcher placement changed");
if (!current.providerRef && !input.verifyExisting) {
const generation = current.metadata?.lifecycleGeneration ?? 0;
const intent = { generation, runId: input.runId, agentId: input.agentId, ...(taskLease ? { taskLease } : {}) };
const previous = readPreparationPlacement(current);
if (!isDeepStrictEqual(previous, intent)) {
// The request must survive a lost reply or rollback of the lifecycle transaction.
const [recorded] = await db.update(executionWorkspaces).set({
metadata: sql`coalesce(${executionWorkspaces.metadata}, '{}'::jsonb) || ${JSON.stringify({ workspaceProviderPreparationPlacement: intent })}::jsonb`,
}).where(and(eq(executionWorkspaces.id, current.id), eq(executionWorkspaces.companyId, current.companyId),
sql`${executionWorkspaces.providerRef} is null
and coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${generation}
and coalesce(${executionWorkspaces.metadata}->'workspaceProviderPreparationPlacement', 'null'::jsonb)
= ${JSON.stringify(previous ?? null)}::jsonb`)).returning({ id: executionWorkspaces.id });
if (!recorded) throw conflict("Workspace preparation placement changed");
}
}
const result = await callWorkspaceProvider(tx as unknown as Db, workers, {
workspace,
pluginId,
runId: input.runId,
agentId: input.agentId,
taskLease,
environment: input.environment,
assertOwned: replacement?.assertOwned ?? input.assertOwned,
operation: {
kind: "prepare",
source: sourceInput,
mode: workspace.mode as "isolated_workspace",
strategy: workspace.strategyType as "git_worktree",
branchName: workspace.metadata?.workspaceProviderFresh ? null : workspace.branchName,
branchMode:
(workspace.metadata?.workspaceProviderBranchMode as
| "create"
| "existing"
| undefined) ??
input.branchMode ??
"create",
baseRef: workspace.baseRef,
existing: readWorkspaceProviderBinding(workspace.metadata),
...(input.verifyExisting ? { verifyExisting: true } : {}),
...(workspace.metadata?.workspaceProviderFresh ? { fresh: workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace } : {}),
},
});
if (result.kind !== "prepared")
throw conflict("Workspace preparation is incomplete");
if (replacement && result.branchName !== workspace.branchName)
throw conflict("Workspace Launcher prepared a different branch");
const savedBase = workspace.metadata?.workspaceProviderBaseSha;
if (input.verifyExisting && (result.created || !result.baseSha ||
(saved && result.binding.workspaceRef !== saved.workspaceRef) ||
(workspace.branchName !== null && result.branchName !== workspace.branchName) ||
(typeof savedBase === "string" && result.baseSha !== savedBase))) {
return null;
}
(replacement?.assertOwned ?? input.assertOwned)?.();
await authorize();
const [currentProject] = await tx
.select()
.from(projects)
.where(eq(projects.id, project.id));
if (!currentProject || currentProject.archivedAt)
throw conflict("Project was archived during workspace preparation");
const [task] = await tx
.select()
.from(issues)
.where(eq(issues.id, input.issueId!));
if (task?.executionWorkspaceId !== workspace.id)
throw conflict("Task workspace binding changed during preparation");
if (replacementScope) {
const [currentLease] = await tx.update(environmentLeases).set({
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ workspaceProvider: result.binding })}::jsonb`,
}).where(replacementScope).returning({ id: environmentLeases.id });
if (!currentLease) throw conflict("Workspace Launcher placement changed during preparation");
}
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
const { workspaceProviderPreparationPlacement: _placement, ...metadata } = current.metadata ?? {};
const [updated] = await tx
.update(executionWorkspaces)
.set({
providerRef: result.binding.workspaceRef,
branchName: result.branchName,
status: "active",
lastUsedAt: new Date(),
updatedAt: new Date(),
metadata: {
...metadata,
workspaceProvider: result.binding,
workspaceProviderBaseSha: result.baseSha,
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
workspaceProviderSource: sourceInput,
workspaceProviderRemoved: false,
createdByRuntime: result.branchCreatedByRuntime,
},
})
.where(
and(
eq(executionWorkspaces.id, workspace.id),
eq(executionWorkspaces.companyId, input.companyId),
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${workspace.metadata?.lifecycleGeneration ?? 0}`,
sql`${executionWorkspaces.providerRef} IS NOT DISTINCT FROM ${workspace.providerRef}`,
),
)
.returning();
if (!updated)
throw conflict("Workspace changed before preparation could be recorded");
return updated!;
});
const prepared = await prepare(workspace, placement?.taskLease);
if (prepared && placement && input.onTaskPrepared) {
let preparedTaskId = placement.taskLease.providerLeaseId;
let preparedWorkspace = prepared;
input.onTaskPrepared((taskId, assertOwned) => {
if (taskId === preparedTaskId) return;
return prepare(preparedWorkspace, { ...placement.taskLease, providerLeaseId: taskId }, { leaseId: placement.leaseId, assertOwned }).then(updated => {
if (!updated) throw conflict("Workspace preparation is incomplete");
preparedWorkspace = updated;
preparedTaskId = taskId;
});
});
}
return prepared;
}
/** Recover a lost prepare result even when the originating run has already stopped. */
async function reconcilePluginWorkspaceReservation(
db: Db,
workers: WorkspaceWorkers | undefined,
snapshot: Workspace,
): Promise<Workspace> {
const [workspace] = await db
.select()
.from(executionWorkspaces)
.where(
and(
eq(executionWorkspaces.id, snapshot.id),
eq(executionWorkspaces.companyId, snapshot.companyId),
),
);
if (!workspace) throw conflict("Workspace is unavailable");
if (readWorkspaceProviderBinding(workspace.metadata) || workspace.metadata?.workspaceProviderRemoval ||
workspace.metadata?.workspaceProviderRemoved === true) return workspace;
const owner = workspace.metadata?.workspaceProviderOwner;
const retired = typeof owner === "string" ? await retireAbandonedPluginWorkspaceReservation(db, workspace) : null;
if (retired) return retired;
if (typeof owner !== "string" || !workspace.projectId || !workspace.metadata?.workspaceProviderSource)
throw conflict("Workspace storage reservation is unavailable");
const operation = workspaceProviderOperationSchema.parse({
kind: "prepare",
source: workspace.metadata.workspaceProviderSource,
mode: workspace.mode,
strategy: workspace.strategyType,
branchName: workspace.metadata.workspaceProviderFresh ? null : workspace.branchName,
branchMode: workspace.metadata.workspaceProviderBranchMode ?? "create",
baseRef: workspace.baseRef,
existing: null,
...(workspace.metadata?.workspaceProviderFresh ? { fresh: workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace } : {}),
});
if (operation.kind !== "prepare") throw conflict("Workspace preparation is unavailable");
const result = await withPluginWorkspaceReplay({
pluginId: owner, companyId: workspace.companyId, projectId: workspace.projectId,
projectWorkspaceId: operation.source.id, executionWorkspaceId: workspace.id,
}, () => callWorkspaceProvider(db, workers, {
workspace,
pluginId: owner,
...readPreparationPlacement(workspace),
operation,
}));
if (result.kind !== "prepared")
throw conflict("Workspace reconciliation is incomplete");
const { workspaceProviderPreparationPlacement: _placement, ...metadata } = workspace.metadata ?? {};
const [updated] = await db.transaction(async (tx) => {
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
return tx
.update(executionWorkspaces)
.set({
providerRef: result.binding.workspaceRef,
branchName: result.branchName,
updatedAt: new Date(),
metadata: {
...metadata,
workspaceProvider: result.binding,
workspaceProviderBaseSha: result.baseSha,
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
createdByRuntime: result.branchCreatedByRuntime,
},
})
.where(
and(
eq(executionWorkspaces.id, workspace.id),
eq(executionWorkspaces.companyId, workspace.companyId),
sql`${executionWorkspaces.providerRef} IS NULL`,
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${workspace.metadata?.lifecycleGeneration ?? 0}`,
),
)
.returning();
});
if (!updated)
throw conflict("Workspace changed during reconciliation; retry inspection");
return updated;
}
async function inspectBinding(
db: Db,
workers: WorkspaceWorkers | undefined,
workspace: Workspace,
recoveryPullRequest?: PluginWorkspaceRecoveryPullRequest,
) {
const binding = readWorkspaceProviderBinding(workspace.metadata);
if (!binding) throw conflict("Workspace storage binding is unavailable");
const result = await callWorkspaceProvider(db, workers, {
workspace,
operation: { kind: "inspect", binding, ...(recoveryPullRequest ? { recoveryPullRequest } : {}) },
});
if (result.kind !== "inspection")
throw conflict("Workspace inspection is incomplete");
return result;
}
/** Reconcile durable lifecycle intent under the same lock for either read path. */
async function withRetainedWorkspace<T>(db: Db, workers: WorkspaceWorkers | undefined,
workspace: Workspace, operation: (workspace: Workspace) => Promise<T>) {
return withExecutionWorkspaceLifecycleLock(db, workspace.id, async () => {
workspace = await reconcilePluginWorkspaceReservation(db, workers, workspace);
if (workspace.metadata?.workspaceProviderRestoration)
workspace = await finishPendingRestoration(db, workers, workspace);
if (workspace.metadata?.workspaceProviderRemoval)
workspace = await finishPendingRemoval(db, workers, workspace);
return operation(workspace);
});
}
export async function inspectPluginWorkspace(db: Db, workers: WorkspaceWorkers | undefined, workspace: Workspace, recoveryPullRequest?: PluginWorkspaceRecoveryPullRequest) {
return withRetainedWorkspace(db, workers, workspace, (current) =>
current.metadata?.workspaceProviderRemoved === true
? Promise.resolve(removedInspection()) : inspectBinding(db, workers, current, recoveryPullRequest));
}
type PluginWorkspaceFinalizationPlan = { kind: "inspect" } | { kind: "finalize"; taskLease?: TaskPlacement };
export async function resolvePluginWorkspaceFinalization(db: Db, workspace: Workspace,
context: { runId: string; agentId: string; taskLease?: TaskPlacement | null }): Promise<PluginWorkspaceFinalizationPlan | undefined> {
if (workspace.providerType !== "plugin") return undefined;
if (context.taskLease) return { kind: "finalize", taskLease: context.taskLease };
const binding = readWorkspaceProviderBinding(workspace.metadata);
if (!binding) throw conflict("Workspace storage ownership is missing");
const selected = await pluginRegistryService(db).getById(binding.pluginId);
if (!selected) throw conflict("The workspace storage plugin is unavailable");
// Existing providers continue to use their established inspection contract.
if (selected.manifestJson.workspaceProvider?.finalizationVersion !== 1) return { kind: "inspect" };
const portableDrivers = new Set(selected.manifestJson.environmentDrivers?.filter(driver =>
driver.sessionHandoffVersion === 1).map(driver => driver.driverKey));
let taskLease: TaskPlacement | undefined;
if (context.taskLease === undefined && portableDrivers.size > 0) {
const leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, workspace.companyId),
eq(environmentLeases.executionWorkspaceId, workspace.id),
eq(environmentLeases.heartbeatRunId, context.runId),
eq(environmentLeases.status, "active"),
));
const placements = leases.flatMap(lease => {
const driverKey = lease.metadata?.driverKey;
return lease.metadata?.pluginId === binding.pluginId && lease.metadata?.agentId === context.agentId &&
lease.metadata?.runnerTaskLease === true && lease.environmentId && lease.providerLeaseId &&
typeof driverKey === "string" && portableDrivers.has(driverKey)
? [{ driverKey, environmentId: lease.environmentId, providerLeaseId: lease.providerLeaseId }] : [];
});
if (placements.length > 1) throw conflict("Workspace Launcher placement is ambiguous");
taskLease = placements[0];
}
return { kind: "finalize", taskLease };
}
/** Completion does not need the deletion-safety proof supplied by inspect. */
export async function finalizePluginWorkspace(db: Db, workers: WorkspaceWorkers | undefined,
workspace: Workspace, context: { runId: string; agentId: string; taskLease?: TaskPlacement | null;
plan?: PluginWorkspaceFinalizationPlan }) {
return withRetainedWorkspace(db, workers, workspace, async (current) => {
if (current.metadata?.workspaceProviderRemoved === true)
throw conflict("Workspace was removed before finalization");
const binding = readWorkspaceProviderBinding(current.metadata);
if (!binding) throw conflict("Workspace storage ownership is missing");
if (context.plan && !isDeepStrictEqual(binding, readWorkspaceProviderBinding(workspace.metadata)))
throw conflict("Workspace storage ownership changed");
const plan = context.plan ?? await resolvePluginWorkspaceFinalization(db, current, context);
if (!plan) throw conflict("The workspace storage plugin is unavailable");
if (plan.kind === "inspect") return inspectBinding(db, workers, current);
const taskLease = plan.taskLease;
const result = await callWorkspaceProvider(db, workers, {
workspace: current, runId: context.runId, agentId: context.agentId, taskLease, operation: { kind: "finalize", binding },
});
if (result.kind !== "finalized") throw conflict("Workspace finalization is incomplete");
return result;
});
}
const removedInspection = () => ({
kind: "inspection" as const,
headSha: null,
branchName: null,
hasDirtyTrackedFiles: false,
hasUntrackedFiles: false,
hasLocalOnlyData: false,
isMergedIntoBase: null,
remotelyRecoverable: true,
});
async function finishPendingRestoration(
db: Db,
workers: WorkspaceWorkers | undefined,
workspace: Workspace,
): Promise<Workspace> {
const pending = workspace.metadata?.workspaceProviderRestoration as {
generation?: unknown;
operation?: unknown;
};
const operation = workspaceProviderOperationSchema.safeParse(
pending?.operation,
);
const generation =
(workspace.metadata?.lifecycleGeneration as number | undefined) ?? 0;
if (
!operation.success ||
operation.data.kind !== "prepare" ||
pending.generation !== generation + 1 ||
JSON.stringify(operation.data.existing) !==
JSON.stringify(readWorkspaceProviderBinding(workspace.metadata)) ||
!["archived", "cleanup_failed"].includes(workspace.status)
)
throw conflict("Workspace restoration intent is unavailable or stale");
const result = await callWorkspaceProvider(db, workers, {
workspace,
operation: operation.data,
advanceGeneration: true,
});
if (result.kind !== "prepared")
throw conflict("Workspace restoration is incomplete");
const [updated] = await db.transaction(async (tx) => {
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
return tx
.update(executionWorkspaces)
.set({
providerRef: result.binding.workspaceRef,
branchName: result.branchName,
updatedAt: new Date(),
metadata: {
...workspace.metadata,
lifecycleGeneration: generation + 1,
workspaceProvider: result.binding,
workspaceProviderBaseSha: result.baseSha,
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
workspaceProviderRemoved: false,
workspaceProviderRestoration: null,
},
})
.where(
and(
eq(executionWorkspaces.id, workspace.id),
eq(executionWorkspaces.companyId, workspace.companyId),
eq(executionWorkspaces.status, workspace.status),
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${generation}`,
sql`${executionWorkspaces.metadata}->'workspaceProviderRestoration' = ${JSON.stringify(pending)}::jsonb`,
),
)
.returning();
});
if (!updated)
throw conflict("Workspace changed during restoration reconciliation");
return updated;
}
/** Replay the recorded delete before trying to inspect a checkout that may already be gone. */
async function finishPendingRemoval(
db: Db,
workers: WorkspaceWorkers | undefined,
workspace: Workspace,
): Promise<Workspace> {
if (workspace.metadata?.workspaceProviderRemoved === true) return workspace;
const pending = workspace.metadata?.workspaceProviderRemoval as
| { generation?: unknown; operation?: unknown; operationId?: unknown }
| undefined;
const parsed = workspaceProviderOperationSchema.safeParse(pending?.operation);
const binding = readWorkspaceProviderBinding(workspace.metadata);
if (
!parsed.success ||
parsed.data.kind !== "remove" ||
pending?.generation !== (workspace.metadata?.lifecycleGeneration ?? 0) ||
JSON.stringify(parsed.data.binding) !== JSON.stringify(binding)
)
throw conflict("Workspace removal intent is unavailable or stale");
if (pending.operationId === undefined && parsed.data.recoveryPullRequest)
throw conflict("Workspace removal identity is ambiguous; reconciliation requires the original request");
const operationId = createHash("sha256").update(JSON.stringify([workspace.id, pending.generation, parsed.data])).digest("hex");
if (pending.operationId !== undefined && pending.operationId !== operationId)
throw conflict("Workspace removal identity does not match its recorded payload");
const owner = workspace.metadata?.workspaceProviderOwner;
const result = await callWorkspaceProvider(db, workers, {
workspace,
pluginId: typeof owner === "string" ? owner : undefined,
operation: parsed.data,
removalOperationId: operationId,
});
if (result.kind !== "removed")
throw conflict("Workspace removal is incomplete");
if (workspace.metadata?.workspaceProviderFresh)
await releaseProviderRecoveryTransfers(workspace.companyId, workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace);
const [updated] = await db.transaction(async (tx) => {
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
return tx
.update(executionWorkspaces)
.set({
...(workspace.status === "cleanup_failed"
? { status: "archived", cleanupReason: null }
: {}),
metadata: {
...workspace.metadata,
workspaceProviderRemoval: null,
workspaceProviderRemoved: true,
},
updatedAt: new Date(),
})
.where(
and(
eq(executionWorkspaces.id, workspace.id),
eq(executionWorkspaces.companyId, workspace.companyId),
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${pending!.generation}`,
),
)
.returning();
});
if (!updated)
throw conflict("Workspace changed during removal reconciliation");
return updated;
}
export async function removePluginWorkspace(
db: Db,
workers: WorkspaceWorkers | undefined,
workspace: Workspace,
expectedHeadSha: string | null,
recoveryPullRequest?: PluginWorkspaceRecoveryPullRequest,
beforeRemove?: () => Promise<void>,
) {
workspace = await reconcilePluginWorkspaceReservation(db, workers, workspace);
if (workspace.metadata?.workspaceProviderRestoration)
throw conflict("Workspace restoration must be reconciled before removal");
if (workspace.metadata?.workspaceProviderRemoved === true) {
if (workspace.metadata?.workspaceProviderFresh)
await releaseProviderRecoveryTransfers(workspace.companyId, workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace);
return;
}
if (workspace.metadata?.workspaceProviderRemoval) {
await finishPendingRemoval(db, workers, workspace);
return;
}
const binding = readWorkspaceProviderBinding(workspace.metadata);
if (!binding) throw conflict("Workspace storage binding is unavailable");
const inspection = await inspectBinding(db, workers, workspace, recoveryPullRequest);
if (
inspection.hasDirtyTrackedFiles ||
inspection.hasUntrackedFiles ||
inspection.hasLocalOnlyData ||
inspection.headSha !== expectedHeadSha ||
(inspection.remotelyRecoverable !== true &&
inspection.isMergedIntoBase !== true)
) {
throw conflict(
"Workspace has local changes or cannot be recovered remotely",
);
}
await beforeRemove?.();
const generation = workspace.metadata?.lifecycleGeneration ?? 0;
const selectedProvider = await requireWorkspaceProviderWorker(db, workers, workspace.companyId, binding.pluginId);
const operation = workspaceProviderOperationSchema.parse({ kind: "remove", binding, expectedHeadSha,
...(recoveryPullRequest && selectedProvider.manifestJson.workspaceProvider?.recoveryPullRequestVersion === 1 ? { recoveryPullRequest } : {}) });
const operationId = createHash("sha256").update(JSON.stringify([workspace.id, generation, operation])).digest("hex");
const [pending] = await db.transaction(async (tx) => {
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
return tx
.update(executionWorkspaces)
.set({
metadata: {
...workspace.metadata,
workspaceProviderRemoval: { generation, operation, operationId },
},
updatedAt: new Date(),
})
.where(
and(
eq(executionWorkspaces.id, workspace.id),
eq(executionWorkspaces.companyId, workspace.companyId),
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${generation}`,
eq(executionWorkspaces.providerRef, binding.workspaceRef),
),
)
.returning();
});
if (!pending)
throw conflict("Workspace changed before removal could be recorded");
await finishPendingRemoval(db, workers, pending);
}
/** An authorized storage edit atomically retires closed, removed dependencies under the reopen fence. */
export async function retireRemovedPluginWorkspaceDependents(
db: Db,
projectId: string,
sourceId?: string,
) {
const rows = await db
.select({ id: executionWorkspaces.id })
.from(executionWorkspaces)
.where(
and(
eq(executionWorkspaces.projectId, projectId),
eq(executionWorkspaces.providerType, "plugin"),
sourceId
? eq(executionWorkspaces.projectWorkspaceId, sourceId)
: undefined,
),
);
for (const { id } of rows.sort((a, b) => a.id.localeCompare(b.id))) {
const acquired = await db.execute(
sql`select pg_try_advisory_xact_lock(hashtextextended(${`execution_workspace_lifecycle:${id}`}, 0)) as acquired`,
);
if (!acquired[0]?.acquired)
throw conflict("Workspace lifecycle is busy; retry the source edit");
const [row] = await db
.select()
.from(executionWorkspaces)
.where(eq(executionWorkspaces.id, id));
if (!row || row.metadata?.workspaceProviderRetired === true) continue;
if (
row.metadata?.workspaceProviderRemoved !== true ||
row.metadata?.workspaceProviderRestoration ||
!["archived", "cleanup_failed"].includes(row.status)
) {
throw conflict(
"Retire managed execution workspaces before changing their storage owner or repository source",
);
}
await db
.update(executionWorkspaces)
.set({
metadata: { ...row.metadata, workspaceProviderRetired: true },
updatedAt: new Date(),
})
.where(eq(executionWorkspaces.id, row.id));
}
}
/** Called inside Core's existing lifecycle fence after task authorization. */
export async function restorePluginWorkspace(
db: Db,
workers: WorkspaceWorkers | undefined,
workspace: Workspace,
) {
workspace = await reconcilePluginWorkspaceReservation(db, workers, workspace);
if (workspace.metadata?.workspaceProviderRetired === true)
throw conflict("Workspace storage was retired");
if (workspace.metadata?.workspaceProviderRemoval)
workspace = await finishPendingRemoval(db, workers, workspace);
const binding = readWorkspaceProviderBinding(workspace.metadata);
const source = workspace.metadata?.workspaceProviderSource;
if (!binding || !source)
throw conflict("Workspace restoration recipe is unavailable");
const operation = workspaceProviderOperationSchema.parse({
kind: "prepare",
source,
mode: workspace.mode,
strategy: workspace.strategyType,
branchName: workspace.metadata?.workspaceProviderFresh ? null : workspace.branchName,
branchMode: workspace.metadata?.workspaceProviderFresh ? "create" : "existing",
baseRef: workspace.baseRef,
existing: binding,
...(workspace.metadata?.workspaceProviderFresh ? { fresh: workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace } : {}),
});
const [pending] = await db.transaction(async (tx) => {
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
return tx
.update(executionWorkspaces)
.set({
metadata: {
...workspace.metadata,
workspaceProviderRemoved: false,
workspaceProviderRestoration: {
generation:
((workspace.metadata?.lifecycleGeneration as number) ?? 0) + 1,
operation,
},
},
updatedAt: new Date(),
})
.where(
and(
eq(executionWorkspaces.id, workspace.id),
eq(executionWorkspaces.companyId, workspace.companyId),
sql`coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${workspace.metadata?.lifecycleGeneration ?? 0}`,
),
)
.returning();
});
if (!pending)
throw conflict("Workspace changed before restoration could be recorded");
workspace = pending;
const result = await callWorkspaceProvider(db, workers, {
workspace,
operation,
advanceGeneration: true,
});
if (result.kind !== "prepared")
throw conflict("Workspace restoration is incomplete");
return {
...workspace.metadata,
workspaceProvider: result.binding,
workspaceProviderRemoved: false,
workspaceProviderRestoration: null,
workspaceProviderBaseSha: result.baseSha,
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
};
}