mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path > - Paperclip lets plugins own managed Git workspaces. > - A portable Runner task can move to a different physical allocation. > - The checkout must be present on the allocation that launches the Runner. > - Workspace preparation and finalization need exact placement when they retry. > - This change checks the placement at Runner admission and records it before finalization. ## Linked Issues or Issue Description **What existing behavior does this improve?** Workspace preparation and finalization for portable Runner tasks. **Current behavior** A task replacement can change the allocation after Git preparation. A lost finalization reply can leave retry without the original placement. **Proposed behavior** Prepare the checkout on the actual allocation before admission. Record finalization placement so an exact retry can use it. Portable ownership and controller recovery are in [#21066](https://github.com/paperclipai/paperclip/pull/21066). ## What Changed - Reserve the managed workspace before acquiring the portable task lease. - Pass the task placement to the workspace provider. Reuse the described task while placement stays the same and prepare again before admitting a replacement. - Record the initial preparation's run, agent, generation, and optional task placement before dispatch. Inspection and retries on the same Launcher replay the same request; VM and sandbox preparation use the same recovery path. - Keep provider request identity separate from Git credential authority: reservation reconciliation uses the existing project-consent path in a host-only scope bound to the plugin and workspace. Foreground preparation still requires an active run; consent, repository, and revocation checks are unchanged. - Acquire the current Launcher before retrying preparation. If placement changes, use ordinary preparation on the replacement so the provider can settle the prior writer and recover the retained checkout. No additional provider discovery call is added. - Select finalization placement before publishing the operation record, then dispatch and recover using that saved choice. Retain the Launcher during bounded finalization retries. - Preserve interrupted pre-upgrade finalization requests without placement, including their original operation IDs and provider routing. No migration or new durable mode flag is required. ## Verification - Final revision `74ea3e6712`: server typecheck and exact SDK source-pin verification (538 inputs) passed. - All 185 tests passed across the full plugin worker manager, project Git authorization, and plugin workspace integration suites (including 56 workspace tests). Independent verification reran the three lost-reply cases: inspection, same-Launcher retry, and replacement after deleting the original Launcher. The tests assert the correct request identity, durable intent, and clearing that intent after success. - Before the final retry-accounting fix, all 323 tests passed across workspace admission, runtime execution, native workspace finalization, plugin workspace integration, and terminal cleanup retry. Regressions exercise crashes after operation insertion but before dispatch in both finalization paths, exact replay after lease release, and retry after ambiguous placement clears without publishing an incomplete operation. Pre-upgrade interrupted requests retain their operation IDs despite an available portable lease. - Final retry-accounting fix: all 84 native workspace finalizer, failure, and supersession tests passed. A real missing-plugin regression first failed on the prior commit, then verified healthy-run progress, three-attempt exhaustion, terminal cleanup, and absence of a misleading workspace operation. Independent verification also passed the legacy and crash-before-dispatch cases. - Worker-process tests cover both preparation RPC methods: the provider still receives the original run and agent, maintenance receives project-scoped credential authority, and foreground or mismatched-scope requests remain denied. Database tests confirm a stopped run cannot authorize foreground Git access while valid project consent can authorize maintenance. - Existing metadata protection coverage proves callers cannot overwrite the private preparation intent or read it through workspace responses. - Two existing Launcher Git tests passed: retry preserves dirty files and the original base commit without fetching, and retry resumes the saved checkout intent while rejecting a changed pending branch. - Implementation was delegated to GPT-6.1 Sol and independently reviewed. The GitHub stack's automatic rebase preserved the original workspace-placement patch. - [Full CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706) passed on `74ea3e6712`, including server and workspace tests, Runner, end-to-end tests, typecheck, build, and canary dry run. Ordinary Greptile and [Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388) both scored `74ea3e6712` 5/5, with no unresolved findings. ## Risks - Live Daytona end-to-end qualification is pending. The checkout record persists on the mounted volume; replacement preparation relies on the provider confirming the previous writer is settled before using that retained checkout. Local tests cover Core request routing and Launcher's durable checkout recovery separately. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
ea400ecb07
commit
7f69ef8b44
23 files changed
+1029
-243
No files matched your search
@@ -1,10 +1,12 @@
|
||||
import { isDeepStrictEqual } from "node:util";
|
||||
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
|
||||
import { assertProjectStorageEnvironment, effectiveProjectWorkspacePolicy, requireWorkspaceProviderWorker, type WorkspaceWorkers } from "./project-workspace-provider.js";
|
||||
import { releaseProviderRecoveryTransfers, stageProviderFreshWorkspace } from "./provider-fresh-workspace.js";
|
||||
import { releaseProviderRecoveryTransfers } from "./provider-fresh-workspace.js";
|
||||
import { retireAbandonedPluginWorkspaceReservation } from "./plugin-workspace-reservations.js";
|
||||
import { randomUUID, createHash } from "node:crypto";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import {
|
||||
environmentLeases,
|
||||
executionWorkspaces,
|
||||
heartbeatRuns,
|
||||
issues,
|
||||
@@ -15,27 +17,45 @@ import {
|
||||
import {
|
||||
readWorkspaceProviderBinding,
|
||||
type WorkspaceProviderBinding,
|
||||
type ExecutionWorkspace,
|
||||
} from "@paperclipai/shared";
|
||||
import {
|
||||
pluginWorkspaceParamsSchema,
|
||||
workspaceProviderOperationSchema,
|
||||
type PluginWorkspaceOperation,
|
||||
type PluginWorkspaceRecoveryPullRequest,
|
||||
type PluginWorkspaceParams,
|
||||
type PluginWorkspaceResult,
|
||||
type PluginFreshWorkspace,
|
||||
} from "@paperclipai/plugin-sdk";
|
||||
import { invokeWorkspaceProvider } from "./plugin-workspace-preparation.js";
|
||||
import { callWorkspaceProvider } from "./plugin-workspace-preparation.js";
|
||||
import { withPluginWorkspaceReplay } from "./plugin-workspace-replay-context.js";
|
||||
import { pluginRegistryService } from "./plugin-registry.js";
|
||||
import { notifyTerminalWorkspaceWork } from "./terminal-workspace-work.js";
|
||||
import { withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifecycle-lock.js";
|
||||
import { WorkspaceProviderFailureError, workspaceProviderCallFailure } from "./workspace-provider-failure.js";
|
||||
import {
|
||||
assertTaskWorkspaceExecutionAccess,
|
||||
assertTaskWorkspaceSourceProjectAccess,
|
||||
} from "./task-workspace-access.js";
|
||||
import { forbidden, conflict } from "../errors.js";
|
||||
|
||||
export { callWorkspaceProvider } from "./plugin-workspace-preparation.js";
|
||||
export type { WorkspaceWorkers } from "./project-workspace-provider.js";
|
||||
type Workspace = typeof executionWorkspaces.$inferSelect;
|
||||
type TaskPlacement = NonNullable<PluginWorkspaceParams["taskLease"]>;
|
||||
const preparationPlacementSchema = pluginWorkspaceParamsSchema.pick({
|
||||
generation: true, runId: true, agentId: true, taskLease: true,
|
||||
});
|
||||
|
||||
function readPreparationPlacement(workspace: Workspace) {
|
||||
const saved = workspace.metadata?.workspaceProviderPreparationPlacement;
|
||||
if (saved === undefined) return undefined;
|
||||
const placement = preparationPlacementSchema.parse(saved);
|
||||
if (!placement.runId || !placement.agentId ||
|
||||
placement.generation !== (workspace.metadata?.lifecycleGeneration ?? 0))
|
||||
throw conflict("Workspace preparation placement changed");
|
||||
return placement;
|
||||
}
|
||||
type EnsureTaskWorkspace = (taskId: string, assertOwned: () => void) => Promise<void> | undefined;
|
||||
|
||||
export async function pluginWorkspaceSupportsMergeAncestry(db: Db, workers: WorkspaceWorkers | undefined, workspace: Workspace) {
|
||||
const pluginId = readWorkspaceProviderBinding(workspace.metadata)?.pluginId;
|
||||
@@ -44,102 +64,6 @@ export async function pluginWorkspaceSupportsMergeAncestry(db: Db, workers: Work
|
||||
return selected.manifestJson.workspaceProvider?.supportsMergeAncestry !== false;
|
||||
}
|
||||
|
||||
/** No paths or provider credentials are accepted from task callers. */
|
||||
export async function callWorkspaceProvider(
|
||||
db: Db,
|
||||
workers: WorkspaceWorkers | undefined,
|
||||
input: {
|
||||
workspace: Workspace;
|
||||
operation: PluginWorkspaceOperation;
|
||||
pluginId?: string;
|
||||
advanceGeneration?: boolean;
|
||||
runId?: string | null;
|
||||
agentId?: string | null;
|
||||
removalOperationId?: string;
|
||||
},
|
||||
) {
|
||||
const { workspace } = input;
|
||||
const binding = readWorkspaceProviderBinding(workspace.metadata);
|
||||
const pluginId = binding?.pluginId ?? input.pluginId;
|
||||
if (!pluginId || !workspace.projectId)
|
||||
throw conflict("Workspace storage ownership is missing");
|
||||
const selectedProvider = await requireWorkspaceProviderWorker(db, workers, workspace.companyId, pluginId, true);
|
||||
const verifyExisting = input.operation.kind === "prepare" && input.operation.verifyExisting === true;
|
||||
if (verifyExisting && selectedProvider.manifestJson.workspaceProvider?.workspaceVerificationVersion !== 1)
|
||||
throw conflict("Workspace verification is unavailable");
|
||||
if (input.operation.kind === "prepare" && input.operation.fresh && selectedProvider.manifestJson.workspaceProvider?.freshWorkspaceVersion !== 1)
|
||||
throw conflict("Fresh workspace preparation is unavailable");
|
||||
if (input.operation.kind === "finalize" &&
|
||||
(selectedProvider.manifestJson.workspaceProvider?.finalizationVersion !== 1 || !input.runId || !input.agentId))
|
||||
throw conflict("Workspace finalization is unavailable");
|
||||
const generation =
|
||||
typeof workspace.metadata?.lifecycleGeneration === "number"
|
||||
? workspace.metadata.lifecycleGeneration
|
||||
: 0;
|
||||
const operationGeneration = generation + (input.advanceGeneration ? 1 : 0);
|
||||
const operation = workspaceProviderOperationSchema.parse(input.operation);
|
||||
if (input.removalOperationId !== undefined && (operation.kind !== "remove" ||
|
||||
input.removalOperationId !== createHash("sha256").update(JSON.stringify([workspace.id, operationGeneration, operation])).digest("hex")))
|
||||
throw conflict("Workspace removal identity does not match its recorded payload");
|
||||
if ((operation.kind === "inspect" || operation.kind === "remove") && input.removalOperationId === undefined
|
||||
&& selectedProvider.manifestJson.workspaceProvider?.recoveryPullRequestVersion !== 1) delete operation.recoveryPullRequest;
|
||||
if (operation.kind === "prepare" && operation.fresh && !verifyExisting) {
|
||||
if (!workspace.sourceIssueId) throw conflict("Fresh workspace task ownership is missing");
|
||||
operation.fresh = await stageProviderFreshWorkspace({ db, companyId: workspace.companyId,
|
||||
issueId: workspace.sourceIssueId, ...operation.fresh, existing: operation.fresh });
|
||||
}
|
||||
const params: Omit<PluginWorkspaceParams, "deadline"> = {
|
||||
pluginId,
|
||||
companyId: workspace.companyId,
|
||||
projectId: workspace.projectId,
|
||||
executionWorkspaceId: workspace.id,
|
||||
operationId: input.removalOperationId ?? (
|
||||
operation.kind === "inspect"
|
||||
? randomUUID()
|
||||
: createHash("sha256")
|
||||
.update(
|
||||
JSON.stringify([workspace.id, operationGeneration, operation,
|
||||
...(operation.kind === "finalize" ? [input.runId, input.agentId] : [])]),
|
||||
)
|
||||
.digest("hex")),
|
||||
generation: operationGeneration,
|
||||
runId: input.runId ?? null,
|
||||
agentId: input.agentId ?? null,
|
||||
operation,
|
||||
};
|
||||
let result: PluginWorkspaceResult;
|
||||
try {
|
||||
result = await invokeWorkspaceProvider(workers!, params);
|
||||
} catch (error) {
|
||||
// A worker that never received the call, a lost reply and a provider-reported
|
||||
// failure need different recovery; only a closed provider reason crosses over.
|
||||
if (error instanceof WorkspaceProviderFailureError) throw error;
|
||||
throw workspaceProviderCallFailure(error);
|
||||
}
|
||||
// Recheck enablement and ownership after an external effect, including an ambiguous retry.
|
||||
await requireWorkspaceProviderWorker(db, workers, workspace.companyId, pluginId);
|
||||
const [fresh] = await db
|
||||
.select()
|
||||
.from(executionWorkspaces)
|
||||
.where(
|
||||
and(
|
||||
eq(executionWorkspaces.id, workspace.id),
|
||||
eq(executionWorkspaces.companyId, workspace.companyId),
|
||||
),
|
||||
);
|
||||
if (
|
||||
!fresh ||
|
||||
fresh.status !== workspace.status ||
|
||||
(fresh.metadata?.lifecycleGeneration ?? 0) !== generation ||
|
||||
fresh.providerRef !== workspace.providerRef
|
||||
) {
|
||||
throw conflict("Workspace changed during storage preparation");
|
||||
}
|
||||
if (result.kind === "prepared" && result.binding.pluginId !== pluginId)
|
||||
throw conflict("Workspace provider ownership changed");
|
||||
return result;
|
||||
}
|
||||
|
||||
export async function preparePluginWorkspace(
|
||||
db: Db,
|
||||
workers: WorkspaceWorkers | undefined,
|
||||
@@ -162,6 +86,8 @@ export async function preparePluginWorkspace(
|
||||
fresh?: PluginFreshWorkspace;
|
||||
expectedBindingRevision?: number;
|
||||
verifyExisting?: true;
|
||||
acquireTaskLease?: (workspace: Pick<ExecutionWorkspace, "id" | "mode">) => Promise<{ leaseId: string; taskLease: TaskPlacement }>;
|
||||
onTaskPrepared?: (ensure: EnsureTaskWorkspace) => void;
|
||||
},
|
||||
) {
|
||||
const [previous] = input.existingWorkspaceId
|
||||
@@ -260,11 +186,11 @@ export async function preparePluginWorkspace(
|
||||
? null
|
||||
: input.responsibleUserId,
|
||||
};
|
||||
const authorize = async () => {
|
||||
const authorize = async (workspaceId = existing?.id) => {
|
||||
await assertTaskWorkspaceExecutionAccess(db, actor, input.companyId, {
|
||||
issueId: input.issueId!,
|
||||
...(existing
|
||||
? { workspaceId: existing.id }
|
||||
...(workspaceId
|
||||
? { workspaceId }
|
||||
: { projectWorkspaceId: input.projectWorkspaceId }),
|
||||
});
|
||||
await assertTaskWorkspaceSourceProjectAccess(
|
||||
@@ -417,7 +343,12 @@ export async function preparePluginWorkspace(
|
||||
.where(eq(issues.id, task.id));
|
||||
return row!;
|
||||
}));
|
||||
return withExecutionWorkspaceLifecycleLock(db, workspace.id, async (tx) => {
|
||||
const driverKey = input.environment?.driver === "plugin" ? input.environment.config?.driverKey : undefined;
|
||||
const placement = selectedProvider.manifestJson.environmentDrivers?.some(driver =>
|
||||
driver.driverKey === driverKey && driver.sessionHandoffVersion === 1)
|
||||
? await input.acquireTaskLease?.({ id: workspace.id, mode: mode as ExecutionWorkspace["mode"] }) : undefined;
|
||||
const prepare = (workspace: Workspace, taskLease?: TaskPlacement, replacement?: { leaseId: string; assertOwned: () => void }) => withExecutionWorkspaceLifecycleLock(db, workspace.id, async (tx) => {
|
||||
replacement?.assertOwned();
|
||||
const [current] = await tx
|
||||
.select()
|
||||
.from(executionWorkspaces)
|
||||
@@ -433,17 +364,50 @@ export async function preparePluginWorkspace(
|
||||
) {
|
||||
throw conflict("Workspace lifecycle changed; reopen before preparing");
|
||||
}
|
||||
if (taskLease) await authorize(replacement ? workspace.id : undefined);
|
||||
const replacementScope = replacement && taskLease ? and(
|
||||
eq(environmentLeases.id, replacement.leaseId),
|
||||
eq(environmentLeases.companyId, input.companyId),
|
||||
eq(environmentLeases.heartbeatRunId, input.runId),
|
||||
eq(environmentLeases.executionWorkspaceId, workspace.id),
|
||||
eq(environmentLeases.environmentId, taskLease.environmentId),
|
||||
eq(environmentLeases.providerLeaseId, taskLease.providerLeaseId),
|
||||
eq(environmentLeases.status, "active"),
|
||||
sql`${environmentLeases.metadata}->>'pluginId' = ${pluginId} and ${environmentLeases.metadata}->>'driverKey' = ${taskLease.driverKey}`,
|
||||
sql`${environmentLeases.releasedAt} is null and (${environmentLeases.expiresAt} is null or ${environmentLeases.expiresAt} > clock_timestamp())`,
|
||||
sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
|
||||
) : undefined;
|
||||
if (replacementScope && !(await tx.select({ id: environmentLeases.id }).from(environmentLeases).where(replacementScope)).length)
|
||||
throw conflict("Workspace Launcher placement changed");
|
||||
if (!current.providerRef && !input.verifyExisting) {
|
||||
const generation = current.metadata?.lifecycleGeneration ?? 0;
|
||||
const intent = { generation, runId: input.runId, agentId: input.agentId, ...(taskLease ? { taskLease } : {}) };
|
||||
const previous = readPreparationPlacement(current);
|
||||
if (!isDeepStrictEqual(previous, intent)) {
|
||||
// The request must survive a lost reply or rollback of the lifecycle transaction.
|
||||
const [recorded] = await db.update(executionWorkspaces).set({
|
||||
metadata: sql`coalesce(${executionWorkspaces.metadata}, '{}'::jsonb) || ${JSON.stringify({ workspaceProviderPreparationPlacement: intent })}::jsonb`,
|
||||
}).where(and(eq(executionWorkspaces.id, current.id), eq(executionWorkspaces.companyId, current.companyId),
|
||||
sql`${executionWorkspaces.providerRef} is null
|
||||
and coalesce((${executionWorkspaces.metadata}->>'lifecycleGeneration')::int, 0) = ${generation}
|
||||
and coalesce(${executionWorkspaces.metadata}->'workspaceProviderPreparationPlacement', 'null'::jsonb)
|
||||
= ${JSON.stringify(previous ?? null)}::jsonb`)).returning({ id: executionWorkspaces.id });
|
||||
if (!recorded) throw conflict("Workspace preparation placement changed");
|
||||
}
|
||||
}
|
||||
const result = await callWorkspaceProvider(tx as unknown as Db, workers, {
|
||||
workspace,
|
||||
pluginId,
|
||||
runId: input.runId,
|
||||
agentId: input.agentId,
|
||||
taskLease,
|
||||
assertOwned: replacement?.assertOwned,
|
||||
operation: {
|
||||
kind: "prepare",
|
||||
source: sourceInput,
|
||||
mode: workspace.mode as "isolated_workspace",
|
||||
strategy: workspace.strategyType as "git_worktree",
|
||||
branchName: workspace.metadata?.workspaceProviderFresh ? null : branchName,
|
||||
branchName: workspace.metadata?.workspaceProviderFresh ? null : workspace.branchName,
|
||||
branchMode:
|
||||
(workspace.metadata?.workspaceProviderBranchMode as
|
||||
| "create"
|
||||
@@ -452,13 +416,15 @@ export async function preparePluginWorkspace(
|
||||
input.branchMode ??
|
||||
"create",
|
||||
baseRef: workspace.baseRef,
|
||||
existing: saved,
|
||||
existing: readWorkspaceProviderBinding(workspace.metadata),
|
||||
...(input.verifyExisting ? { verifyExisting: true } : {}),
|
||||
...(workspace.metadata?.workspaceProviderFresh ? { fresh: workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace } : {}),
|
||||
},
|
||||
});
|
||||
if (result.kind !== "prepared")
|
||||
throw conflict("Workspace preparation is incomplete");
|
||||
if (replacement && result.branchName !== workspace.branchName)
|
||||
throw conflict("Workspace Launcher prepared a different branch");
|
||||
const savedBase = workspace.metadata?.workspaceProviderBaseSha;
|
||||
if (input.verifyExisting && (result.created || !result.baseSha ||
|
||||
(saved && result.binding.workspaceRef !== saved.workspaceRef) ||
|
||||
@@ -479,7 +445,14 @@ export async function preparePluginWorkspace(
|
||||
.where(eq(issues.id, input.issueId!));
|
||||
if (task?.executionWorkspaceId !== workspace.id)
|
||||
throw conflict("Task workspace binding changed during preparation");
|
||||
if (replacementScope) {
|
||||
const [currentLease] = await tx.update(environmentLeases).set({
|
||||
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ workspaceProvider: result.binding })}::jsonb`,
|
||||
}).where(replacementScope).returning({ id: environmentLeases.id });
|
||||
if (!currentLease) throw conflict("Workspace Launcher placement changed during preparation");
|
||||
}
|
||||
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
|
||||
const { workspaceProviderPreparationPlacement: _placement, ...metadata } = current.metadata ?? {};
|
||||
const [updated] = await tx
|
||||
.update(executionWorkspaces)
|
||||
.set({
|
||||
@@ -489,7 +462,7 @@ export async function preparePluginWorkspace(
|
||||
lastUsedAt: new Date(),
|
||||
updatedAt: new Date(),
|
||||
metadata: {
|
||||
...workspace.metadata,
|
||||
...metadata,
|
||||
workspaceProvider: result.binding,
|
||||
workspaceProviderBaseSha: result.baseSha,
|
||||
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
|
||||
@@ -511,6 +484,19 @@ export async function preparePluginWorkspace(
|
||||
throw conflict("Workspace changed before preparation could be recorded");
|
||||
return updated!;
|
||||
});
|
||||
const prepared = await prepare(workspace, placement?.taskLease);
|
||||
if (placement && input.onTaskPrepared) {
|
||||
let preparedTaskId = placement.taskLease.providerLeaseId;
|
||||
let preparedWorkspace = prepared;
|
||||
input.onTaskPrepared((taskId, assertOwned) => {
|
||||
if (taskId === preparedTaskId) return;
|
||||
return prepare(preparedWorkspace, { ...placement.taskLease, providerLeaseId: taskId }, { leaseId: placement.leaseId, assertOwned }).then(updated => {
|
||||
preparedWorkspace = updated;
|
||||
preparedTaskId = taskId;
|
||||
});
|
||||
});
|
||||
}
|
||||
return prepared;
|
||||
}
|
||||
|
||||
/** Recover a lost prepare result even when the originating run has already stopped. */
|
||||
@@ -534,7 +520,7 @@ async function reconcilePluginWorkspaceReservation(
|
||||
const owner = workspace.metadata?.workspaceProviderOwner;
|
||||
const retired = typeof owner === "string" ? await retireAbandonedPluginWorkspaceReservation(db, workspace) : null;
|
||||
if (retired) return retired;
|
||||
if (typeof owner !== "string" || !workspace.metadata?.workspaceProviderSource)
|
||||
if (typeof owner !== "string" || !workspace.projectId || !workspace.metadata?.workspaceProviderSource)
|
||||
throw conflict("Workspace storage reservation is unavailable");
|
||||
const operation = workspaceProviderOperationSchema.parse({
|
||||
kind: "prepare",
|
||||
@@ -547,13 +533,19 @@ async function reconcilePluginWorkspaceReservation(
|
||||
existing: null,
|
||||
...(workspace.metadata?.workspaceProviderFresh ? { fresh: workspace.metadata.workspaceProviderFresh as PluginFreshWorkspace } : {}),
|
||||
});
|
||||
const result = await callWorkspaceProvider(db, workers, {
|
||||
if (operation.kind !== "prepare") throw conflict("Workspace preparation is unavailable");
|
||||
const result = await withPluginWorkspaceReplay({
|
||||
pluginId: owner, companyId: workspace.companyId, projectId: workspace.projectId,
|
||||
projectWorkspaceId: operation.source.id, executionWorkspaceId: workspace.id,
|
||||
}, () => callWorkspaceProvider(db, workers, {
|
||||
workspace,
|
||||
pluginId: owner,
|
||||
...readPreparationPlacement(workspace),
|
||||
operation,
|
||||
});
|
||||
}));
|
||||
if (result.kind !== "prepared")
|
||||
throw conflict("Workspace reconciliation is incomplete");
|
||||
const { workspaceProviderPreparationPlacement: _placement, ...metadata } = workspace.metadata ?? {};
|
||||
const [updated] = await db.transaction(async (tx) => {
|
||||
await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
|
||||
return tx
|
||||
@@ -563,7 +555,7 @@ async function reconcilePluginWorkspaceReservation(
|
||||
branchName: result.branchName,
|
||||
updatedAt: new Date(),
|
||||
metadata: {
|
||||
...workspace.metadata,
|
||||
...metadata,
|
||||
workspaceProvider: result.binding,
|
||||
workspaceProviderBaseSha: result.baseSha,
|
||||
...(result.destinationRef !== undefined ? { workspaceProviderDestinationRef: result.destinationRef } : {}),
|
||||
@@ -621,20 +613,58 @@ export async function inspectPluginWorkspace(db: Db, workers: WorkspaceWorkers |
|
||||
? Promise.resolve(removedInspection()) : inspectBinding(db, workers, current, recoveryPullRequest));
|
||||
}
|
||||
|
||||
type PluginWorkspaceFinalizationPlan = { kind: "inspect" } | { kind: "finalize"; taskLease?: TaskPlacement };
|
||||
|
||||
export async function resolvePluginWorkspaceFinalization(db: Db, workspace: Workspace,
|
||||
context: { runId: string; agentId: string; taskLease?: TaskPlacement | null }): Promise<PluginWorkspaceFinalizationPlan | undefined> {
|
||||
if (workspace.providerType !== "plugin") return undefined;
|
||||
if (context.taskLease) return { kind: "finalize", taskLease: context.taskLease };
|
||||
const binding = readWorkspaceProviderBinding(workspace.metadata);
|
||||
if (!binding) throw conflict("Workspace storage ownership is missing");
|
||||
const selected = await pluginRegistryService(db).getById(binding.pluginId);
|
||||
if (!selected) throw conflict("The workspace storage plugin is unavailable");
|
||||
// Existing providers continue to use their established inspection contract.
|
||||
if (selected.manifestJson.workspaceProvider?.finalizationVersion !== 1) return { kind: "inspect" };
|
||||
const portableDrivers = new Set(selected.manifestJson.environmentDrivers?.filter(driver =>
|
||||
driver.sessionHandoffVersion === 1).map(driver => driver.driverKey));
|
||||
let taskLease: TaskPlacement | undefined;
|
||||
if (context.taskLease === undefined && portableDrivers.size > 0) {
|
||||
const leases = await db.select().from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, workspace.companyId),
|
||||
eq(environmentLeases.executionWorkspaceId, workspace.id),
|
||||
eq(environmentLeases.heartbeatRunId, context.runId),
|
||||
eq(environmentLeases.status, "active"),
|
||||
));
|
||||
const placements = leases.flatMap(lease => {
|
||||
const driverKey = lease.metadata?.driverKey;
|
||||
return lease.metadata?.pluginId === binding.pluginId && lease.metadata?.agentId === context.agentId &&
|
||||
lease.metadata?.runnerTaskLease === true && lease.environmentId && lease.providerLeaseId &&
|
||||
typeof driverKey === "string" && portableDrivers.has(driverKey)
|
||||
? [{ driverKey, environmentId: lease.environmentId, providerLeaseId: lease.providerLeaseId }] : [];
|
||||
});
|
||||
if (placements.length > 1) throw conflict("Workspace Launcher placement is ambiguous");
|
||||
taskLease = placements[0];
|
||||
}
|
||||
return { kind: "finalize", taskLease };
|
||||
}
|
||||
|
||||
/** Completion does not need the deletion-safety proof supplied by inspect. */
|
||||
export async function finalizePluginWorkspace(db: Db, workers: WorkspaceWorkers | undefined,
|
||||
workspace: Workspace, context: { runId: string; agentId: string }) {
|
||||
workspace: Workspace, context: { runId: string; agentId: string; taskLease?: TaskPlacement | null;
|
||||
plan?: PluginWorkspaceFinalizationPlan }) {
|
||||
return withRetainedWorkspace(db, workers, workspace, async (current) => {
|
||||
if (current.metadata?.workspaceProviderRemoved === true)
|
||||
throw conflict("Workspace was removed before finalization");
|
||||
const binding = readWorkspaceProviderBinding(current.metadata);
|
||||
if (!binding) throw conflict("Workspace storage ownership is missing");
|
||||
const selected = await requireWorkspaceProviderWorker(db, workers, current.companyId, binding.pluginId);
|
||||
// Existing providers continue to use their established inspection contract.
|
||||
if (selected.manifestJson.workspaceProvider?.finalizationVersion !== 1)
|
||||
return inspectBinding(db, workers, current);
|
||||
if (context.plan && !isDeepStrictEqual(binding, readWorkspaceProviderBinding(workspace.metadata)))
|
||||
throw conflict("Workspace storage ownership changed");
|
||||
const plan = context.plan ?? await resolvePluginWorkspaceFinalization(db, current, context);
|
||||
if (!plan) throw conflict("The workspace storage plugin is unavailable");
|
||||
if (plan.kind === "inspect") return inspectBinding(db, workers, current);
|
||||
const taskLease = plan.taskLease;
|
||||
const result = await callWorkspaceProvider(db, workers, {
|
||||
workspace: current, ...context, operation: { kind: "finalize", binding },
|
||||
workspace: current, runId: context.runId, agentId: context.agentId, taskLease, operation: { kind: "finalize", binding },
|
||||
});
|
||||
if (result.kind !== "finalized") throw conflict("Workspace finalization is incomplete");
|
||||
return result;
|
||||
|
||||
Reference in new issue
Block a user