mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 14:19:00 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online import preserves old local files before the web server can use a disposable filesystem. > - A parent import must authenticate saved child archives before it can omit their directories. > - The operation journal preserves import identity, but completed child checks existed only in process memory. > - A container restart therefore repeated downloads, decryption and restoration that had already succeeded. > - This pull request saves a versioned verification receipt on each immutable child snapshot. > - A replacement process can reuse those checks while it reacquires locks and verifies current local files. ## Linked Issues or Issue Description Refs #19854 for verification reuse within a live source fence. Refs #19873 for cohort import. Refs #20298 for the separate remote-hold admission fix. Searched related workspace verification and restart PRs; none provides durable child verification. **What happened?** Replacing a server container interrupts a long online import. The original operation and snapshot survive, but a later invocation downloads, decrypts and restores completed child archives again. The current cache ends with the source-lock owner. **Expected behavior** Completed immutable checks survive process and scratch loss. Resume keeps the same operation and snapshot, reuses only correctly bound verification, and publishes once. Current local files and Git state must still be checked under fresh source locks. **Steps to reproduce** 1. Import two nested child workspaces, then start the parent import. 2. Kill the importer after a child verification finishes. Remove its scratch directory. 3. Resume in a new process. Before this fix, the completed child is downloaded and restored again. **Paperclip version or commit** The regression fails on the previous migration verifier. The final branch includes master at `189fefb99183953be37f0d1084275fbb0f35d6e2`. **Deployment mode** Managed Linux server with PostgreSQL and explicit S3 storage configuration. ## What Changed - Add optional opaque storage namespace identity. S3 binds it to the explicit endpoint, region, bucket, normalized prefix and addressing style. Implicit endpoints retain full verification. - Save a versioned child-snapshot receipt only after canonical crypto, manifest and file restoration succeed and a second HEAD confirms the same strong ETag and size. - Bind reuse to the complete snapshot reference, key material, child lineage, source operation and storage namespace. Require a fresh strong HEAD and canonical validation that the current secret provider can unwrap the saved snapshot key. - Keep current directory/Git capture and source-owner checks on every invocation. Read-only verifier callers do not write checkpoints. - Merge checkpoint metadata under a short row lock without changing parent journal hashes, snapshot retention or update timestamps. Bound reads and writes. Lock timeouts, statement timeouts, deadlocks and serialization conflicts are safe cache misses. Other errors remain fatal. - Extend real PostgreSQL/Git/archive tests with process death, scratch removal, resume, changed bindings and single publication. ## Verification - Baseline RED: two import invocations read each child twice. With durable receipts, each child is read once across both invocations. - Real process regression passed on the final rebased head: SIGKILL the owned importer after one checkpoint, delete all scratch, and resume in a new Node process. It retains the same operation and snapshot, reads each child once across both processes, and records one publication. - 53 focused cases passed in 104.86 seconds on the final production bytes before rebase. They cover existing nested/cohort behavior, namespace identity, changed source/reference/key/lineage/ETag, unknown proof versions, weak validators and bounded row contention. - After clean rebase and stronger scratch/snapshot assertions, the real process case passed again in 18.54 seconds. Server TypeScript, module boundaries and diff checks passed on final head `9da68ee7ce4a778201ef85e21a11c49eeea7bc16`. Required CI supplies final-head broad coverage. - The initial restricted-network dependency setup failed; unchanged preflight passed with normal network access. Initial TypeScript failures in test pipe assertions were corrected. No failed attempt is counted as a pass. - Review found a missing current-master-key check. A real restart regression with all child checkpoints complete reproduced unsafe publication after switching to a different valid master key. Follow-up `7ea92c2abd9ffdb2e7dd81cecf62c8fc7a6e74a5` delegates to the canonical snapshot-key owner before both durable and in-memory cache hits. - The fixed process regressions passed 2/2. Ten key-authority and existing identity/source guard cases passed. The wrong-key process now refuses with zero additional child payload GETs, no publication, and unchanged original journal/snapshot rows. Server typecheck, module boundaries and diff checks passed on the follow-up. - Review also found that transient database failures could make the optional checkpoint cache stop a valid import. Read and save now fall back only for SQLSTATE `55P03`, `57014`, `40P01`, and `40001`. Connection loss, identity changes, crypto failures, programming errors and cancellation remain fatal. Reads use a read-only transaction with one-second lock and eight-second statement limits. - The checkpoint database-fault regression failed two cases before the fix. All four recoverable import cases then passed. The final selected run passed 21 cases, including fatal-error and cancellation controls, real row contention, both process-restart cases and key/identity guards. Server typecheck and boundaries passed. Fault injection is limited to the checkpoint database boundary; the importer, PostgreSQL, Git, archives, storage and publication remain real. - Latest master was merged without rewriting published history. Six real import/process-restart cases passed again on final head `8c8e4213a680ebb5c8d6864416bc3a4f271e1e76` in 38.03 seconds. [Required CI](https://github.com/paperclipai/paperclip/actions/runs/42658248222) finished with 54 successful checks and four intentional skips; Snyk passed. [Fresh review](https://github.com/paperclipai/paperclip/pull/20311#issuecomment-6745354381) is 5/5 with all four threads addressed and resolved. Hosted verification after normal release delivery remains pending. ## Risks - A stale receipt could skip authentication of a different object. Complete reference/lineage/namespace binding, fresh strong HEAD and row-locked save validation prevent that reuse. Missing or unsupported hints trigger full verification. - Mutable local files can change between attempts. Their source locks and fresh captures remain required; a checkpoint never certifies current local bytes. - Checkpoint persistence can contend with other snapshot owners. It has one-second lock and eight-second statement limits; the four classified transient SQL errors retain the normal fully verified path. Other database errors stop the import so lost source ownership cannot be hidden. - No schema, archive format, public API or saved-run identity change. Older snapshots without receipts remain readable and get full verification. Existing data and recovery obligations stay intact. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. Context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
810 lines
65 KiB
TypeScript
810 lines
65 KiB
TypeScript
import { createHash, randomUUID } from "node:crypto";
|
|
import { z } from "zod";
|
|
import fs from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { and, eq, inArray, ne, or, sql } from "drizzle-orm";
|
|
import {
|
|
companies, environmentLeases, executionWorkspaceRepositories, executionWorkspaceRuntimeLeases, executionWorkspaceSnapshots, executionWorkspaces,
|
|
heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, projects, workspaceOperations, workspaceRuntimeServices, type Db,
|
|
} from "@paperclipai/db";
|
|
import { readRemoteWorkspaceAuthority, remoteWorkspaceSourceSchema, type RemoteWorkspaceAuthority, type RemoteWorkspaceSource } from "@paperclipai/shared";
|
|
import { offlineMigrationExecutionWhere } from "./offline-migration-execution.js";
|
|
import { inspectFinalizedWorkspaceRecoveryHold, type FinalizedWorkspaceRecoveryHold } from "./finalized-workspace-recovery-migration.js";
|
|
import { workspaceExportCauses } from "./workspace-recovery-source.js";
|
|
import type { StorageService } from "../storage/types.js";
|
|
import { persistActivity, publishActivity } from "./activity-log.js";
|
|
import { tryAcquireExecutionWorkspaceLifecycleLock, withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifecycle-lock.js";
|
|
import { captureRemoteWorkspaceDirectory, restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js";
|
|
import { beginWorkspaceSnapshot, loadWorkspaceSnapshotArchive, recoverWorkspaceSnapshotArchive, uploadWorkspaceSnapshot } from "./remote-workspace-snapshot-storage.js";
|
|
import { notifyRemoteWorkspaceSnapshots } from "./remote-workspace-snapshot-retention.js";
|
|
import { materializeTaskDirectorySnapshot, removeTaskDirectoryTree } from "./isolated-task-directory-snapshot.js";
|
|
import { loadTaskDirectoryRestorePoint, mayBeIsolatedTaskDirectoryWorkspace, readTaskDirectoryRetention, validateTaskDirectoryRestorePointRef } from "./isolated-task-directory-retention.js";
|
|
import { inspectArchivedGitMigrationSource, lockArchivedGitMigrationSources, materializeArchivedGitMigrationSource, type ArchivedGitMigrationSource } from "./remote-workspace-migration-git.js";
|
|
import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
|
|
import { isLegacyWorkspaceTaskRelated, readLegacyWorkspaceRecoveryAllocations, LEGACY_WORKSPACE_RECOVERY_MIGRATION, type LegacyWorkspaceRecoverySource } from "./legacy-workspace-recovery-migration.js";
|
|
import { createNestedWorkspaceImportVerifier, onlineOrdinarySourceBlockers, offlineWorkspaceMigrationBlockers, prepareNestedWorkspaceImports, validateImportedWorkspaceChildren, workspaceMigrationLayoutBlockers, type WorkspaceMigrationCohort } from "./remote-workspace-migration-tree.js";
|
|
|
|
import { inspectOnlineWorkspaceSource, requiresOnlineWorkspaceSource, type OnlineWorkspaceSource } from "./online-workspace-source.js";
|
|
import { withWorkspaceSourceLocks, withIndependentWorkspaceSourceLocks, IsolatedRepositorySourceBusy } from "./workspace-source-lock.js";
|
|
import { inspectOnlineLegacyWorkspaceRecoverySource, inspectOnlineLegacyWorkspaceRecoveryHold, lockOnlineLegacyWorkspaceRecoverySources, type OnlineLegacyWorkspaceRecoveryHold } from "./online-legacy-workspace-recovery.js";
|
|
import { inspectNativeWorkspaceMigration, nativeWorkspaceMigrationMarker } from "./native-runtime/native-workspace-migration.js";
|
|
|
|
const SCHEMA = "paperclip.remote-workspace-migration.v1";
|
|
type Workspace = typeof executionWorkspaces.$inferSelect;
|
|
type ReadDb = Pick<Db, "select">;
|
|
type OnlineRecoveryHold = OnlineLegacyWorkspaceRecoveryHold & { actionId: string };
|
|
async function lockMigrationParents(db: ReadDb, companyId: string, issueId: string | null, recoveryIssueIds: readonly string[] = []) {
|
|
await db.select({ id: companies.id }).from(companies).where(eq(companies.id, companyId)).for("key share");
|
|
for (const id of [...new Set([...(issueId ? [issueId] : []), ...recoveryIssueIds])].sort())
|
|
await db.select({ id: issues.id }).from(issues).where(and(
|
|
eq(issues.companyId, companyId), eq(issues.id, id))).for("update", recoveryIssueIds.length ? { noWait: true } : undefined);
|
|
}
|
|
|
|
const object = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
|
function canonical(value: unknown): string {
|
|
if (Array.isArray(value)) return `[${value.map(canonical).join(",")}]`;
|
|
if (value && typeof value === "object") return `{${Object.entries(value).sort(([a], [b]) => a.localeCompare(b)).map(([key, item]) => `${JSON.stringify(key)}:${canonical(item)}`).join(",")}}`;
|
|
return JSON.stringify(value) ?? "null";
|
|
}
|
|
function fingerprint(workspace: Workspace) {
|
|
const metadata = { ...workspace.metadata }; delete metadata.remoteWorkspaceMigration;
|
|
return createHash("sha256").update(canonical({ companyId: workspace.companyId, id: workspace.id, projectId: workspace.projectId,
|
|
projectWorkspaceId: workspace.projectWorkspaceId, sourceIssueId: workspace.sourceIssueId, mode: workspace.mode, strategyType: workspace.strategyType,
|
|
providerType: workspace.providerType, cwd: workspace.cwd, providerRef: workspace.providerRef, repoUrl: workspace.repoUrl,
|
|
baseRef: workspace.baseRef, branchName: workspace.branchName, status: workspace.status, closedAt: workspace.closedAt?.toISOString(), metadata })).digest("hex");
|
|
}
|
|
const gitSourceProof = (source: ArchivedGitMigrationSource | null) => source ? {
|
|
commit: source.commit, sourcePath: source.sourcePath, cleanupOperationId: source.cleanupOperationId,
|
|
...(source.retainedBranch ? { retainedBranch: source.retainedBranch } : {}),
|
|
} : null;
|
|
|
|
async function isUnmaterializedRepositoryRequest(workspace: Workspace,
|
|
repository: typeof executionWorkspaceRepositories.$inferSelect) {
|
|
if (!workspace.cwd || repository.state !== "pending" || repository.pinnedCommit
|
|
|| repository.branchName || repository.failureCode) return false;
|
|
const relative = repository.relativePath;
|
|
if (!relative || relative.includes("\\") || path.posix.isAbsolute(relative)
|
|
|| path.posix.normalize(relative) !== relative || relative.split("/").some(part => !part || part === "." || part === "..")) return false;
|
|
const root = await fs.lstat(workspace.cwd).catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code === "ENOENT") return null;
|
|
throw error;
|
|
});
|
|
if (!root?.isDirectory() || root.isSymbolicLink()) return false;
|
|
let parent = workspace.cwd;
|
|
for (const part of relative.split("/").slice(0, -1)) {
|
|
parent = path.join(parent, part);
|
|
const entry = await fs.lstat(parent).catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code === "ENOENT") return null;
|
|
throw error;
|
|
});
|
|
if (!entry) return true;
|
|
if (!entry.isDirectory() || entry.isSymbolicLink()) return false;
|
|
}
|
|
return !await fs.lstat(path.join(workspace.cwd, relative)).catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code === "ENOENT") return null;
|
|
throw error;
|
|
});
|
|
}
|
|
|
|
async function inspect(db: ReadDb, input: { companyId: string; workspaceId: string; operationId?: string; writersStopped?: boolean; onlineSource?: OnlineWorkspaceSource; onlineSourceUnproven?: boolean }, cohort?: WorkspaceMigrationCohort) {
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.workspaceId)));
|
|
if (!workspace) throw new Error("remote_workspace_migration_workspace_missing");
|
|
const authority = readRemoteWorkspaceAuthority(workspace.metadata);
|
|
const marker = object(workspace.metadata?.remoteWorkspaceMigration);
|
|
const allowedOperation = input.operationId ?? (marker.schema === SCHEMA && typeof marker.operationId === "string" ? marker.operationId : undefined);
|
|
const reasons: string[] = [];
|
|
if (authority) return { workspace, reasons, authority, marker, source: authority.source, issueBinding: null, retainedDirectory: null, archivedGitSource: null, legacyRecoverySources: [] as LegacyWorkspaceRecoverySource[], finalizedRecoveryHolds: [] as FinalizedWorkspaceRecoveryHold[], onlineRecoveryHolds: [] as OnlineRecoveryHold[] };
|
|
if (input.writersStopped) reasons.push(...await offlineWorkspaceMigrationBlockers(db));
|
|
if (input.onlineSourceUnproven) reasons.push("online_workspace_source_scope_unproven");
|
|
if (input.onlineSource) {
|
|
const currentScope = await inspectOnlineWorkspaceSource(db, workspace);
|
|
if (canonical(currentScope) !== canonical(input.onlineSource)) reasons.push("online_workspace_source_scope_changed");
|
|
else reasons.push(...await onlineOrdinarySourceBlockers(db as Db, workspace.companyId, currentScope!, workspace, allowedOperation, cohort));
|
|
}
|
|
if (!["local_fs", "git_worktree"].includes(workspace.providerType)) reasons.push("workspace_provider_unsupported");
|
|
if (!["active", "idle", "in_review", "archived"].includes(workspace.status)) reasons.push("workspace_not_active");
|
|
if (!workspace.sourceIssueId) reasons.push("task_binding_required");
|
|
const nestedMigration = workspace.metadata?.nestedCloneMigrationPending;
|
|
if (nestedMigration != null && (typeof nestedMigration !== "object" || Array.isArray(nestedMigration)
|
|
|| Object.keys(nestedMigration).length > 0)) reasons.push("nested_clone_migration_unresolved");
|
|
const cwd = workspace.cwd ?? workspace.providerRef;
|
|
let retainedDirectory = null as ReturnType<typeof readTaskDirectoryRetention>;
|
|
let archivedGitSource: ArchivedGitMigrationSource | null = null;
|
|
if (!cwd || !path.isAbsolute(cwd)) reasons.push("local_source_missing");
|
|
else {
|
|
const entry = await fs.lstat(cwd).catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code === "ENOENT") return null;
|
|
throw error;
|
|
});
|
|
if (!entry && workspace.status === "archived" && mayBeIsolatedTaskDirectoryWorkspace(workspace)) {
|
|
const retained = readTaskDirectoryRetention(workspace);
|
|
if (retained?.removed === true && retained.restorePoint && retained.generation === retained.restorePoint.generation) {
|
|
if (retained.restorePoint.expiredAt) reasons.push("restore_point_expired");
|
|
else {
|
|
try { validateTaskDirectoryRestorePointRef(retained.restorePoint, workspace); retainedDirectory = retained; }
|
|
catch { reasons.push("restore_point_invalid"); }
|
|
}
|
|
}
|
|
}
|
|
if (!entry && !retainedDirectory && (workspace.status === "archived" || input.writersStopped)) {
|
|
try { archivedGitSource = await inspectArchivedGitMigrationSource(db, workspace, { allowActiveRetainedBranch: input.writersStopped, operationId: allowedOperation }); }
|
|
catch (error) { reasons.push(error instanceof Error && error.message === "remote_workspace_migration_restore_point_expired"
|
|
? "restore_point_expired" : "archived_git_source_unavailable"); }
|
|
}
|
|
if (!retainedDirectory && !archivedGitSource && (!entry?.isDirectory() || entry.isSymbolicLink())) reasons.push("local_source_unsafe_or_missing");
|
|
if (entry?.isDirectory() && !entry.isSymbolicLink()) reasons.push(...await workspaceMigrationLayoutBlockers(db, workspace, input.writersStopped, input.onlineSource));
|
|
}
|
|
const [issue] = workspace.sourceIssueId ? await db.select().from(issues).where(and(eq(issues.companyId, workspace.companyId), eq(issues.id, workspace.sourceIssueId))) : [];
|
|
// Historical records still own their saved files after the task selects a
|
|
// different workspace. Importing them must not reopen or rebind that task.
|
|
if (workspace.sourceIssueId && !issue) reasons.push("task_binding_changed");
|
|
const [project] = workspace.projectId ? await db.select({ policy: projects.executionWorkspacePolicy }).from(projects)
|
|
.where(and(eq(projects.companyId, workspace.companyId), eq(projects.id, workspace.projectId))) : [];
|
|
const issuePolicy = object(issue?.executionWorkspaceSettings), projectPolicy = object(project?.policy);
|
|
if (workspace.mode === "shared_workspace" && (issuePolicy.sharedWorkspaceConcurrency ?? (projectPolicy.enabled === true ? projectPolicy.sharedWorkspaceConcurrency : undefined)) === "allow") reasons.push("parallel_shared_writers_unsupported");
|
|
if (issue?.executionRunId || issue?.checkoutRunId) reasons.push("task_writer_bound");
|
|
const [run] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, workspace.companyId),
|
|
input.writersStopped ? offlineMigrationExecutionWhere() : inArray(heartbeatRuns.status, ["queued", "running"]), sql`(${heartbeatRuns.nativeIssueId} = ${workspace.sourceIssueId}::uuid
|
|
or ${heartbeatRuns.issueId} = ${workspace.sourceIssueId}::uuid
|
|
or ${heartbeatRuns.contextSnapshot}->>'executionWorkspaceId' = ${workspace.id}
|
|
or ${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'workspaceId' = ${workspace.id})`,
|
|
)).limit(1);
|
|
if (run) reasons.push("active_execution");
|
|
const referencedRuns = await db.select({ id: heartbeatRuns.id, status: heartbeatRuns.status, runtimeMode: heartbeatRuns.runtimeMode,
|
|
issueId: heartbeatRuns.nativeIssueId, controllerLeaseExpiresAt: heartbeatRuns.controllerLeaseExpiresAt,
|
|
reference: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'`,
|
|
}).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, workspace.companyId),
|
|
sql`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'workspaceId' = ${workspace.id}`,
|
|
sql`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'state' in ('prepared', 'finalized')`,
|
|
));
|
|
const legacyRecoverySources: LegacyWorkspaceRecoverySource[] = [];
|
|
const onlineRecoveryAllowed = Boolean(input.onlineSource) && !input.writersStopped;
|
|
for (const run of referencedRuns.filter(run => object(run.reference).state === "prepared").sort((a, b) => a.id.localeCompare(b.id))) {
|
|
if (onlineRecoveryAllowed) {
|
|
const source = await inspectOnlineLegacyWorkspaceRecoverySource(db, { workspace, runId: run.id });
|
|
if (source) legacyRecoverySources.push(source);
|
|
else reasons.push("remote_copyback_unresolved");
|
|
continue;
|
|
}
|
|
const reference = readNativeWorkspaceSyncReference(run.reference);
|
|
const [owner] = await db.select({ phase: nativeRunFinalizations.phase, leaseOwner: nativeRunFinalizations.leaseOwner,
|
|
leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt, resultId: nativeRunFinalizations.resultId })
|
|
.from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, workspace.companyId), eq(nativeRunFinalizations.runId, run.id)));
|
|
const unsettled = owner && ["observed", "retryable_failure"].includes(owner.phase) && owner.resultId === null;
|
|
const owned = owner?.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt > new Date());
|
|
if (!input.writersStopped || !reference || run.runtimeMode !== "native" || !run.issueId
|
|
|| !await isLegacyWorkspaceTaskRelated(db, { companyId: workspace.companyId, issueId: run.issueId, sourceIssueId: workspace.sourceIssueId })
|
|
|| !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(run.status)
|
|
|| run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > new Date()
|
|
|| !owner || owned || !(["committed", "terminal_failure"].includes(owner.phase) || unsettled)) reasons.push("remote_copyback_unresolved");
|
|
else legacyRecoverySources.push({ runId: run.id, issueId: run.issueId, reference,
|
|
...(unsettled ? { unresolvedFinalization: { phase: owner.phase, leaseOwner: owner.leaseOwner,
|
|
leaseExpiresAt: owner.leaseExpiresAt?.toISOString() ?? null } } : {}) });
|
|
}
|
|
const finalizations = workspace.sourceIssueId ? await db.select({ id: nativeRunFinalizations.runId, phase: nativeRunFinalizations.phase,
|
|
leaseOwner: nativeRunFinalizations.leaseOwner, leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt,
|
|
resultId: nativeRunFinalizations.resultId, run: heartbeatRuns }).from(nativeRunFinalizations)
|
|
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId), eq(heartbeatRuns.id, nativeRunFinalizations.runId))).where(and(
|
|
eq(nativeRunFinalizations.companyId, workspace.companyId), eq(nativeRunFinalizations.issueId, workspace.sourceIssueId),
|
|
sql`${nativeRunFinalizations.phase} not in ('committed', 'terminal_failure')`,
|
|
)) : [];
|
|
for (const row of finalizations) {
|
|
if (legacyRecoverySources.some(source => source.runId === row.id && source.unresolvedFinalization)) continue;
|
|
if (onlineRecoveryAllowed) {
|
|
const source = await inspectOnlineLegacyWorkspaceRecoverySource(db, { workspace, runId: row.id });
|
|
if (source) { legacyRecoverySources.push(source); continue; }
|
|
}
|
|
const stopped = input.writersStopped && row.run.runtimeMode === "native"
|
|
&& ["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(row.run.status)
|
|
&& !(row.run.controllerLeaseExpiresAt && row.run.controllerLeaseExpiresAt > new Date())
|
|
&& !(row.leaseOwner && (!row.leaseExpiresAt || row.leaseExpiresAt > new Date()));
|
|
const reference = readNativeWorkspaceSyncReference(row.run.runnerProfileJson?.nativeWorkspaceSync);
|
|
// The finalizer retries from its operation receipt, not the sync descriptor.
|
|
// Without that receipt it would try exporting the newly migrated workspace,
|
|
// which intentionally has no live location. Keep the local authority until
|
|
// the existing owner records completion; then assessment/status can resume.
|
|
if (stopped && reference?.state === "finalized") {
|
|
const [finalized] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
|
|
eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.heartbeatRunId, row.id),
|
|
eq(workspaceOperations.issueId, row.run.nativeIssueId!), eq(workspaceOperations.phase, "workspace_finalize"),
|
|
eq(workspaceOperations.status, "succeeded"),
|
|
)).limit(1);
|
|
if (finalized) continue;
|
|
}
|
|
if (stopped && reference && reference.workspaceId !== workspace.id) {
|
|
const [other] = await db.select({ providerType: executionWorkspaces.providerType, cwd: executionWorkspaces.cwd,
|
|
sourceIssueId: executionWorkspaces.sourceIssueId, metadata: executionWorkspaces.metadata }).from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, workspace.companyId), eq(executionWorkspaces.id, reference.workspaceId)));
|
|
// The offline batch must migrate every local record before the app can
|
|
// start. Another workspace's coordinator is checked with its own source.
|
|
if (other && ((other.cwd && ["local_fs", "git_worktree"].includes(other.providerType)) || readRemoteWorkspaceAuthority(other.metadata))
|
|
&& await isLegacyWorkspaceTaskRelated(db, { companyId: workspace.companyId, issueId: row.run.nativeIssueId, sourceIssueId: other.sourceIssueId })) continue;
|
|
}
|
|
if (stopped && row.run.runnerProfileJson?.nativeWorkspaceSync == null && row.run.runnerProfileJson?.remoteWorkspace == null
|
|
&& row.resultId === null && ["observed", "retryable_failure"].includes(row.phase)) {
|
|
legacyRecoverySources.push({ runId: row.id, issueId: row.run.nativeIssueId!, reference: null,
|
|
recoveryAllocations: await readLegacyWorkspaceRecoveryAllocations(db, row.run),
|
|
unresolvedFinalization: { phase: row.phase, leaseOwner: row.leaseOwner,
|
|
leaseExpiresAt: row.leaseExpiresAt?.toISOString() ?? null } });
|
|
continue;
|
|
}
|
|
reasons.push("execution_finalization_unresolved");
|
|
}
|
|
legacyRecoverySources.sort((a, b) => a.runId.localeCompare(b.runId));
|
|
const [operation] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
|
|
eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id),
|
|
allowedOperation ? ne(workspaceOperations.id, allowedOperation) : undefined,
|
|
sql`(${workspaceOperations.status} in ('running', 'pending') or ${workspaceOperations.metadata}->>'uncertain' = 'true')`,
|
|
)).limit(1);
|
|
if (operation) reasons.push("workspace_operation_unresolved");
|
|
const [lease] = await db.select({ id: executionWorkspaceRuntimeLeases.id }).from(executionWorkspaceRuntimeLeases).where(and(
|
|
eq(executionWorkspaceRuntimeLeases.companyId, workspace.companyId), eq(executionWorkspaceRuntimeLeases.executionWorkspaceId, workspace.id),
|
|
sql`${executionWorkspaceRuntimeLeases.expiresAt} > clock_timestamp()`,
|
|
)).limit(1);
|
|
if (lease) reasons.push("workspace_runtime_owned");
|
|
// A control lease can expire while the service process remains alive. Require
|
|
// settled service state independently, including inherited shared services.
|
|
const runtime = object(workspace.metadata?.config).workspaceRuntime;
|
|
const inheritedSourceId = workspace.mode === "shared_workspace" && workspace.projectWorkspaceId
|
|
&& (!runtime || typeof runtime !== "object" || Array.isArray(runtime)) ? workspace.projectWorkspaceId : null;
|
|
const [service] = await db.select({ id: workspaceRuntimeServices.id }).from(workspaceRuntimeServices).where(and(
|
|
eq(workspaceRuntimeServices.companyId, workspace.companyId), ne(workspaceRuntimeServices.status, "stopped"),
|
|
or(eq(workspaceRuntimeServices.executionWorkspaceId, workspace.id), inheritedSourceId ? and(
|
|
eq(workspaceRuntimeServices.projectWorkspaceId, inheritedSourceId), eq(workspaceRuntimeServices.scopeType, "project_workspace"),
|
|
) : undefined),
|
|
)).limit(1);
|
|
if (service) reasons.push("workspace_runtime_service_unsettled");
|
|
const recoveryIssueIds = [...new Set([workspace.sourceIssueId, ...referencedRuns.map(run => run.issueId),
|
|
...legacyRecoverySources.map(source => source.issueId)].filter((id): id is string => Boolean(id)))];
|
|
const recoveries = recoveryIssueIds.length ? await db.select().from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, workspace.companyId), inArray(issueRecoveryActions.sourceIssueId, recoveryIssueIds),
|
|
inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
|
)) : [];
|
|
const finalizedRecoveryHolds: FinalizedWorkspaceRecoveryHold[] = [];
|
|
const onlineRecoveryHolds: OnlineRecoveryHold[] = [];
|
|
for (const recovery of recoveries.sort((a, b) => a.id.localeCompare(b.id))) {
|
|
if (onlineRecoveryAllowed) {
|
|
const held = await inspectOnlineLegacyWorkspaceRecoveryHold(db, { workspace, recovery });
|
|
if (!held) { reasons.push("workspace_recovery_unresolved"); continue; }
|
|
onlineRecoveryHolds.push({ actionId: recovery.id, ...held });
|
|
if (held.preparedSource && !legacyRecoverySources.some(source => source.runId === held.preparedSource!.runId))
|
|
legacyRecoverySources.push(held.preparedSource);
|
|
if (held.finalizedHold) {
|
|
finalizedRecoveryHolds.push(held.finalizedHold);
|
|
if (held.finalizedHold.source && !legacyRecoverySources.some(source => source.runId === held.finalizedHold!.source!.runId))
|
|
legacyRecoverySources.push(held.finalizedHold.source);
|
|
}
|
|
continue;
|
|
}
|
|
if (!input.writersStopped || object(recovery.evidence?.systemRecovery).state === "running") {
|
|
reasons.push("workspace_recovery_unresolved"); continue;
|
|
}
|
|
if (legacyRecoverySources.some(source => source.runId === recovery.evidence?.runId && source.reference?.state !== "finalized")) continue;
|
|
const held = await inspectFinalizedWorkspaceRecoveryHold(db, { companyId: workspace.companyId,
|
|
workspaceId: workspace.id, recovery });
|
|
if (held) {
|
|
finalizedRecoveryHolds.push(held);
|
|
if (held.source && !legacyRecoverySources.some(source => source.runId === held.source!.runId)) legacyRecoverySources.push(held.source);
|
|
continue;
|
|
}
|
|
// These warnings govern task disposition, not file authority. Keep them
|
|
// unchanged while the offline owner saves the workspace. Unknown recovery
|
|
// kinds and any declared file source still require explicit preservation.
|
|
if (!["stranded_assigned_issue", "configuration_validation", "missing_disposition", "deliberate_wait_without_target"].includes(recovery.kind)
|
|
|| workspaceExportCauses.includes(recovery.cause)
|
|
|| ["workspaceRestoreRecovery", "workspaceRecovery", "historicalWorkspaceSource", "systemRecovery", "nativeFailureBlock"]
|
|
.some(key => recovery.evidence?.[key] != null)) reasons.push("workspace_recovery_unresolved");
|
|
}
|
|
legacyRecoverySources.sort((a, b) => a.runId.localeCompare(b.runId));
|
|
const repositories = await db.select().from(executionWorkspaceRepositories).where(and(eq(executionWorkspaceRepositories.companyId, workspace.companyId),
|
|
eq(executionWorkspaceRepositories.executionWorkspaceId, workspace.id)));
|
|
for (const repository of repositories.filter(repository => repository.state !== "ready")) {
|
|
if (!input.writersStopped || !await isUnmaterializedRepositoryRequest(workspace, repository)) reasons.push("repository_preparation_unresolved");
|
|
}
|
|
const source: RemoteWorkspaceSource = workspace.repoUrl ? { kind: "git", repoUrl: workspace.repoUrl,
|
|
ref: archivedGitSource?.commit ?? workspace.baseRef, branchName: workspace.branchName } : { kind: "empty" };
|
|
const materialized = repositories.filter(repository => repository.state === "ready");
|
|
if (materialized.length) source.repositories = materialized.map(repository => ({ id: repository.id, relativePath: repository.relativePath,
|
|
repoUrl: repository.repoUrl, ref: repository.requestedRef, ...(repository.pinnedCommit ? { pinnedCommit: repository.pinnedCommit } : {}) }));
|
|
return { workspace, reasons: [...new Set(reasons)], authority, marker, source, issueBinding: issue?.executionWorkspaceId ?? null, retainedDirectory, archivedGitSource, legacyRecoverySources, finalizedRecoveryHolds, onlineRecoveryHolds };
|
|
}
|
|
|
|
export async function inventoryRemoteWorkspaceMigration(db: Db, input: { companyId: string; workspaceId: string; writersStopped?: boolean }) {
|
|
const [workspace] = !input.writersStopped ? await db.select().from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.workspaceId))) : [];
|
|
const onlineSource = workspace && !readRemoteWorkspaceAuthority(workspace.metadata)
|
|
? await inspectOnlineWorkspaceSource(db, workspace) ?? undefined : undefined;
|
|
const result = await inspect(db, { ...input, onlineSource, onlineSourceUnproven: Boolean(workspace && !onlineSource
|
|
&& !readRemoteWorkspaceAuthority(workspace.metadata) && await requiresOnlineWorkspaceSource(workspace)) });
|
|
return { companyId: input.companyId, workspaceId: input.workspaceId, eligible: result.reasons.length === 0,
|
|
alreadyMigrated: Boolean(result.authority), reasons: result.reasons, localPath: result.workspace.cwd ?? result.workspace.providerRef,
|
|
source: result.source, operationId: typeof result.marker.operationId === "string" ? result.marker.operationId : null,
|
|
snapshotId: result.authority?.snapshotId ?? null };
|
|
}
|
|
|
|
type MigrationInput = {
|
|
db: Db; storage: StorageService; companyId: string; workspaceId: string; apply?: boolean; source?: RemoteWorkspaceSource;
|
|
writersStopped?: boolean; signal?: AbortSignal;
|
|
};
|
|
type MigrationTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
|
|
|
async function resolveMigrationSource(input: MigrationInput) {
|
|
const [sourceWorkspace] = !input.writersStopped ? await input.db.select().from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.workspaceId))) : [];
|
|
const onlineSource = sourceWorkspace && !readRemoteWorkspaceAuthority(sourceWorkspace.metadata)
|
|
? await inspectOnlineWorkspaceSource(input.db, sourceWorkspace) ?? undefined : undefined;
|
|
if (sourceWorkspace && !onlineSource && !readRemoteWorkspaceAuthority(sourceWorkspace.metadata)
|
|
&& await requiresOnlineWorkspaceSource(sourceWorkspace)) {
|
|
throw new Error("remote_workspace_migration_blocked:online_workspace_source_scope_unproven");
|
|
}
|
|
return onlineSource;
|
|
}
|
|
|
|
async function reserveMigration(input: MigrationInput, onlineSource?: OnlineWorkspaceSource, cohort?: WorkspaceMigrationCohort,
|
|
expectedSourceFingerprint?: string) {
|
|
return input.db.transaction(async tx => {
|
|
const [selected] = await tx.select({ sourceIssueId: executionWorkspaces.sourceIssueId }).from(executionWorkspaces)
|
|
.where(and(eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.workspaceId)));
|
|
if (!selected) throw new Error("remote_workspace_migration_workspace_missing");
|
|
await lockMigrationParents(tx, input.companyId, selected.sourceIssueId);
|
|
const [workspace] = await tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.workspaceId))).for("update");
|
|
if (!workspace) throw new Error("remote_workspace_migration_workspace_missing");
|
|
if (workspace.sourceIssueId !== selected.sourceIssueId) throw new Error("remote_workspace_migration_source_changed");
|
|
const current = await inspect(tx as unknown as Db, { ...input, onlineSource }, cohort);
|
|
if (current.authority) return { workspace, authority: current.authority, operation: null, source: current.source, sourceFingerprint: "", issueBinding: current.issueBinding, retainedDirectory: null, archivedGitSource: null, legacyRecoverySources: [] as LegacyWorkspaceRecoverySource[], finalizedRecoveryHolds: [] as FinalizedWorkspaceRecoveryHold[], onlineRecoveryHolds: [] as OnlineRecoveryHold[] };
|
|
if (current.reasons.length) throw new Error(`remote_workspace_migration_blocked:${current.reasons.join(",")}`);
|
|
const nativeSessionContinuity = await inspectNativeWorkspaceMigration(tx as unknown as Db, workspace);
|
|
const source = remoteWorkspaceSourceSchema.parse(input.source ?? current.source);
|
|
const sourceFingerprint = fingerprint(workspace);
|
|
if (cohort && sourceFingerprint !== expectedSourceFingerprint) throw new Error("workspace_cohort_source_changed");
|
|
const marker = current.marker;
|
|
if (marker.schema === SCHEMA && typeof marker.operationId === "string") {
|
|
if (marker.sourceFingerprint !== sourceFingerprint || canonical(marker.source) !== canonical(source)) throw new Error("remote_workspace_migration_source_changed");
|
|
const [operation] = await tx.select().from(workspaceOperations).where(and(eq(workspaceOperations.companyId, workspace.companyId),
|
|
eq(workspaceOperations.executionWorkspaceId, workspace.id), eq(workspaceOperations.id, marker.operationId), eq(workspaceOperations.phase, "remote_workspace_migration")));
|
|
if (!operation) throw new Error("remote_workspace_migration_operation_missing");
|
|
if (canonical(operation.metadata?.cohortMigration ?? null) !== canonical(cohort ?? null)) throw new Error("remote_workspace_cohort_resume_required");
|
|
if (canonical(operation.metadata?.nativeSessionContinuity ?? null) !== canonical(nativeSessionContinuity)) throw new Error("native_workspace_migration_source_changed");
|
|
// Older operations only admitted the task's current workspace. Preserve
|
|
// that binding when resuming an import journaled before this field existed.
|
|
const issueBinding = Object.hasOwn(operation.metadata ?? {}, "sourceIssueBinding")
|
|
? operation.metadata!.sourceIssueBinding : workspace.id;
|
|
if (issueBinding !== current.issueBinding) throw new Error("remote_workspace_migration_source_changed");
|
|
if (Boolean(operation.metadata?.retainedDirectory) !== Boolean(current.retainedDirectory)) throw new Error("remote_workspace_migration_source_changed");
|
|
if (canonical(operation.metadata?.archivedGitSource ?? null) !== canonical(gitSourceProof(current.archivedGitSource))) throw new Error("remote_workspace_migration_source_changed");
|
|
if (canonical(operation.metadata?.finalizedRecoveryHolds ?? []) !== canonical(current.finalizedRecoveryHolds)) throw new Error("remote_workspace_migration_source_changed");
|
|
if (canonical(operation.metadata?.legacyRecoverySources ?? []) !== canonical(current.legacyRecoverySources)) throw new Error("remote_workspace_migration_source_changed");
|
|
if (canonical(operation.metadata?.onlineRecoveryHolds ?? []) !== canonical(current.onlineRecoveryHolds)) throw new Error("remote_workspace_migration_source_changed");
|
|
return { workspace, authority: null, operation, source, sourceFingerprint, issueBinding, retainedDirectory: current.retainedDirectory, archivedGitSource: current.archivedGitSource, legacyRecoverySources: current.legacyRecoverySources, finalizedRecoveryHolds: current.finalizedRecoveryHolds, onlineRecoveryHolds: current.onlineRecoveryHolds };
|
|
}
|
|
const [operation] = await tx.insert(workspaceOperations).values({ ...(cohort ? { id: cohort.members.find(member => member.workspaceId === input.workspaceId)!.operationId } : {}), companyId: workspace.companyId, executionWorkspaceId: workspace.id,
|
|
issueId: workspace.sourceIssueId, cwd: workspace.cwd ?? workspace.providerRef, phase: "remote_workspace_migration", status: "running",
|
|
metadata: { schema: SCHEMA, sourceFingerprint, source, sourceIssueBinding: current.issueBinding,
|
|
...(cohort ? { cohortMigration: cohort } : {}),
|
|
...(nativeSessionContinuity ? { nativeSessionContinuity } : {}),
|
|
nestedWorkspaceScope: { version: 1, state: "pending" },
|
|
...(current.finalizedRecoveryHolds.length ? { finalizedRecoveryHolds: current.finalizedRecoveryHolds } : {}),
|
|
...(current.legacyRecoverySources.length ? { legacyRecoverySources: current.legacyRecoverySources } : {}),
|
|
...(current.onlineRecoveryHolds.length ? { onlineRecoveryHolds: current.onlineRecoveryHolds } : {}),
|
|
...(current.archivedGitSource ? { archivedGitSource: gitSourceProof(current.archivedGitSource) } : {}),
|
|
...(current.retainedDirectory ? { retainedDirectory: { restorePointId: current.retainedDirectory.restorePoint.id,
|
|
generation: current.retainedDirectory.generation, digest: current.retainedDirectory.restorePoint.digest } } : {}) } }).returning();
|
|
await notifyRemoteWorkspaceSnapshots(tx, `workspace:${workspace.id}`);
|
|
await tx.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspaceMigration: {
|
|
schema: SCHEMA, operationId: operation!.id, state: "running", sourceFingerprint, source,
|
|
} }, updatedAt: new Date() }).where(eq(executionWorkspaces.id, workspace.id));
|
|
return { workspace, authority: null, operation: operation!, source, sourceFingerprint, issueBinding: current.issueBinding, retainedDirectory: current.retainedDirectory, archivedGitSource: current.archivedGitSource, legacyRecoverySources: current.legacyRecoverySources, finalizedRecoveryHolds: current.finalizedRecoveryHolds, onlineRecoveryHolds: current.onlineRecoveryHolds };
|
|
});
|
|
}
|
|
type ReservedMigration = Awaited<ReturnType<typeof reserveMigration>>;
|
|
type MigrationPlan = { input: MigrationInput; onlineSource?: OnlineWorkspaceSource; reserved: ReservedMigration };
|
|
|
|
async function lockMigrationRecovery(lock: MigrationTransaction, plans: MigrationPlan[], cohort?: WorkspaceMigrationCohort) {
|
|
const online = plans.filter(plan => plan.onlineSource);
|
|
if (!online.length) return;
|
|
if (!await lockOnlineLegacyWorkspaceRecoverySources(lock as unknown as Db, { companyId: online[0].input.companyId,
|
|
sources: online.flatMap(({ reserved, onlineSource }) => [...reserved.legacyRecoverySources, ...onlineSource!.companions.flatMap(proof => proof.sources)]),
|
|
holds: online.flatMap(({ reserved, onlineSource }) => [...reserved.onlineRecoveryHolds, ...onlineSource!.companions.flatMap(proof => proof.holds)]) }))
|
|
throw new Error("remote_workspace_migration_blocked:remote_copyback_owned");
|
|
for (const { input, onlineSource, reserved } of online) {
|
|
const current = await inspect(input.db, { ...input, onlineSource, operationId: reserved.operation!.id }, cohort);
|
|
if (current.reasons.length || canonical(current.legacyRecoverySources) !== canonical(reserved.legacyRecoverySources)
|
|
|| canonical(current.onlineRecoveryHolds) !== canonical(reserved.onlineRecoveryHolds))
|
|
throw new Error("remote_workspace_migration_recovery_source_changed");
|
|
}
|
|
}
|
|
|
|
function createMigrationChildVerifier(input: MigrationInput, onlineSource: OnlineWorkspaceSource | undefined, sourceLock: MigrationTransaction) {
|
|
if (onlineSource?.kind !== "git") return undefined;
|
|
const verifier = createNestedWorkspaceImportVerifier({ db: input.db, storage: input.storage, companyId: input.companyId,
|
|
sourceLock, projectIds: onlineSource.projectIds, issueIds: onlineSource.issueIds, signal: input.signal, persistVerification: true });
|
|
let pending = Promise.resolve();
|
|
return {
|
|
prepare(options: Parameters<typeof prepareNestedWorkspaceImports>[0]) {
|
|
const work = pending.then(() => verifier.prepare(options));
|
|
pending = work.then(() => undefined, () => undefined);
|
|
return work;
|
|
},
|
|
async close() { await pending; await verifier.close(); },
|
|
};
|
|
}
|
|
|
|
async function captureMigration(input: MigrationInput, reserved: ReservedMigration, onlineSource?: OnlineWorkspaceSource,
|
|
childVerifier?: ReturnType<typeof createMigrationChildVerifier>) {
|
|
let operation = reserved.operation!;
|
|
const snapshot = await beginWorkspaceSnapshot(input.db, { companyId: input.companyId, workspaceId: input.workspaceId,
|
|
operationId: operation.id, generation: 1, epoch: 1 });
|
|
const scratch = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "pc-workspace-migration-")));
|
|
try {
|
|
input.signal?.throwIfAborted();
|
|
const retained = reserved.retainedDirectory?.restorePoint;
|
|
const archivePath = path.join(scratch, "workspace.pcws");
|
|
const recovered = reserved.archivedGitSource ? await recoverWorkspaceSnapshotArchive({
|
|
db: input.db, storage: input.storage, companyId: input.companyId, workspaceId: input.workspaceId,
|
|
snapshotId: snapshot.id, targetPath: archivePath, signal: input.signal,
|
|
}) : null;
|
|
const sourceRoot = recovered ? null : retained ? await materializeTaskDirectorySnapshot({
|
|
payload: await loadTaskDirectoryRestorePoint({ storage: input.storage, workspace: reserved.workspace, ref: retained, recoverMissingEmpty: true, signal: input.signal }),
|
|
target: path.join(scratch, "retained-source"), expectedDigest: retained.digest,
|
|
}) : reserved.archivedGitSource ? await materializeArchivedGitMigrationSource({ db: input.db, storage: input.storage,
|
|
workspace: reserved.workspace, source: reserved.archivedGitSource, scratch }) : reserved.workspace.cwd ?? reserved.workspace.providerRef!;
|
|
const scope = object(operation.metadata?.nestedWorkspaceScope);
|
|
if (operation.metadata?.nestedWorkspaceScope !== undefined && (scope.version !== 1 || !["pending", "fixed"].includes(String(scope.state))))
|
|
throw new Error("nested_workspace_scope_invalid");
|
|
// Journals predating scoped captures included every child. Preserve that
|
|
// archive identity even if a later batch has already imported the leaves.
|
|
const recordedChildren = scope.state === "fixed" ? validateImportedWorkspaceChildren(scope.children)
|
|
: scope.state === "pending" ? undefined : [];
|
|
const nested = (input.writersStopped || onlineSource?.kind === "git") && !retained && !reserved.archivedGitSource
|
|
? await (childVerifier?.prepare ?? prepareNestedWorkspaceImports)({ ...input, workspace: reserved.workspace, scratch, recordedChildren, gitSourceRoots: onlineSource?.gitSourceRoots }) : [];
|
|
if (scope.state === "pending") {
|
|
const [saved] = await input.db.update(workspaceOperations).set({ metadata: { ...operation.metadata,
|
|
nestedWorkspaceScope: { version: 1, state: "fixed", children: nested } }, updatedAt: new Date() })
|
|
.where(and(eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.id, operation.id),
|
|
eq(workspaceOperations.status, "running"))).returning();
|
|
if (!saved) throw new Error("remote_workspace_migration_operation_changed");
|
|
operation = saved;
|
|
}
|
|
const exclude = nested.map(child => child.relativePath);
|
|
const captured = recovered ?? await captureRemoteWorkspaceDirectory({ cwd: sourceRoot!, archivePath, exclude, requireNoGit: onlineSource?.kind === "ordinary", gitSourceRoots: onlineSource?.gitSourceRoots, signal: input.signal });
|
|
const restoredArchive = path.join(scratch, "verified.pcws"), restored = path.join(scratch, "restored");
|
|
const verifiedArchivePath = process.platform === "win32" ? undefined : restoredArchive;
|
|
const ref = await uploadWorkspaceSnapshot({ db: input.db, storage: input.storage, snapshotId: snapshot.id,
|
|
companyId: input.companyId, workspaceId: input.workspaceId, archivePath, manifestDigest: captured.manifestDigest, signal: input.signal,
|
|
verifiedArchivePath,
|
|
// Private Git reconstruction can change administrative bytes between
|
|
// attempts. Its journaled logical recipe is checked at reservation and
|
|
// again under source locks before authority publication below.
|
|
...(reserved.archivedGitSource ? { unpublishedRecapture: { sourceOperationId: operation.id } } : {}),
|
|
});
|
|
if (!verifiedArchivePath) await loadWorkspaceSnapshotArchive({ storage: input.storage, reference: ref,
|
|
targetPath: restoredArchive, signal: input.signal });
|
|
await restoreRemoteWorkspaceArchive({ archivePath: restoredArchive, targetDir: restored, expectedManifestDigest: ref.manifestDigest, signal: input.signal });
|
|
if (!recovered) {
|
|
const currentArchive = path.join(scratch, "current.pcws");
|
|
const current = await captureRemoteWorkspaceDirectory({ cwd: sourceRoot!, archivePath: currentArchive, exclude, requireNoGit: onlineSource?.kind === "ordinary", gitSourceRoots: onlineSource?.gitSourceRoots, signal: input.signal });
|
|
if (current.manifestDigest !== captured.manifestDigest) throw new Error("remote_workspace_migration_source_changed");
|
|
}
|
|
if (nested.length) {
|
|
const verifyScratch = path.join(scratch, "verify-nested"); await fs.mkdir(verifyScratch);
|
|
const verified = await (childVerifier?.prepare ?? prepareNestedWorkspaceImports)({ ...input, workspace: reserved.workspace,
|
|
scratch: verifyScratch, recordedChildren: nested, gitSourceRoots: onlineSource?.gitSourceRoots });
|
|
if (canonical(verified) !== canonical(nested)) throw new Error("nested_workspace_source_changed");
|
|
}
|
|
input.signal?.throwIfAborted();
|
|
return { operation, ref, nested, scratch };
|
|
} catch (error) { await removeTaskDirectoryTree(scratch); throw error; }
|
|
}
|
|
type CapturedMigration = Awaited<ReturnType<typeof captureMigration>>;
|
|
|
|
async function validateMigrationPublication(tx: MigrationTransaction, input: MigrationInput, reserved: ReservedMigration,
|
|
captured: CapturedMigration, onlineSource?: OnlineWorkspaceSource, cohort?: WorkspaceMigrationCohort) {
|
|
const { operation, ref, nested } = captured;
|
|
if (cohort) {
|
|
const [journal] = await tx.select().from(workspaceOperations).where(and(eq(workspaceOperations.companyId, input.companyId),
|
|
eq(workspaceOperations.id, operation.id), eq(workspaceOperations.executionWorkspaceId, input.workspaceId))).for("update", { noWait: true });
|
|
if (!journal || journal.status !== "running" || journal.finishedAt !== null || journal.phase !== "remote_workspace_migration"
|
|
|| journal.issueId !== operation.issueId || journal.cwd !== operation.cwd
|
|
|| canonical(journal.metadata) !== canonical(operation.metadata)) throw new Error("workspace_cohort_journal_changed");
|
|
}
|
|
await lockMigrationParents(tx, input.companyId, reserved.workspace.sourceIssueId, onlineSource ? [
|
|
...reserved.legacyRecoverySources.map(source => source.issueId),
|
|
...onlineSource.companions.flatMap(proof => proof.sources.map(source => source.issueId)),
|
|
] : []);
|
|
for (const actionId of [...new Set([...reserved.finalizedRecoveryHolds, ...reserved.onlineRecoveryHolds].map(held => held.actionId))].sort()) {
|
|
await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, actionId))).for("update", { noWait: true });
|
|
}
|
|
const protectedRunIds = [...new Set([...reserved.legacyRecoverySources.map(source => source.runId),
|
|
...reserved.finalizedRecoveryHolds.map(held => held.runId)])].sort();
|
|
for (const runId of protectedRunIds) {
|
|
await tx.select({ id: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
|
|
.where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, runId))).for("update", { noWait: true });
|
|
await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, runId))).for("update", { noWait: true });
|
|
}
|
|
const [workspace] = await tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.workspaceId))).for("update");
|
|
if (!workspace) throw new Error("remote_workspace_migration_workspace_missing");
|
|
const existing = readRemoteWorkspaceAuthority(workspace.metadata);
|
|
if (existing?.snapshotId === ref.id) return null;
|
|
if (existing || fingerprint(workspace) !== reserved.sourceFingerprint
|
|
|| object(workspace.metadata?.remoteWorkspaceMigration).operationId !== operation.id) throw new Error("remote_workspace_migration_publication_changed");
|
|
for (const source of reserved.legacyRecoverySources) {
|
|
for (const allocation of source.recoveryAllocations ?? []) await tx.select({ id: environmentLeases.id })
|
|
.from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
|
|
eq(environmentLeases.id, allocation.leaseId))).for("update", { noWait: true });
|
|
}
|
|
const validation = await inspect(tx as unknown as Db, { ...input, operationId: operation.id, onlineSource }, cohort);
|
|
if (!input.source && canonical(validation.source) !== canonical(reserved.source)) throw new Error("remote_workspace_migration_source_changed");
|
|
if (validation.issueBinding !== reserved.issueBinding) throw new Error("remote_workspace_migration_source_changed");
|
|
if (Boolean(validation.retainedDirectory) !== Boolean(reserved.retainedDirectory)) throw new Error("remote_workspace_migration_source_changed");
|
|
if (canonical(gitSourceProof(validation.archivedGitSource)) !== canonical(gitSourceProof(reserved.archivedGitSource))) throw new Error("remote_workspace_migration_source_changed");
|
|
if (validation.reasons.length) throw new Error(`remote_workspace_migration_blocked:${validation.reasons.join(",")}`);
|
|
if (reserved.archivedGitSource) await lockArchivedGitMigrationSources(tx as unknown as Db, workspace, reserved.archivedGitSource);
|
|
const [uploaded] = await tx.select().from(executionWorkspaceSnapshots).where(and(eq(executionWorkspaceSnapshots.companyId, workspace.companyId),
|
|
eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id), eq(executionWorkspaceSnapshots.id, ref.id))).for("update");
|
|
if (!uploaded || uploaded.status !== "uploaded" || uploaded.sourceOperationId !== operation.id) throw new Error("remote_workspace_migration_snapshot_changed");
|
|
for (const child of [...nested].sort((a, b) => a.snapshotId.localeCompare(b.snapshotId))) {
|
|
const [saved] = await tx.select().from(executionWorkspaceSnapshots).where(and(
|
|
eq(executionWorkspaceSnapshots.companyId, workspace.companyId), eq(executionWorkspaceSnapshots.id, child.snapshotId),
|
|
eq(executionWorkspaceSnapshots.executionWorkspaceId, child.workspaceId),
|
|
)).for("update");
|
|
if (!saved || !["committed", "superseded"].includes(saved.status) || saved.manifestDigest !== child.manifestDigest
|
|
|| saved.sourceOperationId !== child.operationId) throw new Error("nested_workspace_snapshot_changed");
|
|
}
|
|
if (canonical(validation.finalizedRecoveryHolds) !== canonical(reserved.finalizedRecoveryHolds)) throw new Error("remote_workspace_migration_source_changed");
|
|
if (canonical(validation.onlineRecoveryHolds) !== canonical(reserved.onlineRecoveryHolds)) throw new Error("remote_workspace_migration_recovery_source_changed");
|
|
if (canonical(validation.legacyRecoverySources) !== canonical(reserved.legacyRecoverySources)) throw new Error("remote_workspace_migration_source_changed");
|
|
const continuity = await inspectNativeWorkspaceMigration(tx as unknown as Db, workspace);
|
|
if (canonical(continuity) !== canonical(operation.metadata?.nativeSessionContinuity ?? null)) throw new Error("native_workspace_migration_source_changed");
|
|
return { workspace, uploaded, continuity };
|
|
}
|
|
type ValidatedPublication = NonNullable<Awaited<ReturnType<typeof validateMigrationPublication>>>;
|
|
|
|
async function publishMigration(tx: MigrationTransaction, reserved: ReservedMigration, captured: CapturedMigration, validated: ValidatedPublication) {
|
|
const { operation, ref, nested } = captured, { workspace, uploaded, continuity } = validated;
|
|
const authority: RemoteWorkspaceAuthority = { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: ref.id,
|
|
source: reserved.source, location: null, writerRunId: null, dirty: false };
|
|
const metadata: Record<string, unknown> = { ...workspace.metadata, remoteWorkspace: authority }; delete metadata.remoteWorkspaceMigration;
|
|
if (continuity) metadata.nativeWorkspaceMigration = nativeWorkspaceMigrationMarker(operation.id);
|
|
if (reserved.legacyRecoverySources.length) {
|
|
metadata.legacyWorkspaceRecoveryMigration = { schema: LEGACY_WORKSPACE_RECOVERY_MIGRATION, operationId: operation.id };
|
|
for (const source of reserved.legacyRecoverySources) {
|
|
const markerKey = source.reference?.state === "finalized"
|
|
? "finalizedWorkspaceRecoveryMigration" : "legacyWorkspaceRecoveryMigration";
|
|
// The offline deployment fence has stopped the old controller. The
|
|
// locked validation above must still match its journaled expired
|
|
// lease exactly. Release only that coordinator lease so explicit
|
|
// file recovery can proceed; its phase/checkpoint and the provider's
|
|
// independent stop obligation remain unchanged.
|
|
if (source.unresolvedFinalization?.leaseOwner) await tx.update(nativeRunFinalizations).set({
|
|
leaseOwner: null, leaseExpiresAt: null,
|
|
recoveryHistory: sql`${nativeRunFinalizations.recoveryHistory} || ${JSON.stringify([{
|
|
event: "legacy_workspace_migration_owner_released", operationId: operation.id,
|
|
workspaceId: workspace.id, previousOwner: source.unresolvedFinalization,
|
|
}])}::jsonb`, updatedAt: new Date(),
|
|
}).where(and(eq(nativeRunFinalizations.companyId, workspace.companyId),
|
|
eq(nativeRunFinalizations.runId, source.runId), eq(nativeRunFinalizations.issueId, source.issueId)));
|
|
await tx.update(heartbeatRuns).set({
|
|
runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) || ${JSON.stringify({
|
|
[markerKey]: { schema: LEGACY_WORKSPACE_RECOVERY_MIGRATION,
|
|
operationId: operation.id, workspaceId: workspace.id },
|
|
})}::jsonb`, updatedAt: new Date(),
|
|
}).where(and(eq(heartbeatRuns.companyId, workspace.companyId), eq(heartbeatRuns.id, source.runId)));
|
|
}
|
|
const prior = Array.isArray(metadata.remoteWorkspaceRecoverySourceRunIds) ? metadata.remoteWorkspaceRecoverySourceRunIds : [];
|
|
metadata.remoteWorkspaceRecoverySourceRunIds = [...new Set([...prior, ...reserved.legacyRecoverySources.map(source => source.runId)])];
|
|
}
|
|
await notifyRemoteWorkspaceSnapshots(tx, `workspace:${workspace.id}`);
|
|
await tx.update(executionWorkspaceSnapshots).set({ status: "committed", deleteAfter: null,
|
|
metadata: { ...uploaded.metadata, ...(nested.length ? { importedWorkspaceChildren: nested } : {}) },
|
|
updatedAt: new Date() }).where(eq(executionWorkspaceSnapshots.id, ref.id));
|
|
await tx.update(executionWorkspaces).set({ providerType: "cloud_sandbox", providerRef: null, cwd: `/workspaces/${workspace.id}`, metadata,
|
|
updatedAt: new Date() }).where(eq(executionWorkspaces.id, workspace.id));
|
|
await tx.update(workspaceOperations).set({ status: "succeeded", finishedAt: new Date(), updatedAt: new Date(),
|
|
metadata: { ...operation.metadata, snapshotId: ref.id, manifestDigest: ref.manifestDigest } }).where(eq(workspaceOperations.id, operation.id));
|
|
return (await persistActivity(tx as unknown as Db, { companyId: workspace.companyId, actorType: "system", actorId: "workspace-migration",
|
|
action: "workspace.remote_storage_migrated", entityType: "execution_workspace", entityId: workspace.id,
|
|
details: { snapshotId: ref.id, operationId: operation.id, manifestDigest: ref.manifestDigest } })).publication;
|
|
}
|
|
|
|
function publicationError(error: unknown): never {
|
|
const failure = object(error), cause = object(failure.cause);
|
|
if ((cause.code ?? failure.code) !== "55P03") throw error;
|
|
throw new Error("remote_workspace_migration_blocked:execution_rows_busy", { cause: error });
|
|
}
|
|
|
|
/** Administrative import. The CLI enforces external deployment prerequisites. */
|
|
export async function migrateRemoteWorkspace(input: MigrationInput) {
|
|
if (!input.apply) return { ...await inventoryRemoteWorkspaceMigration(input.db, input), applied: false };
|
|
const onlineSource = await resolveMigrationSource(input);
|
|
let committedPublication: Parameters<typeof publishActivity>[0] | undefined;
|
|
const migrate = async (lock: MigrationTransaction) => {
|
|
const reserved = await reserveMigration(input, onlineSource);
|
|
if (reserved.authority) return { ...await inventoryRemoteWorkspaceMigration(input.db, input), applied: true, replayed: true };
|
|
await lockMigrationRecovery(lock, [{ input, onlineSource, reserved }]);
|
|
const childVerifier = createMigrationChildVerifier(input, onlineSource, lock);
|
|
let scratch: string | undefined;
|
|
try {
|
|
const captured = await captureMigration(input, reserved, onlineSource, childVerifier);
|
|
scratch = captured.scratch;
|
|
await lock.execute(sql`select 1`);
|
|
const publicationDb = onlineSource ? lock as unknown as Db : input.db;
|
|
const publication = await publicationDb.transaction(async tx => {
|
|
const validated = await validateMigrationPublication(tx, input, reserved, captured, onlineSource);
|
|
return validated ? publishMigration(tx, reserved, captured, validated) : null;
|
|
}).catch(publicationError);
|
|
if (publication) { if (onlineSource) committedPublication = publication; else publishActivity(publication); }
|
|
return { ...await inventoryRemoteWorkspaceMigration(publicationDb, input), applied: true, replayed: false };
|
|
} finally { await childVerifier?.close(); if (scratch) await removeTaskDirectoryTree(scratch); }
|
|
};
|
|
if (onlineSource) {
|
|
const result = await withWorkspaceSourceLocks(input.db, {
|
|
companyId: input.companyId, issueIds: onlineSource.issueIds,
|
|
projectIds: onlineSource.projectIds,
|
|
}, async lock => {
|
|
for (const workspaceId of onlineSource.workspaceIds) {
|
|
if (!await tryAcquireExecutionWorkspaceLifecycleLock(lock, workspaceId))
|
|
throw new IsolatedRepositorySourceBusy("workspace_source_busy");
|
|
}
|
|
return migrate(lock);
|
|
}, true);
|
|
if (committedPublication) publishActivity(committedPublication);
|
|
return result;
|
|
}
|
|
return withExecutionWorkspaceLifecycleLock(input.db, input.workspaceId, migrate);
|
|
}
|
|
|
|
const cohortSchema = z.object({
|
|
schema: z.literal("paperclip.remote-workspace-cohort.v1"), id: z.string().uuid(), companyId: z.string().uuid(),
|
|
sourceProof: z.string().regex(/^[a-f0-9]{64}$/),
|
|
members: z.array(z.object({ workspaceId: z.string().uuid(), operationId: z.string().uuid() }).strict()).min(1).max(8),
|
|
}).strict();
|
|
|
|
export async function migrateRemoteWorkspaceCohort(input: {
|
|
db: Db; storage: StorageService; companyId: string; workspaceIds: readonly string[];
|
|
apply?: boolean; concurrency?: number; signal: AbortSignal;
|
|
}) {
|
|
const companyId = z.string().uuid().parse(input.companyId).toLowerCase();
|
|
const workspaceIds = z.array(z.string().uuid()).min(1).max(8).parse(input.workspaceIds).map(id => id.toLowerCase()).sort();
|
|
const concurrency = z.number().int().min(1).max(8).parse(input.concurrency ?? 2);
|
|
if (new Set(workspaceIds).size !== workspaceIds.length) throw new Error("workspace_cohort_duplicate_member");
|
|
input.signal.throwIfAborted();
|
|
const initial = await input.db.select().from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, companyId), inArray(executionWorkspaces.id, workspaceIds))).orderBy(executionWorkspaces.id);
|
|
if (initial.length !== workspaceIds.length) throw new Error("workspace_cohort_member_missing");
|
|
const imported = initial.map(workspace => Boolean(readRemoteWorkspaceAuthority(workspace.metadata)));
|
|
if (imported.every(Boolean)) {
|
|
const snapshotIds = initial.flatMap(workspace => readRemoteWorkspaceAuthority(workspace.metadata)?.snapshotId ?? []);
|
|
if (snapshotIds.length !== workspaceIds.length) throw new Error("workspace_cohort_membership_changed");
|
|
const saved = await input.db.select({ snapshot: executionWorkspaceSnapshots, operation: workspaceOperations })
|
|
.from(executionWorkspaceSnapshots).innerJoin(workspaceOperations, and(
|
|
eq(workspaceOperations.companyId, executionWorkspaceSnapshots.companyId),
|
|
eq(workspaceOperations.id, executionWorkspaceSnapshots.sourceOperationId)))
|
|
.where(and(eq(executionWorkspaceSnapshots.companyId, companyId), inArray(executionWorkspaceSnapshots.id, snapshotIds),
|
|
inArray(executionWorkspaceSnapshots.executionWorkspaceId, workspaceIds)));
|
|
const parsed = cohortSchema.safeParse(saved[0]?.operation.metadata?.cohortMigration);
|
|
if (!parsed.success) throw new Error("workspace_cohort_membership_changed");
|
|
const cohort = parsed.data;
|
|
if (saved.length !== workspaceIds.length || cohort.companyId !== companyId
|
|
|| canonical(cohort.members.map(member => member.workspaceId)) !== canonical(workspaceIds)
|
|
|| new Set(cohort.members.map(member => member.operationId)).size !== workspaceIds.length
|
|
|| saved.some(({ snapshot, operation }) => operation.status !== "succeeded" || operation.phase !== "remote_workspace_migration"
|
|
|| operation.executionWorkspaceId !== snapshot.executionWorkspaceId || !["committed", "superseded"].includes(snapshot.status)
|
|
|| operation.metadata?.snapshotId !== snapshot.id || operation.metadata?.manifestDigest !== snapshot.manifestDigest
|
|
|| cohort.members.find(member => member.workspaceId === snapshot.executionWorkspaceId)?.operationId !== operation.id
|
|
|| canonical(operation.metadata?.cohortMigration) !== canonical(cohort))) throw new Error("workspace_cohort_membership_changed");
|
|
return { companyId, cohortId: cohort.id, applied: Boolean(input.apply), replayed: true,
|
|
members: await Promise.all(workspaceIds.map(workspaceId => inventoryRemoteWorkspaceMigration(input.db, { companyId, workspaceId }))) };
|
|
}
|
|
if (imported.some(Boolean)) throw new Error("workspace_cohort_membership_changed");
|
|
const stop = new AbortController(), signal = AbortSignal.any([input.signal, stop.signal]);
|
|
const members = await Promise.all(workspaceIds.map(async workspaceId => {
|
|
const memberInput: MigrationInput = { db: input.db, storage: input.storage, companyId, workspaceId, apply: true, signal };
|
|
const onlineSource = await resolveMigrationSource(memberInput);
|
|
if (!onlineSource) throw new Error("workspace_cohort_online_source_required");
|
|
return { input: memberInput, onlineSource };
|
|
}));
|
|
const scopeIdentity = (scope: OnlineWorkspaceSource) => canonical({ kind: scope.kind,
|
|
projectIds: scope.projectIds, issueIds: scope.issueIds, workspaceIds: scope.workspaceIds });
|
|
if (members.some(member => scopeIdentity(member.onlineSource) !== scopeIdentity(members[0].onlineSource)))
|
|
throw new Error("workspace_cohort_source_scope_mismatch");
|
|
for (const member of members) for (const other of members) {
|
|
if (member !== other && member.onlineSource.root !== other.onlineSource.root
|
|
&& member.onlineSource.root.startsWith(other.onlineSource.root + path.sep)) throw new Error("workspace_cohort_leaf_members_required");
|
|
}
|
|
const sourceProof = createHash("sha256").update(canonical({ scope: scopeIdentity(members[0].onlineSource),
|
|
members: initial.map(workspace => ({ workspaceId: workspace.id, fingerprint: fingerprint(workspace) })) })).digest("hex");
|
|
const pending = [];
|
|
for (const workspace of initial) {
|
|
const marker = object(workspace.metadata?.remoteWorkspaceMigration);
|
|
if (marker.schema !== SCHEMA || typeof marker.operationId !== "string") continue;
|
|
const [operation] = await input.db.select().from(workspaceOperations).where(and(
|
|
eq(workspaceOperations.companyId, companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id),
|
|
eq(workspaceOperations.id, marker.operationId), eq(workspaceOperations.phase, "remote_workspace_migration")));
|
|
if (!operation?.metadata?.cohortMigration) throw new Error("workspace_cohort_foreign_journal");
|
|
if (operation.status !== "running" || operation.finishedAt !== null) throw new Error("workspace_cohort_journal_changed");
|
|
pending.push(cohortSchema.parse(operation.metadata.cohortMigration));
|
|
}
|
|
const cohort: WorkspaceMigrationCohort = pending[0] ?? { schema: "paperclip.remote-workspace-cohort.v1", id: randomUUID(), companyId,
|
|
sourceProof, members: workspaceIds.map(workspaceId => ({ workspaceId, operationId: randomUUID() })) };
|
|
if (cohort.companyId !== companyId || cohort.sourceProof !== sourceProof
|
|
|| canonical(cohort.members.map(member => member.workspaceId)) !== canonical(workspaceIds)
|
|
|| new Set(cohort.members.map(member => member.operationId)).size !== workspaceIds.length
|
|
|| pending.some(marker => canonical(marker) !== canonical(cohort))) throw new Error("workspace_cohort_membership_changed");
|
|
const preflight = async (db: ReadDb) => {
|
|
for (const member of members) {
|
|
const current = await inspect(db, { ...member.input, onlineSource: member.onlineSource }, cohort);
|
|
if (current.reasons.length) throw new Error(`remote_workspace_migration_blocked:${current.reasons.join(",")}`);
|
|
if (fingerprint(current.workspace) !== fingerprint(initial.find(workspace => workspace.id === member.input.workspaceId)!))
|
|
throw new Error("workspace_cohort_source_changed");
|
|
}
|
|
};
|
|
if (!input.apply) { await preflight(input.db); return { companyId, cohortId: cohort.id, applied: false, eligible: true, workspaceIds }; }
|
|
const publications: Parameters<typeof publishActivity>[0][] = [];
|
|
const result = await withIndependentWorkspaceSourceLocks(input.db, { companyId,
|
|
projectIds: members[0].onlineSource.projectIds, issueIds: members[0].onlineSource.issueIds }, async lock => {
|
|
for (const workspaceId of members[0].onlineSource.workspaceIds) {
|
|
if (!await tryAcquireExecutionWorkspaceLifecycleLock(lock, workspaceId)) throw new IsolatedRepositorySourceBusy("workspace_source_busy");
|
|
}
|
|
await preflight(input.db);
|
|
const plans: MigrationPlan[] = [];
|
|
for (const member of members) {
|
|
const reserved = await reserveMigration(member.input, member.onlineSource, cohort,
|
|
fingerprint(initial.find(workspace => workspace.id === member.input.workspaceId)!));
|
|
if (reserved.authority) throw new Error("workspace_cohort_membership_changed");
|
|
plans.push({ ...member, reserved });
|
|
}
|
|
await lockMigrationRecovery(lock, plans, cohort);
|
|
const captures: Array<CapturedMigration | undefined> = Array(plans.length);
|
|
const childVerifier = createMigrationChildVerifier(plans[0].input, plans[0].onlineSource, lock);
|
|
let next = 0, failure: unknown;
|
|
const worker = async () => {
|
|
while (!signal.aborted && next < plans.length) {
|
|
const index = next++, plan = plans[index];
|
|
try { captures[index] = await captureMigration(plan.input, plan.reserved, plan.onlineSource, childVerifier); }
|
|
catch (error) { failure ??= error; stop.abort(error); }
|
|
}
|
|
};
|
|
try {
|
|
const peers = await Promise.allSettled(Array.from({ length: Math.min(concurrency, plans.length) }, worker));
|
|
for (const peer of peers) if (peer.status === "rejected") failure ??= peer.reason;
|
|
if (failure) throw failure;
|
|
signal.throwIfAborted();
|
|
await lock.execute(sql`select 1`);
|
|
await lock.transaction(async tx => {
|
|
await lockMigrationParents(tx, companyId, null, plans.flatMap(plan => [
|
|
plan.reserved.workspace.sourceIssueId!, ...plan.reserved.legacyRecoverySources.map(source => source.issueId),
|
|
...plan.onlineSource!.companions.flatMap(proof => proof.sources.map(source => source.issueId)),
|
|
]));
|
|
const validated = [];
|
|
for (const [index, plan] of plans.entries()) {
|
|
signal.throwIfAborted();
|
|
const current = await validateMigrationPublication(tx, plan.input, plan.reserved, captures[index]!, plan.onlineSource, cohort);
|
|
if (!current) throw new Error("workspace_cohort_publication_changed");
|
|
validated.push(current);
|
|
}
|
|
for (const [index, plan] of plans.entries()) {
|
|
signal.throwIfAborted();
|
|
publications.push(await publishMigration(tx, plan.reserved, captures[index]!, validated[index]));
|
|
}
|
|
signal.throwIfAborted();
|
|
}).catch(publicationError);
|
|
const results = [];
|
|
for (const workspaceId of workspaceIds) results.push(await inventoryRemoteWorkspaceMigration(lock as unknown as Db, { companyId, workspaceId }));
|
|
return { companyId, cohortId: cohort.id, applied: true, replayed: false, members: results };
|
|
} catch (error) { failure = error; throw error; }
|
|
finally {
|
|
await childVerifier?.close();
|
|
const cleanup = await Promise.allSettled(captures.filter((capture): capture is CapturedMigration => Boolean(capture))
|
|
.map(capture => removeTaskDirectoryTree(capture.scratch)));
|
|
const errors = cleanup.flatMap(outcome => outcome.status === "rejected" ? [outcome.reason] : []);
|
|
if (errors.length) throw new AggregateError(failure ? [failure, ...errors] : errors, "workspace_cohort_cleanup_failed");
|
|
if (!failure) signal.throwIfAborted();
|
|
}
|
|
}, true);
|
|
for (const publication of publications) publishActivity(publication);
|
|
return result;
|
|
}
|