From 203511852618cb729928d06c1e0353ef022da394 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:36:22 -0600 Subject: [PATCH] fix(workspaces): preserve archived data during remote migration (#18826) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspace storage lets the web server run without a persistent volume. > - Existing deployments also contain archived and historical task workspaces. > - Retention can remove their folders while preserving encrypted restore points or Git reconstruction records. > - The importer rejected those records and only accepted the task's current workspace. > - This pull request imports verified saved data without reopening or rebinding tasks. > - The benefit is preservation of existing work during an explicit storage migration. ## Linked Issues or Issue Description Refs #18403 and #17472. **What happened?** An explicit remote-workspace import rejected archived records, historical task bindings, and removed folders with valid retained data. **Expected behavior** Import the recorded files or exact Git commit. Preserve archived state and the task's current binding. Refuse missing, expired, corrupt, or changed sources. **Steps to reproduce** Archive a task directory with a restore point, remove its folder, and run the remote-workspace import. Repeat with a Git restore point containing an unpushed commit and staged and unstaged edits. The previous importer reports an inactive workspace or missing source. ## What Changed - Preserve lifecycle state and current task binding while importing archived and historical records. Recheck the binding before publication and on resume. - Expose the task-directory owner's validated restore-point reader. Reuse it for ordinary restoration and migration. - Restore archived directories and Git data into private scratch space. Support saved Git recipes and exact legacy branch-deletion receipts. - Recover pruned unpushed Git commits from the encrypted bundle and its explicit prerequisites. - Keep independent snapshot download, restoration, source checks, writer guards, encryption, and atomic publication. - Document the supported import sources and the remaining tenant-enrollment gaps. ## Verification - Before production edits, two new regressions failed on archived status and historical task binding. - Final focused verification: 58 tests pass across remote-workspace migration and isolated task-directory restoration. - Real PostgreSQL, Git repositories, encrypted storage, and restored file bytes cover cold restoration after removing the old folder and source object, pruned unpushed history, index and worktree edits, corrupt/missing/expired sources, binding changes, and source-folder races. - A Git fixture initially lacked runtime ownership metadata. Correcting the fixture made the valid retained-source case pass; no production validation was weakened. - Full repository typecheck and build passed. Full `pnpm test:run` stopped in chat integration shard 2 after two failures: a retry-notice timeout and a subsequent fake-timer error. All four cases in that retry-notice group passed in a separate focused rerun without source changes; the complete suite has not passed locally. All exact-head CI jobs passed. The separate Paperclip Review app check is still queued; all existing review threads are resolved. Server typecheck passed after the review fixes. - Both review findings were reproduced: the real cleanup owner's verified-tip receipt was rejected, and a lost upload reply for a legacy Git source failed with `operation_content_changed`. The fixes accept the owner's scoped receipt metadata and rebuild reconstructed Git indexes from their staged trees. Both regressions now pass, including retained staged and unstaged edits. - Additional coverage restores from a fresh HTTP clone after deleting the original repository, imports a nested local checkout with encrypted settings, and rejects an unapproved nested source without publishing remote authority. - This PR does not claim a completed tenant enrollment or a new zero-downtime deployment proof. ## Risks Imports can fail when old receipts or repository objects are unavailable. They retain their migration journal and leave the original state intact. Operators must resolve pending writers and recovery before importing. Shared physical roots, nested worktrees, and unresolved sandbox recovery require separate migration work. This change does not remove those blockers or alter automatic retention. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. This session does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task ID or instance-derived details - [x] I have run the focused tests locally and they pass; full typecheck/build passed and the full-suite failure is documented - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DEVELOPING.md | 16 ++ ...-existing-workspace-ephemeral-migration.md | 45 +++ .../isolated-task-directory-retention.ts | 30 +- .../remote-workspace-migration-git.ts | 142 ++++++++++ .../remote-workspace-migration.test.ts | 263 +++++++++++++++++- .../services/remote-workspace-migration.ts | 74 ++++- 6 files changed, 547 insertions(+), 23 deletions(-) create mode 100644 doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md create mode 100644 server/src/services/remote-workspace-migration-git.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 182afe1d850..78621549793 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1878,6 +1878,22 @@ or recovery, unsafe paths, and explicitly concurrent shared writers block import. An interrupted apply retains its journal and admission hold; retry the same command to reconcile it before starting another writer. +An import preserves the workspace's lifecycle state and the task's current +workspace binding. It can import an archived or superseded workspace without +reopening the task or selecting that older workspace for future turns. A changed +task binding during the import blocks publication. + +If an archived folder has already been removed, the importer can reconstruct it +in private scratch space from its encrypted task-directory restore point or +validated Git restoration recipe. For older Git workspaces without a recipe, +it requires a successful branch-deletion receipt and the exact saved commit in +the verified source repository. It never substitutes today's default branch. +Git restore points retain unpushed history and staged and unstaged edits, even +when the source repository has pruned the saved branch. Missing, expired, +corrupt, or unbound sources block the import. Publication still requires an +independent download and restore of the new encrypted snapshot. The old folder +remains absent, and the importer does not change the original restore point. + Supported sources include empty task folders, repository URLs, and managed mutable repositories. Authorized mentioned projects attach all their configured Git repositories as writable copies under `.paperclip-repositories/` in the task diff --git a/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md b/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md new file mode 100644 index 00000000000..65c7dd3d351 --- /dev/null +++ b/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md @@ -0,0 +1,45 @@ +# Preserve existing workspaces during ephemeral enrollment + +The deployment endpoint remains a volume-free Cloud tenant, followed by a real +rolling deployment while an agent works and new work is admitted. Enrollment must +preserve existing task files and historical recovery sources. It must not reset +task workspaces, manufacture successful copyback, or weaken migration guards. + +## Evidence and remaining work + +| Required outcome | Current evidence | Remaining gap | Owner / next action | +| --- | --- | --- | --- | +| Existing active task retains files | The current import encrypts, uploads, independently restores, compares the source again and publishes under the lifecycle lock. A live completed task imported successfully with its original directory retained. | The importer only accepts an active workspace with the current task binding. | Workspace migration owner: support historical records without changing the task binding or lifecycle state. | +| Archived task retains its saved files | Retention records include encrypted task-directory restore points, Git restore points and verified Git reconstruction recipes. | The importer rejects missing local folders even when an existing restore point can reconstruct them. | Restore-point owners: reconstruct into migration scratch space, reject expired/missing/corrupt sources, and feed the verified result into the normal import publication. | +| Shared project roots preserve all task files | Legacy roots contain nested task worktrees and multiple records can reference one physical folder. | A naive root capture recursively imports those other workspaces; the current per-record writer check does not fence aliases. | Workspace migration owner: inventory physical overlaps, fence all writers, and preserve nested workspace coverage before excluding any directory. | +| Failed runs retain unsaved files | Provider inspection distinguishes surviving archived sandboxes from missing allocations. Existing recovery validation accepts some sources and rejects others for missing provenance or ambiguous allocation ownership. | Recovery must settle or durably preserve each exact source. A successful old preservation action does not necessarily update the legacy sync descriptor. | Workspace recovery owner: verify existing preservation receipts, recover eligible sources, and retain explicit unresolved obligations for unavailable/ambiguous sources. Do not replay agent work. | +| Tenant is eligible without its volume | External DB and S3 are configured; the original volume remains attached. | Workspaces, local run logs and remaining owner-maintained data must pass the packaged maintenance receipt. | Cloud enrollment owner, after workspace migration is complete. | +| Ordinary upgrades remain available | The no-drain Cloud deployment path is merged and qualified on another enrolled tenant. | Exact final app image qualification and a live deployment on this tenant. | Cloud deployment owner: qualify, enroll, then prove reads, submissions and active-run adoption through an actual replacement. | + +## Acceptance boundaries + +The first implementation slice supports archived and historical records, +encrypted task-directory restoration, Git restoration recipes, and legacy +branch-deletion receipts. PostgreSQL and real-file tests prove cold restoration +of saved bytes, index state, and unpushed commits, including after pruning the +old Git object store. Invalid or unavailable retained data still blocks import. +Shared-root migration and recovery-obligation reconciliation remain separate +gaps; this slice does not make a whole tenant eligible for enrollment. + +- Import existing files or a verified retained source. A missing folder alone is + not permission to publish an empty workspace or choose a newer Git branch. +- Preserve archived status, task bindings, task history, unpushed Git history, + staged and unstaged edits, and saved recovery copies. +- Preserve original files and the detached original volume until verified + migration receipts and the retention interval permit removal. +- Parent roots may omit a nested workspace only with durable, inspectable + coverage of that exact nested directory. Arbitrary exclusion globs are not an + acceptable data migration. +- Resume interrupted imports with their original identity; cancellation may + release an empty reservation only after the producer is stopped and no upload + was journaled. Preserve an audit receipt. +- Verify restored file bytes independently, including after making the old + source unavailable in an isolated test. Exercise current and historical + bindings, archived sources, overlap/writer races, and unavailable sources. +- Unrelated tenants, production services, and agent task execution remain outside + the migration's mutation scope. diff --git a/server/src/services/isolated-task-directory-retention.ts b/server/src/services/isolated-task-directory-retention.ts index cb5a8dc8a37..b4c83c70e66 100644 --- a/server/src/services/isolated-task-directory-retention.ts +++ b/server/src/services/isolated-task-directory-retention.ts @@ -124,7 +124,7 @@ function restorePointObjectKey(companyId: string, workspaceId: string, generatio return `${companyId}/${NAMESPACE}/${workspaceId}/${generation}/task-directory/${id}.pcrp`; } -function validateRestorePointRef(ref: TaskDirectoryRestorePointRef | null | undefined, workspace: { companyId: string; id: string }) { +export function validateTaskDirectoryRestorePointRef(ref: TaskDirectoryRestorePointRef | null | undefined, workspace: { companyId: string; id: string }) { if (!ref || ref.schema !== TASK_DIRECTORY_RESTORE_POINT_SCHEMA || typeof ref.id !== "string" || !UUID.test(ref.id) || !Number.isSafeInteger(ref.generation) || ref.generation < 1 || ref.objectKey !== restorePointObjectKey(workspace.companyId, workspace.id, ref.generation, ref.id) @@ -136,6 +136,25 @@ function validateRestorePointRef(ref: TaskDirectoryRestorePointRef | null | unde return ref; } +/** Read a retained directory without reopening its workspace or replacing its + * original path. Administrative migration uses the same identity and byte checks + * as ordinary restore, and never treats an expired object as an empty folder. */ +export async function loadTaskDirectoryRestorePoint(input: { + storage: StorageService; + workspace: { companyId: string; id: string }; + ref: TaskDirectoryRestorePointRef; + signal?: AbortSignal; +}) { + const ref = validateTaskDirectoryRestorePointRef(input.ref, input.workspace); + if (ref.expiredAt) throw new TaskDirectoryRestoreFailure("restore_point_expired"); + const ciphertext = await readRestorePointObject(input.storage, input.workspace.companyId, ref.objectKey, input.signal).catch(() => { + throw new TaskDirectoryRestoreFailure("restore_point_unavailable"); + }); + if (ciphertext.length !== ref.byteSize) throw new TaskDirectoryRestoreFailure("ciphertext_size_mismatch"); + return openRestorePoint({ ciphertext, sha256: ref.sha256, key: ref.key, + identity: { companyId: input.workspace.companyId, workspaceId: input.workspace.id, restorePointId: ref.id, generation: ref.generation } }); +} + function skipReason(error: unknown) { if (error instanceof TaskDirectorySnapshotSkip) return error.reason; if (error instanceof IsolatedRepositorySourceBusy) return "source_busy"; @@ -341,7 +360,7 @@ export async function restoreIsolatedTaskDirectory(input: { const parent = path.dirname(cwd); await fs.mkdir(parent, { recursive: true, mode: 0o700 }); if (await fs.realpath(parent) !== parent) throw new TaskDirectoryRestoreFailure("task_directory_identity_changed"); - const ref = validateRestorePointRef(state.restorePoint, input.workspace); + const ref = validateTaskDirectoryRestorePointRef(state.restorePoint, input.workspace); if (await directoryState(cwd) === "present") { if (!state.removed) return { restored: false, expired: false }; if ((await fs.readdir(cwd)).length === 0) await fs.rmdir(cwd); @@ -355,12 +374,7 @@ export async function restoreIsolatedTaskDirectory(input: { await fs.mkdir(cwd, { mode: 0o700 }); return { restored: false, expired: true }; } - const ciphertext = await readRestorePointObject(input.storage, input.workspace.companyId, ref.objectKey, input.signal).catch(() => { - throw new TaskDirectoryRestoreFailure("restore_point_unavailable"); - }); - if (ciphertext.length !== ref.byteSize) throw new TaskDirectoryRestoreFailure("ciphertext_size_mismatch"); - const payload = await openRestorePoint({ ciphertext, sha256: ref.sha256, key: ref.key, - identity: { companyId: input.workspace.companyId, workspaceId: input.workspace.id, restorePointId: ref.id, generation: ref.generation } }); + const payload = await loadTaskDirectoryRestorePoint({ ...input, ref }); const staging = await materializeTaskDirectorySnapshot({ payload, target: cwd, expectedDigest: ref.digest, signal: input.signal }); try { await fs.rename(staging, cwd); diff --git a/server/src/services/remote-workspace-migration-git.ts b/server/src/services/remote-workspace-migration-git.ts new file mode 100644 index 00000000000..66d82639c38 --- /dev/null +++ b/server/src/services/remote-workspace-migration-git.ts @@ -0,0 +1,142 @@ +import { execFile } from "node:child_process"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { promisify } from "node:util"; +import { and, desc, eq, gte, or, sql } from "drizzle-orm"; +import { executionWorkspaces, projectWorkspaces, workspaceOperations, type Db } from "@paperclipai/db"; +import { PROJECT_REPOSITORIES_DIR } from "@paperclipai/adapter-utils/git-workspace-sync"; +import type { StorageService } from "../storage/types.js"; +import { createIssueGitRemoteAuthProvider } from "./git-credentials.js"; +import { applyGitRestorePointToWorktree, ensureGitRestorePointCommit, loadGitRestorePoint } from "./git-restore-point-storage.js"; +import { restoreGitWorkspaceContents } from "./git-workspace-contents.js"; +import { ensureSavedGitCommit, gitRestorePointExpired, managedSourcePaths, readGitCacheRestoration, + verifiedGitCacheSource, type GitCacheRestoration } from "./git-workspace-restoration.js"; +import { localRepositoryPath } from "./local-repository-path.js"; +import { materializeManagedProjectWorkspace } from "./managed-repository-checkout.js"; + +type Workspace = typeof executionWorkspaces.$inferSelect; +export type ArchivedGitMigrationSource = { + commit: string; + sourcePath: string; + cleanupOperationId: string | null; + recipe: GitCacheRestoration | null; +}; +const execute = promisify(execFile); +const unavailable = () => new Error("remote_workspace_migration_archived_git_source_unavailable"); +async function git(cwd: string, args: string[]) { + return (await execute("git", ["-c", "core.fsmonitor=false", "-c", `core.hooksPath=${os.devNull}`, "-C", cwd, ...args], { + timeout: 120_000, maxBuffer: 1024 * 1024, + env: { ...process.env, GIT_CONFIG_NOSYSTEM: "1", GIT_CONFIG_GLOBAL: os.devNull, + GIT_TERMINAL_PROMPT: "0", GIT_OPTIONAL_LOCKS: "0" }, + })).stdout.trim(); +} + +/** A removed branch must be reconstructed from its saved commit or its exact + * successful deletion receipt. Today's default branch is never a substitute. */ +export async function inspectArchivedGitMigrationSource(db: Pick, workspace: Workspace): Promise { + if (workspace.status !== "archived" || !workspace.repoUrl || !workspace.branchName || !workspace.sourceIssueId) return null; + const recipe = readGitCacheRestoration(workspace); + if (recipe) { + if (gitRestorePointExpired(recipe)) throw new Error("remote_workspace_migration_restore_point_expired"); + return { commit: recipe.commit, sourcePath: recipe.sourcePath, recipe, cleanupOperationId: null }; + } + if (workspace.providerType !== "git_worktree" || !workspace.closedAt || workspace.metadata?.gitBranchOwnershipVersion !== 1) return null; + const [operation] = await db.select().from(workspaceOperations).where(and( + eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id), + eq(workspaceOperations.phase, "worktree_cleanup"), eq(workspaceOperations.status, "succeeded"), + or(eq(workspaceOperations.command, `git branch -d ${workspace.branchName}`), and( + sql`${workspaceOperations.metadata}->>'cleanupAction' = 'branch_delete'`, + sql`${workspaceOperations.metadata}->>'workspaceId' = ${workspace.id}`, + sql`${workspaceOperations.metadata}->>'branchName' = ${workspace.branchName}`, + sql`${workspaceOperations.metadata}->>'workspacePath' = ${workspace.providerRef ?? workspace.cwd}`, + )), gte(workspaceOperations.startedAt, workspace.closedAt), + )).orderBy(desc(workspaceOperations.startedAt), desc(workspaceOperations.id)).limit(1); + const receipt = operation?.stdoutExcerpt?.trim(), prefix = `Deleted branch ${workspace.branchName} (was `; + if (!operation || !receipt?.startsWith(prefix) || !receipt.endsWith(").")) return null; + const abbreviation = receipt.slice(prefix.length, -2); + if (!/^[a-f0-9]{7,40}$/.test(abbreviation)) throw unavailable(); + const [projectWorkspace] = workspace.projectWorkspaceId ? await db.select({ cwd: projectWorkspaces.cwd, repoUrl: projectWorkspaces.repoUrl }) + .from(projectWorkspaces).where(and(eq(projectWorkspaces.id, workspace.projectWorkspaceId), + eq(projectWorkspaces.companyId, workspace.companyId), workspace.projectId ? eq(projectWorkspaces.projectId, workspace.projectId) : undefined)) : []; + const sourcePath = projectWorkspace?.cwd ?? managedSourcePaths(workspace)[0]; + if (!sourcePath || projectWorkspace?.repoUrl && projectWorkspace.repoUrl !== workspace.repoUrl) throw unavailable(); + await verifiedGitCacheSource(workspace, sourcePath); + const commit = await git(sourcePath, ["rev-parse", "--verify", `${abbreviation}^{commit}`]); + if (!/^[a-f0-9]{40}$/.test(commit)) throw unavailable(); + return { commit, sourcePath, cleanupOperationId: operation.id, recipe: null }; +} + +/** Restore only into private migration scratch. Original repositories, task + * folders, lifecycle state and restore-point metadata remain unchanged. */ +export async function materializeArchivedGitMigrationSource(input: { + db: Db; storage: StorageService; workspace: Workspace; source: ArchivedGitMigrationSource; scratch: string; +}) { + const { workspace, source } = input, cwd = path.join(input.scratch, "retained-git"); + if (!workspace.sourceIssueId || !workspace.repoUrl || !workspace.branchName) throw unavailable(); + const resolveGitAuth = createIssueGitRemoteAuthProvider(input.db, workspace.companyId, workspace.sourceIssueId); + const present = await fs.lstat(source.sourcePath).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (present) { + const verified = await verifiedGitCacheSource(workspace, source.sourcePath); + await fs.mkdir(cwd, { mode: 0o700 }); await git(cwd, ["init", "--quiet"]); + // Fetch objects, never the current working tree of a shared project root. + const presentCommit = await git(verified, ["cat-file", "-e", `${source.commit}^{commit}`]).then(() => true, () => false); + if (presentCommit) { + await git(cwd, ["fetch", "--no-tags", "--no-recurse-submodules", "--", verified, source.commit]); + } else if (source.recipe?.restorePoint && !source.recipe.restorePoint.headOnRemote) { + // The archived branch can have been pruned from the base while its + // encrypted bundle still owns the unpushed commits. Import only that + // bundle's explicit prerequisites, then let the restore-point owner + // verify and import the saved history below. + const { manifest } = await loadGitRestorePoint({ storage: input.storage, workspace, ref: source.recipe.restorePoint }); + for (const commit of manifest.prerequisites) { + const local = await git(verified, ["cat-file", "-e", `${commit}^{commit}`]).then(() => true, () => false); + if (local) await git(cwd, ["fetch", "--no-tags", "--no-recurse-submodules", "--", verified, commit]); + else await ensureSavedGitCommit(cwd, workspace.repoUrl, commit, resolveGitAuth); + } + } + } else { + if (!source.recipe) throw unavailable(); + await materializeManagedProjectWorkspace(cwd, { repoUrl: workspace.repoUrl, resolveGitAuth }); + } + if (source.recipe?.restorePoint) await ensureGitRestorePointCommit({ storage: input.storage, workspace, + ref: source.recipe.restorePoint, repositoryCwd: cwd }); + await ensureSavedGitCommit(cwd, workspace.repoUrl, source.commit, resolveGitAuth); + await git(cwd, ["check-ref-format", `refs/heads/${workspace.branchName}`]); + await git(cwd, ["checkout", "--detach", source.commit]); + await git(cwd, ["update-ref", `refs/heads/${workspace.branchName}`, source.commit]); + await git(cwd, ["symbolic-ref", "HEAD", `refs/heads/${workspace.branchName}`]); + await git(cwd, ["config", "remote.origin.url", workspace.repoUrl]); + if (source.recipe?.restorePoint) await applyGitRestorePointToWorktree({ storage: input.storage, workspace, + ref: source.recipe.restorePoint, repositoryCwd: cwd, worktreePath: cwd }); + const approvedLocalSources = new Set(); + await restoreGitWorkspaceContents({ cwd, companyId: workspace.companyId, workspaceId: workspace.id, + projectId: workspace.projectId, contents: source.recipe?.contents, storage: input.storage, resolveGitAuth, + authorizeSource: async saved => { + if (!workspace.projectId) throw unavailable(); + const [row] = await input.db.select().from(projectWorkspaces).where(and(eq(projectWorkspaces.id, saved.id), + eq(projectWorkspaces.companyId, workspace.companyId), eq(projectWorkspaces.projectId, workspace.projectId))); + if (!row || row.repoUrl !== saved.repoUrl) throw unavailable(); + const local = localRepositoryPath(saved.repoUrl); if (local) approvedLocalSources.add(local); + }, + prepareLocalSource: async local => { + if (!approvedLocalSources.has(local)) throw unavailable(); + const identity = await fs.realpath(local); + return async () => { if (await fs.realpath(local) !== identity) throw unavailable(); }; + }, + }); + // A checkout records scratch-file inode and time fields in its index. Those + // fields must not change the saved archive identity after an upload reply is + // lost. Rebuild only these reconstructed indexes from their staged trees; + // the original repository and unstaged files remain untouched. + for (const repository of [cwd, ...(source.recipe?.contents?.checkouts ?? []) + .map(checkout => path.join(cwd, PROJECT_REPOSITORIES_DIR, checkout.name))]) { + const tree = await git(repository, ["write-tree"]); + await git(repository, ["read-tree", "--empty"]); + await git(repository, ["read-tree", tree]); + } + return cwd; +} diff --git a/server/src/services/remote-workspace-migration.test.ts b/server/src/services/remote-workspace-migration.test.ts index 0e55b224eab..e2345e3cb88 100644 --- a/server/src/services/remote-workspace-migration.test.ts +++ b/server/src/services/remote-workspace-migration.test.ts @@ -4,6 +4,8 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { promisify } from "node:util"; +import { createServer, type Server } from "node:http"; +import { pathToFileURL } from "node:url"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { eq } from "drizzle-orm"; import { activityLog, applyPendingMigrations, closeRegisteredClients, companies, createDb, executionWorkspaceSnapshots, @@ -16,7 +18,14 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro import { inventoryRemoteWorkspaceMigration, migrateRemoteWorkspace } from "./remote-workspace-migration.js"; import { getWorkspaceSnapshot, loadWorkspaceSnapshotArchive } from "./remote-workspace-snapshot-storage.js"; import { restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js"; -import { removeTaskDirectoryTree } from "./isolated-task-directory-snapshot.js"; +import { captureTaskDirectorySnapshot, removeTaskDirectoryTree } from "./isolated-task-directory-snapshot.js"; +import { captureRestorePoint, sealRestorePoint } from "./git-restore-point.js"; +import type { TaskDirectoryRestorePointRef } from "./isolated-task-directory-retention.js"; +import { resolveManagedProjectWorkspaceDir } from "../home-paths.js"; +import { cleanupExecutionWorkspaceArtifacts } from "./workspace-runtime.js"; +import { workspaceOperationService } from "./workspace-operations.js"; +import { managedProjectCheckoutName } from "./managed-project-checkout-name.js"; +import { captureGitWorkspaceContents, readManagedCheckoutContents } from "./git-workspace-contents.js"; const execute = promisify(execFile); const git = async (cwd: string, ...args: string[]) => (await execute("git", ["-C", cwd, ...args], { env: { ...process.env, @@ -28,6 +37,7 @@ if (url && !/^remote_workspace_[a-z_]*test$/.test(new URL(url).pathname.slice(1) const support = url ? { supported: true } : await getEmbeddedPostgresTestSupport(); describe.skipIf(!support.supported)("remote workspace migration", () => { let db: ReturnType, root: string, storage: StorageService; + const servers: Server[] = []; let temporary: Awaited> | undefined; beforeAll(async () => { if (url) { await applyPendingMigrations(url); db = createDb(url); } @@ -37,6 +47,7 @@ describe.skipIf(!support.supported)("remote workspace migration", () => { beforeEach(async () => { vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "61".repeat(32)); root = await fs.mkdtemp(path.join(os.tmpdir(), "pc-migration-test-")); storage = createStorageService(createLocalDiskStorageProvider(path.join(root, "objects"))); }); afterEach(async () => { + await Promise.all(servers.splice(0).map(server => new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve())))); vi.unstubAllEnvs(); await db.delete(activityLog); await db.delete(workspaceOperations); await db.delete(executionWorkspaceSnapshots); await db.delete(workspaceRuntimeServices); await db.delete(executionWorkspaceRuntimeLeases); await db.delete(issues); await db.delete(executionWorkspaces); await db.delete(projectWorkspaces); await db.delete(projects); @@ -53,6 +64,256 @@ describe.skipIf(!support.supported)("remote workspace migration", () => { return { companyId, workspaceId, issueId: issue!.id, cwd }; } async function workspace(id: string) { return (await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, id)))[0]!; } + async function gitArchiveFixture(repoUrl = "https://example.invalid/repository.git") { + const f = await seed(), projectId = randomUUID(), projectWorkspaceId = randomUUID(), branchName = "saved-task"; + vi.stubEnv("PAPERCLIP_HOME", root); vi.stubEnv("PAPERCLIP_INSTANCE_ID", "default"); + const base = resolveManagedProjectWorkspaceDir({ companyId: f.companyId, projectId, repoName: "repository" }); + await fs.mkdir(base, { recursive: true }); await git(base, "init", "-q", "-b", "main"); + await fs.writeFile(path.join(base, "base.txt"), "base\n"); await git(base, "add", "."); await git(base, "commit", "-qm", "base"); + await git(base, "remote", "add", "origin", repoUrl); const upstream = await git(base, "rev-parse", "HEAD"); + const cwd = path.join(base, ".paperclip", "worktrees", branchName); + await git(base, "worktree", "add", "-b", branchName, cwd); + await fs.copyFile(path.join(f.cwd, "saved.txt"), path.join(cwd, "saved.txt")); await fs.rm(f.cwd, { recursive: true }); + await git(cwd, "add", "."); await git(cwd, "commit", "-qm", "saved task commit"); const commit = await git(cwd, "rev-parse", "HEAD"); + await db.insert(projects).values({ id: projectId, companyId: f.companyId, name: "Archived project" }); + await db.insert(projectWorkspaces).values({ id: projectWorkspaceId, companyId: f.companyId, projectId, name: "Source", cwd: base, repoUrl }); + await db.update(executionWorkspaces).set({ cwd, providerRef: cwd, projectId, projectWorkspaceId, repoUrl, baseRef: "main", + branchName, strategyType: "git_worktree", providerType: "git_worktree", metadata: { createdByRuntime: true, gitBranchOwnershipVersion: 1 } }).where(eq(executionWorkspaces.id, f.workspaceId)); + return { ...f, cwd, base, upstream, commit, projectId, projectWorkspaceId, repoUrl, branchName }; + } + it.each(["legacy", "verified", "retry"])("recovers an archived Git workspace from its exact branch-deletion receipt (%s)", async receiptKind => { + const f = await gitArchiveFixture(), closedAt = new Date(); + await db.update(executionWorkspaces).set({ status: "archived", closedAt }).where(eq(executionWorkspaces.id, f.workspaceId)); + if (receiptKind !== "verified") { + await git(f.base, "merge", "--ff-only", f.branchName); + await git(f.base, "worktree", "remove", f.cwd); + const receipt = await git(f.base, "branch", "-d", f.branchName); + await db.insert(workspaceOperations).values({ companyId: f.companyId, executionWorkspaceId: f.workspaceId, + phase: "worktree_cleanup", status: "succeeded", command: `git branch -d ${f.branchName}`, + stdoutExcerpt: receipt, startedAt: new Date(), finishedAt: new Date() }); + } else { + const cleaned = await cleanupExecutionWorkspaceArtifacts({ workspace: await workspace(f.workspaceId), + projectWorkspace: { cwd: f.base, cleanupCommand: null }, expectedBranchHeadSha: f.commit, runCleanupCommands: false, + recorder: workspaceOperationService(db).createRecorder({ companyId: f.companyId, executionWorkspaceId: f.workspaceId, issueId: f.issueId }) }); + expect(cleaned.warnings).toEqual([]); + } + await fs.writeFile(path.join(f.base, "base.txt"), "newer default branch\n"); + await git(f.base, "add", "base.txt"); await git(f.base, "commit", "-qm", "new default branch work"); + if (receiptKind === "retry") { + const put = storage.putFile.bind(storage); let fail = true; + storage.putFile = async input => { const result = await put(input); if (fail) { fail = false; throw new Error("upload reply lost"); } return result; }; + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("upload reply lost"); + } + await migrateRemoteWorkspace({ db, storage, ...f, apply: true }); + const row = await workspace(f.workspaceId), authority = readRemoteWorkspaceAuthority(row.metadata)!; + expect(row.status).toBe("archived"); expect(authority.source).toMatchObject({ kind: "git", ref: f.commit }); + await fs.rm(f.base, { recursive: true }); + const ref = await getWorkspaceSnapshot(db, { ...f, snapshotId: authority.snapshotId! }); + const archivePath = path.join(root, "legacy-git.pcws"), targetDir = path.join(root, "restored-legacy-git"); + await loadWorkspaceSnapshotArchive({ storage, reference: ref!, targetPath: archivePath }); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref!.manifestDigest }); + expect(await git(targetDir, "rev-parse", "HEAD")).toBe(f.commit); + expect(await fs.readFile(path.join(targetDir, "base.txt"), "utf8")).toBe("base\n"); + expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("original saved work\n"); + }); + async function serveRepository() { + const directory = path.join(root, "http"), bare = path.join(directory, "repository.git"); + await fs.mkdir(bare, { recursive: true }); await git(bare, "init", "--bare", "--initial-branch=main"); + const server = createServer(async (request, response) => { + const relative = new URL(request.url!, "http://localhost").pathname.slice(1); + const target = path.resolve(directory, relative); + if (!target.startsWith(`${directory}/`)) { response.writeHead(403).end(); return; } + try { response.end(await fs.readFile(target)); } catch { response.writeHead(404).end(); } + }); + await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); servers.push(server); + const address = server.address(); if (!address || typeof address === "string") throw new Error("test server not bound"); + return { bare, repoUrl: `http://127.0.0.1:${address.port}/repository.git` }; + } + it.each(["present", "pruned", "retry", "missing-base", "nested", "unapproved-nested", "missing", "expired", "corrupt"])("preserves archived Git edits or refuses an unavailable restore point (%s)", async state => { + const remote = state === "missing-base" ? await serveRepository() : null; + const f = await gitArchiveFixture(remote?.repoUrl); + if (remote) { await git(remote.bare, "fetch", f.base, "main:refs/heads/main"); await git(remote.bare, "update-server-info"); } + await fs.writeFile(path.join(f.cwd, "saved.txt"), "staged edit\n"); await git(f.cwd, "add", "saved.txt"); + await fs.writeFile(path.join(f.cwd, "saved.txt"), "unstaged edit\n"); + const captured = (await captureRestorePoint({ cwd: f.cwd, upstreamCommits: [f.upstream] }))!; + const id = randomUUID(), generation = 1, sealed = await sealRestorePoint(captured.payload, + { companyId: f.companyId, workspaceId: f.workspaceId, restorePointId: id, generation }); + const objectKey = `${f.companyId}/git-restore-points/${f.workspaceId}/${generation}/${id}.pcrp`; + await storage.putFile({ companyId: f.companyId, namespace: "git-restore-points", objectKey, + originalFilename: "git.pcrp", contentType: "application/octet-stream", body: sealed.ciphertext }); + const restorePoint = { schema: "paperclip.git-restore-point.v1", id, generation, objectKey, byteSize: sealed.ciphertext.length, + sha256: sealed.sha256, key: sealed.key, head: captured.manifest.head, index: captured.manifest.index, + worktree: captured.manifest.worktree, ignored: captured.manifest.ignored, modes: captured.manifest.modes, headOnRemote: false }; + let contents; + if (state === "nested" || state === "unapproved-nested") { + const sourceCwd = path.join(root, "nested-source"); + await fs.mkdir(sourceCwd); await git(sourceCwd, "init", "-q", "-b", "main"); + await fs.writeFile(path.join(sourceCwd, "nested.txt"), "nested source work\n"); + await git(sourceCwd, "add", "."); await git(sourceCwd, "commit", "-qm", "nested source"); + const source = { id: randomUUID(), repoUrl: pathToFileURL(sourceCwd).href, repoRef: "main", cwd: sourceCwd }; + const name = managedProjectCheckoutName(source), nestedCwd = path.join(f.cwd, ".paperclip-repositories", name); + await git(f.cwd, "clone", source.repoUrl, nestedCwd); await git(nestedCwd, "checkout", "--detach"); + await fs.mkdir(path.join(nestedCwd, ".claude")); + await fs.writeFile(path.join(nestedCwd, ".claude", "settings.local.json"), '{"permissions":{"allow":["Read"]}}'); + contents = await captureGitWorkspaceContents(f.cwd, { companyId: f.companyId, workspaceId: f.workspaceId }, + [await readManagedCheckoutContents(nestedCwd, name, source)]); + if (state === "nested") await db.insert(projectWorkspaces).values({ ...source, companyId: f.companyId, projectId: f.projectId, name: "Nested source" }); + } + await db.update(executionWorkspaces).set({ status: "archived", closedAt: new Date(), metadata: { + createdByRuntime: true, gitBranchOwnershipVersion: 1, gitCacheRestoration: { schema: "paperclip.git-cache-restoration.v2", companyId: f.companyId, + workspaceId: f.workspaceId, sourceIssueId: f.issueId, repoUrl: f.repoUrl, commit: f.commit, branchName: f.branchName, + sourceKind: "managed_project", projectId: f.projectId, sourcePath: f.base, worktreePath: f.cwd, + gitBranchOwnershipVersion: 1, verifiedAt: new Date().toISOString(), restorePoint, contents }, + } }).where(eq(executionWorkspaces.id, f.workspaceId)); + await git(f.base, "worktree", "remove", "--force", f.cwd); await git(f.base, "branch", "-D", f.branchName); + if (state === "missing-base") await fs.rm(f.base, { recursive: true }); + if (state === "unapproved-nested") { + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow(); + expect(readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)).toBeNull(); + expect((await workspace(f.workspaceId)).status).toBe("archived"); + return; + } + if (state === "pruned") { + await git(f.base, "reflog", "expire", "--expire=now", "--all"); + await git(f.base, "gc", "--prune=now"); + await expect(git(f.base, "cat-file", "-e", `${f.commit}^{commit}`)).rejects.toThrow(); + } + if (["missing", "expired", "corrupt"].includes(state)) { + if (state === "missing") await storage.deleteObject(f.companyId, objectKey); + else { + const row = await workspace(f.workspaceId); + const recipe = row.metadata!.gitCacheRestoration as Record; + await db.update(executionWorkspaces).set({ metadata: { ...row.metadata, gitCacheRestoration: { ...recipe, + restorePoint: { ...restorePoint, ...(state === "expired" ? { expiredAt: new Date().toISOString() } : { sha256: "00".repeat(32) }) }, + } } }).where(eq(executionWorkspaces.id, f.workspaceId)); + } + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow(); + const row = await workspace(f.workspaceId); + expect(row).toMatchObject({ status: "archived", providerType: "git_worktree", cwd: f.cwd }); + expect(readRemoteWorkspaceAuthority(row.metadata)).toBeNull(); + expect(await fs.stat(f.cwd).catch(() => null)).toBeNull(); + return; + } + if (state === "retry") { + const put = storage.putFile.bind(storage); let fail = true; + storage.putFile = async input => { const result = await put(input); if (fail) { fail = false; throw new Error("upload reply lost"); } return result; }; + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("upload reply lost"); + } + await migrateRemoteWorkspace({ db, storage, ...f, apply: true }); + const authority = readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)!; + await fs.rm(f.base, { recursive: true, force: true }); await fs.rm(path.join(root, "nested-source"), { recursive: true, force: true }); await storage.deleteObject(f.companyId, objectKey); + const ref = await getWorkspaceSnapshot(db, { ...f, snapshotId: authority.snapshotId! }); + const archivePath = path.join(root, "retained-git.pcws"), targetDir = path.join(root, "restored-retained-git"); + await loadWorkspaceSnapshotArchive({ storage, reference: ref!, targetPath: archivePath }); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref!.manifestDigest }); + expect(await git(targetDir, "rev-parse", "HEAD")).toBe(f.commit); + if (state === "nested") { + expect(await fs.readFile(path.join(targetDir, ".paperclip-repositories", contents!.checkouts[0]!.name, "nested.txt"), "utf8")).toBe("nested source work\n"); + expect(JSON.parse(await fs.readFile(path.join(targetDir, ".paperclip-repositories", contents!.checkouts[0]!.name, ".claude/settings.local.json"), "utf8"))) + .toEqual({ permissions: { allow: ["Read"] } }); + } + expect(await git(targetDir, "show", ":saved.txt")).toBe("staged edit"); + expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("unstaged edit\n"); + }); + async function retainDirectory(f: Awaited>) { + const captured = await captureTaskDirectorySnapshot(f.cwd), id = randomUUID(), generation = 1; + const sealed = await sealRestorePoint(captured.payload, { companyId: f.companyId, workspaceId: f.workspaceId, restorePointId: id, generation }); + const objectKey = `${f.companyId}/git-restore-points/${f.workspaceId}/${generation}/task-directory/${id}.pcrp`; + await storage.putFile({ companyId: f.companyId, namespace: "git-restore-points", objectKey, + originalFilename: "retained.pcrp", contentType: "application/octet-stream", body: sealed.ciphertext }); + const ref: TaskDirectoryRestorePointRef = { schema: "paperclip.task-directory-restore-point.v1", id, generation, + objectKey, sha256: sealed.sha256, byteSize: sealed.ciphertext.length, key: sealed.key, + digest: captured.digest, files: captured.files, excluded: captured.excluded }; + await db.update(executionWorkspaces).set({ status: "archived", closedAt: new Date(), metadata: { + isolatedTaskDirectoryRetention: { schema: "paperclip.task-directory-retention.v1", generation, removed: true, + trash: path.join(root, "removed-trash"), restorePoint: ref }, + } }).where(eq(executionWorkspaces.id, f.workspaceId)); + await fs.rm(f.cwd, { recursive: true }); + return ref; + } + it("imports an archived directory from its encrypted restore point after the old folder is gone", async () => { + const f = await seed(), retained = await retainDirectory(f); + expect(await inventoryRemoteWorkspaceMigration(db, f)).toMatchObject({ eligible: true }); + await migrateRemoteWorkspace({ db, storage, ...f, apply: true }); + const saved = await workspace(f.workspaceId), authority = readRemoteWorkspaceAuthority(saved.metadata)!; + expect(saved.status).toBe("archived"); + expect(await fs.stat(f.cwd).catch(() => null)).toBeNull(); + await storage.deleteObject(f.companyId, retained.objectKey); + const ref = await getWorkspaceSnapshot(db, { ...f, snapshotId: authority.snapshotId! }); + const archivePath = path.join(root, "retained.pcws"), targetDir = path.join(root, "restored-retained"); + await loadWorkspaceSnapshotArchive({ storage, reference: ref!, targetPath: archivePath }); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref!.manifestDigest }); + expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("original saved work\n"); + }); + it.each(["missing", "expired", "corrupt"])("does not replace an unavailable retained directory with an empty workspace (%s)", async failure => { + const f = await seed(), retained = await retainDirectory(f); + if (failure === "missing") await storage.deleteObject(f.companyId, retained.objectKey); + else { + const row = await workspace(f.workspaceId); + const retention = row.metadata!.isolatedTaskDirectoryRetention as Record; + await db.update(executionWorkspaces).set({ metadata: { ...row.metadata, isolatedTaskDirectoryRetention: { + ...retention, restorePoint: { ...retained, ...(failure === "expired" ? { expiredAt: new Date().toISOString() } : { sha256: "00".repeat(32) }) }, + } } }).where(eq(executionWorkspaces.id, f.workspaceId)); + } + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow(); + const row = await workspace(f.workspaceId); + expect(row).toMatchObject({ status: "archived", providerType: "local_fs", cwd: f.cwd }); + expect(readRemoteWorkspaceAuthority(row.metadata)).toBeNull(); + expect(await fs.stat(f.cwd).catch(() => null)).toBeNull(); + }); + it("keeps newly restored local files when they appear during an archived import", async () => { + const f = await seed(); await retainDirectory(f); + const put = storage.putFile.bind(storage); + storage.putFile = async input => { + const result = await put(input); + await fs.mkdir(f.cwd); await fs.writeFile(path.join(f.cwd, "saved.txt"), "new local edits\n"); + return result; + }; + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("remote_workspace_migration_source_changed"); + expect(readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)).toBeNull(); + expect(await fs.readFile(path.join(f.cwd, "saved.txt"), "utf8")).toBe("new local edits\n"); + }); + it("preserves an archived workspace without reopening its task or losing its files", async () => { + const f = await seed(), closedAt = new Date("2026-10-20T12:00:00Z"); + await db.update(executionWorkspaces).set({ status: "archived", closedAt }).where(eq(executionWorkspaces.id, f.workspaceId)); + await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId)); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).resolves.toMatchObject({ applied: true }); + const saved = await workspace(f.workspaceId), authority = readRemoteWorkspaceAuthority(saved.metadata)!; + expect(saved.status).toBe("archived"); expect(saved.closedAt).toEqual(closedAt); + const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId)); + expect(issue).toMatchObject({ status: "done", executionWorkspaceId: f.workspaceId }); + const ref = await getWorkspaceSnapshot(db, { ...f, snapshotId: authority.snapshotId! }); + await fs.rm(f.cwd, { recursive: true }); + const archivePath = path.join(root, "archived.pcws"), targetDir = path.join(root, "restored-archived"); + await loadWorkspaceSnapshotArchive({ storage, reference: ref!, targetPath: archivePath }); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref!.manifestDigest }); + expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("original saved work\n"); + }); + it("imports a historical workspace without replacing the task's current workspace", async () => { + const f = await seed(), currentId = randomUUID(), currentCwd = path.join(root, "current"); + await fs.mkdir(currentCwd); await fs.writeFile(path.join(currentCwd, "saved.txt"), "newer current work\n"); + await db.insert(executionWorkspaces).values({ id: currentId, companyId: f.companyId, sourceIssueId: f.issueId, + mode: "isolated_workspace", strategyType: "project_primary", providerType: "local_fs", name: "Current workspace", cwd: currentCwd }); + await db.update(issues).set({ executionWorkspaceId: currentId }).where(eq(issues.id, f.issueId)); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).resolves.toMatchObject({ applied: true }); + const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId)); + expect(issue?.executionWorkspaceId).toBe(currentId); + expect(await fs.readFile(path.join(currentCwd, "saved.txt"), "utf8")).toBe("newer current work\n"); + expect(readRemoteWorkspaceAuthority((await workspace(currentId)).metadata)).toBeNull(); + expect(readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)?.snapshotId).toBeTruthy(); + }); + it("refuses publication when the task changes its workspace during import", async () => { + const f = await seed(), put = storage.putFile.bind(storage); + storage.putFile = async input => { + const result = await put(input); + await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.id, f.issueId)); + return result; + }; + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("remote_workspace_migration_source_changed"); + expect(readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)).toBeNull(); + expect(await fs.readFile(path.join(f.cwd, "saved.txt"), "utf8")).toBe("original saved work\n"); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("remote_workspace_migration_source_changed"); + }); it("defaults to dry run and preserves both database authority and local files", async () => { const f = await seed(); expect(await migrateRemoteWorkspace({ db, storage, ...f })).toMatchObject({ eligible: true, applied: false, alreadyMigrated: false }); expect((await workspace(f.workspaceId)).providerType).toBe("local_fs"); diff --git a/server/src/services/remote-workspace-migration.ts b/server/src/services/remote-workspace-migration.ts index b26dcfcf1b7..47dd4c7435f 100644 --- a/server/src/services/remote-workspace-migration.ts +++ b/server/src/services/remote-workspace-migration.ts @@ -14,7 +14,9 @@ import { withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifec import { captureRemoteWorkspaceDirectory, restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js"; import { beginWorkspaceSnapshot, loadWorkspaceSnapshotArchive, uploadWorkspaceSnapshot } from "./remote-workspace-snapshot-storage.js"; import { notifyRemoteWorkspaceSnapshots } from "./remote-workspace-snapshot-retention.js"; -import { removeTaskDirectoryTree } from "./isolated-task-directory-snapshot.js"; +import { materializeTaskDirectorySnapshot, removeTaskDirectoryTree } from "./isolated-task-directory-snapshot.js"; +import { loadTaskDirectoryRestorePoint, mayBeIsolatedTaskDirectoryWorkspace, readTaskDirectoryRetention, validateTaskDirectoryRestorePointRef } from "./isolated-task-directory-retention.js"; +import { inspectArchivedGitMigrationSource, materializeArchivedGitMigrationSource, type ArchivedGitMigrationSource } from "./remote-workspace-migration-git.js"; const SCHEMA = "paperclip.remote-workspace-migration.v1"; type Workspace = typeof executionWorkspaces.$inferSelect; @@ -32,6 +34,9 @@ function fingerprint(workspace: Workspace) { providerType: workspace.providerType, cwd: workspace.cwd, providerRef: workspace.providerRef, repoUrl: workspace.repoUrl, baseRef: workspace.baseRef, branchName: workspace.branchName, status: workspace.status, closedAt: workspace.closedAt?.toISOString(), metadata })).digest("hex"); } +const gitSourceProof = (source: ArchivedGitMigrationSource | null) => source ? { + commit: source.commit, sourcePath: source.sourcePath, cleanupOperationId: source.cleanupOperationId, +} : null; async function inspect(db: ReadDb, input: { companyId: string; workspaceId: string; operationId?: string }) { const [workspace] = await db.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.workspaceId))); @@ -40,21 +45,43 @@ async function inspect(db: ReadDb, input: { companyId: string; workspaceId: stri const marker = object(workspace.metadata?.remoteWorkspaceMigration); const allowedOperation = input.operationId ?? (marker.schema === SCHEMA && typeof marker.operationId === "string" ? marker.operationId : undefined); const reasons: string[] = []; - if (authority) return { workspace, reasons, authority, marker, source: authority.source }; + if (authority) return { workspace, reasons, authority, marker, source: authority.source, issueBinding: null, retainedDirectory: null, archivedGitSource: null }; if (!["local_fs", "git_worktree"].includes(workspace.providerType)) reasons.push("workspace_provider_unsupported"); - if (!["active", "idle", "in_review"].includes(workspace.status)) reasons.push("workspace_not_active"); + if (!["active", "idle", "in_review", "archived"].includes(workspace.status)) reasons.push("workspace_not_active"); if (!workspace.sourceIssueId) reasons.push("task_binding_required"); const nestedMigration = workspace.metadata?.nestedCloneMigrationPending; if (nestedMigration != null && (typeof nestedMigration !== "object" || Array.isArray(nestedMigration) || Object.keys(nestedMigration).length > 0)) reasons.push("nested_clone_migration_unresolved"); const cwd = workspace.cwd ?? workspace.providerRef; + let retainedDirectory = null as ReturnType; + let archivedGitSource: ArchivedGitMigrationSource | null = null; if (!cwd || !path.isAbsolute(cwd)) reasons.push("local_source_missing"); else { - const entry = await fs.lstat(cwd).catch(() => null); - if (!entry?.isDirectory() || entry.isSymbolicLink()) reasons.push("local_source_unsafe_or_missing"); + const entry = await fs.lstat(cwd).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (!entry && workspace.status === "archived" && mayBeIsolatedTaskDirectoryWorkspace(workspace)) { + const retained = readTaskDirectoryRetention(workspace); + if (retained?.removed === true && retained.restorePoint && retained.generation === retained.restorePoint.generation) { + if (retained.restorePoint.expiredAt) reasons.push("restore_point_expired"); + else { + try { validateTaskDirectoryRestorePointRef(retained.restorePoint, workspace); retainedDirectory = retained; } + catch { reasons.push("restore_point_invalid"); } + } + } + } + if (!entry && workspace.status === "archived" && !retainedDirectory) { + try { archivedGitSource = await inspectArchivedGitMigrationSource(db, workspace); } + catch (error) { reasons.push(error instanceof Error && error.message === "remote_workspace_migration_restore_point_expired" + ? "restore_point_expired" : "archived_git_source_unavailable"); } + } + if (!retainedDirectory && !archivedGitSource && (!entry?.isDirectory() || entry.isSymbolicLink())) reasons.push("local_source_unsafe_or_missing"); } const [issue] = workspace.sourceIssueId ? await db.select().from(issues).where(and(eq(issues.companyId, workspace.companyId), eq(issues.id, workspace.sourceIssueId))) : []; - if (workspace.sourceIssueId && (!issue || issue.executionWorkspaceId !== workspace.id)) reasons.push("task_binding_changed"); + // Historical records still own their saved files after the task selects a + // different workspace. Importing them must not reopen or rebind that task. + if (workspace.sourceIssueId && !issue) reasons.push("task_binding_changed"); const [project] = workspace.projectId ? await db.select({ policy: projects.executionWorkspacePolicy }).from(projects) .where(and(eq(projects.companyId, workspace.companyId), eq(projects.id, workspace.projectId))) : []; const issuePolicy = object(issue?.executionWorkspaceSettings), projectPolicy = object(project?.policy); @@ -109,10 +136,10 @@ async function inspect(db: ReadDb, input: { companyId: string; workspaceId: stri eq(executionWorkspaceRepositories.executionWorkspaceId, workspace.id))); if (repositories.some(repository => repository.state !== "ready")) reasons.push("repository_preparation_unresolved"); const source: RemoteWorkspaceSource = workspace.repoUrl ? { kind: "git", repoUrl: workspace.repoUrl, - ref: workspace.baseRef, branchName: workspace.branchName } : { kind: "empty" }; + ref: archivedGitSource?.commit ?? workspace.baseRef, branchName: workspace.branchName } : { kind: "empty" }; if (repositories.length) source.repositories = repositories.map(repository => ({ id: repository.id, relativePath: repository.relativePath, repoUrl: repository.repoUrl, ref: repository.requestedRef, ...(repository.pinnedCommit ? { pinnedCommit: repository.pinnedCommit } : {}) })); - return { workspace, reasons: [...new Set(reasons)], authority, marker, source }; + return { workspace, reasons: [...new Set(reasons)], authority, marker, source, issueBinding: issue?.executionWorkspaceId ?? null, retainedDirectory, archivedGitSource }; } export async function inventoryRemoteWorkspaceMigration(db: Db, input: { companyId: string; workspaceId: string }) { @@ -134,7 +161,7 @@ export async function migrateRemoteWorkspace(input: { if (!workspace) throw new Error("remote_workspace_migration_workspace_missing"); if (workspace.sourceIssueId) await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, workspace.companyId), eq(issues.id, workspace.sourceIssueId))).for("update"); const current = await inspect(tx as unknown as Db, input); - if (current.authority) return { workspace, authority: current.authority, operation: null, source: current.source, sourceFingerprint: "" }; + if (current.authority) return { workspace, authority: current.authority, operation: null, source: current.source, sourceFingerprint: "", issueBinding: current.issueBinding, retainedDirectory: null, archivedGitSource: null }; if (current.reasons.length) throw new Error(`remote_workspace_migration_blocked:${current.reasons.join(",")}`); const source = remoteWorkspaceSourceSchema.parse(input.source ?? current.source); const sourceFingerprint = fingerprint(workspace); @@ -144,16 +171,26 @@ export async function migrateRemoteWorkspace(input: { const [operation] = await tx.select().from(workspaceOperations).where(and(eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id), eq(workspaceOperations.id, marker.operationId), eq(workspaceOperations.phase, "remote_workspace_migration"))); if (!operation) throw new Error("remote_workspace_migration_operation_missing"); - return { workspace, authority: null, operation, source, sourceFingerprint }; + // Older operations only admitted the task's current workspace. Preserve + // that binding when resuming an import journaled before this field existed. + const issueBinding = Object.hasOwn(operation.metadata ?? {}, "sourceIssueBinding") + ? operation.metadata!.sourceIssueBinding : workspace.id; + if (issueBinding !== current.issueBinding) throw new Error("remote_workspace_migration_source_changed"); + if (Boolean(operation.metadata?.retainedDirectory) !== Boolean(current.retainedDirectory)) throw new Error("remote_workspace_migration_source_changed"); + if (canonical(operation.metadata?.archivedGitSource ?? null) !== canonical(gitSourceProof(current.archivedGitSource))) throw new Error("remote_workspace_migration_source_changed"); + return { workspace, authority: null, operation, source, sourceFingerprint, issueBinding, retainedDirectory: current.retainedDirectory, archivedGitSource: current.archivedGitSource }; } const [operation] = await tx.insert(workspaceOperations).values({ companyId: workspace.companyId, executionWorkspaceId: workspace.id, issueId: workspace.sourceIssueId, cwd: workspace.cwd ?? workspace.providerRef, phase: "remote_workspace_migration", status: "running", - metadata: { schema: SCHEMA, sourceFingerprint, source } }).returning(); + metadata: { schema: SCHEMA, sourceFingerprint, source, sourceIssueBinding: current.issueBinding, + ...(current.archivedGitSource ? { archivedGitSource: gitSourceProof(current.archivedGitSource) } : {}), + ...(current.retainedDirectory ? { retainedDirectory: { restorePointId: current.retainedDirectory.restorePoint.id, + generation: current.retainedDirectory.generation, digest: current.retainedDirectory.restorePoint.digest } } : {}) } }).returning(); await notifyRemoteWorkspaceSnapshots(tx, `workspace:${workspace.id}`); await tx.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspaceMigration: { schema: SCHEMA, operationId: operation!.id, state: "running", sourceFingerprint, source, } }, updatedAt: new Date() }).where(eq(executionWorkspaces.id, workspace.id)); - return { workspace, authority: null, operation: operation!, source, sourceFingerprint }; + return { workspace, authority: null, operation: operation!, source, sourceFingerprint, issueBinding: current.issueBinding, retainedDirectory: current.retainedDirectory, archivedGitSource: current.archivedGitSource }; }); if (reserved.authority) return { ...await inventoryRemoteWorkspaceMigration(input.db, input), applied: true, replayed: true }; const operation = reserved.operation!; @@ -161,15 +198,21 @@ export async function migrateRemoteWorkspace(input: { operationId: operation.id, generation: 1, epoch: 1 }); const scratch = await fs.mkdtemp(path.join(os.tmpdir(), "pc-workspace-migration-")); try { + const retained = reserved.retainedDirectory?.restorePoint; + const sourceRoot = retained ? await materializeTaskDirectorySnapshot({ + payload: await loadTaskDirectoryRestorePoint({ storage: input.storage, workspace: reserved.workspace, ref: retained }), + target: path.join(scratch, "retained-source"), expectedDigest: retained.digest, + }) : reserved.archivedGitSource ? await materializeArchivedGitMigrationSource({ db: input.db, storage: input.storage, + workspace: reserved.workspace, source: reserved.archivedGitSource, scratch }) : reserved.workspace.cwd ?? reserved.workspace.providerRef!; const archivePath = path.join(scratch, "workspace.pcws"); - const captured = await captureRemoteWorkspaceDirectory({ cwd: reserved.workspace.cwd ?? reserved.workspace.providerRef!, archivePath }); + const captured = await captureRemoteWorkspaceDirectory({ cwd: sourceRoot, archivePath }); const ref = await uploadWorkspaceSnapshot({ db: input.db, storage: input.storage, snapshotId: snapshot.id, companyId: input.companyId, workspaceId: input.workspaceId, archivePath, manifestDigest: captured.manifestDigest }); const restoredArchive = path.join(scratch, "verified.pcws"), restored = path.join(scratch, "restored"); await loadWorkspaceSnapshotArchive({ storage: input.storage, reference: ref, targetPath: restoredArchive }); await restoreRemoteWorkspaceArchive({ archivePath: restoredArchive, targetDir: restored, expectedManifestDigest: ref.manifestDigest }); const currentArchive = path.join(scratch, "current.pcws"); - const current = await captureRemoteWorkspaceDirectory({ cwd: reserved.workspace.cwd ?? reserved.workspace.providerRef!, archivePath: currentArchive }); + const current = await captureRemoteWorkspaceDirectory({ cwd: sourceRoot, archivePath: currentArchive }); if (current.manifestDigest !== captured.manifestDigest) throw new Error("remote_workspace_migration_source_changed"); await lock.execute(sql`select 1`); const publication = await input.db.transaction(async tx => { @@ -181,6 +224,9 @@ export async function migrateRemoteWorkspace(input: { || object(workspace.metadata?.remoteWorkspaceMigration).operationId !== operation.id) throw new Error("remote_workspace_migration_publication_changed"); if (workspace.sourceIssueId) await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, workspace.sourceIssueId)).for("update"); const validation = await inspect(tx as unknown as Db, { ...input, operationId: operation.id }); + if (validation.issueBinding !== reserved.issueBinding) throw new Error("remote_workspace_migration_source_changed"); + if (Boolean(validation.retainedDirectory) !== Boolean(reserved.retainedDirectory)) throw new Error("remote_workspace_migration_source_changed"); + if (canonical(gitSourceProof(validation.archivedGitSource)) !== canonical(gitSourceProof(reserved.archivedGitSource))) throw new Error("remote_workspace_migration_source_changed"); if (validation.reasons.length) throw new Error(`remote_workspace_migration_blocked:${validation.reasons.join(",")}`); const [uploaded] = await tx.select().from(executionWorkspaceSnapshots).where(and(eq(executionWorkspaceSnapshots.companyId, workspace.companyId), eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id), eq(executionWorkspaceSnapshots.id, ref.id))).for("update");