diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 4e976e80ba4..f21ffd3348d 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -3055,6 +3055,19 @@ Both enable switches default off. Rollback disables further expiration while leaving saved recipes and restoration support intact. Recovery archives, runner backups, non-Git workspaces, and immutable skill snapshots are unaffected. +### Isolated repository source re-home + +Workspaces created with a private per-issue clone under +`isolated-repository-sources/` can be moved onto the project's shared managed +clone, after which the private clone is deleted. Enable it with +`PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true`; it runs only when +`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and defaults off. Set +`PAPERCLIP_ISOLATED_SOURCE_REHOME_DRY_RUN=true` to log candidates without changing +anything. Each attempt logs `execution_workspace.isolated_source_rehome` with its +outcome, skip reason and reclaimed bytes. See +[recoverable disk cleanup](recoverable-disk-cleanup.md#isolated-repository-sources) +for the conditions and the verification it performs. + ### Terminal workspace compaction A finished task often keeps its worktree for a long time: the reaper archives a diff --git a/doc/recoverable-disk-cleanup.md b/doc/recoverable-disk-cleanup.md index e7ef3ba52f4..c31954027e6 100644 --- a/doc/recoverable-disk-cleanup.md +++ b/doc/recoverable-disk-cleanup.md @@ -313,3 +313,94 @@ processed in batches of 50; a full batch continues immediately. Rows have no foreign keys, so they outlive a deleted workspace, issue or company long enough to remove the stored object. Each run that changes anything logs its counts. + +## Isolated repository sources + +For a while, workspace preparation gave some isolated `git_worktree` workspaces their +own full clone of the project repository under +`instances//isolated-repository-sources///checkout`, with the +issue's worktree inside it. Workspace metadata marks these with `isolatedRemoteSource`. +New workspaces no longer get one; they are worktrees of the project's shared managed +clone. Each old clone still holds a full `.git` and an unused default-branch checkout. + +With `PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true` (and git cache expiration enabled), +the terminal workspace reaper moves each such open workspace onto the shared clone and +then deletes its private clone. It runs under the same source lock, workspace lifecycle +lock and Git lock files as git cache expiration, so no run, expiration, reopen or +restore can interleave. A workspace is moved only when: + +- no run, native finalization, environment lease or runtime service is still live for it, + and it is not waiting for a reopen to be consumed (the issue may be open); +- the project has a shared managed clone for the same `repoUrl`; +- its worktree sits at the standard `.paperclip/worktrees/` location, and the + branch is absent from the shared clone, points to an ancestor of the worktree's HEAD + there, and is not checked out in another of its worktrees; +- the shared clone's local config sets no filter drivers, includes, attribute files, worktree + config or command settings (`core.fsmonitor`, `core.hooksPath`, `core.sshCommand` and + similar), and it has no `info/attributes`, because the move checks out files there; +- its Git state is transferable: no stash or replacement refs, no in-progress merge, + rebase or other worktree-specific state, no hidden index flags, no shallow history, and + nothing a restore point refuses (submodules, filtered paths, special files, untracked + nested repositories, oversized contents). + +The move captures a restore point of the old worktree against commits both repositories +share, imports it into the shared clone, creates or fast-forwards the issue branch, adds +the new worktree at the same HEAD and applies the restore point (index, worktree changes, +untracked and ignored files, file modes). The `HEAD` reflog and `ORIG_HEAD` are copied. +What restore points leave out is moved by rename on the same filesystem: regenerable +directories such as `node_modules`, untracked `.env` and `.env.*` files, +`.claude/settings.local.json`, and each nested `.paperclip-repositories/*` checkout as a +whole. A standalone nested clone moves as is. A nested checkout that is a worktree of its own +shared clone moves too, and the move runs `git worktree repair` in that clone so its +registration follows the new path. Its `paperclip//` branch is renamed to the prefix of +the new path, because that prefix proves the checkout owns the branch (a rollback moves it +back, repairs it and renames the branch back). Items inside a moved nested checkout, such as +its `node_modules`, move with it. Any other +ignored file that a restore point does not save keeps the workspace in +place. + +The new worktree must then match the old one exactly (HEAD, index, worktree, ignored files, +modes and reflog), and every moved item must exist. Only then does one database update +point the workspace at the new path, drop `isolatedRemoteSource`, rewrite a saved +restoration recipe to the shared clone, and record `isolatedRepositoryRehome` +(`{ at, from, to, cleanedAt }`). Any failure before that update removes the new worktree, +moves items back, restores the branch, and leaves the old worktree untouched. Before it +changes the shared clone, the move records `isolatedRepositoryRehomePending` (target, branch, +previous branch tip and planned moves). If the server stops before the update, the next +attempt uses that record to move items back, remove the new worktree and restore the +branch, then starts again. A dry run reports such a record without undoing it. If an item +exists in both places (for example a process recreated `.env` in the old worktree), the +rollback stops with `rehome_rollback_conflict` and keeps the new worktree and the record, so +neither copy is lost; later attempts retry the rollback once the conflict is resolved. A new worktree is removed only when the move created it: a +target that appears from elsewhere is never deleted. + +After the update, the old worktree is removed and the clone is renamed to trash and +deleted when nothing else needs it: no other workspace or project workspace refers to it +(a workspace already expired with its worktree removed does not count), no runtime +service runs in it, it has no other worktree, its own checkout has no changes or files +other than regenerable ones, it has only branch, tag and remote-tracking refs, and every +commit its local refs and reflogs hold is either on a remote-tracking ref or present in the +shared clone. A clone with more than 5,000 refs, reflog entries or such commits is kept +(`clone_history_budget`). A clone another workspace still uses is kept (`cloneRetained`); that +workspace's own move removes it later. If the server stops after the update, a later sweep +finishes the removal and sets `cleanedAt`, also after the workspace has been archived. + +Each attempt logs `event: "execution_workspace.isolated_source_rehome"` with the outcome, +skip reason, candidate bytes and reclaimed bytes. A workspace that is skipped, or whose +clone is kept for a reason that can change, is retried an hour later through the reaper's +existing deadlines; nothing runs while there is no candidate. +`PAPERCLIP_ISOLATED_SOURCE_REHOME_DRY_RUN=true` reports candidates without changing +anything. + +Re-homed workspaces get a new path. An agent that resumes an older session may still +remember the old path; the run's working directory comes from the updated workspace. + +Reopening or restoring an expired workspace whose recipe came from a private clone no +longer recreates that clone when the project has a shared managed clone for the same +repository, its branch there is absent or at the saved commit, and the shared clone has +every commit the saved restore point bundle needs as a base. A recipe with worktree-backed +nested checkouts on branches keeps the old behavior, because their branch names are bound to +the old path. The workspace is +rebuilt as a worktree of the shared clone at the standard location, with the saved commit +and restore point imported and applied as before. Without a shared clone, the old +behavior applies. diff --git a/server/src/__tests__/isolated-repository-rehome.test.ts b/server/src/__tests__/isolated-repository-rehome.test.ts new file mode 100644 index 00000000000..6631ebfc0c2 --- /dev/null +++ b/server/src/__tests__/isolated-repository-rehome.test.ts @@ -0,0 +1,475 @@ +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { execFileSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { pathToFileURL } from "node:url"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { eq, sql } from "drizzle-orm"; +import { agents, companies, createDb, executionWorkspaceRestorePoints, executionWorkspaces, heartbeatRuns, issues, projects } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveManagedProjectWorkspaceDir } from "../home-paths.js"; +import { materializeManagedProjectWorkspace, ensureManagedRepositoriesIgnored } from "../services/managed-repository-checkout.js"; +import { isolatedRemoteSourcePath, materializeIsolatedRemoteWorkspaceSource } from "../services/isolated-remote-workspace-source.js"; +import { realizeExecutionWorkspace } from "../services/workspace-runtime.js"; +import { executionWorkspaceService } from "../services/execution-workspaces.js"; +import { expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js"; +import { currentRestorePointSnapshot } from "../services/git-restore-point.js"; +import { rehomeIsolatedRepositoryWorkspace } from "../services/isolated-repository-rehome.js"; +import { addNestedRepositoryWorktree, nestedRepositoryBaseCandidates, nestedWorktreeBranchPrefix, readLinkedNestedCheckout } from "../services/nested-repository-worktrees.js"; +import { getStorageService } from "../storage/index.js"; + +const gitEnv = { ...process.env, GIT_AUTHOR_NAME: "Fixture", GIT_AUTHOR_EMAIL: "fixture@example.test", + GIT_COMMITTER_NAME: "Fixture", GIT_COMMITTER_EMAIL: "fixture@example.test", GIT_TERMINAL_PROMPT: "0" }; +const git = (cwd: string, ...args: string[]) => execFileSync("git", args, { cwd, env: gitEnv, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim(); +const exists = (target: string) => fs.lstat(target).then(() => true, () => false); +const DAY_MS = 24 * 60 * 60 * 1000; + +const support = await getEmbeddedPostgresTestSupport(); +const describeEmbeddedPostgres = support.supported ? describe : describe.skip; +describeEmbeddedPostgres("isolated repository source re-home", () => { + let db: ReturnType; + let database: Awaited>; + let root: string; + const previousHome = process.env.PAPERCLIP_HOME; + const previousInstance = process.env.PAPERCLIP_INSTANCE_ID; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("isolated-repository-rehome-"); + db = createDb(database.connectionString); + root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-isolated-rehome-")); + process.env.PAPERCLIP_HOME = path.join(root, "home"); + process.env.PAPERCLIP_INSTANCE_ID = "default"; + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "41".repeat(32)); + }); + afterEach(async () => { + await db.delete(executionWorkspaceRestorePoints); + await db.execute(sql`TRUNCATE TABLE companies CASCADE`); + }); + afterAll(async () => { + vi.unstubAllEnvs(); + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome; + if (previousInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; else process.env.PAPERCLIP_INSTANCE_ID = previousInstance; + await db.$client.end({ timeout: 1 }); await database.cleanup(); await fs.rm(root, { recursive: true, force: true }); + }); + + async function fixture(options: { sharedBase?: boolean; issueStatus?: "in_progress" | "done" } = {}) { + const companyId = randomUUID(), issueId = randomUUID(), workspaceId = randomUUID(), projectId = randomUUID(); + const location = path.join(root, companyId), author = path.join(location, "author"), remote = path.join(location, "remote.git"); + await fs.mkdir(author, { recursive: true }); + git(author, "init", "--initial-branch=main"); + await fs.writeFile(path.join(author, "marker.txt"), "commit A\n"); + await fs.writeFile(path.join(author, "remove-me.txt"), "tracked\n"); + await fs.writeFile(path.join(author, ".gitignore"), "node_modules/\n.env\n.claude/settings.local.json\n*.tsbuildinfo\n"); + git(author, "add", "."); git(author, "commit", "-m", "A"); + git(location, "clone", "--bare", author, remote); + const done = options.issueStatus === "done"; + await db.insert(companies).values({ id: companyId, name: "Re-home fixture", issuePrefix: `RH${companyId.slice(0, 6)}` }); + await db.insert(projects).values({ id: projectId, companyId, name: "Fixture project" }); + await db.insert(issues).values({ id: issueId, companyId, projectId, title: "Open task", status: done ? "done" : "in_progress", + ...(done ? { completedAt: new Date(Date.now() - 4 * DAY_MS) } : {}) }); + const repoUrl = pathToFileURL(remote).href; + const sharedPath = resolveManagedProjectWorkspaceDir({ companyId, projectId, repoName: "remote" }); + if (options.sharedBase !== false) await materializeManagedProjectWorkspace(sharedPath, { repoUrl }); + const clone = await materializeIsolatedRemoteWorkspaceSource({ companyId, issueId, repoUrl, resolveGitAuth: async () => null }); + const realized = await realizeExecutionWorkspace({ base: { baseCwd: clone, source: "task_session", projectId: null, workspaceId: null, repoUrl, repoRef: "origin/main" }, + issue: { id: issueId, identifier: "RH-1", title: "Fixture" }, agent: { id: null, name: "Fixture", companyId }, + config: { workspaceStrategy: { type: "git_worktree", branchTemplate: "fixture-task", baseRef: "origin/main" } }, skipHostProvision: true }); + await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, projectId, sourceIssueId: issueId, mode: "isolated_workspace", strategyType: "git_worktree", + name: "Fixture", status: "active", providerType: "local_fs", cwd: realized.cwd, providerRef: realized.cwd, repoUrl, baseRef: "origin/main", + branchName: realized.branchName, metadata: { createdByRuntime: realized.branchCreatedByRuntime, gitBranchOwnershipVersion: 1, baseRefSha: realized.baseRefSha, + isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId, issueId, repoUrl } } }); + await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId)); + const row = () => db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).then(rows => rows[0]!); + const target = path.join(await fs.realpath(root), path.relative(root, sharedPath), ".paperclip", "worktrees", "fixture-task"); + return { companyId, issueId, workspaceId, projectId, cwd: realized.cwd, clone, sharedPath, target, repoUrl, remote, author, row }; + } + + async function makeWork(cwd: string) { + await fs.writeFile(path.join(cwd, "marker.txt"), "commit B (unpushed)\n"); + git(cwd, "commit", "-am", "B unpushed"); + const unpushed = git(cwd, "rev-parse", "HEAD"); + await fs.writeFile(path.join(cwd, "marker.txt"), "staged edit\n"); + git(cwd, "add", "marker.txt"); + await fs.writeFile(path.join(cwd, "marker.txt"), "unstaged edit\n"); + await fs.rm(path.join(cwd, "remove-me.txt")); + await fs.mkdir(path.join(cwd, "notes")); + await fs.writeFile(path.join(cwd, "notes", "todo.md"), "untracked work\n"); + await fs.chmod(path.join(cwd, "notes", "todo.md"), 0o600); + await fs.writeFile(path.join(cwd, "build.tsbuildinfo"), "ignored output\n"); + await fs.mkdir(path.join(cwd, "node_modules", "pkg"), { recursive: true }); + await fs.writeFile(path.join(cwd, "node_modules", "pkg", "index.js"), "module.exports = 1;\n"); + await fs.writeFile(path.join(cwd, ".env"), "LOCAL_SETTING=1\n"); + await fs.mkdir(path.join(cwd, ".claude")); + await fs.writeFile(path.join(cwd, ".claude", "settings.local.json"), "{\"local\":true}\n"); + return unpushed; + } + + async function addLinkedNested(f: Awaited>) { + const location = path.dirname(f.remote), author = path.join(location, "docs-author"), remote = path.join(location, "docs.git"); + await fs.mkdir(author, { recursive: true }); + git(author, "init", "--initial-branch=main"); + await fs.writeFile(path.join(author, "guide.md"), "guide\n"); git(author, "add", "."); git(author, "commit", "-m", "docs"); + git(location, "clone", "--bare", author, remote); + const repoUrl = pathToFileURL(remote).href; + const base = (await materializeManagedProjectWorkspace(nestedRepositoryBaseCandidates({ companyId: f.companyId, projectId: f.projectId, repoUrl })[0]!, { repoUrl })).cwd; + await ensureManagedRepositoriesIgnored(f.cwd); + const checkout = path.join(f.cwd, ".paperclip-repositories", "docs-linked"); + await fs.mkdir(path.dirname(checkout), { recursive: true }); + const { branchName } = await addNestedRepositoryWorktree({ base, cwd: checkout, repoUrl, repoRef: null }); + await fs.writeFile(path.join(checkout, "guide.md"), "edited in the nested worktree\n"); + return { base, checkout, branchName, realBase: await fs.realpath(base) }; + } + + const worktreePaths = (base: string) => git(base, "worktree", "list", "--porcelain").split("\n").filter(line => line.startsWith("worktree ")).map(line => line.slice(9)); + + const gitView = (cwd: string) => ({ branch: git(cwd, "rev-parse", "--abbrev-ref", "HEAD"), head: git(cwd, "rev-parse", "HEAD"), + status: git(cwd, "status", "--porcelain=v1", "--untracked-files=all"), reflog: git(cwd, "reflog", "show", "--format=%H", "HEAD") }); + + async function expectUntouched(f: Awaited>, before: Awaited>, cwd = f.cwd) { + const current = await f.row(); + expect(current.cwd).toBe(f.cwd); + expect(current.metadata?.isolatedRemoteSource).toBeDefined(); + expect(current.metadata?.isolatedRepositoryRehome).toBeUndefined(); + expect(await currentRestorePointSnapshot(cwd)).toEqual(before); + expect(await exists(f.target)).toBe(false); + if (await exists(f.sharedPath)) expect(() => git(f.sharedPath, "rev-parse", "--verify", "-q", "refs/heads/fixture-task")).toThrow(); + } + + it("re-homes an open workspace onto the shared project clone with its exact state", async () => { + const f = await fixture(); + const unpushed = await makeWork(f.cwd); + await ensureManagedRepositoriesIgnored(f.cwd); + await fs.appendFile(path.join(f.cwd, ".gitignore"), "/.paperclip-repositories/\n"); + const nested = path.join(f.cwd, ".paperclip-repositories", "docs"); + await fs.mkdir(nested, { recursive: true }); + git(nested, "init", "--initial-branch=main"); + await fs.writeFile(path.join(nested, "readme.md"), "nested\n"); + await fs.writeFile(path.join(nested, ".gitignore"), "node_modules/\n"); git(nested, "add", "."); git(nested, "commit", "-m", "nested"); + await fs.mkdir(path.join(nested, "node_modules", "dep"), { recursive: true }); + await fs.writeFile(path.join(nested, "node_modules", "dep", "index.js"), "nested dependency\n"); + const before = await currentRestorePointSnapshot(f.cwd); + const view = gitView(f.cwd); + + const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row()); + expect(result).toMatchObject({ outcome: "rehomed", rehomed: true, cloneRemoved: true, reason: null, cloneKeptReason: null, to: f.target }); + expect(result.reclaimedBytes).toBeGreaterThan(0); + + expect(await currentRestorePointSnapshot(f.target)).toEqual(before); + expect(gitView(f.target)).toEqual(view); + expect(view.head).toBe(unpushed); + expect(git(f.target, "rev-parse", "--path-format=absolute", "--git-common-dir")).toBe(path.join(await fs.realpath(f.sharedPath), ".git")); + expect(git(f.target, "show", ":marker.txt")).toBe("staged edit"); + expect(await fs.readFile(path.join(f.target, "marker.txt"), "utf8")).toBe("unstaged edit\n"); + expect(await exists(path.join(f.target, "remove-me.txt"))).toBe(false); + expect((await fs.stat(path.join(f.target, "notes", "todo.md"))).mode & 0o777).toBe(0o600); + expect(await fs.readFile(path.join(f.target, "build.tsbuildinfo"), "utf8")).toBe("ignored output\n"); + expect(await fs.readFile(path.join(f.target, "node_modules", "pkg", "index.js"), "utf8")).toBe("module.exports = 1;\n"); + expect(await fs.readFile(path.join(f.target, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n"); + expect(await fs.readFile(path.join(f.target, ".claude", "settings.local.json"), "utf8")).toBe("{\"local\":true}\n"); + expect(git(path.join(f.target, ".paperclip-repositories", "docs"), "log", "-1", "--format=%s")).toBe("nested"); + expect(await fs.readFile(path.join(f.target, ".paperclip-repositories", "docs", "node_modules", "dep", "index.js"), "utf8")).toBe("nested dependency\n"); + + const row = await f.row(); + expect(row).toMatchObject({ cwd: f.target, providerRef: f.target, branchName: "fixture-task" }); + expect(row.metadata?.isolatedRemoteSource).toBeUndefined(); + expect(row.metadata?.isolatedRepositoryRehome).toMatchObject({ from: f.cwd, to: f.target, cleanedAt: expect.any(String) }); + expect(await exists(f.cwd)).toBe(false); + expect(await exists(f.clone)).toBe(false); + expect(await exists(isolatedRemoteSourcePath(f.companyId, f.issueId))).toBe(false); + expect(await rehomeIsolatedRepositoryWorkspace(db, row)).toMatchObject({ rehomed: false, reason: "not_candidate" }); + }); + + it("moves worktree-backed nested checkouts and repairs their registration in their shared clone", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const nested = await addLinkedNested(f); + const oldCheckout = await fs.realpath(nested.checkout); + const previousPrefix = await nestedWorktreeBranchPrefix(oldCheckout); + const before = await currentRestorePointSnapshot(f.cwd); + const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row()); + expect(result).toMatchObject({ rehomed: true, cloneRemoved: true, reason: null }); + expect(await currentRestorePointSnapshot(f.target)).toEqual(before); + const moved = path.join(f.target, ".paperclip-repositories", "docs-linked"); + expect(await readLinkedNestedCheckout(moved)).toMatchObject({ base: nested.realBase, checkout: moved }); + expect(worktreePaths(nested.base)).toContain(moved); + expect(worktreePaths(nested.base)).not.toContain(oldCheckout); + expect(nested.branchName!.startsWith(previousPrefix)).toBe(true); + const renamed = `${await nestedWorktreeBranchPrefix(moved)}${nested.branchName!.slice(previousPrefix.length)}`; + expect(renamed).not.toBe(nested.branchName); + expect(git(moved, "symbolic-ref", "--short", "HEAD")).toBe(renamed); + expect(() => git(nested.base, "rev-parse", "--verify", "-q", `refs/heads/${nested.branchName}`)).toThrow(); + expect(git(moved, "status", "--porcelain")).toBe("M guide.md"); + expect(git(nested.base, "worktree", "list", "--porcelain")).not.toContain("prunable"); + }); + + it("restores the registration of worktree-backed nested checkouts when a move rolls back", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const nested = await addLinkedNested(f); + const oldCheckout = await fs.realpath(nested.checkout); + const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { beforeVerify: async ({ from, to }) => { + expect(worktreePaths(nested.base)).toContain(path.join(to, ".paperclip-repositories", "docs-linked")); + await fs.writeFile(path.join(from, "late.txt"), "written by a concurrent run\n"); + } } }); + expect(result).toMatchObject({ rehomed: false, reason: "snapshot_mismatch" }); + expect(await readLinkedNestedCheckout(nested.checkout)).toMatchObject({ base: nested.realBase, checkout: oldCheckout }); + expect(worktreePaths(nested.base)).toContain(oldCheckout); + expect(git(nested.checkout, "symbolic-ref", "--short", "HEAD")).toBe(nested.branchName); + expect(git(nested.checkout, "status", "--porcelain")).toBe("M guide.md"); + expect(await exists(f.target)).toBe(false); + }); + + it("keeps a workspace with a live run", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const agentId = randomUUID(); + await db.insert(agents).values({ id: agentId, companyId: f.companyId, name: "Fixture", role: "engineer", adapterType: "process" }); + await db.insert(heartbeatRuns).values({ id: randomUUID(), companyId: f.companyId, agentId, issueId: f.issueId, status: "running" }); + const before = await currentRestorePointSnapshot(f.cwd); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: false, reason: "live_or_unresolved_consumer" }); + await expectUntouched(f, before); + }); + + it("keeps a workspace whose branch already exists in the shared clone at an unrelated commit", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const foreign = git(f.sharedPath, "commit-tree", "HEAD^{tree}", "-p", "HEAD", "-m", "foreign"); + git(f.sharedPath, "update-ref", "refs/heads/fixture-task", foreign); + const before = await currentRestorePointSnapshot(f.cwd); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: false, reason: "base_branch_diverged" }); + expect(git(f.sharedPath, "rev-parse", "refs/heads/fixture-task")).toBe(foreign); + git(f.sharedPath, "update-ref", "-d", "refs/heads/fixture-task"); + await expectUntouched(f, before); + }); + + it("keeps a workspace when the project has no shared clone", async () => { + const f = await fixture({ sharedBase: false }); + await makeWork(f.cwd); + const before = await currentRestorePointSnapshot(f.cwd); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: false, reason: "no_shared_base" }); + await expectUntouched(f, before); + }); + + it("keeps a workspace with state a restore point cannot transfer", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const nested = path.join(f.cwd, "vendor-checkout"); + await fs.mkdir(nested); + git(nested, "init", "--initial-branch=main"); + const view = gitView(f.cwd); + const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row()); + expect(result).toMatchObject({ rehomed: false, reason: "untracked_nested_repository" }); + expect(gitView(f.cwd)).toEqual(view); + expect(await exists(path.join(f.cwd, "node_modules", "pkg", "index.js"))).toBe(true); + await fs.rm(nested, { recursive: true }); + await expectUntouched(f, await currentRestorePointSnapshot(f.cwd)); + }); + + it("removes the new worktree and keeps the old one when the workspace changes during the move", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const before = await currentRestorePointSnapshot(f.cwd); + const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { beforeVerify: async ({ from, to }) => { + expect(await exists(path.join(to, "node_modules"))).toBe(true); + await fs.writeFile(path.join(from, "late.txt"), "written by a concurrent run\n"); + } } }); + expect(result).toMatchObject({ rehomed: false, reason: "snapshot_mismatch" }); + await fs.rm(path.join(f.cwd, "late.txt")); + await expectUntouched(f, before); + expect(await fs.readFile(path.join(f.cwd, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n"); + expect(await fs.readFile(path.join(f.cwd, "node_modules", "pkg", "index.js"), "utf8")).toBe("module.exports = 1;\n"); + expect(git(f.sharedPath, "worktree", "list", "--porcelain").split("\n").filter(line => line.startsWith("worktree "))).toHaveLength(1); + }); + + it("undoes a move interrupted before the database update and then completes it", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const before = await currentRestorePointSnapshot(f.cwd); + const from = await fs.realpath(f.cwd); + const head = git(f.cwd, "rev-parse", "HEAD"); + git(f.sharedPath, "fetch", "-q", f.clone, "refs/heads/fixture-task:refs/heads/fixture-task"); + git(f.sharedPath, "worktree", "add", "-q", f.target, "fixture-task"); + await fs.rename(path.join(f.cwd, "node_modules"), path.join(f.target, "node_modules")); + await fs.rename(path.join(f.cwd, ".env"), path.join(f.target, ".env")); + const pending = { base: f.sharedPath, from, target: f.target, branch: "fixture-task", head, previousTip: null, moves: [".env", "node_modules"] }; + const current = await f.row(); + await db.update(executionWorkspaces).set({ metadata: { ...current.metadata, isolatedRepositoryRehomePending: pending } }).where(eq(executionWorkspaces.id, f.workspaceId)); + + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { dryRun: true })) + .toMatchObject({ outcome: "dry_run", rehomed: false, reason: "interrupted_rehome_pending" }); + expect(await exists(path.join(f.target, "node_modules", "pkg", "index.js"))).toBe(true); + expect(await exists(path.join(f.cwd, ".env"))).toBe(false); + expect(git(f.sharedPath, "rev-parse", "refs/heads/fixture-task")).toBe(head); + expect((await f.row()).metadata?.isolatedRepositoryRehomePending).toEqual(pending); + + const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row()); + expect(result).toMatchObject({ rehomed: true, cloneRemoved: true, reason: null }); + expect(await currentRestorePointSnapshot(f.target)).toEqual(before); + expect(await fs.readFile(path.join(f.target, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n"); + expect(await fs.readFile(path.join(f.target, "node_modules", "pkg", "index.js"), "utf8")).toBe("module.exports = 1;\n"); + expect((await f.row()).metadata?.isolatedRepositoryRehomePending).toBeUndefined(); + }); + + it("never deletes a target worktree another workspace created during the move", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const before = await currentRestorePointSnapshot(f.cwd); + const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { beforeWorktreeAdd: async ({ to }) => { + await fs.mkdir(to, { recursive: true }); + await fs.writeFile(path.join(to, "other.txt"), "another workspace\n"); + } } }); + expect(result).toMatchObject({ rehomed: false, reason: "target_worktree_exists" }); + expect(await fs.readFile(path.join(f.target, "other.txt"), "utf8")).toBe("another workspace\n"); + await fs.rm(f.target, { recursive: true }); + await expectUntouched(f, before); + expect((await f.row()).metadata?.isolatedRepositoryRehomePending).toBeUndefined(); + }); + + it("keeps both copies when a file reappears in the old worktree before verification", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { beforeVerify: async ({ from }) => { + await fs.writeFile(path.join(from, ".env"), "RECREATED=1\n"); + } } }); + expect(result).toMatchObject({ rehomed: false, reason: "rehome_rollback_conflict" }); + expect(await fs.readFile(path.join(f.target, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n"); + expect(await fs.readFile(path.join(f.cwd, ".env"), "utf8")).toBe("RECREATED=1\n"); + expect((await f.row()).metadata?.isolatedRepositoryRehomePending).toMatchObject({ target: f.target }); + expect((await f.row()).cwd).toBe(f.cwd); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: false, reason: "rehome_rollback_conflict" }); + expect(await fs.readFile(path.join(f.target, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n"); + await fs.rm(path.join(f.cwd, ".env")); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: true, reason: null }); + expect(await fs.readFile(path.join(f.target, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n"); + }); + + it("refuses to check out from a shared clone whose config runs filters", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const marker = path.join(path.dirname(f.remote), "smudge-ran"); + git(f.sharedPath, "config", "filter.evil.smudge", `sh -c 'touch ${marker}; cat'`); + git(f.sharedPath, "config", "filter.evil.required", "true"); + const before = await currentRestorePointSnapshot(f.cwd); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: false, reason: "unsupported_base_git_config" }); + expect(await exists(marker)).toBe(false); + await expectUntouched(f, before); + }); + + it("keeps a clone with more refs than the cleanup budget", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { afterDatabaseUpdate: async () => { throw new Error("simulated crash"); } } }); + const head = git(f.clone, "rev-parse", "HEAD"); + execFileSync("git", ["-C", f.clone, "update-ref", "--stdin"], { input: Array.from({ length: 5_001 }, (_, index) => `create refs/tags/extra-${index} ${head}`).join("\n") + "\n" }); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "clone_history_budget" }); + expect(await exists(f.clone)).toBe(true); + expect((await f.row()).metadata?.isolatedRepositoryRehome).toMatchObject({ cleanedAt: null }); + execFileSync("git", ["-C", f.clone, "update-ref", "--stdin"], { input: Array.from({ length: 5_001 }, (_, index) => `delete refs/tags/extra-${index}`).join("\n") + "\n" }); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ outcome: "cleaned", cloneRemoved: true }); + }); + + it("keeps the clone while another workspace still references it", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await db.insert(executionWorkspaces).values({ companyId: f.companyId, projectId: f.projectId, sourceIssueId: f.issueId, mode: "isolated_workspace", + strategyType: "git_worktree", name: "Sibling", status: "active", providerType: "local_fs", cwd: path.join(f.clone, "sibling"), repoUrl: f.repoUrl, + metadata: { isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId: f.companyId, issueId: f.issueId, repoUrl: f.repoUrl } } }); + const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row()); + expect(result).toMatchObject({ rehomed: true, cloneRemoved: false, cloneKeptReason: "base_has_consumers" }); + expect(await exists(f.clone)).toBe(true); + expect(await exists(f.cwd)).toBe(false); + expect((await f.row()).metadata?.isolatedRepositoryRehome).toMatchObject({ cloneRetained: "base_has_consumers", cleanedAt: expect.any(String) }); + }); + + it("finishes removing the clone on a later sweep after a crash following the database update", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const before = await currentRestorePointSnapshot(f.cwd); + const crashed = await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { afterDatabaseUpdate: async () => { throw new Error("simulated crash"); } } }); + expect(crashed).toMatchObject({ rehomed: true, cloneRemoved: false }); + expect(await f.row()).toMatchObject({ cwd: f.target, metadata: expect.objectContaining({ isolatedRepositoryRehome: expect.objectContaining({ cleanedAt: null }) }) }); + expect(await exists(f.clone)).toBe(true); + expect(await currentRestorePointSnapshot(f.target)).toEqual(before); + + await db.update(executionWorkspaces).set({ status: "archived", closedAt: new Date() }).where(eq(executionWorkspaces.id, f.workspaceId)); + const service = executionWorkspaceService(db, { gitCacheExpirationEnabled: true, isolatedSourceRehomeEnabled: true }); + await service.sweepTerminalWorkspaces(); + expect(await exists(f.clone)).toBe(false); + expect(await exists(f.cwd)).toBe(false); + expect((await f.row()).metadata?.isolatedRepositoryRehome).toMatchObject({ cleanedAt: expect.any(String) }); + expect(await currentRestorePointSnapshot(f.target)).toEqual(before); + expect(await service.nextTerminalWorkspaceSweepAt()).toBeNull(); + }); + + it("is disabled by default and changes nothing in dry run", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const before = await currentRestorePointSnapshot(f.cwd); + await executionWorkspaceService(db, { gitCacheExpirationEnabled: true }).sweepTerminalWorkspaces(); + await executionWorkspaceService(db, { isolatedSourceRehomeEnabled: true }).sweepTerminalWorkspaces(); + await expectUntouched(f, before); + const dryRun = executionWorkspaceService(db, { gitCacheExpirationEnabled: true, isolatedSourceRehomeEnabled: true, isolatedSourceRehomeDryRun: true }); + await dryRun.sweepTerminalWorkspaces(); + await expectUntouched(f, before); + expect(await exists(f.clone)).toBe(true); + expect(await dryRun.nextTerminalWorkspaceSweepAt()).toBeGreaterThan(Date.now()); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { dryRun: true })).toMatchObject({ outcome: "dry_run", rehomed: false, to: f.target }); + await expectUntouched(f, before); + }); + + it("restores an expired isolated workspace into the shared project clone without recreating its clone", async () => { + const f = await fixture({ issueStatus: "done" }); + const unpushed = await makeWork(f.cwd); + await fs.rm(path.join(f.cwd, "node_modules"), { recursive: true }); + const expired = await expireReconstructableGitWorkspace(db, await f.row(), { dryRun: false, expireBase: true, now: new Date(), + restorePoint: { storage: getStorageService(), notBefore: new Date(Date.now() - DAY_MS), keyDurable: true } }); + expect(expired).toMatchObject({ expired: true, restorePoint: true }); + await fs.rm(f.clone, { recursive: true, force: true }); + + const reopened = await executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId: f.workspaceId, + issue: { id: f.issueId, companyId: f.companyId, projectId: f.projectId }, actor: { actorType: "user", agentId: null }, + authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } }); + expect(reopened).toMatchObject({ ok: true, reopened: true }); + expect(await exists(f.clone)).toBe(false); + const row = await f.row(); + expect(row).toMatchObject({ status: "active", cwd: f.target, providerRef: f.target }); + expect(row.metadata?.isolatedRemoteSource).toBeUndefined(); + expect(row.metadata?.gitCacheRestoration).toMatchObject({ sourceKind: "managed_project", sourcePath: f.sharedPath, worktreePath: f.target, + restorePoint: { appliedAt: expect.any(String) } }); + expect(git(f.target, "rev-parse", "HEAD")).toBe(unpushed); + expect(git(f.target, "rev-parse", "--path-format=absolute", "--git-common-dir")).toBe(path.join(await fs.realpath(f.sharedPath), ".git")); + expect(git(f.target, "show", ":marker.txt")).toBe("staged edit"); + expect(await fs.readFile(path.join(f.target, "marker.txt"), "utf8")).toBe("unstaged edit\n"); + expect(await fs.readFile(path.join(f.target, "notes", "todo.md"), "utf8")).toBe("untracked work\n"); + expect(await fs.readFile(path.join(f.target, "build.tsbuildinfo"), "utf8")).toBe("ignored output\n"); + }); + + it("restores through the private clone when the shared clone lacks the restore point's base commits", async () => { + const f = await fixture({ issueStatus: "done" }); + await fs.writeFile(path.join(f.author, "marker.txt"), "commit C\n"); + git(f.author, "commit", "-qam", "C"); + git(f.author, "push", "-q", f.remote, "main"); + const upstream = git(f.author, "rev-parse", "HEAD"); + git(f.cwd, "fetch", "-q", "origin"); + git(f.cwd, "reset", "-q", "--hard", "origin/main"); + await fs.writeFile(path.join(f.cwd, "marker.txt"), "commit D (unpushed)\n"); + git(f.cwd, "commit", "-qam", "D unpushed"); + const unpushed = git(f.cwd, "rev-parse", "HEAD"); + await fs.writeFile(path.join(f.cwd, "notes.txt"), "untracked work\n"); + const expired = await expireReconstructableGitWorkspace(db, await f.row(), { dryRun: false, expireBase: true, now: new Date(), + restorePoint: { storage: getStorageService(), notBefore: new Date(Date.now() - DAY_MS), keyDurable: true } }); + expect(expired).toMatchObject({ expired: true, restorePoint: true }); + await fs.rm(f.clone, { recursive: true, force: true }); + expect(() => git(f.sharedPath, "cat-file", "-e", `${upstream}^{commit}`)).toThrow(); + + const reopened = await executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId: f.workspaceId, + issue: { id: f.issueId, companyId: f.companyId, projectId: f.projectId }, actor: { actorType: "user", agentId: null }, + authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } }); + expect(reopened).toMatchObject({ ok: true, reopened: true }); + const row = await f.row(); + expect(row.cwd).toBe(f.cwd); + expect(row.metadata?.isolatedRemoteSource).toBeDefined(); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed); + expect(await fs.readFile(path.join(f.cwd, "notes.txt"), "utf8")).toBe("untracked work\n"); + }); +}); diff --git a/server/src/config.ts b/server/src/config.ts index d6bb2f6cbab..d9acf20c592 100644 --- a/server/src/config.ts +++ b/server/src/config.ts @@ -82,6 +82,8 @@ export interface Config { gitCacheExpirationEnabled: boolean; gitBaseExpirationEnabled: boolean; gitCacheExpirationDryRun: boolean; + isolatedSourceRehomeEnabled: boolean; + isolatedSourceRehomeDryRun: boolean; workspaceReaperCooldownDays: number; workspaceCompactionDelayHours: number | null; skillVersionRetentionEnabled: boolean; @@ -396,6 +398,8 @@ export function loadConfig(): Config { gitCacheExpirationEnabled: process.env.PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED === "true", gitBaseExpirationEnabled: process.env.PAPERCLIP_GIT_BASE_EXPIRATION_ENABLED === "true", gitCacheExpirationDryRun: process.env.PAPERCLIP_GIT_CACHE_EXPIRATION_DRY_RUN !== "false", + isolatedSourceRehomeEnabled: process.env.PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED === "true", + isolatedSourceRehomeDryRun: process.env.PAPERCLIP_ISOLATED_SOURCE_REHOME_DRY_RUN === "true", workspaceReaperCooldownDays, workspaceCompactionDelayHours, skillVersionRetentionEnabled: process.env.PAPERCLIP_SKILL_VERSION_RETENTION_ENABLED !== "false", diff --git a/server/src/index.ts b/server/src/index.ts index ef5da7d36e4..5eb3d88c599 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1485,6 +1485,8 @@ async function startServerWithDatabaseTeardown( gitCacheExpirationEnabled: config.gitCacheExpirationEnabled, gitBaseExpirationEnabled: config.gitBaseExpirationEnabled, gitCacheExpirationDryRun: config.gitCacheExpirationDryRun, + isolatedSourceRehomeEnabled: config.isolatedSourceRehomeEnabled, + isolatedSourceRehomeDryRun: config.isolatedSourceRehomeDryRun, gitRestorePoints: config.gitRestorePointsEnabled ? { storage: storageService, terminalDelayMs: config.gitRestorePointTerminalDelayDays * 24 * 60 * 60 * 1000, diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index 6d01ef4c5d0..0b922eeecc2 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -6,6 +6,7 @@ import { measureRunPhase } from "./execution-preparation-timing.js"; import { assertExecutionWorkspaceRecordAccess, assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-access.js"; import { verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js"; import { expireReconstructableGitWorkspace, gitWorkspaceLastActivityAt, hasPendingGitExpiration } from "./git-workspace-expiration.js"; +import { hasPendingIsolatedRepositoryRehomeCleanup, isIsolatedRepositoryRehomeCandidate, rehomeExpiredIsolatedRestoration, rehomeIsolatedRepositoryWorkspace } from "./isolated-repository-rehome.js"; import type { StorageService } from "../storage/types.js"; import { readGitCacheRestoration, restoreGitCacheBase, recoverGitCacheExpirationLocks, GitCacheRestorationFailure, gitRestorePointExpired } from "./git-workspace-restoration.js"; @@ -301,6 +302,8 @@ export type ExecutionWorkspaceServiceOptions = { gitCacheExpirationEnabled?: boolean; gitBaseExpirationEnabled?: boolean; gitCacheExpirationDryRun?: boolean; + isolatedSourceRehomeEnabled?: boolean; + isolatedSourceRehomeDryRun?: boolean; gitRestorePoints?: { storage: StorageService; terminalDelayMs: number; keyDurable: boolean; idleMs?: number | null }; terminalWorkspaceInspectionBackoffMs?: number; resolvePullRequestDetails?: PullRequestMergeDetailsResolver; @@ -1742,6 +1745,9 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic let terminalSweepCursor: { updatedAt: Date; id: string } | null = null; const terminalInspections = new Map; nextAt: number }>(); const terminalWorkspaceDeadlines = new Map(); + const isolatedRehomeDeadlines = new Map(); + const isolatedRehomeNotBefore = new Map(); + const ISOLATED_REHOME_RETRY_MS = 60 * 60 * 1000; let terminalSweepRescanRequested = false; // The reaper freezes an upper bound on updatedAt at the start of each // rotation. The scan only reads candidates at or below the bound, so a steady @@ -3246,7 +3252,10 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic and(eq(executionWorkspaces.providerType, "plugin"), inArray(executionWorkspaces.status, ["archived", "cleanup_failed"]), sql`(${executionWorkspaces.metadata}->>'workspaceProviderRemoved') IS DISTINCT FROM 'true'`), opts.gitCacheExpirationEnabled ? and(inArray(executionWorkspaces.status, ["archived", "cleanup_failed"]), - sql`${executionWorkspaces.metadata}->'gitCacheExpiration'->>'completed' = 'false'`) : undefined), + sql`${executionWorkspaces.metadata}->'gitCacheExpiration'->>'completed' = 'false'`) : undefined, + opts.gitCacheExpirationEnabled && opts.isolatedSourceRehomeEnabled ? and(inArray(executionWorkspaces.status, ["archived", "cleanup_failed"]), + sql`(${executionWorkspaces.metadata}->'isolatedRepositoryRehome') IS NOT NULL`, + sql`(${executionWorkspaces.metadata}->'isolatedRepositoryRehome'->>'cleanedAt') IS NULL`) : undefined), ); // Continue the scan after the previous sweep's last row. The keyset // predicate uses the same (updatedAt, id) order as the query, so each @@ -3261,6 +3270,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic if (!cursor) { terminalSweepBoundary = now(); terminalWorkspaceDeadlines.clear(); + isolatedRehomeDeadlines.clear(); terminalSweepRescanRequested = false; } const boundary = terminalSweepBoundary; @@ -3311,12 +3321,35 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic }; const readinessByWorkingTree = new Map>(); - for (const workspace of candidates) { + for (let workspace of candidates) { signal?.throwIfAborted(); // Trash deletion is part of compaction, so a sweep that skips compaction // also leaves leftover trash for a later sweep. if (options.compact !== false) await purgeCompactionTrashIfDue(workspace); else deferCompactionTrashPurge(workspace); + if (opts.gitCacheExpirationEnabled && opts.isolatedSourceRehomeEnabled + && (isIsolatedRepositoryRehomeCandidate(workspace) || hasPendingIsolatedRepositoryRehomeCleanup(workspace))) { + const dueAt = isolatedRehomeNotBefore.get(workspace.id) ?? 0; + if (dueAt > now().getTime()) isolatedRehomeDeadlines.set(workspace.id, dueAt); + else { + const dryRun = opts.isolatedSourceRehomeDryRun ?? false; + const rehome = await rehomeIsolatedRepositoryWorkspace(db, workspace, { dryRun, now: now(), signal }); + logger.info({ event: "execution_workspace.isolated_source_rehome", executionWorkspaceId: workspace.id, + companyId: workspace.companyId, dryRun, ...rehome }, "Isolated repository source re-home inspected"); + const [after] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace.id)); + if (after && (isIsolatedRepositoryRehomeCandidate(after) || hasPendingIsolatedRepositoryRehomeCleanup(after))) { + if (isolatedRehomeNotBefore.size >= 4096) isolatedRehomeNotBefore.delete(isolatedRehomeNotBefore.keys().next().value!); + isolatedRehomeNotBefore.set(workspace.id, now().getTime() + ISOLATED_REHOME_RETRY_MS); + isolatedRehomeDeadlines.set(workspace.id, now().getTime() + ISOLATED_REHOME_RETRY_MS); + } else { + isolatedRehomeNotBefore.delete(workspace.id); + isolatedRehomeDeadlines.delete(workspace.id); + } + if (!after) continue; + workspace = after; + } + if (isClosedExecutionWorkspaceStatus(workspace.status) && !hasPendingGitExpiration(workspace)) continue; + } const issueTree = await listWorkspaceIssueTree(workspace); const sourceIssue = issueTree.find((issue) => issue.id === workspace.sourceIssueId); const reopenPending = metadataHasReopenPendingConsumption( @@ -3752,8 +3785,8 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic }, nextTerminalWorkspaceSweepAt: async () => { - if (terminalWorkspaceDeadlines.size === 0) return null; - const next = [...terminalWorkspaceDeadlines.values()].reduce((earliest, deadline) => Math.min(earliest, deadline), Infinity); + if (terminalWorkspaceDeadlines.size === 0 && isolatedRehomeDeadlines.size === 0) return null; + const next = [...terminalWorkspaceDeadlines.values(), ...isolatedRehomeDeadlines.values()].reduce((earliest, deadline) => Math.min(earliest, deadline), Infinity); const nowMs = now().getTime(); if (next <= nowMs && opts.terminalWorkspaceInspectionBackoffMs) { return Math.min(nowMs + opts.terminalWorkspaceInspectionBackoffMs, @@ -3825,7 +3858,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic if (!freshSnapshot || freshSnapshot.sourceIssueId !== sourceIssueId) { return { ok: false, code: "not_reopenable", message: "Execution workspace source changed during reopen" }; } - const snapshot = freshSnapshot; + const snapshot = freshSnapshot.providerType === "plugin" ? freshSnapshot : await rehomeExpiredIsolatedRestoration(db, freshSnapshot); if (snapshot.providerType === "plugin") { return withExecutionWorkspaceLifecycleLock(db, snapshot.id, async tx => { const [row] = await tx.select().from(executionWorkspaces).where(and( diff --git a/server/src/services/git-restore-point-storage.ts b/server/src/services/git-restore-point-storage.ts index 32fb3dc15df..87bcae36578 100644 --- a/server/src/services/git-restore-point-storage.ts +++ b/server/src/services/git-restore-point-storage.ts @@ -178,10 +178,12 @@ export async function restorePointHistoryUnchanged(ref: GitRestorePointRef, cwd: return ref.history === undefined || await restorePointHistoryDigest(cwd, signal) === ref.history; } +const modesKey = (modes: Array<{ path: string; mode: number }>) => JSON.stringify(modes.map(entry => [entry.path, entry.mode])); + export function restorePointSnapshotMatches(ref: GitRestorePointRef, snapshot: { head: string; index: string | null; worktree: string | null; ignored: string | null; modes: Array<{ path: string; mode: number }> }) { return snapshot.head === ref.head && snapshot.index === ref.index && snapshot.worktree === ref.worktree && snapshot.ignored === (ref.ignored ?? null) - && JSON.stringify(snapshot.modes) === JSON.stringify(ref.modes); + && modesKey(snapshot.modes) === modesKey(ref.modes); } export async function abandonGitRestorePointsByKey(db: Db, workspaceId: string, objectKeys: string[]) { @@ -234,7 +236,7 @@ export async function loadGitRestorePoint(input: { const { manifest } = decodeRestorePointPayload(payload); if (manifest.head !== input.ref.head || manifest.index !== input.ref.index || manifest.worktree !== input.ref.worktree || (manifest.ignored ?? null) !== (input.ref.ignored ?? null) - || JSON.stringify(manifest.modes) !== JSON.stringify(input.ref.modes)) { + || modesKey(manifest.modes) !== modesKey(input.ref.modes)) { throw new RestorePointRestoreFailure("restore_point_manifest_mismatch"); } return { payload, manifest }; diff --git a/server/src/services/git-workspace-expiration.ts b/server/src/services/git-workspace-expiration.ts index 516ca8d5fb5..ff9e6c7ffc5 100644 --- a/server/src/services/git-workspace-expiration.ts +++ b/server/src/services/git-workspace-expiration.ts @@ -17,7 +17,7 @@ import type { GitRemoteAuthProvider } from "./git-credentials.js"; import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, assertReproducibleManagedCheckout, provenManagedCheckoutHistory, inspectGitCacheRecipe, inspectGitCacheRecipeForRestorePoint, readGitCacheRestoration, GIT_CACHE_RESTORATION_SCHEMA_V2, verifyRemoteGitCommit, gitCacheExpirationLockLocations, recoverGitCacheExpirationLocks, GitCacheRestorationFailure, type ManagedCheckoutProver, gitCacheCommand, readGitCacheExpiration, finishGitCacheTrash, type GitCacheExpiration, type GitCacheRestoration, GIT_CACHE_RESTORATION_KEY, GIT_CACHE_EXPIRATION_REASON } from "./git-workspace-restoration.js"; -async function allocatedBytes(target: string, shouldContinue: () => boolean): Promise { +export async function allocatedBytes(target: string, shouldContinue: () => boolean): Promise { if (!shouldContinue()) throw new Error("scan_stopped"); const entry = await fs.lstat(target); if (!entry.isDirectory() && !entry.isFile() && !entry.isSymbolicLink()) throw new Error("unsupported_filesystem_entry"); @@ -100,7 +100,7 @@ async function restorePointBoundary(cwd: string, repoUrl: string, headOnRemote: } /** Re-read the whole tree and every runtime/recovery consumer immediately before deletion. */ -function expirationScope(workspace: Workspace) { +export function expirationScope(workspace: Workspace) { return sql` WITH RECURSIVE tree AS ( SELECT id, status, completed_at, cancelled_at, updated_at FROM ${issues} diff --git a/server/src/services/git-workspace-restoration.ts b/server/src/services/git-workspace-restoration.ts index 85ba8036811..67b7c08a67e 100644 --- a/server/src/services/git-workspace-restoration.ts +++ b/server/src/services/git-workspace-restoration.ts @@ -54,7 +54,7 @@ type Workspace = { cwd: string | null; providerRef: string | null; status?: string; metadata: Record | null; }; -function managedSourcePaths(workspace: Workspace) { +export function managedSourcePaths(workspace: Workspace) { if (!workspace.projectId || !workspace.repoUrl) return []; let repoName: string | null = null; try { repoName = new URL(workspace.repoUrl).pathname.replace(/\/+$/, "").split("/").pop()?.replace(/\.git$/i, "") || null; } catch { return []; } @@ -62,7 +62,7 @@ function managedSourcePaths(workspace: Workspace) { return [root, `${root}-${createHash("sha256").update(workspace.repoUrl).digest("hex").slice(0, 12)}`]; } -async function verifiedGitCacheSource(workspace: Workspace, sourcePath: string) { +export async function verifiedGitCacheSource(workspace: Workspace, sourcePath: string) { const privateSource = recordedIsolatedRemoteSource(workspace); if (privateSource) { if (sourcePath !== privateSource) throw new GitCacheRestorationFailure(); diff --git a/server/src/services/isolated-repository-rehome.ts b/server/src/services/isolated-repository-rehome.ts new file mode 100644 index 00000000000..233ca303d88 --- /dev/null +++ b/server/src/services/isolated-repository-rehome.ts @@ -0,0 +1,622 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { createHash } from "node:crypto"; +import { execFile as execFileCallback } from "node:child_process"; +import { promisify } from "node:util"; +import { and, eq, ne, sql } from "drizzle-orm"; +import { executionWorkspaces, projectWorkspaces, workspaceRuntimeServices, heartbeatRuns, nativeRunFinalizations, environmentLeases, + withDedicatedDbConnection, type Db } from "@paperclipai/db"; +import { PROJECT_REPOSITORIES_DIR } from "@paperclipai/adapter-utils/git-workspace-sync"; +import { IsolatedRepositorySourceBusy, isolatedRemoteSourcePath, verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js"; +import { acquireGitExpirationLocks, recoverGitExpirationLocks, type GitExpirationLockLocation } from "./git-workspace-expiration-locks.js"; +import { allocatedBytes, describeGitExpirationFailure, expirationScope, gitExpirationSkipReason, type GitExpirationFailure } from "./git-workspace-expiration.js"; +import { finishGitCacheTrash, managedSourcePaths, readGitCacheExpiration, readGitCacheRestoration, verifiedGitCacheSource, + GIT_CACHE_RESTORATION_KEY, type GitCacheRestoration } from "./git-workspace-restoration.js"; +import { applyRestorePoint, captureRestorePoint, currentRestorePointSnapshot, importRestorePoint, restorePointSavesIgnoredFile, + RestorePointRestoreFailure, RestorePointSkip, type RestorePointLimits, type RestorePointSnapshot } from "./git-restore-point.js"; +import { findRegenerableWorkspaceDirs, REGENERABLE_WORKSPACE_DIR_NAMES } from "./workspace-compaction.js"; +import { ensureManagedRepositoriesIgnored } from "./managed-repository-checkout.js"; +import { withWorkspaceLockKeepalive } from "./native-runtime/native-workspace-lock-keepalive.js"; +import { canonicalRepositoryRoot } from "./execution-workspace-source-roots.js"; +import { withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifecycle-lock.js"; +import { loadGitRestorePoint } from "./git-restore-point-storage.js"; +import { nestedWorktreeBranchPrefix, readLinkedNestedCheckout, repairMovedNestedRepositoryWorktree, withNestedRepositoryBaseLock } from "./nested-repository-worktrees.js"; +import { getStorageService } from "../storage/index.js"; + +const execFile = promisify(execFileCallback); +const SHA = /^[0-9a-f]{40}$/; +const OPEN_STATUSES = ["active", "idle", "in_review"]; +const CLOSED_STATUSES = ["archived", "cleanup_failed"]; +const ADMIN_ENTRIES = new Set(["HEAD", "ORIG_HEAD", "FETCH_HEAD", "index", "commondir", "gitdir", "logs", "refs", "COMMIT_EDITMSG"]); +const MAX_BOUNDARY_COMMITS = 64; +const CLONE_HISTORY_BUDGET = 5_000; +const UNSAFE_BASE_CONFIG = /^(filter\.|include\.|includeif\.|core\.(fsmonitor|hookspath|attributesfile|worktree|gitproxy|sshcommand|askpass|editor|pager|alternaterefscommand)$|extensions\.worktreeconfig$)/; +export const ISOLATED_REPOSITORY_REHOME_KEY = "isolatedRepositoryRehome"; +export const ISOLATED_REPOSITORY_REHOME_PENDING_KEY = "isolatedRepositoryRehomePending"; + +type Workspace = typeof executionWorkspaces.$inferSelect; +type Tx = Parameters[0]>[0]; +export type IsolatedRepositoryRehomeRecord = { at: string; from: string; to: string; cleanedAt: string | null; cloneRetained?: string }; +export type IsolatedRepositoryRehomePending = { base: string; from: string; target: string; branch: string; head: string; previousTip: string | null; moves: string[] }; +export type IsolatedRepositoryRehomeHooks = { + beforeWorktreeAdd?: (input: { from: string; to: string }) => Promise; + beforeVerify?: (input: { from: string; to: string }) => Promise; + afterDatabaseUpdate?: (workspace: Workspace) => Promise; +}; +export type IsolatedRepositoryRehomeResult = { + outcome: "rehomed" | "dry_run" | "skipped" | "cleaned" | "clone_kept"; + rehomed: boolean; cloneRemoved: boolean; reason: string | null; cloneKeptReason: string | null; + candidateBytes: number; reclaimedBytes: number; from: string | null; to: string | null; failure?: GitExpirationFailure; +}; + +export function isIsolatedRepositoryRehomeCandidate(workspace: Workspace) { + return workspace.providerType !== "plugin" && workspace.mode === "isolated_workspace" && workspace.strategyType === "git_worktree" + && OPEN_STATUSES.includes(workspace.status) && !workspace.closedAt && workspace.metadata?.isolatedRemoteSource !== undefined; +} + +function readRehome(workspace: Pick) { + const value = workspace.metadata?.[ISOLATED_REPOSITORY_REHOME_KEY] as IsolatedRepositoryRehomeRecord | undefined; + return value && typeof value === "object" && typeof value.from === "string" && typeof value.to === "string" ? value : null; +} + +export function hasPendingIsolatedRepositoryRehomeCleanup(workspace: Workspace) { + const rehome = readRehome(workspace); + return Boolean(rehome && !rehome.cleanedAt && workspace.providerType !== "plugin"); +} + +async function git(cwd: string, args: string[], options: { signal?: AbortSignal; input?: string; timeout?: number } = {}) { + const child = execFile("git", ["-c", "core.fsmonitor=false", "-c", `core.hooksPath=${os.devNull}`, "-C", cwd, ...args], { + timeout: options.timeout ?? 120_000, signal: options.signal, maxBuffer: 256 * 1024 * 1024, + env: { ...process.env, GIT_TERMINAL_PROMPT: "0", GIT_OPTIONAL_LOCKS: "0" }, + }); + if (options.input !== undefined) child.child.stdin?.end(options.input); + return (await child).stdout; +} + +const shas = (output: string) => output.split("\n").map(line => line.trim()).filter(line => SHA.test(line)); +const nul = (output: string) => output.split("\0").filter(Boolean); + +async function exists(target: string) { + return fs.lstat(target).then(() => true, (error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return false; throw error; }); +} + +async function presentCommits(cwd: string, commits: Iterable, signal?: AbortSignal) { + const unique = [...new Set(commits)].filter(sha => SHA.test(sha)); + if (unique.length === 0) return new Set(); + const output = await git(cwd, ["cat-file", "--batch-check=%(objectname) %(objecttype)"], { signal, input: unique.join("\n") + "\n" }); + return new Set(output.split("\n").map(line => line.split(" ")).filter(([sha, type]) => SHA.test(sha ?? "") && type === "commit").map(([sha]) => sha!)); +} + +function lockToken(workspaceId: string) { + return createHash("sha256").update(`paperclip-isolated-repository-rehome:${workspaceId}`).digest("hex"); +} + +async function recoverRehomeLocks(clone: string, branch: string | null, workspaceId: string) { + const worktrees = path.join(clone, ".git", "worktrees"); + const admins = await fs.readdir(worktrees).catch(() => [] as string[]); + await recoverGitExpirationLocks(rehomeLockLocations(clone, admins, branch), lockToken(workspaceId)).catch(() => undefined); +} + +function rehomeLockLocations(clone: string, admins: string[], branch: string | null): GitExpirationLockLocation[] { + const locations: GitExpirationLockLocation[] = admins.flatMap(admin => ["index.lock", "HEAD.lock"] + .map(name => ({ root: clone, relativePath: path.join(".git", "worktrees", admin, name), checkout: null }))); + if (branch) locations.push({ root: clone, relativePath: path.join(".git", "refs", "heads", `${branch}.lock`), checkout: null }); + return locations; +} + +async function rehomeEligible(tx: Db | Tx, workspace: Workspace) { + if (workspace.metadata?.reopenPendingConsumption === true) return false; + const rows = await tx.execute<{ allowed: boolean }>(sql`${expirationScope(workspace)} + SELECT ( + NOT EXISTS (SELECT 1 FROM ${executionWorkspaces} other WHERE other.id <> ${workspace.id} + AND (other.cwd = ${workspace.cwd} OR other.provider_ref = ${workspace.cwd} + OR left(other.cwd, length(${workspace.cwd}) + 1) = ${workspace.cwd} || '/')) + AND NOT EXISTS (SELECT 1 FROM relevant_runs WHERE status NOT IN ('succeeded', 'failed', 'timed_out', 'cancelled', 'interrupted') + OR finished_at IS NULL OR controller_lease_expires_at > now()) + AND NOT EXISTS (SELECT 1 FROM ${nativeRunFinalizations} f WHERE f.company_id = ${workspace.companyId} + AND (f.issue_id IN (SELECT id FROM tree) OR f.run_id IN (SELECT id FROM relevant_runs)) + AND (f.phase <> 'committed' OR f.lease_owner IS NOT NULL OR f.recovery_state IS NOT NULL)) + AND NOT EXISTS (SELECT 1 FROM ${environmentLeases} l WHERE l.company_id = ${workspace.companyId} + AND (l.execution_workspace_id = ${workspace.id} OR l.heartbeat_run_id IN (SELECT id FROM relevant_runs)) + AND (l.status = 'pending_cleanup' OR l.cleanup_status = 'failed' OR l.metadata ? 'nativeWorkspaceExportResume' + OR (l.released_at IS NULL AND NOT coalesce(l.status = 'retained' AND l.lease_policy = 'reuse_by_environment' AND l.cleanup_status = 'success', false)))) + AND NOT EXISTS (SELECT 1 FROM ${workspaceRuntimeServices} WHERE company_id = ${workspace.companyId} + AND (execution_workspace_id = ${workspace.id} OR cwd = ${workspace.cwd} + OR left(cwd, length(${workspace.cwd}) + 1) = ${workspace.cwd} || '/') AND status <> 'stopped') + ) AS allowed`); + return rows[0]?.allowed === true; +} + +async function sharedManagedBase(workspace: Workspace) { + if (!workspace.projectId || !workspace.repoUrl) throw new Error("no_shared_base"); + let mismatch = false; + for (const candidate of managedSourcePaths(workspace)) { + if (!await exists(path.join(candidate, ".git"))) continue; + try { return await verifiedGitCacheSource({ ...workspace, metadata: null }, candidate); } catch { mismatch = true; } + } + throw new Error(mismatch ? "shared_base_repo_mismatch" : "no_shared_base"); +} + +function standardWorktreeSuffix(branch: string) { + if (!branch || branch.split("/").some(part => !part || part === "." || part === "..") || branch.includes("\\")) throw new Error("unsupported_branch_name"); + return path.join(".paperclip", "worktrees", branch); +} + +async function targetWorktree(db: Db | Tx, workspace: Workspace, base: string, branch: string, signal?: AbortSignal) { + const target = path.join(await fs.realpath(base), standardWorktreeSuffix(branch)); + if (await exists(target)) throw new Error("target_worktree_exists"); + const [other] = await db.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(ne(executionWorkspaces.id, workspace.id), + sql`(${executionWorkspaces.cwd} = ${target} OR ${executionWorkspaces.providerRef} = ${target})`)).limit(1); + if (other) throw new Error("target_worktree_exists"); + const checkedOut = (await git(base, ["worktree", "list", "--porcelain"], { signal })).split("\n").includes(`branch refs/heads/${branch}`); + if (checkedOut) throw new Error("base_branch_checked_out"); + return target; +} + +async function assertSafeBaseConfig(base: string, signal?: AbortSignal) { + const keys = (await git(base, ["config", "--local", "--name-only", "--list"], { signal })).split("\n").map(key => key.trim().toLowerCase()).filter(Boolean); + if (keys.some(key => UNSAFE_BASE_CONFIG.test(key))) throw new Error("unsupported_base_git_config"); + if (await exists(path.join(base, ".git", "info", "attributes"))) throw new Error("unsupported_base_git_config"); +} + +async function inspectWorktree(workspace: Workspace, clone: string, signal?: AbortSignal) { + const cwd = workspace.cwd; + if (!cwd || !workspace.branchName || !workspace.repoUrl || (workspace.providerRef && workspace.providerRef !== cwd)) throw new Error("unproven_workspace_ownership"); + const stat = await fs.lstat(cwd).catch(() => null); + if (!stat) throw new Error("missing_worktree"); + if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("unsafe_worktree_path"); + const real = await fs.realpath(cwd); + const cloneReal = await fs.realpath(clone); + if (real !== path.join(cloneReal, standardWorktreeSuffix(workspace.branchName))) throw new Error("nonstandard_worktree_location"); + if (!(await fs.lstat(path.join(real, ".git"))).isFile()) throw new Error("not_linked_worktree"); + const common = (await git(real, ["rev-parse", "--path-format=absolute", "--git-common-dir"], { signal })).trim(); + if (common !== path.join(cloneReal, ".git")) throw new Error("source_identity_changed"); + if ((await git(real, ["symbolic-ref", "--short", "HEAD"], { signal })).trim() !== workspace.branchName) throw new Error("branch_identity_changed"); + const admin = (await git(real, ["rev-parse", "--absolute-git-dir"], { signal })).trim(); + if (path.dirname(admin) !== path.join(common, "worktrees")) throw new Error("unproven_worktree_admin"); + if ((await fs.readdir(admin)).some(name => !ADMIN_ENTRIES.has(name))) throw new Error("unsupported_worktree_git_state"); + const refs = async (dir: string): Promise => { + for (const entry of await fs.readdir(dir, { withFileTypes: true }).catch(() => [])) { + if (!entry.isDirectory()) throw new Error("unique_worktree_refs"); + await refs(path.join(dir, entry.name)); + } + }; + await refs(path.join(admin, "refs")); + if ((await fs.readdir(path.join(admin, "logs")).catch(() => [] as string[])).some(name => name !== "HEAD")) throw new Error("unknown_worktree_git_logs"); + if (await exists(path.join(common, "shallow"))) throw new Error("unsupported_local_git_state"); + if ((await git(real, ["for-each-ref", "--format=%(refname)", "refs/stash", "refs/replace"], { signal })).trim()) throw new Error("unique_git_refs"); + if ((await git(real, ["ls-files", "-v"], { signal })).split("\n").some(line => /^[a-zS] /.test(line))) throw new Error("unsupported_index_flags"); + return { cwd: real, admin, head: (await git(real, ["rev-parse", "--verify", "HEAD^{commit}"], { signal })).trim() }; +} + +async function transferPlan(cwd: string, signal?: AbortSignal) { + const regenerable = await findRegenerableWorkspaceDirs(cwd, { signal }); + const nestedRoot = path.join(cwd, PROJECT_REPOSITORIES_DIR); + const nestedStat = await fs.lstat(nestedRoot).catch(() => null); + if (nestedStat && (!nestedStat.isDirectory() || nestedStat.isSymbolicLink())) throw new Error("untransferable_managed_checkouts"); + const nested = nestedStat ? (await fs.readdir(nestedRoot)).map(name => path.posix.join(PROJECT_REPOSITORIES_DIR, name)) : []; + const covered = (file: string) => file === `${PROJECT_REPOSITORIES_DIR}/` || file.startsWith(`${PROJECT_REPOSITORIES_DIR}/`) + || regenerable.some(dir => file.startsWith(`${dir}/`)); + const env = (file: string) => /^\.env(\..*)?$/.test(path.posix.basename(file)); + const untracked = nul(await git(cwd, ["ls-files", "-z", "--others", "--exclude-standard"], { signal })); + const ignored = nul(await git(cwd, ["ls-files", "-z", "--others", "--ignored", "--exclude-standard"], { signal })); + const files = [...new Set([...untracked.filter(file => !covered(file) && env(file)), + ...ignored.filter(file => !covered(file) && (env(file) || file === ".claude/settings.local.json"))])].sort(); + if (ignored.some(file => !covered(file) && !files.includes(file) && !restorePointSavesIgnoredFile(file))) throw new Error("untransferable_ignored_files"); + if (files.some(file => file.endsWith("/"))) throw new Error("untransferable_ignored_files"); + const plan = [...regenerable, ...nested, ...files]; + return plan.filter(item => !plan.some(other => other !== item && item.startsWith(`${other}/`))); +} + +async function boundary(cwd: string, base: string, head: string, signal?: AbortSignal) { + const remoteTips = async (repo: string) => shas(await git(repo, ["for-each-ref", "--format=%(objectname)", "--sort=-committerdate", "refs/remotes"], { signal })); + const candidates = [head, ...await remoteTips(base), ...await remoteTips(cwd)]; + const inBase = await presentCommits(base, candidates, signal); + const inClone = await presentCommits(cwd, candidates, signal); + return [...new Set(candidates)].filter(sha => inBase.has(sha) && inClone.has(sha)).slice(0, MAX_BOUNDARY_COMMITS); +} + +function snapshotKey(snapshot: RestorePointSnapshot) { + return JSON.stringify([snapshot.head, snapshot.index, snapshot.worktree, snapshot.ignored, snapshot.modes]); +} + +async function historyKey(cwd: string, admin: string, signal?: AbortSignal) { + const reflog = await git(cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal }).catch(() => ""); + const original = await fs.readFile(path.join(admin, "ORIG_HEAD"), "utf8").catch(() => ""); + return JSON.stringify([shas(reflog), original.trim()]); +} + +async function repairLinkedCheckout(checkout: string) { + const marker = await fs.lstat(path.join(checkout, ".git")).catch(() => null); + if (!marker?.isFile()) return; + const common = (await git(checkout, ["rev-parse", "--path-format=absolute", "--git-common-dir"])).trim(); + await repairMovedNestedRepositoryWorktree(path.dirname(common), checkout); +} + +async function retargetNestedBranch(checkout: string, from: string) { + const marker = await fs.lstat(path.join(checkout, ".git")).catch(() => null); + if (!marker?.isFile()) return; + const head = (await git(checkout, ["symbolic-ref", "--quiet", "--short", "HEAD"]).catch(() => "")).trim(); + const previous = await nestedWorktreeBranchPrefix(from); + if (!head.startsWith(previous)) return; + const next = `${await nestedWorktreeBranchPrefix(checkout)}${head.slice(previous.length)}`; + if (next === head) return; + const base = path.dirname((await git(checkout, ["rev-parse", "--path-format=absolute", "--git-common-dir"])).trim()); + await withNestedRepositoryBaseLock(base, async () => { + if ((await git(base, ["rev-parse", "--verify", "-q", `refs/heads/${next}`]).catch(() => "")).trim()) throw new Error("nested_branch_conflict"); + await git(checkout, ["branch", "-m", head, next]); + }); +} + +async function linkedCheckoutBases(cwd: string, moves: string[]) { + const bases = new Map(); + for (const relative of moves.filter(item => item.startsWith(`${PROJECT_REPOSITORIES_DIR}/`))) { + const linked = await readLinkedNestedCheckout(path.join(cwd, relative)).catch(() => { throw new Error("untransferable_managed_checkouts"); }); + if (linked) bases.set(relative, linked.base); + } + return bases; +} + +function readPending(workspace: Workspace) { + const value = workspace.metadata?.[ISOLATED_REPOSITORY_REHOME_PENDING_KEY] as IsolatedRepositoryRehomePending | undefined; + if (value === undefined) return null; + if (!value || typeof value !== "object" || ![value.base, value.from, value.target, value.branch, value.head].every(item => typeof item === "string") + || !SHA.test(value.head) || (value.previousTip !== null && !SHA.test(value.previousTip)) || !Array.isArray(value.moves) + || value.moves.some(item => typeof item !== "string" || !item || path.isAbsolute(item) || item.split("/").includes(".."))) throw new Error("invalid_rehome_record"); + return value; +} + +async function writePending(db: Db, workspaceId: string, value: IsolatedRepositoryRehomePending | null) { + await db.update(executionWorkspaces).set({ metadata: value + ? sql`coalesce(${executionWorkspaces.metadata}, '{}'::jsonb) || jsonb_build_object(${ISOLATED_REPOSITORY_REHOME_PENDING_KEY}::text, ${JSON.stringify(value)}::jsonb)` + : sql`${executionWorkspaces.metadata} - ${ISOLATED_REPOSITORY_REHOME_PENDING_KEY}::text` }).where(eq(executionWorkspaces.id, workspaceId)); +} + +async function provenTarget(db: Db | Tx, workspaceId: string, pending: IsolatedRepositoryRehomePending) { + const stat = await fs.lstat(pending.target).catch(() => null); + if (!stat?.isDirectory() || stat.isSymbolicLink() || !(await fs.lstat(path.join(pending.target, ".git")).catch(() => null))?.isFile()) return false; + const common = (await git(pending.target, ["rev-parse", "--path-format=absolute", "--git-common-dir"]).catch(() => "")).trim(); + if (common !== path.join(await fs.realpath(pending.base), ".git")) return false; + if ((await git(pending.target, ["symbolic-ref", "-q", "HEAD"]).catch(() => "")).trim() !== `refs/heads/${pending.branch}`) return false; + const [other] = await db.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(ne(executionWorkspaces.id, workspaceId), + sql`(${executionWorkspaces.cwd} = ${pending.target} OR ${executionWorkspaces.providerRef} = ${pending.target})`)).limit(1); + return !other; +} + +async function undoMove(db: Db | Tx, workspaceId: string, pending: IsolatedRepositoryRehomePending, ownsTarget?: boolean) { + if (ownsTarget ?? await provenTarget(db, workspaceId, pending)) { + for (const relative of pending.moves) { + if (await exists(path.join(pending.target, relative)) && await exists(path.join(pending.from, relative))) throw new Error("rehome_rollback_conflict"); + } + for (const relative of [...pending.moves].reverse()) { + const moved = path.join(pending.target, relative), original = path.join(pending.from, relative); + if (!await exists(moved) || await exists(original)) continue; + await fs.mkdir(path.dirname(original), { recursive: true }); + await fs.rename(moved, original); + await repairLinkedCheckout(original).catch(() => undefined); + await retargetNestedBranch(original, moved).catch(() => undefined); + } + await git(pending.base, ["worktree", "remove", "--force", "--force", pending.target]).catch(() => undefined); + await fs.rm(pending.target, { recursive: true, force: true }); + await git(pending.base, ["worktree", "prune"]).catch(() => undefined); + } + const branchRef = `refs/heads/${pending.branch}`; + const tip = (await git(pending.base, ["rev-parse", "--verify", "-q", `${branchRef}^{commit}`]).catch(() => "")).trim(); + if (tip !== pending.head || (await git(pending.base, ["worktree", "list", "--porcelain"])).split("\n").includes(`branch ${branchRef}`)) return; + if (pending.previousTip) await git(pending.base, ["update-ref", branchRef, pending.previousTip, pending.head]); + else { + await git(pending.base, ["update-ref", "-d", branchRef, pending.head]); + for (const key of ["remote", "merge"]) await git(pending.base, ["config", "--local", "--unset-all", `branch.${pending.branch}.${key}`]).catch(() => undefined); + } +} + +function withRehomeLocks(db: Db, workspace: Workspace, work: (tx: Tx) => Promise) { + return withIsolatedRepositorySourceLock(db, workspace.companyId, workspace.sourceIssueId!, () => + withDedicatedDbConnection(db, connection => connection.transaction(async tx => { + const rows = await tx.execute<{ locked: boolean; idle_timeout_ms: string | number | null }>(sql`select + pg_try_advisory_xact_lock(hashtextextended(${`execution_workspace_lifecycle:${workspace.id}`}, 0)) as locked, + (select setting::bigint from pg_settings where name = 'idle_in_transaction_session_timeout') as idle_timeout_ms`); + if (!rows[0]?.locked) throw new Error("lifecycle_busy"); + return withWorkspaceLockKeepalive(tx, Number(rows[0]?.idle_timeout_ms), () => work(tx)); + })), true); +} + +function emptyResult(): IsolatedRepositoryRehomeResult { + return { outcome: "skipped", rehomed: false, cloneRemoved: false, reason: null, cloneKeptReason: null, + candidateBytes: 0, reclaimedBytes: 0, from: null, to: null }; +} + +function skip(result: IsolatedRepositoryRehomeResult, error: unknown) { + const reason = error instanceof IsolatedRepositorySourceBusy ? "source_busy" : error instanceof RestorePointSkip ? error.reason + : error instanceof RestorePointRestoreFailure ? error.reason : gitExpirationSkipReason(error); + if (reason === "inspection_failed") result.failure = describeGitExpirationFailure(error); + return reason; +} + +export async function rehomeIsolatedRepositoryWorkspace(db: Db, snapshot: Workspace, options: { + dryRun?: boolean; now?: Date; signal?: AbortSignal; limits?: Partial; hooks?: IsolatedRepositoryRehomeHooks; +} = {}): Promise { + const result = emptyResult(); + const candidate = isIsolatedRepositoryRehomeCandidate(snapshot); + if (!snapshot.sourceIssueId || (!candidate && !hasPendingIsolatedRepositoryRehomeCleanup(snapshot))) return { ...result, reason: "not_candidate" }; + try { + return await withRehomeLocks(db, snapshot, async tx => { + const [row] = await db.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, snapshot.id), eq(executionWorkspaces.companyId, snapshot.companyId))); + if (!row || row.updatedAt.getTime() !== snapshot.updatedAt.getTime()) throw new Error("workspace_changed"); + if (!candidate) return await cleanupRehomedSource(db, tx, row, result, options); + if (!await rehomeEligible(tx, row)) throw new Error("live_or_unresolved_consumer"); + let recipe: GitCacheRestoration | null = null; + try { recipe = row.metadata?.[GIT_CACHE_RESTORATION_KEY] === undefined ? null : readGitCacheRestoration(row); } + catch { throw new Error("invalid_restoration_recipe"); } + if (recipe && recipe.sourceKind !== "isolated_remote") throw new Error("unsupported_restoration_recipe"); + const clone = (await verifiedRecordedIsolatedRemoteSource(row))!; + await recoverRehomeLocks(clone, row.branchName, row.id); + const old = await inspectWorktree(row, clone, options.signal); + const base = await sharedManagedBase(row); + const branch = row.branchName!; + const branchRef = `refs/heads/${branch}`; + const interrupted = readPending(row); + if (interrupted) { + if (interrupted.base !== base || interrupted.from !== old.cwd || interrupted.branch !== branch + || interrupted.target !== path.join(await fs.realpath(base), standardWorktreeSuffix(branch))) throw new Error("invalid_rehome_record"); + if (options.dryRun) return { ...result, outcome: "dry_run" as const, reason: "interrupted_rehome_pending", from: row.cwd, to: interrupted.target }; + await undoMove(tx, row.id, interrupted); + await writePending(db, row.id, null); + } + const target = await targetWorktree(tx, row, base, branch, options.signal); + await assertSafeBaseConfig(base, options.signal); + const previousTip = (await git(base, ["rev-parse", "--verify", "-q", `${branchRef}^{commit}`], { signal: options.signal }).catch(() => "")).trim() || null; + if (previousTip && !await git(old.cwd, ["merge-base", "--is-ancestor", previousTip, "HEAD"], { signal: options.signal }).then(() => true, () => false)) { + throw new Error("base_branch_diverged"); + } + const moves = await transferPlan(old.cwd, options.signal); + const linkedBases = await linkedCheckoutBases(old.cwd, moves); + result.from = row.cwd; result.to = target; + result.candidateBytes = await allocatedBytes(clone, () => { options.signal?.throwIfAborted(); return true; }).catch(() => 0); + if (options.dryRun) return { ...result, outcome: "dry_run" as const }; + const admins = [path.basename(old.admin)]; + const locks = await acquireGitExpirationLocks(rehomeLockLocations(clone, admins, branch), lockToken(row.id)) + .catch(() => { throw new Error("worktree_git_busy"); }); + const pending: IsolatedRepositoryRehomePending = { base, from: old.cwd, target, branch, head: old.head, previousTip, moves }; + let recorded = false, ownsTarget = false, committed = false; + try { + const before = await currentRestorePointSnapshot(old.cwd, options.signal); + const beforeHistory = await historyKey(old.cwd, old.admin, options.signal); + if (before.head !== old.head) throw new Error("head_changed"); + const captured = await captureRestorePoint({ cwd: old.cwd, upstreamCommits: await boundary(old.cwd, base, old.head, options.signal), + limits: options.limits, signal: options.signal }); + const manifest = captured ? await importRestorePoint({ repositoryCwd: base, payload: captured.payload, signal: options.signal }) : null; + const reflog = shas(await git(old.cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal: options.signal }).catch(() => "")); + const original = (await fs.readFile(path.join(old.admin, "ORIG_HEAD"), "utf8").catch(() => "")).trim(); + const history = [old.head, ...reflog, ...(SHA.test(original) ? [original] : [])]; + const present = await presentCommits(base, history, options.signal); + if (history.some(sha => !present.has(sha))) throw new Error("history_not_transferred"); + await writePending(db, row.id, pending); + recorded = true; + if (previousTip) await git(base, ["update-ref", branchRef, old.head, previousTip], { signal: options.signal }); + else { + await git(base, ["update-ref", branchRef, old.head, ""], { signal: options.signal }); + for (const key of ["remote", "merge"]) { + const value = (await git(old.cwd, ["config", "--local", "--get", `branch.${branch}.${key}`]).catch(() => "")).trim(); + if (value) await git(base, ["config", "--local", `branch.${branch}.${key}`, value], { signal: options.signal }); + } + } + await fs.mkdir(path.dirname(target), { recursive: true }); + await assertSafeBaseConfig(base, options.signal); + await options.hooks?.beforeWorktreeAdd?.({ from: old.cwd, to: target }); + if (await exists(target)) throw new Error("target_worktree_exists"); + await git(base, ["worktree", "add", target, branch], { signal: options.signal, timeout: 15 * 60_000 }); + ownsTarget = true; + if (manifest) await applyRestorePoint({ cwd: target, manifest, signal: options.signal }); + const newAdmin = (await git(target, ["rev-parse", "--absolute-git-dir"], { signal: options.signal })).trim(); + await fs.mkdir(path.join(newAdmin, "logs"), { recursive: true }); + await fs.copyFile(path.join(old.admin, "logs", "HEAD"), path.join(newAdmin, "logs", "HEAD")).catch(async (error: NodeJS.ErrnoException) => { + if (error.code !== "ENOENT") throw error; + await fs.rm(path.join(newAdmin, "logs", "HEAD"), { force: true }); + }); + if (original) await fs.copyFile(path.join(old.admin, "ORIG_HEAD"), path.join(newAdmin, "ORIG_HEAD")); + if (moves.some(item => item.startsWith(`${PROJECT_REPOSITORIES_DIR}/`))) await ensureManagedRepositoriesIgnored(target); + for (const relative of moves) { + const destination = path.join(target, relative); + if (await exists(destination)) throw new Error("transfer_conflict"); + await fs.mkdir(path.dirname(destination), { recursive: true }); + await fs.rename(path.join(old.cwd, relative), destination); + if (linkedBases.has(relative)) { + await repairLinkedCheckout(destination); + await retargetNestedBranch(destination, path.join(old.cwd, relative)); + } + } + await options.hooks?.beforeVerify?.({ from: old.cwd, to: target }); + const after = await currentRestorePointSnapshot(old.cwd, options.signal); + const moved = await currentRestorePointSnapshot(target, options.signal); + if (snapshotKey(after) !== snapshotKey(before) || snapshotKey(moved) !== snapshotKey(before)) throw new Error("snapshot_mismatch"); + if (await historyKey(old.cwd, old.admin, options.signal) !== beforeHistory || await historyKey(target, newAdmin, options.signal) !== beforeHistory) { + throw new Error("history_mismatch"); + } + if ((await git(target, ["symbolic-ref", "--short", "HEAD"], { signal: options.signal })).trim() !== branch) throw new Error("branch_identity_changed"); + for (const relative of moves) { + if (!await exists(path.join(target, relative)) || await exists(path.join(old.cwd, relative))) throw new Error("transfer_incomplete"); + const base = linkedBases.get(relative); + if (!base) continue; + const linked = await readLinkedNestedCheckout(path.join(target, relative)).catch(() => null); + const head = linked ? (await git(linked.checkout, ["symbolic-ref", "--quiet", "--short", "HEAD"]).catch(() => "")).trim() : ""; + if (linked?.base !== base || (head.startsWith("paperclip/") && !head.startsWith(linked.branchPrefix))) throw new Error("transfer_incomplete"); + } + const at = (options.now ?? new Date()).toISOString(); + await db.transaction(async updateTx => { + const [current] = await updateTx.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, row.id)).for("update"); + if (!current || current.updatedAt.getTime() !== row.updatedAt.getTime()) throw new Error("workspace_changed"); + if (!await rehomeEligible(updateTx, current)) throw new Error("live_or_unresolved_consumer"); + const metadata: Record = { ...current.metadata }; + delete metadata.isolatedRemoteSource; + delete metadata.gitCacheExpiration; + delete metadata[ISOLATED_REPOSITORY_REHOME_PENDING_KEY]; + if (recipe) metadata[GIT_CACHE_RESTORATION_KEY] = { ...recipe, sourceKind: "managed_project", sourcePath: base, worktreePath: target }; + metadata[ISOLATED_REPOSITORY_REHOME_KEY] = { at, from: current.cwd ?? old.cwd, to: target, cleanedAt: null } satisfies IsolatedRepositoryRehomeRecord; + await updateTx.update(executionWorkspaces).set({ cwd: target, providerRef: current.providerRef ? target : null, metadata, updatedAt: new Date() }) + .where(eq(executionWorkspaces.id, row.id)); + }); + committed = true; + } finally { + try { + if (!committed && recorded) { + await undoMove(tx, row.id, pending, ownsTarget); + await writePending(db, row.id, null); + } + } finally { await locks.release().catch(() => undefined); } + } + result.rehomed = true; + result.outcome = "rehomed"; + const [updated] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, row.id)); + await options.hooks?.afterDatabaseUpdate?.(updated!); + return await cleanupRehomedSource(db, tx, updated!, result, options); + }); + } catch (error) { + options.signal?.throwIfAborted(); + result.reason = skip(result, error); + return result; + } +} + +async function cloneConsumers(db: Db | Tx, workspace: Workspace, clone: string) { + const others = await db.select().from(executionWorkspaces).where(and(ne(executionWorkspaces.id, workspace.id), + sql`((${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'companyId' = ${workspace.companyId} + AND ${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'issueId' = ${workspace.sourceIssueId}) + OR ${executionWorkspaces.cwd} = ${clone} OR ${executionWorkspaces.providerRef} = ${clone} + OR left(${executionWorkspaces.cwd}, length(${clone}) + 1) = ${clone} || '/')`)); + const expired = (row: Workspace) => { + try { return CLOSED_STATUSES.includes(row.status) && Boolean(readGitCacheRestoration(row)) && Boolean(readGitCacheExpiration(row)?.workspaceRemoved); } + catch { return false; } + }; + if (others.some(row => !expired(row))) return "base_has_consumers"; + const [project] = await db.select({ id: projectWorkspaces.id }).from(projectWorkspaces).where(sql`${projectWorkspaces.cwd} = ${clone} + OR left(${projectWorkspaces.cwd}, length(${clone}) + 1) = ${clone} || '/'`).limit(1); + if (project) return "base_has_consumers"; + const [service] = await db.select({ id: workspaceRuntimeServices.id }).from(workspaceRuntimeServices).where(sql` + ${workspaceRuntimeServices.status} <> 'stopped' AND (${workspaceRuntimeServices.cwd} = ${clone} + OR left(${workspaceRuntimeServices.cwd}, length(${clone}) + 1) = ${clone} || '/')`).limit(1); + return service ? "base_has_live_services" : null; +} + +async function cloneDisposable(clone: string, base: string, signal?: AbortSignal) { + const worktrees = (await git(clone, ["worktree", "list", "--porcelain"], { signal })).split("\n").filter(line => line.startsWith("worktree ")); + if (worktrees.length !== 1) return "base_has_worktrees"; + if ((await git(clone, ["status", "--porcelain=v1", "-z", "--untracked-files=no"], { signal })).length) return "clone_checkout_has_files"; + const untracked = nul(await git(clone, ["ls-files", "-z", "--others", "--exclude-standard"], { signal })); + const ignored = nul(await git(clone, ["ls-files", "-z", "--others", "--ignored", "--exclude-standard", "--directory"], { signal })); + if (untracked.length || ignored.some(file => !file.split("/").slice(0, -1).some(part => REGENERABLE_WORKSPACE_DIR_NAMES.has(part)))) return "clone_checkout_has_files"; + const refs = (await git(clone, ["for-each-ref", `--count=${CLONE_HISTORY_BUDGET + 1}`, "--format=%(objectname) %(refname)"], { signal })).split("\n").filter(Boolean); + if (refs.length > CLONE_HISTORY_BUDGET) return "clone_history_budget"; + if (refs.some(ref => !/^[0-9a-f]{40} refs\/(heads|tags|remotes)\//.test(ref))) return "unsupported_clone_refs"; + const reflog = (await git(clone, ["reflog", "show", "--all", "-n", String(CLONE_HISTORY_BUDGET + 1), "--format=%H %gD"], { signal }).catch(() => "")) + .split("\n").map(line => line.trim().split(" ")).filter(([sha]) => SHA.test(sha ?? "")); + if (reflog.length > CLONE_HISTORY_BUDGET) return "clone_history_budget"; + const local = [...refs.filter(ref => !/ refs\/remotes\//.test(ref)).map(ref => ref.split(" ")[0]!), + ...reflog.filter(([, selector]) => !(selector ?? "").startsWith("refs/remotes/")).map(([sha]) => sha!)]; + if (local.length === 0) return null; + const unique = shas(await git(clone, ["rev-list", `--max-count=${CLONE_HISTORY_BUDGET + 1}`, "--stdin"], { signal, input: [...new Set(local)].join("\n") + "\n--not\n--remotes\n" })); + if (unique.length > CLONE_HISTORY_BUDGET) return "clone_history_budget"; + const present = await presentCommits(base, unique, signal); + return unique.some(sha => !present.has(sha)) ? "unique_clone_history" : null; +} + +async function cleanupRehomedSource(db: Db, tx: Tx, row: Workspace, result: IsolatedRepositoryRehomeResult, + options: { dryRun?: boolean; now?: Date; signal?: AbortSignal }): Promise { + const rehome = readRehome(row); + if (!rehome || rehome.cleanedAt) return result; + result.from ??= rehome.from; result.to ??= rehome.to; + if (row.cwd !== rehome.to || !row.sourceIssueId) throw new Error("workspace_changed"); + const clone = isolatedRemoteSourcePath(row.companyId, row.sourceIssueId); + const parent = path.dirname(clone); + const worktreeTrash = path.join(parent, `.rehomed-worktree-${row.id}`); + const cloneTrash = path.join(parent, `.rehomed-checkout-${row.id}`); + const base = await sharedManagedBase(row); + const from = await canonicalRepositoryRoot(rehome.from); + const cloneReal = await exists(clone) ? await fs.realpath(clone) : null; + const keep = async (reason: string, retained: boolean) => { + result.cloneKeptReason = reason; + if (!result.rehomed) result.outcome = "clone_kept"; + if (retained && !options.dryRun) await markCleaned(db, row.id, options.now, reason); + return result; + }; + if (cloneReal && path.relative(cloneReal, from).startsWith("..")) return keep("unproven_worktree_admin", false); + if (options.dryRun) { + if (cloneReal) result.candidateBytes = await allocatedBytes(clone, () => { options.signal?.throwIfAborted(); return true; }).catch(() => 0); + result.outcome = "dry_run"; + return result; + } + if (!await rehomeEligible(tx, row)) return keep("live_or_unresolved_consumer", false); + if (cloneReal) { + await recoverRehomeLocks(clone, row.branchName, row.id); + const fromStat = await fs.lstat(from).catch(() => null); + if (fromStat) { + if (!fromStat.isDirectory() || fromStat.isSymbolicLink()) return keep("unsafe_worktree_path", false); + const common = (await git(from, ["rev-parse", "--path-format=absolute", "--git-common-dir"], { signal: options.signal }).catch(() => "")).trim(); + if (common !== path.join(cloneReal, ".git")) return keep("source_identity_changed", false); + result.reclaimedBytes += await allocatedBytes(from, () => true).catch(() => 0); + if (await exists(worktreeTrash)) await fs.rm(worktreeTrash, { recursive: true, force: true }); + await fs.rename(from, worktreeTrash); + } + await git(clone, ["worktree", "prune"], { signal: options.signal }); + } + if (await exists(worktreeTrash)) await fs.rm(worktreeTrash, { recursive: true, force: true }); + if (cloneReal) { + const consumer = await cloneConsumers(tx, row, clone); + if (consumer) return keep(consumer, consumer === "base_has_consumers"); + const disposable = await cloneDisposable(clone, base, options.signal); + if (disposable) return keep(disposable, false); + const bytes = await allocatedBytes(clone, () => true).catch(() => 0); + if (await exists(cloneTrash)) await fs.rm(cloneTrash, { recursive: true, force: true }); + await fs.rename(clone, cloneTrash); + result.reclaimedBytes += bytes; + } + if (await exists(cloneTrash)) { + await fs.rm(cloneTrash, { recursive: true, force: true }); + result.cloneRemoved = true; + } + await fs.rmdir(parent).catch(() => undefined); + if (!result.rehomed) result.outcome = "cleaned"; + await markCleaned(db, row.id, options.now); + return result; +} + +async function markCleaned(db: Db, workspaceId: string, now?: Date, retained?: string) { + const patch = { cleanedAt: (now ?? new Date()).toISOString(), ...(retained ? { cloneRetained: retained } : {}) }; + await db.update(executionWorkspaces).set({ + metadata: sql`jsonb_set(${executionWorkspaces.metadata}, ${`{${ISOLATED_REPOSITORY_REHOME_KEY}}`}::text[], + (${executionWorkspaces.metadata}->${ISOLATED_REPOSITORY_REHOME_KEY}) || ${JSON.stringify(patch)}::jsonb)`, + }).where(eq(executionWorkspaces.id, workspaceId)); +} + +export async function rehomeExpiredIsolatedRestoration(db: Db, snapshot: Workspace): Promise { + if (!CLOSED_STATUSES.includes(snapshot.status) || snapshot.metadata?.isolatedRemoteSource === undefined || !snapshot.branchName) return snapshot; + try { + const recipe = readGitCacheRestoration(snapshot); + if (!recipe || recipe.sourceKind !== "isolated_remote" || !readGitCacheExpiration(snapshot)?.workspaceRemoved || await exists(recipe.worktreePath)) return snapshot; + if (await canonicalRepositoryRoot(recipe.worktreePath) !== path.join(await canonicalRepositoryRoot(recipe.sourcePath), standardWorktreeSuffix(recipe.branchName))) return snapshot; + const base = await sharedManagedBase(snapshot); + const target = await targetWorktree(db, snapshot, base, recipe.branchName); + const tip = (await git(base, ["rev-parse", "--verify", "-q", `refs/heads/${recipe.branchName}^{commit}`]).catch(() => "")).trim(); + if (tip && tip !== recipe.commit) return snapshot; + if ((recipe.contents?.checkouts ?? []).some(checkout => checkout.worktree && checkout.branchName)) return snapshot; + if (recipe.restorePoint && !recipe.restorePoint.headOnRemote && !recipe.restorePoint.expiredAt) { + const { manifest } = await loadGitRestorePoint({ storage: getStorageService(), workspace: { companyId: snapshot.companyId, id: snapshot.id }, ref: recipe.restorePoint }); + const present = await presentCommits(base, manifest.prerequisites); + if (manifest.prerequisites.some(sha => !present.has(sha))) return snapshot; + } + await finishGitCacheTrash(snapshot); + return await withExecutionWorkspaceLifecycleLock(db, snapshot.id, async tx => { + const [row] = await tx.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, snapshot.id)).for("update"); + if (!row || row.updatedAt.getTime() !== snapshot.updatedAt.getTime()) return row ?? snapshot; + const metadata: Record = { ...row.metadata }; + delete metadata.isolatedRemoteSource; + delete metadata.gitCacheExpiration; + metadata[GIT_CACHE_RESTORATION_KEY] = { ...recipe, sourceKind: "managed_project", sourcePath: base, worktreePath: target } satisfies GitCacheRestoration; + metadata[ISOLATED_REPOSITORY_REHOME_KEY] = { at: new Date().toISOString(), from: recipe.worktreePath, to: target, cleanedAt: null } satisfies IsolatedRepositoryRehomeRecord; + const [updated] = await tx.update(executionWorkspaces).set({ cwd: target, providerRef: row.providerRef ? target : null, metadata, updatedAt: new Date() }) + .where(eq(executionWorkspaces.id, row.id)).returning(); + return updated ?? row; + }); + } catch { return snapshot; } +}