diff --git a/doc/recoverable-disk-cleanup.md b/doc/recoverable-disk-cleanup.md index 5642e560250..1fa8a1a892e 100644 --- a/doc/recoverable-disk-cleanup.md +++ b/doc/recoverable-disk-cleanup.md @@ -428,7 +428,9 @@ restore can interleave. A workspace is moved only when: - its Git state is transferable: no stash or replacement refs, no in-progress merge, rebase or other worktree-specific state, no hidden index flags, no shallow history, and nothing a restore point refuses (submodules, filtered paths, special files, untracked - nested repositories, oversized contents). + nested repositories). The transfer is local, so it uses much larger restore point limits + than uploads do: 500,000 files, 1 GiB per file and in total, a 1.5 GiB bundle and + 1,000,000 commits. The move captures a restore point of the old worktree against commits both repositories share, imports it into the shared clone, creates or fast-forwards the issue branch, adds @@ -461,16 +463,48 @@ rollback stops with `rehome_rollback_conflict` and keeps the new worktree and th neither copy is lost; later attempts retry the rollback once the conflict is resolved. A new worktree is removed only when the move created it: a target that appears from elsewhere is never deleted. -After the update, the old worktree is removed and the clone is renamed to trash and -deleted when nothing else needs it: no other workspace or project workspace refers to it -(a workspace already expired with its worktree removed does not count), no runtime -service runs in it, it has no other worktree, its own checkout has no changes or files -other than regenerable ones, it has only branch, tag and remote-tracking refs, and every -commit its local refs and reflogs hold is either on a remote-tracking ref or present in the -shared clone. A clone with more than 5,000 refs, reflog entries or such commits is kept -(`clone_history_budget`). A clone another workspace still uses is kept (`cloneRetained`); that -workspace's own move removes it later. If the server stops after the update, a later sweep -finishes the removal and sets `cleanedAt`, also after the workspace has been archived. +After the update, the old worktree is removed (its ownership is proven by `git rev-parse` +or, when its registration in the clone was already pruned, by its `.git` pointer into the +clone's `worktrees` directory). The clone is then renamed to trash and deleted when nothing +else needs it and everything in it is reproducible: + +- no other workspace or project workspace refers to it (a workspace already expired with its + worktree removed does not count) and no runtime service runs in it; +- after `git worktree prune`, it has no worktree besides its own checkout; +- its own checkout has no tracked changes and no untracked files (only Paperclip's + compaction trash is allowed, even inside a directory with a regenerable name), and its + ignored files are only inside regenerable directories (such as `node_modules`) or Paperclip's compaction + trash. Empty directories, such as the `.paperclip/worktrees` parent left by the old + worktree, do not count; +- it has only branch, tag and remote-tracking refs (no stash, notes or replacement refs); +- every commit that its local branches, local tags, reflogs (other than remote-tracking + ones) or `ORIG_HEAD` hold is on one of its remote-tracking refs, present in the shared + clone, or found on the remote. The remote is read fresh (branches and tags) with the + company's Git identity, falling back to anonymous access, and only when some commit is in + neither place. `HEAD` itself counts too, so a detached checkout is covered. Remote branches and release tags that the shared clone never fetched do + not keep a clone. + +A clone with more than 5,000 refs, reflog entries or such commits is kept +(`clone_history_budget`), and so is one with more than 1,000 ignored directories outside +regenerable ones (`clone_file_budget`); those directories are read in one Git call. The log entry names up to five paths, worktrees, refs or commits +that kept a clone (`cloneKeptPaths`). A clone another workspace still uses is kept +(`cloneRetained`); that workspace's own move removes it later. If the server stops after +the update, a later sweep finishes the removal and sets `cleanedAt`, also after the +workspace has been archived. + +The reaper also sweeps orphaned clones under `isolated-repository-sources/` once per server +start: clones that no execution workspace refers to (no open workspace with matching +`isolatedRemoteSource`, and no closed one that would need the clone to reopen) and that no +pending move or cleanup still owns. The repository must be proven by the database: the +workspaces of that issue must record exactly one repository URL, and the clone's `origin` +must equal it (`unproven_repository` or `source_identity_changed` otherwise), so a rewritten +`origin` is never fetched or trusted. It applies the same reproducibility check, using the +shared clone of a workspace of the same issue and repository when one exists, and logs +`event: "execution_workspace.isolated_source_orphan_cleanup"`. It inspects at most eight +issue directories per reaper sweep and continues from that position in the next sweep, so a +slow remote cannot hold up other workspace cleanup. When the walk ends and it kept any +orphan, it looks again a day later; otherwise it registers no further work. It also deletes leftover +trash directories from interrupted removals. Each attempt logs `event: "execution_workspace.isolated_source_rehome"` with the outcome, skip reason, candidate bytes and reclaimed bytes. A workspace that is skipped, or whose diff --git a/server/src/__tests__/isolated-repository-rehome.test.ts b/server/src/__tests__/isolated-repository-rehome.test.ts index 6631ebfc0c2..889b0ffa63f 100644 --- a/server/src/__tests__/isolated-repository-rehome.test.ts +++ b/server/src/__tests__/isolated-repository-rehome.test.ts @@ -14,8 +14,8 @@ import { isolatedRemoteSourcePath, materializeIsolatedRemoteWorkspaceSource } fr import { realizeExecutionWorkspace } from "../services/workspace-runtime.js"; import { executionWorkspaceService } from "../services/execution-workspaces.js"; import { expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js"; -import { currentRestorePointSnapshot } from "../services/git-restore-point.js"; -import { rehomeIsolatedRepositoryWorkspace } from "../services/isolated-repository-rehome.js"; +import { captureRestorePoint, currentRestorePointSnapshot } from "../services/git-restore-point.js"; +import { rehomeIsolatedRepositoryWorkspace, sweepOrphanedIsolatedRepositorySources } from "../services/isolated-repository-rehome.js"; import { addNestedRepositoryWorktree, nestedRepositoryBaseCandidates, nestedWorktreeBranchPrefix, readLinkedNestedCheckout } from "../services/nested-repository-worktrees.js"; import { getStorageService } from "../storage/index.js"; @@ -44,6 +44,7 @@ describeEmbeddedPostgres("isolated repository source re-home", () => { afterEach(async () => { await db.delete(executionWorkspaceRestorePoints); await db.execute(sql`TRUNCATE TABLE companies CASCADE`); + await fs.rm(path.join(root, "home", "instances", "default", "isolated-repository-sources"), { recursive: true, force: true }); }); afterAll(async () => { vi.unstubAllEnvs(); @@ -472,4 +473,173 @@ describeEmbeddedPostgres("isolated repository source re-home", () => { expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed); expect(await fs.readFile(path.join(f.cwd, "notes.txt"), "utf8")).toBe("untracked work\n"); }); + + async function crashedRehome(f: Awaited>) { + const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { afterDatabaseUpdate: async () => { throw new Error("simulated crash"); } } }); + expect(result).toMatchObject({ rehomed: true, cloneRemoved: false }); + return result; + } + + async function publishUnfetchedHistory(f: Awaited>) { + git(f.author, "checkout", "-q", "-b", "feature"); + await fs.writeFile(path.join(f.author, "feature.txt"), "feature\n"); git(f.author, "add", "."); git(f.author, "commit", "-qm", "feature"); + const feature = git(f.author, "rev-parse", "HEAD"); + git(f.author, "checkout", "-q", "main"); + const release = git(f.author, "commit-tree", "HEAD^{tree}", "-p", "HEAD", "-m", "release"); + git(f.author, "tag", "v1", release); + git(f.author, "push", "-q", f.remote, "feature", "v1"); + git(f.clone, "fetch", "-q", "--tags", "origin"); + return { feature, release }; + } + + it("deletes a clean clone whose only extra history is remote branches and tags the shared clone never fetched", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const { feature, release } = await publishUnfetchedHistory(f); + await fs.appendFile(path.join(f.clone, ".git", "info", "exclude"), "\n.paperclip/\n"); + await crashedRehome(f); + expect(() => git(f.sharedPath, "cat-file", "-e", `${release}^{commit}`)).toThrow(); + expect(() => git(f.sharedPath, "cat-file", "-e", `${feature}^{commit}`)).toThrow(); + expect(git(f.clone, "rev-parse", "refs/tags/v1")).toBe(release); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ outcome: "cleaned", cloneRemoved: true, cloneKeptReason: null }); + expect(await exists(f.clone)).toBe(false); + }); + + it("keeps a clone with a local branch commit that is neither upstream nor in the shared clone", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + const local = git(f.clone, "commit-tree", "HEAD^{tree}", "-p", "HEAD", "-m", "local only"); + git(f.clone, "branch", "local-work", local); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "unique_clone_history", cloneKeptPaths: [local] }); + expect(await exists(f.clone)).toBe(true); + expect((await f.row()).metadata?.isolatedRepositoryRehome).toMatchObject({ cleanedAt: null }); + }); + + it("keeps a clone whose detached HEAD holds the only copy of a commit", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + const detached = git(f.clone, "commit-tree", "HEAD^{tree}", "-p", "HEAD", "-m", "detached only"); + git(f.clone, "checkout", "-q", "--detach", detached); + git(f.clone, "reflog", "expire", "--expire=now", "--all"); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "unique_clone_history", cloneKeptPaths: [detached] }); + expect(await exists(f.clone)).toBe(true); + }); + + it("keeps a clone whose own checkout has changes or unexplained ignored files", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + await fs.writeFile(path.join(f.clone, "marker.txt"), "edited in the clone\n"); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "clone_checkout_has_files", cloneKeptPaths: ["marker.txt"] }); + git(f.clone, "checkout", "--", "marker.txt"); + await fs.writeFile(path.join(f.clone, "cache.tsbuildinfo"), "ignored but not regenerable\n"); + await fs.mkdir(path.join(f.clone, "node_modules", "dep"), { recursive: true }); + await fs.writeFile(path.join(f.clone, "node_modules", "dep", "index.js"), "regenerable\n"); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "clone_checkout_has_files", cloneKeptPaths: ["cache.tsbuildinfo"] }); + await fs.rm(path.join(f.clone, "cache.tsbuildinfo")); + await fs.mkdir(path.join(f.clone, "build")); + await fs.writeFile(path.join(f.clone, "build", "notes.txt"), "untracked, not ignored\n"); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "clone_checkout_has_files", cloneKeptPaths: ["build/"] }); + await fs.rm(path.join(f.clone, "build"), { recursive: true }); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ outcome: "cleaned", cloneRemoved: true }); + }); + + it("keeps a clone with another registered worktree", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + const extra = path.join(path.dirname(f.remote), "extra-worktree"); + git(f.clone, "worktree", "add", "-q", "-b", "extra", extra); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "base_has_worktrees" }); + expect(await exists(f.clone)).toBe(true); + }); + + it("removes the old worktree and clone when the old worktree's registration was already pruned", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + await fs.rm(path.join(f.clone, ".git", "worktrees", "fixture-task"), { recursive: true }); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ outcome: "cleaned", cloneRemoved: true, cloneKeptReason: null }); + expect(await exists(f.cwd)).toBe(false); + }); + + it("sweeps orphaned clones that no workspace needs and keeps those with unique history", async () => { + const orphan = await fixture(); + await makeWork(orphan.cwd); + await publishUnfetchedHistory(orphan); + await crashedRehome(orphan); + git(orphan.clone, "worktree", "remove", "--force", orphan.cwd); + const current = await orphan.row(); + await db.update(executionWorkspaces).set({ metadata: { ...current.metadata, isolatedRepositoryRehome: { + ...current.metadata?.isolatedRepositoryRehome as Record, cleanedAt: new Date().toISOString() } } }).where(eq(executionWorkspaces.id, orphan.workspaceId)); + const unknown = await fixture(); + await makeWork(unknown.cwd); + await crashedRehome(unknown); + git(unknown.clone, "worktree", "remove", "--force", unknown.cwd); + await db.delete(executionWorkspaces).where(eq(executionWorkspaces.id, unknown.workspaceId)); + + const first = await sweepOrphanedIsolatedRepositorySources(db, { dryRun: true, limit: 1 }); + expect(first.results).toHaveLength(1); + expect(first.next).toEqual(expect.any(String)); + const second = await sweepOrphanedIsolatedRepositorySources(db, { dryRun: true, limit: 1, after: first.next }); + expect(second).toMatchObject({ results: [expect.anything()], next: null }); + expect(new Set([...first.results, ...second.results].map(result => result.issueId))).toEqual(new Set([orphan.issueId, unknown.issueId])); + const dryRun = executionWorkspaceService(db, { gitCacheExpirationEnabled: true, isolatedSourceRehomeEnabled: true, isolatedSourceRehomeDryRun: true }); + await dryRun.sweepTerminalWorkspaces(); + expect(await exists(orphan.clone)).toBe(true); + + const service = executionWorkspaceService(db, { gitCacheExpirationEnabled: true, isolatedSourceRehomeEnabled: true }); + await service.sweepTerminalWorkspaces(); + expect(await exists(orphan.clone)).toBe(false); + expect(await exists(unknown.clone)).toBe(true); + expect(await service.nextTerminalWorkspaceSweepAt()).toBeGreaterThan(Date.now() + 23 * 60 * 60 * 1000); + }); + + it("re-homes a workspace whose changes exceed the default restore point limits", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const large = Buffer.alloc(65 * 1024 * 1024, 7); + await fs.writeFile(path.join(f.cwd, "notes", "large.bin"), large); + await fs.writeFile(path.join(f.cwd, "large.tsbuildinfo"), large); + await expect(captureRestorePoint({ cwd: f.cwd, upstreamCommits: [] })).rejects.toMatchObject({ reason: "restore_point_too_large" }); + const before = await currentRestorePointSnapshot(f.cwd); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: true, cloneRemoved: true, reason: null }); + expect(await currentRestorePointSnapshot(f.target)).toEqual(before); + expect((await fs.readFile(path.join(f.target, "notes", "large.bin"))).equals(large)).toBe(true); + expect((await fs.readFile(path.join(f.target, "large.tsbuildinfo"))).equals(large)).toBe(true); + }); + + it("keeps an orphaned clone whose origin no longer matches the recorded repository", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + git(f.clone, "worktree", "remove", "--force", f.cwd); + const current = await f.row(); + await db.update(executionWorkspaces).set({ metadata: { ...current.metadata, isolatedRepositoryRehome: { + ...current.metadata?.isolatedRepositoryRehome as Record, cleanedAt: new Date().toISOString() } } }).where(eq(executionWorkspaces.id, f.workspaceId)); + const decoy = path.join(path.dirname(f.remote), "decoy.git"); + git(path.dirname(f.remote), "clone", "-q", "--bare", f.clone, decoy); + git(f.clone, "remote", "set-url", "origin", pathToFileURL(decoy).href); + const { results } = await sweepOrphanedIsolatedRepositorySources(db); + expect(results).toMatchObject([{ issueId: f.issueId, removed: false, reason: "source_identity_changed" }]); + expect(await exists(f.clone)).toBe(true); + await db.delete(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId)); + expect((await sweepOrphanedIsolatedRepositorySources(db)).results).toMatchObject([{ issueId: f.issueId, removed: false, reason: "unproven_repository" }]); + }); + + it("keeps a clone with more ignored directories than the cleanup budget", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + await fs.appendFile(path.join(f.clone, ".git", "info", "exclude"), "\nscratch-*/\n"); + for (let index = 0; index <= 1_000; index += 1) { + await fs.mkdir(path.join(f.clone, `scratch-${index}`, "node_modules"), { recursive: true }); + await fs.writeFile(path.join(f.clone, `scratch-${index}`, "node_modules", "x.js"), "x\n"); + } + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "clone_file_budget" }); + for (let index = 1; index <= 1_000; index += 1) await fs.rm(path.join(f.clone, `scratch-${index}`), { recursive: true }); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ outcome: "cleaned", cloneRemoved: true }); + }); }); diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index d5c456c8b06..9d42edcc6e1 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -6,7 +6,7 @@ import { measureRunPhase } from "./execution-preparation-timing.js"; import { assertExecutionWorkspaceRecordAccess, assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-access.js"; import { verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js"; import { expireReconstructableGitWorkspace, gitWorkspaceLastActivityAt, hasPendingGitExpiration } from "./git-workspace-expiration.js"; -import { hasPendingIsolatedRepositoryRehomeCleanup, isIsolatedRepositoryRehomeCandidate, rehomeExpiredIsolatedRestoration, rehomeIsolatedRepositoryWorkspace } from "./isolated-repository-rehome.js"; +import { hasPendingIsolatedRepositoryRehomeCleanup, isIsolatedRepositoryRehomeCandidate, rehomeExpiredIsolatedRestoration, rehomeIsolatedRepositoryWorkspace, sweepOrphanedIsolatedRepositorySources } from "./isolated-repository-rehome.js"; import { hasPendingNestedCloneMigration, isNestedCloneMigrationCandidate, nextNestedCloneMigrationBatch, pendingNestedCloneMigrationNames, listNestedCloneMigrationNames, migrateNestedRepositoryClone, NESTED_CLONE_MIGRATION_RETRY_REASONS, type NestedCloneMigrationRotation } from "./nested-clone-migration.js"; import type { StorageService } from "../storage/types.js"; @@ -1756,6 +1756,10 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic const nestedCloneMigrationNotBefore = new Map(); const nestedCloneMigrationRetryAt = new Map(); const nestedCloneMigrationRotations = new Map(); + const ISOLATED_ORPHAN_RETRY_MS = 24 * 60 * 60 * 1000; + let isolatedOrphanSweepDueAt = 0; + let isolatedOrphanCursor: string | null = null; + let isolatedOrphanKept = false; let terminalSweepRescanRequested = false; // The reaper freezes an upper bound on updatedAt at the start of each // rotation. The scan only reads candidates at or below the bound, so a steady @@ -3283,6 +3287,25 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic isolatedRehomeDeadlines.clear(); nestedCloneMigrationDeadlines.clear(); terminalSweepRescanRequested = false; + if (opts.gitCacheExpirationEnabled && opts.isolatedSourceRehomeEnabled && isolatedOrphanSweepDueAt <= now().getTime()) { + const dryRun = opts.isolatedSourceRehomeDryRun ?? false; + const batch = await sweepOrphanedIsolatedRepositorySources(db, { dryRun, signal, after: isolatedOrphanCursor }).catch((error: unknown) => { + signal?.throwIfAborted(); + logger.warn({ err: error }, "isolated repository source orphan sweep failed; it retries later"); + return null; + }); + for (const orphan of batch?.results ?? []) { + logger.info({ event: "execution_workspace.isolated_source_orphan_cleanup", dryRun, ...orphan }, "Orphaned isolated repository source inspected"); + } + isolatedOrphanKept ||= !batch || batch.results.some(orphan => !orphan.removed); + isolatedOrphanCursor = batch?.next ?? null; + if (isolatedOrphanCursor) isolatedOrphanSweepDueAt = now().getTime(); + else { + isolatedOrphanSweepDueAt = isolatedOrphanKept ? now().getTime() + ISOLATED_ORPHAN_RETRY_MS : Number.POSITIVE_INFINITY; + isolatedOrphanKept = false; + } + } + if (Number.isFinite(isolatedOrphanSweepDueAt) && isolatedOrphanSweepDueAt > 0) isolatedRehomeDeadlines.set("isolated-source-orphans", isolatedOrphanSweepDueAt); } const boundary = terminalSweepBoundary; const boundaryFilter = boundary diff --git a/server/src/services/git-restore-point.ts b/server/src/services/git-restore-point.ts index 5c949c5208f..698b1395ade 100644 --- a/server/src/services/git-restore-point.ts +++ b/server/src/services/git-restore-point.ts @@ -1,4 +1,4 @@ -import { execFile as execFileCallback } from "node:child_process"; +import { execFile as execFileCallback, spawn } from "node:child_process"; import { createCipheriv, createDecipheriv, createHash, randomBytes } from "node:crypto"; import fs from "node:fs/promises"; import os from "node:os"; @@ -105,10 +105,7 @@ function isolatedEnv(extra: Record = {}) { } async function git(cwd: string, args: string[], options: GitOptions = {}): Promise { - const child = execFile("git", [ - "-c", "core.fsmonitor=false", "-c", "core.autocrlf=false", "-c", "core.safecrlf=false", - "-c", `core.hooksPath=${os.devNull}`, "-C", cwd, ...args, - ], { + const child = execFile("git", gitArgs(cwd, args), { env: isolatedEnv(options.env), encoding: "buffer", maxBuffer: options.maxBuffer ?? 64 * 1024 * 1024, @@ -120,6 +117,32 @@ async function git(cwd: string, args: string[], options: GitOptions = {}): Promi return stdout as Buffer; } +function gitArgs(cwd: string, args: string[]) { + return ["-c", "core.fsmonitor=false", "-c", "core.autocrlf=false", "-c", "core.safecrlf=false", "-c", `core.hooksPath=${os.devNull}`, "-C", cwd, ...args]; +} + +async function gitPipe(cwd: string, from: string[], to: string[], signal?: AbortSignal) { + const options = { env: isolatedEnv(), signal, timeout: 15 * 60_000 }; + const source = spawn("git", gitArgs(cwd, from), { ...options, stdio: ["ignore", "pipe", "pipe"] }); + const sink = spawn("git", gitArgs(cwd, to), { ...options, stdio: ["pipe", "ignore", "pipe"] }); + source.stdout.pipe(sink.stdin); + sink.stdin.on("error", () => {}); + const finished = (child: ReturnType) => new Promise((resolve, reject) => { + let stderr = ""; + child.stderr?.on("data", (chunk: Buffer) => { if (stderr.length < 4096) stderr += chunk.toString("utf8"); }); + child.on("error", reject); + child.on("close", (code) => code === 0 ? resolve() : reject(Object.assign(new Error(`git exited with ${code}`), { code, stderr }))); + }); + const done = [finished(source), finished(sink)]; + try { await Promise.all(done); } + catch (error) { + source.kill("SIGKILL"); + sink.kill("SIGKILL"); + await Promise.allSettled(done); + throw error; + } +} + async function gitText(cwd: string, args: string[], options: GitOptions = {}) { return (await git(cwd, args, options)).toString("utf8").trim(); } @@ -232,21 +255,21 @@ async function rawIgnoredTree(cwd: string, env: Record, temporar } async function writeIgnoredFiles(cwd: string, commit: string, signal?: AbortSignal) { - const listing = nulList(await git(cwd, ["ls-tree", "-r", "-z", `${commit}^{tree}`], { signal })); + const listing = nulList(await git(cwd, ["ls-tree", "-r", "-z", "--long", `${commit}^{tree}`], { signal })); const entries = listing.map((line) => { - const match = /^(100644|100755|120000) blob ([0-9a-f]{40})\t(.+)$/s.exec(line); + const match = /^(100644|100755|120000) blob ([0-9a-f]{40}) +(\d+)\t(.+)$/s.exec(line); if (!match) throw new RestorePointRestoreFailure("invalid_payload"); - const target = path.join(cwd, match[3]!); - if (path.isAbsolute(match[3]!) || match[3]!.split("/").includes("..") || !target.startsWith(`${cwd}${path.sep}`)) { + const target = path.join(cwd, match[4]!); + if (path.isAbsolute(match[4]!) || match[4]!.split("/").includes("..") || !target.startsWith(`${cwd}${path.sep}`)) { throw new RestorePointRestoreFailure("invalid_payload"); } - return { mode: match[1]!, blob: match[2]!, target }; + return { mode: match[1]!, blob: match[2]!, size: Number(match[3]), target }; }); for (const entry of entries) { if (await fs.lstat(entry.target).then(() => true, () => false)) throw new RestorePointRestoreFailure("ignored_files_conflict"); } const contents = await git(cwd, ["cat-file", "--batch"], { signal, input: entries.map((entry) => entry.blob).join("\n") + "\n", - maxBuffer: 2 * DEFAULT_RESTORE_POINT_LIMITS.maxTotalBytes }); + maxBuffer: entries.reduce((total, entry) => total + entry.size + 128, 4096) }); const blobs: Buffer[] = []; let offset = 0; for (const entry of entries) { @@ -487,8 +510,12 @@ export async function applyRestorePoint(input: { cwd: string; manifest: RestoreP const head = await gitText(cwd, ["rev-parse", "--verify", "HEAD^{commit}"], { signal: input.signal }); if (head !== manifest.head) throw new RestorePointRestoreFailure("head_mismatch"); if (manifest.index && manifest.worktree) { - const diff = await git(cwd, ["diff", "--binary", "--no-color", "--no-ext-diff", "--no-textconv", "--no-renames", "HEAD", manifest.worktree], { signal: input.signal }); - if (diff.length > 0) await git(cwd, ["apply", "--binary", "--whitespace=nowarn", "-"], { input: diff, signal: input.signal }); + const changed = await git(cwd, ["diff", "--quiet", "--no-ext-diff", "--no-textconv", "HEAD", manifest.worktree], { signal: input.signal }).then(() => false, (error: { code?: unknown }) => { + if (error.code === 1) return true; + throw error; + }); + if (changed) await gitPipe(cwd, ["diff", "--binary", "--no-color", "--no-ext-diff", "--no-textconv", "--no-renames", "HEAD", manifest.worktree], + ["apply", "--binary", "--whitespace=nowarn", "-"], input.signal); await git(cwd, ["read-tree", `${manifest.index}^{tree}`], { signal: input.signal }); } if (manifest.ignored) await writeIgnoredFiles(cwd, manifest.ignored, input.signal); diff --git a/server/src/services/git-workspace-restoration.ts b/server/src/services/git-workspace-restoration.ts index 67b7c08a67e..f71c1851e7c 100644 --- a/server/src/services/git-workspace-restoration.ts +++ b/server/src/services/git-workspace-restoration.ts @@ -402,7 +402,7 @@ async function tipsMissingUpstream(checkout: string, tips: string[], repoUrl: st return tips.filter(tip => missing.has(tip)).sort(); } -async function hasCommitsMissingUpstream(checkout: string, localTips: string[], repoUrl: string, +export async function hasCommitsMissingUpstream(checkout: string, localTips: string[], repoUrl: string, resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal, covered: string[] = []) { return (await commitsMissingUpstream(checkout, localTips, repoUrl, resolveGitAuth, signal, covered, 1)).length > 0; } diff --git a/server/src/services/isolated-repository-rehome.ts b/server/src/services/isolated-repository-rehome.ts index 1b342e52fbe..c8c02f85578 100644 --- a/server/src/services/isolated-repository-rehome.ts +++ b/server/src/services/isolated-repository-rehome.ts @@ -1,21 +1,21 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import { createHash } from "node:crypto"; +import { createHash, randomBytes } from "node:crypto"; import { execFile as execFileCallback } from "node:child_process"; import { promisify } from "node:util"; -import { and, eq, ne, sql } from "drizzle-orm"; +import { and, eq, ne, or, sql } from "drizzle-orm"; import { executionWorkspaces, projectWorkspaces, workspaceRuntimeServices, heartbeatRuns, nativeRunFinalizations, environmentLeases, withDedicatedDbConnection, type Db } from "@paperclipai/db"; import { PROJECT_REPOSITORIES_DIR } from "@paperclipai/adapter-utils/git-workspace-sync"; import { IsolatedRepositorySourceBusy, isolatedRemoteSourcePath, verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js"; import { acquireGitExpirationLocks, recoverGitExpirationLocks, type GitExpirationLockLocation } from "./git-workspace-expiration-locks.js"; -import { allocatedBytes, describeGitExpirationFailure, expirationScope, gitExpirationSkipReason, type GitExpirationFailure } from "./git-workspace-expiration.js"; -import { finishGitCacheTrash, managedSourcePaths, readGitCacheExpiration, readGitCacheRestoration, verifiedGitCacheSource, +import { allocatedBytes, describeGitExpirationFailure, expirationScope, gitExpirationSkipReason, withAnonymousGitFallback, type GitExpirationFailure } from "./git-workspace-expiration.js"; +import { finishGitCacheTrash, hasCommitsMissingUpstream, managedSourcePaths, readGitCacheExpiration, readGitCacheRestoration, verifiedGitCacheSource, GIT_CACHE_RESTORATION_KEY, type GitCacheRestoration } from "./git-workspace-restoration.js"; import { applyRestorePoint, captureRestorePoint, currentRestorePointSnapshot, importRestorePoint, restorePointSavesIgnoredFile, RestorePointRestoreFailure, RestorePointSkip, type RestorePointLimits, type RestorePointSnapshot } from "./git-restore-point.js"; -import { findRegenerableWorkspaceDirs, REGENERABLE_WORKSPACE_DIR_NAMES } from "./workspace-compaction.js"; +import { findRegenerableWorkspaceDirs, REGENERABLE_WORKSPACE_DIR_NAMES, WORKSPACE_COMPACTION_TRASH_DIR } from "./workspace-compaction.js"; import { ensureManagedRepositoriesIgnored } from "./managed-repository-checkout.js"; import { withWorkspaceLockKeepalive } from "./native-runtime/native-workspace-lock-keepalive.js"; import { canonicalRepositoryRoot } from "./execution-workspace-source-roots.js"; @@ -23,6 +23,8 @@ import { withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifec import { loadGitRestorePoint } from "./git-restore-point-storage.js"; import { nestedWorktreeBranchPrefix, readLinkedNestedCheckout, repairMovedNestedRepositoryWorktree, withNestedRepositoryBaseLock } from "./nested-repository-worktrees.js"; import { getStorageService } from "../storage/index.js"; +import { createGitRemoteAuthProvider } from "./git-credentials.js"; +import { resolvePaperclipInstanceRoot } from "../home-paths.js"; const execFile = promisify(execFileCallback); const SHA = /^[0-9a-f]{40}$/; @@ -31,7 +33,16 @@ const CLOSED_STATUSES = ["archived", "cleanup_failed"]; const ADMIN_ENTRIES = new Set(["HEAD", "ORIG_HEAD", "FETCH_HEAD", "index", "commondir", "gitdir", "logs", "refs", "COMMIT_EDITMSG"]); const MAX_BOUNDARY_COMMITS = 64; const CLONE_HISTORY_BUDGET = 5_000; +const CLONE_DIRECTORY_BUDGET = 1_000; const UNSAFE_BASE_CONFIG = /^(filter\.|include\.|includeif\.|core\.(fsmonitor|hookspath|attributesfile|worktree|gitproxy|sshcommand|askpass|editor|pager|alternaterefscommand)$|extensions\.worktreeconfig$)/; +const GIB = 1024 * 1024 * 1024; +const COMPACTION_TRASH_PATH = `.paperclip/worktrees/${WORKSPACE_COMPACTION_TRASH_DIR}`; +const SOURCE_ID = /^[a-zA-Z0-9_-]+$/; +const ORPHAN_HANDLED_ELSEWHERE = new Set(["base_has_consumers", "rehome_pending", "rehome_cleanup_pending"]); +const SOURCE_TRASH = /^\.(rehomed-checkout|rehomed-worktree|orphaned-checkout)-[a-zA-Z0-9_-]+$/; +export const ISOLATED_REHOME_RESTORE_POINT_LIMITS: RestorePointLimits = { + maxFiles: 500_000, maxFileBytes: GIB, maxTotalBytes: GIB, maxCommits: 1_000_000, maxBundleBytes: 1.5 * GIB, +}; export const ISOLATED_REPOSITORY_REHOME_KEY = "isolatedRepositoryRehome"; export const ISOLATED_REPOSITORY_REHOME_PENDING_KEY = "isolatedRepositoryRehomePending"; @@ -46,7 +57,7 @@ export type IsolatedRepositoryRehomeHooks = { }; export type IsolatedRepositoryRehomeResult = { outcome: "rehomed" | "dry_run" | "skipped" | "cleaned" | "clone_kept"; - rehomed: boolean; cloneRemoved: boolean; reason: string | null; cloneKeptReason: string | null; + rehomed: boolean; cloneRemoved: boolean; reason: string | null; cloneKeptReason: string | null; cloneKeptPaths?: string[]; candidateBytes: number; reclaimedBytes: number; from: string | null; to: string | null; failure?: GitExpirationFailure; }; @@ -419,7 +430,7 @@ export async function rehomeIsolatedRepositoryWorkspace(db: Db, snapshot: Worksp const beforeHistory = await historyKey(old.cwd, old.admin, options.signal); if (before.head !== old.head) throw new Error("head_changed"); const captured = await captureRestorePoint({ cwd: old.cwd, upstreamCommits: await sharedHistoryBoundary(old.cwd, base, old.head, options.signal), - limits: options.limits, signal: options.signal }); + limits: options.limits ?? ISOLATED_REHOME_RESTORE_POINT_LIMITS, signal: options.signal }); const manifest = captured ? await importRestorePoint({ repositoryCwd: base, payload: captured.payload, signal: options.signal }) : null; const reflog = shas(await git(old.cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal: options.signal }).catch(() => "")); const original = (await fs.readFile(path.join(old.admin, "ORIG_HEAD"), "utf8").catch(() => "")).trim(); @@ -496,10 +507,10 @@ export async function rehomeIsolatedRepositoryWorkspace(db: Db, snapshot: Worksp } } -async function cloneConsumers(db: Db | Tx, workspace: Workspace, clone: string) { - const others = await db.select().from(executionWorkspaces).where(and(ne(executionWorkspaces.id, workspace.id), - sql`((${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'companyId' = ${workspace.companyId} - AND ${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'issueId' = ${workspace.sourceIssueId}) +async function cloneConsumers(db: Db | Tx, companyId: string, issueId: string, excludeId: string | null, clone: string) { + const others = await db.select().from(executionWorkspaces).where(and(excludeId ? ne(executionWorkspaces.id, excludeId) : undefined, + sql`((${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'companyId' = ${companyId} + AND ${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'issueId' = ${issueId}) OR ${executionWorkspaces.cwd} = ${clone} OR ${executionWorkspaces.providerRef} = ${clone} OR left(${executionWorkspaces.cwd}, length(${clone}) + 1) = ${clone} || '/')`)); const expired = (row: Workspace) => { @@ -516,26 +527,79 @@ async function cloneConsumers(db: Db | Tx, workspace: Workspace, clone: string) return service ? "base_has_live_services" : null; } -async function cloneDisposable(clone: string, base: string, signal?: AbortSignal) { - const worktrees = (await git(clone, ["worktree", "list", "--porcelain"], { signal })).split("\n").filter(line => line.startsWith("worktree ")); - if (worktrees.length !== 1) return "base_has_worktrees"; - if ((await git(clone, ["status", "--porcelain=v1", "-z", "--untracked-files=no"], { signal })).length) return "clone_checkout_has_files"; - const untracked = nul(await git(clone, ["ls-files", "-z", "--others", "--exclude-standard"], { signal })); - const ignored = nul(await git(clone, ["ls-files", "-z", "--others", "--ignored", "--exclude-standard", "--directory"], { signal })); - if (untracked.length || ignored.some(file => !file.split("/").slice(0, -1).some(part => REGENERABLE_WORKSPACE_DIR_NAMES.has(part)))) return "clone_checkout_has_files"; - const refs = (await git(clone, ["for-each-ref", `--count=${CLONE_HISTORY_BUDGET + 1}`, "--format=%(objectname) %(refname)"], { signal })).split("\n").filter(Boolean); - if (refs.length > CLONE_HISTORY_BUDGET) return "clone_history_budget"; - if (refs.some(ref => !/^[0-9a-f]{40} refs\/(heads|tags|remotes)\//.test(ref))) return "unsupported_clone_refs"; - const reflog = (await git(clone, ["reflog", "show", "--all", "-n", String(CLONE_HISTORY_BUDGET + 1), "--format=%H %gD"], { signal }).catch(() => "")) - .split("\n").map(line => line.trim().split(" ")).filter(([sha]) => SHA.test(sha ?? "")); - if (reflog.length > CLONE_HISTORY_BUDGET) return "clone_history_budget"; - const local = [...refs.filter(ref => !/ refs\/remotes\//.test(ref)).map(ref => ref.split(" ")[0]!), - ...reflog.filter(([, selector]) => !(selector ?? "").startsWith("refs/remotes/")).map(([sha]) => sha!)]; - if (local.length === 0) return null; - const unique = shas(await git(clone, ["rev-list", `--max-count=${CLONE_HISTORY_BUDGET + 1}`, "--stdin"], { signal, input: [...new Set(local)].join("\n") + "\n--not\n--remotes\n" })); - if (unique.length > CLONE_HISTORY_BUDGET) return "clone_history_budget"; - const present = await presentCommits(base, unique, signal); - return unique.some(sha => !present.has(sha)) ? "unique_clone_history" : null; +function reproducibleCheckoutPath(file: string) { + const parts = file.split("/").filter(Boolean); + const directories = file.endsWith("/") ? parts : parts.slice(0, -1); + return directories.some(part => REGENERABLE_WORKSPACE_DIR_NAMES.has(part)) + || file === `${COMPACTION_TRASH_PATH}/` || file.startsWith(`${COMPACTION_TRASH_PATH}/`); +} + +type CloneCheck = { reason: string | null; paths: string[] }; +type MissingUpstream = (commits: string[], covered: string[]) => Promise; + +async function cloneDisposable(clone: string, base: string | null, missingUpstream: MissingUpstream, signal?: AbortSignal): Promise { + const kept = (reason: string, paths: string[] = []): CloneCheck => ({ reason, paths: paths.slice(0, 5) }); + try { + await git(clone, ["worktree", "prune"], { signal }); + const worktrees = (await git(clone, ["worktree", "list", "--porcelain"], { signal })).split("\n").filter(line => line.startsWith("worktree ")); + if (worktrees.length !== 1) return kept("base_has_worktrees", worktrees.slice(1).map(line => line.slice(9))); + const tracked = nul(await git(clone, ["status", "--porcelain=v1", "-z", "--untracked-files=no"], { signal })); + if (tracked.length) return kept("clone_checkout_has_files", tracked.map(entry => entry.slice(3))); + const untracked = nul(await git(clone, ["ls-files", "-z", "--others", "--exclude-standard", "--directory", "--no-empty-directory"], { signal })) + .filter(file => file !== `${COMPACTION_TRASH_PATH}/` && !file.startsWith(`${COMPACTION_TRASH_PATH}/`)); + if (untracked.length) return kept("clone_checkout_has_files", untracked); + const ignored = nul(await git(clone, ["ls-files", "-z", "--others", "--ignored", "--exclude-standard", "--directory", "--no-empty-directory"], { signal })) + .filter(entry => !reproducibleCheckoutPath(entry)); + const ignoredFiles = ignored.filter(entry => !entry.endsWith("/")); + if (ignoredFiles.length) return kept("clone_checkout_has_files", ignoredFiles); + const ignoredDirectories = ignored.filter(entry => entry.endsWith("/")); + if (ignoredDirectories.length > CLONE_DIRECTORY_BUDGET) return kept("clone_file_budget"); + if (ignoredDirectories.length) { + const unexplained = nul(await git(clone, ["--literal-pathspecs", "ls-files", "-z", "--others", "--ignored", "--exclude-standard", "--", ...ignoredDirectories], { signal })) + .filter(file => !reproducibleCheckoutPath(file)); + if (unexplained.length) return kept("clone_checkout_has_files", unexplained); + } + const refs = (await git(clone, ["for-each-ref", `--count=${CLONE_HISTORY_BUDGET + 1}`, "--format=%(objectname) %(refname)"], { signal })).split("\n").filter(Boolean); + if (refs.length > CLONE_HISTORY_BUDGET) return kept("clone_history_budget"); + const unexpected = refs.filter(ref => !/^[0-9a-f]{40} refs\/(heads|tags|remotes)\//.test(ref)); + if (unexpected.length) return kept("unsupported_clone_refs", unexpected.map(ref => ref.slice(41))); + const reflog = (await git(clone, ["reflog", "show", "--all", "-n", String(CLONE_HISTORY_BUDGET + 1), "--format=%H %gD"], { signal }).catch(() => "")) + .split("\n").map(line => line.trim().split(" ")).filter(([sha]) => SHA.test(sha ?? "")); + if (reflog.length > CLONE_HISTORY_BUDGET) return kept("clone_history_budget"); + const original = (await fs.readFile(path.join(clone, ".git", "ORIG_HEAD"), "utf8").catch(() => "")).trim(); + const head = (await git(clone, ["rev-parse", "--verify", "-q", "HEAD^{commit}"], { signal }).catch(() => "")).trim(); + const tips = [...new Set([...refs.filter(ref => !/ refs\/remotes\//.test(ref)).map(ref => ref.split(" ")[0]!), + ...reflog.filter(([, selector]) => !(selector ?? "").startsWith("refs/remotes/")).map(([sha]) => sha!), ...(SHA.test(original) ? [original] : []), ...(SHA.test(head) ? [head] : [])])]; + const tipsInBase = base ? await presentCommits(base, tips, signal) : new Set(); + const pending = tips.filter(sha => !tipsInBase.has(sha)); + if (pending.length === 0) return { reason: null, paths: [] }; + const unique = shas(await git(clone, ["rev-list", `--max-count=${CLONE_HISTORY_BUDGET + 1}`, "--stdin"], { signal, + input: [...pending, "--not", "--remotes", ...tipsInBase].join("\n") + "\n" })); + if (unique.length > CLONE_HISTORY_BUDGET) return kept("clone_history_budget"); + const uniqueInBase = base ? await presentCommits(base, unique, signal) : new Set(); + const missing = unique.filter(sha => !uniqueInBase.has(sha)); + if (missing.length === 0) return { reason: null, paths: [] }; + return await missingUpstream(missing, [...tipsInBase, ...uniqueInBase]) ? kept("unique_clone_history", missing) : { reason: null, paths: [] }; + } catch (error) { + signal?.throwIfAborted(); + return kept(gitExpirationSkipReason(error)); + } +} + +async function worktreeOfClone(worktree: string, cloneReal: string, signal?: AbortSignal) { + const admins = path.join(cloneReal, ".git", "worktrees"); + const common = (await git(worktree, ["rev-parse", "--path-format=absolute", "--git-common-dir"], { signal }).catch(() => "")).trim(); + if (common) return common === path.join(cloneReal, ".git"); + const marker = await fs.lstat(path.join(worktree, ".git")).catch(() => null); + if (!marker?.isFile() || marker.size > 4096) return false; + const pointer = /^gitdir: (.+)\n?$/.exec(await fs.readFile(path.join(worktree, ".git"), "utf8"))?.[1]; + if (!pointer) return false; + return path.dirname(await canonicalRepositoryRoot(path.resolve(worktree, pointer))) === admins; +} + +function upstreamCheck(db: Db, companyId: string, issueId: string, clone: string, repoUrl: string, signal?: AbortSignal): MissingUpstream { + const auth = withAnonymousGitFallback(createGitRemoteAuthProvider(db, companyId, { issueId })); + return (commits, covered) => hasCommitsMissingUpstream(clone, commits, repoUrl, auth, signal, covered); } async function cleanupRehomedSource(db: Db, tx: Tx, row: Workspace, result: IsolatedRepositoryRehomeResult, @@ -569,8 +633,7 @@ async function cleanupRehomedSource(db: Db, tx: Tx, row: Workspace, result: Isol const fromStat = await fs.lstat(from).catch(() => null); if (fromStat) { if (!fromStat.isDirectory() || fromStat.isSymbolicLink()) return keep("unsafe_worktree_path", false); - const common = (await git(from, ["rev-parse", "--path-format=absolute", "--git-common-dir"], { signal: options.signal }).catch(() => "")).trim(); - if (common !== path.join(cloneReal, ".git")) return keep("source_identity_changed", false); + if (!await worktreeOfClone(from, cloneReal, options.signal)) return keep("source_identity_changed", false); result.reclaimedBytes += await allocatedBytes(from, () => true).catch(() => 0); if (await exists(worktreeTrash)) await fs.rm(worktreeTrash, { recursive: true, force: true }); await fs.rename(from, worktreeTrash); @@ -579,10 +642,15 @@ async function cleanupRehomedSource(db: Db, tx: Tx, row: Workspace, result: Isol } if (await exists(worktreeTrash)) await fs.rm(worktreeTrash, { recursive: true, force: true }); if (cloneReal) { - const consumer = await cloneConsumers(tx, row, clone); + const consumer = await cloneConsumers(tx, row.companyId, row.sourceIssueId, row.id, clone); if (consumer) return keep(consumer, consumer === "base_has_consumers"); - const disposable = await cloneDisposable(clone, base, options.signal); - if (disposable) return keep(disposable, false); + const repoUrl = (await git(clone, ["config", "--local", "--get", "remote.origin.url"], { signal: options.signal }).catch(() => "")).trim(); + if (repoUrl !== row.repoUrl) return keep("source_identity_changed", false); + const disposable = await cloneDisposable(clone, base, upstreamCheck(db, row.companyId, row.sourceIssueId, clone, repoUrl, options.signal), options.signal); + if (disposable.reason) { + result.cloneKeptPaths = disposable.paths; + return keep(disposable.reason, false); + } const bytes = await allocatedBytes(clone, () => true).catch(() => 0); if (await exists(cloneTrash)) await fs.rm(cloneTrash, { recursive: true, force: true }); await fs.rename(clone, cloneTrash); @@ -637,3 +705,82 @@ export async function rehomeExpiredIsolatedRestoration(db: Db, snapshot: Workspa }); } catch { return snapshot; } } + +export type OrphanedIsolatedSourceResult = { + companyId: string; issueId: string; removed: boolean; reason: string | null; paths: string[]; + candidateBytes: number; reclaimedBytes: number; failure?: GitExpirationFailure; +}; + +async function sourceDirectories(parent: string) { + const entries = await fs.readdir(parent, { withFileTypes: true }).catch(() => []); + return entries.filter(entry => entry.isDirectory() && SOURCE_ID.test(entry.name)).map(entry => entry.name).sort(); +} + +export async function sweepOrphanedIsolatedRepositorySources(db: Db, options: { dryRun?: boolean; signal?: AbortSignal; after?: string | null; limit?: number } = {}) { + const root = path.join(resolvePaperclipInstanceRoot(), "isolated-repository-sources"); + const results: OrphanedIsolatedSourceResult[] = []; + const limit = options.limit ?? 8; + let inspected = 0; + let lastPosition: string | null = null; + const [afterCompany, afterIssue] = options.after?.split("/") ?? []; + for (const companyId of await sourceDirectories(root)) { + for (const issueId of await sourceDirectories(path.join(root, companyId))) { + options.signal?.throwIfAborted(); + if (afterCompany !== undefined && (companyId < afterCompany || (companyId === afterCompany && issueId <= afterIssue!))) continue; + if (inspected >= limit) return { results, next: lastPosition }; + inspected += 1; + lastPosition = `${companyId}/${issueId}`; + const result: OrphanedIsolatedSourceResult = { companyId, issueId, removed: false, reason: null, paths: [], candidateBytes: 0, reclaimedBytes: 0 }; + try { + await withIsolatedRepositorySourceLock(db, companyId, issueId, async () => { + const clone = isolatedRemoteSourcePath(companyId, issueId); + const parent = path.dirname(clone); + for (const entry of await fs.readdir(parent, { withFileTypes: true })) { + if (!options.dryRun && entry.isDirectory() && SOURCE_TRASH.test(entry.name)) await fs.rm(path.join(parent, entry.name), { recursive: true, force: true }); + } + const stat = await fs.lstat(clone).catch(() => null); + if (!stat) return; + const storage = await fs.lstat(path.join(clone, ".git")).catch(() => null); + if (!stat.isDirectory() || stat.isSymbolicLink() || !storage?.isDirectory() || storage.isSymbolicLink() + || await fs.realpath(clone) !== path.join(await fs.realpath(parent), "checkout")) throw new Error("source_identity_changed"); + const consumer = await cloneConsumers(db, companyId, issueId, null, clone); + if (consumer) throw new Error(consumer); + const rows = await db.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, companyId), or(eq(executionWorkspaces.sourceIssueId, issueId), + sql`${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'issueId' = ${issueId}`))); + if (rows.some(row => row.metadata?.[ISOLATED_REPOSITORY_REHOME_PENDING_KEY] !== undefined)) throw new Error("rehome_pending"); + if (rows.some(row => hasPendingIsolatedRepositoryRehomeCleanup(row))) throw new Error("rehome_cleanup_pending"); + const recorded = new Set(rows.map(row => (row.metadata?.isolatedRemoteSource as { repoUrl?: unknown } | undefined)?.repoUrl ?? row.repoUrl) + .filter((url): url is string => typeof url === "string" && url.length > 0)); + if (recorded.size !== 1) throw new Error("unproven_repository"); + const repoUrl = [...recorded][0]!; + if ((await git(clone, ["config", "--local", "--null", "--get-all", "remote.origin.url"], { signal: options.signal }).catch(() => "")) !== `${repoUrl}\0`) { + throw new Error("source_identity_changed"); + } + const anchor = rows.find(row => row.projectId && row.repoUrl === repoUrl); + const base = anchor ? await sharedManagedBase(anchor).catch(() => null) : null; + const check = await cloneDisposable(clone, base, upstreamCheck(db, companyId, issueId, clone, repoUrl, options.signal), options.signal); + if (check.reason) { + result.paths = check.paths; + throw new Error(check.reason); + } + result.candidateBytes = await allocatedBytes(clone, () => { options.signal?.throwIfAborted(); return true; }).catch(() => 0); + if (options.dryRun) return; + const trash = path.join(parent, `.orphaned-checkout-${randomBytes(8).toString("hex")}`); + await fs.rename(clone, trash); + await fs.rm(trash, { recursive: true, force: true }); + result.removed = true; + result.reclaimedBytes = result.candidateBytes; + await fs.rmdir(parent).catch(() => undefined); + await fs.rmdir(path.dirname(parent)).catch(() => undefined); + }, true); + } catch (error) { + options.signal?.throwIfAborted(); + result.reason = error instanceof IsolatedRepositorySourceBusy ? "source_busy" : gitExpirationSkipReason(error); + if (result.reason === "inspection_failed") result.failure = describeGitExpirationFailure(error); + } + if (result.reason && ORPHAN_HANDLED_ELSEWHERE.has(result.reason)) continue; + if (result.removed || result.reason || result.candidateBytes) results.push(result); + } + } + return { results, next: null }; +}