From 785d54636e1c8513ee5503d602daf6a5ee1970d6 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 09:17:28 -0700 Subject: [PATCH] Delete re-homed isolated clones when their contents are reproducible (#17410) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Isolated task workspaces are Git worktrees on the server disk, and the terminal workspace reaper keeps that disk usage bounded > - #17369 moves old workspaces off their private per-issue clones onto the project's shared clone and then deletes the private clone > - On one instance, the first live sweep moved 12 workspaces but deleted none of their clones (about 600 MB each), and 3 workspaces could not move because their restore point was "too large" > - The kept clones are not referenced by any workspace, so no other cleanup removes them > - This pull request replaces the clone check with a real reproducibility check, sweeps the orphaned clones, lifts the restore point limits for local transfers, and fixes a false `source_identity_changed` > - The benefit is that the disk space #17369 was meant to reclaim is reclaimed, without deleting unique work ## Linked Issues or Issue Description Refs #17369. Bug report described in-PR (bug report template). **What happened?** On one instance, with `PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true` and dry run off, the first sweep re-homed 12 workspaces, skipped 5 with `live_or_unresolved_consumer` and 3 with `restore_point_too_large`. No clone was removed: 11 logged `cloneKeptReason: "clone_checkout_has_files"` and 1 logged `"source_identity_changed"`. Each clone's own checkout was clean (`git status --porcelain --untracked-files=all` empty). Its only history that the shared clone lacked was remote branches and release tags that the shared clone never fetched, which exist on the remote. An operator removed 11 clones by hand after verifying them. **Expected behavior** A re-homed clone whose checkout is clean and whose local history is in the shared clone or on the remote is deleted. A clone with real unique work is kept. Workspaces whose changes exceed the upload limits still move, because the transfer is local. **Steps to reproduce** 1. Use a project repository whose `.gitignore` (or the clone's `info/exclude`) contains `.paperclip/`, with a release tag on a commit that is not on any branch. 2. Re-home an isolated workspace of that repository with #17369. 3. The old worktree is removed, the empty `.paperclip/worktrees` directory stays, and `git ls-files --others --ignored --directory` lists `.paperclip/`. The clone is kept with `clone_checkout_has_files`. Without the ignore line, the local release tag keeps it with `unique_clone_history`. 4. Add a file larger than 64 MiB to a workspace. The move skips with `restore_point_too_large`. **Paperclip version or commit:** `master` at aa0037a1378 (rebased on ee29e2850d0, after #17407). **Deployment mode:** any. **Agent adapter(s) involved:** not adapter-specific (core). ## What Changed - New clone reproducibility check (`cloneDisposable` in `server/src/services/isolated-repository-rehome.ts`), used before deleting a re-homed clone: - `git worktree prune`, then only the clone's own worktree may remain (`base_has_worktrees`); - no tracked changes and no untracked files (`clone_checkout_has_files`); the only untracked exception is Paperclip's compaction trash; - ignored files only inside regenerable directories (`node_modules`, `dist`, ...) or Paperclip's compaction trash. Ignored entries are listed with `--no-empty-directory`, and a non-regenerable ignored directory is expanded file by file, so empty directories such as the old worktree's `.paperclip/worktrees` parent no longer count; - only `refs/heads`, `refs/tags` and `refs/remotes` (no stash, notes or replacement refs: `unsupported_clone_refs`); - every commit reachable from local branches, local tags, non-remote reflogs, `ORIG_HEAD` and `HEAD` (so a detached checkout counts) must be on a remote-tracking ref, present in the shared clone, or on the remote. The remote check reuses `hasCommitsMissingUpstream` (now exported), which fetches the remote's branches and tags into a temporary store, and it runs only when some commit is in neither place. A local tag is accepted when its commit is on the remote. Otherwise the clone is kept (`unique_clone_history`), or kept with `managed_checkout_upstream_unavailable` when the remote cannot be read; - existing 5,000 budgets for refs, reflog entries and unique commits (`clone_history_budget`), plus at most 1,000 ignored directories outside regenerable ones, read in one Git call (`clone_file_budget`); - the log entry now includes `cloneKeptPaths`: up to five paths, worktrees, refs or commits that kept the clone. - Cleanup also checks that the clone's `origin` still matches the workspace `repoUrl` before deleting it. - Old worktree ownership (#4): it is proven by `git rev-parse --git-common-dir`, or, when that fails, by the worktree's `.git` pointer into `/.git/worktrees/`. - Orphan sweep (`sweepOrphanedIsolatedRepositorySources`): at the first reaper rotation after server start (feature on), it walks `isolated-repository-sources///checkout`. It holds the isolated source lock and skips clones that a workspace still needs (an open workspace with matching `isolatedRemoteSource`, a closed one that is not expired with its worktree removed, a project workspace or a running service), clones with a pending move, and clones whose re-homed workspace still has its own cleanup pending. The repository URL comes only from the database: the issue's workspaces must record exactly one URL and the clone's `origin` must equal it (`unproven_repository` or `source_identity_changed` otherwise), so a rewritten `origin` is never fetched. It then applies the same check (using the shared clone of a workspace of the same issue and repository when one exists), renames to trash and deletes. It logs `event: "execution_workspace.isolated_source_orphan_cleanup"` and removes leftover trash directories from interrupted removals. It inspects at most eight issue directories per reaper sweep and resumes from the saved position in the next sweep, so slow remotes cannot hold up other cleanup. When a full walk kept any orphan, it registers a deadline one day later in the reaper's existing deadline set. Otherwise it registers nothing until the next server start. No new timer. - Restore point limits for re-home (#3): `ISOLATED_REHOME_RESTORE_POINT_LIMITS` (500,000 files, 1 GiB per file and in total, 1.5 GiB bundle, 1,000,000 commits) replaces the upload defaults for this local transfer. Verification is unchanged (exact snapshot equality). In `git-restore-point.ts`, `applyRestorePoint` now streams `git diff --binary` into `git apply` instead of buffering it (and skips the pipe when there is no diff, so no new Git version is needed). On a failure it kills both processes and waits for both. The ignored-file read sizes its buffer from the tree instead of the default limit. - Docs: `doc/recoverable-disk-cleanup.md`. ### What I found for `source_identity_changed` (#4) In #17369, cleanup ran `git rev-parse --git-common-dir` in the old worktree and used `""` when the command failed. Then it compared that to the clone's `.git` path, so any Git failure in the old worktree was reported as `source_identity_changed`. Git fails there when the old worktree's registration in the clone (`/.git/worktrees/`) is gone while the directory still exists. Its `.git` file then points to a missing directory. I reproduced this by deleting the registration after a re-home: the old code keeps the clone with `source_identity_changed`, the new code removes the old worktree and the clone. I could not inspect the instance, so the exact event that removed that one registration is not confirmed. The check was a false positive in that case, and the fix proves ownership from the `.git` pointer instead. A worktree whose pointer leads outside the clone is still kept. ## Verification - `pnpm exec vitest run src/__tests__/isolated-repository-rehome.test.ts` in `server/`: 28 passed (real Git repositories, embedded Postgres). New cases: - a clean clone whose only extra history is a remote branch and a release tag on a commit outside any branch, both absent from the shared clone, with `.paperclip/` ignored in the clone: deleted; - a clone with a local branch commit that is neither upstream nor in the shared clone: kept (`unique_clone_history`, the commit is in `cloneKeptPaths`); - a clone with a tracked change, then an ignored non-regenerable file next to an ignored `node_modules`, then an untracked `build/notes.txt`: each kept (`clone_checkout_has_files` with the path), then deleted after cleanup; - a clone with a second registered worktree: kept (`base_has_worktrees`); - a clone whose detached `HEAD` holds the only copy of a commit (reflogs expired): kept (`unique_clone_history`); - an old worktree whose registration was pruned: old worktree and clone removed; - an orphan whose `origin` was rewritten is kept with `source_identity_changed`, and with `unproven_repository` once no workspace records its repository; - 1,001 ignored directories keep a clone (`clone_file_budget`); - orphan sweep: a batch of one returns a resume position and the next batch finishes the walk; a dry run changes nothing. Then an orphan whose history is in the shared clone or on the remote is deleted, and an orphan with no workspace rows and an unpushed commit is kept, with the next deadline one day later; - a workspace with 65 MiB untracked and ignored files (default capture refuses them with `restore_point_too_large`) moves with an exact snapshot match. - After rebasing on #17407 (nested clone migration, which reuses helpers from the re-home module): `isolated-repository-rehome`, `nested-clone-migration`, `execution-workspaces-service`, `execution-workspaces-routes`, `execution-workspace-reopen`, `execution-workspace-compaction`, `execution-workspace-policy`, `execution-workspace-runtime-control-conflict`, `execution-workspace-runtime-lease-route`, `git-workspace-expiration`, `git-restore-point`, `git-restore-point-expiration`: 405 passed; typecheck passes. - Related suites in `server/` (before the rebase): `git-workspace-expiration`, `git-restore-point`, `git-restore-point-expiration`, `execution-workspace-reopen`, `execution-workspaces-service`, `execution-workspace-compaction`, `heartbeat-git-restore-point-resume`, `heartbeat-project-repositories`, `git-reproducible-managed-checkouts`, `isolated-remote-workspace-source`, with the new suite: 398 passed. - `pnpm run typecheck` in `server/` passes. - Mutation checks, each failing a test: - remote proof replaced by "always unique": the unfetched-history and orphan tests fail; - remote proof replaced by "never unique": the local-commit and orphan tests fail; - `.git` pointer fallback removed: the pruned-registration test fails; - re-home limits removed: the large-file test fails; - orphan sweep removed from the reaper: the orphan test fails; - worktree count check removed: the second-worktree test fails; - regenerable-directory exception applied to untracked files: the dirty-checkout test fails; - orphan batch limit removed: the orphan test fails; - orphan URL taken from the clone again: the rewritten-origin test fails; - ignored directory budget removed: the many-directory test fails; - `HEAD` left out of the checked commits: the detached-HEAD test fails. - Dropping only `--no-empty-directory` does not fail a test, because the per-directory expansion also accepts an empty ignored directory. The two mechanisms are redundant on purpose. - Not run: the full server suite, or the change on a live instance. ## Risks - Deleting a clone now depends on reading the remote when some local commit is neither on a remote-tracking ref nor in the shared clone. If the remote cannot be read, the clone is kept. - Commits that only remote-tracking refs hold (for example a remote branch deleted upstream after the clone fetched it) are treated as not unique, as before. They were pushed once. - The orphan sweep deletes clones that no workspace refers to. A closed workspace that is expired with its worktree removed does not count as a user, because reopening it rebuilds from the shared clone or clones again. - Re-home transfers can now use up to about 1.5 GiB of memory for a single restore point payload, so a very large workspace uses more memory during its move than before. - The orphan sweep runs once per server start and then only while it keeps an orphan, so a clone that becomes orphaned later waits for the next restart. - No migration. No schema change. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file editing, Git, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/recoverable-disk-cleanup.md | 56 ++++- .../isolated-repository-rehome.test.ts | 174 +++++++++++++- server/src/services/execution-workspaces.ts | 25 +- server/src/services/git-restore-point.ts | 53 +++-- .../src/services/git-workspace-restoration.ts | 2 +- .../services/isolated-repository-rehome.ts | 219 +++++++++++++++--- 6 files changed, 465 insertions(+), 64 deletions(-) diff --git a/doc/recoverable-disk-cleanup.md b/doc/recoverable-disk-cleanup.md index 5642e560250..1fa8a1a892e 100644 --- a/doc/recoverable-disk-cleanup.md +++ b/doc/recoverable-disk-cleanup.md @@ -428,7 +428,9 @@ restore can interleave. A workspace is moved only when: - its Git state is transferable: no stash or replacement refs, no in-progress merge, rebase or other worktree-specific state, no hidden index flags, no shallow history, and nothing a restore point refuses (submodules, filtered paths, special files, untracked - nested repositories, oversized contents). + nested repositories). The transfer is local, so it uses much larger restore point limits + than uploads do: 500,000 files, 1 GiB per file and in total, a 1.5 GiB bundle and + 1,000,000 commits. The move captures a restore point of the old worktree against commits both repositories share, imports it into the shared clone, creates or fast-forwards the issue branch, adds @@ -461,16 +463,48 @@ rollback stops with `rehome_rollback_conflict` and keeps the new worktree and th neither copy is lost; later attempts retry the rollback once the conflict is resolved. A new worktree is removed only when the move created it: a target that appears from elsewhere is never deleted. -After the update, the old worktree is removed and the clone is renamed to trash and -deleted when nothing else needs it: no other workspace or project workspace refers to it -(a workspace already expired with its worktree removed does not count), no runtime -service runs in it, it has no other worktree, its own checkout has no changes or files -other than regenerable ones, it has only branch, tag and remote-tracking refs, and every -commit its local refs and reflogs hold is either on a remote-tracking ref or present in the -shared clone. A clone with more than 5,000 refs, reflog entries or such commits is kept -(`clone_history_budget`). A clone another workspace still uses is kept (`cloneRetained`); that -workspace's own move removes it later. If the server stops after the update, a later sweep -finishes the removal and sets `cleanedAt`, also after the workspace has been archived. +After the update, the old worktree is removed (its ownership is proven by `git rev-parse` +or, when its registration in the clone was already pruned, by its `.git` pointer into the +clone's `worktrees` directory). The clone is then renamed to trash and deleted when nothing +else needs it and everything in it is reproducible: + +- no other workspace or project workspace refers to it (a workspace already expired with its + worktree removed does not count) and no runtime service runs in it; +- after `git worktree prune`, it has no worktree besides its own checkout; +- its own checkout has no tracked changes and no untracked files (only Paperclip's + compaction trash is allowed, even inside a directory with a regenerable name), and its + ignored files are only inside regenerable directories (such as `node_modules`) or Paperclip's compaction + trash. Empty directories, such as the `.paperclip/worktrees` parent left by the old + worktree, do not count; +- it has only branch, tag and remote-tracking refs (no stash, notes or replacement refs); +- every commit that its local branches, local tags, reflogs (other than remote-tracking + ones) or `ORIG_HEAD` hold is on one of its remote-tracking refs, present in the shared + clone, or found on the remote. The remote is read fresh (branches and tags) with the + company's Git identity, falling back to anonymous access, and only when some commit is in + neither place. `HEAD` itself counts too, so a detached checkout is covered. Remote branches and release tags that the shared clone never fetched do + not keep a clone. + +A clone with more than 5,000 refs, reflog entries or such commits is kept +(`clone_history_budget`), and so is one with more than 1,000 ignored directories outside +regenerable ones (`clone_file_budget`); those directories are read in one Git call. The log entry names up to five paths, worktrees, refs or commits +that kept a clone (`cloneKeptPaths`). A clone another workspace still uses is kept +(`cloneRetained`); that workspace's own move removes it later. If the server stops after +the update, a later sweep finishes the removal and sets `cleanedAt`, also after the +workspace has been archived. + +The reaper also sweeps orphaned clones under `isolated-repository-sources/` once per server +start: clones that no execution workspace refers to (no open workspace with matching +`isolatedRemoteSource`, and no closed one that would need the clone to reopen) and that no +pending move or cleanup still owns. The repository must be proven by the database: the +workspaces of that issue must record exactly one repository URL, and the clone's `origin` +must equal it (`unproven_repository` or `source_identity_changed` otherwise), so a rewritten +`origin` is never fetched or trusted. It applies the same reproducibility check, using the +shared clone of a workspace of the same issue and repository when one exists, and logs +`event: "execution_workspace.isolated_source_orphan_cleanup"`. It inspects at most eight +issue directories per reaper sweep and continues from that position in the next sweep, so a +slow remote cannot hold up other workspace cleanup. When the walk ends and it kept any +orphan, it looks again a day later; otherwise it registers no further work. It also deletes leftover +trash directories from interrupted removals. Each attempt logs `event: "execution_workspace.isolated_source_rehome"` with the outcome, skip reason, candidate bytes and reclaimed bytes. A workspace that is skipped, or whose diff --git a/server/src/__tests__/isolated-repository-rehome.test.ts b/server/src/__tests__/isolated-repository-rehome.test.ts index 6631ebfc0c2..889b0ffa63f 100644 --- a/server/src/__tests__/isolated-repository-rehome.test.ts +++ b/server/src/__tests__/isolated-repository-rehome.test.ts @@ -14,8 +14,8 @@ import { isolatedRemoteSourcePath, materializeIsolatedRemoteWorkspaceSource } fr import { realizeExecutionWorkspace } from "../services/workspace-runtime.js"; import { executionWorkspaceService } from "../services/execution-workspaces.js"; import { expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js"; -import { currentRestorePointSnapshot } from "../services/git-restore-point.js"; -import { rehomeIsolatedRepositoryWorkspace } from "../services/isolated-repository-rehome.js"; +import { captureRestorePoint, currentRestorePointSnapshot } from "../services/git-restore-point.js"; +import { rehomeIsolatedRepositoryWorkspace, sweepOrphanedIsolatedRepositorySources } from "../services/isolated-repository-rehome.js"; import { addNestedRepositoryWorktree, nestedRepositoryBaseCandidates, nestedWorktreeBranchPrefix, readLinkedNestedCheckout } from "../services/nested-repository-worktrees.js"; import { getStorageService } from "../storage/index.js"; @@ -44,6 +44,7 @@ describeEmbeddedPostgres("isolated repository source re-home", () => { afterEach(async () => { await db.delete(executionWorkspaceRestorePoints); await db.execute(sql`TRUNCATE TABLE companies CASCADE`); + await fs.rm(path.join(root, "home", "instances", "default", "isolated-repository-sources"), { recursive: true, force: true }); }); afterAll(async () => { vi.unstubAllEnvs(); @@ -472,4 +473,173 @@ describeEmbeddedPostgres("isolated repository source re-home", () => { expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed); expect(await fs.readFile(path.join(f.cwd, "notes.txt"), "utf8")).toBe("untracked work\n"); }); + + async function crashedRehome(f: Awaited>) { + const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { afterDatabaseUpdate: async () => { throw new Error("simulated crash"); } } }); + expect(result).toMatchObject({ rehomed: true, cloneRemoved: false }); + return result; + } + + async function publishUnfetchedHistory(f: Awaited>) { + git(f.author, "checkout", "-q", "-b", "feature"); + await fs.writeFile(path.join(f.author, "feature.txt"), "feature\n"); git(f.author, "add", "."); git(f.author, "commit", "-qm", "feature"); + const feature = git(f.author, "rev-parse", "HEAD"); + git(f.author, "checkout", "-q", "main"); + const release = git(f.author, "commit-tree", "HEAD^{tree}", "-p", "HEAD", "-m", "release"); + git(f.author, "tag", "v1", release); + git(f.author, "push", "-q", f.remote, "feature", "v1"); + git(f.clone, "fetch", "-q", "--tags", "origin"); + return { feature, release }; + } + + it("deletes a clean clone whose only extra history is remote branches and tags the shared clone never fetched", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const { feature, release } = await publishUnfetchedHistory(f); + await fs.appendFile(path.join(f.clone, ".git", "info", "exclude"), "\n.paperclip/\n"); + await crashedRehome(f); + expect(() => git(f.sharedPath, "cat-file", "-e", `${release}^{commit}`)).toThrow(); + expect(() => git(f.sharedPath, "cat-file", "-e", `${feature}^{commit}`)).toThrow(); + expect(git(f.clone, "rev-parse", "refs/tags/v1")).toBe(release); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ outcome: "cleaned", cloneRemoved: true, cloneKeptReason: null }); + expect(await exists(f.clone)).toBe(false); + }); + + it("keeps a clone with a local branch commit that is neither upstream nor in the shared clone", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + const local = git(f.clone, "commit-tree", "HEAD^{tree}", "-p", "HEAD", "-m", "local only"); + git(f.clone, "branch", "local-work", local); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "unique_clone_history", cloneKeptPaths: [local] }); + expect(await exists(f.clone)).toBe(true); + expect((await f.row()).metadata?.isolatedRepositoryRehome).toMatchObject({ cleanedAt: null }); + }); + + it("keeps a clone whose detached HEAD holds the only copy of a commit", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + const detached = git(f.clone, "commit-tree", "HEAD^{tree}", "-p", "HEAD", "-m", "detached only"); + git(f.clone, "checkout", "-q", "--detach", detached); + git(f.clone, "reflog", "expire", "--expire=now", "--all"); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "unique_clone_history", cloneKeptPaths: [detached] }); + expect(await exists(f.clone)).toBe(true); + }); + + it("keeps a clone whose own checkout has changes or unexplained ignored files", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + await fs.writeFile(path.join(f.clone, "marker.txt"), "edited in the clone\n"); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "clone_checkout_has_files", cloneKeptPaths: ["marker.txt"] }); + git(f.clone, "checkout", "--", "marker.txt"); + await fs.writeFile(path.join(f.clone, "cache.tsbuildinfo"), "ignored but not regenerable\n"); + await fs.mkdir(path.join(f.clone, "node_modules", "dep"), { recursive: true }); + await fs.writeFile(path.join(f.clone, "node_modules", "dep", "index.js"), "regenerable\n"); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "clone_checkout_has_files", cloneKeptPaths: ["cache.tsbuildinfo"] }); + await fs.rm(path.join(f.clone, "cache.tsbuildinfo")); + await fs.mkdir(path.join(f.clone, "build")); + await fs.writeFile(path.join(f.clone, "build", "notes.txt"), "untracked, not ignored\n"); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "clone_checkout_has_files", cloneKeptPaths: ["build/"] }); + await fs.rm(path.join(f.clone, "build"), { recursive: true }); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ outcome: "cleaned", cloneRemoved: true }); + }); + + it("keeps a clone with another registered worktree", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + const extra = path.join(path.dirname(f.remote), "extra-worktree"); + git(f.clone, "worktree", "add", "-q", "-b", "extra", extra); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "base_has_worktrees" }); + expect(await exists(f.clone)).toBe(true); + }); + + it("removes the old worktree and clone when the old worktree's registration was already pruned", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + await fs.rm(path.join(f.clone, ".git", "worktrees", "fixture-task"), { recursive: true }); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ outcome: "cleaned", cloneRemoved: true, cloneKeptReason: null }); + expect(await exists(f.cwd)).toBe(false); + }); + + it("sweeps orphaned clones that no workspace needs and keeps those with unique history", async () => { + const orphan = await fixture(); + await makeWork(orphan.cwd); + await publishUnfetchedHistory(orphan); + await crashedRehome(orphan); + git(orphan.clone, "worktree", "remove", "--force", orphan.cwd); + const current = await orphan.row(); + await db.update(executionWorkspaces).set({ metadata: { ...current.metadata, isolatedRepositoryRehome: { + ...current.metadata?.isolatedRepositoryRehome as Record, cleanedAt: new Date().toISOString() } } }).where(eq(executionWorkspaces.id, orphan.workspaceId)); + const unknown = await fixture(); + await makeWork(unknown.cwd); + await crashedRehome(unknown); + git(unknown.clone, "worktree", "remove", "--force", unknown.cwd); + await db.delete(executionWorkspaces).where(eq(executionWorkspaces.id, unknown.workspaceId)); + + const first = await sweepOrphanedIsolatedRepositorySources(db, { dryRun: true, limit: 1 }); + expect(first.results).toHaveLength(1); + expect(first.next).toEqual(expect.any(String)); + const second = await sweepOrphanedIsolatedRepositorySources(db, { dryRun: true, limit: 1, after: first.next }); + expect(second).toMatchObject({ results: [expect.anything()], next: null }); + expect(new Set([...first.results, ...second.results].map(result => result.issueId))).toEqual(new Set([orphan.issueId, unknown.issueId])); + const dryRun = executionWorkspaceService(db, { gitCacheExpirationEnabled: true, isolatedSourceRehomeEnabled: true, isolatedSourceRehomeDryRun: true }); + await dryRun.sweepTerminalWorkspaces(); + expect(await exists(orphan.clone)).toBe(true); + + const service = executionWorkspaceService(db, { gitCacheExpirationEnabled: true, isolatedSourceRehomeEnabled: true }); + await service.sweepTerminalWorkspaces(); + expect(await exists(orphan.clone)).toBe(false); + expect(await exists(unknown.clone)).toBe(true); + expect(await service.nextTerminalWorkspaceSweepAt()).toBeGreaterThan(Date.now() + 23 * 60 * 60 * 1000); + }); + + it("re-homes a workspace whose changes exceed the default restore point limits", async () => { + const f = await fixture(); + await makeWork(f.cwd); + const large = Buffer.alloc(65 * 1024 * 1024, 7); + await fs.writeFile(path.join(f.cwd, "notes", "large.bin"), large); + await fs.writeFile(path.join(f.cwd, "large.tsbuildinfo"), large); + await expect(captureRestorePoint({ cwd: f.cwd, upstreamCommits: [] })).rejects.toMatchObject({ reason: "restore_point_too_large" }); + const before = await currentRestorePointSnapshot(f.cwd); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: true, cloneRemoved: true, reason: null }); + expect(await currentRestorePointSnapshot(f.target)).toEqual(before); + expect((await fs.readFile(path.join(f.target, "notes", "large.bin"))).equals(large)).toBe(true); + expect((await fs.readFile(path.join(f.target, "large.tsbuildinfo"))).equals(large)).toBe(true); + }); + + it("keeps an orphaned clone whose origin no longer matches the recorded repository", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + git(f.clone, "worktree", "remove", "--force", f.cwd); + const current = await f.row(); + await db.update(executionWorkspaces).set({ metadata: { ...current.metadata, isolatedRepositoryRehome: { + ...current.metadata?.isolatedRepositoryRehome as Record, cleanedAt: new Date().toISOString() } } }).where(eq(executionWorkspaces.id, f.workspaceId)); + const decoy = path.join(path.dirname(f.remote), "decoy.git"); + git(path.dirname(f.remote), "clone", "-q", "--bare", f.clone, decoy); + git(f.clone, "remote", "set-url", "origin", pathToFileURL(decoy).href); + const { results } = await sweepOrphanedIsolatedRepositorySources(db); + expect(results).toMatchObject([{ issueId: f.issueId, removed: false, reason: "source_identity_changed" }]); + expect(await exists(f.clone)).toBe(true); + await db.delete(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId)); + expect((await sweepOrphanedIsolatedRepositorySources(db)).results).toMatchObject([{ issueId: f.issueId, removed: false, reason: "unproven_repository" }]); + }); + + it("keeps a clone with more ignored directories than the cleanup budget", async () => { + const f = await fixture(); + await makeWork(f.cwd); + await crashedRehome(f); + await fs.appendFile(path.join(f.clone, ".git", "info", "exclude"), "\nscratch-*/\n"); + for (let index = 0; index <= 1_000; index += 1) { + await fs.mkdir(path.join(f.clone, `scratch-${index}`, "node_modules"), { recursive: true }); + await fs.writeFile(path.join(f.clone, `scratch-${index}`, "node_modules", "x.js"), "x\n"); + } + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "clone_file_budget" }); + for (let index = 1; index <= 1_000; index += 1) await fs.rm(path.join(f.clone, `scratch-${index}`), { recursive: true }); + expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ outcome: "cleaned", cloneRemoved: true }); + }); }); diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index d5c456c8b06..9d42edcc6e1 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -6,7 +6,7 @@ import { measureRunPhase } from "./execution-preparation-timing.js"; import { assertExecutionWorkspaceRecordAccess, assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-access.js"; import { verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js"; import { expireReconstructableGitWorkspace, gitWorkspaceLastActivityAt, hasPendingGitExpiration } from "./git-workspace-expiration.js"; -import { hasPendingIsolatedRepositoryRehomeCleanup, isIsolatedRepositoryRehomeCandidate, rehomeExpiredIsolatedRestoration, rehomeIsolatedRepositoryWorkspace } from "./isolated-repository-rehome.js"; +import { hasPendingIsolatedRepositoryRehomeCleanup, isIsolatedRepositoryRehomeCandidate, rehomeExpiredIsolatedRestoration, rehomeIsolatedRepositoryWorkspace, sweepOrphanedIsolatedRepositorySources } from "./isolated-repository-rehome.js"; import { hasPendingNestedCloneMigration, isNestedCloneMigrationCandidate, nextNestedCloneMigrationBatch, pendingNestedCloneMigrationNames, listNestedCloneMigrationNames, migrateNestedRepositoryClone, NESTED_CLONE_MIGRATION_RETRY_REASONS, type NestedCloneMigrationRotation } from "./nested-clone-migration.js"; import type { StorageService } from "../storage/types.js"; @@ -1756,6 +1756,10 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic const nestedCloneMigrationNotBefore = new Map(); const nestedCloneMigrationRetryAt = new Map(); const nestedCloneMigrationRotations = new Map(); + const ISOLATED_ORPHAN_RETRY_MS = 24 * 60 * 60 * 1000; + let isolatedOrphanSweepDueAt = 0; + let isolatedOrphanCursor: string | null = null; + let isolatedOrphanKept = false; let terminalSweepRescanRequested = false; // The reaper freezes an upper bound on updatedAt at the start of each // rotation. The scan only reads candidates at or below the bound, so a steady @@ -3283,6 +3287,25 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic isolatedRehomeDeadlines.clear(); nestedCloneMigrationDeadlines.clear(); terminalSweepRescanRequested = false; + if (opts.gitCacheExpirationEnabled && opts.isolatedSourceRehomeEnabled && isolatedOrphanSweepDueAt <= now().getTime()) { + const dryRun = opts.isolatedSourceRehomeDryRun ?? false; + const batch = await sweepOrphanedIsolatedRepositorySources(db, { dryRun, signal, after: isolatedOrphanCursor }).catch((error: unknown) => { + signal?.throwIfAborted(); + logger.warn({ err: error }, "isolated repository source orphan sweep failed; it retries later"); + return null; + }); + for (const orphan of batch?.results ?? []) { + logger.info({ event: "execution_workspace.isolated_source_orphan_cleanup", dryRun, ...orphan }, "Orphaned isolated repository source inspected"); + } + isolatedOrphanKept ||= !batch || batch.results.some(orphan => !orphan.removed); + isolatedOrphanCursor = batch?.next ?? null; + if (isolatedOrphanCursor) isolatedOrphanSweepDueAt = now().getTime(); + else { + isolatedOrphanSweepDueAt = isolatedOrphanKept ? now().getTime() + ISOLATED_ORPHAN_RETRY_MS : Number.POSITIVE_INFINITY; + isolatedOrphanKept = false; + } + } + if (Number.isFinite(isolatedOrphanSweepDueAt) && isolatedOrphanSweepDueAt > 0) isolatedRehomeDeadlines.set("isolated-source-orphans", isolatedOrphanSweepDueAt); } const boundary = terminalSweepBoundary; const boundaryFilter = boundary diff --git a/server/src/services/git-restore-point.ts b/server/src/services/git-restore-point.ts index 5c949c5208f..698b1395ade 100644 --- a/server/src/services/git-restore-point.ts +++ b/server/src/services/git-restore-point.ts @@ -1,4 +1,4 @@ -import { execFile as execFileCallback } from "node:child_process"; +import { execFile as execFileCallback, spawn } from "node:child_process"; import { createCipheriv, createDecipheriv, createHash, randomBytes } from "node:crypto"; import fs from "node:fs/promises"; import os from "node:os"; @@ -105,10 +105,7 @@ function isolatedEnv(extra: Record = {}) { } async function git(cwd: string, args: string[], options: GitOptions = {}): Promise { - const child = execFile("git", [ - "-c", "core.fsmonitor=false", "-c", "core.autocrlf=false", "-c", "core.safecrlf=false", - "-c", `core.hooksPath=${os.devNull}`, "-C", cwd, ...args, - ], { + const child = execFile("git", gitArgs(cwd, args), { env: isolatedEnv(options.env), encoding: "buffer", maxBuffer: options.maxBuffer ?? 64 * 1024 * 1024, @@ -120,6 +117,32 @@ async function git(cwd: string, args: string[], options: GitOptions = {}): Promi return stdout as Buffer; } +function gitArgs(cwd: string, args: string[]) { + return ["-c", "core.fsmonitor=false", "-c", "core.autocrlf=false", "-c", "core.safecrlf=false", "-c", `core.hooksPath=${os.devNull}`, "-C", cwd, ...args]; +} + +async function gitPipe(cwd: string, from: string[], to: string[], signal?: AbortSignal) { + const options = { env: isolatedEnv(), signal, timeout: 15 * 60_000 }; + const source = spawn("git", gitArgs(cwd, from), { ...options, stdio: ["ignore", "pipe", "pipe"] }); + const sink = spawn("git", gitArgs(cwd, to), { ...options, stdio: ["pipe", "ignore", "pipe"] }); + source.stdout.pipe(sink.stdin); + sink.stdin.on("error", () => {}); + const finished = (child: ReturnType) => new Promise((resolve, reject) => { + let stderr = ""; + child.stderr?.on("data", (chunk: Buffer) => { if (stderr.length < 4096) stderr += chunk.toString("utf8"); }); + child.on("error", reject); + child.on("close", (code) => code === 0 ? resolve() : reject(Object.assign(new Error(`git exited with ${code}`), { code, stderr }))); + }); + const done = [finished(source), finished(sink)]; + try { await Promise.all(done); } + catch (error) { + source.kill("SIGKILL"); + sink.kill("SIGKILL"); + await Promise.allSettled(done); + throw error; + } +} + async function gitText(cwd: string, args: string[], options: GitOptions = {}) { return (await git(cwd, args, options)).toString("utf8").trim(); } @@ -232,21 +255,21 @@ async function rawIgnoredTree(cwd: string, env: Record, temporar } async function writeIgnoredFiles(cwd: string, commit: string, signal?: AbortSignal) { - const listing = nulList(await git(cwd, ["ls-tree", "-r", "-z", `${commit}^{tree}`], { signal })); + const listing = nulList(await git(cwd, ["ls-tree", "-r", "-z", "--long", `${commit}^{tree}`], { signal })); const entries = listing.map((line) => { - const match = /^(100644|100755|120000) blob ([0-9a-f]{40})\t(.+)$/s.exec(line); + const match = /^(100644|100755|120000) blob ([0-9a-f]{40}) +(\d+)\t(.+)$/s.exec(line); if (!match) throw new RestorePointRestoreFailure("invalid_payload"); - const target = path.join(cwd, match[3]!); - if (path.isAbsolute(match[3]!) || match[3]!.split("/").includes("..") || !target.startsWith(`${cwd}${path.sep}`)) { + const target = path.join(cwd, match[4]!); + if (path.isAbsolute(match[4]!) || match[4]!.split("/").includes("..") || !target.startsWith(`${cwd}${path.sep}`)) { throw new RestorePointRestoreFailure("invalid_payload"); } - return { mode: match[1]!, blob: match[2]!, target }; + return { mode: match[1]!, blob: match[2]!, size: Number(match[3]), target }; }); for (const entry of entries) { if (await fs.lstat(entry.target).then(() => true, () => false)) throw new RestorePointRestoreFailure("ignored_files_conflict"); } const contents = await git(cwd, ["cat-file", "--batch"], { signal, input: entries.map((entry) => entry.blob).join("\n") + "\n", - maxBuffer: 2 * DEFAULT_RESTORE_POINT_LIMITS.maxTotalBytes }); + maxBuffer: entries.reduce((total, entry) => total + entry.size + 128, 4096) }); const blobs: Buffer[] = []; let offset = 0; for (const entry of entries) { @@ -487,8 +510,12 @@ export async function applyRestorePoint(input: { cwd: string; manifest: RestoreP const head = await gitText(cwd, ["rev-parse", "--verify", "HEAD^{commit}"], { signal: input.signal }); if (head !== manifest.head) throw new RestorePointRestoreFailure("head_mismatch"); if (manifest.index && manifest.worktree) { - const diff = await git(cwd, ["diff", "--binary", "--no-color", "--no-ext-diff", "--no-textconv", "--no-renames", "HEAD", manifest.worktree], { signal: input.signal }); - if (diff.length > 0) await git(cwd, ["apply", "--binary", "--whitespace=nowarn", "-"], { input: diff, signal: input.signal }); + const changed = await git(cwd, ["diff", "--quiet", "--no-ext-diff", "--no-textconv", "HEAD", manifest.worktree], { signal: input.signal }).then(() => false, (error: { code?: unknown }) => { + if (error.code === 1) return true; + throw error; + }); + if (changed) await gitPipe(cwd, ["diff", "--binary", "--no-color", "--no-ext-diff", "--no-textconv", "--no-renames", "HEAD", manifest.worktree], + ["apply", "--binary", "--whitespace=nowarn", "-"], input.signal); await git(cwd, ["read-tree", `${manifest.index}^{tree}`], { signal: input.signal }); } if (manifest.ignored) await writeIgnoredFiles(cwd, manifest.ignored, input.signal); diff --git a/server/src/services/git-workspace-restoration.ts b/server/src/services/git-workspace-restoration.ts index 67b7c08a67e..f71c1851e7c 100644 --- a/server/src/services/git-workspace-restoration.ts +++ b/server/src/services/git-workspace-restoration.ts @@ -402,7 +402,7 @@ async function tipsMissingUpstream(checkout: string, tips: string[], repoUrl: st return tips.filter(tip => missing.has(tip)).sort(); } -async function hasCommitsMissingUpstream(checkout: string, localTips: string[], repoUrl: string, +export async function hasCommitsMissingUpstream(checkout: string, localTips: string[], repoUrl: string, resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal, covered: string[] = []) { return (await commitsMissingUpstream(checkout, localTips, repoUrl, resolveGitAuth, signal, covered, 1)).length > 0; } diff --git a/server/src/services/isolated-repository-rehome.ts b/server/src/services/isolated-repository-rehome.ts index 1b342e52fbe..c8c02f85578 100644 --- a/server/src/services/isolated-repository-rehome.ts +++ b/server/src/services/isolated-repository-rehome.ts @@ -1,21 +1,21 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import { createHash } from "node:crypto"; +import { createHash, randomBytes } from "node:crypto"; import { execFile as execFileCallback } from "node:child_process"; import { promisify } from "node:util"; -import { and, eq, ne, sql } from "drizzle-orm"; +import { and, eq, ne, or, sql } from "drizzle-orm"; import { executionWorkspaces, projectWorkspaces, workspaceRuntimeServices, heartbeatRuns, nativeRunFinalizations, environmentLeases, withDedicatedDbConnection, type Db } from "@paperclipai/db"; import { PROJECT_REPOSITORIES_DIR } from "@paperclipai/adapter-utils/git-workspace-sync"; import { IsolatedRepositorySourceBusy, isolatedRemoteSourcePath, verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js"; import { acquireGitExpirationLocks, recoverGitExpirationLocks, type GitExpirationLockLocation } from "./git-workspace-expiration-locks.js"; -import { allocatedBytes, describeGitExpirationFailure, expirationScope, gitExpirationSkipReason, type GitExpirationFailure } from "./git-workspace-expiration.js"; -import { finishGitCacheTrash, managedSourcePaths, readGitCacheExpiration, readGitCacheRestoration, verifiedGitCacheSource, +import { allocatedBytes, describeGitExpirationFailure, expirationScope, gitExpirationSkipReason, withAnonymousGitFallback, type GitExpirationFailure } from "./git-workspace-expiration.js"; +import { finishGitCacheTrash, hasCommitsMissingUpstream, managedSourcePaths, readGitCacheExpiration, readGitCacheRestoration, verifiedGitCacheSource, GIT_CACHE_RESTORATION_KEY, type GitCacheRestoration } from "./git-workspace-restoration.js"; import { applyRestorePoint, captureRestorePoint, currentRestorePointSnapshot, importRestorePoint, restorePointSavesIgnoredFile, RestorePointRestoreFailure, RestorePointSkip, type RestorePointLimits, type RestorePointSnapshot } from "./git-restore-point.js"; -import { findRegenerableWorkspaceDirs, REGENERABLE_WORKSPACE_DIR_NAMES } from "./workspace-compaction.js"; +import { findRegenerableWorkspaceDirs, REGENERABLE_WORKSPACE_DIR_NAMES, WORKSPACE_COMPACTION_TRASH_DIR } from "./workspace-compaction.js"; import { ensureManagedRepositoriesIgnored } from "./managed-repository-checkout.js"; import { withWorkspaceLockKeepalive } from "./native-runtime/native-workspace-lock-keepalive.js"; import { canonicalRepositoryRoot } from "./execution-workspace-source-roots.js"; @@ -23,6 +23,8 @@ import { withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifec import { loadGitRestorePoint } from "./git-restore-point-storage.js"; import { nestedWorktreeBranchPrefix, readLinkedNestedCheckout, repairMovedNestedRepositoryWorktree, withNestedRepositoryBaseLock } from "./nested-repository-worktrees.js"; import { getStorageService } from "../storage/index.js"; +import { createGitRemoteAuthProvider } from "./git-credentials.js"; +import { resolvePaperclipInstanceRoot } from "../home-paths.js"; const execFile = promisify(execFileCallback); const SHA = /^[0-9a-f]{40}$/; @@ -31,7 +33,16 @@ const CLOSED_STATUSES = ["archived", "cleanup_failed"]; const ADMIN_ENTRIES = new Set(["HEAD", "ORIG_HEAD", "FETCH_HEAD", "index", "commondir", "gitdir", "logs", "refs", "COMMIT_EDITMSG"]); const MAX_BOUNDARY_COMMITS = 64; const CLONE_HISTORY_BUDGET = 5_000; +const CLONE_DIRECTORY_BUDGET = 1_000; const UNSAFE_BASE_CONFIG = /^(filter\.|include\.|includeif\.|core\.(fsmonitor|hookspath|attributesfile|worktree|gitproxy|sshcommand|askpass|editor|pager|alternaterefscommand)$|extensions\.worktreeconfig$)/; +const GIB = 1024 * 1024 * 1024; +const COMPACTION_TRASH_PATH = `.paperclip/worktrees/${WORKSPACE_COMPACTION_TRASH_DIR}`; +const SOURCE_ID = /^[a-zA-Z0-9_-]+$/; +const ORPHAN_HANDLED_ELSEWHERE = new Set(["base_has_consumers", "rehome_pending", "rehome_cleanup_pending"]); +const SOURCE_TRASH = /^\.(rehomed-checkout|rehomed-worktree|orphaned-checkout)-[a-zA-Z0-9_-]+$/; +export const ISOLATED_REHOME_RESTORE_POINT_LIMITS: RestorePointLimits = { + maxFiles: 500_000, maxFileBytes: GIB, maxTotalBytes: GIB, maxCommits: 1_000_000, maxBundleBytes: 1.5 * GIB, +}; export const ISOLATED_REPOSITORY_REHOME_KEY = "isolatedRepositoryRehome"; export const ISOLATED_REPOSITORY_REHOME_PENDING_KEY = "isolatedRepositoryRehomePending"; @@ -46,7 +57,7 @@ export type IsolatedRepositoryRehomeHooks = { }; export type IsolatedRepositoryRehomeResult = { outcome: "rehomed" | "dry_run" | "skipped" | "cleaned" | "clone_kept"; - rehomed: boolean; cloneRemoved: boolean; reason: string | null; cloneKeptReason: string | null; + rehomed: boolean; cloneRemoved: boolean; reason: string | null; cloneKeptReason: string | null; cloneKeptPaths?: string[]; candidateBytes: number; reclaimedBytes: number; from: string | null; to: string | null; failure?: GitExpirationFailure; }; @@ -419,7 +430,7 @@ export async function rehomeIsolatedRepositoryWorkspace(db: Db, snapshot: Worksp const beforeHistory = await historyKey(old.cwd, old.admin, options.signal); if (before.head !== old.head) throw new Error("head_changed"); const captured = await captureRestorePoint({ cwd: old.cwd, upstreamCommits: await sharedHistoryBoundary(old.cwd, base, old.head, options.signal), - limits: options.limits, signal: options.signal }); + limits: options.limits ?? ISOLATED_REHOME_RESTORE_POINT_LIMITS, signal: options.signal }); const manifest = captured ? await importRestorePoint({ repositoryCwd: base, payload: captured.payload, signal: options.signal }) : null; const reflog = shas(await git(old.cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal: options.signal }).catch(() => "")); const original = (await fs.readFile(path.join(old.admin, "ORIG_HEAD"), "utf8").catch(() => "")).trim(); @@ -496,10 +507,10 @@ export async function rehomeIsolatedRepositoryWorkspace(db: Db, snapshot: Worksp } } -async function cloneConsumers(db: Db | Tx, workspace: Workspace, clone: string) { - const others = await db.select().from(executionWorkspaces).where(and(ne(executionWorkspaces.id, workspace.id), - sql`((${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'companyId' = ${workspace.companyId} - AND ${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'issueId' = ${workspace.sourceIssueId}) +async function cloneConsumers(db: Db | Tx, companyId: string, issueId: string, excludeId: string | null, clone: string) { + const others = await db.select().from(executionWorkspaces).where(and(excludeId ? ne(executionWorkspaces.id, excludeId) : undefined, + sql`((${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'companyId' = ${companyId} + AND ${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'issueId' = ${issueId}) OR ${executionWorkspaces.cwd} = ${clone} OR ${executionWorkspaces.providerRef} = ${clone} OR left(${executionWorkspaces.cwd}, length(${clone}) + 1) = ${clone} || '/')`)); const expired = (row: Workspace) => { @@ -516,26 +527,79 @@ async function cloneConsumers(db: Db | Tx, workspace: Workspace, clone: string) return service ? "base_has_live_services" : null; } -async function cloneDisposable(clone: string, base: string, signal?: AbortSignal) { - const worktrees = (await git(clone, ["worktree", "list", "--porcelain"], { signal })).split("\n").filter(line => line.startsWith("worktree ")); - if (worktrees.length !== 1) return "base_has_worktrees"; - if ((await git(clone, ["status", "--porcelain=v1", "-z", "--untracked-files=no"], { signal })).length) return "clone_checkout_has_files"; - const untracked = nul(await git(clone, ["ls-files", "-z", "--others", "--exclude-standard"], { signal })); - const ignored = nul(await git(clone, ["ls-files", "-z", "--others", "--ignored", "--exclude-standard", "--directory"], { signal })); - if (untracked.length || ignored.some(file => !file.split("/").slice(0, -1).some(part => REGENERABLE_WORKSPACE_DIR_NAMES.has(part)))) return "clone_checkout_has_files"; - const refs = (await git(clone, ["for-each-ref", `--count=${CLONE_HISTORY_BUDGET + 1}`, "--format=%(objectname) %(refname)"], { signal })).split("\n").filter(Boolean); - if (refs.length > CLONE_HISTORY_BUDGET) return "clone_history_budget"; - if (refs.some(ref => !/^[0-9a-f]{40} refs\/(heads|tags|remotes)\//.test(ref))) return "unsupported_clone_refs"; - const reflog = (await git(clone, ["reflog", "show", "--all", "-n", String(CLONE_HISTORY_BUDGET + 1), "--format=%H %gD"], { signal }).catch(() => "")) - .split("\n").map(line => line.trim().split(" ")).filter(([sha]) => SHA.test(sha ?? "")); - if (reflog.length > CLONE_HISTORY_BUDGET) return "clone_history_budget"; - const local = [...refs.filter(ref => !/ refs\/remotes\//.test(ref)).map(ref => ref.split(" ")[0]!), - ...reflog.filter(([, selector]) => !(selector ?? "").startsWith("refs/remotes/")).map(([sha]) => sha!)]; - if (local.length === 0) return null; - const unique = shas(await git(clone, ["rev-list", `--max-count=${CLONE_HISTORY_BUDGET + 1}`, "--stdin"], { signal, input: [...new Set(local)].join("\n") + "\n--not\n--remotes\n" })); - if (unique.length > CLONE_HISTORY_BUDGET) return "clone_history_budget"; - const present = await presentCommits(base, unique, signal); - return unique.some(sha => !present.has(sha)) ? "unique_clone_history" : null; +function reproducibleCheckoutPath(file: string) { + const parts = file.split("/").filter(Boolean); + const directories = file.endsWith("/") ? parts : parts.slice(0, -1); + return directories.some(part => REGENERABLE_WORKSPACE_DIR_NAMES.has(part)) + || file === `${COMPACTION_TRASH_PATH}/` || file.startsWith(`${COMPACTION_TRASH_PATH}/`); +} + +type CloneCheck = { reason: string | null; paths: string[] }; +type MissingUpstream = (commits: string[], covered: string[]) => Promise; + +async function cloneDisposable(clone: string, base: string | null, missingUpstream: MissingUpstream, signal?: AbortSignal): Promise { + const kept = (reason: string, paths: string[] = []): CloneCheck => ({ reason, paths: paths.slice(0, 5) }); + try { + await git(clone, ["worktree", "prune"], { signal }); + const worktrees = (await git(clone, ["worktree", "list", "--porcelain"], { signal })).split("\n").filter(line => line.startsWith("worktree ")); + if (worktrees.length !== 1) return kept("base_has_worktrees", worktrees.slice(1).map(line => line.slice(9))); + const tracked = nul(await git(clone, ["status", "--porcelain=v1", "-z", "--untracked-files=no"], { signal })); + if (tracked.length) return kept("clone_checkout_has_files", tracked.map(entry => entry.slice(3))); + const untracked = nul(await git(clone, ["ls-files", "-z", "--others", "--exclude-standard", "--directory", "--no-empty-directory"], { signal })) + .filter(file => file !== `${COMPACTION_TRASH_PATH}/` && !file.startsWith(`${COMPACTION_TRASH_PATH}/`)); + if (untracked.length) return kept("clone_checkout_has_files", untracked); + const ignored = nul(await git(clone, ["ls-files", "-z", "--others", "--ignored", "--exclude-standard", "--directory", "--no-empty-directory"], { signal })) + .filter(entry => !reproducibleCheckoutPath(entry)); + const ignoredFiles = ignored.filter(entry => !entry.endsWith("/")); + if (ignoredFiles.length) return kept("clone_checkout_has_files", ignoredFiles); + const ignoredDirectories = ignored.filter(entry => entry.endsWith("/")); + if (ignoredDirectories.length > CLONE_DIRECTORY_BUDGET) return kept("clone_file_budget"); + if (ignoredDirectories.length) { + const unexplained = nul(await git(clone, ["--literal-pathspecs", "ls-files", "-z", "--others", "--ignored", "--exclude-standard", "--", ...ignoredDirectories], { signal })) + .filter(file => !reproducibleCheckoutPath(file)); + if (unexplained.length) return kept("clone_checkout_has_files", unexplained); + } + const refs = (await git(clone, ["for-each-ref", `--count=${CLONE_HISTORY_BUDGET + 1}`, "--format=%(objectname) %(refname)"], { signal })).split("\n").filter(Boolean); + if (refs.length > CLONE_HISTORY_BUDGET) return kept("clone_history_budget"); + const unexpected = refs.filter(ref => !/^[0-9a-f]{40} refs\/(heads|tags|remotes)\//.test(ref)); + if (unexpected.length) return kept("unsupported_clone_refs", unexpected.map(ref => ref.slice(41))); + const reflog = (await git(clone, ["reflog", "show", "--all", "-n", String(CLONE_HISTORY_BUDGET + 1), "--format=%H %gD"], { signal }).catch(() => "")) + .split("\n").map(line => line.trim().split(" ")).filter(([sha]) => SHA.test(sha ?? "")); + if (reflog.length > CLONE_HISTORY_BUDGET) return kept("clone_history_budget"); + const original = (await fs.readFile(path.join(clone, ".git", "ORIG_HEAD"), "utf8").catch(() => "")).trim(); + const head = (await git(clone, ["rev-parse", "--verify", "-q", "HEAD^{commit}"], { signal }).catch(() => "")).trim(); + const tips = [...new Set([...refs.filter(ref => !/ refs\/remotes\//.test(ref)).map(ref => ref.split(" ")[0]!), + ...reflog.filter(([, selector]) => !(selector ?? "").startsWith("refs/remotes/")).map(([sha]) => sha!), ...(SHA.test(original) ? [original] : []), ...(SHA.test(head) ? [head] : [])])]; + const tipsInBase = base ? await presentCommits(base, tips, signal) : new Set(); + const pending = tips.filter(sha => !tipsInBase.has(sha)); + if (pending.length === 0) return { reason: null, paths: [] }; + const unique = shas(await git(clone, ["rev-list", `--max-count=${CLONE_HISTORY_BUDGET + 1}`, "--stdin"], { signal, + input: [...pending, "--not", "--remotes", ...tipsInBase].join("\n") + "\n" })); + if (unique.length > CLONE_HISTORY_BUDGET) return kept("clone_history_budget"); + const uniqueInBase = base ? await presentCommits(base, unique, signal) : new Set(); + const missing = unique.filter(sha => !uniqueInBase.has(sha)); + if (missing.length === 0) return { reason: null, paths: [] }; + return await missingUpstream(missing, [...tipsInBase, ...uniqueInBase]) ? kept("unique_clone_history", missing) : { reason: null, paths: [] }; + } catch (error) { + signal?.throwIfAborted(); + return kept(gitExpirationSkipReason(error)); + } +} + +async function worktreeOfClone(worktree: string, cloneReal: string, signal?: AbortSignal) { + const admins = path.join(cloneReal, ".git", "worktrees"); + const common = (await git(worktree, ["rev-parse", "--path-format=absolute", "--git-common-dir"], { signal }).catch(() => "")).trim(); + if (common) return common === path.join(cloneReal, ".git"); + const marker = await fs.lstat(path.join(worktree, ".git")).catch(() => null); + if (!marker?.isFile() || marker.size > 4096) return false; + const pointer = /^gitdir: (.+)\n?$/.exec(await fs.readFile(path.join(worktree, ".git"), "utf8"))?.[1]; + if (!pointer) return false; + return path.dirname(await canonicalRepositoryRoot(path.resolve(worktree, pointer))) === admins; +} + +function upstreamCheck(db: Db, companyId: string, issueId: string, clone: string, repoUrl: string, signal?: AbortSignal): MissingUpstream { + const auth = withAnonymousGitFallback(createGitRemoteAuthProvider(db, companyId, { issueId })); + return (commits, covered) => hasCommitsMissingUpstream(clone, commits, repoUrl, auth, signal, covered); } async function cleanupRehomedSource(db: Db, tx: Tx, row: Workspace, result: IsolatedRepositoryRehomeResult, @@ -569,8 +633,7 @@ async function cleanupRehomedSource(db: Db, tx: Tx, row: Workspace, result: Isol const fromStat = await fs.lstat(from).catch(() => null); if (fromStat) { if (!fromStat.isDirectory() || fromStat.isSymbolicLink()) return keep("unsafe_worktree_path", false); - const common = (await git(from, ["rev-parse", "--path-format=absolute", "--git-common-dir"], { signal: options.signal }).catch(() => "")).trim(); - if (common !== path.join(cloneReal, ".git")) return keep("source_identity_changed", false); + if (!await worktreeOfClone(from, cloneReal, options.signal)) return keep("source_identity_changed", false); result.reclaimedBytes += await allocatedBytes(from, () => true).catch(() => 0); if (await exists(worktreeTrash)) await fs.rm(worktreeTrash, { recursive: true, force: true }); await fs.rename(from, worktreeTrash); @@ -579,10 +642,15 @@ async function cleanupRehomedSource(db: Db, tx: Tx, row: Workspace, result: Isol } if (await exists(worktreeTrash)) await fs.rm(worktreeTrash, { recursive: true, force: true }); if (cloneReal) { - const consumer = await cloneConsumers(tx, row, clone); + const consumer = await cloneConsumers(tx, row.companyId, row.sourceIssueId, row.id, clone); if (consumer) return keep(consumer, consumer === "base_has_consumers"); - const disposable = await cloneDisposable(clone, base, options.signal); - if (disposable) return keep(disposable, false); + const repoUrl = (await git(clone, ["config", "--local", "--get", "remote.origin.url"], { signal: options.signal }).catch(() => "")).trim(); + if (repoUrl !== row.repoUrl) return keep("source_identity_changed", false); + const disposable = await cloneDisposable(clone, base, upstreamCheck(db, row.companyId, row.sourceIssueId, clone, repoUrl, options.signal), options.signal); + if (disposable.reason) { + result.cloneKeptPaths = disposable.paths; + return keep(disposable.reason, false); + } const bytes = await allocatedBytes(clone, () => true).catch(() => 0); if (await exists(cloneTrash)) await fs.rm(cloneTrash, { recursive: true, force: true }); await fs.rename(clone, cloneTrash); @@ -637,3 +705,82 @@ export async function rehomeExpiredIsolatedRestoration(db: Db, snapshot: Workspa }); } catch { return snapshot; } } + +export type OrphanedIsolatedSourceResult = { + companyId: string; issueId: string; removed: boolean; reason: string | null; paths: string[]; + candidateBytes: number; reclaimedBytes: number; failure?: GitExpirationFailure; +}; + +async function sourceDirectories(parent: string) { + const entries = await fs.readdir(parent, { withFileTypes: true }).catch(() => []); + return entries.filter(entry => entry.isDirectory() && SOURCE_ID.test(entry.name)).map(entry => entry.name).sort(); +} + +export async function sweepOrphanedIsolatedRepositorySources(db: Db, options: { dryRun?: boolean; signal?: AbortSignal; after?: string | null; limit?: number } = {}) { + const root = path.join(resolvePaperclipInstanceRoot(), "isolated-repository-sources"); + const results: OrphanedIsolatedSourceResult[] = []; + const limit = options.limit ?? 8; + let inspected = 0; + let lastPosition: string | null = null; + const [afterCompany, afterIssue] = options.after?.split("/") ?? []; + for (const companyId of await sourceDirectories(root)) { + for (const issueId of await sourceDirectories(path.join(root, companyId))) { + options.signal?.throwIfAborted(); + if (afterCompany !== undefined && (companyId < afterCompany || (companyId === afterCompany && issueId <= afterIssue!))) continue; + if (inspected >= limit) return { results, next: lastPosition }; + inspected += 1; + lastPosition = `${companyId}/${issueId}`; + const result: OrphanedIsolatedSourceResult = { companyId, issueId, removed: false, reason: null, paths: [], candidateBytes: 0, reclaimedBytes: 0 }; + try { + await withIsolatedRepositorySourceLock(db, companyId, issueId, async () => { + const clone = isolatedRemoteSourcePath(companyId, issueId); + const parent = path.dirname(clone); + for (const entry of await fs.readdir(parent, { withFileTypes: true })) { + if (!options.dryRun && entry.isDirectory() && SOURCE_TRASH.test(entry.name)) await fs.rm(path.join(parent, entry.name), { recursive: true, force: true }); + } + const stat = await fs.lstat(clone).catch(() => null); + if (!stat) return; + const storage = await fs.lstat(path.join(clone, ".git")).catch(() => null); + if (!stat.isDirectory() || stat.isSymbolicLink() || !storage?.isDirectory() || storage.isSymbolicLink() + || await fs.realpath(clone) !== path.join(await fs.realpath(parent), "checkout")) throw new Error("source_identity_changed"); + const consumer = await cloneConsumers(db, companyId, issueId, null, clone); + if (consumer) throw new Error(consumer); + const rows = await db.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, companyId), or(eq(executionWorkspaces.sourceIssueId, issueId), + sql`${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'issueId' = ${issueId}`))); + if (rows.some(row => row.metadata?.[ISOLATED_REPOSITORY_REHOME_PENDING_KEY] !== undefined)) throw new Error("rehome_pending"); + if (rows.some(row => hasPendingIsolatedRepositoryRehomeCleanup(row))) throw new Error("rehome_cleanup_pending"); + const recorded = new Set(rows.map(row => (row.metadata?.isolatedRemoteSource as { repoUrl?: unknown } | undefined)?.repoUrl ?? row.repoUrl) + .filter((url): url is string => typeof url === "string" && url.length > 0)); + if (recorded.size !== 1) throw new Error("unproven_repository"); + const repoUrl = [...recorded][0]!; + if ((await git(clone, ["config", "--local", "--null", "--get-all", "remote.origin.url"], { signal: options.signal }).catch(() => "")) !== `${repoUrl}\0`) { + throw new Error("source_identity_changed"); + } + const anchor = rows.find(row => row.projectId && row.repoUrl === repoUrl); + const base = anchor ? await sharedManagedBase(anchor).catch(() => null) : null; + const check = await cloneDisposable(clone, base, upstreamCheck(db, companyId, issueId, clone, repoUrl, options.signal), options.signal); + if (check.reason) { + result.paths = check.paths; + throw new Error(check.reason); + } + result.candidateBytes = await allocatedBytes(clone, () => { options.signal?.throwIfAborted(); return true; }).catch(() => 0); + if (options.dryRun) return; + const trash = path.join(parent, `.orphaned-checkout-${randomBytes(8).toString("hex")}`); + await fs.rename(clone, trash); + await fs.rm(trash, { recursive: true, force: true }); + result.removed = true; + result.reclaimedBytes = result.candidateBytes; + await fs.rmdir(parent).catch(() => undefined); + await fs.rmdir(path.dirname(parent)).catch(() => undefined); + }, true); + } catch (error) { + options.signal?.throwIfAborted(); + result.reason = error instanceof IsolatedRepositorySourceBusy ? "source_busy" : gitExpirationSkipReason(error); + if (result.reason === "inspection_failed") result.failure = describeGitExpirationFailure(error); + } + if (result.reason && ORPHAN_HANDLED_ELSEWHERE.has(result.reason)) continue; + if (result.removed || result.reason || result.candidateBytes) results.push(result); + } + } + return { results, next: null }; +}