From c1d567ff6a6bf9ca2e8ad7ca9adc3d5e3ad475d5 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 00:56:06 -0600 Subject: [PATCH] fix(workspaces): isolate archived nested Git reconstruction (#19409) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace import must preserve archived task files and Git history. > - A saved nested checkout can be a linked worktree whose branch belongs to its old absolute path. > - Archived import reconstructs that checkout in a new scratch directory. > - Ordinary linked recovery rejects the old branch at that new path. It can also write into a cache used by other import workers. > - This pull request reconstructs archived nested checkouts as private standalone repositories. > - Each import preserves saved files and extra history without changing shared worktree registrations or branches. ## Linked Issues or Issue Description **What happened?** Archived workspace import fails when a saved nested linked checkout has a branch owned by the old path. The new scratch path fails the linked-worktree ownership check. The existing cache lock is process-local, so separate import processes also share mutable Git state without a common lock. **Expected behavior** Archived import must create an independent portable copy of the saved checkout. It must retain the saved commit, branch name, staged edits, working files, ignored recovery files, settings, and extra saved commits. It must keep the source cache unchanged and enforce source authorization. **Steps to reproduce** 1. Create a managed nested linked checkout with an unpushed commit and staged and unstaged edits. 2. Save an encrypted restore point, including ignored files and an extra history tip. 3. Restore its saved contents under a different absolute workspace path. 4. Before this repair, the new path cannot adopt the branch owned by the old path. **Paperclip version or commit** The real Git regression fails before the repair at `fc1f5f3052`. The final branch includes the merged interrupted-upload and nested-source authorization fix from #19396. **Deployment mode** Archived remote workspace migration with real Git repositories, disposable PostgreSQL, and encrypted object storage. Local integration tests use the filesystem storage provider. Builds on merged #19402 and #19396. Related #19374 concerns interrupted clone staging; it does not implement private archived reconstruction. No duplicate open repair was found. ## What Changed - Enable private repository reconstruction only for archived migration. Ordinary linked recovery keeps its path ownership guard. - Add an object-only source to the managed checkout owner. A verified cache supplies only the saved commit and authenticated bundle prerequisites through scoped fetches. The private copy includes no unrelated task refs or objects, hard links, object alternates, or current cache files. - Fetch missing authenticated bundle prerequisites before importing a saved bundle in both private reconstruction and ordinary linked recovery. A pruned cache can use the approved remote; unavailable history still blocks restore. - Keep every authenticated restore point's extra commit under a private ref in root and nested private copies so portable archive capture and garbage collection retain it. Ordinary recovery keeps its existing history and expiration rules. - Share the verified, bounded ignore-file copy between root and nested reconstruction. Sources remain unchanged. - Retain source authorization before and after reconstruction and the final source-row locks from #19396. - Test concurrent private copies, source rejection, foreign task refs and objects, ordinary linked prerequisite recovery, a real migration, and cold restore after removing all original repositories. Test hard links, oversized excludes, unsafe info paths, and replacement during a descriptor read. ## Verification - Final affected run: 292 tests pass across ten suites. These cover real Git repositories, encrypted restore points, disposable PostgreSQL migration, portable archives, the restore-point ledger, source rejection, and concurrent copies. - Full repository typecheck and build pass after the scoped-object repair. Final server typecheck and full build also pass after restricting preserved refs to archive mode. - RED then GREEN: a linked checkout at a different path; a pruned bundle prerequisite still available from the approved remote; foreign task refs and objects entering the archive; root extra history disappearing after cold restore; and archive-only refs blocking ordinary standalone expiration. - Two concurrent private copies retain the saved head, branch, staged and unstaged edits, ignored recovery bytes, settings, and modes. Source refs, index, worktree registrations, and working files remain unchanged. - Delete the original checkout, cache, remote, and private copies after portable capture. Cold restore retains the saved files and extra history after garbage collection. No alternates or unrelated task objects remain. - Ordinary linked recovery retains its path ownership check and recovers pruned prerequisites. New and existing ordinary standalone destinations have no archive-only refs and remain eligible for expiration after normal object cleanup. - Unsafe exclude tests reject hard links, oversized files, nonregular paths, info-directory substitution, and replacement during the descriptor read. They preserve the target and replacement source bytes. - Preserve the failed attempts: an initial migration fixture changed its home after constructing the source path. Later expiration fixture attempts used a filtered current-head tip, omitted the local-source callback, or used cleanup that generated Git server-info files or left packed unreachable objects. Correct those setup assumptions before the actual regressions fail. Final affected suites pass. - Two earlier local full-suite attempts were interrupted before editing valid review findings. Neither is reported as a full-suite PASS. Final-head full CI passes; review is 5/5 with no open findings. ## Risks Archived imports use additional private disk space for independent Git objects. Missing, changed, or unsafe sources still block import. The repair does not weaken the default linked-worktree ownership check, change stored formats, or update live provider state. The shared checkout addition is optional and has no effect for existing callers that do not supply an object source. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DEVELOPING.md | 8 + ...-existing-workspace-ephemeral-migration.md | 11 + .../archived-nested-repository.test.ts | 212 ++++++++++++++++++ .../__tests__/git-repository-excludes.test.ts | 60 +++++ .../src/services/git-repository-excludes.ts | 37 +++ .../src/services/git-restore-point-storage.ts | 4 + server/src/services/git-workspace-contents.ts | 39 +++- .../services/managed-repository-checkout.ts | 29 ++- .../remote-workspace-migration-git.ts | 39 +--- .../remote-workspace-migration.test.ts | 44 +++- 10 files changed, 434 insertions(+), 49 deletions(-) create mode 100644 server/src/__tests__/archived-nested-repository.test.ts create mode 100644 server/src/__tests__/git-repository-excludes.test.ts create mode 100644 server/src/services/git-repository-excludes.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 9c35a2ccadd..7de63c7af15 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1945,6 +1945,14 @@ point. This keeps saved ignored files, including recovery receipts and bundles, in their original classification. The source ignore file stays unchanged. Symlinked, oversized, or changing source ignore files block reconstruction. +Archived imports restore nested linked checkouts as private standalone +repositories. The saved branch name belongs to the old path, so reconstruction +does not register a new worktree or reset a branch in the shared project cache. +A verified cache can supply Git objects without copying its current working +files. Each private copy retains the restore point's extra commits under local +refs so they survive portable archive capture and garbage collection. Ordinary +linked-worktree recovery still requires its original path ownership. + Shared physical folders require Cloud's offline enrollment maintenance process. A task-level hold cannot fence another workspace record pointing at the same folder. Maintenance verifies that legacy deployments have stopped and that no diff --git a/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md b/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md index 0cb9258466d..f382f2fd1f9 100644 --- a/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md +++ b/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md @@ -41,6 +41,17 @@ namespace- and origin-checked cache. It reconstructs only the recorded commit, never today's working tree. Ordinary private-checkout and restore-point validation stays strict. Missing/revoked receipts, foreign caches and changed origins still block publication. +Archived reconstruction of a saved nested linked checkout uses a private +standalone repository at the new path. It reads only the saved commit and +bundle prerequisites from the verified shared cache, and imports authenticated +restore-point history into private refs. Other tasks' branch refs and unrelated +objects do not enter the archive. Missing prerequisites use the approved remote +or block restoration. Root and nested private copies retain all saved extra +commits after portable capture and garbage collection. Ordinary linked recovery +keeps its path-owned branch check and its existing worktree history owner. +Verified local ignore rules preserve the saved classification of recovery files; +unsafe or changed exclude files block import without changing the source cache. + The next slice adds leaf-first offline migration of overlapping folders, verified nested snapshot dependencies, interruption recovery, and dependency retention. Recovery-obligation reconciliation remains a separate gap; these slices do not diff --git a/server/src/__tests__/archived-nested-repository.test.ts b/server/src/__tests__/archived-nested-repository.test.ts new file mode 100644 index 00000000000..ff64db1f4e6 --- /dev/null +++ b/server/src/__tests__/archived-nested-repository.test.ts @@ -0,0 +1,212 @@ +import { execFileSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; +import { afterEach, beforeEach, expect, it, vi } from "vitest"; +import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; +import { createStorageService } from "../storage/service.js"; +import { captureRestorePoint, currentRestorePointSnapshot, sealRestorePoint } from "../services/git-restore-point.js"; +import { assertReproducibleManagedCheckout } from "../services/git-workspace-restoration.js"; +import { GIT_RESTORE_POINT_SCHEMA } from "../services/git-restore-point-storage.js"; +import { captureGitWorkspaceContents, readManagedCheckoutContents, restoreGitWorkspaceContents } from "../services/git-workspace-contents.js"; +import { managedProjectCheckoutName } from "../services/managed-project-checkout-name.js"; +import { addNestedRepositoryWorktree, nestedRepositoryBaseCandidates, restoreNestedRepositoryWorktree } from "../services/nested-repository-worktrees.js"; +import { captureRemoteWorkspaceDirectory, restoreRemoteWorkspaceArchive } from "../services/remote-workspace-snapshot-archive.js"; + +let root: string; + +function git(cwd: string, ...args: string[]) { + return execFileSync("git", ["-C", cwd, "-c", "commit.gpgsign=false", ...args], { encoding: "utf8", + env: { ...process.env, GIT_CONFIG_NOSYSTEM: "1", GIT_CONFIG_GLOBAL: os.devNull, + GIT_AUTHOR_NAME: "Fixture", GIT_AUTHOR_EMAIL: "fixture@example.test", GIT_COMMITTER_NAME: "Fixture", GIT_COMMITTER_EMAIL: "fixture@example.test" } }).trim(); +} + +beforeEach(async () => { + root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "pc-archived-nested-"))); + vi.stubEnv("PAPERCLIP_HOME", path.join(root, "home")); + vi.stubEnv("PAPERCLIP_INSTANCE_ID", "default"); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32)); +}); + +afterEach(async () => { + vi.unstubAllEnvs(); + await fs.rm(root, { recursive: true, force: true }); +}); + +it.each(["warm", "pruned", "missing-prerequisite", "ordinary-pruned"])("privately restores an archived linked checkout at a new path and keeps its files and extra history after cold restore (cache: %s)", async cache => { + const pruned = cache !== "warm"; + const identity = { companyId: randomUUID(), workspaceId: randomUUID() }, projectId = randomUUID(); + const remote = path.join(root, "remote"), old = path.join(root, "old"); + await fs.mkdir(remote); await fs.mkdir(old); + git(remote, "init", "-q", "-b", "main"); + await fs.writeFile(path.join(remote, ".gitignore"), ".claude/settings.local.json\n"); + await fs.writeFile(path.join(remote, "note.txt"), "remote\n"); + git(remote, "add", "."); git(remote, "commit", "-qm", "initial"); + const initial = git(remote, "rev-parse", "HEAD"); + let upstream = initial; + const source = { id: randomUUID(), repoUrl: pathToFileURL(remote).href, repoRef: "main", cwd: null }; + const name = managedProjectCheckoutName(source); + const [base] = nestedRepositoryBaseCandidates({ ...identity, projectId, repoUrl: source.repoUrl }); + await fs.mkdir(path.dirname(base!), { recursive: true }); git(root, "clone", "-q", source.repoUrl, base!); + if (pruned) { + await fs.writeFile(path.join(remote, "remote-advance.txt"), "new prerequisite\n"); + git(remote, "add", "remote-advance.txt"); git(remote, "commit", "-qm", "advance remote"); + upstream = git(remote, "rev-parse", "HEAD"); + } + git(base!, "checkout", "-qb", "paperclip/another-task/private"); + await fs.writeFile(path.join(base!, "other-task-only.txt"), "another task's private work\n"); + git(base!, "add", "other-task-only.txt"); git(base!, "commit", "-qm", "other task"); + const foreignCommit = git(base!, "rev-parse", "HEAD"); + git(base!, "checkout", "-q", "main"); + const oldNested = path.join(old, ".paperclip-repositories", name); + await fs.mkdir(path.dirname(oldNested), { recursive: true }); + await addNestedRepositoryWorktree({ base: base!, cwd: oldNested, repoUrl: source.repoUrl, repoRef: source.repoRef }); + await fs.writeFile(path.join(oldNested, "note.txt"), "unpushed\n"); git(oldNested, "commit", "-qam", "saved"); + const head = git(oldNested, "rev-parse", "HEAD"); + await fs.writeFile(path.join(oldNested, "extra.txt"), "keep abandoned history\n"); + git(oldNested, "add", "extra.txt"); git(oldNested, "commit", "-qm", "extra"); + const extra = git(oldNested, "rev-parse", "HEAD"); git(oldNested, "reset", "--hard", head); + await fs.writeFile(path.join(oldNested, "note.txt"), "staged\n"); git(oldNested, "add", "note.txt"); + await fs.writeFile(path.join(oldNested, "note.txt"), "unstaged\n"); + await fs.writeFile(path.join(base!, ".git", "info", "exclude"), "evidence/\n"); + await fs.mkdir(path.join(oldNested, "evidence")); + await fs.writeFile(path.join(oldNested, "evidence", "repair.json"), "preserved repair\n", { mode: 0o600 }); + await fs.mkdir(path.join(oldNested, ".claude")); + const settings = '{"env":{"SAVED":"private"}}\n'; + await fs.writeFile(path.join(oldNested, ".claude", "settings.local.json"), settings, { mode: 0o600 }); + const captured = (await captureRestorePoint({ cwd: oldNested, upstreamCommits: [upstream], preserveCommits: [extra] }))!; + expect(captured.manifest.preserved).toContain(extra); + const id = randomUUID(), generation = 1; + const sealed = await sealRestorePoint(captured.payload, { ...identity, restorePointId: id, generation }); + const storage = createStorageService(createLocalDiskStorageProvider(path.join(root, "objects"))); + const objectKey = `${identity.companyId}/git-restore-points/${identity.workspaceId}/${generation}/checkouts/${name}/${id}.pcrp`; + await storage.putFile({ companyId: identity.companyId, namespace: "git-restore-points", originalFilename: null, + contentType: "application/octet-stream", objectKey, body: sealed.ciphertext }); + const descriptor = await readManagedCheckoutContents(oldNested, name, source); + const { manifest } = captured; + const contents = await captureGitWorkspaceContents(old, identity, [{ ...descriptor, restorePoint: { + schema: GIT_RESTORE_POINT_SCHEMA, id, objectKey, sha256: sealed.sha256, byteSize: sealed.ciphertext.length, + key: sealed.key, generation, head: manifest.head, index: manifest.index, worktree: manifest.worktree, + ignored: manifest.ignored, modes: manifest.modes, headOnRemote: false, + } }]); + const expectedSnapshot = await currentRestorePointSnapshot(oldNested); + if (pruned) { + expect(manifest.prerequisites).toContain(upstream); + git(base!, "worktree", "remove", "--force", oldNested); + git(base!, "branch", "-D", descriptor.branchName!); + git(base!, "update-ref", "refs/remotes/origin/main", initial); + git(base!, "reflog", "expire", "--expire=now", "--all"); git(base!, "gc", "--prune=now"); + expect(() => git(base!, "cat-file", "-e", `${upstream}^{commit}`)).toThrow(); + } + await fs.writeFile(path.join(base!, "cache-only.txt"), "unrelated active cache files\n"); + const sourceState = () => ({ refs: git(base!, "show-ref"), worktrees: git(base!, "worktree", "list", "--porcelain"), + status: git(base!, "status", "--porcelain"), index: git(base!, "ls-files", "--stage"), checkout: pruned ? null : git(oldNested, "status", "--porcelain") }); + const before = sourceState(); + const moved = path.join(root, "new"); await fs.mkdir(moved); + const restore = (isolateRepositories: boolean, authorizeSource = async () => {}, cwd = moved) => restoreGitWorkspaceContents({ cwd, + ...identity, projectId, contents, storage, isolateRepositories, + prepareLocalSource: async () => async () => {}, resolveGitAuth: async () => null, authorizeSource }); + if (!pruned) await expect(restore(false)).rejects.toThrow("could not be restored"); + else { + await fs.mkdir(path.join(moved, ".paperclip-repositories"), { recursive: true }); + await expect(restoreNestedRepositoryWorktree({ base: base!, cwd: path.join(moved, ".paperclip-repositories", name), + commit: descriptor.commit, branchName: descriptor.branchName, repoRef: source.repoRef, afterAdd: async () => {} })) + .rejects.toThrow("unowned_nested_worktree_branch"); + } + await expect(restore(true, async () => { throw new Error("source_access_revoked"); })).rejects.toThrow("could not be restored"); + const recordedBase = contents.checkouts[0]!.worktree!.base; + contents.checkouts[0]!.worktree!.base = remote; + await expect(restore(true)).rejects.toThrow("could not be restored"); + contents.checkouts[0]!.worktree!.base = recordedBase; + if (cache === "missing-prerequisite") { + git(remote, "reset", "--hard", initial); + git(remote, "reflog", "expire", "--expire=now", "--all"); git(remote, "gc", "--prune=now"); + await expect(restore(true)).rejects.toThrow("could not be restored"); + expect(sourceState()).toEqual(before); + return; + } + if (cache === "ordinary-pruned") { + await restore(false, undefined, old); + expect((await fs.lstat(path.join(oldNested, ".git"))).isFile()).toBe(true); + expect(await currentRestorePointSnapshot(oldNested)).toEqual(expectedSnapshot); + expect(git(oldNested, "reflog", "show", "--format=%H", "HEAD").split("\n")).toContain(extra); + return; + } + const other = path.join(root, "other-import"); await fs.mkdir(other); + await Promise.all([restore(true), restore(true, undefined, other)]); + expect(await currentRestorePointSnapshot(path.join(other, ".paperclip-repositories", name))) + .toEqual(expectedSnapshot); + const nested = path.join(moved, ".paperclip-repositories", name); + expect((await fs.lstat(path.join(nested, ".git"))).isDirectory()).toBe(true); + expect(git(nested, "for-each-ref", "--format=%(refname)", "refs/remotes/origin/paperclip/another-task")).toBe(""); + expect(() => git(nested, "cat-file", "-e", `${foreignCommit}^{commit}`)).toThrow(); + expect(await currentRestorePointSnapshot(nested)).toEqual(expectedSnapshot); + expect(git(nested, "symbolic-ref", "--short", "HEAD")).toBe(descriptor.branchName); + expect(await fs.readFile(path.join(nested, ".claude", "settings.local.json"), "utf8")).toBe(settings); + await expect(fs.access(path.join(nested, "cache-only.txt"))).rejects.toMatchObject({ code: "ENOENT" }); + expect(sourceState()).toEqual(before); + const archivePath = path.join(root, "workspace.tar"); + const archive = await captureRemoteWorkspaceDirectory({ cwd: moved, archivePath }); + await fs.rm(old, { recursive: true }); await fs.rm(base!, { recursive: true }); + await fs.rm(moved, { recursive: true }); await fs.rm(other, { recursive: true }); await fs.rm(remote, { recursive: true }); + const cold = path.join(root, "cold"); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir: cold, expectedManifestDigest: archive.manifestDigest }); + const coldNested = path.join(cold, ".paperclip-repositories", name); + expect(git(coldNested, "rev-parse", "HEAD")).toBe(head); + expect(git(coldNested, "show", ":note.txt")).toBe("staged"); + expect(await fs.readFile(path.join(coldNested, "note.txt"), "utf8")).toBe("unstaged\n"); + expect(await fs.readFile(path.join(coldNested, "evidence", "repair.json"), "utf8")).toBe("preserved repair\n"); + expect(await fs.readFile(path.join(coldNested, ".claude", "settings.local.json"), "utf8")).toBe(settings); + git(coldNested, "reflog", "expire", "--expire=now", "--all"); git(coldNested, "gc", "--prune=now"); + expect(git(coldNested, "show", `${extra}:extra.txt`)).toBe("keep abandoned history"); + await expect(fs.access(path.join(coldNested, ".git", "objects", "info", "alternates"))).rejects.toMatchObject({ code: "ENOENT" }); +}, 60_000); + +it.each(["new", "existing"])("keeps archive-only refs out of ordinary standalone recovery (%s)", async state => { + const identity = { companyId: randomUUID(), workspaceId: randomUUID() }; + const remote = path.join(root, "remote"), old = path.join(root, "old"), moved = path.join(root, "moved"); + await fs.mkdir(remote); await fs.mkdir(old); await fs.mkdir(moved); + git(remote, "init", "-q", "-b", "main"); + await fs.writeFile(path.join(remote, "note.txt"), "remote\n"); + git(remote, "add", "."); git(remote, "commit", "-qm", "initial"); + const upstream = git(remote, "rev-parse", "HEAD"); + const source = { id: randomUUID(), repoUrl: pathToFileURL(remote).href, repoRef: "main", cwd: null }; + const name = managedProjectCheckoutName(source), original = path.join(old, ".paperclip-repositories", name); + await fs.mkdir(path.dirname(original), { recursive: true }); git(root, "clone", "-q", source.repoUrl, original); + await fs.writeFile(path.join(original, "note.txt"), "saved local commit\n"); + git(original, "commit", "-qam", "saved local commit"); + const head = git(original, "rev-parse", "HEAD"); + await fs.writeFile(path.join(original, "abandoned.txt"), "saved extra history\n"); + git(original, "add", "abandoned.txt"); git(original, "commit", "-qm", "extra history"); + const extra = git(original, "rev-parse", "HEAD"); git(original, "reset", "--hard", head); + const captured = (await captureRestorePoint({ cwd: original, upstreamCommits: [upstream] }))!; + expect(captured.manifest.preserved).toContain(extra); + const id = randomUUID(), generation = 1, sealed = await sealRestorePoint(captured.payload, { ...identity, restorePointId: id, generation }); + const storage = createStorageService(createLocalDiskStorageProvider(path.join(root, "objects"))); + const objectKey = `${identity.companyId}/git-restore-points/${identity.workspaceId}/${generation}/checkouts/${name}/${id}.pcrp`; + await storage.putFile({ companyId: identity.companyId, namespace: "git-restore-points", originalFilename: null, + contentType: "application/octet-stream", objectKey, body: sealed.ciphertext }); + const { manifest } = captured; + const descriptor = await readManagedCheckoutContents(original, name, source); + const contents = await captureGitWorkspaceContents(old, identity, [{ ...descriptor, restorePoint: { + schema: GIT_RESTORE_POINT_SCHEMA, id, objectKey, sha256: sealed.sha256, byteSize: sealed.ciphertext.length, + key: sealed.key, generation, head: manifest.head, index: manifest.index, worktree: manifest.worktree, + ignored: manifest.ignored, modes: manifest.modes, headOnRemote: false, + } }]); + const target = path.join(moved, ".paperclip-repositories", name); + if (state === "existing") { + await fs.mkdir(path.dirname(target), { recursive: true }); git(root, "clone", "-q", "--no-hardlinks", original, target); + git(target, "remote", "set-url", "origin", source.repoUrl); + } + await restoreGitWorkspaceContents({ cwd: moved, ...identity, contents, storage, + prepareLocalSource: async () => async () => {}, resolveGitAuth: async () => null, authorizeSource: async () => {} }); + expect(git(target, "for-each-ref", "--format=%(refname)", "refs/paperclip-preserved")).toBe(""); + expect(await fs.readFile(path.join(target, "note.txt"), "utf8")).toBe("saved local commit\n"); + git(target, "reflog", "expire", "--expire=now", "--all"); + git(target, "-c", "repack.updateServerInfo=false", "repack", "-Ad"); + git(target, "prune", "--expire", "now"); + await expect(assertReproducibleManagedCheckout(target, { repoUrl: source.repoUrl, resolveGitAuth: async () => null, restorePoint: true })) + .resolves.toMatchObject({ headOnRemote: false }); +}); diff --git a/server/src/__tests__/git-repository-excludes.test.ts b/server/src/__tests__/git-repository-excludes.test.ts new file mode 100644 index 00000000000..c38c4bb60ca --- /dev/null +++ b/server/src/__tests__/git-repository-excludes.test.ts @@ -0,0 +1,60 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { execFileSync } from "node:child_process"; +import { afterEach, beforeEach, expect, it, vi } from "vitest"; +import { copyVerifiedGitExcludes } from "../services/git-repository-excludes.js"; + +let root: string, source: string, target: string; +const original = "keep-target-unchanged\n"; +const exclude = (cwd: string) => path.join(cwd, ".git", "info", "exclude"); + +beforeEach(async () => { + root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "pc-git-excludes-"))); + source = path.join(root, "source"); target = path.join(root, "target"); + for (const cwd of [source, target]) { + await fs.mkdir(cwd); + execFileSync("git", ["init", "--quiet", cwd], { env: { ...process.env, GIT_CONFIG_NOSYSTEM: "1", GIT_CONFIG_GLOBAL: os.devNull } }); + } + await fs.writeFile(exclude(source), "saved-recovery/\n"); + await fs.writeFile(exclude(target), original); +}); + +afterEach(async () => { + vi.restoreAllMocks(); + await fs.rm(root, { recursive: true, force: true }); +}); + +it.each(["hardlink", "oversized", "directory", "info-symlink", "info-file"])("rejects an unsafe source (%s) without changing the target", async kind => { + if (kind === "hardlink") await fs.link(exclude(source), path.join(root, "linked-exclude")); + if (kind === "oversized") await fs.writeFile(exclude(source), Buffer.alloc(1024 * 1024 + 1)); + if (kind === "directory") { await fs.rm(exclude(source)); await fs.mkdir(exclude(source)); } + if (kind === "info-symlink" || kind === "info-file") { + const info = path.dirname(exclude(source)), outside = path.join(root, "outside-info"); + await fs.rename(info, outside); + if (kind === "info-symlink") await fs.symlink(outside, info); + else await fs.writeFile(info, "not a directory\n"); + } + await expect(copyVerifiedGitExcludes(source, target)).rejects.toThrow("unsafe_git_excludes"); + expect(await fs.readFile(exclude(target), "utf8")).toBe(original); +}); + +it("rejects an exclude replaced during the descriptor read", async () => { + const replacement = path.join(root, "replacement"); await fs.writeFile(replacement, "new-source-policy/\n"); + const open = fs.open.bind(fs); + vi.spyOn(fs, "open").mockImplementation(async (...args) => { + const handle = await open(...args); + if (String(args[0]) === exclude(source)) { + const read = handle.read.bind(handle); let changed = false; + vi.spyOn(handle, "read").mockImplementation(async (...readArgs) => { + const result = await Reflect.apply(read, handle, readArgs); + if (!changed) { changed = true; await fs.rename(replacement, exclude(source)); } + return result; + }); + } + return handle; + }); + await expect(copyVerifiedGitExcludes(source, target)).rejects.toThrow("unsafe_git_excludes"); + expect(await fs.readFile(exclude(target), "utf8")).toBe(original); + expect(await fs.readFile(exclude(source), "utf8")).toBe("new-source-policy/\n"); +}); diff --git a/server/src/services/git-repository-excludes.ts b/server/src/services/git-repository-excludes.ts new file mode 100644 index 00000000000..b5482685af8 --- /dev/null +++ b/server/src/services/git-repository-excludes.ts @@ -0,0 +1,37 @@ +import { constants } from "node:fs"; +import fs from "node:fs/promises"; +import path from "node:path"; +import { gitCacheCommand } from "./git-workspace-restoration.js"; + +export async function copyVerifiedGitExcludes(source: string, cwd: string) { + const common = await fs.realpath(path.resolve(source, (await gitCacheCommand(source, ["rev-parse", "--path-format=absolute", "--git-common-dir"])).trim())); + const info = path.join(common, "info"), file = path.join(info, "exclude"); + const directory = await fs.lstat(info).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (!directory) return; + if (!directory.isDirectory() || directory.isSymbolicLink()) throw new Error("unsafe_git_excludes"); + const handle = await fs.open(file, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK) + .catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return null; throw new Error("unsafe_git_excludes"); }); + if (!handle) return; + try { + const before = await handle.stat({ bigint: true }); + if (!before.isFile() || before.nlink !== 1n || before.size > 1024n * 1024n) throw new Error("unsafe_git_excludes"); + const bytes = Buffer.alloc(Number(before.size) + 1); let length = 0; + while (length < bytes.length) { + const read = await handle.read(bytes, length, bytes.length - length, length); + if (!read.bytesRead) break; + length += read.bytesRead; + } + const after = await fs.lstat(file, { bigint: true }); + if (!after.isFile() || after.isSymbolicLink() || BigInt(length) !== before.size + || before.dev !== after.dev || before.ino !== after.ino || before.size !== after.size || before.mtimeNs !== after.mtimeNs + || before.mode !== after.mode || before.ctimeNs !== after.ctimeNs || before.nlink !== after.nlink + || await fs.realpath(info) !== info) throw new Error("unsafe_git_excludes"); + const target = path.join(cwd, ".git", "info", "exclude"); + await fs.writeFile(target, bytes.subarray(0, length)); + await fs.chmod(target, Number(before.mode & 0o777n)); + } finally { await handle.close(); } +} + diff --git a/server/src/services/git-restore-point-storage.ts b/server/src/services/git-restore-point-storage.ts index bb32379d92a..311d3c4da20 100644 --- a/server/src/services/git-restore-point-storage.ts +++ b/server/src/services/git-restore-point-storage.ts @@ -282,6 +282,7 @@ export async function applyGitRestorePointToWorktree(input: { ref: GitRestorePointRef; repositoryCwd: string; worktreePath: string; + preserveImportedHistory?: boolean; signal?: AbortSignal; }) { const current = await currentRestorePointSnapshot(input.worktreePath, input.signal); @@ -291,6 +292,9 @@ export async function applyGitRestorePointToWorktree(input: { } const { payload } = await loadGitRestorePoint(input); const manifest = await importRestorePoint({ repositoryCwd: input.repositoryCwd, payload, signal: input.signal }); + if (input.preserveImportedHistory) { + for (const commit of manifest.preserved) await gitCacheCommand(input.repositoryCwd, ["update-ref", `refs/paperclip-preserved/${commit}`, commit], input.signal); + } if (!matches) await applyRestorePoint({ cwd: input.worktreePath, manifest, signal: input.signal }); } diff --git a/server/src/services/git-workspace-contents.ts b/server/src/services/git-workspace-contents.ts index e0f7802da6e..9e138816044 100644 --- a/server/src/services/git-workspace-contents.ts +++ b/server/src/services/git-workspace-contents.ts @@ -12,6 +12,7 @@ import { managedProjectCheckoutName } from "./managed-project-checkout-name.js"; import { materializeManagedProjectWorkspace } from "./managed-repository-checkout.js"; import { ensureSavedGitCommit, gitCacheCommand, GitCacheRestorationFailure } from "./git-workspace-restoration.js"; import { applyGitRestorePointToWorktree, ensureGitRestorePointCommit, loadGitRestorePoint, type GitRestorePointRef } from "./git-restore-point-storage.js"; +import { copyVerifiedGitExcludes } from "./git-repository-excludes.js"; import { importRestorePoint } from "./git-restore-point.js"; import { isNestedWorktreeBranchName, isVerifiedNestedRepositoryBase, nestedRepositoryBaseCandidates, readLinkedNestedCheckout, recordedNestedRepositoryBase, restoreNestedRepositoryWorktree, withNestedRepositoryBaseLock } from "./nested-repository-worktrees.js"; import { getStorageService } from "../storage/index.js"; @@ -34,6 +35,13 @@ export type GitWorkspaceContents = { checkouts: Array; }; +async function ensureCheckoutPrerequisites(cwd: string, checkout: ManagedCheckoutContents, + manifest: { prerequisites: readonly string[] }, resolveGitAuth: GitRemoteAuthProvider) { + const commits = new Set(manifest.prerequisites); + if (checkout.restorePoint?.headOnRemote) commits.add(checkout.commit); + for (const commit of commits) await ensureSavedGitCommit(cwd, checkout.source.repoUrl, commit, resolveGitAuth); +} + async function entry(target: string) { return fs.lstat(target).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return null; @@ -203,6 +211,7 @@ async function restoreSettings(cwd: string, settings: Settings, identity: Identi export async function restoreGitWorkspaceContents(input: Identity & { cwd: string; contents?: GitWorkspaceContents; projectId?: string | null; prepareLocalSource?: (sourceCwd: string) => Promise<() => Promise>; storage?: StorageService; + isolateRepositories?: boolean; resolveGitAuth: GitRemoteAuthProvider; authorizeSource: (source: ManagedCheckoutSource) => Promise }) { if (!input.contents) return; validateGitWorkspaceContents(input.contents); @@ -220,7 +229,7 @@ export async function restoreGitWorkspaceContents(input: Identity & { cwd: strin const candidates = checkout.worktree && input.projectId ? nestedRepositoryBaseCandidates({ companyId: input.companyId, projectId: input.projectId, repoUrl: checkout.source.repoUrl }) : null; if (checkout.worktree && !candidates) throw new GitCacheRestorationFailure(); - if (existing && checkout.worktree) { + if (existing && checkout.worktree && !input.isolateRepositories) { const linked = existing.isDirectory() && !existing.isSymbolicLink() ? await readLinkedNestedCheckout(destination, candidates!) : null; if (!linked || linked.checkout !== destination || !await isVerifiedNestedRepositoryBase(linked.base, candidates!, checkout.source.repoUrl)) { throw new GitCacheRestorationFailure(); @@ -235,11 +244,12 @@ export async function restoreGitWorkspaceContents(input: Identity & { cwd: strin const actual = await readManagedCheckoutContents(destination, checkout.name, checkout.source); if (actual.commit !== checkout.commit || actual.branchName !== checkout.branchName) throw new GitCacheRestorationFailure(); if (checkout.restorePoint) await applyGitRestorePointToWorktree({ storage: input.storage ?? getStorageService(), workspace, - ref: checkout.restorePoint, repositoryCwd: destination, worktreePath: destination }); + ref: checkout.restorePoint, repositoryCwd: destination, worktreePath: destination, + preserveImportedHistory: input.isolateRepositories }); await restoreSettings(destination, checkout.settings, input, relative); continue; } - if (checkout.worktree) { + if (checkout.worktree && !input.isolateRepositories) { const recordedBase = await recordedNestedRepositoryBase(checkout.worktree.base, candidates!); if (!recordedBase) throw new GitCacheRestorationFailure(); if (!await isVerifiedNestedRepositoryBase(recordedBase, candidates!, checkout.source.repoUrl)) { @@ -252,6 +262,7 @@ export async function restoreGitWorkspaceContents(input: Identity & { cwd: strin let preserved: string[] = []; if (checkout.restorePoint) { const { payload, manifest } = await loadGitRestorePoint({ storage: input.storage ?? getStorageService(), workspace, ref: checkout.restorePoint }); + await ensureCheckoutPrerequisites(base, checkout, manifest, input.resolveGitAuth); await importRestorePoint({ repositoryCwd: base, payload }); preserved = manifest.preserved; } @@ -269,20 +280,38 @@ export async function restoreGitWorkspaceContents(input: Identity & { cwd: strin }); continue; } + let gitObjectSource: string | undefined; + if (checkout.worktree) { + const recorded = await recordedNestedRepositoryBase(checkout.worktree.base, candidates!); + if (!recorded) throw new GitCacheRestorationFailure(); + if (await entry(recorded)) { + if (!await isVerifiedNestedRepositoryBase(recorded, candidates!, checkout.source.repoUrl)) throw new GitCacheRestorationFailure(); + gitObjectSource = await fs.realpath(recorded); + } + } + const savedRestorePoint = checkout.restorePoint && input.isolateRepositories + ? await loadGitRestorePoint({ storage: input.storage ?? getStorageService(), workspace, ref: checkout.restorePoint }) : null; const result = await materializeManagedProjectWorkspace(destination, { repoUrl: checkout.source.repoUrl, + gitObjectSource: gitObjectSource ? { cwd: gitObjectSource, commits: [checkout.commit, ...(savedRestorePoint?.manifest.prerequisites ?? [])] } : undefined, resolveGitAuth: input.resolveGitAuth, beforePublish: async temporary => { - if (checkout.restorePoint) await ensureGitRestorePointCommit({ storage: input.storage ?? getStorageService(), workspace, + if (gitObjectSource && checkout.restorePoint) await copyVerifiedGitExcludes(gitObjectSource, temporary); + if (savedRestorePoint) { + const { payload, manifest } = savedRestorePoint; + await ensureCheckoutPrerequisites(temporary, checkout, manifest, input.resolveGitAuth); + await importRestorePoint({ repositoryCwd: temporary, payload }); + } else if (checkout.restorePoint) await ensureGitRestorePointCommit({ storage: input.storage ?? getStorageService(), workspace, ref: checkout.restorePoint, repositoryCwd: temporary }); await ensureSavedGitCommit(temporary, checkout.source.repoUrl, checkout.commit, input.resolveGitAuth); if (checkout.branchName) await gitCacheCommand(temporary, ["check-ref-format", "--branch", checkout.branchName]); await gitCacheCommand(temporary, checkout.branchName ? ["checkout", "-B", checkout.branchName, checkout.commit] : ["checkout", "--detach", checkout.commit]); if (checkout.restorePoint) await applyGitRestorePointToWorktree({ storage: input.storage ?? getStorageService(), workspace, - ref: checkout.restorePoint, repositoryCwd: temporary, worktreePath: temporary }); + ref: checkout.restorePoint, repositoryCwd: temporary, worktreePath: temporary, preserveImportedHistory: input.isolateRepositories }); await gitCacheCommand(temporary, ["reflog", "expire", "--expire=now", "--all"]); await restoreSettings(temporary, checkout.settings, input, relative); await input.authorizeSource(checkout.source); await recheck?.(); + if (gitObjectSource && !await isVerifiedNestedRepositoryBase(gitObjectSource, candidates!, checkout.source.repoUrl)) throw new GitCacheRestorationFailure(); } }); if (result.warning) throw new GitCacheRestorationFailure(); } diff --git a/server/src/services/managed-repository-checkout.ts b/server/src/services/managed-repository-checkout.ts index d6ac5bc6576..b077b9ab55f 100644 --- a/server/src/services/managed-repository-checkout.ts +++ b/server/src/services/managed-repository-checkout.ts @@ -14,6 +14,21 @@ import { resolveRepositoryObjectCache, readRepositoryObjectCache, publishReposit const execFile = promisify(execFileCallback); const MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS = 10 * 60 * 1000; +async function copyManagedGitObjects(cwd: string, repoUrl: string, source: { cwd: string; commits: readonly string[] }) { + const git = (directory: string, args: string[]) => execFile("git", ["-C", directory, ...args], { + env: { ...sanitizeRuntimeServiceBaseEnv(process.env), GIT_TERMINAL_PROMPT: "0" }, + timeout: MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS, + }); + const present = (directory: string, commit: string) => git(directory, ["cat-file", "-e", `${commit}^{commit}`]).then(() => true, () => false); + if (source.commits.some(commit => !/^[a-f0-9]{40}$/.test(commit))) throw new Error("Invalid managed Git object source"); + await git(cwd, ["init", "--quiet"]); + await git(cwd, ["remote", "add", "origin", repoUrl]); + for (const commit of new Set(source.commits)) { + if (await present(cwd, commit) || !await present(source.cwd, commit)) continue; + await git(cwd, ["fetch", "--no-tags", "--no-recurse-submodules", "--no-write-fetch-head", "--no-auto-maintenance", "--", source.cwd, commit]); + } +} + /** The requested branch, tag, or commit is absent from a complete fresh clone. */ export class RepositoryRefNotFoundError extends Error { constructor(message: string, readonly requestedRef: string) { @@ -44,6 +59,7 @@ export async function materializeManagedProjectWorkspace( repoUrl: string | null; repoRef?: string | null; localSource?: string | null; + gitObjectSource?: { cwd: string; commits: readonly string[] }; resolveGitAuth?: GitRemoteAuthProvider | null; beforeCopy?: (cloneCwd: string) => Promise; beforePublish?: (cloneCwd: string) => Promise; @@ -51,6 +67,7 @@ export async function materializeManagedProjectWorkspace( objectCacheCompanyDirectory?: string; }, ): Promise<{ cwd: string; warning: string | null }> { + if (input.localSource && input.gitObjectSource) throw new Error("Conflicting managed checkout sources"); await fs.mkdir(path.dirname(cwd), { recursive: true }); const stats = await fs.stat(cwd).catch(() => null); @@ -81,14 +98,14 @@ export async function materializeManagedProjectWorkspace( await fs.rm(cwd, { recursive: true, force: true }); } - let auth = input.resolveGitAuth && !input.localSource + let auth = input.resolveGitAuth && !input.localSource && !input.gitObjectSource ? await measureRunPhase("workspace.checkout.git_auth", () => input.resolveGitAuth!(input.repoUrl!)) : null; const cloneTmpDir = await fs.mkdtemp(input.stagingParent ? path.join(input.stagingParent, `${path.basename(cwd)}.clone-`) : `${cwd}.clone-`); const cloneSource = input.localSource ?? input.repoUrl; - let objectCache = input.objectCacheCompanyDirectory && !input.localSource + let objectCache = input.objectCacheCompanyDirectory && !input.localSource && !input.gitObjectSource ? await measureRunPhase("workspace.checkout.cache_lookup", async () => { const cache = await resolveRepositoryObjectCache(input.objectCacheCompanyDirectory!, input.repoUrl!, auth); if (!cache) recordRepositoryObjectCacheDetail({ outcome: "disabled" }); @@ -102,9 +119,10 @@ export async function materializeManagedProjectWorkspace( try { await input.beforeCopy?.(cloneTmpDir); try { - const clone = (referencePath: string | null) => measureRunPhase("workspace.checkout.git_clone", () => { + const clone = (referencePath: string | null) => measureRunPhase("workspace.checkout.git_clone", async () => { recordRepositoryObjectCacheDetail({ referenceSupplied: referencePath !== null }); - return execFile( + if (input.gitObjectSource) return copyManagedGitObjects(cloneTmpDir, input.repoUrl!, input.gitObjectSource); + await execFile( "git", [...(auth?.configArgs ?? []), ...(referencePath ? ["-c", "repack.updateServerInfo=false"] : []), "clone", "--no-hardlinks", ...(referencePath ? ["--reference-if-able", referencePath, "--dissociate"] : []), "--", cloneSource, cloneTmpDir], { @@ -126,7 +144,7 @@ export async function materializeManagedProjectWorkspace( await clone(null); } } catch (error) { - const connectionFailure = !input.localSource ? classifyGitCloneFailure(input.repoUrl, error) : null; + const connectionFailure = !input.localSource && !input.gitObjectSource ? classifyGitCloneFailure(input.repoUrl, error) : null; if (connectionFailure) throw new GitConnectionFailureError( error instanceof Error ? error.message : "Git clone failed", connectionFailure, ); @@ -162,6 +180,7 @@ export async function materializeManagedProjectWorkspace( ); } } + if (input.gitObjectSource) await execFile("git", ["-C", cloneTmpDir, "remote", "set-url", "origin", input.repoUrl], { timeout: 10_000 }); await input.beforePublish?.(cloneTmpDir); } catch (error) { await fs diff --git a/server/src/services/remote-workspace-migration-git.ts b/server/src/services/remote-workspace-migration-git.ts index 242e38c9e53..a80623b7ec3 100644 --- a/server/src/services/remote-workspace-migration-git.ts +++ b/server/src/services/remote-workspace-migration-git.ts @@ -1,5 +1,4 @@ import { execFile } from "node:child_process"; -import { constants } from "node:fs"; import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; @@ -10,6 +9,7 @@ import { PROJECT_REPOSITORIES_DIR } from "@paperclipai/adapter-utils/git-workspa import type { StorageService } from "../storage/types.js"; import { createIssueGitRemoteAuthProvider } from "./git-credentials.js"; import { applyGitRestorePointToWorktree, ensureGitRestorePointCommit, loadGitRestorePoint } from "./git-restore-point-storage.js"; +import { copyVerifiedGitExcludes } from "./git-repository-excludes.js"; import { restoreGitWorkspaceContents, type ManagedCheckoutSource } from "./git-workspace-contents.js"; import { ensureSavedGitCommit, gitRestorePointExpired, managedSourcePaths, readGitCacheRestoration, verifiedGitCacheSource, verifiedManagedGitObjectSource, type GitCacheRestoration } from "./git-workspace-restoration.js"; @@ -181,37 +181,6 @@ export async function lockArchivedGitMigrationSources(db: Pick, wo } } -async function copyArchivedGitExcludes(source: string, cwd: string) { - const common = await fs.realpath(path.resolve(source, await git(source, ["rev-parse", "--path-format=absolute", "--git-common-dir"]))); - const info = path.join(common, "info"), file = path.join(info, "exclude"); - const directory = await fs.lstat(info).catch((error: NodeJS.ErrnoException) => { - if (error.code === "ENOENT") return null; - throw error; - }); - if (!directory) return; - if (!directory.isDirectory() || directory.isSymbolicLink()) throw unavailable(); - const handle = await fs.open(file, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK) - .catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return null; throw unavailable(); }); - if (!handle) return; - try { - const before = await handle.stat({ bigint: true }); - if (!before.isFile() || before.nlink !== 1n || before.size > 1024n * 1024n) throw unavailable(); - const bytes = Buffer.alloc(Number(before.size) + 1); let length = 0; - while (length < bytes.length) { - const read = await handle.read(bytes, length, bytes.length - length, length); - if (!read.bytesRead) break; - length += read.bytesRead; - } - const after = await fs.lstat(file, { bigint: true }); - if (!after.isFile() || after.isSymbolicLink() || BigInt(length) !== before.size - || before.dev !== after.dev || before.ino !== after.ino || before.size !== after.size || before.mtimeNs !== after.mtimeNs - || before.mode !== after.mode || before.ctimeNs !== after.ctimeNs || before.nlink !== after.nlink - || await fs.realpath(info) !== info) throw unavailable(); - const target = path.join(cwd, ".git", "info", "exclude"); - await fs.writeFile(target, bytes.subarray(0, length)); - await fs.chmod(target, Number(before.mode & 0o777n)); - } finally { await handle.close(); } -} export async function materializeArchivedGitMigrationSource(input: { db: Db; storage: StorageService; workspace: Workspace; source: ArchivedGitMigrationSource; scratch: string; @@ -232,7 +201,7 @@ export async function materializeArchivedGitMigrationSource(input: { : await verifiedReceiptObjectSource(workspace, source.sourcePath); if (source.retainedBranch && (await git(verified, ["rev-parse", "--verify", `refs/heads/${source.retainedBranch}^{commit}`])) !== source.commit) throw unavailable(); await fs.mkdir(cwd, { mode: 0o700 }); await git(cwd, ["init", "--quiet"]); - if (source.recipe?.restorePoint) await copyArchivedGitExcludes(verified, cwd); + if (source.recipe?.restorePoint) await copyVerifiedGitExcludes(verified, cwd).catch(() => { throw unavailable(); }); // Fetch objects, never the current working tree of a shared project root. const presentCommit = await git(verified, ["cat-file", "-e", `${source.commit}^{commit}`]).then(() => true, () => false); if (presentCommit) { @@ -262,10 +231,10 @@ export async function materializeArchivedGitMigrationSource(input: { await git(cwd, ["symbolic-ref", "HEAD", `refs/heads/${workspace.branchName}`]); await git(cwd, ["config", "remote.origin.url", workspace.repoUrl]); if (source.recipe?.restorePoint) await applyGitRestorePointToWorktree({ storage: input.storage, workspace, - ref: source.recipe.restorePoint, repositoryCwd: cwd, worktreePath: cwd }); + ref: source.recipe.restorePoint, repositoryCwd: cwd, worktreePath: cwd, preserveImportedHistory: true }); const approvedLocalSources = new Set(); await restoreGitWorkspaceContents({ cwd, companyId: workspace.companyId, workspaceId: workspace.id, - projectId: workspace.projectId, contents: source.recipe?.contents, storage: input.storage, resolveGitAuth, + projectId: workspace.projectId, contents: source.recipe?.contents, storage: input.storage, resolveGitAuth, isolateRepositories: true, authorizeSource: async saved => { await authorizeNestedSource(input.db, workspace, saved); const local = localRepositoryPath(saved.repoUrl); if (local) approvedLocalSources.add(local); diff --git a/server/src/services/remote-workspace-migration.test.ts b/server/src/services/remote-workspace-migration.test.ts index 474bef43691..9da4182d8f1 100644 --- a/server/src/services/remote-workspace-migration.test.ts +++ b/server/src/services/remote-workspace-migration.test.ts @@ -29,6 +29,7 @@ import { resumeNativeWorkspaceSync } from "./native-runtime/native-workspace-syn import { claimNativeSessionResumptions } from "./native-runtime/native-finalization-reconciler.js"; import { claimNativeRestartRecoveries } from "./native-runtime/native-restart-recovery.js"; import { attemptNativeWorkspaceFinalization } from "./native-runtime/native-workspace-finalizer.js"; +import { addNestedRepositoryWorktree, nestedRepositoryBaseCandidates } from "./nested-repository-worktrees.js"; import { managedProjectCheckoutName } from "./managed-project-checkout-name.js"; import { captureGitWorkspaceContents, readManagedCheckoutContents } from "./git-workspace-contents.js"; import { isolatedRemoteSourcePath } from "./isolated-remote-workspace-source.js"; @@ -660,10 +661,16 @@ describe.skipIf(!support.supported)("remote workspace migration", () => { const address = server.address(); if (!address || typeof address === "string") throw new Error("test server not bound"); return { bare, repoUrl: `http://127.0.0.1:${address.port}/repository.git` }; } - it.each(["present", "pruned", "retry", "source-excludes", "source-excludes-symlink", "missing-base", "missing-base-retry", "missing-base-uploaded-retry", "missing-base-changed-retry", "nested", "nested-retry", "nested-source-changed-retry", "unapproved-nested", "missing", "expired", "corrupt"])("preserves archived Git edits or refuses an unavailable restore point (%s)", async state => { + it.each(["present", "root-history", "pruned", "retry", "source-excludes", "source-excludes-symlink", "missing-base", "missing-base-retry", "missing-base-uploaded-retry", "missing-base-changed-retry", "nested", "nested-linked", "nested-retry", "nested-source-changed-retry", "unapproved-nested", "missing", "expired", "corrupt"])("preserves archived Git edits or refuses an unavailable restore point (%s)", async state => { const remote = state.startsWith("missing-base") ? await serveRepository() : null; const f = await gitArchiveFixture(remote?.repoUrl); if (remote) { await git(remote.bare, "fetch", f.base, "main:refs/heads/main"); await git(remote.bare, "update-server-info"); } + let rootHistory: string | undefined; + if (state === "root-history") { + await fs.writeFile(path.join(f.cwd, "history-only.txt"), "saved root history\n"); + await git(f.cwd, "add", "history-only.txt"); await git(f.cwd, "commit", "-qm", "root history"); + rootHistory = await git(f.cwd, "rev-parse", "HEAD"); await git(f.cwd, "reset", "--hard", f.commit); + } await fs.writeFile(path.join(f.cwd, "saved.txt"), "staged edit\n"); await git(f.cwd, "add", "saved.txt"); await fs.writeFile(path.join(f.cwd, "saved.txt"), "unstaged edit\n"); if (state.startsWith("source-excludes")) { @@ -675,6 +682,7 @@ describe.skipIf(!support.supported)("remote workspace migration", () => { } const sourceExcludes = state.startsWith("source-excludes") ? await fs.readFile(path.join(f.base, ".git", "info", "exclude")) : null; const captured = (await captureRestorePoint({ cwd: f.cwd, upstreamCommits: [f.upstream] }))!; + if (rootHistory) expect(captured.manifest.preserved).toContain(rootHistory); const id = randomUUID(), generation = 1, sealed = await sealRestorePoint(captured.payload, { companyId: f.companyId, workspaceId: f.workspaceId, restorePointId: id, generation }); const objectKey = `${f.companyId}/git-restore-points/${f.workspaceId}/${generation}/${id}.pcrp`; @@ -691,11 +699,28 @@ describe.skipIf(!support.supported)("remote workspace migration", () => { await git(sourceCwd, "add", "."); await git(sourceCwd, "commit", "-qm", "nested source"); const source = { id: randomUUID(), repoUrl: pathToFileURL(sourceCwd).href, repoRef: "main", cwd: sourceCwd }; const name = managedProjectCheckoutName(source), nestedCwd = path.join(f.cwd, ".paperclip-repositories", name); - await git(f.cwd, "clone", source.repoUrl, nestedCwd); await git(nestedCwd, "checkout", "--detach"); + let nestedPoint; + if (state === "nested-linked") { + const [base] = nestedRepositoryBaseCandidates({ companyId: f.companyId, projectId: f.projectId, repoUrl: source.repoUrl }); + await fs.mkdir(path.dirname(base!), { recursive: true }); await git(root, "clone", source.repoUrl, base!); + await fs.mkdir(path.dirname(nestedCwd), { recursive: true }); + await addNestedRepositoryWorktree({ base: base!, cwd: nestedCwd, repoUrl: source.repoUrl, repoRef: "main" }); + await fs.writeFile(path.join(nestedCwd, "nested.txt"), "nested staged edit\n"); await git(nestedCwd, "add", "nested.txt"); + await fs.writeFile(path.join(nestedCwd, "nested.txt"), "nested unstaged edit\n"); + const captured = (await captureRestorePoint({ cwd: nestedCwd, upstreamCommits: [await git(sourceCwd, "rev-parse", "HEAD")] }))!; + const id = randomUUID(), generation = 1, sealed = await sealRestorePoint(captured.payload, + { companyId: f.companyId, workspaceId: f.workspaceId, restorePointId: id, generation }); + const objectKey = `${f.companyId}/git-restore-points/${f.workspaceId}/${generation}/checkouts/${name}/${id}.pcrp`; + await storage.putFile({ companyId: f.companyId, namespace: "git-restore-points", objectKey, + originalFilename: null, contentType: "application/octet-stream", body: sealed.ciphertext }); + nestedPoint = { schema: "paperclip.git-restore-point.v1" as const, id, generation, objectKey, byteSize: sealed.ciphertext.length, + sha256: sealed.sha256, key: sealed.key, head: captured.manifest.head, index: captured.manifest.index, + worktree: captured.manifest.worktree, ignored: captured.manifest.ignored, modes: captured.manifest.modes, headOnRemote: true }; + } else { await git(f.cwd, "clone", source.repoUrl, nestedCwd); await git(nestedCwd, "checkout", "--detach"); } await fs.mkdir(path.join(nestedCwd, ".claude")); await fs.writeFile(path.join(nestedCwd, ".claude", "settings.local.json"), '{"permissions":{"allow":["Read"]}}'); contents = await captureGitWorkspaceContents(f.cwd, { companyId: f.companyId, workspaceId: f.workspaceId }, - [await readManagedCheckoutContents(nestedCwd, name, source)]); + [{ ...await readManagedCheckoutContents(nestedCwd, name, source), ...(nestedPoint ? { restorePoint: nestedPoint } : {}) }]); if (state !== "unapproved-nested") await db.insert(projectWorkspaces).values({ ...source, companyId: f.companyId, projectId: f.projectId, name: "Nested source" }); } await db.update(executionWorkspaces).set({ status: "archived", closedAt: new Date(), metadata: { @@ -808,8 +833,19 @@ describe.skipIf(!support.supported)("remote workspace migration", () => { await loadWorkspaceSnapshotArchive({ storage, reference: ref!, targetPath: archivePath }); await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref!.manifestDigest }); expect(await git(targetDir, "rev-parse", "HEAD")).toBe(f.commit); + if (rootHistory) { + await git(targetDir, "reflog", "expire", "--expire=now", "--all"); await git(targetDir, "gc", "--prune=now"); + expect(await git(targetDir, "show", `${rootHistory}:history-only.txt`)).toBe("saved root history"); + } + if (state === "nested-linked") { + await fs.rm(contents!.checkouts[0]!.worktree!.base, { recursive: true }); + const nested = path.join(targetDir, ".paperclip-repositories", contents!.checkouts[0]!.name); + expect((await fs.lstat(path.join(nested, ".git"))).isDirectory()).toBe(true); + expect(await fs.readFile(path.join(nested, "nested.txt"), "utf8")).toBe("nested unstaged edit\n"); + expect(await git(nested, "show", ":nested.txt")).toBe("nested staged edit"); + } if (state.startsWith("nested")) { - expect(await fs.readFile(path.join(targetDir, ".paperclip-repositories", contents!.checkouts[0]!.name, "nested.txt"), "utf8")).toBe("nested source work\n"); + if (state !== "nested-linked") expect(await fs.readFile(path.join(targetDir, ".paperclip-repositories", contents!.checkouts[0]!.name, "nested.txt"), "utf8")).toBe("nested source work\n"); expect(JSON.parse(await fs.readFile(path.join(targetDir, ".paperclip-repositories", contents!.checkouts[0]!.name, ".claude/settings.local.json"), "utf8"))) .toEqual({ permissions: { allow: ["Read"] } }); }