Files
PaperClipAI/packages/adapter-utils/src/git-workspace-sync.test.ts
T
DottaandPaperclip 0f14d26123 fix(runtime): allow bounded large untracked workspace snapshots (#14194)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Remote runs need a snapshot of the task workspace before the agent
starts.
> - The snapshot lists untracked filenames through the shared Git scan
scheduler.
> - A generated directory with a few thousand long filenames can exceed
the 1 MiB output limit.
> - This stops setup and prevents the agent from continuing its task.
> - This change gives that listing a 32 MiB bound and keeps the explicit
file snapshot.
> - Normal generated trees can now pass setup, while larger snapshots
still fail at a finite limit.

## Linked Issues or Issue Description

Refs #11572 and #12214 for the existing bounded scan and ignore-scan
protections.

**What happened?**

An agent continuation failed during workspace setup with `Workspace Git
scan exceeded its output limit`. A real Git fixture reproduces the
untracked-file path: 5,000 long filenames in one generated directory
exceed its 1 MiB output limit.

**Expected behavior**

The workspace snapshot must support ordinary generated trees with
thousands of files. It must keep a finite output bound and select
explicit files before staging.

**Steps to reproduce**

1. Commit a base file in a Git repository.
2. Add 5,000 untracked files with long names in one new directory.
3. Call `readGitWorkspaceSnapshot` with the normal scan limits.
4. Observe the output-limit error before this change.

**Paperclip version or commit**

Base commit: `640dee1`.

**Deployment mode**

Remote sandbox execution from source.

## What Changed

- Increase the untracked-file snapshot output bound from 1 MiB to 32
MiB.
- Keep explicit file selection, the shared scheduler, the timeout, and
the other scan bounds.
- Test that 40,000 long filenames above the old 8 MiB bound reach the
snapshot. Reuse those files with deeper paths to prove that output above
32 MiB still fails.
- Test that files created after the snapshot, including an ignored
secret, stay out of the overlay archive.
- Check the workspace root identity and reject selected paths with
symlinked parent directories before upload.
- Test root replacement after path resolution, including root-level
selected files.
- Preserve existing workspace root aliases by capturing the resolved
root before snapshot selection. Test that later alias retargeting cannot
change the archive contents.
- Stop staging on permission and I/O errors; continue to allow missing
files.
- Test these failures and preserve selected symlink entries.
- Accept valid case-renamed directories by checking ancestor file types.
A modeled case-insensitive regression failed before this correction and
now passes.
- Give the 40,000-file fixture enough time to remove its files.
- Document the larger bound and staging behavior.

## Verification

- Red: the 5,000-file regression failed with `stdout maxBuffer length
exceeded` before the fix.
- A separate check through the real server scheduler reproduced
`workspace_git_scan_output_limit` on the original code. The revised code
selected all 5,000 files.
- The late-file regression failed against the first PR revision because
the archive contained `drafts/late.secret`. It passes with the final
explicit-file approach.
- The staging regressions failed before the review fix: a substituted
parent directory and permission/I/O errors were accepted. All three
cases now stop before upload.
- Green: 134 tests passed across `git-workspace-sync.test.ts` and
`sandbox-managed-runtime.test.ts` with Vitest 4.1.11. This includes
complete selection above 8 MiB and rejection above 32 MiB.
- `pnpm --filter @paperclipai/adapter-utils... typecheck` passed after
the revision.
- The module-boundary check and `git diff --check` passed.
- `pnpm -r typecheck` and `pnpm build` stopped in the Rust runner steps
because this environment has no `cargo` executable.
- The full `pnpm test:run` attempt ended with `SIGKILL` during the
general server suite. It did not finish. That full-suite result belongs
to the earlier revision. Fresh checks are required for this revision.

- The new regression fails at the old 8 MiB bound with `stdout maxBuffer
length exceeded`. All 134 focused tests pass with the 32 MiB change.
- The affected typechecks and module-boundary check pass. The full local
typecheck requires Cargo, which is absent in this environment.
- Final verification for `b43e9bc95e55382c6a9bfe200487c164770c8be8`: 54
successful checks/statuses and two skipped Storybook checks. No checks
remain pending or failed.
- The [CI
run](https://github.com/paperclipai/paperclip/actions/runs/36315569674)
passes on attempt 2. The first attempt had one unrelated preview-fixture
readiness timeout. That exact test passed locally; its CI shard passed
on the single rerun.
- [Greptile reports
5/5](https://github.com/paperclipai/paperclip/pull/14194#issuecomment-5852045501)
on this revision. All review threads are resolved.
- [Security review accepts the documented memory
tradeoff](https://github.com/paperclipai/paperclip/pull/14194#discussion_r4115149465)
for this finite mitigation. The separate streaming follow-up will remove
full-list buffering.
- This revision also passes affected local typechecks and the
module-boundary check. Full local typecheck/build stop because Cargo is
absent. The full local Vitest attempt was stopped after about 18 minutes
once all remote gates passed; it did not complete locally.

## Risks

- Each untracked-file scan can buffer up to 32 MiB instead of 1 MiB. The
scheduler still limits concurrent scans and execution time.
- Snapshots above 32 MiB still fail with the existing error. Tracked and
ignored-file scan bounds stay unchanged.
- A workspace that replaces a selected path’s parent with a symlink now
fails staging.
- Detailed logs from the reported host were unavailable. The exact
command that exceeded its limit on that host is unconfirmed.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. The runtime does not expose a more specific model build or
context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-27 06:42:19 -05:00

1033 lines
49 KiB
TypeScript

import { execFile as execFileCallback } from "node:child_process";
import { lstat, mkdir, mkdtemp, readFile, readlink, rename, rm, stat, symlink, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { promisify } from "node:util";
import { afterEach, describe, expect, it } from "vitest";
import {
buildRemoteGitDeltaBundleScript,
createImportedGitRef,
createRemoteGitExportRef,
deleteLocalGitRef,
fetchGitBundleIntoLocalRef,
integrateImportedGitHead,
isMissingGitPrerequisiteError,
readGitWorkspaceSnapshot,
ReferencedSourceIgnoreScanLimitExceededError,
readReferencedSourceGitIgnoredPaths,
REFERENCED_SOURCE_IGNORE_MAX_ENTRY_COUNT,
REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES,
runLocalGit,
sanitizeGitRemoteUrl,
setExpensiveWorkspaceGitExecutor,
withShallowGitWorkspaceClone,
} from "./git-workspace-sync.js";
const execFile = promisify(execFileCallback);
async function git(cwd: string, args: string[]): Promise<string> {
return (await runLocalGit(cwd, args)).stdout.trim();
}
describe("git workspace sync", () => {
const cleanupDirs: string[] = [];
afterEach(async () => {
setExpensiveWorkspaceGitExecutor(null);
while (cleanupDirs.length > 0) {
const dir = cleanupDirs.pop();
if (!dir) continue;
await rm(dir, { recursive: true, force: true }).catch(() => undefined);
}
}, 30_000); // The output-limit fixture removes 40,000 files on teardown.
it("delegates every host-side full-tree enumeration to the registered scheduler", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-scheduler-hook-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await writeFile(path.join(repo, "untracked.txt"), "untracked\n", "utf8");
const operations: string[] = [];
setExpensiveWorkspaceGitExecutor(async (input) => {
operations.push(input.operation);
return await runLocalGit(input.localDir, [...input.args], {
timeout: input.timeout,
maxBuffer: input.maxBuffer,
});
});
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot?.overlayPaths).toContain("untracked.txt");
expect(operations.sort()).toEqual([
"adapter_sync.deleted_files",
"adapter_sync.ignored_files",
"adapter_sync.overlay_diff",
"adapter_sync.untracked_files",
]);
});
it("keeps every filename byte for a padded name in each of the four anchor lanes", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-anchor-whitespace-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
// Deleted lane: commit the file first (in isolation, before anything else
// is staged), then remove it from the work tree.
const deletedName = " deleted padded ";
await writeFile(path.join(repo, deletedName), "deleted\n", "utf8");
await git(repo, ["add", deletedName]);
await git(repo, ["commit", "-qm", "add deleted padded"]);
await rm(path.join(repo, deletedName));
// Overlay lane, staged-new half: `git diff --diff-filter=ACMRTUXB HEAD`
// reports a staged-but-uncommitted file as added.
const overlayName = " overlay padded ";
await writeFile(path.join(repo, overlayName), "overlay\n", "utf8");
await git(repo, ["add", overlayName]);
// Overlay lane, untracked half: `ls-files --others --exclude-standard`.
const untrackedName = " untracked padded ";
await writeFile(path.join(repo, untrackedName), "untracked\n", "utf8");
// Ignored lane: a double-wildcard pattern avoids the separate rule that
// Git trims an unescaped trailing space in a .gitignore PATTERN itself;
// the padding under test lives in the matched FILE name.
const ignoredName = " ignored padded ";
await writeFile(path.join(repo, ".gitignore"), "*ignored*padded*\n", "utf8");
await writeFile(path.join(repo, ignoredName), "ignored\n", "utf8");
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot?.overlayPaths).toContain(overlayName);
expect(snapshot?.overlayPaths).toContain(untrackedName);
expect(snapshot?.deletedPaths).toContain(deletedName);
expect(snapshot?.ignoredPaths).toContain(ignoredName);
});
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "workspace_git_scan_output_limit", "workspace_git_scan_cancelled", "workspace_git_scan_failed"])("preserves %s instead of reporting a non-Git folder", async (code) => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-scan-failure-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const failure = Object.assign(new Error("Git enumeration failed"), { code });
setExpensiveWorkspaceGitExecutor(async (input) => {
if (input.operation === "adapter_sync.ignored_files") throw failure;
return runLocalGit(input.localDir, [...input.args]);
});
await expect(readGitWorkspaceSnapshot(repo, false)).rejects.toBe(failure);
});
it("lists ignored paths without traversing ignored directory contents", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-ignored-scan-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await writeFile(path.join(repo, ".gitignore"), "dependencies/\n*.secret\n");
await mkdir(path.join(repo, "dependencies", "nested"), { recursive: true });
await writeFile(path.join(repo, "dependencies", "nested", "private"), "private");
await writeFile(path.join(repo, "token.secret"), "private");
let ignoredArgs: readonly string[] = [];
setExpensiveWorkspaceGitExecutor(async (input) => {
if (input.operation === "adapter_sync.ignored_files") ignoredArgs = input.args;
return runLocalGit(input.localDir, [...input.args]);
});
expect((await readGitWorkspaceSnapshot(repo))?.ignoredPaths).toEqual(["dependencies", "token.secret"]);
expect(ignoredArgs).toEqual(["ls-files", "--others", "--ignored", "--exclude-standard", "--directory", "-z"]);
});
it("snapshots a generated directory with more than 1 MiB of filenames", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-large-untracked-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const generatedDir = path.join(repo, "storybook-output");
await mkdir(generatedDir);
const names = Array.from({ length: 5_000 }, (_, index) => `${"asset-".repeat(36)}${index}.js`);
for (let start = 0; start < names.length; start += 100) {
await Promise.all(names.slice(start, start + 100).map((name) => writeFile(path.join(generatedDir, name), "")));
}
const raw = await runLocalGit(repo, ["ls-files", "--others", "--exclude-standard", "-z"], {
maxBuffer: 2 * 1024 * 1024,
});
expect(Buffer.byteLength(raw.stdout)).toBeGreaterThan(1024 * 1024);
setExpensiveWorkspaceGitExecutor((input) => runLocalGit(input.localDir, [...input.args], {
timeout: input.timeout,
maxBuffer: input.maxBuffer,
}));
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot?.overlayPaths).toEqual(
names.map((name) => `storybook-output/${name}`).sort((left, right) => left.localeCompare(right)),
);
// A larger tree exceeds the old 8 MiB bound but fits the new 32 MiB bound.
for (let start = 5_000; start < 40_000; start += 100) {
await Promise.all(Array.from({ length: 100 }, (_, index) => writeFile(
path.join(generatedDir, `${"asset-".repeat(36)}${start + index}.js`), "",
)));
}
const largerRaw = await runLocalGit(repo, ["ls-files", "--others", "--exclude-standard", "-z"], {
maxBuffer: 32 * 1024 * 1024,
});
expect(Buffer.byteLength(largerRaw.stdout)).toBeGreaterThan(8 * 1024 * 1024);
const largerSnapshot = await readGitWorkspaceSnapshot(repo);
expect(largerSnapshot?.overlayPaths).toEqual(
largerRaw.stdout.split("\0").filter(Boolean).sort((left, right) => left.localeCompare(right)),
);
// Reuse the files with longer parent paths to exceed 32 MiB without
// creating hundreds of thousands of files solely to test the bound.
const deepParent = path.join(repo, ...Array.from({ length: 4 }, () => "nested-".repeat(30)));
await mkdir(deepParent, { recursive: true });
await rename(generatedDir, path.join(deepParent, "storybook-output"));
expect(Buffer.byteLength(largerRaw.stdout) + 40_000 * (path.relative(repo, deepParent).length + 1))
.toBeGreaterThan(32 * 1024 * 1024);
await expect(readGitWorkspaceSnapshot(repo)).rejects.toMatchObject({
code: "ERR_CHILD_PROCESS_STDIO_MAXBUFFER",
});
}, 60_000);
async function createRepo(rootDir: string): Promise<string> {
const repo = path.join(rootDir, "repo");
await mkdir(repo, { recursive: true });
await git(repo, ["init"]);
await git(repo, ["checkout", "-b", "main"]);
await git(repo, ["config", "user.name", "Paperclip Test"]);
await git(repo, ["config", "user.email", "test@paperclip.dev"]);
await writeFile(path.join(repo, "tracked.txt"), "base\n", "utf8");
await git(repo, ["add", "tracked.txt"]);
await git(repo, ["commit", "-m", "base"]);
return repo;
}
it("does not classify a selected repository subfolder as a cloneable repository root", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-selected-folder-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const selectedDir = path.join(repo, "project");
await mkdir(selectedDir);
await writeFile(path.join(selectedDir, "draft.md"), "selected work\n");
expect(await git(selectedDir, ["rev-parse", "--is-inside-work-tree"])).toBe("true");
expect(await readGitWorkspaceSnapshot(selectedDir)).toBeNull();
expect((await readGitWorkspaceSnapshot(repo))?.headCommit).toBe(await git(repo, ["rev-parse", "HEAD"]));
});
it("creates a shallow standalone clone from the local HEAD snapshot", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-sync-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
await rm(path.join(repo, "tracked.txt"));
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot).toMatchObject({
headCommit: baseHead,
branchName: "main",
deletedPaths: ["tracked.txt"],
});
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
expect((await lstat(path.join(cloneDir, ".git"))).isDirectory()).toBe(true);
await expect(readFile(path.join(cloneDir, ".git", "shallow"), "utf8")).resolves.toContain(baseHead);
expect(await git(cloneDir, ["rev-list", "--count", "HEAD"])).toBe("1");
expect(await git(cloneDir, ["branch", "--show-current"])).toBe("main");
await expect(readFile(path.join(cloneDir, "tracked.txt"), "utf8")).resolves.toBe("base\n");
});
});
it.skipIf(process.platform === "win32")("preserves nested repository symlinks after the temporary clone is removed", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-nested-links-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const nested = await createRepo(path.join(repo, ".paperclip-repositories"));
await writeFile(path.join(repo, ".git/info/exclude"), ".paperclip-repositories/\n");
await mkdir(path.join(nested, "skills", "demo"), { recursive: true });
await mkdir(path.join(nested, ".claude", "skills"), { recursive: true });
await writeFile(path.join(nested, "skills", "demo", "SKILL.md"), "skill content\n");
const links = [
[".claude/skills/demo", "../../skills/demo"],
["skill.md", "skills/demo/SKILL.md"],
["skill-alias", ".claude/skills/demo"],
["future", "future.txt"],
] as const;
for (const [name, target] of links) await symlink(target, path.join(nested, name));
await git(nested, ["add", "."]);
await git(nested, ["commit", "-m", "add repository links"]);
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot?.repositories).toHaveLength(1);
await withShallowGitWorkspaceClone({ localDir: repo, snapshot: snapshot! }, async (cloneDir) => {
// The nested clone's callback has already returned and deleted its temp
// directory. Relative links must keep their repository meaning here.
const copied = path.join(cloneDir, ".paperclip-repositories", "repo");
for (const [name, target] of links) {
expect((await lstat(path.join(copied, name))).isSymbolicLink()).toBe(true);
expect(await readlink(path.join(copied, name))).toBe(target);
}
expect(await readFile(path.join(copied, ".claude/skills/demo/SKILL.md"), "utf8")).toBe("skill content\n");
expect(await readFile(path.join(copied, "skill-alias/SKILL.md"), "utf8")).toBe("skill content\n");
await expect(stat(path.join(copied, "future"))).rejects.toMatchObject({ code: "ENOENT" });
expect(await git(copied, ["status", "--porcelain"])).toBe("");
});
expect(await git(nested, ["status", "--porcelain"])).toBe("");
});
it("copies the workspace origin remote into the shallow clone", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-origin-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await git(repo, ["remote", "add", "origin", "https://github.com/example/repo.git"]);
const snapshot = await readGitWorkspaceSnapshot(repo);
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
expect(await git(cloneDir, ["remote", "get-url", "origin"])).toBe("https://github.com/example/repo.git");
});
});
it("scrubs credentials from the origin remote before copying it", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-origin-scrub-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await git(repo, ["remote", "add", "origin", "https://x-access-token:sekret@github.com/example/repo.git"]);
const snapshot = await readGitWorkspaceSnapshot(repo);
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
expect(await git(cloneDir, ["remote", "get-url", "origin"])).toBe("https://github.com/example/repo.git");
});
});
it("leaves the shallow clone remote-less when the workspace has no origin", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-no-origin-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const snapshot = await readGitWorkspaceSnapshot(repo);
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
await expect(git(cloneDir, ["remote", "get-url", "origin"])).rejects.toThrow();
});
});
it("drops a filesystem-path origin instead of copying it into the shallow clone", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-path-origin-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await git(repo, ["remote", "add", "origin", path.join(rootDir, "elsewhere.git")]);
const snapshot = await readGitWorkspaceSnapshot(repo);
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
await expect(git(cloneDir, ["remote", "get-url", "origin"])).rejects.toThrow();
});
});
it("pushes new commits from the shallow clone to an origin that holds the base commit", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-shallow-push-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const upstream = path.join(rootDir, "upstream.git");
await mkdir(upstream, { recursive: true });
await git(upstream, ["init", "--bare"]);
await git(repo, ["remote", "add", "origin", upstream]);
await git(repo, ["push", "origin", "main"]);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
const snapshot = await readGitWorkspaceSnapshot(repo);
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
await git(cloneDir, ["config", "user.name", "Paperclip Sandbox"]);
await git(cloneDir, ["config", "user.email", "sandbox@paperclip.dev"]);
await writeFile(path.join(cloneDir, "change.txt"), "sandbox change\n", "utf8");
await git(cloneDir, ["add", "change.txt"]);
await git(cloneDir, ["commit", "-m", "sandbox change"]);
const cloneHead = await git(cloneDir, ["rev-parse", "HEAD"]);
// A filesystem-path origin is dropped by the allowlist, so configure the
// remote explicitly — the property under test is the push itself: the
// clone is shallow (single grafted commit), but the boundary commit
// already exists on the origin, so the push pack closes without full
// ancestry. That is what makes transported branches publishable.
await git(cloneDir, ["remote", "add", "origin", upstream]);
await git(cloneDir, ["push", "origin", "HEAD:refs/heads/sandbox-change"]);
expect(await git(upstream, ["rev-parse", "refs/heads/sandbox-change"])).toBe(cloneHead);
expect(await git(upstream, ["merge-base", "refs/heads/main", "refs/heads/sandbox-change"])).toBe(baseHead);
});
});
it("builds thin git delta bundles relative to the imported base", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-delta-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot).not.toBeNull();
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (remoteDir) => {
const emptyBundle = path.join(rootDir, "empty.bundle");
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir,
baseSha: baseHead,
exportRef: createRemoteGitExportRef("test"),
bundlePath: emptyBundle,
})]);
expect((await stat(emptyBundle)).size).toBe(0);
await git(remoteDir, ["config", "user.name", "Paperclip Remote"]);
await git(remoteDir, ["config", "user.email", "remote@paperclip.dev"]);
await writeFile(path.join(remoteDir, "tracked.txt"), "remote\n", "utf8");
await git(remoteDir, ["commit", "-am", "remote update"]);
const remoteHead = await git(remoteDir, ["rev-parse", "HEAD"]);
const deltaBundle = path.join(rootDir, "delta.bundle");
const importedRef = createImportedGitRef("test");
const exportRef = createRemoteGitExportRef("test");
try {
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir,
baseSha: baseHead,
exportRef,
bundlePath: deltaBundle,
})]);
expect((await stat(deltaBundle)).size).toBeGreaterThan(0);
const importedHead = await fetchGitBundleIntoLocalRef({
localDir: repo,
bundlePath: deltaBundle,
exportRef,
importedRef,
baseSha: baseHead,
});
expect(importedHead).toBe(remoteHead);
expect(await git(repo, ["rev-list", "--count", importedRef, "--not", baseHead])).toBe("1");
} finally {
await deleteLocalGitRef({ localDir: repo, ref: importedRef });
}
});
});
it("imports a diverged sandbox HEAD even when the host no longer holds baseSha", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-diverge-"));
cleanupDirs.push(rootDir);
// Host holds only the shared ancestor B (the eventual merge-base), not the
// recorded base H — the state a shared workspace lands in when it is reset
// between export and import.
const host = await createRepo(rootDir);
const mergeBase = await git(host, ["rev-parse", "HEAD"]);
// Sandbox holds B, an advanced commit H (the recorded baseSha), and a
// local-only commit S that forked from B and diverges from H.
const sandbox = path.join(rootDir, "sandbox");
await git(rootDir, ["clone", host, sandbox]);
await git(sandbox, ["config", "user.name", "Paperclip Remote"]);
await git(sandbox, ["config", "user.email", "remote@paperclip.dev"]);
await writeFile(path.join(sandbox, "advance.txt"), "advance\n", "utf8");
await git(sandbox, ["add", "-A"]);
await git(sandbox, ["commit", "-m", "advance"]);
const baseSha = await git(sandbox, ["rev-parse", "HEAD"]);
await git(sandbox, ["reset", "--hard", mergeBase]);
await writeFile(path.join(sandbox, "local.txt"), "local\n", "utf8");
await git(sandbox, ["add", "-A"]);
await git(sandbox, ["commit", "-m", "local-only"]);
const sandboxHead = await git(sandbox, ["rev-parse", "HEAD"]);
// The host genuinely lacks baseSha; the old thin bundle would name it as an
// unsatisfiable prerequisite.
await expect(git(host, ["cat-file", "-e", `${baseSha}^{commit}`])).rejects.toThrow();
const bundle = path.join(rootDir, "diverge.bundle");
const exportRef = createRemoteGitExportRef("test");
const importedRef = createImportedGitRef("test");
try {
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir: sandbox,
baseSha,
exportRef,
bundlePath: bundle,
})]);
expect((await stat(bundle)).size).toBeGreaterThan(0);
const importedHead = await fetchGitBundleIntoLocalRef({
localDir: host,
bundlePath: bundle,
exportRef,
importedRef,
baseSha,
});
expect(importedHead).toBe(sandboxHead);
// The host received the local-only commit and its parent (the merge-base).
expect(await git(host, ["cat-file", "-e", `${sandboxHead}^{commit}`])).toBe("");
} finally {
await deleteLocalGitRef({ localDir: host, ref: importedRef });
}
});
it("re-exports a full bundle that imports when the host holds neither baseSha nor the merge-base", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-ancestor-"));
cleanupDirs.push(rootDir);
// Host was reset to a strict ancestor of the eventual merge-base: it holds
// only the very first commit, not baseSha and not the fork point.
const host = await createRepo(rootDir);
const ancestor = await git(host, ["rev-parse", "HEAD"]);
const sandbox = path.join(rootDir, "sandbox");
await git(rootDir, ["clone", host, sandbox]);
await git(sandbox, ["config", "user.name", "Paperclip Remote"]);
await git(sandbox, ["config", "user.email", "remote@paperclip.dev"]);
// Advance the merge-base past the host, then baseSha past that, then a
// divergent local commit — so merge-base(baseSha, HEAD) is itself a commit
// the host does not hold.
await writeFile(path.join(sandbox, "fork.txt"), "fork\n", "utf8");
await git(sandbox, ["add", "-A"]);
await git(sandbox, ["commit", "-m", "fork point"]);
const forkPoint = await git(sandbox, ["rev-parse", "HEAD"]);
await writeFile(path.join(sandbox, "advance.txt"), "advance\n", "utf8");
await git(sandbox, ["add", "-A"]);
await git(sandbox, ["commit", "-m", "advance"]);
const baseSha = await git(sandbox, ["rev-parse", "HEAD"]);
await git(sandbox, ["reset", "--hard", forkPoint]);
await writeFile(path.join(sandbox, "local.txt"), "local\n", "utf8");
await git(sandbox, ["add", "-A"]);
await git(sandbox, ["commit", "-m", "local-only"]);
const sandboxHead = await git(sandbox, ["rev-parse", "HEAD"]);
// Host holds only the initial commit; it lacks both baseSha and the fork point.
expect(await git(host, ["rev-parse", "HEAD"])).toBe(ancestor);
await expect(git(host, ["cat-file", "-e", `${forkPoint}^{commit}`])).rejects.toThrow();
const exportRef = createRemoteGitExportRef("test");
const importedRef = createImportedGitRef("test");
// The delta bundle (relative to the merge-base = fork point) names a
// prerequisite the host lacks, so its import fails and is detected.
const deltaBundle = path.join(rootDir, "delta.bundle");
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir: sandbox,
baseSha,
exportRef,
bundlePath: deltaBundle,
})]);
let deltaError: unknown;
try {
await fetchGitBundleIntoLocalRef({ localDir: host, bundlePath: deltaBundle, exportRef, importedRef, baseSha });
} catch (error) {
deltaError = error;
}
expect(deltaError).toBeDefined();
expect(isMissingGitPrerequisiteError(deltaError)).toBe(true);
// The forced full bundle is self-contained and imports into the same host.
const fullBundle = path.join(rootDir, "full.bundle");
try {
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir: sandbox,
baseSha,
exportRef,
bundlePath: fullBundle,
forceFullBundle: true,
})]);
const importedHead = await fetchGitBundleIntoLocalRef({
localDir: host,
bundlePath: fullBundle,
exportRef,
importedRef,
baseSha,
});
expect(importedHead).toBe(sandboxHead);
} finally {
await deleteLocalGitRef({ localDir: host, ref: importedRef });
}
});
it("falls back to a full self-contained bundle when the sandbox lacks baseSha", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-full-"));
cleanupDirs.push(rootDir);
const sandbox = await createRepo(rootDir);
await writeFile(path.join(sandbox, "more.txt"), "more\n", "utf8");
await git(sandbox, ["add", "-A"]);
await git(sandbox, ["commit", "-m", "more"]);
const sandboxHead = await git(sandbox, ["rev-parse", "HEAD"]);
// A fresh, unrelated host that shares no history with the sandbox.
const host = path.join(rootDir, "fresh-host");
await mkdir(host, { recursive: true });
await git(host, ["init"]);
const bundle = path.join(rootDir, "full.bundle");
const exportRef = createRemoteGitExportRef("test");
const importedRef = createImportedGitRef("test");
try {
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir: sandbox,
// A base the sandbox does not have forces the full-bundle fallback.
baseSha: "0000000000000000000000000000000000000000",
exportRef,
bundlePath: bundle,
})]);
expect((await stat(bundle)).size).toBeGreaterThan(0);
const importedHead = await fetchGitBundleIntoLocalRef({
localDir: host,
bundlePath: bundle,
exportRef,
importedRef,
baseSha: "0000000000000000000000000000000000000000",
});
expect(importedHead).toBe(sandboxHead);
} finally {
await deleteLocalGitRef({ localDir: host, ref: importedRef });
}
});
it("creates the concurrent-history merge commit with a deterministic identity", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-merge-identity-"));
cleanupDirs.push(rootDir);
// No repo-local user.name/user.email on purpose: execution hosts are
// containers without git config, where commit-tree cannot auto-detect an
// identity. Setup commits pass their identity inline so only the merge
// commit under test depends on the sync-supplied identity.
const setupIdentity = ["-c", "user.name=Setup", "-c", "user.email=setup@paperclip.dev"];
const repo = path.join(rootDir, "repo");
await mkdir(repo, { recursive: true });
await git(repo, ["init"]);
await git(repo, ["checkout", "-b", "main"]);
await writeFile(path.join(repo, "tracked.txt"), "base\n", "utf8");
await git(repo, ["add", "tracked.txt"]);
await git(repo, [...setupIdentity, "commit", "-m", "base"]);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
await writeFile(path.join(repo, "local.txt"), "local\n", "utf8");
await git(repo, ["add", "local.txt"]);
await git(repo, [...setupIdentity, "commit", "-m", "local advance"]);
const currentHead = await git(repo, ["rev-parse", "HEAD"]);
await git(repo, ["checkout", "-b", "imported", baseHead]);
await writeFile(path.join(repo, "imported.txt"), "imported\n", "utf8");
await git(repo, ["add", "imported.txt"]);
await git(repo, [...setupIdentity, "commit", "-m", "sandbox change"]);
const importedHead = await git(repo, ["rev-parse", "HEAD"]);
await git(repo, ["checkout", "main"]);
// Ambient identity env vars would override the `-c` flags and make the
// assertion machine-dependent, so clear them for the call under test.
const identityEnvKeys = ["GIT_AUTHOR_NAME", "GIT_AUTHOR_EMAIL", "GIT_COMMITTER_NAME", "GIT_COMMITTER_EMAIL", "EMAIL"];
const savedEnv = new Map(identityEnvKeys.map((key) => [key, process.env[key]]));
for (const key of identityEnvKeys) delete process.env[key];
try {
await integrateImportedGitHead({ localDir: repo, importedHead });
} finally {
for (const [key, value] of savedEnv) {
if (value === undefined) delete process.env[key];
else process.env[key] = value;
}
}
const parents = (await git(repo, ["rev-list", "--parents", "-1", "HEAD"])).split(" ");
expect(parents.slice(1)).toEqual([currentHead, importedHead]);
expect(await git(repo, ["log", "-1", "--format=%an|%ae|%cn|%ce"]))
.toBe("Paperclip|noreply@paperclip.ing|Paperclip|noreply@paperclip.ing");
expect(await git(repo, ["log", "-1", "--format=%s"]))
.toBe(`Paperclip remote git sync merge ${importedHead.slice(0, 12)}`);
const mergedTree = await git(repo, ["ls-tree", "--name-only", "HEAD"]);
expect(mergedTree).toContain("local.txt");
expect(mergedTree).toContain("imported.txt");
});
it("grafts an imported head onto the current head when histories share no ancestor", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-graft-"));
cleanupDirs.push(rootDir);
const setupIdentity = ["-c", "user.name=Setup", "-c", "user.email=setup@paperclip.dev"];
const repo = path.join(rootDir, "repo");
await mkdir(repo, { recursive: true });
await git(repo, ["init"]);
await git(repo, ["checkout", "-b", "main"]);
await writeFile(path.join(repo, "tracked.txt"), "base\n", "utf8");
await git(repo, ["add", "tracked.txt"]);
await git(repo, [...setupIdentity, "commit", "-m", "base"]);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
await writeFile(path.join(repo, "local.txt"), "local\n", "utf8");
await git(repo, ["add", "local.txt"]);
await git(repo, [...setupIdentity, "commit", "-m", "local advance"]);
const currentHead = await git(repo, ["rev-parse", "HEAD"]);
// The shape a depth-1 shallow clone produces after `git commit --amend`:
// a parentless root commit that shares no ancestor with the host history.
const importedTree = await git(repo, ["rev-parse", `${baseHead}^{tree}`]);
const importedHead = await git(repo, [...setupIdentity, "commit-tree", importedTree, "-m", "sandbox rewrite"]);
await integrateImportedGitHead({ localDir: repo, importedHead });
const parents = (await git(repo, ["rev-list", "--parents", "-1", "HEAD"])).split(" ");
expect(parents.slice(1)).toEqual([currentHead]);
// The imported tree is taken wholesale: no base exists to merge against.
expect(await git(repo, ["rev-parse", "HEAD^{tree}"])).toBe(importedTree);
expect(await git(repo, ["log", "-1", "--format=%s"])).toBe("sandbox rewrite");
const body = await git(repo, ["log", "-1", "--format=%B"]);
expect(body).toContain(`Paperclip remote git sync graft ${importedHead.slice(0, 12)}`);
expect(body).toContain("shares no ancestor");
});
it("does not graft when merge-base fails for a reason other than missing ancestry", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-no-graft-"));
cleanupDirs.push(rootDir);
const setupIdentity = ["-c", "user.name=Setup", "-c", "user.email=setup@paperclip.dev"];
const repo = path.join(rootDir, "repo");
await mkdir(repo, { recursive: true });
await git(repo, ["init"]);
await git(repo, ["checkout", "-b", "main"]);
await writeFile(path.join(repo, "tracked.txt"), "base\n", "utf8");
await git(repo, ["add", "tracked.txt"]);
await git(repo, [...setupIdentity, "commit", "-m", "base"]);
const currentHead = await git(repo, ["rev-parse", "HEAD"]);
// A well-formed sha the repository does not hold: merge-base fails with an
// object error (exit 128), not the no-ancestor signal (exit 1). The graft
// must not fire, and the integration keeps its loud failure.
const missingHead = "0123456789abcdef0123456789abcdef01234567";
await expect(integrateImportedGitHead({ localDir: repo, importedHead: missingHead }))
.rejects.toThrow(/Failed to merge concurrent remote git histories/);
expect(await git(repo, ["rev-parse", "HEAD"])).toBe(currentHead);
});
describe("readReferencedSourceGitIgnoredPaths", () => {
it("returns null for a directory that is not a Git work tree", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-nogit-"));
cleanupDirs.push(rootDir);
const plainDir = path.join(rootDir, "plain");
await mkdir(plainDir, { recursive: true });
await writeFile(path.join(plainDir, "file.txt"), "body\n", "utf8");
await expect(readReferencedSourceGitIgnoredPaths(plainDir)).resolves.toBeNull();
});
it.each([
{ code: "workspace_git_scan_failed", exitCode: 128, signal: null, nonGit: true },
{ code: "workspace_git_scan_timeout", exitCode: 128, signal: null, nonGit: false },
{ code: "workspace_git_scan_cancelled", exitCode: 128, signal: null, nonGit: false },
{ code: "workspace_git_scan_output_limit", exitCode: 128, signal: null, nonGit: false },
{ code: "workspace_git_scan_failed", exitCode: null, signal: "SIGTERM", nonGit: false },
])("classifies scheduled non-repository failures without swallowing $code/$signal", async ({ code, exitCode, signal, nonGit }) => {
const error = Object.assign(new Error("Workspace Git scan failed"), {
code,
details: { exitCode, signal, stderr: "fatal: not a git repository (or any of the parent directories): .git" },
});
setExpensiveWorkspaceGitExecutor(async () => { throw error; });
try {
const result = readReferencedSourceGitIgnoredPaths("/plain-workspace");
if (nonGit) await expect(result).resolves.toBeNull();
else await expect(result).rejects.toBe(error);
} finally {
setExpensiveWorkspaceGitExecutor(null);
}
});
it("reads the repository top level and the ignored paths of a Git work tree", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-git-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await writeFile(path.join(repo, ".gitignore"), "secret.env\nbuild/\n", "utf8");
await writeFile(path.join(repo, "secret.env"), "TOKEN=abc\n", "utf8");
await mkdir(path.join(repo, "build"), { recursive: true });
await writeFile(path.join(repo, "build", "out.js"), "artifact\n", "utf8");
const scan = await readReferencedSourceGitIgnoredPaths(repo);
expect(scan?.toplevel).toBe(await git(repo, ["rev-parse", "--show-toplevel"]));
expect(scan?.ignoredPaths).toEqual(["build", "secret.env"]);
});
it("preserves trailing whitespace in an ignored path entry", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-trailing-ws-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
// A wildcard pattern avoids the separate rule that git trims an
// unescaped trailing space in a .gitignore pattern itself; the trailing
// space under test lives in the matched FILE name, not the pattern.
const paddedName = "secret.env ";
await writeFile(path.join(repo, ".gitignore"), "secret.env*\n", "utf8");
await writeFile(path.join(repo, paddedName), "TOKEN=abc\n", "utf8");
const scan = await readReferencedSourceGitIgnoredPaths(repo);
expect(scan?.ignoredPaths).toEqual([paddedName]);
});
it("fails closed when the parsed ignored-entry count exceeds the bound", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-bound-count-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
// Synthesize the `git ls-files --others --ignored -z` output directly,
// rather than creating ten thousand real files, by intercepting the
// scan at the executor seam. The parser must reject this before it
// sorts or re-relativizes the list.
const overLimitCount = REFERENCED_SOURCE_IGNORE_MAX_ENTRY_COUNT + 1;
const syntheticIgnored = `${Array.from({ length: overLimitCount }, (_, index) => `entry-${index}`).join("\0")}\0`;
setExpensiveWorkspaceGitExecutor(async (input) => {
if (input.operation === "referenced_source.ignored_files") {
return { stdout: syntheticIgnored, stderr: "" };
}
return await runLocalGit(input.localDir, [...input.args], {
timeout: input.timeout,
maxBuffer: input.maxBuffer,
env: input.env,
});
});
await expect(readReferencedSourceGitIgnoredPaths(repo)).rejects.toBeInstanceOf(
ReferencedSourceIgnoreScanLimitExceededError,
);
});
it("fails closed when the summed UTF-8 byte size of ignored paths exceeds the bound", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-bound-bytes-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
// One entry alone exceeds the byte bound, well under the entry-count bound.
const hugeEntry = "a".repeat(REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES + 1);
const syntheticIgnored = `${hugeEntry}\0`;
setExpensiveWorkspaceGitExecutor(async (input) => {
if (input.operation === "referenced_source.ignored_files") {
return { stdout: syntheticIgnored, stderr: "" };
}
return await runLocalGit(input.localDir, [...input.args], {
timeout: input.timeout,
maxBuffer: input.maxBuffer,
env: input.env,
});
});
await expect(readReferencedSourceGitIgnoredPaths(repo)).rejects.toBeInstanceOf(
ReferencedSourceIgnoreScanLimitExceededError,
);
});
it("fails closed on the byte bound while it is still accumulating, before it would ever reach a later entry-count breach", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-bound-order-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
// Three entries alone cross the byte bound. Many more small entries
// follow, so the FULL response also carries more than the entry-count
// bound. A parser that fully builds the list before checking either
// bound (post-parse) would report the entry-count breach, because it
// checks that bound first against the whole materialized list. A
// parser that checks both bounds while the list accumulates rejects on
// the byte bound instead, the moment the third entry crosses it, well
// before the count bound is ever reached.
const oversizedEntry = "a".repeat(Math.ceil(REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES / 2) + 1);
const bigEntries = Array.from({ length: 3 }, (_, index) => `${oversizedEntry}-${index}`);
const trailingEntries = Array.from(
{ length: REFERENCED_SOURCE_IGNORE_MAX_ENTRY_COUNT + 10 },
(_, index) => `trailing-${index}`,
);
const syntheticIgnored = `${[...bigEntries, ...trailingEntries].join("\0")}\0`;
setExpensiveWorkspaceGitExecutor(async (input) => {
if (input.operation === "referenced_source.ignored_files") {
return { stdout: syntheticIgnored, stderr: "" };
}
return await runLocalGit(input.localDir, [...input.args], {
timeout: input.timeout,
maxBuffer: input.maxBuffer,
env: input.env,
});
});
await expect(readReferencedSourceGitIgnoredPaths(repo)).rejects.toThrow(/UTF-8 bytes/);
});
it("bounds the raw command-output allowance to the ignore-scan limits, not the general-purpose full-tree ceiling", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-raw-buffer-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
let observedMaxBuffer: number | undefined;
setExpensiveWorkspaceGitExecutor(async (input) => {
if (input.operation === "referenced_source.ignored_files") {
observedMaxBuffer = input.maxBuffer;
}
return await runLocalGit(input.localDir, [...input.args], {
timeout: input.timeout,
maxBuffer: input.maxBuffer,
env: input.env,
});
});
await readReferencedSourceGitIgnoredPaths(repo);
// Enough headroom for a scan within bounds to complete, but a small
// multiple of the byte bound — not the far larger allowance the
// anchor workspace's general-purpose full-tree reads use.
expect(observedMaxBuffer).toBeGreaterThan(REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES);
expect(observedMaxBuffer).toBeLessThan(16 * 1024 * 1024);
});
it("does not fail closed on a huge amount of unrelated tracked-change and untracked noise, when the ignored set itself stays in bounds", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-mixed-status-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await writeFile(path.join(repo, ".gitignore"), "secret.env\n", "utf8");
await writeFile(path.join(repo, "secret.env"), "TOKEN=abc\n", "utf8");
// Many long-named, untracked, NOT-ignored files at the repository root.
// `git status` reports one record per file (root-level files are never
// collapsed the way an entirely untracked directory is), so this alone
// makes the raw `git status --ignored` response exceed the raw buffer
// bound this scan used to apply to the WHOLE response, well before the
// parser ever got to discard these non-ignored records. The ignored set
// above stays a single small entry throughout.
const noiseNameLength = 220;
const noiseFileCount = 30_000;
const noiseNames = Array.from(
{ length: noiseFileCount },
(_, index) => `${"n".repeat(noiseNameLength - 6)}${String(index).padStart(6, "0")}`,
);
const writeConcurrency = 200;
for (let start = 0; start < noiseNames.length; start += writeConcurrency) {
const batch = noiseNames.slice(start, start + writeConcurrency);
await Promise.all(batch.map((name) => writeFile(path.join(repo, name), "", "utf8")));
}
// Confirm this test actually reproduces the reported defect precondition:
// the raw `git status --ignored` response for this repository state is
// larger than the 4 MiB raw buffer bound the scan used to apply to the
// whole response, not just to the declared ignored-set limits. A large
// explicit maxBuffer is required here only to observe that raw size;
// the scan under test never issues this command.
const rawStatusResult = await runLocalGit(
repo,
["status", "--ignored", "--porcelain=v1", "-z", "--untracked-files=normal"],
{ maxBuffer: 16 * 1024 * 1024 },
);
expect(Buffer.byteLength(rawStatusResult.stdout, "utf8")).toBeGreaterThan(REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES * 2);
const scan = await readReferencedSourceGitIgnoredPaths(repo);
expect(scan?.ignoredPaths).toEqual(["secret.env"]);
});
it("routes both scan commands through the registered scheduler instead of spawning git directly", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-scheduler-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await writeFile(path.join(repo, ".gitignore"), "build/\n", "utf8");
await mkdir(path.join(repo, "build"), { recursive: true });
await writeFile(path.join(repo, "build", "out.js"), "artifact\n", "utf8");
const operations: string[] = [];
setExpensiveWorkspaceGitExecutor(async (input) => {
operations.push(input.operation);
return await runLocalGit(input.localDir, [...input.args], {
timeout: input.timeout,
maxBuffer: input.maxBuffer,
env: input.env,
});
});
const scan = await readReferencedSourceGitIgnoredPaths(repo);
expect(scan?.ignoredPaths).toEqual(["build"]);
// Both the toplevel probe and the ignored-paths read go through the SAME
// process-wide admission seam the anchor workspace's expensive reads
// use. A host process that bounds concurrent scans there also bounds
// referenced-project scans, so a run with many referenced projects
// cannot spawn one unbounded Git process per project.
expect(operations.sort()).toEqual(["referenced_source.ignored_files", "referenced_source.toplevel"]);
});
it("carries the hardened arguments and does not inherit a poisoned GIT_CONFIG_GLOBAL", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-hardened-env-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const badGlobalConfig = path.join(rootDir, "bad-global-gitconfig");
await writeFile(badGlobalConfig, "this is not valid git config syntax [[[\n", "utf8");
const priorGlobal = process.env.GIT_CONFIG_GLOBAL;
process.env.GIT_CONFIG_GLOBAL = badGlobalConfig;
try {
// A plain invocation inherits the poisoned global config and fails to parse it.
await expect(execFile("git", ["-C", repo, "status", "--porcelain"])).rejects.toThrow();
// The hardened helper does not inherit GIT_CONFIG_GLOBAL from this process's
// environment, so it succeeds regardless.
await expect(readReferencedSourceGitIgnoredPaths(repo)).resolves.toMatchObject({ ignoredPaths: [] });
} finally {
if (priorGlobal === undefined) delete process.env.GIT_CONFIG_GLOBAL;
else process.env.GIT_CONFIG_GLOBAL = priorGlobal;
}
});
it("neutralizes a repository-local core.fsmonitor hook", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-fsmonitor-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const markerPath = path.join(rootDir, "pwned.txt");
// A malicious repository-local config: a non-boolean `core.fsmonitor` value
// is a hook COMMAND Git runs on every status-like read. `--no-optional-locks`
// alone does not stop this; only the command-line `-c core.fsmonitor=false`
// override does, because command-line config wins over repository config.
await git(repo, ["config", "core.fsmonitor", `sh -c 'touch ${markerPath}; printf 1'`]);
await readReferencedSourceGitIgnoredPaths(repo);
await expect(stat(markerPath)).rejects.toThrow();
});
});
});
describe("sanitizeGitRemoteUrl", () => {
it("strips userinfo, query, and fragment from http(s) URLs", () => {
expect(sanitizeGitRemoteUrl("https://x-access-token:sekret@github.com/example/repo.git"))
.toBe("https://github.com/example/repo.git");
expect(sanitizeGitRemoteUrl("https://sekret-token@github.com/example/repo.git"))
.toBe("https://github.com/example/repo.git");
expect(sanitizeGitRemoteUrl("http://user:pass@git.internal/example/repo.git"))
.toBe("http://git.internal/example/repo.git");
expect(sanitizeGitRemoteUrl("https://github.com/example/repo.git?private_token=sekret#fragment"))
.toBe("https://github.com/example/repo.git");
});
it("strips password and query from ssh-scheme URLs but keeps the username", () => {
expect(sanitizeGitRemoteUrl("ssh://git@github.com/example/repo.git"))
.toBe("ssh://git@github.com/example/repo.git");
expect(sanitizeGitRemoteUrl("ssh://git:sekret@github.com/example/repo.git"))
.toBe("ssh://git@github.com/example/repo.git");
expect(sanitizeGitRemoteUrl("git+ssh://git@github.com/example/repo.git?key=sekret"))
.toBe("git+ssh://git@github.com/example/repo.git");
});
it("keeps credential-free scp-like remotes unchanged", () => {
expect(sanitizeGitRemoteUrl("git@github.com:example/repo.git"))
.toBe("git@github.com:example/repo.git");
expect(sanitizeGitRemoteUrl("https://github.com/example/repo.git"))
.toBe("https://github.com/example/repo.git");
});
it("drops every shape whose credential surface is unknown", () => {
// Filesystem paths are useless on the execution host and could leak
// host-layout details; unknown schemes and malformed userinfo could carry
// embedded secrets the sanitizer cannot recognize. All fail closed.
expect(sanitizeGitRemoteUrl("/tmp/local/upstream.git")).toBeNull();
expect(sanitizeGitRemoteUrl("ftp://user:pass@host/repo.git")).toBeNull();
expect(sanitizeGitRemoteUrl("user:pass@host:path/repo.git")).toBeNull();
expect(sanitizeGitRemoteUrl("host.example:path/repo.git")).toBeNull();
});
it("returns null for empty input", () => {
expect(sanitizeGitRemoteUrl("")).toBeNull();
expect(sanitizeGitRemoteUrl(" ")).toBeNull();
});
});