Files
PaperClipAI/packages/adapter-utils/src/git-workspace-sync.test.ts
T
DottaandPaperclip 4b38db9622 fix(runtime): stream workspace Git snapshots through disk manifests (#14253)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Managed runs copy a selected workspace to an execution environment
and restore its changes.
> - Git snapshots select the files for that copy and for later recovery.
> - A fixed output limit stops large generated trees before the run can
start.
> - Increasing the limit still keeps the complete filename lists in
memory.
> - This pull request stores those lists and merge baselines in disk
manifests.
> - Large snapshots can now complete with bounded filename buffers and
explicit failure handling.

## Linked Issues or Issue Description

Refs #14194. This is the streaming follow-up to the merged 32 MiB limit
fix.

Related: #13619 and #11621 cover workspace scan admission and demand.
This change keeps the shared scheduler and changes the snapshot data
path.

## What Changed

- Stream changed, untracked, deleted, and ignored paths through the
shared scheduler and the standalone adapter path.
- Use SQLite manifests for file selection, duplicate removal,
ignored-path lookup, baseline capture, and merge lookup.
- Set a configurable 30-minute snapshot deadline. Keep the existing
interactive scan deadlines.
- Wait for each child process and pending sink write before removing
temporary storage after failure or cancellation.
- Use NUL archive lists and bounded deletion batches. Preserve unusual
names, explicit selection, nested repositories, and source root checks.
- Store manifest references in native recovery format v2. Check their
location and digest before recovery reads. Keep v1 descriptors readable.
- Remove temporary manifests at lifecycle completion. Use fixed-size
temporary copy names for long basenames.
- Admit each manifest with a SQLite page allowance based on current disk
capacity. Keep a configurable free-space reserve and fail explicitly
when either limit is reached.
- Preserve a host file that replaces a directory deleted by the sandbox,
and continue the rest of the restore.

## Verification

- Current head `5ab622ec43cd16d35429d79dedee6a5d8e3d2df2` has 54
successful checks/statuses and two skipped Storybook jobs. No checks
failed or remain pending.
- [CI
passed](https://github.com/paperclipai/paperclip/actions/runs/36318966368):
typecheck, build, all test shards, E2E, Rust checks, and the aggregate
verify job.
- [Greptile is
5/5](https://github.com/paperclipai/paperclip/pull/14253#issuecomment-5855670338)
on the current head. All four review threads are resolved. Security
checks passed.
- 229 focused tests passed across Git sync, runtime staging, merge,
manifest integrity, native recovery, and the scheduler (214
adapter/runtime tests and 15 scheduler tests).
- A real 40,000-file fixture produces 43,428,890 filename bytes. The
original standalone and scheduled scans fail. The new test passes all
four filename paths, complete staging, exclusion of late files, unusual
names, and deletion replay.
- Recovery tests reject changed bytes, symlinks, and paths outside the
controller state directory. Adapter-utils typecheck passed.
- A test executor returned buffered output and caused two retry
integration failures. The fixture now uses the shared streaming
scheduler. All 13 tests passed with `corepack pnpm exec vitest run
server/src/__tests__/heartbeat-project-repositories.test.ts`. The same
CI shard now passes.
- Ran `pnpm -r typecheck`, `pnpm test:run`, and `pnpm build` locally.
Each full local command hit SIGKILL/exit 137 in the 4 GiB container.
These local commands did not pass. The current-head CI gates above
provide the full verification.
- A real-Git disk-capacity regression confirms a typed failure and
removal of the incomplete manifest. Repeated writer attempts cannot
exceed the permitted page count.

- Follow-up real Daytona and separate staging qualification passed with
the related archive validator (#14315) and exact-owner finalization fix
(#14314). Three successive turns copied back all 60,000 files with
39,828,890 filename bytes and five unusual names. Independent host
inventories verified every file and the pinned Git HEAD. Native,
provider, session, and process identities stayed fixed; no retry
remained. The task reached Done, and its browser-downloaded final proof
matched exactly. The reusable regression is #14316, including an
assertion of the effective environment idle policy.

## Risks

- SQLite manifests use disk space. Each receives one quarter of the
available capacity above the host reserve at creation. The reserve
defaults to 256 MiB and has a 64 MiB configuration minimum. Disk
capacity, filesystem quotas, per-path limits, Git resource use, and
execution deadlines remain limits.
- Each path and sink chunk has a 64 KiB limit. SQLite connections use a
1 MiB page cache. Invalid or incomplete records fail explicitly.
- Restore transport keeps fixed and configured archive exclusions. A
remotely created Git-ignored file can be transferred, but the host merge
excludes it through the manifest.
- Provider archive buffers, Git and tar memory, repository metadata,
legacy v1 arrays, and the separate referenced-source resolver retain
their own limits. Existing provider safety validators still buffer
textual tar listings: Daytona allows 32 MiB and Kubernetes allows 64
MiB. These separate transport limits can stop a sufficiently large
restore before merge. This change does not claim bounded total process
memory or unlimited transport size.
- New descriptors use v2. Existing v1 recovery remains supported; a
downgrade cannot read v2 descriptors.

## Model Used

OpenAI GPT-6 through Codex. The exact deployment ID and context limit
are not exposed in this run. The agent used code editing, terminal
execution, tests, and GitHub tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-28 08:40:36 -05:00

1053 lines
51 KiB
TypeScript

import { execFile as execFileCallback } from "node:child_process";
import { lstat, mkdir, mkdtemp, readFile, readlink, rename, rm, stat, symlink, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { promisify } from "node:util";
import { workspacePaths } from "./workspace-manifest.js";
import { runWorkspaceGitProcess } from "./workspace-git-stream.js";
import { afterEach, describe, expect, it } from "vitest";
import {
buildRemoteGitDeltaBundleScript,
createImportedGitRef,
createRemoteGitExportRef,
deleteLocalGitRef,
fetchGitBundleIntoLocalRef,
integrateImportedGitHead,
isMissingGitPrerequisiteError,
readGitWorkspaceSnapshot as readRawSnapshot,
disposeGitWorkspaceSnapshot,
type ExpensiveWorkspaceGitInput,
type GitWorkspaceSnapshot,
ReferencedSourceIgnoreScanLimitExceededError,
readReferencedSourceGitIgnoredPaths,
REFERENCED_SOURCE_IGNORE_MAX_ENTRY_COUNT,
REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES,
runLocalGit,
sanitizeGitRemoteUrl,
setExpensiveWorkspaceGitExecutor,
withShallowGitWorkspaceClone,
} from "./git-workspace-sync.js";
const execFile = promisify(execFileCallback);
const snapshots: GitWorkspaceSnapshot[] = [];
async function readGitWorkspaceSnapshot(...args: Parameters<typeof readRawSnapshot>) {
const snapshot = await readRawSnapshot(...args);
if (!snapshot) return null;
snapshots.push(snapshot);
// Small fixture assertions use arrays; production consumes only iterators.
return { ...snapshot, overlayPaths: [...workspacePaths(snapshot.overlayPaths)].sort((a,b)=>a.localeCompare(b)),
deletedPaths: [...workspacePaths(snapshot.deletedPaths)], ignoredPaths: [...workspacePaths(snapshot.ignoredPaths)] };
}
function executeScan(input: ExpensiveWorkspaceGitInput) {
if (!input.onStdout) return runLocalGit(input.localDir, [...input.args], input);
return runWorkspaceGitProcess({ cwd: input.localDir, args: input.args, timeoutMs: input.timeout,
maxStdoutBytes: input.maxBuffer, maxStderrBytes: input.maxBuffer, onStdout: input.onStdout, signal: input.signal, env: input.env });
}
async function git(cwd: string, args: string[]): Promise<string> {
return (await runLocalGit(cwd, args)).stdout.trim();
}
describe("git workspace sync", () => {
const cleanupDirs: string[] = [];
afterEach(async () => {
setExpensiveWorkspaceGitExecutor(null);
for (const snapshot of snapshots.splice(0)) await disposeGitWorkspaceSnapshot(snapshot);
while (cleanupDirs.length > 0) {
const dir = cleanupDirs.pop();
if (!dir) continue;
await rm(dir, { recursive: true, force: true }).catch(() => undefined);
}
}, 30_000); // The output-limit fixture removes 40,000 files on teardown.
it("delegates every host-side full-tree enumeration to the registered scheduler", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-scheduler-hook-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await writeFile(path.join(repo, "untracked.txt"), "untracked\n", "utf8");
const operations: string[] = [];
setExpensiveWorkspaceGitExecutor(async (input) => {
operations.push(input.operation);
return executeScan(input);
});
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot?.overlayPaths).toContain("untracked.txt");
expect(operations.sort()).toEqual([
"adapter_sync.deleted_files",
"adapter_sync.ignored_files",
"adapter_sync.overlay_diff",
"adapter_sync.untracked_files",
]);
});
it("keeps every filename byte for a padded name in each of the four anchor lanes", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-anchor-whitespace-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
// Deleted lane: commit the file first (in isolation, before anything else
// is staged), then remove it from the work tree.
const deletedName = " deleted padded ";
await writeFile(path.join(repo, deletedName), "deleted\n", "utf8");
await git(repo, ["add", deletedName]);
await git(repo, ["commit", "-qm", "add deleted padded"]);
await rm(path.join(repo, deletedName));
// Overlay lane, staged-new half: `git diff --diff-filter=ACMRTUXB HEAD`
// reports a staged-but-uncommitted file as added.
const overlayName = " overlay padded ";
await writeFile(path.join(repo, overlayName), "overlay\n", "utf8");
await git(repo, ["add", overlayName]);
// Overlay lane, untracked half: `ls-files --others --exclude-standard`.
const untrackedName = " untracked padded ";
await writeFile(path.join(repo, untrackedName), "untracked\n", "utf8");
// Ignored lane: a double-wildcard pattern avoids the separate rule that
// Git trims an unescaped trailing space in a .gitignore PATTERN itself;
// the padding under test lives in the matched FILE name.
const ignoredName = " ignored padded ";
await writeFile(path.join(repo, ".gitignore"), "*ignored*padded*\n", "utf8");
await writeFile(path.join(repo, ignoredName), "ignored\n", "utf8");
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot?.overlayPaths).toContain(overlayName);
expect(snapshot?.overlayPaths).toContain(untrackedName);
expect(snapshot?.deletedPaths).toContain(deletedName);
expect(snapshot?.ignoredPaths).toContain(ignoredName);
});
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "workspace_git_scan_output_limit", "workspace_git_scan_cancelled", "workspace_git_scan_failed"])("preserves %s instead of reporting a non-Git folder", async (code) => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-scan-failure-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const failure = Object.assign(new Error("Git enumeration failed"), { code });
setExpensiveWorkspaceGitExecutor(async (input) => {
if (input.operation === "adapter_sync.ignored_files") throw failure;
return executeScan(input);
});
await expect(readGitWorkspaceSnapshot(repo, false)).rejects.toBe(failure);
});
it("lists ignored paths without traversing ignored directory contents", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-ignored-scan-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await writeFile(path.join(repo, ".gitignore"), "dependencies/\n*.secret\n");
await mkdir(path.join(repo, "dependencies", "nested"), { recursive: true });
await writeFile(path.join(repo, "dependencies", "nested", "private"), "private");
await writeFile(path.join(repo, "token.secret"), "private");
let ignoredArgs: readonly string[] = [];
setExpensiveWorkspaceGitExecutor(async (input) => {
if (input.operation === "adapter_sync.ignored_files") ignoredArgs = input.args;
return executeScan(input);
});
expect((await readGitWorkspaceSnapshot(repo))?.ignoredPaths).toEqual(["dependencies", "token.secret"]);
expect(ignoredArgs).toEqual(["ls-files", "--others", "--ignored", "--exclude-standard", "--directory", "-z"]);
});
it("snapshots a generated directory with more than 1 MiB of filenames", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-large-untracked-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const generatedDir = path.join(repo, "storybook-output");
await mkdir(generatedDir);
const names = Array.from({ length: 5_000 }, (_, index) => `${"asset-".repeat(36)}${index}.js`);
for (let start = 0; start < names.length; start += 100) {
await Promise.all(names.slice(start, start + 100).map((name) => writeFile(path.join(generatedDir, name), "")));
}
const raw = await runLocalGit(repo, ["ls-files", "--others", "--exclude-standard", "-z"], {
maxBuffer: 2 * 1024 * 1024,
});
expect(Buffer.byteLength(raw.stdout)).toBeGreaterThan(1024 * 1024);
setExpensiveWorkspaceGitExecutor(executeScan);
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot?.overlayPaths).toEqual(
names.map((name) => `storybook-output/${name}`).sort((left, right) => left.localeCompare(right)),
);
// A larger tree exceeds the old 8 MiB bound but fits the new 32 MiB bound.
for (let start = 5_000; start < 40_000; start += 100) {
await Promise.all(Array.from({ length: 100 }, (_, index) => writeFile(
path.join(generatedDir, `${"asset-".repeat(36)}${start + index}.js`), "",
)));
}
const largerRaw = await runLocalGit(repo, ["ls-files", "--others", "--exclude-standard", "-z"], {
maxBuffer: 32 * 1024 * 1024,
});
expect(Buffer.byteLength(largerRaw.stdout)).toBeGreaterThan(8 * 1024 * 1024);
const largerSnapshot = await readGitWorkspaceSnapshot(repo);
expect(largerSnapshot?.overlayPaths).toEqual(
largerRaw.stdout.split("\0").filter(Boolean).sort((left, right) => left.localeCompare(right)),
);
// Reuse the files with longer parent paths to exceed 32 MiB without
// creating hundreds of thousands of files solely to test the bound.
const deepParent = path.join(repo, ...Array.from({ length: 4 }, () => "nested-".repeat(30)));
await mkdir(deepParent, { recursive: true });
await rename(generatedDir, path.join(deepParent, "storybook-output"));
expect(Buffer.byteLength(largerRaw.stdout) + 40_000 * (path.relative(repo, deepParent).length + 1))
.toBeGreaterThan(32 * 1024 * 1024);
const streamed = await readRawSnapshot(repo);
snapshots.push(streamed!);
let count = 0;
let bytes = 0;
for (const relative of workspacePaths(streamed!.overlayPaths)) { count++; bytes += Buffer.byteLength(relative) + 1; }
expect(count).toBe(40_000);
expect(bytes).toBeGreaterThan(32 * 1024 * 1024);
expect(JSON.stringify(streamed).length).toBeLessThan(4096);
}, 60_000);
async function createRepo(rootDir: string): Promise<string> {
const repo = path.join(rootDir, "repo");
await mkdir(repo, { recursive: true });
await git(repo, ["init"]);
await git(repo, ["checkout", "-b", "main"]);
await git(repo, ["config", "user.name", "Paperclip Test"]);
await git(repo, ["config", "user.email", "test@paperclip.dev"]);
await writeFile(path.join(repo, "tracked.txt"), "base\n", "utf8");
await git(repo, ["add", "tracked.txt"]);
await git(repo, ["commit", "-m", "base"]);
return repo;
}
it("does not classify a selected repository subfolder as a cloneable repository root", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-selected-folder-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const selectedDir = path.join(repo, "project");
await mkdir(selectedDir);
await writeFile(path.join(selectedDir, "draft.md"), "selected work\n");
expect(await git(selectedDir, ["rev-parse", "--is-inside-work-tree"])).toBe("true");
expect(await readGitWorkspaceSnapshot(selectedDir)).toBeNull();
expect((await readGitWorkspaceSnapshot(repo))?.headCommit).toBe(await git(repo, ["rev-parse", "HEAD"]));
});
it("creates a shallow standalone clone from the local HEAD snapshot", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-sync-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
await rm(path.join(repo, "tracked.txt"));
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot).toMatchObject({
headCommit: baseHead,
branchName: "main",
deletedPaths: ["tracked.txt"],
});
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
expect((await lstat(path.join(cloneDir, ".git"))).isDirectory()).toBe(true);
await expect(readFile(path.join(cloneDir, ".git", "shallow"), "utf8")).resolves.toContain(baseHead);
expect(await git(cloneDir, ["rev-list", "--count", "HEAD"])).toBe("1");
expect(await git(cloneDir, ["branch", "--show-current"])).toBe("main");
await expect(readFile(path.join(cloneDir, "tracked.txt"), "utf8")).resolves.toBe("base\n");
});
});
it.skipIf(process.platform === "win32")("preserves nested repository symlinks after the temporary clone is removed", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-nested-links-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const nested = await createRepo(path.join(repo, ".paperclip-repositories"));
await writeFile(path.join(repo, ".git/info/exclude"), ".paperclip-repositories/\n");
await mkdir(path.join(nested, "skills", "demo"), { recursive: true });
await mkdir(path.join(nested, ".claude", "skills"), { recursive: true });
await writeFile(path.join(nested, "skills", "demo", "SKILL.md"), "skill content\n");
const links = [
[".claude/skills/demo", "../../skills/demo"],
["skill.md", "skills/demo/SKILL.md"],
["skill-alias", ".claude/skills/demo"],
["future", "future.txt"],
] as const;
for (const [name, target] of links) await symlink(target, path.join(nested, name));
await git(nested, ["add", "."]);
await git(nested, ["commit", "-m", "add repository links"]);
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot?.repositories).toHaveLength(1);
await withShallowGitWorkspaceClone({ localDir: repo, snapshot: snapshot! }, async (cloneDir) => {
// The nested clone's callback has already returned and deleted its temp
// directory. Relative links must keep their repository meaning here.
const copied = path.join(cloneDir, ".paperclip-repositories", "repo");
for (const [name, target] of links) {
expect((await lstat(path.join(copied, name))).isSymbolicLink()).toBe(true);
expect(await readlink(path.join(copied, name))).toBe(target);
}
expect(await readFile(path.join(copied, ".claude/skills/demo/SKILL.md"), "utf8")).toBe("skill content\n");
expect(await readFile(path.join(copied, "skill-alias/SKILL.md"), "utf8")).toBe("skill content\n");
await expect(stat(path.join(copied, "future"))).rejects.toMatchObject({ code: "ENOENT" });
expect(await git(copied, ["status", "--porcelain"])).toBe("");
});
expect(await git(nested, ["status", "--porcelain"])).toBe("");
});
it("copies the workspace origin remote into the shallow clone", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-origin-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await git(repo, ["remote", "add", "origin", "https://github.com/example/repo.git"]);
const snapshot = await readGitWorkspaceSnapshot(repo);
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
expect(await git(cloneDir, ["remote", "get-url", "origin"])).toBe("https://github.com/example/repo.git");
});
});
it("scrubs credentials from the origin remote before copying it", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-origin-scrub-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await git(repo, ["remote", "add", "origin", "https://x-access-token:sekret@github.com/example/repo.git"]);
const snapshot = await readGitWorkspaceSnapshot(repo);
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
expect(await git(cloneDir, ["remote", "get-url", "origin"])).toBe("https://github.com/example/repo.git");
});
});
it("leaves the shallow clone remote-less when the workspace has no origin", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-no-origin-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const snapshot = await readGitWorkspaceSnapshot(repo);
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
await expect(git(cloneDir, ["remote", "get-url", "origin"])).rejects.toThrow();
});
});
it("drops a filesystem-path origin instead of copying it into the shallow clone", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-path-origin-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await git(repo, ["remote", "add", "origin", path.join(rootDir, "elsewhere.git")]);
const snapshot = await readGitWorkspaceSnapshot(repo);
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
await expect(git(cloneDir, ["remote", "get-url", "origin"])).rejects.toThrow();
});
});
it("pushes new commits from the shallow clone to an origin that holds the base commit", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-shallow-push-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const upstream = path.join(rootDir, "upstream.git");
await mkdir(upstream, { recursive: true });
await git(upstream, ["init", "--bare"]);
await git(repo, ["remote", "add", "origin", upstream]);
await git(repo, ["push", "origin", "main"]);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
const snapshot = await readGitWorkspaceSnapshot(repo);
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
await git(cloneDir, ["config", "user.name", "Paperclip Sandbox"]);
await git(cloneDir, ["config", "user.email", "sandbox@paperclip.dev"]);
await writeFile(path.join(cloneDir, "change.txt"), "sandbox change\n", "utf8");
await git(cloneDir, ["add", "change.txt"]);
await git(cloneDir, ["commit", "-m", "sandbox change"]);
const cloneHead = await git(cloneDir, ["rev-parse", "HEAD"]);
// A filesystem-path origin is dropped by the allowlist, so configure the
// remote explicitly — the property under test is the push itself: the
// clone is shallow (single grafted commit), but the boundary commit
// already exists on the origin, so the push pack closes without full
// ancestry. That is what makes transported branches publishable.
await git(cloneDir, ["remote", "add", "origin", upstream]);
await git(cloneDir, ["push", "origin", "HEAD:refs/heads/sandbox-change"]);
expect(await git(upstream, ["rev-parse", "refs/heads/sandbox-change"])).toBe(cloneHead);
expect(await git(upstream, ["merge-base", "refs/heads/main", "refs/heads/sandbox-change"])).toBe(baseHead);
});
});
it("builds thin git delta bundles relative to the imported base", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-delta-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot).not.toBeNull();
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (remoteDir) => {
const emptyBundle = path.join(rootDir, "empty.bundle");
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir,
baseSha: baseHead,
exportRef: createRemoteGitExportRef("test"),
bundlePath: emptyBundle,
})]);
expect((await stat(emptyBundle)).size).toBe(0);
await git(remoteDir, ["config", "user.name", "Paperclip Remote"]);
await git(remoteDir, ["config", "user.email", "remote@paperclip.dev"]);
await writeFile(path.join(remoteDir, "tracked.txt"), "remote\n", "utf8");
await git(remoteDir, ["commit", "-am", "remote update"]);
const remoteHead = await git(remoteDir, ["rev-parse", "HEAD"]);
const deltaBundle = path.join(rootDir, "delta.bundle");
const importedRef = createImportedGitRef("test");
const exportRef = createRemoteGitExportRef("test");
try {
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir,
baseSha: baseHead,
exportRef,
bundlePath: deltaBundle,
})]);
expect((await stat(deltaBundle)).size).toBeGreaterThan(0);
const importedHead = await fetchGitBundleIntoLocalRef({
localDir: repo,
bundlePath: deltaBundle,
exportRef,
importedRef,
baseSha: baseHead,
});
expect(importedHead).toBe(remoteHead);
expect(await git(repo, ["rev-list", "--count", importedRef, "--not", baseHead])).toBe("1");
} finally {
await deleteLocalGitRef({ localDir: repo, ref: importedRef });
}
});
});
it("imports a diverged sandbox HEAD even when the host no longer holds baseSha", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-diverge-"));
cleanupDirs.push(rootDir);
// Host holds only the shared ancestor B (the eventual merge-base), not the
// recorded base H — the state a shared workspace lands in when it is reset
// between export and import.
const host = await createRepo(rootDir);
const mergeBase = await git(host, ["rev-parse", "HEAD"]);
// Sandbox holds B, an advanced commit H (the recorded baseSha), and a
// local-only commit S that forked from B and diverges from H.
const sandbox = path.join(rootDir, "sandbox");
await git(rootDir, ["clone", host, sandbox]);
await git(sandbox, ["config", "user.name", "Paperclip Remote"]);
await git(sandbox, ["config", "user.email", "remote@paperclip.dev"]);
await writeFile(path.join(sandbox, "advance.txt"), "advance\n", "utf8");
await git(sandbox, ["add", "-A"]);
await git(sandbox, ["commit", "-m", "advance"]);
const baseSha = await git(sandbox, ["rev-parse", "HEAD"]);
await git(sandbox, ["reset", "--hard", mergeBase]);
await writeFile(path.join(sandbox, "local.txt"), "local\n", "utf8");
await git(sandbox, ["add", "-A"]);
await git(sandbox, ["commit", "-m", "local-only"]);
const sandboxHead = await git(sandbox, ["rev-parse", "HEAD"]);
// The host genuinely lacks baseSha; the old thin bundle would name it as an
// unsatisfiable prerequisite.
await expect(git(host, ["cat-file", "-e", `${baseSha}^{commit}`])).rejects.toThrow();
const bundle = path.join(rootDir, "diverge.bundle");
const exportRef = createRemoteGitExportRef("test");
const importedRef = createImportedGitRef("test");
try {
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir: sandbox,
baseSha,
exportRef,
bundlePath: bundle,
})]);
expect((await stat(bundle)).size).toBeGreaterThan(0);
const importedHead = await fetchGitBundleIntoLocalRef({
localDir: host,
bundlePath: bundle,
exportRef,
importedRef,
baseSha,
});
expect(importedHead).toBe(sandboxHead);
// The host received the local-only commit and its parent (the merge-base).
expect(await git(host, ["cat-file", "-e", `${sandboxHead}^{commit}`])).toBe("");
} finally {
await deleteLocalGitRef({ localDir: host, ref: importedRef });
}
});
it("re-exports a full bundle that imports when the host holds neither baseSha nor the merge-base", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-ancestor-"));
cleanupDirs.push(rootDir);
// Host was reset to a strict ancestor of the eventual merge-base: it holds
// only the very first commit, not baseSha and not the fork point.
const host = await createRepo(rootDir);
const ancestor = await git(host, ["rev-parse", "HEAD"]);
const sandbox = path.join(rootDir, "sandbox");
await git(rootDir, ["clone", host, sandbox]);
await git(sandbox, ["config", "user.name", "Paperclip Remote"]);
await git(sandbox, ["config", "user.email", "remote@paperclip.dev"]);
// Advance the merge-base past the host, then baseSha past that, then a
// divergent local commit — so merge-base(baseSha, HEAD) is itself a commit
// the host does not hold.
await writeFile(path.join(sandbox, "fork.txt"), "fork\n", "utf8");
await git(sandbox, ["add", "-A"]);
await git(sandbox, ["commit", "-m", "fork point"]);
const forkPoint = await git(sandbox, ["rev-parse", "HEAD"]);
await writeFile(path.join(sandbox, "advance.txt"), "advance\n", "utf8");
await git(sandbox, ["add", "-A"]);
await git(sandbox, ["commit", "-m", "advance"]);
const baseSha = await git(sandbox, ["rev-parse", "HEAD"]);
await git(sandbox, ["reset", "--hard", forkPoint]);
await writeFile(path.join(sandbox, "local.txt"), "local\n", "utf8");
await git(sandbox, ["add", "-A"]);
await git(sandbox, ["commit", "-m", "local-only"]);
const sandboxHead = await git(sandbox, ["rev-parse", "HEAD"]);
// Host holds only the initial commit; it lacks both baseSha and the fork point.
expect(await git(host, ["rev-parse", "HEAD"])).toBe(ancestor);
await expect(git(host, ["cat-file", "-e", `${forkPoint}^{commit}`])).rejects.toThrow();
const exportRef = createRemoteGitExportRef("test");
const importedRef = createImportedGitRef("test");
// The delta bundle (relative to the merge-base = fork point) names a
// prerequisite the host lacks, so its import fails and is detected.
const deltaBundle = path.join(rootDir, "delta.bundle");
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir: sandbox,
baseSha,
exportRef,
bundlePath: deltaBundle,
})]);
let deltaError: unknown;
try {
await fetchGitBundleIntoLocalRef({ localDir: host, bundlePath: deltaBundle, exportRef, importedRef, baseSha });
} catch (error) {
deltaError = error;
}
expect(deltaError).toBeDefined();
expect(isMissingGitPrerequisiteError(deltaError)).toBe(true);
// The forced full bundle is self-contained and imports into the same host.
const fullBundle = path.join(rootDir, "full.bundle");
try {
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir: sandbox,
baseSha,
exportRef,
bundlePath: fullBundle,
forceFullBundle: true,
})]);
const importedHead = await fetchGitBundleIntoLocalRef({
localDir: host,
bundlePath: fullBundle,
exportRef,
importedRef,
baseSha,
});
expect(importedHead).toBe(sandboxHead);
} finally {
await deleteLocalGitRef({ localDir: host, ref: importedRef });
}
});
it("falls back to a full self-contained bundle when the sandbox lacks baseSha", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-full-"));
cleanupDirs.push(rootDir);
const sandbox = await createRepo(rootDir);
await writeFile(path.join(sandbox, "more.txt"), "more\n", "utf8");
await git(sandbox, ["add", "-A"]);
await git(sandbox, ["commit", "-m", "more"]);
const sandboxHead = await git(sandbox, ["rev-parse", "HEAD"]);
// A fresh, unrelated host that shares no history with the sandbox.
const host = path.join(rootDir, "fresh-host");
await mkdir(host, { recursive: true });
await git(host, ["init"]);
const bundle = path.join(rootDir, "full.bundle");
const exportRef = createRemoteGitExportRef("test");
const importedRef = createImportedGitRef("test");
try {
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir: sandbox,
// A base the sandbox does not have forces the full-bundle fallback.
baseSha: "0000000000000000000000000000000000000000",
exportRef,
bundlePath: bundle,
})]);
expect((await stat(bundle)).size).toBeGreaterThan(0);
const importedHead = await fetchGitBundleIntoLocalRef({
localDir: host,
bundlePath: bundle,
exportRef,
importedRef,
baseSha: "0000000000000000000000000000000000000000",
});
expect(importedHead).toBe(sandboxHead);
} finally {
await deleteLocalGitRef({ localDir: host, ref: importedRef });
}
});
it("creates the concurrent-history merge commit with a deterministic identity", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-merge-identity-"));
cleanupDirs.push(rootDir);
// No repo-local user.name/user.email on purpose: execution hosts are
// containers without git config, where commit-tree cannot auto-detect an
// identity. Setup commits pass their identity inline so only the merge
// commit under test depends on the sync-supplied identity.
const setupIdentity = ["-c", "user.name=Setup", "-c", "user.email=setup@paperclip.dev"];
const repo = path.join(rootDir, "repo");
await mkdir(repo, { recursive: true });
await git(repo, ["init"]);
await git(repo, ["checkout", "-b", "main"]);
await writeFile(path.join(repo, "tracked.txt"), "base\n", "utf8");
await git(repo, ["add", "tracked.txt"]);
await git(repo, [...setupIdentity, "commit", "-m", "base"]);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
await writeFile(path.join(repo, "local.txt"), "local\n", "utf8");
await git(repo, ["add", "local.txt"]);
await git(repo, [...setupIdentity, "commit", "-m", "local advance"]);
const currentHead = await git(repo, ["rev-parse", "HEAD"]);
await git(repo, ["checkout", "-b", "imported", baseHead]);
await writeFile(path.join(repo, "imported.txt"), "imported\n", "utf8");
await git(repo, ["add", "imported.txt"]);
await git(repo, [...setupIdentity, "commit", "-m", "sandbox change"]);
const importedHead = await git(repo, ["rev-parse", "HEAD"]);
await git(repo, ["checkout", "main"]);
// Ambient identity env vars would override the `-c` flags and make the
// assertion machine-dependent, so clear them for the call under test.
const identityEnvKeys = ["GIT_AUTHOR_NAME", "GIT_AUTHOR_EMAIL", "GIT_COMMITTER_NAME", "GIT_COMMITTER_EMAIL", "EMAIL"];
const savedEnv = new Map(identityEnvKeys.map((key) => [key, process.env[key]]));
for (const key of identityEnvKeys) delete process.env[key];
try {
await integrateImportedGitHead({ localDir: repo, importedHead });
} finally {
for (const [key, value] of savedEnv) {
if (value === undefined) delete process.env[key];
else process.env[key] = value;
}
}
const parents = (await git(repo, ["rev-list", "--parents", "-1", "HEAD"])).split(" ");
expect(parents.slice(1)).toEqual([currentHead, importedHead]);
expect(await git(repo, ["log", "-1", "--format=%an|%ae|%cn|%ce"]))
.toBe("Paperclip|noreply@paperclip.ing|Paperclip|noreply@paperclip.ing");
expect(await git(repo, ["log", "-1", "--format=%s"]))
.toBe(`Paperclip remote git sync merge ${importedHead.slice(0, 12)}`);
const mergedTree = await git(repo, ["ls-tree", "--name-only", "HEAD"]);
expect(mergedTree).toContain("local.txt");
expect(mergedTree).toContain("imported.txt");
});
it("grafts an imported head onto the current head when histories share no ancestor", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-graft-"));
cleanupDirs.push(rootDir);
const setupIdentity = ["-c", "user.name=Setup", "-c", "user.email=setup@paperclip.dev"];
const repo = path.join(rootDir, "repo");
await mkdir(repo, { recursive: true });
await git(repo, ["init"]);
await git(repo, ["checkout", "-b", "main"]);
await writeFile(path.join(repo, "tracked.txt"), "base\n", "utf8");
await git(repo, ["add", "tracked.txt"]);
await git(repo, [...setupIdentity, "commit", "-m", "base"]);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
await writeFile(path.join(repo, "local.txt"), "local\n", "utf8");
await git(repo, ["add", "local.txt"]);
await git(repo, [...setupIdentity, "commit", "-m", "local advance"]);
const currentHead = await git(repo, ["rev-parse", "HEAD"]);
// The shape a depth-1 shallow clone produces after `git commit --amend`:
// a parentless root commit that shares no ancestor with the host history.
const importedTree = await git(repo, ["rev-parse", `${baseHead}^{tree}`]);
const importedHead = await git(repo, [...setupIdentity, "commit-tree", importedTree, "-m", "sandbox rewrite"]);
await integrateImportedGitHead({ localDir: repo, importedHead });
const parents = (await git(repo, ["rev-list", "--parents", "-1", "HEAD"])).split(" ");
expect(parents.slice(1)).toEqual([currentHead]);
// The imported tree is taken wholesale: no base exists to merge against.
expect(await git(repo, ["rev-parse", "HEAD^{tree}"])).toBe(importedTree);
expect(await git(repo, ["log", "-1", "--format=%s"])).toBe("sandbox rewrite");
const body = await git(repo, ["log", "-1", "--format=%B"]);
expect(body).toContain(`Paperclip remote git sync graft ${importedHead.slice(0, 12)}`);
expect(body).toContain("shares no ancestor");
});
it("does not graft when merge-base fails for a reason other than missing ancestry", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-no-graft-"));
cleanupDirs.push(rootDir);
const setupIdentity = ["-c", "user.name=Setup", "-c", "user.email=setup@paperclip.dev"];
const repo = path.join(rootDir, "repo");
await mkdir(repo, { recursive: true });
await git(repo, ["init"]);
await git(repo, ["checkout", "-b", "main"]);
await writeFile(path.join(repo, "tracked.txt"), "base\n", "utf8");
await git(repo, ["add", "tracked.txt"]);
await git(repo, [...setupIdentity, "commit", "-m", "base"]);
const currentHead = await git(repo, ["rev-parse", "HEAD"]);
// A well-formed sha the repository does not hold: merge-base fails with an
// object error (exit 128), not the no-ancestor signal (exit 1). The graft
// must not fire, and the integration keeps its loud failure.
const missingHead = "0123456789abcdef0123456789abcdef01234567";
await expect(integrateImportedGitHead({ localDir: repo, importedHead: missingHead }))
.rejects.toThrow(/Failed to merge concurrent remote git histories/);
expect(await git(repo, ["rev-parse", "HEAD"])).toBe(currentHead);
});
describe("readReferencedSourceGitIgnoredPaths", () => {
it("returns null for a directory that is not a Git work tree", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-nogit-"));
cleanupDirs.push(rootDir);
const plainDir = path.join(rootDir, "plain");
await mkdir(plainDir, { recursive: true });
await writeFile(path.join(plainDir, "file.txt"), "body\n", "utf8");
await expect(readReferencedSourceGitIgnoredPaths(plainDir)).resolves.toBeNull();
});
it.each([
{ code: "workspace_git_scan_failed", exitCode: 128, signal: null, nonGit: true },
{ code: "workspace_git_scan_timeout", exitCode: 128, signal: null, nonGit: false },
{ code: "workspace_git_scan_cancelled", exitCode: 128, signal: null, nonGit: false },
{ code: "workspace_git_scan_output_limit", exitCode: 128, signal: null, nonGit: false },
{ code: "workspace_git_scan_failed", exitCode: null, signal: "SIGTERM", nonGit: false },
])("classifies scheduled non-repository failures without swallowing $code/$signal", async ({ code, exitCode, signal, nonGit }) => {
const error = Object.assign(new Error("Workspace Git scan failed"), {
code,
details: { exitCode, signal, stderr: "fatal: not a git repository (or any of the parent directories): .git" },
});
setExpensiveWorkspaceGitExecutor(async () => { throw error; });
try {
const result = readReferencedSourceGitIgnoredPaths("/plain-workspace");
if (nonGit) await expect(result).resolves.toBeNull();
else await expect(result).rejects.toBe(error);
} finally {
setExpensiveWorkspaceGitExecutor(null);
}
});
it("reads the repository top level and the ignored paths of a Git work tree", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-git-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await writeFile(path.join(repo, ".gitignore"), "secret.env\nbuild/\n", "utf8");
await writeFile(path.join(repo, "secret.env"), "TOKEN=abc\n", "utf8");
await mkdir(path.join(repo, "build"), { recursive: true });
await writeFile(path.join(repo, "build", "out.js"), "artifact\n", "utf8");
const scan = await readReferencedSourceGitIgnoredPaths(repo);
expect(scan?.toplevel).toBe(await git(repo, ["rev-parse", "--show-toplevel"]));
expect(scan?.ignoredPaths).toEqual(["build", "secret.env"]);
});
it("preserves trailing whitespace in an ignored path entry", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-trailing-ws-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
// A wildcard pattern avoids the separate rule that git trims an
// unescaped trailing space in a .gitignore pattern itself; the trailing
// space under test lives in the matched FILE name, not the pattern.
const paddedName = "secret.env ";
await writeFile(path.join(repo, ".gitignore"), "secret.env*\n", "utf8");
await writeFile(path.join(repo, paddedName), "TOKEN=abc\n", "utf8");
const scan = await readReferencedSourceGitIgnoredPaths(repo);
expect(scan?.ignoredPaths).toEqual([paddedName]);
});
it("fails closed when the parsed ignored-entry count exceeds the bound", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-bound-count-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
// Synthesize the `git ls-files --others --ignored -z` output directly,
// rather than creating ten thousand real files, by intercepting the
// scan at the executor seam. The parser must reject this before it
// sorts or re-relativizes the list.
const overLimitCount = REFERENCED_SOURCE_IGNORE_MAX_ENTRY_COUNT + 1;
const syntheticIgnored = `${Array.from({ length: overLimitCount }, (_, index) => `entry-${index}`).join("\0")}\0`;
setExpensiveWorkspaceGitExecutor(async (input) => {
if (input.operation === "referenced_source.ignored_files") {
return { stdout: syntheticIgnored, stderr: "" };
}
return await runLocalGit(input.localDir, [...input.args], {
timeout: input.timeout,
maxBuffer: input.maxBuffer,
env: input.env,
});
});
await expect(readReferencedSourceGitIgnoredPaths(repo)).rejects.toBeInstanceOf(
ReferencedSourceIgnoreScanLimitExceededError,
);
});
it("fails closed when the summed UTF-8 byte size of ignored paths exceeds the bound", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-bound-bytes-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
// One entry alone exceeds the byte bound, well under the entry-count bound.
const hugeEntry = "a".repeat(REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES + 1);
const syntheticIgnored = `${hugeEntry}\0`;
setExpensiveWorkspaceGitExecutor(async (input) => {
if (input.operation === "referenced_source.ignored_files") {
return { stdout: syntheticIgnored, stderr: "" };
}
return await runLocalGit(input.localDir, [...input.args], {
timeout: input.timeout,
maxBuffer: input.maxBuffer,
env: input.env,
});
});
await expect(readReferencedSourceGitIgnoredPaths(repo)).rejects.toBeInstanceOf(
ReferencedSourceIgnoreScanLimitExceededError,
);
});
it("fails closed on the byte bound while it is still accumulating, before it would ever reach a later entry-count breach", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-bound-order-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
// Three entries alone cross the byte bound. Many more small entries
// follow, so the FULL response also carries more than the entry-count
// bound. A parser that fully builds the list before checking either
// bound (post-parse) would report the entry-count breach, because it
// checks that bound first against the whole materialized list. A
// parser that checks both bounds while the list accumulates rejects on
// the byte bound instead, the moment the third entry crosses it, well
// before the count bound is ever reached.
const oversizedEntry = "a".repeat(Math.ceil(REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES / 2) + 1);
const bigEntries = Array.from({ length: 3 }, (_, index) => `${oversizedEntry}-${index}`);
const trailingEntries = Array.from(
{ length: REFERENCED_SOURCE_IGNORE_MAX_ENTRY_COUNT + 10 },
(_, index) => `trailing-${index}`,
);
const syntheticIgnored = `${[...bigEntries, ...trailingEntries].join("\0")}\0`;
setExpensiveWorkspaceGitExecutor(async (input) => {
if (input.operation === "referenced_source.ignored_files") {
return { stdout: syntheticIgnored, stderr: "" };
}
return await runLocalGit(input.localDir, [...input.args], {
timeout: input.timeout,
maxBuffer: input.maxBuffer,
env: input.env,
});
});
await expect(readReferencedSourceGitIgnoredPaths(repo)).rejects.toThrow(/UTF-8 bytes/);
});
it("bounds the raw command-output allowance to the ignore-scan limits, not the general-purpose full-tree ceiling", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-raw-buffer-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
let observedMaxBuffer: number | undefined;
setExpensiveWorkspaceGitExecutor(async (input) => {
if (input.operation === "referenced_source.ignored_files") {
observedMaxBuffer = input.maxBuffer;
}
return await runLocalGit(input.localDir, [...input.args], {
timeout: input.timeout,
maxBuffer: input.maxBuffer,
env: input.env,
});
});
await readReferencedSourceGitIgnoredPaths(repo);
// Enough headroom for a scan within bounds to complete, but a small
// multiple of the byte bound — not the far larger allowance the
// anchor workspace's general-purpose full-tree reads use.
expect(observedMaxBuffer).toBeGreaterThan(REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES);
expect(observedMaxBuffer).toBeLessThan(16 * 1024 * 1024);
});
it("does not fail closed on a huge amount of unrelated tracked-change and untracked noise, when the ignored set itself stays in bounds", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-mixed-status-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await writeFile(path.join(repo, ".gitignore"), "secret.env\n", "utf8");
await writeFile(path.join(repo, "secret.env"), "TOKEN=abc\n", "utf8");
// Many long-named, untracked, NOT-ignored files at the repository root.
// `git status` reports one record per file (root-level files are never
// collapsed the way an entirely untracked directory is), so this alone
// makes the raw `git status --ignored` response exceed the raw buffer
// bound this scan used to apply to the WHOLE response, well before the
// parser ever got to discard these non-ignored records. The ignored set
// above stays a single small entry throughout.
const noiseNameLength = 220;
const noiseFileCount = 30_000;
const noiseNames = Array.from(
{ length: noiseFileCount },
(_, index) => `${"n".repeat(noiseNameLength - 6)}${String(index).padStart(6, "0")}`,
);
const writeConcurrency = 200;
for (let start = 0; start < noiseNames.length; start += writeConcurrency) {
const batch = noiseNames.slice(start, start + writeConcurrency);
await Promise.all(batch.map((name) => writeFile(path.join(repo, name), "", "utf8")));
}
// Confirm this test actually reproduces the reported defect precondition:
// the raw `git status --ignored` response for this repository state is
// larger than the 4 MiB raw buffer bound the scan used to apply to the
// whole response, not just to the declared ignored-set limits. A large
// explicit maxBuffer is required here only to observe that raw size;
// the scan under test never issues this command.
const rawStatusResult = await runLocalGit(
repo,
["status", "--ignored", "--porcelain=v1", "-z", "--untracked-files=normal"],
{ maxBuffer: 16 * 1024 * 1024 },
);
expect(Buffer.byteLength(rawStatusResult.stdout, "utf8")).toBeGreaterThan(REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES * 2);
const scan = await readReferencedSourceGitIgnoredPaths(repo);
expect(scan?.ignoredPaths).toEqual(["secret.env"]);
});
it("routes both scan commands through the registered scheduler instead of spawning git directly", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-scheduler-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
await writeFile(path.join(repo, ".gitignore"), "build/\n", "utf8");
await mkdir(path.join(repo, "build"), { recursive: true });
await writeFile(path.join(repo, "build", "out.js"), "artifact\n", "utf8");
const operations: string[] = [];
setExpensiveWorkspaceGitExecutor(async (input) => {
operations.push(input.operation);
return await runLocalGit(input.localDir, [...input.args], {
timeout: input.timeout,
maxBuffer: input.maxBuffer,
env: input.env,
});
});
const scan = await readReferencedSourceGitIgnoredPaths(repo);
expect(scan?.ignoredPaths).toEqual(["build"]);
// Both the toplevel probe and the ignored-paths read go through the SAME
// process-wide admission seam the anchor workspace's expensive reads
// use. A host process that bounds concurrent scans there also bounds
// referenced-project scans, so a run with many referenced projects
// cannot spawn one unbounded Git process per project.
expect(operations.sort()).toEqual(["referenced_source.ignored_files", "referenced_source.toplevel"]);
});
it("carries the hardened arguments and does not inherit a poisoned GIT_CONFIG_GLOBAL", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-hardened-env-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const badGlobalConfig = path.join(rootDir, "bad-global-gitconfig");
await writeFile(badGlobalConfig, "this is not valid git config syntax [[[\n", "utf8");
const priorGlobal = process.env.GIT_CONFIG_GLOBAL;
process.env.GIT_CONFIG_GLOBAL = badGlobalConfig;
try {
// A plain invocation inherits the poisoned global config and fails to parse it.
await expect(execFile("git", ["-C", repo, "status", "--porcelain"])).rejects.toThrow();
// The hardened helper does not inherit GIT_CONFIG_GLOBAL from this process's
// environment, so it succeeds regardless.
await expect(readReferencedSourceGitIgnoredPaths(repo)).resolves.toMatchObject({ ignoredPaths: [] });
} finally {
if (priorGlobal === undefined) delete process.env.GIT_CONFIG_GLOBAL;
else process.env.GIT_CONFIG_GLOBAL = priorGlobal;
}
});
it("neutralizes a repository-local core.fsmonitor hook", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-fsmonitor-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const markerPath = path.join(rootDir, "pwned.txt");
// A malicious repository-local config: a non-boolean `core.fsmonitor` value
// is a hook COMMAND Git runs on every status-like read. `--no-optional-locks`
// alone does not stop this; only the command-line `-c core.fsmonitor=false`
// override does, because command-line config wins over repository config.
await git(repo, ["config", "core.fsmonitor", `sh -c 'touch ${markerPath}; printf 1'`]);
await readReferencedSourceGitIgnoredPaths(repo);
await expect(stat(markerPath)).rejects.toThrow();
});
});
});
describe("sanitizeGitRemoteUrl", () => {
it("strips userinfo, query, and fragment from http(s) URLs", () => {
expect(sanitizeGitRemoteUrl("https://x-access-token:sekret@github.com/example/repo.git"))
.toBe("https://github.com/example/repo.git");
expect(sanitizeGitRemoteUrl("https://sekret-token@github.com/example/repo.git"))
.toBe("https://github.com/example/repo.git");
expect(sanitizeGitRemoteUrl("http://user:pass@git.internal/example/repo.git"))
.toBe("http://git.internal/example/repo.git");
expect(sanitizeGitRemoteUrl("https://github.com/example/repo.git?private_token=sekret#fragment"))
.toBe("https://github.com/example/repo.git");
});
it("strips password and query from ssh-scheme URLs but keeps the username", () => {
expect(sanitizeGitRemoteUrl("ssh://git@github.com/example/repo.git"))
.toBe("ssh://git@github.com/example/repo.git");
expect(sanitizeGitRemoteUrl("ssh://git:sekret@github.com/example/repo.git"))
.toBe("ssh://git@github.com/example/repo.git");
expect(sanitizeGitRemoteUrl("git+ssh://git@github.com/example/repo.git?key=sekret"))
.toBe("git+ssh://git@github.com/example/repo.git");
});
it("keeps credential-free scp-like remotes unchanged", () => {
expect(sanitizeGitRemoteUrl("git@github.com:example/repo.git"))
.toBe("git@github.com:example/repo.git");
expect(sanitizeGitRemoteUrl("https://github.com/example/repo.git"))
.toBe("https://github.com/example/repo.git");
});
it("drops every shape whose credential surface is unknown", () => {
// Filesystem paths are useless on the execution host and could leak
// host-layout details; unknown schemes and malformed userinfo could carry
// embedded secrets the sanitizer cannot recognize. All fail closed.
expect(sanitizeGitRemoteUrl("/tmp/local/upstream.git")).toBeNull();
expect(sanitizeGitRemoteUrl("ftp://user:pass@host/repo.git")).toBeNull();
expect(sanitizeGitRemoteUrl("user:pass@host:path/repo.git")).toBeNull();
expect(sanitizeGitRemoteUrl("host.example:path/repo.git")).toBeNull();
});
it("returns null for empty input", () => {
expect(sanitizeGitRemoteUrl("")).toBeNull();
expect(sanitizeGitRemoteUrl(" ")).toBeNull();
});
});