Files
PaperClipAI/packages/adapter-utils/src/git-workspace-sync.test.ts
T
Devin FoleyandPaperclip 2c98c8e1e5 Fix sandbox git publishing and large workspace uploads (#8422)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Sandbox and SSH runtimes need to preserve agent work across isolated
execution environments
> - Git-backed workspaces were being copied mostly as filesystem
archives, which breaks when `.git` points outside the mounted workspace
and makes sandbox agents unable to publish their own branches
> - Large ignored dependency trees could also be swept into the sandbox
overlay, causing multi-GB transfers and max-string failures in some
sandbox clients
> - This pull request makes sandbox runtime setup use a git-backed HEAD
sync plus a small dirty/untracked overlay, and bounds sandbox file
transfers so large archives do not need one huge string
> - The benefit is that sandbox agents can commit and push from a usable
git checkout without uploading dependency trees such as `node_modules`

## Linked Issues or Issue Description

Refs #8395

No public duplicate issue or PR was found after searches for `sandbox
git workspace`, `git push sandbox`, and `node_modules sandbox upload`.

Bug report:

- What happened: sandbox-backed agent workspaces could receive a `.git`
file that pointed at host-only git state, leaving the sandbox unable to
run normal git workflows. The sandbox overlay upload could also include
ignored dependency directories, creating very large transfers.
- Expected behavior: sandbox and remote runtimes should prepare a usable
git-backed workspace, copy only the necessary workspace overlay, and
restore git history plus file changes without depending on a host-only
`.git` path.
- Steps to reproduce:
1. Run an agent in a sandbox-backed workspace whose local git checkout
is a worktree.
2. Ask the agent to complete a GitHub workflow that requires commit/push
access.
3. Observe that git operations can fail inside the sandbox, and ignored
dependency trees can be uploaded as part of the workspace overlay.
- Paperclip version or commit: reproduced against `master` before this
PR, base `7aa212296eb1`.
- Deployment mode: local dev / sandbox-backed runtime.
- Installation method: built from source.
- Agent adapters involved: local adapters using shared adapter-utils
runtime preparation.
- Database mode: not database-related.
- Access context: agent runtime.
- Local verification environment: Node.js v25.6.1, pnpm 9.15.4, macOS
arm64.
- Privacy checklist: all pasted output was reviewed for secrets, private
hostnames, local usernames, and internal instance links.

## What Changed

- Added a GitHub workflow push preflight so agent runs can detect
missing push credentials when a workflow explicitly needs GitHub
publishing.
- Added shared git workspace sync helpers for shallow HEAD import/export
and dirty/untracked overlay tracking.
- Updated sandbox managed runtime setup to use git history plus a
selected overlay instead of uploading the full local workspace for
git-backed workspaces.
- Bounded sandbox archive upload/download paths so large payloads stream
or chunk instead of materializing one oversized string.
- Excluded `.git` and ignored dependency trees from sandbox upload,
download, and restore baselines while preserving local ignored
directories during sync-back.
- Added focused tests for git workspace sync, sandbox overlay selection,
transfer chunking, and heartbeat push-preflight behavior.

## Verification

- `pnpm exec vitest run
packages/adapter-utils/src/git-workspace-sync.test.ts
packages/adapter-utils/src/sandbox-managed-runtime.test.ts
packages/adapter-utils/src/command-managed-runtime.test.ts
server/src/__tests__/heartbeat-project-env.test.ts
server/src/__tests__/heartbeat-workspace-session.test.ts`
- `pnpm --filter @paperclipai/adapter-utils typecheck`
- `pnpm --filter @paperclipai/server typecheck`
- `pnpm build`
- Public-hygiene scan of the PR diff and commit messages for internal
issue ids, local paths, private hostnames, and obvious token patterns.

## Risks

- Medium risk: this changes sandbox runtime synchronization semantics
for git-backed workspaces, especially around dirty tracked files,
untracked files, deleted paths, and ignored files.
- The main mitigation is focused test coverage for upload contents,
restore exclusions, and git round-trip behavior.
- The SSH runtime keeps the current bundle-based implementation from
`master`; this PR only aligns shared excludes and sandbox behavior with
that model.

## Model Used

OpenAI Codex, GPT-5-based coding agent, tool-enabled shell/git/GitHub
workflow, with code execution and repository inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-06-20 22:03:55 -07:00

129 lines
4.7 KiB
TypeScript

import { execFile as execFileCallback } from "node:child_process";
import { lstat, mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { promisify } from "node:util";
import { afterEach, describe, expect, it } from "vitest";
import {
buildRemoteGitDeltaBundleScript,
createImportedGitRef,
createRemoteGitExportRef,
deleteLocalGitRef,
fetchGitBundleIntoLocalRef,
readGitWorkspaceSnapshot,
runLocalGit,
withShallowGitWorkspaceClone,
} from "./git-workspace-sync.js";
const execFile = promisify(execFileCallback);
async function git(cwd: string, args: string[]): Promise<string> {
return (await runLocalGit(cwd, args)).stdout.trim();
}
describe("git workspace sync", () => {
const cleanupDirs: string[] = [];
afterEach(async () => {
while (cleanupDirs.length > 0) {
const dir = cleanupDirs.pop();
if (!dir) continue;
await rm(dir, { recursive: true, force: true }).catch(() => undefined);
}
});
async function createRepo(rootDir: string): Promise<string> {
const repo = path.join(rootDir, "repo");
await mkdir(repo, { recursive: true });
await git(repo, ["init"]);
await git(repo, ["checkout", "-b", "main"]);
await git(repo, ["config", "user.name", "Paperclip Test"]);
await git(repo, ["config", "user.email", "test@paperclip.dev"]);
await writeFile(path.join(repo, "tracked.txt"), "base\n", "utf8");
await git(repo, ["add", "tracked.txt"]);
await git(repo, ["commit", "-m", "base"]);
return repo;
}
it("creates a shallow standalone clone from the local HEAD snapshot", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-sync-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
await rm(path.join(repo, "tracked.txt"));
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot).toMatchObject({
headCommit: baseHead,
branchName: "main",
deletedPaths: ["tracked.txt"],
});
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
expect((await lstat(path.join(cloneDir, ".git"))).isDirectory()).toBe(true);
await expect(readFile(path.join(cloneDir, ".git", "shallow"), "utf8")).resolves.toContain(baseHead);
expect(await git(cloneDir, ["rev-list", "--count", "HEAD"])).toBe("1");
expect(await git(cloneDir, ["branch", "--show-current"])).toBe("main");
await expect(readFile(path.join(cloneDir, "tracked.txt"), "utf8")).resolves.toBe("base\n");
});
});
it("builds thin git delta bundles relative to the imported base", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-delta-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot).not.toBeNull();
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (remoteDir) => {
const emptyBundle = path.join(rootDir, "empty.bundle");
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir,
baseSha: baseHead,
exportRef: createRemoteGitExportRef("test"),
bundlePath: emptyBundle,
})]);
expect((await stat(emptyBundle)).size).toBe(0);
await git(remoteDir, ["config", "user.name", "Paperclip Remote"]);
await git(remoteDir, ["config", "user.email", "remote@paperclip.dev"]);
await writeFile(path.join(remoteDir, "tracked.txt"), "remote\n", "utf8");
await git(remoteDir, ["commit", "-am", "remote update"]);
const remoteHead = await git(remoteDir, ["rev-parse", "HEAD"]);
const deltaBundle = path.join(rootDir, "delta.bundle");
const importedRef = createImportedGitRef("test");
const exportRef = createRemoteGitExportRef("test");
try {
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir,
baseSha: baseHead,
exportRef,
bundlePath: deltaBundle,
})]);
expect((await stat(deltaBundle)).size).toBeGreaterThan(0);
const importedHead = await fetchGitBundleIntoLocalRef({
localDir: repo,
bundlePath: deltaBundle,
exportRef,
importedRef,
baseSha: baseHead,
});
expect(importedHead).toBe(remoteHead);
expect(await git(repo, ["rev-list", "--count", importedRef, "--not", baseHead])).toBe("1");
} finally {
await deleteLocalGitRef({ localDir: repo, ref: importedRef });
}
});
});
});