mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-10 20:50:08 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox and SSH runtimes need to preserve agent work across isolated execution environments > - Git-backed workspaces were being copied mostly as filesystem archives, which breaks when `.git` points outside the mounted workspace and makes sandbox agents unable to publish their own branches > - Large ignored dependency trees could also be swept into the sandbox overlay, causing multi-GB transfers and max-string failures in some sandbox clients > - This pull request makes sandbox runtime setup use a git-backed HEAD sync plus a small dirty/untracked overlay, and bounds sandbox file transfers so large archives do not need one huge string > - The benefit is that sandbox agents can commit and push from a usable git checkout without uploading dependency trees such as `node_modules` ## Linked Issues or Issue Description Refs #8395 No public duplicate issue or PR was found after searches for `sandbox git workspace`, `git push sandbox`, and `node_modules sandbox upload`. Bug report: - What happened: sandbox-backed agent workspaces could receive a `.git` file that pointed at host-only git state, leaving the sandbox unable to run normal git workflows. The sandbox overlay upload could also include ignored dependency directories, creating very large transfers. - Expected behavior: sandbox and remote runtimes should prepare a usable git-backed workspace, copy only the necessary workspace overlay, and restore git history plus file changes without depending on a host-only `.git` path. - Steps to reproduce: 1. Run an agent in a sandbox-backed workspace whose local git checkout is a worktree. 2. Ask the agent to complete a GitHub workflow that requires commit/push access. 3. Observe that git operations can fail inside the sandbox, and ignored dependency trees can be uploaded as part of the workspace overlay. - Paperclip version or commit: reproduced against `master` before this PR, base `7aa212296eb1`. - Deployment mode: local dev / sandbox-backed runtime. - Installation method: built from source. - Agent adapters involved: local adapters using shared adapter-utils runtime preparation. - Database mode: not database-related. - Access context: agent runtime. - Local verification environment: Node.js v25.6.1, pnpm 9.15.4, macOS arm64. - Privacy checklist: all pasted output was reviewed for secrets, private hostnames, local usernames, and internal instance links. ## What Changed - Added a GitHub workflow push preflight so agent runs can detect missing push credentials when a workflow explicitly needs GitHub publishing. - Added shared git workspace sync helpers for shallow HEAD import/export and dirty/untracked overlay tracking. - Updated sandbox managed runtime setup to use git history plus a selected overlay instead of uploading the full local workspace for git-backed workspaces. - Bounded sandbox archive upload/download paths so large payloads stream or chunk instead of materializing one oversized string. - Excluded `.git` and ignored dependency trees from sandbox upload, download, and restore baselines while preserving local ignored directories during sync-back. - Added focused tests for git workspace sync, sandbox overlay selection, transfer chunking, and heartbeat push-preflight behavior. ## Verification - `pnpm exec vitest run packages/adapter-utils/src/git-workspace-sync.test.ts packages/adapter-utils/src/sandbox-managed-runtime.test.ts packages/adapter-utils/src/command-managed-runtime.test.ts server/src/__tests__/heartbeat-project-env.test.ts server/src/__tests__/heartbeat-workspace-session.test.ts` - `pnpm --filter @paperclipai/adapter-utils typecheck` - `pnpm --filter @paperclipai/server typecheck` - `pnpm build` - Public-hygiene scan of the PR diff and commit messages for internal issue ids, local paths, private hostnames, and obvious token patterns. ## Risks - Medium risk: this changes sandbox runtime synchronization semantics for git-backed workspaces, especially around dirty tracked files, untracked files, deleted paths, and ignored files. - The main mitigation is focused test coverage for upload contents, restore exclusions, and git round-trip behavior. - The SSH runtime keeps the current bundle-based implementation from `master`; this PR only aligns shared excludes and sandbox behavior with that model. ## Model Used OpenAI Codex, GPT-5-based coding agent, tool-enabled shell/git/GitHub workflow, with code execution and repository inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
129 lines
4.7 KiB
TypeScript
129 lines
4.7 KiB
TypeScript
import { execFile as execFileCallback } from "node:child_process";
|
|
import { lstat, mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { promisify } from "node:util";
|
|
import { afterEach, describe, expect, it } from "vitest";
|
|
|
|
import {
|
|
buildRemoteGitDeltaBundleScript,
|
|
createImportedGitRef,
|
|
createRemoteGitExportRef,
|
|
deleteLocalGitRef,
|
|
fetchGitBundleIntoLocalRef,
|
|
readGitWorkspaceSnapshot,
|
|
runLocalGit,
|
|
withShallowGitWorkspaceClone,
|
|
} from "./git-workspace-sync.js";
|
|
|
|
const execFile = promisify(execFileCallback);
|
|
|
|
async function git(cwd: string, args: string[]): Promise<string> {
|
|
return (await runLocalGit(cwd, args)).stdout.trim();
|
|
}
|
|
|
|
describe("git workspace sync", () => {
|
|
const cleanupDirs: string[] = [];
|
|
|
|
afterEach(async () => {
|
|
while (cleanupDirs.length > 0) {
|
|
const dir = cleanupDirs.pop();
|
|
if (!dir) continue;
|
|
await rm(dir, { recursive: true, force: true }).catch(() => undefined);
|
|
}
|
|
});
|
|
|
|
async function createRepo(rootDir: string): Promise<string> {
|
|
const repo = path.join(rootDir, "repo");
|
|
await mkdir(repo, { recursive: true });
|
|
await git(repo, ["init"]);
|
|
await git(repo, ["checkout", "-b", "main"]);
|
|
await git(repo, ["config", "user.name", "Paperclip Test"]);
|
|
await git(repo, ["config", "user.email", "test@paperclip.dev"]);
|
|
await writeFile(path.join(repo, "tracked.txt"), "base\n", "utf8");
|
|
await git(repo, ["add", "tracked.txt"]);
|
|
await git(repo, ["commit", "-m", "base"]);
|
|
return repo;
|
|
}
|
|
|
|
it("creates a shallow standalone clone from the local HEAD snapshot", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-sync-"));
|
|
cleanupDirs.push(rootDir);
|
|
const repo = await createRepo(rootDir);
|
|
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
|
|
await rm(path.join(repo, "tracked.txt"));
|
|
|
|
const snapshot = await readGitWorkspaceSnapshot(repo);
|
|
expect(snapshot).toMatchObject({
|
|
headCommit: baseHead,
|
|
branchName: "main",
|
|
deletedPaths: ["tracked.txt"],
|
|
});
|
|
|
|
await withShallowGitWorkspaceClone({
|
|
localDir: repo,
|
|
snapshot: snapshot!,
|
|
}, async (cloneDir) => {
|
|
expect((await lstat(path.join(cloneDir, ".git"))).isDirectory()).toBe(true);
|
|
await expect(readFile(path.join(cloneDir, ".git", "shallow"), "utf8")).resolves.toContain(baseHead);
|
|
expect(await git(cloneDir, ["rev-list", "--count", "HEAD"])).toBe("1");
|
|
expect(await git(cloneDir, ["branch", "--show-current"])).toBe("main");
|
|
await expect(readFile(path.join(cloneDir, "tracked.txt"), "utf8")).resolves.toBe("base\n");
|
|
});
|
|
});
|
|
|
|
it("builds thin git delta bundles relative to the imported base", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-delta-"));
|
|
cleanupDirs.push(rootDir);
|
|
const repo = await createRepo(rootDir);
|
|
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
|
|
const snapshot = await readGitWorkspaceSnapshot(repo);
|
|
expect(snapshot).not.toBeNull();
|
|
|
|
await withShallowGitWorkspaceClone({
|
|
localDir: repo,
|
|
snapshot: snapshot!,
|
|
}, async (remoteDir) => {
|
|
const emptyBundle = path.join(rootDir, "empty.bundle");
|
|
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
|
|
remoteDir,
|
|
baseSha: baseHead,
|
|
exportRef: createRemoteGitExportRef("test"),
|
|
bundlePath: emptyBundle,
|
|
})]);
|
|
expect((await stat(emptyBundle)).size).toBe(0);
|
|
|
|
await git(remoteDir, ["config", "user.name", "Paperclip Remote"]);
|
|
await git(remoteDir, ["config", "user.email", "remote@paperclip.dev"]);
|
|
await writeFile(path.join(remoteDir, "tracked.txt"), "remote\n", "utf8");
|
|
await git(remoteDir, ["commit", "-am", "remote update"]);
|
|
const remoteHead = await git(remoteDir, ["rev-parse", "HEAD"]);
|
|
|
|
const deltaBundle = path.join(rootDir, "delta.bundle");
|
|
const importedRef = createImportedGitRef("test");
|
|
const exportRef = createRemoteGitExportRef("test");
|
|
try {
|
|
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
|
|
remoteDir,
|
|
baseSha: baseHead,
|
|
exportRef,
|
|
bundlePath: deltaBundle,
|
|
})]);
|
|
expect((await stat(deltaBundle)).size).toBeGreaterThan(0);
|
|
|
|
const importedHead = await fetchGitBundleIntoLocalRef({
|
|
localDir: repo,
|
|
bundlePath: deltaBundle,
|
|
exportRef,
|
|
importedRef,
|
|
baseSha: baseHead,
|
|
});
|
|
expect(importedHead).toBe(remoteHead);
|
|
expect(await git(repo, ["rev-list", "--count", importedRef, "--not", baseHead])).toBe("1");
|
|
} finally {
|
|
await deleteLocalGitRef({ localDir: repo, ref: importedRef });
|
|
}
|
|
});
|
|
});
|
|
});
|