fix(adapter-utils): honor .gitignore for referenced-project staging (#12184)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Sandbox adapters stage project files before an agent starts.
> - Referenced projects ignored Git-ignored paths and copied large local
directories.
> - This behavior increased staging time and disk use, and it differed
from anchor workspaces.
> - This pull request resolves Git-ignored paths once and shares that
result across all referenced-project consumers.
> - The benefit is smaller, faster, and consistent project staging.

## Linked Issues or Issue Description

No public GitHub issue exists for this bug.

**What happened?**
Referenced-project staging copied Git-ignored paths, except for a fixed
list of heavy directory names. A large repository therefore used much
more time and disk space than the same repository in an anchor
workspace.

**Expected behavior**
Referenced-project staging should exclude the same Git-ignored paths
that the workspace staging path excludes.

**Steps to reproduce**
1. Create a referenced project with a large Git-ignored directory.
2. Start a sandbox or SSH run that stages the referenced project.
3. Observe that the ignored directory enters the staged content.

**Paperclip version or commit**
Commit `9964b034bbff24e700c8eccf5a8b1fc3daa44bf2`.

**Deployment mode**
Built from source.

## What Changed

- Resolve each referenced project's Git-ignored paths once before
staging.
- Carry the resolved paths as a required field on
`SandboxAdditionalSource`.
- Reuse the resolved paths in sandbox staging, SSH staging, and
content-signature code.
- Harden the read-only Git helper with a bounded process, a reduced
environment, and disabled system and global configuration.
- Fail closed on Git errors, timeouts, and invalid path relations.
- Escape tar glob metacharacters in ignore-derived exclude entries.
- Add and update unit tests for the resolver and its three consumers.

## Verification

- `pnpm vitest run --config packages/adapter-utils/vitest.config.ts`
passes 266 tests locally.
- `pnpm exec tsc --noEmit -p packages/adapter-utils/tsconfig.json`
passes locally.
- CI must pass on this pull request.
- Greptile must report 5/5 with no unresolved comments before merge.

## Risks

- A Git error or timeout now prevents staging for the affected
referenced project.
- The resolver uses a bounded read-only Git process and fails closed by
design.
- The change stays inside `packages/adapter-utils` and does not change
the database schema.

## Model Used

Claude Sonnet 5 (Anthropic) assisted the implementation with code
execution and tool use. The exact context window and reasoning mode are
not recorded.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed 2026-08-25 14:22:47 -07:00
1 parent 9fc2f594ae
commit 6880213de5
12 files changed
+902 -49

No files matched your search

@@ -22,6 +22,7 @@ import {
prepareAdapterExecutionTargetRuntime,
readAdapterExecutionTarget,
resolveAdapterExecutionTargetTimeout,
resolveReferencedSourceIgnore,
runAdapterExecutionTargetShellCommand,
startAdapterExecutionTargetPaperclipBridge,
startAdapterExecutionTargetProcessSessionBridge,
@@ -31,6 +32,7 @@ import {
type AdapterExecutionTargetTimeoutResolution,
type AdapterManagedRuntimeAsset,
type PreparedAdapterExecutionTargetRuntime,
type ReferencedSourceIgnoreResolution,
type SandboxAdditionalSource,
} from "@paperclipai/adapter-utils/execution-target";
import type { DuplexLossReason } from "../duplex-observability.js";
@@ -519,9 +521,14 @@ export function finalizeLaunchEnvironment(
}
// Directory names the staging path never ships for a referenced project (heavy
// build/cache output and git history). The content signature skips them so it
// reflects only the staged tree and never reads their bytes. Keep this set equal
// to the staging excludes in the sandbox and remote runtimes.
// build/cache output and git history), applied regardless of the project's
// ignore resolution. The content signature skips them so it reflects only the
// staged tree and never reads their bytes. Keep this set equal to the fixed
// excludes the sandbox and SSH runtimes always apply. A project's OWN resolved
// Git-ignored paths (see `resolveReferencedSourceIgnore`) are matched
// separately, by relative path, inside `referencedSourceContentSignature` — that
// is the real invariant now: the signature and both staging lanes must consume
// the SAME one resolution per project, not just this fixed name list.
const REFERENCED_SOURCE_SIGNATURE_SKIP_DIRS = new Set([
"node_modules",
"vendor",
@@ -551,12 +558,30 @@ const REFERENCED_SOURCE_SIGNATURE_SKIP_DIRS = new Set([
* re-checkout that restores the same size and timestamp. The byte hash busts on any
* content change, so the fingerprint busts and the next launch stages the current
* tree. The walk skips the heavy build, cache, and git directories the staging path
* never ships, and records a symlink by its target text without following it. On a
* read error the function returns a stable marker, so the fingerprint does not churn
* while staging surfaces the real error. The walk runs only when the run carries
* referenced projects (the multi-project sync path).
* never ships, plus the project's own resolved Git-ignored paths, and records a
* symlink by its target text without following it. On a read error the function
* returns a stable marker, so the fingerprint does not churn while staging
* surfaces the real error. The walk runs only when the run carries referenced
* projects (the multi-project sync path).
*
* `ignoreResolution` is the ONE resolution `resolveReferencedSourceIgnore`
* computed for this project — the same one the sandbox lane and the SSH lane
* consume. A `failed` resolution skips the walk entirely and returns a stable
* marker instead, because a failed project is not staged and its bytes are not
* read anywhere.
*/
async function referencedSourceContentSignature(localPath: string): Promise<string> {
export async function referencedSourceContentSignature(
localPath: string,
ignoreResolution: ReferencedSourceIgnoreResolution,
): Promise<string> {
if (ignoreResolution.kind === "failed") {
return `unreadable:${ignoreResolution.reason}`;
}
const isIgnoredByGitResolution = (relativePath: string): boolean =>
ignoreResolution.kind === "git" &&
ignoreResolution.ignoredPaths.some(
(entry) => relativePath === entry || relativePath.startsWith(`${entry}/`),
);
const hash = createHash("sha256");
const walk = async (relative: string): Promise<void> => {
const current = relative ? path.join(localPath, relative) : localPath;
@@ -564,6 +589,9 @@ async function referencedSourceContentSignature(localPath: string): Promise<stri
dirents.sort((left, right) => (left.name < right.name ? -1 : left.name > right.name ? 1 : 0));
for (const dirent of dirents) {
const next = relative ? path.posix.join(relative, dirent.name) : dirent.name;
if (isIgnoredByGitResolution(next)) {
continue;
}
if (dirent.isDirectory()) {
if (REFERENCED_SOURCE_SIGNATURE_SKIP_DIRS.has(dirent.name)) {
continue;
@@ -1545,32 +1573,56 @@ async function buildRuntime(input: {
const additionalSourceRecords = (
Array.isArray(realizationContext.additional) ? realizationContext.additional : []
).map((entry) => parseObject(entry));
const additionalSources: SandboxAdditionalSource[] = additionalSourceRecords
.map((entry) => ({ localPath: asString(entry.path, ""), projectId: asString(entry.projectId, "") }))
const additionalSourceCandidates = additionalSourceRecords
.map((entry) => ({
localPath: asString(entry.path, ""),
projectId: asString(entry.projectId, ""),
projectWorkspaceId: asString(entry.projectWorkspaceId, ""),
repoUrl: asString(entry.repoUrl, ""),
repoRef: asString(entry.repoRef, ""),
}))
.filter((entry) => entry.localPath.length > 0 && entry.projectId.length > 0);
// Resolve each referenced project's Git-ignored paths ONCE, here, before any
// staging site runs. The sandbox lane, the SSH lane, and the content signature
// below all consume this SAME resolution per project, so they can never apply
// a different exclusion set to the same project. See
// `resolveReferencedSourceIgnore` for the fail-closed rules.
const additionalSourcesWithIgnore = await Promise.all(
additionalSourceCandidates.map(async (entry) => ({
...entry,
ignoreResolution: await resolveReferencedSourceIgnore(entry.localPath),
})),
);
const additionalSources: SandboxAdditionalSource[] = additionalSourcesWithIgnore.map((entry) => ({
localPath: entry.localPath,
projectId: entry.projectId,
ignoreResolution: entry.ignoreResolution,
}));
// Stable identity of the referenced-project set for the session fingerprint.
// The staged-runtime cache reuses already-staged referenced-project trees on a
// compatible resume, so the fingerprint must change when the set OR a project's
// pinned checkout changes. Without this, a resume reuses a stale staged tree.
// Fold in each project's id, host path, workspace id, and pinned ref; sort by
// projectId so the identity depends on the set, not the record order.
const additionalSourcesIdentityBase = additionalSourceRecords
const additionalSourcesIdentityBase = additionalSourcesWithIgnore
.map((entry) => ({
projectId: asString(entry.projectId, ""),
localPath: asString(entry.path, ""),
projectWorkspaceId: asString(entry.projectWorkspaceId, ""),
repoUrl: asString(entry.repoUrl, ""),
repoRef: asString(entry.repoRef, ""),
projectId: entry.projectId,
localPath: entry.localPath,
projectWorkspaceId: entry.projectWorkspaceId,
repoUrl: entry.repoUrl,
repoRef: entry.repoRef,
ignoreResolution: entry.ignoreResolution,
}))
.filter((entry) => entry.localPath.length > 0 && entry.projectId.length > 0)
.sort((a, b) => (a.projectId < b.projectId ? -1 : a.projectId > b.projectId ? 1 : 0));
// Metadata alone does not change on a content-only checkout change (same host
// path and pinned ref, new file bytes). Fold in each tree's content signature so
// a file add, remove, or edit busts the fingerprint and the resume re-stages.
// The signature reads the same `ignoreResolution` the staging sites above use,
// so it never disagrees with what was actually shipped.
const additionalSourcesIdentity = await Promise.all(
additionalSourcesIdentityBase.map(async (entry) => ({
additionalSourcesIdentityBase.map(async ({ ignoreResolution, ...entry }) => ({
...entry,
contentSignature: await referencedSourceContentSignature(entry.localPath),
contentSignature: await referencedSourceContentSignature(entry.localPath, ignoreResolution),
})),
);
// Referenced-project workspace hints exposed to the agent through PAPERCLIP_WORKSPACES_JSON. The