mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-06 10:48:12 +02:00
fix(adapter-utils): honor .gitignore for referenced-project staging (#12184)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Sandbox adapters stage project files before an agent starts. > - Referenced projects ignored Git-ignored paths and copied large local directories. > - This behavior increased staging time and disk use, and it differed from anchor workspaces. > - This pull request resolves Git-ignored paths once and shares that result across all referenced-project consumers. > - The benefit is smaller, faster, and consistent project staging. ## Linked Issues or Issue Description No public GitHub issue exists for this bug. **What happened?** Referenced-project staging copied Git-ignored paths, except for a fixed list of heavy directory names. A large repository therefore used much more time and disk space than the same repository in an anchor workspace. **Expected behavior** Referenced-project staging should exclude the same Git-ignored paths that the workspace staging path excludes. **Steps to reproduce** 1. Create a referenced project with a large Git-ignored directory. 2. Start a sandbox or SSH run that stages the referenced project. 3. Observe that the ignored directory enters the staged content. **Paperclip version or commit** Commit `9964b034bbff24e700c8eccf5a8b1fc3daa44bf2`. **Deployment mode** Built from source. ## What Changed - Resolve each referenced project's Git-ignored paths once before staging. - Carry the resolved paths as a required field on `SandboxAdditionalSource`. - Reuse the resolved paths in sandbox staging, SSH staging, and content-signature code. - Harden the read-only Git helper with a bounded process, a reduced environment, and disabled system and global configuration. - Fail closed on Git errors, timeouts, and invalid path relations. - Escape tar glob metacharacters in ignore-derived exclude entries. - Add and update unit tests for the resolver and its three consumers. ## Verification - `pnpm vitest run --config packages/adapter-utils/vitest.config.ts` passes 266 tests locally. - `pnpm exec tsc --noEmit -p packages/adapter-utils/tsconfig.json` passes locally. - CI must pass on this pull request. - Greptile must report 5/5 with no unresolved comments before merge. ## Risks - A Git error or timeout now prevents staging for the affected referenced project. - The resolver uses a bounded read-only Git process and fails closed by design. - The change stays inside `packages/adapter-utils` and does not change the database schema. ## Model Used Claude Sonnet 5 (Anthropic) assisted the implementation with code execution and tool use. The exact context window and reasoning mode are not recorded. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
9fc2f594ae
commit
6880213de5
12 files changed
+902
-49
No files matched your search
@@ -22,6 +22,7 @@ import {
|
||||
prepareAdapterExecutionTargetRuntime,
|
||||
readAdapterExecutionTarget,
|
||||
resolveAdapterExecutionTargetTimeout,
|
||||
resolveReferencedSourceIgnore,
|
||||
runAdapterExecutionTargetShellCommand,
|
||||
startAdapterExecutionTargetPaperclipBridge,
|
||||
startAdapterExecutionTargetProcessSessionBridge,
|
||||
@@ -31,6 +32,7 @@ import {
|
||||
type AdapterExecutionTargetTimeoutResolution,
|
||||
type AdapterManagedRuntimeAsset,
|
||||
type PreparedAdapterExecutionTargetRuntime,
|
||||
type ReferencedSourceIgnoreResolution,
|
||||
type SandboxAdditionalSource,
|
||||
} from "@paperclipai/adapter-utils/execution-target";
|
||||
import type { DuplexLossReason } from "../duplex-observability.js";
|
||||
@@ -519,9 +521,14 @@ export function finalizeLaunchEnvironment(
|
||||
}
|
||||
|
||||
// Directory names the staging path never ships for a referenced project (heavy
|
||||
// build/cache output and git history). The content signature skips them so it
|
||||
// reflects only the staged tree and never reads their bytes. Keep this set equal
|
||||
// to the staging excludes in the sandbox and remote runtimes.
|
||||
// build/cache output and git history), applied regardless of the project's
|
||||
// ignore resolution. The content signature skips them so it reflects only the
|
||||
// staged tree and never reads their bytes. Keep this set equal to the fixed
|
||||
// excludes the sandbox and SSH runtimes always apply. A project's OWN resolved
|
||||
// Git-ignored paths (see `resolveReferencedSourceIgnore`) are matched
|
||||
// separately, by relative path, inside `referencedSourceContentSignature` — that
|
||||
// is the real invariant now: the signature and both staging lanes must consume
|
||||
// the SAME one resolution per project, not just this fixed name list.
|
||||
const REFERENCED_SOURCE_SIGNATURE_SKIP_DIRS = new Set([
|
||||
"node_modules",
|
||||
"vendor",
|
||||
@@ -551,12 +558,30 @@ const REFERENCED_SOURCE_SIGNATURE_SKIP_DIRS = new Set([
|
||||
* re-checkout that restores the same size and timestamp. The byte hash busts on any
|
||||
* content change, so the fingerprint busts and the next launch stages the current
|
||||
* tree. The walk skips the heavy build, cache, and git directories the staging path
|
||||
* never ships, and records a symlink by its target text without following it. On a
|
||||
* read error the function returns a stable marker, so the fingerprint does not churn
|
||||
* while staging surfaces the real error. The walk runs only when the run carries
|
||||
* referenced projects (the multi-project sync path).
|
||||
* never ships, plus the project's own resolved Git-ignored paths, and records a
|
||||
* symlink by its target text without following it. On a read error the function
|
||||
* returns a stable marker, so the fingerprint does not churn while staging
|
||||
* surfaces the real error. The walk runs only when the run carries referenced
|
||||
* projects (the multi-project sync path).
|
||||
*
|
||||
* `ignoreResolution` is the ONE resolution `resolveReferencedSourceIgnore`
|
||||
* computed for this project — the same one the sandbox lane and the SSH lane
|
||||
* consume. A `failed` resolution skips the walk entirely and returns a stable
|
||||
* marker instead, because a failed project is not staged and its bytes are not
|
||||
* read anywhere.
|
||||
*/
|
||||
async function referencedSourceContentSignature(localPath: string): Promise<string> {
|
||||
export async function referencedSourceContentSignature(
|
||||
localPath: string,
|
||||
ignoreResolution: ReferencedSourceIgnoreResolution,
|
||||
): Promise<string> {
|
||||
if (ignoreResolution.kind === "failed") {
|
||||
return `unreadable:${ignoreResolution.reason}`;
|
||||
}
|
||||
const isIgnoredByGitResolution = (relativePath: string): boolean =>
|
||||
ignoreResolution.kind === "git" &&
|
||||
ignoreResolution.ignoredPaths.some(
|
||||
(entry) => relativePath === entry || relativePath.startsWith(`${entry}/`),
|
||||
);
|
||||
const hash = createHash("sha256");
|
||||
const walk = async (relative: string): Promise<void> => {
|
||||
const current = relative ? path.join(localPath, relative) : localPath;
|
||||
@@ -564,6 +589,9 @@ async function referencedSourceContentSignature(localPath: string): Promise<stri
|
||||
dirents.sort((left, right) => (left.name < right.name ? -1 : left.name > right.name ? 1 : 0));
|
||||
for (const dirent of dirents) {
|
||||
const next = relative ? path.posix.join(relative, dirent.name) : dirent.name;
|
||||
if (isIgnoredByGitResolution(next)) {
|
||||
continue;
|
||||
}
|
||||
if (dirent.isDirectory()) {
|
||||
if (REFERENCED_SOURCE_SIGNATURE_SKIP_DIRS.has(dirent.name)) {
|
||||
continue;
|
||||
@@ -1545,32 +1573,56 @@ async function buildRuntime(input: {
|
||||
const additionalSourceRecords = (
|
||||
Array.isArray(realizationContext.additional) ? realizationContext.additional : []
|
||||
).map((entry) => parseObject(entry));
|
||||
const additionalSources: SandboxAdditionalSource[] = additionalSourceRecords
|
||||
.map((entry) => ({ localPath: asString(entry.path, ""), projectId: asString(entry.projectId, "") }))
|
||||
const additionalSourceCandidates = additionalSourceRecords
|
||||
.map((entry) => ({
|
||||
localPath: asString(entry.path, ""),
|
||||
projectId: asString(entry.projectId, ""),
|
||||
projectWorkspaceId: asString(entry.projectWorkspaceId, ""),
|
||||
repoUrl: asString(entry.repoUrl, ""),
|
||||
repoRef: asString(entry.repoRef, ""),
|
||||
}))
|
||||
.filter((entry) => entry.localPath.length > 0 && entry.projectId.length > 0);
|
||||
// Resolve each referenced project's Git-ignored paths ONCE, here, before any
|
||||
// staging site runs. The sandbox lane, the SSH lane, and the content signature
|
||||
// below all consume this SAME resolution per project, so they can never apply
|
||||
// a different exclusion set to the same project. See
|
||||
// `resolveReferencedSourceIgnore` for the fail-closed rules.
|
||||
const additionalSourcesWithIgnore = await Promise.all(
|
||||
additionalSourceCandidates.map(async (entry) => ({
|
||||
...entry,
|
||||
ignoreResolution: await resolveReferencedSourceIgnore(entry.localPath),
|
||||
})),
|
||||
);
|
||||
const additionalSources: SandboxAdditionalSource[] = additionalSourcesWithIgnore.map((entry) => ({
|
||||
localPath: entry.localPath,
|
||||
projectId: entry.projectId,
|
||||
ignoreResolution: entry.ignoreResolution,
|
||||
}));
|
||||
// Stable identity of the referenced-project set for the session fingerprint.
|
||||
// The staged-runtime cache reuses already-staged referenced-project trees on a
|
||||
// compatible resume, so the fingerprint must change when the set OR a project's
|
||||
// pinned checkout changes. Without this, a resume reuses a stale staged tree.
|
||||
// Fold in each project's id, host path, workspace id, and pinned ref; sort by
|
||||
// projectId so the identity depends on the set, not the record order.
|
||||
const additionalSourcesIdentityBase = additionalSourceRecords
|
||||
const additionalSourcesIdentityBase = additionalSourcesWithIgnore
|
||||
.map((entry) => ({
|
||||
projectId: asString(entry.projectId, ""),
|
||||
localPath: asString(entry.path, ""),
|
||||
projectWorkspaceId: asString(entry.projectWorkspaceId, ""),
|
||||
repoUrl: asString(entry.repoUrl, ""),
|
||||
repoRef: asString(entry.repoRef, ""),
|
||||
projectId: entry.projectId,
|
||||
localPath: entry.localPath,
|
||||
projectWorkspaceId: entry.projectWorkspaceId,
|
||||
repoUrl: entry.repoUrl,
|
||||
repoRef: entry.repoRef,
|
||||
ignoreResolution: entry.ignoreResolution,
|
||||
}))
|
||||
.filter((entry) => entry.localPath.length > 0 && entry.projectId.length > 0)
|
||||
.sort((a, b) => (a.projectId < b.projectId ? -1 : a.projectId > b.projectId ? 1 : 0));
|
||||
// Metadata alone does not change on a content-only checkout change (same host
|
||||
// path and pinned ref, new file bytes). Fold in each tree's content signature so
|
||||
// a file add, remove, or edit busts the fingerprint and the resume re-stages.
|
||||
// The signature reads the same `ignoreResolution` the staging sites above use,
|
||||
// so it never disagrees with what was actually shipped.
|
||||
const additionalSourcesIdentity = await Promise.all(
|
||||
additionalSourcesIdentityBase.map(async (entry) => ({
|
||||
additionalSourcesIdentityBase.map(async ({ ignoreResolution, ...entry }) => ({
|
||||
...entry,
|
||||
contentSignature: await referencedSourceContentSignature(entry.localPath),
|
||||
contentSignature: await referencedSourceContentSignature(entry.localPath, ignoreResolution),
|
||||
})),
|
||||
);
|
||||
// Referenced-project workspace hints exposed to the agent through PAPERCLIP_WORKSPACES_JSON. The
|
||||
|
||||
Reference in new issue
Block a user