mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 14:16:39 +01:00
## Thinking Path > - Paperclip coordinates agent work and preserves its results. > - Remote execution copies completed workspace changes back to the host. > - An agent can fetch a published branch with shallow Git history. > - A full Git bundle omits the shallow boundary metadata and can fail host connectivity checks. > - This pull request carries the boundaries through the full-bundle fallback and imports through normal Git fetch. > - Recovery must also recognize completed native work on a reusable lease without treating it as ordinary executor reuse. > - Completed work can be saved without another provider turn, while existing host history remains connected. ## Linked Issues or Issue Description **What happened?** Sandbox finalization can fail with a missing parent commit after an agent checks out a separately fetched shallow branch. Re-exporting a full bundle repeats the failure because Git bundles do not encode shallow boundaries. **Expected behavior** Import the completed branch and saved files. Preserve existing host ancestry and concurrent host edits. Keep normal Git connectivity and workspace merge checks. **Steps to reproduce** 1. Prepare a sandbox from one host commit. 2. Fetch a different published branch with depth one and check it out. 3. Save a result file and restore the sandbox workspace. **Paperclip version or commit** Reproduced against master before this repair. The regression fails with the missing-parent error before the production change. Related public context: #15268 covers rewritten history integration. This repair handles shallow bundle transport before integration. Searches found no equivalent open repair. ## What Changed - Export shallow boundaries beside the full bundle when the delta import lacks history. - Unpack into a temporary shared bare repository. Keep parent connections already available on the host and import with Git's shallow fetch protocol. - Reject malformed boundary values and retain existing connectivity, branch, and concurrent-work merge guards. Keep boundary probes within 15 seconds without consuming the separate 60-second Git import budget. - Let source-bound preservation recover reusable leases only for the finalizer-selected, accepted successful native result and matching completion contract. Retain provider-stop, physical ownership and host-writer checks; ordinary reusable sources remain blocked. - Classify invalid Git metadata separately from omittable unsafe archives. Keep native finalization failed and preserve service-owned file recovery; carry the sender's dedicated rejection across the command boundary without transient retries. - Share the failure allowlist across bounded run/activity projections and cancelled-preservation selection. Migration `0416` rebuilds the matching partial index; task chat retains the non-retryable recovery notice. - Add real Git, filesystem sandbox and database recovery regressions. Update execution semantics and affected verification maps. ## Verification - Red: real filesystem sandbox restore failed with the same missing-parent/revision-walk error before the export repair. - Green: 240 focused Git, workspace restore, sandbox transport and native-finalization tests pass. Coverage includes both transfer paths, malformed/oversized metadata, a genuinely shallow host with its original boundary intact, and concurrent host work with both merge parents verified. - Red/green: a 14-second simulated probe left the final real Git fetch only 45,893 ms before the budget split. It now retains over 55 seconds after the probes; a 16-second probe exhausts its own budget before host import. - Red/green: an oversized read progress report was retried three times before its distinct metadata classification. It now aborts the first read before payload consumption, leaves the host tip unchanged and does not retry. - Red/green: the real metadata error was swallowed by best-effort export and native finalization incorrectly returned success. Both regressions now pass; all 240 focused Git/restore/native-finalization tests pass. A PostgreSQL/filesystem fault-injection test also verifies retained files, failed run, service-owned recovery barrier, no preservation receipt and no new wake. - Red/green: duplicate metadata produced 47 Git process starts before deduplication; the bounded-work regression now passes. Oversized byte and boundary inputs fail before Git probes. - Recovery red: both reusable lease policies exhausted recovery despite a selected accepted successful result. The assembled PostgreSQL/filesystem path now passes for retained and both reusable policies, including idempotent re-admission after exhausted automatic recovery, saved files, no agent wake, and once-only existing-result finalization. - Source verification controls reject ordinary reusable leases, rejected/failed/unselected results, another physical allocation owner, and an active host writer without changing recorded state. - The 78-test PostgreSQL/filesystem recovery/source suite passed before the metadata classification correction. Its new persisted metadata-failure regression also passes. Full workspace typecheck and build pass at final head `69f10daab2`. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Full local `pnpm test:run` attempts did not finish. They are not counted as passes. Early focused runs exceeded 5-second and 30-second fixture deadlines; the final serial suite passed all 176 tests with a 30-second per-test budget in 56.8 seconds. - Red/green: both bounded PostgreSQL projections dropped the new code in oversized results. All 19 source/projection cases now pass, plus an actual index-predicate test that excludes a metadata repair hold and still selects ordinary cancelled work. - Red/green: an oversized sender `.git/shallow` lost its error type before host validation. The real transport test now verifies one typed rejection, no shallow-payload read, and an unchanged host tip. All 465 command/sandbox/task-chat tests pass, and token/migration gates pass. - Full typecheck/build pass after these final contract corrections at `69f10daab2`; all 20 PostgreSQL source/projection/index contracts pass. All current-head CI checks passed; Greptile is 5/5 with no unresolved review threads. Merged #18230 supplies the CI-cache and Linux shell-fixture baseline repairs; their duplicate changes have been removed from this PR. The final Runner lane passed after one justified rerun of an unchanged authentication fixture timeout (all 11 fixture tests also pass locally). Previous CI failures and incomplete local runs remain recorded; live deployment and retained-run recovery remain separate verification. ## Risks - The fallback creates a temporary shared bare clone. Metadata is limited to 64 KiB and 256 unique boundaries, duplicate/parent probes are cached, and Git import has a 60-second command budget plus a separate 15-second boundary-probe budget. It runs only after a missing-history import failure. - Incorrect boundaries could hide real host ancestry. The importer retains available parent connections, and regressions cover that case. - The full-bundle fallback can transfer a bundle twice. This preserves the existing fast delta path; removing that extra transfer is outside this correctness repair. - Preservation eligibility expands only for completed native exports on reusable leases. It creates no provider turn and leaves ordinary reusable leases ineligible. Database/filesystem regressions cover result selection and competing physical/host ownership. - The migration rebuilds one partial index; no tables, columns or user records change. There are no agent execution admission, model, trust, recovery permission, workflow or lockfile changes. ## Model Used OpenAI Codex, GPT-6, with reasoning, code execution, and browser tools. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1217 lines
54 KiB
TypeScript
1217 lines
54 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { execFile } from "node:child_process";
|
|
import { promises as fs } from "node:fs";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { createWorkspaceManifest, workspacePaths, WorkspaceNulParser, type WorkspacePaths } from "./workspace-manifest.js";
|
|
import { runWorkspaceGitProcess } from "./workspace-git-stream.js";
|
|
import { preserveWorkspaceRestoreErrorDiagnostic, withWorkspaceRestoreGitCommand, type WorkspaceRestoreGitCommand } from "./workspace-restore-diagnostics.js";
|
|
|
|
export interface GitCommandResult {
|
|
stdout: string;
|
|
stderr: string;
|
|
}
|
|
|
|
export interface GitWorkspaceSnapshot {
|
|
headCommit: string;
|
|
branchName: string | null;
|
|
originUrl?: string | null;
|
|
overlayPaths: WorkspacePaths;
|
|
deletedPaths: WorkspacePaths;
|
|
ignoredPaths: WorkspacePaths;
|
|
/** Managed, editable repositories inside the task workspace. */
|
|
repositories?: Array<{ path: string; snapshot: GitWorkspaceSnapshot }>;
|
|
}
|
|
|
|
export const PROJECT_REPOSITORIES_DIR = ".paperclip-repositories";
|
|
|
|
export interface ExpensiveWorkspaceGitInput {
|
|
localDir: string;
|
|
args: readonly string[];
|
|
operation: string;
|
|
timeout: number;
|
|
maxBuffer: number;
|
|
onStdout?: (chunk: Buffer) => Promise<void> | void;
|
|
signal?: AbortSignal;
|
|
/**
|
|
* Optional environment override for the invocation. Absent for the anchor
|
|
* workspace's own full-tree walks (they inherit the process environment, a
|
|
* directory this process already controls). A referenced-project scan sets
|
|
* this to its hardened environment (see {@link buildHardenedGitEnv}), so a
|
|
* host executor that honors it still runs the read hardened even though it
|
|
* dispatches through the same seam as the anchor's reads.
|
|
*/
|
|
env?: NodeJS.ProcessEnv;
|
|
}
|
|
|
|
export type ExpensiveWorkspaceGitExecutor = (
|
|
input: ExpensiveWorkspaceGitInput,
|
|
) => Promise<GitCommandResult>;
|
|
|
|
let expensiveWorkspaceGitExecutor: ExpensiveWorkspaceGitExecutor | null = null;
|
|
|
|
/**
|
|
* The workspace Git scan scheduler's typed code for a saturated queue
|
|
* (`server/src/services/workspace-git-operation-scheduler.ts`,
|
|
* `WORKSPACE_GIT_SCAN_ERROR_CODES.saturated`). Declared again here because
|
|
* `adapter-utils` cannot import from `server` (the reverse direction is
|
|
* allowed, not this one); `server` carries a test that asserts the two
|
|
* literals stay equal. `resolveReferencedSourceIgnore` in
|
|
* `sandbox-managed-runtime.ts` reads this code off a caught error's `code`
|
|
* property, never off its message text, to retry only a saturated queue and
|
|
* fail closed on every other Git scan error.
|
|
*/
|
|
export const WORKSPACE_GIT_SCAN_SATURATED_CODE = "workspace_git_scan_saturated";
|
|
|
|
/**
|
|
* Lets a host process apply its process-wide admission policy to the adapter
|
|
* package's full-tree Git walks. Standalone adapter-utils consumers retain the
|
|
* streaming process fallback for filename scans.
|
|
*/
|
|
export function setExpensiveWorkspaceGitExecutor(executor: ExpensiveWorkspaceGitExecutor | null): void {
|
|
expensiveWorkspaceGitExecutor = executor;
|
|
}
|
|
|
|
export const GIT_ARCHIVE_EXCLUDES = [".git", ".git/*"] as const;
|
|
|
|
/**
|
|
* Identity flags for commits the sync machinery itself creates (the merge
|
|
* commits that reconcile concurrent histories). Execution hosts are often
|
|
* containers with no git config and no resolvable hostname, so git cannot
|
|
* auto-detect an identity there and `commit-tree` hard-fails with "Author
|
|
* identity unknown" — which fails the whole run at finalize. Passing the
|
|
* identity per invocation keeps every deployment working without host
|
|
* configuration; `GIT_AUTHOR_*` / `GIT_COMMITTER_*` environment variables
|
|
* still take precedence over `-c` when an operator sets them.
|
|
*/
|
|
export const GIT_SYNC_COMMIT_IDENTITY_ARGS = [
|
|
"-c",
|
|
"user.name=Paperclip",
|
|
"-c",
|
|
"user.email=noreply@paperclip.ing",
|
|
] as const;
|
|
|
|
function shellQuote(value: string) {
|
|
return `'${value.replace(/'/g, `'\"'\"'`)}'`;
|
|
}
|
|
|
|
/** Refresh restored stat metadata without staging dirty working-tree content. */
|
|
export function buildRefreshWorkspaceGitIndexCommand(directory: string): string {
|
|
return `if git -C ${shellQuote(directory)} -c core.fsmonitor=false update-index -q --refresh >/dev/null; then :; `
|
|
+ 'else index_status=$?; if [ "$index_status" -ne 1 ]; then exit "$index_status"; fi; fi';
|
|
}
|
|
|
|
export async function runLocalGit(
|
|
localDir: string,
|
|
args: string[],
|
|
options: {
|
|
timeout?: number;
|
|
maxBuffer?: number;
|
|
env?: NodeJS.ProcessEnv;
|
|
} = {},
|
|
): Promise<GitCommandResult> {
|
|
return await new Promise<GitCommandResult>((resolve, reject) => {
|
|
execFile(
|
|
"git",
|
|
["-C", localDir, ...args],
|
|
{
|
|
timeout: options.timeout ?? 15_000,
|
|
maxBuffer: options.maxBuffer ?? 1024 * 128,
|
|
env: options.env ?? process.env,
|
|
},
|
|
(error, stdout, stderr) => {
|
|
if (error) {
|
|
reject(Object.assign(error, { stdout: stdout ?? "", stderr: stderr ?? "" }));
|
|
return;
|
|
}
|
|
resolve({
|
|
stdout: stdout ?? "",
|
|
stderr: stderr ?? "",
|
|
});
|
|
},
|
|
);
|
|
});
|
|
}
|
|
|
|
async function runExpensiveWorkspaceGit(
|
|
localDir: string,
|
|
args: string[],
|
|
operation: string,
|
|
options: { timeout: number; maxBuffer: number; env?: NodeJS.ProcessEnv; onStdout?: ExpensiveWorkspaceGitInput["onStdout"]; signal?: AbortSignal },
|
|
): Promise<GitCommandResult> {
|
|
if (expensiveWorkspaceGitExecutor) {
|
|
return await expensiveWorkspaceGitExecutor({
|
|
localDir,
|
|
args,
|
|
operation,
|
|
timeout: options.timeout,
|
|
maxBuffer: options.maxBuffer,
|
|
env: options.env,
|
|
onStdout: options.onStdout,
|
|
signal: options.signal,
|
|
});
|
|
}
|
|
if (options.onStdout) return runWorkspaceGitProcess({
|
|
cwd: localDir, args, timeoutMs: options.timeout, maxStdoutBytes: options.maxBuffer,
|
|
maxStderrBytes: options.maxBuffer, onStdout: options.onStdout, signal: options.signal, env: options.env,
|
|
});
|
|
return await runLocalGit(localDir, args, options);
|
|
}
|
|
|
|
const ownedSnapshots = new WeakMap<GitWorkspaceSnapshot, string>();
|
|
|
|
export async function disposeGitWorkspaceSnapshot(snapshot: GitWorkspaceSnapshot | null): Promise<void> {
|
|
if (!snapshot) return;
|
|
const ownedDirectory = ownedSnapshots.get(snapshot);
|
|
if (!ownedDirectory) return;
|
|
ownedSnapshots.delete(snapshot);
|
|
for (const repository of snapshot.repositories ?? []) await disposeGitWorkspaceSnapshot(repository.snapshot);
|
|
await fs.rm(ownedDirectory, { recursive: true, force: true });
|
|
}
|
|
|
|
/** Snapshot deadlines include disk backpressure. Operators can allow up to 24h. */
|
|
export function workspaceSnapshotTimeoutMs(): number {
|
|
const configured = Number(process.env.PAPERCLIP_WORKSPACE_GIT_SNAPSHOT_TIMEOUT_MS);
|
|
return Number.isFinite(configured) && configured >= 1000 ? Math.min(configured, 86_400_000) : 30 * 60_000;
|
|
}
|
|
|
|
export async function readManagedWorkspaceRepositories(localDir: string, options: { signal?: AbortSignal } = {}): Promise<NonNullable<GitWorkspaceSnapshot["repositories"]>> {
|
|
const repositories: NonNullable<GitWorkspaceSnapshot["repositories"]> = [];
|
|
try {
|
|
|
|
const root = path.join(localDir, PROJECT_REPOSITORIES_DIR);
|
|
const rootStat = await fs.lstat(root).catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code === "ENOENT") return null;
|
|
throw error;
|
|
});
|
|
if (rootStat) {
|
|
if (!rootStat.isDirectory() || rootStat.isSymbolicLink()) throw new Error("Invalid project repositories directory");
|
|
for await (const entry of await fs.opendir(root)) {
|
|
if (!entry.isDirectory() || !/^[a-zA-Z0-9_-]+$/.test(entry.name)) throw new Error("Invalid project repository directory");
|
|
const relative = `${PROJECT_REPOSITORIES_DIR}/${entry.name}`;
|
|
const snapshot = await readGitWorkspaceSnapshot(path.join(localDir, relative), false, options);
|
|
if (!snapshot) throw new Error(`Project repository is not a Git checkout: ${relative}`);
|
|
repositories.push({ path: relative, snapshot });
|
|
}
|
|
}
|
|
|
|
return repositories;
|
|
} catch (error) {
|
|
for (const repo of repositories) await disposeGitWorkspaceSnapshot(repo.snapshot);
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
export async function readGitWorkspaceSnapshot(localDir: string, includeRepositories = true, options: { signal?: AbortSignal } = {}): Promise<GitWorkspaceSnapshot | null> {
|
|
const repositories: NonNullable<GitWorkspaceSnapshot["repositories"]> = [];
|
|
// Only repository discovery may report an ordinary directory. A failed
|
|
// snapshot of a confirmed repository must never fall back to directory sync.
|
|
let insideWorkTree: GitCommandResult;
|
|
try {
|
|
insideWorkTree = await runLocalGit(localDir, ["rev-parse", "--is-inside-work-tree"], {
|
|
timeout: 10_000,
|
|
maxBuffer: 16 * 1024,
|
|
});
|
|
} catch (error) {
|
|
if (repositories.length === 0 && isNotAGitRepositoryError(error)) return null;
|
|
throw error;
|
|
}
|
|
if (insideWorkTree.stdout.trim() !== "true") {
|
|
return null;
|
|
}
|
|
|
|
const toplevelResult = await runLocalGit(localDir, ["rev-parse", "--show-toplevel"], {
|
|
timeout: 10_000,
|
|
maxBuffer: 16 * 1024,
|
|
});
|
|
// Git discovers a parent repository from a nested project directory, but
|
|
// that directory is not a fetch source. Keep the selected workspace
|
|
// boundary: subfolders use directory sync instead of importing the parent.
|
|
const [workspacePath, repositoryPath] = await Promise.all([
|
|
fs.realpath(localDir),
|
|
fs.realpath(toplevelResult.stdout.trim()),
|
|
]);
|
|
if (workspacePath !== repositoryPath) return null;
|
|
|
|
const writer = await createWorkspaceManifest();
|
|
const controller = new AbortController();
|
|
const abort = () => controller.abort();
|
|
options.signal?.addEventListener("abort", abort, { once: true });
|
|
if (options.signal?.aborted) abort();
|
|
try {
|
|
if (includeRepositories) repositories.push(...await readManagedWorkspaceRepositories(localDir, options));
|
|
|
|
const scan = async (args: string[], operation: string, category: string) => {
|
|
const parser = new WorkspaceNulParser((record) => {
|
|
const relative = category === "ignored" ? record.replace(/\/+$/, "") : record;
|
|
if (category === "ignored" && repositories.length && relative === PROJECT_REPOSITORIES_DIR) return;
|
|
// Managed children have their own exact file selections. Never stage a
|
|
// parent Git directory record that could admit files created later.
|
|
if (repositories.some((repo) => relative === repo.path || relative.startsWith(`${repo.path}/`))) return;
|
|
writer.add(category, relative);
|
|
});
|
|
const result = await runExpensiveWorkspaceGit(localDir, args, operation, {
|
|
timeout: workspaceSnapshotTimeoutMs(), maxBuffer: 64 * 1024,
|
|
signal: controller.signal,
|
|
onStdout: (chunk) => writer.batch(() => parser.write(chunk)),
|
|
});
|
|
if (result.stdout) throw new Error("Workspace Git executor did not stream stdout");
|
|
try { parser.finish(); }
|
|
catch (error) { throw Object.assign(error as Error, { code: "workspace_git_scan_failed" }); }
|
|
};
|
|
// Cancellation is followed by an all-settled barrier BEFORE storage cleanup.
|
|
const tasks = [
|
|
runLocalGit(localDir, ["rev-parse", "HEAD"], { timeout: 10_000, maxBuffer: 16 * 1024 }),
|
|
runLocalGit(localDir, ["rev-parse", "--abbrev-ref", "HEAD"], { timeout: 10_000, maxBuffer: 16 * 1024 }),
|
|
scan(["diff", "--name-only", "-z", "--diff-filter=ACMRTUXB", "HEAD", "--"], "adapter_sync.overlay_diff", "overlay"),
|
|
scan(["ls-files", "--others", "--exclude-standard", "-z"], "adapter_sync.untracked_files", "overlay"),
|
|
scan(["diff", "--name-only", "-z", "--diff-filter=D", "HEAD", "--"], "adapter_sync.deleted_files", "deleted"),
|
|
scan(["ls-files", "--others", "--ignored", "--exclude-standard", "--directory", "-z"], "adapter_sync.ignored_files", "ignored"),
|
|
];
|
|
let firstError: unknown;
|
|
const settled = await Promise.allSettled(tasks.map((task) => task.catch((error) => {
|
|
if (firstError === undefined) firstError = error;
|
|
controller.abort();
|
|
throw error;
|
|
})));
|
|
if (firstError !== undefined) throw firstError;
|
|
if (controller.signal.aborted) throw Object.assign(new Error("Workspace snapshot cancelled"), { code: "workspace_git_scan_cancelled" });
|
|
for (const repo of repositories) {
|
|
for (const [category, paths] of [["overlay", repo.snapshot.overlayPaths], ["deleted", repo.snapshot.deletedPaths], ["ignored", repo.snapshot.ignoredPaths]] as const) {
|
|
for (const relative of workspacePaths(paths)) writer.add(category, `${repo.path}/${relative}`);
|
|
}
|
|
}
|
|
const head = (settled[0] as PromiseFulfilledResult<GitCommandResult>).value.stdout.trim();
|
|
const branch = (settled[1] as PromiseFulfilledResult<GitCommandResult>).value.stdout.trim();
|
|
const snapshot: GitWorkspaceSnapshot = {
|
|
headCommit: head, branchName: branch && branch !== "HEAD" ? branch : null,
|
|
originUrl: await readSanitizedOriginRemoteUrl(localDir),
|
|
overlayPaths: writer.paths("overlay"), deletedPaths: writer.paths("deleted"), ignoredPaths: writer.paths("ignored"),
|
|
...(repositories.length ? { repositories } : {}),
|
|
};
|
|
writer.close();
|
|
ownedSnapshots.set(snapshot, path.dirname(writer.filePath));
|
|
return snapshot;
|
|
} catch (error) {
|
|
writer.close(false);
|
|
for (const repo of repositories) await disposeGitWorkspaceSnapshot(repo.snapshot);
|
|
await fs.rm(path.dirname(writer.filePath), { recursive: true, force: true });
|
|
throw error;
|
|
} finally { options.signal?.removeEventListener("abort", abort); }
|
|
}
|
|
|
|
/** The `git ls-files --others --ignored` output for one directory, read by {@link readReferencedSourceGitIgnoredPaths}. */
|
|
export interface ReferencedSourceGitIgnoreScan {
|
|
/** The absolute repository top level `git rev-parse --show-toplevel` reports. */
|
|
toplevel: string;
|
|
/** Ignored paths, relative to `toplevel`, trailing slashes stripped, sorted. */
|
|
ignoredPaths: string[];
|
|
}
|
|
|
|
/**
|
|
* Build the environment for a hardened, read-only Git invocation against a
|
|
* directory this process does not control (a referenced project, not the
|
|
* anchor workspace). Two protections apply:
|
|
*
|
|
* - Drop every inherited `GIT_*` variable, so an already-set override in this
|
|
* process's own environment cannot change how the read-only command runs.
|
|
* - Point the global config file at `/dev/null` (in addition to the
|
|
* command-line `GIT_CONFIG_NOSYSTEM=1` the caller sets), so neither this
|
|
* host's global nor system Git configuration can add a setting the
|
|
* read-only command was not built to expect.
|
|
*
|
|
* This does not defend against the directory's OWN repository-local
|
|
* configuration; the command-line `-c core.fsmonitor=false` override in
|
|
* {@link runHardenedReadOnlyGit} does that instead, because command-line
|
|
* config always wins over repository-local config.
|
|
*/
|
|
function buildHardenedGitEnv(): NodeJS.ProcessEnv {
|
|
const env: NodeJS.ProcessEnv = {};
|
|
for (const [key, value] of Object.entries(process.env)) {
|
|
if (key.startsWith("GIT_") || value === undefined) continue;
|
|
env[key] = value;
|
|
}
|
|
env.GIT_CONFIG_NOSYSTEM = "1";
|
|
env.GIT_CONFIG_GLOBAL = "/dev/null";
|
|
return env;
|
|
}
|
|
|
|
/**
|
|
* Run a read-only Git command against a directory this process does not
|
|
* control, hardened against a hostile repository-local configuration, and
|
|
* dispatched through {@link runExpensiveWorkspaceGit} — the SAME process-wide
|
|
* admission seam the anchor workspace's expensive full-tree reads use. A host
|
|
* process that registers a bounded scheduler there (see
|
|
* `setExpensiveWorkspaceGitExecutor`) governs referenced-project scans too, so
|
|
* a run with many referenced projects cannot spawn one unbounded Git process
|
|
* per project; each request queues behind the same concurrency limit.
|
|
*
|
|
* Every call still carries `--no-optional-locks` (never blocks on, or is
|
|
* blocked by, a concurrent Git process in the directory) and
|
|
* `-c core.fsmonitor=false` (neutralizes a repository-local `core.fsmonitor`
|
|
* setting that would otherwise run an arbitrary configured program on this
|
|
* read). See {@link buildHardenedGitEnv} for the paired environment hardening,
|
|
* carried through the executor's optional `env` field so hardening survives
|
|
* the hop through a host-registered scheduler.
|
|
*/
|
|
async function runHardenedReadOnlyGit(
|
|
localDir: string,
|
|
args: string[],
|
|
operation: string,
|
|
options: { timeout: number; maxBuffer: number },
|
|
): Promise<GitCommandResult> {
|
|
return await runExpensiveWorkspaceGit(
|
|
localDir,
|
|
["-c", "core.fsmonitor=false", "--no-optional-locks", ...args],
|
|
operation,
|
|
{ timeout: options.timeout, maxBuffer: options.maxBuffer, env: buildHardenedGitEnv() },
|
|
);
|
|
}
|
|
|
|
/**
|
|
* True when a failed `git` invocation failed specifically because `localDir`
|
|
* is not inside a Git work tree — Git's own "not a git repository" fatal
|
|
* error. Distinguishes the expected non-Git case from a real failure (a
|
|
* timeout, a permissions error, a corrupt repository), which must still
|
|
* surface as a failure and never look like "no Git tree here".
|
|
*/
|
|
function isNotAGitRepositoryError(error: unknown): boolean {
|
|
// The host scheduler keeps bounded subprocess diagnostics under details.
|
|
// Only a completed Git exit may establish that no repository exists.
|
|
if (error && typeof error === "object" && "code" in error &&
|
|
typeof error.code === "string" && error.code.startsWith("workspace_git_scan_")) {
|
|
const details = "details" in error && error.details && typeof error.details === "object"
|
|
? error.details as Record<string, unknown> : {};
|
|
return error.code === "workspace_git_scan_failed" && details.exitCode === 128 && details.signal === null &&
|
|
typeof details.stderr === "string" && /not a git repository/i.test(details.stderr);
|
|
}
|
|
const stderr = error && typeof error === "object" && "stderr" in error ? String((error as { stderr: unknown }).stderr) : "";
|
|
const message = error instanceof Error ? error.message : String(error);
|
|
return /not a git repository/i.test(stderr) || /not a git repository/i.test(message);
|
|
}
|
|
|
|
/** Bound on the number of parsed ignored entries `readReferencedSourceGitIgnoredPaths` accepts before it fails closed. */
|
|
export const REFERENCED_SOURCE_IGNORE_MAX_ENTRY_COUNT = 10_000;
|
|
|
|
/** Bound on the summed UTF-8 byte length of the resolved ignored-path strings `readReferencedSourceGitIgnoredPaths` accepts before it fails closed. */
|
|
export const REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES = 2 * 1024 * 1024;
|
|
|
|
/**
|
|
* Bound on the raw `git ls-files --others --ignored` output
|
|
* `readReferencedSourceGitIgnoredPaths` lets Node buffer, kept proportionate
|
|
* to {@link REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES} instead of the far
|
|
* larger allowance the anchor workspace's general-purpose full-tree reads
|
|
* use. The command reports only ignored entries (see the invocation below),
|
|
* so this raw allowance is not exposed to an unrelated tracked-change or
|
|
* ordinary-untracked record count — a repository with a huge diff or a huge
|
|
* untracked set never grows this command's output. The parser below still
|
|
* enforces the real entry-count and byte bounds while it reads each record,
|
|
* so this value only needs headroom for the NUL delimiter and the trailing
|
|
* slash on every entry, not room for an oversized ignored-path list to land
|
|
* in memory in the first place.
|
|
*/
|
|
const REFERENCED_SOURCE_IGNORE_MAX_RAW_BUFFER = REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES * 2;
|
|
|
|
/**
|
|
* Thrown by {@link readReferencedSourceGitIgnoredPaths} when the parsed
|
|
* ignored-path list breaches {@link REFERENCED_SOURCE_IGNORE_MAX_ENTRY_COUNT}
|
|
* or {@link REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES}, so the caller can
|
|
* classify the failure as a bound breach instead of a plain Git read error.
|
|
* The message never leaves this package: `resolveReferencedSourceIgnore`
|
|
* replaces it with a fixed category before the failure reaches any consumer.
|
|
*/
|
|
export class ReferencedSourceIgnoreScanLimitExceededError extends Error {
|
|
constructor(message: string) {
|
|
super(message);
|
|
this.name = "ReferencedSourceIgnoreScanLimitExceededError";
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Read the Git-ignored paths of a referenced-project host directory, for the
|
|
* staging path to exclude them (see `resolveReferencedSourceIgnore` in
|
|
* `sandbox-managed-runtime.ts`). Every command runs through
|
|
* {@link runHardenedReadOnlyGit}, because the directory is a host checkout the
|
|
* staging code does not control, unlike the anchor workspace.
|
|
*
|
|
* Returns `null` when `localDir` is not a Git work tree — the caller keeps
|
|
* today's fixed excludes for that case. Throws on any other Git error, a
|
|
* timeout, malformed output, or a bound breach (see
|
|
* {@link ReferencedSourceIgnoreScanLimitExceededError}), so the caller can
|
|
* fail closed and skip staging that one project instead of shipping it
|
|
* unfiltered.
|
|
*/
|
|
export async function readReferencedSourceGitIgnoredPaths(
|
|
localDir: string,
|
|
): Promise<ReferencedSourceGitIgnoreScan | null> {
|
|
let toplevel: string;
|
|
try {
|
|
const toplevelResult = await runHardenedReadOnlyGit(
|
|
localDir,
|
|
["rev-parse", "--show-toplevel"],
|
|
"referenced_source.toplevel",
|
|
{ timeout: 15_000, maxBuffer: 64 * 1024 },
|
|
);
|
|
toplevel = toplevelResult.stdout.trim();
|
|
} catch (error) {
|
|
if (isNotAGitRepositoryError(error)) {
|
|
return null;
|
|
}
|
|
throw error;
|
|
}
|
|
if (!toplevel) {
|
|
throw new Error(`git rev-parse --show-toplevel returned an empty path for ${localDir}`);
|
|
}
|
|
|
|
// `ls-files --others --ignored --exclude-standard` reports only ignored
|
|
// entries — unlike `git status --ignored`, it never also reports a tracked
|
|
// change or an ordinary untracked file. A repository with a huge diff or a
|
|
// huge untracked set (unrelated to what is ignored) cannot inflate this
|
|
// command's raw output, so the raw buffer bound below only ever has to
|
|
// cover the declared ignored-set limits, not an unbounded amount of
|
|
// unrelated status noise ahead of them.
|
|
// `--directory` collapses an entirely ignored directory into one entry with
|
|
// a trailing slash, matching `git status --ignored`'s traditional mode.
|
|
// `--full-name` reports paths relative to the repository toplevel, so this
|
|
// still matches the toplevel-relative shape `resolveReferencedSourceIgnore`
|
|
// re-relativizes against, regardless of `localDir`'s position under it.
|
|
const ignoredResult = await runHardenedReadOnlyGit(
|
|
localDir,
|
|
["ls-files", "--others", "--ignored", "--exclude-standard", "--directory", "--full-name", "-z"],
|
|
"referenced_source.ignored_files",
|
|
{ timeout: 60_000, maxBuffer: REFERENCED_SOURCE_IGNORE_MAX_RAW_BUFFER },
|
|
);
|
|
|
|
// Read one NUL-delimited record at a time and enforce both bounds while the
|
|
// ignored-entry list accumulates, instead of splitting and mapping the
|
|
// whole response into a list first and only then checking its size. A
|
|
// pathologically large ignore set (a huge repository, or one crafted to
|
|
// hold many ignored entries) must fail closed the moment it breaches a
|
|
// bound, without this scan first retaining and transforming the full
|
|
// oversized response.
|
|
//
|
|
// Do not trim each entry: `-z` already delimits entries with a NUL byte, so
|
|
// a leading or trailing space in an entry is part of the path itself, not
|
|
// padding to remove. A length check finds the one genuinely empty record
|
|
// `-z` appends after the last NUL, without eating a real path's own
|
|
// leading or trailing whitespace.
|
|
const rawIgnored = ignoredResult.stdout;
|
|
const parsedIgnoredEntries: string[] = [];
|
|
let totalIgnoredBytes = 0;
|
|
let recordStart = 0;
|
|
while (recordStart < rawIgnored.length) {
|
|
const nulIndex = rawIgnored.indexOf("\0", recordStart);
|
|
const recordEnd = nulIndex === -1 ? rawIgnored.length : nulIndex;
|
|
const record = rawIgnored.slice(recordStart, recordEnd);
|
|
recordStart = nulIndex === -1 ? rawIgnored.length : nulIndex + 1;
|
|
|
|
const entry = record.replace(/\/+$/, "");
|
|
if (entry.length === 0) {
|
|
continue;
|
|
}
|
|
|
|
if (parsedIgnoredEntries.length + 1 > REFERENCED_SOURCE_IGNORE_MAX_ENTRY_COUNT) {
|
|
throw new ReferencedSourceIgnoreScanLimitExceededError(
|
|
`referenced project ignore scan found more than ${REFERENCED_SOURCE_IGNORE_MAX_ENTRY_COUNT} ignored entries`,
|
|
);
|
|
}
|
|
totalIgnoredBytes += Buffer.byteLength(entry, "utf8");
|
|
if (totalIgnoredBytes > REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES) {
|
|
throw new ReferencedSourceIgnoreScanLimitExceededError(
|
|
`referenced project ignore scan exceeded ${REFERENCED_SOURCE_IGNORE_MAX_TOTAL_BYTES} UTF-8 bytes of ignored paths`,
|
|
);
|
|
}
|
|
parsedIgnoredEntries.push(entry);
|
|
}
|
|
|
|
// The list is bounded by both checks above, so sorting and re-relativizing
|
|
// it here never costs more than the accepted bounds allow.
|
|
const ignoredPaths = parsedIgnoredEntries.sort((left, right) => left.localeCompare(right));
|
|
|
|
return { toplevel, ignoredPaths };
|
|
}
|
|
|
|
// scp-like ssh remote (`user@host:path`). The syntax has no password slot, so
|
|
// it cannot embed a secret. Conservative shape: exactly one `@`, no colon in
|
|
// the user segment (a colon there could smuggle credential-looking material),
|
|
// no scheme separator (a `://` form parses as a URL and never reaches this).
|
|
const SCP_LIKE_REMOTE_PATTERN = /^[^@:/\s]+@[^@:/\s]+:\S+$/;
|
|
|
|
/**
|
|
* Reduce a git remote URL to a credential-free form before it is copied into a
|
|
* transported workspace, or null when the URL must not be carried at all.
|
|
* Allowlist, fail closed: only shapes whose credential surface is fully known
|
|
* are kept — http(s) with userinfo/query/fragment stripped (tokens ride in any
|
|
* of those), ssh/git schemes with password/query/fragment stripped, and
|
|
* scp-like `user@host:path` (no password slot exists in that syntax). Every
|
|
* other form — filesystem paths, unknown schemes, unparseable strings — is
|
|
* dropped rather than risk persisting an embedded secret in the execution
|
|
* host's git config.
|
|
*/
|
|
export function sanitizeGitRemoteUrl(url: string): string | null {
|
|
const trimmed = url.trim();
|
|
if (!trimmed) {
|
|
return null;
|
|
}
|
|
try {
|
|
const parsed = new URL(trimmed);
|
|
if (parsed.protocol === "http:" || parsed.protocol === "https:") {
|
|
parsed.username = "";
|
|
parsed.password = "";
|
|
parsed.search = "";
|
|
parsed.hash = "";
|
|
return parsed.toString();
|
|
}
|
|
if (parsed.protocol === "ssh:" || parsed.protocol === "git:" || parsed.protocol === "git+ssh:") {
|
|
// The username (conventionally `git`) is addressing, not a secret; a
|
|
// password or query string can be, so those are stripped.
|
|
parsed.password = "";
|
|
parsed.search = "";
|
|
parsed.hash = "";
|
|
return parsed.toString();
|
|
}
|
|
return null;
|
|
} catch {
|
|
return SCP_LIKE_REMOTE_PATTERN.test(trimmed) ? trimmed : null;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* The workspace's `origin` remote URL with credentials scrubbed, or null when
|
|
* the workspace has no `origin` remote (or is not a git repository).
|
|
*/
|
|
export async function readSanitizedOriginRemoteUrl(localDir: string): Promise<string | null> {
|
|
try {
|
|
const result = await runLocalGit(localDir, ["remote", "get-url", "origin"], {
|
|
timeout: 10_000,
|
|
maxBuffer: 16 * 1024,
|
|
});
|
|
return sanitizeGitRemoteUrl(result.stdout.trim());
|
|
} catch {
|
|
return null;
|
|
}
|
|
}
|
|
|
|
export async function copyCloneTree(source: string, target: string): Promise<void> {
|
|
await fs.mkdir(target, { recursive: true });
|
|
for await (const entry of await fs.opendir(source)) {
|
|
const from = path.join(source, entry.name);
|
|
const to = path.join(target, entry.name);
|
|
if (entry.isDirectory()) await copyCloneTree(from, to);
|
|
else if (entry.isSymbolicLink()) await fs.symlink(await fs.readlink(from), to);
|
|
else { await fs.copyFile(from, to); await fs.chmod(to, (await fs.stat(from)).mode); }
|
|
}
|
|
}
|
|
|
|
export async function withShallowGitWorkspaceClone<T>(
|
|
input: {
|
|
localDir: string;
|
|
snapshot: GitWorkspaceSnapshot;
|
|
},
|
|
fn: (cloneDir: string) => Promise<T>,
|
|
): Promise<T> {
|
|
const cloneDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-workspace-"));
|
|
const tempRef = `refs/paperclip/git-sync/import/${randomUUID()}`;
|
|
try {
|
|
const originUrl = input.snapshot.originUrl !== undefined ? input.snapshot.originUrl : await readSanitizedOriginRemoteUrl(input.localDir);
|
|
await runLocalGit(input.localDir, ["update-ref", tempRef, input.snapshot.headCommit], {
|
|
timeout: 10_000,
|
|
maxBuffer: 16 * 1024,
|
|
});
|
|
await runLocalGit(cloneDir, ["init"], {
|
|
timeout: 10_000,
|
|
maxBuffer: 64 * 1024,
|
|
});
|
|
if (originUrl) {
|
|
// The clone is what lands in the sandbox. Without `origin`, the branch
|
|
// there reads as an unpublishable root snapshot even though its head is a
|
|
// commit the upstream remote already holds — so fetch (to reconnect
|
|
// ancestry) and push (to publish the branch; the shallow boundary commit
|
|
// is already on the remote, so the pack closes) are both mechanically
|
|
// possible once the remote is carried over. Best-effort: a failure to
|
|
// record the remote must not fail the transport.
|
|
await runLocalGit(cloneDir, ["remote", "add", "origin", originUrl], {
|
|
timeout: 10_000,
|
|
maxBuffer: 16 * 1024,
|
|
}).catch(() => undefined);
|
|
}
|
|
await runLocalGit(cloneDir, ["fetch", "--depth=1", input.localDir, tempRef], {
|
|
timeout: 60_000,
|
|
maxBuffer: 1024 * 1024,
|
|
});
|
|
await runLocalGit(
|
|
cloneDir,
|
|
input.snapshot.branchName
|
|
? ["checkout", "--force", "-B", input.snapshot.branchName, "FETCH_HEAD"]
|
|
: ["checkout", "--force", "--detach", "FETCH_HEAD"],
|
|
{
|
|
timeout: 60_000,
|
|
maxBuffer: 1024 * 1024,
|
|
},
|
|
);
|
|
await runLocalGit(cloneDir, ["reset", "--hard", input.snapshot.headCommit], {
|
|
timeout: 60_000,
|
|
maxBuffer: 1024 * 1024,
|
|
});
|
|
for (const repository of input.snapshot.repositories ?? []) {
|
|
await withShallowGitWorkspaceClone({
|
|
localDir: path.join(input.localDir, repository.path),
|
|
snapshot: repository.snapshot,
|
|
}, async (nestedClone) => {
|
|
// Preserve repository-relative links. fs.cp otherwise rewrites them to
|
|
// absolute paths into nestedClone, which is deleted after this callback
|
|
// and is outside the workspace when the sandbox restores its files.
|
|
await copyCloneTree(nestedClone, path.join(cloneDir, repository.path));
|
|
});
|
|
}
|
|
if (input.snapshot.repositories?.length) {
|
|
await fs.appendFile(path.join(cloneDir, ".git/info/exclude"), `\n/${PROJECT_REPOSITORIES_DIR}/\n`);
|
|
}
|
|
return await fn(cloneDir);
|
|
} finally {
|
|
await runLocalGit(input.localDir, ["update-ref", "-d", tempRef], {
|
|
timeout: 10_000,
|
|
maxBuffer: 16 * 1024,
|
|
}).catch(() => undefined);
|
|
await fs.rm(cloneDir, { recursive: true, force: true }).catch(() => undefined);
|
|
}
|
|
}
|
|
|
|
export function createImportedGitRef(scope = "remote"): string {
|
|
return `refs/paperclip/git-sync/imported/${scope}/${randomUUID()}`;
|
|
}
|
|
|
|
export function createRemoteGitExportRef(scope = "remote"): string {
|
|
return `refs/paperclip/git-sync/export/${scope}/${randomUUID()}`;
|
|
}
|
|
|
|
export async function deleteLocalGitRef(input: {
|
|
localDir: string;
|
|
ref: string;
|
|
}): Promise<void> {
|
|
await runLocalGit(input.localDir, ["update-ref", "-d", input.ref], {
|
|
timeout: 10_000,
|
|
maxBuffer: 16 * 1024,
|
|
}).catch(() => undefined);
|
|
}
|
|
|
|
export const GIT_SHALLOW_METADATA_EXIT_CODE = 47;
|
|
export const GIT_SHALLOW_METADATA_ERROR_CODE = "WORKSPACE_RESTORE_INVALID_GIT_METADATA";
|
|
|
|
/** Invalid restore metadata is an integrity rejection, not a dropped transfer. */
|
|
export class GitShallowMetadataError extends Error {
|
|
readonly code = GIT_SHALLOW_METADATA_ERROR_CODE;
|
|
}
|
|
|
|
export const GIT_SHALLOW_MAX_BYTES = 64 * 1024;
|
|
export const GIT_SHALLOW_MAX_BOUNDARIES = 256;
|
|
|
|
export async function fetchGitBundleIntoLocalRef(input: {
|
|
localDir: string;
|
|
bundlePath: string;
|
|
exportRef: string;
|
|
importedRef: string;
|
|
baseSha: string;
|
|
/** Shallow boundaries accompanying a full bundle, which cannot encode them. */
|
|
shallowPath?: string;
|
|
}): Promise<string> {
|
|
const bundleSize = (await fs.stat(input.bundlePath).catch(() => null))?.size ?? 0;
|
|
if (bundleSize === 0) {
|
|
return input.baseSha;
|
|
}
|
|
|
|
const shallowSize = input.shallowPath ? (await fs.stat(input.shallowPath)).size : 0;
|
|
if (shallowSize > GIT_SHALLOW_MAX_BYTES) {
|
|
throw new GitShallowMetadataError(`Git shallow metadata exceeds its byte limit (${shallowSize} > ${GIT_SHALLOW_MAX_BYTES})`);
|
|
}
|
|
const shallow = input.shallowPath ? await fs.readFile(input.shallowPath, "utf8") : "";
|
|
const shallowBytes = Buffer.byteLength(shallow);
|
|
if (shallowBytes > GIT_SHALLOW_MAX_BYTES)
|
|
throw new GitShallowMetadataError(`Git shallow metadata exceeds its byte limit (${shallowBytes} > ${GIT_SHALLOW_MAX_BYTES})`);
|
|
if (shallow.trim()) {
|
|
const boundaries = [...new Set(shallow.trim().split("\n"))];
|
|
if (boundaries.length > GIT_SHALLOW_MAX_BOUNDARIES) throw new GitShallowMetadataError(`Git shallow metadata exceeds its boundary limit (${boundaries.length} > ${GIT_SHALLOW_MAX_BOUNDARIES})`);
|
|
let deadline = Date.now() + 60_000;
|
|
const importGit = (cwd: string, args: string[], options: NonNullable<Parameters<typeof runLocalGit>[2]> = {}) => {
|
|
const remaining = deadline - Date.now();
|
|
if (remaining <= 0) throw new Error("Git shallow history import timed out");
|
|
return runLocalGit(cwd, args, { ...options, timeout: Math.min(options.timeout ?? 15_000, remaining) });
|
|
};
|
|
const objectFormat = (await importGit(input.localDir, ["rev-parse", "--show-object-format"])).stdout.trim();
|
|
const oid = objectFormat === "sha256" ? /^[a-f0-9]{64}$/ : /^[a-f0-9]{40}$/;
|
|
if (!boundaries.every((boundary) => oid.test(boundary))) throw new GitShallowMetadataError(`Invalid Git shallow boundaries for host object format ${objectFormat}`);
|
|
const staging = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-bundle-"));
|
|
try {
|
|
// Git bundles have no shallow protocol. Unpack into a private repository
|
|
// carrying the sender's boundaries, then let ordinary Git fetch negotiate
|
|
// which boundaries the host actually needs. Never overwrite host shallow
|
|
// metadata or disable connectivity checks.
|
|
await importGit(input.localDir, ["clone", "--bare", "--shared", input.localDir, staging], { timeout: 60_000 });
|
|
const advertised = await importGit(staging, ["bundle", "unbundle", input.bundlePath], {
|
|
timeout: 60_000, maxBuffer: 1024 * 1024,
|
|
});
|
|
const head = advertised.stdout.split("\n").find((line) => line.endsWith(` ${input.exportRef}`))?.split(" ")[0];
|
|
if (!head || !oid.test(head)) throw new Error("Git bundle does not advertise the requested export ref");
|
|
const retainedBoundaries = (await fs.readFile(path.join(staging, "shallow"), "utf8").catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code !== "ENOENT") throw error;
|
|
return "";
|
|
})).trim().split("\n").filter(Boolean);
|
|
// Boundary inspection has its own small budget. It must not consume the
|
|
// transfer budget needed to save a valid large bundle after the probes.
|
|
const probeStarted = Date.now(), probeDeadline = probeStarted + 15_000;
|
|
const probeGit = (args: string[]) => {
|
|
const remaining = probeDeadline - Date.now();
|
|
if (remaining <= 0) throw new Error("Git shallow boundary inspection timed out");
|
|
return runLocalGit(staging, args, { timeout: remaining });
|
|
};
|
|
const knownCommits = new Map<string, boolean>();
|
|
const hasCommit = async (commit: string): Promise<boolean> => {
|
|
if (knownCommits.has(commit)) return knownCommits.get(commit)!;
|
|
try {
|
|
await probeGit(["rev-parse", "--verify", "--quiet", `${commit}^{commit}`]);
|
|
knownCommits.set(commit, true);
|
|
return true;
|
|
} catch (error) {
|
|
if ((error as { code?: unknown }).code === 1) {
|
|
knownCommits.set(commit, false);
|
|
return false;
|
|
}
|
|
throw error;
|
|
}
|
|
};
|
|
for (const boundary of boundaries) {
|
|
// A shallow sender can stop before history the host already holds.
|
|
// Keep that connection when possible; adding its boundary unconditionally
|
|
// would hide a real merge-base and reject concurrent host changes.
|
|
if (!await hasCommit(boundary)) continue; // An unrelated tip is not in this bundle.
|
|
const parents = await probeGit(["rev-parse", `${boundary}^@`]);
|
|
let missingParent = false;
|
|
for (const parent of parents.stdout.trim().split("\n").filter(Boolean)) {
|
|
if (!await hasCommit(parent)) {
|
|
missingParent = true;
|
|
break;
|
|
}
|
|
}
|
|
if (missingParent) retainedBoundaries.push(boundary);
|
|
}
|
|
deadline += Date.now() - probeStarted;
|
|
if (retainedBoundaries.length) await fs.writeFile(path.join(staging, "shallow"), `${[...new Set(retainedBoundaries)].join("\n")}\n`);
|
|
await importGit(staging, ["update-ref", input.exportRef, head]);
|
|
await importGit(input.localDir, ["fetch", "--force", "--update-shallow", staging, `${input.exportRef}:${input.importedRef}`], {
|
|
timeout: 60_000, maxBuffer: 1024 * 1024,
|
|
});
|
|
} finally {
|
|
await fs.rm(staging, { recursive: true, force: true });
|
|
}
|
|
} else {
|
|
await runLocalGit(input.localDir, ["fetch", "--force", input.bundlePath, `${input.exportRef}:${input.importedRef}`], {
|
|
timeout: 60_000,
|
|
maxBuffer: 1024 * 1024,
|
|
});
|
|
}
|
|
const importedHead = await runLocalGit(input.localDir, ["rev-parse", input.importedRef], {
|
|
timeout: 10_000,
|
|
maxBuffer: 16 * 1024,
|
|
});
|
|
return importedHead.stdout.trim();
|
|
}
|
|
|
|
/** Substrings git emits when a bundle names a prerequisite the importer lacks. */
|
|
const GIT_MISSING_PREREQUISITE_MARKERS = [
|
|
"did not send all necessary objects",
|
|
"lacks these prerequisite commits",
|
|
"revision walk setup failed",
|
|
];
|
|
|
|
/**
|
|
* True when a bundle import failed because the host repository does not hold a
|
|
* commit the (delta) bundle assumes as a prerequisite. Such a failure is
|
|
* recoverable by re-exporting a full, self-contained bundle from the still-live
|
|
* sandbox rather than discarding the run.
|
|
*/
|
|
export function isMissingGitPrerequisiteError(error: unknown): boolean {
|
|
const message = error instanceof Error ? error.message : String(error);
|
|
return GIT_MISSING_PREREQUISITE_MARKERS.some((marker) => message.includes(marker));
|
|
}
|
|
|
|
export function buildRemoteGitDeltaBundleScript(input: {
|
|
remoteDir: string;
|
|
baseSha: string;
|
|
exportRef: string;
|
|
bundlePath: string;
|
|
statusPath?: string;
|
|
catBundle?: boolean;
|
|
cleanupBundle?: boolean;
|
|
/**
|
|
* Skip the delta boundary entirely and always emit a full, self-contained
|
|
* bundle (no prerequisites). Used as the recovery path when a delta import
|
|
* failed because the host lacked the bundle's prerequisite.
|
|
*/
|
|
forceFullBundle?: boolean;
|
|
/** Export the sender's shallow boundaries alongside a full bundle. */
|
|
shallowPath?: string;
|
|
}): string {
|
|
const remoteDir = shellQuote(input.remoteDir);
|
|
const bundlePath = shellQuote(input.bundlePath);
|
|
const exportRef = shellQuote(input.exportRef);
|
|
const baseSha = shellQuote(input.baseSha);
|
|
const statusPath = input.statusPath ? shellQuote(input.statusPath) : null;
|
|
const cleanupParts = [
|
|
`rm -f ${bundlePath}`,
|
|
...(input.shallowPath ? [`rm -f ${shellQuote(input.shallowPath)}`] : []),
|
|
...(statusPath ? [`rm -f ${statusPath}`] : []),
|
|
`git -C ${remoteDir} update-ref -d ${exportRef} >/dev/null 2>&1 || true`,
|
|
];
|
|
return [
|
|
"set -e",
|
|
input.cleanupBundle ? `cleanup() { ${cleanupParts.join("; ")}; }` : "",
|
|
input.cleanupBundle ? "trap cleanup EXIT" : "",
|
|
`mkdir -p ${shellQuote(path.posix.dirname(input.bundlePath))}`,
|
|
`rm -f ${bundlePath}`,
|
|
...(input.shallowPath ? [
|
|
`shallow_file=$(git -C ${remoteDir} rev-parse --path-format=absolute --git-path shallow)`,
|
|
`if [ -f "$shallow_file" ]; then`,
|
|
` dd if="$shallow_file" of=${shellQuote(input.shallowPath)} bs=${GIT_SHALLOW_MAX_BYTES + 1} count=1 2>/dev/null`,
|
|
`else : > ${shellQuote(input.shallowPath)}; fi`,
|
|
`[ "$(wc -c < ${shellQuote(input.shallowPath)})" -le ${GIT_SHALLOW_MAX_BYTES} ] || { printf '%s\\n' 'Git shallow metadata exceeds its byte limit' >&2; exit ${GIT_SHALLOW_METADATA_EXIT_CODE}; }`,
|
|
] : []),
|
|
// Choose the bundle boundary. A thin bundle `HEAD --not <baseSha>` records
|
|
// baseSha as a prerequisite the importer (host) must already hold. That
|
|
// assumption breaks in two real cases, and then `git fetch` on the host
|
|
// hard-fails with "did not send all necessary objects" and the run's work
|
|
// is lost:
|
|
// 1. The sandbox HEAD has diverged from baseSha (e.g. a local-only branch
|
|
// that forked from an older commit) — the host may still hold baseSha,
|
|
// but a repo whose history is inconsistent cannot satisfy the walk.
|
|
// 2. The host workspace no longer holds baseSha at import time (a shared
|
|
// workspace that was reset/re-realized between export and import).
|
|
// Bundle relative to the merge-base of baseSha and HEAD instead: that
|
|
// merge-base is an ancestor of baseSha, so any host that holds baseSha (or
|
|
// an ancestor of it) can satisfy the prerequisite, while the bundle stays a
|
|
// delta. When baseSha is absent from the sandbox — or no merge-base exists,
|
|
// or the caller forces it after a delta import failed on a missing
|
|
// prerequisite — fall back to a full, self-contained bundle with no
|
|
// prerequisites.
|
|
...(input.forceFullBundle
|
|
? [`bundle_base=""`]
|
|
: [
|
|
`if git -C ${remoteDir} cat-file -e ${baseSha}^{commit} 2>/dev/null; then`,
|
|
` bundle_base=$(git -C ${remoteDir} merge-base ${baseSha} HEAD 2>/dev/null || true)`,
|
|
"else",
|
|
` bundle_base=""`,
|
|
"fi",
|
|
]),
|
|
// An empty bundle means "still at baseSha" to the importer. A reset to an
|
|
// older ancestor has no delta commits either, but must carry its new tip.
|
|
// Use the full-bundle path so Git advertises that tip instead of losing it.
|
|
`if [ -n "$bundle_base" ] && [ "$bundle_base" != ${baseSha} ] && [ "$bundle_base" = "$(git -C ${remoteDir} rev-parse HEAD)" ]; then`,
|
|
` bundle_base=""`,
|
|
"fi",
|
|
`if [ -n "$bundle_base" ]; then`,
|
|
` commit_count=$(git -C ${remoteDir} rev-list --count HEAD --not "$bundle_base")`,
|
|
"else",
|
|
` commit_count=$(git -C ${remoteDir} rev-list --count HEAD)`,
|
|
"fi",
|
|
'if [ "$commit_count" -gt 0 ]; then',
|
|
` git -C ${remoteDir} update-ref ${exportRef} HEAD`,
|
|
` if [ -n "$bundle_base" ]; then`,
|
|
` git -C ${remoteDir} bundle create ${bundlePath} ${exportRef} --not "$bundle_base" >/dev/null`,
|
|
" else",
|
|
` git -C ${remoteDir} bundle create ${bundlePath} ${exportRef} >/dev/null`,
|
|
" fi",
|
|
"else",
|
|
` : > ${bundlePath}`,
|
|
"fi",
|
|
statusPath
|
|
? [
|
|
`git -C ${remoteDir} status --porcelain=v1 --untracked-files=normal -z > ${shellQuote(`${input.statusPath}.git-status`)}`,
|
|
`if [ ! -s ${shellQuote(`${input.statusPath}.git-status`)} ]; then`,
|
|
` printf clean > ${statusPath}`,
|
|
"else",
|
|
` printf dirty > ${statusPath}`,
|
|
"fi",
|
|
`rm -f -- ${shellQuote(`${input.statusPath}.git-status`)}`,
|
|
].join("\n")
|
|
: "",
|
|
input.catBundle ? `cat ${bundlePath}` : "",
|
|
].filter(Boolean).join("\n");
|
|
}
|
|
|
|
// Labels are fixed at the integration call sites, never derived from Git arguments.
|
|
function runIntegrationGit(
|
|
command: WorkspaceRestoreGitCommand,
|
|
...args: Parameters<typeof runLocalGit>
|
|
): Promise<GitCommandResult> {
|
|
return withWorkspaceRestoreGitCommand(command, () => runLocalGit(...args));
|
|
}
|
|
|
|
/**
|
|
* Preserve imported work whose history does not connect to the local one.
|
|
*
|
|
* The dominant real-world cause is a history rewrite inside a transported
|
|
* workspace: transported clones are depth-1 shallow, so the boundary commit
|
|
* reads as parentless there and `git commit --amend` rewrites it into a root
|
|
* commit that shares no ancestor with the host history. A tree merge is
|
|
* impossible without a common ancestor, and failing the integration would
|
|
* discard the run's work. Instead, squash-graft the imported tree onto the
|
|
* current head as a single commit that reuses the imported head's message,
|
|
* with a trailer recording the graft. Concurrent local-only commits keep
|
|
* their place in history as the graft's ancestry; the imported tree is taken
|
|
* wholesale because no base exists to merge against. The caller advances the
|
|
* branch ref to the returned commit.
|
|
*/
|
|
export async function createUnrelatedHistoryGraftCommit(input: {
|
|
localDir: string;
|
|
currentHead: string;
|
|
importedHead: string;
|
|
syncLabel: string;
|
|
}): Promise<string> {
|
|
const importedTree = (await runIntegrationGit("rev_parse", input.localDir, ["rev-parse", `${input.importedHead}^{tree}`], {
|
|
timeout: 10_000,
|
|
maxBuffer: 16 * 1024,
|
|
})).stdout.trim();
|
|
const importedMessage = (await runIntegrationGit("log", input.localDir, ["log", "-1", "--format=%B", input.importedHead], {
|
|
timeout: 10_000,
|
|
maxBuffer: 256 * 1024,
|
|
})).stdout;
|
|
const message = [
|
|
importedMessage.trim(),
|
|
"",
|
|
`(${input.syncLabel} graft ${input.importedHead.slice(0, 12)}: imported history shares no ancestor with ${input.currentHead.slice(0, 12)})`,
|
|
].join("\n");
|
|
const graftCommit = await runIntegrationGit(
|
|
"commit_tree", input.localDir,
|
|
[...GIT_SYNC_COMMIT_IDENTITY_ARGS, "commit-tree", importedTree, "-p", input.currentHead, "-m", message],
|
|
{
|
|
timeout: 60_000,
|
|
maxBuffer: 64 * 1024,
|
|
},
|
|
);
|
|
return graftCommit.stdout.trim();
|
|
}
|
|
|
|
async function updateLocalGitHead(input: {
|
|
localDir: string;
|
|
newHead: string;
|
|
oldHead: string;
|
|
branchName: string | null;
|
|
}): Promise<void> {
|
|
// Prepare the ref transaction before checking the symbolic HEAD identity.
|
|
// Git holds HEAD.lock (and the branch lock when attached) until commit/abort,
|
|
// so a checkout cannot redirect the write after this check. --no-deref also
|
|
// prevents a detached write from following a newly attached branch.
|
|
await withWorkspaceRestoreGitCommand("update_ref", () => new Promise<void>((resolve, reject) => {
|
|
let identityError: unknown;
|
|
let prepared = false;
|
|
let output = "";
|
|
const child = execFile("git", ["-C", input.localDir, "update-ref", "--stdin"], {
|
|
timeout: 15_000,
|
|
maxBuffer: 64 * 1024,
|
|
}, (error, stdout, stderr) => {
|
|
if (identityError) reject(identityError);
|
|
else if (error) reject(Object.assign(error, { stdout, stderr }));
|
|
else if (!stdout.includes("commit: ok\n")) reject(new Error("Git HEAD transaction did not commit."));
|
|
else resolve();
|
|
});
|
|
// Early Git errors close stdin; the process callback reports the error.
|
|
child.stdin!.on("error", () => {});
|
|
child.stdout!.on("data", (chunk: string | Buffer) => {
|
|
output += chunk.toString();
|
|
if (prepared || !output.includes("prepare: ok\n")) return;
|
|
prepared = true;
|
|
void (async () => {
|
|
try {
|
|
const branchName = (await runIntegrationGit("symbolic_ref", input.localDir, ["symbolic-ref", "--quiet", "--short", "HEAD"], {
|
|
timeout: 10_000,
|
|
}).catch((error) => {
|
|
if (error.code === 1) return { stdout: "" };
|
|
throw error;
|
|
})).stdout.trim() || null;
|
|
if (branchName !== input.branchName) {
|
|
throw new Error("Workspace branch changed while remote work was running.");
|
|
}
|
|
child.stdin!.end("commit\n");
|
|
} catch (error) {
|
|
identityError = error;
|
|
child.stdin!.end("abort\n");
|
|
}
|
|
})();
|
|
});
|
|
child.stdin!.write([
|
|
"start",
|
|
...(input.branchName === null ? ["option no-deref"] : []),
|
|
`update HEAD ${input.newHead} ${input.oldHead}`,
|
|
"prepare",
|
|
"",
|
|
].join("\n"));
|
|
}));
|
|
}
|
|
|
|
export async function integrateImportedGitHead(input: {
|
|
localDir: string;
|
|
importedHead: string;
|
|
/** The host Git identity captured before staging this run. */
|
|
baseline?: Pick<GitWorkspaceSnapshot, "headCommit" | "branchName">;
|
|
}): Promise<void> {
|
|
const isConcurrentRefUpdateError = (error: unknown) => {
|
|
const message = error instanceof Error ? error.message : String(error);
|
|
return message.includes("cannot lock ref") && message.includes("expected");
|
|
};
|
|
|
|
for (let attempt = 0; attempt < 5; attempt += 1) {
|
|
const snapshot = {
|
|
headCommit: (await runIntegrationGit("rev_parse", input.localDir, ["rev-parse", "HEAD"])).stdout.trim(),
|
|
branchName: (await runIntegrationGit("symbolic_ref", input.localDir, ["symbolic-ref", "--quiet", "--short", "HEAD"]).catch((error) => {
|
|
if (error.code === 1) return { stdout: "" };
|
|
throw error;
|
|
})).stdout.trim() || null,
|
|
};
|
|
|
|
if (input.baseline && snapshot.branchName !== input.baseline.branchName) {
|
|
throw new Error("Workspace branch changed while remote work was running.");
|
|
}
|
|
const currentHead = snapshot.headCommit;
|
|
if (!currentHead || currentHead === input.importedHead) return;
|
|
|
|
// `git merge-base` exits 1 when the commits share no ancestor — the only
|
|
// outcome that authorizes the graft fallback below. Every other failure
|
|
// (timeout, missing object, repository error) must keep failing the
|
|
// integration instead of silently rewriting the tip.
|
|
let noCommonAncestor = false;
|
|
const mergeBase = await runIntegrationGit("merge_base", input.localDir, ["merge-base", currentHead, input.importedHead], {
|
|
timeout: 10_000,
|
|
maxBuffer: 16 * 1024,
|
|
}).catch((error: unknown) => {
|
|
noCommonAncestor = (error as { code?: unknown } | null)?.code === 1;
|
|
return null;
|
|
});
|
|
const mergeBaseHead = mergeBase?.stdout.trim() ?? "";
|
|
|
|
// A rebase/amend rewrites the sandbox tip without a concurrent host edit.
|
|
// Adopt that history when the host still matches the run's starting tip;
|
|
// merging the old and rewritten commits can reintroduce resolved conflicts.
|
|
// Keep the expected-old-value check: if the host advances during this write,
|
|
// retry against its new tip and use the normal concurrent-history path.
|
|
if (mergeBaseHead === currentHead || (mergeBaseHead && currentHead === input.baseline?.headCommit)) {
|
|
try {
|
|
await updateLocalGitHead({
|
|
localDir: input.localDir, newHead: input.importedHead, oldHead: currentHead, branchName: snapshot.branchName,
|
|
});
|
|
return;
|
|
} catch (error) {
|
|
if (isConcurrentRefUpdateError(error) && attempt < 4) continue;
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
if (mergeBaseHead === input.importedHead) {
|
|
return;
|
|
}
|
|
|
|
if (noCommonAncestor) {
|
|
if (input.baseline && currentHead !== input.baseline.headCommit) {
|
|
throw new Error("Cannot restore unrelated remote history after the host advanced.");
|
|
}
|
|
// No common ancestor — merging is impossible and failing here would
|
|
// discard the imported work. Graft it onto the current head instead;
|
|
// see createUnrelatedHistoryGraftCommit.
|
|
const graftCommit = await createUnrelatedHistoryGraftCommit({
|
|
localDir: input.localDir,
|
|
currentHead,
|
|
importedHead: input.importedHead,
|
|
syncLabel: "Paperclip remote git sync",
|
|
});
|
|
try {
|
|
await updateLocalGitHead({
|
|
localDir: input.localDir, newHead: graftCommit, oldHead: currentHead, branchName: snapshot.branchName,
|
|
});
|
|
return;
|
|
} catch (error) {
|
|
if (isConcurrentRefUpdateError(error) && attempt < 4) continue;
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
let mergedTree;
|
|
try {
|
|
mergedTree = await runIntegrationGit("merge_tree", input.localDir, ["merge-tree", "--write-tree", currentHead, input.importedHead], {
|
|
timeout: 60_000,
|
|
maxBuffer: 256 * 1024,
|
|
});
|
|
} catch (error) {
|
|
const reason = error instanceof Error ? error.message : String(error);
|
|
throw preserveWorkspaceRestoreErrorDiagnostic(new Error(
|
|
`Failed to merge concurrent remote git histories for ${currentHead.slice(0, 12)} and ${input.importedHead.slice(0, 12)}: ${reason}`,
|
|
), error);
|
|
}
|
|
const mergedTreeId = mergedTree.stdout.trim().split("\n")[0]?.trim() ?? "";
|
|
if (!mergedTreeId) {
|
|
throw new Error("Failed to compute a merged git tree for workspace restore.");
|
|
}
|
|
|
|
const mergeCommit = await runIntegrationGit(
|
|
"commit_tree", input.localDir,
|
|
[
|
|
...GIT_SYNC_COMMIT_IDENTITY_ARGS,
|
|
"commit-tree",
|
|
mergedTreeId,
|
|
"-p",
|
|
currentHead,
|
|
"-p",
|
|
input.importedHead,
|
|
"-m",
|
|
`Paperclip remote git sync merge ${input.importedHead.slice(0, 12)}`,
|
|
],
|
|
{
|
|
timeout: 60_000,
|
|
maxBuffer: 64 * 1024,
|
|
},
|
|
);
|
|
try {
|
|
await updateLocalGitHead({
|
|
localDir: input.localDir, newHead: mergeCommit.stdout.trim(), oldHead: currentHead, branchName: snapshot.branchName,
|
|
});
|
|
return;
|
|
} catch (error) {
|
|
if (isConcurrentRefUpdateError(error) && attempt < 4) continue;
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
throw new Error(`Failed to integrate concurrent remote git history for ${input.importedHead.slice(0, 12)} after multiple retries.`);
|
|
}
|
|
|
|
export async function resetLocalGitIndexToHead(input: {
|
|
localDir: string;
|
|
checkWorkingTreeClean?: boolean;
|
|
}): Promise<void> {
|
|
try {
|
|
await runLocalGit(input.localDir, ["reset", "--quiet", "HEAD", "--", "."], {
|
|
timeout: 60_000,
|
|
maxBuffer: 1024 * 1024,
|
|
});
|
|
} catch (error) {
|
|
const detail = error && typeof error === "object"
|
|
? [
|
|
(error as { message?: unknown }).message,
|
|
(error as { stderr?: unknown }).stderr,
|
|
(error as { stdout?: unknown }).stdout,
|
|
].filter((value): value is string => typeof value === "string" && value.trim().length > 0).join("\n")
|
|
: String(error);
|
|
throw preserveWorkspaceRestoreErrorDiagnostic(
|
|
new Error(`Failed to reset local git index to HEAD after workspace restore: ${detail}`), error,
|
|
);
|
|
}
|
|
|
|
const hasDiff = async (args: string[]) => {
|
|
try { await runLocalGit(input.localDir, args, { timeout: workspaceSnapshotTimeoutMs(), maxBuffer: 64 * 1024 }); return false; }
|
|
catch (error) { if ((error as { code?: unknown }).code === 1) return true; throw error; }
|
|
};
|
|
if (await hasDiff(["diff", "--quiet", "--cached", "HEAD", "--"])) {
|
|
throw new Error("Workspace restore left staged git index changes after reset");
|
|
}
|
|
if (input.checkWorkingTreeClean && await hasDiff(["diff", "--quiet", "HEAD", "--"])) {
|
|
console.warn("[paperclip] Workspace restore preserved local working tree changes after clean sandbox restore.");
|
|
}
|
|
}
|