fix: preserve repository cache negotiation refs (#16687)

## Thinking Path

> - Paperclip manages agent work and its execution environments.
> - A fresh repository task needs an independent checkout before its
runner can start.
> - The object cache from #16487 should reduce repeated network
transfer.
> - The cache kept Git objects but dropped remote-tracking refs during
its bare clone.
> - Git needs those tips to negotiate which objects it already has.
> - This pull request preserves the tips in a versioned mirror cache and
records explicit cache outcomes.
> - The benefit is less repeated transfer and clearer evidence when a
cache does not help.

## Linked Issues or Issue Description

Refs #16487 and #16454. No duplicate cache-negotiation fix was found in
the public PR search.

**What happened?**

A valid object cache could make a fresh clone slower. A controlled
staging probe took 45.87 seconds and received 184.39 MB even though the
reference already held the objects. An earlier probe took 52.33 seconds.
An uncached clone took 20.94 seconds.

`git clone --bare` copied the pristine checkout's objects, local
branches, and tags, but omitted its remote-tracking refs. The source had
about 1,200 remote branches. Git could not use the missing tips to
negotiate those objects.

**Expected behavior**

Keep the pristine reference's negotiation tips. Each task must still
authenticate online, use the current remote refs, and end with
independent Git objects.

**Steps to reproduce**

1. Clone a repository with many remote branches into a fresh checkout.
2. Publish a reference with `git clone --bare --no-hardlinks` from that
checkout.
3. Clone the remote with `--reference-if-able` and `--dissociate`.
4. Compare with a reference that retains the remote-tracking tips. Use
Git Trace2 and count received pack bytes.

**Paperclip version or commit**

The staging runtime was `27ab902d3db2770ada79c9f42a9c06e1d346f34b`, with
Git 2.47.3. The cache implementation was unchanged from #16487.

**Deployment mode**

Docker on a persistent application volume. The probe used the
application file owner and the public Paperclip repository. It did not
start agent tasks or alter the application cache.

## What Changed

- Publish pristine object caches with `--mirror` to retain
remote-tracking tips.
- Put cache format version 2 in the key and receipt. The first checkout
builds a new generation. Existing generations stay under the current
storage limits.
- Keep the existing online authentication, current remote refs,
credential and company isolation, independent objects, quotas, and
fallback behavior.
- Add closed Honeycomb cache outcomes and a boolean that states whether
a reference was supplied to clone.
- Document that a lookup hit or supplied reference does not prove Git
avoided transfer.

## Verification

- Controlled staging component experiment: a disposable reference copy
held exactly the same 210,057 objects as the original. It added 1,187
advertised branch tips whose commits were already in those objects.
- Before: 45.87 seconds and 184.39 MB received, one sample in this
comparison.
- After: 12.20 and 14.58 seconds; median 13.39 seconds. Received pack
sizes were 1.99 and 1.95 MB.
- All comparison clones used the same current remote HEAD and ended
without an alternates file.
- The remaining repack took 9.72–9.81 seconds. Checkout file writing
took 0.86–0.96 seconds.
- The staging experiment restored tips on a disposable cache copy. It
did not deploy this patch or measure complete agent latency.
- Confirmed both staging scratch locations were empty afterward. The
original cache timestamps stayed unchanged. The temporary operator SSH
key was revoked and its local files removed.
- TypeScript 5.9.3 transpilation reported zero syntax diagnostics across
the four changed TypeScript files. `git diff --check` passed.
- The new negotiation regression fails with the old `--bare` publisher
and passes with the mirror publisher.
- Focused Vitest suites: 26 tests passed across repository-object-cache
and execution-preparation-timing. These cover independent objects,
current refs, unavailable/revoked remotes, scope separation, invalid
storage, tracing isolation, and endpoint-off behavior.
- The first full focused run passed 24 tests but failed its existing
localhost HTTP fixture because the sandbox denied `listen` on
`127.0.0.1`. The rerun with local loopback permission passed all 26
tests.
- Server semantic typechecking passed with `tsc -p server/tsconfig.json
--noEmit` after building the required workspace declarations. Initial
attempts lacked generated dependency outputs; the completed dependency
build and rerun passed.
- A full application build was not run locally. CI passed: 55 checks
passed and two optional checks were skipped. Greptile is 5/5 on the
current head, with both findings resolved.

## Risks

- Version 2 causes one cold cache rebuild. Old cache generations retain
their disk usage, and the existing quota can prevent publication.
- A cache still needs a full repack to produce an independent task
checkout. This PR does not remove that cost.
- The component experiment establishes the negotiation effect on
staging. It does not establish an end-to-end agent speedup or the
under-30-second goal.
- The new trace fields remain behind the existing OTLP gate. Their
values are closed enums or a boolean. They export no refs, paths, URLs,
cache keys, credentials, or error text.

## Model Used

OpenAI Codex, GPT-6 family, with code and tool use. The exact serving
model ID and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-18 21:28:20 -05:00
1 parent aaeff418cf
commit 023f4a5d0c
7 files changed
+188 -36

No files matched your search

+18
View File
@@ -321,6 +321,24 @@ Asset writes and verification may add only non-negative integer
`paperclip.execution.fileCount` and `paperclip.execution.bytes` attributes.
The closed keys are declared in `EXECUTION_PREPARATION_SPAN_ATTRS` in
`packages/adapter-utils/src/acpx-engine/startup-timing.ts`.
Repository checkout cache phases also carry the closed
`paperclip.execution.repositoryCache.outcome` value: `disabled`, `hit`, `miss`,
`published`, `already_present`, `contended`, `capacity`, `budget`, `invalid`, or `error`.
The clone span carries the boolean
`paperclip.execution.repositoryCache.reference_supplied`, including `false` for
a fallback clone. A lookup hit means a validated cache was found, and supplying
a reference does not prove Git avoided a download. These attributes carry no
cache keys, remote URLs, paths, refs, credentials, or error text.
Pristine repository-object cache version 2 mirrors the fresh checkout's refs,
including remote-tracking refs. Version 1 copied their objects into a bare
repository but omitted the remote-tracking tips. Git could then download those
objects again during negotiation. The version is part of the cache key, so the
first checkout creates a new generation and later checkouts can reuse its tips.
Existing generations remain under the same storage limits. Cached refs stay
private to the company and credential scope. Task checkouts still authenticate
online, use the current remote refs, and dissociate their objects from the cache.
Other execution phases carry only the existing hashed run ID; manual boundaries
have no attributes. Exceptions set status without exporting their messages.
@@ -51,6 +51,8 @@ export const EXECUTION_PREPARATION_SPAN_ATTRS = {
runId: "paperclip.task.run.run_id",
fileCount: "paperclip.execution.fileCount",
bytes: "paperclip.execution.bytes",
repositoryCacheOutcome: "paperclip.execution.repositoryCache.outcome",
repositoryCacheReferenceSupplied: "paperclip.execution.repositoryCache.reference_supplied",
} as const;
/**
@@ -11,11 +11,13 @@ import { withDirectoryMergeLock } from "@paperclipai/adapter-utils/workspace-res
import { materializeManagedProjectWorkspace } from "../services/managed-repository-checkout.js";
import { assertReproducibleBaseGitStorage } from "../services/git-workspace-restoration.js";
import { readRepositoryObjectCache, resolveRepositoryObjectCache, publishRepositoryObjectCache } from "../services/repository-object-cache.js";
import * as preparationTiming from "../services/execution-preparation-timing.js";
const execFile = promisify(execute);
const roots: string[] = [];
afterEach(async () => {
vi.unstubAllEnvs();
vi.restoreAllMocks();
await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true })));
});
async function fixture() {
@@ -33,6 +35,30 @@ async function fixture() {
return { root, source, git, company, repoUrl, prepare };
}
it("retains remote branch tips so a primed clone transfers no existing objects", async () => {
const f = await fixture();
await f.git(f.source, "checkout", "-b", "feature");
await fs.writeFile(path.join(f.source, "feature.txt"), "unmerged feature content\n");
await f.git(f.source, "add", ".");
await f.git(f.source, "commit", "-m", "feature");
const featureHead = await f.git(f.source, "rev-parse", "HEAD");
await f.git(f.source, "checkout", "main");
await f.prepare("first");
const cache = (await resolveRepositoryObjectCache(f.company, f.repoUrl, null))!;
expect(await f.git(cache.directory, "rev-parse", "refs/remotes/origin/feature")).toBe(featureHead);
const packTrace = path.join(f.root, "received.pack");
vi.stubEnv("GIT_TRACE_PACKFILE", packTrace);
await f.prepare("second");
expect(await fs.stat(packTrace).then(stat => stat.size, () => 0)).toBe(0);
const target = path.join(f.root, "second");
expect(await f.git(target, "rev-parse", "origin/feature")).toBe(featureHead);
await expect(fs.stat(path.join(target, ".git/objects/info/alternates"))).rejects.toMatchObject({ code: "ENOENT" });
await fs.rm(f.company, { recursive: true });
expect(await f.git(target, "show", "origin/feature:feature.txt")).toBe("unmerged feature content");
await f.git(target, "fsck", "--full");
});
it("uses current remote refs and produces an independent checkout without private cached refs", async () => {
const f = await fixture();
await f.git(f.source, "checkout", "-b", "private");
@@ -157,11 +183,23 @@ it("skips oversized repositories without creating a cache generation", async ()
const cache = (await resolveRepositoryObjectCache(f.company, f.repoUrl, null))!;
const oversized = path.join(f.source, ".git/oversized");
await fs.writeFile(oversized, ""); await fs.truncate(oversized, 2 * 1024 ** 3 + 1);
const detail = vi.spyOn(preparationTiming, "recordRepositoryObjectCacheDetail");
await publishRepositoryObjectCache(cache, f.source);
expect(detail).toHaveBeenLastCalledWith({ outcome: "capacity" });
expect(await fs.readdir(f.company)).toEqual([]);
expect(await f.git(f.source, "show", "HEAD:work.txt")).toBe("initial");
});
it("reports a symlink found during storage scanning as an invalid cache skip", async () => {
const f = await fixture();
const cache = (await resolveRepositoryObjectCache(f.company, f.repoUrl, null))!;
await fs.symlink(f.source, path.join(f.source, ".git/unexpected-link"));
const detail = vi.spyOn(preparationTiming, "recordRepositoryObjectCacheDetail");
await publishRepositoryObjectCache(cache, f.source);
expect(detail).toHaveBeenLastCalledWith({ outcome: "invalid" });
expect(await fs.readdir(f.company)).toEqual([]);
});
it.each(["https://user:secret@github.com/fixture/repo", "https://github.com/fixture/repo?token=secret", "https://github.com/fixture/repo#secret"])("does not persist credential-bearing URLs: %s", async (repoUrl) => {
const f = await fixture();
expect(await resolveRepositoryObjectCache(f.company, repoUrl, null)).toBeNull();
@@ -1,11 +1,11 @@
import { afterEach, beforeEach, expect, it, vi } from "vitest";
import { createRunPreparationTiming, measureRunPhase, withRunExecutionTiming } from "./execution-preparation-timing.js";
import { createRunPreparationTiming, measureRunPhase, recordRepositoryObjectCacheDetail, withRunExecutionTiming } from "./execution-preparation-timing.js";
const handles = vi.hoisted(() => ({ getTracer: vi.fn(), startActiveSpan: vi.fn() }));
vi.mock("@opentelemetry/api", () => ({
ROOT_CONTEXT: "root", SpanStatusCode: { ERROR: 2 },
trace: { getTracer: handles.getTracer, getActiveSpan: () => undefined },
}));
vi.mock("@opentelemetry/api", async importOriginal => {
const api = await importOriginal<typeof import("@opentelemetry/api")>();
return { ...api, trace: { ...api.trace, getTracer: handles.getTracer, getActiveSpan: () => undefined } };
});
beforeEach(() => {
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test");
handles.getTracer.mockReturnValue({ startActiveSpan: handles.startActiveSpan });
@@ -43,7 +43,10 @@ it.each([undefined, "", " "])("does not consult tracing for endpoint %s", async
await expect(createRunPreparationTiming("run", "native.controller")("backend_setup", () => 42)).resolves.toBe(42);
for (const phase of ["execution.usage_startup", "execution.mcp_handoff", "execution.dispatch_admission", "execution.budget_reservation"] as const) {
const result = {};
const work = vi.fn(() => result);
const work = vi.fn(() => {
recordRepositoryObjectCacheDetail({ outcome: "hit", referenceSupplied: true });
return result;
});
await expect(withRunExecutionTiming("run", () => measureRunPhase(phase, work))).resolves.toBe(result);
expect(work).toHaveBeenCalledOnce();
const original = new Error("original startup failure");
@@ -59,3 +62,46 @@ it.each([undefined, "", " "])("does not consult tracing for endpoint %s", async
error.mockRestore();
}
});
it("keeps cache outcomes and clone references on their owning concurrent phases", async () => {
const spans: Array<{ name: string; attributes: Record<string, unknown> }> = [];
handles.startActiveSpan.mockImplementation((name, options, _context, callback) => {
const record = { name, attributes: { ...options.attributes } };
spans.push(record);
return callback({
end: vi.fn(), setStatus: vi.fn(),
setAttribute: (key: string, value: unknown) => { record.attributes[key] = value; },
});
});
await Promise.all([
withRunExecutionTiming("first", () => measureRunPhase("workspace.checkout.cache_lookup", async () => {
await Promise.resolve();
recordRepositoryObjectCacheDetail({ outcome: "hit" });
})),
withRunExecutionTiming("second", () => measureRunPhase("workspace.checkout.git_clone", async () => {
await Promise.resolve();
recordRepositoryObjectCacheDetail({ referenceSupplied: false });
})),
]);
expect(spans.find(span => span.name === "workspace.checkout.cache_lookup")?.attributes).toMatchObject({
"paperclip.execution.repositoryCache.outcome": "hit",
});
expect(spans.find(span => span.name === "workspace.checkout.git_clone")?.attributes).toMatchObject({
"paperclip.execution.repositoryCache.reference_supplied": false,
});
expect(spans.filter(span => span.name === "task.execution_attempt").every(span =>
Object.keys(span.attributes).every(key => key === "paperclip.task.run.run_id"))).toBe(true);
});
it("does not let cache attribute failures replace the operation result", async () => {
handles.startActiveSpan.mockImplementation((_name, _options, _context, callback) => callback({
end: vi.fn(), setStatus: vi.fn(),
setAttribute: () => { throw new Error("tracer failed"); },
}));
const work = vi.fn(() => {
recordRepositoryObjectCacheDetail({ outcome: "published", referenceSupplied: true });
return 42;
});
await expect(withRunExecutionTiming("run", () => measureRunPhase("workspace.checkout.cache_publish", work))).resolves.toBe(42);
expect(work).toHaveBeenCalledOnce();
});
@@ -84,6 +84,26 @@ type RunScope = { runId: string; parent?: Context; ended: boolean; details: Map<
const ownership = new AsyncLocalStorage<RunScope>();
const endpointEnabled = () => Boolean(process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim());
const repositoryCacheOutcomes = ["disabled", "hit", "miss", "published", "already_present", "contended", "capacity", "budget", "invalid", "error"] as const;
export type RepositoryObjectCacheOutcome = typeof repositoryCacheOutcomes[number];
export function recordRepositoryObjectCacheDetail(detail: {
outcome?: RepositoryObjectCacheOutcome;
referenceSupplied?: boolean;
}): void {
const owned = ownership.getStore();
if (!endpointEnabled() || !owned?.parent || owned.ended) return;
try {
const span = trace.getSpan(owned.parent);
if (detail.outcome && repositoryCacheOutcomes.includes(detail.outcome)) {
span?.setAttribute(EXECUTION_PREPARATION_SPAN_ATTRS.repositoryCacheOutcome, detail.outcome);
}
if (typeof detail.referenceSupplied === "boolean") {
span?.setAttribute(EXECUTION_PREPARATION_SPAN_ATTRS.repositoryCacheReferenceSupplied, detail.referenceSupplied);
}
} catch {}
}
export function getRunExecutionTraceContext(runId: string): Context | undefined {
const owned = ownership.getStore();
return endpointEnabled() && owned?.runId === runId && !owned.ended ? owned.parent : undefined;
@@ -8,7 +8,7 @@ import { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js";
import { describeGitAuthFailure, scrubGitCredentialText, type GitRemoteAuthProvider } from "./git-credentials.js";
import { classifyGitCloneFailure, GitConnectionFailureError, readGitConnectionFailure } from "./git-connection-failure.js";
import { isWorkspaceGitScanError, WorkspaceGitScanError } from "./workspace-git-operation-scheduler.js";
import { measureRunPhase } from "./execution-preparation-timing.js";
import { measureRunPhase, recordRepositoryObjectCacheDetail } from "./execution-preparation-timing.js";
import { resolveRepositoryObjectCache, readRepositoryObjectCache, publishRepositoryObjectCache } from "./repository-object-cache.js";
const execFile = promisify(execFileCallback);
@@ -65,23 +65,34 @@ export async function materializeManagedProjectWorkspace(
: `${cwd}.clone-`);
const cloneSource = input.localSource ?? input.repoUrl;
let objectCache = input.objectCacheCompanyDirectory && !input.localSource
? await measureRunPhase("workspace.checkout.cache_lookup", () => resolveRepositoryObjectCache(input.objectCacheCompanyDirectory!, input.repoUrl!, auth)) : null;
const reference = objectCache ? await measureRunPhase("workspace.checkout.cache_lookup", () => readRepositoryObjectCache(objectCache!)) : null;
? await measureRunPhase("workspace.checkout.cache_lookup", async () => {
const cache = await resolveRepositoryObjectCache(input.objectCacheCompanyDirectory!, input.repoUrl!, auth);
if (!cache) recordRepositoryObjectCacheDetail({ outcome: "disabled" });
return cache;
}) : null;
const reference = objectCache ? await measureRunPhase("workspace.checkout.cache_lookup", async () => {
const cached = await readRepositoryObjectCache(objectCache!);
recordRepositoryObjectCacheDetail({ outcome: cached ? "hit" : "miss" });
return cached;
}) : null;
try {
await input.beforeCopy?.(cloneTmpDir);
try {
const clone = (referencePath: string | null) => measureRunPhase("workspace.checkout.git_clone", () => execFile(
"git",
[...(auth?.configArgs ?? []), ...(referencePath ? ["-c", "repack.updateServerInfo=false"] : []), "clone", "--no-hardlinks", ...(referencePath ? ["--reference-if-able", referencePath, "--dissociate"] : []), "--", cloneSource, cloneTmpDir],
{
env: {
...sanitizeRuntimeServiceBaseEnv(process.env),
GIT_TERMINAL_PROMPT: "0",
...(auth?.env ?? {}),
const clone = (referencePath: string | null) => measureRunPhase("workspace.checkout.git_clone", () => {
recordRepositoryObjectCacheDetail({ referenceSupplied: referencePath !== null });
return execFile(
"git",
[...(auth?.configArgs ?? []), ...(referencePath ? ["-c", "repack.updateServerInfo=false"] : []), "clone", "--no-hardlinks", ...(referencePath ? ["--reference-if-able", referencePath, "--dissociate"] : []), "--", cloneSource, cloneTmpDir],
{
env: {
...sanitizeRuntimeServiceBaseEnv(process.env),
GIT_TERMINAL_PROMPT: "0",
...(auth?.env ?? {}),
},
timeout: MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS,
},
timeout: MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS,
},
));
);
});
try { await clone(reference); }
catch (error) {
if (!reference) throw error;
+33 -16
View File
@@ -7,13 +7,21 @@ import { tryWithDirectoryMergeLock } from "@paperclipai/adapter-utils/workspace-
import { assertWorkspaceManifestDiskSpace } from "@paperclipai/adapter-utils/workspace-manifest";
import type { GitAuthInvocation } from "./git-credentials.js";
import { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js";
import { recordRepositoryObjectCacheDetail, type RepositoryObjectCacheOutcome } from "./execution-preparation-timing.js";
const execFile = promisify(execute);
const REPOSITORY_BYTES = 2 * 1024 ** 3;
const COMPANY_BYTES = 4 * 1024 ** 3;
const COMPANY_REPOSITORIES = 64;
const CACHE_VERSION = 2;
const hash = (value: string) => createHash("sha256").update(value).digest("hex");
class RepositoryObjectCacheSkip extends Error {
constructor(readonly outcome: "capacity" | "budget" | "invalid") {
super("Repository object cache skipped");
}
}
export interface RepositoryObjectCache {
companyDirectory: string;
directory: string;
@@ -36,7 +44,7 @@ export async function resolveRepositoryObjectCache(companyDirectory: string, rep
}
companyDirectory = await fs.realpath(companyDirectory);
const scope = auth ? [auth.source, auth.secretName, auth.configArgs, Object.entries(auth.env).sort(([a], [b]) => a.localeCompare(b))] : null;
const key = hash(JSON.stringify([repoUrl, scope]));
const key = hash(JSON.stringify([CACHE_VERSION, repoUrl, scope]));
return { companyDirectory, directory: path.join(companyDirectory, key), key, repoUrl };
} catch { return null; }
}
@@ -55,7 +63,7 @@ export async function readRepositoryObjectCache(cache: RepositoryObjectCache): P
const stat = await fs.lstat(receiptPath);
if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 4096) return null;
const receipt = JSON.parse(await fs.readFile(receiptPath, "utf8"));
if (receipt.version !== 1 || receipt.key !== cache.key) return null;
if (receipt.version !== CACHE_VERSION || receipt.key !== cache.key) return null;
const options = { timeout: 10000, maxBuffer: 16384, env: sanitizeRuntimeServiceBaseEnv(process.env) };
const bare = await execFile("git", ["-C", cache.directory, "rev-parse", "--is-bare-repository"], options);
const origin = await execFile("git", ["-C", cache.directory, "config", "--local", "--null", "--get-all", "remote.origin.url"], options);
@@ -67,45 +75,54 @@ export async function readRepositoryObjectCache(cache: RepositoryObjectCache): P
async function storageBytes(directory: string, ceiling: number, budget: { entries: number; deadline: number }): Promise<number> {
let bytes = 0;
for await (const entry of await fs.opendir(directory)) {
if (--budget.entries < 0 || performance.now() > budget.deadline) throw new Error("Repository object cache work budget exhausted");
if (--budget.entries < 0 || performance.now() > budget.deadline) throw new RepositoryObjectCacheSkip("budget");
const target = path.join(directory, entry.name);
const stat = await fs.lstat(target);
if (stat.isSymbolicLink()) throw new Error("Invalid repository object cache entry");
if (stat.isSymbolicLink()) throw new RepositoryObjectCacheSkip("invalid");
bytes += stat.isDirectory() ? await storageBytes(target, ceiling - bytes, budget) : stat.size;
if (bytes > ceiling) throw new Error("Repository object cache capacity exhausted");
if (bytes > ceiling) throw new RepositoryObjectCacheSkip("capacity");
}
return bytes;
}
export async function publishRepositoryObjectCache(cache: RepositoryObjectCache, checkout: string): Promise<void> {
try {
if (await readRepositoryObjectCache(cache)) return;
await tryWithDirectoryMergeLock(cache.companyDirectory, async () => {
if (await readRepositoryObjectCache(cache)) {
recordRepositoryObjectCacheDetail({ outcome: "already_present" });
return;
}
const outcome = await tryWithDirectoryMergeLock(cache.companyDirectory, async (): Promise<RepositoryObjectCacheOutcome> => {
const company = await fs.lstat(cache.companyDirectory);
if (!company.isDirectory() || company.isSymbolicLink() || await fs.realpath(cache.companyDirectory) !== cache.companyDirectory) return;
if (!company.isDirectory() || company.isSymbolicLink() || await fs.realpath(cache.companyDirectory) !== cache.companyDirectory) return "invalid";
const budget = { entries: 4096, deadline: performance.now() + 5000 };
let repositories = 0, bytes = 0;
for await (const entry of await fs.opendir(cache.companyDirectory)) {
if (++repositories >= COMPANY_REPOSITORIES) return;
if (++repositories >= COMPANY_REPOSITORIES) return "capacity";
const directory = path.join(cache.companyDirectory, entry.name);
const stat = await fs.lstat(directory);
if (!stat.isDirectory() || stat.isSymbolicLink()) return;
if (!stat.isDirectory() || stat.isSymbolicLink()) return "invalid";
bytes += await storageBytes(directory, COMPANY_BYTES - bytes, budget);
}
if (await fs.lstat(cache.directory).then(() => true, () => false)) return;
if (await fs.lstat(cache.directory).then(() => true, () => false)) return "invalid";
const incoming = await storageBytes(path.join(checkout, ".git"), REPOSITORY_BYTES, budget) + 65536;
if (bytes + incoming > COMPANY_BYTES) return;
if (bytes + incoming > COMPANY_BYTES) return "capacity";
assertWorkspaceManifestDiskSpace(cache.companyDirectory, incoming);
const temporary = path.join(cache.companyDirectory, `.pending-${randomUUID()}`);
try {
const options = () => ({ timeout: Math.max(1, Math.ceil(budget.deadline - performance.now())), maxBuffer: 16384, env: sanitizeRuntimeServiceBaseEnv(process.env) });
await execFile("git", ["clone", "--bare", "--no-hardlinks", "--", checkout, temporary], options());
await execFile("git", ["clone", "--mirror", "--no-hardlinks", "--", checkout, temporary], options());
await execFile("git", ["-C", temporary, "remote", "set-url", "origin", cache.repoUrl], options());
await fs.writeFile(path.join(temporary, "paperclip-object-cache.json"), JSON.stringify({ version: 1, key: cache.key }));
await fs.writeFile(path.join(temporary, "paperclip-object-cache.json"), JSON.stringify({ version: CACHE_VERSION, key: cache.key }));
const actual = await storageBytes(temporary, REPOSITORY_BYTES, budget);
if (bytes + actual > COMPANY_BYTES) return;
if (bytes + actual > COMPANY_BYTES) return "capacity";
await fs.rename(temporary, cache.directory);
return "published";
} finally { await fs.rm(temporary, { recursive: true, force: true }); }
});
} catch {}
recordRepositoryObjectCacheDetail({ outcome: outcome ?? "contended" });
} catch (error) {
const code = (error as NodeJS.ErrnoException)?.code;
recordRepositoryObjectCacheDetail({ outcome: error instanceof RepositoryObjectCacheSkip
? error.outcome : code === "ENOSPC" || code === "workspace_git_scan_failed" ? "capacity" : "error" });
}
}