From 023f4a5d0cd452c540ddd0deb4a0b81ef5e2ff92 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 18 Oct 2026 21:28:20 -0500 Subject: [PATCH] fix: preserve repository cache negotiation refs (#16687) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages agent work and its execution environments. > - A fresh repository task needs an independent checkout before its runner can start. > - The object cache from #16487 should reduce repeated network transfer. > - The cache kept Git objects but dropped remote-tracking refs during its bare clone. > - Git needs those tips to negotiate which objects it already has. > - This pull request preserves the tips in a versioned mirror cache and records explicit cache outcomes. > - The benefit is less repeated transfer and clearer evidence when a cache does not help. ## Linked Issues or Issue Description Refs #16487 and #16454. No duplicate cache-negotiation fix was found in the public PR search. **What happened?** A valid object cache could make a fresh clone slower. A controlled staging probe took 45.87 seconds and received 184.39 MB even though the reference already held the objects. An earlier probe took 52.33 seconds. An uncached clone took 20.94 seconds. `git clone --bare` copied the pristine checkout's objects, local branches, and tags, but omitted its remote-tracking refs. The source had about 1,200 remote branches. Git could not use the missing tips to negotiate those objects. **Expected behavior** Keep the pristine reference's negotiation tips. Each task must still authenticate online, use the current remote refs, and end with independent Git objects. **Steps to reproduce** 1. Clone a repository with many remote branches into a fresh checkout. 2. Publish a reference with `git clone --bare --no-hardlinks` from that checkout. 3. Clone the remote with `--reference-if-able` and `--dissociate`. 4. Compare with a reference that retains the remote-tracking tips. Use Git Trace2 and count received pack bytes. **Paperclip version or commit** The staging runtime was `27ab902d3db2770ada79c9f42a9c06e1d346f34b`, with Git 2.47.3. The cache implementation was unchanged from #16487. **Deployment mode** Docker on a persistent application volume. The probe used the application file owner and the public Paperclip repository. It did not start agent tasks or alter the application cache. ## What Changed - Publish pristine object caches with `--mirror` to retain remote-tracking tips. - Put cache format version 2 in the key and receipt. The first checkout builds a new generation. Existing generations stay under the current storage limits. - Keep the existing online authentication, current remote refs, credential and company isolation, independent objects, quotas, and fallback behavior. - Add closed Honeycomb cache outcomes and a boolean that states whether a reference was supplied to clone. - Document that a lookup hit or supplied reference does not prove Git avoided transfer. ## Verification - Controlled staging component experiment: a disposable reference copy held exactly the same 210,057 objects as the original. It added 1,187 advertised branch tips whose commits were already in those objects. - Before: 45.87 seconds and 184.39 MB received, one sample in this comparison. - After: 12.20 and 14.58 seconds; median 13.39 seconds. Received pack sizes were 1.99 and 1.95 MB. - All comparison clones used the same current remote HEAD and ended without an alternates file. - The remaining repack took 9.72–9.81 seconds. Checkout file writing took 0.86–0.96 seconds. - The staging experiment restored tips on a disposable cache copy. It did not deploy this patch or measure complete agent latency. - Confirmed both staging scratch locations were empty afterward. The original cache timestamps stayed unchanged. The temporary operator SSH key was revoked and its local files removed. - TypeScript 5.9.3 transpilation reported zero syntax diagnostics across the four changed TypeScript files. `git diff --check` passed. - The new negotiation regression fails with the old `--bare` publisher and passes with the mirror publisher. - Focused Vitest suites: 26 tests passed across repository-object-cache and execution-preparation-timing. These cover independent objects, current refs, unavailable/revoked remotes, scope separation, invalid storage, tracing isolation, and endpoint-off behavior. - The first full focused run passed 24 tests but failed its existing localhost HTTP fixture because the sandbox denied `listen` on `127.0.0.1`. The rerun with local loopback permission passed all 26 tests. - Server semantic typechecking passed with `tsc -p server/tsconfig.json --noEmit` after building the required workspace declarations. Initial attempts lacked generated dependency outputs; the completed dependency build and rerun passed. - A full application build was not run locally. CI passed: 55 checks passed and two optional checks were skipped. Greptile is 5/5 on the current head, with both findings resolved. ## Risks - Version 2 causes one cold cache rebuild. Old cache generations retain their disk usage, and the existing quota can prevent publication. - A cache still needs a full repack to produce an independent task checkout. This PR does not remove that cost. - The component experiment establishes the negotiation effect on staging. It does not establish an end-to-end agent speedup or the under-30-second goal. - The new trace fields remain behind the existing OTLP gate. Their values are closed enums or a boolean. They export no refs, paths, URLs, cache keys, credentials, or error text. ## Model Used OpenAI Codex, GPT-6 family, with code and tool use. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/observability.md | 18 ++++++ .../src/acpx-engine/startup-timing.ts | 2 + .../__tests__/repository-object-cache.test.ts | 38 ++++++++++++ .../execution-preparation-timing.test.ts | 58 +++++++++++++++++-- .../services/execution-preparation-timing.ts | 20 +++++++ .../services/managed-repository-checkout.ts | 39 ++++++++----- .../src/services/repository-object-cache.ts | 49 +++++++++++----- 7 files changed, 188 insertions(+), 36 deletions(-) diff --git a/doc/observability.md b/doc/observability.md index 1052b718785..3bfa51657d0 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -321,6 +321,24 @@ Asset writes and verification may add only non-negative integer `paperclip.execution.fileCount` and `paperclip.execution.bytes` attributes. The closed keys are declared in `EXECUTION_PREPARATION_SPAN_ATTRS` in `packages/adapter-utils/src/acpx-engine/startup-timing.ts`. +Repository checkout cache phases also carry the closed +`paperclip.execution.repositoryCache.outcome` value: `disabled`, `hit`, `miss`, +`published`, `already_present`, `contended`, `capacity`, `budget`, `invalid`, or `error`. +The clone span carries the boolean +`paperclip.execution.repositoryCache.reference_supplied`, including `false` for +a fallback clone. A lookup hit means a validated cache was found, and supplying +a reference does not prove Git avoided a download. These attributes carry no +cache keys, remote URLs, paths, refs, credentials, or error text. + +Pristine repository-object cache version 2 mirrors the fresh checkout's refs, +including remote-tracking refs. Version 1 copied their objects into a bare +repository but omitted the remote-tracking tips. Git could then download those +objects again during negotiation. The version is part of the cache key, so the +first checkout creates a new generation and later checkouts can reuse its tips. +Existing generations remain under the same storage limits. Cached refs stay +private to the company and credential scope. Task checkouts still authenticate +online, use the current remote refs, and dissociate their objects from the cache. + Other execution phases carry only the existing hashed run ID; manual boundaries have no attributes. Exceptions set status without exporting their messages. diff --git a/packages/adapter-utils/src/acpx-engine/startup-timing.ts b/packages/adapter-utils/src/acpx-engine/startup-timing.ts index 590d1f4f2ab..2c3fbe8f4cd 100644 --- a/packages/adapter-utils/src/acpx-engine/startup-timing.ts +++ b/packages/adapter-utils/src/acpx-engine/startup-timing.ts @@ -51,6 +51,8 @@ export const EXECUTION_PREPARATION_SPAN_ATTRS = { runId: "paperclip.task.run.run_id", fileCount: "paperclip.execution.fileCount", bytes: "paperclip.execution.bytes", + repositoryCacheOutcome: "paperclip.execution.repositoryCache.outcome", + repositoryCacheReferenceSupplied: "paperclip.execution.repositoryCache.reference_supplied", } as const; /** diff --git a/server/src/__tests__/repository-object-cache.test.ts b/server/src/__tests__/repository-object-cache.test.ts index 4c5454d6f81..5dc9eb9683b 100644 --- a/server/src/__tests__/repository-object-cache.test.ts +++ b/server/src/__tests__/repository-object-cache.test.ts @@ -11,11 +11,13 @@ import { withDirectoryMergeLock } from "@paperclipai/adapter-utils/workspace-res import { materializeManagedProjectWorkspace } from "../services/managed-repository-checkout.js"; import { assertReproducibleBaseGitStorage } from "../services/git-workspace-restoration.js"; import { readRepositoryObjectCache, resolveRepositoryObjectCache, publishRepositoryObjectCache } from "../services/repository-object-cache.js"; +import * as preparationTiming from "../services/execution-preparation-timing.js"; const execFile = promisify(execute); const roots: string[] = []; afterEach(async () => { vi.unstubAllEnvs(); + vi.restoreAllMocks(); await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true }))); }); async function fixture() { @@ -33,6 +35,30 @@ async function fixture() { return { root, source, git, company, repoUrl, prepare }; } +it("retains remote branch tips so a primed clone transfers no existing objects", async () => { + const f = await fixture(); + await f.git(f.source, "checkout", "-b", "feature"); + await fs.writeFile(path.join(f.source, "feature.txt"), "unmerged feature content\n"); + await f.git(f.source, "add", "."); + await f.git(f.source, "commit", "-m", "feature"); + const featureHead = await f.git(f.source, "rev-parse", "HEAD"); + await f.git(f.source, "checkout", "main"); + await f.prepare("first"); + const cache = (await resolveRepositoryObjectCache(f.company, f.repoUrl, null))!; + expect(await f.git(cache.directory, "rev-parse", "refs/remotes/origin/feature")).toBe(featureHead); + + const packTrace = path.join(f.root, "received.pack"); + vi.stubEnv("GIT_TRACE_PACKFILE", packTrace); + await f.prepare("second"); + expect(await fs.stat(packTrace).then(stat => stat.size, () => 0)).toBe(0); + const target = path.join(f.root, "second"); + expect(await f.git(target, "rev-parse", "origin/feature")).toBe(featureHead); + await expect(fs.stat(path.join(target, ".git/objects/info/alternates"))).rejects.toMatchObject({ code: "ENOENT" }); + await fs.rm(f.company, { recursive: true }); + expect(await f.git(target, "show", "origin/feature:feature.txt")).toBe("unmerged feature content"); + await f.git(target, "fsck", "--full"); +}); + it("uses current remote refs and produces an independent checkout without private cached refs", async () => { const f = await fixture(); await f.git(f.source, "checkout", "-b", "private"); @@ -157,11 +183,23 @@ it("skips oversized repositories without creating a cache generation", async () const cache = (await resolveRepositoryObjectCache(f.company, f.repoUrl, null))!; const oversized = path.join(f.source, ".git/oversized"); await fs.writeFile(oversized, ""); await fs.truncate(oversized, 2 * 1024 ** 3 + 1); + const detail = vi.spyOn(preparationTiming, "recordRepositoryObjectCacheDetail"); await publishRepositoryObjectCache(cache, f.source); + expect(detail).toHaveBeenLastCalledWith({ outcome: "capacity" }); expect(await fs.readdir(f.company)).toEqual([]); expect(await f.git(f.source, "show", "HEAD:work.txt")).toBe("initial"); }); +it("reports a symlink found during storage scanning as an invalid cache skip", async () => { + const f = await fixture(); + const cache = (await resolveRepositoryObjectCache(f.company, f.repoUrl, null))!; + await fs.symlink(f.source, path.join(f.source, ".git/unexpected-link")); + const detail = vi.spyOn(preparationTiming, "recordRepositoryObjectCacheDetail"); + await publishRepositoryObjectCache(cache, f.source); + expect(detail).toHaveBeenLastCalledWith({ outcome: "invalid" }); + expect(await fs.readdir(f.company)).toEqual([]); +}); + it.each(["https://user:secret@github.com/fixture/repo", "https://github.com/fixture/repo?token=secret", "https://github.com/fixture/repo#secret"])("does not persist credential-bearing URLs: %s", async (repoUrl) => { const f = await fixture(); expect(await resolveRepositoryObjectCache(f.company, repoUrl, null)).toBeNull(); diff --git a/server/src/services/execution-preparation-timing.test.ts b/server/src/services/execution-preparation-timing.test.ts index 17bda2f4191..05798901709 100644 --- a/server/src/services/execution-preparation-timing.test.ts +++ b/server/src/services/execution-preparation-timing.test.ts @@ -1,11 +1,11 @@ import { afterEach, beforeEach, expect, it, vi } from "vitest"; -import { createRunPreparationTiming, measureRunPhase, withRunExecutionTiming } from "./execution-preparation-timing.js"; +import { createRunPreparationTiming, measureRunPhase, recordRepositoryObjectCacheDetail, withRunExecutionTiming } from "./execution-preparation-timing.js"; const handles = vi.hoisted(() => ({ getTracer: vi.fn(), startActiveSpan: vi.fn() })); -vi.mock("@opentelemetry/api", () => ({ - ROOT_CONTEXT: "root", SpanStatusCode: { ERROR: 2 }, - trace: { getTracer: handles.getTracer, getActiveSpan: () => undefined }, -})); +vi.mock("@opentelemetry/api", async importOriginal => { + const api = await importOriginal(); + return { ...api, trace: { ...api.trace, getTracer: handles.getTracer, getActiveSpan: () => undefined } }; +}); beforeEach(() => { vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test"); handles.getTracer.mockReturnValue({ startActiveSpan: handles.startActiveSpan }); @@ -43,7 +43,10 @@ it.each([undefined, "", " "])("does not consult tracing for endpoint %s", async await expect(createRunPreparationTiming("run", "native.controller")("backend_setup", () => 42)).resolves.toBe(42); for (const phase of ["execution.usage_startup", "execution.mcp_handoff", "execution.dispatch_admission", "execution.budget_reservation"] as const) { const result = {}; - const work = vi.fn(() => result); + const work = vi.fn(() => { + recordRepositoryObjectCacheDetail({ outcome: "hit", referenceSupplied: true }); + return result; + }); await expect(withRunExecutionTiming("run", () => measureRunPhase(phase, work))).resolves.toBe(result); expect(work).toHaveBeenCalledOnce(); const original = new Error("original startup failure"); @@ -59,3 +62,46 @@ it.each([undefined, "", " "])("does not consult tracing for endpoint %s", async error.mockRestore(); } }); + +it("keeps cache outcomes and clone references on their owning concurrent phases", async () => { + const spans: Array<{ name: string; attributes: Record }> = []; + handles.startActiveSpan.mockImplementation((name, options, _context, callback) => { + const record = { name, attributes: { ...options.attributes } }; + spans.push(record); + return callback({ + end: vi.fn(), setStatus: vi.fn(), + setAttribute: (key: string, value: unknown) => { record.attributes[key] = value; }, + }); + }); + await Promise.all([ + withRunExecutionTiming("first", () => measureRunPhase("workspace.checkout.cache_lookup", async () => { + await Promise.resolve(); + recordRepositoryObjectCacheDetail({ outcome: "hit" }); + })), + withRunExecutionTiming("second", () => measureRunPhase("workspace.checkout.git_clone", async () => { + await Promise.resolve(); + recordRepositoryObjectCacheDetail({ referenceSupplied: false }); + })), + ]); + expect(spans.find(span => span.name === "workspace.checkout.cache_lookup")?.attributes).toMatchObject({ + "paperclip.execution.repositoryCache.outcome": "hit", + }); + expect(spans.find(span => span.name === "workspace.checkout.git_clone")?.attributes).toMatchObject({ + "paperclip.execution.repositoryCache.reference_supplied": false, + }); + expect(spans.filter(span => span.name === "task.execution_attempt").every(span => + Object.keys(span.attributes).every(key => key === "paperclip.task.run.run_id"))).toBe(true); +}); + +it("does not let cache attribute failures replace the operation result", async () => { + handles.startActiveSpan.mockImplementation((_name, _options, _context, callback) => callback({ + end: vi.fn(), setStatus: vi.fn(), + setAttribute: () => { throw new Error("tracer failed"); }, + })); + const work = vi.fn(() => { + recordRepositoryObjectCacheDetail({ outcome: "published", referenceSupplied: true }); + return 42; + }); + await expect(withRunExecutionTiming("run", () => measureRunPhase("workspace.checkout.cache_publish", work))).resolves.toBe(42); + expect(work).toHaveBeenCalledOnce(); +}); diff --git a/server/src/services/execution-preparation-timing.ts b/server/src/services/execution-preparation-timing.ts index 94f1a67862b..e92e42ee985 100644 --- a/server/src/services/execution-preparation-timing.ts +++ b/server/src/services/execution-preparation-timing.ts @@ -84,6 +84,26 @@ type RunScope = { runId: string; parent?: Context; ended: boolean; details: Map< const ownership = new AsyncLocalStorage(); const endpointEnabled = () => Boolean(process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim()); +const repositoryCacheOutcomes = ["disabled", "hit", "miss", "published", "already_present", "contended", "capacity", "budget", "invalid", "error"] as const; +export type RepositoryObjectCacheOutcome = typeof repositoryCacheOutcomes[number]; + +export function recordRepositoryObjectCacheDetail(detail: { + outcome?: RepositoryObjectCacheOutcome; + referenceSupplied?: boolean; +}): void { + const owned = ownership.getStore(); + if (!endpointEnabled() || !owned?.parent || owned.ended) return; + try { + const span = trace.getSpan(owned.parent); + if (detail.outcome && repositoryCacheOutcomes.includes(detail.outcome)) { + span?.setAttribute(EXECUTION_PREPARATION_SPAN_ATTRS.repositoryCacheOutcome, detail.outcome); + } + if (typeof detail.referenceSupplied === "boolean") { + span?.setAttribute(EXECUTION_PREPARATION_SPAN_ATTRS.repositoryCacheReferenceSupplied, detail.referenceSupplied); + } + } catch {} +} + export function getRunExecutionTraceContext(runId: string): Context | undefined { const owned = ownership.getStore(); return endpointEnabled() && owned?.runId === runId && !owned.ended ? owned.parent : undefined; diff --git a/server/src/services/managed-repository-checkout.ts b/server/src/services/managed-repository-checkout.ts index 1e6205381b4..3d5dd027c0f 100644 --- a/server/src/services/managed-repository-checkout.ts +++ b/server/src/services/managed-repository-checkout.ts @@ -8,7 +8,7 @@ import { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js"; import { describeGitAuthFailure, scrubGitCredentialText, type GitRemoteAuthProvider } from "./git-credentials.js"; import { classifyGitCloneFailure, GitConnectionFailureError, readGitConnectionFailure } from "./git-connection-failure.js"; import { isWorkspaceGitScanError, WorkspaceGitScanError } from "./workspace-git-operation-scheduler.js"; -import { measureRunPhase } from "./execution-preparation-timing.js"; +import { measureRunPhase, recordRepositoryObjectCacheDetail } from "./execution-preparation-timing.js"; import { resolveRepositoryObjectCache, readRepositoryObjectCache, publishRepositoryObjectCache } from "./repository-object-cache.js"; const execFile = promisify(execFileCallback); @@ -65,23 +65,34 @@ export async function materializeManagedProjectWorkspace( : `${cwd}.clone-`); const cloneSource = input.localSource ?? input.repoUrl; let objectCache = input.objectCacheCompanyDirectory && !input.localSource - ? await measureRunPhase("workspace.checkout.cache_lookup", () => resolveRepositoryObjectCache(input.objectCacheCompanyDirectory!, input.repoUrl!, auth)) : null; - const reference = objectCache ? await measureRunPhase("workspace.checkout.cache_lookup", () => readRepositoryObjectCache(objectCache!)) : null; + ? await measureRunPhase("workspace.checkout.cache_lookup", async () => { + const cache = await resolveRepositoryObjectCache(input.objectCacheCompanyDirectory!, input.repoUrl!, auth); + if (!cache) recordRepositoryObjectCacheDetail({ outcome: "disabled" }); + return cache; + }) : null; + const reference = objectCache ? await measureRunPhase("workspace.checkout.cache_lookup", async () => { + const cached = await readRepositoryObjectCache(objectCache!); + recordRepositoryObjectCacheDetail({ outcome: cached ? "hit" : "miss" }); + return cached; + }) : null; try { await input.beforeCopy?.(cloneTmpDir); try { - const clone = (referencePath: string | null) => measureRunPhase("workspace.checkout.git_clone", () => execFile( - "git", - [...(auth?.configArgs ?? []), ...(referencePath ? ["-c", "repack.updateServerInfo=false"] : []), "clone", "--no-hardlinks", ...(referencePath ? ["--reference-if-able", referencePath, "--dissociate"] : []), "--", cloneSource, cloneTmpDir], - { - env: { - ...sanitizeRuntimeServiceBaseEnv(process.env), - GIT_TERMINAL_PROMPT: "0", - ...(auth?.env ?? {}), + const clone = (referencePath: string | null) => measureRunPhase("workspace.checkout.git_clone", () => { + recordRepositoryObjectCacheDetail({ referenceSupplied: referencePath !== null }); + return execFile( + "git", + [...(auth?.configArgs ?? []), ...(referencePath ? ["-c", "repack.updateServerInfo=false"] : []), "clone", "--no-hardlinks", ...(referencePath ? ["--reference-if-able", referencePath, "--dissociate"] : []), "--", cloneSource, cloneTmpDir], + { + env: { + ...sanitizeRuntimeServiceBaseEnv(process.env), + GIT_TERMINAL_PROMPT: "0", + ...(auth?.env ?? {}), + }, + timeout: MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS, }, - timeout: MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS, - }, - )); + ); + }); try { await clone(reference); } catch (error) { if (!reference) throw error; diff --git a/server/src/services/repository-object-cache.ts b/server/src/services/repository-object-cache.ts index 9a0590739f2..68dc23df1fe 100644 --- a/server/src/services/repository-object-cache.ts +++ b/server/src/services/repository-object-cache.ts @@ -7,13 +7,21 @@ import { tryWithDirectoryMergeLock } from "@paperclipai/adapter-utils/workspace- import { assertWorkspaceManifestDiskSpace } from "@paperclipai/adapter-utils/workspace-manifest"; import type { GitAuthInvocation } from "./git-credentials.js"; import { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js"; +import { recordRepositoryObjectCacheDetail, type RepositoryObjectCacheOutcome } from "./execution-preparation-timing.js"; const execFile = promisify(execute); const REPOSITORY_BYTES = 2 * 1024 ** 3; const COMPANY_BYTES = 4 * 1024 ** 3; const COMPANY_REPOSITORIES = 64; +const CACHE_VERSION = 2; const hash = (value: string) => createHash("sha256").update(value).digest("hex"); +class RepositoryObjectCacheSkip extends Error { + constructor(readonly outcome: "capacity" | "budget" | "invalid") { + super("Repository object cache skipped"); + } +} + export interface RepositoryObjectCache { companyDirectory: string; directory: string; @@ -36,7 +44,7 @@ export async function resolveRepositoryObjectCache(companyDirectory: string, rep } companyDirectory = await fs.realpath(companyDirectory); const scope = auth ? [auth.source, auth.secretName, auth.configArgs, Object.entries(auth.env).sort(([a], [b]) => a.localeCompare(b))] : null; - const key = hash(JSON.stringify([repoUrl, scope])); + const key = hash(JSON.stringify([CACHE_VERSION, repoUrl, scope])); return { companyDirectory, directory: path.join(companyDirectory, key), key, repoUrl }; } catch { return null; } } @@ -55,7 +63,7 @@ export async function readRepositoryObjectCache(cache: RepositoryObjectCache): P const stat = await fs.lstat(receiptPath); if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 4096) return null; const receipt = JSON.parse(await fs.readFile(receiptPath, "utf8")); - if (receipt.version !== 1 || receipt.key !== cache.key) return null; + if (receipt.version !== CACHE_VERSION || receipt.key !== cache.key) return null; const options = { timeout: 10000, maxBuffer: 16384, env: sanitizeRuntimeServiceBaseEnv(process.env) }; const bare = await execFile("git", ["-C", cache.directory, "rev-parse", "--is-bare-repository"], options); const origin = await execFile("git", ["-C", cache.directory, "config", "--local", "--null", "--get-all", "remote.origin.url"], options); @@ -67,45 +75,54 @@ export async function readRepositoryObjectCache(cache: RepositoryObjectCache): P async function storageBytes(directory: string, ceiling: number, budget: { entries: number; deadline: number }): Promise { let bytes = 0; for await (const entry of await fs.opendir(directory)) { - if (--budget.entries < 0 || performance.now() > budget.deadline) throw new Error("Repository object cache work budget exhausted"); + if (--budget.entries < 0 || performance.now() > budget.deadline) throw new RepositoryObjectCacheSkip("budget"); const target = path.join(directory, entry.name); const stat = await fs.lstat(target); - if (stat.isSymbolicLink()) throw new Error("Invalid repository object cache entry"); + if (stat.isSymbolicLink()) throw new RepositoryObjectCacheSkip("invalid"); bytes += stat.isDirectory() ? await storageBytes(target, ceiling - bytes, budget) : stat.size; - if (bytes > ceiling) throw new Error("Repository object cache capacity exhausted"); + if (bytes > ceiling) throw new RepositoryObjectCacheSkip("capacity"); } return bytes; } export async function publishRepositoryObjectCache(cache: RepositoryObjectCache, checkout: string): Promise { try { - if (await readRepositoryObjectCache(cache)) return; - await tryWithDirectoryMergeLock(cache.companyDirectory, async () => { + if (await readRepositoryObjectCache(cache)) { + recordRepositoryObjectCacheDetail({ outcome: "already_present" }); + return; + } + const outcome = await tryWithDirectoryMergeLock(cache.companyDirectory, async (): Promise => { const company = await fs.lstat(cache.companyDirectory); - if (!company.isDirectory() || company.isSymbolicLink() || await fs.realpath(cache.companyDirectory) !== cache.companyDirectory) return; + if (!company.isDirectory() || company.isSymbolicLink() || await fs.realpath(cache.companyDirectory) !== cache.companyDirectory) return "invalid"; const budget = { entries: 4096, deadline: performance.now() + 5000 }; let repositories = 0, bytes = 0; for await (const entry of await fs.opendir(cache.companyDirectory)) { - if (++repositories >= COMPANY_REPOSITORIES) return; + if (++repositories >= COMPANY_REPOSITORIES) return "capacity"; const directory = path.join(cache.companyDirectory, entry.name); const stat = await fs.lstat(directory); - if (!stat.isDirectory() || stat.isSymbolicLink()) return; + if (!stat.isDirectory() || stat.isSymbolicLink()) return "invalid"; bytes += await storageBytes(directory, COMPANY_BYTES - bytes, budget); } - if (await fs.lstat(cache.directory).then(() => true, () => false)) return; + if (await fs.lstat(cache.directory).then(() => true, () => false)) return "invalid"; const incoming = await storageBytes(path.join(checkout, ".git"), REPOSITORY_BYTES, budget) + 65536; - if (bytes + incoming > COMPANY_BYTES) return; + if (bytes + incoming > COMPANY_BYTES) return "capacity"; assertWorkspaceManifestDiskSpace(cache.companyDirectory, incoming); const temporary = path.join(cache.companyDirectory, `.pending-${randomUUID()}`); try { const options = () => ({ timeout: Math.max(1, Math.ceil(budget.deadline - performance.now())), maxBuffer: 16384, env: sanitizeRuntimeServiceBaseEnv(process.env) }); - await execFile("git", ["clone", "--bare", "--no-hardlinks", "--", checkout, temporary], options()); + await execFile("git", ["clone", "--mirror", "--no-hardlinks", "--", checkout, temporary], options()); await execFile("git", ["-C", temporary, "remote", "set-url", "origin", cache.repoUrl], options()); - await fs.writeFile(path.join(temporary, "paperclip-object-cache.json"), JSON.stringify({ version: 1, key: cache.key })); + await fs.writeFile(path.join(temporary, "paperclip-object-cache.json"), JSON.stringify({ version: CACHE_VERSION, key: cache.key })); const actual = await storageBytes(temporary, REPOSITORY_BYTES, budget); - if (bytes + actual > COMPANY_BYTES) return; + if (bytes + actual > COMPANY_BYTES) return "capacity"; await fs.rename(temporary, cache.directory); + return "published"; } finally { await fs.rm(temporary, { recursive: true, force: true }); } }); - } catch {} + recordRepositoryObjectCacheDetail({ outcome: outcome ?? "contended" }); + } catch (error) { + const code = (error as NodeJS.ErrnoException)?.code; + recordRepositoryObjectCacheDetail({ outcome: error instanceof RepositoryObjectCacheSkip + ? error.outcome : code === "ENOSPC" || code === "workspace_git_scan_failed" ? "capacity" : "error" }); + } }