mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 16:27:18 +02:00
perf: reduce fresh workspace clone and sandbox staging waits (#16487)
## Thinking Path > - Paperclip manages AI agents and their work. > - Each run needs a private workspace and safe persistence. > - Fresh isolated repository sources download upstream Git history again. > - Sandbox staging also verifies an optional seed cache before it uploads the workspace. > - That verification holds a company lock during a full extraction and hash scan. > - This pull request reuses pristine Git objects through an authenticated remote clone. > - It also overlaps seed verification with upload and moves verification outside the publication lock. > - The benefit is less repeated transfer and less serial preparation work, with the same task isolation and recovery contract. ## Linked Issues or Issue Description **What existing behavior does this improve?** Fresh task-private repository preparation and inbound sandbox workspace staging. **Current behavior** Each isolated remote source clones the full upstream history. Optional seed verification then extracts and hashes another full tree before upload. One workspace's verification can hold the company cache lock while other workspaces wait. **Proposed behavior** Use a company and credential scoped cache of pristine upstream objects for GitHub HTTPS sources. Each task still clones from the current remote. It disconnects from the cache before publication. Run seed verification alongside upload. Hold the company lock only for final capacity checks and publication. **Reason and benefit** These operations lie on the workspace preparation path. Removing repeated Git pack transfer and serial verification can reduce startup time. Local measurements below cover these boundaries. They do not establish hosted agent latency. Related: #16425, #16454, #16398, #16461, #16468, #16473. The cache inspection and durable folder proposal in #13376 covers separate product work. ## What Changed - Add a pristine upstream Git object cache for fresh isolated GitHub HTTPS sources. Key it by company, repository and current credential fingerprint. Do not store credentials. - Keep online clone authorization and current remote refs. Use `--reference-if-able --dissociate` so task objects remain independent. A reference failure retries once online without the cache and resolves credentials again. - Cap cache storage at 64 entries and 4 GiB per company, with 2 GiB per entry. Skip contended publication. Bound priming by a cooperative five-second budget and a 4,096-entry scan allowance. Keep the existing free-space reserve. - Overlap optional seed verification and required workspace upload. Settle both before returning or deleting temporary files. Recheck quotas and company directory identity under the publication lock. - Keep the independent fresh upstream reproducibility proof, task commits, dirty source copies, ignored files, nested repositories and durable recovery behavior. - Suppress optional Git server-info generation during dissociation so generated storage passes the existing safe-expiration checks. This uses [the supported Git setting](https://git-scm.com/docs/git-config/2.43.0#Documentation/git-config.txt-repackupdateServerInfo). - Add bounded timing phases, regression coverage and two repeatable local benchmarks. No first-party Telemetry or dependency changes. ## Verification - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - Affected adapter suites: 240 tests passed. Native workspace persistence/history and tracing: 33 tests passed. Server checkout/isolation/tracing batch: 53 tests passed. Final object-cache authority, path and capacity suite: 12 tests passed. These batches overlap. - Cache and expiration/restoration lifecycle batch: 34 tests passed. A direct generated-storage check covers cached clones. - Real HTTP Git fixture: a matching cache does not bypass revoked remote access. Other fixtures cover current refs/tags, hidden cached commits, independent objects after cache removal, corruption fallback, credential/company scope, quota limits and publication contention. - Initial restricted runs hit SQLite write and Unix socket permission failures. The adapter batch passed with disposable writable instance state and socket access. The first typecheck rejected the test use of `Promise.withResolvers`; typed deferred gates now use the supported library contract. - Repository benchmark on final head `9047368bf3b7fa2061e00d698b8b51cf4ae1da32`: five alternating samples, real local `file://` Git transport, synthetic 32 MiB history, no injected latency. Median 1143.8 ms to 240.5 ms; ranges 1085.7–1486.6 ms and 188.0–254.8 ms. Received Git pack bytes fell from 33,569,788 to zero. Initial cache priming took 1208.4 ms and is reported separately. - Sandbox benchmark, five alternating samples per mode on 10,000 files with real packing, extraction and verification: with a controlled 3-second transfer delay, median 9.450 s to 7.620 s (ranges 9.280–12.115 s and 7.296–7.983 s). With no injected delay, median 6.808 s to 6.450 s (ranges 6.255–7.029 s and 5.709–6.902 s). The no-delay ranges overlap; local copying competes with verification for disk. Both modes verify final tree contents, ignored-file exclusion and Git HEAD. These are synthetic local fixtures, not hosted agent runs. - Reproduce with `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-repository-object-cache.ts` and `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-workspace-staging.ts --baseline-ref=307eb16c0c --transfer-delay-ms=3000`. - Prior head `af766bc820` received Greptile 5/5 with no findings. Its E2E shard 7 failed in `plan-task-fixture` cleanup: company deletion raced an `activity_log` foreign key. That fixture has no provider run or Git workspace. The failure is retained here; final-head CI must pass. The corrected corruption fixture now forces a pack and verifies a real damaged archive. Current head `9047368bf3b7fa2061e00d698b8b51cf4ae1da32` has Greptile 5/5 with no review threads or findings. Final head has 55 passing checks and two optional Storybook skips. [The first workspace-shard attempt](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116655485350) passed all 1,778 cases, then reported a late `EPIPE` from unchanged `server-utils.ts` while the ACP reuse test was active. That case mocks workspace preparation and passed locally in isolation. [Its single retry on the same commit](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116659728034) passed, as did the dependent verification gate. The earlier E2E cleanup fixture passed on this final head. Both initial failures remain recorded here. A full local suite was not duplicated; the affected boundaries were run locally and the complete suite runs in CI. ## Risks - First cache population adds local copying. Cold clones, credential rotation, capacity exhaustion and a corrupt cache retain the online clone path. No eviction is added. Cached data is bounded, but many companies can each consume their allowance. - Individual filesystem calls and cleanup can exceed the cooperative priming budget. Failed reference clones make one additional online attempt. - Parallel verification and upload compete for disk and CPU. The local no-delay benchmark shows a smaller gain than a slow-transfer case. Verification still checks contents before cache publication, and required upload failures remain failures. - Local fixtures do not prove hosted throughput or the 15–30-second end-to-end target. The fresh checkout still writes its working tree, and model startup, copyback and other preparation costs remain. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed in this session. Used reasoning, source editing, shell tools and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal task identifiers - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
647116dd4b
commit
ce0eff7d1f
15 files changed
+708
-38
No files matched your search
@@ -0,0 +1,183 @@
|
||||
import { execFile as execute } from "node:child_process";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { promisify } from "node:util";
|
||||
import { createServer } from "node:http";
|
||||
import { createReadStream } from "node:fs";
|
||||
import { afterEach, expect, it, vi } from "vitest";
|
||||
import { withDirectoryMergeLock } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
||||
import { materializeManagedProjectWorkspace } from "../services/managed-repository-checkout.js";
|
||||
import { assertReproducibleBaseGitStorage } from "../services/git-workspace-restoration.js";
|
||||
import { readRepositoryObjectCache, resolveRepositoryObjectCache, publishRepositoryObjectCache } from "../services/repository-object-cache.js";
|
||||
|
||||
const execFile = promisify(execute);
|
||||
const roots: string[] = [];
|
||||
afterEach(async () => {
|
||||
vi.unstubAllEnvs();
|
||||
await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true })));
|
||||
});
|
||||
async function fixture() {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "repository-object-cache-")); roots.push(root);
|
||||
vi.stubEnv("PAPERCLIP_HOME", path.join(root, "home"));
|
||||
const source = path.join(root, "source"); await fs.mkdir(source);
|
||||
const git = async (cwd: string, ...args: string[]) => (await execFile("git", ["-C", cwd, ...args])).stdout.trim();
|
||||
await git(source, "init", "-b", "main");
|
||||
await git(source, "config", "user.name", "Fixture"); await git(source, "config", "user.email", "fixture@example.test");
|
||||
await fs.writeFile(path.join(source, "work.txt"), "initial\n");
|
||||
await git(source, "add", "."); await git(source, "commit", "-m", "initial");
|
||||
const repoUrl = pathToFileURL(source).href;
|
||||
const company = path.join(root, "cache", "company");
|
||||
const prepare = (name: string) => materializeManagedProjectWorkspace(path.join(root, name), { repoUrl, objectCacheCompanyDirectory: company });
|
||||
return { root, source, git, company, repoUrl, prepare };
|
||||
}
|
||||
|
||||
it("uses current remote refs and produces an independent checkout without private cached refs", async () => {
|
||||
const f = await fixture();
|
||||
await f.git(f.source, "checkout", "-b", "private");
|
||||
await fs.writeFile(path.join(f.source, "private.txt"), "private branch bytes");
|
||||
await f.git(f.source, "add", "."); await f.git(f.source, "commit", "-m", "private");
|
||||
const privateHead = await f.git(f.source, "rev-parse", "HEAD");
|
||||
await f.git(f.source, "checkout", "main");
|
||||
await f.prepare("first");
|
||||
const cache = (await resolveRepositoryObjectCache(f.company, f.repoUrl, null))!;
|
||||
expect(await readRepositoryObjectCache(cache)).toBe(cache.directory);
|
||||
await f.git(f.source, "branch", "-D", "private");
|
||||
await fs.writeFile(path.join(f.source, "work.txt"), "current\n");
|
||||
await f.git(f.source, "commit", "-am", "current"); await f.git(f.source, "tag", "current-tag");
|
||||
const currentHead = await f.git(f.source, "rev-parse", "HEAD");
|
||||
await f.prepare("second");
|
||||
const target = path.join(f.root, "second");
|
||||
expect(await f.git(target, "rev-parse", "HEAD")).toBe(currentHead);
|
||||
expect(await f.git(target, "rev-parse", "current-tag")).toBe(currentHead);
|
||||
expect(await fs.readFile(path.join(target, "work.txt"), "utf8")).toBe("current\n");
|
||||
await expect(f.git(target, "cat-file", "-e", privateHead)).rejects.toThrow();
|
||||
await expect(fs.stat(path.join(target, ".git/objects/info/alternates"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await assertReproducibleBaseGitStorage(target);
|
||||
await fs.rm(f.company, { recursive: true });
|
||||
await f.git(target, "fsck", "--full");
|
||||
expect(await f.git(target, "show", "HEAD~1:work.txt")).toBe("initial");
|
||||
});
|
||||
|
||||
it("never substitutes a cached checkout for an unavailable remote", async () => {
|
||||
const f = await fixture(); await f.prepare("first");
|
||||
await fs.rename(f.source, `${f.source}-unavailable`);
|
||||
await expect(f.prepare("denied")).rejects.toThrow("Failed to prepare managed checkout");
|
||||
await expect(fs.stat(path.join(f.root, "denied"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
|
||||
it("separates company and current credential scopes", async () => {
|
||||
const f = await fixture();
|
||||
const auth = { source: "company_secret" as const, secretName: "GITHUB_TOKEN", configArgs: [], env: { TOKEN: "first-token" } };
|
||||
const first = await resolveRepositoryObjectCache(f.company, f.repoUrl, auth);
|
||||
const other = await resolveRepositoryObjectCache(f.company, f.repoUrl, { ...auth, env: { TOKEN: "new-token" } });
|
||||
const company = await resolveRepositoryObjectCache(path.join(f.root, "cache/other-company"), f.repoUrl, auth);
|
||||
expect(first!.directory).not.toBe(other!.directory); expect(first!.directory).not.toBe(company!.directory);
|
||||
expect(JSON.stringify(first)).not.toContain("first-token");
|
||||
});
|
||||
|
||||
it("skips contended optional cache publication without delaying a fresh checkout", async () => {
|
||||
const f = await fixture();
|
||||
await resolveRepositoryObjectCache(f.company, f.repoUrl, null);
|
||||
await withDirectoryMergeLock(f.company, async () => {
|
||||
await f.prepare("first");
|
||||
expect(await fs.readdir(f.company)).toEqual([]);
|
||||
});
|
||||
expect(await fs.readFile(path.join(f.root, "first/work.txt"), "utf8")).toBe("initial\n");
|
||||
});
|
||||
|
||||
it("rebuilds from the remote when the cached pack is corrupt", async () => {
|
||||
const f = await fixture(); await f.prepare("first");
|
||||
const cache = (await resolveRepositoryObjectCache(f.company, f.repoUrl, null))!;
|
||||
await f.git(cache.directory, "gc", "--prune=now");
|
||||
const packs = (await fs.readdir(path.join(cache.directory, "objects/pack"))).filter(entry => entry.endsWith(".pack"));
|
||||
expect(packs.length).toBeGreaterThan(0);
|
||||
for (const entry of packs) {
|
||||
const file = path.join(cache.directory, "objects/pack", entry);
|
||||
await fs.chmod(file, 0o600);
|
||||
await fs.writeFile(file, "corrupted");
|
||||
}
|
||||
await f.git(path.join(f.root, "first"), "fsck", "--full");
|
||||
await f.prepare("second");
|
||||
expect(await fs.readFile(path.join(f.root, "second/work.txt"), "utf8")).toBe("initial\n");
|
||||
});
|
||||
|
||||
it("still authenticates to the HTTP remote when a matching object cache exists", async () => {
|
||||
const f = await fixture();
|
||||
const remote = path.join(f.root, "remote.git");
|
||||
await execFile("git", ["clone", "--bare", f.source, remote]);
|
||||
await f.git(remote, "update-server-info");
|
||||
let revoked = false, rejected = 0;
|
||||
const server = createServer(async (request, response) => {
|
||||
if (revoked || request.headers.authorization !== "Bearer fixture-token") {
|
||||
rejected++; response.writeHead(403); response.end(); return;
|
||||
}
|
||||
const pathname = new URL(request.url!, "http://localhost").pathname;
|
||||
if (!/^\/[a-zA-Z0-9/._-]+$/.test(pathname) || pathname.includes("..")) { response.writeHead(404); response.end(); return; }
|
||||
const file = path.join(remote, pathname);
|
||||
if (!await fs.stat(file).then(stat => stat.isFile(), () => false)) { response.writeHead(404); response.end(); return; }
|
||||
response.setHeader("Content-Type", "text/plain");
|
||||
createReadStream(file).pipe(response);
|
||||
});
|
||||
await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
|
||||
try {
|
||||
const address = server.address() as { port: number };
|
||||
const repoUrl = `http://127.0.0.1:${address.port}`;
|
||||
const resolveGitAuth = vi.fn(async () => ({ source: "company_secret" as const, secretName: "fixture", configArgs: [], env: {
|
||||
GIT_CONFIG_COUNT: "1", GIT_CONFIG_KEY_0: "http.extraHeader", GIT_CONFIG_VALUE_0: "Authorization: Bearer fixture-token",
|
||||
} }));
|
||||
const prepare = (name: string) => materializeManagedProjectWorkspace(path.join(f.root, name), {
|
||||
repoUrl, objectCacheCompanyDirectory: f.company, resolveGitAuth,
|
||||
});
|
||||
await prepare("authorized");
|
||||
expect(await fs.readdir(f.company)).toHaveLength(1);
|
||||
revoked = true;
|
||||
await expect(prepare("revoked")).rejects.toThrow("403");
|
||||
expect(rejected).toBeGreaterThan(0); expect(resolveGitAuth.mock.calls.length).toBeGreaterThanOrEqual(2);
|
||||
await expect(fs.stat(path.join(f.root, "revoked"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(await fs.readFile(path.join(f.root, "authorized/.git/config"), "utf8")).not.toContain("fixture-token");
|
||||
} finally { await new Promise<void>((resolve, reject) => server.close(error => error ? reject(error) : resolve())); }
|
||||
});
|
||||
|
||||
it("does not follow a cache directory symlink or grow an over-capacity company", async () => {
|
||||
const f = await fixture();
|
||||
const cache = (await resolveRepositoryObjectCache(f.company, f.repoUrl, null))!;
|
||||
await fs.symlink(f.source, cache.directory);
|
||||
expect(await readRepositoryObjectCache(cache)).toBeNull();
|
||||
await fs.unlink(cache.directory);
|
||||
for (let i = 0; i < 64; i++) await fs.mkdir(path.join(f.company, `retained-${i}`));
|
||||
await publishRepositoryObjectCache(cache, f.source);
|
||||
expect(await fs.readdir(f.company)).toHaveLength(64);
|
||||
await expect(fs.stat(cache.directory)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
|
||||
it("skips oversized repositories without creating a cache generation", async () => {
|
||||
const f = await fixture();
|
||||
const cache = (await resolveRepositoryObjectCache(f.company, f.repoUrl, null))!;
|
||||
const oversized = path.join(f.source, ".git/oversized");
|
||||
await fs.writeFile(oversized, ""); await fs.truncate(oversized, 2 * 1024 ** 3 + 1);
|
||||
await publishRepositoryObjectCache(cache, f.source);
|
||||
expect(await fs.readdir(f.company)).toEqual([]);
|
||||
expect(await f.git(f.source, "show", "HEAD:work.txt")).toBe("initial");
|
||||
});
|
||||
|
||||
it.each(["https://user:secret@github.com/fixture/repo", "https://github.com/fixture/repo?token=secret", "https://github.com/fixture/repo#secret"])("does not persist credential-bearing URLs: %s", async (repoUrl) => {
|
||||
const f = await fixture();
|
||||
expect(await resolveRepositoryObjectCache(f.company, repoUrl, null)).toBeNull();
|
||||
await expect(fs.stat(f.company)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
|
||||
it("does not initialize or publish through a replaced cache parent", async () => {
|
||||
const f = await fixture();
|
||||
const other = path.join(f.root, "other"); await fs.mkdir(other);
|
||||
await fs.symlink(other, path.dirname(f.company));
|
||||
expect(await resolveRepositoryObjectCache(f.company, f.repoUrl, null)).toBeNull();
|
||||
expect(await fs.readdir(other)).toEqual([]);
|
||||
await fs.unlink(path.dirname(f.company));
|
||||
const cache = (await resolveRepositoryObjectCache(f.company, f.repoUrl, null))!;
|
||||
await fs.rename(path.dirname(f.company), path.join(f.root, "retained"));
|
||||
await fs.mkdir(path.join(other, "company")); await fs.symlink(other, path.dirname(f.company));
|
||||
await publishRepositoryObjectCache(cache, f.source);
|
||||
expect(await fs.readdir(path.join(other, "company"))).toEqual([]);
|
||||
});
|
||||
@@ -43,6 +43,7 @@ export type ExecutionPhase =
|
||||
| "workspace.materialize.directories" | "workspace.materialize.source_identity"
|
||||
| "workspace.checkout.lookup" | "workspace.checkout.git_auth" | "workspace.checkout.git_clone"
|
||||
| "workspace.checkout.git_ref" | "workspace.checkout.publish"
|
||||
| "workspace.checkout.cache_lookup" | "workspace.checkout.cache_publish"
|
||||
| "runtime_selection.review" | "runtime_selection.contract_read" | "runtime_selection.contract_ensure"
|
||||
| "runtime_selection.resume" | "runtime_selection.recovery_profile" | "runtime_selection.profile" | "runtime_selection.interactions"
|
||||
| "runtime_selection.provider_discovery" | "runtime_selection.bootstrap" | "runtime_selection.run_write" | "runtime_selection.finalization_write"
|
||||
|
||||
@@ -6,7 +6,7 @@ import fs from "node:fs/promises";
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
import type { TaskWorkspaceIntent } from "@paperclipai/shared";
|
||||
import type { GitRemoteAuthProvider } from "./git-credentials.js";
|
||||
import { isGitHubHttpsRemoteUrl, type GitRemoteAuthProvider } from "./git-credentials.js";
|
||||
import { resolvePaperclipInstanceRoot } from "../home-paths.js";
|
||||
import { materializeManagedProjectWorkspace } from "./managed-repository-checkout.js";
|
||||
import { measureRunPhase } from "./execution-preparation-timing.js";
|
||||
@@ -139,6 +139,8 @@ export async function materializeIsolatedRemoteWorkspaceSource(input: {
|
||||
});
|
||||
const result = await materializeManagedProjectWorkspace(canonicalCwd, {
|
||||
repoUrl: input.repoUrl, resolveGitAuth: input.resolveGitAuth,
|
||||
objectCacheCompanyDirectory: isGitHubHttpsRemoteUrl(input.repoUrl)
|
||||
? path.join(await fs.realpath(resolvePaperclipInstanceRoot()), "repository-object-cache", input.companyId) : undefined,
|
||||
});
|
||||
if (result.warning) throw new Error("Isolated repository source is not a Git checkout");
|
||||
await measureRunPhase("workspace.materialize.source_identity", () => assertRemoteSourceIdentity(cwd, input.repoUrl));
|
||||
|
||||
@@ -9,6 +9,7 @@ import { describeGitAuthFailure, scrubGitCredentialText, type GitRemoteAuthProvi
|
||||
import { classifyGitCloneFailure, GitConnectionFailureError, readGitConnectionFailure } from "./git-connection-failure.js";
|
||||
import { isWorkspaceGitScanError, WorkspaceGitScanError } from "./workspace-git-operation-scheduler.js";
|
||||
import { measureRunPhase } from "./execution-preparation-timing.js";
|
||||
import { resolveRepositoryObjectCache, readRepositoryObjectCache, publishRepositoryObjectCache } from "./repository-object-cache.js";
|
||||
|
||||
const execFile = promisify(execFileCallback);
|
||||
const MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS = 10 * 60 * 1000;
|
||||
@@ -23,6 +24,7 @@ export async function materializeManagedProjectWorkspace(
|
||||
beforeCopy?: (cloneCwd: string) => Promise<void>;
|
||||
beforePublish?: (cloneCwd: string) => Promise<void>;
|
||||
stagingParent?: string;
|
||||
objectCacheCompanyDirectory?: string;
|
||||
},
|
||||
): Promise<{ cwd: string; warning: string | null }> {
|
||||
await fs.mkdir(path.dirname(cwd), { recursive: true });
|
||||
@@ -55,19 +57,22 @@ export async function materializeManagedProjectWorkspace(
|
||||
await fs.rm(cwd, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
const auth = input.resolveGitAuth && !input.localSource
|
||||
let auth = input.resolveGitAuth && !input.localSource
|
||||
? await measureRunPhase("workspace.checkout.git_auth", () => input.resolveGitAuth!(input.repoUrl!))
|
||||
: null;
|
||||
const cloneTmpDir = await fs.mkdtemp(input.stagingParent
|
||||
? path.join(input.stagingParent, `${path.basename(cwd)}.clone-`)
|
||||
: `${cwd}.clone-`);
|
||||
const cloneSource = input.localSource ?? input.repoUrl;
|
||||
let objectCache = input.objectCacheCompanyDirectory && !input.localSource
|
||||
? await measureRunPhase("workspace.checkout.cache_lookup", () => resolveRepositoryObjectCache(input.objectCacheCompanyDirectory!, input.repoUrl!, auth)) : null;
|
||||
const reference = objectCache ? await measureRunPhase("workspace.checkout.cache_lookup", () => readRepositoryObjectCache(objectCache!)) : null;
|
||||
try {
|
||||
await input.beforeCopy?.(cloneTmpDir);
|
||||
try {
|
||||
await measureRunPhase("workspace.checkout.git_clone", () => execFile(
|
||||
const clone = (referencePath: string | null) => measureRunPhase("workspace.checkout.git_clone", () => execFile(
|
||||
"git",
|
||||
[...(auth?.configArgs ?? []), "clone", "--no-hardlinks", "--", cloneSource, cloneTmpDir],
|
||||
[...(auth?.configArgs ?? []), ...(referencePath ? ["-c", "repack.updateServerInfo=false"] : []), "clone", "--no-hardlinks", ...(referencePath ? ["--reference-if-able", referencePath, "--dissociate"] : []), "--", cloneSource, cloneTmpDir],
|
||||
{
|
||||
env: {
|
||||
...sanitizeRuntimeServiceBaseEnv(process.env),
|
||||
@@ -77,6 +82,14 @@ export async function materializeManagedProjectWorkspace(
|
||||
timeout: MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS,
|
||||
},
|
||||
));
|
||||
try { await clone(reference); }
|
||||
catch (error) {
|
||||
if (!reference) throw error;
|
||||
for (const name of await fs.readdir(cloneTmpDir)) await fs.rm(path.join(cloneTmpDir, name), { recursive: true, force: true });
|
||||
objectCache = null;
|
||||
auth = input.resolveGitAuth ? await input.resolveGitAuth(input.repoUrl) : null;
|
||||
await clone(null);
|
||||
}
|
||||
} catch (error) {
|
||||
const connectionFailure = !input.localSource ? classifyGitCloneFailure(input.repoUrl, error) : null;
|
||||
if (connectionFailure) throw new GitConnectionFailureError(
|
||||
@@ -84,6 +97,7 @@ export async function materializeManagedProjectWorkspace(
|
||||
);
|
||||
throw error;
|
||||
}
|
||||
if (objectCache && !reference) await measureRunPhase("workspace.checkout.cache_publish", () => publishRepositoryObjectCache(objectCache!, cloneTmpDir));
|
||||
if (input.localSource) {
|
||||
const snapshot = await readGitWorkspaceSnapshot(input.localSource, false);
|
||||
if (!snapshot) throw new Error("Configured repository folder is not a Git checkout");
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { execFile as execute } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
import { tryWithDirectoryMergeLock } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
||||
import { assertWorkspaceManifestDiskSpace } from "@paperclipai/adapter-utils/workspace-manifest";
|
||||
import type { GitAuthInvocation } from "./git-credentials.js";
|
||||
import { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js";
|
||||
|
||||
const execFile = promisify(execute);
|
||||
const REPOSITORY_BYTES = 2 * 1024 ** 3;
|
||||
const COMPANY_BYTES = 4 * 1024 ** 3;
|
||||
const COMPANY_REPOSITORIES = 64;
|
||||
const hash = (value: string) => createHash("sha256").update(value).digest("hex");
|
||||
|
||||
export interface RepositoryObjectCache {
|
||||
companyDirectory: string;
|
||||
directory: string;
|
||||
key: string;
|
||||
repoUrl: string;
|
||||
}
|
||||
|
||||
export async function resolveRepositoryObjectCache(companyDirectory: string, repoUrl: string, auth: GitAuthInvocation | null): Promise<RepositoryObjectCache | null> {
|
||||
try {
|
||||
const remote = new URL(repoUrl);
|
||||
if (remote.username || remote.password || remote.search || remote.hash) return null;
|
||||
const parent = path.dirname(companyDirectory);
|
||||
await fs.mkdir(parent, { recursive: true, mode: 0o700 });
|
||||
for (const directory of [parent, companyDirectory]) {
|
||||
if (directory === companyDirectory) await fs.mkdir(directory, { mode: 0o700 }).catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code !== "EEXIST") throw error;
|
||||
});
|
||||
const stat = await fs.lstat(directory);
|
||||
if (!stat.isDirectory() || stat.isSymbolicLink()) return null;
|
||||
}
|
||||
companyDirectory = await fs.realpath(companyDirectory);
|
||||
const scope = auth ? [auth.source, auth.secretName, auth.configArgs, Object.entries(auth.env).sort(([a], [b]) => a.localeCompare(b))] : null;
|
||||
const key = hash(JSON.stringify([repoUrl, scope]));
|
||||
return { companyDirectory, directory: path.join(companyDirectory, key), key, repoUrl };
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
export async function readRepositoryObjectCache(cache: RepositoryObjectCache): Promise<string | null> {
|
||||
try {
|
||||
for (const directory of [cache.companyDirectory, cache.directory]) {
|
||||
const stat = await fs.lstat(directory);
|
||||
if (!stat.isDirectory() || stat.isSymbolicLink()) return null;
|
||||
}
|
||||
if (await fs.realpath(cache.companyDirectory) !== cache.companyDirectory) return null;
|
||||
const objects = await fs.lstat(path.join(cache.directory, "objects"));
|
||||
if (!objects.isDirectory() || objects.isSymbolicLink()) return null;
|
||||
if (await fs.lstat(path.join(cache.directory, "objects/info/alternates")).then(() => true, () => false)) return null;
|
||||
const receiptPath = path.join(cache.directory, "paperclip-object-cache.json");
|
||||
const stat = await fs.lstat(receiptPath);
|
||||
if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 4096) return null;
|
||||
const receipt = JSON.parse(await fs.readFile(receiptPath, "utf8"));
|
||||
if (receipt.version !== 1 || receipt.key !== cache.key) return null;
|
||||
const options = { timeout: 10000, maxBuffer: 16384, env: sanitizeRuntimeServiceBaseEnv(process.env) };
|
||||
const bare = await execFile("git", ["-C", cache.directory, "rev-parse", "--is-bare-repository"], options);
|
||||
const origin = await execFile("git", ["-C", cache.directory, "config", "--local", "--null", "--get-all", "remote.origin.url"], options);
|
||||
if (bare.stdout.trim() !== "true" || origin.stdout !== `${cache.repoUrl}\0`) return null;
|
||||
return cache.directory;
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
async function storageBytes(directory: string, ceiling: number, budget: { entries: number; deadline: number }): Promise<number> {
|
||||
let bytes = 0;
|
||||
for await (const entry of await fs.opendir(directory)) {
|
||||
if (--budget.entries < 0 || performance.now() > budget.deadline) throw new Error("Repository object cache work budget exhausted");
|
||||
const target = path.join(directory, entry.name);
|
||||
const stat = await fs.lstat(target);
|
||||
if (stat.isSymbolicLink()) throw new Error("Invalid repository object cache entry");
|
||||
bytes += stat.isDirectory() ? await storageBytes(target, ceiling - bytes, budget) : stat.size;
|
||||
if (bytes > ceiling) throw new Error("Repository object cache capacity exhausted");
|
||||
}
|
||||
return bytes;
|
||||
}
|
||||
|
||||
export async function publishRepositoryObjectCache(cache: RepositoryObjectCache, checkout: string): Promise<void> {
|
||||
try {
|
||||
if (await readRepositoryObjectCache(cache)) return;
|
||||
await tryWithDirectoryMergeLock(cache.companyDirectory, async () => {
|
||||
const company = await fs.lstat(cache.companyDirectory);
|
||||
if (!company.isDirectory() || company.isSymbolicLink() || await fs.realpath(cache.companyDirectory) !== cache.companyDirectory) return;
|
||||
const budget = { entries: 4096, deadline: performance.now() + 5000 };
|
||||
let repositories = 0, bytes = 0;
|
||||
for await (const entry of await fs.opendir(cache.companyDirectory)) {
|
||||
if (++repositories >= COMPANY_REPOSITORIES) return;
|
||||
const directory = path.join(cache.companyDirectory, entry.name);
|
||||
const stat = await fs.lstat(directory);
|
||||
if (!stat.isDirectory() || stat.isSymbolicLink()) return;
|
||||
bytes += await storageBytes(directory, COMPANY_BYTES - bytes, budget);
|
||||
}
|
||||
if (await fs.lstat(cache.directory).then(() => true, () => false)) return;
|
||||
const incoming = await storageBytes(path.join(checkout, ".git"), REPOSITORY_BYTES, budget) + 65536;
|
||||
if (bytes + incoming > COMPANY_BYTES) return;
|
||||
assertWorkspaceManifestDiskSpace(cache.companyDirectory, incoming);
|
||||
const temporary = path.join(cache.companyDirectory, `.pending-${randomUUID()}`);
|
||||
try {
|
||||
const options = () => ({ timeout: Math.max(1, Math.ceil(budget.deadline - performance.now())), maxBuffer: 16384, env: sanitizeRuntimeServiceBaseEnv(process.env) });
|
||||
await execFile("git", ["clone", "--bare", "--no-hardlinks", "--", checkout, temporary], options());
|
||||
await execFile("git", ["-C", temporary, "remote", "set-url", "origin", cache.repoUrl], options());
|
||||
await fs.writeFile(path.join(temporary, "paperclip-object-cache.json"), JSON.stringify({ version: 1, key: cache.key }));
|
||||
const actual = await storageBytes(temporary, REPOSITORY_BYTES, budget);
|
||||
if (bytes + actual > COMPANY_BYTES) return;
|
||||
await fs.rename(temporary, cache.directory);
|
||||
} finally { await fs.rm(temporary, { recursive: true, force: true }); }
|
||||
});
|
||||
} catch {}
|
||||
}
|
||||
Reference in new issue
Block a user