perf: reduce fresh workspace clone and sandbox staging waits (#16487)

## Thinking Path

> - Paperclip manages AI agents and their work.
> - Each run needs a private workspace and safe persistence.
> - Fresh isolated repository sources download upstream Git history
again.
> - Sandbox staging also verifies an optional seed cache before it
uploads the workspace.
> - That verification holds a company lock during a full extraction and
hash scan.
> - This pull request reuses pristine Git objects through an
authenticated remote clone.
> - It also overlaps seed verification with upload and moves
verification outside the publication lock.
> - The benefit is less repeated transfer and less serial preparation
work, with the same task isolation and recovery contract.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Fresh task-private repository preparation and inbound sandbox workspace
staging.

**Current behavior**

Each isolated remote source clones the full upstream history. Optional
seed verification then extracts and hashes another full tree before
upload. One workspace's verification can hold the company cache lock
while other workspaces wait.

**Proposed behavior**

Use a company and credential scoped cache of pristine upstream objects
for GitHub HTTPS sources. Each task still clones from the current
remote. It disconnects from the cache before publication. Run seed
verification alongside upload. Hold the company lock only for final
capacity checks and publication.

**Reason and benefit**

These operations lie on the workspace preparation path. Removing
repeated Git pack transfer and serial verification can reduce startup
time. Local measurements below cover these boundaries. They do not
establish hosted agent latency.

Related: #16425, #16454, #16398, #16461, #16468, #16473. The cache
inspection and durable folder proposal in #13376 covers separate product
work.

## What Changed

- Add a pristine upstream Git object cache for fresh isolated GitHub
HTTPS sources. Key it by company, repository and current credential
fingerprint. Do not store credentials.
- Keep online clone authorization and current remote refs. Use
`--reference-if-able --dissociate` so task objects remain independent. A
reference failure retries once online without the cache and resolves
credentials again.
- Cap cache storage at 64 entries and 4 GiB per company, with 2 GiB per
entry. Skip contended publication. Bound priming by a cooperative
five-second budget and a 4,096-entry scan allowance. Keep the existing
free-space reserve.
- Overlap optional seed verification and required workspace upload.
Settle both before returning or deleting temporary files. Recheck quotas
and company directory identity under the publication lock.
- Keep the independent fresh upstream reproducibility proof, task
commits, dirty source copies, ignored files, nested repositories and
durable recovery behavior.
- Suppress optional Git server-info generation during dissociation so
generated storage passes the existing safe-expiration checks. This uses
[the supported Git
setting](https://git-scm.com/docs/git-config/2.43.0#Documentation/git-config.txt-repackupdateServerInfo).
- Add bounded timing phases, regression coverage and two repeatable
local benchmarks. No first-party Telemetry or dependency changes.

## Verification

- `pnpm -r typecheck`: passed.
- `pnpm build`: passed.
- Affected adapter suites: 240 tests passed. Native workspace
persistence/history and tracing: 33 tests passed. Server
checkout/isolation/tracing batch: 53 tests passed. Final object-cache
authority, path and capacity suite: 12 tests passed. These batches
overlap.
- Cache and expiration/restoration lifecycle batch: 34 tests passed. A
direct generated-storage check covers cached clones.
- Real HTTP Git fixture: a matching cache does not bypass revoked remote
access. Other fixtures cover current refs/tags, hidden cached commits,
independent objects after cache removal, corruption fallback,
credential/company scope, quota limits and publication contention.
- Initial restricted runs hit SQLite write and Unix socket permission
failures. The adapter batch passed with disposable writable instance
state and socket access. The first typecheck rejected the test use of
`Promise.withResolvers`; typed deferred gates now use the supported
library contract.
- Repository benchmark on final head
`9047368bf3b7fa2061e00d698b8b51cf4ae1da32`: five alternating samples,
real local `file://` Git transport, synthetic 32 MiB history, no
injected latency. Median 1143.8 ms to 240.5 ms; ranges 1085.7–1486.6 ms
and 188.0–254.8 ms. Received Git pack bytes fell from 33,569,788 to
zero. Initial cache priming took 1208.4 ms and is reported separately.
- Sandbox benchmark, five alternating samples per mode on 10,000 files
with real packing, extraction and verification: with a controlled
3-second transfer delay, median 9.450 s to 7.620 s (ranges 9.280–12.115
s and 7.296–7.983 s). With no injected delay, median 6.808 s to 6.450 s
(ranges 6.255–7.029 s and 5.709–6.902 s). The no-delay ranges overlap;
local copying competes with verification for disk. Both modes verify
final tree contents, ignored-file exclusion and Git HEAD. These are
synthetic local fixtures, not hosted agent runs.
- Reproduce with `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-repository-object-cache.ts` and `pnpm --filter
@paperclipai/server exec tsx ../scripts/benchmark-workspace-staging.ts
--baseline-ref=307eb16c0c --transfer-delay-ms=3000`.
- Prior head `af766bc820` received Greptile 5/5 with no findings. Its
E2E shard 7 failed in `plan-task-fixture` cleanup: company deletion
raced an `activity_log` foreign key. That fixture has no provider run or
Git workspace. The failure is retained here; final-head CI must pass.
The corrected corruption fixture now forces a pack and verifies a real
damaged archive. Current head `9047368bf3b7fa2061e00d698b8b51cf4ae1da32`
has Greptile 5/5 with no review threads or findings. Final head has 55
passing checks and two optional Storybook skips. [The first
workspace-shard
attempt](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116655485350)
passed all 1,778 cases, then reported a late `EPIPE` from unchanged
`server-utils.ts` while the ACP reuse test was active. That case mocks
workspace preparation and passed locally in isolation. [Its single retry
on the same
commit](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116659728034)
passed, as did the dependent verification gate. The earlier E2E cleanup
fixture passed on this final head. Both initial failures remain recorded
here. A full local suite was not duplicated; the affected boundaries
were run locally and the complete suite runs in CI.

## Risks

- First cache population adds local copying. Cold clones, credential
rotation, capacity exhaustion and a corrupt cache retain the online
clone path. No eviction is added. Cached data is bounded, but many
companies can each consume their allowance.
- Individual filesystem calls and cleanup can exceed the cooperative
priming budget. Failed reference clones make one additional online
attempt.
- Parallel verification and upload compete for disk and CPU. The local
no-delay benchmark shows a smaller gain than a slow-transfer case.
Verification still checks contents before cache publication, and
required upload failures remain failures.
- Local fixtures do not prove hosted throughput or the 15–30-second
end-to-end target. The fresh checkout still writes its working tree, and
model startup, copyback and other preparation costs remain.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and context
window are not exposed in this session. Used reasoning, source editing,
shell tools and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal task
identifiers
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

packages/adapter-utils/src/sandbox-managed-runtime.ts

@@ -1640,38 +1640,43 @@ export async function prepareSandboxManagedRuntime(input: {
             workspaceUploadBytes += (await fs.stat(workspaceTarPath)).size;
           });
 
-          if (seedGeneration && !cachedSeed && input.workspaceDurableSeed) {
-            const seed = input.workspaceDurableSeed;
-            await runStepSpan("workspace.seed_publish", () => publishWorkspaceSeedGeneration(input.workspaceSeedCacheDirectory!, seedGeneration,
-              { ...seed, gitArchivePath: gitSnapshot || repositories.length ? seed.gitArchivePath : null }, {
-                companyDirectory: input.workspaceSeedCacheCompanyDirectory,
-                verify: () => runStepSpan("seed.verify", async () => {
-                  const verifiedRoot = path.join(tempDir, "verify-seed");
-                  try {
-                    if (gitSnapshot || repositories.length) await runStepSpan("workspace.seed_extract", () => extractTarballToDirectory({ archivePath: path.join(tempDir, "git-workspace.tar"), localDir: verifiedRoot }));
-                    await runStepSpan("workspace.seed_extract", () => extractTarballToDirectory({ archivePath: path.join(tempDir, "workspace.tar"), localDir: verifiedRoot, exclude: repositoryMetadataExcludes }));
-                    for (const relative of workspacePaths(deletedPaths)) {
-                      let parent = verifiedRoot;
-                      for (const segment of relative.split("/").slice(0, -1)) {
-                        parent = path.join(parent, segment);
-                        const stat = await fs.lstat(parent).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return null; throw error; });
-                        if (stat && !stat.isDirectory()) return false;
+          const publishSeed = async () => {
+            if (seedGeneration && !cachedSeed && input.workspaceDurableSeed) {
+              const seed = input.workspaceDurableSeed;
+              await runStepSpan("workspace.seed_publish", () => publishWorkspaceSeedGeneration(input.workspaceSeedCacheDirectory!, seedGeneration,
+                { ...seed, gitArchivePath: gitSnapshot || repositories.length ? seed.gitArchivePath : null }, {
+                  companyDirectory: input.workspaceSeedCacheCompanyDirectory,
+                  verify: () => runStepSpan("seed.verify", async () => {
+                    const verifiedRoot = path.join(tempDir, "verify-seed");
+                    try {
+                      if (gitSnapshot || repositories.length) await runStepSpan("workspace.seed_extract", () => extractTarballToDirectory({ archivePath: path.join(tempDir, "git-workspace.tar"), localDir: verifiedRoot }));
+                      await runStepSpan("workspace.seed_extract", () => extractTarballToDirectory({ archivePath: path.join(tempDir, "workspace.tar"), localDir: verifiedRoot, exclude: repositoryMetadataExcludes }));
+                      for (const relative of workspacePaths(deletedPaths)) {
+                        let parent = verifiedRoot;
+                        for (const segment of relative.split("/").slice(0, -1)) {
+                          parent = path.join(parent, segment);
+                          const stat = await fs.lstat(parent).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return null; throw error; });
+                          if (stat && !stat.isDirectory()) return false;
+                        }
+                        await fs.rm(path.join(verifiedRoot, relative), { recursive: true, force: true });
                       }
-                      await fs.rm(path.join(verifiedRoot, relative), { recursive: true, force: true });
-                    }
-                    const verified = await runStepSpan("workspace.seed_snapshot", () => captureWorkspaceSnapshot(verifiedRoot, { exclude: baselineSnapshot!.exclude, ignoredPaths: baselineSnapshot!.ignoredPaths, diskBacked: true }));
-                    try { return directorySnapshotSha256(verified) === directorySnapshotSha256(baselineSnapshot!); }
-                    finally { await disposeDirectorySnapshot(verified); }
-                  } finally { await runStepSpan("workspace.seed_cleanup", () => fs.rm(verifiedRoot, { recursive: true, force: true })); }
-                }),
-              }));
+                      const verified = await runStepSpan("workspace.seed_snapshot", () => captureWorkspaceSnapshot(verifiedRoot, { exclude: baselineSnapshot!.exclude, ignoredPaths: baselineSnapshot!.ignoredPaths, diskBacked: true }));
+                      try { return directorySnapshotSha256(verified) === directorySnapshotSha256(baselineSnapshot!); }
+                      finally { await disposeDirectorySnapshot(verified); }
+                    } finally { await runStepSpan("workspace.seed_cleanup", () => fs.rm(verifiedRoot, { recursive: true, force: true })); }
+                  }),
+                }));
+            }
+          };
+          if (!stageWorkspace) {
+            await publishSeed();
+            return;
           }
-          if (!stageWorkspace) return;
 
           // One confined `syncIn` for the whole merged workspace file set. The confine
           // guard covers every mapping BEFORE any bytes upload (fail-closed): a source
           // or target escape in EITHER tar mapping stops the upload of both.
-          await stageConfinedSyncIn({
+          const staging = stageConfinedSyncIn({
             files: workspaceFiles,
             postUploadCommands: workspacePostUploadCommands,
             sourceRoots: [tempDir],
@@ -1680,6 +1685,8 @@ export async function prepareSandboxManagedRuntime(input: {
             statusPhase: "config_sync",
             progressBytes: workspaceUploadBytes,
           });
+          const [staged] = await Promise.allSettled([staging, publishSeed()]);
+          if (staged.status === "rejected") throw staged.reason;
         }),
       );
     }

packages/adapter-utils/src/sandbox-managed-seed.test.ts

@@ -8,6 +8,12 @@ import { createTarballFromDirectory, prepareSandboxManagedRuntime, type SandboxS
 import { captureDirectorySnapshot } from "./workspace-restore-merge.js";
 
 const execute = promisify(execFile);
+function deferred<T>() {
+  let resolve!: (value: T | PromiseLike<T>) => void;
+  const promise = new Promise<T>(done => { resolve = done; });
+  return { promise, resolve };
+}
+
 const roots: string[] = [];
 afterEach(async () => {
   vi.unstubAllEnvs();
@@ -48,9 +54,6 @@ it.each([undefined, "all"] as const)("keeps nested Git metadata sanitized in col
       for (const file of operation.files) {
         await fs.mkdir(path.dirname(file.targetPath), { recursive: true });
         await fs.cp(file.sourcePath, file.targetPath, { recursive: true, verbatimSymlinks: true });
-        if (file.sourcePath.endsWith("/workspace.tar")) {
-          await expect(fs.stat(path.join(path.dirname(file.sourcePath), "verify-seed"))).rejects.toMatchObject({ code: "ENOENT" });
-        }
       }
       for (const command of operation.postUploadCommands ?? []) await run(command.command);
     }
@@ -111,3 +114,45 @@ it.each([undefined, "all"] as const)("keeps nested Git metadata sanitized in col
   }
   await recovered.cleanupWorkspaceSnapshot(); await cached.cleanupWorkspaceSnapshot();
 }, 30000);
+
+it.each([false, true])("overlaps seed verification and upload and drains both before %s failure settles", async (failUpload) => {
+  const temp = await fs.mkdtemp(path.join(os.tmpdir(), "managed-seed-overlap-")); roots.push(temp);
+  vi.stubEnv("PAPERCLIP_HOME", path.join(temp, "home"));
+  const host = path.join(temp, "host"); await fs.mkdir(host);
+  await fs.writeFile(path.join(host, "work.txt"), "original\n");
+  const entered = deferred<void>();
+  const release = deferred<void>();
+  let verificationFinished = false, settled = false;
+  const preparing = prepareSandboxManagedRuntime({
+    spec: { provider: "test", sandboxId: "test", remoteCwd: path.join(temp, "remote"), timeoutMs: 30000, apiKey: null, transport: "sandbox" },
+    adapterKey: "test", workspaceLocalDir: host, workspaceFileMode: "all",
+    workspaceSeedCacheDirectory: path.join(temp, "cache"),
+    workspaceDurableSeed: { workspaceArchivePath: path.join(temp, "seed.tar") },
+    runtimeSpan: async (name, work) => {
+      if (name !== "seed.verify") return work();
+      entered.resolve();
+      await release.promise;
+      try { return await work(); } finally { verificationFinished = true; }
+    },
+    client: {
+      makeDir: async () => {}, run: async () => {}, writeFile: async () => {},
+      readFile: async () => Buffer.alloc(0), listFiles: async () => [], remove: async () => {},
+      syncIn: async () => {
+        await entered.promise;
+        expect(verificationFinished).toBe(false);
+        if (failUpload) throw new Error("upload failed");
+        return { operations: [] };
+      },
+    },
+  });
+  const result = preparing.then(runtime => { settled = true; return { runtime }; }, error => { settled = true; return { error }; });
+  try {
+    await entered.promise;
+    await new Promise(resolve => setImmediate(resolve));
+    expect(settled).toBe(false);
+  } finally { release.resolve(); }
+  const outcome = await result;
+  expect(verificationFinished).toBe(true);
+  if ("error" in outcome) { expect(failUpload).toBe(true); expect(outcome.error.message).toBe("upload failed"); }
+  else { expect(failUpload).toBe(false); await outcome.runtime.cleanupWorkspaceSnapshot(); }
+});

packages/adapter-utils/src/workspace-restore-merge.ts

@@ -556,6 +556,16 @@ export async function withDirectoryPublicationLock<T>(
   }, env);
 }
 
+export async function tryWithDirectoryMergeLock<T>(targetDir: string, fn: () => Promise<T>): Promise<T | undefined> {
+  const canonicalTargetDir = await fs.realpath(targetDir);
+  try {
+    return await withCanonicalDirectoryMergeLock(canonicalTargetDir, fn, process.env, undefined, 0);
+  } catch (error) {
+    if ((error as NodeJS.ErrnoException).code === WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE) return undefined;
+    throw error;
+  }
+}
+
 async function withCanonicalDirectoryMergeLock<T>(
   canonicalTargetDir: string,
   fn: (canonicalTargetDir: string) => Promise<T>,

packages/adapter-utils/src/workspace-seed-cache.test.ts

@@ -5,6 +5,12 @@ import path from "node:path";
 import { afterEach, expect, it, vi } from "vitest";
 import { publishWorkspaceSeedGeneration, readWorkspaceSeedGeneration } from "./workspace-seed-cache.js";
 
+function deferred<T>() {
+  let resolve!: (value: T | PromiseLike<T>) => void;
+  const promise = new Promise<T>(done => { resolve = done; });
+  return { promise, resolve };
+}
+
 const roots: string[] = [];
 afterEach(async () => {
   vi.restoreAllMocks(); vi.unstubAllEnvs();
@@ -112,3 +118,48 @@ it("treats optional verification storage failure as a cache miss", async () => {
   expect(await fs.readdir(f.company)).toEqual([]);
   expect(await fs.readFile(f.archive, "utf8")).toBe("run recovery bytes");
 });
+
+it("does not hold the company publication lock during slow verification", async () => {
+  const f = await fixture();
+  const entered = deferred<void>();
+  const release = deferred<boolean>();
+  const slow = publishWorkspaceSeedGeneration(f.workspace, generation(1), f.seed, {
+    companyDirectory: f.company, verify: () => { entered.resolve(); return release.promise; },
+  });
+  try {
+    await entered.promise;
+    expect(await publishWorkspaceSeedGeneration(path.join(f.company, "second"), generation(2), f.seed,
+      { companyDirectory: f.company })).toBe(true);
+    expect(await fs.readdir(f.company)).toEqual(["second"]);
+  } finally { release.resolve(true); await slow; }
+  expect(await readWorkspaceSeedGeneration(f.workspace, generation(1))).not.toBeNull();
+});
+
+it("rechecks aggregate capacity after concurrent verifications finish", async () => {
+  const f = await fixture();
+  const release = deferred<boolean>();
+  const entered = vi.fn(() => release.promise);
+  const publications = ["first", "second"].map((name, index) => publishWorkspaceSeedGeneration(
+    path.join(f.company, name), generation(index), f.seed,
+    { companyDirectory: f.company, limits: { companyGenerations: 1 }, verify: entered },
+  ));
+  try { await vi.waitFor(() => expect(entered).toHaveBeenCalledTimes(2)); }
+  finally { release.resolve(true); }
+  expect((await Promise.all(publications)).sort()).toEqual([false, true]);
+  expect(await fs.readdir(f.company)).toHaveLength(1);
+  expect(await fs.readFile(f.archive, "utf8")).toBe("run recovery bytes");
+});
+
+it("rejects a company directory replaced during verification", async () => {
+  const f = await fixture();
+  const replacement = path.join(f.root, "other-company"); await fs.mkdir(replacement);
+  expect(await publishWorkspaceSeedGeneration(f.workspace, generation(1), f.seed, {
+    companyDirectory: f.company, verify: async () => {
+      await fs.rename(f.company, `${f.company}-retained`);
+      await fs.symlink(replacement, f.company);
+      return true;
+    },
+  })).toBe(false);
+  expect(await fs.readdir(replacement)).toEqual([]);
+  expect(await fs.readFile(f.archive, "utf8")).toBe("run recovery bytes");
+});

packages/adapter-utils/src/workspace-seed-cache.ts

@@ -83,7 +83,10 @@ export async function publishWorkspaceSeedGeneration(
     await fs.mkdir(company, { recursive: true, mode: 0o700 });
     const companyStat = await fs.lstat(company);
     if (!companyStat.isDirectory() || companyStat.isSymbolicLink()) return false;
-    return await withDirectoryMergeLock(company, async () => {
+    const capacity = async (): Promise<"existing" | "available" | false> => {
+      const currentCompany = await fs.lstat(company);
+      if (!currentCompany.isDirectory() || currentCompany.isSymbolicLink()
+        || currentCompany.dev !== companyStat.dev || currentCompany.ino !== companyStat.ino) return false;
       let totalGenerations = 0, totalBytes = 0, workspaces = 0;
       let workspace = { generations: 0, bytes: 0 };
       let workspaceExists = false;
@@ -99,7 +102,7 @@ export async function publishWorkspaceSeedGeneration(
         if (totalGenerations > limits.companyGenerations || totalBytes > limits.companyBytes) return false;
         if (directory === root) { workspace = usage; workspaceExists = true; }
       }
-      if (workspaceExists && await fs.lstat(path.join(root, generation)).then(() => true, () => false)) return true;
+      if (workspaceExists && await fs.lstat(path.join(root, generation)).then(() => true, () => false)) return "existing";
       let incomingBytes = 4096;
       for (const archive of [seed.workspaceArchivePath, seed.gitArchivePath]) {
         if (!archive) continue;
@@ -111,7 +114,14 @@ export async function publishWorkspaceSeedGeneration(
         || totalGenerations + 1 > limits.companyGenerations || totalBytes + incomingBytes > limits.companyBytes
         || workspaces + (workspaceExists ? 0 : 1) > limits.companyWorkspaces) return false;
       assertWorkspaceManifestDiskSpace(company, incomingBytes);
-      if (options.verify && !await options.verify()) return false;
+      return "available";
+    };
+    const initialCapacity = await capacity();
+    if (initialCapacity !== "available") return initialCapacity === "existing";
+    if (options.verify && !await options.verify()) return false;
+    return await withDirectoryMergeLock(company, async () => {
+      const currentCapacity = await capacity();
+      if (currentCapacity !== "available") return currentCapacity === "existing";
       await fs.mkdir(root, { recursive: true, mode: 0o700 });
       const temporary = path.join(root, `.pending-${randomUUID()}`);
       await fs.mkdir(temporary, { mode: 0o700 });