perf: reduce fresh workspace clone and sandbox staging waits (#16487)

## Thinking Path

> - Paperclip manages AI agents and their work.
> - Each run needs a private workspace and safe persistence.
> - Fresh isolated repository sources download upstream Git history
again.
> - Sandbox staging also verifies an optional seed cache before it
uploads the workspace.
> - That verification holds a company lock during a full extraction and
hash scan.
> - This pull request reuses pristine Git objects through an
authenticated remote clone.
> - It also overlaps seed verification with upload and moves
verification outside the publication lock.
> - The benefit is less repeated transfer and less serial preparation
work, with the same task isolation and recovery contract.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Fresh task-private repository preparation and inbound sandbox workspace
staging.

**Current behavior**

Each isolated remote source clones the full upstream history. Optional
seed verification then extracts and hashes another full tree before
upload. One workspace's verification can hold the company cache lock
while other workspaces wait.

**Proposed behavior**

Use a company and credential scoped cache of pristine upstream objects
for GitHub HTTPS sources. Each task still clones from the current
remote. It disconnects from the cache before publication. Run seed
verification alongside upload. Hold the company lock only for final
capacity checks and publication.

**Reason and benefit**

These operations lie on the workspace preparation path. Removing
repeated Git pack transfer and serial verification can reduce startup
time. Local measurements below cover these boundaries. They do not
establish hosted agent latency.

Related: #16425, #16454, #16398, #16461, #16468, #16473. The cache
inspection and durable folder proposal in #13376 covers separate product
work.

## What Changed

- Add a pristine upstream Git object cache for fresh isolated GitHub
HTTPS sources. Key it by company, repository and current credential
fingerprint. Do not store credentials.
- Keep online clone authorization and current remote refs. Use
`--reference-if-able --dissociate` so task objects remain independent. A
reference failure retries once online without the cache and resolves
credentials again.
- Cap cache storage at 64 entries and 4 GiB per company, with 2 GiB per
entry. Skip contended publication. Bound priming by a cooperative
five-second budget and a 4,096-entry scan allowance. Keep the existing
free-space reserve.
- Overlap optional seed verification and required workspace upload.
Settle both before returning or deleting temporary files. Recheck quotas
and company directory identity under the publication lock.
- Keep the independent fresh upstream reproducibility proof, task
commits, dirty source copies, ignored files, nested repositories and
durable recovery behavior.
- Suppress optional Git server-info generation during dissociation so
generated storage passes the existing safe-expiration checks. This uses
[the supported Git
setting](https://git-scm.com/docs/git-config/2.43.0#Documentation/git-config.txt-repackupdateServerInfo).
- Add bounded timing phases, regression coverage and two repeatable
local benchmarks. No first-party Telemetry or dependency changes.

## Verification

- `pnpm -r typecheck`: passed.
- `pnpm build`: passed.
- Affected adapter suites: 240 tests passed. Native workspace
persistence/history and tracing: 33 tests passed. Server
checkout/isolation/tracing batch: 53 tests passed. Final object-cache
authority, path and capacity suite: 12 tests passed. These batches
overlap.
- Cache and expiration/restoration lifecycle batch: 34 tests passed. A
direct generated-storage check covers cached clones.
- Real HTTP Git fixture: a matching cache does not bypass revoked remote
access. Other fixtures cover current refs/tags, hidden cached commits,
independent objects after cache removal, corruption fallback,
credential/company scope, quota limits and publication contention.
- Initial restricted runs hit SQLite write and Unix socket permission
failures. The adapter batch passed with disposable writable instance
state and socket access. The first typecheck rejected the test use of
`Promise.withResolvers`; typed deferred gates now use the supported
library contract.
- Repository benchmark on final head
`9047368bf3b7fa2061e00d698b8b51cf4ae1da32`: five alternating samples,
real local `file://` Git transport, synthetic 32 MiB history, no
injected latency. Median 1143.8 ms to 240.5 ms; ranges 1085.7–1486.6 ms
and 188.0–254.8 ms. Received Git pack bytes fell from 33,569,788 to
zero. Initial cache priming took 1208.4 ms and is reported separately.
- Sandbox benchmark, five alternating samples per mode on 10,000 files
with real packing, extraction and verification: with a controlled
3-second transfer delay, median 9.450 s to 7.620 s (ranges 9.280–12.115
s and 7.296–7.983 s). With no injected delay, median 6.808 s to 6.450 s
(ranges 6.255–7.029 s and 5.709–6.902 s). The no-delay ranges overlap;
local copying competes with verification for disk. Both modes verify
final tree contents, ignored-file exclusion and Git HEAD. These are
synthetic local fixtures, not hosted agent runs.
- Reproduce with `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-repository-object-cache.ts` and `pnpm --filter
@paperclipai/server exec tsx ../scripts/benchmark-workspace-staging.ts
--baseline-ref=307eb16c0c --transfer-delay-ms=3000`.
- Prior head `af766bc820` received Greptile 5/5 with no findings. Its
E2E shard 7 failed in `plan-task-fixture` cleanup: company deletion
raced an `activity_log` foreign key. That fixture has no provider run or
Git workspace. The failure is retained here; final-head CI must pass.
The corrected corruption fixture now forces a pack and verifies a real
damaged archive. Current head `9047368bf3b7fa2061e00d698b8b51cf4ae1da32`
has Greptile 5/5 with no review threads or findings. Final head has 55
passing checks and two optional Storybook skips. [The first
workspace-shard
attempt](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116655485350)
passed all 1,778 cases, then reported a late `EPIPE` from unchanged
`server-utils.ts` while the ACP reuse test was active. That case mocks
workspace preparation and passed locally in isolation. [Its single retry
on the same
commit](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116659728034)
passed, as did the dependent verification gate. The earlier E2E cleanup
fixture passed on this final head. Both initial failures remain recorded
here. A full local suite was not duplicated; the affected boundaries
were run locally and the complete suite runs in CI.

## Risks

- First cache population adds local copying. Cold clones, credential
rotation, capacity exhaustion and a corrupt cache retain the online
clone path. No eviction is added. Cached data is bounded, but many
companies can each consume their allowance.
- Individual filesystem calls and cleanup can exceed the cooperative
priming budget. Failed reference clones make one additional online
attempt.
- Parallel verification and upload compete for disk and CPU. The local
no-delay benchmark shows a smaller gain than a slow-transfer case.
Verification still checks contents before cache publication, and
required upload failures remain failures.
- Local fixtures do not prove hosted throughput or the 15–30-second
end-to-end target. The fresh checkout still writes its working tree, and
model startup, copyback and other preparation costs remain.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and context
window are not exposed in this session. Used reasoning, source editing,
shell tools and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal task
identifiers
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/DEVELOPING.md

@@ -686,6 +686,35 @@ per-query results and latency measurements; scale measurements are opt-in.
 
 ### Repository workspace admission benchmark
 
+Measure a fresh checkout with and without the repository object cache:
+
+```sh
+pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-repository-object-cache.ts
+```
+
+This uses a synthetic 32 MiB repository and the real `file://` Git transport.
+It reports cache priming separately, then alternates five fresh checkouts per
+mode. Git's pack trace records received bytes. Each checkout has independent
+objects and passes `git fsck`. There is no injected latency or hosted network
+measurement.
+
+Measure cold sandbox staging with a disposable 10,000-file Git repository:
+
+```sh
+pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-workspace-staging.ts --baseline-ref=origin/master
+pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-workspace-staging.ts --baseline-ref=origin/master --transfer-delay-ms=3000
+```
+
+The benchmark alternates the baseline and current implementations. Both pack,
+verify, copy and extract real archives, then compare the resulting tree and Git
+HEAD. It retains dirty, deleted and untracked-file semantics and excludes ignored
+files. The transport is a local copy. The optional delay models a slow transfer;
+it does not measure hosted network or agent latency. The report contains every
+sample, whole-staging wall time, archive bytes and phase durations. Seed
+verification and upload can overlap, so do not sum their durations. The script
+temporarily imports baseline source beside the current source and removes it on
+exit. Run without other repository edits or heavy I/O for a useful comparison.
+
 Measure the real repository preparation service with a disposable embedded
 PostgreSQL database and synthetic history, without invoking an agent:
 
@@ -1104,6 +1133,35 @@ When effective run config changes, Paperclip may intentionally skip a saved adap
 
 An explicit `reuse_existing` Git workspace keeps its recorded source repository, including after its project-workspace row is deleted or the project primary changes. The server verifies the original company, project, repository origin, and worktree registration before reuse. It does not attach a deleted association to the new primary. A conflicting explicit project-workspace selection, missing source identity, or an unverified legacy local checkout blocks the run. Restore the original source. Before intentionally clearing the existing-workspace binding to choose a new repository, review and preserve its retained work. Missing worktrees may be reconstructed only from the verified original source; current project defaults are not a fallback.
 
+### Isolated repository object cache
+
+Fresh task-private GitHub HTTPS sources use a pristine upstream object cache at
+`<instance root>/repository-object-cache/<company>/<opaque scope>`. The scope
+includes the repository URL and current credential fingerprint. Credentials
+are not stored in the cache. Other remote providers retain the ordinary clone
+path. Existing task sources and local dirty source copies keep their current
+behavior.
+
+Every new checkout still contacts the remote with current Git credentials.
+`git clone --reference-if-able --dissociate` reuses objects but selects the
+remote's current refs and creates independent storage. An unavailable or denied
+remote cannot fall back to cached files. A failed reference clone retries once
+without the cache and resolves credentials again. Task commits and dirty files
+never populate this cache. Reproducibility checks continue to use a fresh upstream object store for
+their independent proof; they do not use this cache.
+
+Cache publication is optional and never waits for another publisher. It keeps
+at most 64 entries and 4 GiB per company, with 2 GiB per entry. Temporary cache
+construction can use one additional entry's space; the existing manifest free
+space reserve applies. Priming uses a five-second cooperative work budget and
+a 4,096-entry scan allowance; individual filesystem calls and cleanup can
+exceed that time. Existing entries and abandoned pending entries count
+against capacity. A full, corrupt or unavailable cache leaves ordinary cloning
+available. No cache eviction is performed. Credential rotation selects another
+scope and can reduce reuse after capacity is exhausted. There is no link from
+an admitted task checkout back to this cache, so cache removal cannot remove
+task Git objects.
+
 ## Workspace Git Scan Protection
 
 Paperclip applies one process-wide scheduler to expensive host-side workspace Git enumeration, including changed-file browsing, runtime/finalization cleanliness guards, and adapter sandbox-sync snapshots. The scheduler defaults to two active scans and a bounded queue of 32. Identical buffered scans of the same canonical worktree share one subprocess, while successful changed-file listings are cached for 10 seconds. Streaming snapshot scans have caller-owned sinks, so they use separate jobs in the same queue and are never cached or coalesced. Correctness-sensitive runtime guards bypass the result cache.

doc/observability.md

@@ -289,7 +289,7 @@ attempt or owning phase:
 | `workspace.sources.*` | Source inventory queries, canonicalization, one filesystem alias scan, repository scan, and configured project scan. |
 | `workspace.readers`, `workspace.readers.*`, `workspace.access.*` | Retained-reader groups, identity, trust, project and source authorization, privacy and inventory reads. |
 | `workspace.provenance`, `workspace.notification`, `workspace.write` | Preserved ownership, notification statements, and record writes. |
-| `workspace.materialize.*`, `workspace.checkout.*` | Private source-directory setup and identity verification, existing checkout lookup, Git credential resolution, clone, requested-ref checkout, and atomic publication. The enclosing `heartbeat.prepare.workspace_source_materialize` includes the complete source-materialization path. |
+| `workspace.materialize.*`, `workspace.checkout.*` | Private source-directory setup and identity verification, existing checkout lookup, Git credential resolution, clone, optional pristine-object cache lookup/publication (`cache_lookup`, `cache_publish`), requested-ref checkout, and atomic publication. The enclosing `heartbeat.prepare.workspace_source_materialize` includes the complete source-materialization path. |
 | `runtime_selection.*` | Completion contracts, session/recovery reads, profiles, interaction materialization, backend discovery, bootstrap, run/coordinator writes, transaction entry/settlement, and workspace synchronization. |
 | `runtime_context.instructions`, `.skills`, `.mcp` | The three concurrent runtime-context branches. Their durations overlap. |
 | `runtime_context.asset_*` | Source reads, manifest hashing, cache lookup, cold materialization, and exact file verification on cold and warm paths. |

doc/plans/2026-10-09-task-workspaces.md

@@ -71,7 +71,7 @@ Sparse filesystem checkpoints use disk-backed full manifests, changed bytes/dele
 
 A returned sparse snapshot must describe a coherent tree before any host mutation. Every nested entry requires an immediate parent marked as a directory in that same returned snapshot, including baseline-unchanged files. Checking every directory also verifies recursive ancestry. Top-level entries use the implicit root. Invalid trees fail without overwriting retained host files and keep recovery evidence. The check uses the existing indexed manifest interface; it adds no host-path scans or unbounded in-memory inventory.
 
-Seed recovery deliberately uses **no delta chain** in this implementation. An unchanged generation reuses its verified content-addressed immutable full seed; a changed generation creates a fresh full seed. Sparse output transfer is separate from seed recovery. This revises the sketch's 16-delta compaction proposal: composing recovery generations would add a new failure surface before the simpler cache has been qualified. Existing references retain old seeds; no automatic eviction/deletion policy is introduced. Optional cache publication is capped at 16 generations/1 GiB per workspace and 256 generations/256 workspace directories/8 GiB per company. A company-scoped filesystem lock serializes quota checks and publication across workspace roots. Archive and receipt bytes, corrupt entries and abandoned pending generations count against capacity; publication also preserves the manifest storage free-space reserve (256 MiB by default), accounting for a full-copy fallback. Saturation or cache failure skips publication and keeps the already-written per-run recovery seed. Existing over-capacity caches stop growing without deleting admitted evidence. This deliberately trades cache hit rate after saturation for bounded additional storage and unchanged recovery semantics; no global cross-company quota is added. Changed input generations still pay full snapshot creation. New cache entries are reconstructed and checked against the admitted baseline before publication, so concurrent writes cannot label an archive as the wrong generation; this adds cold verification work. A corrupted cache entry remains a miss and falls back to a fresh seed; automatic cache repair and garbage collection are deferred. The existing Git seed contract captures the selected branch/HEAD, not unrelated refs.
+Seed recovery deliberately uses **no delta chain** in this implementation. An unchanged generation reuses its verified content-addressed immutable full seed; a changed generation creates a fresh full seed. Sparse output transfer is separate from seed recovery. This revises the sketch's 16-delta compaction proposal: composing recovery generations would add a new failure surface before the simpler cache has been qualified. Existing references retain old seeds; no automatic eviction/deletion policy is introduced. Optional cache publication is capped at 16 generations/1 GiB per workspace and 256 generations/256 workspace directories/8 GiB per company. A company-scoped filesystem lock serializes final quota checks and publication across workspace roots. Optional seed reconstruction and baseline verification run outside that lock, alongside inbound upload. Preparation settles both operations before returning or removing temporary archives. Publication rechecks capacity and directory identity under the lock after verification; a competing publication can consume the remaining capacity and cause this optional cache write to be skipped. Archive and receipt bytes, corrupt entries and abandoned pending generations count against capacity; publication also preserves the manifest storage free-space reserve (256 MiB by default), accounting for a full-copy fallback. Saturation or cache failure skips publication and keeps the already-written per-run recovery seed. Existing over-capacity caches stop growing without deleting admitted evidence. This deliberately trades cache hit rate after saturation for bounded additional storage and unchanged recovery semantics; no global cross-company quota is added. Changed input generations still pay full snapshot creation. New cache entries are reconstructed and checked against the admitted baseline before publication, so concurrent writes cannot label an archive as the wrong generation; this adds cold verification work. A corrupted cache entry remains a miss and falls back to a fresh seed; automatic cache repair and garbage collection are deferred. The existing Git seed contract captures the selected branch/HEAD, not unrelated refs.
 
 Ordinary settlement remains host-durable before successful completion. Preserve export-only recovery and exact-source lease retention. Preserve the specific unsafe-export omission exception and its run-log-only diagnostic. Do not change global merge conflict semantics as a side effect.