perf: avoid flushing separate Git cache seed archives (#18308)

## Thinking Path

> - Paperclip manages work done by AI agents.
> - Native sandbox runs prepare a workspace before the agent can
respond.
> - A shared seed cache avoids repeated Git copies and archive
preparation.
> - Cache publication also flushes a branch-neutral Git seed that is
separate from recovery archives.
> - This cache can be rebuilt. Each reader checks the archive bytes
before reuse.
> - This change omits that separate cache file flush and measures the
remaining preparation gaps.
> - The benefit is less work before agent execution and clearer timing
evidence.

## Linked Issues or Issue Description

Refs #18240.

**What existing behavior does this improve?**

Workspace seed cache publication and latency diagnostics for native
sandbox preparation.

**Current behavior**

Publication flushes both cache archives to disk under the company lock.
Slow storage can delay the publishing run and cache readers. A cache hit
also combines private copying, hashing, and last-used bookkeeping in one
span. Early controller preparation has gaps before environment
selection.

**Proposed behavior**

Keep the existing flush for recovery-backed workspace and Git archives.
Omit the flush only for a separate branch-neutral Git cache archive.
Reject an alternate cache file that is the recovery file or its
hardlink. Keep archive checksums, private copies, atomic publication,
quotas, and company locking. Measure cache read steps and the existing
early preparation operations.

## What Changed

- Skip the flush only for the separate branch-neutral Git seed used by
the company cache.
- Preserve the existing flush for recovery-backed archives, including
hardlink storage. Reject cache-only files that alias recovery bytes
before or during publication.
- Add a regression with a blocked cache-only Git flush. The deployed
code waits; the new code publishes verified cache bytes without waiting.
Test storage failure for both recovery-backed archives and same-path,
hardlink, and replacement alias cases.
- Add cache copy, hash, and bookkeeping spans beneath seed lookup.
- Add timing around existing agent, conversation, runtime, issue,
identity, trust, and task context preparation. Keep their execution
order.
- Update the trust-order guard to recognize the timed call while
continuing to assert trust resolution precedes workspace selection and
credential setup.
- Document the cache contract and the new spans. Keep the old flush span
in the allowlist for compatibility with older observations.

## Verification

- The blocked-flush regression fails on the old code and passes with
this change.
- Focused tests (`vitest run` on the six affected cache, timing, and
native-sync suites) cover cache corruption, private-copy integrity,
quota and eviction races, branch binding, per-run recovery, native
workspace synchronization, trace parentage, and tracing disabled or
failing.
- 115 focused tests and both typechecks pass after rebase on current
master. The tests include recovery storage failures and an archive
replaced with a recovery hardlink during verification.
- All seven trust tests pass with the actual service and migrations,
using a fresh database on an existing local PostgreSQL server. The local
embedded server could not allocate host semaphores, so only its test
startup helper was substituted; assertions and service code were
unchanged.
- A real SDK export with a retained public seed verifies that cache-read
detail spans have the lookup as their parent. The endpoint-off control
produces no spans.
- Current revision `28dea63432d462cad160906f8dfc2d21c81d3446`: all 53
checks pass, two optional checks are skipped, Greptile is 5/5, and no
review threads are open.
- No new staging latency claim is made by this PR. Hosted measurement
remains necessary after delivery.

## Risks

A crash or power loss can discard the cache-only Git seed. A later read
rejects missing or corrupt cache bytes and prepares the workspace again.
Recovery-backed archives retain their previous flush behavior. The
change adds no flush to warm per-run persistence. Hashes and
private-copy checks remain required. The new spans use the existing
opt-in exporter and closed names; they record no task content.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, and code
execution. The session does not expose the exact model SKU or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/services/execution-preparation-timing.test.ts

@@ -46,7 +46,8 @@ it.each([undefined, "", "  "])("does not consult tracing for endpoint %s", async
     await expect(createRunPreparationTiming("run", "native.controller")("backend_setup", () => 42)).resolves.toBe(42);
     await expect(withRunWorkspaceFinalizationTiming("run", () => 42)).resolves.toBe(42);
     for (const phase of ["execution.usage_startup", "execution.mcp_handoff", "execution.dispatch_admission", "execution.budget_reservation",
-      "skills.inventory", "skills.inventory.wait", "skills.cache.acquire", "skills.version.materialize", "skills.consume"] as const) {
+      "skills.inventory", "skills.inventory.wait", "skills.cache.acquire", "skills.version.materialize", "skills.consume",
+      "workspace_sync.seed_read_copy", "workspace_sync.seed_read_hash", "workspace_sync.seed_read_touch"] as const) {
       const result = {};
       const work = vi.fn(() => {
         recordRepositoryObjectCacheDetail({ outcome: "hit", referenceSupplied: true });

server/src/services/execution-preparation-timing.ts

@@ -4,6 +4,8 @@ import { createHash } from "node:crypto";
 import { isSpanContextValid, ROOT_CONTEXT, SpanStatusCode, trace, type Context } from "@opentelemetry/api";
 
 type PreparationPhase = "environment_selection" | "execution_configuration" | "session"
+  | "agent_read" | "chat_completion" | "conversation" | "runtime_state" | "issue_context"
+  | "run_identity" | "trust_preset" | "task_context"
   | "workspace" | "github_environment" | "github_launchers" | "run_output"
   | "workspace_validation" | "runtime_services" | "instructions" | "runtime_selection"
   | "configuration_mentions" | "configuration_github_identity" | "configuration_secrets"
@@ -84,6 +86,7 @@ export type ExecutionPhase =
   | "workspace_sync.seed_generation" | "workspace_sync.seed_lookup" | "workspace_sync.seed_publish"
   | "workspace_sync.seed_cache_copy" | "workspace_sync.seed_cache_sync" | "workspace_sync.seed_cache_hash"
   | "workspace_sync.seed_cache_capacity" | "workspace_sync.seed_cache_evict" | "workspace_sync.seed_cache_locked_publish"
+  | "workspace_sync.seed_read_copy" | "workspace_sync.seed_read_hash" | "workspace_sync.seed_read_touch"
   | "workspace_sync.git_clone" | "workspace_sync.git_archive" | "workspace_sync.seed_replay" | "workspace_sync.seed_persist"
   | "workspace_sync.git_index_refresh"
   | "workspace_sync.overlay_copy" | "workspace_sync.overlay_archive" | "workspace_sync.deleted_manifest"
@@ -251,6 +254,9 @@ const workspaceSyncDetailPhases = {
   "workspace.ignore_rules": "workspace_sync.ignore_rules",
   "workspace.seed_generation": "workspace_sync.seed_generation",
   "workspace.seed_lookup": "workspace_sync.seed_lookup",
+  "workspace.seed_read_copy": "workspace_sync.seed_read_copy",
+  "workspace.seed_read_hash": "workspace_sync.seed_read_hash",
+  "workspace.seed_read_touch": "workspace_sync.seed_read_touch",
   "workspace.seed_publish": "workspace_sync.seed_publish",
   "workspace.seed_cache_copy": "workspace_sync.seed_cache_copy",
   "workspace.seed_cache_sync": "workspace_sync.seed_cache_sync",

server/src/services/heartbeat.ts

@@ -1977,8 +1977,8 @@ export function heartbeatService(
     let identityRedactor = createAgentIdentityRedactor();
 
     try {
-      let agent = await getAgent(run.agentId);
-      if (!agent) {
+      const loadedAgent = await timePreparation("agent_read", () => getAgent(run.agentId));
+      if (!loadedAgent) {
         await setRunStatus(runId, "failed", {
           error: "Agent not found",
           errorCode: "agent_not_found",
@@ -1992,6 +1992,7 @@ export function heartbeatService(
         if (failedRun) await releaseIssueExecutionAndPromote(failedRun);
         return;
       }
+      let agent = loadedAgent;
 
       // The claimed adapter identity is immutable recovery evidence. Do not
       // execute a newly selected adapter under a previous adapter's claim.
@@ -2011,8 +2012,8 @@ export function heartbeatService(
         await finalizeAgentStatus(agent.id, "cancelled");
         return;
       }
-      run = await prepareChatCompletionTurn(db, run);
-      const preparedConversation = await prepareConversationTurn(db, run);
+      run = await timePreparation("chat_completion", () => prepareChatCompletionTurn(db, run));
+      const preparedConversation = await timePreparation("conversation", () => prepareConversationTurn(db, run));
       run = { ...run, contextSnapshot: preparedConversation.context };
       if (preparedConversation.reset) {
         const contextSnapshot = { ...preparedConversation.context, conversationReset: true };
@@ -2025,7 +2026,7 @@ export function heartbeatService(
         await finalizeAgentStatus(agent.id, "succeeded");
         return;
       }
-      const runtime = await ensureRuntimeState(agent);
+      const runtime = await timePreparation("runtime_state", () => ensureRuntimeState(agent));
       const context = parseObject(run.contextSnapshot);
       const authorizeFailedChatRetryExecution = () =>
         db.transaction((tx) =>
@@ -2103,7 +2104,7 @@ export function heartbeatService(
         }
       }
       let issueContext = issueId
-        ? await getIssueExecutionContext(agent.companyId, issueId)
+        ? await timePreparation("issue_context", () => getIssueExecutionContext(agent.companyId, issueId))
         : null;
       const issueDependencyReadiness = issueId
         ? await issuesSvc
@@ -2351,7 +2352,7 @@ export function heartbeatService(
           issueContext,
           routineEnvContext,
         });
-      const identityContext = await initializeRunIdentity(db, {
+      const identityContext = await timePreparation("run_identity", () => initializeRunIdentity(db, {
         companyId: agent.companyId,
         runId: run.id,
         responsibleUserId,
@@ -2385,7 +2386,7 @@ export function heartbeatService(
           readNonEmptyString(context.executionIdentityCause) ??
           readNonEmptyString(context.wakeReason) ??
           "dispatch",
-      });
+      }));
       // Initialization has persisted the active context, including an explicit
       // absence of identity inherited from an automatic continuation.
       responsibleUserId = identityContext.responsibleUserId;
@@ -2443,7 +2444,7 @@ export function heartbeatService(
           parsedIssueExecutionWorkspaceSettings,
           runtimeWorkspaceSelectionEnabled,
         );
-      const retainedTrust = await resolveAndRetainRunTrustPreset(db, {
+      const retainedTrust = await timePreparation("trust_preset", () => resolveAndRetainRunTrustPreset(db, {
         companyId: agent.companyId,
         agentId: agent.id,
         runId: run.id,
@@ -2463,7 +2464,7 @@ export function heartbeatService(
               executionPolicy: issueContext.executionPolicy,
             }
           : null,
-      });
+      }));
       const trustPreset = retainedTrust.trustPreset;
       if (retainedTrust.executionPolicy !== undefined) {
         // Later launch-context writes must preserve the boundary already made
@@ -2603,12 +2604,12 @@ export function heartbeatService(
       }).where(eq(heartbeatRuns.id, run.id));
       const {
         continuationSummary, executionContinuation, paperclipWakePayload, safeWakeComments, safeWakeCommentContext,
-      } = await prepareHeartbeatTaskContext(db, {
+      } = await timePreparation("task_context", () => prepareHeartbeatTaskContext(db, {
         run, agent,
         task: { issueRef, issueContext, wakeCommentContext, context, taskSession },
         policy: { trustPreset, experimentalInstanceSettings, recovering: Boolean(runOptions.nativeRestartRecovery || runOptions.nativeLeaseOwner) },
         services: { issuesSvc, getPinnedSkillTestContext },
-      });
+      }));
       const sourceWorkspaceConfig = buildExecutionWorkspaceAdapterConfig({
         agentConfig: config,
         projectPolicy: projectExecutionWorkspacePolicy,