fix: replay prepared warm seeds when generation caching is unavailable (#16399)

Replay this preparation’s verified private workspace seed after failed warm validation, including when the generation cache refuses publication. Preserve host-current semantics and rebuild if the private seed is missing or changed.

Verified 89 focused cases, full typecheck/build, exact-head CI/E2E/security and Apex 5/5. Live PAP latency acceptance remains separate.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

doc/plans/2026-10-09-task-workspaces.md

@@ -67,7 +67,7 @@ Preserve simultaneous local/SSH tasks in an explicitly configured identical phys
 
 Concurrent remote sharing requires proven common physical placement, separate session authority and coherent snapshot capability. Unsupported new concurrent sharing returns a typed unavailable result; do not instantiate separate clones and label them shared. Already-admitted descriptors remain recoverable unchanged. Local authoritative roots need no replica copyback; SSH retains its actual existing persistence contract.
 
-Sparse filesystem checkpoints use disk-backed full manifests, changed bytes/deletions, safe-link and existing ignore semantics, and separate nested Git history. Capture still enumerates and hashes files with stable-read validation; a metadata hash cache is deferred. No-change sends zero file-content payload where capability is supported; manifest metadata still transfers; unsupported transport uses explicit confined full fallback. Sparse capture capacity failures before outbound data is read also use the existing confined full transfer, so a tiny output can recover when the sandbox cannot preserve the sparse manifest reserve. Returned manifest, path, digest and payload validation failures remain errors and never trigger fallback. Full checkpoint exports validate the completed staging root before merge, reject unsupported entries, and omit absolute or escaping links without following them; safe relative links remain supported. Reuse immutable durable seed generations, not only warm sandbox contents; reuse still verifies archive digests. If warm identity validation requires a host-current transfer, that fallback retains the same company/workspace seed-cache scope, so a verified generation can supply the transfer without a second Git clone and archive rebuild. Missing or invalid generations still rebuild from the host. Existing run references pin seeds/deltas until all admitted runs/recovery obligations release them. Retain the final host rescan because the run ownership fence does not exclude external writers to shared folders; do not reuse a stale post-merge manifest.
+Sparse filesystem checkpoints use disk-backed full manifests, changed bytes/deletions, safe-link and existing ignore semantics, and separate nested Git history. Capture still enumerates and hashes files with stable-read validation; a metadata hash cache is deferred. No-change sends zero file-content payload where capability is supported; manifest metadata still transfers; unsupported transport uses explicit confined full fallback. Sparse capture capacity failures before outbound data is read also use the existing confined full transfer, so a tiny output can recover when the sandbox cannot preserve the sparse manifest reserve. Returned manifest, path, digest and payload validation failures remain errors and never trigger fallback. Full checkpoint exports validate the completed staging root before merge, reject unsupported entries, and omit absolute or escaping links without following them; safe relative links remain supported. Reuse immutable durable seed generations, not only warm sandbox contents; reuse still verifies archive digests. If warm identity validation requires a host-current transfer, that fallback retains the same company/workspace seed-cache scope, so a verified generation can supply the transfer without a second Git clone and archive rebuild. Missing or invalid generations still rebuild from the host. When the generation cache declines a new snapshot, warm fallback may replay the current preparation's private durable seed after its existing file-type and digest checks. It does not raise the cache quotas or reuse a prior run's seed. Missing or changed current-run archives retain the host rebuild path. Existing run references pin seeds/deltas until all admitted runs/recovery obligations release them. Retain the final host rescan because the run ownership fence does not exclude external writers to shared folders; do not reuse a stale post-merge manifest.
 
 A returned sparse snapshot must describe a coherent tree before any host mutation. Every nested entry requires an immediate parent marked as a directory in that same returned snapshot, including baseline-unchanged files. Checking every directory also verifies recursive ancestry. Top-level entries use the implicit root. Invalid trees fail without overwriting retained host files and keep recovery evidence. The check uses the existing indexed manifest interface; it adds no host-path scans or unbounded in-memory inventory.
 

server/src/__tests__/native-workspace-sync-history.test.ts

@@ -1,5 +1,5 @@
 import { execFile } from "node:child_process";
-import { mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
+import { mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises";
 import os from "node:os";
 import path from "node:path";
 import { afterEach, describe, expect, it, vi } from "vitest";
@@ -7,6 +7,7 @@ import { heartbeatRuns, type Db } from "@paperclipai/db";
 import type { EnvironmentLease } from "@paperclipai/shared";
 import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
 import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
+import * as workspaceSeedCache from "@paperclipai/adapter-utils/workspace-seed-cache";
 import * as gitWorkspaceSync from "@paperclipai/adapter-utils/git-workspace-sync";
 import { runLocalGit } from "@paperclipai/adapter-utils/git-workspace-sync";
 import { prepareNativeWorkspaceSync, nativeWorkspaceSyncInternals } from "../services/native-runtime/native-workspace-sync.js";
@@ -46,8 +47,10 @@ describe("native warm workspace Git history", () => {
     await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true })));
   });
 
-  it.each(["unchanged", "host_commit", "host_branch", "host_origin", "remote_commit", "remote_commit_corrupt_seed", "remote_commit_missing_seed", "nested_commit", "plain_nested_commit"] as const)(
+  it.each(["unchanged", "host_commit", "host_branch", "host_origin", "remote_commit", "remote_commit_corrupt_seed", "remote_commit_missing_seed", "remote_commit_cache_refused", "host_bytes_cache_refused", "remote_commit_missing_run_seed", "remote_commit_corrupt_run_seed", "nested_commit", "plain_nested_commit"] as const)(
     "checks Git identity before reusing a warm sandbox: %s", async (change) => {
+      const invalidRunSeed = change === "remote_commit_missing_run_seed" || change === "remote_commit_corrupt_run_seed";
+      const cacheRefused = change === "remote_commit_cache_refused" || change === "host_bytes_cache_refused" || invalidRunSeed;
       const packClone = vi.spyOn(gitWorkspaceSync, "withShallowGitWorkspaceClone");
       const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-warm-history-"));
       roots.push(root);
@@ -113,7 +116,10 @@ describe("native warm workspace Git history", () => {
         await git(host, "checkout", "-b", "other");
       } else if (change === "host_origin") {
         await git(host, "remote", "add", "origin", "https://github.com/paperclipai/new-source.git");
-      } else if (change === "remote_commit" || change === "remote_commit_corrupt_seed" || change === "remote_commit_missing_seed") {
+      } else if (change === "host_bytes_cache_refused") {
+        await writeFile(path.join(host, "file.txt"), "updated host content\n");
+        await writeFile(path.join(host, "extra.txt"), "new host file\n");
+      } else if (change.startsWith("remote_commit")) {
         await git(remote, "-c", "user.name=Test", "-c", "user.email=test@paperclip.dev", "commit", "--allow-empty", "-m", "remote only");
       }
       if (change === "remote_commit_corrupt_seed" || change === "remote_commit_missing_seed") {
@@ -124,24 +130,47 @@ describe("native warm workspace Git history", () => {
         await rm(archive);
         if (change === "remote_commit_corrupt_seed") await writeFile(archive, "invalid cached archive");
       }
+      if (cacheRefused) {
+        await rm(path.join(process.env.PAPERCLIP_HOME!, "instances", "test", "native-workspace-sync", "generations", "company", "workspace"), { recursive: true, force: true });
+        vi.spyOn(workspaceSeedCache, "publishWorkspaceSeedGeneration").mockResolvedValue(false);
+      }
       const hostRepo = change === "plain_nested_commit" ? path.join(host, nestedPath) : host;
       const remoteRepo = change === "plain_nested_commit" ? path.join(remote, nestedPath) : remote;
+      const expectedContents = await readFile(path.join(hostRepo, "file.txt"), "utf8");
       const expectedHead = await git(hostRepo, "rev-parse", "HEAD");
       const expectedBranch = await git(hostRepo, "symbolic-ref", "--short", "HEAD");
       const nestedHead = change === "nested_commit" || change === "plain_nested_commit" ? await git(path.join(host, nestedPath), "rev-parse", "HEAD") : null;
       runnerProfileJson = {};
       packClone.mockClear();
+      let invalidatedRunSeed = false;
+      const secondRunner: CommandManagedRuntimeRunner = {
+        execute: async operation => {
+          const result = await runner.execute(operation);
+          if (invalidRunSeed && !invalidatedRunSeed && operation.args?.some(arg => arg.includes("cat ") && arg.includes("workspace-sync-v1.json"))) {
+            invalidatedRunSeed = true;
+            const seed = nativeWorkspaceSyncInternals.durableSeedPaths("second");
+            await rm(seed.workspaceArchivePath);
+            if (change === "remote_commit_corrupt_run_seed") await writeFile(seed.workspaceArchivePath, "invalid prepared archive");
+          }
+          return result;
+        },
+      };
       const second = (await prepareNativeWorkspaceSync({ ...input, runId: "second", target: {
-        ...target, sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" },
+        ...target, runner: secondRunner, sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" },
       } }))!;
       try {
         if (change === "remote_commit") expect(packClone).not.toHaveBeenCalled();
+        if (cacheRefused) expect(packClone).toHaveBeenCalledTimes(invalidRunSeed ? 2 : 1);
+        if (invalidRunSeed) expect(invalidatedRunSeed).toBe(true);
         if (change === "remote_commit_corrupt_seed" || change === "remote_commit_missing_seed") expect(packClone).toHaveBeenCalled();
         expect(second.mode).toBe(change === "unchanged" ? "adopt_remote" : "host_current");
+        expect(await readFile(path.join(remoteRepo, "file.txt"), "utf8")).toBe(expectedContents);
+        if (change === "host_bytes_cache_refused") expect(await readFile(path.join(remote, "extra.txt"), "utf8")).toBe("new host file\n");
         expect(await git(remoteRepo, "rev-parse", "HEAD")).toBe(expectedHead);
         expect(await git(remoteRepo, "symbolic-ref", "--short", "HEAD")).toBe(expectedBranch);
         if (change === "host_origin") expect(await git(remote, "remote", "get-url", "origin")).toBe("https://github.com/paperclipai/new-source.git");
         await second.restoreWorkspace();
+        expect(await readFile(path.join(hostRepo, "file.txt"), "utf8")).toBe(expectedContents);
         expect(await git(hostRepo, "rev-parse", "HEAD")).toBe(expectedHead);
         if (nestedHead) {
           expect(await git(path.join(remote, nestedPath), "rev-parse", "HEAD")).toBe(nestedHead);

server/src/services/native-runtime/native-workspace-sync.ts

@@ -991,6 +991,10 @@ export async function prepareNativeWorkspaceSync(input: {
     if (!currentSnapshot) {
       throw new Error("native_workspace_sync_snapshot_missing");
     }
+    let workspaceArchiveSha256 = await sha256File(seedPaths.workspaceArchivePath);
+    let gitArchiveSha256 = currentSnapshot.gitSnapshot || currentSnapshot.repositories?.length
+      ? await sha256File(seedPaths.gitArchivePath)
+      : null;
     if (acquisition === "resumed" && priorStamp) {
       const currentHostSha256 = directorySnapshotSha256(
         currentSnapshot.baseline,
@@ -1002,11 +1006,19 @@ export async function prepareNativeWorkspaceSync(input: {
         mode = "adopt_remote";
       } else {
         mode = "host_current";
+        let preparedSeed: WorkspaceDurableSeedPaths | undefined;
+        try {
+          preparedSeed = await verifiedDurableSeed({ runId: input.runId,
+            seed: { workspaceArchiveSha256, gitArchiveSha256 },
+            gitSnapshot: currentSnapshot.gitSnapshot, repositories: currentSnapshot.repositories });
+        } catch (error) {
+          if (!(error instanceof Error) || error.message !== "workspace_sync_out_unrecoverable") throw error;
+        }
         runtime = await prepareRuntime({
           runId: input.runId,
           target,
           workspaceLocalDir: input.workspaceLocalDir,
-          mode,
+          mode: preparedSeed ? "durable_seed" : mode,
           seedCacheCompanyDirectory,
           seedCacheDirectory,
           baseline: currentSnapshot.baseline,
@@ -1014,23 +1026,19 @@ export async function prepareNativeWorkspaceSync(input: {
           repositories: currentSnapshot.repositories,
           onPhase: input.onPhase,
           onCheckpoint: input.onCheckpoint,
-          durableSeed: {
-            workspaceArchivePath: seedPaths.workspaceArchivePath,
-            gitArchivePath: seedPaths.gitArchivePath,
-          },
+          durableSeed: preparedSeed ?? seedPaths,
         });
+        if (!preparedSeed) {
+          workspaceArchiveSha256 = await sha256File(seedPaths.workspaceArchivePath);
+          gitArchiveSha256 = currentSnapshot.gitSnapshot || currentSnapshot.repositories?.length
+            ? await sha256File(seedPaths.gitArchivePath) : null;
+        }
       }
     }
     const snapshot = runtime.workspaceSyncSnapshot;
     if (!snapshot) throw new Error("native_workspace_sync_snapshot_missing");
     await persistSnapshotManifests(input.runId, snapshot);
     const now = new Date().toISOString();
-    const workspaceArchiveSha256 = await sha256File(
-      seedPaths.workspaceArchivePath,
-    );
-    const gitArchiveSha256 = snapshot.gitSnapshot || snapshot.repositories?.length
-      ? await sha256File(seedPaths.gitArchivePath)
-      : null;
     descriptor = {
       schema: DESCRIPTOR_SCHEMA,
       binding: {