fix(daytona): gzip transport compression for sandbox images without zstd (#20104)

## Thinking Path

- Per-turn remote sandboxes are the default, so every remote turn
uploads the workspace into a fresh Daytona sandbox.
- The Daytona plugin compresses large uploads with zstd, but only when
the sandbox has a `zstd` binary. Without one, the upload goes raw: 220
MB for the paperclip seed.
- The default Paperclip runner image has zstd, so it is already covered.
A sandbox image without zstd is not: for example a user-supplied
`config.snapshot` built on plain `debian:bookworm`. Such an image still
has gzip.
- This adds gzip as a fallback codec for those images. It is a
robustness fallback, not a speedup for the default image.

## Linked Issues or Issue Description

Refs #20100

**What happened?**
On a Daytona sandbox image without a `zstd` binary, every remote turn
uploads the workspace seed uncompressed. Measured live with the
paperclip seed: raw takes 18.6–28.8 s, against 6.1–16.0 s compressed.

**Steps to reproduce**
1. Use a Daytona snapshot whose image has gzip but no zstd.
2. Run a remote task on a large repository and inspect the `transfer`
span: there is no `transfer.compression.codec` attribute, and the full
raw size is uploaded.

**Expected behavior**
Large workspace tarballs are compressed in transit whenever the sandbox
can decompress them.

## What Changed

- `packages/plugins/sandbox-providers/daytona/src/file-sync.ts`:
- The existing availability probe (the same round trip as `mkdir`) also
reports `gzip`.
- Codec choice: zstd when the sandbox has it and Node supports it
(unchanged), otherwise gzip, otherwise raw (unchanged).
- gzip runs at level 1. On the 220 MB paperclip seed it produces 69.6 MB
in 1.26 s on the host; zstd -3 produces 57.6 MB in 0.54 s.
- In the sandbox, gzip decompresses with `gzip -dc -- <scratch> > <raw
scratch>`. It reuses the existing explicit `chmod`, atomic `mv -f`,
failure sweep, private host temp directory, and minimum-saving fallback.
- A failed or partial decompress exits 49, the raw scratch is never
promoted, and every scratch file is swept. The workspace sentinel and
`baselineSha256` checks still run before `turn.start`.
- The `transfer.compression.codec` span attribute is now `zstd` or
`gzip` (still a closed set).
- Tests in `file-sync.test.ts` (red on master):
- A real shell promotion on a sandbox that reports no zstd: the upload
is `.gz` and under half the size, `gzip -dc` runs, and the target is
byte-identical with mode 0600 and no scratch left.
- A truncated gzip upload fails the sync, never creates the target, and
leaves no scratch.
- Codec preference: zstd when both are present, gzip when only gzip is,
raw when neither is.

## Verification

- `npx vitest run` in `packages/plugins/sandbox-providers/daytona`: all
pass. The existing zstd suite runs on this host because it has zstd.
- `tsc --noEmit -p packages/plugins/sandbox-providers/daytona`: clean.
- Live Daytona run on the runner image, 2 vCPU, 220 MB seed, through the
real `performSyncIn` with each codec forced through the probe. The
sha256 matched every time.

  | Codec | Times |
  |---|---|
  | raw | 28.8 s, 18.6 s, 20.2 s, 22.8 s |
  | zstd | 6.9 s, 6.1 s, 16.0 s, 8.5 s |
  | gzip | 9.8 s, 16.8 s |

Network variance is large. Both codecs roughly halve the raw upload or
better.

## Risks

- **Host CPU:** about 1.3 s of streamed gzip per large upload, on the
libuv thread pool, and only when the sandbox lacks zstd. Files under 8
MB, or that save less than 10%, still go raw.
- **No compressor at all:** a sandbox without gzip or zstd uploads raw,
as today.
- **Probe caching:** the probe runs on every sync, on the `mkdir` round
trip it already makes, so it adds no latency. It stays uncached,
following the existing design's rule that one principal's observation is
never reused for another.
- **Observability:** one new value, `gzip`, in an existing closed-set
span attribute. The endpoint gate is unchanged.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

packages/plugins/sandbox-providers/daytona/src/file-sync.test.ts

@@ -266,6 +266,8 @@ function createRecordingSandbox(input: {
 function createRealExecSandbox(input?: {
   uploadOverride?: (uploads: Array<{ source: string; destination: string }>) => Promise<boolean>;
   commandEnv?: NodeJS.ProcessEnv;
+  /** Report no `zstd` binary to the availability probe, like the Daytona runner image. */
+  hideZstd?: boolean;
 }) {
   const commands: RecordedCommand[] = [];
   return {
@@ -275,7 +277,8 @@ function createRealExecSandbox(input?: {
         executeCommand: async (command: string) => {
           commands.push({ command });
           const result = spawnSync("/bin/sh", ["-c", command], { encoding: "utf8", maxBuffer: 64 * 1024 * 1024, env: input?.commandEnv });
-          return { exitCode: result.status ?? 1, result: (result.stdout ?? "") + (result.stderr ?? "") };
+          const output = (result.stdout ?? "") + (result.stderr ?? "");
+          return { exitCode: result.status ?? 1, result: input?.hideZstd ? output.replace("PAPERCLIP_ZSTD_AVAILABLE\n", "") : output };
         },
       },
       fs: {
@@ -1159,6 +1162,96 @@ describe.skipIf(!gnuTar)("automatic unsafe workspace export recovery", () => {
 });
 
 
+describe("daytona file-sync inbound gzip transport fallback (sandbox without zstd)", () => {
+  // The Daytona runner image has gzip but no zstd binary. Without this fallback
+  // every remote turn uploaded the whole uncompressed workspace seed (about
+  // 220 MB for the paperclip repository, about 13 s per turn).
+  it("compresses with gzip on the host and decompresses in-sandbox to a byte-identical file", async () => {
+    const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-gzip-"));
+    try {
+      const remoteDir = path.join(root, "remote");
+      await fs.mkdir(remoteDir);
+      const sourcePath = path.join(root, "workspace-upload.tar");
+      await writeCompressibleFile(sourcePath, ZSTD_MIN_SOURCE_BYTES_FOR_TEST + 1024);
+      const targetPath = path.posix.join(remoteDir, "workspace-upload.tar");
+      const uploaded: Array<{ destination: string; bytes: number }> = [];
+      const { sandbox, commands } = createRealExecSandbox({
+        hideZstd: true,
+        uploadOverride: async (uploads) => {
+          for (const upload of uploads) {
+            uploaded.push({ destination: upload.destination, bytes: (await fs.stat(upload.source)).size });
+            await fs.copyFile(upload.source, upload.destination);
+          }
+          return true;
+        },
+      });
+      await performSyncIn({ sandbox: sandbox as never, remoteDir, timeoutSeconds: 30,
+        operations: [{ operationId: "op-1", files: [{ sourcePath, targetPath, kind: "file" }] }] });
+
+      expect(uploaded).toHaveLength(1);
+      expect(uploaded[0]!.destination.endsWith(".gz")).toBe(true);
+      expect(uploaded[0]!.bytes).toBeLessThan((ZSTD_MIN_SOURCE_BYTES_FOR_TEST + 1024) / 2);
+      expect(commands.some((entry) => entry.command.includes("gzip -dc"))).toBe(true);
+      expect(await sha256OfFile(targetPath)).toBe(await sha256OfFile(sourcePath));
+      expect((await fs.stat(targetPath)).mode & 0o777).toBe(0o600);
+      expect((await fs.readdir(remoteDir)).filter((name) => name.includes(".paperclip-upload"))).toHaveLength(0);
+    } finally {
+      await fs.rm(root, { recursive: true, force: true });
+    }
+  });
+
+  it("never promotes a truncated gzip upload: gzip's CRC check fails the sync and the scratch is swept", async () => {
+    const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-gzip-"));
+    try {
+      const remoteDir = path.join(root, "remote");
+      await fs.mkdir(remoteDir);
+      const sourcePath = path.join(root, "workspace-upload.tar");
+      await writeCompressibleFile(sourcePath, ZSTD_MIN_SOURCE_BYTES_FOR_TEST + 1024);
+      const targetPath = path.posix.join(remoteDir, "workspace-upload.tar");
+      const { sandbox } = createRealExecSandbox({
+        hideZstd: true,
+        uploadOverride: async (uploads) => {
+          for (const upload of uploads) {
+            const bytes = await fs.readFile(upload.source);
+            await fs.writeFile(upload.destination, bytes.subarray(0, Math.floor(bytes.length / 2)));
+          }
+          return true;
+        },
+      });
+      await expect(performSyncIn({ sandbox: sandbox as never, remoteDir, timeoutSeconds: 30,
+        operations: [{ operationId: "op-1", files: [{ sourcePath, targetPath, kind: "file" }] }] })).rejects.toThrow(/decompress failed|exit 49/);
+      await expect(fs.stat(targetPath)).rejects.toThrow(/ENOENT/);
+      expect((await fs.readdir(remoteDir)).filter((name) => name.includes(".paperclip-upload"))).toHaveLength(0);
+    } finally {
+      await fs.rm(root, { recursive: true, force: true });
+    }
+  });
+
+  it("prefers zstd when the sandbox has both, and uploads raw when it has neither", async () => {
+    const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-gzip-"));
+    try {
+      const sourcePath = path.join(root, "big.tar");
+      await writeCompressibleFile(sourcePath, ZSTD_MIN_SOURCE_BYTES_FOR_TEST + 1024);
+      const destinationsFor = async (probe: string) => {
+        const uploadedDestinations: string[] = [];
+        const sandbox = {
+          process: { executeCommand: async (command: string) => ({ exitCode: 0, result: command.includes("mkdir -p") ? probe : "" }) },
+          fs: { uploadFiles: async (uploads: Array<{ destination: string }>) => { uploadedDestinations.push(...uploads.map((upload) => upload.destination)); },
+            setFilePermissions: async () => undefined },
+        };
+        await performSyncIn({ sandbox: sandbox as never, remoteDir: "/workspace", timeoutSeconds: 30,
+          operations: [{ operationId: "op-1", files: [{ sourcePath, targetPath: "/workspace/target.bin", kind: "file" }] }] });
+        return uploadedDestinations;
+      };
+      expect((await destinationsFor("PAPERCLIP_ZSTD_AVAILABLE\nPAPERCLIP_GZIP_AVAILABLE\n"))[0]).toMatch(/\.zst$/);
+      expect((await destinationsFor("PAPERCLIP_GZIP_AVAILABLE\n"))[0]).toMatch(/\.gz$/);
+      expect((await destinationsFor(""))[0]).not.toMatch(/\.(gz|zst)$/);
+    } finally {
+      await fs.rm(root, { recursive: true, force: true });
+    }
+  });
+});
+
 describe("precompressed file transport", () => {
   it.each(["gzip", "zstd"])("uploads %s bytes without trying to recompress them", async codec => {
     const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-compressed-transfer-"));

packages/plugins/sandbox-providers/daytona/src/file-sync.ts

@@ -33,8 +33,8 @@ const SPAN_ATTR = {
   // for a download from the sandbox. Operation identity comes from the parent
   // span, so the transfer span never carries an operation label.
   transferDirection: `${SPAN_ATTR_PREFIX}transfer.direction`,
-  // The five zstd-transport-compression attributes. Values are a closed codec
-  // set or a finite number — never a path, a command line, file content, a
+  // The five transport-compression attributes. Values are a closed codec
+  // set (`zstd`, `gzip`) or a finite number — never a path, a command line, file content, a
   // raw identifier, or error text.
   transferCompressionCodec: `${SPAN_ATTR_PREFIX}transfer.compression.codec`,
   transferCompressionWallMs: `${SPAN_ATTR_PREFIX}transfer.compression.wall_ms`,
@@ -513,6 +513,18 @@ const ZSTD_COMPRESSION_LEVEL = 3;
  * (no compression), per the design's fallback rules. */
 const ZSTD_PROBE_MARKER = "PAPERCLIP_ZSTD_AVAILABLE";
 
+/** Marker for a `gzip` binary on `PATH`. The Daytona runner image ships gzip
+ * but no zstd, so gzip is the fallback codec there; without it every remote
+ * turn uploaded the whole uncompressed workspace seed. */
+const GZIP_PROBE_MARKER = "PAPERCLIP_GZIP_AVAILABLE";
+
+/** gzip level for the fallback codec: level 1 compresses a 220 MB repository
+ * tar to about 70 MB in about 1.3 s; higher levels cost far more CPU for little
+ * extra saving. */
+const GZIP_COMPRESSION_LEVEL = 1;
+
+type TransportCodec = "zstd" | "gzip";
+
 /**
  * Feature-detect zstd support on the running Node runtime. `node:zlib` shipped
  * zstd as of Node v22.15.0 / v23.8.0, ahead of this package's declared
@@ -550,13 +562,15 @@ async function hasCompressedHeader(sourcePath: string): Promise<boolean> {
  * step after the directory create, including the post-compression size
  * stat, so a throw there does not leave the directory behind.
  */
-async function compressFileToHostTemp(sourcePath: string): Promise<{ dir: string; path: string; bytesOut: number }> {
-  const dir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-zstd-"));
-  const tempPath = path.join(dir, "artifact.zst");
+async function compressFileToHostTemp(sourcePath: string, codec: TransportCodec): Promise<{ dir: string; path: string; bytesOut: number }> {
+  const dir = await fs.mkdtemp(path.join(os.tmpdir(), `paperclip-daytona-${codec}-`));
+  const tempPath = path.join(dir, codec === "zstd" ? "artifact.zst" : "artifact.gz");
   try {
     await pipeline(
       createReadStream(sourcePath),
-      zlib.createZstdCompress({ params: { [zlib.constants.ZSTD_c_compressionLevel]: ZSTD_COMPRESSION_LEVEL } }),
+      codec === "zstd"
+        ? zlib.createZstdCompress({ params: { [zlib.constants.ZSTD_c_compressionLevel]: ZSTD_COMPRESSION_LEVEL } })
+        : zlib.createGzip({ level: GZIP_COMPRESSION_LEVEL }),
       createWriteStream(tempPath, { flags: "wx", mode: 0o600 }),
     );
     const bytesOut = (await fs.stat(tempPath)).size;
@@ -768,7 +782,9 @@ interface FileMappingPlan {
    * decompression step does. */
   rawScratch: string;
   compressed: null | {
-    /** Reserved `.zst` scratch name, a direct child of `remoteDir`. */
+    /** Codec of the uploaded bytes; selects the in-sandbox decompressor. */
+    codec: TransportCodec;
+    /** Reserved `.zst` (or `.gz`) scratch name, a direct child of `remoteDir`. */
     zstdScratch: string;
     /** Private `0700` host temp directory holding the compressed file, removed
      * (recursively) after upload. */
@@ -817,6 +833,7 @@ async function syncInFileMappings(input: {
   const mkdirAndProbeCommand = [
     mkdirCommand,
     `&& { command -v zstd >/dev/null 2>&1 && echo ${ZSTD_PROBE_MARKER} || true; }`,
+    `&& { command -v gzip >/dev/null 2>&1 && echo ${GZIP_PROBE_MARKER} || true; }`,
   ].join(" ");
   // `ensureDirectory` span: `mkdir -p` (plus the zstd availability probe) —
   // ensure a directory exists before a write.
@@ -828,6 +845,11 @@ async function syncInFileMappings(input: {
   // An absent or unexpected probe answer fails closed: no compression, byte-
   // identical to a sandbox that has no `zstd` binary.
   const sandboxHasZstd = mkdirOutput.includes(ZSTD_PROBE_MARKER);
+  // Prefer zstd (faster, smaller); fall back to gzip, which every Debian image
+  // carries. Node always has gzip, so only zstd needs the runtime feature check.
+  const codec: TransportCodec | null = sandboxHasZstd && isZstdCompressionSupported()
+    ? "zstd"
+    : mkdirOutput.includes(GZIP_PROBE_MARKER) ? "gzip" : null;
 
   // Host-side compression, gated on the probe AND a runtime feature check (a
   // declared `engines.node` floor is an assumption, not a guarantee — always
@@ -837,19 +859,19 @@ async function syncInFileMappings(input: {
   // misses `ZSTD_MIN_SAVING_RATIO`, falls back to the raw path — a fallback is
   // never an error, it reproduces the present behavior exactly.
   const compressCandidates = plans.filter((plan) => plan.sourceSize >= ZSTD_MIN_SOURCE_BYTES);
-  if (sandboxHasZstd && isZstdCompressionSupported() && compressCandidates.length > 0) {
+  if (codec && compressCandidates.length > 0) {
     let compressBytesIn = 0;
     let compressBytesOut = 0;
     await withProviderSpan({
       name: "compress",
       wallMsAttr: SPAN_ATTR.transferCompressionWallMs,
-      attributes: { [SPAN_ATTR.transferCompressionCodec]: "zstd" },
+      attributes: { [SPAN_ATTR.transferCompressionCodec]: codec },
       run: async (span: PluginSpan) => {
         for (const plan of compressCandidates) {
           let hostTempDir: string | null = null;
           try {
             if (await hasCompressedHeader(plan.mapping.sourcePath)) continue;
-            const compressed = await compressFileToHostTemp(plan.mapping.sourcePath);
+            const compressed = await compressFileToHostTemp(plan.mapping.sourcePath, codec);
             hostTempDir = compressed.dir;
             compressBytesIn += plan.sourceSize;
             compressBytesOut += compressed.bytesOut;
@@ -859,7 +881,8 @@ async function syncInFileMappings(input: {
               continue;
             }
             plan.compressed = {
-              zstdScratch: path.posix.join(remoteDir, scratchName(".zst")),
+              codec,
+              zstdScratch: path.posix.join(remoteDir, scratchName(codec === "zstd" ? ".zst" : ".gz")),
               hostTempDir: compressed.dir,
               hostTempPath: compressed.path,
             };
@@ -950,8 +973,13 @@ async function syncInFileMappings(input: {
         // applies no `chmod` when the mapping sets no `mode`, so the two
         // inbound branches do not use the same no-mode default today.
         const targetMode = typeof plan.mapping.mode === "number" ? plan.mapping.mode : 0o600;
+        // gzip writes through a redirect, so the output mode comes from umask;
+        // the `chmod` below sets it explicitly either way.
+        const decompress = plan.compressed.codec === "zstd"
+          ? `zstd -d -o ${shellQuote(plan.rawScratch)} ${shellQuote(plan.compressed.zstdScratch)}`
+          : `gzip -dc -- ${shellQuote(plan.compressed.zstdScratch)} > ${shellQuote(plan.rawScratch)}`;
         renameScript.push(
-          `zstd -d -o ${shellQuote(plan.rawScratch)} ${shellQuote(plan.compressed.zstdScratch)} || { echo "decompress failed"; exit 49; };`,
+          `${decompress} || { echo "decompress failed"; exit 49; };`,
           `chmod ${toOctalModeString(targetMode)} ${shellQuote(plan.rawScratch)} || { echo "chmod failed"; exit 50; };`,
         );
       }
@@ -999,7 +1027,7 @@ async function syncInFileMappings(input: {
     const leftoverCount = await sweepZstdScratchAfterSuccess(sandbox, compressedZstdScratchNames, timeoutSeconds);
     if (leftoverCount > 0) {
       console.warn(
-        `Daytona zstd transport compression: ${leftoverCount} post-promotion scratch file(s) could not be removed after two attempts. The already-promoted target file(s) are unaffected.`,
+        `Daytona transport compression: ${leftoverCount} post-promotion scratch file(s) could not be removed after two attempts. The already-promoted target file(s) are unaffected.`,
       );
     }
   }