From 3bf51913ef779ca5c300a2c6f6af37678199a464 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 8 Nov 2026 14:52:56 -0600 Subject: [PATCH] fix(daytona): gzip transport compression for sandbox images without zstd (#20104) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path - Per-turn remote sandboxes are the default, so every remote turn uploads the workspace into a fresh Daytona sandbox. - The Daytona plugin compresses large uploads with zstd, but only when the sandbox has a `zstd` binary. Without one, the upload goes raw: 220 MB for the paperclip seed. - The default Paperclip runner image has zstd, so it is already covered. A sandbox image without zstd is not: for example a user-supplied `config.snapshot` built on plain `debian:bookworm`. Such an image still has gzip. - This adds gzip as a fallback codec for those images. It is a robustness fallback, not a speedup for the default image. ## Linked Issues or Issue Description Refs #20100 **What happened?** On a Daytona sandbox image without a `zstd` binary, every remote turn uploads the workspace seed uncompressed. Measured live with the paperclip seed: raw takes 18.6–28.8 s, against 6.1–16.0 s compressed. **Steps to reproduce** 1. Use a Daytona snapshot whose image has gzip but no zstd. 2. Run a remote task on a large repository and inspect the `transfer` span: there is no `transfer.compression.codec` attribute, and the full raw size is uploaded. **Expected behavior** Large workspace tarballs are compressed in transit whenever the sandbox can decompress them. ## What Changed - `packages/plugins/sandbox-providers/daytona/src/file-sync.ts`: - The existing availability probe (the same round trip as `mkdir`) also reports `gzip`. - Codec choice: zstd when the sandbox has it and Node supports it (unchanged), otherwise gzip, otherwise raw (unchanged). - gzip runs at level 1. On the 220 MB paperclip seed it produces 69.6 MB in 1.26 s on the host; zstd -3 produces 57.6 MB in 0.54 s. - In the sandbox, gzip decompresses with `gzip -dc -- > `. It reuses the existing explicit `chmod`, atomic `mv -f`, failure sweep, private host temp directory, and minimum-saving fallback. - A failed or partial decompress exits 49, the raw scratch is never promoted, and every scratch file is swept. The workspace sentinel and `baselineSha256` checks still run before `turn.start`. - The `transfer.compression.codec` span attribute is now `zstd` or `gzip` (still a closed set). - Tests in `file-sync.test.ts` (red on master): - A real shell promotion on a sandbox that reports no zstd: the upload is `.gz` and under half the size, `gzip -dc` runs, and the target is byte-identical with mode 0600 and no scratch left. - A truncated gzip upload fails the sync, never creates the target, and leaves no scratch. - Codec preference: zstd when both are present, gzip when only gzip is, raw when neither is. ## Verification - `npx vitest run` in `packages/plugins/sandbox-providers/daytona`: all pass. The existing zstd suite runs on this host because it has zstd. - `tsc --noEmit -p packages/plugins/sandbox-providers/daytona`: clean. - Live Daytona run on the runner image, 2 vCPU, 220 MB seed, through the real `performSyncIn` with each codec forced through the probe. The sha256 matched every time. | Codec | Times | |---|---| | raw | 28.8 s, 18.6 s, 20.2 s, 22.8 s | | zstd | 6.9 s, 6.1 s, 16.0 s, 8.5 s | | gzip | 9.8 s, 16.8 s | Network variance is large. Both codecs roughly halve the raw upload or better. ## Risks - **Host CPU:** about 1.3 s of streamed gzip per large upload, on the libuv thread pool, and only when the sandbox lacks zstd. Files under 8 MB, or that save less than 10%, still go raw. - **No compressor at all:** a sandbox without gzip or zstd uploads raw, as today. - **Probe caching:** the probe runs on every sync, on the `mkdir` round trip it already makes, so it adds no latency. It stays uncached, following the existing design's rule that one principal's observation is never reused for another. - **Observability:** one new value, `gzip`, in an existing closed-set span attribute. The endpoint gate is unchanged. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../daytona/src/file-sync.test.ts | 95 ++++++++++++++++++- .../daytona/src/file-sync.ts | 54 ++++++++--- 2 files changed, 135 insertions(+), 14 deletions(-) diff --git a/packages/plugins/sandbox-providers/daytona/src/file-sync.test.ts b/packages/plugins/sandbox-providers/daytona/src/file-sync.test.ts index 81ed11847a5..466059be3d0 100644 --- a/packages/plugins/sandbox-providers/daytona/src/file-sync.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/file-sync.test.ts @@ -266,6 +266,8 @@ function createRecordingSandbox(input: { function createRealExecSandbox(input?: { uploadOverride?: (uploads: Array<{ source: string; destination: string }>) => Promise; commandEnv?: NodeJS.ProcessEnv; + /** Report no `zstd` binary to the availability probe, like the Daytona runner image. */ + hideZstd?: boolean; }) { const commands: RecordedCommand[] = []; return { @@ -275,7 +277,8 @@ function createRealExecSandbox(input?: { executeCommand: async (command: string) => { commands.push({ command }); const result = spawnSync("/bin/sh", ["-c", command], { encoding: "utf8", maxBuffer: 64 * 1024 * 1024, env: input?.commandEnv }); - return { exitCode: result.status ?? 1, result: (result.stdout ?? "") + (result.stderr ?? "") }; + const output = (result.stdout ?? "") + (result.stderr ?? ""); + return { exitCode: result.status ?? 1, result: input?.hideZstd ? output.replace("PAPERCLIP_ZSTD_AVAILABLE\n", "") : output }; }, }, fs: { @@ -1159,6 +1162,96 @@ describe.skipIf(!gnuTar)("automatic unsafe workspace export recovery", () => { }); +describe("daytona file-sync inbound gzip transport fallback (sandbox without zstd)", () => { + // The Daytona runner image has gzip but no zstd binary. Without this fallback + // every remote turn uploaded the whole uncompressed workspace seed (about + // 220 MB for the paperclip repository, about 13 s per turn). + it("compresses with gzip on the host and decompresses in-sandbox to a byte-identical file", async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-gzip-")); + try { + const remoteDir = path.join(root, "remote"); + await fs.mkdir(remoteDir); + const sourcePath = path.join(root, "workspace-upload.tar"); + await writeCompressibleFile(sourcePath, ZSTD_MIN_SOURCE_BYTES_FOR_TEST + 1024); + const targetPath = path.posix.join(remoteDir, "workspace-upload.tar"); + const uploaded: Array<{ destination: string; bytes: number }> = []; + const { sandbox, commands } = createRealExecSandbox({ + hideZstd: true, + uploadOverride: async (uploads) => { + for (const upload of uploads) { + uploaded.push({ destination: upload.destination, bytes: (await fs.stat(upload.source)).size }); + await fs.copyFile(upload.source, upload.destination); + } + return true; + }, + }); + await performSyncIn({ sandbox: sandbox as never, remoteDir, timeoutSeconds: 30, + operations: [{ operationId: "op-1", files: [{ sourcePath, targetPath, kind: "file" }] }] }); + + expect(uploaded).toHaveLength(1); + expect(uploaded[0]!.destination.endsWith(".gz")).toBe(true); + expect(uploaded[0]!.bytes).toBeLessThan((ZSTD_MIN_SOURCE_BYTES_FOR_TEST + 1024) / 2); + expect(commands.some((entry) => entry.command.includes("gzip -dc"))).toBe(true); + expect(await sha256OfFile(targetPath)).toBe(await sha256OfFile(sourcePath)); + expect((await fs.stat(targetPath)).mode & 0o777).toBe(0o600); + expect((await fs.readdir(remoteDir)).filter((name) => name.includes(".paperclip-upload"))).toHaveLength(0); + } finally { + await fs.rm(root, { recursive: true, force: true }); + } + }); + + it("never promotes a truncated gzip upload: gzip's CRC check fails the sync and the scratch is swept", async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-gzip-")); + try { + const remoteDir = path.join(root, "remote"); + await fs.mkdir(remoteDir); + const sourcePath = path.join(root, "workspace-upload.tar"); + await writeCompressibleFile(sourcePath, ZSTD_MIN_SOURCE_BYTES_FOR_TEST + 1024); + const targetPath = path.posix.join(remoteDir, "workspace-upload.tar"); + const { sandbox } = createRealExecSandbox({ + hideZstd: true, + uploadOverride: async (uploads) => { + for (const upload of uploads) { + const bytes = await fs.readFile(upload.source); + await fs.writeFile(upload.destination, bytes.subarray(0, Math.floor(bytes.length / 2))); + } + return true; + }, + }); + await expect(performSyncIn({ sandbox: sandbox as never, remoteDir, timeoutSeconds: 30, + operations: [{ operationId: "op-1", files: [{ sourcePath, targetPath, kind: "file" }] }] })).rejects.toThrow(/decompress failed|exit 49/); + await expect(fs.stat(targetPath)).rejects.toThrow(/ENOENT/); + expect((await fs.readdir(remoteDir)).filter((name) => name.includes(".paperclip-upload"))).toHaveLength(0); + } finally { + await fs.rm(root, { recursive: true, force: true }); + } + }); + + it("prefers zstd when the sandbox has both, and uploads raw when it has neither", async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-gzip-")); + try { + const sourcePath = path.join(root, "big.tar"); + await writeCompressibleFile(sourcePath, ZSTD_MIN_SOURCE_BYTES_FOR_TEST + 1024); + const destinationsFor = async (probe: string) => { + const uploadedDestinations: string[] = []; + const sandbox = { + process: { executeCommand: async (command: string) => ({ exitCode: 0, result: command.includes("mkdir -p") ? probe : "" }) }, + fs: { uploadFiles: async (uploads: Array<{ destination: string }>) => { uploadedDestinations.push(...uploads.map((upload) => upload.destination)); }, + setFilePermissions: async () => undefined }, + }; + await performSyncIn({ sandbox: sandbox as never, remoteDir: "/workspace", timeoutSeconds: 30, + operations: [{ operationId: "op-1", files: [{ sourcePath, targetPath: "/workspace/target.bin", kind: "file" }] }] }); + return uploadedDestinations; + }; + expect((await destinationsFor("PAPERCLIP_ZSTD_AVAILABLE\nPAPERCLIP_GZIP_AVAILABLE\n"))[0]).toMatch(/\.zst$/); + expect((await destinationsFor("PAPERCLIP_GZIP_AVAILABLE\n"))[0]).toMatch(/\.gz$/); + expect((await destinationsFor(""))[0]).not.toMatch(/\.(gz|zst)$/); + } finally { + await fs.rm(root, { recursive: true, force: true }); + } + }); +}); + describe("precompressed file transport", () => { it.each(["gzip", "zstd"])("uploads %s bytes without trying to recompress them", async codec => { const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-compressed-transfer-")); diff --git a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts index fa1273c1a4b..be958b0371a 100644 --- a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts +++ b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts @@ -33,8 +33,8 @@ const SPAN_ATTR = { // for a download from the sandbox. Operation identity comes from the parent // span, so the transfer span never carries an operation label. transferDirection: `${SPAN_ATTR_PREFIX}transfer.direction`, - // The five zstd-transport-compression attributes. Values are a closed codec - // set or a finite number — never a path, a command line, file content, a + // The five transport-compression attributes. Values are a closed codec + // set (`zstd`, `gzip`) or a finite number — never a path, a command line, file content, a // raw identifier, or error text. transferCompressionCodec: `${SPAN_ATTR_PREFIX}transfer.compression.codec`, transferCompressionWallMs: `${SPAN_ATTR_PREFIX}transfer.compression.wall_ms`, @@ -513,6 +513,18 @@ const ZSTD_COMPRESSION_LEVEL = 3; * (no compression), per the design's fallback rules. */ const ZSTD_PROBE_MARKER = "PAPERCLIP_ZSTD_AVAILABLE"; +/** Marker for a `gzip` binary on `PATH`. The Daytona runner image ships gzip + * but no zstd, so gzip is the fallback codec there; without it every remote + * turn uploaded the whole uncompressed workspace seed. */ +const GZIP_PROBE_MARKER = "PAPERCLIP_GZIP_AVAILABLE"; + +/** gzip level for the fallback codec: level 1 compresses a 220 MB repository + * tar to about 70 MB in about 1.3 s; higher levels cost far more CPU for little + * extra saving. */ +const GZIP_COMPRESSION_LEVEL = 1; + +type TransportCodec = "zstd" | "gzip"; + /** * Feature-detect zstd support on the running Node runtime. `node:zlib` shipped * zstd as of Node v22.15.0 / v23.8.0, ahead of this package's declared @@ -550,13 +562,15 @@ async function hasCompressedHeader(sourcePath: string): Promise { * step after the directory create, including the post-compression size * stat, so a throw there does not leave the directory behind. */ -async function compressFileToHostTemp(sourcePath: string): Promise<{ dir: string; path: string; bytesOut: number }> { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-zstd-")); - const tempPath = path.join(dir, "artifact.zst"); +async function compressFileToHostTemp(sourcePath: string, codec: TransportCodec): Promise<{ dir: string; path: string; bytesOut: number }> { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), `paperclip-daytona-${codec}-`)); + const tempPath = path.join(dir, codec === "zstd" ? "artifact.zst" : "artifact.gz"); try { await pipeline( createReadStream(sourcePath), - zlib.createZstdCompress({ params: { [zlib.constants.ZSTD_c_compressionLevel]: ZSTD_COMPRESSION_LEVEL } }), + codec === "zstd" + ? zlib.createZstdCompress({ params: { [zlib.constants.ZSTD_c_compressionLevel]: ZSTD_COMPRESSION_LEVEL } }) + : zlib.createGzip({ level: GZIP_COMPRESSION_LEVEL }), createWriteStream(tempPath, { flags: "wx", mode: 0o600 }), ); const bytesOut = (await fs.stat(tempPath)).size; @@ -768,7 +782,9 @@ interface FileMappingPlan { * decompression step does. */ rawScratch: string; compressed: null | { - /** Reserved `.zst` scratch name, a direct child of `remoteDir`. */ + /** Codec of the uploaded bytes; selects the in-sandbox decompressor. */ + codec: TransportCodec; + /** Reserved `.zst` (or `.gz`) scratch name, a direct child of `remoteDir`. */ zstdScratch: string; /** Private `0700` host temp directory holding the compressed file, removed * (recursively) after upload. */ @@ -817,6 +833,7 @@ async function syncInFileMappings(input: { const mkdirAndProbeCommand = [ mkdirCommand, `&& { command -v zstd >/dev/null 2>&1 && echo ${ZSTD_PROBE_MARKER} || true; }`, + `&& { command -v gzip >/dev/null 2>&1 && echo ${GZIP_PROBE_MARKER} || true; }`, ].join(" "); // `ensureDirectory` span: `mkdir -p` (plus the zstd availability probe) — // ensure a directory exists before a write. @@ -828,6 +845,11 @@ async function syncInFileMappings(input: { // An absent or unexpected probe answer fails closed: no compression, byte- // identical to a sandbox that has no `zstd` binary. const sandboxHasZstd = mkdirOutput.includes(ZSTD_PROBE_MARKER); + // Prefer zstd (faster, smaller); fall back to gzip, which every Debian image + // carries. Node always has gzip, so only zstd needs the runtime feature check. + const codec: TransportCodec | null = sandboxHasZstd && isZstdCompressionSupported() + ? "zstd" + : mkdirOutput.includes(GZIP_PROBE_MARKER) ? "gzip" : null; // Host-side compression, gated on the probe AND a runtime feature check (a // declared `engines.node` floor is an assumption, not a guarantee — always @@ -837,19 +859,19 @@ async function syncInFileMappings(input: { // misses `ZSTD_MIN_SAVING_RATIO`, falls back to the raw path — a fallback is // never an error, it reproduces the present behavior exactly. const compressCandidates = plans.filter((plan) => plan.sourceSize >= ZSTD_MIN_SOURCE_BYTES); - if (sandboxHasZstd && isZstdCompressionSupported() && compressCandidates.length > 0) { + if (codec && compressCandidates.length > 0) { let compressBytesIn = 0; let compressBytesOut = 0; await withProviderSpan({ name: "compress", wallMsAttr: SPAN_ATTR.transferCompressionWallMs, - attributes: { [SPAN_ATTR.transferCompressionCodec]: "zstd" }, + attributes: { [SPAN_ATTR.transferCompressionCodec]: codec }, run: async (span: PluginSpan) => { for (const plan of compressCandidates) { let hostTempDir: string | null = null; try { if (await hasCompressedHeader(plan.mapping.sourcePath)) continue; - const compressed = await compressFileToHostTemp(plan.mapping.sourcePath); + const compressed = await compressFileToHostTemp(plan.mapping.sourcePath, codec); hostTempDir = compressed.dir; compressBytesIn += plan.sourceSize; compressBytesOut += compressed.bytesOut; @@ -859,7 +881,8 @@ async function syncInFileMappings(input: { continue; } plan.compressed = { - zstdScratch: path.posix.join(remoteDir, scratchName(".zst")), + codec, + zstdScratch: path.posix.join(remoteDir, scratchName(codec === "zstd" ? ".zst" : ".gz")), hostTempDir: compressed.dir, hostTempPath: compressed.path, }; @@ -950,8 +973,13 @@ async function syncInFileMappings(input: { // applies no `chmod` when the mapping sets no `mode`, so the two // inbound branches do not use the same no-mode default today. const targetMode = typeof plan.mapping.mode === "number" ? plan.mapping.mode : 0o600; + // gzip writes through a redirect, so the output mode comes from umask; + // the `chmod` below sets it explicitly either way. + const decompress = plan.compressed.codec === "zstd" + ? `zstd -d -o ${shellQuote(plan.rawScratch)} ${shellQuote(plan.compressed.zstdScratch)}` + : `gzip -dc -- ${shellQuote(plan.compressed.zstdScratch)} > ${shellQuote(plan.rawScratch)}`; renameScript.push( - `zstd -d -o ${shellQuote(plan.rawScratch)} ${shellQuote(plan.compressed.zstdScratch)} || { echo "decompress failed"; exit 49; };`, + `${decompress} || { echo "decompress failed"; exit 49; };`, `chmod ${toOctalModeString(targetMode)} ${shellQuote(plan.rawScratch)} || { echo "chmod failed"; exit 50; };`, ); } @@ -999,7 +1027,7 @@ async function syncInFileMappings(input: { const leftoverCount = await sweepZstdScratchAfterSuccess(sandbox, compressedZstdScratchNames, timeoutSeconds); if (leftoverCount > 0) { console.warn( - `Daytona zstd transport compression: ${leftoverCount} post-promotion scratch file(s) could not be removed after two attempts. The already-promoted target file(s) are unaffected.`, + `Daytona transport compression: ${leftoverCount} post-promotion scratch file(s) could not be removed after two attempts. The already-promoted target file(s) are unaffected.`, ); } }