From 482063aeca7033f877e7aba8baab7a5f81c0091c Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 12 Nov 2026 06:28:27 -0600 Subject: [PATCH] fix(daytona): reuse or repair a read-only skill bundle instead of failing the sync (#21075) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent turns run in a remote Runner in a Daytona sandbox. Before a turn starts, Paperclip copies each selected skill into the sandbox as a read-only bundle. > - A sandbox is often kept between turns, so the read-only bundle is already there when the next turn (or a recovery) copies it again. > - The server writes the same skill files again on its own disk, with new timestamps. The copy compared the bundles with `tar --diff`, which also compares timestamps. So the copy saw a difference, extracted over the read-only directories, and failed with "Cannot open: File exists". > - The turn then stopped and the task stayed blocked until a person sent a message. > - This pull request treats a timestamp-only difference as the same bundle, and repairs a bundle that really differs instead of failing. > - The benefit is that a kept sandbox no longer blocks a task when its skills are copied again. ## Linked Issues or Issue Description **What happened?** On staging, two tasks in 30 minutes stopped with "Daytona syncIn extract command failed (exit 43): tar: .paperclip-repository/...: Cannot open: File exists". One failure was a normal new turn on a kept sandbox. The other was a recovery of an interrupted turn. Neither one started again by itself. **Expected behavior** The turn uses the bundle that is already in the sandbox when its content and permissions match. When the content does not match, the copy repairs it. **Steps to reproduce** 1. Copy a read-only (0555/0444) skill directory into a sandbox. 2. Change only the timestamps of the copy (this is what a new server preparation does). 3. Copy the same skill again. GNU tar fails with "Cannot open: File exists". ## What Changed - `file-sync.ts` (read-only directory copy): run the `tar --diff` check with `LC_ALL=C` and ignore lines that only say "Mod time differs". Bytes, modes, file types and missing files still count as differences. - When the bundle really differs, make its read-only subdirectories writable before the extract. GNU tar then replaces the files and restores the archived directory modes (`--delay-directory-restore`). Extra files in the target stay. - If reopening the directories or the repair extract fails, re-seal only the directories that were reopened for it (their list is recorded first). Writable user directories beside the bundle keep their modes. - `file-sync.test.ts`: the existing GNU tar test now covers a timestamp-only resync (kept, no failure) and a damaged bundle (repaired to the source bytes and modes, instead of the old fail-closed result), and a repair that cannot finish (bundle directories re-sealed to 0555, a writable user directory left at 0755), and a reopen that hits an unreadable user directory (bundle re-sealed, no scratch files left). ## Verification - Red: with the old `file-sync.ts`, the updated test fails with the same error seen on staging: `tar: references/overview.md: Cannot open: File exists`. - Green: `npx vitest run src/file-sync.test.ts` in `packages/plugins/sandbox-providers/daytona` passes 33/33 with GNU tar. - I reproduced the root cause by hand with GNU tar 1.35: a re-extract over a 0555 directory fails with "Cannot open: File exists", and `tar -df` reports only "Mod time differs" for a copy that differs only in timestamps. ## Risks - Low risk. The change affects only read-only (immutable) directory copies on Daytona. - A bundle that really differs is now repaired in place instead of failing the turn. Each file is replaced by tar (unlink and create). The directory modes come back from the archive. - If a target has extra read-only subdirectories that are not in the archive, they become owner-writable after a repair. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, code search, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../daytona/src/file-sync.test.ts | 58 +++++++++++++++++-- .../daytona/src/file-sync.ts | 27 +++++++-- 2 files changed, 75 insertions(+), 10 deletions(-) diff --git a/packages/plugins/sandbox-providers/daytona/src/file-sync.test.ts b/packages/plugins/sandbox-providers/daytona/src/file-sync.test.ts index 466059be3d0..982e904179c 100644 --- a/packages/plugins/sandbox-providers/daytona/src/file-sync.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/file-sync.test.ts @@ -338,20 +338,70 @@ it.skipIf(!gnuTar)("extracts interleaved read-only skill directories with GNU ta expect(await fs.readFile(path.join(target, "references", "agents", "qa.md"), "utf8")).toBe("QA instructions"); expect((await fs.stat(path.join(target, "references", "agents"))).mode & 0o777).toBe(0o555); expect((await fs.stat(path.join(target, "references", "agents", "qa.md"))).mode & 0o777).toBe(0o444); + // Recovery re-materializes the same admitted bytes on the host with new + // timestamps. That is still the same bundle: keep it, never fail on it. + const qa = path.join(target, "references", "agents", "qa.md"); + const agentsDir = path.join(target, "references", "agents"); + const resumedAt = new Date(Date.now() + 3_600_000); + for (const entry of [qa, agentsDir, path.join(target, "references", "overview.md")]) await fs.utimes(entry, resumedAt, resumedAt); + await performSyncIn({ sandbox: sandbox as never, remoteDir, timeoutSeconds: 30, + operations: [{ operationId: "rematerialized-skill-resume", files: [{ sourcePath: source, targetPath: target, kind: "directory", mode: 0o555 }] }] }); + expect((await fs.stat(qa)).mtime.getTime()).toBe(resumedAt.getTime()); // Never treat a corrupted read-only bundle as a cache hit, even if its // file size, permissions and timestamp still match the source archive. - const qa = path.join(target, "references", "agents", "qa.md"); - const before = await fs.stat(qa); + // Repair it in place instead of failing the run. + const sourceTimes = await Promise.all([qa, agentsDir, path.join(target, "references", "overview.md")] + .map((entry) => fs.stat(path.join(source, path.relative(target, entry))))); + const before = sourceTimes[0]!; + await fs.chmod(agentsDir, 0o755); await fs.chmod(qa, 0o644); await fs.writeFile(qa, "XX instructions"); await fs.chmod(qa, 0o444); + await fs.chmod(agentsDir, 0o555); await fs.utimes(qa, before.atime, before.mtime); + await fs.utimes(agentsDir, sourceTimes[1]!.atime, sourceTimes[1]!.mtime); + await fs.utimes(path.join(target, "references", "overview.md"), sourceTimes[2]!.atime, sourceTimes[2]!.mtime); + await performSyncIn({ sandbox: sandbox as never, remoteDir, timeoutSeconds: 30, + operations: [{ operationId: "corrupt-skill-resume", files: [{ sourcePath: source, targetPath: target, kind: "directory", mode: 0o555 }] }] }); + expect(await fs.readFile(qa, "utf8")).toBe("QA instructions"); + expect((await fs.stat(agentsDir)).mode & 0o777).toBe(0o555); + expect((await fs.stat(qa)).mode & 0o777).toBe(0o444); + expect(await fs.readFile(path.join(target, "unrelated.txt"), "utf8")).toBe("keep me"); + expect((await fs.readdir(remoteDir)).filter((name) => name.startsWith(".paperclip-upload"))).toEqual([]); + // A repair that cannot finish still leaves the bundle sealed read-only. + const blocked = path.join(target, "references", "overview.md"); + await fs.chmod(path.join(target, "references"), 0o755); + await fs.rm(blocked); + await fs.mkdir(blocked); + await fs.writeFile(path.join(blocked, "child"), "x"); + await fs.chmod(path.join(target, "references"), 0o555); + // A writable directory the user keeps beside the bundle stays writable. + const userDir = path.join(target, "notes"); + await fs.mkdir(userDir, { mode: 0o755 }); await expect(performSyncIn({ sandbox: sandbox as never, remoteDir, timeoutSeconds: 30, - operations: [{ operationId: "corrupt-skill-resume", files: [{ sourcePath: source, targetPath: target, kind: "directory", mode: 0o555 }] }] })).rejects.toThrow("syncIn extract"); - expect(await fs.readFile(qa, "utf8")).toBe("XX instructions"); + operations: [{ operationId: "blocked-skill-resume", files: [{ sourcePath: source, targetPath: target, kind: "directory", mode: 0o555 }] }] })).rejects.toThrow("syncIn extract"); + for (const dir of ["references", "references/agents"]) { + expect((await fs.stat(path.join(target, dir))).mode & 0o777).toBe(0o555); + } + expect((await fs.stat(userDir)).mode & 0o777).toBe(0o755); + expect((await fs.stat(blocked)).mode & 0o200).toBe(0o200); + await fs.chmod(blocked, 0o700); + expect((await fs.readdir(remoteDir)).filter((name) => name.startsWith(".paperclip-upload"))).toEqual([]); + // A reopen that fails part-way (an unreadable user directory) re-seals the + // bundle and leaves no scratch files behind either. + const unreadable = path.join(target, "private"); + await fs.mkdir(unreadable); + await fs.chmod(unreadable, 0o300); + await expect(performSyncIn({ sandbox: sandbox as never, remoteDir, timeoutSeconds: 30, + operations: [{ operationId: "unreadable-skill-resume", files: [{ sourcePath: source, targetPath: target, kind: "directory", mode: 0o555 }] }] })).rejects.toThrow("syncIn extract"); + for (const dir of ["references", "references/agents"]) { + expect((await fs.stat(path.join(target, dir))).mode & 0o777).toBe(0o555); + } + expect((await fs.stat(unreadable)).mode & 0o777).toBe(0o300); expect((await fs.readdir(remoteDir)).filter((name) => name.startsWith(".paperclip-upload"))).toEqual([]); } finally { + await fs.chmod(path.join(target, "private"), 0o700).catch(() => undefined); for (const base of [source, target]) { for (const dir of ["references/agents", "references"]) await fs.chmod(path.join(base, dir), 0o700).catch(() => undefined); } diff --git a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts index be958b0371a..3dabd49dbb9 100644 --- a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts +++ b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts @@ -1104,27 +1104,42 @@ async function syncInDirectoryMapping(input: { // A resumed sandbox can already contain these immutable 0444/0555 // assets. Repack a private extraction as the sandbox user: tar --diff // otherwise rejects identical bytes because the host UID/GID differ. - // Compare content and modes, never chmod the live bundle or skip - // unverified old files. Extra user files in the target stay untouched. + // Compare content and modes, never skip unverified old files. A + // recovery re-materializes the same admitted bytes on the host with + // new timestamps, so a timestamp-only difference is still a match. + // Extra user files in the target stay untouched. ...(immutable ? [ `compare_dir=${shellQuote(`${remoteTar}.compare`)};`, `compare_tar=${shellQuote(`${remoteTar}.normalized`)};`, `compare_list=${shellQuote(`${remoteTar}.members`)};`, - 'cleanup_compare() { if [ -d "$compare_dir" ]; then find "$compare_dir" -type d -exec chmod u+w {} +; rm -rf "$compare_dir"; fi; rm -f "$compare_tar" "$compare_list"; };', + `compare_out=${shellQuote(`${remoteTar}.differences`)};`, + 'cleanup_compare() { if [ -d "$compare_dir" ]; then find "$compare_dir" -type d -exec chmod u+w {} +; rm -rf "$compare_dir"; fi; rm -f "$compare_tar" "$compare_list" "$compare_out"; };', "trap cleanup_compare EXIT;", 'mkdir -m 700 "$compare_dir" || exit 43;', `tar -xf ${shellQuote(remoteTar)} --no-same-owner --delay-directory-restore -C "$compare_dir" || exit 43;`, '(cd "$compare_dir" && find . -mindepth 1 -maxdepth 1 -print0) > "$compare_list" || exit 43;', 'tar -cf "$compare_tar" --format=pax -C "$compare_dir" --null -T "$compare_list" || exit 43;', - `if tar -df "$compare_tar" -C ${shellQuote(mapping.targetPath)} >/dev/null 2>&1; then rm -f ${shellQuote(remoteTar)}; exit 0; fi;`, + `LC_ALL=C tar -df "$compare_tar" -C ${shellQuote(mapping.targetPath)} > "$compare_out" 2>&1; compare_status=$?;`, + `if [ "$compare_status" -le 1 ] && ! grep -v ': Mod time differs$' "$compare_out" >/dev/null; then rm -f ${shellQuote(remoteTar)}; exit 0; fi;`, "cleanup_compare;", "trap - EXIT;", + // The old copy really differs (a changed or damaged file). Its + // read-only directories would make tar fail with "Cannot open: File + // exists", so reopen them; the extract restores the archived modes. + // Record exactly which ones were reopened so a failed repair re-seals + // only those and never touches writable user directories. A reopen + // that fails part-way (an unreadable directory) re-seals the same way. + `reopened_list=${shellQuote(`${remoteTar}.reopened`)};`, + 'reseal_reopened() { if [ -s "$reopened_list" ]; then xargs -0 chmod u-w < "$reopened_list"; fi; rm -f "$reopened_list"; };', + `find ${shellQuote(mapping.targetPath)} -mindepth 1 -type d ! -perm -u+w -print0 -exec chmod u+w {} + > "$reopened_list" || { reseal_reopened; echo "reopen failed"; exit 43; };`, ] : []), // BSD archives may revisit a directory after its parent's files. Keep // GNU tar from restoring a read-only skill directory's mode before all // of its children are extracted; final permissions remain unchanged. - `tar -xf ${shellQuote(remoteTar)} --delay-directory-restore -C ${shellQuote(mapping.targetPath)} || { echo "extract failed"; exit 43; };`, - `rm -f ${shellQuote(remoteTar)};`, + // A failed repair must not leave the read-only bundle writable. + `tar -xf ${shellQuote(remoteTar)} --delay-directory-restore -C ${shellQuote(mapping.targetPath)} || { ${immutable + ? "reseal_reopened; " : ""}echo "extract failed"; exit 43; };`, + `rm -f ${shellQuote(remoteTar)}${immutable ? ' "$reopened_list"' : ""};`, ].join("\n"); // `extractTarball` span: one round trip — re-check the path, `tar -xf`, and // remove the scratch tarball.