From 8af70b9fae9a0de314b9a4adbd2d6e0eed7fae04 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Wed, 22 Jul 2026 12:37:09 -0700 Subject: [PATCH 01/63] fix(test): drain in-flight heartbeat runs before liveness teardown (#10040) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip runs AI agent heartbeats to manage work; each heartbeat dispatches `executeRun` fire-and-forget, which is intentional for concurrency > - The server escalation test suite (`heartbeat-issue-liveness-escalation.test.ts`) exercises `reconcileIssueGraphLiveness`, which heals a resolved-dependency wake by enqueuing an on-demand heartbeat run > - `enqueueWakeup` → `startNextQueuedRunForAgent` dispatches the run fire-and-forget (`void executeRun(...)`), so the background run outlives the awaited reconcile call > - The test's `afterEach` polled `heartbeat_runs.status` to wait for idle, but that flips to `completed` while `executeRun`'s finally block is still flushing events — the escaping `heartbeat_run_events` insert could land between the events delete and the runs delete, tripping the FK constraint > - This PR fixes the race deterministically by tracking in-flight `executeRun` promises and exposing `heartbeatService.drainActiveRunExecutions()`, which the suite awaits before clearing tables > - The benefit is a permanently reliable escalation test suite with no sleeps, no retry bumps, and no production behavior change ## Linked Issues or Issue Description **What happened?** The `heartbeat-issue-liveness-escalation.test.ts` suite intermittently failed in CI with: ``` delete on table "heartbeat_runs" violates foreign key constraint "heartbeat_run_events_run_id_heartbeat_runs_id_fk" ``` **Expected behavior** `afterEach` cleanup should complete without FK violations. **Steps to reproduce** The race is timing-dependent but surfaces reliably when the teardown window is artificially widened. `reconcileIssueGraphLiveness()` heals resolved-dependency wakes by dispatching a heartbeat run fire-and-forget (`void executeRun(...)`). The old `afterEach` polled `heartbeat_runs.status` — but that flips to `completed` while `executeRun`'s finally block still has pending `heartbeat_run_events` row writes. The escaping insert can land between the events delete and the runs delete. **Paperclip version or commit** Reproducible on current `master` (commit `b57aa9950c707a024156c34b79326a82b2dcca31`) ## What Changed - **`server/src/services/heartbeat.ts`** — tracks all in-flight `executeRun` promises in a module-level `Set`; exposes `heartbeatService(db).drainActiveRunExecutions()`, which loops until the set drains (a completing run can enqueue the next queued run in its finally, so a single `await` is not enough) - **`server/src/server-suites/heartbeat-issue-liveness-escalation.test.ts`** — replaces the poll-on-`heartbeat_runs.status` teardown with `await heartbeatService(db).drainActiveRunExecutions()` before clearing tables; removes the now-unnecessary `waitForHeartbeatRunToComplete` helper ## Verification ```bash # Full file (22 tests) npx vitest run server/src/server-suites/heartbeat-issue-liveness-escalation.test.ts # 12x stress loop (264 test-runs, 0 failures) for i in $(seq 1 12); do npx vitest run server/src/server-suites/heartbeat-issue-liveness-escalation.test.ts || break done # Type check the changed files npx tsc --noEmit ``` - 22/22 tests green locally - 12/12 full-file loop iterations: 264 test-runs / 264 afterEach cycles, 0 failures - Widened-teardown stress variant (failed deterministically before the fix) now passes with the drain ## Risks Low risk. The drain mechanism is additive — it only affects test teardown and could also be wired into graceful shutdown. The fire-and-forget dispatch in production is unchanged. The `Set`-based tracking adds negligible overhead per run dispatch (insert on dispatch, delete on completion). ## Model Used - **Provider:** Anthropic - **Model:** Claude Sonnet 4.6 (`claude-sonnet-4-6`) - **Context window:** 200K tokens - **Mode:** Tool use, code execution, extended reasoning ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Harold Kim Co-authored-by: Paperclip --- ...eartbeat-issue-liveness-escalation.test.ts | 24 ++++++--------- server/src/services/heartbeat.ts | 30 ++++++++++++++++++- 2 files changed, 38 insertions(+), 16 deletions(-) diff --git a/server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts b/server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts index 354a2809cb7..92b85ab14cb 100644 --- a/server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts +++ b/server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts @@ -93,21 +93,15 @@ describeEmbeddedPostgres("heartbeat issue graph liveness escalation", () => { afterEach(async () => { vi.clearAllMocks(); runningProcesses.clear(); - let idlePolls = 0; - for (let attempt = 0; attempt < 100; attempt += 1) { - const runs = await db - .select({ status: heartbeatRuns.status }) - .from(heartbeatRuns); - const hasActiveRun = runs.some((run) => run.status === "queued" || run.status === "running"); - if (!hasActiveRun) { - idlePolls += 1; - if (idlePolls >= 3) break; - } else { - idlePolls = 0; - } - await new Promise((resolve) => setTimeout(resolve, 50)); - } - await new Promise((resolve) => setTimeout(resolve, 50)); + // reconcileIssueGraphLiveness heals dependency wakes by enqueuing an + // on-demand wake, which dispatches a heartbeat run fire-and-forget (see + // startNextQueuedRunForAgent → executeRun in the heartbeat service). That + // background run keeps writing rows (workspace_operations, heartbeat_run_events) + // after the awaited call resolves. Deterministically await those in-flight + // executions before clearing tables — otherwise an escaping heartbeat_run_events + // insert can land between the events delete and the heartbeat_runs delete and + // trip the run_events → runs foreign key. + await heartbeatService(db).drainActiveRunExecutions(); await db.delete(activityLog); await db.delete(heartbeatRunEvents); await db.delete(costEvents); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 6fbe84b699f..9276a9ee817 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -554,6 +554,13 @@ const SESSIONED_LOCAL_ADAPTERS = new Set([ // Routes and the scheduler construct separate heartbeatService instances, but // they must agree on in-process adapter executions when reaping stale runs. const activeRunExecutions = new Set(); +// Background heartbeat executions are dispatched fire-and-forget (see +// startNextQueuedRunForAgent), so the promise that resolves once a run's DB +// writes are fully flushed is otherwise unobservable. Track those promises here +// — shared across service instances like activeRunExecutions above — so callers +// that must guarantee no run write is still in flight (graceful shutdown, and +// tests tearing down a shared database) can await drainActiveRunExecutions(). +const activeRunExecutionPromises = new Set>(); const INLINE_BASE64_IMAGE_DATA_RE = /("type":"image","source":\{"type":"base64","data":")([A-Za-z0-9+/=]{1024,})(")/g; type RuntimeConfigSecretResolver = Pick< @@ -11748,14 +11755,34 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) if (claimedRuns.length === 0) return []; for (const claimedRun of claimedRuns) { - void executeRun(claimedRun.id).catch((err) => { + const execution = executeRun(claimedRun.id).catch((err) => { logger.error({ err, runId: claimedRun.id }, "queued heartbeat execution failed"); }); + // Register the in-flight execution so drainActiveRunExecutions() can await + // it. executeRun resolves only after its finally block finishes flushing + // run rows/events, so awaiting this promise guarantees the run's writes + // have landed before a caller (e.g. a test's afterEach) mutates the DB. + activeRunExecutionPromises.add(execution); + void execution.finally(() => { + activeRunExecutionPromises.delete(execution); + }); } return claimedRuns; }); } + // Await every background heartbeat execution that is currently in flight. A + // draining run can, in its finally block, promote and dispatch the next queued + // run for the same agent — that follow-up execution is registered in the set + // before the parent promise settles, so we loop until the set is empty rather + // than snapshotting once. Callers use this to guarantee no run is still + // writing rows/events (graceful shutdown, deterministic test teardown). + async function drainActiveRunExecutions() { + while (activeRunExecutionPromises.size > 0) { + await Promise.all([...activeRunExecutionPromises]); + } + } + async function executeRun(runId: string) { if ((await getSchedulingSuppression()).suppressed) return; @@ -16912,6 +16939,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) // gate on suppression should prefer this over the env-only resolver. resolveSchedulingSuppression: getSchedulingSuppression, drainRunningRunsForShutdown, + drainActiveRunExecutions, promoteDueScheduledRetries, retryScheduledRetryNow, From 4e8cd757eb696b2d61a0dce75f0b7c0312bfdf3a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:48:54 -0500 Subject: [PATCH 02/63] fix(release): add npm publish crash diagnostics (#10041) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip publishes a coordinated set of packages through its release workflows > - Bundled packages use a pinned npm CLI so trusted publishing works consistently > - The canary publisher now crashes deterministically inside npm before useful output reaches the workflow log > - The npm debug log that contains the underlying failure disappears with the hosted runner > - This pull request upgrades the pinned publish CLI and preserves both verbose HTTP activity and npm debug logs on failure > - The benefit is that the plausible HTTP-layer fix ships immediately, while any remaining CI-only failure becomes diagnosable ## Linked Issues or Issue Description - **Problem:** The canary release workflow fails on the first bundled package with `npm error Exit handler never called!` and no preceding diagnostic output. - **Expected behavior:** Bundled packages publish through trusted publishing, or the workflow retains enough npm diagnostics to identify the actual failure. - **Reproduction:** Run the canary release workflow in GitHub Actions; the failure reproduced on both attempts of run 29948506814. - **Version/commit:** Current `master` after #10024 and #10030. - **Deployment mode:** GitHub-hosted release workflow using Node.js 24 and npm trusted publishing. - Related: #10024, #10030. ## What Changed - Bumped the bundled publish CLI from npm 11.16.0 to npm 11.18.0. - Added `--loglevel verbose` to bundled npm publish invocations. - Dumped the last 300 lines of every npm debug log after failed canary or stable publishes, with common registry credential forms redacted. - Updated release assertions to pin npm 11.18.0 and verify verbose logging. ## Verification - `pnpm test:release-registry` — 66 tests passed. - `bash -n scripts/release-lib.sh`. - Parsed `.github/workflows/release.yml` with Python/PyYAML. - Smoke-tested npm log redaction with representative Authorization, `_authToken`, and token environment values. - Smoke-tested npm debug-log redaction against Authorization, `_authToken`, and `npm_token` examples. - `git diff --check origin/master...HEAD`. - The merge-triggered canary workflow remains the live trusted-publishing verification. ## Risks - Low code risk: changes are isolated to the release publisher and its workflow diagnostics. - npm 11.18.0 could expose a different registry/runtime regression; failure-time debug log dumping makes that actionable. - Verbose npm output increases release log volume but does not change package contents or dist-tags; common credential forms are redacted before debug logs are printed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, GPT-5.3 Codex, tool-enabled coding agent with repository and shell execution; context window size is not exposed in this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .github/workflows/release.yml | 20 ++++++++++++++++++++ scripts/acpx-patch-packaging.test.mjs | 3 ++- scripts/release-lib.sh | 4 ++-- scripts/release-lib.test.mjs | 9 ++++++--- 4 files changed, 30 insertions(+), 6 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index ee36208e470..d46cf1fb594 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -78,6 +78,16 @@ jobs: GITHUB_ACTIONS: "true" run: ./scripts/release.sh canary --skip-verify + - name: Dump npm debug logs + if: failure() + run: | + shopt -s nullglob + for f in "$HOME"/.npm/_logs/*.log; do + echo "===== $f =====" + tail -n 300 "$f" | sed -E \ + -e 's#((authorization|_authToken|_auth|node_auth_token|npm_token)"?[[:space:]]*[:=][[:space:]]*"?)(Bearer[[:space:]]+)?[^",[:space:]]+#\1***REDACTED***#Ig' + done + - name: Push canary tag run: | tag="$(git tag --points-at HEAD | grep '^canary/v' | head -1)" @@ -184,6 +194,16 @@ jobs: fi ./scripts/release.sh "${args[@]}" + - name: Dump npm debug logs + if: failure() + run: | + shopt -s nullglob + for f in "$HOME"/.npm/_logs/*.log; do + echo "===== $f =====" + tail -n 300 "$f" | sed -E \ + -e 's#((authorization|_authToken|_auth|node_auth_token|npm_token)"?[[:space:]]*[:=][[:space:]]*"?)(Bearer[[:space:]]+)?[^",[:space:]]+#\1***REDACTED***#Ig' + done + - name: Push stable tag run: | tag="$(git tag --points-at HEAD | grep '^v' | head -1)" diff --git a/scripts/acpx-patch-packaging.test.mjs b/scripts/acpx-patch-packaging.test.mjs index d51ca8e7efa..8aa45833af6 100644 --- a/scripts/acpx-patch-packaging.test.mjs +++ b/scripts/acpx-patch-packaging.test.mjs @@ -36,7 +36,8 @@ test("bundled package staging materializes publishConfig entrypoints", () => { test("bundled package dry runs preview without querying published versions", () => { assert.match(releaseScript, /run_bundled_npm_pack pack --pack-destination "\$publish_dir"/); assert.match(releaseLib, /BUNDLED_NPM_PACK_VERSION="10\.9\.7"/); - assert.match(releaseLib, /BUNDLED_NPM_PUBLISH_VERSION="11\.16\.0"/); + assert.match(releaseLib, /BUNDLED_NPM_PUBLISH_VERSION="11\.18\.0"/); assert.match(releaseLib, /npx --yes "npm@\$BUNDLED_NPM_PACK_VERSION"/); assert.match(releaseLib, /npx --yes "npm@\$BUNDLED_NPM_PUBLISH_VERSION"/); + assert.match(releaseLib, /"\$@" --loglevel verbose/); }); diff --git a/scripts/release-lib.sh b/scripts/release-lib.sh index 927c36a96e8..a0e2852d753 100644 --- a/scripts/release-lib.sh +++ b/scripts/release-lib.sh @@ -311,14 +311,14 @@ package_publish_tool() { } BUNDLED_NPM_PACK_VERSION="10.9.7" -BUNDLED_NPM_PUBLISH_VERSION="11.16.0" +BUNDLED_NPM_PUBLISH_VERSION="11.18.0" run_bundled_npm_pack() { npx --yes "npm@$BUNDLED_NPM_PACK_VERSION" "$@" } run_bundled_npm_publish() { - npx --yes "npm@$BUNDLED_NPM_PUBLISH_VERSION" "$@" + npx --yes "npm@$BUNDLED_NPM_PUBLISH_VERSION" "$@" --loglevel verbose } run_package_publish() { diff --git a/scripts/release-lib.test.mjs b/scripts/release-lib.test.mjs index e98128d8d65..15bf289df4d 100644 --- a/scripts/release-lib.test.mjs +++ b/scripts/release-lib.test.mjs @@ -91,7 +91,7 @@ exit 1 set -euo pipefail printf 'npx %s\n' "$*" >> "$FAKE_CALL_LOG" [ "$1" = "--yes" ] && shift -[ "$1" = "npm@11.16.0" ] && shift +[ "$1" = "npm@11.18.0" ] && shift exec npm "$@" `, ); @@ -145,8 +145,11 @@ test("publish_package_to_npm uses trusted-publishing-capable npm for bundled dep const result = runPublishHelper({ pnpmMode: "success", publishTool: "npm" }); assert.equal(result.status, 0); - assert.match(result.calls, /^npx --yes npm@11\.16\.0 publish --tag canary --access public$/m); - assert.match(result.calls, /^npm publish --tag canary --access public$/m); + assert.match( + result.calls, + /^npx --yes npm@11\.18\.0 publish --tag canary --access public --loglevel verbose$/m, + ); + assert.match(result.calls, /^npm publish --tag canary --access public --loglevel verbose$/m); assert.doesNotMatch(result.calls, /^pnpm publish/m); }); From 39b94d0e7ab2e64ffda525c92c592b10fe40b5a2 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Wed, 22 Jul 2026 14:00:36 -0700 Subject: [PATCH 03/63] fix(test): gate interaction-continuation retry barrier on terminal write (#10049) --- .../__tests__/heartbeat-process-recovery.test.ts | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/server/src/__tests__/heartbeat-process-recovery.test.ts b/server/src/__tests__/heartbeat-process-recovery.test.ts index 1432021f91e..49008825b42 100644 --- a/server/src/__tests__/heartbeat-process-recovery.test.ts +++ b/server/src/__tests__/heartbeat-process-recovery.test.ts @@ -2211,7 +2211,21 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { .select() .from(heartbeatRuns) .where(eq(heartbeatRuns.agentId, agentId)); - return rows.length >= 2 ? rows : null; + if (rows.length < 2) return null; + // Gate on the *terminal* write of the background recovery, not the + // intermediate retry-run commit. recordPlanApprovalResumeFailureRetry + // writes the system comment first and updates the interaction + // result.resumeFailure last (heartbeat.ts:5627 then :5632), so once + // resumeFailure.status is observed the comment + issue update are also + // committed and every assertion below is race-free. + const interactionRow = await db + .select({ result: issueThreadInteractions.result }) + .from(issueThreadInteractions) + .where(eq(issueThreadInteractions.id, interactionId)) + .then((interactionRows) => interactionRows[0] ?? null); + const result = interactionRow?.result ?? null; + const resumeFailure = result && "resumeFailure" in result ? result.resumeFailure : null; + return resumeFailure?.status === "retrying" ? rows : null; }); expect(runs).toHaveLength(2); From f215444919c8770fb10565780eaef4d63f717dd5 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Wed, 22 Jul 2026 14:54:06 -0700 Subject: [PATCH 04/63] feat(daytona): native file-sync lifecycle hooks over batch transfer (#10028) --- .../daytona/src/file-sync.ts | 787 ++++++++++++++++++ .../daytona/src/plugin.test.ts | 704 +++++++++++++++- .../sandbox-providers/daytona/src/plugin.ts | 60 ++ 3 files changed, 1550 insertions(+), 1 deletion(-) create mode 100644 packages/plugins/sandbox-providers/daytona/src/file-sync.ts diff --git a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts new file mode 100644 index 00000000000..df406dfbc16 --- /dev/null +++ b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts @@ -0,0 +1,787 @@ +import path from "node:path"; +import os from "node:os"; +import { promises as fs } from "node:fs"; +import { randomUUID } from "node:crypto"; +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; +import type { FileDownloadRequest, FileDownloadResponse, FileUpload, Sandbox } from "@daytonaio/sdk"; +import type { + PluginEnvironmentSyncResult, + PluginSyncFileMapping, + PluginSyncOperation, +} from "@paperclipai/plugin-sdk"; + +const execFileAsync = promisify(execFile); + +// Reserved scratch-name stem for staged uploads/downloads and remote tarballs. +// The runtime's base64 fallback stages to `.paperclip-upload`; the native +// transport reuses the same reserved prefix so a provider temp never collides +// with a real target or with the fallback's scratch name. +const SCRATCH_PREFIX = ".paperclip-upload"; + +function scratchName(suffix = ""): string { + return `${SCRATCH_PREFIX}-${randomUUID()}${suffix}`; +} + +/** + * Single-quote a path for safe interpolation into a sandbox shell command. Every + * path handed to `sandbox.process.executeCommand` (tar extract / `mv -f` rename) + * MUST pass through this so a path containing shell metacharacters is transferred + * literally, never interpreted. + */ +function shellQuote(value: string): string { + return `'${value.replace(/'/g, `'"'"'`)}'`; +} + +/** + * Convert a POSIX numeric mode (e.g. `0o600`) to the octal string the Daytona + * SDK's `setFilePermissions` expects (e.g. `"600"`), masked to the permission + * bits so an accidental type flag never widens the mode. + */ +function toOctalModeString(mode: number): string { + return (mode & 0o7777).toString(8).padStart(3, "0"); +} + +/** + * Host-side complete-mediation guard applied as defense-in-depth below the + * orchestrator's own confinement. Every sandbox-side path (the sync target for + * inbound, the sync source for outbound) MUST canonicalize inside the workspace + * remote dir; absolute escapes and `..` traversal are rejected fail-closed before + * any bytes move. Sandbox paths on the server are POSIX. + */ +export function assertConfinedSandboxPath(remoteDir: string, candidate: string, label: string): void { + const normalizedRoot = path.posix.normalize(remoteDir); + const normalized = path.posix.normalize(candidate); + if ( + !path.posix.isAbsolute(normalized) || + normalized === ".." || + normalized.includes("/../") || + normalized.endsWith("/..") + ) { + throw new Error(`Daytona sync ${label} path is not a confined absolute path: ${candidate}`); + } + const prefix = normalizedRoot.endsWith("/") ? normalizedRoot : `${normalizedRoot}/`; + if (normalized !== normalizedRoot && !normalized.startsWith(prefix)) { + throw new Error(`Daytona sync ${label} path escapes the workspace remote dir: ${candidate}`); + } +} + +async function withHostTempDir(fn: (dir: string) => Promise): Promise { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-sync-")); + try { + return await fn(dir); + } finally { + await fs.rm(dir, { recursive: true, force: true }).catch(() => undefined); + } +} + +/** + * Build a host-side tarball of a directory, mirroring the runtime's own + * `createTarballFromDirectory`: archive top-level entries by name (no "." self + * entry), suppress AppleDouble/xattr sidecars, honor `exclude`, and reproduce the + * `followSymlinks` → `-h` mapping so the native path is observationally identical + * to the base64 fallback's tar. + */ +async function createHostTarball(input: { + localDir: string; + archivePath: string; + exclude?: string[]; + followSymlinks?: boolean; +}): Promise { + const excludeArgs = ["._*", ...(input.exclude ?? [])].flatMap((entry) => ["--exclude", entry]); + const entries = (await fs.readdir(input.localDir)).sort((left, right) => left.localeCompare(right)); + if (entries.length === 0) { + // An empty source is valid (blank workspace / empty asset dir). Write a valid + // empty tar (1024-byte zero EOF marker) so extraction is a clean no-op. + await fs.writeFile(input.archivePath, Buffer.alloc(1024)); + return; + } + await execFileAsync( + "tar", + [ + "-c", + "--no-xattrs", + ...(input.followSymlinks ? ["-h"] : []), + "-f", + input.archivePath, + "-C", + input.localDir, + ...excludeArgs, + "--", + ...entries, + ], + { env: { ...process.env, COPYFILE_DISABLE: "1" }, maxBuffer: 32 * 1024 * 1024 }, + ); +} + +/** + * True when `relative` (a POSIX path) escapes its anchoring directory once + * normalized: an absolute path, `..`, or a `..`-leading traversal all break out. + */ +function posixPathEscapes(relative: string): boolean { + const normalized = path.posix.normalize(relative); + return normalized === ".." || normalized.startsWith("../") || path.posix.isAbsolute(normalized); +} + +/** + * Reject a sandbox-authored tarball before extraction if any member would land + * outside the extraction dir. The archive is produced by the (untrusted) sandbox, + * so `tar -xf` on the host must never be handed an archive whose entries carry + * absolute paths or `../` traversal, nor a symlink/hardlink member whose target + * escapes the tree — the latter would let a follow-up member be written through + * the link to an arbitrary host path. Legitimate in-tree relative links (targets + * that resolve back inside the archive, e.g. `shortcut -> nested/data.txt`) are + * preserved. Parses the `-tvf` verbose listing so both member names and link + * targets are inspected; any unparseable line fails closed. + */ +async function assertTarballEntriesConfined(archivePath: string): Promise { + const { stdout } = await execFileAsync("tar", ["-tvf", archivePath], { + env: { ...process.env, COPYFILE_DISABLE: "1" }, + maxBuffer: 32 * 1024 * 1024, + }); + const lines = stdout.split("\n").filter((line) => line.trim().length > 0); + for (const line of lines) { + // GNU tar -tvf: " /