Files
PaperClipAI/packages/plugins/sandbox-providers/daytona
DottaandPaperclip 89e3c23062 fix(daytona): stream files when export copies cannot fit (#18712)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native task must save its files before finalization can release
its writer.
> - Remote workspace capture can produce an archive that fits on the
sandbox disk.
> - Daytona then makes another copy before it downloads that archive.
> - The second copy can run out of space and leave completed work behind
a save hold.
> - This pull request uses the existing bounded stream when that copy
cannot fit.
> - The benefit is a saved archive without a second archive-sized
sandbox allocation.

## Linked Issues or Issue Description

Related: #18644 reduced capture scratch space. #18172 added the
disk-free directory export. #18705 preserves provider checkpoints on a
full disk and reports capture failures. This change handles the later
single-file download copy. It does not depend on #18705 and does not
replace its checkpoint fix.

**What happened?**

A native provider turn returned a successful result and captured its
workspace archive. The Daytona outbound file guard then failed with exit
43 and `No space left on device` while it copied the archive to a
reserved download file. Finalization and file-preservation retries could
not pass this boundary. The workspace kept its writer hold.

**Expected behavior**

Download an existing archive through a confined sandbox process when its
second copy cannot fit. Keep the source and all finalization holds until
the normal save commits.

**Steps to reproduce**

1. Capture a workspace archive in a Daytona sandbox.
2. Leave less free space than the archive needs for another copy.
3. Finalize the task. The outbound single-file guard fails before
download.

**Paperclip version or commit**

Reproduced against `6d72243d8a`. The incident used the native sandbox
runtime. This branch is based on current master.

**Deployment mode**

Cloud with native Paperclip Runner execution and remote workspace
storage.

## What Changed

- Remove partial outbound file copies after a failed snapshot command.
Only copy capacity failures select the stream path. Permission,
confinement, and transport errors still fail.
- Use the existing acknowledged terminal stream to read each file as the
sandbox user. Recheck canonical confinement and regular-file status. Use
a no-follow open. Keep the sandbox disk limit, host free-space reserve,
shared transfer deadline, and bounded teardown.
- Keep host files in private temporary files until the complete batch
has transfer receipts. Never pass a mutable live source to the
privileged download API.
- Add real-file and real-terminal regressions, a disposable Daytona
acceptance case, and workspace verification notes.

## Verification

- Red before repair: a real 5 MiB copy under a controlled file-size
limit failed with exit 43. The same test passes after repair and
compares every source and destination byte.
- The ten capacity cases pass. They cover producer failure after output,
source replacement, stream size limits, permission errors, timeout
teardown, cleanup, and a successful retry. A two-file case completes the
first stream and fails the second. Both old host files stay intact, both
source files survive, and retry saves both files. An isolated
early-publication mutation makes that test fail.
- The initial complete Daytona package suite passes: 537 tests passed, 8
skipped. After the test-only review follow-up, all 60 focused transfer,
stream, and diagnostic tests pass. The package typecheck also passes.
- Real Daytona acceptance passes: an owned Ubuntu 24.04 sandbox stops
and resumes, a copy to `/dev/full` reports ENOSPC, production export
saves the exact archive hash, and a second save after reconnect passes.
The test deletes only its disposable sandbox. This is provider-boundary
acceptance, not a UI journey.
- Assembled finalization acceptance passes with real Postgres and a live
Linux x86_64 Daytona sandbox. The production control-plane port accepts
one fixture result. The production workspace finalizer encounters the
failed copy, saves the files, and commits the task as `done`. A new
database client reads the committed result and exact file hash. The
workspace has no writer or running save, and there is still one run and
one result. No model turn runs. The target resolver is bound to the live
provider fixture. Object storage uses the production encryption and
local-disk service with an S3 admission label; this does not prove an S3
network transfer or a UI journey.
- The live tests used the fixed transfer source with SHA-256
`857b3d8c9ad75170707d95277c17c051ce4582fa79a9c38331d51d2800b36af5`. That
source is identical in this PR head.
- `pnpm -r typecheck` passes locally. The duplicate full local test run
passed all six chat shards before it was stopped during the server lane
in favor of completed current-head CI; a duplicate local build was
skipped. Full current-head CI passes, including typecheck, build,
general and serialized tests, Rust/Runner checks, all eight E2E shards,
and the `ci / e2e` and `ci / verify` aggregate gates.
- Greptile is 5/5 on the current commit with no unresolved review
threads.
- `node scripts/check-maps.mjs` and `git diff --check` pass. Map checks
prove structure only.

## Risks

- The fallback requires the Bash and GNU `dd`/`base64` tools already
required by the existing stream path.
- Terminal streaming is slower than the normal bulk download. It runs
only after a copy capacity failure.
- The workspace capture itself still needs sandbox space. This change
does not fix a full disk before an archive exists.
- The test injects a failed copy sink. It does not exhaust a shared host
disk. No original task was retried, and no deployment or merge is part
of this PR.
- No schema, API permission, writer ownership, or retry-budget change.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
editing, shell tools, browser diagnostics, and automated tests. The
exact serving variant and context-window size are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 06:11:41 -06:00
..

@paperclipai/plugin-daytona

Published Daytona sandbox provider plugin for Paperclip.

This package lives in the Paperclip monorepo, but it is intentionally excluded from the root pnpm workspace and shaped to publish and install like a standalone npm package. That lets operators install it from the Plugins page by package name without introducing root lockfile churn for Daytona's SDK dependencies.

Install

From a Paperclip instance, install:

@paperclipai/plugin-daytona

The host plugin installer runs npm install into the managed plugin directory, so transitive dependencies such as @daytonaio/sdk are pulled in during installation.

Configuration

Configure Daytona from Instance Settings -> Environments, not from the plugin's plugin page.

  • Put the Daytona API key on the sandbox environment itself.
  • When you save an environment, Paperclip stores pasted API keys as company secrets.
  • DAYTONA_API_KEY remains an optional host-level fallback when an environment omits the key.
  • Optional apiUrl and target settings map directly to the Daytona SDK/client configuration. If apiUrl is omitted, the Daytona SDK uses its default endpoint.

Notes:

  • The current published Daytona SDK package is @daytonaio/sdk.
  • The driver supports both snapshot-based and image-based sandbox creation. If both are set, validation rejects the config as ambiguous.
  • Optional ttlMinutes sets a positive whole-minute wall-clock lifetime in the creation request. It applies to image and snapshot sandboxes, including probes and interactive setup. The driver requires the provider to confirm the expiry before it returns the sandbox. It preserves an earlier creation expiry when a caller supplies a later deadline. Daytona destroys the sandbox and its files at expiry, even while active, stopped, paused, or archived. Omit this setting to keep existing lifetime behavior. This is separate from inactivity cleanup.
  • Each cold create uses a unique provider name and ownership labels. If creation fails after Daytona has allocated a sandbox, the driver looks up that exact name, checks every ownership label, and waits for deletion. Failed, missing, or timed-out lookups and failed deletions report unconfirmed cleanup with the provider name; they never return a usable lease. A missing name lookup after an uncertain create is not proof that a delayed provider request cannot create a resource.
  • For plugin-backed sandbox lease acquisition, unresolved creation cleanup crosses the worker RPC as a validated ownership envelope. The host records pending_cleanup before retrying deletion. If only a creation name is known, the provider first returns an ownership-verified sandbox ID without deleting it. The host saves that observation before authorizing deletion. Its existing cleanup sweep and durable spool preserve retries across controller restarts and environment deletion. A missing observed ID confirms cleanup after a lost deletion reply or database update; a name that has never been observed remains unresolved. Provider exceptions and resolved credentials are excluded from that envelope. This requires the matching host and plugin SDK update; probe and custom-image interactive-setup calls still report unconfirmed immediate cleanup without this lease-recovery path.
  • Reusable leases map to Daytona stop/start semantics. Non-reusable leases are deleted on release. A provider-resolved target does not change the identity of an existing sandbox. Release closes the same scoped lease that a later sentinel-verified resume reopens.
  • Idle sleep tracks pending hooks, underlying provider requests, open terminals, and sandbox allocations. A successful stop or delete clears only that sandbox's company, environment, and account scope. Every sandbox deletion waits for completed remote destruction within its existing timeout; acceptance alone cannot clear idle evidence. A timeout does not settle the underlying request. Cleanup that overlaps other work on the same sandbox needs a later quiet stop/delete receipt or read-only observation of the exact stopped or missing sandbox. Proof is captured before the provider observation. Other work that starts or finishes before its reply invalidates that proof, even after the other request has settled. An unobserved creation name is never absence proof. Active resources and uncertain cleanup continue to prevent idle sleep; verified cleanup no longer blocks sleep for the worker's entire lifetime.
  • A command that has already exited returns its final separated log snapshot without opening a follow socket. The initial status check is bounded to 500ms and the caller's command deadline. Each follow connection also makes up to four early status reads, with 25/50/100/200ms delays and the same observation/deadline bounds. A confirmed exit still requires the final separated snapshot, but does not wait for the follow socket to close. Unconfirmed or unavailable status keeps streaming; the early reads stop after their bounded attempt window. The command is dispatched only once.
  • A session log socket can close while its command still runs. The driver requires a recorded command exit before returning a completed execution. It reconnects the log stream once, then polls status and log snapshots at most once per second. Replayed output is removed by byte offset, new output continues to reach the host, and the command is never dispatched again. A socket that delivers no output for 15 seconds switches directly to polling, even if the socket stays open or its initial connection never completes. Late callbacks from that socket are ignored. This recovers pending permission requests and cancellation results from saved output without replaying the command. Healthy initial and reconnected streams retain their existing lifetime under the caller’s RPC/run guard. After a clean close or an idle socket, each status or snapshot read gets a fresh operation timeout; successful recovery polling does not impose a new command lifetime. When both stream attempts fail without a clean close, fallback retains its original timeout budget starting after the stream attempts.
  • The SDK returns full log snapshots, so fallback bandwidth grows with retained output; it has no paged snapshot or stream cancellation API. A recovery-read timeout keeps partial output and explicitly reports whether command exit remains unconfirmed; it does not prove the remote command stopped. Provider/session teardown remains responsible for closing an outstanding socket. A final snapshot reconciles bytes missed by the stream before returning a recorded exit.
  • A sandbox record can survive the loss of its underlying container. Resume treats it as expired only when a fresh provider read confirms the exact missing-container error for that sandbox and marks it unrecoverable. Unknown errors and failed confirmation reads preserve the lease. The host still requires a verified native-runner backup before replacement.
  • Directory export normally builds a sandbox archive and downloads it. If archive creation exhausts disk space, quota, or the file-size limit, export streams the archive through a raw terminal instead. Each bounded chunk waits for a host write acknowledgement. Import still requires successful tar completion and the existing archive/path validation. The total archive cannot exceed the provider-reported sandbox disk allocation, and each host write must leave a 1 GiB free-space reserve. This fallback writes no sandbox archive or command log, preserves source files, and retains the configured transfer timeout. Terminal cleanup has a separate five-second limit and cannot hold the host archive open. It requires Bash and GNU tar, dd, and base64 in the sandbox. It does not repair a provider session that already failed to save its state or increase the sandbox's disk allocation.
  • Single-file export normally downloads a guarded sandbox copy. If that copy exhausts disk space, quota, or the file-size limit, the provider removes partial copies and streams the source through the same bounded sandbox-user channel. The stream rechecks canonical confinement and regular-file identity, refuses a symlink at open, and publishes host files only after a successful transfer receipt. It never hands a mutable source to the privileged download API. Permission, confinement, and transport failures still fail; a failed stream leaves prior host files intact. This removes the second archive-sized copy needed to download an already captured workspace archive. Creating the workspace archive itself still needs sandbox space.

Local development

cd packages/plugins/sandbox-providers/daytona
pnpm install --ignore-workspace --no-lockfile
pnpm build
pnpm test
pnpm typecheck

These commands assume the repo root has already been installed once so the local @paperclipai/plugin-sdk workspace package is available to the compiler during development.

The capacity regression uses Python 3 for a real local PTY, Bash, and GNU core utilities (gtar, gdd, and gbase64 on macOS). It skips when those prerequisites are unavailable or their version checks do not match. With DAYTONA_API_KEY set, run pnpm test src/file-sync-capacity.live.test.ts to exercise an actual stopped and resumed Daytona sandbox, tar and file-copy writes to /dev/full, validated exports, and second exports after reconnect. Each case creates and deletes one disposable 1 GiB Ubuntu sandbox; it does not fill a disk or modify existing sandboxes.

Package layout

  • src/manifest.ts declares the sandbox-provider driver metadata
  • src/plugin.ts implements the environment lifecycle hooks
  • paperclipPlugin.manifest and paperclipPlugin.worker point the host at the built plugin entrypoints in dist/

Interrupted run-lease creation

The host commits a company-scoped creation operation before acquiring a new run lease. The operation UUID becomes paperclip-create-<uuid> and the paperclip-create-attempt label. The existing inspection hook can resolve that name and validate company, environment and run labels without starting a sandbox. Recovery persists the observed provider ID before the existing cleanup worker deletes it. A missing creation name is an uncertain outcome, not proof of removal. The host retains that operation for manual reconciliation and never repeats the creation request blindly. Credential identity must still match admission.

This capability is declared as creationOperationVersion: 1 in manifest version 0.1.10. It covers normal run/ad-hoc lease acquisition and plugin task replacement. Legacy provider environmentProbe and custom-image interactive setup have separate lifecycles and are outside this lease-orphan fallback replacement. The normal Test path which acquires a runtime lease is covered. Unit fixtures exercise lookup and deletion; live provider qualification requires an explicit disposable test account.