## Thinking Path > - Paperclip manages AI agents and preserves their work when execution stops. > - Remote workspace recovery must save the retained files before another executor can start. > - Daytona directory export first writes a complete tar on the source sandbox. > - When that disk is full, each preservation retry fails while writing the same temporary archive. > - This change streams an archive after a storage-capacity failure, without another full copy on the sandbox. > - Existing source ownership, archive validation, and task admission gates remain in force. ## Linked Issues or Issue Description Related: #14402 established safe workspace export and retained-source recovery. This PR handles the remaining temporary-archive storage requirement. **What happened?** A sandbox ran out of disk space. The provider could no longer save its state, and workspace preservation then failed with tar `No space left on device` or `Wrote only ... bytes`. Repeating the export could not resolve that failure because it required free space on the same disk. **Expected behavior** Export retained files without needing a second full copy on the source disk. Reject incomplete transfers and unsafe archives. Preserve the source and newer host files if export fails. **Steps to reproduce** Run the capacity regression with real tar and a file-size limit. The original export fails during archive creation. The fixed export transfers the same bytes through a real PTY. The credential-gated live test uses a real Daytona sandbox and `/dev/full` as the archive sink to reproduce ENOSPC without filling a provider disk. **Paperclip version or commit** Reproduced on `27be859044`; fix rebased onto `6d59f5ef7e`. **Deployment mode** Source checkout on macOS arm64; live provider verification on a disposable Ubuntu 24.04 Daytona sandbox with SDK 0.203.0. ## What Changed - Fall back only after archive storage-capacity errors. Keep the ordinary bulk download path. - Transfer 40 KiB archive chunks over a raw PTY. Wait for each host write acknowledgement and a final receipt acknowledgement. Send long scripts only after raw-mode readiness. - Require successful tar exit and complete framing before the existing host archive validator can import files. Keep the bounded safe-link retry and exclude rules. - Bound total archive bytes by the provider-reported disk allocation, preserve a 1 GiB host free-space reserve, and bound terminal cleanup to five seconds while closing the host file independently. - Add real-tar/PTY regressions, stream failure tests, a credential-gated provider test, and recovery documentation. ## Verification - Red before production changes: the capacity regression failed on the real tar write with exit 153. Green afterward: unchanged bytes, file mode, confined links, and source files survive. A tar producer that fails after sending bytes cannot overwrite newer host files. - 43 focused tests pass. Missing receipts, nonzero exits, transport loss, invalid base64, oversized frames, stalled transfers, and late connection cleanup fail closed. - Live Daytona test passes: stop/start, actual tar ENOSPC, matching file hashes and modes, unsafe-link filtering, intact source, then another stop/start and export with a later saved file. The disposable sandbox was deleted. Initial default-sandbox creation hit account quota; the successful test used a 1 GiB image. Large output bursts exposed a real socket closure, corrected by bounded chunks and verified in the successful run. - Repository `pnpm -r typecheck` and `pnpm build` pass. `node scripts/check-maps.mjs` passes. Standalone plugin typecheck passes. - Standalone plugin build and final typecheck pass. The final broad plugin run passed 482 tests, skipped seven credential-gated cases, and hit one existing 30-second archive-listing timeout. Earlier runs hit both large-listing cases. Both timeouts reproduce with unchanged `file-sync.ts` from master `6d59f5ef7e`, with 22 other archive-listing cases passing on both base and fixed source. Parallel, serial, and GNU-tar local runs retain those timeouts. The serial local `pnpm test:run` was stopped after 36 minutes once the complete current-head CI test suite passed; it is not counted as a full local pass. GitHub Actions on production-code head `9ffbd9e56e283a01c22f005c022b8f6838cf0143` passed typecheck, all general and serialized test shards, Runner verification, build, E2E, and canary dry run. - Review fixes have passed the live Daytona test again. Sparse-file budget enforcement, rejection before host writes, low host reserve, and stalled provider cleanup are covered by the 43 focused tests. Greptile is 5/5 and security scanning passes on `9ffbd9e56e283a01c22f005c022b8f6838cf0143`; both review threads are resolved. - Test portability: `d2280108f5` verifies GNU tool versions and Python before running the real-PTY regression. All 43 focused tests pass with prerequisites installed; restricting PATH to the macOS system tools cleanly skips the three integration cases. Standalone plugin typecheck passes. Current-head CI passes typecheck, build, all test shards, Runner verification, E2E, and canary dry run: https://github.com/paperclipai/paperclip/actions/runs/40910736373. Greptile and Paperclip Review both assess the final head 5/5, security checks pass, and no review threads remain unresolved. - Paperclip Review assessed the final head 5/5 with no actionable defects after the test-portability fix. Remaining accepted tradeoffs are recorded under Risks. - This is live provider-boundary acceptance. It does not prove a full task Retry journey, repair lost provider session state, or recover the original deployed task. ## Risks - Streaming requires Bash and GNU tar, dd, and base64. Missing tools or transfer failure reject export and retain the recovery obligation. - Acknowledged terminal transfer is slower than bulk download, especially across high-latency links. Very large exports can still reach the configured deadline. The ordinary bulk path is unchanged. - The fallback checks host free space before every 40 KiB write; this deliberately favors a strict reserve check over reducing local syscalls. It carries archive-download error context but does not yet emit a dedicated transfer span. Provider cleanup is bounded; a kernel-stalled local filesystem operation can still delay completion and file close. - This change does not increase sandbox capacity or reconstruct provider state that failed to persist. The plugin must reach the target installation before recovery can use it. No merge, deployment, or original-task replay is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code editing, shell execution, and automated testing. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
8.8 KiB
@paperclipai/plugin-daytona
Published Daytona sandbox provider plugin for Paperclip.
This package lives in the Paperclip monorepo, but it is intentionally excluded from the root pnpm workspace and shaped to publish and install like a standalone npm package. That lets operators install it from the Plugins page by package name without introducing root lockfile churn for Daytona's SDK dependencies.
Install
From a Paperclip instance, install:
@paperclipai/plugin-daytona
The host plugin installer runs npm install into the managed plugin directory, so transitive dependencies such as @daytonaio/sdk are pulled in during installation.
Configuration
Configure Daytona from Instance Settings -> Environments, not from the plugin's plugin page.
- Put the Daytona API key on the sandbox environment itself.
- When you save an environment, Paperclip stores pasted API keys as company secrets.
DAYTONA_API_KEYremains an optional host-level fallback when an environment omits the key.- Optional
apiUrlandtargetsettings map directly to the Daytona SDK/client configuration. IfapiUrlis omitted, the Daytona SDK uses its default endpoint.
Notes:
- The current published Daytona SDK package is
@daytonaio/sdk. - The driver supports both
snapshot-based andimage-based sandbox creation. If both are set, validation rejects the config as ambiguous. - Optional
ttlMinutessets a positive whole-minute wall-clock lifetime in the creation request. It applies to image and snapshot sandboxes, including probes and interactive setup. The driver requires the provider to confirm the expiry before it returns the sandbox. It preserves an earlier creation expiry when a caller supplies a later deadline. Daytona destroys the sandbox and its files at expiry, even while active, stopped, paused, or archived. Omit this setting to keep existing lifetime behavior. This is separate from inactivity cleanup. - Each cold create uses a unique provider name and ownership labels. If creation fails after Daytona has allocated a sandbox, the driver looks up that exact name, checks every ownership label, and waits for deletion. Failed, missing, or timed-out lookups and failed deletions report unconfirmed cleanup with the provider name; they never return a usable lease. A missing name lookup after an uncertain create is not proof that a delayed provider request cannot create a resource.
- For plugin-backed sandbox lease acquisition, unresolved creation cleanup crosses the worker RPC as a validated ownership envelope. The host records
pending_cleanupbefore retrying deletion. If only a creation name is known, the provider first returns an ownership-verified sandbox ID without deleting it. The host saves that observation before authorizing deletion. Its existing cleanup sweep and durable spool preserve retries across controller restarts and environment deletion. A missing observed ID confirms cleanup after a lost deletion reply or database update; a name that has never been observed remains unresolved. Provider exceptions and resolved credentials are excluded from that envelope. This requires the matching host and plugin SDK update; probe and custom-image interactive-setup calls still report unconfirmed immediate cleanup without this lease-recovery path. - Reusable leases map to Daytona stop/start semantics. Non-reusable leases are deleted on release. A provider-resolved
targetdoes not change the identity of an existing sandbox. Release closes the same scoped lease that a later sentinel-verified resume reopens. - Idle sleep tracks pending hooks, underlying provider requests, open terminals, and sandbox allocations. A successful stop or delete clears only that sandbox's company, environment, and account scope. Every sandbox deletion waits for completed remote destruction within its existing timeout; acceptance alone cannot clear idle evidence. A timeout does not settle the underlying request. Cleanup that overlaps other work on the same sandbox needs a later quiet stop/delete receipt or read-only observation of the exact stopped or missing sandbox. Proof is captured before the provider observation. Other work that starts or finishes before its reply invalidates that proof, even after the other request has settled. An unobserved creation name is never absence proof. Active resources and uncertain cleanup continue to prevent idle sleep; verified cleanup no longer blocks sleep for the worker's entire lifetime.
- A command that has already exited returns its final separated log snapshot without opening a follow socket. The initial status check is bounded to 500ms and the caller's command deadline. Each follow connection also makes up to four early status reads, with 25/50/100/200ms delays and the same observation/deadline bounds. A confirmed exit still requires the final separated snapshot, but does not wait for the follow socket to close. Unconfirmed or unavailable status keeps streaming; the early reads stop after their bounded attempt window. The command is dispatched only once.
- A session log socket can close while its command still runs. The driver requires a recorded command exit before returning a completed execution. It reconnects the log stream once, then polls status and log snapshots at most once per second. Replayed output is removed by byte offset, new output continues to reach the host, and the command is never dispatched again. A socket that delivers no output for 15 seconds switches directly to polling, even if the socket stays open or its initial connection never completes. Late callbacks from that socket are ignored. This recovers pending permission requests and cancellation results from saved output without replaying the command. Healthy initial and reconnected streams retain their existing lifetime under the caller’s RPC/run guard. After a clean close or an idle socket, each status or snapshot read gets a fresh operation timeout; successful recovery polling does not impose a new command lifetime. When both stream attempts fail without a clean close, fallback retains its original timeout budget starting after the stream attempts.
- The SDK returns full log snapshots, so fallback bandwidth grows with retained output; it has no paged snapshot or stream cancellation API. A recovery-read timeout keeps partial output and explicitly reports whether command exit remains unconfirmed; it does not prove the remote command stopped. Provider/session teardown remains responsible for closing an outstanding socket. A final snapshot reconciles bytes missed by the stream before returning a recorded exit.
- A sandbox record can survive the loss of its underlying container. Resume treats it as expired only when a fresh provider read confirms the exact missing-container error for that sandbox and marks it unrecoverable. Unknown errors and failed confirmation reads preserve the lease. The host still requires a verified native-runner backup before replacement.
- Directory export normally builds a sandbox archive and downloads it. If archive creation exhausts disk space, quota, or the file-size limit, export streams the archive through a raw terminal instead. Each bounded chunk waits for a host write acknowledgement. Import still requires successful tar completion and the existing archive/path validation. The total archive cannot exceed the provider-reported sandbox disk allocation, and each host write must leave a 1 GiB free-space reserve. This fallback writes no sandbox archive or command log, preserves source files, and retains the configured transfer timeout. Terminal cleanup has a separate five-second limit and cannot hold the host archive open. It requires Bash and GNU
tar,dd, andbase64in the sandbox. It does not repair a provider session that already failed to save its state or increase the sandbox's disk allocation.
Local development
cd packages/plugins/sandbox-providers/daytona
pnpm install --ignore-workspace --no-lockfile
pnpm build
pnpm test
pnpm typecheck
These commands assume the repo root has already been installed once so the local @paperclipai/plugin-sdk workspace package is available to the compiler during development.
The capacity regression uses Python 3 for a real local PTY, Bash, and GNU core
utilities (gtar, gdd, and gbase64 on macOS). It skips when those prerequisites
are unavailable or their version checks do not match. With DAYTONA_API_KEY set, run
pnpm test src/file-sync-capacity.live.test.ts to exercise an actual stopped and
resumed Daytona sandbox, a tar write to /dev/full, validated export, and a
second export after reconnect. The test creates and deletes one disposable
1 GiB Ubuntu sandbox; it does not fill a disk or modify existing sandboxes.
Package layout
src/manifest.tsdeclares the sandbox-provider driver metadatasrc/plugin.tsimplements the environment lifecycle hookspaperclipPlugin.manifestandpaperclipPlugin.workerpoint the host at the built plugin entrypoints indist/