## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Cloud agent runs use the Daytona sandbox provider plugin. One shared plugin worker process serves all Daytona calls on a server. > - The host closes a duplex channel through the worker. The host waits 10 seconds for the close acknowledgement. When the acknowledgement is late, the host retires the whole worker with SIGKILL. > - The worker close waits for the remote PTY kill before it releases the socket. The Daytona SDK sends that kill as a toolbox request with a 24-hour default timeout. A stalled sandbox can hold it open for longer than 10 seconds. > - Then the host kills the shared worker. Every other call in flight on that worker fails with `Worker process exited (code=null, signal=SIGKILL)`. This includes other runs' workspace preparation and restore. > - This pull request limits the wait for the remote kill to 5 seconds. Then the worker releases the socket and acknowledges the close. > - The benefit is that one slow sandbox no longer stops unrelated runs on the same server. ## Linked Issues or Issue Description **What happened?** On a hosted staging server, the host logged `duplex channel close not acknowledged; retiring worker` six times in about 90 minutes. Each time, the Daytona worker exited with SIGKILL. Each time, all in-flight calls on that worker failed. Runs in workspace preparation failed with `setup_failed` and `Worker process exited (code=null, signal=SIGKILL)`. Several runs failed together in each burst. **Expected behavior** A slow close of one duplex channel does not stop calls that belong to other channels or other runs. **Steps to reproduce** 1. Open a duplex channel on a Daytona lease. 2. Make the Daytona PTY kill request stall (the SDK default request timeout is 24 hours). 3. Close the channel from the host. The worker does not acknowledge within 10 seconds. 4. The host retires the worker with SIGKILL. All other pending calls fail. **Relevant logs or output** ``` info duplex channel close not acknowledged; retiring worker info retired worker process exited code=null signal=SIGKILL willRestart=true JsonRpcCallError: Worker process exited (code=null, signal=SIGKILL) ``` **Additional context** The login PTY close does not have this problem. It only does the local socket disconnect. A kill that rejects already gets a close acknowledgement, because the session close ignores kill errors. This change gives a slow kill the same result. No related public PR exists. I searched open and closed PRs for this failure. ## What Changed - `duplex-command-stream.ts`: `close()` waits for the remote `kill()` for at most `DUPLEX_CHANNEL_CLOSE_KILL_TIMEOUT_MS` (5 seconds). Then it always releases the socket with `disconnect()`. The kill continues in the background. A kill that answers in time keeps the old order: kill, then disconnect, then the acknowledgement. - `duplex-close-deadline.test.ts` (new): regression tests at the session close and at the worker `onDuplexChannelClose` boundary, and a check that the deadline stays well inside the host deadline. ## Verification - Red before the fix: with the old `duplex-command-stream.ts`, `acknowledges a bound close inside the host deadline when the remote kill never settles` fails (no acknowledgement after 9 seconds), and the session close test times out. - Green after the fix: `pnpm --dir packages/plugins/sandbox-providers/daytona exec vitest run --config vitest.config.ts src/duplex-close-deadline.test.ts src/duplex-command-stream.test.ts src/plugin.test.ts` passes 453 of 453. This includes the existing tests `joins pending teardown for concurrent closes while all channel authority stays revoked` and `does not turn a rejected retired close into an acknowledgement`. - The safety case `still waits for a kill that answers inside the deadline before it releases the socket` passes before and after the fix. - `tsc --noEmit` for the Daytona plugin passes. - In the full Daytona plugin suite, two GNU tar tests in `file-sync.test.ts` fail in my sandbox. They also fail on unchanged `master` in the same sandbox, so they are environment failures. CI covers them. - I did not run a live staging workflow. After the deploy, the staging logs must show no new `duplex channel close not acknowledged; retiring worker` lines from slow kills. ## Risks - Low risk. The change is in one Daytona plugin function. - After a kill timeout, the gateway process can stay alive in the sandbox until the background kill completes or the lease teardown stops the sandbox. The socket is closed, so no bytes can move on the channel. The route stays retired, so the host route cannot open again. This is the same state as when a kill rejects today. - The host contract does not change. A malformed or mismatched acknowledgement still retires the worker. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
@paperclipai/plugin-daytona
Published Daytona sandbox provider plugin for Paperclip.
This package lives in the Paperclip monorepo, but it is intentionally excluded from the root pnpm workspace and shaped to publish and install like a standalone npm package. That lets operators install it from the Plugins page by package name without introducing root lockfile churn for Daytona's SDK dependencies.
Install
From a Paperclip instance, install:
@paperclipai/plugin-daytona
The host plugin installer runs npm install into the managed plugin directory, so transitive dependencies such as @daytonaio/sdk are pulled in during installation.
Configuration
Configure Daytona from Instance Settings -> Environments, not from the plugin's plugin page.
- Put the Daytona API key on the sandbox environment itself.
- When you save an environment, Paperclip stores pasted API keys as company secrets.
DAYTONA_API_KEYremains an optional host-level fallback when an environment omits the key.- Optional
apiUrlandtargetsettings map directly to the Daytona SDK/client configuration. IfapiUrlis omitted, the Daytona SDK uses its default endpoint.
Notes:
- The current published Daytona SDK package is
@daytonaio/sdk. - The driver supports both
snapshot-based andimage-based sandbox creation. If both are set, validation rejects the config as ambiguous. - Optional
ttlMinutessets a positive whole-minute wall-clock lifetime in the creation request. It applies to image and snapshot sandboxes, including probes and interactive setup. The driver requires the provider to confirm the expiry before it returns the sandbox. It preserves an earlier creation expiry when a caller supplies a later deadline. Daytona destroys the sandbox and its files at expiry, even while active, stopped, paused, or archived. Omit this setting to keep existing lifetime behavior. This is separate from inactivity cleanup. - Each cold create uses a unique provider name and ownership labels. If creation fails after Daytona has allocated a sandbox, the driver looks up that exact name, checks every ownership label, and waits for deletion. Failed, missing, or timed-out lookups and failed deletions report unconfirmed cleanup with the provider name; they never return a usable lease. A missing name lookup after an uncertain create is not proof that a delayed provider request cannot create a resource.
- For plugin-backed sandbox lease acquisition, unresolved creation cleanup crosses the worker RPC as a validated ownership envelope. The host records
pending_cleanupbefore retrying deletion. If only a creation name is known, the provider first returns an ownership-verified sandbox ID without deleting it. The host saves that observation before authorizing deletion. Its existing cleanup sweep and durable spool preserve retries across controller restarts and environment deletion. A missing observed ID confirms cleanup after a lost deletion reply or database update; a name that has never been observed remains unresolved. Provider exceptions and resolved credentials are excluded from that envelope. This requires the matching host and plugin SDK update; probe and custom-image interactive-setup calls still report unconfirmed immediate cleanup without this lease-recovery path. - Reusable leases map to Daytona stop/start semantics. Non-reusable leases are deleted on release. A provider-resolved
targetdoes not change the identity of an existing sandbox. Release closes the same scoped lease that a later sentinel-verified resume reopens. - Idle sleep tracks pending hooks, underlying provider requests, open terminals, and sandbox allocations. A successful stop or delete clears only that sandbox's company, environment, and account scope. Every sandbox deletion waits for completed remote destruction within its existing timeout; acceptance alone cannot clear idle evidence. A timeout does not settle the underlying request. Cleanup that overlaps other work on the same sandbox needs a later quiet stop/delete receipt or read-only observation of the exact stopped or missing sandbox. Proof is captured before the provider observation. Other work that starts or finishes before its reply invalidates that proof, even after the other request has settled. An unobserved creation name is never absence proof. Active resources and uncertain cleanup continue to prevent idle sleep; verified cleanup no longer blocks sleep for the worker's entire lifetime.
- A command that has already exited returns its final separated log snapshot without opening a follow socket. The initial status check is bounded to 500ms and the caller's command deadline. Each follow connection also makes up to four early status reads, with 25/50/100/200ms delays and the same observation/deadline bounds. A confirmed exit still requires the final separated snapshot, but does not wait for the follow socket to close. Unconfirmed or unavailable status keeps streaming; the early reads stop after their bounded attempt window. The command is dispatched only once.
- A session log socket can close while its command still runs. The driver requires a recorded command exit before returning a completed execution. It reconnects the log stream once, then polls status and log snapshots at most once per second. Replayed output is removed by byte offset, new output continues to reach the host, and the command is never dispatched again. A socket that delivers no output for 15 seconds switches directly to polling, even if the socket stays open or its initial connection never completes. Late callbacks from that socket are ignored. This recovers pending permission requests and cancellation results from saved output without replaying the command. Healthy initial and reconnected streams retain their existing lifetime under the caller’s RPC/run guard. After a clean close or an idle socket, each status or snapshot read gets a fresh operation timeout; successful recovery polling does not impose a new command lifetime. When both stream attempts fail without a clean close, fallback retains its original timeout budget starting after the stream attempts.
- The SDK returns full log snapshots, so fallback bandwidth grows with retained output; it has no paged snapshot or stream cancellation API. A recovery-read timeout keeps partial output and explicitly reports whether command exit remains unconfirmed; it does not prove the remote command stopped. Provider/session teardown remains responsible for closing an outstanding socket. A final snapshot reconciles bytes missed by the stream before returning a recorded exit.
- A sandbox record can survive the loss of its underlying container. Resume treats it as expired only when a fresh provider read confirms the exact missing-container error for that sandbox and marks it unrecoverable. Unknown errors and failed confirmation reads preserve the lease. The host still requires a verified native-runner backup before replacement.
- Daytona rejects Stop for a sandbox in its
errorstate, for example after the sandbox disk fills. The stop-only hook confirms the retention policy first. If Daytona marks the error recoverable, the hook calls Recover once and waits at most 20 seconds for the sandbox to start. Then it stops the sandbox normally. The hook never calls Recover a second time in the same call. An unrecoverable error, a sandbox that is back inerrorafter recovery, or an unfinished recovery returns a typedPluginEnvironmentStopErrorinstead of the generic stop error. The sandbox is never deleted. - Directory export normally builds a sandbox archive and downloads it. If archive creation exhausts disk space, quota, or the file-size limit, export streams the archive through a raw terminal instead. The sender can send up to sixteen 40 KiB chunks before it waits for their host write acknowledgements. The host writes them in order and rejects a sender that exceeds the pending-frame limit. This bounds queued payloads to 640 KiB while avoiding one network round trip per chunk. Import still requires successful tar completion and the existing archive/path validation. The total archive cannot exceed the provider-reported sandbox disk allocation, and each host write must leave a 1 GiB free-space reserve. This fallback writes no sandbox archive or command log, preserves source files, and retains the configured transfer timeout. Terminal cleanup has a separate five-second limit and cannot hold the host archive open. It requires Bash and GNU
tar,dd, andbase64in the sandbox. It does not repair a provider session that already failed to save its state or increase the sandbox's disk allocation. - Single-file export normally downloads a guarded sandbox copy. If that copy exhausts disk space, quota, or the file-size limit, the provider removes partial copies and streams the source through the same bounded sandbox-user channel. The stream rechecks canonical confinement and regular-file identity, refuses a symlink at open, and publishes host files only after a successful transfer receipt. It never hands a mutable source to the privileged download API. Permission, confinement, and transport failures still fail; a failed stream leaves prior host files intact. This removes the second archive-sized copy needed to download an already captured workspace archive. Creating the workspace archive itself still needs sandbox space.
Native file transfers accept an optional minimumTimeoutMs from the host. The
minimum must be a positive whole number no greater than 30 minutes. File sync
uses the larger of this minimum and the configured timeout. Sandbox start and
command timeouts keep their configured limits. Without the field, behavior is
unchanged.
Local development
cd packages/plugins/sandbox-providers/daytona
pnpm install --ignore-workspace --no-lockfile
pnpm build
pnpm test
pnpm exec vitest run --config vitest.sdk-ingress.config.ts
pnpm typecheck
These commands assume the repo root has already been installed once so the local @paperclipai/plugin-sdk workspace package is available to the compiler during development.
The dedicated vitest.sdk-ingress.config.ts requires this package's installed, pinned Daytona SDK. It sends
local HTTP faults through the real SDK and worker RPC without a provider account
or sandbox. The root workspace does not install standalone provider dependencies;
its normal Daytona tests use mocked SDK boundaries. This target fails if the SDK
is missing and runs all ingress acceptance cases without a conditional skip.
The capacity regression uses Python 3 for a real local PTY, Bash, and GNU core
utilities (gtar, gdd, and gbase64 on macOS). It skips when those prerequisites
are unavailable or their version checks do not match. With DAYTONA_API_KEY set, run
pnpm test src/file-sync-capacity.live.test.ts to exercise an actual stopped and
resumed Daytona sandbox, tar and file-copy writes to /dev/full, validated
exports, and second exports after reconnect. Each case creates and deletes one
disposable 1 GiB Ubuntu sandbox; it does not fill a disk or modify existing sandboxes.
Package layout
src/manifest.tsdeclares the sandbox-provider driver metadatasrc/plugin.tsimplements the environment lifecycle hookspaperclipPlugin.manifestandpaperclipPlugin.workerpoint the host at the built plugin entrypoints indist/
Interrupted run-lease creation
The host commits a company-scoped creation operation before acquiring a new run
lease. The operation UUID becomes paperclip-create-<uuid> and the
paperclip-create-attempt label. The existing inspection hook can resolve that
name and validate company, environment and run labels without starting a sandbox.
Recovery persists the observed provider ID before the existing cleanup worker
deletes it. A missing creation name is an uncertain outcome, not proof of removal.
The host retains that operation for manual reconciliation and never repeats the
creation request blindly. Credential identity must still match admission.
This capability is declared as creationOperationVersion: 1 in manifest version
0.1.10. It covers normal run/ad-hoc lease acquisition and plugin task replacement.
Legacy provider environmentProbe and custom-image interactive setup have separate
lifecycles and are outside this lease-orphan fallback replacement. The normal Test
path which acquires a runtime lease is covered. Unit fixtures exercise lookup and
deletion; live provider qualification requires an explicit disposable test account.