mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Cloud agent runs use the Daytona sandbox provider plugin. One shared plugin worker process serves all Daytona calls on a server. > - The host closes a duplex channel through the worker. The host waits 10 seconds for the close acknowledgement. When the acknowledgement is late, the host retires the whole worker with SIGKILL. > - The worker close waits for the remote PTY kill before it releases the socket. The Daytona SDK sends that kill as a toolbox request with a 24-hour default timeout. A stalled sandbox can hold it open for longer than 10 seconds. > - Then the host kills the shared worker. Every other call in flight on that worker fails with `Worker process exited (code=null, signal=SIGKILL)`. This includes other runs' workspace preparation and restore. > - This pull request limits the wait for the remote kill to 5 seconds. Then the worker releases the socket and acknowledges the close. > - The benefit is that one slow sandbox no longer stops unrelated runs on the same server. ## Linked Issues or Issue Description **What happened?** On a hosted staging server, the host logged `duplex channel close not acknowledged; retiring worker` six times in about 90 minutes. Each time, the Daytona worker exited with SIGKILL. Each time, all in-flight calls on that worker failed. Runs in workspace preparation failed with `setup_failed` and `Worker process exited (code=null, signal=SIGKILL)`. Several runs failed together in each burst. **Expected behavior** A slow close of one duplex channel does not stop calls that belong to other channels or other runs. **Steps to reproduce** 1. Open a duplex channel on a Daytona lease. 2. Make the Daytona PTY kill request stall (the SDK default request timeout is 24 hours). 3. Close the channel from the host. The worker does not acknowledge within 10 seconds. 4. The host retires the worker with SIGKILL. All other pending calls fail. **Relevant logs or output** ``` info duplex channel close not acknowledged; retiring worker info retired worker process exited code=null signal=SIGKILL willRestart=true JsonRpcCallError: Worker process exited (code=null, signal=SIGKILL) ``` **Additional context** The login PTY close does not have this problem. It only does the local socket disconnect. A kill that rejects already gets a close acknowledgement, because the session close ignores kill errors. This change gives a slow kill the same result. No related public PR exists. I searched open and closed PRs for this failure. ## What Changed - `duplex-command-stream.ts`: `close()` waits for the remote `kill()` for at most `DUPLEX_CHANNEL_CLOSE_KILL_TIMEOUT_MS` (5 seconds). Then it always releases the socket with `disconnect()`. The kill continues in the background. A kill that answers in time keeps the old order: kill, then disconnect, then the acknowledgement. - `duplex-close-deadline.test.ts` (new): regression tests at the session close and at the worker `onDuplexChannelClose` boundary, and a check that the deadline stays well inside the host deadline. ## Verification - Red before the fix: with the old `duplex-command-stream.ts`, `acknowledges a bound close inside the host deadline when the remote kill never settles` fails (no acknowledgement after 9 seconds), and the session close test times out. - Green after the fix: `pnpm --dir packages/plugins/sandbox-providers/daytona exec vitest run --config vitest.config.ts src/duplex-close-deadline.test.ts src/duplex-command-stream.test.ts src/plugin.test.ts` passes 453 of 453. This includes the existing tests `joins pending teardown for concurrent closes while all channel authority stays revoked` and `does not turn a rejected retired close into an acknowledgement`. - The safety case `still waits for a kill that answers inside the deadline before it releases the socket` passes before and after the fix. - `tsc --noEmit` for the Daytona plugin passes. - In the full Daytona plugin suite, two GNU tar tests in `file-sync.test.ts` fail in my sandbox. They also fail on unchanged `master` in the same sandbox, so they are environment failures. CI covers them. - I did not run a live staging workflow. After the deploy, the staging logs must show no new `duplex channel close not acknowledged; retiring worker` lines from slow kills. ## Risks - Low risk. The change is in one Daytona plugin function. - After a kill timeout, the gateway process can stay alive in the sandbox until the background kill completes or the lease teardown stops the sandbox. The socket is closed, so no bytes can move on the channel. The route stays retired, so the host route cannot open again. This is the same state as when a kill rejects today. - The host contract does not change. A malformed or mismatched acknowledgement still retires the worker. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>