Files
PaperClipAI/packages/plugins
ef2a49b6df fix(daytona): bound the duplex channel close so a stalled kill cannot retire the worker (#21471)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Cloud agent runs use the Daytona sandbox provider plugin. One shared
plugin worker process serves all Daytona calls on a server.
> - The host closes a duplex channel through the worker. The host waits
10 seconds for the close acknowledgement. When the acknowledgement is
late, the host retires the whole worker with SIGKILL.
> - The worker close waits for the remote PTY kill before it releases
the socket. The Daytona SDK sends that kill as a toolbox request with a
24-hour default timeout. A stalled sandbox can hold it open for longer
than 10 seconds.
> - Then the host kills the shared worker. Every other call in flight on
that worker fails with `Worker process exited (code=null,
signal=SIGKILL)`. This includes other runs' workspace preparation and
restore.
> - This pull request limits the wait for the remote kill to 5 seconds.
Then the worker releases the socket and acknowledges the close.
> - The benefit is that one slow sandbox no longer stops unrelated runs
on the same server.

## Linked Issues or Issue Description

**What happened?**
On a hosted staging server, the host logged `duplex channel close not
acknowledged; retiring worker` six times in about 90 minutes. Each time,
the Daytona worker exited with SIGKILL. Each time, all in-flight calls
on that worker failed. Runs in workspace preparation failed with
`setup_failed` and `Worker process exited (code=null, signal=SIGKILL)`.
Several runs failed together in each burst.

**Expected behavior**
A slow close of one duplex channel does not stop calls that belong to
other channels or other runs.

**Steps to reproduce**
1. Open a duplex channel on a Daytona lease.
2. Make the Daytona PTY kill request stall (the SDK default request
timeout is 24 hours).
3. Close the channel from the host. The worker does not acknowledge
within 10 seconds.
4. The host retires the worker with SIGKILL. All other pending calls
fail.

**Relevant logs or output**
```
info duplex channel close not acknowledged; retiring worker
info retired worker process exited  code=null signal=SIGKILL willRestart=true
JsonRpcCallError: Worker process exited (code=null, signal=SIGKILL)
```

**Additional context**
The login PTY close does not have this problem. It only does the local
socket disconnect. A kill that rejects already gets a close
acknowledgement, because the session close ignores kill errors. This
change gives a slow kill the same result.

No related public PR exists. I searched open and closed PRs for this
failure.

## What Changed

- `duplex-command-stream.ts`: `close()` waits for the remote `kill()`
for at most `DUPLEX_CHANNEL_CLOSE_KILL_TIMEOUT_MS` (5 seconds). Then it
always releases the socket with `disconnect()`. The kill continues in
the background. A kill that answers in time keeps the old order: kill,
then disconnect, then the acknowledgement.
- `duplex-close-deadline.test.ts` (new): regression tests at the session
close and at the worker `onDuplexChannelClose` boundary, and a check
that the deadline stays well inside the host deadline.

## Verification

- Red before the fix: with the old `duplex-command-stream.ts`,
`acknowledges a bound close inside the host deadline when the remote
kill never settles` fails (no acknowledgement after 9 seconds), and the
session close test times out.
- Green after the fix: `pnpm --dir
packages/plugins/sandbox-providers/daytona exec vitest run --config
vitest.config.ts src/duplex-close-deadline.test.ts
src/duplex-command-stream.test.ts src/plugin.test.ts` passes 453 of 453.
This includes the existing tests `joins pending teardown for concurrent
closes while all channel authority stays revoked` and `does not turn a
rejected retired close into an acknowledgement`.
- The safety case `still waits for a kill that answers inside the
deadline before it releases the socket` passes before and after the fix.
- `tsc --noEmit` for the Daytona plugin passes.
- In the full Daytona plugin suite, two GNU tar tests in
`file-sync.test.ts` fail in my sandbox. They also fail on unchanged
`master` in the same sandbox, so they are environment failures. CI
covers them.
- I did not run a live staging workflow. After the deploy, the staging
logs must show no new `duplex channel close not acknowledged; retiring
worker` lines from slow kills.

## Risks

- Low risk. The change is in one Daytona plugin function.
- After a kill timeout, the gateway process can stay alive in the
sandbox until the background kill completes or the lease teardown stops
the sandbox. The socket is closed, so no bytes can move on the channel.
The route stays retired, so the host route cannot open again. This is
the same state as when a kill rejects today.
- The host contract does not change. A malformed or mismatched
acknowledgement still retires the worker.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
2026-11-13 12:17:05 -08:00
..