From ef2a49b6df554a80701d1daf788b2002a512c61c Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 13 Nov 2026 12:17:05 -0800 Subject: [PATCH] fix(daytona): bound the duplex channel close so a stalled kill cannot retire the worker (#21471) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Cloud agent runs use the Daytona sandbox provider plugin. One shared plugin worker process serves all Daytona calls on a server. > - The host closes a duplex channel through the worker. The host waits 10 seconds for the close acknowledgement. When the acknowledgement is late, the host retires the whole worker with SIGKILL. > - The worker close waits for the remote PTY kill before it releases the socket. The Daytona SDK sends that kill as a toolbox request with a 24-hour default timeout. A stalled sandbox can hold it open for longer than 10 seconds. > - Then the host kills the shared worker. Every other call in flight on that worker fails with `Worker process exited (code=null, signal=SIGKILL)`. This includes other runs' workspace preparation and restore. > - This pull request limits the wait for the remote kill to 5 seconds. Then the worker releases the socket and acknowledges the close. > - The benefit is that one slow sandbox no longer stops unrelated runs on the same server. ## Linked Issues or Issue Description **What happened?** On a hosted staging server, the host logged `duplex channel close not acknowledged; retiring worker` six times in about 90 minutes. Each time, the Daytona worker exited with SIGKILL. Each time, all in-flight calls on that worker failed. Runs in workspace preparation failed with `setup_failed` and `Worker process exited (code=null, signal=SIGKILL)`. Several runs failed together in each burst. **Expected behavior** A slow close of one duplex channel does not stop calls that belong to other channels or other runs. **Steps to reproduce** 1. Open a duplex channel on a Daytona lease. 2. Make the Daytona PTY kill request stall (the SDK default request timeout is 24 hours). 3. Close the channel from the host. The worker does not acknowledge within 10 seconds. 4. The host retires the worker with SIGKILL. All other pending calls fail. **Relevant logs or output** ``` info duplex channel close not acknowledged; retiring worker info retired worker process exited code=null signal=SIGKILL willRestart=true JsonRpcCallError: Worker process exited (code=null, signal=SIGKILL) ``` **Additional context** The login PTY close does not have this problem. It only does the local socket disconnect. A kill that rejects already gets a close acknowledgement, because the session close ignores kill errors. This change gives a slow kill the same result. No related public PR exists. I searched open and closed PRs for this failure. ## What Changed - `duplex-command-stream.ts`: `close()` waits for the remote `kill()` for at most `DUPLEX_CHANNEL_CLOSE_KILL_TIMEOUT_MS` (5 seconds). Then it always releases the socket with `disconnect()`. The kill continues in the background. A kill that answers in time keeps the old order: kill, then disconnect, then the acknowledgement. - `duplex-close-deadline.test.ts` (new): regression tests at the session close and at the worker `onDuplexChannelClose` boundary, and a check that the deadline stays well inside the host deadline. ## Verification - Red before the fix: with the old `duplex-command-stream.ts`, `acknowledges a bound close inside the host deadline when the remote kill never settles` fails (no acknowledgement after 9 seconds), and the session close test times out. - Green after the fix: `pnpm --dir packages/plugins/sandbox-providers/daytona exec vitest run --config vitest.config.ts src/duplex-close-deadline.test.ts src/duplex-command-stream.test.ts src/plugin.test.ts` passes 453 of 453. This includes the existing tests `joins pending teardown for concurrent closes while all channel authority stays revoked` and `does not turn a rejected retired close into an acknowledgement`. - The safety case `still waits for a kill that answers inside the deadline before it releases the socket` passes before and after the fix. - `tsc --noEmit` for the Daytona plugin passes. - In the full Daytona plugin suite, two GNU tar tests in `file-sync.test.ts` fail in my sandbox. They also fail on unchanged `master` in the same sandbox, so they are environment failures. CI covers them. - I did not run a live staging workflow. After the deploy, the staging logs must show no new `duplex channel close not acknowledged; retiring worker` lines from slow kills. ## Risks - Low risk. The change is in one Daytona plugin function. - After a kill timeout, the gateway process can stay alive in the sandbox until the background kill completes or the lease teardown stops the sandbox. The socket is closed, so no bytes can move on the channel. The route stays retired, so the host route cannot open again. This is the same state as when a kill rejects today. - The host contract does not change. A malformed or mismatched acknowledgement still retires the worker. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> --- .../daytona/src/duplex-close-deadline.test.ts | 115 ++++++++++++++++++ .../daytona/src/duplex-command-stream.ts | 16 ++- 2 files changed, 130 insertions(+), 1 deletion(-) create mode 100644 packages/plugins/sandbox-providers/daytona/src/duplex-close-deadline.test.ts diff --git a/packages/plugins/sandbox-providers/daytona/src/duplex-close-deadline.test.ts b/packages/plugins/sandbox-providers/daytona/src/duplex-close-deadline.test.ts new file mode 100644 index 00000000000..ae111b048bc --- /dev/null +++ b/packages/plugins/sandbox-providers/daytona/src/duplex-close-deadline.test.ts @@ -0,0 +1,115 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const createSandbox = vi.hoisted(() => vi.fn()); +vi.mock("@daytonaio/sdk", () => ({ + Daytona: class { create = createSandbox; get = vi.fn(); }, + DaytonaNotFoundError: class extends Error {}, + DaytonaTimeoutError: class extends Error {}, +})); + +import { DUPLEX_CHANNEL_CLOSE_KILL_TIMEOUT_MS, openDaytonaDuplexChannelSession } from "./duplex-command-stream.js"; +import plugin, { __resetDaytonaSandboxHandleCacheForTest } from "./plugin.js"; + +const HOST_DUPLEX_CLOSE_TIMEOUT_MS = 10_000; + +function fakePty() { + return { + waitForConnection: vi.fn(async () => {}), + sendInput: vi.fn(async () => {}), + wait: vi.fn(() => new Promise<{ exitCode: number }>(() => {})), + kill: vi.fn(async () => {}), + disconnect: vi.fn(async () => {}), + }; +} + +async function fixture() { + const pty = fakePty(); + const sandbox = { + id: "fixture-sandbox", name: "paperclip-sandbox", state: "started", recoverable: false, target: "us", + errorReason: null, autoDeleteInterval: -1, autoStopInterval: 15, + setAutostopInterval: vi.fn(async () => {}), setAutoDeleteInterval: vi.fn(async () => {}), + setTtl: vi.fn(async () => {}), refreshData: vi.fn(async () => {}), + getWorkDir: vi.fn(async () => "/home/daytona"), getUserHomeDir: vi.fn(async () => "/home/daytona"), + start: vi.fn(async () => {}), stop: vi.fn(async () => {}), delete: vi.fn(async () => {}), + fs: { createFolder: vi.fn(async () => {}), uploadFile: vi.fn(async () => {}), deleteFile: vi.fn(async () => {}) }, + process: { + executeCommand: vi.fn(async () => ({ exitCode: 0, result: "bash", artifacts: { stdout: "bash" } })), + createPty: vi.fn(async () => pty), + }, + }; + createSandbox.mockResolvedValue(sandbox); + await plugin.definition.onEnvironmentAcquireLease!({ + driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment", runId: "fixture-run", + config: { apiKey: "fixture-not-a-key", image: "fixture-image", timeoutMs: 3_000, livenessTimeoutMs: 1_000, reuseLease: true }, + }); + const opened = await plugin.definition.onDuplexChannelOpen!({ + driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment", + providerLeaseId: sandbox.id, command: ["fixture-bridge"], hostRouteId: "fixture-route", + }); + return { pty, opened }; +} + +describe("Daytona duplex channel close deadline", () => { + beforeEach(() => { __resetDaytonaSandboxHandleCacheForTest(); createSandbox.mockReset(); }); + afterEach(() => { vi.useRealTimers(); __resetDaytonaSandboxHandleCacheForTest(); }); + + it("keeps the close kill deadline well inside the host close deadline", () => { + expect(DUPLEX_CHANNEL_CLOSE_KILL_TIMEOUT_MS).toBeLessThanOrEqual(HOST_DUPLEX_CLOSE_TIMEOUT_MS / 2); + }); + + it("releases the session socket at the kill deadline when the remote kill never settles", async () => { + const pty = fakePty(); + const session = await openDaytonaDuplexChannelSession({ createPty: async () => pty }, ["fixture-bridge"]); + pty.kill.mockImplementation(() => new Promise(() => {})); + vi.useFakeTimers(); + + let closed = false; + const closing = session.close().then(() => { closed = true; }); + await vi.advanceTimersByTimeAsync(DUPLEX_CHANNEL_CLOSE_KILL_TIMEOUT_MS - 1); + expect(closed).toBe(false); + expect(pty.disconnect).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(1); + await closing; + expect(pty.disconnect).toHaveBeenCalledTimes(1); + }); + + it("acknowledges a bound close inside the host deadline when the remote kill never settles", async () => { + const { pty, opened } = await fixture(); + pty.kill.mockImplementation(() => new Promise(() => {})); + vi.useFakeTimers(); + + let ack: unknown; + const closing = plugin.definition.onDuplexChannelClose!({ hostRouteId: opened.hostRouteId }) + .then((value) => { ack = value; }); + await vi.advanceTimersByTimeAsync(HOST_DUPLEX_CLOSE_TIMEOUT_MS - 1_000); + expect(ack).toEqual(opened); + await closing; + expect(pty.kill).toHaveBeenCalledTimes(1); + expect(pty.disconnect).toHaveBeenCalledTimes(1); + + await expect(plugin.definition.onDuplexChannelClose!({ hostRouteId: opened.hostRouteId })).resolves.toEqual(opened); + await expect(plugin.definition.onDuplexChannelOpen!({ + driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment", + providerLeaseId: "fixture-sandbox", command: ["fixture-bridge"], hostRouteId: opened.hostRouteId, + })).rejects.toThrow("already been used"); + }); + + it("still waits for a kill that answers inside the deadline before it releases the socket", async () => { + const { pty, opened } = await fixture(); + let finishKill!: () => void; + pty.kill.mockImplementation(() => new Promise((resolve) => { finishKill = resolve; })); + vi.useFakeTimers(); + + let ack: unknown; + const closing = plugin.definition.onDuplexChannelClose!({ hostRouteId: opened.hostRouteId }) + .then((value) => { ack = value; }); + await vi.advanceTimersByTimeAsync(1_000); + expect(ack).toBeUndefined(); + expect(pty.disconnect).not.toHaveBeenCalled(); + + finishKill(); + await closing; + expect(ack).toEqual(opened); + expect(pty.disconnect).toHaveBeenCalledTimes(1); + }); +}); diff --git a/packages/plugins/sandbox-providers/daytona/src/duplex-command-stream.ts b/packages/plugins/sandbox-providers/daytona/src/duplex-command-stream.ts index e96d5a9ef58..1aa84b917a6 100644 --- a/packages/plugins/sandbox-providers/daytona/src/duplex-command-stream.ts +++ b/packages/plugins/sandbox-providers/daytona/src/duplex-command-stream.ts @@ -110,6 +110,20 @@ export interface DaytonaDuplexChannelOptions { onWriteError?: (reason: DaytonaDuplexWriteErrorReason) => void; } +export const DUPLEX_CHANNEL_CLOSE_KILL_TIMEOUT_MS = 5_000; + +async function settleWithin(action: Promise, timeoutMs: number): Promise { + let timer: ReturnType | undefined; + try { + await Promise.race([ + action.catch(() => undefined), + new Promise((resolve) => { timer = setTimeout(resolve, timeoutMs); }), + ]); + } finally { + clearTimeout(timer); + } +} + // The terminal size for the duplex channel PTY. The channel carries bytes, not a // visible UI, so a fixed standard size is enough. A fixed size keeps the launch // deterministic. @@ -279,7 +293,7 @@ export async function openDaytonaDuplexChannelSession( // Stop the child and release the socket. `kill` ends the gateway and `wait` // resolves with the exit; `disconnect` releases the PTY socket. Both are // safe to call more than one time, so close stays idempotent. - await handle.kill().catch(() => undefined); + await settleWithin(handle.kill(), DUPLEX_CHANNEL_CLOSE_KILL_TIMEOUT_MS); await handle.disconnect().catch(() => undefined); }, };