fix(daytona): retain duplex close ownership after lease cleanup (#19150)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote agents use provider plugins to manage sandbox leases and
duplex channels.
> - The host verifies the exact route and session when a bound channel
closes.
> - Daytona lease teardown removed that binding before a later host
close.
> - The incomplete reply made the host retire the shared worker. A later
command then failed during setup.
> - This pull request retains bounded close receipts and joins the
original cleanup.
> - Later work can use the same worker while invalid or unconfirmed
closes still fail closed.

## Linked Issues or Issue Description

**What happened?**

A normal lease teardown can close a duplex channel before the host
closes its bound route. Daytona then replies with only the route ID. The
host requires the original session ID too. It logs `duplex channel close
not acknowledged; retiring worker`, stops the worker, and rejects a
later `environmentExecute` call as stopped.

**Expected behavior**

A later close must identify the original route and session and join its
existing cleanup. It must not revive the channel or restart the worker.
Unknown ownership and an unconfirmed close must retain the existing
failure behavior.

**Steps to reproduce**

1. Acquire two leases through one Daytona provider worker.
2. Open a duplex channel on the first lease.
3. Release that lease, then close the bound channel through the host.
4. Execute a command on the second lease.

**Paperclip version or commit**

The real host/worker regression fails on
`69dea878c0b920b335594debb0286443ba9f6c35`. This branch includes master
`3a72efebe5b050e7594bb309e71a0d5f3072db00`.

**Deployment mode**

Managed Daytona sandbox execution in Cloud. The regression uses a local
real host and SDK worker. Only the external Daytona SDK and PTY are
simulated.

Related work: Refs #19004, Refs #10429, Refs #18065, Refs #19050. These
cover task Retry, managed worker startup, drain release, and stale
provider sessions.

## What Changed

- Retain at most 4096 retired route/session bindings with their original
close promises.
- Remove live channel authority before cleanup starts. Concurrent and
later closes join the same promise and return the stored binding.
- Reject reuse of a retained route ID. Keep unknown and evicted routes
unable to provide a session acknowledgement.
- Add a real child-process host/SDK-worker/Daytona regression with only
the external Daytona SDK and PTY simulated.
- Cover release, cancellation, stop, destroy, resume, a stalled close,
concurrent closes, failed cleanup, forged IDs and bounded eviction.
- Document close receipt scope and its separate provider-termination
checks.

## Verification

- RED before the repair: the actual host and SDK worker reproduced the
retirement log, SIGKILL, stopped state, and later `environmentExecute`
rejection. The direct provider cases also failed because the session ID
was missing.
- Combined-branch provider suite: 364 passed.
- Combined-branch host manager, duplex, managed readiness and composed
suites: 142 passed. The final test-only manifest-loading change passed
all 6 composed cases again.
- Full repository typecheck (`node scripts/typecheck-low-memory.mjs`),
focused server types and the standalone Daytona build passed. The
provider build uses pinned SDK `0.203.0` and TypeScript `5.9.3`.
- The composed cases check the same worker/PID, one later command,
preserved saved content, revoked channel writes/stops, and continued
rejection of commands on closed leases. A stalled close still retires
the worker and dispatches no later command.
- Reproduce with `pnpm --dir server exec vitest run --config
vitest.config.ts
src/__tests__/plugin-worker-manager-daytona-close.test.ts` from a
checkout with the SDK build dependencies prepared. Run the provider
suite with its own Vitest config.
- An initial local typecheck lacked the excluded providers' SDK links
after an install with scripts disabled. The standard root postinstall
linker restored those local links, and full typechecking then passed.
The fixture loads its actual built manifest. Failed setup output is
retained; no type configuration was relaxed.
- The first two composed attempts stopped during fixture setup. They are
not the behavioral RED above. Their failed outputs are retained.
- The first broad local typecheck stopped at an IPC permission error
during preflight. It passed after rerunning with allowed local IPC and
the standard SDK links.
- [Full repository
CI](https://github.com/paperclipai/paperclip/actions/runs/41945022881)
passed all 48 jobs on this head. Selection fell back to the complete
task plan, so the run was unfiltered. The complete check inventory has
54 successes and two expected Storybook skips; the separate Snyk status
passed. Fresh Greptile review on
`daebbeb5005b9d9bd59d32ea30094279048b9213` is [5/5 with no actionable
comments](https://github.com/paperclipai/paperclip/pull/19150#issuecomment-6646621914).
Independent production and fixture review found no actionable findings.
Hosted acceptance of this repair has not run.

## Risks

- A receipt removed at the 4096-entry bound cannot confirm an old bound
close. The existing host guard remains effective in that case.
- Close acknowledgements identify channels. They do not certify remote
process termination or clear unknown external work. Provider termination
receipts and existing recovery gates remain in force.
- Hosted evidence confirms the unacknowledged-close retirement class.
The historical log does not distinguish a missing session ID from a
timeout or another close error. The regression proves and repairs the
missing-binding path; it also preserves timeout retirement.
- The provider is outside the main pnpm workspace. A new Cloud image
must compile its changed source. The unchanged plugin version alone does
not prove activation.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, source editing and tool
execution. The exact model ID and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/__tests__/fixtures/daytona-worker-lifecycle-sdk.mjs

@@ -0,0 +1,77 @@
+import { appendFileSync, readFileSync } from "node:fs";
+
+const sandboxes = new Map();
+let nextId = 0;
+
+function record(event, fields = {}) {
+  appendFileSync(process.env.PAPERCLIP_DAYTONA_FIXTURE_JOURNAL, `${JSON.stringify({ event, ...fields })}\n`);
+}
+
+function sandbox(params) {
+  const id = `fixture-sandbox-${++nextId}`;
+  const files = new Map();
+  const resource = {
+    id,
+    name: params.name,
+    labels: params.labels,
+    state: "started",
+    target: "us",
+    autoDeleteInterval: -1,
+    async getWorkDir() { return "/home/daytona"; },
+    async getUserHomeDir() { return "/home/daytona"; },
+    async refreshData() {},
+    async setTtl() {},
+    async setAutoDeleteInterval(value) { resource.autoDeleteInterval = value; },
+    async start() { resource.state = "started"; record("start", { id }); },
+    async stop() { resource.state = "stopped"; record("stop", { id }); },
+    async delete() { sandboxes.delete(id); record("delete", { id }); },
+    fs: {
+      async createFolder() {},
+      async uploadFile(content, destination) { files.set(destination, content.toString("utf8")); },
+      async deleteFile() {},
+    },
+    process: {
+      async executeCommand(command) {
+        const afterRelease = command.includes("monitor-after-release");
+        record("execute", { id, afterRelease });
+        if (command.includes("fixture-provider-stop")) resource.state = "stopped";
+        const sentinel = command.match(/^cat '([^']+)'/);
+        const result = afterRelease ? readFileSync(process.env.PAPERCLIP_DAYTONA_FIXTURE_SAVED_FILE, "utf8")
+          : sentinel ? files.get(sentinel[1]) ?? "" : "bash";
+        return { exitCode: 0, result, artifacts: { stdout: result } };
+      },
+      async createPty() {
+        let finish;
+        const exited = new Promise(resolve => { finish = resolve; });
+        record("pty.open", { id });
+        return {
+          async waitForConnection() {},
+          async sendInput() { record("pty.write", { id }); },
+          wait() { return exited; },
+          async kill() { record("pty.kill", { id }); finish({ exitCode: 0 }); },
+          async disconnect() {
+            record("pty.disconnect", { id });
+            if (process.env.PAPERCLIP_DAYTONA_FIXTURE_HANG_CLOSE === "1" && id === "fixture-sandbox-1") {
+              await new Promise(() => {});
+            }
+          },
+        };
+      },
+    },
+  };
+  sandboxes.set(id, resource);
+  record("create", { id });
+  return resource;
+}
+
+export class Daytona {
+  async create(params) { return sandbox(params); }
+  async get(id) {
+    const result = sandboxes.get(id);
+    if (!result) throw new DaytonaNotFoundError("Fixture sandbox is absent");
+    return result;
+  }
+}
+
+export class DaytonaNotFoundError extends Error {}
+export class DaytonaTimeoutError extends Error {}

server/src/__tests__/plugin-worker-manager-daytona-close.test.ts

@@ -0,0 +1,97 @@
+import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { fileURLToPath, pathToFileURL } from "node:url";
+import { build } from "esbuild";
+import { expect, it } from "vitest";
+import { createPluginWorkerManager } from "../services/plugin-worker-manager.js";
+
+const repoRoot = fileURLToPath(new URL("../../..", import.meta.url));
+const providerRoot = path.join(repoRoot, "packages/plugins/sandbox-providers/daytona");
+const pluginId = "fixture-daytona-provider";
+
+it.each(["release", "cancel", "stop", "destroy", "resume", "stalled-close"] as const)("preserves worker ownership after %s of a bound duplex channel", async mode => {
+  const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-worker-close-"));
+  const journal = path.join(root, "provider.jsonl"), savedFile = path.join(root, "saved.txt");
+  const workerFile = path.join(root, "worker.mjs");
+  const manager = createPluginWorkerManager();
+  try {
+    await writeFile(savedFile, "saved work remains intact", { mode: 0o600 });
+    await build({
+      entryPoints: { worker: path.join(providerRoot, "src/worker.ts"), manifest: path.join(providerRoot, "src/manifest.ts") },
+      outdir: root,
+      outExtension: { ".js": ".mjs" },
+      platform: "node",
+      format: "esm",
+      target: "node24",
+      bundle: true,
+      logLevel: "silent",
+      nodePaths: [path.join(repoRoot, "server/node_modules")],
+      alias: {
+        "@daytonaio/sdk": path.join(repoRoot, "server/src/__tests__/fixtures/daytona-worker-lifecycle-sdk.mjs"),
+        "@paperclipai/plugin-sdk": path.join(repoRoot, "packages/plugins/sdk/dist/index.js"),
+      },
+      banner: { js: 'import { createRequire } from "node:module"; const require = createRequire(import.meta.url);' },
+    });
+    const { default: manifest } = await import(pathToFileURL(path.join(root, "manifest.mjs")).href);
+    const worker = await manager.startWorker(pluginId, {
+      entrypointPath: workerFile,
+      manifest,
+      config: {},
+      instanceInfo: { instanceId: "fixture-instance", hostVersion: "0.0.0" },
+      apiVersion: 1,
+      hostHandlers: {},
+      env: { PAPERCLIP_DAYTONA_FIXTURE_JOURNAL: journal, PAPERCLIP_DAYTONA_FIXTURE_SAVED_FILE: savedFile,
+        PAPERCLIP_DAYTONA_FIXTURE_HANG_CLOSE: mode === "stalled-close" ? "1" : "0" },
+      duplexChannelLimits: { closeTimeoutMs: 500 },
+    });
+    const pid = worker.diagnostics().pid;
+    const exited = new Promise<void>(resolve => worker.on("exit", () => resolve()));
+    const base = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment",
+      config: { apiKey: "fixture-not-a-credential", image: "fixture-image", timeoutMs: 2_000, livenessTimeoutMs: 250, reuseLease: true } };
+    const acquisition = { ...base, agentId: "fixture-agent", executionWorkspaceId: "fixture-workspace" };
+    const first = await manager.call(pluginId, "environmentAcquireLease", { ...acquisition, runId: "fixture-first-run" });
+    const next = await manager.call(pluginId, "environmentAcquireLease", { ...acquisition, runId: "fixture-next-run" });
+    const channel = await worker.openDuplexChannel({
+      driverKey: base.driverKey, companyId: base.companyId, environmentId: base.environmentId,
+      providerLeaseId: first.providerLeaseId!, command: ["fixture-bridge"],
+    });
+    const teardown = { ...base, providerLeaseId: first.providerLeaseId! };
+    if (mode === "resume") {
+      await manager.call(pluginId, "environmentExecute", { ...base, lease: first,
+        command: "fixture-provider-stop", bypassSession: true });
+      await expect(manager.call(pluginId, "environmentResumeLease", { ...teardown, leaseMetadata: first.metadata }))
+        .resolves.toMatchObject({ providerLeaseId: first.providerLeaseId, metadata: { resumedFromState: "stopped" } });
+    } else if (mode === "stop") await manager.call(pluginId, "environmentStopLease", teardown);
+    else if (mode === "destroy") await manager.call(pluginId, "environmentDestroyLease", teardown);
+    else await manager.call(pluginId, "environmentReleaseLease", { ...teardown, cancelActiveWork: mode === "cancel" });
+    await channel.close();
+    if (mode === "stalled-close") await exited;
+    else await Promise.race([exited, new Promise<void>(resolve => setTimeout(resolve, 100))]);
+    const execution = manager.call(pluginId, "environmentExecute", {
+      ...base, lease: next, command: "printf", args: ["monitor-after-release"], bypassSession: true,
+    });
+    if (mode === "stalled-close") await expect(execution).rejects.toThrow("is stopped");
+    else await expect(execution).resolves.toMatchObject({ exitCode: 0, stdout: "saved work remains intact" });
+    expect(worker.status).toBe(mode === "stalled-close" ? "stopped" : "running");
+    expect(manager.getWorker(pluginId)).toBe(worker);
+    expect(worker.diagnostics().pid).toBe(mode === "stalled-close" ? null : pid);
+    channel.write(new TextEncoder().encode("closed channel must remain closed"));
+    channel.kill();
+    if (mode !== "resume") await expect(manager.call(pluginId, "environmentExecute", {
+      ...base, lease: first, command: "printf", args: ["closed-lease-after-release"], bypassSession: true,
+    })).rejects.toThrow();
+    const events = (await readFile(journal, "utf8")).trim().split("\n").map(line => JSON.parse(line));
+    expect(events.filter(event => event.event === "create")).toHaveLength(2);
+    expect(events.filter(event => event.event === "pty.open")).toHaveLength(1);
+    expect(events.filter(event => event.event === "pty.write")).toHaveLength(1);
+    expect(events.filter(event => event.event === "pty.kill")).toHaveLength(1);
+    expect(events.filter(event => event.event === "pty.disconnect")).toHaveLength(1);
+    expect(events.filter(event => event.event === "execute" && event.afterRelease))
+      .toEqual(mode === "stalled-close" ? [] : [{ event: "execute", id: next.providerLeaseId, afterRelease: true }]);
+    expect(await readFile(savedFile, "utf8")).toBe("saved work remains intact");
+  } finally {
+    await manager.stopAll();
+    await rm(root, { recursive: true, force: true });
+  }
+});