fix(daytona): retain duplex close ownership after lease cleanup (#19150)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote agents use provider plugins to manage sandbox leases and
duplex channels.
> - The host verifies the exact route and session when a bound channel
closes.
> - Daytona lease teardown removed that binding before a later host
close.
> - The incomplete reply made the host retire the shared worker. A later
command then failed during setup.
> - This pull request retains bounded close receipts and joins the
original cleanup.
> - Later work can use the same worker while invalid or unconfirmed
closes still fail closed.

## Linked Issues or Issue Description

**What happened?**

A normal lease teardown can close a duplex channel before the host
closes its bound route. Daytona then replies with only the route ID. The
host requires the original session ID too. It logs `duplex channel close
not acknowledged; retiring worker`, stops the worker, and rejects a
later `environmentExecute` call as stopped.

**Expected behavior**

A later close must identify the original route and session and join its
existing cleanup. It must not revive the channel or restart the worker.
Unknown ownership and an unconfirmed close must retain the existing
failure behavior.

**Steps to reproduce**

1. Acquire two leases through one Daytona provider worker.
2. Open a duplex channel on the first lease.
3. Release that lease, then close the bound channel through the host.
4. Execute a command on the second lease.

**Paperclip version or commit**

The real host/worker regression fails on
`69dea878c0b920b335594debb0286443ba9f6c35`. This branch includes master
`3a72efebe5b050e7594bb309e71a0d5f3072db00`.

**Deployment mode**

Managed Daytona sandbox execution in Cloud. The regression uses a local
real host and SDK worker. Only the external Daytona SDK and PTY are
simulated.

Related work: Refs #19004, Refs #10429, Refs #18065, Refs #19050. These
cover task Retry, managed worker startup, drain release, and stale
provider sessions.

## What Changed

- Retain at most 4096 retired route/session bindings with their original
close promises.
- Remove live channel authority before cleanup starts. Concurrent and
later closes join the same promise and return the stored binding.
- Reject reuse of a retained route ID. Keep unknown and evicted routes
unable to provide a session acknowledgement.
- Add a real child-process host/SDK-worker/Daytona regression with only
the external Daytona SDK and PTY simulated.
- Cover release, cancellation, stop, destroy, resume, a stalled close,
concurrent closes, failed cleanup, forged IDs and bounded eviction.
- Document close receipt scope and its separate provider-termination
checks.

## Verification

- RED before the repair: the actual host and SDK worker reproduced the
retirement log, SIGKILL, stopped state, and later `environmentExecute`
rejection. The direct provider cases also failed because the session ID
was missing.
- Combined-branch provider suite: 364 passed.
- Combined-branch host manager, duplex, managed readiness and composed
suites: 142 passed. The final test-only manifest-loading change passed
all 6 composed cases again.
- Full repository typecheck (`node scripts/typecheck-low-memory.mjs`),
focused server types and the standalone Daytona build passed. The
provider build uses pinned SDK `0.203.0` and TypeScript `5.9.3`.
- The composed cases check the same worker/PID, one later command,
preserved saved content, revoked channel writes/stops, and continued
rejection of commands on closed leases. A stalled close still retires
the worker and dispatches no later command.
- Reproduce with `pnpm --dir server exec vitest run --config
vitest.config.ts
src/__tests__/plugin-worker-manager-daytona-close.test.ts` from a
checkout with the SDK build dependencies prepared. Run the provider
suite with its own Vitest config.
- An initial local typecheck lacked the excluded providers' SDK links
after an install with scripts disabled. The standard root postinstall
linker restored those local links, and full typechecking then passed.
The fixture loads its actual built manifest. Failed setup output is
retained; no type configuration was relaxed.
- The first two composed attempts stopped during fixture setup. They are
not the behavioral RED above. Their failed outputs are retained.
- The first broad local typecheck stopped at an IPC permission error
during preflight. It passed after rerunning with allowed local IPC and
the standard SDK links.
- [Full repository
CI](https://github.com/paperclipai/paperclip/actions/runs/41945022881)
passed all 48 jobs on this head. Selection fell back to the complete
task plan, so the run was unfiltered. The complete check inventory has
54 successes and two expected Storybook skips; the separate Snyk status
passed. Fresh Greptile review on
`daebbeb5005b9d9bd59d32ea30094279048b9213` is [5/5 with no actionable
comments](https://github.com/paperclipai/paperclip/pull/19150#issuecomment-6646621914).
Independent production and fixture review found no actionable findings.
Hosted acceptance of this repair has not run.

## Risks

- A receipt removed at the 4096-entry bound cannot confirm an old bound
close. The existing host guard remains effective in that case.
- Close acknowledgements identify channels. They do not certify remote
process termination or clear unknown external work. Provider termination
receipts and existing recovery gates remain in force.
- Hosted evidence confirms the unacknowledged-close retirement class.
The historical log does not distinguish a missing session ID from a
timeout or another close error. The regression proves and repairs the
missing-binding path; it also preserves timeout retirement.
- The provider is outside the main pnpm workspace. A new Cloud image
must compile its changed source. The unchanged plugin version alone does
not prove activation.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, source editing and tool
execution. The exact model ID and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-04 19:15:50 -06:00
1 parent 11f2a5293d
commit 9e5039f6c8
5 files changed
+301 -4

No files matched your search

+12
View File
@@ -3038,6 +3038,18 @@ Older packs without the declaration retain content verification until the image
is updated. Selected ACPX agent profile checks remain in force.
Daytona detects gzip/zstd headers and skips redundant transport compression.
### Daytona duplex close receipts
Lease teardown removes duplex routes from command admission before it closes
their transports. The provider retains up to 4096 route/session bindings and
their close promises, so a later host close joins the same cleanup and returns
the original binding. These records grant no write, stop, or open authority.
Unknown or evicted routes cannot supply a session acknowledgement. A pending
close still waits; an unconfirmed close still retires the worker through the
existing host guard. The acknowledgement identifies the channel. Provider stop
receipts and unresolved external-work checks still determine process termination.
No command is replayed or worker restarted by this mechanism.
### Remote GitHub launcher staging
Managed GitHub launchers stage their nine text files in one bounded control
@@ -40,6 +40,7 @@ import type { PluginContext } from "@paperclipai/plugin-sdk";
import { environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk";
import manifest from "./manifest.js";
import { parseTarVerboseListingLine, splitLinkEntryOnce } from "./file-sync.js";
import * as duplexCommands from "./duplex-command-stream.js";
describe("Daytona read-only resource inspection", () => {
const params = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment",
@@ -729,6 +730,8 @@ describe("Daytona sandbox provider plugin", () => {
});
expect(killed).toBeGreaterThanOrEqual(1);
expect(disconnected).toBe(1);
await expect(plugin.definition.onDuplexChannelClose!({ hostRouteId: "route-1" }))
.resolves.toEqual({ hostRouteId: "route-1", workerSessionId });
// The channel entry is gone, so a later write is a no-op and reaches no
// process.
@@ -746,6 +749,94 @@ describe("Daytona sandbox provider plugin", () => {
}
});
describe("retired duplex ownership", () => {
async function fixture() {
const params = { driverKey: "daytona", companyId: "company-1", environmentId: "env-1",
config: { apiKey: "fixture-key", image: "fixture-image", timeoutMs: 3_000, livenessTimeoutMs: 1_000, reuseLease: true } };
const pty = { waitForConnection: vi.fn(async () => {}), sendInput: vi.fn(async () => {}),
wait: vi.fn(() => new Promise<{ exitCode: number }>(() => {})),
kill: vi.fn(async () => {}), disconnect: vi.fn(async () => {}) };
const sandbox = createMockSandbox();
const createPty = vi.fn(async () => pty);
Object.assign(sandbox.process, { createPty });
mockCreate.mockResolvedValue(sandbox);
await plugin.definition.onEnvironmentAcquireLease!({ ...params, runId: "run-1" });
const opening = { driverKey: params.driverKey, companyId: params.companyId, environmentId: params.environmentId,
providerLeaseId: sandbox.id, command: ["fixture-bridge"] };
return { params: { ...params, providerLeaseId: sandbox.id }, opening, pty, createPty };
}
it("joins pending teardown for concurrent closes while all channel authority stays revoked", async () => {
const f = await fixture();
const opened = await plugin.definition.onDuplexChannelOpen!({ ...f.opening, hostRouteId: "pending-route" });
let finishKill!: () => void, enteredKill!: () => void;
const entered = new Promise<void>(resolve => { enteredKill = resolve; });
const killing = new Promise<void>(resolve => { finishKill = resolve; });
f.pty.kill.mockImplementation(async () => { enteredKill(); await killing; });
const requests: Promise<unknown>[] = [];
try {
requests.push(plugin.definition.onEnvironmentReleaseLease!(f.params));
await entered;
let replies = 0;
for (let index = 0; index < 2; index++) requests.push(plugin.definition.onDuplexChannelClose!({ hostRouteId: opened.hostRouteId })
.then(value => { replies++; return value; }));
await new Promise<void>(resolve => setImmediate(resolve));
expect(replies).toBe(0);
expect(f.pty.kill).toHaveBeenCalledTimes(1);
expect(f.pty.disconnect).not.toHaveBeenCalled();
await plugin.definition.onDuplexChannelWrite!({ ...opened, data: Buffer.from("closed").toString("base64") });
await plugin.definition.onDuplexChannelStop!(opened);
expect(f.pty.sendInput).toHaveBeenCalledTimes(1);
expect(f.pty.kill).toHaveBeenCalledTimes(1);
await expect(plugin.definition.onDuplexChannelOpen!({ ...f.opening, hostRouteId: opened.hostRouteId }))
.rejects.toThrow("already been used");
expect(f.createPty).toHaveBeenCalledTimes(1);
finishKill();
const result = await Promise.all(requests);
expect(result.slice(1)).toEqual([opened, opened]);
await expect(plugin.definition.onDuplexChannelClose!({ hostRouteId: opened.hostRouteId })).resolves.toEqual(opened);
expect(f.pty.kill).toHaveBeenCalledTimes(1);
expect(f.pty.disconnect).toHaveBeenCalledTimes(1);
await expect(plugin.definition.onDuplexChannelClose!({ hostRouteId: "unknown-route", workerSessionId: opened.workerSessionId } as never))
.resolves.toEqual({ hostRouteId: "unknown-route" });
} finally { finishKill(); await Promise.allSettled(requests); }
});
it("does not turn a rejected retired close into an acknowledgement", async () => {
const f = await fixture();
const close = vi.fn().mockRejectedValue(new Error("fixture closure remains unknown"));
const restore = vi.spyOn(duplexCommands, "openDaytonaDuplexChannelSession").mockResolvedValueOnce({
onData() {}, write: vi.fn(), kill: vi.fn(), wait: () => new Promise(() => {}), close,
});
try {
const opened = await plugin.definition.onDuplexChannelOpen!({ ...f.opening, hostRouteId: "failed-route" });
await plugin.definition.onEnvironmentReleaseLease!(f.params);
await expect(plugin.definition.onDuplexChannelClose!({ hostRouteId: opened.hostRouteId }))
.rejects.toThrow("fixture closure remains unknown");
expect(close).toHaveBeenCalledTimes(1);
await expect(plugin.definition.onDuplexChannelOpen!({ ...f.opening, hostRouteId: opened.hostRouteId }))
.rejects.toThrow("already been used");
} finally { restore.mockRestore(); }
});
it("bounds retained acknowledgements and never echoes a supplied session id for evicted routes", async () => {
const f = await fixture();
const first = await plugin.definition.onDuplexChannelOpen!({ ...f.opening, hostRouteId: "oldest-route" });
await plugin.definition.onDuplexChannelClose!({ hostRouteId: first.hostRouteId });
let latest = first;
for (let index = 0; index < 4_096; index++) {
latest = await plugin.definition.onDuplexChannelOpen!({ ...f.opening, hostRouteId: `bounded-route-${index}` });
await plugin.definition.onDuplexChannelClose!({ hostRouteId: latest.hostRouteId });
}
await expect(plugin.definition.onDuplexChannelClose!({ hostRouteId: first.hostRouteId, workerSessionId: first.workerSessionId } as never))
.resolves.toEqual({ hostRouteId: first.hostRouteId });
await expect(plugin.definition.onDuplexChannelClose!({ hostRouteId: latest.hostRouteId, workerSessionId: "forged-session" } as never))
.resolves.toEqual(latest);
expect(f.pty.kill).toHaveBeenCalledTimes(4_097);
expect(f.pty.disconnect).toHaveBeenCalledTimes(4_097);
});
});
it("normalizes config and validates the API key fallback", async () => {
process.env.DAYTONA_API_KEY = "host-key";
@@ -1728,6 +1728,7 @@ export function __resetDaytonaSandboxHandleCacheForTest(): void {
sandboxHandleWritableDirs.reset();
sandboxHandleSessionStore.reset();
runnerIngressGenerationStore.reset();
retiredDaytonaDuplexChannelByRoute.clear();
}
/**
@@ -2489,12 +2490,24 @@ interface DaytonaDuplexChannelEntry {
}
const daytonaDuplexChannelByRoute = new Map<string, DaytonaDuplexChannelEntry>();
const daytonaDuplexChannelBySession = new Map<string, DaytonaDuplexChannelEntry>();
const retiredDaytonaDuplexChannelByRoute = new Map<string, { workerSessionId: string; closed: Promise<void> }>();
const MAX_RETIRED_DAYTONA_DUPLEX_CHANNELS = 4096;
function forgetDaytonaDuplexChannel(entry: DaytonaDuplexChannelEntry): void {
daytonaDuplexChannelByRoute.delete(entry.hostRouteId);
daytonaDuplexChannelBySession.delete(entry.workerSessionId);
}
function closeDaytonaDuplexChannel(entry: DaytonaDuplexChannelEntry): Promise<void> {
forgetDaytonaDuplexChannel(entry);
const closed = Promise.resolve().then(() => entry.session.close());
if (retiredDaytonaDuplexChannelByRoute.size >= MAX_RETIRED_DAYTONA_DUPLEX_CHANNELS) {
retiredDaytonaDuplexChannelByRoute.delete(retiredDaytonaDuplexChannelByRoute.keys().next().value!);
}
retiredDaytonaDuplexChannelByRoute.set(entry.hostRouteId, { workerSessionId: entry.workerSessionId, closed });
return closed;
}
// Close every open duplex channel that belongs to one provider lease and drop its
// entry. The lease teardown hooks (release, destroy, resume) call this, so a
// channel never outlives the sandbox that carries it. The close kills the child
@@ -2504,8 +2517,7 @@ async function closeDaytonaDuplexChannelsForLease(providerLeaseId: string): Prom
const matches = [...daytonaDuplexChannelByRoute.values()].filter(
(entry) => entry.providerLeaseId === providerLeaseId,
);
for (const entry of matches) forgetDaytonaDuplexChannel(entry);
await Promise.allSettled(matches.map(entry => entry.session.close()));
await Promise.allSettled(matches.map(closeDaytonaDuplexChannel));
}
async function stopLease(params: PluginEnvironmentReleaseLeaseParams): Promise<PluginEnvironmentTerminationReceipt> {
@@ -3716,6 +3728,9 @@ const plugin = definePlugin(trackDaytonaHooks({
// `ctx.duplexChannel`, bound to the returned worker session id. Fail closed when
// no cached sandbox matches the lease.
async onDuplexChannelOpen(params) {
if (daytonaDuplexChannelByRoute.has(params.hostRouteId) || retiredDaytonaDuplexChannelByRoute.has(params.hostRouteId)) {
throw new Error("Daytona duplex channel: host route has already been used.");
}
const sandbox = await sandboxHandleCache.findByProviderLeaseId(params.providerLeaseId);
if (!sandbox) {
throw new Error(
@@ -3794,10 +3809,14 @@ const plugin = definePlugin(trackDaytonaHooks({
const entry = daytonaDuplexChannelByRoute.get(params.hostRouteId);
if (entry) {
const boundWorkerSessionId = entry.workerSessionId;
forgetDaytonaDuplexChannel(entry);
await entry.session.close().catch(() => undefined);
await closeDaytonaDuplexChannel(entry).catch(() => undefined);
return { hostRouteId: params.hostRouteId, workerSessionId: boundWorkerSessionId };
}
const retired = retiredDaytonaDuplexChannelByRoute.get(params.hostRouteId);
if (retired) {
await retired.closed;
return { hostRouteId: params.hostRouteId, workerSessionId: retired.workerSessionId };
}
return { hostRouteId: params.hostRouteId };
},
@@ -3814,6 +3833,7 @@ const plugin = definePlugin(trackDaytonaHooks({
const openChannels = [...daytonaDuplexChannelByRoute.values()];
daytonaDuplexChannelByRoute.clear();
daytonaDuplexChannelBySession.clear();
retiredDaytonaDuplexChannelByRoute.clear();
for (const entry of openChannels) {
await entry.session.close().catch(() => undefined);
}
@@ -0,0 +1,77 @@
import { appendFileSync, readFileSync } from "node:fs";
const sandboxes = new Map();
let nextId = 0;
function record(event, fields = {}) {
appendFileSync(process.env.PAPERCLIP_DAYTONA_FIXTURE_JOURNAL, `${JSON.stringify({ event, ...fields })}\n`);
}
function sandbox(params) {
const id = `fixture-sandbox-${++nextId}`;
const files = new Map();
const resource = {
id,
name: params.name,
labels: params.labels,
state: "started",
target: "us",
autoDeleteInterval: -1,
async getWorkDir() { return "/home/daytona"; },
async getUserHomeDir() { return "/home/daytona"; },
async refreshData() {},
async setTtl() {},
async setAutoDeleteInterval(value) { resource.autoDeleteInterval = value; },
async start() { resource.state = "started"; record("start", { id }); },
async stop() { resource.state = "stopped"; record("stop", { id }); },
async delete() { sandboxes.delete(id); record("delete", { id }); },
fs: {
async createFolder() {},
async uploadFile(content, destination) { files.set(destination, content.toString("utf8")); },
async deleteFile() {},
},
process: {
async executeCommand(command) {
const afterRelease = command.includes("monitor-after-release");
record("execute", { id, afterRelease });
if (command.includes("fixture-provider-stop")) resource.state = "stopped";
const sentinel = command.match(/^cat '([^']+)'/);
const result = afterRelease ? readFileSync(process.env.PAPERCLIP_DAYTONA_FIXTURE_SAVED_FILE, "utf8")
: sentinel ? files.get(sentinel[1]) ?? "" : "bash";
return { exitCode: 0, result, artifacts: { stdout: result } };
},
async createPty() {
let finish;
const exited = new Promise(resolve => { finish = resolve; });
record("pty.open", { id });
return {
async waitForConnection() {},
async sendInput() { record("pty.write", { id }); },
wait() { return exited; },
async kill() { record("pty.kill", { id }); finish({ exitCode: 0 }); },
async disconnect() {
record("pty.disconnect", { id });
if (process.env.PAPERCLIP_DAYTONA_FIXTURE_HANG_CLOSE === "1" && id === "fixture-sandbox-1") {
await new Promise(() => {});
}
},
};
},
},
};
sandboxes.set(id, resource);
record("create", { id });
return resource;
}
export class Daytona {
async create(params) { return sandbox(params); }
async get(id) {
const result = sandboxes.get(id);
if (!result) throw new DaytonaNotFoundError("Fixture sandbox is absent");
return result;
}
}
export class DaytonaNotFoundError extends Error {}
export class DaytonaTimeoutError extends Error {}
@@ -0,0 +1,97 @@
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import { build } from "esbuild";
import { expect, it } from "vitest";
import { createPluginWorkerManager } from "../services/plugin-worker-manager.js";
const repoRoot = fileURLToPath(new URL("../../..", import.meta.url));
const providerRoot = path.join(repoRoot, "packages/plugins/sandbox-providers/daytona");
const pluginId = "fixture-daytona-provider";
it.each(["release", "cancel", "stop", "destroy", "resume", "stalled-close"] as const)("preserves worker ownership after %s of a bound duplex channel", async mode => {
const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-worker-close-"));
const journal = path.join(root, "provider.jsonl"), savedFile = path.join(root, "saved.txt");
const workerFile = path.join(root, "worker.mjs");
const manager = createPluginWorkerManager();
try {
await writeFile(savedFile, "saved work remains intact", { mode: 0o600 });
await build({
entryPoints: { worker: path.join(providerRoot, "src/worker.ts"), manifest: path.join(providerRoot, "src/manifest.ts") },
outdir: root,
outExtension: { ".js": ".mjs" },
platform: "node",
format: "esm",
target: "node24",
bundle: true,
logLevel: "silent",
nodePaths: [path.join(repoRoot, "server/node_modules")],
alias: {
"@daytonaio/sdk": path.join(repoRoot, "server/src/__tests__/fixtures/daytona-worker-lifecycle-sdk.mjs"),
"@paperclipai/plugin-sdk": path.join(repoRoot, "packages/plugins/sdk/dist/index.js"),
},
banner: { js: 'import { createRequire } from "node:module"; const require = createRequire(import.meta.url);' },
});
const { default: manifest } = await import(pathToFileURL(path.join(root, "manifest.mjs")).href);
const worker = await manager.startWorker(pluginId, {
entrypointPath: workerFile,
manifest,
config: {},
instanceInfo: { instanceId: "fixture-instance", hostVersion: "0.0.0" },
apiVersion: 1,
hostHandlers: {},
env: { PAPERCLIP_DAYTONA_FIXTURE_JOURNAL: journal, PAPERCLIP_DAYTONA_FIXTURE_SAVED_FILE: savedFile,
PAPERCLIP_DAYTONA_FIXTURE_HANG_CLOSE: mode === "stalled-close" ? "1" : "0" },
duplexChannelLimits: { closeTimeoutMs: 500 },
});
const pid = worker.diagnostics().pid;
const exited = new Promise<void>(resolve => worker.on("exit", () => resolve()));
const base = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment",
config: { apiKey: "fixture-not-a-credential", image: "fixture-image", timeoutMs: 2_000, livenessTimeoutMs: 250, reuseLease: true } };
const acquisition = { ...base, agentId: "fixture-agent", executionWorkspaceId: "fixture-workspace" };
const first = await manager.call(pluginId, "environmentAcquireLease", { ...acquisition, runId: "fixture-first-run" });
const next = await manager.call(pluginId, "environmentAcquireLease", { ...acquisition, runId: "fixture-next-run" });
const channel = await worker.openDuplexChannel({
driverKey: base.driverKey, companyId: base.companyId, environmentId: base.environmentId,
providerLeaseId: first.providerLeaseId!, command: ["fixture-bridge"],
});
const teardown = { ...base, providerLeaseId: first.providerLeaseId! };
if (mode === "resume") {
await manager.call(pluginId, "environmentExecute", { ...base, lease: first,
command: "fixture-provider-stop", bypassSession: true });
await expect(manager.call(pluginId, "environmentResumeLease", { ...teardown, leaseMetadata: first.metadata }))
.resolves.toMatchObject({ providerLeaseId: first.providerLeaseId, metadata: { resumedFromState: "stopped" } });
} else if (mode === "stop") await manager.call(pluginId, "environmentStopLease", teardown);
else if (mode === "destroy") await manager.call(pluginId, "environmentDestroyLease", teardown);
else await manager.call(pluginId, "environmentReleaseLease", { ...teardown, cancelActiveWork: mode === "cancel" });
await channel.close();
if (mode === "stalled-close") await exited;
else await Promise.race([exited, new Promise<void>(resolve => setTimeout(resolve, 100))]);
const execution = manager.call(pluginId, "environmentExecute", {
...base, lease: next, command: "printf", args: ["monitor-after-release"], bypassSession: true,
});
if (mode === "stalled-close") await expect(execution).rejects.toThrow("is stopped");
else await expect(execution).resolves.toMatchObject({ exitCode: 0, stdout: "saved work remains intact" });
expect(worker.status).toBe(mode === "stalled-close" ? "stopped" : "running");
expect(manager.getWorker(pluginId)).toBe(worker);
expect(worker.diagnostics().pid).toBe(mode === "stalled-close" ? null : pid);
channel.write(new TextEncoder().encode("closed channel must remain closed"));
channel.kill();
if (mode !== "resume") await expect(manager.call(pluginId, "environmentExecute", {
...base, lease: first, command: "printf", args: ["closed-lease-after-release"], bypassSession: true,
})).rejects.toThrow();
const events = (await readFile(journal, "utf8")).trim().split("\n").map(line => JSON.parse(line));
expect(events.filter(event => event.event === "create")).toHaveLength(2);
expect(events.filter(event => event.event === "pty.open")).toHaveLength(1);
expect(events.filter(event => event.event === "pty.write")).toHaveLength(1);
expect(events.filter(event => event.event === "pty.kill")).toHaveLength(1);
expect(events.filter(event => event.event === "pty.disconnect")).toHaveLength(1);
expect(events.filter(event => event.event === "execute" && event.afterRelease))
.toEqual(mode === "stalled-close" ? [] : [{ event: "execute", id: next.providerLeaseId, afterRelease: true }]);
expect(await readFile(savedFile, "utf8")).toBe("saved work remains intact");
} finally {
await manager.stopAll();
await rm(root, { recursive: true, force: true });
}
});