mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
fix(daytona): retain duplex close ownership after lease cleanup (#19150)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote agents use provider plugins to manage sandbox leases and duplex channels. > - The host verifies the exact route and session when a bound channel closes. > - Daytona lease teardown removed that binding before a later host close. > - The incomplete reply made the host retire the shared worker. A later command then failed during setup. > - This pull request retains bounded close receipts and joins the original cleanup. > - Later work can use the same worker while invalid or unconfirmed closes still fail closed. ## Linked Issues or Issue Description **What happened?** A normal lease teardown can close a duplex channel before the host closes its bound route. Daytona then replies with only the route ID. The host requires the original session ID too. It logs `duplex channel close not acknowledged; retiring worker`, stops the worker, and rejects a later `environmentExecute` call as stopped. **Expected behavior** A later close must identify the original route and session and join its existing cleanup. It must not revive the channel or restart the worker. Unknown ownership and an unconfirmed close must retain the existing failure behavior. **Steps to reproduce** 1. Acquire two leases through one Daytona provider worker. 2. Open a duplex channel on the first lease. 3. Release that lease, then close the bound channel through the host. 4. Execute a command on the second lease. **Paperclip version or commit** The real host/worker regression fails on `69dea878c0b920b335594debb0286443ba9f6c35`. This branch includes master `3a72efebe5b050e7594bb309e71a0d5f3072db00`. **Deployment mode** Managed Daytona sandbox execution in Cloud. The regression uses a local real host and SDK worker. Only the external Daytona SDK and PTY are simulated. Related work: Refs #19004, Refs #10429, Refs #18065, Refs #19050. These cover task Retry, managed worker startup, drain release, and stale provider sessions. ## What Changed - Retain at most 4096 retired route/session bindings with their original close promises. - Remove live channel authority before cleanup starts. Concurrent and later closes join the same promise and return the stored binding. - Reject reuse of a retained route ID. Keep unknown and evicted routes unable to provide a session acknowledgement. - Add a real child-process host/SDK-worker/Daytona regression with only the external Daytona SDK and PTY simulated. - Cover release, cancellation, stop, destroy, resume, a stalled close, concurrent closes, failed cleanup, forged IDs and bounded eviction. - Document close receipt scope and its separate provider-termination checks. ## Verification - RED before the repair: the actual host and SDK worker reproduced the retirement log, SIGKILL, stopped state, and later `environmentExecute` rejection. The direct provider cases also failed because the session ID was missing. - Combined-branch provider suite: 364 passed. - Combined-branch host manager, duplex, managed readiness and composed suites: 142 passed. The final test-only manifest-loading change passed all 6 composed cases again. - Full repository typecheck (`node scripts/typecheck-low-memory.mjs`), focused server types and the standalone Daytona build passed. The provider build uses pinned SDK `0.203.0` and TypeScript `5.9.3`. - The composed cases check the same worker/PID, one later command, preserved saved content, revoked channel writes/stops, and continued rejection of commands on closed leases. A stalled close still retires the worker and dispatches no later command. - Reproduce with `pnpm --dir server exec vitest run --config vitest.config.ts src/__tests__/plugin-worker-manager-daytona-close.test.ts` from a checkout with the SDK build dependencies prepared. Run the provider suite with its own Vitest config. - An initial local typecheck lacked the excluded providers' SDK links after an install with scripts disabled. The standard root postinstall linker restored those local links, and full typechecking then passed. The fixture loads its actual built manifest. Failed setup output is retained; no type configuration was relaxed. - The first two composed attempts stopped during fixture setup. They are not the behavioral RED above. Their failed outputs are retained. - The first broad local typecheck stopped at an IPC permission error during preflight. It passed after rerunning with allowed local IPC and the standard SDK links. - [Full repository CI](https://github.com/paperclipai/paperclip/actions/runs/41945022881) passed all 48 jobs on this head. Selection fell back to the complete task plan, so the run was unfiltered. The complete check inventory has 54 successes and two expected Storybook skips; the separate Snyk status passed. Fresh Greptile review on `daebbeb5005b9d9bd59d32ea30094279048b9213` is [5/5 with no actionable comments](https://github.com/paperclipai/paperclip/pull/19150#issuecomment-6646621914). Independent production and fixture review found no actionable findings. Hosted acceptance of this repair has not run. ## Risks - A receipt removed at the 4096-entry bound cannot confirm an old bound close. The existing host guard remains effective in that case. - Close acknowledgements identify channels. They do not certify remote process termination or clear unknown external work. Provider termination receipts and existing recovery gates remain in force. - Hosted evidence confirms the unacknowledged-close retirement class. The historical log does not distinguish a missing session ID from a timeout or another close error. The regression proves and repairs the missing-binding path; it also preserves timeout retirement. - The provider is outside the main pnpm workspace. A new Cloud image must compile its changed source. The unchanged plugin version alone does not prove activation. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source editing and tool execution. The exact model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
11f2a5293d
commit
9e5039f6c8
5 files changed
+301
-4
No files matched your search
@@ -3038,6 +3038,18 @@ Older packs without the declaration retain content verification until the image
|
||||
is updated. Selected ACPX agent profile checks remain in force.
|
||||
Daytona detects gzip/zstd headers and skips redundant transport compression.
|
||||
|
||||
### Daytona duplex close receipts
|
||||
|
||||
Lease teardown removes duplex routes from command admission before it closes
|
||||
their transports. The provider retains up to 4096 route/session bindings and
|
||||
their close promises, so a later host close joins the same cleanup and returns
|
||||
the original binding. These records grant no write, stop, or open authority.
|
||||
Unknown or evicted routes cannot supply a session acknowledgement. A pending
|
||||
close still waits; an unconfirmed close still retires the worker through the
|
||||
existing host guard. The acknowledgement identifies the channel. Provider stop
|
||||
receipts and unresolved external-work checks still determine process termination.
|
||||
No command is replayed or worker restarted by this mechanism.
|
||||
|
||||
### Remote GitHub launcher staging
|
||||
|
||||
Managed GitHub launchers stage their nine text files in one bounded control
|
||||
|
||||
@@ -40,6 +40,7 @@ import type { PluginContext } from "@paperclipai/plugin-sdk";
|
||||
import { environmentSyncErrorData, readEnvironmentSyncErrorDiagnostic, withEnvironmentSyncErrorCapture, environmentCreationCleanupErrorData, readEnvironmentCreationCleanupError, readEnvironmentAcquisitionDiagnostic } from "@paperclipai/plugin-sdk";
|
||||
import manifest from "./manifest.js";
|
||||
import { parseTarVerboseListingLine, splitLinkEntryOnce } from "./file-sync.js";
|
||||
import * as duplexCommands from "./duplex-command-stream.js";
|
||||
|
||||
describe("Daytona read-only resource inspection", () => {
|
||||
const params = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment",
|
||||
@@ -729,6 +730,8 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
});
|
||||
expect(killed).toBeGreaterThanOrEqual(1);
|
||||
expect(disconnected).toBe(1);
|
||||
await expect(plugin.definition.onDuplexChannelClose!({ hostRouteId: "route-1" }))
|
||||
.resolves.toEqual({ hostRouteId: "route-1", workerSessionId });
|
||||
|
||||
// The channel entry is gone, so a later write is a no-op and reaches no
|
||||
// process.
|
||||
@@ -746,6 +749,94 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
}
|
||||
});
|
||||
|
||||
describe("retired duplex ownership", () => {
|
||||
async function fixture() {
|
||||
const params = { driverKey: "daytona", companyId: "company-1", environmentId: "env-1",
|
||||
config: { apiKey: "fixture-key", image: "fixture-image", timeoutMs: 3_000, livenessTimeoutMs: 1_000, reuseLease: true } };
|
||||
const pty = { waitForConnection: vi.fn(async () => {}), sendInput: vi.fn(async () => {}),
|
||||
wait: vi.fn(() => new Promise<{ exitCode: number }>(() => {})),
|
||||
kill: vi.fn(async () => {}), disconnect: vi.fn(async () => {}) };
|
||||
const sandbox = createMockSandbox();
|
||||
const createPty = vi.fn(async () => pty);
|
||||
Object.assign(sandbox.process, { createPty });
|
||||
mockCreate.mockResolvedValue(sandbox);
|
||||
await plugin.definition.onEnvironmentAcquireLease!({ ...params, runId: "run-1" });
|
||||
const opening = { driverKey: params.driverKey, companyId: params.companyId, environmentId: params.environmentId,
|
||||
providerLeaseId: sandbox.id, command: ["fixture-bridge"] };
|
||||
return { params: { ...params, providerLeaseId: sandbox.id }, opening, pty, createPty };
|
||||
}
|
||||
|
||||
it("joins pending teardown for concurrent closes while all channel authority stays revoked", async () => {
|
||||
const f = await fixture();
|
||||
const opened = await plugin.definition.onDuplexChannelOpen!({ ...f.opening, hostRouteId: "pending-route" });
|
||||
let finishKill!: () => void, enteredKill!: () => void;
|
||||
const entered = new Promise<void>(resolve => { enteredKill = resolve; });
|
||||
const killing = new Promise<void>(resolve => { finishKill = resolve; });
|
||||
f.pty.kill.mockImplementation(async () => { enteredKill(); await killing; });
|
||||
const requests: Promise<unknown>[] = [];
|
||||
try {
|
||||
requests.push(plugin.definition.onEnvironmentReleaseLease!(f.params));
|
||||
await entered;
|
||||
let replies = 0;
|
||||
for (let index = 0; index < 2; index++) requests.push(plugin.definition.onDuplexChannelClose!({ hostRouteId: opened.hostRouteId })
|
||||
.then(value => { replies++; return value; }));
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(replies).toBe(0);
|
||||
expect(f.pty.kill).toHaveBeenCalledTimes(1);
|
||||
expect(f.pty.disconnect).not.toHaveBeenCalled();
|
||||
await plugin.definition.onDuplexChannelWrite!({ ...opened, data: Buffer.from("closed").toString("base64") });
|
||||
await plugin.definition.onDuplexChannelStop!(opened);
|
||||
expect(f.pty.sendInput).toHaveBeenCalledTimes(1);
|
||||
expect(f.pty.kill).toHaveBeenCalledTimes(1);
|
||||
await expect(plugin.definition.onDuplexChannelOpen!({ ...f.opening, hostRouteId: opened.hostRouteId }))
|
||||
.rejects.toThrow("already been used");
|
||||
expect(f.createPty).toHaveBeenCalledTimes(1);
|
||||
finishKill();
|
||||
const result = await Promise.all(requests);
|
||||
expect(result.slice(1)).toEqual([opened, opened]);
|
||||
await expect(plugin.definition.onDuplexChannelClose!({ hostRouteId: opened.hostRouteId })).resolves.toEqual(opened);
|
||||
expect(f.pty.kill).toHaveBeenCalledTimes(1);
|
||||
expect(f.pty.disconnect).toHaveBeenCalledTimes(1);
|
||||
await expect(plugin.definition.onDuplexChannelClose!({ hostRouteId: "unknown-route", workerSessionId: opened.workerSessionId } as never))
|
||||
.resolves.toEqual({ hostRouteId: "unknown-route" });
|
||||
} finally { finishKill(); await Promise.allSettled(requests); }
|
||||
});
|
||||
|
||||
it("does not turn a rejected retired close into an acknowledgement", async () => {
|
||||
const f = await fixture();
|
||||
const close = vi.fn().mockRejectedValue(new Error("fixture closure remains unknown"));
|
||||
const restore = vi.spyOn(duplexCommands, "openDaytonaDuplexChannelSession").mockResolvedValueOnce({
|
||||
onData() {}, write: vi.fn(), kill: vi.fn(), wait: () => new Promise(() => {}), close,
|
||||
});
|
||||
try {
|
||||
const opened = await plugin.definition.onDuplexChannelOpen!({ ...f.opening, hostRouteId: "failed-route" });
|
||||
await plugin.definition.onEnvironmentReleaseLease!(f.params);
|
||||
await expect(plugin.definition.onDuplexChannelClose!({ hostRouteId: opened.hostRouteId }))
|
||||
.rejects.toThrow("fixture closure remains unknown");
|
||||
expect(close).toHaveBeenCalledTimes(1);
|
||||
await expect(plugin.definition.onDuplexChannelOpen!({ ...f.opening, hostRouteId: opened.hostRouteId }))
|
||||
.rejects.toThrow("already been used");
|
||||
} finally { restore.mockRestore(); }
|
||||
});
|
||||
|
||||
it("bounds retained acknowledgements and never echoes a supplied session id for evicted routes", async () => {
|
||||
const f = await fixture();
|
||||
const first = await plugin.definition.onDuplexChannelOpen!({ ...f.opening, hostRouteId: "oldest-route" });
|
||||
await plugin.definition.onDuplexChannelClose!({ hostRouteId: first.hostRouteId });
|
||||
let latest = first;
|
||||
for (let index = 0; index < 4_096; index++) {
|
||||
latest = await plugin.definition.onDuplexChannelOpen!({ ...f.opening, hostRouteId: `bounded-route-${index}` });
|
||||
await plugin.definition.onDuplexChannelClose!({ hostRouteId: latest.hostRouteId });
|
||||
}
|
||||
await expect(plugin.definition.onDuplexChannelClose!({ hostRouteId: first.hostRouteId, workerSessionId: first.workerSessionId } as never))
|
||||
.resolves.toEqual({ hostRouteId: first.hostRouteId });
|
||||
await expect(plugin.definition.onDuplexChannelClose!({ hostRouteId: latest.hostRouteId, workerSessionId: "forged-session" } as never))
|
||||
.resolves.toEqual(latest);
|
||||
expect(f.pty.kill).toHaveBeenCalledTimes(4_097);
|
||||
expect(f.pty.disconnect).toHaveBeenCalledTimes(4_097);
|
||||
});
|
||||
});
|
||||
|
||||
it("normalizes config and validates the API key fallback", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
|
||||
|
||||
@@ -1728,6 +1728,7 @@ export function __resetDaytonaSandboxHandleCacheForTest(): void {
|
||||
sandboxHandleWritableDirs.reset();
|
||||
sandboxHandleSessionStore.reset();
|
||||
runnerIngressGenerationStore.reset();
|
||||
retiredDaytonaDuplexChannelByRoute.clear();
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -2489,12 +2490,24 @@ interface DaytonaDuplexChannelEntry {
|
||||
}
|
||||
const daytonaDuplexChannelByRoute = new Map<string, DaytonaDuplexChannelEntry>();
|
||||
const daytonaDuplexChannelBySession = new Map<string, DaytonaDuplexChannelEntry>();
|
||||
const retiredDaytonaDuplexChannelByRoute = new Map<string, { workerSessionId: string; closed: Promise<void> }>();
|
||||
const MAX_RETIRED_DAYTONA_DUPLEX_CHANNELS = 4096;
|
||||
|
||||
function forgetDaytonaDuplexChannel(entry: DaytonaDuplexChannelEntry): void {
|
||||
daytonaDuplexChannelByRoute.delete(entry.hostRouteId);
|
||||
daytonaDuplexChannelBySession.delete(entry.workerSessionId);
|
||||
}
|
||||
|
||||
function closeDaytonaDuplexChannel(entry: DaytonaDuplexChannelEntry): Promise<void> {
|
||||
forgetDaytonaDuplexChannel(entry);
|
||||
const closed = Promise.resolve().then(() => entry.session.close());
|
||||
if (retiredDaytonaDuplexChannelByRoute.size >= MAX_RETIRED_DAYTONA_DUPLEX_CHANNELS) {
|
||||
retiredDaytonaDuplexChannelByRoute.delete(retiredDaytonaDuplexChannelByRoute.keys().next().value!);
|
||||
}
|
||||
retiredDaytonaDuplexChannelByRoute.set(entry.hostRouteId, { workerSessionId: entry.workerSessionId, closed });
|
||||
return closed;
|
||||
}
|
||||
|
||||
// Close every open duplex channel that belongs to one provider lease and drop its
|
||||
// entry. The lease teardown hooks (release, destroy, resume) call this, so a
|
||||
// channel never outlives the sandbox that carries it. The close kills the child
|
||||
@@ -2504,8 +2517,7 @@ async function closeDaytonaDuplexChannelsForLease(providerLeaseId: string): Prom
|
||||
const matches = [...daytonaDuplexChannelByRoute.values()].filter(
|
||||
(entry) => entry.providerLeaseId === providerLeaseId,
|
||||
);
|
||||
for (const entry of matches) forgetDaytonaDuplexChannel(entry);
|
||||
await Promise.allSettled(matches.map(entry => entry.session.close()));
|
||||
await Promise.allSettled(matches.map(closeDaytonaDuplexChannel));
|
||||
}
|
||||
|
||||
async function stopLease(params: PluginEnvironmentReleaseLeaseParams): Promise<PluginEnvironmentTerminationReceipt> {
|
||||
@@ -3716,6 +3728,9 @@ const plugin = definePlugin(trackDaytonaHooks({
|
||||
// `ctx.duplexChannel`, bound to the returned worker session id. Fail closed when
|
||||
// no cached sandbox matches the lease.
|
||||
async onDuplexChannelOpen(params) {
|
||||
if (daytonaDuplexChannelByRoute.has(params.hostRouteId) || retiredDaytonaDuplexChannelByRoute.has(params.hostRouteId)) {
|
||||
throw new Error("Daytona duplex channel: host route has already been used.");
|
||||
}
|
||||
const sandbox = await sandboxHandleCache.findByProviderLeaseId(params.providerLeaseId);
|
||||
if (!sandbox) {
|
||||
throw new Error(
|
||||
@@ -3794,10 +3809,14 @@ const plugin = definePlugin(trackDaytonaHooks({
|
||||
const entry = daytonaDuplexChannelByRoute.get(params.hostRouteId);
|
||||
if (entry) {
|
||||
const boundWorkerSessionId = entry.workerSessionId;
|
||||
forgetDaytonaDuplexChannel(entry);
|
||||
await entry.session.close().catch(() => undefined);
|
||||
await closeDaytonaDuplexChannel(entry).catch(() => undefined);
|
||||
return { hostRouteId: params.hostRouteId, workerSessionId: boundWorkerSessionId };
|
||||
}
|
||||
const retired = retiredDaytonaDuplexChannelByRoute.get(params.hostRouteId);
|
||||
if (retired) {
|
||||
await retired.closed;
|
||||
return { hostRouteId: params.hostRouteId, workerSessionId: retired.workerSessionId };
|
||||
}
|
||||
return { hostRouteId: params.hostRouteId };
|
||||
},
|
||||
|
||||
@@ -3814,6 +3833,7 @@ const plugin = definePlugin(trackDaytonaHooks({
|
||||
const openChannels = [...daytonaDuplexChannelByRoute.values()];
|
||||
daytonaDuplexChannelByRoute.clear();
|
||||
daytonaDuplexChannelBySession.clear();
|
||||
retiredDaytonaDuplexChannelByRoute.clear();
|
||||
for (const entry of openChannels) {
|
||||
await entry.session.close().catch(() => undefined);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
import { appendFileSync, readFileSync } from "node:fs";
|
||||
|
||||
const sandboxes = new Map();
|
||||
let nextId = 0;
|
||||
|
||||
function record(event, fields = {}) {
|
||||
appendFileSync(process.env.PAPERCLIP_DAYTONA_FIXTURE_JOURNAL, `${JSON.stringify({ event, ...fields })}\n`);
|
||||
}
|
||||
|
||||
function sandbox(params) {
|
||||
const id = `fixture-sandbox-${++nextId}`;
|
||||
const files = new Map();
|
||||
const resource = {
|
||||
id,
|
||||
name: params.name,
|
||||
labels: params.labels,
|
||||
state: "started",
|
||||
target: "us",
|
||||
autoDeleteInterval: -1,
|
||||
async getWorkDir() { return "/home/daytona"; },
|
||||
async getUserHomeDir() { return "/home/daytona"; },
|
||||
async refreshData() {},
|
||||
async setTtl() {},
|
||||
async setAutoDeleteInterval(value) { resource.autoDeleteInterval = value; },
|
||||
async start() { resource.state = "started"; record("start", { id }); },
|
||||
async stop() { resource.state = "stopped"; record("stop", { id }); },
|
||||
async delete() { sandboxes.delete(id); record("delete", { id }); },
|
||||
fs: {
|
||||
async createFolder() {},
|
||||
async uploadFile(content, destination) { files.set(destination, content.toString("utf8")); },
|
||||
async deleteFile() {},
|
||||
},
|
||||
process: {
|
||||
async executeCommand(command) {
|
||||
const afterRelease = command.includes("monitor-after-release");
|
||||
record("execute", { id, afterRelease });
|
||||
if (command.includes("fixture-provider-stop")) resource.state = "stopped";
|
||||
const sentinel = command.match(/^cat '([^']+)'/);
|
||||
const result = afterRelease ? readFileSync(process.env.PAPERCLIP_DAYTONA_FIXTURE_SAVED_FILE, "utf8")
|
||||
: sentinel ? files.get(sentinel[1]) ?? "" : "bash";
|
||||
return { exitCode: 0, result, artifacts: { stdout: result } };
|
||||
},
|
||||
async createPty() {
|
||||
let finish;
|
||||
const exited = new Promise(resolve => { finish = resolve; });
|
||||
record("pty.open", { id });
|
||||
return {
|
||||
async waitForConnection() {},
|
||||
async sendInput() { record("pty.write", { id }); },
|
||||
wait() { return exited; },
|
||||
async kill() { record("pty.kill", { id }); finish({ exitCode: 0 }); },
|
||||
async disconnect() {
|
||||
record("pty.disconnect", { id });
|
||||
if (process.env.PAPERCLIP_DAYTONA_FIXTURE_HANG_CLOSE === "1" && id === "fixture-sandbox-1") {
|
||||
await new Promise(() => {});
|
||||
}
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
};
|
||||
sandboxes.set(id, resource);
|
||||
record("create", { id });
|
||||
return resource;
|
||||
}
|
||||
|
||||
export class Daytona {
|
||||
async create(params) { return sandbox(params); }
|
||||
async get(id) {
|
||||
const result = sandboxes.get(id);
|
||||
if (!result) throw new DaytonaNotFoundError("Fixture sandbox is absent");
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
export class DaytonaNotFoundError extends Error {}
|
||||
export class DaytonaTimeoutError extends Error {}
|
||||
@@ -0,0 +1,97 @@
|
||||
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { build } from "esbuild";
|
||||
import { expect, it } from "vitest";
|
||||
import { createPluginWorkerManager } from "../services/plugin-worker-manager.js";
|
||||
|
||||
const repoRoot = fileURLToPath(new URL("../../..", import.meta.url));
|
||||
const providerRoot = path.join(repoRoot, "packages/plugins/sandbox-providers/daytona");
|
||||
const pluginId = "fixture-daytona-provider";
|
||||
|
||||
it.each(["release", "cancel", "stop", "destroy", "resume", "stalled-close"] as const)("preserves worker ownership after %s of a bound duplex channel", async mode => {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-worker-close-"));
|
||||
const journal = path.join(root, "provider.jsonl"), savedFile = path.join(root, "saved.txt");
|
||||
const workerFile = path.join(root, "worker.mjs");
|
||||
const manager = createPluginWorkerManager();
|
||||
try {
|
||||
await writeFile(savedFile, "saved work remains intact", { mode: 0o600 });
|
||||
await build({
|
||||
entryPoints: { worker: path.join(providerRoot, "src/worker.ts"), manifest: path.join(providerRoot, "src/manifest.ts") },
|
||||
outdir: root,
|
||||
outExtension: { ".js": ".mjs" },
|
||||
platform: "node",
|
||||
format: "esm",
|
||||
target: "node24",
|
||||
bundle: true,
|
||||
logLevel: "silent",
|
||||
nodePaths: [path.join(repoRoot, "server/node_modules")],
|
||||
alias: {
|
||||
"@daytonaio/sdk": path.join(repoRoot, "server/src/__tests__/fixtures/daytona-worker-lifecycle-sdk.mjs"),
|
||||
"@paperclipai/plugin-sdk": path.join(repoRoot, "packages/plugins/sdk/dist/index.js"),
|
||||
},
|
||||
banner: { js: 'import { createRequire } from "node:module"; const require = createRequire(import.meta.url);' },
|
||||
});
|
||||
const { default: manifest } = await import(pathToFileURL(path.join(root, "manifest.mjs")).href);
|
||||
const worker = await manager.startWorker(pluginId, {
|
||||
entrypointPath: workerFile,
|
||||
manifest,
|
||||
config: {},
|
||||
instanceInfo: { instanceId: "fixture-instance", hostVersion: "0.0.0" },
|
||||
apiVersion: 1,
|
||||
hostHandlers: {},
|
||||
env: { PAPERCLIP_DAYTONA_FIXTURE_JOURNAL: journal, PAPERCLIP_DAYTONA_FIXTURE_SAVED_FILE: savedFile,
|
||||
PAPERCLIP_DAYTONA_FIXTURE_HANG_CLOSE: mode === "stalled-close" ? "1" : "0" },
|
||||
duplexChannelLimits: { closeTimeoutMs: 500 },
|
||||
});
|
||||
const pid = worker.diagnostics().pid;
|
||||
const exited = new Promise<void>(resolve => worker.on("exit", () => resolve()));
|
||||
const base = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment",
|
||||
config: { apiKey: "fixture-not-a-credential", image: "fixture-image", timeoutMs: 2_000, livenessTimeoutMs: 250, reuseLease: true } };
|
||||
const acquisition = { ...base, agentId: "fixture-agent", executionWorkspaceId: "fixture-workspace" };
|
||||
const first = await manager.call(pluginId, "environmentAcquireLease", { ...acquisition, runId: "fixture-first-run" });
|
||||
const next = await manager.call(pluginId, "environmentAcquireLease", { ...acquisition, runId: "fixture-next-run" });
|
||||
const channel = await worker.openDuplexChannel({
|
||||
driverKey: base.driverKey, companyId: base.companyId, environmentId: base.environmentId,
|
||||
providerLeaseId: first.providerLeaseId!, command: ["fixture-bridge"],
|
||||
});
|
||||
const teardown = { ...base, providerLeaseId: first.providerLeaseId! };
|
||||
if (mode === "resume") {
|
||||
await manager.call(pluginId, "environmentExecute", { ...base, lease: first,
|
||||
command: "fixture-provider-stop", bypassSession: true });
|
||||
await expect(manager.call(pluginId, "environmentResumeLease", { ...teardown, leaseMetadata: first.metadata }))
|
||||
.resolves.toMatchObject({ providerLeaseId: first.providerLeaseId, metadata: { resumedFromState: "stopped" } });
|
||||
} else if (mode === "stop") await manager.call(pluginId, "environmentStopLease", teardown);
|
||||
else if (mode === "destroy") await manager.call(pluginId, "environmentDestroyLease", teardown);
|
||||
else await manager.call(pluginId, "environmentReleaseLease", { ...teardown, cancelActiveWork: mode === "cancel" });
|
||||
await channel.close();
|
||||
if (mode === "stalled-close") await exited;
|
||||
else await Promise.race([exited, new Promise<void>(resolve => setTimeout(resolve, 100))]);
|
||||
const execution = manager.call(pluginId, "environmentExecute", {
|
||||
...base, lease: next, command: "printf", args: ["monitor-after-release"], bypassSession: true,
|
||||
});
|
||||
if (mode === "stalled-close") await expect(execution).rejects.toThrow("is stopped");
|
||||
else await expect(execution).resolves.toMatchObject({ exitCode: 0, stdout: "saved work remains intact" });
|
||||
expect(worker.status).toBe(mode === "stalled-close" ? "stopped" : "running");
|
||||
expect(manager.getWorker(pluginId)).toBe(worker);
|
||||
expect(worker.diagnostics().pid).toBe(mode === "stalled-close" ? null : pid);
|
||||
channel.write(new TextEncoder().encode("closed channel must remain closed"));
|
||||
channel.kill();
|
||||
if (mode !== "resume") await expect(manager.call(pluginId, "environmentExecute", {
|
||||
...base, lease: first, command: "printf", args: ["closed-lease-after-release"], bypassSession: true,
|
||||
})).rejects.toThrow();
|
||||
const events = (await readFile(journal, "utf8")).trim().split("\n").map(line => JSON.parse(line));
|
||||
expect(events.filter(event => event.event === "create")).toHaveLength(2);
|
||||
expect(events.filter(event => event.event === "pty.open")).toHaveLength(1);
|
||||
expect(events.filter(event => event.event === "pty.write")).toHaveLength(1);
|
||||
expect(events.filter(event => event.event === "pty.kill")).toHaveLength(1);
|
||||
expect(events.filter(event => event.event === "pty.disconnect")).toHaveLength(1);
|
||||
expect(events.filter(event => event.event === "execute" && event.afterRelease))
|
||||
.toEqual(mode === "stalled-close" ? [] : [{ event: "execute", id: next.providerLeaseId, afterRelease: true }]);
|
||||
expect(await readFile(savedFile, "utf8")).toBe("saved work remains intact");
|
||||
} finally {
|
||||
await manager.stopAll();
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
Reference in new issue
Block a user