mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 01:50:13 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox providers such as Daytona run as plugin workers that the host manages in `server/src/services/plugin-worker-manager.ts` > - The host retires a worker when it cannot prove that a login pseudo-terminal or a duplex channel route is gone (fail closed) > - Retirement used `killProcess()`, which marks a deliberate stop, so the handle stayed `stopped` and no restart followed > - Every later call then failed with `Cannot call "<method>" — worker for "<id>" is stopped` until the server process restarted > - This pull request makes a retirement replace the worker with a fresh process through the existing backoff timer, with a bound on repeated retirements > - The benefit is that one unconfirmed close no longer makes a sandbox provider unavailable for the whole server ## Linked Issues or Issue Description Refs #19150 **What happened?** A Daytona plugin worker was retired after an unacknowledged duplex close. After that, every run that used the Daytona environment failed setup with `Cannot call "environmentExecute" — worker for <plugin> is stopped`. 15 runs failed in about 12 minutes. The plugin stayed unavailable until the server restarted. #19150 removed the specific trigger. It keeps fail-closed retirement for close replies that the host cannot confirm, so the same outage can still occur. **Expected behavior** The retired process, its routes, and its tombstones are never reused. The plugin comes back on a fresh process after a short, bounded backoff. A deliberate stop (plugin disable, upgrade, shutdown) stays stopped. **Steps to reproduce** 1. Start a plugin worker and open a duplex channel whose close acknowledgement is wrong or missing. 2. Close the channel. The host retires the worker. 3. Call any worker method. Before this change, the call fails with `is stopped` forever. ## What Changed - Add `retireWorkerProcess()`. It kills the process but does not mark a deliberate stop. The three fail-closed retirement paths use it instead of `killProcess()`. - In the exit handler, a retired exit settles every route as before, then counts the retirement and schedules a fresh process through the existing backoff timer (`crashed` → `backoff` → `running`). It emits the existing `crash` event with `willRestart`. - A deliberate stop wins. If `stop()` runs before the exit, the exit ends `stopped`. If `stop()` runs during backoff, it cancels the restart. - Bound repeated retirements: at most 5 retirements in the 10-minute crash window. A successful restart does not reset this count (it does reset the crash count), so a worker that starts cleanly and is retired again cannot loop forever. After the bound, the worker stays `crashed`. - `scheduleRestart()` / `computeBackoffMs()` take the attempt number, so the crash path and the retirement path each use their own counter. - Add an optional `crashRecovery` start option (`minBackoffMs`, `maxConsecutiveRetirements`) so tests can use short waits. - Update the Daytona worker close test: a stalled close now shows `backoff`, then a fresh `running` process with a new pid. ## Verification - Red before the fix: the three new tests in `server/src/__tests__/plugin-worker-manager-duplex.test.ts` (`plugin worker replacement after a fail-closed retirement`) failed with `expected 'stopped' to be 'running'` / `'backoff'`. - Green after the fix. The tests prove: - the retired route and a sibling route on the same worker both settle with `{ exitCode: null }`; - the fresh process has a new pid, a write to a retired route does not revive it, and a new route on the fresh process echoes data; - `stop()` during backoff stays `stopped` with no process; - with `maxConsecutiveRetirements: 2`, the third retirement leaves the worker `crashed` and `call()` rejects. - The real Daytona plugin worker test (`plugin-worker-manager-daytona-close.test.ts`, `stalled-close`) now shows the replacement with the actual provider bundle. The closed lease still rejects on the fresh process. - `cd server && npx vitest run src/__tests__/plugin src/__tests__/managed-plugin src/services/plugin` → 48 files, 649 tests pass. The new tests passed 5 runs in a row. - `npx tsc --noEmit -p server` shows no errors in the changed files. - Not run: a live staging run. This PR does not deploy. ## Risks - A plugin that keeps causing retirements now restarts up to 5 times in 10 minutes (1s, 2s, 4s, 8s, 16s backoff) instead of staying dead. After that, it stays `crashed`, as a crash loop does now. - A retirement now emits `plugin.worker.crashed`, so host-service cleanup runs for the plugin, the same as for a crash. A restart emits `plugin.worker.restarted`. A retirement also counts in `totalCrashes`, so idle sleep treats the worker as it treats a crashed worker. - A call made during backoff fails with `is backoff` rather than `is stopped`. The outage is bounded to the backoff, not to a server restart. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
106 lines
6.3 KiB
TypeScript
106 lines
6.3 KiB
TypeScript
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { fileURLToPath, pathToFileURL } from "node:url";
|
|
import { build } from "esbuild";
|
|
import { expect, it, vi } from "vitest";
|
|
import { createPluginWorkerManager } from "../services/plugin-worker-manager.js";
|
|
|
|
const repoRoot = fileURLToPath(new URL("../../..", import.meta.url));
|
|
const providerRoot = path.join(repoRoot, "packages/plugins/sandbox-providers/daytona");
|
|
const pluginId = "fixture-daytona-provider";
|
|
|
|
it.each(["release", "cancel", "stop", "destroy", "resume", "stalled-close"] as const)("preserves worker ownership after %s of a bound duplex channel", async mode => {
|
|
const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-worker-close-"));
|
|
const journal = path.join(root, "provider.jsonl"), savedFile = path.join(root, "saved.txt");
|
|
const workerFile = path.join(root, "worker.mjs");
|
|
const manager = createPluginWorkerManager();
|
|
try {
|
|
await writeFile(savedFile, "saved work remains intact", { mode: 0o600 });
|
|
await build({
|
|
entryPoints: { worker: path.join(providerRoot, "src/worker.ts"), manifest: path.join(providerRoot, "src/manifest.ts") },
|
|
outdir: root,
|
|
outExtension: { ".js": ".mjs" },
|
|
platform: "node",
|
|
format: "esm",
|
|
target: "node24",
|
|
bundle: true,
|
|
logLevel: "silent",
|
|
nodePaths: [path.join(repoRoot, "server/node_modules")],
|
|
alias: {
|
|
"@daytonaio/sdk": path.join(repoRoot, "server/src/__tests__/fixtures/daytona-worker-lifecycle-sdk.mjs"),
|
|
"@paperclipai/plugin-sdk": path.join(repoRoot, "packages/plugins/sdk/dist/index.js"),
|
|
},
|
|
banner: { js: 'import { createRequire } from "node:module"; const require = createRequire(import.meta.url);' },
|
|
});
|
|
const { default: manifest } = await import(pathToFileURL(path.join(root, "manifest.mjs")).href);
|
|
const worker = await manager.startWorker(pluginId, {
|
|
entrypointPath: workerFile,
|
|
manifest,
|
|
config: {},
|
|
instanceInfo: { instanceId: "fixture-instance", hostVersion: "0.0.0" },
|
|
apiVersion: 1,
|
|
hostHandlers: {},
|
|
env: { PAPERCLIP_DAYTONA_FIXTURE_JOURNAL: journal, PAPERCLIP_DAYTONA_FIXTURE_SAVED_FILE: savedFile,
|
|
PAPERCLIP_DAYTONA_FIXTURE_HANG_CLOSE: mode === "stalled-close" ? "1" : "0" },
|
|
duplexChannelLimits: { closeTimeoutMs: 500 },
|
|
});
|
|
const pid = worker.diagnostics().pid;
|
|
const exited = new Promise<void>(resolve => worker.on("exit", () => resolve()));
|
|
const base = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment",
|
|
config: { apiKey: "fixture-not-a-credential", image: "fixture-image", timeoutMs: 2_000, livenessTimeoutMs: 250, reuseLease: true } };
|
|
const acquisition = { ...base, agentId: "fixture-agent", executionWorkspaceId: "fixture-workspace" };
|
|
const first = await manager.call(pluginId, "environmentAcquireLease", { ...acquisition, runId: "fixture-first-run" });
|
|
const next = await manager.call(pluginId, "environmentAcquireLease", { ...acquisition, runId: "fixture-next-run" });
|
|
const channel = await worker.openDuplexChannel({
|
|
driverKey: base.driverKey, companyId: base.companyId, environmentId: base.environmentId,
|
|
providerLeaseId: first.providerLeaseId!, command: ["fixture-bridge"],
|
|
});
|
|
const teardown = { ...base, providerLeaseId: first.providerLeaseId! };
|
|
if (mode === "resume") {
|
|
await manager.call(pluginId, "environmentExecute", { ...base, lease: first,
|
|
command: "fixture-provider-stop", bypassSession: true });
|
|
await expect(manager.call(pluginId, "environmentResumeLease", { ...teardown, leaseMetadata: first.metadata }))
|
|
.resolves.toMatchObject({ providerLeaseId: first.providerLeaseId, metadata: { resumedFromState: "stopped" } });
|
|
} else if (mode === "stop") await manager.call(pluginId, "environmentStopLease", teardown);
|
|
else if (mode === "destroy") await manager.call(pluginId, "environmentDestroyLease", teardown);
|
|
else await manager.call(pluginId, "environmentReleaseLease", { ...teardown, cancelActiveWork: mode === "cancel" });
|
|
await channel.close();
|
|
if (mode === "stalled-close") await exited;
|
|
else await Promise.race([exited, new Promise<void>(resolve => setTimeout(resolve, 100))]);
|
|
const execution = manager.call(pluginId, "environmentExecute", {
|
|
...base, lease: next, command: "printf", args: ["monitor-after-release"], bypassSession: true,
|
|
});
|
|
if (mode === "stalled-close") {
|
|
// The unconfirmed close retires the process. The plugin is unavailable only
|
|
// while the bounded restart backoff runs, then a fresh process replaces it.
|
|
await expect(execution).rejects.toThrow("is backoff");
|
|
await vi.waitFor(() => expect(worker.status).toBe("running"), { timeout: 5_000 });
|
|
expect(worker.diagnostics().pid).not.toBe(pid);
|
|
expect(worker.diagnostics().pid).not.toBeNull();
|
|
} else {
|
|
await expect(execution).resolves.toMatchObject({ exitCode: 0, stdout: "saved work remains intact" });
|
|
expect(worker.diagnostics().pid).toBe(pid);
|
|
}
|
|
expect(worker.status).toBe("running");
|
|
expect(manager.getWorker(pluginId)).toBe(worker);
|
|
channel.write(new TextEncoder().encode("closed channel must remain closed"));
|
|
channel.kill();
|
|
if (mode !== "resume") await expect(manager.call(pluginId, "environmentExecute", {
|
|
...base, lease: first, command: "printf", args: ["closed-lease-after-release"], bypassSession: true,
|
|
})).rejects.toThrow();
|
|
const events = (await readFile(journal, "utf8")).trim().split("\n").map(line => JSON.parse(line));
|
|
expect(events.filter(event => event.event === "create")).toHaveLength(2);
|
|
expect(events.filter(event => event.event === "pty.open")).toHaveLength(1);
|
|
expect(events.filter(event => event.event === "pty.write")).toHaveLength(1);
|
|
expect(events.filter(event => event.event === "pty.kill")).toHaveLength(1);
|
|
expect(events.filter(event => event.event === "pty.disconnect")).toHaveLength(1);
|
|
expect(events.filter(event => event.event === "execute" && event.afterRelease))
|
|
.toEqual(mode === "stalled-close" ? [] : [{ event: "execute", id: next.providerLeaseId, afterRelease: true }]);
|
|
expect(await readFile(savedFile, "utf8")).toBe("saved work remains intact");
|
|
} finally {
|
|
await manager.stopAll();
|
|
await rm(root, { recursive: true, force: true });
|
|
}
|
|
});
|