Files
PaperClipAI/server/src/__tests__/plugin-worker-manager-daytona-close.test.ts
T
DottaandClaude Opus 5.5 0c9962a9bc fix(plugins): replace a retired worker with a fresh process (#19245)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Sandbox providers such as Daytona run as plugin workers that the
host manages in `server/src/services/plugin-worker-manager.ts`
> - The host retires a worker when it cannot prove that a login
pseudo-terminal or a duplex channel route is gone (fail closed)
> - Retirement used `killProcess()`, which marks a deliberate stop, so
the handle stayed `stopped` and no restart followed
> - Every later call then failed with `Cannot call "<method>" — worker
for "<id>" is stopped` until the server process restarted
> - This pull request makes a retirement replace the worker with a fresh
process through the existing backoff timer, with a bound on repeated
retirements
> - The benefit is that one unconfirmed close no longer makes a sandbox
provider unavailable for the whole server

## Linked Issues or Issue Description

Refs #19150

**What happened?**
A Daytona plugin worker was retired after an unacknowledged duplex
close. After that, every run that used the Daytona environment failed
setup with `Cannot call "environmentExecute" — worker for <plugin> is
stopped`. 15 runs failed in about 12 minutes. The plugin stayed
unavailable until the server restarted. #19150 removed the specific
trigger. It keeps fail-closed retirement for close replies that the host
cannot confirm, so the same outage can still occur.

**Expected behavior**
The retired process, its routes, and its tombstones are never reused.
The plugin comes back on a fresh process after a short, bounded backoff.
A deliberate stop (plugin disable, upgrade, shutdown) stays stopped.

**Steps to reproduce**
1. Start a plugin worker and open a duplex channel whose close
acknowledgement is wrong or missing.
2. Close the channel. The host retires the worker.
3. Call any worker method. Before this change, the call fails with `is
stopped` forever.

## What Changed

- Add `retireWorkerProcess()`. It kills the process but does not mark a
deliberate stop. The three fail-closed retirement paths use it instead
of `killProcess()`.
- In the exit handler, a retired exit settles every route as before,
then counts the retirement and schedules a fresh process through the
existing backoff timer (`crashed` → `backoff` → `running`). It emits the
existing `crash` event with `willRestart`.
- A deliberate stop wins. If `stop()` runs before the exit, the exit
ends `stopped`. If `stop()` runs during backoff, it cancels the restart.
- Bound repeated retirements: at most 5 retirements in the 10-minute
crash window. A successful restart does not reset this count (it does
reset the crash count), so a worker that starts cleanly and is retired
again cannot loop forever. After the bound, the worker stays `crashed`.
- `scheduleRestart()` / `computeBackoffMs()` take the attempt number, so
the crash path and the retirement path each use their own counter.
- Add an optional `crashRecovery` start option (`minBackoffMs`,
`maxConsecutiveRetirements`) so tests can use short waits.
- Update the Daytona worker close test: a stalled close now shows
`backoff`, then a fresh `running` process with a new pid.

## Verification

- Red before the fix: the three new tests in
`server/src/__tests__/plugin-worker-manager-duplex.test.ts` (`plugin
worker replacement after a fail-closed retirement`) failed with
`expected 'stopped' to be 'running'` / `'backoff'`.
- Green after the fix. The tests prove:
- the retired route and a sibling route on the same worker both settle
with `{ exitCode: null }`;
- the fresh process has a new pid, a write to a retired route does not
revive it, and a new route on the fresh process echoes data;
  - `stop()` during backoff stays `stopped` with no process;
- with `maxConsecutiveRetirements: 2`, the third retirement leaves the
worker `crashed` and `call()` rejects.
- The real Daytona plugin worker test
(`plugin-worker-manager-daytona-close.test.ts`, `stalled-close`) now
shows the replacement with the actual provider bundle. The closed lease
still rejects on the fresh process.
- `cd server && npx vitest run src/__tests__/plugin
src/__tests__/managed-plugin src/services/plugin` → 48 files, 649 tests
pass. The new tests passed 5 runs in a row.
- `npx tsc --noEmit -p server` shows no errors in the changed files.
- Not run: a live staging run. This PR does not deploy.

## Risks

- A plugin that keeps causing retirements now restarts up to 5 times in
10 minutes (1s, 2s, 4s, 8s, 16s backoff) instead of staying dead. After
that, it stays `crashed`, as a crash loop does now.
- A retirement now emits `plugin.worker.crashed`, so host-service
cleanup runs for the plugin, the same as for a crash. A restart emits
`plugin.worker.restarted`. A retirement also counts in `totalCrashes`,
so idle sleep treats the worker as it treats a crashed worker.
- A call made during backoff fails with `is backoff` rather than `is
stopped`. The outage is bounded to the backoff, not to a server restart.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 08:37:16 -06:00

106 lines
6.3 KiB
TypeScript

import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import { build } from "esbuild";
import { expect, it, vi } from "vitest";
import { createPluginWorkerManager } from "../services/plugin-worker-manager.js";
const repoRoot = fileURLToPath(new URL("../../..", import.meta.url));
const providerRoot = path.join(repoRoot, "packages/plugins/sandbox-providers/daytona");
const pluginId = "fixture-daytona-provider";
it.each(["release", "cancel", "stop", "destroy", "resume", "stalled-close"] as const)("preserves worker ownership after %s of a bound duplex channel", async mode => {
const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-daytona-worker-close-"));
const journal = path.join(root, "provider.jsonl"), savedFile = path.join(root, "saved.txt");
const workerFile = path.join(root, "worker.mjs");
const manager = createPluginWorkerManager();
try {
await writeFile(savedFile, "saved work remains intact", { mode: 0o600 });
await build({
entryPoints: { worker: path.join(providerRoot, "src/worker.ts"), manifest: path.join(providerRoot, "src/manifest.ts") },
outdir: root,
outExtension: { ".js": ".mjs" },
platform: "node",
format: "esm",
target: "node24",
bundle: true,
logLevel: "silent",
nodePaths: [path.join(repoRoot, "server/node_modules")],
alias: {
"@daytonaio/sdk": path.join(repoRoot, "server/src/__tests__/fixtures/daytona-worker-lifecycle-sdk.mjs"),
"@paperclipai/plugin-sdk": path.join(repoRoot, "packages/plugins/sdk/dist/index.js"),
},
banner: { js: 'import { createRequire } from "node:module"; const require = createRequire(import.meta.url);' },
});
const { default: manifest } = await import(pathToFileURL(path.join(root, "manifest.mjs")).href);
const worker = await manager.startWorker(pluginId, {
entrypointPath: workerFile,
manifest,
config: {},
instanceInfo: { instanceId: "fixture-instance", hostVersion: "0.0.0" },
apiVersion: 1,
hostHandlers: {},
env: { PAPERCLIP_DAYTONA_FIXTURE_JOURNAL: journal, PAPERCLIP_DAYTONA_FIXTURE_SAVED_FILE: savedFile,
PAPERCLIP_DAYTONA_FIXTURE_HANG_CLOSE: mode === "stalled-close" ? "1" : "0" },
duplexChannelLimits: { closeTimeoutMs: 500 },
});
const pid = worker.diagnostics().pid;
const exited = new Promise<void>(resolve => worker.on("exit", () => resolve()));
const base = { driverKey: "daytona", companyId: "fixture-company", environmentId: "fixture-environment",
config: { apiKey: "fixture-not-a-credential", image: "fixture-image", timeoutMs: 2_000, livenessTimeoutMs: 250, reuseLease: true } };
const acquisition = { ...base, agentId: "fixture-agent", executionWorkspaceId: "fixture-workspace" };
const first = await manager.call(pluginId, "environmentAcquireLease", { ...acquisition, runId: "fixture-first-run" });
const next = await manager.call(pluginId, "environmentAcquireLease", { ...acquisition, runId: "fixture-next-run" });
const channel = await worker.openDuplexChannel({
driverKey: base.driverKey, companyId: base.companyId, environmentId: base.environmentId,
providerLeaseId: first.providerLeaseId!, command: ["fixture-bridge"],
});
const teardown = { ...base, providerLeaseId: first.providerLeaseId! };
if (mode === "resume") {
await manager.call(pluginId, "environmentExecute", { ...base, lease: first,
command: "fixture-provider-stop", bypassSession: true });
await expect(manager.call(pluginId, "environmentResumeLease", { ...teardown, leaseMetadata: first.metadata }))
.resolves.toMatchObject({ providerLeaseId: first.providerLeaseId, metadata: { resumedFromState: "stopped" } });
} else if (mode === "stop") await manager.call(pluginId, "environmentStopLease", teardown);
else if (mode === "destroy") await manager.call(pluginId, "environmentDestroyLease", teardown);
else await manager.call(pluginId, "environmentReleaseLease", { ...teardown, cancelActiveWork: mode === "cancel" });
await channel.close();
if (mode === "stalled-close") await exited;
else await Promise.race([exited, new Promise<void>(resolve => setTimeout(resolve, 100))]);
const execution = manager.call(pluginId, "environmentExecute", {
...base, lease: next, command: "printf", args: ["monitor-after-release"], bypassSession: true,
});
if (mode === "stalled-close") {
// The unconfirmed close retires the process. The plugin is unavailable only
// while the bounded restart backoff runs, then a fresh process replaces it.
await expect(execution).rejects.toThrow("is backoff");
await vi.waitFor(() => expect(worker.status).toBe("running"), { timeout: 5_000 });
expect(worker.diagnostics().pid).not.toBe(pid);
expect(worker.diagnostics().pid).not.toBeNull();
} else {
await expect(execution).resolves.toMatchObject({ exitCode: 0, stdout: "saved work remains intact" });
expect(worker.diagnostics().pid).toBe(pid);
}
expect(worker.status).toBe("running");
expect(manager.getWorker(pluginId)).toBe(worker);
channel.write(new TextEncoder().encode("closed channel must remain closed"));
channel.kill();
if (mode !== "resume") await expect(manager.call(pluginId, "environmentExecute", {
...base, lease: first, command: "printf", args: ["closed-lease-after-release"], bypassSession: true,
})).rejects.toThrow();
const events = (await readFile(journal, "utf8")).trim().split("\n").map(line => JSON.parse(line));
expect(events.filter(event => event.event === "create")).toHaveLength(2);
expect(events.filter(event => event.event === "pty.open")).toHaveLength(1);
expect(events.filter(event => event.event === "pty.write")).toHaveLength(1);
expect(events.filter(event => event.event === "pty.kill")).toHaveLength(1);
expect(events.filter(event => event.event === "pty.disconnect")).toHaveLength(1);
expect(events.filter(event => event.event === "execute" && event.afterRelease))
.toEqual(mode === "stalled-close" ? [] : [{ event: "execute", id: next.providerLeaseId, afterRelease: true }]);
expect(await readFile(savedFile, "utf8")).toBe("saved work remains intact");
} finally {
await manager.stopAll();
await rm(root, { recursive: true, force: true });
}
});