mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(plugins): retry a plugin activation whose worker initialize timed out (#20982)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins, sandbox providers such as Daytona among them, run as worker
processes that the server activates at boot. Activation sends the worker
an `initialize` RPC with a 15-second timeout.
> - When `initialize` timed out, the loader marked the plugin `error`
for the life of the server process. Only the next boot or an operator
retried it.
> - For a sandbox provider, that means every Daytona run on that server
fails. New runs fail with `lease_acquire_failed … plugin is currently
error`. Reattachments fail with "The exact sandbox lease cannot be
resumed by its verified provider". Retry hits the same dead plugin.
> - A worker that is slow or stalled for a moment at boot is transient,
not a broken plugin.
> - This pull request retries an activation whose `initialize` timed
out, with a bounded backoff. Meanwhile the server keeps serving, and
runs that need the plugin see a restarting worker, which setup retries.
Deterministic failures stay a permanent error.
> - The benefit is that a short plugin stall at boot no longer disables
a provider until the next deploy.
## Linked Issues or Issue Description
Refs #20966, #20857
**What happened?**
Chaos wave `w-20261111T232850-s9521` (cell CR4, first version) stopped
the replacement server's Daytona plugin worker while the server was
activating it. The worker's `initialize` RPC timed out after 15 s
(`INITIALIZE_TIMEOUT_MS`). `plugin-loader.ts` then called
`lifecycleManager.markError`, and the plugin stayed `error`. Task t03's
reattachment failed in `resumeRunLease`, and its Retry did not start a
new run. Task t04 failed 18 s after launch with `lease_acquire_failed`
("…plugin is currently error"), before its own fault.
**Expected behavior**
An activation timeout is retried, and the plugin ends `ready` when the
worker answers. Runs that need the plugin meanwhile are retried within
their bounded setup budget. A bad manifest, a worker that exits during
`initialize`, or a version mismatch is still a permanent error.
**Steps to reproduce**
1. Start the server.
2. Stop (SIGSTOP) a plugin worker for more than 15 s while the server
activates it.
3. Resume the worker. The plugin stays `error` until the next boot.
**Paperclip version or commit**
master at dfd2a6efca.
## What Changed
- `plugin-worker-manager.ts`: the "Worker initialize failed" error keeps
its cause, the `JsonRpcCallError` with `PLUGIN_RPC_ERROR_CODES.TIMEOUT`.
- `plugin-loader.ts`, `activatePlugin`:
- An activation whose error chain has the TIMEOUT code is retried after
5, 15, 45 and 120 s (`activationTimeoutRetryDelaysMs`).
- Before each retry, the loader tears down the partial runtime and keeps
the plugin `ready`. It sets `lastError` to "Activation timed out;
retrying in Ns (retry n of 4): …" so operators can see the state.
- A retry stops when the plugin is no longer `ready` or its worker
already runs. Shutdown clears pending retries.
- After the last retry, the loader calls `markError` with "Activation
timed out after 4 retries: …". Any other failure calls `markError` with
"Activation failed: …" at once, as before.
- `pluginLoader.requestActivationTimeoutRecovery(plugin)`: re-activates
a plugin in `error` whose `lastError` starts with "Activation timed
out", at most once per minute.
- `app.ts` exposes it through
`workerManager.requestActivationTimeoutRecovery`.
- `environment-runtime.ts`:
- Lease acquisition, for both pinned and key-resolved providers, calls
`requestActivationTimeoutRecovery` for such a plugin and reports
`worker_unavailable` (`PluginWorkerUnavailableError`, which setup
retries) instead of the terminal "plugin is currently error".
- `resumeRunLease` reports a stopped worker as
`PluginWorkerUnavailableError` when the plugin is `ready`, or is
re-activating from a timeout, instead of the generic "cannot be resumed
by its verified provider".
- This covers every plugin kind: activation retry is in the shared
loader. The on-demand path is in the sandbox-provider resolution, which
is the path a run waits on.
- `doc/plugins/PLUGIN_SPEC.md` §12 describes the rule.
## Verification
- Red on master:
`server/src/__tests__/plugin-loader-activation-timeout.test.ts` uses a
real PostgreSQL plugin registry and a bundled image fixture, and all 3
of its tests fail on master. On master, "initialize times out once, then
succeeds" stays `error` and the worker starts only once.
- Green, all 3 pass:
- A timeout followed by success ends `ready`, with a running worker and
no `markError`.
- Timeouts past the bound end `error` with the "Activation timed out
after 2 retries" message, and an on-demand request re-activates the
plugin to `ready`.
- A worker crash during `initialize` is a permanent error, with no retry
and no on-demand re-activation.
- `environment-runtime.test.ts` has new tests:
- A sandbox plugin errored by activation timeouts gives
`PluginWorkerUnavailableError` and requests re-activation. When no
re-activation is in flight, the error stays terminal.
- `resumeRunLease` on a stopped worker of a `ready` or re-activating
plugin gives `PluginWorkerUnavailableError` and sends no provider call.
- These files pass: `environment-runtime`,
`environment-run-orchestrator`, `plugin-loader-error-retry`,
`plugin-environment-driver-ready-recovery`,
`bundled-plugin-registration`, `plugin-lifecycle-restart`,
`plugin-lifecycle-delivery` and `plugin-dev-watcher`.
- `tsc --noEmit` passes for `server`.
- Chaos: cell CR6 (a plugin stalled during activation at boot) runs
before and after this change. The results are in a comment.
## Risks
- A plugin whose `initialize` keeps timing out now takes about 3 minutes
to reach `error`, instead of 15 s. During that time runs that need it
see a restarting worker and are retried within their setup budget.
- On-demand re-activation is limited to one attempt a minute per plugin,
so a stuck plugin cannot crash-loop.
- Nothing changes for deterministic activation failures.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
776e900b48
commit
a9a08d8169
9 files changed
+424
-6
No files matched your search
@@ -1178,6 +1178,26 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
expect(offlineManager.call).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("treats a sandbox plugin errored by activation timeouts as restarting while it re-activates", async () => {
|
||||
const { companyId, environment, runId, pluginId } = await seedReusablePluginSandboxLease();
|
||||
const input = { companyId, environment, issueId: null, heartbeatRunId: runId, persistedExecutionWorkspace: null };
|
||||
await db.update(plugins).set({ status: "error", lastError: "Activation timed out after 4 retries: Worker initialize failed" })
|
||||
.where(eq(plugins.id, pluginId));
|
||||
const requestActivationTimeoutRecovery = vi.fn(() => true);
|
||||
const manager = { isRunning: () => false, call: vi.fn(), requestActivationTimeoutRecovery } as unknown as PluginWorkerManager;
|
||||
const refusal = await environmentRuntimeService(db, { pluginWorkerManager: manager, pluginWorkerReadyTimeoutMs: 0 })
|
||||
.acquireRunLease(input).catch(error => error);
|
||||
expect(refusal).toBeInstanceOf(PluginWorkerUnavailableError);
|
||||
expect(requestActivationTimeoutRecovery).toHaveBeenCalledWith(expect.objectContaining({ id: pluginId, status: "error" }));
|
||||
|
||||
requestActivationTimeoutRecovery.mockReturnValue(false);
|
||||
const terminal = await environmentRuntimeService(db, { pluginWorkerManager: manager, pluginWorkerReadyTimeoutMs: 0 })
|
||||
.acquireRunLease(input).catch(error => error);
|
||||
expect(terminal).not.toBeInstanceOf(PluginWorkerUnavailableError);
|
||||
expect((terminal as Error).message).toContain("but that plugin is currently error");
|
||||
expect(manager.call).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("retains a successful reusable sandbox lease without stopping the provider resource", async () => {
|
||||
const { pluginId, runId, reusableLease } = await seedReusablePluginSandboxLease();
|
||||
const workerManager = {
|
||||
@@ -4008,6 +4028,21 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
expect(call).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it.each(["ready", "error"] as const)("refuses an exact resume as a retryable stopped worker while the %s plugin re-activates", async status => {
|
||||
const seeded = await seedReusablePluginSandboxLease("paperclip_runner");
|
||||
const lease = (await environmentService(db).getLeaseById(seeded.reusableLease.id))!;
|
||||
await db.update(plugins).set({ status, lastError: status === "error" ? "Activation timed out after 4 retries: initialize" : null })
|
||||
.where(eq(plugins.id, seeded.pluginId));
|
||||
const call = vi.fn();
|
||||
const requestActivationTimeoutRecovery = vi.fn(() => true);
|
||||
const workerManager = { isRunning: () => false, call, getWorker: () => undefined, requestActivationTimeoutRecovery } as unknown as PluginWorkerManager;
|
||||
const error = await environmentRuntimeService(db, { pluginWorkerManager: workerManager })
|
||||
.resumeRunLease({ environment: seeded.environment, lease }).catch(caught => caught);
|
||||
expect(error).toBeInstanceOf(PluginWorkerUnavailableError);
|
||||
expect(call).not.toHaveBeenCalled();
|
||||
expect(requestActivationTimeoutRecovery).toHaveBeenCalledTimes(status === "error" ? 1 : 0);
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ timeoutMs: 300_000, rpcTimeoutMs: 330_000 },
|
||||
{ timeoutMs: 1234, rpcTimeoutMs: 31_234 },
|
||||
|
||||
@@ -7,7 +7,7 @@ import { distributionBundleDigest, readDistributionPluginCatalog } from "../../s
|
||||
import { derivePluginDatabaseNamespace } from "../../services/plugin-database.js";
|
||||
import { createBundledPluginPackageResolver } from "../../services/bundled-plugin-package.js";
|
||||
|
||||
export function bundledImageFixture(manifestOverrides: Partial<PaperclipPluginManifestV1> = {}) {
|
||||
export function bundledImageFixture(manifestOverrides: Partial<PaperclipPluginManifestV1> = {}, options: { workerPrelude?: string } = {}) {
|
||||
const root = realpathSync(mkdtempSync(path.join(os.tmpdir(), "bundled-image-")));
|
||||
const packageRoot = path.join(root, "distribution", "fixture");
|
||||
const manifest: PaperclipPluginManifestV1 = {
|
||||
@@ -22,6 +22,7 @@ export function bundledImageFixture(manifestOverrides: Partial<PaperclipPluginMa
|
||||
}));
|
||||
writeFileSync(path.join(packageRoot, "dist", "manifest.js"), `export default ${JSON.stringify(manifest)};`);
|
||||
writeFileSync(path.join(packageRoot, "dist", "worker.js"), `import readline from "node:readline";
|
||||
${options.workerPrelude ?? ""}
|
||||
if (process.argv.includes("--source")) process.stdout.write(import.meta.url);
|
||||
else {
|
||||
const configs = {};
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
import { existsSync, mkdtempSync, rmSync } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { applyPendingMigrations, closeRegisteredClients, createDb, plugins } from "@paperclipai/db";
|
||||
import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES } from "@paperclipai/plugin-sdk";
|
||||
import { PLUGIN_ACTIVATION_TIMEOUT_ERROR_PREFIX, pluginLoader, type PluginRuntimeServices } from "../services/plugin-loader.js";
|
||||
import { createPluginWorkerManager } from "../services/plugin-worker-manager.js";
|
||||
import { pluginRegistryService } from "../services/plugin-registry.js";
|
||||
import { bundledImageFixture } from "./helpers/bundled-image-fixture.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
|
||||
const externalDatabaseUrl = process.env.PAPERCLIP_BUNDLED_PLUGIN_TEST_DATABASE_URL?.trim();
|
||||
const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
const describePostgres = support.supported ? describe : describe.skip;
|
||||
|
||||
const initializeTimeout = (pluginId: string) => new Error(`Worker initialize failed for "${pluginId}": RPC call "initialize" timed out after 15000ms`,
|
||||
{ cause: new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.TIMEOUT, message: 'RPC call "initialize" timed out after 15000ms' }) });
|
||||
|
||||
describePostgres("plugin activation after a worker initialize timeout", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
const roots: string[] = [];
|
||||
beforeAll(async () => {
|
||||
if (externalDatabaseUrl) {
|
||||
await applyPendingMigrations(externalDatabaseUrl);
|
||||
database = { connectionString: externalDatabaseUrl, cleanup: () => closeRegisteredClients(externalDatabaseUrl) };
|
||||
} else {
|
||||
database = await startEmbeddedPostgresTestDatabase("plugin-activation-timeout-");
|
||||
}
|
||||
db = createDb(database.connectionString);
|
||||
}, 90_000);
|
||||
afterEach(async () => {
|
||||
await db.delete(plugins);
|
||||
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
afterAll(async () => { await database?.cleanup(); });
|
||||
|
||||
async function setup(startOutcomes: Array<"timeout" | "crash" | "ok">, beforeRecoveredStart?: (pluginId: string) => Promise<void>) {
|
||||
const image = bundledImageFixture();
|
||||
roots.push(image.root);
|
||||
const options = {
|
||||
localPluginDir: path.join(image.root, "home"), prebuiltOnly: true,
|
||||
resolvePackageRoot: image.resolver.resolve, assertPackageActivation: image.resolver.assertActivation,
|
||||
enableLocalFilesystem: false, enableNpmDiscovery: false,
|
||||
activationTimeoutRetryDelaysMs: [20, 20], activationTimeoutOnDemandIntervalMs: 0,
|
||||
};
|
||||
await pluginLoader(db, options).installPlugin({ localPath: image.packageRoot });
|
||||
const registry = pluginRegistryService(db);
|
||||
const row = (await registry.getByKey(image.manifest.id))!;
|
||||
await registry.updateStatus(row.id, { status: "ready" });
|
||||
let running = false;
|
||||
let handle: { status: "crashed" | "running" } | undefined;
|
||||
const startWorker = vi.fn(async (pluginId: string) => {
|
||||
if (handle) throw new Error(`Worker already registered for plugin ${pluginId}`);
|
||||
const outcome = startOutcomes.shift() ?? "ok";
|
||||
if (outcome !== "ok") {
|
||||
handle = { status: "crashed" };
|
||||
if (outcome === "timeout") throw initializeTimeout(pluginId);
|
||||
throw new Error(`Worker initialize failed for "${pluginId}": worker exited with code 1`);
|
||||
}
|
||||
if (startWorker.mock.calls.length > 1) await beforeRecoveredStart?.(pluginId);
|
||||
handle = { status: "running" };
|
||||
running = true;
|
||||
});
|
||||
const markError = vi.fn(async (pluginId: string, error: string) => registry.updateStatus(pluginId, { status: "error", lastError: error }));
|
||||
const services = {
|
||||
workerManager: {
|
||||
startWorker, call: vi.fn(async () => ({})), isRunning: vi.fn(() => running),
|
||||
stopWorker: vi.fn(async () => { running = false; handle = undefined; }), stopAll: vi.fn(async () => { running = false; handle = undefined; }),
|
||||
getWorker: vi.fn(() => handle),
|
||||
},
|
||||
lifecycleManager: { markError },
|
||||
jobStore: { syncJobDeclarations: vi.fn() }, jobScheduler: { registerPlugin: vi.fn(), unregisterPlugin: vi.fn(), stop: vi.fn() },
|
||||
toolDispatcher: { unregisterPluginTools: vi.fn() },
|
||||
eventBus: { forPlugin: vi.fn(), subscriptionCount: vi.fn(() => 0), clearPlugin: vi.fn() },
|
||||
buildHostHandlers: vi.fn(() => ({})), instanceInfo: { instanceId: "fixture", hostVersion: "1.0.0" },
|
||||
} as unknown as PluginRuntimeServices;
|
||||
const loader = pluginLoader(db, options, services);
|
||||
return { loader, registry, row, startWorker, markError, isRunning: () => running };
|
||||
}
|
||||
|
||||
it("retries an activation whose worker initialize timed out and ends ready with a running worker", async () => {
|
||||
const { loader, registry, row, startWorker, markError, isRunning } = await setup(["timeout", "ok"]);
|
||||
const result = await loader.loadAll();
|
||||
expect(result.failed).toBe(1);
|
||||
expect((await registry.getById(row.id))!).toMatchObject({ status: "ready",
|
||||
lastError: expect.stringContaining(PLUGIN_ACTIVATION_TIMEOUT_ERROR_PREFIX) });
|
||||
await vi.waitFor(() => expect(isRunning()).toBe(true), { timeout: 5_000 });
|
||||
expect(startWorker).toHaveBeenCalledTimes(2);
|
||||
expect(markError).not.toHaveBeenCalled();
|
||||
await vi.waitFor(async () => expect((await registry.getById(row.id))!).toMatchObject({ status: "ready", lastError: null }));
|
||||
await loader.shutdownAll();
|
||||
});
|
||||
|
||||
it("does not bring back a plugin an operator disabled while its activation was timing out", async () => {
|
||||
const { loader, registry, row, startWorker } = await setup(["timeout", "ok"]);
|
||||
await loader.loadAll();
|
||||
await registry.updateStatus(row.id, { status: "disabled" });
|
||||
await new Promise(resolve => setTimeout(resolve, 200));
|
||||
expect(startWorker).toHaveBeenCalledOnce();
|
||||
expect((await registry.getById(row.id))!.status).toBe("disabled");
|
||||
expect(loader.requestActivationTimeoutRecovery({ id: row.id, status: "error", lastError: `${PLUGIN_ACTIVATION_TIMEOUT_ERROR_PREFIX} after 2 retries` })).toBe(true);
|
||||
await loader.stopActivationTimeoutRecovery();
|
||||
expect(startWorker).toHaveBeenCalledOnce();
|
||||
expect((await registry.getById(row.id))!.status).toBe("disabled");
|
||||
});
|
||||
|
||||
it("stops a recovered worker when an operator disabled the plugin during its activation", async () => {
|
||||
let disable: ((pluginId: string) => Promise<void>) | undefined;
|
||||
const { loader, registry, row, startWorker, isRunning } = await setup(["timeout", "ok"], async pluginId => { await disable?.(pluginId); });
|
||||
disable = async pluginId => { await registry.updateStatus(pluginId, { status: "disabled" }); };
|
||||
await loader.loadAll();
|
||||
await vi.waitFor(() => expect(startWorker).toHaveBeenCalledTimes(2), { timeout: 5_000 });
|
||||
await vi.waitFor(() => expect(isRunning()).toBe(false), { timeout: 5_000 });
|
||||
await loader.stopActivationTimeoutRecovery();
|
||||
expect(isRunning()).toBe(false);
|
||||
expect((await registry.getById(row.id))!.status).toBe("disabled");
|
||||
});
|
||||
|
||||
it("cancels a pending retry on shutdown", async () => {
|
||||
const { loader, startWorker } = await setup(["timeout", "ok"]);
|
||||
await loader.loadAll();
|
||||
await loader.stopActivationTimeoutRecovery();
|
||||
await new Promise(resolve => setTimeout(resolve, 200));
|
||||
expect(startWorker).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("marks a plugin error once its activation keeps timing out past the bound", async () => {
|
||||
const { loader, registry, row, startWorker, markError, isRunning } = await setup(["timeout", "timeout", "timeout"]);
|
||||
await loader.loadAll();
|
||||
await vi.waitFor(() => expect(markError).toHaveBeenCalledOnce(), { timeout: 5_000 });
|
||||
expect(startWorker).toHaveBeenCalledTimes(3);
|
||||
expect(markError).toHaveBeenCalledWith(row.id, expect.stringMatching(new RegExp(`^${PLUGIN_ACTIVATION_TIMEOUT_ERROR_PREFIX} after 2 retries`)));
|
||||
expect(isRunning()).toBe(false);
|
||||
const errored = (await registry.getById(row.id))!;
|
||||
expect(errored.status).toBe("error");
|
||||
|
||||
expect(loader.requestActivationTimeoutRecovery(errored)).toBe(true);
|
||||
await vi.waitFor(() => expect(isRunning()).toBe(true), { timeout: 5_000 });
|
||||
expect(startWorker).toHaveBeenCalledTimes(4);
|
||||
expect((await registry.getById(row.id))!.status).toBe("ready");
|
||||
await loader.shutdownAll();
|
||||
});
|
||||
|
||||
it("recovers a real worker whose first initialize stalled past the host timeout", async () => {
|
||||
const markerRoot = mkdtempSync(path.join(os.tmpdir(), "plugin-stall-marker-"));
|
||||
roots.push(markerRoot);
|
||||
const marker = path.join(markerRoot, "stalled");
|
||||
const image = bundledImageFixture({}, { workerPrelude: `import { existsSync as stalledBefore, writeFileSync as markStalled } from "node:fs";
|
||||
if (!process.argv.includes("--source") && !stalledBefore(${JSON.stringify(marker)})) { markStalled(${JSON.stringify(marker)}, "stalled"); setInterval(() => {}, 60_000); await new Promise(() => {}); }` });
|
||||
roots.push(image.root);
|
||||
const options = {
|
||||
localPluginDir: path.join(image.root, "home"), prebuiltOnly: true,
|
||||
resolvePackageRoot: image.resolver.resolve, assertPackageActivation: image.resolver.assertActivation,
|
||||
enableLocalFilesystem: false, enableNpmDiscovery: false, activationTimeoutRetryDelaysMs: [50],
|
||||
};
|
||||
await pluginLoader(db, options).installPlugin({ localPath: image.packageRoot });
|
||||
const registry = pluginRegistryService(db);
|
||||
const row = (await registry.getByKey(image.manifest.id))!;
|
||||
await registry.updateStatus(row.id, { status: "ready" });
|
||||
const manager = createPluginWorkerManager();
|
||||
const markError = vi.fn(async (pluginId: string, error: string) => registry.updateStatus(pluginId, { status: "error", lastError: error }));
|
||||
const services = {
|
||||
workerManager: manager, lifecycleManager: { markError },
|
||||
jobStore: { syncJobDeclarations: vi.fn() }, jobScheduler: { registerPlugin: vi.fn(), unregisterPlugin: vi.fn(), stop: vi.fn() },
|
||||
toolDispatcher: { unregisterPluginTools: vi.fn() },
|
||||
eventBus: { forPlugin: vi.fn(), subscriptionCount: vi.fn(() => 0), clearPlugin: vi.fn() },
|
||||
buildHostHandlers: vi.fn(() => ({})), instanceInfo: { instanceId: "fixture", hostVersion: "1.0.0" },
|
||||
} as unknown as PluginRuntimeServices;
|
||||
const loader = pluginLoader(db, options, services);
|
||||
try {
|
||||
const result = await loader.loadAll();
|
||||
expect(result.failed).toBe(1);
|
||||
expect(existsSync(marker)).toBe(true);
|
||||
await vi.waitFor(() => expect(manager.isRunning(row.id)).toBe(true), { timeout: 10_000 });
|
||||
expect(await manager.call(row.id, "health", {})).toMatchObject({ status: "ok" });
|
||||
expect(markError).not.toHaveBeenCalled();
|
||||
await vi.waitFor(async () => expect((await registry.getById(row.id))!).toMatchObject({ status: "ready", lastError: null }));
|
||||
} finally {
|
||||
await loader.shutdownAll();
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
it("keeps a deterministic activation failure a permanent error without retrying it", async () => {
|
||||
const { loader, registry, row, startWorker, markError } = await setup(["crash", "ok"]);
|
||||
await loader.loadAll();
|
||||
expect(markError).toHaveBeenCalledWith(row.id, expect.stringMatching(/^Activation failed: /));
|
||||
await new Promise(resolve => setTimeout(resolve, 100));
|
||||
expect(startWorker).toHaveBeenCalledOnce();
|
||||
const errored = (await registry.getById(row.id))!;
|
||||
expect(errored.status).toBe("error");
|
||||
expect(loader.requestActivationTimeoutRecovery(errored)).toBe(false);
|
||||
await loader.shutdownAll();
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user