Files
PaperClipAI/server/src/__tests__/plugin-idle-manager.test.ts
T
Devin FoleyandPaperclip 3d1d5294d9 Drain idle plugin workers before automatic sleep (#15599)
## Thinking Path

> - Paperclip runs autonomous work through agents, schedules, and
plugins.
> - Automatic idle sleep must preserve accepted work and unfinished
cleanup.
> - Every enabled plugin currently blocks sleep, including unused
providers.
> - An enabled flag or package version cannot prove that a worker is
idle.
> - This change adds a live worker drain under the existing owned hold.
> - Idle workers can allow sleep while unknown work stays protected.

## Linked Issues or Issue Description

Refs #15522. Related: #15391 adds plugin readiness for agent admission;
this change concerns instance sleep and does not replace that contract.

**Subsystem affected**

Plugin worker lifecycle and automatic idle sleep.

**Problem or motivation**

A workspace with no pending work cannot sleep when any plugin is
enabled. Removing that check alone would lose accepted RPCs, background
tasks, or cleanup after a caller timeout.

**Proposed solution**

Require a live `onIdleDrain` handshake from each worker. Close admission
in both processes for the exact owner and expiry. Count accepted work
until completion. Continue checking durable work separately.

## What Changed

- Add bounded worker holds, exact-owner release, automatic expiry, and
an abort signal for plugin-owned background work.
- Count host and worker requests through their real completion receipts.
Keep timed-out work counted. Check active notifications and terminal
routes.
- Accept enabled plugins only when their current workers provide
matching runtime receipts. Missing workers, old SDKs, crashes, invalid
replies, and unknown cleanup still prevent sleep.
- Let unused Daytona workers opt in. Once a worker contacts the
provider, it remains a blocker for that process lifetime. This
restriction avoids treating its existing timeout and terminal-close
behavior as a cleanup receipt.
- Document the plugin author contract. No schema or user-facing API is
added.

## Verification

- All hosted CI checks passed on 6ffb682894, including all server,
runner, workspace, build, typecheck, and end-to-end checks. One
unrelated OpenCode transport test hit a five-second timeout and passed
on rerun.
- `pnpm -r typecheck` passed.
- `pnpm build` passed.
- Focused Vitest: 293 tests passed across worker admission, real
child-process RPC, durable idle checks, and Daytona.
- Adjacent worker manager and duplex tests: 119 passed.
- Real child-process tests cover late worker completion, a host write
after caller timeout, stale owners, closed admission, and release.
- SDK tests cover hook failures, busy workers, expiry without another
request, and late acknowledgements after expiry.
- Initial long local run: 17,202 passed, 10 failed. Five failures
involved parent-directory skill fixtures; four of those were already
reproduced and passed in an isolated worktree. One HTTP test had a
socket hang-up; all 66 tests in that file passed on rerun. The other
four used new tests with runtime modules loaded before the follow-up
edits; all pass in the fresh focused run above. A clean build and
full-suite rerun at the final commit are running in an isolated
worktree. No full-suite pass is claimed.
- Follow-up: 44 focused tests passed, including repeated owned holds
without an intervening normal request. Worker status changes now
invalidate an in-progress scan.
- Apex follow-up: 13 idle tests, 119 adjacent worker tests, 71
instance-settings route tests, and server typecheck passed. A real
queued notification completes its write while preparation is in flight;
synchronous and async session callback failures are logged.
- Diff whitespace and added-line secret/private-reference scans passed.

## Risks

- Plugins are trusted code. A hook must account for work outside SDK
RPCs and keep it quiescent until its signal aborts. There is no
package-name or manifest-only exemption.
- This first change targets unused workers. Used Daytona workers remain
awake until remote cleanup receipts are complete. Restarting does not
bypass durable lease and recovery checks.
- Unknown completion remains a blocker, including after a worker crash.
This can retain cost but cannot authorize sleep from a timeout alone.
- Enabled jobs, schedules, credentials, integrations, and other durable
work remain blockers. Durable wake ownership is separate follow-up work.
- Rollback restores the blanket plugin blocker. No data migration is
required.

## Model Used

OpenAI Codex, GPT-6. Used repository inspection, reasoning, code
editing, and command execution. The runtime did not report an exact
model revision or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing public issues or described the issue
in this PR
- [x] I have not referenced internal or instance-local issues or links
- [x] My branch name describes the change and contains no internal
identifiers
- [x] Focused local tests pass; full CI is green (long local rerun
status is recorded above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile Apex is 5/5 on 6ffb682894 with no open findings
- [x] I will address all review comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-08 12:57:32 -07:00

99 lines
5.0 KiB
TypeScript

import { fileURLToPath } from "node:url";
import { createRequire } from "node:module";
import { setTimeout as delay } from "node:timers/promises";
import { afterEach, describe, expect, it } from "vitest";
import { createPluginWorkerHandle } from "../services/plugin-worker-manager.js";
import { startTaskDrain, stopTaskDrain } from "../services/task-admission.js";
const manifest = { id: "test.idle", apiVersion: 1 as const, version: "1.0.0", displayName: "Idle fixture",
description: "Idle fixture", author: "Paperclip", categories: ["automation" as const], capabilities: [], entrypoints: { worker: "worker.cjs" } };
function worker(write = false, hostHandler = async () => undefined) {
return createPluginWorkerHandle("test.idle", {
entrypointPath: fileURLToPath(new URL("./fixtures/plugin-worker-idle.cjs", import.meta.url)),
execArgv: ["--import", createRequire(import.meta.url).resolve("tsx")],
manifest, config: {}, instanceInfo: { instanceId: "fixture", hostVersion: "1.0.0" }, apiVersion: 1,
env: { IDLE_TEST_WRITE: write ? "1" : "0" },
hostHandlers: { "state.set": hostHandler, "events.subscribe": async () => ({}) },
});
}
function hold() {
const lease = startTaskDrain({ purpose: "idle", ttlMs: 30_000 });
return { ownerId: lease.ownerId!, expiresAt: lease.expiresAt!.getTime() };
}
describe("plugin manager idle receipts", () => {
afterEach(() => stopTaskDrain());
it("retains timed-out worker operations and admits work after the owned hold releases", async () => {
const handle = worker();
await handle.start();
try {
await expect(handle.call("environmentProbe", { driverKey: "fixture", companyId: "fixture", environmentId: "fixture", config: { delayMs: 200 } }, 10)).rejects.toThrow("timed out");
const lease = hold();
expect(await handle.prepareIdleSleep!(lease)).toBe("present");
await expect.poll(() => handle.prepareIdleSleep!(lease)).toBe("none");
await expect(handle.call("health", {})).rejects.toThrow("held");
expect(await handle.prepareIdleSleep!({ ...lease, ownerId: "stale" })).toBe("unknown");
stopTaskDrain(); handle.releaseIdleSleep!();
expect(await handle.call("health", {})).toMatchObject({ status: "ok" });
expect(await handle.prepareIdleSleep!(hold())).toBe("none");
stopTaskDrain();
// No intervening request or explicit worker release: the next controller
// attempt must discard the worker manager's previous ownership record.
expect(await handle.prepareIdleSleep!(hold())).toBe("none");
} finally { stopTaskDrain(); await handle.stop(); }
});
it("keeps a host write counted after the caller stops waiting", async () => {
let finish!: () => void;
const handle = worker(true, () => new Promise<void>((resolve) => { finish = resolve; }));
await handle.start();
try {
await expect(handle.call("health", {}, 30)).rejects.toThrow("timed out");
const lease = hold();
expect(await handle.prepareIdleSleep!(lease)).toBe("present");
finish();
await expect.poll(() => handle.prepareIdleSleep!(lease)).toBe("none");
} finally { stopTaskDrain(); await handle.stop(); }
});
it("lets a queued notification finish its write while idle preparation is in flight", async () => {
let finish!: () => void;
let writes = 0;
const handle = worker(false, () => new Promise<void>((resolve) => {
writes++;
finish = resolve;
}));
await handle.start();
try {
// Finish setup and its subscription receipt before queuing the event.
await handle.call("health", {});
handle.notify("onEvent", { event: { eventType: "fixture.write" } });
// No await: the host has set its hold before it can read the event's
// worker-to-host write. The worker must still be allowed to complete it.
const lease = hold();
const preparation = handle.prepareIdleSleep!(lease);
expect(await preparation).not.toBe("none");
await expect.poll(() => writes).toBe(1);
expect(await handle.prepareIdleSleep!(lease)).toBe("present");
finish();
await expect.poll(() => handle.prepareIdleSleep!(lease)).toBe("none");
} finally { finish?.(); stopTaskDrain(); await handle.stop(); }
});
it("resumes crash recovery after an idle hold without discarding the restart", async () => {
const handle = worker();
await handle.start();
handle.on("crash", () => { hold(); });
try {
await expect(handle.call("environmentProbe", { driverKey: "fixture", companyId: "fixture", environmentId: "fixture", config: { crash: true } })).rejects.toThrow();
await delay(1_600);
expect(handle.status).toBe("backoff");
expect(handle.diagnostics().nextRestartAt).not.toBeNull();
stopTaskDrain();
await expect.poll(() => handle.status, { timeout: 5_000 }).toBe("running");
// Recovery restores service, but lost completion receipts remain unsafe.
expect(await handle.prepareIdleSleep!(hold())).toBe("unknown");
} finally { stopTaskDrain(); await handle.stop(); }
});
});