Files
Devin FoleyandPaperclip 4415665d67 fix(server): allow proved idle waits and verify historical cleanup (#16633)
## Thinking Path

> - Paperclip manages agents, task continuations, integrations, and
their execution resources.
> - Automatic idle sleep needs durable evidence that accepted work and
provider cleanup are settled.
> - Response-driven questions, passive board escalations, and completed
connection receipts currently look like unfinished work.
> - Legacy ephemeral leases and terminal image sessions need actual
deletion proof before their history can become idle.
> - This PR adds narrow evidence-based exceptions and read-only provider
verification while preserving queues, outcomes, and ownership fences.
> - Idle instances can sleep without losing responses, automatic
reconciliation, or cleanup obligations.

## Linked Issues or Issue Description

**What happened?**

Instances with no executable work can remain awake because pending human
responses, passive recovery records, outbound integration configuration,
or completed delivery history are counted as background work. Older
failed ephemeral leases and finished setup sessions also lack the
evidence needed to distinguish retained resources from deleted
allocations.

**Expected behavior**

Permit idle sleep only for proved response-driven waits and settled
history. Continue blocking real queued work, inbound integrations
without a wake contract, unfinished accounting/retries, retained
resources, and uncertain provider cleanup.

**Steps to reproduce**

Finish an issue's producing run, leave a pending question or
confirmation, and request an owned scheduled idle report. Separately
retain a processed connection receipt or a failed ephemeral lease whose
provider allocation has already been deleted. The old checks keep those
instances awake even after executable work has finished.

Related public groundwork: #15522, #15599, #15761, #15897, #15907, and
#16325. Searched existing public idle-sleep PRs; this addresses
additional blockers in the current predicates.

## What Changed

- Recognize supported response-driven waits and passive board recoveries
without changing task state, response audiences, or continuation
policies. Bind passive issue generation into routine-only alarms too.
Keep automatically polled merge-confirmation cards awake.
- Treat processed events, delivered intents, disabled drafts, and
outbound MCP configuration as history/configuration where proved safe.
Preserve pending deliveries, downstream wake requests, and inbound
GitHub bindings.
- Permit only pristine Slack setup drafts with a matching disabled
channel connection, no provisioning registration, and no prior provider
identity/activity. A saved slash-command name remains inert
configuration.
- Repair missed routine execution-issue status hooks in bounded
awake-only maintenance. Lock and re-read exact owners, apply existing
completion/transient-failure rules, and preserve normal continuation
after an issue resumes.
- Allow human-only connection setup to wait only when no reusable active
connection of its purpose exists; available access reblocks sleep for
reconciliation. The delivery worker shares this proof, releases its
local idle hold, and retains maintenance only while already awake.
Passive-only ticks pause before admission during an owned drain,
allowing slow backups to finish; new producer notifications still
reconcile and invalidate the backup.
- Add optional `environmentInspectLease` SDK discovery/dispatch and
read-only Daytona/fake-provider implementations. Old workers remain
conservative. Daytona status inspection uses a fresh provider GET
without reconnecting or poisoning unused-worker eligibility.
- Bind historical credential reads to an unrotated initial secret
version and its immutable resolved-value hash; reject ambiguous
histories and ambient Daytona credential/API-endpoint fallbacks before
querying provider state.
- Verify bounded rotating batches of historical resources on
already-awake Cloud instances, outside readiness. Store host-owned
deletion evidence bound to exact row identity/version and reject
concurrent ownership changes. Preserve failure outcomes, cleanup/export
intents, and run continuation authority.
- Preserve cancellation provenance, defer contended routine rows
individually, and cap actual inspection RPCs across caller timeouts and
worker exits. Add database, SDK, provider, configuration-scope, and race
regression coverage and update idle protocol documentation.

## Verification

- `pnpm -r typecheck` and `pnpm build` passed on `24fd1c0f5ec`; server
typecheck/build passed again after the final coordinator-only fix in
`60f23e6e4b6`.
- Final server regressions: 258 sleep-safety/connection cases and all 17
coordinator cases passed. Earlier focused checks also covered 331
combined idle/cleanup cases, 34 custom-image cases, 39
credential/cleanup cases, 39 connection-history cases, and 80 existing
routine cases. Counts overlap and are not additive.
- SDK: 153 passed. Daytona: 406 passed with 14 existing opt-in skips.
The provider implementation also typechecks against the locked SDK used
by the Cloud wrapper.
- Dependency boundaries were exercised with real child workers holding
provider inspection replies beyond caller timeouts, a real PostgreSQL
row lock delaying one routine owner, and real connection creation →
queue notification → worker hold release → resumption after access is
enabled without another notification. A controlled six-second backup
goes through the real coordinator, task drain, work signals, and
idle-safety generation checks; maintenance permits completion, while a
new committed transaction invalidates it.
- Full local `pnpm test:run` completed its general-server phase with
21,122 passing, 29 failing, and 109 skipped tests; later phases did not
run. That long run spanned source edits and is not a final-revision
pass. The affected Slack draft assertion passes on current source.
Restoring `/usr/sbin` for `lsof` produced 233 passing tests and two
Git-fixture failures across the other five files (3 existing skips).
Both remaining assertions pass with command-scoped
`init.defaultBranch=master`; this host defaults to main and leaves both
branches in their fixture. No source/global Git setting changed for that
comparison. Original failure logs are retained.
- A parallel typecheck/build attempt raced generated Runner outputs and
failed an import. Running the build sequentially passed with no source
change.
- Local database checks use the maintained test-constructor seam with
native PostgreSQL 16.14 because bundled PostgreSQL 18 is unreliable on
this host. No database assertions were skipped in the focused runs. An
interrupted combined run was superseded by the separate passing
coordinator and safety/connection runs; no owned test database processes
remain running.
- Apex reviewed final head `60f23e6e4b615a08591e19d86443a6c35c091ce1`:
**5/5**, zero unresolved threads. All current-head CI gates are green,
with expected policy skips. The branch is conflict-free and GitHub
reports a clean merge state.
- Current-head CI initially failed `plan-task-fixture.spec.ts:66` during
fixture deletion: the final company DELETE hit
`activity_log_company_id_companies_id_fk` after the feature assertions
completed. The deletion path is unchanged here; the final diff only
changes drain-gated passive maintenance. The preceding revision passed
this shard. Failure evidence is retained in [the original CI
job](https://github.com/paperclipai/paperclip/actions/runs/39006936642/job/117321244396);
the targeted retry passed on the unchanged commit ([retry
job](https://github.com/paperclipai/paperclip/actions/runs/39006936642/job/117322535013)).
This shows recovery, not that the underlying company-deletion race is
resolved.
- These checks do not claim staging/production deployment or live
provider cleanup proof. Provider tests simulate responses. Full Cloud
staging verification requires a compatible Core/private-plugin
composition after the Core revision is on master.

## Risks

Sleep eligibility changes require the existing owned drain, final
generation validation, local-work checks, and plugin drain. Unsupported
states fail closed. Queued agent input and routine execution remain
blocked. Outbound MCP exemptions exclude inbound GitHub grants; other
unsupported inbound integrations remain awake.

The inspection RPC is optional and requires a matching live worker
method. Its actual outstanding calls are capped at two across workers
even after the caller times out. Historical lease inspection requires
its recorded plugin/config; setup inspection requires an unchanged
recorded config fingerprint. Not-found under replacement account
credentials is not historical deletion proof: ambiguous/rotated secret
histories defer verification, and initial-version reads must match their
recorded value hashes. Missing provenance (including legacy image
sessions without a recorded config fingerprint), live/stopped resources,
provider authorization failures, and concurrent changes defer
verification. Routine repair replays existing
completion/transient-failure behavior and timestamps at reconciliation
time; it does not execute or discard deferred comments. No migration or
destructive provider operation is added. The new marker does not grant
run-resume authority. Full Cloud staging validation requires a
compatible Core/private-plugin composition; no production deployment is
included.

## Model Used

OpenAI GPT-6 Codex, with repository/tool access and code execution.
Exact model identifier and context-window size are not exposed by this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run relevant tests locally; passing focused checks and
full-run limitations are documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-17 07:01:32 -07:00

1232 lines
44 KiB
TypeScript

import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { createInterface } from "node:readline";
import { PassThrough } from "node:stream";
import { pathToFileURL } from "node:url";
import { afterEach, describe, expect, it } from "vitest";
import { definePlugin } from "../src/define-plugin.js";
import {
createRequest,
createErrorResponse,
createSuccessResponse,
isJsonRpcRequest,
isJsonRpcResponse,
isJsonRpcNotification,
parseMessage,
PLUGIN_RPC_ERROR_CODES,
serializeMessage,
type JsonRpcNotification,
type JsonRpcResponse,
type PluginInvocationContext,
} from "../src/protocol.js";
import { isWorkerEntrypoint, startWorkerRpcHost } from "../src/worker-rpc-host.js";
describe("isWorkerEntrypoint", () => {
const tempRoots: string[] = [];
afterEach(() => {
for (const tempRoot of tempRoots.splice(0)) {
fs.rmSync(tempRoot, { recursive: true, force: true });
}
});
function createTempRoot(): string {
const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-sdk-worker-"));
tempRoots.push(tempRoot);
return tempRoot;
}
it("matches an entrypoint reached through a symlinked directory", () => {
const tempRoot = createTempRoot();
const realDir = path.join(tempRoot, "real");
const linkDir = path.join(tempRoot, "link");
fs.mkdirSync(realDir);
fs.symlinkSync(realDir, linkDir, "dir");
const workerPath = path.join(realDir, "worker.js");
fs.writeFileSync(workerPath, "");
expect(
isWorkerEntrypoint(
path.join(linkDir, "worker.js"),
pathToFileURL(workerPath).toString(),
),
).toBe(true);
});
it("does not match a different entrypoint", () => {
const tempRoot = createTempRoot();
const workerPath = path.join(tempRoot, "worker.js");
const otherPath = path.join(tempRoot, "other.js");
fs.writeFileSync(workerPath, "");
fs.writeFileSync(otherPath, "");
expect(
isWorkerEntrypoint(
otherPath,
pathToFileURL(workerPath).toString(),
),
).toBe(false);
});
});
describe("worker resource lifecycle delivery", () => {
it("advertises and dispatches the native lifecycle callback", async () => {
const stdin = new PassThrough(), stdout = new PassThrough();
const lines = createInterface({ input: stdout });
const plugin = definePlugin({
async setup() {},
async onResourceLifecycle(event) { return { eventId: event.id, status: "retry" }; },
});
const worker = startWorkerRpcHost({ plugin, stdin, stdout });
async function call(method: string, params: unknown) {
const response = new Promise<JsonRpcResponse>(resolve => {
const handler = (line: string) => {
const message = parseMessage(line);
if (isJsonRpcResponse(message)) { lines.off("line", handler); resolve(message); }
};
lines.on("line", handler);
});
stdin.write(serializeMessage(createRequest(method, params, method)));
return response;
}
try {
const initialized = await call("initialize", {
manifest: { id: "test.lifecycle", apiVersion: 1, version: "1.0.0", displayName: "Lifecycle", description: "Test", author: "Tests", categories: ["automation"], capabilities: ["events.subscribe"], resourceLifecycle: ["project"], entrypoints: { worker: "worker.js" } },
config: {}, databaseNamespace: null,
});
expect(initialized).toMatchObject({ result: { supportedMethods: expect.arrayContaining(["resourceLifecycle"]) } });
expect(await call("resourceLifecycle", { id: "17", companyId: "company", resourceType: "project", resourceId: "project", action: "delete", createdAt: "2020-01-01T00:00:00Z" }))
.toMatchObject({ result: { eventId: "17", status: "retry" } });
} finally { worker.stop(); lines.close(); stdin.destroy(); stdout.destroy(); }
});
});
describe("worker performAction context", () => {
it("does not derive context companyId from caller params without host actor context", async () => {
const hostToWorker = new PassThrough();
const workerToHost = new PassThrough();
const hostReadline = createInterface({ input: workerToHost });
const pending = new Map<string, (response: JsonRpcResponse) => void>();
let nextRequestId = 1;
const plugin = definePlugin({
async setup(ctx) {
ctx.actions.register("inspect", async (params, context) => ({
paramsCompanyId: params.companyId,
actor: context.actor,
companyId: context.companyId,
}));
},
});
const worker = startWorkerRpcHost({
plugin,
stdin: hostToWorker,
stdout: workerToHost,
});
function callWorker(method: string, params: unknown) {
const id = `host-${nextRequestId++}`;
const result = new Promise<unknown>((resolve, reject) => {
pending.set(id, (response) => {
if ("error" in response && response.error) {
reject(new Error(response.error.message));
return;
}
resolve((response as { result?: unknown }).result);
});
});
hostToWorker.write(serializeMessage(createRequest(method, params, id)));
return result;
}
hostReadline.on("line", (line) => {
const message = parseMessage(line);
if (!isJsonRpcResponse(message)) return;
pending.get(String(message.id))?.(message);
pending.delete(String(message.id));
});
try {
await expect(callWorker("initialize", {
manifest: {
id: "paperclip.test-worker-context",
apiVersion: 1,
version: "1.0.0",
displayName: "Worker Context Test",
description: "Test plugin",
author: "Paperclip",
categories: ["automation"],
capabilities: [],
entrypoints: {},
},
config: {},
databaseNamespace: null,
})).resolves.toMatchObject({ ok: true });
await expect(callWorker("performAction", {
key: "inspect",
params: { companyId: "spoofed-company" },
})).resolves.toEqual({
paramsCompanyId: "spoofed-company",
actor: {
type: "system",
userId: null,
agentId: null,
runId: null,
companyId: null,
},
companyId: null,
});
} finally {
worker.stop();
hostReadline.close();
hostToWorker.destroy();
workerToHost.destroy();
}
});
});
describe("worker invocation scope propagation", () => {
it("keeps overlapping company scopes local to each getData invocation", async () => {
const hostToWorker = new PassThrough();
const workerToHost = new PassThrough();
const hostReadline = createInterface({ input: workerToHost });
const pending = new Map<string, (response: JsonRpcResponse) => void>();
const nestedInvocationIds: string[] = [];
const invocationCompanies = new Map([
["invocation-a", "company-a"],
["invocation-b", "company-b"],
]);
let releaseCompanyA: (() => void) | null = null;
let nextRequestId = 1;
const plugin = definePlugin({
async setup(ctx) {
ctx.data.register("probe", async (params) => {
if (params.label === "a") {
await new Promise<void>((resolve) => {
releaseCompanyA = resolve;
});
}
const company = await ctx.companies.get(String(params.requestedCompanyId));
return { label: params.label, company };
});
},
});
const worker = startWorkerRpcHost({
plugin,
stdin: hostToWorker,
stdout: workerToHost,
});
function callWorker(method: string, params: unknown, invocation?: PluginInvocationContext) {
const id = `host-${nextRequestId++}`;
const request = {
...createRequest(method, params, id),
...(invocation ? { paperclipInvocation: invocation } : {}),
};
const result = new Promise<unknown>((resolve, reject) => {
pending.set(id, (response) => {
if ("error" in response && response.error) {
reject(new Error(response.error.message));
return;
}
resolve((response as { result?: unknown }).result);
});
});
hostToWorker.write(serializeMessage(request));
return result;
}
hostReadline.on("line", (line) => {
const message = parseMessage(line);
if (isJsonRpcResponse(message)) {
pending.get(String(message.id))?.(message);
pending.delete(String(message.id));
return;
}
if (!isJsonRpcRequest(message)) return;
if (message.method !== "companies.get") return;
const invocationId = (message as { paperclipInvocationId?: string }).paperclipInvocationId ?? "";
const requestedCompanyId = (message.params as { companyId?: string }).companyId;
const allowedCompanyId = invocationCompanies.get(invocationId);
nestedInvocationIds.push(invocationId);
if (requestedCompanyId !== allowedCompanyId) {
hostToWorker.write(serializeMessage(createErrorResponse(
message.id,
PLUGIN_RPC_ERROR_CODES.CAPABILITY_DENIED,
`requested company "${requestedCompanyId}" but invocation "${invocationId}" is scoped to "${allowedCompanyId}"`,
)));
return;
}
hostToWorker.write(serializeMessage(createSuccessResponse(message.id, {
id: requestedCompanyId,
})));
if (invocationId === "invocation-b") {
releaseCompanyA?.();
}
});
try {
await callWorker("initialize", {
manifest: {
id: "paperclip.scope-test",
apiVersion: 1,
version: "1.0.0",
displayName: "Scope test",
description: "Scope test",
author: "Paperclip",
categories: ["automation"],
capabilities: ["companies.read"],
entrypoints: { worker: "dist/worker.js" },
},
config: {},
instanceInfo: { instanceId: "test", hostVersion: "0.0.0" },
apiVersion: 1,
});
const companyARequest = callWorker(
"getData",
{
key: "probe",
companyId: "company-a",
params: { label: "a", requestedCompanyId: "company-b" },
},
{ id: "invocation-a", scope: { companyId: "company-a" } },
);
const companyAExpectation = expect(companyARequest).rejects.toThrow(
/requested company "company-b"/,
);
const companyBRequest = callWorker(
"getData",
{
key: "probe",
companyId: "company-b",
params: { label: "b", requestedCompanyId: "company-b" },
},
{ id: "invocation-b", scope: { companyId: "company-b" } },
);
await expect(companyBRequest).resolves.toEqual({
label: "b",
company: { id: "company-b" },
});
await companyAExpectation;
expect(nestedInvocationIds).toEqual(["invocation-b", "invocation-a"]);
} finally {
worker.stop();
hostReadline.close();
hostToWorker.destroy();
workerToHost.destroy();
}
});
});
describe("worker configChanged cross-tenant guard", () => {
// Spin up a worker-rpc-host wired to in-memory streams and expose a
// request/response `callWorker` plus `initialize`/`stop` helpers.
function makeWorker(plugin: ReturnType<typeof definePlugin>) {
const hostToWorker = new PassThrough();
const workerToHost = new PassThrough();
const hostReadline = createInterface({ input: workerToHost });
const pending = new Map<string, (response: JsonRpcResponse) => void>();
let nextRequestId = 1;
const worker = startWorkerRpcHost({
plugin,
stdin: hostToWorker,
stdout: workerToHost,
});
function callWorker(method: string, params: unknown) {
const id = `host-${nextRequestId++}`;
const result = new Promise<unknown>((resolve, reject) => {
pending.set(id, (response) => {
if ("error" in response && response.error) {
reject(
Object.assign(new Error(response.error.message), {
code: response.error.code,
}),
);
return;
}
resolve((response as { result?: unknown }).result);
});
});
hostToWorker.write(serializeMessage(createRequest(method, params, id)));
return result;
}
hostReadline.on("line", (line) => {
const message = parseMessage(line);
if (!isJsonRpcResponse(message)) return;
pending.get(String(message.id))?.(message);
pending.delete(String(message.id));
});
async function initialize() {
return await callWorker("initialize", {
manifest: {
id: "paperclip.config-guard-test",
apiVersion: 1,
version: "1.0.0",
displayName: "Config Guard Test",
description: "Test plugin",
author: "Paperclip",
categories: ["automation"],
capabilities: [],
entrypoints: {},
},
config: {},
databaseNamespace: null,
});
}
function stop() {
worker.stop();
hostReadline.close();
hostToWorker.destroy();
workerToHost.destroy();
}
return { callWorker, initialize, stop };
}
it.each([false, true])("advertises and dispatches stop-only only with an explicit hook: %s", async supported => {
let releases = 0, stops = 0;
const worker = makeWorker(definePlugin({ async setup() {},
async onEnvironmentReleaseLease() { releases++; return { providerLeaseId: "allocation", state: "destroyed" }; },
...(supported ? { async onEnvironmentStopLease() { stops++; return { providerLeaseId: "allocation", state: "stopped" as const }; } } : {}),
}));
try {
const initialized = await worker.initialize() as { supportedMethods: string[] };
expect(initialized.supportedMethods.includes("environmentStopLease")).toBe(supported);
const stopped = worker.callWorker("environmentStopLease", { driverKey: "fixture", companyId: "company", environmentId: "environment", providerLeaseId: "allocation", config: {} });
if (supported) await expect(stopped).resolves.toEqual({ providerLeaseId: "allocation", state: "stopped" });
else await expect(stopped).rejects.toThrow();
expect(stops).toBe(supported ? 1 : 0);
expect(releases).toBe(0);
} finally { worker.stop(); }
});
it.each([false, true])("advertises read-only inspection only when implemented: %s", async supported => {
let inspections = 0, mutations = 0;
const worker = makeWorker(definePlugin({ async setup() {},
async onEnvironmentReleaseLease() { mutations++; },
async onEnvironmentDestroyLease() { mutations++; },
...(supported ? { async onEnvironmentInspectLease() { inspections++; return { providerLeaseId: "allocation", state: "destroyed" as const }; } } : {}),
}));
try {
const initialized = await worker.initialize() as { supportedMethods: string[] };
expect(initialized.supportedMethods.includes("environmentInspectLease")).toBe(supported);
const result = worker.callWorker("environmentInspectLease", { driverKey: "fixture", companyId: "company", environmentId: "environment", providerLeaseId: "allocation", config: {} });
if (supported) await expect(result).resolves.toEqual({ providerLeaseId: "allocation", state: "destroyed" });
else await expect(result).rejects.toThrow();
expect(inspections).toBe(supported ? 1 : 0);
expect(mutations).toBe(0);
} finally { worker.stop(); }
});
it("fails closed when a second, distinct company's config would overwrite a single-tenant worker", async () => {
const applied: Array<{ companyId: string | null; token: unknown }> = [];
const plugin = definePlugin({
async setup() {},
async onConfigChanged(newConfig, context) {
applied.push({
companyId: context?.companyId ?? null,
token: newConfig.slackBotToken,
});
},
});
const { callWorker, initialize, stop } = makeWorker(plugin);
try {
await initialize();
// Company A's config is delivered first (deterministic ORDER BY companyId
// in the loader) and applied.
await expect(
callWorker("configChanged", {
config: { companyId: "company-a", slackBotToken: "xoxb-A" },
companyId: "company-a",
}),
).resolves.toBeNull();
// Company B's *distinct* config must be rejected rather than silently
// collapsing the single worker onto B's bot token (the vulnerability).
await expect(
callWorker("configChanged", {
config: { companyId: "company-b", slackBotToken: "xoxb-B" },
companyId: "company-b",
}),
).rejects.toMatchObject({
code: PLUGIN_RPC_ERROR_CODES.CROSS_TENANT_CONFIG,
});
// The worker stayed bound to company A; company B never reached the
// plugin. Against the pre-fix code this array would be
// [company-a, company-b] (last-write-wins collapse).
expect(applied).toEqual([{ companyId: "company-a", token: "xoxb-A" }]);
} finally {
stop();
}
});
it("allows an idempotent replay of the same config under a different scope row", async () => {
// Mirrors the live single-tenant gateway: several plugin_config rows keyed
// by distinct row companyIds but all embedding the same config. Replaying
// them must be a no-op, not a fail-closed rejection.
const appliedScopes: Array<string | null> = [];
const plugin = definePlugin({
async setup() {},
async onConfigChanged(_newConfig, context) {
appliedScopes.push(context?.companyId ?? null);
},
});
const { callWorker, initialize, stop } = makeWorker(plugin);
try {
await initialize();
const embedded = { companyId: "company-a", slackBotToken: "xoxb-A" };
await callWorker("configChanged", {
config: { ...embedded },
companyId: "row-scope-1",
});
await expect(
callWorker("configChanged", {
config: { ...embedded },
companyId: "row-scope-2",
}),
).resolves.toBeNull();
expect(appliedScopes).toEqual(["row-scope-1", "row-scope-2"]);
} finally {
stop();
}
});
it("threads per-company config to a plugin that opts into multiCompanyConfig", async () => {
const applied: Array<{ companyId: string | null; token: unknown }> = [];
const plugin = definePlugin({
multiCompanyConfig: true,
async setup() {},
async onConfigChanged(newConfig, context) {
applied.push({
companyId: context?.companyId ?? null,
token: newConfig.slackBotToken,
});
},
});
const { callWorker, initialize, stop } = makeWorker(plugin);
try {
await initialize();
await callWorker("configChanged", {
config: { companyId: "company-a", slackBotToken: "xoxb-A" },
companyId: "company-a",
});
await expect(
callWorker("configChanged", {
config: { companyId: "company-b", slackBotToken: "xoxb-B" },
companyId: "company-b",
}),
).resolves.toBeNull();
// Both companies' configs delivered, each tagged with its own scope.
expect(applied).toEqual([
{ companyId: "company-a", token: "xoxb-A" },
{ companyId: "company-b", token: "xoxb-B" },
]);
} finally {
stop();
}
});
});
describe("worker provider tracer", () => {
it("default plugin tracer is a no-op that starts and ends a span without throwing", async () => {
const { NOOP_PLUGIN_TRACER } = await import("../src/types.js");
const span = NOOP_PLUGIN_TRACER.startSpan("pack", { attributes: { a: 1 } });
expect(() => {
span.setAttribute("b", 2);
span.setStatus({ code: 1 });
span.end();
}).not.toThrow();
});
// Drive a plugin data handler that opens a provider span, and capture the
// worker→host traffic. The host injects a `traceparent` on the invocation, so
// the worker must emit one `span.record` request that echoes the invocation id
// and carries the span name and attributes.
async function runSpanProbe(invocation: PluginInvocationContext) {
const hostToWorker = new PassThrough();
const workerToHost = new PassThrough();
const hostReadline = createInterface({ input: workerToHost });
const pending = new Map<string, (response: JsonRpcResponse) => void>();
const spanRecords: Array<{ params: unknown; invocationId?: string }> = [];
let nextRequestId = 1;
const plugin = definePlugin({
async setup(ctx) {
ctx.data.register("probe", async () => {
const span = ctx.tracer.startSpan("pack", {
attributes: { "paperclip.sandbox.startup.pack.wall_ms": 12 },
});
span.setAttribute("paperclip.sandbox.startup.provider", "daytona");
span.end();
return { ok: true };
});
},
});
const worker = startWorkerRpcHost({ plugin, stdin: hostToWorker, stdout: workerToHost });
function callWorker(method: string, params: unknown, inv?: PluginInvocationContext) {
const id = `host-${nextRequestId++}`;
const request = {
...createRequest(method, params, id),
...(inv ? { paperclipInvocation: inv } : {}),
};
const result = new Promise<unknown>((resolve, reject) => {
pending.set(id, (response) => {
if ("error" in response && response.error) {
reject(new Error(response.error.message));
return;
}
resolve((response as { result?: unknown }).result);
});
});
hostToWorker.write(serializeMessage(request));
return result;
}
hostReadline.on("line", (line) => {
const message = parseMessage(line);
if (isJsonRpcResponse(message)) {
pending.get(String(message.id))?.(message);
pending.delete(String(message.id));
return;
}
if (!isJsonRpcRequest(message)) return;
if (message.method === "span.record") {
spanRecords.push({
params: message.params,
invocationId: (message as { paperclipInvocationId?: string }).paperclipInvocationId,
});
hostToWorker.write(serializeMessage(createSuccessResponse(message.id, null)));
}
});
try {
await callWorker("initialize", {
manifest: {
id: "paperclip.tracer-test",
apiVersion: 1,
version: "1.0.0",
displayName: "Tracer test",
description: "Tracer test",
author: "Paperclip",
categories: ["automation"],
capabilities: ["environment.drivers.register"],
entrypoints: { worker: "dist/worker.js" },
},
config: {},
instanceInfo: { instanceId: "test", hostVersion: "0.0.0" },
apiVersion: 1,
});
await callWorker("getData", { key: "probe", companyId: "company-a", params: {} }, invocation);
// Let the fire-and-forget span.record flush.
await new Promise((resolve) => setTimeout(resolve, 20));
return spanRecords;
} finally {
worker.stop();
hostReadline.close();
hostToWorker.destroy();
workerToHost.destroy();
}
}
it("emits one span.record with the name and attributes when a host trace context is active", async () => {
const spanRecords = await runSpanProbe({
id: "invocation-a",
scope: { companyId: "company-a" },
traceparent: "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01",
});
expect(spanRecords).toHaveLength(1);
const record = spanRecords[0]!;
expect(record.invocationId).toBe("invocation-a");
expect(record.params).toMatchObject({
name: "pack",
attributes: {
"paperclip.sandbox.startup.pack.wall_ms": 12,
"paperclip.sandbox.startup.provider": "daytona",
},
});
});
it("sends a finite startTimeMs and endTimeMs with endTimeMs >= startTimeMs", async () => {
const spanRecords = await runSpanProbe({
id: "invocation-a",
scope: { companyId: "company-a" },
traceparent: "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01",
});
expect(spanRecords).toHaveLength(1);
const params = spanRecords[0]!.params as {
startTimeMs?: number;
endTimeMs?: number;
};
expect(Number.isFinite(params.startTimeMs)).toBe(true);
expect(Number.isFinite(params.endTimeMs)).toBe(true);
expect(params.endTimeMs!).toBeGreaterThanOrEqual(params.startTimeMs!);
});
it("emits no span.record when the invocation carries no traceparent (tracing off)", async () => {
const spanRecords = await runSpanProbe({
id: "invocation-a",
scope: { companyId: "company-a" },
});
expect(spanRecords).toHaveLength(0);
});
});
describe("worker execute.log emitter", () => {
// Run one data handler that calls `ctx.execution.log`, and capture the
// `execute.log` notifications the worker sends to the host.
async function runExecuteLogProbe(
invocation: PluginInvocationContext | undefined,
entries: Array<{ stream: "stdout" | "stderr"; chunk: string }>,
) {
const hostToWorker = new PassThrough();
const workerToHost = new PassThrough();
const hostReadline = createInterface({ input: workerToHost });
const pending = new Map<string, (response: JsonRpcResponse) => void>();
const logRecords: Array<{ params: unknown; invocationId?: string }> = [];
let nextRequestId = 1;
const plugin = definePlugin({
async setup(ctx) {
ctx.data.register("emit-logs", async () => {
for (const entry of entries) {
ctx.execution.log(entry.stream, entry.chunk);
}
return { ok: true };
});
},
});
const worker = startWorkerRpcHost({ plugin, stdin: hostToWorker, stdout: workerToHost });
function callWorker(method: string, params: unknown, inv?: PluginInvocationContext) {
const id = `host-${nextRequestId++}`;
const request = {
...createRequest(method, params, id),
...(inv ? { paperclipInvocation: inv } : {}),
};
const result = new Promise<unknown>((resolve, reject) => {
pending.set(id, (response) => {
if ("error" in response && response.error) {
reject(new Error(response.error.message));
return;
}
resolve((response as { result?: unknown }).result);
});
});
hostToWorker.write(serializeMessage(request));
return result;
}
hostReadline.on("line", (line) => {
const message = parseMessage(line);
if (isJsonRpcResponse(message)) {
pending.get(String(message.id))?.(message);
pending.delete(String(message.id));
return;
}
// `execute.log` is a fire-and-forget notification (no id), so it is not a
// JSON-RPC request. Match on the method name directly.
if ((message as { method?: string }).method === "execute.log") {
logRecords.push({
params: (message as { params?: unknown }).params,
invocationId: (message as { paperclipInvocationId?: string }).paperclipInvocationId,
});
}
});
try {
await callWorker("initialize", {
manifest: {
id: "paperclip.execute-log-test",
apiVersion: 1,
version: "1.0.0",
displayName: "Execute log test",
description: "Execute log test",
author: "Paperclip",
categories: ["automation"],
capabilities: ["environment.drivers.register"],
entrypoints: { worker: "dist/worker.js" },
},
config: {},
instanceInfo: { instanceId: "test", hostVersion: "0.0.0" },
apiVersion: 1,
});
await callWorker("getData", { key: "emit-logs", companyId: "company-a", params: {} }, invocation);
// Let the fire-and-forget execute.log notifications flush.
await new Promise((resolve) => setTimeout(resolve, 20));
return logRecords;
} finally {
worker.stop();
hostReadline.close();
hostToWorker.destroy();
workerToHost.destroy();
}
}
it("stamps the active invocation id on each execute.log notification", async () => {
const records = await runExecuteLogProbe(
{ id: "invocation-a", scope: { companyId: "company-a" } },
[
{ stream: "stdout", chunk: "one" },
{ stream: "stderr", chunk: "two" },
],
);
expect(records).toHaveLength(2);
expect(records[0]).toEqual({
params: { stream: "stdout", chunk: "one" },
invocationId: "invocation-a",
});
expect(records[1]).toEqual({
params: { stream: "stderr", chunk: "two" },
invocationId: "invocation-a",
});
});
it("drops an empty chunk before it reaches the host", async () => {
const records = await runExecuteLogProbe(
{ id: "invocation-a", scope: { companyId: "company-a" } },
[
{ stream: "stdout", chunk: "" },
{ stream: "stdout", chunk: "kept" },
],
);
expect(records).toEqual([
{ params: { stream: "stdout", chunk: "kept" }, invocationId: "invocation-a" },
]);
});
});
describe("worker setup-token pseudo-terminal dispatch", () => {
it("dispatches open, input, stop, and close, and streams output and exit as notifications", async () => {
const hostToWorker = new PassThrough();
const workerToHost = new PassThrough();
const hostReadline = createInterface({ input: workerToHost });
const pending = new Map<string, (response: JsonRpcResponse) => void>();
const notifications: JsonRpcNotification[] = [];
let nextRequestId = 1;
// The fake session the opener returns. The test drives its output and exit.
let emitOutput: ((chunk: string) => void) | null = null;
let resolveWait: ((value: { exitCode: number | null }) => void) | null = null;
const inputs: string[] = [];
let killed = 0;
let closed = 0;
// The worker emits output and exit through `ctx.loginPty`, bound to the
// worker session id. The test drives them through the captured emitters.
const controllablePlugin = definePlugin({
async setup(ctx) {
emitOutput = (chunk: string) => ctx.loginPty.output("route-1", "ws-1", chunk);
resolveWait = (value) => ctx.loginPty.exit("route-1", "ws-1", value.exitCode);
},
async onLoginPtyOpen(params) {
// The open carries the host route id, the closed command key, and the
// validated session home. The worker returns a worker session id for the
// output binding only. The open carries no command string.
expect(params.hostRouteId).toBe("route-1");
expect(params.loginCommandKey).toBe("claude");
expect(params.sessionHome).toBe(
"/tmp/paperclip-adapter-login/11111111-2222-4333-8444-555555555555",
);
expect(params.providerLeaseId).toBe("lease-1");
return { workerSessionId: "ws-1" };
},
async onLoginPtyInput(params) {
inputs.push(params.data);
},
async onLoginPtyStop() {
killed += 1;
},
async onLoginPtyClose(params) {
// The close keys on the host route id and returns a bound acknowledgement.
closed += 1;
return { hostRouteId: params.hostRouteId };
},
});
const worker = startWorkerRpcHost({
plugin: controllablePlugin,
stdin: hostToWorker,
stdout: workerToHost,
});
function callWorker(method: string, params: unknown) {
const id = `host-${nextRequestId++}`;
const result = new Promise<unknown>((resolve, reject) => {
pending.set(id, (response) => {
if ("error" in response && response.error) {
reject(new Error(response.error.message));
return;
}
resolve((response as { result?: unknown }).result);
});
});
hostToWorker.write(serializeMessage(createRequest(method, params, id)));
return result;
}
hostReadline.on("line", (line) => {
const message = parseMessage(line);
if (isJsonRpcResponse(message)) {
pending.get(String(message.id))?.(message);
pending.delete(String(message.id));
return;
}
if (isJsonRpcNotification(message)) {
notifications.push(message as JsonRpcNotification);
}
});
try {
await expect(
callWorker("initialize", {
manifest: {
id: "paperclip.login-pty",
apiVersion: 1,
version: "1.0.0",
displayName: "Login PTY Test",
description: "Test plugin",
author: "Paperclip",
categories: ["automation"],
capabilities: [],
entrypoints: {},
},
config: {},
databaseNamespace: null,
}),
).resolves.toMatchObject({
ok: true,
supportedMethods: expect.arrayContaining([
"loginPtyOpen",
"loginPtyInput",
"loginPtyStop",
"loginPtyClose",
]),
});
await expect(
callWorker("loginPtyOpen", {
hostRouteId: "route-1",
driverKey: "daytona",
companyId: "company-1",
environmentId: "env-1",
providerLeaseId: "lease-1",
loginCommandKey: "claude",
sessionHome: "/tmp/paperclip-adapter-login/11111111-2222-4333-8444-555555555555",
}),
).resolves.toEqual({ workerSessionId: "ws-1" });
// The worker streams output as a notification bound to the worker session id.
emitOutput?.("prompt output");
await callWorker("loginPtyInput", { workerSessionId: "ws-1", data: "browser-code" });
await callWorker("loginPtyStop", { workerSessionId: "ws-1" });
resolveWait?.({ exitCode: 0 });
await expect(
callWorker("loginPtyClose", { hostRouteId: "route-1" }),
).resolves.toEqual({ hostRouteId: "route-1" });
await new Promise((resolve) => setImmediate(resolve));
expect(inputs).toEqual(["browser-code"]);
expect(killed).toBe(1);
expect(closed).toBe(1);
const outputNotes = notifications.filter(
(note) => note.method === "loginPty.output",
);
expect(outputNotes.map((note) => note.params)).toEqual([
{ hostRouteId: "route-1", workerSessionId: "ws-1", chunk: "prompt output" },
]);
const exitNotes = notifications.filter(
(note) => note.method === "loginPty.exit",
);
expect(exitNotes.map((note) => note.params)).toEqual([
{ hostRouteId: "route-1", workerSessionId: "ws-1", exitCode: 0 },
]);
} finally {
worker.stop();
hostReadline.close();
}
});
});
describe("worker duplex channel dispatch", () => {
it("dispatches open, write, stop, and close, and reports the duplex methods", async () => {
const hostToWorker = new PassThrough();
const workerToHost = new PassThrough();
const hostReadline = createInterface({ input: workerToHost });
const pending = new Map<string, (response: JsonRpcResponse) => void>();
let nextRequestId = 1;
const writes: string[] = [];
let stopped = 0;
let closed = 0;
// The plugin declares the four duplex channel handlers. The open returns a
// worker session id. The close keys on the host route id and returns a bound
// acknowledgement.
const controllablePlugin = definePlugin({
async setup() {},
async onDuplexChannelOpen(params) {
expect(params.hostRouteId).toBe("route-1");
expect(params.command).toEqual(["paperclip-bridge"]);
expect(params.providerLeaseId).toBe("lease-1");
return { workerSessionId: "ws-1" };
},
async onDuplexChannelWrite(params) {
writes.push(params.data);
},
async onDuplexChannelStop() {
stopped += 1;
},
async onDuplexChannelClose(params) {
closed += 1;
return { hostRouteId: params.hostRouteId };
},
});
const worker = startWorkerRpcHost({
plugin: controllablePlugin,
stdin: hostToWorker,
stdout: workerToHost,
});
function callWorker(method: string, params: unknown) {
const id = `host-${nextRequestId++}`;
const result = new Promise<unknown>((resolve, reject) => {
pending.set(id, (response) => {
if ("error" in response && response.error) {
reject(new Error(response.error.message));
return;
}
resolve((response as { result?: unknown }).result);
});
});
hostToWorker.write(serializeMessage(createRequest(method, params, id)));
return result;
}
hostReadline.on("line", (line) => {
const message = parseMessage(line);
if (isJsonRpcResponse(message)) {
pending.get(String(message.id))?.(message);
pending.delete(String(message.id));
}
});
try {
await expect(
callWorker("initialize", {
manifest: {
id: "paperclip.duplex-channel",
apiVersion: 1,
version: "1.0.0",
displayName: "Duplex Channel Test",
description: "Test plugin",
author: "Paperclip",
categories: ["automation"],
capabilities: [],
entrypoints: {},
},
config: {},
databaseNamespace: null,
}),
).resolves.toMatchObject({
ok: true,
supportedMethods: expect.arrayContaining([
"duplexChannelOpen",
"duplexChannelWrite",
"duplexChannelStop",
"duplexChannelClose",
]),
});
await expect(
callWorker("duplexChannelOpen", {
hostRouteId: "route-1",
driverKey: "daytona",
companyId: "company-1",
environmentId: "env-1",
providerLeaseId: "lease-1",
command: ["paperclip-bridge"],
}),
).resolves.toEqual({ workerSessionId: "ws-1" });
await callWorker("duplexChannelWrite", { workerSessionId: "ws-1", data: "payload" });
await callWorker("duplexChannelStop", { workerSessionId: "ws-1" });
await expect(
callWorker("duplexChannelClose", { hostRouteId: "route-1" }),
).resolves.toEqual({ hostRouteId: "route-1" });
expect(writes).toEqual(["payload"]);
expect(stopped).toBe(1);
expect(closed).toBe(1);
} finally {
worker.stop();
hostReadline.close();
}
});
it("reports no duplex methods when the plugin declares no duplex handlers", async () => {
const hostToWorker = new PassThrough();
const workerToHost = new PassThrough();
const hostReadline = createInterface({ input: workerToHost });
const pending = new Map<string, (response: JsonRpcResponse) => void>();
let nextRequestId = 1;
// A plugin with no duplex handlers advertises no duplex method. The Phase 1
// capability `duplexCommandStream` still resolves false for this provider,
// because the prerequisite verb `duplexChannelOpen` is absent.
const barePlugin = definePlugin({
async setup() {},
});
const worker = startWorkerRpcHost({
plugin: barePlugin,
stdin: hostToWorker,
stdout: workerToHost,
});
function callWorker(method: string, params: unknown) {
const id = `host-${nextRequestId++}`;
const result = new Promise<unknown>((resolve, reject) => {
pending.set(id, (response) => {
if ("error" in response && response.error) {
reject(new Error(response.error.message));
return;
}
resolve((response as { result?: unknown }).result);
});
});
hostToWorker.write(serializeMessage(createRequest(method, params, id)));
return result;
}
hostReadline.on("line", (line) => {
const message = parseMessage(line);
if (isJsonRpcResponse(message)) {
pending.get(String(message.id))?.(message);
pending.delete(String(message.id));
}
});
try {
const result = (await callWorker("initialize", {
manifest: {
id: "paperclip.bare",
apiVersion: 1,
version: "1.0.0",
displayName: "Bare Test",
description: "Test plugin",
author: "Paperclip",
categories: ["automation"],
capabilities: [],
entrypoints: {},
},
config: {},
databaseNamespace: null,
})) as { supportedMethods: string[] };
expect(result.supportedMethods).not.toContain("duplexChannelOpen");
expect(result.supportedMethods).not.toContain("duplexChannelWrite");
expect(result.supportedMethods).not.toContain("duplexChannelStop");
expect(result.supportedMethods).not.toContain("duplexChannelClose");
} finally {
worker.stop();
hostReadline.close();
}
});
});
describe("AI connection router RPC", () => {
it("advertises and calls only an implemented routing hook", async () => {
const input = new PassThrough(), output = new PassThrough();
const reader = createInterface({ input: output });
const request = { companyId: "company", taskKey: "task", candidates: [], memberOrder: [] };
let received: unknown;
const plugin = definePlugin({ async setup() {}, onRouteAiConnection(params) { received = params; return { kind: "selected", memberId: "authorized-member" }; } });
const worker = startWorkerRpcHost({ plugin, stdin: input, stdout: output });
let sequence = 0;
const pending = new Map<string, (value: unknown) => void>();
reader.on("line", line => { const response = parseMessage(line); if (isJsonRpcResponse(response)) { pending.get(String(response.id))?.(response); pending.delete(String(response.id)); } });
const call = (method: string, params: unknown) => new Promise<JsonRpcResponse>(resolve => { const id = String(++sequence); pending.set(id, value => resolve(value as JsonRpcResponse)); input.write(serializeMessage(createRequest(method, params, id))); });
try {
const initialized = await call("initialize", { manifest: { id: "fixture.router", apiVersion: 1, version: "1.0.0", displayName: "Router", description: "Fixture", author: "Tests", categories: ["connector"], capabilities: ["ai.connections.route"], entrypoints: {} }, config: {}, databaseNamespace: null });
expect((initialized as { result: { supportedMethods: string[] } }).result.supportedMethods).toContain("routeAiConnection");
expect((await call("routeAiConnection", request) as { result: unknown }).result).toEqual({ kind: "selected", memberId: "authorized-member" });
expect(received).toEqual(request);
} finally { worker.stop(); reader.close(); input.destroy(); output.destroy(); }
});
});
describe("Durable lifecycle inbox RPC", () => {
it("sends company-scoped reads and acknowledgments through the worker SDK", async () => {
const input = new PassThrough(), output = new PassThrough();
const reader = createInterface({ input: output });
const calls: Array<{ method: string; params: unknown }> = [];
const event = { id: "7", companyId: "company-a", resourceType: "agent", resourceId: "agent-a", action: "pause", createdAt: new Date(0).toISOString() };
const plugin = definePlugin({ async setup(ctx) {
ctx.data.register("drain", async ({ companyId }) => {
const events = await ctx.events.listLifecycle(companyId!, 10, "6");
await ctx.events.acknowledgeLifecycle(companyId!, events[0].id);
return events;
});
} });
const worker = startWorkerRpcHost({ plugin, stdin: input, stdout: output });
let sequence = 0;
const pending = new Map<string, (value: JsonRpcResponse) => void>();
reader.on("line", line => {
const message = parseMessage(line);
if (isJsonRpcResponse(message)) {
pending.get(String(message.id))?.(message);
pending.delete(String(message.id));
} else if (isJsonRpcRequest(message)) {
calls.push({ method: message.method, params: message.params });
input.write(serializeMessage(createSuccessResponse(message.id, message.method === "events.listLifecycle" ? [event] : null)));
}
});
const call = (method: string, params: unknown) => new Promise<JsonRpcResponse>(resolve => {
const id = String(++sequence);
pending.set(id, resolve);
input.write(serializeMessage(createRequest(method, params, id)));
});
try {
await call("initialize", { manifest: { id: "fixture.lifecycle", apiVersion: 1, version: "1.0.0", displayName: "Lifecycle", description: "Fixture", author: "Tests", categories: ["automation"], capabilities: ["events.subscribe"], entrypoints: {} }, config: {}, databaseNamespace: null });
expect((await call("getData", { key: "drain", companyId: "company-a", params: {} }) as { result: unknown }).result).toEqual([event]);
expect(calls).toEqual([
{ method: "events.listLifecycle", params: { companyId: "company-a", limit: 10, afterId: "6" } },
{ method: "events.acknowledgeLifecycle", params: { companyId: "company-a", eventId: "7" } },
]);
} finally { worker.stop(); reader.close(); input.destroy(); output.destroy(); }
});
});