mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Plugins can serve connector tools from their worker (`transport: "plugin"`, added in #22004, #22019 and #22201). > - The tool gateway runs these tools with a fixed 10 s default timeout. > - Some plugin tools call backends that take 12–20 s. These calls fail every time with "Tool execution timed out", even though the plugin's own HTTP client waits much longer. > - The plugin connector design said that the connector, not a fixed gateway constant, sets the timeout. > - This pull request lets a plugin method or a single static tool declare `timeoutMs`, and gives plugin-served tools a 30 s default. > - The benefit is that a plugin connector can serve slow tools like other connectors can, within the same 60 s gateway ceiling. ## Linked Issues or Issue Description Refs #22004, #22019, #22201. Refs #20746. **Relationship to #20746.** This PR builds alongside #20746. It does not supersede it. #20746 changes the default for legacy top-level plugin tools (provider `paperclip_plugin`) to 60 s. This PR changes only plugin-served connector tools (provider `paperclip_plugin_connection`, `transport: "plugin"`). The two PRs touch different branches of the gateway and can merge in either order. This PR does not pin the legacy default in its tests, so #20746 does not break them. **Current behavior** A plugin connector tool always times out after 10 s, unless the caller sends `timeoutMs`. Agents and runners do not send it. The worker RPC also stops at 30 s. **Proposed behavior** A plugin can declare `timeoutMs` (1000–60000 ms) on a `transport: "plugin"` method or on one static tool. The gateway uses that value. If nothing is declared, the gateway waits 30 s. **Reason and benefit** Plugin tools whose backends take longer than 10 s can now work. Plugin authors control the value in the manifest, and the gateway keeps its 60 s ceiling. **Breaking changes** None. Existing manifests stay valid. Only plugin-served tools get the new 30 s default. ## What Changed - `packages/shared`: `timeoutMs` is a new optional field on `ConnectionMethodDef` and on `PluginConnectorToolDef`. The validator accepts integers from 1000 to 60000 only. It rejects `timeoutMs` on methods that do not use `transport: "plugin"`. New exported constants: `PLUGIN_CONNECTOR_TOOL_TIMEOUT_MIN_MS`, `PLUGIN_CONNECTOR_TOOL_TIMEOUT_MAX_MS`, and `PLUGIN_CONNECTOR_TOOL_DEFAULT_TIMEOUT_MS` (30000). - `server/src/services/plugin-connector-runtime.ts`: `pluginConnectionToolTimeoutMs()` resolves the timeout in this order: the tool's value, then the method's value, then 30 s. The worker `executeTool` RPC now gets the gateway timeout plus 1 s. Thus the gateway's `tool_timeout` error is what callers see, and the 30 s RPC default no longer cuts off a longer declared timeout. - `server/src/services/tool-gateway.ts`: `executePluginConnectionTool` now applies the timeout itself, after it loads the connection. All three plugin paths use it: session, assigned-tool and direct calls (`executeTool`), test calls (`executeTestCall`), and approved action execution. If a caller sends `timeoutMs`, that value wins, as before. It is still clamped to 60 s. Approved actions keep their fixed 60 s budget. - Other transports do not change. Remote MCP, local stdio, built-in and legacy plugin tools keep their timeouts. (#20746 separately proposes a 60 s default for legacy plugin tools.) - Docs: `doc/plugins/PLUGIN_SPEC.md` §11A and the plugin SDK README describe the field, its range, and the default. The SDK manifest type gets the field from `@paperclipai/shared`. ### Why 30 s 30 s is the same as the plugin worker manager's default RPC timeout. So, before this change, the worker already allowed 30 s, and only the gateway's 10 s limit stopped the call. With 30 s, the common 12–20 s backend case works without a declaration. It is also well under the 60 s ceiling. Tools that need more time can declare up to 60 s. ## Verification - New `server/src/__tests__/plugin-connector-tool-timeout.test.ts` runs the real gateway invocation path against an embedded Postgres and a fake worker. Fake timers make the 12 s backend free in wall-clock time: - With method `timeoutMs: 30000`, a 12 s tool succeeds through `executeTool` and through `executeTestCall`. The worker RPC timeout is 31000. - With tool `timeoutMs: 5000`, a 12 s tool fails at 5 s with `504 tool_timeout`. - With no declaration, a 12 s tool succeeds (it failed at the old 10 s), and a 45 s tool times out at 30 s. - Manifest values above 60000, below 1000, or not integers are rejected. A caller `timeoutMs: 120000` is clamped to 60 s. - A legacy plugin tool (`paperclip_plugin`, a different provider path) never reaches the connector runtime, and a caller deadline still applies. This test does not pin the legacy default, because #20746 changes it. - Remote MCP, local stdio and REST timeouts do not change in this PR. Their existing timeout tests in `tool-gateway.test.ts` and `google-workspace-rest-transport.test.ts` still apply. - Red/green: with the source change reverted, 4 of these 5 tests fail. The legacy-tool test passes both ways, as expected. - `packages/shared/src/plugin-apps.test.ts`: validation cases for the range, non-integer values, and non-plugin methods. - Commands run: - `pnpm --filter @paperclipai/shared typecheck && pnpm --filter @paperclipai/server typecheck && pnpm --filter @paperclipai/plugin-sdk typecheck` - `cd server && npx vitest run src/__tests__/plugin-connector-tool-timeout.test.ts src/__tests__/plugin-connectors-plugin-transport.test.ts src/__tests__/plugin-connector-agent-grant.test.ts` (20 passed) - `cd packages/shared && npx vitest run src/plugin-apps.test.ts src/app-definitions.test.ts` (76 passed) - `pnpm check:module-boundaries` - Full suite not run locally. CI supplies it. ## Risks - Low risk. Plugin-served tools now wait up to 30 s instead of 10 s by default. An agent turn can wait longer for a slow or stuck plugin tool, but never longer than the existing 60 s ceiling. - The connection row lookup now runs before the timeout timer starts. This is one indexed query. - Dynamic tool lists (`kind: "dynamic"`) use the method's `timeoutMs`. Per-tool values apply to static tools only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
207 lines
14 KiB
TypeScript
207 lines
14 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { mkdtemp, rm } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import {
|
|
agents, companies, companyMemberships, createDb, heartbeatRuns, issues, plugins, pluginConnectorLegacyAccess,
|
|
toolProfileBindings, toolProfileEntries, toolProfiles,
|
|
} from "@paperclipai/db";
|
|
import { pluginAppSlug, pluginManifestV1Schema, type PaperclipPluginManifestV1 } from "@paperclipai/shared";
|
|
import { toolAccessService } from "../services/tool-access.js";
|
|
import { createToolGatewayService } from "../services/tool-gateway.js";
|
|
import { setPluginConnectorWorkers } from "../services/plugin-connector-runtime.js";
|
|
import { completeLegacyPluginToolAccess } from "../services/plugin-connector-legacy-access.js";
|
|
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
|
|
import type { PluginToolDispatcher } from "../services/plugin-tool-dispatcher.js";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
|
|
|
const actor = { actorType: "user" as const, actorId: "plugin-timeout-admin" };
|
|
const PLUGIN_KEY = "example.slow";
|
|
const APP = "reports";
|
|
const SLUG = pluginAppSlug(PLUGIN_KEY, APP);
|
|
const BACKEND_DELAY_MS = 12_000;
|
|
|
|
function slowManifest(method: { timeoutMs?: number; quickTimeoutMs?: number } = {}) {
|
|
return {
|
|
id: PLUGIN_KEY, apiVersion: 1, version: "1.0.0", displayName: "Slow reports", description: "Slow backend",
|
|
author: "Tests", categories: ["connector"], capabilities: ["connectors.provide"], entrypoints: { worker: "worker.js" },
|
|
apps: [{
|
|
schemaVersion: 1, slug: APP, name: "Reports", description: "Build reports", categories: ["data"],
|
|
branding: { logoUrl: "/brands/apps/api-key-generic.svg" }, urlPatterns: [],
|
|
methods: [{
|
|
key: "none", transport: "plugin", auth: "none", ownershipModes: ["customer"], whenToUse: "No sign-in.",
|
|
guidanceMd: "Nothing to set up.", riskTier: "S1",
|
|
...(method.timeoutMs !== undefined ? { timeoutMs: method.timeoutMs } : {}),
|
|
tools: { kind: "static", list: [
|
|
{ name: "build_report", description: "Build a report.", risk: "read", parametersSchema: { type: "object" } },
|
|
{ name: "quick_check", description: "Check quickly.", risk: "read", parametersSchema: { type: "object" },
|
|
...(method.quickTimeoutMs !== undefined ? { timeoutMs: method.quickTimeoutMs } : {}) },
|
|
] },
|
|
}],
|
|
}],
|
|
};
|
|
}
|
|
|
|
function slowWorkers(delayMs: number) {
|
|
const calls: Array<{ method: string; params: any; timeoutMs?: number }> = [];
|
|
const manager = {
|
|
isRunning: () => true,
|
|
getWorker: () => ({ status: "running" }),
|
|
call: vi.fn((_id: string, method: string, params: any, timeoutMs?: number) => {
|
|
calls.push({ method, params: structuredClone(params), timeoutMs });
|
|
if (method !== "executeTool") return Promise.reject(new Error(`unexpected ${method}`));
|
|
return new Promise((resolve) => setTimeout(() => resolve({ content: `ran ${params.toolName}` }), delayMs));
|
|
}),
|
|
} as unknown as PluginWorkerManager;
|
|
return { manager, calls };
|
|
}
|
|
|
|
const support = await getEmbeddedPostgresTestSupport();
|
|
(support.supported ? describe : describe.skip)("plugin connector tool timeouts", () => {
|
|
let db: ReturnType<typeof createDb>;
|
|
let temp: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let keyDir: string;
|
|
beforeAll(async () => {
|
|
keyDir = await mkdtemp(join(tmpdir(), "plugin-timeout-secrets-"));
|
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", join(keyDir, "key"));
|
|
temp = await startEmbeddedPostgresTestDatabase("paperclip-plugin-timeout-");
|
|
db = createDb(temp.connectionString);
|
|
}, 60_000);
|
|
afterAll(async () => { await temp?.cleanup(); vi.unstubAllEnvs(); if (keyDir) await rm(keyDir, { recursive: true, force: true }); });
|
|
afterEach(async () => { vi.useRealTimers(); setPluginConnectorWorkers(null); await db.delete(plugins); });
|
|
|
|
async function setup(manifestJson: PaperclipPluginManifestV1, options: { delayMs?: number; dispatcher?: PluginToolDispatcher } = {}) {
|
|
const [company] = await db.insert(companies).values({ name: "Plugin timeouts", issuePrefix: `PO${randomUUID().slice(0, 6)}` }).returning();
|
|
await db.insert(companyMemberships).values({ companyId: company.id, principalType: "user", principalId: actor.actorId, status: "active", membershipRole: "admin" });
|
|
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Reporter", role: "engineer", adapterType: "process" }).returning();
|
|
await db.insert(plugins).values({ pluginKey: manifestJson.id, packageName: `@test/${manifestJson.id}`, version: "1.0.0", manifestJson, status: "ready" });
|
|
const workers = slowWorkers(options.delayMs ?? BACKEND_DELAY_MS);
|
|
setPluginConnectorWorkers(workers.manager);
|
|
const service = toolAccessService(db);
|
|
const gateway = createToolGatewayService(db, {
|
|
toolActionSigningSecret: "plugin-timeout-signing",
|
|
...(options.dispatcher ? { pluginToolDispatcher: options.dispatcher } : {}),
|
|
});
|
|
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Report", assigneeAgentId: agent.id }).returning();
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, issueId: issue.id, scopeKind: "issue",
|
|
invocationSource: "on_demand", status: "running", responsibleUserId: actor.actorId, contextSnapshot: { issueId: issue.id } }).returning();
|
|
const session = await gateway.createSession({ companyId: company.id, agentId: agent.id, runId: run.id, issueId: issue.id });
|
|
return { company, agent, workers, service, gateway, token: session.token };
|
|
}
|
|
|
|
async function connectReports(f: Awaited<ReturnType<typeof setup>>) {
|
|
const connected = await f.service.connectGalleryApp(f.company.id, { galleryKey: SLUG, connectionMethodKey: "none" }, actor);
|
|
await f.service.finishGalleryAppConnection(f.company.id, connected.connectionId, {
|
|
enabledCatalogEntryIds: connected.catalog.map(entry => entry.id), askFirstCatalogEntryIds: [], access: { agentIds: [f.agent.id] } }, actor);
|
|
const tools = (await f.gateway.listToolsForSession(f.token)).filter(tool => tool.providerType === "paperclip_plugin_connection");
|
|
const tool = (name: string) => tools.find(candidate => candidate.upstreamToolName === name)!;
|
|
return { connectionId: connected.connectionId, buildReport: tool("build_report"), quickCheck: tool("quick_check") };
|
|
}
|
|
|
|
async function settleAfter<T>(f: { workers: { calls: unknown[] } }, start: () => Promise<T>, advanceMs: number) {
|
|
const before = f.workers.calls.length;
|
|
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
|
const pending = start().then((value) => ({ value }), (error: unknown) => ({ error: error as { status?: number; reasonCode?: string } }));
|
|
await vi.waitFor(() => expect(f.workers.calls.length).toBe(before + 1), { timeout: 20_000, interval: 5 });
|
|
await vi.advanceTimersByTimeAsync(advanceMs);
|
|
const outcome = await pending;
|
|
vi.useRealTimers();
|
|
return outcome;
|
|
}
|
|
|
|
const timedOut = { error: expect.objectContaining({ status: 504, reasonCode: "tool_timeout" }) };
|
|
|
|
it("waits for the method's declared timeoutMs on session and test calls", async () => {
|
|
const f = await setup(pluginManifestV1Schema.parse(slowManifest({ timeoutMs: 30_000 })) as PaperclipPluginManifestV1);
|
|
const { connectionId, buildReport } = await connectReports(f);
|
|
|
|
const call = await settleAfter(f, () => f.gateway.executeTool({ sessionToken: f.token, tool: buildReport.name, parameters: {}, idempotencyKey: randomUUID() }), BACKEND_DELAY_MS);
|
|
expect(call).toMatchObject({ value: { status: "completed" } });
|
|
expect(f.workers.calls.at(-1)).toMatchObject({ method: "executeTool", timeoutMs: 31_000 });
|
|
|
|
const tested = await settleAfter(f, () => f.gateway.executeTestCall({ companyId: f.company.id, agentId: f.agent.id, userId: actor.actorId,
|
|
connectionId, toolName: buildReport.name, parameters: {} }), BACKEND_DELAY_MS);
|
|
expect(tested).toMatchObject({ value: { decision: "allowed" } });
|
|
expect(tested).not.toHaveProperty("value.error");
|
|
});
|
|
|
|
it("times out at a tool's declared timeoutMs when the backend is slower", async () => {
|
|
const f = await setup(pluginManifestV1Schema.parse(slowManifest({ timeoutMs: 30_000, quickTimeoutMs: 5_000 })) as PaperclipPluginManifestV1);
|
|
const { quickCheck } = await connectReports(f);
|
|
const call = await settleAfter(f, () => f.gateway.executeTool({ sessionToken: f.token, tool: quickCheck.name, parameters: {}, idempotencyKey: randomUUID() }), 5_000);
|
|
expect(call).toMatchObject(timedOut);
|
|
expect(f.workers.calls.at(-1)).toMatchObject({ timeoutMs: 6_000 });
|
|
});
|
|
|
|
it("prefers an explicit caller timeout below the declared value", async () => {
|
|
const f = await setup(pluginManifestV1Schema.parse(slowManifest({ timeoutMs: 30_000 })) as PaperclipPluginManifestV1);
|
|
const { buildReport } = await connectReports(f);
|
|
const call = await settleAfter(f, () => f.gateway.executeTool({ sessionToken: f.token, tool: buildReport.name, parameters: {},
|
|
timeoutMs: 5_000, idempotencyKey: randomUUID() }), 5_000);
|
|
expect(call).toMatchObject(timedOut);
|
|
expect(f.workers.calls.at(-1)).toMatchObject({ timeoutMs: 6_000 });
|
|
});
|
|
|
|
it("waits 30 s for plugin tools that declare no timeout", async () => {
|
|
const f = await setup(pluginManifestV1Schema.parse(slowManifest()) as PaperclipPluginManifestV1);
|
|
const { buildReport } = await connectReports(f);
|
|
const call = await settleAfter(f, () => f.gateway.executeTool({ sessionToken: f.token, tool: buildReport.name, parameters: {}, idempotencyKey: randomUUID() }), BACKEND_DELAY_MS);
|
|
expect(call).toMatchObject({ value: { status: "completed" } });
|
|
expect(f.workers.calls.at(-1)).toMatchObject({ timeoutMs: 31_000 });
|
|
|
|
setPluginConnectorWorkers(null);
|
|
const slower = slowWorkers(45_000);
|
|
f.workers.calls.length = 0;
|
|
setPluginConnectorWorkers(slower.manager);
|
|
const late = await settleAfter({ workers: slower }, () => f.gateway.executeTool({ sessionToken: f.token, tool: buildReport.name, parameters: {}, idempotencyKey: randomUUID() }), 30_000);
|
|
expect(late).toMatchObject(timedOut);
|
|
});
|
|
|
|
it("rejects declared timeouts outside 1-60 s and clamps caller timeouts to the 60 s ceiling", async () => {
|
|
expect(pluginManifestV1Schema.safeParse(slowManifest({ timeoutMs: 60_000 })).success).toBe(true);
|
|
for (const bad of [{ timeoutMs: 60_001 }, { timeoutMs: 120_000 }, { timeoutMs: 999 }, { timeoutMs: 1_500.5 }, { quickTimeoutMs: 90_000 }]) {
|
|
expect(pluginManifestV1Schema.safeParse(slowManifest(bad)).success).toBe(false);
|
|
}
|
|
|
|
const f = await setup(pluginManifestV1Schema.parse(slowManifest()) as PaperclipPluginManifestV1, { delayMs: 90_000 });
|
|
const { buildReport } = await connectReports(f);
|
|
const call = await settleAfter(f, () => f.gateway.executeTool({ sessionToken: f.token, tool: buildReport.name, parameters: {},
|
|
timeoutMs: 120_000, idempotencyKey: randomUUID() }), 60_000);
|
|
expect(call).toMatchObject(timedOut);
|
|
expect(f.workers.calls.at(-1)).toMatchObject({ timeoutMs: 61_000 });
|
|
});
|
|
it("leaves legacy plugin tools on their own deadline path", async () => {
|
|
const legacyManifest = pluginManifestV1Schema.parse({
|
|
id: "example.legacy-slow", apiVersion: 1, version: "1.0.0", displayName: "Legacy slow", description: "Legacy tools", author: "Tests",
|
|
categories: ["connector"], capabilities: ["agent.tools.register"], entrypoints: { worker: "worker.js" },
|
|
tools: [{ name: "read-status", displayName: "Read status", description: "Read status", parametersSchema: { type: "object" } }],
|
|
}) as PaperclipPluginManifestV1;
|
|
const legacyPluginDbId = randomUUID();
|
|
const dispatched: string[] = [];
|
|
const dispatcher: PluginToolDispatcher = {
|
|
initialize: async () => {}, teardown: () => {},
|
|
listToolsForAgent: () => [{ name: "example.legacy-slow:read-status", displayName: "Read status", description: "Read status",
|
|
parametersSchema: { type: "object" }, pluginId: legacyPluginDbId }],
|
|
getTool: () => null,
|
|
executeTool: (toolName) => {
|
|
dispatched.push(toolName);
|
|
return new Promise((resolve) => setTimeout(() => resolve({ pluginId: "example.legacy-slow", toolName: "read-status", result: { content: "ok" } }), BACKEND_DELAY_MS));
|
|
},
|
|
registerPluginTools: () => {}, unregisterPluginTools: () => {}, toolCount: () => 1, getRegistry: () => { throw new Error("unused"); },
|
|
};
|
|
const f = await setup(legacyManifest, { dispatcher });
|
|
const [profile] = await db.insert(toolProfiles).values({ companyId: f.company.id, profileKey: `legacy-${randomUUID()}`, name: `Legacy ${randomUUID()}`, defaultAction: "deny" }).returning();
|
|
await db.insert(toolProfileBindings).values({ companyId: f.company.id, profileId: profile.id, targetType: "agent", targetId: f.agent.id });
|
|
await db.insert(toolProfileEntries).values({ companyId: f.company.id, profileId: profile.id, selectorType: "tool_name", effect: "include", toolName: "example.legacy-slow:read-status" });
|
|
await db.insert(pluginConnectorLegacyAccess).values({ companyId: f.company.id, pluginKey: "example.legacy-slow" });
|
|
expect(await completeLegacyPluginToolAccess(db)).toBe(1);
|
|
const tool = (await f.gateway.listToolsForSession(f.token)).find(candidate => candidate.name === "example.legacy-slow:read-status")!;
|
|
expect(tool.providerType).toBe("paperclip_plugin");
|
|
const call = await settleAfter({ workers: { calls: dispatched } }, () => f.gateway.executeTool({ sessionToken: f.token, tool: tool.name, parameters: {},
|
|
timeoutMs: 5_000, idempotencyKey: randomUUID() }), 5_000);
|
|
expect(call).toMatchObject(timedOut);
|
|
expect(f.workers.calls).toEqual([]);
|
|
});
|
|
});
|