mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-18 15:39:47 +02:00
fix(connectors): avoid redundant run preparation discovery (#16461)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Run preparation selects connector skills and tools under current permissions. > - An assignment check can build an unrelated company tool catalog and wait for a shared discovery slot. > - Native startup repeats assignment discovery to obtain static tool definitions. > - Slack also locks and reconciles an unchanged catalog, then repeats task authorization. > - This pull request removes that redundant startup work and preserves live checks at tool execution. > - The benefit is less preparation work, less contention, and unchanged permission and approval enforcement. ## Linked Issues or Issue Description Related: #16454 (detailed preparation tracing), #16425 (workspace preparation performance). **What happened?** Connector configuration added avoidable work to run preparation. Browser Use assignment authorized the whole company catalog, including other connector families. Four connector resolvers ran in series. Native controller startup repeated all assignment resolution when any connector skill was pinned. Slack reconciled an unchanged catalog under an advisory lock and checked task authority twice. **Expected behavior** Prepare only the connector data needed by the run. Reuse static tool definitions from pinned skills. Preserve current company, assignment, identity, profile, policy, grant, and approval checks when an operation executes. **Steps to reproduce** 1. Configure Browser Use and an unrelated MCP catalog with 500 tools for one company. 2. Resolve Browser Use resources for an authorized task. The old path reads and authorizes the unrelated catalog. 3. Occupy both full discovery slots. The old assignment check waits for those slots. 4. Resolve a linked Slack task repeatedly with an existing current catalog. The old path still acquires a catalog lock and repeats authority checks. **Paperclip version or commit** Base: `68a09dcb5f50e3fb6ceaaeac2706d96db97fc9b3`. **Deployment mode** Observed in hosted native runs. The local reproduction uses a source checkout and real embedded PostgreSQL. ## What Changed - Restrict Browser Use catalog reads and policy decisions to active Browser Use connections. Preserve guest restrictions, reviewed catalog filters, live grants, and deterministic resource order. Keep assignment checks out of the full discovery queue. - Resolve the four independent connector families concurrently. Wait for outstanding work to settle before propagating a failure, and retain registration order. - Register native direct connector tool definitions from pinned connector skills. Keep live assignment resolution and governed execution on every tool call. - Compare Slack catalog hashes before reconciliation. Current catalogs skip the catalog transaction and lock. Check task authority once after current profile checks. Initial setup and upgrades retain locked reconciliation and checks before and after it. - Precompute immutable bundled Slack descriptor hashes. Preserve disabled entries, profile state, removed bindings, and review metadata. - Document the startup paths and add focused contention, revocation, catalog upgrade, and failure-settlement regressions. - Repair a pre-existing CI guard false positive: the filename `docker-compose-secrets.test.mjs` matched `secrets.`. Check GitHub secret expressions instead, with dot, bracket, and whole-context controls. No workflow changes. ## Verification - Conflict resolution: merge `b1566ca494f4af6e8d4418fa37393c5caee02582` from master into head `55140b1609ae7bd21a038eb75a48c2683299e4b4`. Combine both secret-expression guards and regression cases. Preserve current workspace, native recovery, and Daytona changes. All 467 policy tests, 787 focused native/connector/Browser Use tests, and server typecheck passed locally after integration. Fresh full CI passed all 55 checks on this head, with two optional Storybook jobs skipped. Greptile reviewed this exact head at 5/5 with no unresolved threads. - `pnpm -r typecheck` — passed. - `pnpm build` — passed. - Focused native startup, connector concurrency, and AgentMail suites — 763 tests passed. - Complete Browser Use suite — 33 tests passed. Focused Slack catalog, linked task, retained provenance/revocation, and pinned native tool tests passed. The complete runner tool authority suite passed (26 tests). The 1,337-case chat suite passed 1,335 tests and had two failures: the original new transaction-count assertion and a 15-second timeout in the publication-batch case. Both cases passed an isolated rerun. The publication timeout occurred during concurrent local checks; the passing rerun does not erase it. CI must still pass the chat shards. - The first focused run found two incorrect new assertions: policy evaluation performs a second catalog lookup, and an explicit allow policy overrides a profile default. The tests now distinguish the scoped catalog query and revoke the governing policy. The focused Browser Use rerun passed all 33 cases after these assertion fixes. - Full-suite test verification used the CI shards of `pnpm test:run`. Local runs focused on the changed boundaries; the complete repository test command was not duplicated locally. - `node --test '.github/scripts/tests/*.test.mjs'` — all 467 policy tests passed. The CI failure reproduced on the unchanged base guard before this correction. Actual GitHub secret expressions still match the guard. - CI on the final head passed all chat shards, typecheck, build, Runner checks, browser shards, and other test lanes. One existing workspace-runtime case hit `EADDRINUSE` for a randomly selected test listener in server shard 1/12 (1,162 tests passed in that shard). The shard passed its single retry on the same SHA with no source changes. The first failure remains recorded here. Previous reviewed head `7191ac40691ddb1ef38b86665c95cc14c57ea390`: all 55 checks passed, two optional Storybook jobs skipped, and no pending or failed checks. Greptile is 5/5 on this head with no unresolved review threads. The external security scan passed. Local before/after measurements use the old and new implementations against the same fixture. Both paths are warmed. Seven samples alternate execution order. The database is real embedded PostgreSQL. No latency is injected. These measurements do not establish hosted speed. | Local path | Before median (range) | After median (range) | Work reduction | | --- | --- | --- | --- | | Browser Use assignment with 500 unrelated MCP tools | 44.020 ms (33.994–107.328) | 26.502 ms (20.385–99.037) | 26 → 22 statements; unrelated catalog and discovery queue removed | | Slack assignment with current catalog and linked responsible user | 49.723 ms (43.624–80.653) | 22.141 ms (19.781–30.818) | 5 → 2 database transactions; no catalog lock on the current path | The contention regression fails against the old implementation after a bounded five-second wait, then passes against the optimized implementation. It occupies the real global discovery scheduler and verifies that Browser Use assignment completes without releasing those slots. It records catalog SQL to verify connection scope. Revocation cases remove policy, catalog availability, and grants between calls. Existing Slack integration cases exercise retained provenance, removed profile bindings, identity revocation, approved writes, and current responsible-user restrictions. ## Risks - Pinned native tool definitions can remain visible after access is revoked. They confer no authority. Invocation rechecks current assignments and fails closed before an operation executes. Newly available connector families still require a newly admitted skill selection. - Concurrent resolution can increase instantaneous database reads to four connector families per configuration call. It is a fixed bound. Outstanding work settles before a failed preparation returns. - Slack skips reconciliation only when all bundled version hashes and an existing profile match. Missing or changed definitions still use the existing locked path. Operator revocations remain effective. - This PR addresses connector preparation. Workspace transfer, provider startup, copyback, and queue delay can still keep total latency above the user's goal. Hosted improvement needs a deployed before/after trace. - No schema, lockfile, workflow, telemetry event, or authorization cache changes. ## Model Used OpenAI GPT-6 (Codex), with reasoning, repository inspection, code execution, and GitHub tools. The runtime does not expose the exact serving model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
b1566ca494
commit
5ec93dae01
13 files changed
+247
-65
No files matched your search
@@ -241,11 +241,12 @@ test("workflow exporter runs trusted code and keeps keys out of the PR jobs", ()
|
||||
const pr = readFileSync(new URL("../../workflows/pr-trusted.yml", import.meta.url), "utf8");
|
||||
// A script filename such as docker-compose-secrets.test.mjs is not a
|
||||
// GitHub secret lookup. Check expression context, including bracket access.
|
||||
const secretReference = /HONEYCOMB.*KEY|\$\{\{[^}]*\bsecrets\s*(?:\.|\[)/;
|
||||
const secretReference = /HONEYCOMB.*KEY|\$\{\{[^}]*\bsecrets\b/;
|
||||
assert.doesNotMatch("run: node --test scripts/docker-compose-secrets.test.mjs", secretReference);
|
||||
for (const lookup of ["${{ secrets.PR_TOKEN }}", "${{ secrets['PR_TOKEN'] }}", "${{\n secrets.PR_TOKEN\n }}"]) {
|
||||
for (const lookup of ["${{ secrets.PR_TOKEN }}", "${{ secrets['PR_TOKEN'] }}", "${{\n secrets.PR_TOKEN\n }}", "${{ toJSON(secrets) }}", "${{ secrets.HONEYCOMB_API_KEY }}"]) {
|
||||
assert.match(lookup, secretReference);
|
||||
}
|
||||
|
||||
assert.doesNotMatch(pr, secretReference);
|
||||
assert.match(pr, /if: always\(\) && env.PAPERCLIP_CI_PROFILE == 'true'/);
|
||||
assert.doesNotMatch(pr, /PAPERCLIP_CI_PROFILE:.*labels/);
|
||||
|
||||
+24
-1
@@ -279,7 +279,8 @@ attempt or owning phase:
|
||||
|
||||
Detail phases emit at most 16 spans of each name per attempt. Larger reader and
|
||||
asset inventories continue under the containing group without unbounded per-row
|
||||
or per-file spans. Work, query order, authorization, and locks are unchanged.
|
||||
or per-file spans. Adding detail spans does not itself change work, query order,
|
||||
authorization, or locks.
|
||||
Manual transaction and dispatch boundaries retain their actual lifetime.
|
||||
Asset writes and verification may add only non-negative integer
|
||||
`paperclip.execution.fileCount` and `paperclip.execution.bytes` attributes.
|
||||
@@ -295,6 +296,28 @@ can continue through later bookkeeping and resource cleanup. A later recovery
|
||||
invocation has its own attempt; this does not manufacture one continuous trace
|
||||
across queueing, process restarts, or an unavailable controller.
|
||||
|
||||
### Connector preparation work
|
||||
|
||||
Run configuration resolves the four connector families concurrently and preserves
|
||||
registration order. Browser Use assignment reads only active Browser Use
|
||||
connections and their eligible catalog entries. It applies the same tool policy,
|
||||
grant, company, and restricted GitHub guest checks without joining the full tool
|
||||
discovery queue or resolving unrelated Slack, GitHub, and Speko tools.
|
||||
|
||||
Native startup registers direct connector tool definitions from the run's pinned
|
||||
skills. It does not repeat assignment discovery. These definitions do not grant
|
||||
access: each invocation still resolves current assignments and enters the existing
|
||||
permission and approval gateway. A retained session may advertise a tool whose
|
||||
access was revoked; the call fails under the current authorization state.
|
||||
|
||||
Slack assignment checks stored catalog hashes against the bundled definitions.
|
||||
A current catalog and existing profile need no catalog transaction or advisory
|
||||
lock. Current profile bindings and entries still determine availability, followed
|
||||
by one live task authority check. Missing or changed catalog definitions use the
|
||||
existing locked reconciliation path, with authority checks before and after the
|
||||
update. Disabled catalog entries and operator-removed bindings remain disabled.
|
||||
No authorization result is cached across calls.
|
||||
|
||||
### Reading a slow startup in Honeycomb
|
||||
|
||||
Filter `paperclip.task.run.run_id` to the first 12 hex characters of
|
||||
|
||||
@@ -22,6 +22,7 @@ import {
|
||||
issues,
|
||||
toolConnections,
|
||||
toolApplications,
|
||||
toolCatalogEntries,
|
||||
toolConnectionInstalls,
|
||||
toolProfiles,
|
||||
toolProfileBindings,
|
||||
@@ -419,6 +420,59 @@ const actor = { actorType: "user" as const, actorId: "browser-reviewer" };
|
||||
expect(f.request).not.toHaveBeenCalled();
|
||||
} finally { discovery.mockRestore(); }
|
||||
});
|
||||
it("resolves only the Browser Use catalog even while full discovery is saturated", async () => {
|
||||
const f = await fixture();
|
||||
const [application] = await db.insert(toolApplications).values({
|
||||
companyId: f.company.id, name: "Unrelated MCP", type: "mcp_http", status: "active",
|
||||
}).returning();
|
||||
const [connection] = await db.insert(toolConnections).values({
|
||||
companyId: f.company.id, applicationId: application.id, name: "Unrelated MCP",
|
||||
uid: randomUUID(), transport: "mcp_remote", status: "active", enabled: true, healthStatus: "ok",
|
||||
}).returning();
|
||||
await db.insert(toolCatalogEntries).values(Array.from({ length: 500 }, (_, i) => ({
|
||||
companyId: f.company.id, applicationId: application.id, connectionId: connection.id,
|
||||
entryKind: "tool" as const, name: `unrelated_${i}`, toolName: `unrelated_${i}`,
|
||||
inputSchema: { type: "object" }, riskLevel: "read" as const, status: "active" as const,
|
||||
versionHash: "fixture", schemaHash: "fixture",
|
||||
})));
|
||||
const { recordingDb } = await import("./helpers/tool-gateway-listing-fixture.js");
|
||||
const { toolDiscoveryScheduler } = await import("../services/tool-discovery-scheduler.js");
|
||||
const releases: Array<() => void> = [];
|
||||
const blockers = [0, 1].map(() => toolDiscoveryScheduler.run(() => new Promise<void>(resolve => releases.push(resolve))));
|
||||
await vi.waitFor(() => expect(releases).toHaveLength(2));
|
||||
const recorded = recordingDb(db);
|
||||
const gateway = createToolGatewayService(recorded.db);
|
||||
const pending = gateway.browserUseResources(f.binding);
|
||||
let timeout: ReturnType<typeof setTimeout> | undefined;
|
||||
try {
|
||||
const resources = await Promise.race([pending, new Promise<never>((_, reject) => {
|
||||
timeout = setTimeout(() => reject(new Error("Assignment waited for unrelated discovery")), 5_000);
|
||||
})]);
|
||||
expect(resources).toEqual([{ id: f.connection.connectionId, connectionId: f.connection.connectionId, label: "Browser Use Cloud" }]);
|
||||
const catalogReads = recorded.statements.flatMap((query, index) =>
|
||||
query.includes('from "tool_catalog_entries"') && query.includes('inner join "tool_connections"') ? [recorded.statementParams[index]] : []);
|
||||
expect(catalogReads).toHaveLength(1);
|
||||
expect(catalogReads[0]).toContain(f.connection.connectionId);
|
||||
expect(catalogReads[0]).not.toContain(connection.id);
|
||||
expect(recorded.statements.some(query => query.includes('from "chat_endpoints"'))).toBe(false);
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
releases.forEach(release => release());
|
||||
await Promise.all(blockers);
|
||||
await pending;
|
||||
}
|
||||
});
|
||||
it.each(["policy", "catalog", "grant"])("rechecks Browser Use %s revocation without a cached assignment", async kind => {
|
||||
const f = await fixture();
|
||||
expect(await f.gateway.browserUseResources(f.binding)).toHaveLength(1);
|
||||
if (kind === "policy") await db.insert(toolPolicies).values({
|
||||
companyId: f.company.id, name: "Revoke Browser Use", policyType: "block",
|
||||
selectors: { connectionId: f.connection.connectionId }, priority: 100,
|
||||
});
|
||||
if (kind === "catalog") await db.update(toolCatalogEntries).set({ quarantinedAt: new Date() }).where(eq(toolCatalogEntries.connectionId, f.connection.connectionId));
|
||||
if (kind === "grant") await db.update(connectionGrants).set({ status: "revoked" }).where(eq(connectionGrants.id, f.grant.id));
|
||||
expect(await f.gateway.browserUseResources(f.binding)).toEqual([]);
|
||||
});
|
||||
it("withholds Cloud instructions when connection tools are disabled", async () => {
|
||||
const f = await fixture();
|
||||
await db.update(toolConnections).set({ enabled: false }).where(eq(toolConnections.id, f.connection.connectionId));
|
||||
|
||||
@@ -8096,10 +8096,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
|
||||
it("refreshes Slack tool guidance without restoring disabled catalog policies", async () => {
|
||||
const { SLACK_TOOLS } = await import("@paperclipai/shared");
|
||||
const { syncSlackBotTools } = await import("../services/connectors/slack-catalog.js");
|
||||
const { slackBotCatalogIsCurrent, syncSlackBotTools } = await import("../services/connectors/slack-catalog.js");
|
||||
const fixture = await seedCompany();
|
||||
const { endpoint } = await configuredSlackEndpoint(fixture);
|
||||
expect(await slackBotCatalogIsCurrent(db, endpoint)).toBe(false);
|
||||
await db.transaction(tx => syncSlackBotTools(tx, endpoint, "owner-user", true));
|
||||
expect(await slackBotCatalogIsCurrent(db, endpoint)).toBe(true);
|
||||
const tool = SLACK_TOOLS.find(tool => tool.name === "slack_post_message")!;
|
||||
// Simulate the previous hash contract, which omitted descriptions.
|
||||
const oldHash = createHash("sha256").update(JSON.stringify({ name: tool.name, inputSchema: tool.inputSchema, risk: tool.risk, scopes: tool.scopes })).digest("hex");
|
||||
@@ -8108,7 +8110,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
description: "Old posting guidance", versionHash: oldHash, schemaHash: oldHash,
|
||||
status: "removed", reviewedByUserId: "owner-user", reviewedAt,
|
||||
}).where(and(eq(toolCatalogEntries.connectionId, endpoint.connectionId), eq(toolCatalogEntries.toolName, tool.name)));
|
||||
expect(await slackBotCatalogIsCurrent(db, endpoint)).toBe(false);
|
||||
await db.transaction(tx => syncSlackBotTools(tx, endpoint, "different-operator", true));
|
||||
expect(await slackBotCatalogIsCurrent(db, endpoint)).toBe(true);
|
||||
const [updated] = await db.select().from(toolCatalogEntries).where(and(eq(toolCatalogEntries.connectionId, endpoint.connectionId), eq(toolCatalogEntries.toolName, tool.name)));
|
||||
expect(updated).toMatchObject({ description: tool.description, status: "removed", reviewedByUserId: "owner-user", reviewedAt });
|
||||
expect(updated!.versionHash).not.toBe(oldHash);
|
||||
@@ -8132,6 +8136,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
expect(assignments.map(item => item.key)).toEqual(["slack"]);
|
||||
expect(assignments[0].resources[0]).toMatchObject({ id: endpoint.id, metadata: { requesterId: "UROUTINE" } });
|
||||
expect(assignments[0].tools.some(tool => tool.name === "slack_open_dm")).toBe(true);
|
||||
const transactions = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
expect(await resolveConnectorAssignments(db, binding)).toEqual(assignments);
|
||||
expect(transactions).toHaveBeenCalledTimes(2);
|
||||
} finally { transactions.mockRestore(); }
|
||||
|
||||
await expect(resolveSlackTaskAuthority(db, binding)).resolves.toMatchObject({ conversation: null, userId: "owner-user", slackUserId: "UROUTINE" });
|
||||
const upstream = vi.fn(async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||
const method = String(input).split("/").at(-1);
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { resolveConnectorAssignments } from "./connector-runtime.js";
|
||||
|
||||
const resolvers = vi.hoisted(() => ({
|
||||
browser: vi.fn(), speko: vi.fn(), slack: vi.fn(), inboxes: vi.fn(),
|
||||
}));
|
||||
vi.mock("./native-runtime/assigned-mcp-tools.js", () => ({ getAssignedMcpGateway: () => ({ browserUseResources: resolvers.browser }) }));
|
||||
vi.mock("./voice/speko-agent-tools.js", () => ({ spekoAssignedResources: resolvers.speko }));
|
||||
vi.mock("./connectors/slack.js", () => ({ slackAssignedResource: resolvers.slack, executeGovernedSlackTool: vi.fn() }));
|
||||
vi.mock("./email-channels.js", () => ({ emailChannelService: () => ({ assignedInboxes: resolvers.inboxes }) }));
|
||||
|
||||
afterEach(() => vi.resetAllMocks());
|
||||
|
||||
function deferred<T>() {
|
||||
let resolve!: (value: T) => void;
|
||||
let reject!: (error: Error) => void;
|
||||
const promise = new Promise<T>((yes, no) => { resolve = yes; reject = no; });
|
||||
return { promise, resolve, reject };
|
||||
}
|
||||
|
||||
describe("connector assignment preparation", () => {
|
||||
it("starts independent resolvers together and retains connector order", async () => {
|
||||
const pending = Array.from({ length: 4 }, () => deferred<Array<{ id: string; label: string; connectionId: string; address: string }>>());
|
||||
Object.values(resolvers).forEach((resolver, i) => resolver.mockReturnValue(pending[i].promise));
|
||||
const assignments = resolveConnectorAssignments({} as Db, { companyId: "company", agentId: "agent" });
|
||||
await vi.waitFor(() => Object.values(resolvers).forEach(resolver => expect(resolver).toHaveBeenCalledOnce()));
|
||||
for (const i of [3, 2, 1, 0]) pending[i].resolve([{ id: `resource-${i}`, connectionId: `connection-${i}`, label: "Resource", address: "inbox@example.com" }]);
|
||||
expect((await assignments).map(assignment => assignment.key)).toEqual(["browser-use-cloud", "speko", "slack", "agentmail"]);
|
||||
});
|
||||
|
||||
it("settles outstanding resolution before propagating a failed connector", async () => {
|
||||
const slack = deferred<[]>();
|
||||
resolvers.browser.mockResolvedValue([]);
|
||||
resolvers.speko.mockRejectedValue(new Error("Speko unavailable"));
|
||||
resolvers.slack.mockReturnValue(slack.promise);
|
||||
resolvers.inboxes.mockResolvedValue([]);
|
||||
let settled = false;
|
||||
const assignments = resolveConnectorAssignments({} as Db, { companyId: "company", agentId: "agent" });
|
||||
const result = expect(assignments).rejects.toThrow("Speko unavailable").finally(() => { settled = true; });
|
||||
await vi.waitFor(() => expect(resolvers.slack).toHaveBeenCalledOnce());
|
||||
expect(settled).toBe(false);
|
||||
slack.resolve([]);
|
||||
await result;
|
||||
});
|
||||
});
|
||||
@@ -128,19 +128,25 @@ export async function resolveConnectorAssignments(
|
||||
db: Db,
|
||||
binding: AgentBinding,
|
||||
): Promise<ConnectorAssignment[]> {
|
||||
const assignments: ConnectorAssignment[] = [];
|
||||
for (const connector of connectors) {
|
||||
const results = await Promise.allSettled(connectors.map(async (connector) => {
|
||||
const resources = await measureRunPhase(connector.resolvePhase, () => connector.resolve(db, binding));
|
||||
if (resources.length)
|
||||
assignments.push({
|
||||
key: connector.key,
|
||||
label: connector.label,
|
||||
skillKey: skillKey(connector),
|
||||
resources,
|
||||
tools: connector.tools,
|
||||
});
|
||||
}
|
||||
return assignments;
|
||||
return resources.length ? [{
|
||||
key: connector.key,
|
||||
label: connector.label,
|
||||
skillKey: skillKey(connector),
|
||||
resources,
|
||||
tools: connector.tools,
|
||||
}] : [];
|
||||
}));
|
||||
return results.flatMap(result => {
|
||||
if (result.status === "rejected") throw result.reason;
|
||||
return result.value;
|
||||
});
|
||||
}
|
||||
|
||||
export function connectorToolDefinitionsForSkills(skillKeys: ReadonlySet<string>): Pick<ConnectorAssignment, "tools">[] {
|
||||
return connectors.filter(connector => skillKeys.has(skillKey(connector)) && connector.tools.length > 0)
|
||||
.map(connector => ({ tools: connector.tools }));
|
||||
}
|
||||
|
||||
export function isConnectorSkill(key: string) {
|
||||
|
||||
@@ -187,7 +187,7 @@ export async function slackEndpointCandidates(
|
||||
binding: Pick<SlackTaskBinding, "companyId" | "agentId" | "endpointId">,
|
||||
) {
|
||||
return db
|
||||
.select({ id: chatEndpoints.id })
|
||||
.select({ id: chatEndpoints.id, connectionId: chatEndpoints.connectionId })
|
||||
.from(chatEndpoints)
|
||||
.where(
|
||||
and(
|
||||
|
||||
@@ -20,6 +20,36 @@ import {
|
||||
resolveSlackTaskAuthority,
|
||||
slackEndpointCandidates,
|
||||
} from "./slack-authority.js";
|
||||
const catalogDefinitions = SLACK_TOOLS.map(tool => ({
|
||||
tool,
|
||||
hash: createHash("sha256").update(JSON.stringify({
|
||||
name: tool.name,
|
||||
description: tool.description,
|
||||
inputSchema: tool.inputSchema,
|
||||
risk: tool.risk,
|
||||
scopes: tool.scopes,
|
||||
})).digest("hex"),
|
||||
}));
|
||||
|
||||
export async function slackBotCatalogIsCurrent(
|
||||
db: Db,
|
||||
endpoint: { id: string; companyId: string; connectionId: string },
|
||||
): Promise<boolean> {
|
||||
const [entries, profiles] = await Promise.all([
|
||||
db.select({ name: toolCatalogEntries.name, versionHash: toolCatalogEntries.versionHash })
|
||||
.from(toolCatalogEntries).where(and(
|
||||
eq(toolCatalogEntries.companyId, endpoint.companyId),
|
||||
eq(toolCatalogEntries.connectionId, endpoint.connectionId),
|
||||
)),
|
||||
db.select({ id: toolProfiles.id }).from(toolProfiles).where(and(
|
||||
eq(toolProfiles.companyId, endpoint.companyId),
|
||||
eq(toolProfiles.profileKey, `slack-bot:${endpoint.id}`),
|
||||
)).limit(1),
|
||||
]);
|
||||
const hashes = new Map(entries.map(entry => [entry.name, entry.versionHash]));
|
||||
return profiles.length > 0 && catalogDefinitions.every(({ tool, hash }) => hashes.get(`slack_bot:${tool.name}`) === hash);
|
||||
}
|
||||
|
||||
export async function syncSlackBotTools(
|
||||
tx: Parameters<Parameters<Db["transaction"]>[0]>[0],
|
||||
endpoint: typeof chatEndpoints.$inferSelect,
|
||||
@@ -46,18 +76,7 @@ export async function syncSlackBotTools(
|
||||
const entriesByName = new Map(
|
||||
existingEntries.map((entry) => [entry.name, entry]),
|
||||
);
|
||||
for (const tool of SLACK_TOOLS) {
|
||||
const hash = createHash("sha256")
|
||||
.update(
|
||||
JSON.stringify({
|
||||
name: tool.name,
|
||||
description: tool.description,
|
||||
inputSchema: tool.inputSchema,
|
||||
risk: tool.risk,
|
||||
scopes: tool.scopes,
|
||||
}),
|
||||
)
|
||||
.digest("hex");
|
||||
for (const { tool, hash } of catalogDefinitions) {
|
||||
const values = {
|
||||
companyId: endpoint.companyId,
|
||||
applicationId: connection.applicationId,
|
||||
|
||||
@@ -2,7 +2,7 @@ import { authorizeSlackDocument } from "./slack-document.js";
|
||||
import { slackDelivery } from "./slack-delivery.js";
|
||||
import { slackMessage } from "./slack-message.js";
|
||||
import { executeSlackWrite } from "./slack-writes.js";
|
||||
import { syncSlackBotTools } from "./slack-catalog.js";
|
||||
import { slackBotCatalogIsCurrent, syncSlackBotTools } from "./slack-catalog.js";
|
||||
import { SLACK_TOOLS } from "@paperclipai/shared";
|
||||
import { measureRunPhase } from "../execution-preparation-timing.js";
|
||||
import { measureRunTransaction } from "../execution-transaction-timing.js";
|
||||
@@ -34,28 +34,24 @@ import {
|
||||
async function slackAssignedEndpoint(
|
||||
db: Db,
|
||||
binding: Partial<SlackTaskBinding> & { companyId: string; agentId: string },
|
||||
endpoint: { id: string; connectionId: string },
|
||||
) {
|
||||
if (!binding.runId || !binding.issueId) return [];
|
||||
let authority: Awaited<ReturnType<typeof resolveSlackTaskAuthority>>;
|
||||
try {
|
||||
authority = await measureRunPhase("connectors.slack.authority", () => resolveSlackTaskAuthority(
|
||||
db,
|
||||
binding as SlackTaskBinding,
|
||||
));
|
||||
} catch (error) {
|
||||
if (
|
||||
error &&
|
||||
typeof error === "object" &&
|
||||
"status" in error &&
|
||||
error.status === 403
|
||||
)
|
||||
return [];
|
||||
throw error;
|
||||
const authorize = async () => {
|
||||
try {
|
||||
return await measureRunPhase("connectors.slack.authority", () => resolveSlackTaskAuthority(db, binding as SlackTaskBinding));
|
||||
} catch (error) {
|
||||
if (error && typeof error === "object" && "status" in error && error.status === 403) return null;
|
||||
throw error;
|
||||
}
|
||||
};
|
||||
if (!await slackBotCatalogIsCurrent(db, { ...endpoint, companyId: binding.companyId })) {
|
||||
const authority = await authorize();
|
||||
if (!authority || authority.endpoint.connectionId !== endpoint.connectionId) return [];
|
||||
await measureRunTransaction(db, "connectors.slack.catalog", (tx) =>
|
||||
syncSlackBotTools(tx, authority.endpoint, authority.userId, true),
|
||||
);
|
||||
}
|
||||
// Catalog updates are idempotent and preserve configured policies.
|
||||
await measureRunTransaction(db, "connectors.slack.catalog", (tx) =>
|
||||
syncSlackBotTools(tx, authority.endpoint, authority.userId, true),
|
||||
);
|
||||
const [enabledProfile] = await measureRunPhase("connectors.slack.profile", () => db
|
||||
.select({ id: toolProfiles.id })
|
||||
.from(toolProfiles)
|
||||
@@ -72,20 +68,21 @@ async function slackAssignedEndpoint(
|
||||
toolProfileEntries,
|
||||
and(
|
||||
eq(toolProfileEntries.profileId, toolProfiles.id),
|
||||
eq(toolProfileEntries.connectionId, authority.endpoint.connectionId),
|
||||
eq(toolProfileEntries.connectionId, endpoint.connectionId),
|
||||
eq(toolProfileEntries.effect, "include"),
|
||||
),
|
||||
)
|
||||
.where(
|
||||
and(
|
||||
eq(toolProfiles.companyId, binding.companyId),
|
||||
eq(toolProfiles.profileKey, `slack-bot:${authority.endpoint.id}`),
|
||||
eq(toolProfiles.profileKey, `slack-bot:${endpoint.id}`),
|
||||
eq(toolProfiles.status, "active"),
|
||||
),
|
||||
)
|
||||
.limit(1));
|
||||
if (!enabledProfile) return [];
|
||||
authority = await measureRunPhase("connectors.slack.authority", () => resolveSlackTaskAuthority(db, binding as SlackTaskBinding));
|
||||
const authority = await authorize();
|
||||
if (!authority || authority.endpoint.connectionId !== endpoint.connectionId) return [];
|
||||
return [
|
||||
{
|
||||
id: authority.endpoint.id,
|
||||
@@ -113,7 +110,7 @@ export async function slackAssignedResource(
|
||||
...(await slackAssignedEndpoint(db, {
|
||||
...binding,
|
||||
endpointId: endpoint.id,
|
||||
})),
|
||||
}, endpoint)),
|
||||
);
|
||||
}
|
||||
return resources;
|
||||
|
||||
@@ -35,7 +35,7 @@ import { prepareVerifiedRemoteProviderPack } from "./remote-provider-pack.js";
|
||||
import { selectRemotePiCompanion } from "./remote-pi-companion.js";
|
||||
import { readNativeLocalProcessStop, PROCESS_START_REQUESTED } from "../native-local-process-stop.js";
|
||||
import { remoteLeaseCleanupScope } from "../remote-execution-termination.js";
|
||||
import { resolveConnectorAssignments, isConnectorSkill } from "../connector-runtime.js";
|
||||
import { connectorToolDefinitionsForSkills } from "../connector-runtime.js";
|
||||
import { activeNativeExecutionRunIds } from "./native-active-runs.js";
|
||||
import { preserveNativeFailedWorkspace } from "./native-failed-workspace-recovery.js";
|
||||
import {
|
||||
@@ -11447,8 +11447,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
input.execution.binding,
|
||||
));
|
||||
const pinnedSkills = new Set("runtimeContext" in input.execution ? input.execution.runtimeContext.skills.map((skill) => skill.key) : []);
|
||||
const connectorAssignments = [...pinnedSkills].some(isConnectorSkill)
|
||||
? await timeController("connector_assignments", () => resolveConnectorAssignments(input.db, input.execution.binding)) : [];
|
||||
const connectorAssignments = connectorToolDefinitionsForSkills(pinnedSkills);
|
||||
const reviewRun = await input.db.select({ contextSnapshot: heartbeatRuns.contextSnapshot })
|
||||
.from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, input.execution.binding.runId),
|
||||
@@ -11489,7 +11488,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
)).limit(1).then(rows => rows[0]) : undefined;
|
||||
const authority = new PaperclipRunnerToolAuthority(input.db, {
|
||||
...(nativeReview ? { nativeReview } : {}),
|
||||
connectorAssignments: connectorAssignments.filter((assignment) => pinnedSkills.has(assignment.skillKey)),
|
||||
connectorAssignments,
|
||||
assignedMcpTools,
|
||||
companyId: input.execution.binding.companyId,
|
||||
issueId: input.execution.binding.issueId,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { connectorToolDefinitionsForSkills } from "../connector-runtime.js";
|
||||
import * as cloudIdentity from "../cloud-runtime-identity.js";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { randomUUID } from "node:crypto";
|
||||
@@ -96,6 +97,18 @@ describe("PaperclipRunnerToolAuthority", () => {
|
||||
await temporary?.cleanup();
|
||||
});
|
||||
|
||||
it("registers pinned connector definitions without discovery and rejects revoked assignments at use", async () => {
|
||||
const connectorAssignments = connectorToolDefinitionsForSkills(new Set([
|
||||
"paperclipai/paperclip/slack", "paperclipai/paperclip/browser-use-cloud", "custom/agentmail",
|
||||
]));
|
||||
expect(connectorAssignments).toHaveLength(1);
|
||||
const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, connectorAssignments });
|
||||
expect(authority.definitions().some(tool => tool.name === "slack_history")).toBe(true);
|
||||
expect(authority.definitions().some(tool => String(tool.name).startsWith("agentmail_"))).toBe(false);
|
||||
await expect(authority.execute({ tool: "slack_history", callId: randomUUID(), arguments: { channel: "CEXAMPLE" } }))
|
||||
.rejects.toThrow("This connector is no longer assigned or authorized");
|
||||
});
|
||||
|
||||
it("advertises only real bindings and reads the bound task", async () => {
|
||||
const authority = new PaperclipRunnerToolAuthority(db, {
|
||||
companyId,
|
||||
|
||||
@@ -139,7 +139,7 @@ type Binding = {
|
||||
apiUrl?: string;
|
||||
storage?: StorageService;
|
||||
/** Server-owned suppression for baseline evals; true never overrides operator opt-in. */
|
||||
connectorAssignments?: ConnectorAssignment[];
|
||||
connectorAssignments?: Pick<ConnectorAssignment, "tools">[];
|
||||
assignedMcpTools?: Awaited<ReturnType<typeof createAssignedMcpTools>>;
|
||||
apiToolsEnabled?: boolean;
|
||||
workMode?: "standard" | "planning" | "ask";
|
||||
|
||||
@@ -1328,7 +1328,9 @@ export function createToolGatewayService(
|
||||
|
||||
async function connectedMcpToolsForCompany(
|
||||
companyId: string,
|
||||
selectedConnectionIds?: string[],
|
||||
): Promise<ToolGatewayDescriptor[]> {
|
||||
if (selectedConnectionIds?.length === 0) return [];
|
||||
// Read each connection and application row once. Selecting them in the
|
||||
// catalog join repeats the full connection row (config included) for
|
||||
// every tool, and one connection can expose hundreds of tools. Both reads
|
||||
@@ -1354,6 +1356,7 @@ export function createToolGatewayService(
|
||||
eq(toolCatalogEntries.status, "active"),
|
||||
isNull(toolCatalogEntries.quarantinedAt),
|
||||
connectedMcpConnectionFilter(companyId),
|
||||
selectedConnectionIds ? inArray(toolConnections.id, selectedConnectionIds) : undefined,
|
||||
),
|
||||
)
|
||||
.orderBy(toolConnections.name, toolCatalogEntries.name);
|
||||
@@ -9135,19 +9138,30 @@ export function createToolGatewayService(
|
||||
|
||||
async browserUseResources(binding: { companyId: string; agentId: string; runId?: string; issueId?: string }) {
|
||||
if (!binding.runId || !binding.issueId) return [];
|
||||
const [connection] = await measureRunPhase("connectors.browser_use.connection", () => db.select({ id: toolConnections.id }).from(toolConnections).where(and(
|
||||
const connections = await measureRunPhase("connectors.browser_use.connection", () => db.select({ id: toolConnections.id }).from(toolConnections).where(and(
|
||||
eq(toolConnections.companyId, binding.companyId), eq(toolConnections.transport, "rest_api"),
|
||||
eq(toolConnections.enabled, true), eq(toolConnections.status, "active"),
|
||||
sql`${toolConnections.config}->>'sourceTemplateKey' = 'browser-use-cloud'`,
|
||||
)).limit(1));
|
||||
if (!connection) return [];
|
||||
)).orderBy(toolConnections.name, toolConnections.id));
|
||||
if (!connections.length) return [];
|
||||
const session: ToolGatewaySession = { ...binding, runId: binding.runId, issueId: binding.issueId, id: randomUUID(), token: "", projectId: null, createdAt: new Date(), expiresAt: new Date(Date.now() + 60000) };
|
||||
const tools = (await measureRunPhase("connectors.browser_use.catalog", () => listToolsForContext(session))).filter(t => t.providerType === "provider_rest");
|
||||
await assertAgentInCompany(binding.companyId, binding.agentId);
|
||||
if (await githubGuestBotConnectionForSession(db, session)) return [];
|
||||
const tools = await measureRunPhase("connectors.browser_use.catalog", async () => {
|
||||
const candidates = await connectedMcpToolsForCompany(binding.companyId, connections.map(connection => connection.id));
|
||||
const decisions = await decideToolsForListing(candidates, tool => policyInputForTool({ session, tool }));
|
||||
return decisions.filter(({ decision }) => decision.allowed || decision.decision === "require_approval").map(({ tool }) => tool);
|
||||
});
|
||||
const visible = new Set(tools.map(tool => tool.connectionId));
|
||||
const resources: Array<{ id: string; label: string; connectionId: string }> = [];
|
||||
for (const connectionId of new Set(tools.map(t => t.connectionId!))) {
|
||||
const [connection] = await measureRunPhase("connectors.browser_use.connection", () => db.select().from(toolConnections).where(and(eq(toolConnections.companyId, binding.companyId), eq(toolConnections.id, connectionId))));
|
||||
for (const candidate of connections) {
|
||||
if (!visible.has(candidate.id)) continue;
|
||||
const [connection] = await measureRunPhase("connectors.browser_use.connection", () => db.select().from(toolConnections).where(and(
|
||||
eq(toolConnections.companyId, binding.companyId), eq(toolConnections.id, candidate.id),
|
||||
eq(toolConnections.enabled, true), eq(toolConnections.status, "active"),
|
||||
)));
|
||||
if (!connection || !isBrowserUseConnection(connection)) continue;
|
||||
try { await measureRunPhase("connectors.browser_use.grant", () => resolveConnectionGrant(session, connection, false)); resources.push({ id: connectionId, connectionId, label: "Browser Use Cloud" }); } catch { /* Unavailable grants do not contribute a runtime skill. */ }
|
||||
try { await measureRunPhase("connectors.browser_use.grant", () => resolveConnectionGrant(session, connection, false)); resources.push({ id: connection.id, connectionId: connection.id, label: "Browser Use Cloud" }); } catch { /* Unavailable grants do not contribute a runtime skill. */ }
|
||||
}
|
||||
return resources;
|
||||
},
|
||||
|
||||
Reference in new issue
Block a user