Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-29 04:25:52 +01:00
Code Issues Packages Projects Releases Wiki Activity

fix(server): schedule tool health from committed changes and deadlines (#17554)

Browse Source 
## Thinking Path

> - Paperclip manages AI-agent work and its connected tools.
> - Idle hosted instances need their database and network activity to
stop.
> - The maintenance tick queries tool connections even when no health
check is due.
> - Existing connection rows already contain eligibility and the last
health observation.
> - This pull request reconstructs health deadlines and refreshes them
after committed changes.
> - The shared coordinator preserves retries, drain, and shutdown
without another private timer.

## Linked Issues or Issue Description

**What happened?**

The aggregate maintenance tick scans connection health every 30 seconds.
Empty instances and connections with fresh health observations still
cause SQL activity.

**Expected behavior**

No eligible connection means no recurring health SQL. Eligible
connections retain the existing 15-minute proactive check. Configuration
changes and health observations update the schedule after commit.

**Steps to reproduce**

Start a server with heartbeat scheduling enabled and no tool
connections. Let startup finish and observe two maintenance ticks. The
old code repeats the health inventory query. The new startup regression
verifies that those ticks do not invoke the health inventory.

**Paperclip version or commit**

This branch includes merged parent #17518 and master
`96727b3cfefb61f85ffe1fbc29a9b6dd6837efe8`. The 47 changed paths belong
to the single tool-connection health duty. No database file differs from
master.

Related: #14638 covers runtime-auth exclusions. #15448, #15325, and
#11910 concern MCP failure classification. This PR preserves those
provider policies and does not replace their work.

## What Changed

- Keep the GitHub error-projection unit fixture at the health-write
boundary and verify the real transaction path in the PostgreSQL
health-worker suite.

- Add a health schedule owned by tool access. Read lean eligibility
fields and saved observation timestamps.
- Register health work with the existing coordinator. Refresh only dirty
connection/application/company scopes using bounded inventory pages,
check due connections in fair company batches, continue concrete due
backlog after one second, and retain failures with backoff.
- Notify before supported eligibility and observation writes in their
owning transactions. Cover API/gallery setup, OAuth, catalog/gateway
observations, email, voice, access removal, deletion, and settings.
- Commit connection creation with grant, binding, and runtime-slot SQL
before its wake. Keep provider calls outside the new transactions.
- Keep provider-to-transport mapping in the chat catalog. SDK-only chat
changes do not wake health work.
- Replace aggregate health scans, retain the public manual sweep, and
document the producer audit and remaining sleep limits.
- Keep metadata-only example and Smoke Lab installs scoped. Lock and
re-read application type before publishing a real eligibility change.
- Keep failed inventory reads on their own retry deadline while healthy
due batches continue.
- Apply fixture migrations before using external disposable databases in
all six producer suites.

## Verification

Published head: `8c1d293d1ae2b7db753ab20f5a2a2edec1fb0a62`. Fresh CI and
reviews are required for this exact head.

- **623 distinct targeted cases pass.** The final four complete health,
fair-batching, GitHub health recovery, and startup suites pass 80 cases
after the retry fix. Five complete producer suites pass 543 cases: tool
access 469, access 18, email 45, GitHub events four, and Smoke Lab
seven. The earlier 26 health cases are part of this total, not
additional cases.
- The six producer suites first passed against separate, initially empty
PostgreSQL 17 databases at `acbf262843`. They apply their own
migrations. The subsequent repair changes only the health worker and its
regression; its complete final consumer suites pass on that integrated
parent. Server typecheck and build pass with the final repair.
- **Installer amplification:** five real installs with 205 unrelated
connections reproduced 1,035 example rows and 1,045 Smoke Lab rows.
Repeated metadata installs now load only changed fixture connections.
The same cases restore an excluded application through each installer
and require all 205 unrelated IDs to re-enter the schedule.
- **Concurrent eligibility:** a real PostgreSQL lock-wait case requires
the installer to notify after a concurrent type change. A mutation that
retains the lock but uses the old lookup fails the settled-hint
assertion; original source was restored byte for byte.
- **Retry isolation:** the failing baseline read one unavailable company
three times while another company's 60 due checks drained. Final tests
require one initial read, at least 30 seconds between failed reads,
prompt healthy progress, eventual recovery, and SQL quiet after
settlement.
- Real-service producer coverage includes AgentMail activation/pause,
GitHub installation changes, and member access removal. Lifecycle
coverage includes outer commit/rollback, startup reconstruction,
drain/resume, failure, and shutdown.
- Failure history and exact evidence scope: [health
acceptance](https://github.com/paperclipai/paperclip/blob/8c1d293d1ae2b7db753ab20f5a2a2edec1fb0a62/doc/idle-tool-connection-health.md#acceptance-follow-up-after-the-plugin-parent-merged-october-27),
[producer
audit](https://github.com/paperclipai/paperclip/blob/8c1d293d1ae2b7db753ab20f5a2a2edec1fb0a62/doc/plans/2026-10-24-tool-connection-health-sleep-audit.md).
Prior remote failures and earlier 706-case evidence remain historical.
Startup tests emitted listener-count warnings; all assertions passed. No
full local repository-suite, deployed sleep, fleet savings, or live
provider qualification claim is made.

## Risks

- Supported live writers must use the transaction signals. Independent
SQL writers and other replicas need their own wake integration.
- Configured eligible connections still have a real proactive health
obligation. This does not prove Cloud alarm coverage or host/database
suspension.
- Existing manual probes and catalog operations can overlap. This
migration does not introduce a credential-version CAS or change provider
failure policy.
- Voice remains eligible under the previous health policy. Its generic
probe can still report an unsupported transport; changing that behavior
is separate work.
- No database migration, new table, workflow, or lockfile change. This
PR includes its merged parent and current inspected master; later stack
PRs and all current-head acceptance gates remain separate.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools, code execution, and
reasoning. The exact serving version and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-27 21:37:50 -05:00
1 parent 5cfa2bf26d
commit 94a780286f
47 files changed
+1795 -246
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

server/src/services/access.ts
+2
View File
@@ -1,3 +1,4 @@
import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js";
import { and, eq, inArray, ne, notInArray, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
@@ -323,6 +324,7 @@ export function accessService(db: Db) {
&& grant.credentialSecretRefs.length > 0
&& grant.credentialSecretRefs.every((ref) => !connectionSecretIdsToRemove.has(ref.secretId)),
);
if (!hasUnaffectedActiveGrant) await notifyToolConnectionHealth(tx, { companyId, id: connection.id });
await tx.update(toolConnections).set({
credentialRefs,
credentialSecretRefs,
server/src/services/chat-channels.ts
+2
View File
@@ -1,3 +1,4 @@
import { notifyChatConnectionHealth } from "./chat-channels/connection-health-notifications.js";
import { credentialFingerprint } from "./chat-channels/credential-fingerprint.js";
import { createChatSdkMessageReceiver } from "./chat-channels/runtime/sdk-messages.js";
import { admitDiscordSdkMessage } from "./chat-channels/discord/messages.js";
@@ -2295,6 +2296,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
updatedAt: new Date(),
})
.where(eq(chatConversations.id, conversation!.id));
await notifyChatConnectionHealth(taskTx, taskEndpoint);
await taskTx
.update(toolConnections)
.set({
server/src/services/chat-channels/configuration.ts
+7
View File
@@ -1,3 +1,4 @@
import { notifyChatConnectionHealth } from "./connection-health-notifications.js";
import type { ConnectionManagementContext } from "../connection-management.js";
import { CHAT_HUMAN_CREDENTIAL_FIELDS } from "./credentials.js";
import { preparePhotonConfiguration, createPhotonConfiguration } from "./imessage-photon/setup.js";
@@ -207,6 +208,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
updatedAt: pausedAt,
})
.where(eq(chatEndpoints.id, endpoint.id));
await notifyChatConnectionHealth(tx, endpoint);
await tx
.update(toolConnections)
.set({ status: "disabled", enabled: false, updatedAt: new Date() })
@@ -351,6 +353,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
})
.where(eq(chatEndpoints.id, endpoint.id))
.returning();
await notifyChatConnectionHealth(tx, endpoint);
await tx
.update(toolConnections)
.set({
@@ -396,6 +399,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
sql`coalesce((${chatEndpoints.setup}->>'runtimeGeneration')::integer, 0) = ${runtimeGeneration(activatedEndpoint.setup)}`,
),
);
await notifyChatConnectionHealth(tx, endpoint);
await tx
.update(toolConnections)
.set({
@@ -500,6 +504,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
.update(chatConversations)
.set({ state: "endpoint_removed", updatedAt: new Date() })
.where(eq(chatConversations.endpointId, endpoint.id));
await notifyChatConnectionHealth(tx, endpoint);
await tx
.update(toolConnections)
.set({ status: "archived", enabled: false, updatedAt: new Date() })
@@ -829,6 +834,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
updatedAt,
})
.where(eq(chatEndpoints.id, endpoint.id));
await notifyChatConnectionHealth(tx, endpoint);
await tx
.update(toolConnections)
.set({
@@ -891,6 +897,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
updatedAt: new Date(),
})
.where(eq(chatEndpoints.id, endpoint.id));
await notifyChatConnectionHealth(tx, endpoint);
await tx
.update(toolConnections)
.set({
server/src/services/chat-channels/connection-health-notifications.ts
+8
View File
@@ -0,0 +1,8 @@
import type { ChatProvider } from "@paperclipai/shared";
import { notifyToolConnectionHealth } from "../tool-connection-health-notifications.js";
import { chatConnectionTransport } from "./provider-catalog.js";
export async function notifyChatConnectionHealth(transaction: object, endpoint: { companyId: string; connectionId: string; provider: ChatProvider }) {
if (chatConnectionTransport(endpoint.provider) === "chat_sdk") return;
await notifyToolConnectionHealth(transaction, { companyId: endpoint.companyId, id: endpoint.connectionId });
}
server/src/services/chat-channels/endpoints.ts
+2 -1
View File
@@ -1,3 +1,4 @@
import { chatConnectionTransport } from "./provider-catalog.js";
import type { ConnectionManagementContext } from "../connection-management.js";
import { slackSetupState } from "./slack/setup.js";
import { assertChatExecutionDefaultsAccess } from "../chat-execution-defaults.js";
@@ -333,7 +334,7 @@ function providerSetupState(
connectionKind: "managed",
connectionPurpose: "channel",
ownership: "customer",
transport: input.provider === "speko" ? "voice" : "chat_sdk",
transport: chatConnectionTransport(input.provider),
authKind: "api_key",
credentialPolicy: "shared",
status: "draft",
server/src/services/chat-channels/interactions/callbacks.ts
+2
View File
@@ -1,3 +1,4 @@
import { notifyChatConnectionHealth } from "../connection-health-notifications.js";
import { slackQuestionFormValidationError } from "../slack/question-forms.js";
import { chatInteractionForm, chatItemVerdicts } from "../../chat-interaction-forms.js";
import { createDiscordQuestionForms } from "../discord/question-forms.js";
@@ -426,6 +427,7 @@ export function createChatInteractionCallbacks(db: Db, options: InteractionCallb
updatedAt: processedAt,
})
.where(eq(chatEndpoints.id, endpoint.id));
await notifyChatConnectionHealth(tx, currentEndpoint);
await tx
.update(toolConnections)
.set({
server/src/services/chat-channels/provider-catalog.ts
+6
View File
@@ -106,3 +106,9 @@ const CAPABILITIES: Record<ChatProvider, ChatAdapterCapabilities> = {
},
};
export { PROVIDER_LABELS, CAPABILITIES };
export function chatConnectionTransport(provider: ChatProvider): "voice" | "rest_api" | "chat_sdk" {
if (provider === "speko") return "voice";
if (provider === "agentmail") return "rest_api";
return "chat_sdk";
}
server/src/services/chat-channels/provider-effects.ts
+2
View File
@@ -1,3 +1,4 @@
import { notifyChatConnectionHealth } from "./connection-health-notifications.js";
import { createTelegramCallbackNoticeAuthority } from "./telegram/callback-notices.js";
import { slackThreadChannelId } from "./slack/messages.js";
@@ -325,6 +326,7 @@ export function createProviderEffects(db: Db, options: ProviderEffectsOptions) {
updatedAt: new Date(),
})
.where(eq(chatEndpoints.id, currentEndpoint.id));
await notifyChatConnectionHealth(tx, currentEndpoint);
await tx
.update(toolConnections)
.set({
server/src/services/chat-channels/publications/failures.ts
+2
View File
@@ -1,3 +1,4 @@
import { notifyChatConnectionHealth } from "../connection-health-notifications.js";
import { isOutboundAttachmentHydrationError, isOutboundAttachmentValidationError } from "../outbound-attachments.js";
import type { EndpointRow, DbTransaction, LifecycleRuntimeFence } from "../runtime-types.js";
@@ -197,6 +198,7 @@ export function createChatPublicationFailures(db: Db, options: PublicationFailur
updatedAt: new Date(),
})
.where(eq(chatEndpoints.id, currentEndpoint.id));
await notifyChatConnectionHealth(tx, currentEndpoint);
await tx
.update(toolConnections)
.set({
server/src/services/chat-channels/resource-reconciliation.ts
+3
View File
@@ -1,3 +1,4 @@
import { notifyChatConnectionHealth } from "./connection-health-notifications.js";
import type { ConnectionManagementContext } from "../connection-management.js";
import { imessageResourceParticipants } from "./imessage-photon/resources.js";
import { createGitHubResources, reconcileGitHubRepositoryCoordinates } from "./github/resources.js";
@@ -523,6 +524,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
updatedAt: now,
})
.where(eq(chatEndpoints.id, currentEndpoint.id));
await notifyChatConnectionHealth(tx, currentEndpoint);
await tx
.update(toolConnections)
.set({
@@ -552,6 +554,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
updatedAt: now,
})
.where(eq(chatEndpoints.id, currentEndpoint.id));
await notifyChatConnectionHealth(tx, currentEndpoint);
await tx
.update(toolConnections)
.set({
server/src/services/chat-channels/setup-probes.ts
+2
View File
@@ -1,3 +1,4 @@
import { notifyChatConnectionHealth } from "./connection-health-notifications.js";
import type { ConnectionManagementContext } from "../connection-management.js";
import type { EndpointRow, DbOrTransaction } from "./runtime-types.js";
@@ -329,6 +330,7 @@ export function createSetupProbes(db: Db, options: SetupProbesOptions) {
code: "chat_endpoint_test_superseded",
});
}
await notifyChatConnectionHealth(tx, endpoint);
await tx
.update(toolConnections)
.set({
server/src/services/company-deletion.ts
+2
View File
@@ -1,3 +1,4 @@
import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js";
import { notifyCompanyRetentionWork } from "./retention-work-notifications.js";
import { notifyToolReviewChange } from "./tool-review-work-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
@@ -119,6 +120,7 @@ export async function deleteCompany(db: Db, id: string) {
await tx.delete(executionWorkspaces).where(eq(executionWorkspaces.companyId, id));
await tx.delete(projects).where(eq(projects.companyId, id));
await agentLifecycleCompanyDeletion.deleteCompanyData(tx, id);
await notifyToolConnectionHealth(tx, id);
const rows = await tx
.delete(companies)
.where(eq(companies.id, id))
server/src/services/delivery-work-notifications.ts
+1 -1
View File
@@ -18,7 +18,7 @@ export const DELIVERY_QUEUES = {
decisionContinuation: "decision-continuation",
decisionArchiveDelivery: "decision-archive-delivery",
toolConnectionHealth: "tool-connection-health",
toolReviewProjection: "tool-review-projection",
toolReviewInterruption: "tool-review-interruption",
toolReviewRecovery: "tool-review-recovery",
server/src/services/email-channels.ts
+8 -4
View File
@@ -1,3 +1,4 @@
import { notifyToolConnectionHealth, writeToolConnectionHealthChange } from "./tool-connection-health-notifications.js";
import type { ConnectionManagementContext } from "./connection-management.js";
import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
@@ -923,6 +924,7 @@ export function emailChannelService(db: Db, options: EmailChannelOptions) {
updatedAt: now,
})
.where(eq(chatEndpoints.id, endpoint.id));
await notifyToolConnectionHealth(tx, { companyId: endpoint.companyId, id: endpoint.connectionId });
await tx
.update(toolConnections)
.set({ status: "active", enabled: true, healthStatus: "ok" })
@@ -1994,7 +1996,7 @@ export function emailChannelService(db: Db, options: EmailChannelOptions) {
.update(chatEndpoints)
.set({ lastError: null, healthMessage: "Connected" })
.where(eq(chatEndpoints.id, endpoint.id));
await db
await writeToolConnectionHealthChange(db, { companyId: endpoint.companyId, id: endpoint.connectionId }, tx => tx
.update(toolConnections)
.set({
healthStatus: "ok",
@@ -2002,7 +2004,7 @@ export function emailChannelService(db: Db, options: EmailChannelOptions) {
lastError: null,
healthCheckedAt: new Date(),
})
.where(eq(toolConnections.id, endpoint.connectionId));
.where(eq(toolConnections.id, endpoint.connectionId)));
}
}
async function maintainSocket(endpoint: Endpoint) {
@@ -2102,7 +2104,7 @@ export function emailChannelService(db: Db, options: EmailChannelOptions) {
});
}
async function markError(endpoint: Endpoint, error: string) {
await db
await writeToolConnectionHealthChange(db, { companyId: endpoint.companyId, id: endpoint.connectionId }, tx => tx
.update(toolConnections)
.set({
healthStatus: "degraded",
@@ -2110,7 +2112,7 @@ export function emailChannelService(db: Db, options: EmailChannelOptions) {
lastError: error,
healthCheckedAt: new Date(),
})
.where(eq(toolConnections.id, endpoint.connectionId));
.where(eq(toolConnections.id, endpoint.connectionId)));
await db
.update(chatEndpoints)
.set({ lastError: error, healthMessage: error, updatedAt: new Date() })
@@ -2372,6 +2374,7 @@ export function emailChannelService(db: Db, options: EmailChannelOptions) {
.update(chatEndpoints)
.set({ status: "paused" })
.where(eq(chatEndpoints.id, endpoint.id));
await notifyToolConnectionHealth(tx, { companyId: endpoint.companyId, id: endpoint.connectionId });
await tx
.update(toolConnections)
.set({ enabled: false, status: "disabled" })
@@ -2467,6 +2470,7 @@ export function emailChannelService(db: Db, options: EmailChannelOptions) {
updatedAt: new Date(),
})
.where(eq(chatEndpoints.id, id));
await notifyToolConnectionHealth(tx, { companyId: endpoint.companyId, id: endpoint.connectionId });
await tx
.update(toolConnections)
.set({
server/src/services/email-connections.ts
+3 -2
View File
@@ -1,3 +1,4 @@
import { writeToolConnectionHealthChange } from "./tool-connection-health-notifications.js";
import { randomUUID } from "node:crypto";
import { and, eq, isNull, ne, sql } from "drizzle-orm";
import {
@@ -268,7 +269,7 @@ export function emailConnectionService(
actorSource: actor.localImplicit ? "local_implicit" : "session",
},
);
await db
await writeToolConnectionHealthChange(db, connection, tx => tx
.update(toolConnections)
.set({
uid: `agentmail-account-${input.idempotencyKey}`,
@@ -276,7 +277,7 @@ export function emailConnectionService(
healthMessage: "Connected",
healthCheckedAt: new Date(),
})
.where(eq(toolConnections.id, connection.id));
.where(eq(toolConnections.id, connection.id)));
if (input.grantKind === "user")
await db
.update(connectionGrants)
server/src/services/github-connection-events.ts
+2
View File
@@ -1,3 +1,4 @@
import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js";
import { dispatchGitHubBotCloudEvent } from "./chat-github-cloud-ingress.js";
import { chatEndpoints } from "@paperclipai/db";
import { randomUUID } from "node:crypto";
@@ -328,6 +329,7 @@ export function githubConnectionEventService(
};
await database.update(connectionGrants).set({ providerTenant, updatedAt: now() })
.where(and(eq(connectionGrants.id, binding.grantId), eq(connectionGrants.companyId, binding.companyId)));
await notifyToolConnectionHealth(database, { companyId: binding.companyId, id: binding.connectionId });
await database.update(toolConnections).set({
healthStatus: unavailable ? "failed" : "ok",
healthMessage: unavailable
server/src/services/instance-settings.ts
+2
View File
@@ -1,3 +1,4 @@
import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import type { Db } from "@paperclipai/db";
import { companies, instanceSettings } from "@paperclipai/db";
@@ -613,6 +614,7 @@ export function instanceSettingsService(db: Db, options: InstanceSettingsService
await notifyDeliveryWork(tx, DELIVERY_QUEUES.mcpEvents, "*");
await notifyDeliveryWork(tx, DELIVERY_QUEUES.dotEvents, "*");
await notifyDeliveryWork(tx, DELIVERY_QUEUES.grokDispatch);
await notifyToolConnectionHealth(tx, "*");
return tx
.update(instanceSettings)
.set({
server/src/services/smoke-lab.ts
+58 -46
View File
@@ -1,3 +1,4 @@
import { notifyToolConnectionHealth, writeToolConnectionHealthChange } from "./tool-connection-health-notifications.js";
import { spawn, type ChildProcess } from "node:child_process";
import { createHash, randomUUID } from "node:crypto";
import { existsSync } from "node:fs";
@@ -690,14 +691,20 @@ export function smokeLabService(db: Db, options: {
));
const now = new Date();
if (existing) {
const [updated] = await db.update(toolApplications).set({
name: input.name,
description: input.description,
type: input.type,
status: "active",
metadata: { smokeLab: true },
updatedAt: now,
}).where(eq(toolApplications.id, existing.id)).returning();
const [updated] = await db.transaction(async tx => {
const [current] = await tx.select({ type: toolApplications.type }).from(toolApplications)
.where(and(eq(toolApplications.id, existing.id), eq(toolApplications.companyId, input.companyId)))
.for("no key update");
if (current && current.type !== input.type) await notifyToolConnectionHealth(tx, { companyId: input.companyId, applicationId: existing.id });
return tx.update(toolApplications).set({
name: input.name,
description: input.description,
type: input.type,
status: "active",
metadata: { smokeLab: true },
updatedAt: now,
}).where(eq(toolApplications.id, existing.id)).returning();
});
return { row: updated ?? existing, created: false };
}
const [created] = await db.insert(toolApplications).values({
@@ -742,41 +749,45 @@ export function smokeLabService(db: Db, options: {
lastHealthAt: now,
updatedAt: now,
};
const ensureDefaultOrganizationGrant = async (connectionId: string) => {
const [existingGrant] = await db.select({ id: connectionGrants.id }).from(connectionGrants).where(and(
eq(connectionGrants.companyId, input.companyId),
eq(connectionGrants.connectionId, connectionId),
eq(connectionGrants.kind, "organization"),
eq(connectionGrants.isDefault, true),
));
if (existingGrant) return;
await db.insert(connectionGrants).values({
const connectionId = existing?.id ?? randomUUID();
return writeToolConnectionHealthChange(db, { companyId: input.companyId, id: connectionId }, async tx => {
const ensureDefaultOrganizationGrant = async (connectionId: string) => {
const [existingGrant] = await tx.select({ id: connectionGrants.id }).from(connectionGrants).where(and(
eq(connectionGrants.companyId, input.companyId),
eq(connectionGrants.connectionId, connectionId),
eq(connectionGrants.kind, "organization"),
eq(connectionGrants.isDefault, true),
));
if (existingGrant) return;
await tx.insert(connectionGrants).values({
companyId: input.companyId,
connectionId,
kind: "organization",
status: "active",
isDefault: true,
credentialSecretRefs: [],
createdAt: now,
updatedAt: now,
});
};
if (existing) {
const [updated] = await tx.update(toolConnections).set(values).where(eq(toolConnections.id, existing.id)).returning();
await ensureDefaultOrganizationGrant(existing.id);
return { row: updated ?? existing, created: false };
}
const [created] = await tx.insert(toolConnections).values({
id: connectionId,
companyId: input.companyId,
connectionId,
kind: "organization",
status: "active",
isDefault: true,
credentialSecretRefs: [],
name: input.name,
uid: `smoke-lab/${input.name.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "")}`,
...values,
createdByAgentId: input.actor?.actorType === "agent" ? input.actor.agentId : null,
createdByUserId: input.actor?.actorType === "user" ? input.actor.actorId : null,
createdAt: now,
updatedAt: now,
});
};
if (existing) {
const [updated] = await db.update(toolConnections).set(values).where(eq(toolConnections.id, existing.id)).returning();
await ensureDefaultOrganizationGrant(existing.id);
return { row: updated ?? existing, created: false };
}
const [created] = await db.insert(toolConnections).values({
companyId: input.companyId,
name: input.name,
uid: `smoke-lab/${input.name.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "")}`,
...values,
createdByAgentId: input.actor?.actorType === "agent" ? input.actor.agentId : null,
createdByUserId: input.actor?.actorType === "user" ? input.actor.actorId : null,
createdAt: now,
}).returning();
await ensureDefaultOrganizationGrant(created.id);
return { row: created, created: true };
}).returning();
await ensureDefaultOrganizationGrant(created.id);
return { row: created, created: true };
});
}
async function ensureStdioTemplate(companyId: string, actor?: SmokeLabActorInfo) {
@@ -963,20 +974,21 @@ export function smokeLabService(db: Db, options: {
eq(toolConnections.name, HTTP_CONNECTION_NAME),
));
if (!connection || !httpSidecar) return;
const sidecarUrl = httpSidecar.url;
const now = new Date();
await db.update(toolConnections).set({
await writeToolConnectionHealthChange(db, connection, tx => tx.update(toolConnections).set({
config: {
...asRecord(connection.config),
url: `${httpSidecar.url}/mcp`,
catalogUrl: `${httpSidecar.url}/catalog`,
toolCallUrl: `${httpSidecar.url}/tools/call`,
url: `${sidecarUrl}/mcp`,
catalogUrl: `${sidecarUrl}/catalog`,
toolCallUrl: `${sidecarUrl}/tools/call`,
},
healthStatus: "ok",
healthMessage: "Smoke Lab HTTP fixture sidecar is running.",
healthCheckedAt: now,
lastHealthAt: now,
updatedAt: now,
}).where(eq(toolConnections.id, connection.id));
}).where(eq(toolConnections.id, connection.id)));
}
async function startHttpSidecar(companyId?: string) {
server/src/services/tool-access.ts
+106 -82
View File
@@ -1,3 +1,5 @@
import { notifyToolConnectionHealth, writeToolConnectionHealthChange } from "./tool-connection-health-notifications.js";
import { connectionHealthSchedule } from "./tool-access/catalog/health-schedule.js";
import type { ConnectionManagementContext } from "./connection-management.js";
import { connectionActorUserId } from "./tool-access/actor.js";
import { restConnectorFor } from "./connectors/rest-registry.js";
@@ -1219,8 +1221,12 @@ export function toolAccessService(
safeDefault: true,
};
if (existing) {
const [updated] = await db
.update(toolApplications)
const [updated] = await db.transaction(async tx => {
const [current] = await tx.select({ type: toolApplications.type }).from(toolApplications)
.where(and(eq(toolApplications.id, existing.id), eq(toolApplications.companyId, companyId)))
.for("no key update");
if (current && current.type !== "mcp_stdio") await notifyToolConnectionHealth(tx, { companyId, applicationId: existing.id });
return tx.update(toolApplications)
.set({
name: definition.applicationName,
description: definition.applicationDescription,
@@ -1232,6 +1238,7 @@ export function toolAccessService(
})
.where(eq(toolApplications.id, existing.id))
.returning();
});
return { row: updated, created: false };
}
const [created] = await db
@@ -1262,11 +1269,46 @@ export function toolAccessService(
quarantineNewEntries: true,
};
if (existing) {
const [updated] = await db
.update(toolConnections)
.set({
const updated = await writeToolConnectionHealthChange(db, existing, async tx => {
const [updated] = await tx
.update(toolConnections)
.set({
applicationId,
name: definition.connectionName,
transport: "local_stdio",
status: "active",
enabled: true,
config,
transportConfig: config,
credentialRefs: [],
credentialSecretRefs: [],
updatedAt: new Date(),
})
.where(eq(toolConnections.id, existing.id))
.returning();
await ensureDefaultOrganizationGrant(updated, tx);
await syncCredentialBindings(updated, [], tx);
await createToolRuntimeSlots(tx, runtimeSupervisor).ensureRuntimeSlot(updated);
return updated;
});
emitConnectionUpdated(updated, existing, "example");
return { row: updated, created: false };
}
const connectionId = randomUUID();
const created = await writeToolConnectionHealthChange(db, { companyId, id: connectionId }, async tx => {
const [created] = await tx
.insert(toolConnections)
.values({
id: connectionId,
companyId,
applicationId,
name: definition.connectionName,
uid: connectionUid(
"paperclip",
definition.connectionName,
connectionId,
),
connectionKind: "managed",
transport: "local_stdio",
status: "active",
enabled: true,
@@ -1274,42 +1316,13 @@ export function toolAccessService(
transportConfig: config,
credentialRefs: [],
credentialSecretRefs: [],
updatedAt: new Date(),
})
.where(eq(toolConnections.id, existing.id))
.returning();
await ensureDefaultOrganizationGrant(updated);
await syncCredentialBindings(updated);
await ensureRuntimeSlot(updated);
emitConnectionUpdated(updated, existing, "example");
return { row: updated, created: false };
}
const connectionId = randomUUID();
const [created] = await db
.insert(toolConnections)
.values({
id: connectionId,
companyId,
applicationId,
name: definition.connectionName,
uid: connectionUid(
"paperclip",
definition.connectionName,
connectionId,
),
connectionKind: "managed",
transport: "local_stdio",
status: "active",
enabled: true,
config,
transportConfig: config,
credentialRefs: [],
credentialSecretRefs: [],
})
.returning();
await ensureDefaultOrganizationGrant(created);
await syncCredentialBindings(created);
await ensureRuntimeSlot(created);
await ensureDefaultOrganizationGrant(created, tx);
await syncCredentialBindings(created, [], tx);
await createToolRuntimeSlots(tx, runtimeSupervisor).ensureRuntimeSlot(created);
return created;
});
emitConnectionCreated(created, "example");
return { row: created, created: true };
}
@@ -2475,7 +2488,7 @@ export function toolAccessService(
personalIdentityUserId || dedicatedAgentId ? [] : credentialSecretRefs;
if (revivedConnectionPrevious) {
forgetMcpHttpSessions(revivedConnectionPrevious.id);
[connectionRow] = await db
[connectionRow] = await writeToolConnectionHealthChange(db, revivedConnectionPrevious, tx => tx
.update(toolConnections)
.set({
name,
@@ -2497,7 +2510,7 @@ export function toolAccessService(
updatedAt: new Date(),
})
.where(eq(toolConnections.id, revivedConnectionPrevious.id))
.returning();
.returning());
} else {
const connectionId = randomUUID();
[connectionRow] = await db
@@ -2920,6 +2933,7 @@ export function toolAccessService(
connectionSetupMutationFingerprint(attemptedConnection),
);
if (connectionMutationIsStillCurrent) {
await notifyToolConnectionHealth(tx, revivedConnectionPrevious);
await tx
.update(toolConnections)
.set({
@@ -2995,7 +3009,7 @@ export function toolAccessService(
// surface the restoration failure instead of returning only the
// original provider error.
try {
await db
await writeToolConnectionHealthChange(db, revivedConnectionPrevious, tx => tx
.update(toolConnections)
.set({
status: "draft",
@@ -3011,7 +3025,7 @@ export function toolAccessService(
eq(toolConnections.id, revivedConnectionPrevious.id),
eq(toolConnections.companyId, companyId),
),
);
));
} catch (quarantineError) {
identityRollbackError = new AggregateError(
[rollbackError, quarantineError],
@@ -3020,9 +3034,10 @@ export function toolAccessService(
}
}
} else if (connectionRow) {
await db
const failedConnectionId = connectionRow.id;
await writeToolConnectionHealthChange(db, { companyId, id: failedConnectionId }, tx => tx
.delete(toolConnections)
.where(eq(toolConnections.id, connectionRow.id))
.where(eq(toolConnections.id, failedConnectionId)))
.catch(() => undefined);
}
if (
@@ -3136,6 +3151,7 @@ export function toolAccessService(
await tx.select({ id: toolApplications.id }).from(toolApplications)
.where(and(eq(toolApplications.id, connection.applicationId), eq(toolApplications.companyId, companyId)))
.for("update");
await notifyToolConnectionHealth(tx, connection);
const [row] = await tx.update(toolConnections).set({ config, transportConfig: config, status: "active", enabled: true, ...(input.agentInstructions !== undefined ? { agentInstructions: input.agentInstructions } : {}), updatedAt: new Date() })
.where(and(eq(toolConnections.id, connection.id), eq(toolConnections.companyId, companyId))).returning();
await tx.update(toolApplications).set({ status: "active", updatedAt: new Date() }).where(and(eq(toolApplications.id, connection.applicationId), eq(toolApplications.companyId, companyId)));
@@ -3493,6 +3509,7 @@ export function toolAccessService(
},
tx,
);
await notifyToolConnectionHealth(tx, connection);
const [updatedConnection] = await tx
.update(toolConnections)
.set({ status: "active", enabled: true, ...(input.agentInstructions !== undefined ? { agentInstructions: input.agentInstructions } : {}), updatedAt: new Date() })
@@ -4525,40 +4542,43 @@ export function toolAccessService(
await actor?.managementContext?.reauthorize();
const connectionId = randomUUID();
const binding = actorBinding(actor);
const [row] = await db
.insert(toolConnections)
.values({
id: connectionId,
companyId,
applicationId,
name: input.name,
uid: connectionUid(applicationNamespace, input.name, connectionId),
connectionKind: input.connectionKind ?? "managed",
ownership: input.ownership ?? "customer",
transport,
authKind: input.authKind ?? "none",
credentialPolicy:
input.credentialPolicy ??
(input.authKind === "oauth" ? "per_user" : "shared"),
status: input.status ?? "draft",
enabled: input.enabled ?? false,
agentInstructions: input.agentInstructions !== undefined ? input.agentInstructions : defaultConnectionAgentInstructions(
typeof config.sourceTemplateKey === "string" ? getConnectableAppDefinition(config.sourceTemplateKey)?.agentInstructions : null,
),
config,
transportConfig: isGoogleSheetsConnectionConfig(config)
? config
: (input.transportConfig ?? config),
credentialRefs: input.credentialRefs ?? [],
credentialSecretRefs: input.credentialSecretRefs ?? [],
createdByAgentId:
binding.actorType === "agent" ? binding.actorId : null,
createdByUserId: connectionActorUserId(actor),
})
.returning();
await ensureDefaultOrganizationGrant(row);
await syncCredentialBindings(row);
await ensureRuntimeSlot(row);
const row = await writeToolConnectionHealthChange(db, { companyId, id: connectionId }, async tx => {
const [row] = await tx
.insert(toolConnections)
.values({
id: connectionId,
companyId,
applicationId,
name: input.name,
uid: connectionUid(applicationNamespace, input.name, connectionId),
connectionKind: input.connectionKind ?? "managed",
ownership: input.ownership ?? "customer",
transport,
authKind: input.authKind ?? "none",
credentialPolicy:
input.credentialPolicy ??
(input.authKind === "oauth" ? "per_user" : "shared"),
status: input.status ?? "draft",
enabled: input.enabled ?? false,
agentInstructions: input.agentInstructions !== undefined ? input.agentInstructions : defaultConnectionAgentInstructions(
typeof config.sourceTemplateKey === "string" ? getConnectableAppDefinition(config.sourceTemplateKey)?.agentInstructions : null,
),
config,
transportConfig: isGoogleSheetsConnectionConfig(config)
? config
: (input.transportConfig ?? config),
credentialRefs: input.credentialRefs ?? [],
credentialSecretRefs: input.credentialSecretRefs ?? [],
createdByAgentId:
binding.actorType === "agent" ? binding.actorId : null,
createdByUserId: connectionActorUserId(actor),
})
.returning();
await ensureDefaultOrganizationGrant(row, tx);
await syncCredentialBindings(row, [], tx);
await createToolRuntimeSlots(tx, runtimeSupervisor).ensureRuntimeSlot(row);
return row;
});
emitConnectionCreated(row, "api");
return toConnection(row);
},
@@ -4651,9 +4671,12 @@ export function toolAccessService(
await syncCredentialBindings(rows[0]!, [], tx);
return rows;
})
: await write(db);
if (existing.transport !== "chat_sdk") await syncCredentialBindings(row);
await ensureRuntimeSlot(row);
: await writeToolConnectionHealthChange(db, existing, async tx => {
const rows = await write(tx);
await syncCredentialBindings(rows[0]!, [], tx);
await createToolRuntimeSlots(tx, runtimeSupervisor).ensureRuntimeSlot(rows[0]!);
return rows;
});
emitConnectionUpdated(row, existing, "api");
return toConnection(row);
},
@@ -4704,6 +4727,7 @@ export function toolAccessService(
listAppsNeedingAttention,
sweepConnectionHealth,
scheduledHealth: connectionHealthSchedule(db, checkConnectionHealth),
sweepGitHubConnectionContinuity,
...catalogReader,
server/src/services/tool-access/catalog/health-schedule.ts
+62
View File
@@ -0,0 +1,62 @@
import { logger } from "../../../middleware/logger.js";
import { and, asc, eq, gt, ne, sql } from "drizzle-orm";
import { toolApplications, toolConnections, type Db } from "@paperclipai/db";
import { isConnectorFeatureEnabled } from "@paperclipai/shared";
import { instanceSettingsService } from "../../instance-settings.js";
import { connectionSetupApp } from "../connections/setup-policy.js";
import type { ConnectionHealth } from "./health.js";
export type ConnectionHealthDeadline = { id: string; companyId: string; applicationId: string; dueAt: number };
const HEALTH_FRESHNESS_MS = 15 * 60_000;
export function connectionHealthSchedule(db: Db, checkHealth: ConnectionHealth["checkConnectionHealth"]) {
async function load(companyId?: string, connectionId?: string, applicationId?: string, signal?: AbortSignal): Promise<ConnectionHealthDeadline[]> {
const deadlines: ConnectionHealthDeadline[] = [];
let cursor: string | undefined;
while (true) {
signal?.throwIfAborted();
const rows = await db.select({
id: toolConnections.id,
companyId: toolConnections.companyId,
applicationId: toolConnections.applicationId,
checkedAt: toolConnections.healthCheckedAt,
applicationKey: toolApplications.applicationKey,
sourceTemplateKey: sql<string | null>`coalesce(${toolConnections.config}->>'sourceTemplateKey', ${toolConnections.transportConfig}->>'sourceTemplateKey')`,
}).from(toolConnections).innerJoin(toolApplications, and(
eq(toolApplications.id, toolConnections.applicationId),
eq(toolApplications.companyId, toolConnections.companyId),
)).where(and(
eq(toolConnections.enabled, true), eq(toolConnections.status, "active"),
ne(toolConnections.transport, "chat_sdk"), ne(toolConnections.transport, "runtime_auth"),
ne(toolApplications.type, "paperclip_plugin"),
companyId === undefined ? undefined : eq(toolConnections.companyId, companyId),
connectionId === undefined ? undefined : eq(toolConnections.id, connectionId),
applicationId === undefined ? undefined : eq(toolConnections.applicationId, applicationId),
cursor === undefined ? undefined : gt(toolConnections.id, cursor),
)).orderBy(asc(toolConnections.id)).limit(100);
const definitions = rows.map(row => ({ row, app: connectionSetupApp(row.applicationKey, row.sourceTemplateKey) }));
const settings = definitions.some(({ app }) => app?.experimentalFeatureKey)
? await instanceSettingsService(db).getExperimental() : null;
deadlines.push(...definitions.filter(({ app }) => !app || !settings || isConnectorFeatureEnabled(app, settings))
.map(({ row }) => ({ id: row.id, companyId: row.companyId, applicationId: row.applicationId,
dueAt: row.checkedAt === null ? 0 : row.checkedAt.getTime() + HEALTH_FRESHNESS_MS })));
if (rows.length < 100) return deadlines;
cursor = rows[rows.length - 1].id;
}
}
async function check(id: string): Promise<ConnectionHealthDeadline | null> {
const [before] = await load(undefined, id);
if (!before || before.dueAt > Date.now()) return before ?? null;
let failure: unknown;
try { await checkHealth(id, { actorType: "system", actorId: "tool_health_sweep" }); }
catch (error) {
failure = error;
logger.warn({ companyId: before.companyId, connectionId: id }, "Scheduled tool connection health check failed");
}
const [after] = await load(before.companyId, id);
if (failure && after && after.dueAt <= Date.now()) throw failure;
return after ?? null;
}
return { load, check };
}
server/src/services/tool-access/catalog/health.ts
+3 -2
View File
@@ -1,3 +1,4 @@
import { writeToolConnectionHealthChange } from "../../tool-connection-health-notifications.js";
import { connectionActorUserId } from "../actor.js";
import { assertSupportedConnection } from "../connections/setup-policy.js";
import { connectionSetupUrl, connectionReconnectUrl } from "../identifiers.js";
@@ -47,7 +48,7 @@ export function createConnectionHealth(db: Db, { now, audit, getConnectionRow, d
message: string | null,
) {
const now = new Date();
const [updated] = await db
const [updated] = await writeToolConnectionHealthChange(db, connection, tx => tx
.update(toolConnections)
.set({
healthStatus: status,
@@ -58,7 +59,7 @@ export function createConnectionHealth(db: Db, { now, audit, getConnectionRow, d
updatedAt: now,
})
.where(eq(toolConnections.id, connection.id))
.returning();
.returning());
if (connection.transport === "local_stdio") {
await db
.update(toolRuntimeSlots)
server/src/services/tool-access/catalog/publication.ts
+3 -2
View File
@@ -1,3 +1,4 @@
import { writeToolConnectionHealthChange } from "../../tool-connection-health-notifications.js";
import { connectorToolPolicy } from "../../connectors/tool-policy.js";
import { connectionSetupUrl, connectionReconnectUrl } from "../identifiers.js";
import { createToolRuntimeSlots } from "../runtime/slots.js";
@@ -269,7 +270,7 @@ export function createToolCatalogPublication(db: Db, { now, audit, getConnection
: refreshOptions.enableAllByDefault
? { ...connection.transportConfig, quarantineNewEntries: false }
: connection.transportConfig;
const [updatedConnection] = await db
const [updatedConnection] = await writeToolConnectionHealthChange(db, connection, tx => tx
.update(toolConnections)
.set({
config: normalizedConfig,
@@ -287,7 +288,7 @@ export function createToolCatalogPublication(db: Db, { now, audit, getConnection
updatedAt: refreshedAt,
})
.where(eq(toolConnections.id, connection.id))
.returning();
.returning());
if (connection.transport === "local_stdio") {
await ensureRuntimeSlot(updatedConnection);
server/src/services/tool-access/connections/removal.ts
+2
View File
@@ -1,3 +1,4 @@
import { notifyToolConnectionHealth } from "../../tool-connection-health-notifications.js";
import { toConnection } from "./projection.js";
import { actorBinding } from "../actor.js";
import { vercelConnectResourcesFor } from "../vercel-credentials.js";
@@ -538,6 +539,7 @@ async function removeConnection(
.returning({ id: connectionTokenIssuances.id });
const archived = await db.transaction(async (tx) => {
await notifyToolConnectionHealth(tx, connection);
await tx.select({ id: toolApplications.id }).from(toolApplications)
.where(and(eq(toolApplications.id, connection.applicationId), eq(toolApplications.companyId, connection.companyId)))
.for("update");
server/src/services/tool-access/connections/setup-policy.ts
+6 -2
View File
@@ -14,6 +14,11 @@ export function assertSupportedConnection(connection: typeof toolConnections.$in
}
}
export function connectionSetupApp(applicationKey: unknown, sourceTemplateKey: unknown) {
return applicationKey != null ? getAppDefinitionForApplicationKey(applicationKey)
: typeof sourceTemplateKey === "string" ? getConnectableAppDefinition(sourceTemplateKey) : null;
}
export function createConnectionSetupPolicy(db: Db) {
async function assertExperimentalConnectorSetupEnabled(provider: unknown, existing = false) {
const app = typeof provider === "string" ? getConnectableAppDefinition(provider) : null;
@@ -36,8 +41,7 @@ export function createConnectionSetupPolicy(db: Db) {
resolvedApplicationKey = application?.applicationKey;
}
const marker = connection.config?.sourceTemplateKey ?? connection.transportConfig?.sourceTemplateKey;
const app = resolvedApplicationKey != null ? getAppDefinitionForApplicationKey(resolvedApplicationKey)
: typeof marker === "string" ? getConnectableAppDefinition(marker) : null;
const app = connectionSetupApp(resolvedApplicationKey, marker);
if (app) await assertExperimentalConnectorSetupEnabled(app.slug, connection.status !== "draft");
}
return { assertExperimentalConnectorSetupEnabled, assertConnectorConnectionSetupEnabled };
server/src/services/tool-access/oauth/access-finalization.ts
+33 -27
View File
@@ -1,3 +1,4 @@
import { notifyToolConnectionHealth, writeToolConnectionHealthChange } from "../../tool-connection-health-notifications.js";
import { connectionActorUserId } from "../actor.js";
import type { ActorInfo } from "../types.js";
import { connectionMethodForConnection } from "../connection-methods.js";
@@ -114,33 +115,37 @@ async function finalizeOAuthAccess(
) {
throw conflict("This connection already uses a company identity");
}
[connection] = await db
.update(toolConnections)
.set({
credentialPolicy: "per_user",
credentialRefs: connection.credentialRefs.filter(
(ref) => ref.name !== "oauth.access_token",
),
credentialSecretRefs: connection.credentialSecretRefs.filter(
(ref) =>
ref.configPath !== "oauth.access_token" &&
ref.configPath !== "oauth.refresh_token",
),
status: "active",
enabled: true,
updatedAt: new Date(),
})
.where(
and(
eq(toolConnections.id, connection.id),
eq(toolConnections.companyId, companyId),
),
)
.returning();
await syncCredentialBindings(
connection,
personalGrant.credentialSecretRefs,
);
connection = await writeToolConnectionHealthChange(db, connection, async tx => {
const [updated] = await tx
.update(toolConnections)
.set({
credentialPolicy: "per_user",
credentialRefs: connection.credentialRefs.filter(
(ref) => ref.name !== "oauth.access_token",
),
credentialSecretRefs: connection.credentialSecretRefs.filter(
(ref) =>
ref.configPath !== "oauth.access_token" &&
ref.configPath !== "oauth.refresh_token",
),
status: "active",
enabled: true,
updatedAt: new Date(),
})
.where(
and(
eq(toolConnections.id, connection.id),
eq(toolConnections.companyId, companyId),
),
)
.returning();
await syncCredentialBindings(
updated,
personalGrant.credentialSecretRefs,
tx,
);
return updated;
});
} else if (
connection.credentialPolicy !== "shared" ||
connection.credentialSecretRefs.length === 0
@@ -325,6 +330,7 @@ async function finalizeOAuthAccess(
updatedAt: nowAt,
})
.where(eq(connectionGrants.id, personalGrant.id));
await notifyToolConnectionHealth(tx, connection);
[connection] = await tx
.update(toolConnections)
.set({
server/src/services/tool-access/oauth/cloud-callback.ts
+2
View File
@@ -1,3 +1,4 @@
import { notifyToolConnectionHealth } from "../../tool-connection-health-notifications.js";
import { managedConnectorProfile } from "./client-settings.js";
import type { ActorInfo } from "../types.js";
import { oauthConfig, normalizeOauthScopes } from "./metadata.js";
@@ -356,6 +357,7 @@ export function createCloudOAuthCallback(db: Db, dependencies: Dependencies) {
scopes: [...profile.scopes],
},
};
await notifyToolConnectionHealth(tx, connection);
[connection] = await tx
.update(toolConnections)
.set({
server/src/services/tool-access/oauth/connection-refresh.ts
+3 -2
View File
@@ -1,3 +1,4 @@
import { writeToolConnectionHealthChange } from "../../tool-connection-health-notifications.js";
import type { ActorInfo } from "../types.js";
import { createOAuthClientProtocol, storedOAuthTokenEndpointAuthMethod } from "./client.js";
import { oauthConfig, normalizeOauthScopes } from "./metadata.js";
@@ -59,7 +60,7 @@ const oauthRefreshFlights = new Map<string, Promise<unknown>>();
reauthorizationRequiredAt: now().toISOString(),
},
};
const [updated] = await db
const [updated] = await writeToolConnectionHealthChange(db, connection, tx => tx
.update(toolConnections)
.set({
status: "draft",
@@ -87,7 +88,7 @@ const oauthRefreshFlights = new Map<string, Promise<unknown>>();
)`,
),
)
.returning();
.returning());
if (updated) {
await syncCredentialBindings(updated);
emitConnectionUpdated(updated, connection, "credential_refresh");
server/src/services/tool-access/oauth/direct-callback.ts
+3
View File
@@ -1,3 +1,4 @@
import { notifyToolConnectionHealth } from "../../tool-connection-health-notifications.js";
import { credentialScope } from "./client-settings.js";
import { withoutOAuthRefreshLease } from "./refresh-leases.js";
import type { ActorInfo } from "../types.js";
@@ -332,6 +333,7 @@ export function createDirectOAuthCallback(db: Db, dependencies: Dependencies) {
},
},
};
await notifyToolConnectionHealth(tx, connection);
const [updatedConnection] = await tx
.update(toolConnections)
.set({
@@ -611,6 +613,7 @@ export function createDirectOAuthCallback(db: Db, dependencies: Dependencies) {
},
},
};
await notifyToolConnectionHealth(tx, connection);
const [updatedConnection] = await tx
.update(toolConnections)
.set({
server/src/services/tool-access/oauth/grant-refresh.ts
+3 -2
View File
@@ -1,3 +1,4 @@
import { writeToolConnectionHealthChange } from "../../tool-connection-health-notifications.js";
import type { ActorInfo } from "../types.js";
import { createOAuthClientProtocol, storedOAuthTokenEndpointAuthMethod } from "./client.js";
import { oauthConfig, normalizeOauthScopes, resolveGrantedOauthScopes } from "./metadata.js";
@@ -393,7 +394,7 @@ export function createOAuthGrantRefresh(db: Db, { now, store, leases, currentClo
connection.id,
connection.companyId,
);
const [reauthorizationRequired] = await db
const [reauthorizationRequired] = await writeToolConnectionHealthChange(db, latestConnection, tx => tx
.update(toolConnections)
.set({
status: "draft",
@@ -417,7 +418,7 @@ export function createOAuthGrantRefresh(db: Db, { now, store, leases, currentClo
eq(toolConnections.companyId, latestConnection.companyId),
),
)
.returning();
.returning());
await syncCredentialBindings(reauthorizationRequired);
emitConnectionUpdated(
reauthorizationRequired,
server/src/services/tool-access/oauth/vercel-callback.ts
+3 -2
View File
@@ -1,3 +1,4 @@
import { writeToolConnectionHealthChange } from "../../tool-connection-health-notifications.js";
import { connectionActorUserId } from "../actor.js";
import { vercelConnectHttpError, vercelCredentialFor, vercelConnectResourcesFor } from "../vercel-credentials.js";
import type { ActorInfo } from "../types.js";
@@ -182,7 +183,7 @@ export function createVercelOAuthCallback(db: Db, dependencies: Dependencies) {
status: connection.status,
enabled: connection.enabled,
};
[connection] = await db
[connection] = await writeToolConnectionHealthChange(db, connection, tx => tx
.update(toolConnections)
.set({
status: "active",
@@ -197,7 +198,7 @@ export function createVercelOAuthCallback(db: Db, dependencies: Dependencies) {
eq(toolConnections.companyId, connection.companyId),
),
)
.returning();
.returning());
emitConnectionUpdated(
connection,
preActivationLifecycle,
server/src/services/tool-connection-health-notifications.ts
+27
View File
@@ -0,0 +1,27 @@
import type { Db } from "@paperclipai/db";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
type HealthChange = string | { companyId: string; id: string } | { companyId: string; applicationId: string };
export type HealthScope = { companyId?: string; connectionId?: string; applicationId?: string };
export function healthScope(hint: string): HealthScope {
if (hint === "*") return {};
const [companyId, kind, id, extra] = hint.split(":");
if (companyId && kind === undefined) return { companyId };
if (companyId && id && extra === undefined) {
if (kind === "connection") return { companyId, connectionId: id };
if (kind === "application") return { companyId, applicationId: id };
}
throw new Error("Invalid tool connection health scope");
}
export const notifyToolConnectionHealth = (transaction: object, target: HealthChange) =>
notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolConnectionHealth, typeof target === "string" ? target
: "id" in target ? `${target.companyId}:connection:${target.id}` : `${target.companyId}:application:${target.applicationId}`);
export function writeToolConnectionHealthChange<T>(db: Db, target: HealthChange, write: (transaction: Db) => Promise<T>): Promise<T> {
return db.transaction(async tx => {
await notifyToolConnectionHealth(tx, target);
return write(tx as unknown as Db);
});
}
server/src/services/tool-connection-health-work.ts
+104
View File
@@ -0,0 +1,104 @@
import { healthScope, type HealthScope } from "./tool-connection-health-notifications.js";
import { takeCompanyWorkBatch } from "./company-work-batch.js";
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
import type { ConnectionHealthDeadline } from "./tool-access/catalog/health-schedule.js";
export function registerToolConnectionHealthWork(
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
service: {
load(companyId?: string, connectionId?: string, applicationId?: string, signal?: AbortSignal): Promise<ConnectionHealthDeadline[]>;
check(id: string): Promise<ConnectionHealthDeadline | null>;
},
canRun: () => boolean,
) {
const dirty = new Set<string>(["*"]);
const deadlines = new Map<string, ConnectionHealthDeadline>();
const failures = new Map<string, { count: number; retryAt: number }>();
const refreshFailures = new Map<string, { scope: HealthScope; retryAt: number }>();
const matches = (entry: ConnectionHealthDeadline, scope: HealthScope) =>
(scope.companyId === undefined || entry.companyId === scope.companyId)
&& (scope.connectionId === undefined || entry.id === scope.connectionId)
&& (scope.applicationId === undefined || entry.applicationId === scope.applicationId);
const retryMs = 30_000;
let earliest = 0;
let lastCompany: string | null = null;
function request(hints?: ReadonlySet<string>) {
for (const hint of hints ?? ["*"]) dirty.add(hint);
}
function next() {
let result: number | null = null;
for (const scope of dirty) {
const at = Math.max(earliest, refreshFailures.get(scope)?.retryAt ?? 0);
result = result === null ? at : Math.min(result, at);
}
for (const entry of deadlines.values()) {
let at = Math.max(earliest, entry.dueAt, failures.get(entry.id)?.retryAt ?? 0);
for (const failure of refreshFailures.values()) if (matches(entry, failure.scope)) at = Math.max(at, failure.retryAt);
result = result === null ? at : Math.min(result, at);
}
return result;
}
return coordinator.register(DELIVERY_QUEUES.toolConnectionHealth, {
retryMs,
pausePassiveMaintenance: true,
onSettled: request,
run: async signal => {
if (!canRun() || Date.now() < earliest) return;
earliest = Date.now() + retryMs;
const scopes = dirty.has("*") ? ["*"] : [...dirty];
dirty.clear();
const errors: unknown[] = [];
const unavailable: HealthScope[] = [];
for (let index = 0; index < scopes.length; index++) {
const scope = scopes[index];
if (signal.aborted || !canRun()) { for (const remaining of scopes.slice(index)) dirty.add(remaining); return; }
let target: HealthScope | undefined;
try {
target = healthScope(scope);
if ((refreshFailures.get(scope)?.retryAt ?? 0) > Date.now()) {
dirty.add(scope); unavailable.push(target); continue;
}
const rows = await service.load(target.companyId, target.connectionId, target.applicationId, signal);
if (target.connectionId) {
const current = deadlines.get(target.connectionId);
if (current && matches(current, target)) deadlines.delete(current.id);
} else for (const entry of deadlines.values()) if (matches(entry, target)) deadlines.delete(entry.id);
for (const entry of rows) deadlines.set(entry.id, entry);
if (scope === "*") refreshFailures.clear(); else refreshFailures.delete(scope);
} catch (error) {
const retryScope = target ? scope : "*";
refreshFailures.set(retryScope, { scope: target ?? {}, retryAt: Date.now() + retryMs });
dirty.add(retryScope); unavailable.push(target ?? {}); errors.push(error);
}
}
for (const id of failures.keys()) {
const current = deadlines.get(id);
if (!current || current.dueAt > Date.now()) failures.delete(id);
}
const due = [...deadlines.values()].filter(entry => !unavailable.some(scope => matches(entry, scope))
&& Math.max(entry.dueAt, failures.get(entry.id)?.retryAt ?? 0) <= Date.now())
.sort((a, b) => a.dueAt - b.dueAt || a.id.localeCompare(b.id));
const batch = takeCompanyWorkBatch(due, 25, lastCompany);
lastCompany = batch.lastCompany;
for (const entry of batch.entries) {
if (signal.aborted || !canRun()) break;
try {
const current = await service.check(entry.id);
if (current) deadlines.set(entry.id, current); else deadlines.delete(entry.id);
failures.delete(entry.id);
} catch (error) {
const count = Math.min((failures.get(entry.id)?.count ?? 0) + 1, 6);
failures.set(entry.id, { count, retryAt: Date.now() + Math.min(15 * 60_000, retryMs * 2 ** (count - 1)) });
errors.push(error);
}
}
if (due.length > batch.entries.length) earliest = Date.now() + 1_000;
if (errors.length) throw new AggregateError(errors, "Tool connection health checks could not settle");
},
// Keep the hints/deadlines for resume, but release settled intent while
// paused. The coordinator independently fences unresolved transactions.
hasPending: async () => canRun() && dirty.size > 0,
nextRunAt: next,
});
}
server/src/services/tool-gateway.ts
+3 -2
View File
@@ -1,3 +1,4 @@
import { writeToolConnectionHealthChange } from "./tool-connection-health-notifications.js";
import { writeApprovalChange } from "./approval-work-notifications.js";
import { toolReviewProjectionService } from "./tool-review-projection.js";
import { toolReviewInterruptionService } from "./tool-review-interruption.js";
@@ -3882,7 +3883,7 @@ export function createToolGatewayService(
message: string | null,
) {
const now = new Date();
await db
await writeToolConnectionHealthChange(db, connection, tx => tx
.update(toolConnections)
.set({
healthStatus: status,
@@ -3892,7 +3893,7 @@ export function createToolGatewayService(
lastError: status === "ok" ? null : message,
updatedAt: now,
})
.where(eq(toolConnections.id, connection.id));
.where(eq(toolConnections.id, connection.id)));
}
function grantRefForCredential(