refactor(server): split chat channel workflows into provider and shared modules (#16080)

Move provider operations into channel folders and shared chat work into named interaction, setup, admission, delivery and publication workflows. Consolidate reviewed PRs #16085, #16089, #16096 and #16105. Preserve upstream execution-default and recovery behavior. The main service is 7,917 lines.

All 55 current-head CI checks pass, with two optional Storybook skips. Greptile is 5/5 with no findings. All 1,294 current chat integration cases, repository-wide typecheck and build pass locally. Broad local tool-route timing/socket failures are recorded in the PR.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-13 07:26:32 -05:00
1 parent 60bc0d5fee
commit eb06c9fedd
43 files changed
+13705 -11475

No files matched your search

@@ -0,0 +1,159 @@
# Chat channel workflow refactor
The user authorized six hours of sustained work starting at 2026-10-13 02:14 UTC,
with a target of fewer than 8,000 lines in `server/src/services/chat-channels.ts`.
The starting commit is `7eaa9cfa9ce04398a5eb0211a86aacaaa01959dd`: 19,153 lines,
762,442 bytes. At kickoff, the preceding ten PRs were reviewed and green, but not yet merged.
The user authorizes publishing and merging reviewed changes to paperclipai/paperclip.
The previous plan prioritized too many small helper cuts. This plan prioritizes
complete workflows, provider ownership, and meaningful reductions per PR. Shared
transactions and admission policy belong in shared workflow modules. Provider
transport, envelope checks, setup, and native interactions belong in provider
folders. Do not put shared code in one provider's folder or use a mutable context
bag to move the entire service unchanged.
| Priority | Responsibility and destination | Estimated net reduction | Contract and state | Risk and verification |
| --- | --- | ---: | --- | --- |
| 1 | Native interaction callbacks and continuation: `interactions/`, `telegram/confirmation-actions.ts`, `discord/question-forms.ts` | 2,900 | Current actor/runtime/destination readers; durable token and interaction transactions; deferred wakeups. | Preserve token replay, audit order, provider acknowledgement, post-commit recovery, and live fault barriers. Existing public service, HTTP and real-database interaction tests before and after. |
| 2 | Endpoint configuration and provider setup: provider `setup.ts`, shared credential persistence/configuration | 1,600–2,200 | Credential lease and current endpoint; provider registration outcome; stable native identity. | Preserve fail-closed persistence, webhook/receiver ownership, partial setup cleanup, and credential rotation races. Existing setup and runtime tests. |
| 3 | Message admission and provider policy: shared intake workflow, GitHub message policy, provider envelope projections | 2,400–3,000 | Delivery claim and conversation/task admission; explicit provider policy results; per-service message provenance. | Preserve row locks, deduplication, author authority, task generations, attachments, and GitHub automatic/explicit admission. Existing real-database ingress and replay tests. |
| 4 | Durable delivery and lifecycle: shared queue, provider lifecycle inputs | 1,700–2,000 | Conversation leases, replay decode, ordered drains, mutation reconciliation; shared live receipt state. | Preserve stale lease exclusion, event order, deletion/edit races, deferred startup and shutdown. Existing concurrency and lifecycle tests. |
| 5 | Outbound publication: provider delivery modules and shared publication worker | 2,700–3,400 | Current-source authorization; exact transport parts; attempts, unknown outcomes, completion receipts and cleanup. | Preserve no duplicate sends, ambiguous acceptance, credential fencing, file recovery, concurrency and current task-control completion. Existing assembled publication and file tests. |
| 6, if needed | Operator recovery and endpoint projections | 800–1,200 | Explicit replay/resolution and company-scoped board calls. | Preserve retry audit, current policy and authorization. Existing operator-resolution integration tests. |
The estimates are not a PR quota. Inspect every full execution path before moving
it. Increase size only where a cohesive responsibility has a narrow contract.
After every reviewed slice, remap the remaining file and report the measured size.
For each slice: run a behavioral baseline before editing production; retain
existing bodies and SQL; compare normalized AST and statement order; run server
typecheck, module boundaries and public-path tests. Review all intentional callback
and dispatch substitutions. Preserve live option reads with getters or callbacks.
Keep per-service state per service and shared cache/lease ownership shared.
Open one PR per cohesive extraction with the full repository template. Verify its
current-head review and all CI gates before opening the next extraction. Build each
on its immediate predecessor. Run repository-wide typecheck, tests and build once
at the final handoff; investigate and disclose any failures or unreached coverage.
The earlier batch recorded local timing/host cleanup failures; green CI and passing
focused retries do not erase that evidence. No timeout or assertion weakening.
Public application fixtures prove assembled service/HTTP/database behavior. They
do not prove live third-party execution. Use only owned fixture data and processes.
## First extraction
The native interaction workflow reduces the service from 19,153 to 16,117 lines
and from 762,442 to 656,146 bytes. Telegram owns confirmation resolution and
settlement. Discord owns correction draft retention and accepted-answer cleanup.
Shared callbacks retain token/source/actor checks and atomic question resolution.
Shared continuation work retains durable wakeup recovery. SDK state persistence
has a direct module import and retains the public compatibility export.
The same 62 public-path interaction cases pass before and after. Seven supporting
suites pass all 70 cases. Server typecheck and module boundaries pass. AST checks
cover nine moved declarations, three provider bodies, and retained service order.
Fault barriers remain live getters. No assertion or timeout changed.
## Second extraction
Endpoint configuration and provider setup reduce the service to 14,241 lines and
580,227 bytes. Shared modules own endpoint records, credential identity/persistence,
and the credential-leased configuration state machine. Channel folders own Telegram
registration, GitHub repair, Speko tool installation, Photon configuration and
inspection, and Slack setup projection. Registration dependencies remain live.
All 1,278 chat cases passed on the first PR before this move. The final setup wiring
passes 268 selected integration cases, server typecheck, and module boundaries.
AST checks cover 25 declarations and nine provider bodies. One final test attempt
could not start PostgreSQL because this Mac exhausted IPC slots; after reclaiming
one unused zero-attachment segment with a dead creator, the same suite passed.
## Third extraction
The publication workflow reduces the service to 11,198 lines and 462,108 bytes.
Authorization, durable preparation, transport, failures, attempt ownership, task
control completion, and bounded endpoint scheduling now have named modules.
Photon owns prompt and reply delivery. Discord and Telegram own durable Markdown
attachment preparation. Shared native splitting remains shared across providers.
The same 301 public-path publication cases pass before and after. Eight supporting
suites pass all 185 cases. Server typecheck, module boundaries, and AST checks pass.
The AST audit covers 26 declarations, three provider bodies, and retained service
order. Shared endpoint lanes and Teams cooldowns retain their original maps;
shutdown and publication wake listeners stay live. Early callback registration
uses typed forwarding functions instead of reading factories before initialization.
## Fourth extraction
Ordered delivery and lifecycle reconciliation reduce the service to 9,287 lines
and 388,734 bytes. The delivery ledger owns ordering and runtime-fence projections.
The conversation queue owns exact leases, reconstruction and ordered replay.
The lifecycle workflow owns edit/delete admission and reconciliation. Teams owns
authenticated route recording. GitHub owns restoration of durable message policy
into the same per-service provenance maps.
The same 225 lifecycle, ordering, lease, replay and recovery cases pass before and
after. Typecheck and module boundaries pass. AST checks cover 26 declarations, the
GitHub restoration body, and retained service order. Live maps and lease hooks retain
their original lifetime. The publication predecessor's packaging retry passed unchanged. This fourth
slice is published as PR #16096.
## Fifth extraction
Inbound authority, envelope preparation, durable admission and resource admission
reduce the service to 7,899 lines and 335,607 bytes. The shared admission module
keeps endpoint locking, runtime and credential fences, gateway lease renewal,
deduplication and delivery insertion in one transaction. Teams, Telegram and
Photon own their destination rules and use that same transaction. The main
message path now calls preparation, durable admission and resource admission
before the retained task workflow.
The same 278 selected public-path cases pass before and after the first move.
All 1,278 existing chat integration cases pass on the final provider dispatch
in 309.47 seconds. Server typecheck and module boundaries pass. One full-suite attempt
could not allocate PostgreSQL shared memory before any test ran. Only a verified
unused 56-byte segment with a dead creator and zero attachments was reclaimed
before retrying. No active database was stopped.
AST checks cover 21 moved declarations, envelope and resource phase bodies,
three provider destination bodies, and the complete admission transaction after
restoring the explicit provider calls. The service still owns the per-call
gateway ownership handoff. The envelope retains its per-call recovery map and
the original authenticated runtime. No assertion or timeout changed.
## Stack landing
The preceding ten PRs reached master through #16058. The five workflow PRs are
#16080, #16085, #16089, #16096 and #16105. Each extraction has individual behavioral
evidence and a 5/5 review. The four child PRs merged into their predecessors after
each combined revision passed its own checks and review. Every consolidation
preserved the tested tip tree exactly.
The final master integration leaves the service at 7,917 lines and 337,195 bytes.
The 18-line increase over the extraction tip preserves upstream execution-default
selection and authorization. Endpoint serialization and credential-leased updates
retain that policy in the extracted endpoint module. Task creation and failed-run
recovery retain their upstream behavior. All 107 extracted declarations match
current master after reversing the documented binding substitutions. The endpoint,
task-creation and recovery functions also pass exact normalized AST comparisons.
Repository-wide typecheck and build pass on this integration. The first full local
test command passed 18,391 server tests and skipped 105, but one unchanged tool-access
test hit its 15-second timeout. That case passed alone. The UI and CLI groups passed
8,706 tests. Shared and adapter groups passed 4,751 tests and skipped 20. A database
group could not initialize one PostgreSQL fixture. This Mac has a 32-segment shared
memory limit and concurrent fixture databases. The unchanged database cases are
being rerun one file at a time before the remaining chat and serialized route lanes.
Keep the original failed runs in the verification record. No assertion or timeout
was changed. The root PR remains open for fresh review and CI against master.
The service now has 67 direct functions. The largest retained paths are message
task handling (1,449 lines), webhook dispatch (685), operator action resolution
(488), and publication resolution (343). Provider setup, transport preparation,
native interactions and destination admission live in their channel folders.
Shared admission, delivery, lifecycle and publication transactions have named
workflow modules. The retained message path calls those phases and keeps task
mutation and per-call ownership together.
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,968 @@
import { preparePhotonConfiguration, createPhotonConfiguration } from "./imessage-photon/setup.js";
import { slackRuntimeConfig } from "./slack/setup.js";
import { createTelegramSetup } from "./telegram/setup.js";
import { createGitHubSetup } from "./github/setup.js";
import { createSpekoSetup } from "./speko/setup.js";
import { createGitHubAppSetup } from "./github/app-setup.js";
import type { VerifiedProviderIdentity } from "./runtime-types.js";
import { isProvisionedSlackSetup } from "@paperclipai/shared";
import { slackCallbackMatchesPublicUrl } from "./slack/callback-verification.js";
import { createDiscordCommands } from "./discord/commands.js";
import { createTelegramMaintenance } from "./telegram/maintenance.js";
import type { EndpointRow, RuntimeContext, InternalSetupState } from "./runtime-types.js";
import { createChatResourceReconciliation } from "./resource-reconciliation.js";
import { notifyChatDeliveryWork, notifyChatEndpointWork } from "../chat-work-notifications.js";
import type { ChatEndpointRuntime } from "../voice/voice-runtime.js";
import { chatCredentialMutationLease, type CredentialMutationLeaseGuard } from "../chat-credential-mutation-lease.js";
import { slackChatRegistrationService } from "../chat-slack-registration.js";
import {
createHash,
} from "node:crypto";
import {
and,
eq,
inArray,
isNull,
ne,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatConversations,
chatDeliveries,
chatEndpoints,
toolApplications,
toolConnections,
} from "@paperclipai/db";
import type {
ChatEndpoint,
ChatEndpointSetupState,
ChatProvider,
ConfigureChatEndpointInput,
} from "@paperclipai/shared";
import {
MAX_ATTACHMENT_BYTES,
} from "../../attachment-types.js";
import {
badRequest,
conflict,
notFound,
unprocessable,
} from "../../errors.js";
import {
type ResolvedChatSdkProviderConfig,
} from "../chat-sdk-runtime.js";
import {
logActivity,
} from "../activity-log.js";
import type { EndpointRecord } from "./runtime-types.js";
import type { createChatCredentials } from "./credentials.js";
type ChatConfigurationOptions = Pick<ReturnType<typeof createChatCredentials>, "normalizedCredentials" | "verifyCredentials" | "nativeBotIdentityMatches" | "legacyGitHubLabelsMatchVerifiedCredentials" | "assertNativeBotIdentityAvailable" | "isNativeBotIdentityUniqueViolation" | "nativeBotIdentityConflict" | "persistCredentials" | "clearCredentials" | "resolveCredentials" | "prepareProviderInventory"> & {
endpointRecord: (endpointId: string) => Promise<EndpointRecord | null>;
get: (endpointId: string) => Promise<ChatEndpoint>;
withCredentialMutationLease: ReturnType<typeof chatCredentialMutationLease>;
getWebhookPublicBaseUrl: () => string | null;
invalidateRuntime: (endpointId: string) => Promise<boolean>;
reconcileProviderResourceRows: ReturnType<typeof createChatResourceReconciliation>["reconcileProviderResourceRows"];
reconcileDiscordCommands: ReturnType<typeof createDiscordCommands>["reconcileDiscordCommands"];
runtimeFor: (endpoint: EndpointRow, input?: { requireDiscordOwnership?: boolean; waitForDiscordOwnership?: boolean }) => Promise<ChatEndpointRuntime>;
runtimeContextForRecord: (record: EndpointRecord) => RuntimeContext;
stageTelegramMaintenance: ReturnType<typeof createTelegramMaintenance>["stageTelegramMaintenance"];
processTelegramMaintenance: ReturnType<typeof createTelegramMaintenance>["processTelegramMaintenance"];
recordTelegramStopSubscription: ReturnType<typeof createTelegramMaintenance>["recordTelegramStopSubscription"];
fetchImpl: typeof globalThis.fetch;
redactError: (error: unknown) => string;
runtimeGeneration: (setup: ChatEndpointSetupState) => number;
leasedChatProvider: (provider: string) => boolean;
PROVIDER_LABELS: Record<ChatProvider, string>;
SUPPLIED_CREDENTIAL_KEYS: Record<ChatProvider, readonly string[]>;
PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS: number;
isSupportedTelegramWebhookBaseUrl: (url: string | null) => boolean;
slackRegistration: Pick<ReturnType<typeof slackChatRegistrationService>, "cleanup">;
githubWizard: Pick<ReturnType<typeof createGitHubAppSetup>["githubWizard"], "repairCloud">;
nativeBotIdentityClaimBarrier?: () => Promise<void>;
allowLocalVoiceBoard?: boolean;
};
export function createChatConfiguration(db: Db, options: ChatConfigurationOptions) {
const { endpointRecord, get, withCredentialMutationLease, getWebhookPublicBaseUrl, invalidateRuntime, nativeBotIdentityMatches, assertNativeBotIdentityAvailable, isNativeBotIdentityUniqueViolation, nativeBotIdentityConflict, legacyGitHubLabelsMatchVerifiedCredentials, normalizedCredentials, verifyCredentials, persistCredentials, clearCredentials, resolveCredentials, prepareProviderInventory, reconcileProviderResourceRows, reconcileDiscordCommands, runtimeFor, runtimeContextForRecord, stageTelegramMaintenance, processTelegramMaintenance, recordTelegramStopSubscription, fetchImpl, redactError, runtimeGeneration, leasedChatProvider, PROVIDER_LABELS, SUPPLIED_CREDENTIAL_KEYS, PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS, isSupportedTelegramWebhookBaseUrl } = options;
const { persistPhotonConfiguration } = createPhotonConfiguration(db, { invalidateRuntime });
const { registerTelegramWebhook } = createTelegramSetup(db, { fetchImpl, getWebhookPublicBaseUrl, PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS, runtimeContextForRecord, recordTelegramStopSubscription, stageTelegramMaintenance, processTelegramMaintenance });
const { repairGitHubCloudSetup, repairGitHubWebhookSetup } = createGitHubSetup(db, { fetchImpl, getWebhookPublicBaseUrl, get githubWizard() { return options.githubWizard; } });
const { installSpekoVoiceTools, configureSpekoTools } = createSpekoSetup(db, { fetchImpl, getWebhookPublicBaseUrl, get allowLocalVoiceBoard() { return options.allowLocalVoiceBoard; } });
async function configure(
endpointId: string,
input: ConfigureChatEndpointInput,
actorUserId?: string | null,
) {
const record = await endpointRecord(endpointId);
if (!record) throw notFound("Chat endpoint not found");
if (record.endpoint.provider === "agentmail") throw badRequest("Use the email inbox API for AgentMail");
if (input.photon && record.endpoint.provider !== "imessage-photon") throw badRequest("Photon configuration is only valid for iMessage Photon");
const suppliedCredentialKeys = Object.keys(input.credentials ?? {});
if (suppliedCredentialKeys.length > 0) {
const credentialAction =
input.action === "configure" || input.action === "reconnect";
const allowedKeys = new Set(
credentialAction
? SUPPLIED_CREDENTIAL_KEYS[record.endpoint.provider]
: [],
);
const unsupportedKeys = suppliedCredentialKeys.filter(
(key) => !allowedKeys.has(key),
);
if (unsupportedKeys.length > 0) {
throw unprocessable(
`Unsupported ${PROVIDER_LABELS[record.endpoint.provider]} credential fields for ${input.action}: ${unsupportedKeys.join(", ")}`,
{
code: "chat_endpoint_credentials_invalid",
provider: record.endpoint.provider,
action: input.action,
unsupportedKeys,
},
);
}
}
return withCredentialMutationLease(record.endpoint, (credentialLease) =>
configureWithCredentialLease(
endpointId,
input,
credentialLease,
actorUserId,
),
);
}
async function configureWithCredentialLease(
endpointId: string,
input: ConfigureChatEndpointInput,
credentialLease: CredentialMutationLeaseGuard,
actorUserId?: string | null,
) {
const record = await endpointRecord(endpointId);
if (!record) throw notFound("Chat endpoint not found");
const endpoint = record.endpoint;
if (input.action === "pause") {
if (endpoint.status !== "active") {
throw conflict("Only an active chat connection can be paused", {
code: "chat_endpoint_not_active",
});
}
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const pausedAt = new Date();
const current = await tx
.select({ setup: chatEndpoints.setup, status: chatEndpoints.status })
.from(chatEndpoints)
.where(eq(chatEndpoints.id, endpoint.id))
.for("update")
.then((rows) => rows[0] ?? null);
if (!current || current.status !== "active") {
throw conflict("Only an active chat connection can be paused", {
code: "chat_endpoint_not_active",
});
}
await notifyChatEndpointWork(tx);
await tx
.update(chatEndpoints)
.set({
status: "paused",
setup: {
...current.setup,
runtimeGeneration: runtimeGeneration(current.setup) + (endpoint.provider === "imessage-photon" ? 0 : 1),
} as InternalSetupState,
updatedAt: pausedAt,
})
.where(eq(chatEndpoints.id, endpoint.id));
await tx
.update(toolConnections)
.set({ status: "disabled", enabled: false, updatedAt: new Date() })
.where(eq(toolConnections.id, endpoint.connectionId));
await notifyChatDeliveryWork(tx);
await tx
.update(chatDeliveries)
.set({
state: "filtered",
nextAttemptAt: null,
redactedError: "Connection was paused before processing",
processedAt: pausedAt,
updatedAt: pausedAt,
})
.where(
and(
endpoint.provider === "imessage-photon" ? sql`false` : undefined,
eq(chatDeliveries.endpointId, endpoint.id),
inArray(chatDeliveries.state, ["received", "retry"]),
),
);
await credentialLease.assertOwned(tx);
});
await invalidateRuntime(endpoint.id);
await logActivity(db, {
companyId: endpoint.companyId,
actorType: "user",
actorId: actorUserId ?? "board",
action: "chat_endpoint.paused",
entityType: "tool_connection",
entityId: endpoint.connectionId,
details: { endpointId: endpoint.id },
});
return get(endpoint.id);
}
if (input.action === "resume") {
if (endpoint.status !== "paused" || endpoint.setup.step !== "complete") {
throw conflict(
"Only a previously active chat connection can be resumed",
{
code: "chat_endpoint_not_resumable",
},
);
}
let credentials = await resolveCredentials(endpoint).catch(() => {
throw conflict("Reconnect the provider credentials before resuming", {
code: "chat_endpoint_credentials_missing",
});
});
const identity = await verifyCredentials(endpoint.provider, credentials);
const upgradingLegacyGitHubIdentity =
endpoint.provider === "github" &&
!endpoint.botExternalId &&
legacyGitHubLabelsMatchVerifiedCredentials(endpoint, identity);
if (
!upgradingLegacyGitHubIdentity &&
!nativeBotIdentityMatches(endpoint.provider, endpoint, identity)
) {
throw conflict(
"The provider credentials now identify a different bot; reconnect this connection instead",
{
code: "chat_bot_identity_changed",
},
);
}
await assertNativeBotIdentityAvailable(endpoint, identity);
const prepared = await prepareProviderInventory(endpoint, credentials);
if (
endpoint.provider === "github" &&
prepared.credentials.installationId !== credentials.installationId
) {
await persistCredentials(
endpoint,
prepared.credentials,
credentialLease,
actorUserId,
);
}
credentials = prepared.credentials;
if (prepared.inventory)
await reconcileProviderResourceRows(
endpoint,
prepared.inventory,
credentialLease,
);
let activatedEndpoint!: EndpointRow;
try {
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const resumedAt = new Date();
const current = await tx
.select()
.from(chatEndpoints)
.where(eq(chatEndpoints.id, endpoint.id))
.for("update")
.then((rows) => rows[0] ?? null);
if (!current || current.status !== "paused") {
throw conflict(
"Only a previously active chat connection can be resumed",
{
code: "chat_endpoint_not_resumable",
},
);
}
// Fail closed for any legacy or racing delivery that remained open
// while this endpoint was paused. Resume must never execute traffic
// that Paperclip acknowledged during the inactive interval.
await notifyChatDeliveryWork(tx);
await tx
.update(chatDeliveries)
.set({
state: "filtered",
nextAttemptAt: null,
redactedError: "Connection was paused when this event arrived",
processedAt: resumedAt,
updatedAt: resumedAt,
})
.where(
and(
endpoint.provider === "imessage-photon" ? sql`false` : undefined,
eq(chatDeliveries.endpointId, endpoint.id),
inArray(chatDeliveries.state, ["received", "retry"]),
),
);
await notifyChatEndpointWork(tx);
[activatedEndpoint] = await tx
.update(chatEndpoints)
.set({
status: "active",
...(upgradingLegacyGitHubIdentity
? { botExternalId: identity.botExternalId }
: {}),
healthMessage: "Connected",
lastError: null,
setup: {
...current.setup,
...(endpoint.provider === "imessage-photon" ? { photonIntakeAfter: resumedAt.toISOString() } : {}),
runtimeGeneration: runtimeGeneration(current.setup) + (endpoint.provider === "imessage-photon" ? 0 : 1),
} as InternalSetupState,
updatedAt: resumedAt,
})
.where(eq(chatEndpoints.id, endpoint.id))
.returning();
await tx
.update(toolConnections)
.set({
status: "active",
enabled: true,
healthStatus: "healthy",
healthMessage: "Connected",
lastError: null,
healthCheckedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(toolConnections.id, endpoint.connectionId));
await credentialLease.assertOwned(tx);
});
await invalidateRuntime(endpoint.id);
if (activatedEndpoint.provider === "discord")
await reconcileDiscordCommands(endpoint.id, credentialLease, true);
await runtimeFor(activatedEndpoint, {
requireDiscordOwnership: leasedChatProvider(activatedEndpoint.provider),
waitForDiscordOwnership: leasedChatProvider(activatedEndpoint.provider),
});
} catch (error) {
await invalidateRuntime(endpoint.id).catch(() => undefined);
if (activatedEndpoint) {
const failure = redactError(error);
const failedAt = new Date();
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatEndpointWork(tx);
await tx
.update(chatEndpoints)
.set({
status: "attention",
healthMessage: "Provider resume needs attention",
lastError: failure,
updatedAt: failedAt,
})
.where(
and(
eq(chatEndpoints.id, endpoint.id),
eq(chatEndpoints.status, "active"),
sql`coalesce((${chatEndpoints.setup}->>'runtimeGeneration')::integer, 0) = ${runtimeGeneration(activatedEndpoint.setup)}`,
),
);
await tx
.update(toolConnections)
.set({
status: "disabled",
enabled: false,
healthStatus: "degraded",
healthMessage: "Provider resume failed",
lastError: failure,
healthCheckedAt: failedAt,
updatedAt: failedAt,
})
.where(eq(toolConnections.id, endpoint.connectionId));
await notifyChatDeliveryWork(tx);
await tx
.update(chatDeliveries)
.set({
state: "filtered",
nextAttemptAt: null,
redactedError: "Connection activation failed",
processedAt: failedAt,
updatedAt: failedAt,
})
.where(
and(
eq(chatDeliveries.endpointId, endpoint.id),
inArray(chatDeliveries.state, ["received", "retry"]),
),
);
await credentialLease.assertOwned(tx);
});
}
throw error;
}
await logActivity(db, {
companyId: endpoint.companyId,
actorType: "user",
actorId: actorUserId ?? "board",
action: "chat_endpoint.resumed",
entityType: "tool_connection",
entityId: endpoint.connectionId,
details: { endpointId: endpoint.id },
});
return get(endpoint.id);
}
if (input.action === "remove") {
if (endpoint.status === "archived") {
if (endpoint.provider === "slack") await options.slackRegistration.cleanup(endpoint.id, credentialLease);
// Archival is the durable ingress fence and intentionally commits
// before secret-store cleanup. If that cleanup failed, a repeated
// remove is the recovery operation; once refs are empty, retain the
// normal already-removed conflict contract.
if (record.credentialSecretRefs.length > 0) {
await invalidateRuntime(endpoint.id).catch(() => undefined);
if (endpoint.provider === "telegram") {
await credentialLease.assertOwned();
const action = await stageTelegramMaintenance(
endpoint,
"remove_endpoint",
);
if (action)
await processTelegramMaintenance(
action.id,
true,
credentialLease,
);
} else {
await clearCredentials(endpoint, credentialLease);
}
return get(endpoint.id);
}
throw conflict("This chat connection has already been removed", {
code: "chat_endpoint_already_removed",
});
}
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const current = await tx
.select({ setup: chatEndpoints.setup, status: chatEndpoints.status })
.from(chatEndpoints)
.where(eq(chatEndpoints.id, endpoint.id))
.for("update")
.then((rows) => rows[0] ?? null);
if (!current || current.status === "archived") {
throw conflict("This chat connection has already been removed", {
code: "chat_endpoint_already_removed",
});
}
await notifyChatEndpointWork(tx);
await tx
.update(chatEndpoints)
.set({
status: "archived",
setup: {
...current.setup,
runtimeGeneration: runtimeGeneration(current.setup) + 1,
} as InternalSetupState,
archivedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(chatEndpoints.id, endpoint.id));
await tx
.update(chatConversations)
.set({ state: "endpoint_removed", updatedAt: new Date() })
.where(eq(chatConversations.endpointId, endpoint.id));
await tx
.update(toolConnections)
.set({ status: "archived", enabled: false, updatedAt: new Date() })
.where(eq(toolConnections.id, endpoint.connectionId));
const otherConnection = await tx
.select({ id: toolConnections.id })
.from(toolConnections)
.where(
and(
eq(toolConnections.companyId, endpoint.companyId),
eq(toolConnections.applicationId, record.applicationId),
ne(toolConnections.id, endpoint.connectionId),
ne(toolConnections.status, "archived"),
),
)
.limit(1)
.then((rows) => rows[0] ?? null);
if (!otherConnection) {
await tx
.update(toolApplications)
.set({
status: "archived",
archivedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(toolApplications.id, record.applicationId));
}
await credentialLease.assertOwned(tx);
});
if (endpoint.provider === "slack") await options.slackRegistration.cleanup(endpoint.id, credentialLease);
await invalidateRuntime(endpoint.id).catch(() => undefined);
if (endpoint.provider === "telegram") {
await credentialLease.assertOwned();
const action = await stageTelegramMaintenance(
endpoint,
"remove_endpoint",
);
if (action) {
await processTelegramMaintenance(action.id, true, credentialLease);
}
} else {
await clearCredentials(endpoint, credentialLease);
}
return get(endpoint.id);
}
if (
input.action !== "configure" &&
input.action !== "reconnect" &&
input.action !== "verify"
) {
throw unprocessable("Unsupported chat endpoint setup action");
}
if (!getWebhookPublicBaseUrl() && endpoint.provider !== "discord" && endpoint.provider !== "imessage-photon" && !(endpoint.provider === "github" && endpoint.setup.github?.cloudRegistrationId)) {
throw unprocessable(
`A public HTTPS Paperclip URL is required before connecting ${PROVIDER_LABELS[endpoint.provider]}`,
);
}
if (
endpoint.provider === "telegram" &&
(input.action === "configure" || input.action === "reconnect") &&
!isSupportedTelegramWebhookBaseUrl(getWebhookPublicBaseUrl())
) {
throw unprocessable(
"Telegram webhooks require PAPERCLIP_CHAT_WEBHOOK_PUBLIC_URL to use HTTPS on port 443, 80, 88, or 8443",
{
code: "chat_telegram_webhook_url_unsupported",
provider: "telegram",
supportedPorts: [443, 80, 88, 8443],
},
);
}
if (input.action === "verify") {
if (
endpoint.provider !== "slack" ||
endpoint.status !== "verifying" ||
endpoint.setup.step !== "provider_setup"
) {
throw conflict(
"Provider verification is not available at this setup step",
{
code: "chat_endpoint_invalid_setup_step",
},
);
}
if (!endpoint.setup.webhookVerifiedAt) {
throw conflict("Paperclip has not received a verified Slack callback yet", {
code: "chat_webhook_not_verified",
});
}
} else if (
input.action === "configure" &&
!["draft", "attention", "revoked"].includes(endpoint.status) &&
// GitHub installation discovery can advance the status before the
// wizard has configured its runtime. Finish that exact setup stage;
// configured connections still require the explicit reconnect flow.
!(
endpoint.provider === "github" &&
endpoint.status === "verifying" &&
endpoint.setup.step === "provider_setup"
)
) {
throw conflict(
"This connection is already configured; use its reconnect flow if credentials need repair",
{
code: "chat_endpoint_already_configured",
},
);
} else if (
input.action === "reconnect" &&
!["verifying", "active", "attention", "revoked", "paused"].includes(
endpoint.status,
)
) {
throw conflict(
"This chat connection does not currently need reconnecting",
{
code: "chat_endpoint_not_reconnectable",
},
);
}
if (
endpoint.provider === "github" &&
(input.action === "configure" || input.action === "reconnect") &&
!endpoint.setup.webhookVerifiedAt
) {
throw conflict(
"GitHub has not delivered a signed webhook ping for this secret yet",
{
code: "chat_webhook_not_verified",
},
);
}
let verifiedCurrentIdentity: VerifiedProviderIdentity = endpoint;
if (
input.action === "reconnect" &&
endpoint.provider === "github" &&
!endpoint.botExternalId
) {
const storedCredentials = await resolveCredentials(endpoint).catch(() => {
throw conflict(
"Reconnect the existing GitHub App credentials before changing them",
{
code: "chat_bot_identity_unverifiable",
},
);
});
const storedIdentity = await verifyCredentials(
"github",
storedCredentials,
);
if (
!legacyGitHubLabelsMatchVerifiedCredentials(endpoint, storedIdentity)
) {
throw conflict(
"The stored credentials no longer verify the GitHub App that owns this connection",
{
code: "chat_bot_identity_unverifiable",
},
);
}
verifiedCurrentIdentity = storedIdentity;
}
const previousSpekoSigningSecret = endpoint.provider === "speko"
? (await resolveCredentials(endpoint).catch(() => ({} as Record<string, string>))).signingSecret : undefined;
let credentials =
input.credentials && Object.keys(input.credentials).length > 0
? await normalizedCredentials(endpoint, input.credentials)
: await resolveCredentials(endpoint).catch(() => {
throw unprocessable(
`${PROVIDER_LABELS[endpoint.provider]} credentials are required`,
);
});
if (endpoint.provider === "imessage-photon") {
credentials = preparePhotonConfiguration(endpoint, input, credentials);
}
const identity = await verifyCredentials(endpoint.provider, credentials);
// Once setup has claimed a provider bot identity, every credential repair
// must prove that same identity before secrets can be replaced. A process
// interruption can leave the endpoint in `attention` after the claim but
// before credential persistence; a repeated first-setup request must not
// turn that recovery state into an identity-swap escape hatch.
if (input.action === "reconnect" || Boolean(endpoint.botExternalId)) {
if (
!nativeBotIdentityMatches(
endpoint.provider,
verifiedCurrentIdentity,
identity,
)
) {
throw conflict(
"The provider credentials identify a different bot; create a new connection for that bot instead",
{ code: "chat_bot_identity_changed" },
);
}
}
await assertNativeBotIdentityAvailable(endpoint, identity);
const prepared = await prepareProviderInventory(endpoint, credentials);
const credentialsChangedByDiscovery =
endpoint.provider === "github" &&
prepared.credentials.installationId !== credentials.installationId;
credentials = prepared.credentials;
await options.nativeBotIdentityClaimBarrier?.();
// Claim a fresh provider identity before any newly supplied secrets are
// persisted. The provider-global unique indexes are the multi-process
// arbitration boundary when two setup requests both pass the optimistic
// preflight. A process that stops after this small claim leaves a
// fail-closed attention endpoint that the same operator can reconnect;
// it never leaves credentials bound to the losing endpoint.
if (!endpoint.botExternalId) {
try {
const claimed = await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatEndpointWork(tx);
const rows = await tx
.update(chatEndpoints)
.set({
status: "attention",
providerAccountId: identity.providerAccountId ?? null,
providerAccountLabel: identity.providerAccountLabel ?? null,
botExternalId: identity.botExternalId ?? null,
botUsername: identity.botUsername ?? null,
botDisplayName:
identity.botLabel ??
endpoint.botDisplayName ??
record.assignedAgentName,
healthMessage: "Provider setup must be completed",
lastError: null,
updatedAt: new Date(),
})
.where(
and(
eq(chatEndpoints.id, endpoint.id),
eq(chatEndpoints.status, endpoint.status),
isNull(chatEndpoints.botExternalId),
),
)
.returning({ id: chatEndpoints.id });
await credentialLease.assertOwned(tx);
return rows;
});
if (claimed.length !== 1) {
throw conflict("This chat connection changed during setup", {
code: "chat_endpoint_setup_changed",
});
}
} catch (error) {
if (isNativeBotIdentityUniqueViolation(error)) {
throw nativeBotIdentityConflict(endpoint.provider);
}
throw error;
}
}
if (endpoint.provider === "imessage-photon") {
await persistPhotonConfiguration(endpoint, credentials, credentialLease);
}
if (
(input.credentials && Object.keys(input.credentials).length > 0) ||
credentialsChangedByDiscovery
)
await persistCredentials(
endpoint,
credentials,
credentialLease,
actorUserId,
);
if (prepared.inventory)
await reconcileProviderResourceRows(
endpoint,
prepared.inventory,
credentialLease,
);
const updatedAt = new Date();
const waitingForSlackConfiguration =
endpoint.provider === "slack" &&
(input.action === "configure" || input.action === "reconnect");
try {
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const current = await tx
.select({ setup: chatEndpoints.setup })
.from(chatEndpoints)
.where(eq(chatEndpoints.id, endpoint.id))
.for("update")
.then((rows) => rows[0] ?? null);
if (!current) throw notFound("Chat endpoint not found");
if (endpoint.provider === "speko") {
await installSpekoVoiceTools(tx, endpoint, actorUserId);
}
const observedSlackUrl = (current.setup as InternalSetupState).slackCallbackSurfaces?.events?.url;
const preserveSlackVerification = isProvisionedSlackSetup(current.setup.slackSetupMethod)
&& (current.setup as InternalSetupState).slackVerificationSigningFingerprint === createHash("sha256").update(credentials.signingSecret ?? "").digest("hex")
&& Boolean(observedSlackUrl && slackCallbackMatchesPublicUrl(observedSlackUrl, `${getWebhookPublicBaseUrl()}/api/chat-webhooks/${endpoint.publicId}/slack`));
await notifyChatEndpointWork(tx);
await tx
.update(chatEndpoints)
.set({
status: "verifying",
providerAccountId: identity.providerAccountId ?? null,
providerAccountLabel: identity.providerAccountLabel ?? null,
botExternalId: identity.botExternalId ?? null,
botUsername: identity.botUsername ?? null,
botDisplayName:
identity.botLabel ??
endpoint.botDisplayName ??
record.assignedAgentName,
healthMessage: waitingForSlackConfiguration
? "Finish provider webhook configuration"
: "Waiting for a test conversation",
lastError: null,
lastEventAt: null,
setup: {
...current.setup,
...(endpoint.provider === "github" ? { github: { ...current.setup.github, stage: current.setup.github?.stage ?? "connect", appOwnerType: identity.githubOwnerType, appOwnerLogin: identity.githubOwnerLogin } } : {}),
step: waitingForSlackConfiguration ? "provider_setup" : "test",
...(endpoint.provider === "imessage-photon" ? { photonIntakeAfter: (current.setup as InternalSetupState).photonIntakeAfter ?? updatedAt.toISOString() } : {}),
testStartedAt: waitingForSlackConfiguration
? null
: updatedAt.toISOString(),
webhookVerifiedAt: waitingForSlackConfiguration
? preserveSlackVerification ? (current.setup.webhookVerifiedAt ?? null) : null
: (current.setup.webhookVerifiedAt ?? null),
runtimeGeneration: runtimeGeneration(current.setup) + 1,
} as InternalSetupState,
updatedAt,
})
.where(eq(chatEndpoints.id, endpoint.id));
await tx
.update(toolConnections)
.set({
status: "active",
enabled: true,
healthStatus: "healthy",
healthMessage: "Provider credentials verified",
healthCheckedAt: updatedAt,
updatedAt,
})
.where(eq(toolConnections.id, endpoint.connectionId));
await tx
.update(toolApplications)
.set({ status: "active", archivedAt: null, updatedAt })
.where(eq(toolApplications.id, record.applicationId));
await credentialLease.assertOwned(tx);
});
} catch (error) {
if (isNativeBotIdentityUniqueViolation(error)) {
throw nativeBotIdentityConflict(endpoint.provider);
}
throw error;
}
try {
const next = await endpointRecord(endpoint.id);
if (!next) throw notFound("Chat endpoint not found");
await invalidateRuntime(endpoint.id);
if (endpoint.provider === "github" && input.action === "reconnect" && endpoint.setup.github?.cloudRegistrationId) {
await repairGitHubCloudSetup(endpoint, credentialLease, actorUserId);
}
if (endpoint.provider === "github" && input.action === "reconnect" && !endpoint.setup.github?.cloudRegistrationId) {
await repairGitHubWebhookSetup(endpoint, credentials, credentialLease, actorUserId);
}
if (next.endpoint.provider === "speko") {
await configureSpekoTools(endpoint, credentials, credentialLease, previousSpekoSigningSecret);
}
if (next.endpoint.provider === "discord")
await reconcileDiscordCommands(endpoint.id, credentialLease, true);
await runtimeFor(next.endpoint, {
requireDiscordOwnership: next.endpoint.provider === "discord",
waitForDiscordOwnership: next.endpoint.provider === "discord",
});
if (endpoint.provider === "telegram" && getWebhookPublicBaseUrl()) {
await registerTelegramWebhook(endpoint, input, credentials, credentialLease, next);
}
} catch (error) {
const failure = redactError(error);
await invalidateRuntime(endpoint.id).catch(() => undefined);
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatEndpointWork(tx);
await tx
.update(chatEndpoints)
.set({
status: "attention",
healthMessage: "Provider setup needs attention",
lastError: failure,
updatedAt: new Date(),
})
.where(eq(chatEndpoints.id, endpoint.id));
await tx
.update(toolConnections)
.set({
healthStatus: "degraded",
healthMessage: "Provider setup failed",
lastError: failure,
healthCheckedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(toolConnections.id, endpoint.connectionId));
await credentialLease.assertOwned(tx);
});
throw unprocessable(failure, { code: "chat_provider_setup_failed" });
}
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await logActivity(tx as unknown as Db, {
companyId: endpoint.companyId,
actorType: "user",
actorId: actorUserId ?? "board",
action:
input.action === "reconnect"
? "chat_endpoint.reconnected"
: "chat_endpoint.credentials_verified",
entityType: "tool_connection",
entityId: endpoint.connectionId,
details: { endpointId: endpoint.id, provider: endpoint.provider },
});
await credentialLease.assertOwned(tx);
});
return get(endpoint.id);
}
function runtimeConfig(
endpoint: EndpointRow,
userName: string,
credentials: Record<string, string>,
): ResolvedChatSdkProviderConfig {
if (endpoint.provider === "speko") throw conflict("Speko uses the voice runtime");
if (endpoint.provider === "imessage-photon") return {
provider: "imessage-photon", userName,
intakeAfter: Date.parse(String((endpoint.setup as InternalSetupState).photonIntakeAfter ?? endpoint.setup.testStartedAt ?? endpoint.createdAt.toISOString())),
credentials: { allocation: credentials.allocation === "shared" ? "shared" : "dedicated", projectId: credentials.projectId, lineId: credentials.lineId, projectSecret: credentials.projectSecret, phoneNumber: endpoint.botExternalId! },
};
if (endpoint.provider === "slack")
return slackRuntimeConfig(endpoint, userName, credentials);
if (endpoint.provider === "telegram")
return {
provider: "telegram",
userName,
maxDownloadBytes: MAX_ATTACHMENT_BYTES,
credentials: {
botToken: credentials.botToken,
secretToken: credentials.webhookSecret,
},
};
if (endpoint.provider === "discord")
return {
provider: "discord",
userName,
credentials: {
applicationId: credentials.applicationId,
botToken: credentials.botToken,
guildId: credentials.guildId,
},
};
if (endpoint.provider === "microsoft-teams")
return {
provider: "microsoft-teams",
userName,
credentials: {
appId: credentials.clientId,
appPassword: credentials.clientSecret,
appTenantId: credentials.tenantId,
appType: "SingleTenant",
},
};
return {
provider: "github",
userName,
credentials: {
appId: credentials.appId,
privateKey: credentials.privateKey,
installationId: credentials.installationId
? Number(credentials.installationId)
: undefined,
webhookSecret: credentials.webhookSecret,
},
};
}
return { configure, configureWithCredentialLease, runtimeConfig };
}
@@ -0,0 +1,507 @@
import { prepareSlackInventory } from "./slack/inventory.js";
import { prepareGitHubInventory } from "./github/inventory.js";
import { prepareDiscordInventory } from "./discord/inventory.js";
import type { VerifiedProviderIdentity } from "./runtime-types.js";
import { verifySlackCredentials } from "./slack/credentials.js";
import { verifyDiscordCredentials } from "./discord/credentials.js";
import { verifyTelegramCredentials } from "./telegram/credentials.js";
import { verifyGitHubCredentials } from "./github/credentials.js";
import { verifySpekoCredentials } from "./speko/credentials.js";
import { verifyTeamsCredentials } from "./microsoft-teams/credentials.js";
import { createPhotonCredentialChecks } from "./imessage-photon/credentials.js";
import type { EndpointRow, DbOrTransaction } from "./runtime-types.js";
import { type CredentialMutationLeaseGuard } from "../chat-credential-mutation-lease.js";
import { photonSharedScope } from "../photon/cloud.js";
import { photonChannelConfigurationSchema } from "@paperclipai/shared";
import {
randomBytes,
randomUUID,
} from "node:crypto";
import {
and,
eq,
inArray,
ne,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatEndpoints,
companySecretBindings,
toolConnections,
} from "@paperclipai/db";
import type {
ChatProvider,
ToolCredentialSecretRef,
} from "@paperclipai/shared";
import { isUniqueViolation } from "../../db-errors.js";
import {
conflict,
notFound,
unprocessable,
} from "../../errors.js";
import { logger } from "../../middleware/logger.js";
import { type ChatProviderInventoryResult } from "../chat-provider-inventory.js";
import {
normalizeMicrosoftTeamsCredentialIds,
} from "../chat-teams-credentials.js";
import { secretService } from "../secrets.js";
type ChatCredentialsOptions = {
secrets: ReturnType<typeof secretService>;
PROVIDER_LABELS: Record<ChatProvider, string>;
REQUIRED_CREDENTIALS: Record<Exclude<ChatProvider, "github">, readonly string[]>;
PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS: number;
fetchImpl: typeof globalThis.fetch;
redactError: (error: unknown) => string;
credentialFingerprint: (refs: ToolCredentialSecretRef[]) => string;
verifyPhotonCredentials: ReturnType<typeof createPhotonCredentialChecks>["verifyPhotonCredentials"];
};
export function createChatCredentials(db: Db, options: ChatCredentialsOptions) {
const { secrets, PROVIDER_LABELS, REQUIRED_CREDENTIALS, PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS, fetchImpl, redactError, credentialFingerprint, verifyPhotonCredentials } = options;
async function normalizedCredentials(
endpoint: EndpointRow,
supplied: Record<string, string> | undefined,
) {
const values = await resolveCredentials(endpoint).catch(
() => ({}) as Record<string, string>,
);
for (const [key, rawValue] of Object.entries(supplied ?? {})) {
if (typeof rawValue === "string" && rawValue.trim())
values[key] = key === "privateKey" ? rawValue : rawValue.trim();
}
if (endpoint.provider === "telegram" && !values.webhookSecret) {
values.webhookSecret = randomBytes(32).toString("hex");
}
const required =
endpoint.provider === "github"
? ["appId", "privateKey", "webhookSecret"]
: REQUIRED_CREDENTIALS[endpoint.provider];
const missing = required.filter((key) => !values[key]);
if (missing.length > 0) {
throw unprocessable(
`Missing required ${PROVIDER_LABELS[endpoint.provider]} credentials: ${missing.join(", ")}`,
);
}
return endpoint.provider === "microsoft-teams"
? normalizeMicrosoftTeamsCredentialIds(values)
: values;
}
async function verifyCredentials(
provider: ChatProvider,
credentials: Record<string, string>,
): Promise<VerifiedProviderIdentity> {
if (provider === "imessage-photon") return verifyPhotonCredentials(credentials);
if (provider === "slack") return verifySlackCredentials(credentials, fetchImpl, PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS);
if (provider === "discord") return verifyDiscordCredentials(credentials, fetchImpl, redactError);
if (provider === "telegram") return verifyTelegramCredentials(credentials, fetchImpl, PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS);
if (provider === "github") return verifyGitHubCredentials(credentials, fetchImpl, PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS);
if (provider === "speko") return verifySpekoCredentials(credentials, fetchImpl);
return verifyTeamsCredentials(credentials, fetchImpl, PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS);
}
function nativeBotIdentityKey(
provider: ChatProvider,
identity: VerifiedProviderIdentity,
): string | null {
const account = identity.providerAccountId?.trim().toLowerCase();
const bot = identity.botExternalId?.trim().toLowerCase();
if (provider === "github" || provider === "discord") {
if (bot) return `${provider}:app:${bot}`;
return null;
}
if (!account || !bot) return null;
return `${provider}:${account}:${bot}`;
}
function providerUsesGlobalApplicationIdentity(
provider: ChatProvider,
): boolean {
return (
provider === "github" ||
provider === "discord" ||
provider === "microsoft-teams" || provider === "imessage-photon"
);
}
function nativeBotIdentityMatches(
provider: ChatProvider,
current: VerifiedProviderIdentity,
incoming: VerifiedProviderIdentity,
): boolean {
const currentKey = nativeBotIdentityKey(provider, current);
return (
currentKey !== null &&
currentKey === nativeBotIdentityKey(provider, incoming)
);
}
function legacyGitHubLabelsMatchVerifiedCredentials(
current: VerifiedProviderIdentity,
verified: VerifiedProviderIdentity,
): boolean {
return (
!current.botExternalId &&
Boolean(verified.botExternalId) &&
current.providerAccountId?.trim().toLowerCase() ===
verified.providerAccountId?.trim().toLowerCase() &&
current.botUsername?.trim().toLowerCase() ===
verified.botUsername?.trim().toLowerCase()
);
}
async function assertNativeBotIdentityAvailable(
endpoint: EndpointRow,
identity: VerifiedProviderIdentity,
): Promise<void> {
const key = nativeBotIdentityKey(endpoint.provider, identity);
if (!key) {
throw unprocessable(
`${PROVIDER_LABELS[endpoint.provider]} did not return a stable native bot identity`,
);
}
const candidates = await db
.select({
id: chatEndpoints.id,
companyId: chatEndpoints.companyId,
assignedAgentId: chatEndpoints.assignedAgentId,
providerAccountId: chatEndpoints.providerAccountId,
botExternalId: chatEndpoints.botExternalId,
botUsername: chatEndpoints.botUsername,
})
.from(chatEndpoints)
.where(
and(
providerUsesGlobalApplicationIdentity(endpoint.provider)
? undefined
: eq(chatEndpoints.companyId, endpoint.companyId),
eq(chatEndpoints.provider, endpoint.provider),
ne(chatEndpoints.id, endpoint.id),
endpoint.provider === "imessage-photon" ? ne(chatEndpoints.status, "archived") : inArray(chatEndpoints.status, [
"verifying",
"active",
"paused",
"attention",
]),
),
);
const conflictEndpoint = candidates.find((candidate) => {
if (!providerUsesGlobalApplicationIdentity(endpoint.provider)) {
return nativeBotIdentityKey(endpoint.provider, candidate) === key;
}
const candidateAppId = candidate.botExternalId?.trim().toLowerCase();
const incomingAppId = identity.botExternalId?.trim().toLowerCase();
if (candidateAppId && incomingAppId)
return candidateAppId === incomingAppId;
return (
candidate.providerAccountId?.trim().toLowerCase() ===
identity.providerAccountId?.trim().toLowerCase() &&
candidate.botUsername?.trim().toLowerCase() ===
identity.botUsername?.trim().toLowerCase()
);
});
if (conflictEndpoint) {
throw conflict(
`This ${PROVIDER_LABELS[endpoint.provider]} bot already represents another Paperclip agent connection`,
conflictEndpoint.companyId === endpoint.companyId
? {
code: "chat_bot_identity_in_use",
endpointId: conflictEndpoint.id,
assignedAgentId: conflictEndpoint.assignedAgentId,
}
: { code: "chat_bot_identity_in_use" },
);
}
}
function isNativeBotIdentityUniqueViolation(error: unknown): boolean {
return (
isUniqueViolation(error, "chat_endpoints_photon_number_uq") ||
isUniqueViolation(error, "chat_endpoints_live_bot_external_uq") ||
isUniqueViolation(error, "chat_endpoints_live_discord_bot_external_uq") ||
isUniqueViolation(
error,
"chat_endpoints_live_global_app_bot_external_uq",
) ||
isUniqueViolation(error, "chat_endpoints_live_bot_username_uq")
);
}
function nativeBotIdentityConflict(provider: ChatProvider) {
return conflict(
`This ${PROVIDER_LABELS[provider]} bot already represents another Paperclip agent connection`,
{ code: "chat_bot_identity_in_use" },
);
}
async function persistCredentials(
endpoint: EndpointRow,
credentials: Record<string, string>,
credentialLease: CredentialMutationLeaseGuard,
actorUserId?: string | null,
) {
await credentialLease.assertOwned();
const previousRefs = await db
.select({ refs: toolConnections.credentialSecretRefs })
.from(toolConnections)
.where(
and(
eq(toolConnections.companyId, endpoint.companyId),
eq(toolConnections.id, endpoint.connectionId),
),
)
.then((rows) => rows[0]?.refs ?? []);
const refs: ToolCredentialSecretRef[] = [];
try {
for (const [key, value] of Object.entries(credentials)) {
if (endpoint.provider === "imessage-photon" && key !== "projectSecret") continue;
await credentialLease.assertOwned();
const suffix = randomUUID();
const secret = await secrets.create(
endpoint.companyId,
{
name: `chat.${endpoint.provider}.${endpoint.id.slice(0, 8)}.${key}.${suffix.slice(0, 8)}`,
key: `CHAT_${endpoint.provider.replaceAll("-", "_")}_${endpoint.id.replaceAll("-", "_")}_${key}_${suffix}`.toUpperCase(),
provider: "local_encrypted",
managedMode: "paperclip_managed",
value,
description: `${PROVIDER_LABELS[endpoint.provider]} channel credential for endpoint ${endpoint.id}`,
},
{ userId: actorUserId ?? undefined },
);
refs.push({
secretId: secret.id,
versionSelector: "latest",
configPath: `credentials.${key}`,
required: true,
label: key,
projectionClass: "unclassified",
});
await credentialLease.assertOwned();
}
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const current = await tx
.select({ refs: toolConnections.credentialSecretRefs })
.from(toolConnections)
.where(
and(
eq(toolConnections.companyId, endpoint.companyId),
eq(toolConnections.id, endpoint.connectionId),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
if (!current) throw notFound("Chat connection not found");
if (
credentialFingerprint(current.refs) !==
credentialFingerprint(previousRefs)
) {
throw conflict("Chat credentials changed during rotation", {
code: "chat_endpoint_credentials_changed",
});
}
await tx
.delete(companySecretBindings)
.where(
and(
eq(companySecretBindings.companyId, endpoint.companyId),
eq(companySecretBindings.targetType, "tool_connection"),
eq(companySecretBindings.targetId, endpoint.connectionId),
),
);
if (refs.length > 0) {
await tx.insert(companySecretBindings).values(
refs.map((ref) => ({
companyId: endpoint.companyId,
secretId: ref.secretId,
targetType: "tool_connection",
targetId: endpoint.connectionId,
configPath: ref.configPath,
versionSelector: String(ref.versionSelector ?? "latest"),
required: ref.required,
label: ref.label ?? null,
projectionClass: ref.projectionClass ?? "unclassified",
projectionAllowlistKey: ref.projectionAllowlistKey ?? null,
})),
);
}
await tx
.update(toolConnections)
.set({ credentialSecretRefs: refs, updatedAt: new Date() })
.where(eq(toolConnections.id, endpoint.connectionId));
await credentialLease.assertOwned(tx);
});
} catch (error) {
// A lost database acknowledgement can be ambiguous even when the
// transaction committed. Delete only secrets proven unbound after the
// failure; when that proof cannot be read, leak safely for later cleanup
// instead of deleting a possibly-authoritative credential.
const boundSecretIds =
refs.length === 0
? new Set<string>()
: await db
.select({ secretId: companySecretBindings.secretId })
.from(companySecretBindings)
.where(
and(
eq(companySecretBindings.companyId, endpoint.companyId),
inArray(
companySecretBindings.secretId,
refs.map((ref) => ref.secretId),
),
),
)
.then((rows) => new Set(rows.map((row) => row.secretId)))
.catch(() => null);
if (boundSecretIds) {
await Promise.allSettled(
refs
.filter((ref) => !boundSecretIds.has(ref.secretId))
.map((ref) => secrets.remove(ref.secretId)),
);
}
throw error;
}
const currentIds = new Set(refs.map((ref) => ref.secretId));
const retired = previousRefs.filter((ref) => !currentIds.has(ref.secretId));
const results = await Promise.allSettled(
retired.map((ref) => secrets.remove(ref.secretId)),
);
if (results.some((result) => result.status === "rejected")) {
logger.warn(
{ endpointId: endpoint.id },
"chat credential rotation left an unbound retired secret for later cleanup",
);
}
return refs;
}
async function clearCredentials(
endpoint: EndpointRow,
credentialLease: CredentialMutationLeaseGuard,
) {
let refs: ToolCredentialSecretRef[] = [];
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const current = await tx
.select({ refs: toolConnections.credentialSecretRefs })
.from(toolConnections)
.where(
and(
eq(toolConnections.companyId, endpoint.companyId),
eq(toolConnections.id, endpoint.connectionId),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
if (!current) throw notFound("Chat connection not found");
refs = current.refs;
await tx
.delete(companySecretBindings)
.where(
and(
eq(companySecretBindings.companyId, endpoint.companyId),
eq(companySecretBindings.targetType, "tool_connection"),
eq(companySecretBindings.targetId, endpoint.connectionId),
),
);
await tx
.update(toolConnections)
.set({ credentialSecretRefs: [], updatedAt: new Date() })
.where(eq(toolConnections.id, endpoint.connectionId));
await credentialLease.assertOwned(tx);
});
const results = await Promise.allSettled(
refs.map((ref) => secrets.remove(ref.secretId)),
);
if (results.some((result) => result.status === "rejected")) {
logger.warn(
{ endpointId: endpoint.id },
"chat endpoint removal left an unbound retired secret for later cleanup",
);
}
}
async function resolveCredentialRefs(
endpoint: EndpointRow,
refs: ToolCredentialSecretRef[],
database: DbOrTransaction = db,
): Promise<Record<string, string>> {
const values: Record<string, string> = {};
for (const ref of refs) {
const key = ref.configPath.replace(/^credentials\./, "");
values[key] = await (database === db ? secrets : secretService(database as Db)).resolveSecretValue(
endpoint.companyId,
ref.secretId,
ref.versionSelector ?? "latest",
{
consumerType: "tool_connection",
consumerId: endpoint.connectionId,
configPath: ref.configPath,
actorType: "system",
actorId: null,
},
);
}
return values;
}
async function resolveCredentials(
endpoint: EndpointRow,
database: DbOrTransaction = db,
): Promise<Record<string, string>> {
const connection = await database
.select({ refs: toolConnections.credentialSecretRefs, config: toolConnections.config })
.from(toolConnections)
.where(
and(
eq(toolConnections.companyId, endpoint.companyId),
eq(toolConnections.id, endpoint.connectionId),
),
)
.then((rows) => rows[0] ?? null);
if (!connection) throw notFound("Chat connection not found");
const values = await resolveCredentialRefs(endpoint, connection.refs, database);
if (endpoint.provider === "imessage-photon") {
const configuration = photonChannelConfigurationSchema.parse(connection.config.photon);
return { ...values, ...configuration, lineId: configuration.allocation === "shared" ? photonSharedScope(configuration.projectId) : configuration.lineId };
}
return values;
}
async function prepareProviderInventory(
endpoint: EndpointRow,
credentials: Record<string, string>,
): Promise<{
credentials: Record<string, string>;
inventory: ChatProviderInventoryResult | null;
managementUrl?: string;
}> {
try {
if (endpoint.provider === "slack") return await prepareSlackInventory(credentials, fetchImpl);
if (endpoint.provider === "github") return await prepareGitHubInventory(credentials, fetchImpl);
if (endpoint.provider === "discord") return await prepareDiscordInventory(credentials, fetchImpl);
return { credentials, inventory: null };
} catch (error) {
throw unprocessable(redactError(error), {
code: "chat_provider_inventory_failed",
});
}
}
return { normalizedCredentials, verifyCredentials, nativeBotIdentityMatches, legacyGitHubLabelsMatchVerifiedCredentials, assertNativeBotIdentityAvailable, isNativeBotIdentityUniqueViolation, nativeBotIdentityConflict, persistCredentials, clearCredentials, resolveCredentialRefs, resolveCredentials, prepareProviderInventory };
}
@@ -0,0 +1,120 @@
import { createChatPublicationReplacement } from "../publication-replacement.js";
import { notifyChatPublicationWork } from "../../chat-work-notifications.js";
import {
eq,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatPublications,
} from "@paperclipai/db";
import type {
SafeChatPublicationPayload,
} from "@paperclipai/shared";
import {
discordMarkdownRequiresAttachment,
} from "../../chat-discord.js";
type PublicationPreparationOptions = {
runPublicationToReplace: ReturnType<typeof createChatPublicationReplacement>["runPublicationToReplace"];
};
export function createDiscordPublicationPreparation(db: Db, options: PublicationPreparationOptions) {
const { runPublicationToReplace } = options;
async function prepareDiscordPublication(publication: typeof chatPublications.$inferSelect, persisted: SafeChatPublicationPayload): Promise<typeof chatPublications.$inferSelect> {
if (
persisted.transportPart ||
persisted.card ||
persisted.interactionId ||
!discordMarkdownRequiresAttachment(persisted.text)
) {
return publication;
}
// A normal response can replace its run's working placeholder. Discord's
// adapter cannot add a file while editing, so first turn that placeholder
// into a durable hand-off and then publish the complete Markdown as its
// own ordered attachment row. Publications without a replacement lane
// remain one provider call: a short caption and the lossless file.
const replacesProgressMessage = Boolean(
await runPublicationToReplace(publication, persisted),
);
return db.transaction(async (tx) => {
const current = await tx
.select()
.from(chatPublications)
.where(eq(chatPublications.id, publication.id))
.for("update")
.then((rows) => rows[0] ?? null);
if (!current) return publication;
const currentPayload = current.payload as SafeChatPublicationPayload;
if (
currentPayload.transportPart ||
currentPayload.card ||
currentPayload.interactionId ||
!["pending", "retry"].includes(current.state) ||
!discordMarkdownRequiresAttachment(currentPayload.text)
) {
return current;
}
const batchId = current.id;
const updatedAt = new Date();
const transportPart = (
index: number,
count: number,
mode: "inline" | "discord_markdown_attachment",
) => ({
batchId,
count,
index,
mode,
orderKey: `${batchId}:${String(index).padStart(4, "0")}`,
});
if (!replacesProgressMessage) {
const attachmentPayload: SafeChatPublicationPayload = {
...currentPayload,
transportPart: transportPart(0, 1, "discord_markdown_attachment"),
};
await notifyChatPublicationWork(tx);
await tx
.update(chatPublications)
.set({ payload: attachmentPayload, updatedAt })
.where(eq(chatPublications.id, current.id));
return { ...current, payload: attachmentPayload, updatedAt };
}
const handoffPayload: SafeChatPublicationPayload = {
...currentPayload,
text: "This response needs a separate attachment because it exceeds the message limit.",
transportPart: transportPart(0, 2, "inline"),
};
delete handoffPayload.attachmentIds;
await notifyChatPublicationWork(tx);
await tx
.update(chatPublications)
.set({ payload: handoffPayload, updatedAt })
.where(eq(chatPublications.id, current.id));
const attachmentPayload: SafeChatPublicationPayload = {
...currentPayload,
transportPart: transportPart(1, 2, "discord_markdown_attachment"),
};
delete attachmentPayload.progressState;
await notifyChatPublicationWork(tx);
await tx.insert(chatPublications).values({
companyId: current.companyId,
endpointId: current.endpointId,
conversationId: current.conversationId,
issueId: current.issueId,
commentId: current.commentId,
idempotencyKey: `discord-markdown-attachment:${current.id}`,
payload: attachmentPayload,
state: "pending",
createdAt: sql`(select ${chatPublications.createdAt} from ${chatPublications} where ${chatPublications.id} = ${current.id})`,
updatedAt,
});
return { ...current, payload: handoffPayload, updatedAt };
});
}
return { prepareDiscordPublication };
}
@@ -0,0 +1,189 @@
import { createChatSdkStatePersistence } from "../sdk-state.js";
import type { DbTransaction, RuntimeContext, LifecycleRuntimeFence } from "../runtime-types.js";
import {
and,
eq,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatConversations,
chatExternalPrincipals,
chatPublications,
issueThreadInteractions,
} from "@paperclipai/db";
import type {
SafeChatPublicationPayload,
AskUserQuestionsInteraction,
} from "@paperclipai/shared";
import {
forbidden,
} from "../../../errors.js";
import { logger } from "../../../middleware/logger.js";
import {
type ChatSdkCallbackEvent,
} from "../../chat-sdk-runtime.js";
import type {
ChatSdkStatePersistence,
} from "../../chat-sdk-state.js";
import {
deleteDiscordQuestionFormCorrection,
retainDiscordQuestionFormCorrection,
} from "../../chat-discord-question-forms.js";
import { issueThreadInteractionService } from "../../issue-thread-interactions.js";
import {
isChatQuestionFormOpenActionId,
loadChatQuestionFormSubmissionToken,
resolveChatQuestionFormOpen,
} from "../../chat-question-forms.js";
import type {
ModalResponse,
ModalSubmitEvent,
} from "chat";
import type { EndpointRecord } from "../runtime-types.js";
import type { ChatQuestionFormValidationResult } from "../../chat-question-forms.js";
type LoadedForm = NonNullable<Awaited<ReturnType<typeof loadChatQuestionFormSubmissionToken>>>;
type DiscordFormActor = { principal: typeof chatExternalPrincipals.$inferSelect; userId: string };
type DiscordFormCorrectionInput = {
record: EndpointRecord; event: ChatSdkCallbackEvent<ModalSubmitEvent>; runtimeContext: RuntimeContext;
conversation: typeof chatConversations.$inferSelect; principal: DiscordFormActor; loaded: LoadedForm;
originalPublication: typeof chatPublications.$inferSelect; providerMessageId: string;
interaction: AskUserQuestionsInteraction;
payload: SafeChatPublicationPayload; validation: Exclude<ChatQuestionFormValidationResult, { ok: true }>;
deny: (code: string) => Promise<ModalResponse>;
};
type DiscordQuestionFormsOptions = {
requireCurrentExternalActionAuthorization: (tx: DbTransaction, input: { conversationId: string; endpointId: string; expectedUserId: string; principalId: string; runtimeContext: LifecycleRuntimeFence }) => Promise<void>;
isExternalActionAuthorizationChange: (error: unknown) => boolean;
persistence: ChatSdkStatePersistence;
};
export function createDiscordQuestionForms(db: Db, options: DiscordQuestionFormsOptions) {
const { requireCurrentExternalActionAuthorization, isExternalActionAuthorizationChange, persistence } = options;
const denialError = () => forbidden("This chat form is not a current Paperclip question");
async function handleDiscordFormCorrection(input: DiscordFormCorrectionInput) {
const { record, event, runtimeContext, conversation, principal, loaded, originalPublication, providerMessageId, interaction, payload, validation, deny } = input;
const openAction = payload.card?.actions?.find(
(action) =>
action.type === "callback" &&
isChatQuestionFormOpenActionId(action.actionId),
);
const openActionId =
openAction?.type === "callback" ? openAction.actionId : null;
if (!openActionId) return deny("chat_modal_open_token_not_current");
try {
return await db.transaction(async (tx) => {
await requireCurrentExternalActionAuthorization(tx, {
conversationId: conversation.id,
endpointId: record.endpoint.id,
expectedUserId: principal.userId!,
principalId: principal.principal.id,
runtimeContext,
});
const pending = await tx
.select({ id: issueThreadInteractions.id })
.from(issueThreadInteractions)
.where(
and(
eq(
issueThreadInteractions.companyId,
record.endpoint.companyId,
),
eq(issueThreadInteractions.issueId, conversation.issueId),
eq(issueThreadInteractions.id, interaction.id),
eq(issueThreadInteractions.status, "pending"),
),
)
.for("update")
.then((rows) => rows[0]);
const resolved = pending
? await resolveChatQuestionFormOpen(tx, {
companyId: record.endpoint.companyId,
endpointId: record.endpoint.id,
conversationId: conversation.id,
interaction,
openActionId,
})
: null;
if (
!resolved ||
resolved.submitActionId !== loaded.payload.formActionId ||
resolved.publicationId !== originalPublication.id
)
throw denialError();
return retainDiscordQuestionFormCorrection(
createChatSdkStatePersistence(tx as unknown as Db),
{
companyId: record.endpoint.companyId,
endpointId: record.endpoint.id,
},
{
principalId: principal.principal.id,
userId: principal.userId!,
externalUserId: event.event.user.userId,
conversationId: conversation.id,
publicationId: originalPublication.id,
providerMessageId,
threadId: conversation.externalThreadId,
interactionId: interaction.id,
openActionId,
submitActionId: resolved.submitActionId,
parentExpiresAt: loaded.payload.expiresAt,
modal: resolved.modal,
values: event.event.values,
fieldErrors: validation.fieldErrors,
},
);
});
} catch (error) {
if (
isExternalActionAuthorizationChange(error) ||
(error &&
typeof error === "object" &&
"status" in error &&
error.status === 403)
)
return deny("chat_modal_correction_not_authorized");
throw error;
}
}
async function acceptDiscordQuestionForm(input: { record: EndpointRecord; event: ChatSdkCallbackEvent<ModalSubmitEvent>; principal: DiscordFormActor; loaded: LoadedForm }): Promise<ModalResponse> {
const { record, event, principal, loaded } = input;
try {
await deleteDiscordQuestionFormCorrection(
persistence,
{
companyId: record.endpoint.companyId,
endpointId: record.endpoint.id,
},
{
principalId: principal.principal.id,
userId: principal.userId!,
externalUserId: event.event.user.userId,
},
loaded.payload.formActionId,
);
} catch {
// The answer receipt is already committed. Cleanup is retryable on
// exact replay/expiry access and cannot relabel an accepted answer.
logger.warn(
{
endpointId: record.endpoint.id,
code: "discord_question_correction_cleanup_failed",
},
"Discord correction draft cleanup will be retried",
);
}
return { action: "clear" };
}
return { handleDiscordFormCorrection, acceptDiscordQuestionForm };
}
@@ -0,0 +1,476 @@
import { slackSetupState } from "./slack/setup.js";
import { assertChatExecutionDefaultsAccess } from "../chat-execution-defaults.js";
import type { EndpointRow } from "./runtime-types.js";
import { chatCredentialMutationLease } from "../chat-credential-mutation-lease.js";
import { slackChatRegistrationService, slackRegistrationProjection } from "../chat-slack-registration.js";
import { chatSlackRegistrations } from "@paperclipai/db";
import { defaultSlackAppConfiguration } from "@paperclipai/shared";
import {
randomBytes,
randomUUID,
} from "node:crypto";
import {
and,
desc,
eq,
ne,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
agents,
chatEndpoints,
toolApplications,
toolConnections,
} from "@paperclipai/db";
import type {
ChatAdapterCapabilities,
ChatEndpoint,
ChatProvider,
CreateChatEndpointInput,
UpdateChatEndpointInput,
} from "@paperclipai/shared";
import {
isAgentStatusInvokable,
} from "@paperclipai/shared";
import {
badRequest,
conflict,
notFound,
unprocessable,
} from "../../errors.js";
import {
logActivity,
} from "../activity-log.js";
import type { EndpointRecord } from "./runtime-types.js";
type ChatEndpointsOptions = {
endpointRecord: (endpointId: string) => Promise<EndpointRecord | null>;
CAPABILITIES: Record<ChatProvider, ChatAdapterCapabilities>;
PROVIDER_LABELS: Record<ChatProvider, string>;
withCredentialMutationLease: ReturnType<typeof chatCredentialMutationLease>;
getPublicBaseUrl: () => string | null;
getWebhookPublicBaseUrl: () => string | null;
slackRegistration: Pick<ReturnType<typeof slackChatRegistrationService>, "cleanup" | "managerGrants">;
};
export function createChatEndpoints(db: Db, options: ChatEndpointsOptions) {
const { endpointRecord, CAPABILITIES, PROVIDER_LABELS, withCredentialMutationLease, getPublicBaseUrl, getWebhookPublicBaseUrl } = options;
function iso(value: Date | null | undefined): string | null {
return value ? value.toISOString() : null;
}
function providerSetupState(
endpoint: Pick<
EndpointRow,
| "provider"
| "providerAccountId"
| "publicId"
| "status"
| "setup"
| "botUsername"
>,
publicBaseUrl: string | null,
assignedAgentName?: string | null,
) {
const path = endpoint.provider === "speko" ? `/api/voice-webhooks/${endpoint.publicId}/tools` : `/api/chat-webhooks/${endpoint.publicId}/${endpoint.provider}`;
const webhookUrl = publicBaseUrl ? `${publicBaseUrl}${path}` : null;
const step = endpoint.status === "active" ? "complete" : endpoint.setup.step;
switch (endpoint.provider) {
case "speko":
return { step, providerUrl: "https://platform.speko.ai/", webhookUrl } as const;
case "imessage-photon": return { step, providerUrl: "https://photon.codes/", testStartedAt: endpoint.setup.testStartedAt } as const;
case "agentmail": return endpoint.setup;
case "slack": {
return slackSetupState({ endpoint, webhookUrl, step, assignedAgentName });
}
case "github":
return {
step,
github: endpoint.setup.github,
testStartedAt: endpoint.setup.testStartedAt,
testSkipped: endpoint.setup.testSkipped,
authorizationUrl: "https://github.com/settings/apps/new",
providerUrl: "https://github.com/settings/installations",
webhookUrl,
webhookVerifiedAt: endpoint.setup.webhookVerifiedAt ?? null,
} as const;
case "discord":
return {
step,
authorizationUrl: "https://discord.com/developers/applications",
providerUrl: endpoint.providerAccountId
? `https://discord.com/channels/${encodeURIComponent(endpoint.providerAccountId)}`
: "https://discord.com/channels/@me",
webhookUrl,
} as const;
case "microsoft-teams":
return {
step,
authorizationUrl: "https://dev.teams.microsoft.com/apps",
providerUrl: "https://teams.microsoft.com/",
messagingEndpoint: webhookUrl,
} as const;
case "telegram":
return {
step,
providerUrl: endpoint.botUsername
? `https://t.me/${encodeURIComponent(endpoint.botUsername)}`
: "https://t.me/BotFather",
webhookUrl,
} as const;
}
}
function serializeEndpoint(
row: Awaited<ReturnType<typeof endpointRecord>>,
): ChatEndpoint {
if (!row) throw notFound("Chat endpoint not found");
const endpoint = row.endpoint;
return {
id: endpoint.id,
companyId: endpoint.companyId,
connectionId: endpoint.connectionId,
publicationMode: endpoint.publicationMode,
externalExecutionPolicy: endpoint.externalExecutionPolicy,
provider: endpoint.provider,
publicId: endpoint.publicId,
status: endpoint.status,
deploymentMode: endpoint.deploymentMode,
assignedAgentId: endpoint.assignedAgentId,
assignedAgentName: row.assignedAgentName,
sponsorUserId: endpoint.sponsorUserId,
providerAccountId: endpoint.providerAccountId,
providerAccountLabel: endpoint.providerAccountLabel,
botExternalId: endpoint.botExternalId,
botUsername: endpoint.botUsername,
botLabel: endpoint.botDisplayName ?? row.assignedAgentName,
botAvatarUrl: endpoint.botAvatarUrl,
communicationInstructions: endpoint.communicationInstructions,
executionDefaults: endpoint.executionDefaults,
...(endpoint.provider === "imessage-photon" && endpoint.botExternalId ? { photonAllocation: endpoint.botExternalId.startsWith("photon-project:") ? "shared" as const : "dedicated" as const } : {}),
allowDirectMessages: endpoint.allowDirectMessages,
requireAtMention: endpoint.requireAtMention,
allowGroupChats: endpoint.allowGroupChats,
allowUnlinkedPeople: endpoint.allowUnlinkedPeople,
replyMode: "subscribed",
capabilities: endpoint.capabilities,
setup: {
...providerSetupState(
endpoint,
getWebhookPublicBaseUrl(),
row.assignedAgentName,
),
...(endpoint.provider === "github"
? {
webhookSecretConfigured: row.credentialSecretRefs.some(
(ref) => ref.configPath === "credentials.webhookSecret",
),
}
: endpoint.provider === "speko" ? {
webhookSecretConfigured: row.credentialSecretRefs.some(
(ref) => ref.configPath === "credentials.signingSecret",
),
} : {}),
},
healthMessage: endpoint.healthMessage,
lastError: endpoint.lastError,
lastActivityAt: iso(endpoint.lastEventAt),
lastPublicationAt: iso(endpoint.lastPublicationAt),
activatedAt: iso(endpoint.activatedAt),
createdAt: endpoint.createdAt.toISOString(),
updatedAt: endpoint.updatedAt.toISOString(),
};
}
async function list(companyId: string) {
const rows = await db
.select({
endpoint: chatEndpoints,
assignedAgentName: agents.name,
connectionName: toolConnections.name,
applicationId: toolConnections.applicationId,
credentialSecretRefs: toolConnections.credentialSecretRefs,
})
.from(chatEndpoints)
.innerJoin(
agents,
and(
eq(agents.id, chatEndpoints.assignedAgentId),
eq(agents.companyId, chatEndpoints.companyId),
),
)
.innerJoin(
toolConnections,
and(
eq(toolConnections.id, chatEndpoints.connectionId),
eq(toolConnections.companyId, chatEndpoints.companyId),
),
)
.where(
and(
eq(chatEndpoints.companyId, companyId),
ne(chatEndpoints.status, "archived"),
),
)
.orderBy(desc(chatEndpoints.updatedAt));
return rows.map((row) => serializeEndpoint(row));
}
async function get(endpointId: string) {
const result = serializeEndpoint(await endpointRecord(endpointId));
if (result.provider === "slack") {
const [registration] = await db.select().from(chatSlackRegistrations).where(eq(chatSlackRegistrations.endpointId, endpointId));
result.setup.slackRegistration = registration ? slackRegistrationProjection(registration) : undefined;
const publicOrigin = getPublicBaseUrl();
result.setup.slackOAuthCallbackUri = publicOrigin ? `${publicOrigin}/api/chat-slack/oauth/callback` : null;
}
return result;
}
async function create(
companyId: string,
input: CreateChatEndpointInput,
actorUserId?: string | null,
) {
if ((input.provider as string) === "agentmail") throw badRequest("Use the email inbox setup API for AgentMail");
if ((input.slackApp || input.slackSetupMethod) && input.provider !== "slack") throw unprocessable("Slack app details only apply to Slack connections");
if (input.slackSetupMethod === "managed" && options.slackRegistration.managerGrants.defaultMethod() !== "managed") throw conflict("Managed Slack setup is unavailable");
const agent = await db
.select({ id: agents.id, name: agents.name, status: agents.status })
.from(agents)
.where(
and(
eq(agents.companyId, companyId),
eq(agents.id, input.assignedAgentId),
),
)
.then((rows) => rows[0] ?? null);
if (!agent)
throw unprocessable("The selected agent does not belong to this company");
if (!isAgentStatusInvokable(agent.status)) {
throw unprocessable(
"The selected agent must be active before it can be connected to chat",
{
code: "chat_agent_not_invokable",
},
);
}
const endpointId = randomUUID();
const publicId = randomBytes(32).toString("base64url");
const applicationId = input.applicationId ?? randomUUID();
const connectionId = randomUUID();
const suffix = endpointId.slice(0, 8);
const name = (
input.name?.trim() || `${PROVIDER_LABELS[input.provider]} — ${agent.name}`
).slice(0, 145);
await db.transaction(async (tx) => {
if (input.applicationId) {
const application = await tx
.select({
id: toolApplications.id,
applicationKey: toolApplications.applicationKey,
metadata: toolApplications.metadata,
status: toolApplications.status,
})
.from(toolApplications)
.where(
and(
eq(toolApplications.companyId, companyId),
eq(toolApplications.id, input.applicationId),
),
)
.then((rows) => rows[0] ?? null);
if (!application) throw notFound("App not found");
const sourceTemplateKey =
typeof application.metadata.sourceTemplateKey === "string"
? application.metadata.sourceTemplateKey
: null;
if (
application.status === "archived" ||
(application.applicationKey !== input.provider &&
sourceTemplateKey !== input.provider)
) {
throw unprocessable(
`The selected App is not the ${PROVIDER_LABELS[input.provider]} connector`,
);
}
} else {
await tx.insert(toolApplications).values({
id: applicationId,
companyId,
applicationKey: `chat:${input.provider}:${endpointId}`,
name: `${name} ${suffix}`,
description: `People talk to ${agent.name} through ${PROVIDER_LABELS[input.provider]}.`,
type: "chat",
status: "draft",
metadata: { sourceTemplateKey: input.provider, purpose: "channel" },
ownerUserId: actorUserId ?? null,
});
}
await tx.insert(toolConnections).values({
id: connectionId,
companyId,
applicationId,
name,
uid: `chat-${input.provider}-${endpointId}`,
connectionKind: "managed",
connectionPurpose: "channel",
ownership: "customer",
transport: input.provider === "speko" ? "voice" : "chat_sdk",
authKind: "api_key",
credentialPolicy: "shared",
status: "draft",
enabled: false,
config: { provider: input.provider },
transportConfig: {},
createdByUserId: actorUserId ?? null,
});
await tx.insert(chatEndpoints).values({
id: endpointId,
companyId,
connectionId,
provider: input.provider,
publicId,
assignedAgentId: agent.id,
sponsorUserId: actorUserId ?? null,
// Group chat is a broad Teams reach grant rather than a discovered
// destination toggle. Keep it closed until the operator explicitly
// enables that surface, even when this process is running against a
// database created before the column default was hardened.
requireAtMention: input.provider === "slack",
allowGroupChats: !["microsoft-teams", "speko"].includes(input.provider),
allowUnlinkedPeople: !["slack", "imessage-photon", "speko"].includes(input.provider),
capabilities: CAPABILITIES[input.provider],
setup: {
step: "provider_setup",
...(input.provider === "slack"
? { slackApp: input.slackApp ?? defaultSlackAppConfiguration(agent.name),
command: (input.slackApp ?? defaultSlackAppConfiguration(agent.name)).command, slackSetupMethod: input.slackSetupMethod ?? options.slackRegistration.managerGrants.defaultMethod() }
: {}),
},
});
});
await logActivity(db, {
companyId,
actorType: "user",
actorId: actorUserId ?? "board",
action: "chat_endpoint.created",
entityType: "tool_connection",
entityId: connectionId,
details: {
endpointId,
provider: input.provider,
assignedAgentId: agent.id,
},
});
return get(endpointId);
}
async function update(
endpointId: string,
input: UpdateChatEndpointInput,
actorUserId?: string | null,
) {
const initial = await endpointRecord(endpointId);
if (!initial) throw notFound("Chat endpoint not found");
if (initial.endpoint.provider === "agentmail" && Object.keys(input).some((key) => key !== "executionDefaults")) throw badRequest("Use the email inbox API for AgentMail");
await withCredentialMutationLease(
initial.endpoint,
async (credentialLease) => {
// Reach changes share the provider-transport lane with publications.
// Once a disable returns, no send that sampled the previous grant can
// still begin or finish behind it.
const existing = await endpointRecord(endpointId);
if (!existing) throw notFound("Chat endpoint not found");
const values: Partial<typeof chatEndpoints.$inferInsert> = {
updatedAt: new Date(),
};
if (input.communicationInstructions !== undefined) {
if (existing.endpoint.provider !== "slack") {
throw unprocessable("Communication instructions are currently supported for Slack connections");
}
values.communicationInstructions = input.communicationInstructions;
}
if (input.slackSetupMethod === "managed" && options.slackRegistration.managerGrants.defaultMethod() !== "managed") throw conflict("Managed Slack setup is unavailable");
if (input.slackSetupMethod && (existing.endpoint.provider !== "slack" || existing.endpoint.botExternalId || existing.endpoint.status !== "draft")) {
throw conflict("Setup method can only change before connecting the app");
}
if (input.slackApp || input.slackSetupMethod) {
const [registration] = await db.select().from(chatSlackRegistrations).where(eq(chatSlackRegistrations.endpointId, endpointId));
if (registration && (registration.managerGrantId || registration.status !== "failed")) {
if (registration.managerGrantId || input.slackSetupMethod === "managed" || input.slackApp || input.slackSetupMethod === "automatic") throw conflict("Slack app details are locked after creation starts");
// Manual recovery fences pending authorization under the same credential lease.
await options.slackRegistration.cleanup(endpointId, credentialLease);
}
}
if (input.slackSetupMethod) {
values.setup = { ...existing.endpoint.setup, slackSetupMethod: input.slackSetupMethod };
}
if (input.slackApp) {
if (existing.endpoint.provider !== "slack") {
throw unprocessable("Slack app details only apply to Slack connections");
}
if (existing.endpoint.status !== "draft" || existing.endpoint.botExternalId) {
throw conflict("Slack app details can only be edited before connecting the app");
}
values.setup = {
...(values.setup ?? existing.endpoint.setup),
slackApp: input.slackApp,
command: input.slackApp.command,
};
}
if (input.allowDirectMessages !== undefined)
values.allowDirectMessages = input.allowDirectMessages;
if (input.requireAtMention !== undefined) {
if (existing.endpoint.provider !== "slack")
throw unprocessable("Require at-mention only applies to Slack connections");
values.requireAtMention = input.requireAtMention;
}
if (input.allowGroupChats && existing.endpoint.provider === "imessage-photon" && (!existing.endpoint.botExternalId || existing.endpoint.botExternalId.startsWith("photon-project:")))
throw unprocessable("Photon shared channels support direct messages only; groups require a dedicated channel");
if (input.allowGroupChats !== undefined)
values.allowGroupChats = input.allowGroupChats;
if (input.allowUnlinkedPeople !== undefined)
values.allowUnlinkedPeople = input.allowUnlinkedPeople;
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
if (input.executionDefaults !== undefined) {
await assertChatExecutionDefaultsAccess(db, {
companyId: existing.endpoint.companyId,
assigneeAgentId: values.assignedAgentId ?? existing.endpoint.assignedAgentId,
actor: { type: "board", userId: actorUserId, source: actorUserId ? "session" : "local_implicit" },
defaults: input.executionDefaults,
}, tx);
values.executionDefaults = input.executionDefaults;
}
await tx
.update(chatEndpoints)
.set(values)
.where(eq(chatEndpoints.id, endpointId));
await logActivity(tx as unknown as Db, {
companyId: existing.endpoint.companyId,
actorType: "user",
actorId: actorUserId ?? "board",
action: "chat_endpoint.updated",
entityType: "tool_connection",
entityId: existing.endpoint.connectionId,
details: { endpointId, fields: Object.keys(input) },
});
await credentialLease.assertOwned(tx);
});
},
);
return get(endpointId);
}
return { list, get, create, update };
}
@@ -0,0 +1,32 @@
import type { ChatEndpointRuntime } from "../../voice/voice-runtime.js";
import type { GitHubReviewEventContext, GitHubIssueEventContext, GitHubReviewPolicy } from "@paperclipai/shared";
import {
chatDeliveries,
} from "@paperclipai/db";
import type {
Message,
} from "chat";
type GitHubMessageProvenanceOptions = {
githubManualMessages: WeakMap<object, { policy: GitHubReviewPolicy; revision: number; event: "mention" | "comment" }>;
githubAutomaticMessages: WeakMap<object, { context: GitHubReviewEventContext; revision: number; policy: GitHubReviewPolicy }>;
githubIssueMessages: WeakMap<object, { context: GitHubIssueEventContext; revision: number; policy: GitHubReviewPolicy }>;
};
export function createGitHubMessageProvenance(options: GitHubMessageProvenanceOptions) {
const { githubManualMessages, githubAutomaticMessages, githubIssueMessages } = options;
function restoreGitHubMessageProvenance(delivery: typeof chatDeliveries.$inferSelect, endpointRuntime: Pick<ChatEndpointRuntime, "provider">, message: Message): void {
const githubManual = delivery.normalizedEvent.githubManual;
if (endpointRuntime.provider === "github" && githubManual && typeof githubManual === "object") githubManualMessages.set(message, githubManual as { policy: GitHubReviewPolicy; revision: number; event: "mention" | "comment" });
const githubAutomatic = delivery.normalizedEvent.githubAutomatic;
if (endpointRuntime.provider === "github" && githubAutomatic && typeof githubAutomatic === "object") {
githubAutomaticMessages.set(message, githubAutomatic as { context: GitHubReviewEventContext; revision: number; policy: GitHubReviewPolicy });
}
const githubIssue = delivery.normalizedEvent.githubIssue;
if (endpointRuntime.provider === "github" && githubIssue && typeof githubIssue === "object") {
githubIssueMessages.set(message, githubIssue as { context: GitHubIssueEventContext; revision: number; policy: GitHubReviewPolicy });
}
}
return { restoreGitHubMessageProvenance };
}
@@ -0,0 +1,69 @@
import { createGitHubAppSetup } from "./app-setup.js";
import type { EndpointRow } from "../runtime-types.js";
import { type CredentialMutationLeaseGuard } from "../../chat-credential-mutation-lease.js";
import { githubAppJwt } from "../../chat-github-client.js";
import type { Db } from "@paperclipai/db";
import {
forbidden,
} from "../../../errors.js";
import {
logActivity,
} from "../../activity-log.js";
import {
resyncGitHubAppWebhook,
} from "../../chat-github-webhook-config.js";
type GitHubSetupOptions = {
fetchImpl: typeof globalThis.fetch;
getWebhookPublicBaseUrl: () => string | null;
githubWizard: Pick<ReturnType<typeof createGitHubAppSetup>["githubWizard"], "repairCloud">;
};
export function createGitHubSetup(db: Db, options: GitHubSetupOptions) {
const { fetchImpl, getWebhookPublicBaseUrl } = options;
async function repairGitHubCloudSetup(endpoint: EndpointRow, credentialLease: CredentialMutationLeaseGuard, actorUserId?: string | null) {
if (!actorUserId) throw forbidden("A configuring member is required to recover this App");
await credentialLease.assertOwned();
await options.githubWizard.repairCloud(endpoint.id, actorUserId);
await credentialLease.assertOwned();
}
async function repairGitHubWebhookSetup(endpoint: EndpointRow, credentials: Record<string, string>, credentialLease: CredentialMutationLeaseGuard, actorUserId?: string | null) {
// The immutable App and its installation were verified above. Repair
// only its callback using the stored secret; never reinstall it or
// change repository access. Keep historical signed-ping evidence, but
// require the fresh test round trip established above before activation.
const auditWebhookSync = async (action: string) => {
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await logActivity(tx as unknown as Db, {
companyId: endpoint.companyId,
actorType: "user",
actorId: actorUserId ?? "board",
action,
entityType: "tool_connection",
entityId: endpoint.connectionId,
details: { endpointId: endpoint.id, provider: "github" },
});
await credentialLease.assertOwned(tx);
});
};
await auditWebhookSync("chat_endpoint.webhook_sync_started");
await credentialLease.assertOwned();
await resyncGitHubAppWebhook({
fetch: fetchImpl,
appToken: githubAppJwt(credentials.appId, credentials.privateKey),
webhookUrl: `${getWebhookPublicBaseUrl()}/api/chat-webhooks/${endpoint.publicId}/github`,
webhookSecret: credentials.webhookSecret,
});
await auditWebhookSync("chat_endpoint.webhook_synced");
}
return { repairGitHubCloudSetup, repairGitHubWebhookSetup };
}
@@ -0,0 +1,29 @@
import type { Thread } from "chat";
import { chatEndpointResources } from "@paperclipai/db";
import type { EndpointRow, DbTransaction, DbOrTransaction } from "../runtime-types.js";
type DestinationAdmissionInput = {
endpoint: EndpointRow;
currentEndpoint: Pick<EndpointRow, "status" | "allowGroupChats">;
thread: Thread;
};
type DestinationAdmissionOptions = {
ensureResource: (endpoint: EndpointRow, thread: Thread, enableForSetup: boolean, database?: DbOrTransaction) => Promise<typeof chatEndpointResources.$inferSelect>;
};
export function createPhotonDestinationAdmission(options: DestinationAdmissionOptions) {
const { ensureResource } = options;
async function admitPhotonDestination(input: DestinationAdmissionInput, tx: DbTransaction) {
const { endpoint, currentEndpoint, thread } = input;
let destinationAccepting = true;
const resource = await ensureResource(endpoint, thread, false, tx);
destinationAccepting =
resource.enabled &&
resource.availability === "available" &&
currentEndpoint.allowGroupChats;
return destinationAccepting;
}
return { admitPhotonDestination };
}
@@ -1,4 +1,285 @@
import type { SafeChatPublicationPayload } from "@paperclipai/shared";
import { createChatPublicationReplacement } from "../publication-replacement.js";
import { type ChatRuntime } from "../../chat-runtime.js";
import type { ChatEndpointRuntime } from "../../voice/voice-runtime.js";
import { nativePhotonInteraction, publishPhotonPrompt, type PhotonPromptReceipt, type PhotonInteractionBinding } from "../../photon/interactions.js";
import { PhotonError } from "../../photon/cloud.js";
import { PhotonChatAdapter } from "../../photon/adapter.js";
import {
and,
desc,
eq,
or,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatActions,
chatEndpointResources,
chatMessageLinks,
heartbeatRuns,
issueThreadInteractions,
} from "@paperclipai/db";
import type {
ChatEndpointSetupState,
SafeChatPublicationPayload,
} from "@paperclipai/shared";
import {
type ChatSdkRuntime,
} from "../../chat-sdk-runtime.js";
import { safeChatTaskUrl } from "../../chat-task-url.js";
import { issueThreadInteractionService } from "../../issue-thread-interactions.js";
import type {
FileUpload,
} from "chat";
import type { EndpointRecord } from "../runtime-types.js";
import type { PublicationTransportInput } from "../publications/types.js";
import type { Thread } from "chat";
type PhotonPublicationDeliveryOptions = {
endpointRecord: (endpointId: string) => Promise<EndpointRecord | null>;
runtimeGeneration: (setup: ChatEndpointSetupState) => number;
runtime: Pick<ChatRuntime | ChatSdkRuntime, "get">;
receiptReactionCompletionRunId: ReturnType<typeof createChatPublicationReplacement>["receiptReactionCompletionRunId"];
getTaskBaseUrl: () => string | null | undefined;
};
export function createPhotonPublicationDelivery(db: Db, options: PhotonPublicationDeliveryOptions) {
const { endpointRecord, runtimeGeneration, runtime, receiptReactionCompletionRunId, getTaskBaseUrl } = options;
async function deliverPhotonPublication(input: PublicationTransportInput, endpointRuntime: ChatEndpointRuntime, thread: Pick<Thread, "id">, text: string, files: FileUpload[]) {
const adapter = endpointRuntime.getProviderAdapter();
if (!(adapter instanceof PhotonChatAdapter))
throw new Error("Photon publication runtime unavailable");
const assertCurrent = async () => {
await input.beforePhotonWrite?.();
const current = await endpointRecord(input.endpoint.id);
if (
!current ||
!["verifying", "active"].includes(current.endpoint.status) ||
runtimeGeneration(current.endpoint.setup) !==
runtimeGeneration(input.endpoint.setup) ||
runtime.get(input.endpoint.id) !== endpointRuntime
)
throw new PhotonError(
"rejected",
"Photon publication authority changed",
);
const resource = await db
.select()
.from(chatEndpointResources)
.where(
and(
eq(chatEndpointResources.companyId, current.endpoint.companyId),
eq(chatEndpointResources.endpointId, current.endpoint.id),
eq(chatEndpointResources.id, input.conversation.resourceId!),
),
)
.then((rows) => rows[0]);
if (
resource?.availability !== "available" ||
(!input.conversation.isDirectMessage && !resource.enabled) ||
(input.conversation.isDirectMessage &&
!current.endpoint.allowDirectMessages)
)
throw new PhotonError(
"rejected",
"Photon conversation is no longer enabled",
);
};
const retry = await db
.select()
.from(chatActions)
.where(
and(
eq(chatActions.endpointId, input.endpoint.id),
eq(chatActions.kind, "photon_publication_retry"),
eq(
chatActions.providerActionId,
`photon-retry:${input.publication.id}:${input.publication.attempts + 1}`,
),
eq(chatActions.status, "processed"),
),
)
.then((rows) => rows[0]);
const retryUnknown = Boolean(retry);
const nextQuestion =
/^photon-question:([a-zA-Z0-9_-]{8,24}):(\d{1,2})$/.exec(
input.publication.idempotencyKey,
);
if (
input.payload.interactionId &&
(input.publication.idempotencyKey ===
`interaction:${input.payload.interactionId}:${input.endpoint.id}` ||
nextQuestion)
) {
const action = await db
.select()
.from(chatActions)
.where(
and(
eq(chatActions.companyId, input.endpoint.companyId),
eq(chatActions.endpointId, input.endpoint.id),
eq(chatActions.kind, "photon_interaction"),
nextQuestion
? eq(chatActions.providerActionId, `photon:${nextQuestion[1]}`)
: eq(
sql<string>`${chatActions.payload}->>'publicationId'`,
input.publication.id,
),
eq(chatActions.status, "issued"),
),
)
.then((rows) => rows[0]);
const interaction = (
await issueThreadInteractionService(db).listForIssue(
input.publication.issueId,
)
).find((candidate) => candidate.id === input.payload.interactionId);
if (
action &&
interaction?.status === "pending" &&
nativePhotonInteraction(interaction)
) {
const promptGuard = async () => {
await assertCurrent();
const current = await db
.select({ status: issueThreadInteractions.status })
.from(issueThreadInteractions)
.where(
and(
eq(
issueThreadInteractions.companyId,
input.endpoint.companyId,
),
eq(issueThreadInteractions.id, interaction.id),
),
)
.then((rows) => rows[0]);
if (current?.status !== "pending")
throw new PhotonError(
"rejected",
"This interaction has already been resolved",
);
};
const receipt = await publishPhotonPrompt({
adapter,
threadId: thread.id,
binding: action.payload as unknown as PhotonInteractionBinding,
interaction,
questionIndex: nextQuestion ? Number(nextQuestion[2]) : 0,
taskUrl: safeChatTaskUrl(
getTaskBaseUrl(),
input.publication.issueId,
),
assertCurrent: promptGuard,
retryUnknown,
});
return { id: receipt.promptMessageGuid };
}
if (nextQuestion)
throw new PhotonError(
"rejected",
"This question is no longer available",
);
}
const reply = await adapter.state.read<{ guid: string | null }>(
`publication-reply:${input.publication.id}`,
);
let replyTo = reply?.guid ?? null;
if (!reply) {
const sourceRunId = await receiptReactionCompletionRunId(
db,
input.publication,
input.payload,
);
if (sourceRunId) {
const source = await db
.select({ guid: chatMessageLinks.providerMessageId })
.from(heartbeatRuns)
.innerJoin(
chatMessageLinks,
and(
eq(chatMessageLinks.companyId, input.endpoint.companyId),
eq(chatMessageLinks.endpointId, input.endpoint.id),
eq(chatMessageLinks.conversationId, input.conversation.id),
eq(chatMessageLinks.direction, "inbound"),
or(
sql`${chatMessageLinks.commentId}::text = ${heartbeatRuns.contextSnapshot}->>'wakeCommentId'`,
sql`coalesce(${heartbeatRuns.contextSnapshot}->'wakeCommentIds', '[]'::jsonb) ? ${chatMessageLinks.commentId}::text`,
),
),
)
.where(
and(
eq(heartbeatRuns.id, sourceRunId),
eq(heartbeatRuns.companyId, input.endpoint.companyId),
),
)
.orderBy(desc(chatMessageLinks.createdAt))
.limit(1)
.then((rows) => rows[0]);
replyTo = source?.guid ?? null;
}
replyTo = (
await adapter.state.update<{ guid: string | null }>(
`publication-reply:${input.publication.id}`,
(current) => current ?? { guid: replyTo },
)
).guid;
}
const sent = await adapter.publish(
thread.id,
input.publication.id,
{ markdown: text, files },
{ assertCurrent, retryUnknown, replyTo: replyTo ?? undefined },
);
if (input.publication.idempotencyKey.startsWith("photon-response:")) {
const notice = await db
.select()
.from(chatActions)
.where(
and(
eq(chatActions.companyId, input.endpoint.companyId),
eq(chatActions.endpointId, input.endpoint.id),
eq(chatActions.kind, "photon_response_notice"),
eq(
chatActions.providerActionId,
`photon-notice:${input.publication.idempotencyKey}`,
),
),
)
.then((rows) => rows[0]);
if (
notice &&
typeof notice.payload.reference === "string" &&
typeof notice.payload.questionIndex === "number" &&
typeof notice.payload.publicationId === "string"
) {
const receipt: PhotonPromptReceipt = {
schema: 1,
reference: notice.payload.reference,
questionIndex: notice.payload.questionIndex,
publicationId: notice.payload.publicationId,
promptMessageGuid: sent.id,
promptMessageGuids: sent.messageIds,
options: {},
};
for (const guid of sent.messageIds)
await adapter.state.update<PhotonPromptReceipt>(
`prompt-message:${guid}`,
(current) => current ?? receipt,
);
}
}
return sent;
}
return { deliverPhotonPublication };
}
export function imessageProgressSupersessionReason(
progress: SafeChatPublicationPayload["progressState"],
@@ -6,4 +287,4 @@ export function imessageProgressSupersessionReason(
return progress && ["queued", "working"].includes(progress)
? "iMessage uses typing instead of progress bubbles"
: null;
}
}
@@ -0,0 +1,88 @@
import type { createPhotonCredentialChecks } from "./credentials.js";
import type { EndpointRow } from "../runtime-types.js";
import { type CredentialMutationLeaseGuard } from "../../chat-credential-mutation-lease.js";
import { photonSharedScope, photonSharedIdentity } from "../../photon/cloud.js";
import { photonChannelConfigurationSchema } from "@paperclipai/shared";
import {
and,
eq,
ne,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatEndpoints,
toolConnections,
} from "@paperclipai/db";
import type {
ConfigureChatEndpointInput,
} from "@paperclipai/shared";
import {
conflict,
notFound,
} from "../../../errors.js";
import type { EndpointRecord } from "../runtime-types.js";
type PhotonSetupInspectionOptions = {
endpointRecord: (endpointId: string) => Promise<EndpointRecord | null>;
inspectPhotonCredentials: ReturnType<typeof createPhotonCredentialChecks>["inspectPhotonCredentials"];
};
export function createPhotonSetupInspection(db: Db, options: PhotonSetupInspectionOptions) {
const { endpointRecord, inspectPhotonCredentials } = options;
async function inspectPhoton(
endpointId: string,
input: { projectId: string; projectSecret: string },
) {
const record = await endpointRecord(endpointId);
if (!record || record.endpoint.provider !== "imessage-photon")
throw notFound("iMessage Photon endpoint not found");
const inspection = structuredClone(await inspectPhotonCredentials(
input.projectId,
input.projectSecret,
));
const reserved = await db
.select({ number: chatEndpoints.botExternalId })
.from(chatEndpoints)
.where(
and(
eq(chatEndpoints.provider, "imessage-photon"),
ne(chatEndpoints.id, endpointId),
ne(chatEndpoints.status, "archived"),
),
);
for (const line of inspection.lines) {
if (reserved.some((row) => row.number === line.phoneNumber)) {
line.eligible = false;
line.unavailableReason =
"This number already belongs to another channel";
}
}
inspection.eligible = inspection.allocation === "shared" ? inspection.eligible && !reserved.some((row) => row.number === photonSharedIdentity(inspection.projectId)) : inspection.lines.some((line) => line.eligible);
return inspection;
}
return { inspectPhoton };
}
export function preparePhotonConfiguration(endpoint: EndpointRow, input: ConfigureChatEndpointInput, credentials: Record<string, string>): Record<string, string> {
const configuration = photonChannelConfigurationSchema.parse(input.photon ?? (credentials.allocation === "shared" ? { allocation: "shared", projectId: credentials.projectId } : { projectId: credentials.projectId, lineId: credentials.lineId }));
const lineId = configuration.allocation === "shared" ? photonSharedScope(configuration.projectId) : configuration.lineId;
if (endpoint.botExternalId && (configuration.projectId !== endpoint.providerAccountId || credentials.lineId && lineId !== credentials.lineId)) throw conflict("A different Photon identity requires a new channel");
return { ...credentials, ...configuration, lineId };
}
export function createPhotonConfiguration(db: Db, options: { invalidateRuntime: (endpointId: string) => Promise<boolean> }) {
const { invalidateRuntime } = options;
async function persistPhotonConfiguration(endpoint: EndpointRow, credentials: Record<string, string>, credentialLease: CredentialMutationLeaseGuard) {
await invalidateRuntime(endpoint.id);
await credentialLease.assertOwned();
if (credentials.allocation === "shared") await db.update(chatEndpoints).set({ allowGroupChats: false }).where(eq(chatEndpoints.id, endpoint.id));
await db.update(toolConnections).set({ config: { provider: endpoint.provider, photon: credentials.allocation === "shared" ? { allocation: "shared", projectId: credentials.projectId } : { allocation: "dedicated", projectId: credentials.projectId, lineId: credentials.lineId } } }).where(and(eq(toolConnections.companyId, endpoint.companyId), eq(toolConnections.id, endpoint.connectionId)));
}
return { persistPhotonConfiguration };
}
@@ -0,0 +1,399 @@
import { createTeamsDestinationAdmission } from "../microsoft-teams/destination-admission.js";
import { createTelegramDestinationAdmission } from "../telegram/destination-admission.js";
import { createPhotonDestinationAdmission } from "../imessage-photon/destination-admission.js";
import { deliveryContentWasRedacted } from "./delivery-ledger.js";
import type { EndpointRow, DbTransaction, DbOrTransaction, InboundRuntimeContext } from "../runtime-types.js";
import { notifyChatDeliveryWork } from "../../chat-work-notifications.js";
import {
githubMessageAddressesAnotherBot,
} from "../../chat-github-events.js";
import {
createHash,
} from "node:crypto";
import {
and,
eq,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatDeliveries,
chatEndpointResources,
chatEndpoints,
toolConnections,
} from "@paperclipai/db";
import type {
ChatEndpointSetupState,
ToolCredentialSecretRef,
} from "@paperclipai/shared";
import type {
Message,
Thread,
} from "chat";
import type { DiscordGatewayOwnership } from "../runtime-types.js";
import type { InboundEnvelope } from "./envelope.js";
import type { ChatSdkMessageCallbackEvent } from "../../chat-sdk-runtime.js";
type InboundAdmissionInput = {
endpoint: EndpointRow;
thread: Thread;
message: Message;
trigger: ChatSdkMessageCallbackEvent["trigger"];
runtimeContext?: InboundRuntimeContext;
admittedDeliveryId: string | null;
deferDrainUntilFollowup: boolean;
scheduledAt: number | null;
envelope: Pick<InboundEnvelope, "providerEventId" | "addressed" | "eventKind" | "normalized" | "redactedDestinationNormalized" | "githubAutomatic" | "githubIssue">;
};
type GatewayAdmissionState = {
lost?: DiscordGatewayOwnership;
renewed?: { expiresAt: Date; ownership: DiscordGatewayOwnership };
};
type InboundAdmissionOptions = {
runtimeGeneration: (setup: ChatEndpointSetupState) => number;
credentialFingerprint: (refs: ToolCredentialSecretRef[]) => string;
leasedChatProvider: (provider: string) => boolean;
renewDiscordGatewayOwnershipForMessageAdmission: (tx: DbTransaction, endpointId: string, context: InboundRuntimeContext) => Promise<{ kind: "owned"; expiresAt: Date; ownership: DiscordGatewayOwnership } | { kind: "lost"; ownership: DiscordGatewayOwnership | null }>;
ensureResource: (endpoint: EndpointRow, thread: Thread, enableForSetup: boolean, database?: DbOrTransaction) => Promise<typeof chatEndpointResources.$inferSelect>;
DISCORD_ROOT_THREAD_CONFIRMATION_DELAY_MS: number;
};
export function createInboundAdmission(db: Db, options: InboundAdmissionOptions) {
const { runtimeGeneration, credentialFingerprint, leasedChatProvider, renewDiscordGatewayOwnershipForMessageAdmission, ensureResource, DISCORD_ROOT_THREAD_CONFIRMATION_DELAY_MS } = options;
const { admitTeamsDestination } = createTeamsDestinationAdmission({ ensureResource });
const { admitTelegramDestination } = createTelegramDestinationAdmission({ ensureResource });
const { admitPhotonDestination } = createPhotonDestinationAdmission({ ensureResource });
async function admitInboundMessage(input: InboundAdmissionInput, discordGatewayAdmissionState: GatewayAdmissionState) {
const { endpoint, thread, message, trigger, runtimeContext, admittedDeliveryId, deferDrainUntilFollowup, scheduledAt, envelope } = input;
const { providerEventId, addressed, eventKind, normalized, redactedDestinationNormalized, githubAutomatic, githubIssue } = envelope;
return await db.transaction(async (tx) => {
const currentEndpoint = await tx
.select({
status: chatEndpoints.status,
setup: chatEndpoints.setup,
allowDirectMessages: chatEndpoints.allowDirectMessages,
requireAtMention: chatEndpoints.requireAtMention,
allowGroupChats: chatEndpoints.allowGroupChats,
botUsername: chatEndpoints.botUsername,
})
.from(chatEndpoints)
.where(eq(chatEndpoints.id, endpoint.id))
.for("update")
.then((rows) => rows[0] ?? null);
if (!currentEndpoint) return null;
// An SDK callback can finish parsing after a pause/resume cycle. Each
// runtime captures the durable endpoint generation at registration;
// resume advances it under the same row lock used by admission. This
// rejects stale callbacks without relying on provider/server clock sync.
const currentRuntimeGeneration = runtimeGeneration(currentEndpoint.setup);
const currentCredentialFingerprint = runtimeContext
? await tx
.select({ refs: toolConnections.credentialSecretRefs })
.from(toolConnections)
.where(
and(
eq(toolConnections.companyId, endpoint.companyId),
eq(toolConnections.id, endpoint.connectionId),
),
)
.then((rows) => credentialFingerprint(rows[0]?.refs ?? []))
: null;
const staleActivation =
runtimeContext !== undefined &&
(runtimeContext.generation !== currentRuntimeGeneration ||
runtimeContext.credentialFingerprint !==
currentCredentialFingerprint);
const endpointAccepting =
["verifying", "active"].includes(currentEndpoint.status) &&
!staleActivation;
if (
endpointAccepting &&
leasedChatProvider(endpoint.provider) &&
runtimeContext &&
!admittedDeliveryId
) {
// The first process-local ownership check can precede a host pause.
// Reclaim the same durable token while the endpoint admission row is
// locked so a standby takeover cannot be followed by an obsolete
// callback write. This renews only the exact token that authenticated
// the callback; generation and credential checks above remain intact.
const gatewayOwnership =
await renewDiscordGatewayOwnershipForMessageAdmission(
tx,
endpoint.id,
runtimeContext,
);
if (gatewayOwnership.kind === "lost") {
if (gatewayOwnership.ownership) {
discordGatewayAdmissionState.lost = gatewayOwnership.ownership;
}
return null;
}
discordGatewayAdmissionState.renewed = gatewayOwnership;
}
let provisionalTeamsSetupReply = false;
let destinationAccepting = true;
if (endpointAccepting && thread.isDM) {
destinationAccepting = currentEndpoint.allowDirectMessages;
}
if (
endpointAccepting &&
endpoint.provider === "microsoft-teams" &&
!thread.isDM
) {
({ destinationAccepting, provisionalTeamsSetupReply } = await admitTeamsDestination({ endpoint, currentEndpoint, thread, message, addressed }, tx));
}
if (
endpointAccepting &&
endpoint.provider === "telegram" &&
!thread.isDM
) {
destinationAccepting = await admitTelegramDestination({ endpoint, currentEndpoint, thread, addressed }, tx);
}
if (
endpointAccepting &&
endpoint.provider === "imessage-photon" &&
!thread.isDM
) {
destinationAccepting = await admitPhotonDestination({ endpoint, currentEndpoint, thread }, tx);
}
const mentionRequired =
endpoint.provider === "slack" &&
currentEndpoint.requireAtMention &&
trigger !== "mention" &&
message.isMention !== true;
// GitHub sends the same comment to every installed App. Subscribed
// conversations are a fallback for unaddressed replies, not permission
// to wake every bot when the person names a different connected App.
// Recheck inside durable admission so recovered deliveries follow the
// same company-scoped routing decision before any task or reaction.
const githubAddressedElsewhere =
endpointAccepting &&
endpoint.provider === "github" &&
!githubAutomatic &&
!githubIssue &&
(await githubMessageAddressesAnotherBot(tx, {
id: endpoint.id,
companyId: endpoint.companyId,
botUsername: currentEndpoint.botUsername,
}, message.text));
const accepting = endpointAccepting && destinationAccepting && !githubAddressedElsewhere && !mentionRequired;
const redactDestinationDelivery =
githubAddressedElsewhere || mentionRequired || (!accepting && thread.isDM) ||
(!thread.isDM &&
(endpoint.provider === "microsoft-teams" ||
endpoint.provider === "telegram" ||
endpoint.provider === "imessage-photon") &&
(!accepting || provisionalTeamsSetupReply));
const ignoredAt = accepting ? null : new Date();
const inactiveReason = !endpointAccepting
? staleActivation
? "Connection activation changed before admission"
: "Connection is not active"
: githubAddressedElsewhere
? "GitHub message explicitly mentions a different connected bot"
: mentionRequired
? "Message did not @mention the bot"
: endpoint.provider === "telegram" && !thread.isDM && !addressed
? "Message did not address the agent"
: "Destination is not enabled in Paperclip";
let candidate = admittedDeliveryId
? await tx
.select()
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.id, admittedDeliveryId),
eq(chatDeliveries.endpointId, endpoint.id),
eq(chatDeliveries.providerEventId, providerEventId),
),
)
.for("update")
.then((rows) => rows[0] ?? null)
: null;
if (!admittedDeliveryId) {
await notifyChatDeliveryWork(tx);
const [delivery] = await tx
.insert(chatDeliveries)
.values({
companyId: endpoint.companyId,
endpointId: endpoint.id,
providerEventId,
deduplicationKey: createHash("sha256")
.update(providerEventId)
.digest("hex"),
eventKind,
normalizedEvent: redactDestinationDelivery
? redactedDestinationNormalized
: deferDrainUntilFollowup && accepting
? {
...normalized,
providerThreadPending: true,
}
: normalized,
state: accepting
? deferDrainUntilFollowup
? "retry"
: "received"
: "filtered",
nextAttemptAt:
accepting && deferDrainUntilFollowup
? new Date(
Date.now() + DISCORD_ROOT_THREAD_CONFIRMATION_DELAY_MS,
)
: accepting && scheduledAt
? new Date(scheduledAt)
: null,
redactedError:
accepting && deferDrainUntilFollowup
? "Waiting for Discord thread creation confirmation"
: accepting
? null
: inactiveReason,
processedAt: ignoredAt,
})
.onConflictDoNothing()
.returning();
candidate = delivery ?? null;
}
if (!candidate && !admittedDeliveryId) {
const existingDelivery = await tx
.select()
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.endpointId, endpoint.id),
eq(chatDeliveries.providerEventId, providerEventId),
),
)
.then((rows) => rows[0] ?? null);
if (existingDelivery) {
const hydrateTeamsDelivery =
endpoint.provider === "microsoft-teams" &&
!thread.isDM &&
accepting &&
!provisionalTeamsSetupReply &&
["received", "retry", "processing"].includes(
existingDelivery.state,
) &&
deliveryContentWasRedacted(existingDelivery.normalizedEvent);
await notifyChatDeliveryWork(tx);
[candidate] = await tx
.update(chatDeliveries)
.set({
// Increment under PostgreSQL's row lock so simultaneous handler
// fanout and provider retries cannot lose duplicate telemetry.
normalizedEvent: redactDestinationDelivery
? sql`${JSON.stringify(redactedDestinationNormalized)}::jsonb
|| jsonb_build_object(
'deduplication',
jsonb_build_object(
'duplicateCount',
coalesce((${chatDeliveries.normalizedEvent}#>>'{deduplication,duplicateCount}')::integer, 0) + 1,
'lastDuplicateAt',
${new Date().toISOString()}::text
)
)`
: hydrateTeamsDelivery
? sql`${JSON.stringify(normalized)}::jsonb
|| jsonb_build_object(
'deduplication',
coalesce(${chatDeliveries.normalizedEvent}->'deduplication', '{}'::jsonb)
|| jsonb_build_object(
'duplicateCount',
coalesce((${chatDeliveries.normalizedEvent}#>>'{deduplication,duplicateCount}')::integer, 0) + 1,
'lastDuplicateAt',
${new Date().toISOString()}::text
)
)`
: sql`coalesce(${chatDeliveries.normalizedEvent}, '{}'::jsonb)
|| jsonb_build_object(
'deduplication',
coalesce(${chatDeliveries.normalizedEvent}->'deduplication', '{}'::jsonb)
|| jsonb_build_object(
'duplicateCount',
coalesce((${chatDeliveries.normalizedEvent}#>>'{deduplication,duplicateCount}')::integer, 0) + 1,
'lastDuplicateAt',
${new Date().toISOString()}::text
)
)`,
...(!accepting &&
["received", "retry"].includes(existingDelivery.state)
? {
state: "filtered" as const,
nextAttemptAt: null,
redactedError: inactiveReason,
processedAt: ignoredAt,
}
: {}),
updatedAt: new Date(),
})
.where(eq(chatDeliveries.id, existingDelivery.id))
.returning();
}
}
if (
candidate &&
admittedDeliveryId &&
accepting &&
!provisionalTeamsSetupReply &&
endpoint.provider === "microsoft-teams" &&
!thread.isDM &&
["received", "retry", "processing"].includes(candidate.state) &&
deliveryContentWasRedacted(candidate.normalizedEvent)
) {
[candidate] = await tx
.update(chatDeliveries)
.set({ normalizedEvent: normalized, updatedAt: new Date() })
.where(eq(chatDeliveries.id, candidate.id))
.returning();
}
if (
candidate &&
admittedDeliveryId &&
!accepting &&
["received", "retry", "processing"].includes(candidate.state)
) {
await notifyChatDeliveryWork(tx);
[candidate] = await tx
.update(chatDeliveries)
.set({
state: "filtered",
...(redactDestinationDelivery
? {
normalizedEvent: redactedDestinationNormalized,
principalId: null,
}
: {}),
nextAttemptAt: null,
redactedError: inactiveReason,
processedAt: ignoredAt,
updatedAt: new Date(),
})
.where(eq(chatDeliveries.id, candidate.id))
.returning();
}
if (!accepting && ignoredAt) {
await tx
.update(chatEndpoints)
.set({ lastEventAt: ignoredAt, updatedAt: ignoredAt })
.where(eq(chatEndpoints.id, endpoint.id));
}
return {
accepting,
candidate: candidate ?? null,
provisionalTeamsSetupReply,
};
});
}
return { admitInboundMessage };
}
@@ -0,0 +1,429 @@
import type { EndpointRow, DbTransaction, DbOrTransaction, LifecycleRuntimeFence } from "../runtime-types.js";
import {
stableExternalPrincipalId,
} from "../message-lifecycle.js";
import { voiceSessionStore } from "../../voice/voice-session-store.js";
import { githubChatPrincipalAccess } from "../../chat-github-access.js";
import {
and,
eq,
inArray,
ne,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatConversations,
chatEndpointResources,
chatExternalPrincipals,
chatIdentityLinks,
companyMemberships,
} from "@paperclipai/db";
import {
HttpError,
forbidden,
} from "../../../errors.js";
import type {
Author,
Thread,
} from "chat";
import { canonicalProviderResourceId, chatSurfaceKind, providerResourceLabelFromThread, providerResourceType, nonDirectDestinationAllowed } from "./surfaces.js";
type ResourceRow = typeof chatEndpointResources.$inferSelect;
type InboundAuthorityOptions = {
voiceStore: Pick<ReturnType<typeof voiceSessionStore>, "authorizePrincipal">;
runtimeCallbackEndpoint: (tx: DbTransaction, endpointId: string, context: LifecycleRuntimeFence, statuses: EndpointRow["status"][]) => Promise<EndpointRow | null>;
};
export function createChatInboundAuthority(db: Db, options: InboundAuthorityOptions) {
const { voiceStore, runtimeCallbackEndpoint } = options;
async function findAuthorizedIdentityLink(endpoint: EndpointRow, userId: string) {
return db.select({ id: chatIdentityLinks.id }).from(chatIdentityLinks)
.innerJoin(companyMemberships, and(
eq(companyMemberships.companyId, chatIdentityLinks.companyId), eq(companyMemberships.principalType, "user"),
eq(companyMemberships.principalId, userId), eq(companyMemberships.status, "active"), ne(companyMemberships.membershipRole, "viewer"),
))
.where(and(eq(chatIdentityLinks.companyId, endpoint.companyId), eq(chatIdentityLinks.endpointId, endpoint.id),
eq(chatIdentityLinks.paperclipUserId, userId), eq(chatIdentityLinks.status, "linked")))
.limit(1).then((rows) => rows[0] ?? null);
}
async function ensurePrincipal(
endpoint: EndpointRow,
author: Author,
raw?: unknown,
) {
const providerAccountId = endpoint.providerAccountId ?? "unknown";
const externalId = stableExternalPrincipalId(
endpoint.provider,
author,
raw,
);
const [principal] = await db
.insert(chatExternalPrincipals)
.values({
companyId: endpoint.companyId,
provider: endpoint.provider,
providerAccountId,
externalId,
kind:
author.isBot === true ? "bot" : author.isSystem ? "system" : "user",
displayName: author.fullName,
handle: author.userName,
isBot: author.isBot === true || author.isMe,
lastSeenAt: new Date(),
})
.onConflictDoUpdate({
target: [
chatExternalPrincipals.companyId,
chatExternalPrincipals.provider,
chatExternalPrincipals.providerAccountId,
chatExternalPrincipals.externalId,
],
set: {
displayName: author.fullName,
handle: author.userName,
isBot: author.isBot === true || author.isMe,
lastSeenAt: new Date(),
updatedAt: new Date(),
},
})
.returning();
if (endpoint.provider === "speko") {
try {
const authorization = await db.transaction((tx) => voiceStore.authorizePrincipal(tx, endpoint.companyId, endpoint.id, externalId));
return { principal, userId: authorization.userId, linkedDenied: false };
} catch (error) {
if (!(error instanceof HttpError && [403, 404].includes(error.status))) throw error;
return { principal, userId: null, linkedDenied: true };
}
}
const link = await db
.select({
userId: chatIdentityLinks.paperclipUserId,
status: chatIdentityLinks.status,
})
.from(chatIdentityLinks)
.where(
and(
eq(chatIdentityLinks.endpointId, endpoint.id),
eq(chatIdentityLinks.principalId, principal.id),
),
)
.then((rows) => rows[0] ?? null);
if (link?.status === "linked" && link.userId) {
const membership = await db
.select({
status: companyMemberships.status,
membershipRole: companyMemberships.membershipRole,
})
.from(companyMemberships)
.where(
and(
eq(companyMemberships.companyId, endpoint.companyId),
eq(companyMemberships.principalType, "user"),
eq(companyMemberships.principalId, link.userId),
),
)
.then((rows) => rows[0] ?? null);
// External chat performs the same task mutations as the board. A linked
// viewer therefore remains read-only instead of gaining write authority
// merely by using a provider account.
if (
membership?.status !== "active" ||
membership.membershipRole === "viewer"
) {
return { principal, userId: null, linkedDenied: true };
}
return { principal, userId: link.userId, linkedDenied: false };
}
return { principal, userId: null, linkedDenied: false };
}
async function sponsorAllowsGuest(endpoint: EndpointRow): Promise<boolean> {
// A null sponsor is the local board/operator context used by self-hosted
// instances. Cloud endpoints retain the creating user and must revalidate
// that user's current company authority on every sponsored guest turn.
if (!endpoint.sponsorUserId) return true;
const membership = await db
.select({
status: companyMemberships.status,
membershipRole: companyMemberships.membershipRole,
})
.from(companyMemberships)
.where(
and(
eq(companyMemberships.companyId, endpoint.companyId),
eq(companyMemberships.principalType, "user"),
eq(companyMemberships.principalId, endpoint.sponsorUserId),
),
)
.then((rows) => rows[0] ?? null);
return (
membership?.status === "active" && membership.membershipRole !== "viewer"
);
}
async function lockCurrentPrincipalAuthorization(
tx: DbOrTransaction,
endpoint: EndpointRow,
principalId: string,
): Promise<{
allowed: boolean;
linkedDenied: boolean;
userId: string | null;
sponsorUserId?: string | null;
}> {
if (endpoint.provider === "speko") {
const principal = await tx.select({ externalId: chatExternalPrincipals.externalId }).from(chatExternalPrincipals)
.where(and(eq(chatExternalPrincipals.id, principalId), eq(chatExternalPrincipals.companyId, endpoint.companyId), eq(chatExternalPrincipals.provider, "speko"))).then((rows) => rows[0]);
if (!principal) return { allowed: false, linkedDenied: true, userId: null };
try { return await voiceStore.authorizePrincipal(tx as import("../../voice/voice-session-store.js").VoiceTransaction, endpoint.companyId, endpoint.id, principal.externalId); }
catch (error) {
if (!(error instanceof HttpError && [403, 404].includes(error.status))) throw error;
return { allowed: false, linkedDenied: true, userId: null };
}
}
const githubAccess = await githubChatPrincipalAccess(tx, endpoint, principalId);
if (githubAccess) return githubAccess;
// Link confirmation already uses this transaction-scoped identity key.
// Taking it at the final task mutation boundary prevents a newly confirmed
// identity from racing the authorization snapshot. Row locks below also
// serialize revocation and membership changes that do not use this key.
await tx.execute(
sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${endpoint.companyId}:${principalId}`}, 0))`,
);
const link = await tx
.select({
status: chatIdentityLinks.status,
userId: chatIdentityLinks.paperclipUserId,
})
.from(chatIdentityLinks)
.where(
and(
eq(chatIdentityLinks.companyId, endpoint.companyId),
eq(chatIdentityLinks.endpointId, endpoint.id),
eq(chatIdentityLinks.principalId, principalId),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
if (link?.status === "linked") {
if (!link.userId) {
return { allowed: false, linkedDenied: true, userId: null };
}
const membership = await tx
.select({
status: companyMemberships.status,
membershipRole: companyMemberships.membershipRole,
})
.from(companyMemberships)
.where(
and(
eq(companyMemberships.companyId, endpoint.companyId),
eq(companyMemberships.principalType, "user"),
eq(companyMemberships.principalId, link.userId),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
const allowed =
membership?.status === "active" &&
membership.membershipRole !== "viewer";
return {
allowed,
linkedDenied: !allowed,
userId: allowed ? link.userId : null,
};
}
if (!endpoint.allowUnlinkedPeople) {
return { allowed: false, linkedDenied: false, userId: null };
}
if (!endpoint.sponsorUserId) {
return { allowed: true, linkedDenied: false, userId: null };
}
const sponsorMembership = await tx
.select({
status: companyMemberships.status,
membershipRole: companyMemberships.membershipRole,
})
.from(companyMemberships)
.where(
and(
eq(companyMemberships.companyId, endpoint.companyId),
eq(companyMemberships.principalType, "user"),
eq(companyMemberships.principalId, endpoint.sponsorUserId),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
return {
allowed:
sponsorMembership?.status === "active" &&
sponsorMembership.membershipRole !== "viewer",
linkedDenied: false,
userId: null,
};
}
async function requireCurrentExternalActionAuthorization(
tx: DbTransaction,
input: {
conversationId: string;
endpointId: string;
expectedUserId: string;
principalId: string;
runtimeContext: LifecycleRuntimeFence;
},
): Promise<void> {
const endpoint = await runtimeCallbackEndpoint(
tx,
input.endpointId,
input.runtimeContext,
["active", "verifying"],
);
// Photon setup already admits linked senders and publishes agent prompts.
// Let those prompts resolve so a clarifying question cannot deadlock the
// actual-reply qualification. Other providers retain their active-only gate.
if (!endpoint || (endpoint.status !== "active" &&
(endpoint.provider !== "imessage-photon" || endpoint.setup.step !== "test"))) {
throw forbidden("This chat action is no longer authorized", {
code: "chat_action_authorization_changed",
});
}
const conversation = await tx
.select()
.from(chatConversations)
.where(
and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
eq(chatConversations.id, input.conversationId),
inArray(chatConversations.state, ["active", "waiting"]),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
if (!conversation) {
throw forbidden("This chat action is no longer authorized", {
code: "chat_action_authorization_changed",
});
}
const resource = conversation.resourceId
? await tx
.select()
.from(chatEndpointResources)
.where(
and(
eq(chatEndpointResources.companyId, endpoint.companyId),
eq(chatEndpointResources.endpointId, endpoint.id),
eq(chatEndpointResources.id, conversation.resourceId),
),
)
.for("update")
.then((rows) => rows[0] ?? null)
: null;
const destinationAllowed = conversation.isDirectMessage
? endpoint.allowDirectMessages
: nonDirectDestinationAllowed(endpoint, resource);
const authorization = await lockCurrentPrincipalAuthorization(
tx,
endpoint,
input.principalId,
);
// Governed actions never fall back to sponsored-guest authority. If the
// provider identity was relinked while this callback waited, reject this
// stale actor snapshot and let a fresh callback use the new identity.
if (
!destinationAllowed ||
!authorization.allowed ||
authorization.userId !== input.expectedUserId
) {
throw forbidden("This chat action is no longer authorized", {
code: "chat_action_authorization_changed",
});
}
}
function isExternalActionAuthorizationChange(error: unknown): boolean {
if (!error || typeof error !== "object") return false;
const value = error as { details?: unknown };
return (
Boolean(value.details) &&
typeof value.details === "object" &&
(value.details as { code?: unknown }).code ===
"chat_action_authorization_changed"
);
}
async function ensureResource(
endpoint: EndpointRow,
thread: Thread,
enabledBySetupActivation: boolean,
database: DbOrTransaction = db,
) {
const type = providerResourceType(
endpoint.provider,
chatSurfaceKind(endpoint.provider, thread),
);
const providerResourceId = canonicalProviderResourceId(
endpoint.provider,
thread,
);
const resourceLabel = providerResourceLabelFromThread(
endpoint.provider,
thread,
);
const [resource] = await database
.insert(chatEndpointResources)
.values({
companyId: endpoint.companyId,
endpointId: endpoint.id,
type,
providerResourceId,
label: resourceLabel.label,
availability: "available",
// Slack can deliver the invitation's app_mention before the membership
// callback. Give either arrival order the same new-channel default.
enabled:
thread.isDM ||
enabledBySetupActivation ||
(endpoint.provider === "slack" && type === "channel"),
})
.onConflictDoUpdate({
target: [
chatEndpointResources.endpointId,
chatEndpointResources.type,
chatEndpointResources.providerResourceId,
],
set: {
// Provider inventory and membership lifecycle own availability.
// A delayed message must not resurrect a destination after the bot
// was removed or an installation lost access.
// Slack callbacks are not guaranteed to carry a channel name. Keep
// an inventory- or conversations.info-derived label instead of
// downgrading it to C… / slack:C… when a delayed message arrives.
...(resourceLabel.fallback
? {}
: {
label: sql<string>`case
when ${chatEndpointResources.availability} = 'available' then ${resourceLabel.label}
else ${chatEndpointResources.label}
end`,
}),
updatedAt: new Date(),
},
})
.returning();
return resource;
}
return { findAuthorizedIdentityLink, ensurePrincipal, sponsorAllowsGuest, lockCurrentPrincipalAuthorization, requireCurrentExternalActionAuthorization, isExternalActionAuthorizationChange, ensureResource };
}
@@ -0,0 +1,767 @@
import { createGitHubMessageProvenance } from "../github/message-provenance.js";
import { createDiscordRootThreads } from "../discord/root-threads.js";
import type { EndpointRow, RuntimeContext, InboundRuntimeContext } from "../runtime-types.js";
import {
telegramDeliveryHasZeroMessageId,
} from "../telegram/messages.js";
import { notifyChatDeliveryWork } from "../../chat-work-notifications.js";
import type { ChatEndpointRuntime } from "../../voice/voice-runtime.js";
import type { GitHubReviewEventContext, GitHubIssueEventContext, GitHubReviewPolicy } from "@paperclipai/shared";
import {
createHash,
randomUUID,
} from "node:crypto";
import {
and,
eq,
inArray,
lte,
notInArray,
or,
sql,
} from "drizzle-orm";
import {
externalThreadIdentityCondition,
} from "../../chat-control-chronology.js";
import type { Db } from "@paperclipai/db";
import {
chatDeliveries,
chatEndpointLeases,
chatEndpoints,
} from "@paperclipai/db";
import {
normalizeContentType,
} from "../../../attachment-types.js";
import { logger } from "../../../middleware/logger.js";
import {
type ChatSdkMessageCallbackEvent,
} from "../../chat-sdk-runtime.js";
import { isExternalChatWaitAuthorizationContention } from "../../native-runtime/chat-attachment-reuse.js";
import {
restoreGitHubAttachmentLimitOmissions,
} from "../../chat-github-attachments.js";
import type {
Attachment,
Message,
Thread,
} from "chat";
import { DELIVERY_PROCESSING_STALE_MS, deliveryContentWasRedacted, setupDestinationActivationSuppressed, normalizedDeliveryThreadId, normalizedLifecycleTargetEventId, normalizedLifecycleEffect, deliveryReady, conversationDeliveryOrder, pendingInboundWakeupCondition, lifecycleRuntimeFence } from "./delivery-ledger.js";
import type { createChatMessageLifecycle } from "./lifecycle.js";
import type { LiveInboundMessage } from "../runtime-types.js";
type DeliveryRow = typeof chatDeliveries.$inferSelect;
type ConversationDeliveryQueueOptions = {
DELIVERY_DRAIN_LIMIT: number;
DELIVERY_LEASE_TTL_MS: number;
discordRootThreadPending: ReturnType<typeof createDiscordRootThreads>["discordRootThreadPending"];
reconcileDiscordRootThread: ReturnType<typeof createDiscordRootThreads>["reconcileDiscordRootThread"];
durableExternalThreadIdentity: (threadId: string) => string;
liveInboundMessages: Map<string, LiveInboundMessage>;
processInboundWakeup: (deliveryId: string) => Promise<boolean>;
processLifecycleDelivery: ReturnType<typeof createChatMessageLifecycle>["processLifecycleDelivery"];
processMessage: (endpoint: EndpointRow, thread: Thread, message: Message, trigger: ChatSdkMessageCallbackEvent["trigger"], ingressOnly?: boolean, recoveredProviderUrl?: string | null, runtimeContext?: InboundRuntimeContext, providerUpdateId?: number, admittedDeliveryId?: string | null, receiptReactionSupported?: boolean, preauthorizedUserId?: string | null, deferDrainUntilFollowup?: boolean, suppressSetupDestinationActivation?: boolean, slackSlashControl?: boolean) => Promise<void>;
redactError: (error: unknown) => string;
runtimeContexts: WeakMap<object, RuntimeContext>;
runtimeFor: (endpoint: EndpointRow) => Promise<ChatEndpointRuntime>;
settleRejectedInboundWakeups: (onlyDeliveryId?: string) => Promise<unknown>;
sanitizeFilename: (value: string | undefined) => string | null;
githubManualMessages: WeakMap<object, { policy: GitHubReviewPolicy; revision: number; event: "mention" | "comment" }>;
githubAutomaticMessages: WeakMap<object, { context: GitHubReviewEventContext; revision: number; policy: GitHubReviewPolicy }>;
githubIssueMessages: WeakMap<object, { context: GitHubIssueEventContext; revision: number; policy: GitHubReviewPolicy }>;
conversationDrainKey: (endpointId: string, threadId: string) => string;
conversationLeaseRenewalIntervalMs?: number;
renewConversationDeliveryLease?: (input: { endpointId: string; leaseKey: string; token: string; expiresAt: Date }) => Promise<boolean>;
};
export function createChatConversationDeliveryQueue(db: Db, options: ConversationDeliveryQueueOptions) {
const { DELIVERY_DRAIN_LIMIT, DELIVERY_LEASE_TTL_MS, discordRootThreadPending, reconcileDiscordRootThread, durableExternalThreadIdentity, liveInboundMessages, processInboundWakeup, processLifecycleDelivery, processMessage, redactError, runtimeContexts, runtimeFor, settleRejectedInboundWakeups, sanitizeFilename, githubManualMessages, githubAutomaticMessages, githubIssueMessages, conversationDrainKey } = options;
const { restoreGitHubMessageProvenance } = createGitHubMessageProvenance({ githubManualMessages, githubAutomaticMessages, githubIssueMessages });
async function earliestOpenConversationDelivery(
endpointId: string,
threadId: string,
): Promise<DeliveryRow | null> {
const externalThreadId = sql<string>`${chatDeliveries.normalizedEvent}->'conversation'->>'externalThreadId'`;
const candidate = await db
.select()
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.endpointId, endpointId),
or(
inArray(chatDeliveries.state, ["received", "retry", "processing"]),
and(
eq(chatDeliveries.state, "processed"),
pendingInboundWakeupCondition(),
),
),
notInArray(chatDeliveries.eventKind, [
"reaction_added",
"reaction_removed",
]),
externalThreadIdentityCondition(externalThreadId, threadId),
),
)
.orderBy(...conversationDeliveryOrder())
.limit(1)
.then((rows) => rows[0] ?? null);
if (!candidate) return null;
// A lifecycle callback can reach Paperclip before its create callback. If
// the root becomes durable while the lifecycle row is waiting, always
// return that exact dependency first even when a coarse or malformed
// provider timestamp would otherwise put the edit/delete at the head.
const targetProviderEventId = normalizedLifecycleTargetEventId(candidate);
if (!targetProviderEventId) return candidate;
return db
.select()
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.endpointId, endpointId),
eq(chatDeliveries.providerEventId, targetProviderEventId),
inArray(chatDeliveries.state, ["received", "retry", "processing"]),
),
)
.limit(1)
.then((rows) => rows[0] ?? candidate);
}
async function acquireConversationDeliveryLease(input: {
companyId: string;
endpointId: string;
threadId: string;
}): Promise<{ leaseKey: string; token: string } | null> {
const now = new Date();
const token = randomUUID();
const leaseKey = `inbound:${createHash("sha256").update(durableExternalThreadIdentity(input.threadId)).digest("hex")}`;
const expiresAt = new Date(now.getTime() + DELIVERY_LEASE_TTL_MS);
const inserted = await db
.insert(chatEndpointLeases)
.values({
companyId: input.companyId,
endpointId: input.endpointId,
leaseKey,
token,
expiresAt,
})
.onConflictDoNothing()
.returning({ id: chatEndpointLeases.id });
if (inserted.length > 0) return { leaseKey, token };
const reclaimed = await db
.update(chatEndpointLeases)
.set({ token, expiresAt, updatedAt: now })
.where(
and(
eq(chatEndpointLeases.companyId, input.companyId),
eq(chatEndpointLeases.endpointId, input.endpointId),
eq(chatEndpointLeases.leaseKey, leaseKey),
lte(chatEndpointLeases.expiresAt, now),
),
)
.returning({ id: chatEndpointLeases.id });
return reclaimed.length > 0 ? { leaseKey, token } : null;
}
async function releaseConversationDeliveryLease(input: {
endpointId: string;
leaseKey: string;
token: string;
}): Promise<void> {
await db
.delete(chatEndpointLeases)
.where(
and(
eq(chatEndpointLeases.endpointId, input.endpointId),
eq(chatEndpointLeases.leaseKey, input.leaseKey),
eq(chatEndpointLeases.token, input.token),
),
);
}
function messageFromDelivery(
delivery: DeliveryRow,
thread: Thread,
endpointRuntime: ChatEndpointRuntime,
): {
message: Message;
receiptReactionSupported: boolean;
suppressSetupDestinationActivation: boolean;
trigger: ChatSdkMessageCallbackEvent["trigger"];
providerUrl: string | null;
} | null {
if (telegramDeliveryHasZeroMessageId(delivery, endpointRuntime.provider))
return null;
const normalized = delivery.normalizedEvent as {
acknowledgement?: { receiptReactionSupported?: unknown };
kind?: unknown;
trigger?: unknown;
principal?: {
externalId?: unknown;
displayName?: unknown;
handle?: unknown;
};
message?: {
providerMessageId?: unknown;
sourceProviderMessageId?: unknown;
text?: unknown;
mentionedBot?: unknown;
attachments?: Array<{
name?: unknown;
mimeType?: unknown;
size?: unknown;
recovery?: unknown;
}>;
attachmentLimitOmissions?: unknown;
photonReply?: { guid?: unknown; part?: unknown };
};
conversation?: { providerUrl?: unknown };
};
const providerMessageId = normalized.message?.providerMessageId;
const externalId = normalized.principal?.externalId;
if (typeof providerMessageId !== "string" || typeof externalId !== "string")
return null;
const isTeams = endpointRuntime.provider === "microsoft-teams";
const referenceOnlyTeamsAttachments = isTeams && !thread.isDM;
const originFence = lifecycleRuntimeFence(delivery);
const storedAttachments = normalized.message?.attachments ?? [];
const attachments = (
isTeams ? storedAttachments.slice(0, 20) : storedAttachments
)
.map((attachment) => {
if (!attachment || typeof attachment !== "object") return null;
const rehydrated =
attachment.recovery &&
(!referenceOnlyTeamsAttachments ||
(typeof attachment.recovery === "object" &&
attachment.recovery !== null &&
"locator" in attachment.recovery &&
typeof attachment.recovery.locator === "object" &&
attachment.recovery.locator !== null &&
"kind" in attachment.recovery.locator &&
attachment.recovery.locator.kind === "teams_inline_image"))
? endpointRuntime.rehydrateAttachment(attachment.recovery, {
threadId: thread.id,
messageId: typeof normalized.message?.sourceProviderMessageId === "string" ? normalized.message.sourceProviderMessageId : providerMessageId,
runtimeGeneration: originFence?.generation,
credentialFingerprint: originFence?.credentialFingerprint,
principalExternalId: externalId,
isDirectMessage: thread.isDM,
})
: null;
if (rehydrated) return rehydrated;
if (
isTeams ||
endpointRuntime.provider === "imessage-photon" ||
(endpointRuntime.provider === "telegram" &&
typeof attachment.recovery === "object" &&
attachment.recovery !== null &&
"locator" in attachment.recovery &&
typeof attachment.recovery.locator === "object" &&
attachment.recovery.locator !== null &&
"kind" in attachment.recovery.locator &&
attachment.recovery.locator.kind === "telegram_media")
) {
// Non-personal references and personal files whose safe download
// capability did not survive restart still belong to this input.
// Keep bounded metadata so ingestion records an explicit omission;
// never persist bearer URLs or fabricate a download capability.
// Even a legacy URL locator cannot enable non-personal downloads.
// A Telegram media locator which fails its exact source proof also
// remains an unavailable input, never an empty attachment-free turn.
return {
type: "file",
name:
sanitizeFilename(
typeof attachment.name === "string"
? attachment.name
: undefined,
) ?? undefined,
mimeType: normalizeContentType(
typeof attachment.mimeType === "string"
? attachment.mimeType
: undefined,
).slice(0, 255),
size:
typeof attachment.size === "number" &&
Number.isSafeInteger(attachment.size) &&
attachment.size >= 0
? attachment.size
: undefined,
} as Attachment;
}
return null;
})
.filter((attachment): attachment is Attachment => Boolean(attachment));
const message = {
id: providerMessageId,
threadId: thread.id,
text:
typeof normalized.message?.text === "string"
? normalized.message.text
: "",
formatted: { type: "root", children: [] },
raw:
endpointRuntime.provider === "imessage-photon" &&
typeof normalized.message?.photonReply?.guid === "string"
? {
replyTargetGuid: normalized.message.photonReply.guid,
threadOriginatorPart: normalized.message.photonReply.part,
}
: endpointRuntime.provider === "github" && typeof normalized.message?.sourceProviderMessageId === "string"
? { githubSourceMessageId: normalized.message.sourceProviderMessageId } : {},
author: {
userId: externalId,
userName:
typeof normalized.principal?.handle === "string"
? normalized.principal.handle
: externalId,
fullName:
typeof normalized.principal?.displayName === "string"
? normalized.principal.displayName
: externalId,
isBot: false,
isMe: false,
isSystem: false,
},
metadata: { dateSent: delivery.receivedAt, edited: false },
attachments,
links: [],
isMention: normalized.message?.mentionedBot === true,
} as unknown as Message;
restoreGitHubMessageProvenance(delivery, endpointRuntime, message);
if (
endpointRuntime.provider === "github" ||
endpointRuntime.provider === "imessage-photon"
) {
restoreGitHubAttachmentLimitOmissions(
message,
normalized.message?.attachmentLimitOmissions,
);
}
const normalizedTrigger = normalized.trigger;
const trigger: ChatSdkMessageCallbackEvent["trigger"] =
normalizedTrigger === "direct_message" ||
normalizedTrigger === "mention" ||
normalizedTrigger === "subscribed_message" ||
normalizedTrigger === "unaddressed_message"
? normalizedTrigger
: normalized.kind === "direct_message"
? "direct_message"
: normalized.kind === "mention"
? "mention"
: "subscribed_message";
return {
message,
// Legacy deliveries all originated from native provider messages and
// therefore retain the historical default.
receiptReactionSupported:
normalized.acknowledgement?.receiptReactionSupported !== false,
suppressSetupDestinationActivation: setupDestinationActivationSuppressed(
delivery.normalizedEvent,
),
trigger,
providerUrl:
typeof normalized.conversation?.providerUrl === "string"
? normalized.conversation.providerUrl
: null,
};
}
/**
* Drain one external conversation in durable receipt order. The database
* lease is endpoint + thread scoped, so separate server processes can work
* on different conversations concurrently but can never mutate the same
* Paperclip task from two inbound turns at once.
*/
async function drainConversationDeliveries(
endpointId: string,
threadId: string,
): Promise<boolean> {
const endpoint = await db
.select()
.from(chatEndpoints)
.where(eq(chatEndpoints.id, endpointId))
.then((rows) => rows[0] ?? null);
if (!endpoint) return false;
if (endpoint.status === "paused" || endpoint.status === "attention")
return false;
const lease = await acquireConversationDeliveryLease({
companyId: endpoint.companyId,
endpointId,
threadId,
});
if (!lease) return false;
let leaseOwned = true;
let renewal: Promise<void> | null = null;
const renewTimer = setInterval(
() => {
if (!leaseOwned || renewal) return;
const expiresAt = new Date(Date.now() + DELIVERY_LEASE_TTL_MS);
renewal = Promise.resolve()
.then(async () => {
const renewed = options.renewConversationDeliveryLease
? await options.renewConversationDeliveryLease({
endpointId,
leaseKey: lease.leaseKey,
token: lease.token,
expiresAt,
})
: await db
.update(chatEndpointLeases)
.set({ expiresAt, updatedAt: new Date() })
.where(
and(
eq(chatEndpointLeases.endpointId, endpointId),
eq(chatEndpointLeases.leaseKey, lease.leaseKey),
eq(chatEndpointLeases.token, lease.token),
),
)
.returning({ id: chatEndpointLeases.id })
.then((rows) => rows.length > 0);
if (!renewed) leaseOwned = false;
})
.catch((error) => {
// A failed renewal has unknown durability. Stop this local drain at
// the next delivery boundary so another worker can never overlap
// later turns after the original lease expires or is reclaimed.
leaseOwned = false;
logger.warn(
{ endpointId, error: redactError(error) },
"could not renew external chat conversation lease",
);
})
.finally(() => {
renewal = null;
});
},
options.conversationLeaseRenewalIntervalMs ?? DELIVERY_LEASE_TTL_MS / 3,
);
renewTimer.unref?.();
try {
for (
let processed = 0;
processed < DELIVERY_DRAIN_LIMIT;
processed += 1
) {
if (!leaseOwned) break;
const delivery = await earliestOpenConversationDelivery(
endpointId,
threadId,
);
if (!delivery || !deliveryReady(delivery, new Date())) break;
if (delivery.state === "processed") {
if (!(await processInboundWakeup(delivery.id))) break;
continue;
}
if (telegramDeliveryHasZeroMessageId(delivery, endpoint.provider)) {
const filtered = await db
.transaction(async (tx) => {
// Do not round-trip a PostgreSQL microsecond timestamp through
// JS Date for a CAS. Lock the exact receipt, then revalidate the
// current source and claim before changing it. A competing live
// claim is a retry boundary, not permission to settle its work.
const [current] = await tx
.select()
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.id, delivery.id),
eq(chatDeliveries.companyId, endpoint.companyId),
eq(chatDeliveries.endpointId, endpoint.id),
),
)
.for("update", { noWait: true });
const filteredAt = new Date();
if (
!current ||
current.state !== delivery.state ||
current.eventKind !== delivery.eventKind ||
!["received", "retry", "processing"].includes(current.state) ||
normalizedDeliveryThreadId(current) !== threadId ||
!deliveryReady(current, filteredAt) ||
!telegramDeliveryHasZeroMessageId(current, endpoint.provider)
)
return false;
await notifyChatDeliveryWork(tx);
const changed = await tx
.update(chatDeliveries)
.set({
state: "filtered",
nextAttemptAt: null,
processedAt: filteredAt,
updatedAt: filteredAt,
redactedError:
"Telegram message ID zero cannot authorize ordinary work",
})
.where(eq(chatDeliveries.id, current.id))
.returning({ id: chatDeliveries.id });
return changed.length === 1;
})
.catch((error: unknown) => {
if (isExternalChatWaitAuthorizationContention(error))
return false;
throw error;
});
if (!filtered) break;
// Retain original evidence/comments; filtering adds no user content
// and cannot hydrate attachments, invoke commands, or send feedback.
await settleRejectedInboundWakeups(delivery.id);
liveInboundMessages.delete(delivery.id);
continue;
}
if (endpoint.status === "archived" || endpoint.status === "revoked") {
await db.transaction(async (tx) => {
await notifyChatDeliveryWork(tx);
return tx
.update(chatDeliveries)
.set({
state: "filtered",
redactedError: "Connection was removed before processing",
processedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(chatDeliveries.id, delivery.id));
});
liveInboundMessages.delete(delivery.id);
continue;
}
if (
endpoint.provider === "discord" &&
discordRootThreadPending(delivery)
) {
// This is the only recovery path for the narrow crash window after
// pre-admission but before Chat's ordinary callback. A successful
// read proves the root thread exists; an explicit 404/410 proves it
// does not. Ambiguous transport and auth failures remain retryable.
if (await reconcileDiscordRootThread(endpoint, delivery, threadId)) {
continue;
}
break;
}
const live = liveInboundMessages.get(delivery.id);
try {
if (
delivery.eventKind === "message_updated" ||
delivery.eventKind === "message_deleted" ||
delivery.eventKind === "message_restored"
) {
await processLifecycleDelivery(endpoint, delivery);
} else if (live) {
await processMessage(
endpoint,
live.thread,
live.message,
live.trigger,
false,
live.providerUrl,
live.runtimeContext,
undefined,
delivery.id,
live.receiptReactionSupported,
undefined,
false,
setupDestinationActivationSuppressed(delivery.normalizedEvent),
);
} else {
if (deliveryContentWasRedacted(delivery.normalizedEvent)) {
const filteredAt = new Date();
await db.transaction(async (tx) => {
await notifyChatDeliveryWork(tx);
return tx
.update(chatDeliveries)
.set({
state: "filtered",
principalId: null,
nextAttemptAt: null,
processedAt: filteredAt,
redactedError:
"Provisional Teams setup reply could not be hydrated from a current provider event",
updatedAt: filteredAt,
})
.where(
and(
eq(chatDeliveries.id, delivery.id),
inArray(chatDeliveries.state, [
"received",
"retry",
"processing",
]),
),
);
});
continue;
}
const endpointRuntime = await runtimeFor(endpoint);
const deliveryRuntimeFence = lifecycleRuntimeFence(delivery);
const currentRuntimeContext = runtimeContexts.get(
endpointRuntime as object,
);
const thread = endpointRuntime.thread(threadId);
const reconstructed = messageFromDelivery(
delivery,
thread,
endpointRuntime,
);
if (!reconstructed) {
await db.transaction(async (tx) => {
await notifyChatDeliveryWork(tx);
return tx
.update(chatDeliveries)
.set({
state: "failed",
redactedError: "Normalized delivery is incomplete",
updatedAt: new Date(),
})
.where(eq(chatDeliveries.id, delivery.id));
});
continue;
}
await processMessage(
endpoint,
thread,
reconstructed.message,
reconstructed.trigger,
false,
reconstructed.providerUrl,
deliveryRuntimeFence
? {
...deliveryRuntimeFence,
endpointRuntime,
}
: currentRuntimeContext,
undefined,
delivery.id,
reconstructed.receiptReactionSupported,
undefined,
false,
reconstructed.suppressSetupDestinationActivation,
);
}
} catch (error) {
logger.warn(
{
endpointId,
deliveryId: delivery.id,
error: redactError(error),
},
"external chat delivery will retry in receipt order",
);
}
const state = await db
.select({ state: chatDeliveries.state })
.from(chatDeliveries)
.where(eq(chatDeliveries.id, delivery.id))
.then((rows) => rows[0]?.state ?? null);
if (
state === "processed" ||
state === "filtered" ||
state === "failed"
) {
if (state !== "processed")
await settleRejectedInboundWakeups(delivery.id);
liveInboundMessages.delete(delivery.id);
continue;
}
// A retry delay or another still-live processing claim is a strict
// head-of-line barrier; later replies may not overtake it.
break;
}
} finally {
clearInterval(renewTimer);
await renewal;
await releaseConversationDeliveryLease({ endpointId, ...lease });
}
const next = await earliestOpenConversationDelivery(endpointId, threadId);
if (next?.state === "processed") {
const [ready] = await db
.select({ id: chatDeliveries.id })
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.id, next.id),
pendingInboundWakeupCondition(true),
),
)
.limit(1);
return Boolean(ready);
}
return Boolean(next && deliveryReady(next, new Date()));
}
async function nextInboundDeliveryAt(): Promise<number | null> {
const rows = await db.select({ delivery: {
id: chatDeliveries.id, endpointId: chatDeliveries.endpointId, providerEventId: chatDeliveries.providerEventId,
state: chatDeliveries.state, eventKind: chatDeliveries.eventKind,
// Retain scheduling metadata only, not message bodies or attachment data.
normalizedEvent: sql<DeliveryRow["normalizedEvent"]>`jsonb_build_object(
'conversation', ${chatDeliveries.normalizedEvent}->'conversation',
'lifecycle', ${chatDeliveries.normalizedEvent}->'lifecycle',
'message', jsonb_build_object(
'targetProviderEventId', ${chatDeliveries.normalizedEvent}->'message'->'targetProviderEventId',
'admissionDependencyEventId', ${chatDeliveries.normalizedEvent}->'message'->'admissionDependencyEventId'))`,
}, at: sql<string | null>`case
when ${chatDeliveries.state} = 'processing' then ${chatDeliveries.updatedAt} + ${DELIVERY_PROCESSING_STALE_MS} * interval '1 millisecond'
when ${chatDeliveries.state} = 'processed' then (
select min(case when a.status = 'processing' then a.updated_at + ${DELIVERY_PROCESSING_STALE_MS} * interval '1 millisecond'
else coalesce((a.result->>'retryAt')::timestamptz, now()) end)
from chat_actions a where a.delivery_id = chat_deliveries.id
and a.company_id = chat_deliveries.company_id and a.endpoint_id = chat_deliveries.endpoint_id
and a.kind = 'inbound_wakeup' and a.status in ('issued', 'processing'))
else coalesce(${chatDeliveries.nextAttemptAt}, now()) end` })
.from(chatDeliveries).where(and(
sql`not exists (select 1 from chat_endpoints e where e.id = ${chatDeliveries.endpointId} and e.provider = 'agentmail')`,
// Provider lifecycle receipts can restore an unavailable endpoint.
// The ordinary conversation drain still waits for operator resume.
or(sql`${chatDeliveries.normalizedEvent}->'lifecycle' is not null`,
sql`not exists (select 1 from chat_endpoints e where e.id = ${chatDeliveries.endpointId} and e.status in ('paused', 'attention'))`),
or(inArray(chatDeliveries.state, ["received", "retry", "processing"]),
and(eq(chatDeliveries.state, "processed"), pendingInboundWakeupCondition())),
)).orderBy(...conversationDeliveryOrder());
// Only an ordered conversation head can run. Reaction and provider
// lifecycle receipts are independent; lifecycle edits may select their
// exact admission dependency before the chronologically earliest row.
const byEvent = new Map(rows.map(row => [`${row.delivery.endpointId}:${row.delivery.providerEventId}`, row]));
const conversations = new Set<string>();
let next: number | null = null;
for (let row of rows) {
const delivery = row.delivery;
const threadId = normalizedDeliveryThreadId(delivery);
if (threadId && !normalizedLifecycleEffect(delivery) && !["reaction_added", "reaction_removed"].includes(delivery.eventKind)) {
const key = conversationDrainKey(delivery.endpointId, threadId);
if (conversations.has(key)) continue;
conversations.add(key);
const dependency = normalizedLifecycleTargetEventId(delivery);
const target = dependency ? byEvent.get(`${delivery.endpointId}:${dependency}`) : undefined;
if (target && ["received", "retry", "processing"].includes(target.delivery.state)) row = target;
}
if (row.at !== null) {
const at = new Date(row.at).getTime();
next = next === null ? at : Math.min(next, at);
}
}
return next;
}
return { drainConversationDeliveries, nextInboundDeliveryAt };
}
@@ -0,0 +1,241 @@
import type { LifecycleRuntimeFence } from "../runtime-types.js";
import {
asc,
sql,
type SQL,
} from "drizzle-orm";
import {
chatActions,
chatDeliveries,
} from "@paperclipai/db";
import {
type ChatProviderLifecycleEffect,
} from "../../chat-provider-lifecycle.js";
type DeliveryRow = typeof chatDeliveries.$inferSelect;
const DELIVERY_PROCESSING_STALE_MS = 60_000;
function deliveryContentWasRedacted(normalizedEvent: unknown): boolean {
if (
!normalizedEvent ||
typeof normalizedEvent !== "object" ||
Array.isArray(normalizedEvent)
)
return false;
const filtering = (normalizedEvent as { filtering?: unknown }).filtering;
return (
Boolean(filtering) &&
typeof filtering === "object" &&
!Array.isArray(filtering) &&
(filtering as { contentRetained?: unknown }).contentRetained === false
);
}
function setupDestinationActivationSuppressed(
normalizedEvent: unknown,
): boolean {
if (
!normalizedEvent ||
typeof normalizedEvent !== "object" ||
Array.isArray(normalizedEvent)
)
return false;
const normalized = normalizedEvent as {
admission?: { origin?: unknown };
admissionAuthorization?: { version?: unknown };
};
return (
normalized.admission?.origin === "provider_confirmed_action" ||
normalized.admissionAuthorization?.version === 1
);
}
function normalizedDeliveryThreadId(delivery: Pick<DeliveryRow, "normalizedEvent">): string | null {
const normalized = delivery.normalizedEvent as {
conversation?: { externalThreadId?: unknown };
};
return typeof normalized.conversation?.externalThreadId === "string"
? normalized.conversation.externalThreadId
: null;
}
function normalizedLifecycleTargetEventId(
delivery: Pick<DeliveryRow, "eventKind" | "normalizedEvent">,
): string | null {
if (
delivery.eventKind !== "message_updated" &&
delivery.eventKind !== "message_deleted" &&
delivery.eventKind !== "message_restored"
)
return null;
const normalized = delivery.normalizedEvent as {
message?: { targetProviderEventId?: unknown; admissionDependencyEventId?: unknown };
};
// An edit can also introduce a new addressed request. Process that durable
// admission before waiting for the old, potentially unadmitted source.
if (typeof normalized.message?.admissionDependencyEventId === "string")
return normalized.message.admissionDependencyEventId;
return typeof normalized.message?.targetProviderEventId === "string"
? normalized.message.targetProviderEventId
: null;
}
function normalizedLifecycleEffect(
delivery: Pick<DeliveryRow, "normalizedEvent">,
): ChatProviderLifecycleEffect | null {
const normalized = delivery.normalizedEvent as { lifecycle?: unknown };
const value = normalized.lifecycle;
if (!value || typeof value !== "object" || Array.isArray(value))
return null;
const effect = value as Record<string, unknown>;
if (
(effect.provider !== "slack" &&
effect.provider !== "github" &&
effect.provider !== "discord" &&
effect.provider !== "microsoft-teams" &&
effect.provider !== "telegram") ||
typeof effect.providerEventId !== "string"
)
return null;
if (
effect.kind === "endpoint" &&
(effect.availability === "available" ||
effect.availability === "attention" ||
effect.availability === "revoked") &&
typeof effect.reason === "string"
)
return effect as ChatProviderLifecycleEffect;
if (
effect.kind === "resource" &&
(effect.availability === "available" ||
effect.availability === "unavailable" ||
effect.availability === "removed") &&
typeof effect.providerResourceId === "string" &&
(effect.previousProviderResourceId === undefined ||
typeof effect.previousProviderResourceId === "string") &&
typeof effect.resourceType === "string" &&
typeof effect.label === "string"
)
return effect as ChatProviderLifecycleEffect;
return null;
}
function deliveryReady(delivery: DeliveryRow, now: Date): boolean {
// Accepted input may still own the conversation head through its wake
// outbox. The action dispatcher checks its own retry/claim deadline.
if (delivery.state === "processed") return true;
if (delivery.state === "received")
return !delivery.nextAttemptAt || delivery.nextAttemptAt <= now;
if (delivery.state === "retry") {
return !delivery.nextAttemptAt || delivery.nextAttemptAt <= now;
}
return (
delivery.state === "processing" &&
delivery.updatedAt <=
new Date(now.getTime() - DELIVERY_PROCESSING_STALE_MS)
);
}
function conversationDeliveryOrder() {
return [
asc(
sql`coalesce(nullif(${chatDeliveries.normalizedEvent}->'message'->>'providerSentAt', '')::timestamptz, ${chatDeliveries.receivedAt})`,
),
// GitHub timestamps and Telegram message dates have one-second
// resolution. GitHub's numeric comment id and Telegram's message_id
// are monotonic within one conversation, so use them before receipt
// order. The Telegram update_id is a final provider-native tie-breaker
// for unusual payloads that lack a usable message_id.
asc(sql`coalesce(
case
when ${chatDeliveries.normalizedEvent}->'message'->>'providerMessageSequence' ~ '^[0-9]+$'
then (${chatDeliveries.normalizedEvent}->'message'->>'providerMessageSequence')::numeric
else null
end,
case
when ${chatDeliveries.normalizedEvent}->'message'->>'providerMessageId' ~ '^[0-9]+$'
then (${chatDeliveries.normalizedEvent}->'message'->>'providerMessageId')::numeric
else null
end
)`),
asc(sql`case
when ${chatDeliveries.normalizedEvent}->'message'->>'providerUpdateId' ~ '^[0-9]+$'
then (${chatDeliveries.normalizedEvent}->'message'->>'providerUpdateId')::numeric
else null
end`),
// GitHub exposes no sortable webhook sequence. When an edit and delete
// share its whole-second updated_at value, preserve the only valid
// lifecycle state transition: update before delete. Provider-native
// update ids (Telegram) remain the stronger preceding key.
asc(sql`case ${chatDeliveries.eventKind}
when 'message_updated' then 1
when 'message_deleted' then 2
when 'message_restored' then 3
else 0
end`),
asc(chatDeliveries.receivedAt),
asc(chatDeliveries.id),
];
}
function pendingInboundWakeupCondition(readyOnly = false): SQL {
return sql`exists (
select 1 from ${chatActions}
where ${chatActions.deliveryId} = ${chatDeliveries.id}
and ${chatActions.endpointId} = ${chatDeliveries.endpointId}
and ${chatActions.companyId} = ${chatDeliveries.companyId}
and ${chatActions.kind} = 'inbound_wakeup'
and ${chatActions.status} in ('issued', 'processing')
${
readyOnly
? sql`and (
(${chatActions.status} = 'issued' and (${chatActions.result}->>'retryAt' is null or (${chatActions.result}->>'retryAt')::timestamptz <= now()))
or (${chatActions.status} = 'processing' and ${chatActions.updatedAt} <= ${new Date(Date.now() - DELIVERY_PROCESSING_STALE_MS).toISOString()}::timestamptz)
)`
: sql``
}
)`;
}
function lifecycleRuntimeFence(
delivery: DeliveryRow,
): LifecycleRuntimeFence | null {
const normalized = delivery.normalizedEvent;
if (
!normalized ||
typeof normalized !== "object" ||
Array.isArray(normalized)
)
return null;
const value = (normalized as Record<string, unknown>).runtimeContext;
if (!value || typeof value !== "object" || Array.isArray(value))
return null;
const context = value as Record<string, unknown>;
return typeof context.generation === "number" &&
Number.isSafeInteger(context.generation) &&
typeof context.credentialFingerprint === "string"
? {
generation: context.generation as number,
credentialFingerprint: context.credentialFingerprint,
}
: null;
}
function normalizedLifecycleWithRuntimeFence(
delivery: DeliveryRow,
runtimeContext: LifecycleRuntimeFence,
): Record<string, unknown> {
const normalized =
delivery.normalizedEvent &&
typeof delivery.normalizedEvent === "object" &&
!Array.isArray(delivery.normalizedEvent)
? (delivery.normalizedEvent as Record<string, unknown>)
: {};
return { ...normalized, runtimeContext };
}
export { DELIVERY_PROCESSING_STALE_MS, deliveryContentWasRedacted, setupDestinationActivationSuppressed, normalizedDeliveryThreadId, normalizedLifecycleTargetEventId, normalizedLifecycleEffect, deliveryReady, conversationDeliveryOrder, pendingInboundWakeupCondition, lifecycleRuntimeFence, normalizedLifecycleWithRuntimeFence };
@@ -0,0 +1,360 @@
import { providerResourceType, durableExternalThreadIdentity, canonicalProviderResourceId, chatSurfaceKind } from "./surfaces.js";
import type { EndpointRow, InboundRuntimeContext } from "../runtime-types.js";
import {
MAX_INBOUND_TEXT,
stableExternalPrincipalId,
} from "../message-lifecycle.js";
import {
telegramMessageSequence,
telegramMessageSentAt,
} from "../telegram/messages.js";
import {
slackMessageSentAt,
} from "../slack/messages.js";
import {
teamsActivitySentAt,
} from "../microsoft-teams/messages.js";
import type { ChatEndpointRuntime } from "../../voice/voice-runtime.js";
import { chatGitHubConfigurations } from "@paperclipai/db";
import type { GitHubReviewEventContext, GitHubIssueEventContext, GitHubReviewPolicy } from "@paperclipai/shared";
import { photonReplyReference } from "../../photon/adapter.js";
import {
and,
eq,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatEndpointResources,
} from "@paperclipai/db";
import type {
ChatEventKind,
} from "@paperclipai/shared";
import {
normalizeContentType,
} from "../../../attachment-types.js";
import {
deriveTeamsPersonalRecipient,
} from "../../chat-teams-personal-recipient.js";
import {
type ChatSdkMessageCallbackEvent,
} from "../../chat-sdk-runtime.js";
import {
githubAttachmentLimitOmissions,
} from "../../chat-github-attachments.js";
import { chatProviderConversationUrl } from "../../chat-provider-links.js";
import type {
Attachment,
Message,
Thread,
} from "chat";
export type InboundEnvelopeInput = {
endpoint: EndpointRow;
thread: Thread;
message: Message;
trigger: ChatSdkMessageCallbackEvent["trigger"];
receiptReactionSupported: boolean;
recoveredProviderUrl: string | null;
runtimeContext?: InboundRuntimeContext;
providerUpdateId?: number;
admittedDeliveryId: string | null;
suppressSetupDestinationActivation: boolean;
slackSlashControl: boolean;
};
type InboundEnvelopeOptions = {
runtimeFor: (endpoint: EndpointRow) => Promise<ChatEndpointRuntime>;
sanitizeFilename: (value: string | undefined) => string | null;
githubManualMessages: WeakMap<object, { policy: GitHubReviewPolicy; revision: number; event: "mention" | "comment" }>;
githubAutomaticMessages: WeakMap<object, { context: GitHubReviewEventContext; revision: number; policy: GitHubReviewPolicy }>;
githubIssueMessages: WeakMap<object, { context: GitHubIssueEventContext; revision: number; policy: GitHubReviewPolicy }>;
};
export function createInboundEnvelope(db: Db, options: InboundEnvelopeOptions) {
const { runtimeFor, sanitizeFilename, githubManualMessages, githubAutomaticMessages, githubIssueMessages } = options;
async function prepareInboundEnvelope(input: InboundEnvelopeInput) {
const { endpoint, thread, message, trigger, receiptReactionSupported, recoveredProviderUrl, runtimeContext, providerUpdateId, admittedDeliveryId, suppressSetupDestinationActivation, slackSlashControl } = input;
const githubAutomatic = endpoint.provider === "github" ? githubAutomaticMessages.get(message) : undefined;
const githubIssue = endpoint.provider === "github" ? githubIssueMessages.get(message) : undefined;
let githubManual = githubManualMessages.get(message);
if (endpoint.provider === "github" && !githubAutomatic && !githubIssue && !githubManual) {
const [config] = await db.select().from(chatGitHubConfigurations).where(eq(chatGitHubConfigurations.endpointId, endpoint.id));
if (config) {
const repository = thread.channelId.replace(/^github:/, "").toLowerCase();
const [repo] = await db.select().from(chatEndpointResources).where(and(eq(chatEndpointResources.endpointId, endpoint.id), eq(chatEndpointResources.providerResourceId, repository)));
const repositoryId = String(repo?.metadata?.providerRepositoryId ?? "");
githubManual = { policy: { ...config.configuration.defaults, ...config.configuration.repositories[repositoryId] }, revision: config.revision, event: trigger === "mention" || message.isMention ? "mention" : "comment" };
}
}
const providerEventId = `${durableExternalThreadIdentity(thread.id)}:${message.id}`;
const surfaceKind = chatSurfaceKind(endpoint.provider, thread);
const addressed =
endpoint.provider === "imessage-photon" ||
trigger === "mention" ||
trigger === "direct_message" ||
message.isMention === true;
const eventKind: ChatEventKind =
trigger === "direct_message"
? "direct_message"
: trigger === "mention"
? "mention"
: "message";
// A callback remains tied to the runtime that authenticated and parsed it.
// In particular, do not resolve the registry again here: pause or rotation
// may already have removed that instance, and recreating a runtime before
// the durable admission fence would either throw or bind old payload data
// to new credentials.
const endpointRuntime =
runtimeContext?.endpointRuntime ?? (await runtimeFor(endpoint));
// Non-personal generic files still require a separate Graph grant. Only
// exact inline image resources from the authenticated original activity
// receive a scope-bound, URL-free recovery locator. Reconstructed input
// can retain that proof, but cannot mint one from legacy fetch metadata.
const teamsNonPersonal =
endpoint.provider === "microsoft-teams" && !thread.isDM;
const attachmentSource = {
threadId: thread.id,
messageId: message.id,
runtimeGeneration: runtimeContext?.generation,
credentialFingerprint: runtimeContext?.credentialFingerprint,
principalExternalId: stableExternalPrincipalId(
endpoint.provider,
message.author,
message.raw,
),
isDirectMessage: thread.isDM,
...(runtimeContext?.endpointRuntime && !admittedDeliveryId
? { message }
: {}),
};
const inlineRecovery = new Map<
Attachment,
ReturnType<ChatEndpointRuntime["attachmentRecoveryDescriptor"]>
>();
const nativeInboundAttachments = teamsNonPersonal
? message.attachments.slice(0, 20).flatMap((attachment) => {
const descriptor = endpointRuntime.attachmentRecoveryDescriptor(
attachment,
attachmentSource,
);
if (descriptor?.locator.kind !== "teams_inline_image") return [];
const restored = endpointRuntime.rehydrateAttachment(
descriptor,
attachmentSource,
);
if (!restored) return [];
inlineRecovery.set(attachment, descriptor);
return [restored];
})
: message.attachments;
const unavailableTeamsReferences = teamsNonPersonal
? Math.min(message.attachments.length, 20) -
nativeInboundAttachments.length
: 0;
const providerSentAt =
endpoint.provider === "microsoft-teams"
? teamsActivitySentAt(message.raw)
: endpoint.provider === "telegram"
? telegramMessageSentAt(message.raw)
: endpoint.provider === "slack"
? slackMessageSentAt(message.raw, message.id)
: message.metadata.dateSent instanceof Date &&
Number.isFinite(message.metadata.dateSent.getTime())
? message.metadata.dateSent
: null;
const providerSentAtSource = providerSentAt
? endpoint.provider === "microsoft-teams"
? "teams_activity_timestamp"
: endpoint.provider === "telegram"
? "telegram_message_date"
: endpoint.provider === "slack"
? "slack_message_ts"
: null
: null;
const providerUrl =
(githubAutomatic
? `https://github.com/${githubAutomatic.context.repository}/pull/${githubAutomatic.context.pullNumber}`
: githubIssue ? `https://github.com/${githubIssue.context.repository}/issues/${githubIssue.context.issueNumber}` : null) ??
// Description/review requests and edits have synthetic ledger IDs, not
// native comment IDs. Their verified source URL must not acquire a fake
// issuecomment anchor from that ledger identity.
(endpoint.provider === "github" && githubManual && !receiptReactionSupported ? recoveredProviderUrl : null) ??
chatProviderConversationUrl({
provider: endpoint.provider,
providerAccountId: endpoint.providerAccountId,
botUsername: endpoint.botUsername,
threadId: thread.id,
providerMessageId: message.id,
raw: message.raw,
}) ?? recoveredProviderUrl;
// Preserve only the closed personal-recipient proof from the runtime that
// authenticated this original activity. A reconstructed delivery's AAD-only
// author and SDK route caches cannot recreate its Bot Framework recipient.
const teamsPersonalRecipient =
endpoint.provider === "microsoft-teams" &&
runtimeContext?.endpointRuntime &&
!admittedDeliveryId &&
endpoint.providerAccountId &&
endpoint.botExternalId
? deriveTeamsPersonalRecipient(message, {
companyId: endpoint.companyId,
endpointId: endpoint.id,
runtimeGeneration: runtimeContext.generation,
credentialFingerprint: runtimeContext.credentialFingerprint,
tenantId: endpoint.providerAccountId,
botAppId: endpoint.botExternalId,
providerEventId,
threadId: thread.id,
isDirectMessage: thread.isDM,
})
: null;
const normalized = {
providerEventId,
kind: eventKind,
trigger,
...(teamsPersonalRecipient ? { teamsPersonalRecipient } : {}),
...(slackSlashControl
? { admission: { origin: "slack_slash_control" } }
: suppressSetupDestinationActivation
? // A slash-command root is provider-confirmed only after an enabled
// destination authorized its transport. Persist that closed origin so
// crash recovery can never reinterpret it as first-time setup traffic
// and undo a later operator reach revocation.
{ admission: { origin: "provider_confirmed_action" } }
: {}),
...(runtimeContext
? {
runtimeContext: {
credentialFingerprint: runtimeContext.credentialFingerprint,
generation: runtimeContext.generation,
},
}
: {}),
acknowledgement: {
// Some provider callbacks represent an auditable user command without
// a provider message that can receive a reaction. Persist the
// distinction so a crash/retry cannot mistake the synthetic ledger id
// for a native message id.
receiptReactionSupported,
},
...(githubAutomatic ? { githubAutomatic } : {}),
...(githubIssue ? { githubIssue } : {}),
...(githubManual ? { githubManual } : {}),
principal: {
// Bot/system authors have already been rejected at this intake boundary.
isBot: false,
externalId: stableExternalPrincipalId(
endpoint.provider,
message.author,
message.raw,
),
displayName: message.author.fullName,
handle: message.author.userName,
},
resource: {
type: providerResourceType(endpoint.provider, surfaceKind),
providerResourceId: canonicalProviderResourceId(
endpoint.provider,
thread,
),
label: thread.channel.name ?? thread.channelId,
},
conversation: {
externalConversationId: thread.channelId,
externalThreadId: thread.id,
label: thread.channel.name ?? thread.channelId,
isDirectMessage: thread.isDM,
providerUrl,
},
message: {
providerMessageId: message.id,
...(endpoint.provider === "github" && typeof (message.raw as { githubSourceMessageId?: unknown })?.githubSourceMessageId === "string" && /^[1-9][0-9]*$/.test((message.raw as { githubSourceMessageId: string }).githubSourceMessageId)
? { sourceProviderMessageId: (message.raw as { githubSourceMessageId: string }).githubSourceMessageId } : {}),
...(endpoint.provider === "imessage-photon"
? { photonReply: photonReplyReference(message.raw) }
: {}),
providerMessageSequence:
endpoint.provider === "telegram"
? telegramMessageSequence(message.raw)
: null,
providerUpdateId:
endpoint.provider === "telegram" ? (providerUpdateId ?? null) : null,
text: message.text.slice(0, MAX_INBOUND_TEXT),
mentionedBot: message.isMention === true,
providerSentAt: providerSentAt?.toISOString() ?? null,
...(providerSentAtSource ? { providerSentAtSource } : {}),
...((endpoint.provider === "github" ||
endpoint.provider === "imessage-photon") &&
(githubAttachmentLimitOmissions(message) ||
Math.max(0, message.attachments.length - 20))
? {
attachmentLimitOmissions:
githubAttachmentLimitOmissions(message) ||
Math.max(0, message.attachments.length - 20),
}
: {}),
attachments: message.attachments.slice(0, 20).map((attachment) => ({
name: sanitizeFilename(attachment.name),
mimeType: normalizeContentType(attachment.mimeType),
size: attachment.size ?? null,
recovery: teamsNonPersonal
? (inlineRecovery.get(attachment) ?? null)
: nativeInboundAttachments.includes(attachment)
? endpointRuntime.attachmentRecoveryDescriptor(
attachment,
["telegram", "imessage-photon"].includes(endpoint.provider)
? attachmentSource
: undefined,
)
: null,
})),
},
};
// Teams RSC can deliver messages from every installed channel, and a
// Telegram bot can receive addressed traffic from a provider-available
// group that the operator has not enabled in Paperclip. Until the reach
// gate admits that destination, retain only identifiers required for
// deduplication and operator-visible filtering—not user text, attachment
// metadata, or an external principal profile.
const redactedDestinationNormalized = {
providerEventId,
kind: eventKind,
trigger,
...(runtimeContext
? {
runtimeContext: {
credentialFingerprint: runtimeContext.credentialFingerprint,
generation: runtimeContext.generation,
},
}
: {}),
resource: {
type: providerResourceType(endpoint.provider, surfaceKind),
providerResourceId: canonicalProviderResourceId(
endpoint.provider,
thread,
),
},
conversation: { externalThreadId: thread.id },
message: {
providerMessageId: message.id,
providerSentAt: providerSentAt?.toISOString() ?? null,
},
filtering: { contentRetained: false },
};
return { githubAutomatic, githubIssue, githubManual, providerEventId, surfaceKind, addressed, eventKind, endpointRuntime, nativeInboundAttachments, unavailableTeamsReferences, providerSentAt, providerUrl, normalized, redactedDestinationNormalized };
}
return { prepareInboundEnvelope };
}
export type InboundEnvelope = Awaited<ReturnType<ReturnType<typeof createInboundEnvelope>["prepareInboundEnvelope"]>>;
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,113 @@
import { createChatInboundAuthority } from "./authorization.js";
import type { EndpointRow } from "../runtime-types.js";
import {
and,
eq,
ne,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatEndpointResources,
chatEndpoints,
} from "@paperclipai/db";
import {
notFound,
} from "../../../errors.js";
import type {
Thread,
} from "chat";
import type { ChatSurfaceKind } from "./surfaces.js";
type MessageResourceAdmissionInput = {
endpoint: EndpointRow;
thread: Thread;
addressed: boolean;
suppressSetupDestinationActivation: boolean;
surfaceKind: ChatSurfaceKind;
};
export function createMessageResourceAdmission(db: Db, options: { ensureResource: ReturnType<typeof createChatInboundAuthority>["ensureResource"] }) {
const { ensureResource } = options;
async function admitMessageResource(input: MessageResourceAdmissionInput) {
const { endpoint, thread, addressed, suppressSetupDestinationActivation, surfaceKind } = input;
const mayEnableSetupDestination =
// Slack channels start enabled. Never use first-test activation to
// override an existing channel the operator explicitly disabled.
endpoint.provider !== "slack" &&
endpoint.provider !== "imessage-photon" &&
!thread.isDM &&
endpoint.status === "verifying" &&
addressed &&
!suppressSetupDestinationActivation &&
!(
endpoint.provider === "microsoft-teams" &&
surfaceKind === "linear_group"
);
const resourceAdmission = mayEnableSetupDestination
? await db.transaction(async (tx) => {
const currentEndpoint = await tx
.select({ status: chatEndpoints.status })
.from(chatEndpoints)
.where(eq(chatEndpoints.id, endpoint.id))
.for("update")
.then((rows) => rows[0] ?? null);
if (!currentEndpoint) throw notFound("Chat endpoint not found");
const enabledResourceCount = await tx
.select({ count: sql<number>`count(*)::int` })
.from(chatEndpointResources)
.where(
and(
eq(chatEndpointResources.endpointId, endpoint.id),
eq(chatEndpointResources.enabled, true),
eq(chatEndpointResources.availability, "available"),
ne(chatEndpointResources.type, "direct_message"),
ne(chatEndpointResources.type, "group_chat"),
),
)
.then((rows) => rows[0]?.count ?? 0);
const enableSetupDestination =
currentEndpoint.status === "verifying" &&
enabledResourceCount === 0;
let resource = await ensureResource(
endpoint,
thread,
enableSetupDestination,
tx,
);
if (enableSetupDestination && !resource.enabled) {
resource = await tx
.update(chatEndpointResources)
.set({ enabled: true, updatedAt: new Date() })
.where(eq(chatEndpointResources.id, resource.id))
.returning()
.then((rows) => rows[0] ?? resource);
}
return { enabledResourceCount, resource };
})
: {
enabledResourceCount: await db
.select({ count: sql<number>`count(*)::int` })
.from(chatEndpointResources)
.where(
and(
eq(chatEndpointResources.endpointId, endpoint.id),
eq(chatEndpointResources.enabled, true),
eq(chatEndpointResources.availability, "available"),
ne(chatEndpointResources.type, "direct_message"),
ne(chatEndpointResources.type, "group_chat"),
),
)
.then((rows) => rows[0]?.count ?? 0),
resource: await ensureResource(endpoint, thread, false),
};
return resourceAdmission;
}
return { admitMessageResource };
}
@@ -0,0 +1,180 @@
import { slackThreadChannelId } from "../slack/messages.js";
import type { EndpointRow } from "../runtime-types.js";
import { baseTeamsConversationId } from "../microsoft-teams/resources.js";
import { slackResourceLabelIsFallback } from "../slack/resources.js";
import { telegramResourceLabelIsFallback } from "../telegram/resources.js";
import {
canonicalTeamsThreadId,
teamsConversationId,
} from "../../chat-control-chronology.js";
import {
chatEndpointResources,
} from "@paperclipai/db";
import type {
ChatProvider,
} from "@paperclipai/shared";
import type {
Thread,
} from "chat";
type ResourceRow = typeof chatEndpointResources.$inferSelect;
type ChatSurfaceKind = "direct_message" | "linear_group" | "native_thread";
function providerResourceType(
provider: ChatProvider,
surfaceKind: ChatSurfaceKind,
): string {
if (surfaceKind === "direct_message") return "direct_message";
if (provider === "github") return "repository";
if (provider === "imessage-photon") return "group_chat";
if (provider === "discord") return "channel";
if (provider === "microsoft-teams")
return surfaceKind === "linear_group" ? "group_chat" : "channel";
// Telegram topics are task/thread boundaries inside one group resource. The
// operator grants access to the chat once rather than having to rediscover
// and enable every topic independently.
if (provider === "telegram") return "chat";
return "channel";
}
function sameTeamsThreadIdentity(left: string, right: string): boolean {
const leftConversationId = teamsConversationId(left);
const rightConversationId = teamsConversationId(right);
return (
leftConversationId !== null &&
rightConversationId !== null &&
leftConversationId === rightConversationId
);
}
function durableExternalThreadIdentity(threadId: string): string {
return canonicalTeamsThreadId(threadId) ?? threadId;
}
function teamsThreadRootMessageId(threadId: string): string | null {
const conversationId = teamsConversationId(threadId);
if (!conversationId) return null;
return /;messageid=([^;]+)/i.exec(conversationId)?.[1] ?? null;
}
function canonicalProviderResourceId(
provider: ChatProvider,
thread: Pick<Thread, "channelId" | "id">,
): string {
if (provider === "slack") return thread.channelId.replace(/^slack:/, "");
if (provider === "github")
return thread.channelId.replace(/^github:/, "").toLowerCase();
if (provider === "discord") {
const parts = thread.channelId.split(":");
return parts[2] ?? thread.channelId;
}
if (provider === "microsoft-teams") {
const conversationId =
teamsConversationId(thread.channelId) ?? teamsConversationId(thread.id);
return conversationId
? baseTeamsConversationId(conversationId)
: baseTeamsConversationId(thread.channelId);
}
if (provider === "telegram")
return thread.channelId.replace(/^telegram:/, "");
return thread.channelId;
}
function providerResourceLabelFromThread(
provider: ChatProvider,
thread: Pick<Thread, "channelId" | "id" | "channel">,
): { label: string; fallback: boolean } {
const providerResourceId = canonicalProviderResourceId(provider, thread);
const candidate = thread.channel.name?.trim();
if (!candidate) return { label: providerResourceId, fallback: true };
if (
(provider === "slack" &&
slackResourceLabelIsFallback(providerResourceId, candidate)) ||
(provider === "telegram" &&
telegramResourceLabelIsFallback(providerResourceId, candidate))
) {
return { label: providerResourceId, fallback: true };
}
return { label: candidate, fallback: false };
}
/**
* Native channel threads keep one issue forever. Linear surfaces instead
* advance through Paperclip task generations because their provider id is
* reused after a task completes or the user explicitly starts a new task.
*/
function chatSurfaceKind(
provider: ChatProvider,
thread: Thread,
): ChatSurfaceKind {
if (thread.isDM) return "direct_message";
if (provider === "imessage-photon") return "linear_group";
if (provider === "telegram") {
return /^telegram:[^:]+:[^:]+$/.test(thread.id)
? "native_thread"
: "linear_group";
}
if (provider === "microsoft-teams") {
const conversationId = teamsConversationId(thread.id);
return conversationId?.includes(";messageid=")
? "native_thread"
: "linear_group";
}
return "native_thread";
}
/**
* Teams group chats are admitted as a class: installation at the provider and
* the endpoint's group-chat toggle are the two gates the operator can
* actually control. Teams channels, and every other non-DM destination,
* retain the explicit per-resource enablement gate shown in Settings.
*/
function nonDirectDestinationAllowed(
endpoint: EndpointRow,
resource: ResourceRow | null | undefined,
): boolean {
if (!resource || resource.availability !== "available") return false;
if (
endpoint.provider === "microsoft-teams" &&
resource.type === "group_chat"
) {
return endpoint.allowGroupChats;
}
return resource.enabled;
}
function linearControlCommand(text: string): "new" | "close" | "status" | null {
const match = /^\/(new|close|status)(?:@[\w.-]+)?\s*$/i.exec(text.trim());
return (
(match?.[1]?.toLowerCase() as "new" | "close" | "status" | undefined) ??
null
);
}
function actionThreadMatchesConversation(
provider: ChatProvider,
actionThreadId: string,
conversationThreadId: string,
): boolean {
if (provider === "microsoft-teams") {
return sameTeamsThreadIdentity(actionThreadId, conversationThreadId);
}
if (provider !== "slack") return actionThreadId === conversationThreadId;
const actionChannelId = slackThreadChannelId(actionThreadId);
return (
actionChannelId !== null &&
actionChannelId === slackThreadChannelId(conversationThreadId)
);
}
export { providerResourceType, sameTeamsThreadIdentity, durableExternalThreadIdentity, teamsThreadRootMessageId, canonicalProviderResourceId, providerResourceLabelFromThread, chatSurfaceKind, nonDirectDestinationAllowed, linearControlCommand, actionThreadMatchesConversation };
export type { ChatSurfaceKind };
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,432 @@
import {
and,
asc,
desc,
eq,
isNotNull,
isNull,
lte,
notInArray,
or,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
agentWakeupRequests,
chatActions,
chatEndpoints,
chatMessageLinks,
issues,
} from "@paperclipai/db";
import { isUniqueViolation } from "../../../db-errors.js";
import { logger } from "../../../middleware/logger.js";
import {
type IssueAssignmentWakeupDeps,
} from "../../issue-assignment-wakeup.js";
import { issueThreadInteractionService } from "../../issue-thread-interactions.js";
type InteractionContinuationsOptions = {
heartbeat: IssueAssignmentWakeupDeps;
redactError: (error: unknown) => string;
CONFIRMATION_WAKEUP_RETRY_BACKOFF_MS: number;
};
export function createChatInteractionContinuations(db: Db, options: InteractionContinuationsOptions) {
const { redactError, CONFIRMATION_WAKEUP_RETRY_BACKOFF_MS } = options;
async function processPendingInteractionWakeups(
limit = 100,
): Promise<number> {
const staleBefore = new Date(Date.now() - 30_000);
const retryBefore = new Date(
Date.now() - CONFIRMATION_WAKEUP_RETRY_BACKOFF_MS,
);
const candidates = await db
.select()
.from(chatActions)
.where(
and(
eq(chatActions.kind, "interaction_wakeup"),
or(
and(
eq(chatActions.status, "issued"),
or(
isNull(chatActions.result),
lte(chatActions.updatedAt, retryBefore),
),
),
and(
eq(chatActions.status, "processing"),
lte(chatActions.updatedAt, staleBefore),
),
),
),
)
.orderBy(asc(chatActions.updatedAt), asc(chatActions.id))
.limit(limit);
let processed = 0;
for (const candidate of candidates) {
const [claimed] = await db
.update(chatActions)
.set({ status: "processing", updatedAt: new Date() })
.where(
and(
eq(chatActions.id, candidate.id),
or(
and(
eq(chatActions.status, "issued"),
or(
isNull(chatActions.result),
lte(chatActions.updatedAt, retryBefore),
),
),
and(
eq(chatActions.status, "processing"),
lte(chatActions.updatedAt, staleBefore),
),
),
),
)
.returning();
if (!claimed) continue;
const payload = claimed.payload;
const requestedByActorType =
payload.requestedByActorType === "user" ||
payload.requestedByActorType === "agent" ||
payload.requestedByActorType === "system"
? payload.requestedByActorType
: typeof payload.requestedByUserId === "string"
? "user"
: null;
const requestedByActorId =
typeof payload.requestedByActorId === "string"
? payload.requestedByActorId
: typeof payload.requestedByUserId === "string"
? payload.requestedByUserId
: null;
if (
payload.version !== 1 ||
typeof payload.interactionId !== "string" ||
typeof payload.interactionKind !== "string" ||
!["accepted", "rejected", "answered", "cancelled", "failed"].includes(
String(payload.interactionStatus),
) ||
typeof payload.issueId !== "string" ||
typeof payload.agentId !== "string" ||
!requestedByActorType ||
!requestedByActorId
) {
await db
.update(chatActions)
.set({
status: "failed",
result: { code: "interaction_wakeup_payload_invalid" },
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, claimed.id),
eq(chatActions.status, "processing"),
),
);
continue;
}
const currentIssue = await db
.select({
assigneeAgentId: issues.assigneeAgentId,
status: issues.status,
})
.from(issues)
.where(
and(
eq(issues.companyId, claimed.companyId),
eq(issues.id, payload.issueId),
),
)
.then((rows) => rows[0] ?? null);
const currentInteraction = currentIssue
? (
await issueThreadInteractionService(db).listForIssue(
payload.issueId,
)
).find((interaction) => interaction.id === payload.interactionId)
: null;
if (
!currentIssue ||
!currentInteraction ||
currentInteraction.companyId !== claimed.companyId ||
currentInteraction.kind !== payload.interactionKind ||
currentInteraction.status !== payload.interactionStatus ||
currentIssue.assigneeAgentId !== payload.agentId ||
currentIssue.status === "done" ||
currentIssue.status === "cancelled"
) {
await db
.update(chatActions)
.set({
status: "processed",
result: { code: "interaction_wakeup_no_longer_applicable" },
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, claimed.id),
eq(chatActions.status, "processing"),
),
);
processed += 1;
continue;
}
const idempotencyKey = `interaction:${payload.interactionId}:${payload.interactionStatus}`;
try {
const sourceCommentId = currentInteraction.sourceCommentId ?? null;
const sourceRunId = currentInteraction.sourceRunId ?? null;
// A resolved interaction is still part of the originating external
// turn. Preserve one verified inbound comment edge on the continuation
// run so queued/working/failure milestones return to this exact
// endpoint and conversation instead of disappearing as an internal
// automation run.
const chatOrigin = claimed.conversationId
? await db
.select({
commentId: chatMessageLinks.commentId,
provider: chatEndpoints.provider,
})
.from(chatEndpoints)
.innerJoin(
chatMessageLinks,
and(
eq(chatMessageLinks.companyId, chatEndpoints.companyId),
eq(chatMessageLinks.endpointId, chatEndpoints.id),
),
)
.where(
and(
eq(chatEndpoints.companyId, claimed.companyId),
eq(chatEndpoints.id, claimed.endpointId),
eq(chatMessageLinks.conversationId, claimed.conversationId),
eq(chatMessageLinks.direction, "inbound"),
isNotNull(chatMessageLinks.commentId),
),
)
.orderBy(
desc(chatMessageLinks.createdAt),
desc(chatMessageLinks.id),
)
.limit(1)
.then((rows) => rows[0] ?? null)
: null;
const wakeCommentId = sourceCommentId ?? chatOrigin?.commentId ?? null;
const existing = await db
.select({
agentId: agentWakeupRequests.agentId,
id: agentWakeupRequests.id,
})
.from(agentWakeupRequests)
.where(
and(
eq(agentWakeupRequests.companyId, claimed.companyId),
eq(agentWakeupRequests.idempotencyKey, idempotencyKey),
notInArray(agentWakeupRequests.status, [
"skipped",
"failed",
"cancelled",
]),
),
)
.limit(1)
.then((rows) => rows[0] ?? null);
const queuedRun = !existing
? await options.heartbeat.wakeup(payload.agentId, {
source: "automation",
triggerDetail: "system",
reason: "issue_commented",
payload: {
issueId: payload.issueId,
interactionId: payload.interactionId,
interactionKind: payload.interactionKind,
interactionStatus: payload.interactionStatus,
sourceCommentId,
sourceRunId,
...(wakeCommentId
? {
wakeCommentId,
wakeCommentIds: [wakeCommentId],
}
: {}),
...(payload.planReviewInteraction &&
typeof payload.planReviewInteraction === "object"
? { planReviewInteraction: payload.planReviewInteraction }
: {}),
mutation: "interaction",
},
idempotencyKey,
allowRunCoalescing: false,
requestedByActorType,
requestedByActorId,
contextSnapshot: {
issueId: payload.issueId,
taskId: payload.issueId,
interactionId: payload.interactionId,
interactionKind: payload.interactionKind,
interactionStatus: payload.interactionStatus,
sourceCommentId,
sourceRunId,
...(payload.planReviewInteraction &&
typeof payload.planReviewInteraction === "object"
? { planReviewInteraction: payload.planReviewInteraction }
: {}),
wakeReason: "issue_commented",
source: chatOrigin
? `chat:${chatOrigin.provider}`
: "external_chat.interaction.resolve",
...(wakeCommentId
? {
wakeCommentId,
wakeCommentIds: [wakeCommentId],
}
: {}),
...(payload.forceFreshSession === true
? { forceFreshSession: true }
: {}),
...(typeof payload.workspaceRefreshReason === "string"
? {
workspaceRefreshReason: payload.workspaceRefreshReason,
}
: {}),
},
})
: null;
const durable =
existing ??
(await db
.select({
agentId: agentWakeupRequests.agentId,
id: agentWakeupRequests.id,
})
.from(agentWakeupRequests)
.where(
and(
eq(agentWakeupRequests.companyId, claimed.companyId),
eq(agentWakeupRequests.idempotencyKey, idempotencyKey),
notInArray(agentWakeupRequests.status, [
"skipped",
"failed",
"cancelled",
]),
),
)
.limit(1)
.then((rows) => rows[0] ?? null));
if (!queuedRun && !durable) {
await db
.update(chatActions)
.set({
status: "issued",
result: { code: "interaction_wakeup_deferred" },
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, claimed.id),
eq(chatActions.status, "processing"),
),
);
continue;
}
await db
.update(chatActions)
.set({
status: "processed",
result: {
code: durable
? durable.agentId === payload.agentId
? "interaction_wakeup_already_durable"
: "interaction_wakeup_coalesced_after_reassignment"
: "interaction_wakeup_queued",
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, claimed.id),
eq(chatActions.status, "processing"),
),
);
processed += 1;
} catch (error) {
const durable = isUniqueViolation(error)
? await db
.select({
agentId: agentWakeupRequests.agentId,
id: agentWakeupRequests.id,
})
.from(agentWakeupRequests)
.where(
and(
eq(agentWakeupRequests.companyId, claimed.companyId),
eq(agentWakeupRequests.idempotencyKey, idempotencyKey),
notInArray(agentWakeupRequests.status, [
"skipped",
"failed",
"cancelled",
]),
),
)
.limit(1)
.then((rows) => rows[0] ?? null)
: null;
const previousResult =
claimed.result && typeof claimed.result === "object"
? (claimed.result as Record<string, unknown>)
: {};
const previousAttempts =
typeof previousResult.attemptCount === "number" &&
Number.isInteger(previousResult.attemptCount) &&
previousResult.attemptCount >= 0
? previousResult.attemptCount
: 0;
const attemptCount = previousAttempts + 1;
await db
.update(chatActions)
.set({
status: durable ? "processed" : "issued",
result: {
code: durable
? durable.agentId === payload.agentId
? "interaction_wakeup_already_durable"
: "interaction_wakeup_coalesced_after_reassignment"
: "interaction_wakeup_failed",
...(!durable ? { attemptCount } : {}),
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, claimed.id),
eq(chatActions.status, "processing"),
),
);
if (!durable) {
logger.warn(
{
endpointId: claimed.endpointId,
interactionId: payload.interactionId,
attemptCount,
error: redactError(error),
},
"failed to queue external interaction continuation wake",
);
}
}
}
return processed;
}
return { processPendingInteractionWakeups };
}
@@ -0,0 +1,68 @@
import type { Message, Thread } from "chat";
import { and, eq, sql } from "drizzle-orm";
import { chatEndpointResources } from "@paperclipai/db";
import type { EndpointRow, DbTransaction, DbOrTransaction } from "../runtime-types.js";
import { nonDirectDestinationAllowed, teamsThreadRootMessageId } from "../inbound/surfaces.js";
type DestinationAdmissionInput = {
endpoint: EndpointRow;
currentEndpoint: Pick<EndpointRow, "status" | "allowGroupChats">;
thread: Thread;
message: Message;
addressed: boolean;
};
type DestinationAdmissionOptions = {
ensureResource: (endpoint: EndpointRow, thread: Thread, enableForSetup: boolean, database?: DbOrTransaction) => Promise<typeof chatEndpointResources.$inferSelect>;
};
export function createTeamsDestinationAdmission(options: DestinationAdmissionOptions) {
const { ensureResource } = options;
async function admitTeamsDestination(input: DestinationAdmissionInput, tx: DbTransaction) {
const { endpoint, currentEndpoint, thread, message, addressed } = input;
let destinationAccepting = true;
let provisionalTeamsSetupReply = false;
let resource = await ensureResource(endpoint, thread, false, tx);
const enabledChannelCount = await tx
.select({ count: sql<number>`count(*)::int` })
.from(chatEndpointResources)
.where(
and(
eq(chatEndpointResources.endpointId, endpoint.id),
eq(chatEndpointResources.enabled, true),
eq(chatEndpointResources.availability, "available"),
eq(chatEndpointResources.type, "channel"),
),
)
.then((rows) => rows[0]?.count ?? 0);
const canActivateFirstSetupChannel =
currentEndpoint.status === "verifying" &&
resource.type === "channel" &&
resource.availability === "available" &&
enabledChannelCount === 0 &&
addressed;
if (canActivateFirstSetupChannel && !resource.enabled) {
resource = await tx
.update(chatEndpointResources)
.set({ enabled: true, updatedAt: new Date() })
.where(eq(chatEndpointResources.id, resource.id))
.returning()
.then((rows) => rows[0] ?? resource);
}
const teamsRootMessageId = teamsThreadRootMessageId(thread.id);
provisionalTeamsSetupReply =
currentEndpoint.status === "verifying" &&
resource.type === "channel" &&
resource.availability === "available" &&
enabledChannelCount === 0 &&
!addressed &&
Boolean(teamsRootMessageId) &&
teamsRootMessageId !== message.id;
destinationAccepting =
nonDirectDestinationAllowed(
{ ...endpoint, allowGroupChats: currentEndpoint.allowGroupChats },
resource,
) || provisionalTeamsSetupReply;
return { destinationAccepting, provisionalTeamsSetupReply };
}
return { admitTeamsDestination };
}
@@ -0,0 +1,55 @@
import type { EndpointRow, DbTransaction, LifecycleRuntimeFence, InboundRuntimeContext } from "../runtime-types.js";
import { type ChatRuntime } from "../../chat-runtime.js";
import type { Db } from "@paperclipai/db";
import {
type ChatSdkRuntime,
} from "../../chat-sdk-runtime.js";
type TeamsRoutesOptions = {
runtime: Pick<ChatRuntime | ChatSdkRuntime, "get">;
runtimeCallbackEndpoint: (tx: DbTransaction, endpointId: string, context: LifecycleRuntimeFence, statuses: EndpointRow["status"][]) => Promise<EndpointRow | null>;
};
function microsoftTeamsServiceUrl(raw: unknown): string | null {
if (!raw || typeof raw !== "object" || Array.isArray(raw)) return null;
const value = (raw as { serviceUrl?: unknown }).serviceUrl;
return typeof value === "string" && value ? value : null;
}
export function createTeamsRoutes(db: Db, options: TeamsRoutesOptions) {
const { runtime, runtimeCallbackEndpoint } = options;
async function recordCurrentMicrosoftTeamsRoute(
endpoint: EndpointRow,
context: InboundRuntimeContext | undefined,
threadId: string,
raw: unknown,
recordAcceptedActivity = true,
): Promise<void> {
if (endpoint.provider !== "microsoft-teams" || !context?.endpointRuntime) {
return;
}
const serviceUrl = microsoftTeamsServiceUrl(raw);
if (!serviceUrl) return;
await db.transaction(async (tx) => {
const current = await runtimeCallbackEndpoint(tx, endpoint.id, context, [
"verifying",
"active",
]);
if (!current || runtime.get(endpoint.id) !== context.endpointRuntime) {
return;
}
await context.endpointRuntime.recordMicrosoftTeamsRoute(
threadId,
serviceUrl,
recordAcceptedActivity ? raw : undefined,
);
});
}
return { recordCurrentMicrosoftTeamsRoute };
}
@@ -0,0 +1,108 @@
import type {
ChatAdapterCapabilities,
ChatProvider,
} from "@paperclipai/shared";
const PROVIDER_LABELS: Record<ChatProvider, string> = {
speko: "Speko",
"imessage-photon": "iMessage Photon",
agentmail: "AgentMail",
slack: "Slack",
github: "GitHub",
discord: "Discord",
"microsoft-teams": "Microsoft Teams",
telegram: "Telegram",
};
const CAPABILITIES: Record<ChatProvider, ChatAdapterCapabilities> = {
speko: { threads: false, directMessages: true, nativeStreaming: false, messageEdits: false, messageDeletes: false, reactions: false, files: false, cards: false, actions: false, modals: false, slashCommands: false, ephemeralMessages: false, proactiveDirectMessages: false },
"imessage-photon": { threads: false, directMessages: true, nativeStreaming: false, messageEdits: true, messageDeletes: false, reactions: false, files: true, cards: false, actions: true, modals: false, slashCommands: false, ephemeralMessages: false, proactiveDirectMessages: false },
agentmail: { threads: true, directMessages: true, nativeStreaming: false, messageEdits: false, messageDeletes: false, reactions: false, files: true, cards: false, actions: false, modals: false, slashCommands: false, ephemeralMessages: false, proactiveDirectMessages: true },
slack: {
threads: true,
directMessages: true,
nativeStreaming: true,
messageEdits: true,
messageDeletes: true,
reactions: true,
files: true,
cards: true,
actions: true,
modals: true,
slashCommands: true,
ephemeralMessages: true,
proactiveDirectMessages: false,
},
github: {
threads: true,
directMessages: false,
nativeStreaming: false,
messageEdits: true,
messageDeletes: true,
reactions: true,
files: false,
cards: true,
actions: false,
modals: false,
slashCommands: false,
ephemeralMessages: false,
proactiveDirectMessages: false,
},
discord: {
threads: true,
directMessages: true,
nativeStreaming: false,
messageEdits: true,
messageDeletes: true,
reactions: true,
files: true,
cards: true,
actions: true,
modals: true,
// Automatic registration upgrades these only after a durable provider
// receipt. A draft or an uncertain/conflicting registration stays false.
slashCommands: false,
ephemeralMessages: false,
proactiveDirectMessages: false,
},
"microsoft-teams": {
threads: true,
directMessages: true,
// Production webhook processing is deferred into Paperclip's durable
// ingress queue. The Teams adapter's request-scoped DM streamer is gone by
// the time agent output is published, so advertise the durable behavior we
// can actually provide. editMessage still lets one run coalesce its
// queued, working, and final states in place.
nativeStreaming: false,
// Paperclip supplements the pinned adapter's missing inbound Bot Framework
// messageUpdate/messageDelete dispatch after provider authentication.
messageEdits: true,
messageDeletes: true,
reactions: true,
files: true,
cards: true,
actions: true,
modals: true,
slashCommands: false,
ephemeralMessages: true,
proactiveDirectMessages: false,
},
telegram: {
threads: true,
directMessages: true,
nativeStreaming: true,
messageEdits: true,
// Telegram's Bot API does not emit an update when a user deletes a
// message, so this capability cannot be offered truthfully.
messageDeletes: false,
reactions: true,
files: true,
cards: true,
actions: true,
modals: false,
slashCommands: false,
ephemeralMessages: false,
proactiveDirectMessages: false,
},
};
export { PROVIDER_LABELS, CAPABILITIES };
@@ -0,0 +1,344 @@
import { createGitHubPublicationPolicy } from "../github/publication-policy.js";
import type { EndpointRow, DbTransaction, DbOrTransaction, LifecycleRuntimeFence } from "../runtime-types.js";
import { createChatFailedRunRetry } from "../failed-run-retry.js";
import { notifyChatPublicationWork } from "../../chat-work-notifications.js";
import { type CredentialMutationLeaseGuard } from "../../chat-credential-mutation-lease.js";
import { authorizeSlackBoardPublication } from "../../slack-board-authority.js";
import { slackPublicationAllowed } from "../../connectors/slack-access.js";
import {
and,
eq,
gt,
inArray,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatActions,
chatConversations,
chatEndpointResources,
chatPublications,
heartbeatRuns,
issueComments,
} from "@paperclipai/db";
import {
authorizeNativeChatReviewPresentation,
NativeChatReviewPresentationContentionError,
} from "../../native-runtime/native-chat-review-presentation.js";
import { isExternalChatWaitAuthorizationContention } from "../../native-runtime/chat-attachment-reuse.js";
import { isExplicitOperatorPublication } from "./policy.js";
import { attachmentFailureNoticeFence } from "./policy.js";
import type { createChatPublicationTaskControl } from "./task-control.js";
type ConversationRow = typeof chatConversations.$inferSelect;
type PublicationAuthorizationOptions = {
authorizeInboundWakePublication: (tx: DbOrTransaction, publication: typeof chatPublications.$inferSelect) => Promise<boolean>;
authorizeRetainedChatSourcePublication: (tx: DbOrTransaction, publication: typeof chatPublications.$inferSelect) => Promise<boolean>;
fetchImpl: typeof globalThis.fetch;
githubCurrentOmissionPreparedText: ReturnType<typeof createGitHubPublicationPolicy>["githubCurrentOmissionPreparedText"];
hasCommittedTaskControlCompletion: ReturnType<typeof createChatPublicationTaskControl>["hasCommittedTaskControlCompletion"];
lockCurrentPrincipalAuthorization: (tx: DbOrTransaction, endpoint: EndpointRow, principalId: string) => Promise<{ allowed: boolean; userId?: string | null }>;
nonDirectDestinationAllowed: (endpoint: EndpointRow, resource: typeof chatEndpointResources.$inferSelect | null | undefined) => boolean;
resolveCredentials: (endpoint: EndpointRow, tx?: DbOrTransaction) => Promise<Record<string, string>>;
runtimeCallbackEndpoint: (tx: DbTransaction, endpointId: string, context: LifecycleRuntimeFence, statuses: EndpointRow["status"][]) => Promise<EndpointRow | null>;
FailedChatRetryPublicationReadError: ReturnType<typeof createChatFailedRunRetry>["FailedChatRetryPublicationReadError"];
};
export function createChatPublicationAuthorization(db: Db, options: PublicationAuthorizationOptions) {
const { authorizeInboundWakePublication, authorizeRetainedChatSourcePublication, fetchImpl, githubCurrentOmissionPreparedText, hasCommittedTaskControlCompletion, lockCurrentPrincipalAuthorization, nonDirectDestinationAllowed, resolveCredentials, runtimeCallbackEndpoint, FailedChatRetryPublicationReadError } = options;
async function claimPublicationTransportAuthorization(input: {
credentialLease: CredentialMutationLeaseGuard;
endpoint: EndpointRow;
publication: typeof chatPublications.$inferSelect;
runtimeContext: LifecycleRuntimeFence;
}): Promise<{
authorizationActionId: string | null;
conversation: ConversationRow;
endpoint: EndpointRow;
preparedText: string | null;
} | null> {
return db.transaction(async (tx) => {
await input.credentialLease.assertOwned(tx);
const currentPublication = await tx
.select({ id: chatPublications.id })
.from(chatPublications)
.where(
and(
eq(chatPublications.id, input.publication.id),
eq(chatPublications.state, "streaming"),
eq(chatPublications.attempts, input.publication.attempts + 1),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
if (!currentPublication) return null;
try {
if (
!(await authorizeRetainedChatSourcePublication(tx, input.publication))
)
return null;
} catch (error) {
if (error instanceof NativeChatReviewPresentationContentionError)
throw error;
throw new FailedChatRetryPublicationReadError();
}
const attachmentFailureFence = attachmentFailureNoticeFence(
input.publication.idempotencyKey,
);
if (
attachmentFailureFence &&
(attachmentFailureFence.generation !==
input.runtimeContext.generation ||
attachmentFailureFence.credentialFingerprint !==
input.runtimeContext.credentialFingerprint)
) {
return null;
}
const endpoint = await runtimeCallbackEndpoint(
tx,
input.publication.endpointId,
input.runtimeContext,
["verifying", "active"],
);
const conversation = endpoint
? await tx
.select()
.from(chatConversations)
.where(
and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
eq(chatConversations.id, input.publication.conversationId),
inArray(chatConversations.state, [
"active",
"waiting",
"completed",
]),
),
)
.for("update")
.then((rows) => rows[0] ?? null)
: null;
const resource =
endpoint && conversation?.resourceId && !conversation.isDirectMessage
? await tx
.select()
.from(chatEndpointResources)
.where(
and(
eq(chatEndpointResources.companyId, endpoint.companyId),
eq(chatEndpointResources.endpointId, endpoint.id),
eq(chatEndpointResources.id, conversation.resourceId),
),
)
.for("update")
.then((rows) => rows[0] ?? null)
: null;
const attachmentFailureNoticeSuperseded =
attachmentFailureFence && conversation
? await tx
.select({ id: chatConversations.id })
.from(chatConversations)
.where(
and(
eq(chatConversations.companyId, conversation.companyId),
eq(chatConversations.endpointId, conversation.endpointId),
eq(
chatConversations.externalThreadId,
conversation.externalThreadId,
),
gt(
chatConversations.sessionGeneration,
conversation.sessionGeneration,
),
),
)
.limit(1)
.then((rows) => rows.length > 0)
: false;
const isTaskControl =
input.publication.idempotencyKey.startsWith("control:");
const providerVisibleCompletion =
conversation?.state === "completed" &&
!isExplicitOperatorPublication(input.publication) &&
(await hasCommittedTaskControlCompletion(conversation.id, tx));
const authorizationAction =
endpoint && isTaskControl
? await tx
.select()
.from(chatActions)
.where(
and(
eq(chatActions.companyId, endpoint.companyId),
eq(chatActions.endpointId, endpoint.id),
eq(chatActions.conversationId, conversation?.id ?? ""),
eq(chatActions.kind, "task_control_authorization"),
eq(
chatActions.providerActionId,
`task-control-authorization:${input.publication.id}`,
),
eq(chatActions.status, "issued"),
),
)
.for("update")
.then((rows) => rows[0] ?? null)
: null;
const destinationAllowed =
endpoint !== null &&
conversation !== null &&
(endpoint.provider !== "slack" || await slackPublicationAllowed(tx, endpoint.companyId, endpoint.id, input.publication.issueId, conversation.externalConversationId, null)) &&
(conversation.isDirectMessage
? endpoint.allowDirectMessages
: nonDirectDestinationAllowed(endpoint, resource));
if (
!endpoint ||
!conversation ||
(attachmentFailureFence &&
((conversation.state !== "active" &&
conversation.state !== "waiting") ||
attachmentFailureNoticeSuperseded)) ||
providerVisibleCompletion ||
(isTaskControl && !authorizationAction?.principalId) ||
!destinationAllowed
) {
return null;
}
let authorizationActionId: string | null = null;
if (endpoint.provider === "slack") {
const credentials = await resolveCredentials(endpoint, tx);
if (!await authorizeSlackBoardPublication(tx, endpoint, conversation, input.publication, credentials.botToken ?? "", fetchImpl)) return null;
}
if (
input.publication.idempotencyKey.startsWith("wake:") &&
!(await authorizeInboundWakePublication(tx, input.publication))
)
return null;
// A native response alongside an older completion review has its own
// narrow authorization. Recheck that exact proof, review and original
// requester at the transport claim, including file publications whose
// comment is the run's earlier internal attachment-selection comment.
if (
input.publication.commentId &&
!isExplicitOperatorPublication(input.publication)
) {
const [origin] = await tx
.select({
runId: heartbeatRuns.id,
resultJson: heartbeatRuns.resultJson,
})
.from(issueComments)
.innerJoin(
heartbeatRuns,
and(
eq(heartbeatRuns.id, issueComments.createdByRunId),
eq(heartbeatRuns.companyId, issueComments.companyId),
),
)
.where(
and(
eq(issueComments.id, input.publication.commentId),
eq(issueComments.companyId, endpoint.companyId),
eq(issueComments.issueId, input.publication.issueId),
),
)
.limit(1);
if (
origin?.resultJson?.finalizationReasonCode ===
"governed_response_waiting"
) {
try {
if (
!(await authorizeNativeChatReviewPresentation(
tx as unknown as Db,
{
companyId: endpoint.companyId,
issueId: input.publication.issueId,
runId: origin.runId,
resultJson: origin.resultJson,
destination: {
endpointId: endpoint.id,
conversationId: conversation.id,
},
},
"nonblocking",
))
)
return null;
} catch (error) {
if (isExternalChatWaitAuthorizationContention(error))
throw new NativeChatReviewPresentationContentionError();
throw error;
}
}
}
const preparedText = await githubCurrentOmissionPreparedText(
tx,
input.publication,
endpoint,
);
if (preparedText === false) return null;
if (authorizationAction?.principalId) {
const authorization = await lockCurrentPrincipalAuthorization(
tx,
endpoint,
authorizationAction.principalId,
);
if (!authorization.allowed) return null;
const [claimedAuthorization] = await tx
.update(chatActions)
.set({
status: "processing",
result: {
attempts: input.publication.attempts + 1,
authorizedUserId: authorization.userId,
credentialFingerprint: input.runtimeContext.credentialFingerprint,
runtimeGeneration: input.runtimeContext.generation,
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, authorizationAction.id),
eq(chatActions.status, "issued"),
),
)
.returning({ id: chatActions.id });
if (!claimedAuthorization) return null;
authorizationActionId = claimedAuthorization.id;
}
if (preparedText !== null) {
// Persist the exact provider text before I/O. A retry must reuse this
// body, not accumulate links or regenerate it from changed settings.
await notifyChatPublicationWork(tx);
await tx
.update(chatPublications)
.set({
payload: { ...input.publication.payload, text: preparedText },
})
.where(eq(chatPublications.id, input.publication.id));
}
await input.credentialLease.assertOwned(tx);
// The publication's streaming row plus the optional authorization action
// are the durable transport claim. Their database locks are released
// before provider I/O. The renewable credential lease intentionally
// remains held so reconnect and reach mutations linearize after this
// committed authorization snapshot and transport attempt.
return {
authorizationActionId,
conversation,
endpoint,
preparedText,
};
});
}
return { claimPublicationTransportAuthorization };
}
@@ -0,0 +1,328 @@
import { isOutboundAttachmentHydrationError, isOutboundAttachmentValidationError } from "../outbound-attachments.js";
import type { EndpointRow, DbTransaction, LifecycleRuntimeFence } from "../runtime-types.js";
import { createChatFailedRunRetry } from "../failed-run-retry.js";
import { notifyChatPublicationWork } from "../../chat-work-notifications.js";
import { type CredentialMutationLeaseGuard } from "../../chat-credential-mutation-lease.js";
import {
and,
eq,
inArray,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatActions,
chatConversations,
chatEndpointResources,
chatEndpoints,
chatPublications,
toolConnections,
} from "@paperclipai/db";
import { logger } from "../../../middleware/logger.js";
import { redactSensitiveText } from "../../../redaction.js";
import {
NativeChatReviewPresentationContentionError,
} from "../../native-runtime/native-chat-review-presentation.js";
import { projectSafeChatPublication } from "../../chat-publication-projection.js";
import { safeChatTaskUrl } from "../../chat-task-url.js";
import { classifyChatPublicationError } from "../../chat-publication-errors.js";
import { ATTACHMENT_FAILURE_NOTICE_PREFIX, definiteAttachmentProviderRejection, attachmentFailureKind } from "./policy.js";
type PublicationFailuresOptions = {
FailedChatRetryPublicationReadError: ReturnType<typeof createChatFailedRunRetry>["FailedChatRetryPublicationReadError"];
MAX_ERROR_TEXT: number;
getTaskBaseUrl: () => string | null | undefined;
invalidateRuntime: (endpointId: string) => Promise<boolean>;
runtimeCallbackEndpoint: (tx: DbTransaction, endpointId: string, context: LifecycleRuntimeFence, statuses: EndpointRow["status"][]) => Promise<EndpointRow | null>;
};
export function createChatPublicationFailures(db: Db, options: PublicationFailuresOptions) {
const { FailedChatRetryPublicationReadError, MAX_ERROR_TEXT, getTaskBaseUrl, invalidateRuntime, runtimeCallbackEndpoint } = options;
async function finalizePublicationFailure(
publication: typeof chatPublications.$inferSelect,
error: unknown,
providerAccepted = false,
runtimeContext?: LifecycleRuntimeFence | null,
credentialLease?: CredentialMutationLeaseGuard,
): Promise<boolean> {
const attempts = publication.attempts + 1;
const disposition = providerAccepted
? {
kind: "delivery_unknown" as const,
reason:
"Provider accepted the publication, but Paperclip could not confirm its durable result",
}
: error instanceof NativeChatReviewPresentationContentionError
? { kind: "retry" as const, retryAfterMs: 250, reason: error.message }
: error instanceof FailedChatRetryPublicationReadError
? {
kind: "retry" as const,
retryAfterMs: 1_000,
reason: error.message,
}
: isOutboundAttachmentValidationError(error)
? {
kind: "failed" as const,
reason: error.message,
}
: isOutboundAttachmentHydrationError(error)
? {
kind: "retry" as const,
retryAfterMs: Math.min(
60_000,
2 ** Math.max(0, attempts) * 1_000,
),
reason: error.message,
}
: classifyChatPublicationError(error, attempts);
const failure = redactSensitiveText(disposition.reason).slice(
0,
MAX_ERROR_TEXT,
);
// Provider-directed rate limits are not delivery failures. Keep the
// durable publication retryable for the provider's full interval instead
// of exhausting the five-attempt transport budget while flood control is
// still active.
const terminalRetry =
disposition.kind === "retry" &&
disposition.providerRateLimit !== true &&
attempts >= 5;
const attachmentFailure = attachmentFailureKind(publication.payload);
const attachmentFailureRuntimeContext =
!providerAccepted &&
disposition.kind === "failed" &&
runtimeContext &&
attachmentFailure &&
definiteAttachmentProviderRejection(error)
? runtimeContext
: null;
let invalidateCurrentRuntime = false;
const finalized = await db.transaction(async (tx) => {
await credentialLease?.assertOwned(tx);
const state =
disposition.kind === "retry" && !terminalRetry
? "retry"
: disposition.kind === "delivery_unknown"
? "delivery_unknown"
: disposition.kind === "resource_unavailable"
? "cancelled"
: "failed";
await notifyChatPublicationWork(tx);
const [ownedPublication] = await tx
.update(chatPublications)
.set({
state,
nextAttemptAt:
disposition.kind === "retry" && !terminalRetry
? new Date(Date.now() + disposition.retryAfterMs)
: null,
redactedError: failure,
updatedAt: new Date(),
})
.where(
and(
eq(chatPublications.id, publication.id),
eq(chatPublications.state, "streaming"),
eq(chatPublications.attempts, attempts),
),
)
.returning({ id: chatPublications.id });
if (!ownedPublication) return false;
if (publication.idempotencyKey.startsWith("control:")) {
await tx
.update(chatActions)
.set({
status:
state === "delivery_unknown"
? "delivery_unknown"
: state === "cancelled"
? "cancelled"
: "issued",
result: {
code:
state === "delivery_unknown"
? "task_control_delivery_unknown"
: state === "cancelled"
? "task_control_destination_unavailable"
: "task_control_transport_failed",
attempts,
retryable: state === "retry",
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.endpointId, publication.endpointId),
eq(
chatActions.providerActionId,
`task-control-authorization:${publication.id}`,
),
eq(chatActions.status, "processing"),
sql`(${chatActions.result}->>'attempts')::int = ${attempts}`,
),
);
}
const currentEndpoint =
runtimeContext &&
(disposition.kind === "endpoint_attention" ||
disposition.kind === "resource_unavailable")
? await runtimeCallbackEndpoint(
tx,
publication.endpointId,
runtimeContext,
["verifying", "active"],
)
: null;
if (disposition.kind === "endpoint_attention" && currentEndpoint) {
await tx
.update(chatEndpoints)
.set({
status: "attention",
healthMessage: "Provider credentials or permissions need attention",
lastError: failure,
updatedAt: new Date(),
})
.where(eq(chatEndpoints.id, currentEndpoint.id));
await tx
.update(toolConnections)
.set({
status: "disabled",
enabled: false,
healthStatus: "degraded",
healthMessage: "Provider credentials or permissions need attention",
lastError: failure,
healthCheckedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(toolConnections.id, currentEndpoint.connectionId));
invalidateCurrentRuntime = true;
} else if (
disposition.kind === "resource_unavailable" &&
currentEndpoint
) {
const binding = await tx
.select({ resourceId: chatConversations.resourceId })
.from(chatConversations)
.where(eq(chatConversations.id, publication.conversationId))
.then((result) => result[0] ?? null);
await tx
.update(chatConversations)
.set({ state: "unavailable", updatedAt: new Date() })
.where(
and(
eq(chatConversations.id, publication.conversationId),
eq(chatConversations.endpointId, currentEndpoint.id),
),
);
if (binding?.resourceId) {
await tx
.update(chatEndpointResources)
.set({ availability: "unavailable", updatedAt: new Date() })
.where(
and(
eq(chatEndpointResources.id, binding.resourceId),
eq(chatEndpointResources.endpointId, currentEndpoint.id),
),
);
}
}
if (attachmentFailureRuntimeContext && attachmentFailure) {
const noticeEndpoint = await runtimeCallbackEndpoint(
tx,
publication.endpointId,
attachmentFailureRuntimeContext,
["verifying", "active"],
);
const noticeConversation = noticeEndpoint
? await tx
.select({
id: chatConversations.id,
state: chatConversations.state,
})
.from(chatConversations)
.where(
and(
eq(chatConversations.companyId, publication.companyId),
eq(chatConversations.endpointId, publication.endpointId),
eq(chatConversations.id, publication.conversationId),
inArray(chatConversations.state, ["active", "waiting"]),
),
)
.for("no key update")
.then((rows) => rows[0] ?? null)
: null;
const providerCanSendTextNotice =
noticeEndpoint &&
["slack", "discord", "telegram"].includes(noticeEndpoint.provider) &&
(attachmentFailure.provider === null ||
noticeEndpoint.provider === attachmentFailure.provider);
if (providerCanSendTextNotice && noticeConversation) {
const taskUrl = safeChatTaskUrl(
getTaskBaseUrl(),
publication.issueId,
);
const noticeText =
attachmentFailure.kind === "generated_response"
? "Paperclip could not send the response attachment. The complete response remains on its Paperclip task for an operator to retry."
: "Paperclip could not send an attachment. The file remains on its Paperclip task for an operator to retry.";
const idempotencyKey = `${ATTACHMENT_FAILURE_NOTICE_PREFIX}${publication.id}:${attachmentFailureRuntimeContext.generation}:${attachmentFailureRuntimeContext.credentialFingerprint}`;
await notifyChatPublicationWork(tx);
await tx
.insert(chatPublications)
.values({
companyId: publication.companyId,
endpointId: publication.endpointId,
conversationId: publication.conversationId,
issueId: publication.issueId,
commentId: null,
idempotencyKey,
payload: projectSafeChatPublication({
classification: "external",
source: "safe_milestone",
text: taskUrl
? `${noticeText} Open task: ${taskUrl}`
: noticeText,
}),
state: "pending",
})
.onConflictDoNothing();
}
}
await credentialLease?.assertOwned(tx);
return true;
});
if (finalized && invalidateCurrentRuntime) {
await credentialLease?.assertOwned();
await invalidateRuntime(publication.endpointId).catch(() => undefined);
}
if (finalized) {
logger.warn(
{
endpointId: publication.endpointId,
publicationId: publication.id,
error: failure,
disposition: disposition.kind,
},
"chat publication failed",
);
}
return finalized;
}
return { finalizePublicationFailure };
}
@@ -0,0 +1,36 @@
import {
and,
eq,
lt,
or,
sql,
} from "drizzle-orm";
import { type AnyPgColumn } from "drizzle-orm/pg-core";
type PublicationOrderTable = {
createdAt: AnyPgColumn;
id: AnyPgColumn;
payload: AnyPgColumn;
};
function publicationTransportOrderKey(table: PublicationOrderTable) {
return sql<string>`coalesce(${table.payload}->'transportPart'->>'orderKey', ${table.id}::text)`;
}
function publicationPrecedes(
earlier: PublicationOrderTable,
later: PublicationOrderTable,
) {
return or(
lt(earlier.createdAt, later.createdAt),
and(
eq(earlier.createdAt, later.createdAt),
lt(
publicationTransportOrderKey(earlier),
publicationTransportOrderKey(later),
),
),
);
}
export { publicationTransportOrderKey, publicationPrecedes };
@@ -0,0 +1,132 @@
import type { LifecycleRuntimeFence } from "../runtime-types.js";
import {
chatPublications,
} from "@paperclipai/db";
import type {
SafeChatPublicationPayload,
} from "@paperclipai/shared";
function isExplicitOperatorPublication(
publication: Pick<typeof chatPublications.$inferSelect, "idempotencyKey">,
): boolean {
return /^(?:explicit:|explicit-board:)/.test(publication.idempotencyKey);
}
const ATTACHMENT_FAILURE_NOTICE_PREFIX = "attachment-failure-notice:";
function attachmentFailureNoticeFence(
idempotencyKey: string,
): LifecycleRuntimeFence | null {
const match =
/^attachment-failure-notice:[0-9a-f-]{36}:(\d+):([0-9a-f]{64})$/i.exec(
idempotencyKey,
);
if (!match) return null;
const generation = Number(match[1]);
if (!Number.isSafeInteger(generation) || generation < 0) return null;
return { generation, credentialFingerprint: match[2]!.toLowerCase() };
}
function definiteAttachmentProviderRejection(error: unknown): boolean {
const pending: Array<{ depth: number; value: unknown }> = [
{ depth: 0, value: error },
];
const seen = new Set<object>();
const statuses: number[] = [];
const names: string[] = [];
const codes: string[] = [];
while (pending.length > 0) {
const current = pending.shift();
if (
!current ||
current.depth > 4 ||
!current.value ||
typeof current.value !== "object" ||
seen.has(current.value)
) {
continue;
}
seen.add(current.value);
const record = current.value as {
cause?: unknown;
code?: unknown;
details?: { code?: unknown; providerStatus?: unknown };
innerHttpError?: { statusCode?: unknown };
name?: unknown;
original?: unknown;
originalError?: unknown;
response?: { status?: unknown };
status?: unknown;
statusCode?: unknown;
};
for (const status of [
record.status,
record.statusCode,
record.response?.status,
record.details?.providerStatus,
record.innerHttpError?.statusCode,
]) {
if (typeof status === "number" && Number.isInteger(status)) {
statuses.push(status);
}
}
if (typeof record.name === "string") names.push(record.name);
for (const code of [record.code, record.details?.code]) {
if (typeof code === "string") codes.push(code);
}
for (const nested of [
record.cause,
record.original,
record.originalError,
]) {
pending.push({ depth: current.depth + 1, value: nested });
}
}
if (
names.some((name) =>
["ValidationError", "NotImplementedError"].includes(name),
) ||
codes.some((code) =>
[
"CHAT_ADAPTER_COMPATIBILITY_ERROR",
"CHAT_PROVIDER_PRETRANSPORT_REJECTED",
"NOT_IMPLEMENTED",
"VALIDATION_ERROR",
].includes(code),
)
) {
return false;
}
// Slack's Web API can reject a file operation with `ok: false` in an HTTP
// 200 response. The SDK's structured platform-error code is still a
// definitive provider rejection; the main disposition classifier has
// already excluded auth, destination, and rate-limit outcomes.
if (codes.includes("slack_webapi_platform_error")) return true;
return statuses.some(
(status) =>
status >= 400 &&
status < 500 &&
![401, 403, 404, 410, 429].includes(status),
);
}
function attachmentFailureKind(
payload: SafeChatPublicationPayload,
):
| { kind: "generated_response"; provider: "discord" | "telegram" }
| { kind: "selected_file"; provider: null }
| null {
if (payload.transportPart?.mode === "discord_markdown_attachment") {
return { kind: "generated_response", provider: "discord" };
}
if (payload.transportPart?.mode === "telegram_markdown_attachment") {
return { kind: "generated_response", provider: "telegram" };
}
if (payload.attachmentIds?.length) {
return { kind: "selected_file", provider: null };
}
return null;
}
export { isExplicitOperatorPublication, ATTACHMENT_FAILURE_NOTICE_PREFIX, attachmentFailureNoticeFence, definiteAttachmentProviderRejection, attachmentFailureKind };
@@ -0,0 +1,150 @@
import { createDiscordPublicationPreparation } from "../discord/publication-preparation.js";
import { createTelegramPublicationPreparation } from "../telegram/publication-preparation.js";
import { createChatPublicationReplacement } from "../publication-replacement.js";
import { notifyChatPublicationWork } from "../../chat-work-notifications.js";
import {
eq,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatEndpoints,
chatPublications,
} from "@paperclipai/db";
import type {
SafeChatPublicationPayload,
} from "@paperclipai/shared";
import {
renderPublicationTransportText,
splitNativePublicationText,
} from "../../chat-publication-text-parts.js";
type PublicationPreparationOptions = {
runPublicationToReplace: ReturnType<typeof createChatPublicationReplacement>["runPublicationToReplace"];
publicationTransportPreparationBarrier?: (input: { publicationId: string }) => Promise<void>;
};
export function createChatPublicationPreparation(db: Db, options: PublicationPreparationOptions) {
const { runPublicationToReplace } = options;
const { prepareDiscordPublication } = createDiscordPublicationPreparation(db, { runPublicationToReplace });
const { prepareTelegramPublication } = createTelegramPublicationPreparation(db, { runPublicationToReplace });
async function ensureDurablePublicationTransport(
publication: typeof chatPublications.$inferSelect,
): Promise<typeof chatPublications.$inferSelect> {
const persisted = publication.payload as SafeChatPublicationPayload;
if (
!persisted ||
typeof persisted !== "object" ||
typeof persisted.text !== "string"
) {
throw Object.assign(
new Error("Durable chat publication payload is malformed"),
{ code: "VALIDATION_ERROR", name: "ValidationError" },
);
}
// Validate closed persisted wrappers even on retries that are already split.
renderPublicationTransportText(persisted);
await options.publicationTransportPreparationBarrier?.({
publicationId: publication.id,
});
const provider = await db
.select({ provider: chatEndpoints.provider })
.from(chatEndpoints)
.where(eq(chatEndpoints.id, publication.endpointId))
.then((rows) => rows[0]?.provider ?? null);
if (
provider === "slack" ||
provider === "github" ||
provider === "microsoft-teams"
) {
if (
persisted.transportPart ||
persisted.card ||
persisted.interactionId ||
persisted.attachmentIds?.length
)
return publication;
const parts = splitNativePublicationText(provider, persisted.text);
if (parts.length === 1) return publication;
return db.transaction(async (tx) => {
const current = await tx
.select()
.from(chatPublications)
.where(eq(chatPublications.id, publication.id))
.for("update")
.then((rows) => rows[0] ?? null);
if (!current) return publication;
const currentPayload = current.payload as SafeChatPublicationPayload;
renderPublicationTransportText(currentPayload);
if (
currentPayload.transportPart ||
currentPayload.card ||
currentPayload.interactionId ||
currentPayload.attachmentIds?.length ||
!["pending", "retry"].includes(current.state)
)
return current;
const currentParts =
currentPayload.text === persisted.text
? parts
: splitNativePublicationText(provider, currentPayload.text);
if (currentParts.length === 1) return current;
const batchId = current.id;
const updatedAt = new Date();
const payloadFor = (index: number): SafeChatPublicationPayload => {
const part = currentParts[index]!;
const payload = {
...currentPayload,
text: part.text,
transportPart: {
batchId,
index,
count: currentParts.length,
mode: "inline" as const,
orderKey: `${batchId}:${String(index).padStart(4, "0")}`,
...(part.prefix ? { prefix: part.prefix } : {}),
...(part.suffix ? { suffix: part.suffix } : {}),
},
};
if (index > 0) delete payload.progressState;
return payload;
};
const rootPayload = payloadFor(0);
await notifyChatPublicationWork(tx);
await tx
.update(chatPublications)
.set({ payload: rootPayload, updatedAt })
.where(eq(chatPublications.id, current.id));
await notifyChatPublicationWork(tx);
await tx.insert(chatPublications).values(
currentParts.slice(1).map((_part, offset) => ({
companyId: current.companyId,
endpointId: current.endpointId,
conversationId: current.conversationId,
issueId: current.issueId,
commentId: current.commentId,
idempotencyKey: `${current.idempotencyKey}:${provider}-part:${offset + 1}`,
payload: payloadFor(offset + 1),
state: "pending" as const,
createdAt: sql`(select ${chatPublications.createdAt} from ${chatPublications} where ${chatPublications.id} = ${current.id})`,
updatedAt,
})),
);
return { ...current, payload: rootPayload, updatedAt };
});
}
if (provider === "discord") {
return prepareDiscordPublication(publication, persisted);
}
if (provider !== "telegram") return publication;
return prepareTelegramPublication(publication, persisted);
}
return { ensureDurablePublicationTransport };
}
@@ -0,0 +1,389 @@
import { notifyChatPublicationWork } from "../../chat-work-notifications.js";
import { unresolvedSlackPublicationCondition } from "../../connectors/slack-publication.js";
import { settleExternalConversation } from "../../slack-conversation-lifecycle.js";
import {
and,
asc,
eq,
gt,
inArray,
isNotNull,
isNull,
lte,
notExists,
notInArray,
or,
sql,
} from "drizzle-orm";
import { alias } from "drizzle-orm/pg-core";
import type { Db } from "@paperclipai/db";
import {
chatActions,
chatEndpointLeases,
chatEndpoints,
chatPublications,
chatTeamsFileTransfers,
} from "@paperclipai/db";
import { logger } from "../../../middleware/logger.js";
import { publicationTransportOrderKey, publicationPrecedes } from "./order.js";
import type { createChatPublicationWorker } from "./worker.js";
type PublicationQueueOptions = {
PUBLICATION_ENDPOINT_CONCURRENCY: number;
enqueueInboundWakeupPublications: () => Promise<unknown>;
processPendingInteractionWakeups: (limit?: number) => Promise<number>;
reconcileTerminalConfirmationActions: (limit?: number) => Promise<number>;
processSelectedPublication: ReturnType<typeof createChatPublicationWorker>["processSelectedPublication"];
publicationEndpointTasks: Map<string, Promise<void>>;
teamsFileRetryAfter: Map<string, number>;
redactError: (error: unknown) => string;
isShuttingDown: () => boolean;
wakePublications: () => void;
};
export function createChatPublicationQueue(db: Db, options: PublicationQueueOptions) {
const { PUBLICATION_ENDPOINT_CONCURRENCY, enqueueInboundWakeupPublications, processPendingInteractionWakeups, reconcileTerminalConfirmationActions, processSelectedPublication, publicationEndpointTasks, teamsFileRetryAfter, redactError, wakePublications, isShuttingDown } = options;
function publicationEligibility({ now, busyEndpointIds = [], coolingTeamsPublicationIds = [], attemptedIds = [] }: { now?: Date; busyEndpointIds?: string[]; coolingTeamsPublicationIds?: string[]; attemptedIds?: string[] } = {}) {
const earlierPublication = alias(chatPublications, "earlier_chat_publications");
return and(
sql`not (${unresolvedSlackPublicationCondition()})`,
// An explicit Board send promises both delivery and work. Keep
// its text/files pending until the durable wake has been accepted.
sql`not exists (select 1 from chat_actions a where a.company_id = ${chatPublications.companyId} and a.endpoint_id = ${chatPublications.endpointId} and a.conversation_id = ${chatPublications.conversationId} and a.kind = 'slack_board_message' and a.status = 'received' and a.payload->>'commentId' = ${chatPublications.commentId}::text)`,
or(
and(
sql`not exists (select 1 from chat_endpoints e where e.id = ${chatPublications.endpointId} and e.publication_mode = 'explicit')`,
inArray(chatPublications.state, ["pending", "retry"]),
notExists(
db
.select({ id: chatTeamsFileTransfers.id })
.from(chatTeamsFileTransfers)
.where(
and(
eq(
chatTeamsFileTransfers.companyId,
chatPublications.companyId,
),
eq(
chatTeamsFileTransfers.publicationId,
chatPublications.id,
),
),
),
),
),
// Dedicated protocol work shares the endpoint concurrency cap,
// never the ordinary publication claim or resend implementation.
sql`exists (${db
.select({ id: chatTeamsFileTransfers.id })
.from(chatTeamsFileTransfers)
.where(
and(
eq(
chatTeamsFileTransfers.companyId,
chatPublications.companyId,
),
eq(
chatTeamsFileTransfers.endpointId,
chatPublications.endpointId,
),
eq(
chatTeamsFileTransfers.conversationId,
chatPublications.conversationId,
),
eq(
chatTeamsFileTransfers.publicationId,
chatPublications.id,
),
sql`exists (${db
.select({ id: chatEndpoints.id })
.from(chatEndpoints)
.where(
and(
eq(chatEndpoints.id, chatPublications.endpointId),
eq(
chatEndpoints.companyId,
chatPublications.companyId,
),
eq(chatEndpoints.provider, "microsoft-teams"),
eq(chatEndpoints.status, "active"),
),
)})`,
or(
inArray(chatTeamsFileTransfers.phase, [
"consent_pending",
"upload_pending",
"file_info_pending",
]),
and(
eq(chatTeamsFileTransfers.phase, "consent_unknown"),
isNotNull(chatTeamsFileTransfers.responseActivityId),
sql`${chatTeamsFileTransfers.privateState}->'response' is not null`,
),
),
),
)})`,
),
busyEndpointIds.length > 0
? notInArray(chatPublications.endpointId, busyEndpointIds)
: undefined,
coolingTeamsPublicationIds.length > 0
? notInArray(chatPublications.id, coolingTeamsPublicationIds)
: undefined,
// Inactive endpoints must not consume the eligibility page or
// acquire an artificial retry deadline. When an operator resumes
// them, due work is immediately eligible; genuine provider retry
// deadlines below remain unchanged.
notExists(
db
.select({ id: chatEndpoints.id })
.from(chatEndpoints)
.where(
and(
eq(chatEndpoints.id, chatPublications.endpointId),
inArray(chatEndpoints.status, ["paused", "attention"]),
),
),
),
now ? or(isNull(chatPublications.nextAttemptAt), lte(chatPublications.nextAttemptAt, now)) : undefined,
attemptedIds.length > 0
? notInArray(chatPublications.id, attemptedIds)
: undefined,
notExists(
db
.select({ id: earlierPublication.id })
.from(earlierPublication)
.where(
and(
eq(
earlierPublication.conversationId,
chatPublications.conversationId,
),
publicationPrecedes(earlierPublication, chatPublications),
inArray(earlierPublication.state, [
"pending",
"retry",
"streaming",
"delivery_unknown",
]),
),
),
),
);
}
async function processPendingPublications(
limit = 25,
{ waitForCompletion = true, maintenance = true }: { waitForCompletion?: boolean; maintenance?: boolean } = {},
) {
if (isShuttingDown()) return 0;
if (maintenance) {
await enqueueInboundWakeupPublications();
await reconcileTerminalConfirmationActions(limit);
}
const now = new Date();
const staleBefore = new Date(now.getTime() - 60_000);
// A process that disappears after the provider accepted a post but before
// Paperclip persisted its message id leaves an ambiguous delivery. Never
// resend it automatically; an operator can explicitly replay after
// checking the provider conversation.
const quarantinedPublications = await db.transaction(async tx => {
const condition = and(
sql`not exists (select 1 from chat_endpoints e where e.id = ${chatPublications.endpointId} and e.publication_mode = 'explicit')`,
eq(chatPublications.state, "streaming"),
lte(chatPublications.updatedAt, staleBefore),
// Teams owns separate staged I/O intents and a longer attempt lease.
// Only that protocol may recover/quarantine its in-flight stages.
notExists(
db
.select({ id: chatTeamsFileTransfers.id })
.from(chatTeamsFileTransfers)
.where(
and(
eq(
chatTeamsFileTransfers.companyId,
chatPublications.companyId,
),
eq(chatTeamsFileTransfers.publicationId, chatPublications.id),
),
),
),
notExists(
db
.select({ id: chatEndpointLeases.id })
.from(chatEndpointLeases)
.where(
and(
eq(chatEndpointLeases.companyId, chatPublications.companyId),
eq(
chatEndpointLeases.endpointId,
chatPublications.endpointId,
),
sql`${chatEndpointLeases.leaseKey} = 'publication:' || ${chatPublications.id}::text || ':' || ${chatPublications.attempts}::text`,
gt(chatEndpointLeases.expiresAt, now),
),
),
),
);
const candidates = await tx.select({ id: chatPublications.id }).from(chatPublications).where(condition).for("update", { skipLocked: true });
if (!candidates.length) return [];
await notifyChatPublicationWork(tx);
return tx
.update(chatPublications)
.set({
state: "delivery_unknown",
nextAttemptAt: null,
redactedError:
"Provider delivery could not be confirmed after the worker stopped. Check the external conversation before replaying.",
updatedAt: now,
})
.where(
and(condition, inArray(chatPublications.id, candidates.map(row => row.id))),
)
.returning({ id: chatPublications.id });
});
if (quarantinedPublications.length > 0) {
await db
.update(chatActions)
.set({
status: "delivery_unknown",
result: {
code: "task_control_delivery_unknown",
retryable: false,
},
updatedAt: now,
})
.where(
and(
eq(chatActions.kind, "task_control_authorization"),
eq(chatActions.status, "processing"),
inArray(
chatActions.providerActionId,
quarantinedPublications.map(
(publication) => `task-control-authorization:${publication.id}`,
),
),
),
);
}
const attemptedIds: string[] = [];
const ownedTasks = new Set<Promise<void>>();
let failed = false;
let firstError: unknown;
try {
while (attemptedIds.length < limit && !failed && !isShuttingDown()) {
if (publicationEndpointTasks.size >= PUBLICATION_ENDPOINT_CONCURRENCY) {
if (!waitForCompletion) break;
await Promise.race(publicationEndpointTasks.values());
continue;
}
const busyEndpointIds = [...publicationEndpointTasks.keys()];
for (const [id, retryAfter] of teamsFileRetryAfter)
if (retryAfter <= Date.now()) teamsFileRetryAfter.delete(id);
const coolingTeamsPublicationIds = [...teamsFileRetryAfter.keys()];
// Select only each conversation's current head before applying the
// global limit. Re-query after every batch so one invocation can still
// drain a conversation's newly unblocked milestones in FIFO order.
const rows = await db
.select()
.from(chatPublications)
.where(
publicationEligibility({ now, busyEndpointIds, coolingTeamsPublicationIds, attemptedIds }),
)
.orderBy(
asc(chatPublications.createdAt),
asc(publicationTransportOrderKey(chatPublications)),
)
.limit(limit - attemptedIds.length);
if (rows.length === 0) {
// The query excluded the busy-endpoint snapshot above. A worker
// may have completed while PostgreSQL selected this empty page,
// exposing a new head that must be queried before declaring idle.
if (busyEndpointIds.some((id) => !publicationEndpointTasks.has(id)))
continue;
if (!waitForCompletion || publicationEndpointTasks.size === 0) break;
await Promise.race(publicationEndpointTasks.values());
continue;
}
let started = false;
for (const selectedPublication of rows) {
if (
publicationEndpointTasks.size >= PUBLICATION_ENDPOINT_CONCURRENCY ||
attemptedIds.length >= limit ||
failed ||
isShuttingDown()
)
break;
if (publicationEndpointTasks.has(selectedPublication.endpointId))
continue;
attemptedIds.push(selectedPublication.id);
started = true;
// Reserve synchronously before provider work; overlapping drains share
// this cap. Different conversations for one bot keep the existing
// endpoint-exclusive credential fence instead of timing out behind it.
const task = Promise.resolve()
.then(async () => {
await processSelectedPublication(selectedPublication);
await settleExternalConversation(db, selectedPublication.companyId, selectedPublication.issueId).catch((err) => {
logger.warn({ err, publicationId: selectedPublication.id }, "Slack conversation settlement deferred to reconciliation");
});
})
.catch((error: unknown) => {
if (!failed) {
failed = true;
firstError = error;
}
if (!waitForCompletion) {
logger.error(
{
endpointId: selectedPublication.endpointId,
publicationId: selectedPublication.id,
error: redactError(error),
},
"scheduled chat publication worker failed",
);
}
})
.finally(() => {
if (
publicationEndpointTasks.get(selectedPublication.endpointId) ===
task
) {
publicationEndpointTasks.delete(selectedPublication.endpointId);
}
ownedTasks.delete(task);
wakePublications();
});
publicationEndpointTasks.set(selectedPublication.endpointId, task);
ownedTasks.add(task);
}
// Re-query while slots remain: one busy endpoint must not fill the
// candidate page and hide later eligible endpoints behind its backlog.
if (!started && publicationEndpointTasks.size > 0) {
if (!waitForCompletion) break;
await Promise.race(publicationEndpointTasks.values());
}
}
} catch (error) {
if (!failed) {
failed = true;
firstError = error;
}
} finally {
// Observe rejection immediately above and join every started worker
// before this invocation returns or propagates its original failure.
if (waitForCompletion) await Promise.all(ownedTasks);
}
if (failed) throw firstError;
if (maintenance) await processPendingInteractionWakeups(limit);
return attemptedIds.length;
}
return { processPendingPublications, publicationEligibility };
}
@@ -0,0 +1,149 @@
import type { DbOrTransaction } from "../runtime-types.js";
import { notifyChatPublicationWork } from "../../chat-work-notifications.js";
import {
and,
eq,
gt,
inArray,
like,
or,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatConversations,
chatPublications,
issues,
} from "@paperclipai/db";
import type {
SafeChatPublicationPayload,
} from "@paperclipai/shared";
import { projectSafeChatPublication } from "../../chat-publication-projection.js";
import { publicationTransportOrderKey } from "./order.js";
export function createChatPublicationTaskControl(db: Db) {
async function currentTaskControlPayload(
publication: typeof chatPublications.$inferSelect,
): Promise<SafeChatPublicationPayload> {
const persisted = publication.payload as SafeChatPublicationPayload;
if (!publication.idempotencyKey.startsWith("control:status:")) {
return persisted;
}
const issue = await db
.select({
identifier: issues.identifier,
status: issues.status,
title: issues.title,
})
.from(issues)
.where(
and(
eq(issues.companyId, publication.companyId),
eq(issues.id, publication.issueId),
),
)
.then((rows) => rows[0] ?? null);
if (!issue) return persisted;
return projectSafeChatPublication({
classification: "external",
source: "task_control",
text: `${issue.identifier}: ${issue.title} — ${issue.status}`,
});
}
function taskControlCompletesConversation(
publication: Pick<typeof chatPublications.$inferSelect, "idempotencyKey">,
): boolean {
return /^control:(?:new|close):/.test(publication.idempotencyKey);
}
async function hasCommittedTaskControlCompletion(
conversationId: string,
database: DbOrTransaction = db,
): Promise<boolean> {
return database
.select({ id: chatPublications.id })
.from(chatPublications)
.where(
and(
eq(chatPublications.conversationId, conversationId),
eq(chatPublications.state, "published"),
or(
like(chatPublications.idempotencyKey, "control:new:%"),
like(chatPublications.idempotencyKey, "control:close:%"),
),
),
)
.limit(1)
.then((rows) => rows.length > 0);
}
async function commitTaskControlCompletion(
tx: Db,
publication: Pick<
typeof chatPublications.$inferSelect,
| "companyId"
| "conversationId"
| "createdAt"
| "endpointId"
| "id"
| "idempotencyKey"
| "payload"
>,
committedAt: Date,
): Promise<void> {
if (!taskControlCompletesConversation(publication)) return;
await tx
.update(chatConversations)
.set({ state: "completed", updatedAt: committedAt })
.where(
and(
eq(chatConversations.companyId, publication.companyId),
eq(chatConversations.endpointId, publication.endpointId),
eq(chatConversations.id, publication.conversationId),
inArray(chatConversations.state, ["active", "waiting"]),
),
);
// The provider-visible completion is the external boundary. Anything
// enqueued later for this old generation stays internal even if its run
// finishes after /new or /close. A racing insert that commits after this
// transaction is rejected by the completed-conversation send guard.
await notifyChatPublicationWork(tx);
await tx
.update(chatPublications)
.set({
state: "cancelled",
nextAttemptAt: null,
redactedError:
"Conversation was completed by a task control before delivery",
updatedAt: committedAt,
})
.where(
and(
eq(chatPublications.companyId, publication.companyId),
eq(chatPublications.endpointId, publication.endpointId),
eq(chatPublications.conversationId, publication.conversationId),
or(
gt(chatPublications.createdAt, publication.createdAt),
and(
eq(chatPublications.createdAt, publication.createdAt),
gt(
publicationTransportOrderKey(chatPublications),
publication.payload.transportPart?.orderKey ?? publication.id,
),
),
),
inArray(chatPublications.state, ["pending", "retry", "streaming"]),
sql`not (${chatPublications.idempotencyKey} like 'explicit:%' or ${chatPublications.idempotencyKey} like 'explicit-board:%')`,
),
);
}
return { currentTaskControlPayload, hasCommittedTaskControlCompletion, commitTaskControlCompletion };
}
@@ -0,0 +1,439 @@
import type { PublicationTransportInput } from "./types.js";
import { createPhotonPublicationDelivery } from "../imessage-photon/publications.js";
import { createOutboundAttachments } from "../outbound-attachments.js";
import { discordMarkdownAttachment } from "../discord/messages.js";
import { telegramMarkdownAttachment } from "../telegram/messages.js";
import type { EndpointRow, DbOrTransaction } from "../runtime-types.js";
import { createChatPublicationReplacement } from "../publication-replacement.js";
import { voiceSessionService } from "../../voice/voice-session-service.js";
import { voiceSessionStore } from "../../voice/voice-session-store.js";
import { type ChatRuntime } from "../../chat-runtime.js";
import type { ChatEndpointRuntime } from "../../voice/voice-runtime.js";
import { chatGitHubConfigurations } from "@paperclipai/db";
import { githubResponseCommentService } from "../../chat-github-response-comments.js";
import {
and,
eq,
isNull,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatMessageLinks,
chatPublications,
} from "@paperclipai/db";
import type {
ChatAdapterCapabilities,
ChatEndpointSetupState,
ChatProvider,
SafeChatPublicationPayload,
} from "@paperclipai/shared";
import { telegramAttachmentForUpload } from "../../chat-telegram-photo.js";
import {
conflict,
forbidden,
} from "../../../errors.js";
import {
type ChatSdkRuntime,
} from "../../chat-sdk-runtime.js";
import { safeChatTaskUrl } from "../../chat-task-url.js";
import {
renderPublicationTransportText,
} from "../../chat-publication-text-parts.js";
import { classifyChatPublicationError } from "../../chat-publication-errors.js";
import {
prepareTeamsInlineImage,
} from "../../chat-teams-inline-images.js";
import {
shouldStreamSafePublicationText,
streamSafePublicationText,
} from "../../chat-publication-stream.js";
import {
isChatQuestionFormOpenActionId,
} from "../../chat-question-forms.js";
import type {
AdapterPostableMessage,
Attachment,
FileUpload,
} from "chat";
import { Actions, Button, Card, CardText, LinkButton } from "chat";
import type { EndpointRecord } from "../runtime-types.js";
import { isExplicitOperatorPublication } from "./policy.js";
type PublicationTransportOptions = {
endpointRecord: (endpointId: string) => Promise<EndpointRecord | null>;
fetchImpl: typeof globalThis.fetch;
getTaskBaseUrl: () => string | null | undefined;
publicationFiles: ReturnType<typeof createOutboundAttachments>["publicationFiles"];
receiptReactionCompletionRunId: ReturnType<typeof createChatPublicationReplacement>["receiptReactionCompletionRunId"];
runIdFromMilestonePublication: ReturnType<typeof createChatPublicationReplacement>["runIdFromMilestonePublication"];
runtime: Pick<ChatRuntime | ChatSdkRuntime, "get">;
runtimeFor: (endpoint: EndpointRow) => Promise<ChatEndpointRuntime>;
runtimeGeneration: (setup: ChatEndpointSetupState) => number;
CAPABILITIES: Record<ChatProvider, ChatAdapterCapabilities>;
voice: Pick<ReturnType<typeof voiceSessionService>, "pushReplies">;
voiceStore: Pick<ReturnType<typeof voiceSessionStore>, "enqueuePublication">;
scheduleMessageProcessing: (task: () => Promise<void>) => void;
};
export function createChatPublicationTransport(db: Db, options: PublicationTransportOptions) {
const { endpointRecord, fetchImpl, getTaskBaseUrl, publicationFiles, receiptReactionCompletionRunId, runIdFromMilestonePublication, runtime, runtimeFor, runtimeGeneration, CAPABILITIES, voice, voiceStore, scheduleMessageProcessing } = options;
const { deliverPhotonPublication } = createPhotonPublicationDelivery(db, { endpointRecord, runtimeGeneration, runtime, receiptReactionCompletionRunId, getTaskBaseUrl });
async function attemptProviderPublication<T>(
send: () => Promise<T>,
): Promise<T> {
return await send();
}
async function editOrPostProviderPublication<T>(
edit: () => Promise<T>,
post: () => Promise<T>,
): Promise<T> {
try {
return await edit();
} catch (error) {
// A definite 404 proves the old progress comment no longer exists. It is
// therefore safe to create a replacement without risking a duplicate.
// Ambiguous transport failures must still enter delivery_unknown.
if (classifyChatPublicationError(error, 1).kind !== "resource_unavailable")
throw error;
return await post();
}
}
function safeCardForPublication(
payload: SafeChatPublicationPayload,
provider: ChatProvider,
) {
if (!payload.card) return null;
const children = [];
if (payload.card.body) children.push(CardText(payload.card.body));
const actions = (payload.card.actions ?? []).map((action) =>
action.type === "callback"
? Button({
id: action.actionId,
label: action.label,
style: action.style ?? "default",
// Telegram encodes both id and value into callback_data, whose hard
// limit is 64 bytes. Its opaque action id resolves through the
// durable token ledger, so repeating the interaction UUID is both
// unnecessary and too large. Other adapters retain the value as a
// defense-in-depth binding supported by their larger envelopes.
value: provider === "telegram" ? undefined : payload.interactionId,
actionType: isChatQuestionFormOpenActionId(action.actionId)
? "modal"
: undefined,
})
: LinkButton({ label: action.label, url: action.url }),
);
if (actions.length) children.push(Actions(actions));
return Card({ title: payload.card.title, children });
}
async function recordExactOutboundPublicationLink(
tx: DbOrTransaction,
input: {
publication: typeof chatPublications.$inferSelect;
providerMessageId: string;
},
): Promise<boolean> {
const inserted = await tx
.insert(chatMessageLinks)
.values({
companyId: input.publication.companyId,
endpointId: input.publication.endpointId,
conversationId: input.publication.conversationId,
publicationId: input.publication.id,
commentId: input.publication.commentId,
providerMessageId: input.providerMessageId,
direction: "outbound",
})
.onConflictDoNothing()
.returning({ id: chatMessageLinks.id });
if (inserted.length) return true;
return tx
.select({ id: chatMessageLinks.id })
.from(chatMessageLinks)
.where(
and(
eq(chatMessageLinks.companyId, input.publication.companyId),
eq(chatMessageLinks.endpointId, input.publication.endpointId),
eq(chatMessageLinks.conversationId, input.publication.conversationId),
eq(chatMessageLinks.publicationId, input.publication.id),
input.publication.commentId
? eq(chatMessageLinks.commentId, input.publication.commentId)
: isNull(chatMessageLinks.commentId),
eq(chatMessageLinks.providerMessageId, input.providerMessageId),
eq(chatMessageLinks.direction, "outbound"),
),
)
.limit(1)
.then((rows) => rows.length > 0);
}
async function postSafePublication(input: PublicationTransportInput) {
if (input.endpoint.provider === "speko") {
const receipt = await voiceStore.enqueuePublication(input.endpoint.companyId, input.publication.id);
scheduleMessageProcessing(() => voice.pushReplies(25, input.endpoint.companyId, input.endpoint.id).then(() => undefined));
return receipt;
}
if (input.endpoint.provider === "github" && ["failed", "completed"].includes(input.payload.progressState ?? "") &&
!input.payload.interactionId && !isExplicitOperatorPublication(input.publication) &&
(await db.select({ endpointId: chatGitHubConfigurations.endpointId }).from(chatGitHubConfigurations).where(and(
eq(chatGitHubConfigurations.companyId, input.endpoint.companyId), eq(chatGitHubConfigurations.endpointId, input.endpoint.id))).limit(1)).length > 0) {
const runId = runIdFromMilestonePublication(input.publication);
if (!runId) throw forbidden("GitHub failure has no run binding");
const receipt = await githubResponseCommentService(db, fetchImpl).failRun({
companyId: input.publication.companyId, endpointId: input.endpoint.id,
issueId: input.publication.issueId, runId, body: input.payload.progressState === "completed"
? "This turn ended without publishing a final response. See the Paperclip task for details."
: renderPublicationTransportText(input.payload),
});
if (!receipt) throw conflict("GitHub response publication is busy");
return { id: receipt.id, threadId: input.conversation.externalThreadId, raw: {}, text: input.payload.text };
}
const endpointRuntime = await runtimeFor(input.endpoint);
const thread = endpointRuntime.thread(input.conversation.externalThreadId);
const card = safeCardForPublication(input.payload, input.endpoint.provider);
let attachments: Attachment[] = [];
let files: FileUpload[] = [];
let text = renderPublicationTransportText(input.payload);
if (input.payload.attachmentIds?.length) {
if (
input.endpoint.provider === "microsoft-teams" &&
!input.conversation.isDirectMessage
) {
// The staged personal-file lane remains authoritative for consent.
// Channel/group pictures need no OneDrive upload or public asset URL:
// send only verified original picture bytes through the native adapter.
const candidates = await publicationFiles(
input.publication,
input.payload,
{ teamsInlineImagesOnly: true },
);
const pictures = await Promise.all(
candidates.map(prepareTeamsInlineImage),
);
if (pictures.length && pictures.every((picture) => picture !== null)) {
files = pictures;
}
}
const nativeFileSurface =
CAPABILITIES[input.endpoint.provider].files &&
input.endpoint.provider !== "microsoft-teams";
if (nativeFileSurface) {
const uploads = await publicationFiles(
input.publication,
input.payload,
);
if (input.endpoint.provider === "telegram") {
attachments = await Promise.all(
uploads.map(telegramAttachmentForUpload),
);
} else {
files = uploads;
}
} else if (files.length === 0) {
const taskUrl = safeChatTaskUrl(
getTaskBaseUrl(),
input.publication.issueId,
);
if (
input.endpoint.provider === "github" ||
input.endpoint.provider === "microsoft-teams"
) {
// GitHub lacks a bot upload contract. Teams reaches this fallback
// only without an admissible personal-consent destination. Replace
// our generated caption, preserving any future custom text.
const generatedFileLabel = /^Shared (.+)\.$/s.exec(text)?.[1] ?? null;
const limitation =
input.endpoint.provider === "github"
? "This GitHub App connection cannot upload file bytes into comments."
: "Direct file delivery isn't available for this Teams conversation.";
if (generatedFileLabel) {
const saved = taskUrl
? `File saved on the Paperclip task: ${generatedFileLabel}.`
: `File saved on the private Paperclip task: ${generatedFileLabel}.`;
text = taskUrl
? `${saved} ${limitation} Download it: ${taskUrl}`
: `${saved} ${limitation}`;
} else {
const handoff = taskUrl
? `Open the file on its Paperclip task: ${taskUrl}`
: "The file remains available only on the private Paperclip task.";
text = `${text}\n\n${limitation} ${handoff}`;
}
} else {
const attachmentFallback = taskUrl
? `Open the task in Paperclip: ${taskUrl}`
: "Open the task in Paperclip to download the attachment.";
text = `${text}\n\n${attachmentFallback}`;
}
}
}
if (input.endpoint.provider === "imessage-photon") {
return deliverPhotonPublication(input, endpointRuntime, thread, text, files);
}
if (
input.endpoint.provider === "discord" &&
input.payload.transportPart?.mode === "discord_markdown_attachment"
) {
files.push(discordMarkdownAttachment(text));
text =
input.payload.transportPart.count > 1
? "Complete response attached."
: "Paperclip attached the complete response because it exceeds Discord’s message limit.";
}
if (
input.endpoint.provider === "telegram" &&
input.payload.transportPart?.mode === "telegram_markdown_attachment"
) {
attachments.push(telegramMarkdownAttachment(text));
text =
input.payload.transportPart.count > 1
? "Complete response attached."
: "Paperclip attached the complete response to preserve its Markdown formatting.";
}
if (card && CAPABILITIES[input.endpoint.provider].cards) {
return await attemptProviderPublication(async () =>
input.replaceProviderMessageId
? await editOrPostProviderPublication(
() =>
thread.adapter.editMessage(
thread.id,
input.replaceProviderMessageId!,
{
card,
fallbackText: text,
...(files.length ? { files } : {}),
},
),
() =>
thread.post({
card,
fallbackText: text,
...(files.length ? { files } : {}),
}),
)
: await thread.post({
card,
fallbackText: text,
...(files.length ? { files } : {}),
}),
);
}
if (attachments.length) {
const attachmentMessage: AdapterPostableMessage = {
attachments,
markdown: text,
};
return await attemptProviderPublication(async () =>
input.replaceProviderMessageId
? await editOrPostProviderPublication(
() =>
thread.adapter.editMessage(
thread.id,
input.replaceProviderMessageId!,
attachmentMessage,
),
() => thread.post(attachmentMessage),
)
: await thread.post(attachmentMessage),
);
}
if (files.length) {
// The pinned Slack adapter performs file upload and text publication as
// two provider calls when both fields are present. A rate limit or
// network failure on the second call would make an automatic retry
// duplicate the already-shared file. File publications have their own
// ordered outbox row, so Slack receives a single native upload here.
const fileMessage: AdapterPostableMessage = {
// Empty Slack markdown keeps the SDK call type-safe while its pinned
// adapter treats this as a file-only upload and skips chat.postMessage.
markdown: input.endpoint.provider === "slack" ? "" : text,
files,
};
if (
input.endpoint.provider === "slack" &&
!input.replaceProviderMessageId &&
!input.payload.card &&
!input.payload.interactionId &&
!input.publication.idempotencyKey.startsWith("control:") &&
input.payload.attachmentIds?.length &&
input.onSlackFileUploadAccepted
) {
return await attemptProviderPublication(
async () =>
await endpointRuntime.postSlackFilePublication(
thread.id,
fileMessage,
input.onSlackFileUploadAccepted!,
),
);
}
return await attemptProviderPublication(async () =>
input.replaceProviderMessageId
? await editOrPostProviderPublication(
() =>
thread.adapter.editMessage(
thread.id,
input.replaceProviderMessageId!,
fileMessage,
),
() => thread.post(fileMessage),
)
: await thread.post(fileMessage),
);
}
if (
!input.replaceProviderMessageId &&
!input.payload.transportPart &&
CAPABILITIES[input.endpoint.provider].nativeStreaming &&
(input.endpoint.provider !== "telegram" ||
!input.conversation.isDirectMessage ||
input.telegramDraftControl !== undefined) &&
shouldStreamSafePublicationText(text)
) {
return await attemptProviderPublication(async () =>
input.telegramDraftControl
? endpointRuntime.streamTelegramDraft(
input.conversation.externalThreadId,
streamSafePublicationText(text),
input.telegramDraftControl,
)
: thread.post(streamSafePublicationText(text)),
);
}
return await attemptProviderPublication(async () =>
input.replaceProviderMessageId
? await editOrPostProviderPublication(
() =>
thread.adapter.editMessage(
thread.id,
input.replaceProviderMessageId!,
{ markdown: text },
),
() => thread.post({ markdown: text }),
)
: await thread.post({ markdown: text }),
);
}
return { postSafePublication, recordExactOutboundPublicationLink };
}
@@ -0,0 +1,18 @@
import type { chatConversations, chatPublications } from "@paperclipai/db";
import type { SafeChatPublicationPayload } from "@paperclipai/shared";
import type { EndpointRow } from "../runtime-types.js";
import type { TelegramDraftControl } from "../../chat-telegram-draft-stop.js";
import type { SlackFileUploadAcceptedReceipt } from "../../chat-sdk-runtime.js";
type ConversationRow = typeof chatConversations.$inferSelect;
export type PublicationTransportInput = {
endpoint: EndpointRow;
conversation: ConversationRow;
publication: typeof chatPublications.$inferSelect;
payload: SafeChatPublicationPayload;
replaceProviderMessageId?: string | null;
telegramDraftControl?: TelegramDraftControl;
beforePhotonWrite?(): Promise<void>;
onSlackFileUploadAccepted?: (
receipt: SlackFileUploadAcceptedReceipt,
) => Promise<void>;
};
File diff suppressed because it is too large. Load diff
@@ -1,3 +1,5 @@
import type { Message, Thread } from "chat";
import type { ChatSdkMessageCallbackEvent } from "../chat-sdk-runtime.js";
import type { Db, chatEndpoints } from "@paperclipai/db";
import type { ToolCredentialSecretRef, ChatEndpointSetupState, ChatEndpointCallbackSurfaces } from "@paperclipai/shared";
import type { ChatEndpointRuntime } from "../voice/voice-runtime.js";
@@ -66,3 +68,26 @@ export type PreparedProviderInventory = {
inventory: import("../chat-provider-inventory.js").ChatProviderInventoryResult | null;
managementUrl?: string;
};
export type LiveInboundMessage = {
endpoint: EndpointRow;
message: Message;
receiptReactionSupported: boolean;
providerUrl: string | null;
runtimeContext?: InboundRuntimeContext;
thread: Thread;
trigger: ChatSdkMessageCallbackEvent["trigger"];
};
export type DiscordGatewayOwnership = {
companyId: string;
context: RuntimeContext;
endpointId: string;
expiresAt: Date;
leaseKey: "discord_gateway_runtime" | "photon_receiver_runtime";
renewTimer: ReturnType<typeof setInterval> | null;
renewal: Promise<void> | null;
stopPromise: Promise<void> | null;
stopping: boolean;
token: string;
};
@@ -0,0 +1,88 @@
import {
and,
eq,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatSdkState,
} from "@paperclipai/db";
import type {
ChatSdkStateCompareAndSetInput,
ChatSdkStateDeleteInput,
ChatSdkStatePersistence,
ChatSdkStateScope,
} from "../chat-sdk-state.js";
export function createChatSdkStatePersistence(db: Db): ChatSdkStatePersistence {
return {
async read(scope: ChatSdkStateScope, key: string) {
return db
.select({
value: chatSdkState.value,
version: chatSdkState.version,
expiresAt: chatSdkState.expiresAt,
})
.from(chatSdkState)
.where(
and(
eq(chatSdkState.companyId, scope.companyId),
eq(chatSdkState.endpointId, scope.endpointId),
eq(chatSdkState.stateKey, key),
),
)
.then((rows) => rows[0] ?? null);
},
async compareAndSet(input: ChatSdkStateCompareAndSetInput) {
if (input.expectedVersion === null) {
const inserted = await db
.insert(chatSdkState)
.values({
companyId: input.companyId,
endpointId: input.endpointId,
stateKey: input.key,
version: 1,
value: input.value,
expiresAt: input.expiresAt,
})
.onConflictDoNothing()
.returning({ id: chatSdkState.id });
return inserted.length === 1;
}
const updated = await db
.update(chatSdkState)
.set({
version: input.expectedVersion + 1,
value: input.value,
expiresAt: input.expiresAt,
updatedAt: new Date(),
})
.where(
and(
eq(chatSdkState.companyId, input.companyId),
eq(chatSdkState.endpointId, input.endpointId),
eq(chatSdkState.stateKey, input.key),
eq(chatSdkState.version, input.expectedVersion),
),
)
.returning({ id: chatSdkState.id });
return updated.length === 1;
},
async deleteIfVersion(input: ChatSdkStateDeleteInput) {
const deleted = await db
.delete(chatSdkState)
.where(
and(
eq(chatSdkState.companyId, input.companyId),
eq(chatSdkState.endpointId, input.endpointId),
eq(chatSdkState.stateKey, input.key),
eq(chatSdkState.version, input.expectedVersion),
),
)
.returning({ id: chatSdkState.id });
return deleted.length === 1;
},
};
}
@@ -0,0 +1,89 @@
import { slackCommandForAgent } from "./commands.js";
import { canonicalCallbackUrl, slackCallbackMatchesPublicUrl } from "./callback-verification.js";
import type { EndpointRow, SlackCallbackSurface, InternalSetupState } from "../runtime-types.js";
import type {
ChatEndpointCallbackSurfaces,
} from "@paperclipai/shared";
export function slackSetupState(input: {
endpoint: Pick<
EndpointRow,
| "provider"
| "providerAccountId"
| "publicId"
| "status"
| "setup"
| "botUsername"
>;
webhookUrl: string | null;
step: EndpointRow["setup"]["step"];
assignedAgentName?: string | null;
}) {
const { endpoint, webhookUrl, step, assignedAgentName } = input;
const observations = (endpoint.setup as InternalSetupState)
.slackCallbackSurfaces;
const currentUrl = webhookUrl ? canonicalCallbackUrl(webhookUrl) : null;
const surfaceState = (
surface: SlackCallbackSurface,
): ChatEndpointCallbackSurfaces[SlackCallbackSurface] => {
const observation = observations?.[surface];
if (!observation) return { status: "unverified" };
return {
status:
currentUrl !== null &&
slackCallbackMatchesPublicUrl(observation.url, currentUrl)
? "current"
: "stale",
observedAt: observation.observedAt,
};
};
const callbackSurfaces: ChatEndpointCallbackSurfaces = {
events: surfaceState("events"),
interactivity: surfaceState("interactivity"),
slashCommands: surfaceState("slashCommands"),
};
return {
step,
testStartedAt: endpoint.setup.testStartedAt ?? null,
testSkipped: endpoint.setup.testSkipped ?? false,
authorizationUrl: "https://api.slack.com/apps?new_app=1",
providerUrl: "https://app.slack.com/",
webhookUrl,
webhookVerifiedAt: endpoint.setup.webhookVerifiedAt ?? null,
callbackSurfaces,
callbacksNeedUpdate: Object.values(callbackSurfaces).some(
(surface) => surface.status === "stale",
),
slackApp: endpoint.setup.slackApp,
slackSetupMethod: endpoint.setup.slackSetupMethod,
slackAvatar: endpoint.setup.slackAvatar,
slackManagerError: endpoint.setup.slackManagerError,
slackAccount: endpoint.setup.slackAccount,
// Slack registers the slash command in the provider configuration.
// Keep that identity immutable when the assigned agent is renamed;
// deriving it remains only a compatibility path for older rows.
command:
typeof endpoint.setup.command === "string"
? endpoint.setup.command
: assignedAgentName
? slackCommandForAgent(assignedAgentName, endpoint.publicId)
: null,
} as const;
}
import type { ResolvedChatSdkProviderConfig } from "../../chat-sdk-runtime.js";
export function slackRuntimeConfig(endpoint: EndpointRow, userName: string, credentials: Record<string, string>): ResolvedChatSdkProviderConfig {
return {
provider: "slack",
userName,
credentials: {
botToken: credentials.botToken,
signingSecret: credentials.signingSecret,
botUserId: endpoint.botExternalId ?? undefined,
},
};
}
@@ -0,0 +1,37 @@
import type { EndpointRow } from "../runtime-types.js";
import { syncSpekoVoiceTools } from "../../voice/speko-agent-tools.js";
import { configureSpekoSessionTools } from "../../voice/speko-tool-setup.js";
import { createSpekoProvider } from "../../voice/speko-provider.js";
import { type CredentialMutationLeaseGuard } from "../../chat-credential-mutation-lease.js";
import type { Db } from "@paperclipai/db";
import {
forbidden,
unprocessable,
} from "../../../errors.js";
import type { DbTransaction } from "../runtime-types.js";
type SpekoSetupOptions = {
fetchImpl: typeof globalThis.fetch;
getWebhookPublicBaseUrl: () => string | null;
allowLocalVoiceBoard?: boolean;
};
export function createSpekoSetup(db: Db, options: SpekoSetupOptions) {
const { fetchImpl, getWebhookPublicBaseUrl } = options;
async function installSpekoVoiceTools(tx: DbTransaction, endpoint: EndpointRow, actorUserId?: string | null) {
if (!actorUserId) throw forbidden("An authenticated connection owner is required to install phone tools");
await syncSpekoVoiceTools(tx, endpoint, actorUserId, true, options.allowLocalVoiceBoard === true);
}
async function configureSpekoTools(endpoint: EndpointRow, credentials: Record<string, string>, credentialLease: CredentialMutationLeaseGuard, previousSpekoSigningSecret: string | undefined) {
const webhookPublicBaseUrl = getWebhookPublicBaseUrl();
if (!webhookPublicBaseUrl?.startsWith("https://")) throw unprocessable("Speko needs an explicitly configured public HTTPS callback origin");
await configureSpekoSessionTools(db, { companyId: endpoint.companyId, endpointId: endpoint.id, agentId: credentials.agentId, callbackUrl: `${webhookPublicBaseUrl}/api/voice-webhooks/${endpoint.publicId}/tools`, signingSecret: credentials.signingSecret, previousSigningSecret: previousSpekoSigningSecret, client: createSpekoProvider(credentials.apiKey, fetchImpl), assertOwned: () => credentialLease.assertOwned() });
}
return { installSpekoVoiceTools, configureSpekoTools };
}
@@ -0,0 +1,656 @@
import type { DbTransaction, RuntimeContext, LifecycleRuntimeFence } from "../runtime-types.js";
import { notifyChatPublicationWork } from "../../chat-work-notifications.js";
import {
and,
asc,
eq,
inArray,
ne,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatActions,
chatConversations,
chatEndpoints,
chatExternalPrincipals,
chatPublications,
issueThreadInteractions,
issues,
} from "@paperclipai/db";
import {
type ChatSdkCallbackEvent,
} from "../../chat-sdk-runtime.js";
import {
logActivity,
} from "../../activity-log.js";
import { projectSafeChatPublication } from "../../chat-publication-projection.js";
import { issueThreadInteractionService } from "../../issue-thread-interactions.js";
import type { EndpointRecord } from "../runtime-types.js";
import type { ActionEvent } from "chat";
import { nativeTelegramConfirmation } from "../../chat-interaction-publications.js";
type TelegramConfirmationInput = {
record: EndpointRecord;
event: ChatSdkCallbackEvent<ActionEvent>;
runtimeContext: RuntimeContext;
conversation: typeof chatConversations.$inferSelect;
principal: { principal: typeof chatExternalPrincipals.$inferSelect; userId: string };
issued: { publication: typeof chatPublications.$inferSelect };
interaction: Awaited<ReturnType<ReturnType<typeof issueThreadInteractionService>["listForIssue"]>>[number];
deny: (known?: { conversationId?: string | null; principalId?: string | null }) => Promise<void>;
safelyKnown: { conversationId: string; principalId: string };
};
type TelegramConfirmationOptions = {
requireCurrentExternalActionAuthorization: (tx: DbTransaction, input: { conversationId: string; endpointId: string; expectedUserId: string; principalId: string; runtimeContext: LifecycleRuntimeFence }) => Promise<void>;
isExternalActionAuthorizationChange: (error: unknown) => boolean;
scheduleMessageProcessing: (task: () => Promise<void>) => void;
processPendingPublications: () => Promise<unknown>;
confirmationResolutionPersistBarrier?: () => Promise<void>;
};
export function createTelegramConfirmationActions(db: Db, options: TelegramConfirmationOptions) {
const { requireCurrentExternalActionAuthorization, isExternalActionAuthorizationChange, scheduleMessageProcessing, processPendingPublications } = options;
async function handleTelegramConfirmation(input: TelegramConfirmationInput) {
const { record, event, runtimeContext, conversation, principal, issued, interaction, deny, safelyKnown } = input;
if (
event.provider !== "telegram" ||
!nativeTelegramConfirmation(interaction)
) {
return deny(safelyKnown);
}
const action = await db
.select()
.from(chatActions)
.where(
and(
eq(chatActions.companyId, record.endpoint.companyId),
eq(chatActions.endpointId, record.endpoint.id),
eq(chatActions.conversationId, conversation.id),
eq(chatActions.kind, "confirmation_response"),
eq(chatActions.providerActionId, event.event.actionId),
),
)
.then((rows) => rows[0] ?? null);
if (!action || action.status !== "issued") return deny(safelyKnown);
const tokenPayload = action.payload;
const tokenExpiresAt =
typeof tokenPayload.expiresAt === "string"
? Date.parse(tokenPayload.expiresAt)
: Number.NaN;
if (!Number.isFinite(tokenExpiresAt) || tokenExpiresAt <= Date.now()) {
await db
.update(chatActions)
.set({
status: "expired",
result: { code: "confirmation_action_token_expired" },
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "issued"),
),
);
return deny(safelyKnown);
}
if (
tokenPayload.version !== 1 ||
tokenPayload.publicationId !== issued.publication.id ||
tokenPayload.interactionId !== interaction.id ||
(tokenPayload.decision !== "accept" &&
tokenPayload.decision !== "reject")
) {
return deny(safelyKnown);
}
const issue = await db
.select({
id: issues.id,
companyId: issues.companyId,
projectId: issues.projectId,
goalId: issues.goalId,
status: issues.status,
assigneeAgentId: issues.assigneeAgentId,
})
.from(issues)
.where(
and(
eq(issues.companyId, record.endpoint.companyId),
eq(issues.id, conversation.issueId),
),
)
.then((rows) => rows[0] ?? null);
if (!issue) {
return deny(safelyKnown);
}
const decision = tokenPayload.decision;
try {
if (decision === "accept") {
await issueThreadInteractionService(db).acceptInteraction(
issue,
interaction.id,
{},
{ userId: principal.userId },
{
beforeResolveInTransaction: async (tx) => {
await requireCurrentExternalActionAuthorization(tx, {
conversationId: conversation.id,
endpointId: record.endpoint.id,
expectedUserId: principal.userId!,
principalId: principal.principal.id,
runtimeContext,
});
},
afterResolveInTransaction: async (tx, resolved) => {
const [owned] = await tx
.update(chatActions)
.set({
principalId: principal.principal.id,
status: "processing",
payload: {
...tokenPayload,
messageId: event.event.messageId,
actionId: event.event.actionId,
value: null,
},
result: {
code: "confirmation_resolution_committed_by_provider",
interactionId: resolved.id,
interactionStatus: resolved.status,
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "issued"),
),
)
.returning({ id: chatActions.id });
if (!owned) {
throw new Error(
"External confirmation action ownership changed before interaction commit",
);
}
},
},
);
} else {
await issueThreadInteractionService(db).rejectInteraction(
issue,
interaction.id,
{},
{ userId: principal.userId },
{
beforeResolveInTransaction: async (tx) => {
await requireCurrentExternalActionAuthorization(tx, {
conversationId: conversation.id,
endpointId: record.endpoint.id,
expectedUserId: principal.userId!,
principalId: principal.principal.id,
runtimeContext,
});
},
afterResolveInTransaction: async (tx, resolved) => {
const [owned] = await tx
.update(chatActions)
.set({
principalId: principal.principal.id,
status: "processing",
payload: {
...tokenPayload,
messageId: event.event.messageId,
actionId: event.event.actionId,
value: null,
},
result: {
code: "confirmation_resolution_committed_by_provider",
interactionId: resolved.id,
interactionStatus: resolved.status,
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "issued"),
),
)
.returning({ id: chatActions.id });
if (!owned) {
throw new Error(
"External confirmation action ownership changed before interaction commit",
);
}
},
},
);
}
} catch (error) {
if (isExternalActionAuthorizationChange(error)) {
return deny(safelyKnown);
}
const currentStatus = await db
.select({ status: issueThreadInteractions.status })
.from(issueThreadInteractions)
.where(eq(issueThreadInteractions.id, interaction.id))
.then((rows) => rows[0]?.status ?? null);
if (currentStatus !== null && currentStatus !== "pending") {
await settleTerminalConfirmationAction(action.id);
const completedAction = await db
.select({ status: chatActions.status, result: chatActions.result })
.from(chatActions)
.where(eq(chatActions.id, action.id))
.then((rows) => rows[0] ?? null);
if (
completedAction?.status === "processed" &&
completedAction.result?.code ===
"confirmation_resolution_committed_by_provider"
) {
return;
}
return deny(safelyKnown);
}
throw error;
}
await options.confirmationResolutionPersistBarrier?.();
if (!(await settleTerminalConfirmationAction(action.id))) {
throw new Error(
"External confirmation action ownership changed before settlement",
);
}
scheduleMessageProcessing(async () => {
await processPendingPublications();
});
return;
}
async function settleTerminalConfirmationAction(
actionId: string,
): Promise<boolean> {
const action = await db
.select()
.from(chatActions)
.where(
and(
eq(chatActions.id, actionId),
eq(chatActions.kind, "confirmation_response"),
),
)
.then((rows) => rows[0] ?? null);
if (action?.status === "processed") return true;
if (!action || action.status !== "processing" || !action.conversationId)
return false;
const token = action.payload;
if (
token.version !== 1 ||
typeof token.publicationId !== "string" ||
typeof token.interactionId !== "string" ||
(token.decision !== "accept" && token.decision !== "reject")
) {
await db
.update(chatActions)
.set({
status: "failed",
result: { code: "confirmation_action_payload_invalid" },
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "processing"),
),
);
return false;
}
const [conversation, publication, endpoint] = await Promise.all([
db
.select()
.from(chatConversations)
.where(
and(
eq(chatConversations.companyId, action.companyId),
eq(chatConversations.endpointId, action.endpointId),
eq(chatConversations.id, action.conversationId),
),
)
.then((rows) => rows[0] ?? null),
db
.select()
.from(chatPublications)
.where(
and(
eq(chatPublications.companyId, action.companyId),
eq(chatPublications.endpointId, action.endpointId),
eq(chatPublications.id, token.publicationId as string),
),
)
.then((rows) => rows[0] ?? null),
db
.select()
.from(chatEndpoints)
.where(
and(
eq(chatEndpoints.companyId, action.companyId),
eq(chatEndpoints.id, action.endpointId),
),
)
.then((rows) => rows[0] ?? null),
]);
if (
!conversation ||
!publication ||
!endpoint ||
endpoint.provider !== "telegram" ||
publication.conversationId !== conversation.id ||
publication.payload.interactionId !== token.interactionId
) {
await db
.update(chatActions)
.set({
status: "failed",
result: { code: "confirmation_action_binding_invalid" },
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "processing"),
),
);
return false;
}
const interaction = (
await issueThreadInteractionService(db).listForIssue(conversation.issueId)
).find((candidate) => candidate.id === token.interactionId);
if (
!interaction ||
interaction.kind !== "request_confirmation" ||
!nativeTelegramConfirmation(interaction)
) {
await db
.update(chatActions)
.set({
status: "failed",
result: { code: "confirmation_interaction_invalid" },
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "processing"),
),
);
return false;
}
if (
interaction.status !== "accepted" &&
interaction.status !== "rejected"
) {
if (interaction.status === "pending") return false;
const [retired] = await db
.update(chatActions)
.set({
status: "expired",
result: {
code: "confirmation_interaction_resolved_elsewhere",
interactionId: interaction.id,
interactionStatus: interaction.status,
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "processing"),
),
)
.returning({ id: chatActions.id });
if (retired) return true;
const winner = await db
.select({ status: chatActions.status })
.from(chatActions)
.where(eq(chatActions.id, action.id))
.then((rows) => rows[0] ?? null);
return winner?.status === "expired" || winner?.status === "processed";
}
const decisionMatches =
(token.decision === "accept" && interaction.status === "accepted") ||
(token.decision === "reject" && interaction.status === "rejected");
if (!decisionMatches) {
await db
.update(chatActions)
.set({
status: "expired",
result: { code: "interaction_resolved_by_sibling_action" },
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "processing"),
),
);
return false;
}
const issue = await db
.select({
id: issues.id,
companyId: issues.companyId,
assigneeAgentId: issues.assigneeAgentId,
status: issues.status,
})
.from(issues)
.where(
and(
eq(issues.companyId, action.companyId),
eq(issues.id, conversation.issueId),
),
)
.then((rows) => rows[0] ?? null);
if (!issue || !interaction.resolvedByUserId) {
await db
.update(chatActions)
.set({
status: "failed",
result: { code: "confirmation_resolution_context_missing" },
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "processing"),
),
);
return false;
}
const resolvedByUserId = interaction.resolvedByUserId;
const shouldWake =
Boolean(issue.assigneeAgentId) &&
issue.status !== "done" &&
issue.status !== "cancelled" &&
(interaction.continuationPolicy === "wake_assignee" ||
(interaction.continuationPolicy === "wake_assignee_on_accept" &&
interaction.status === "accepted"));
const resolutionLabel =
interaction.status === "accepted" ? "Accepted" : "Rejected";
const resolvedByThisProviderAction =
action.result?.code === "confirmation_resolution_committed_by_provider";
return db.transaction(async (tx) => {
const [completed] = await tx
.update(chatActions)
.set({
status: "processed",
result: {
code: resolvedByThisProviderAction
? "confirmation_resolution_committed_by_provider"
: "confirmation_resolution_reconciled",
interactionId: interaction.id,
interactionStatus: interaction.status,
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "processing"),
),
)
.returning({ id: chatActions.id });
if (!completed) {
// A publication sweep may reconcile the durable action while the
// original provider callback is still in flight. Treat that CAS loss
// as success once the winner has processed the same action so the
// provider does not receive a spurious 5xx and redeliver it.
const winner = await tx
.select({ status: chatActions.status })
.from(chatActions)
.where(eq(chatActions.id, action.id))
.then((rows) => rows[0] ?? null);
return winner?.status === "processed";
}
await tx
.update(chatActions)
.set({
status: "expired",
result: { code: "interaction_resolved_by_sibling_action" },
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.companyId, issue.companyId),
eq(chatActions.endpointId, action.endpointId),
eq(chatActions.conversationId, conversation.id),
eq(chatActions.kind, "confirmation_response"),
inArray(chatActions.status, ["issued", "processing"]),
ne(chatActions.id, action.id),
eq(
sql<string>`${chatActions.payload}->>'interactionId'`,
interaction.id,
),
),
);
await notifyChatPublicationWork(tx);
await tx
.insert(chatPublications)
.values({
companyId: issue.companyId,
endpointId: action.endpointId,
conversationId: conversation.id,
issueId: issue.id,
idempotencyKey: `interaction-resolution:${interaction.id}:${action.endpointId}`,
payload: projectSafeChatPublication({
classification: "external",
source: "issue_interaction",
text: `${resolutionLabel}: ${interaction.payload.prompt}`,
interaction: {
id: interaction.id,
card: {
kind: "confirmation",
title: interaction.payload.prompt,
body: resolutionLabel,
actions: [],
},
},
}),
state: "pending",
})
.onConflictDoNothing();
if (shouldWake && issue.assigneeAgentId) {
await tx
.insert(chatActions)
.values({
companyId: issue.companyId,
endpointId: action.endpointId,
conversationId: conversation.id,
principalId: action.principalId,
kind: "interaction_wakeup",
providerActionId: `interaction_wakeup:${interaction.id}`,
payload: {
version: 1,
interactionId: interaction.id,
interactionKind: interaction.kind,
interactionStatus: interaction.status,
issueId: issue.id,
agentId: issue.assigneeAgentId,
sourceCommentId: interaction.sourceCommentId ?? null,
sourceRunId: interaction.sourceRunId ?? null,
requestedByUserId: resolvedByUserId,
requestedByActorType: "user",
requestedByActorId: resolvedByUserId,
},
status: "issued",
})
.onConflictDoNothing();
}
// A provider callback can lose the pending -> terminal interaction race
// to the Paperclip UI. In that case this transaction only retires the
// now-redundant provider action and must not fabricate a second external
// resolution event attributed to the board winner.
if (resolvedByThisProviderAction) {
await logActivity(tx as unknown as Db, {
companyId: issue.companyId,
actorType: "user",
actorId: resolvedByUserId,
action:
interaction.status === "accepted"
? "issue.thread_interaction_accepted"
: "issue.thread_interaction_rejected",
entityType: "issue",
entityId: issue.id,
details: {
source: "external_chat",
endpointId: action.endpointId,
provider: endpoint.provider,
conversationId: conversation.id,
publicationId: publication.id,
providerMessageId: publication.providerMessageId,
interactionId: interaction.id,
interactionKind: interaction.kind,
interactionStatus: interaction.status,
resolutionActorKind: "user",
requestedResolverPolicy: interaction.requestedResolverPolicy,
effectiveResolverPolicy: interaction.effectiveResolverPolicy,
},
});
}
return true;
});
}
async function reconcileTerminalConfirmationActions(
limit = 100,
): Promise<number> {
const candidates = await db
.select({ id: chatActions.id })
.from(chatActions)
.where(
and(
eq(chatActions.kind, "confirmation_response"),
eq(chatActions.status, "processing"),
),
)
.orderBy(asc(chatActions.updatedAt), asc(chatActions.id))
.limit(limit);
let settled = 0;
for (const candidate of candidates) {
if (await settleTerminalConfirmationAction(candidate.id)) settled += 1;
}
return settled;
}
return { settleTerminalConfirmationAction, reconcileTerminalConfirmationActions, handleTelegramConfirmation };
}
@@ -0,0 +1,62 @@
import type { Thread } from "chat";
import { and, eq, ne, sql } from "drizzle-orm";
import { chatEndpointResources } from "@paperclipai/db";
import type { EndpointRow, DbTransaction, DbOrTransaction } from "../runtime-types.js";
import { nonDirectDestinationAllowed } from "../inbound/surfaces.js";
type DestinationAdmissionInput = {
endpoint: EndpointRow;
currentEndpoint: Pick<EndpointRow, "status" | "allowGroupChats">;
thread: Thread;
addressed: boolean;
};
type DestinationAdmissionOptions = {
ensureResource: (endpoint: EndpointRow, thread: Thread, enableForSetup: boolean, database?: DbOrTransaction) => Promise<typeof chatEndpointResources.$inferSelect>;
};
export function createTelegramDestinationAdmission(options: DestinationAdmissionOptions) {
const { ensureResource } = options;
async function admitTelegramDestination(input: DestinationAdmissionInput, tx: DbTransaction) {
const { endpoint, currentEndpoint, thread, addressed } = input;
let destinationAccepting = true;
let resource = await ensureResource(endpoint, thread, false, tx);
const enabledResourceCount = await tx
.select({ count: sql<number>`count(*)::int` })
.from(chatEndpointResources)
.where(
and(
eq(chatEndpointResources.endpointId, endpoint.id),
eq(chatEndpointResources.enabled, true),
eq(chatEndpointResources.availability, "available"),
ne(chatEndpointResources.type, "direct_message"),
ne(chatEndpointResources.type, "group_chat"),
),
)
.then((rows) => rows[0]?.count ?? 0);
const canActivateFirstSetupGroup =
currentEndpoint.status === "verifying" &&
resource.availability === "available" &&
enabledResourceCount === 0 &&
addressed;
if (canActivateFirstSetupGroup && !resource.enabled) {
resource = await tx
.update(chatEndpointResources)
.set({ enabled: true, updatedAt: new Date() })
.where(eq(chatEndpointResources.id, resource.id))
.returning()
.then((rows) => rows[0] ?? resource);
}
// Telegram's privacy-mode contract is explicit: in a group or forum
// topic, every admitted turn must address the bot. A direct reply to a
// bot message is normalized as a mention by the pinned adapter, while
// unrelated subscribed traffic remains unaddressed and is filtered.
destinationAccepting =
addressed &&
nonDirectDestinationAllowed(
{ ...endpoint, allowGroupChats: currentEndpoint.allowGroupChats },
resource,
);
return destinationAccepting;
}
return { admitTelegramDestination };
}
@@ -0,0 +1,192 @@
import { createChatPublicationReplacement } from "../publication-replacement.js";
import { notifyChatPublicationWork } from "../../chat-work-notifications.js";
import {
eq,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatPublications,
} from "@paperclipai/db";
import type {
SafeChatPublicationPayload,
} from "@paperclipai/shared";
import {
splitTelegramPublicationText,
telegramMarkdownRequiresAttachment,
} from "../../chat-publication-stream.js";
type PublicationPreparationOptions = {
runPublicationToReplace: ReturnType<typeof createChatPublicationReplacement>["runPublicationToReplace"];
};
export function createTelegramPublicationPreparation(db: Db, options: PublicationPreparationOptions) {
const { runPublicationToReplace } = options;
async function prepareTelegramPublication(publication: typeof chatPublications.$inferSelect, persisted: SafeChatPublicationPayload): Promise<typeof chatPublications.$inferSelect> {if (
persisted.transportPart ||
persisted.card ||
persisted.interactionId ||
persisted.attachmentIds?.length
) {
return publication;
}
if (telegramMarkdownRequiresAttachment(persisted.text)) {
// Telegram parses each message independently. A fixed-size split can
// turn the second half of a code fence, link, or list into unrelated
// plain text even though concatenating the source parts is lossless.
// Preserve structured long Markdown as one document. When this is a run
// final, first complete its existing progress placeholder, then send the
// file through a separate ordered outbox row because Telegram cannot add
// an attachment while editing that placeholder.
const replacesProgressMessage = Boolean(
await runPublicationToReplace(publication, persisted),
);
return db.transaction(async (tx) => {
const current = await tx
.select()
.from(chatPublications)
.where(eq(chatPublications.id, publication.id))
.for("update")
.then((rows) => rows[0] ?? null);
if (!current) return publication;
const currentPayload = current.payload as SafeChatPublicationPayload;
if (
currentPayload.transportPart ||
currentPayload.card ||
currentPayload.interactionId ||
currentPayload.attachmentIds?.length ||
!["pending", "retry"].includes(current.state) ||
!telegramMarkdownRequiresAttachment(currentPayload.text)
) {
return current;
}
const batchId = current.id;
const updatedAt = new Date();
const transportPart = (
index: number,
count: number,
mode: "inline" | "telegram_markdown_attachment",
) => ({
batchId,
count,
index,
mode,
orderKey: `${batchId}:${String(index).padStart(4, "0")}`,
});
if (!replacesProgressMessage) {
const attachmentPayload: SafeChatPublicationPayload = {
...currentPayload,
transportPart: transportPart(0, 1, "telegram_markdown_attachment"),
};
await notifyChatPublicationWork(tx);
await tx
.update(chatPublications)
.set({ payload: attachmentPayload, updatedAt })
.where(eq(chatPublications.id, current.id));
return { ...current, payload: attachmentPayload, updatedAt };
}
const handoffPayload: SafeChatPublicationPayload = {
...currentPayload,
text: "This response needs a separate attachment because it exceeds the message limit.",
transportPart: transportPart(0, 2, "inline"),
};
delete handoffPayload.attachmentIds;
await notifyChatPublicationWork(tx);
await tx
.update(chatPublications)
.set({ payload: handoffPayload, updatedAt })
.where(eq(chatPublications.id, current.id));
const attachmentPayload: SafeChatPublicationPayload = {
...currentPayload,
transportPart: transportPart(1, 2, "telegram_markdown_attachment"),
};
delete attachmentPayload.progressState;
await notifyChatPublicationWork(tx);
await tx.insert(chatPublications).values({
companyId: current.companyId,
endpointId: current.endpointId,
conversationId: current.conversationId,
issueId: current.issueId,
commentId: current.commentId,
idempotencyKey: `telegram-markdown-attachment:${current.id}`,
payload: attachmentPayload,
state: "pending",
createdAt: sql`(select ${chatPublications.createdAt} from ${chatPublications} where ${chatPublications.id} = ${current.id})`,
updatedAt,
});
return { ...current, payload: handoffPayload, updatedAt };
});
}
const parts = splitTelegramPublicationText(persisted.text);
if (parts.length === 1) return publication;
return db.transaction(async (tx) => {
const current = await tx
.select()
.from(chatPublications)
.where(eq(chatPublications.id, publication.id))
.for("update")
.then((rows) => rows[0] ?? null);
if (!current) return publication;
const currentPayload = current.payload as SafeChatPublicationPayload;
if (
currentPayload.transportPart ||
currentPayload.card ||
currentPayload.interactionId ||
currentPayload.attachmentIds?.length ||
!["pending", "retry"].includes(current.state)
) {
return current;
}
const currentParts = splitTelegramPublicationText(currentPayload.text);
if (currentParts.length === 1) return current;
const batchId = current.id;
const updatedAt = new Date();
const transportPart = (index: number) => ({
batchId,
count: currentParts.length,
index,
orderKey: `${batchId}:${String(index).padStart(4, "0")}`,
});
const rootPayload: SafeChatPublicationPayload = {
...currentPayload,
text: currentParts[0]!,
transportPart: transportPart(0),
};
await notifyChatPublicationWork(tx);
await tx
.update(chatPublications)
.set({ payload: rootPayload, updatedAt })
.where(eq(chatPublications.id, current.id));
const { progressState: _progressState, ...nonProgressPayload } =
currentPayload;
await notifyChatPublicationWork(tx);
await tx.insert(chatPublications).values(
currentParts.slice(1).map((text, offset) => {
const index = offset + 1;
return {
companyId: current.companyId,
endpointId: current.endpointId,
conversationId: current.conversationId,
issueId: current.issueId,
commentId: current.commentId,
idempotencyKey: `${current.idempotencyKey}:telegram-part:${index}`,
payload: {
...nonProgressPayload,
text,
transportPart: transportPart(index),
},
state: "pending" as const,
createdAt: sql`(select ${chatPublications.createdAt} from ${chatPublications} where ${chatPublications.id} = ${current.id})`,
updatedAt,
};
}),
);
return { ...current, payload: rootPayload, updatedAt };
});}
return { prepareTelegramPublication };
}
@@ -0,0 +1,110 @@
import { createTelegramMaintenance } from "./maintenance.js";
import type { EndpointRow, RuntimeContext } from "../runtime-types.js";
import { type CredentialMutationLeaseGuard } from "../../chat-credential-mutation-lease.js";
import type { Db } from "@paperclipai/db";
import type {
ConfigureChatEndpointInput,
} from "@paperclipai/shared";
import {
unprocessable,
} from "../../../errors.js";
import type { EndpointRecord } from "../runtime-types.js";
type TelegramSetupOptions = {
fetchImpl: typeof globalThis.fetch;
getWebhookPublicBaseUrl: () => string | null;
PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS: number;
runtimeContextForRecord: (record: EndpointRecord) => RuntimeContext;
recordTelegramStopSubscription: ReturnType<typeof createTelegramMaintenance>["recordTelegramStopSubscription"];
stageTelegramMaintenance: ReturnType<typeof createTelegramMaintenance>["stageTelegramMaintenance"];
processTelegramMaintenance: ReturnType<typeof createTelegramMaintenance>["processTelegramMaintenance"];
};
export function createTelegramSetup(db: Db, options: TelegramSetupOptions) {
const { fetchImpl, getWebhookPublicBaseUrl, PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS, runtimeContextForRecord, recordTelegramStopSubscription, stageTelegramMaintenance, processTelegramMaintenance } = options;
async function registerTelegramWebhook(endpoint: EndpointRow, input: ConfigureChatEndpointInput, credentials: Record<string, string>, credentialLease: CredentialMutationLeaseGuard, next: EndpointRecord) {
const webhookUrl = `${getWebhookPublicBaseUrl()}/api/chat-webhooks/${endpoint.publicId}/telegram`;
const infoResponse = await fetchImpl(
`https://api.telegram.org/bot${encodeURIComponent(credentials.botToken)}/getWebhookInfo`,
{ signal: AbortSignal.timeout(PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS) },
);
const infoResult = (await infoResponse.json()) as {
ok?: boolean;
description?: string;
result?: { url?: string };
};
if (!infoResponse.ok || !infoResult.ok) {
throw unprocessable(
`Telegram could not inspect the existing webhook: ${infoResult.description ?? infoResponse.status}`,
);
}
await credentialLease.assertOwned();
const response = await fetchImpl(
`https://api.telegram.org/bot${encodeURIComponent(credentials.botToken)}/setWebhook`,
{
method: "POST",
signal: AbortSignal.timeout(PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS),
headers: { "content-type": "application/json" },
body: JSON.stringify({
url: webhookUrl,
secret_token: credentials.webhookSecret,
allowed_updates: [
"message",
"edited_message",
"callback_query",
"stopped_message_generation",
"message_reaction",
"my_chat_member",
],
// A first-time connection may inherit stale updates from a bot
// that was used before Paperclip owned it. Reconnect is
// different: Telegram may be holding legitimate updates while
// the old callback URL is unavailable. Preserve that backlog
// even when an operator is rotating the public origin.
drop_pending_updates: input.action === "configure",
}),
},
);
const result = (await response.json()) as {
ok?: boolean;
description?: string;
result?: unknown;
};
if (!response.ok || !result.ok)
throw unprocessable(
`Telegram could not register the webhook: ${result.description ?? response.status}`,
);
await credentialLease.assertOwned();
if (result.ok === true && result.result === true) {
// Telegram's successful boolean receipt acknowledges the exact URL,
// secret and allowed_updates request above. GET alone cannot prove
// secret_token, which Telegram deliberately does not return.
await recordTelegramStopSubscription(
next.endpoint,
runtimeContextForRecord(next),
credentialLease,
);
}
// Command discovery is optional for ingress, but its provider mutation
// is still a durable, auditable action. A transient or rate-limited
// failure is retried by the maintenance worker without blocking the
// otherwise-valid webhook connection.
const commandAction = await stageTelegramMaintenance(
endpoint,
"register_commands",
);
if (commandAction)
await processTelegramMaintenance(
commandAction.id,
false,
credentialLease,
);
}
return { registerTelegramWebhook };
}