refactor(chat): extract delivery, reaction, and setup boundaries (#16024)

## Thinking Path

> - Paperclip manages AI agents and their work.
> - Chat channels connect that work to provider conversations.
> - One shared service contains callback, delivery, reaction, and setup
logic.
> - These responsibilities have distinct contracts and existing
public-path tests.
> - This PR moves ten cohesive responsibilities into focused modules.
> - The service retains shared runtime ownership and scheduling.
> - Provider behavior is easier to locate and review.

## Linked Issues or Issue Description

Refs #15995, #16026, #16027, #16030, #16031, #16035, #16037, #16038,
#16041, #16043.

This PR consolidates the reviewed extraction stack. The plan is
`doc/plans/2026-10-12-chat-channel-delivery-boundaries-refactor.md`.

## What Changed

- Move Slack callback verification into the Slack folder.
- Move Discord Gateway events and root-thread admission/recovery into
the Discord folder.
- Extract durable receipt reactions and inbound reaction
admission/replay.
- Extract inbound attachment ingestion and outbound integrity-checked
hydration. Preserve the public hydration export.
- Extract durable provider effects. Keep Telegram callback notices in
the Telegram folder.
- Extract conversation/activity reads and setup qualification. Keep
Slack and Teams projections in their channel folders.
- Preserve the merged managed Slack setup behavior, shared workers,
credential leases, and transaction boundaries.
- Reduce `chat-channels.ts` from 26,493 lines / 1,048,769 bytes to
21,460 lines / 862,284 bytes. This removes 5,033 lines from the service.

## Verification

- Each extraction passed its selected existing tests before and after
through public service paths.
- On updated master, all 55 selected Slack cases passed before
extraction. All 194 selected chat/managed-Slack cases passed on the
final stack tip.
- `pnpm -r typecheck`, `pnpm build`, module-boundary checks, and
whitespace checks passed on that tip.
- The normalized syntax audit verifies 69 moved declarations and all
retained statements, including their relative order, against updated
master.
- The eight earlier stack merges retain the exact validated repository
tree. The combined root equals the validated tip merged with current
master. All chat-channel files remain byte-identical to the validated
tip. Current combined-root CI and review are pending.
- Full local verification remains unverified. The unchanged
`workspace-git-snapshot-streaming.test.ts:56` stress test timed out at
300,000 ms in a clean run and again in isolation. The full run was
stopped after the reproduced failure. No timeout or assertion was
weakened.
- These checks use assembled services, real local databases, and
provider fixtures. They do not prove live third-party execution.

Reviewers can run the public-path suites:

```sh
pnpm exec vitest run server/src/__tests__/chat-channels.integration.test.ts server/src/services/chat-outbound-attachment-hydration.test.ts server/src/__tests__/slack-development-manager.test.ts
```

## Risks

- Factory wiring must preserve live values, shared state, and
initialization order. Identity-link construction retains deferred access
to the provider-effects callback.
- Credential fences, authorization transactions, and delivery settlement
must keep their original ordering. Syntax parity and public-path
integration cases cover these boundaries.
- Full local verification has the recorded Git-streaming timeout. The
combined PR must pass master’s required CI before merging into master.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and repository tools. The
exact runtime model ID and context-window size are not exposed in this
session. No separate agent was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-12 22:46:25 +00:00
1 parent 9b1b905e01
commit 64bd6e9e07
19 files changed
+5877 -5071

No files matched your search

@@ -0,0 +1,100 @@
# Chat channel callback and delivery boundaries, batch four
Base: 4b766b872993045a2e787404f0475503683e5ded. Batch three (#15995, #15996, #15999, #16001, #16002)
is merged. The service starts at 26,461 lines / 1,046,711 bytes.
This batch may create up to ten stacked PRs. Merging the new batch is separate work.
| Order / boundary | Destination | Approx. moved / net lines | Contract and state | Risks and existing verification |
| --- | --- | --- | --- | --- |
| 1. Slack callback verification | `slack/callback-verification.ts` | 250 / 230 | Pure signature/workspace/URL inspection plus signed URL verification; explicit vault lookup, URL getter, scheduling and registration callbacks | Preserve signing-secret-only setup, original request, TLS/proxy evidence and workspace checks. Signed verification and callback health cases before/after. |
| 2. Discord Gateway events | `discord/gateway-events.ts` | 200 / 185 | Runtime fence and lease-owner predicates, endpoint lookup and lifecycle callback; original per-runtime event queue and sequence | Preserve stale-owner fencing, event order, transient guild recovery and terminal revocation. Gateway lifecycle cases before/after. |
| 3. Discord root threads | `discord/root-threads.ts` | 500 / 460 | Credential lease, current principal authority, runtime, message admission and conversation drain callbacks | Preserve uncertain thread creation, restart reconciliation, no duplicate starter and pause/credential serialization. Root-thread public-service cases. |
| 4. Receipt reactions | `receipt-reactions.ts` and Slack error classifier beside Slack messages | 545 / 500 | Durable reaction claim/transport/settlement with runtime and credential fences; no scheduling owner moves | Preserve retries, legacy GitHub receipts, terminal-removal supersession and same-attempt settlement. Receipt recovery cases before/after. |
| 5. Inbound reactions | `reactions.ts` | 800 / 740 | Conversation lookup/locks, reaction admission and deferred replay together; current principal/runtime callbacks | Preserve exact event dedupe, completed generation links, bounded replay age and no task wakeup. Reaction race/restart/authorization cases. |
| 6. Inbound attachments | `inbound-attachments.ts` | 600 / 550 | Existing storage and issue service, retained source authorization and live options; return same attachment results | Preserve size limits, source authority around I/O, omissions and text delivery. Provider attachment/restart/stale-authority cases. |
| 7. Outbound attachments | `outbound-attachments.ts`; provider Markdown wrappers beside provider messages | 245 / 220 | Metadata validation and bounded integrity-checked hydration; existing storage object. Keep public hydration export compatible | Preserve stream cleanup, deadlines, byte/digest checks and retry classification. Hydration and publication attachment cases. |
| 8. Provider effects | `provider-effects.ts` | 950 / 850 | Durable reply claim, authorization, transport and settlement; endpoint/runtime/credential/principal callbacks | Preserve uncertain-delivery quarantine, credential fence and transaction ordering. Native callback, task-control and reply recovery cases. Put Telegram-only authority under `telegram/` if its contract is narrow. |
| 9. Conversation and activity reads | `activity.ts` | 540 / 520 | Endpoint lookup and read-only Teams conflict projection; return existing API models | Preserve company/endpoint joins, mixed-source cursor ties, replay eligibility and read-only GETs. Activity, file receipt projection and conversation-state cases. |
| 10. Setup probes | `setup-probes.ts` | 280 / 250 | Credential lease, current identity/principal authority, provider fetch, endpoint projection and live activation hook | Preserve real-message requirement, generation and linked-user activation. Setup-test, reconnect and authorization cases. |
Each slice must earn its place after reading its complete path and callers.
Reassess before implementation; stop short of ten if the remaining contracts are
not cohesive. Keep provider-specific code in channel folders and common durable
workflows in the shared chat-channels directory.
Receiver lease ownership stays shared: the existing Discord-named owner is also
used by Photon. Shared lifecycle/application admission, queue-drain ownership,
worker scheduling, and the largest action/message handlers remain in the service.
Do not hide their dependencies in a mutable context object. Preserve live option
hooks, runtime-map identity, partial-failure cleanup, and initialization order.
Run relevant existing public-service tests before and after each mechanical move.
Compare moved declarations and remaining statements with normalized syntax, then
run server types, module boundaries and whitespace checks. Finish local verification and a fresh
5/5 review before opening the next draft PR. GitHub runner-selection jobs remained
queued without steps for over 25 minutes in this batch. To continue the authorized
stack, publish drafts in review order while tracking CI as pending; CI remains a
merge gate. Run cumulative repository gates at handoff.
Retain failures and skipped/unrun coverage even when an isolated retry passes.
Use fixture-backed assembled service/database tests; do not claim live provider
execution. Remap LOC/bytes and the remaining orchestration after the batch.
## Slice 1: Slack callback verification
- Service: 26,461 lines / 1,046,711 bytes → 26,201 lines / 1,035,260 bytes.
- Destination: 299 lines / 13,038 bytes. Eight declarations move without body
changes. Three existing setup types move unchanged into `runtime-types.ts`.
- The verification factory initializes after Slack registration. It receives the
same registration object, credential resolver, live URL getter and scheduler.
Signature/workspace/URL helpers remain callable by shared webhook orchestration.
- Nine public-service cases passed before and after on the updated master base.
Server typecheck, module boundaries, whitespace and normalized syntax parity
passed. Coverage includes signed URL verification, account notification,
workspace rejection, callback surfaces, TLS termination and Cloud host evidence.
- Next: Discord Gateway event sequencing and health transitions. Keep shared
receiver lease ownership and runtime-map lifetime in the service.
## Batch outcome
All ten candidates produced cohesive boundaries. Shared receiver ownership, queue
management and worker scheduling remain in the service. The only intentional
control-wiring change is deferred access to `stageProviderEffect` from identity
link construction; typecheck and integration tests caught the initial early read.
| Slice | Service lines after | Primary module lines | Before/after cases |
| --- | ---: | ---: | ---: |
| 1. Slack signature, workspace and callback url verification | 26,233 | 300 | 9 |
| 2. Discord gateway event sequencing and endpoint health transitions | 26,029 | 253 | 7 |
| 3. Discord root-thread admission, provider creation and durable recovery | 25,527 | 570 | 4 |
| 4. Durable receipt reaction claims, transport, retries and settlement | 24,975 | 586 | 7 |
| 5. Inbound reaction conversation lookup, admission and deferred replay | 24,160 | 891 | 21 |
| 6. Inbound attachment authorization, bounded ingestion, registration and omissions | 23,565 | 674 | 39 |
| 7. Outbound attachment metadata validation and bounded integrity-checked hydration | 23,298 | 294 | 22 |
| 8. Durable provider reply staging, authorization, transport and recovery | 22,331 | 957 | 24 |
| 9. Conversation summaries and paginated activity projections | 21,741 | 616 | 16 |
| 10. Setup message detection and credential-fenced round-trip qualification | 21,460 | 353 | 7 |
After rebasing on master `8e7b5db822`, the service shrank from
**26,493 lines / 1,048,769 bytes** to **21,460 lines /
862,284 bytes** (5,033 fewer lines, about 19%). Normalized syntax comparison
checks all 69 moved declarations/types and every remaining original statement.
Server types, targeted tests, module boundaries and whitespace checks are the
per-slice gates. Cumulative repository gates and remote reviews are recorded in
the PR descriptions as they complete.
Orchestration measurement: the service function has 150 nested functions
(19,399 lines) and 67 state/wiring statements (459 lines), versus 191 functions
(23,829 lines) and 54 statements (425 lines) at the start. Imports take 400 lines.
The remaining large paths are `processMessage` (2,166), `handleAction` (1,074),
`configureWithCredentialLease` (868), `processSelectedPublication` (729),
`handleModalSubmit` (708), `handleWebhook` (685), and lifecycle processing (657).
A future pass should inspect publication admission/transport/settlement as one
workflow, then inbound delivery authorization and drain ownership. Those cuts
need explicit ownership contracts; avoid separating individual checks from their
transactions merely to reduce the remaining line count.
The rebase preserves the merged managed Slack setup flow, including its shared
`isProvisionedSlackSetup` predicate in the extracted callback verification
module. A fresh syntax audit compares all moved declarations and retained
statement ordering with this updated master.
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,616 @@
import { providerEffectPayload } from "./provider-effects.js";
import { slackSlashTaskRecoveryPayload } from "./slack/task-start.js";
import { GITHUB_RECOVERY_STATE_KEY } from "./github/webhook-ingress.js";
import {
and,
desc,
eq,
inArray,
sql,
} from "drizzle-orm";
import { alias, type AnyPgColumn } from "drizzle-orm/pg-core";
import type { Db } from "@paperclipai/db";
import {
chatActions,
chatConversations,
chatDeliveries,
chatPublications,
chatSdkState,
chatTeamsFileTransfers,
issues,
} from "@paperclipai/db";
import type {
ChatPublicationSummary,
SafeChatPublicationPayload,
} from "@paperclipai/shared";
import {
teamsFileTransferService,
} from "../chat-teams-file-transfers.js";
import {
chatFileTransferResolutionActions,
projectChatFileTransfer,
} from "../chat-publication-batches.js";
import {
badRequest,
notFound,
} from "../../errors.js";
import { chatProviderConversationUrl } from "../chat-provider-links.js";
import type { EndpointRecord } from "./runtime-types.js";
import { slackTaskStartActivityStatus } from "./slack/task-start.js";
import { fileTransferProjectionColumns } from "./microsoft-teams/file-transfer-projection.js";
function publicationSummary(
row: typeof chatPublications.$inferSelect,
): ChatPublicationSummary {
return {
id: row.id,
state: row.state,
providerUrl: row.providerUrl,
attempts: row.attempts,
redactedError: row.redactedError,
nextAttemptAt: row.nextAttemptAt?.toISOString() ?? null,
publishedAt: row.publishedAt?.toISOString() ?? null,
};
}
export { publicationSummary };
type ChatActivityOptions = {
endpointRecord: (endpointId: string) => Promise<EndpointRecord | null>;
teamsTransferProtocol: () => Pick<ReturnType<typeof teamsFileTransferService>, "canCancelConflict">;
};
export function createChatActivity(db: Db, options: ChatActivityOptions) {
const { endpointRecord, teamsTransferProtocol } = options;
async function listConversations(endpointId: string) {
const record = await endpointRecord(endpointId);
if (!record) throw notFound("Chat endpoint not found");
const conversations = await db
.select()
.from(chatConversations)
.where(eq(chatConversations.endpointId, endpointId))
.orderBy(
desc(chatConversations.lastActivityAt),
desc(chatConversations.createdAt),
);
if (!conversations.length) return [];
const issueRows = await db
.select({
id: issues.id,
identifier: issues.identifier,
title: issues.title,
status: issues.status,
updatedAt: issues.updatedAt,
})
.from(issues)
.where(
inArray(
issues.id,
conversations.map((row) => row.issueId),
),
);
const publications = await db
.select()
.from(chatPublications)
.where(eq(chatPublications.endpointId, endpointId))
.orderBy(desc(chatPublications.createdAt));
const issueById = new Map(issueRows.map((row) => [row.id, row]));
const lastPublication = new Map<
string,
typeof chatPublications.$inferSelect
>();
for (const publication of publications)
if (!lastPublication.has(publication.conversationId))
lastPublication.set(publication.conversationId, publication);
return conversations.map((conversation) => {
const issue = issueById.get(conversation.issueId);
const externalUrl =
record.endpoint.provider === "telegram" && conversation.isDirectMessage
? chatProviderConversationUrl({
provider: "telegram",
botUsername: record.endpoint.botUsername,
threadId: conversation.externalThreadId,
providerMessageId: "",
raw: { chat: { type: "private" } },
})
: (conversation.providerUrl ??
(record.endpoint.provider === "slack"
? chatProviderConversationUrl({
provider: "slack",
providerAccountId: record.endpoint.providerAccountId,
threadId: conversation.externalThreadId,
providerMessageId: "",
})
: null));
return {
id: conversation.id,
companyId: conversation.companyId,
endpointId: conversation.endpointId,
resourceId: conversation.resourceId,
externalLabel: conversation.externalLabel,
externalUrl,
externalConversationId: conversation.externalConversationId,
externalThreadId: conversation.externalThreadId,
sessionGeneration: conversation.sessionGeneration,
issueId: conversation.issueId,
issueIdentifier: issue?.identifier ?? null,
issueTitle: issue?.title ?? null,
isDirectMessage: conversation.isDirectMessage,
state:
record.endpoint.provider !== "imessage-photon" &&
(issue?.status === "done" || issue?.status === "cancelled")
? "completed"
: conversation.state,
lastActivityAt: conversation.lastActivityAt?.toISOString() ?? null,
createdAt: conversation.createdAt.toISOString(),
updatedAt: (
conversation.lastActivityAt ?? conversation.updatedAt
).toISOString(),
lastPublicationStatus:
lastPublication.get(conversation.id)?.state ?? null,
};
});
}
async function listActivity(endpointId: string, page?: { limit: number; before?: { createdAt: string; id: string } }) {
const limit = page ? page.limit + 1 : 100;
// Match the millisecond precision of the public timestamp and use a UUID
// tie-breaker so equal timestamps never skip records between pages.
const before = (date: AnyPgColumn, id: AnyPgColumn) => page?.before
? sql`(date_trunc('milliseconds', ${date}), ${id}) < (${page.before.createdAt}::timestamptz, ${page.before.id}::uuid)`
: undefined;
const recoveryIngress = alias(chatActions, "github_recovery_ingress");
const [
deliveries,
publications,
slashActions,
providerEffects,
githubIngressFailures,
slackSessionActions,
githubRecoveries,
githubRecoveryScans,
] = await Promise.all([
db
.select()
.from(chatDeliveries)
.where(and(eq(chatDeliveries.endpointId, endpointId), before(chatDeliveries.createdAt, chatDeliveries.id)))
.orderBy(desc(sql`date_trunc('milliseconds', ${chatDeliveries.createdAt})`), desc(chatDeliveries.id))
.limit(limit),
db
.select()
.from(chatPublications)
.where(and(eq(chatPublications.endpointId, endpointId), before(chatPublications.createdAt, chatPublications.id)))
.orderBy(desc(sql`date_trunc('milliseconds', ${chatPublications.createdAt})`), desc(chatPublications.id))
.limit(limit),
db
.select()
.from(chatActions)
.where(
and(
eq(chatActions.endpointId, endpointId),
before(chatActions.createdAt, chatActions.id),
eq(chatActions.kind, "slash_task_start"),
),
)
.orderBy(desc(sql`date_trunc('milliseconds', ${chatActions.createdAt})`), desc(chatActions.id))
.limit(limit),
db
.select()
.from(chatActions)
.where(
and(
eq(chatActions.endpointId, endpointId),
before(chatActions.createdAt, chatActions.id),
eq(chatActions.kind, "provider_effect"),
eq(chatActions.status, "delivery_unknown"),
),
)
.orderBy(desc(sql`date_trunc('milliseconds', ${chatActions.createdAt})`), desc(chatActions.id))
.limit(limit),
db
.select()
.from(chatActions)
.where(
and(
eq(chatActions.endpointId, endpointId),
before(chatActions.createdAt, chatActions.id),
eq(chatActions.kind, "github_webhook_ingress"),
eq(chatActions.status, "failed"),
sql`coalesce(${chatActions.result}->>'retryable', 'false') = 'false'`,
),
)
.orderBy(desc(sql`date_trunc('milliseconds', ${chatActions.createdAt})`), desc(chatActions.id))
.limit(limit),
db
.select()
.from(chatActions)
.where(
and(
eq(chatActions.endpointId, endpointId),
before(chatActions.updatedAt, chatActions.id),
inArray(chatActions.kind, [
"slack_session_sync",
"slack_session_stop",
]),
),
)
.orderBy(desc(sql`date_trunc('milliseconds', ${chatActions.updatedAt})`), desc(chatActions.id))
.limit(limit),
db
.select({
action: chatActions,
ingressStatus: recoveryIngress.status,
})
.from(chatActions)
.leftJoin(
recoveryIngress,
and(
eq(recoveryIngress.companyId, chatActions.companyId),
eq(recoveryIngress.endpointId, chatActions.endpointId),
eq(recoveryIngress.kind, "github_webhook_ingress"),
sql`${recoveryIngress.providerActionId} = 'github_webhook_ingress:' || (${chatActions.payload}#>>'{original,guid}')`,
),
)
.where(
and(
eq(chatActions.endpointId, endpointId),
before(chatActions.updatedAt, chatActions.id),
eq(chatActions.kind, "github_webhook_recovery"),
),
)
.orderBy(desc(sql`date_trunc('milliseconds', ${chatActions.updatedAt})`), desc(chatActions.id))
.limit(limit),
db
.select()
.from(chatSdkState)
.where(
and(
eq(chatSdkState.endpointId, endpointId),
before(chatSdkState.updatedAt, chatSdkState.id),
eq(chatSdkState.stateKey, GITHUB_RECOVERY_STATE_KEY),
),
)
.limit(1),
]);
// A provider effect can be on another page. Replay safety must still
// consider every unresolved effect associated with these deliveries.
const ambiguousEffects = deliveries.length ? await db.select({ deliveryId: chatActions.deliveryId })
.from(chatActions).where(and(
eq(chatActions.endpointId, endpointId),
eq(chatActions.kind, "provider_effect"),
eq(chatActions.status, "delivery_unknown"),
inArray(chatActions.deliveryId, deliveries.map((row) => row.id)),
)) : [];
const ambiguousProviderEffectDeliveryIds = new Set(ambiguousEffects.map((row) => row.deliveryId));
const transferRows = publications.length
? await db
.select(fileTransferProjectionColumns)
.from(chatTeamsFileTransfers)
.innerJoin(
chatPublications,
and(
eq(chatPublications.id, chatTeamsFileTransfers.publicationId),
eq(chatPublications.companyId, chatTeamsFileTransfers.companyId),
eq(
chatPublications.endpointId,
chatTeamsFileTransfers.endpointId,
),
eq(
chatPublications.conversationId,
chatTeamsFileTransfers.conversationId,
),
),
)
.where(
and(
eq(chatTeamsFileTransfers.companyId, publications[0]!.companyId),
eq(chatTeamsFileTransfers.endpointId, endpointId),
inArray(
chatTeamsFileTransfers.publicationId,
publications.map((row) => row.id),
),
),
)
: [];
const transfers = new Map(
transferRows.map((row) => [row.publicationId, row]),
);
const cancellableConflicts = new Map<string, number>();
for (const transfer of transferRows) {
if (transfer.phase !== "conflict") continue;
const publication = publications.find(
(row) => row.id === transfer.publicationId,
);
if (!publication?.conversationId) continue;
if (
await teamsTransferProtocol().canCancelConflict({
companyId: publication.companyId,
endpointId,
conversationId: publication.conversationId,
publicationId: publication.id,
version: transfer.version,
})
)
cancellableConflicts.set(publication.id, transfer.version);
}
return [
...deliveries.map((row) => {
const normalized =
row.normalizedEvent &&
typeof row.normalizedEvent === "object" &&
!Array.isArray(row.normalizedEvent)
? (row.normalizedEvent as Record<string, unknown>)
: {};
const deduplication =
normalized.deduplication &&
typeof normalized.deduplication === "object" &&
!Array.isArray(normalized.deduplication)
? (normalized.deduplication as Record<string, unknown>)
: null;
const duplicateCount =
typeof deduplication?.duplicateCount === "number"
? deduplication.duplicateCount
: 0;
const label = row.eventKind.replaceAll("_", " ");
const outcome =
row.state === "filtered"
? "ignored"
: row.state === "processed"
? "processed"
: row.state;
return {
id: row.id,
kind: "delivery" as const,
status: row.state,
summary: `${label} ${outcome}${duplicateCount ? ` · ${duplicateCount} duplicate${duplicateCount === 1 ? "" : "s"} ignored` : ""}`,
detail: row.redactedError,
createdAt: row.createdAt.toISOString(),
replayable:
row.state === "failed" &&
Boolean(row.conversationId) &&
!ambiguousProviderEffectDeliveryIds.has(row.id),
resolutionActions: [],
};
}),
...publications.map((row) => {
const transfer = transfers.get(row.id);
const projected = projectChatFileTransfer(
publicationSummary(row),
transfer,
);
const payload = row.payload as Partial<SafeChatPublicationPayload>;
const summary = payload.progressState
? `${payload.progressState.replaceAll("_", " ")} update`
: payload.interactionId
? "Interactive question"
: "Response to external conversation";
return {
id: row.id,
kind: "publication" as const,
status: projected.state,
summary,
detail: projected.redactedError ?? null,
createdAt: row.createdAt.toISOString(),
replayable: !transfer && row.state === "failed",
...(projected.fileTransfer
? { fileTransfer: projected.fileTransfer }
: {}),
resolutionActions: transfer
? chatFileTransferResolutionActions(
projected,
cancellableConflicts.has(row.id)
? {
publicationId: row.id,
version: cancellableConflicts.get(row.id)!,
}
: undefined,
)
: row.state === "delivery_unknown"
? (["mark_delivered", "retry_anyway", "cancel"] as const)
: [],
};
}),
...slashActions.map((row) => {
const recovery = slackSlashTaskRecoveryPayload(row.payload);
// Keep this GET read-only while still surfacing a crashed attempt as
// actionable. The resolution POST persists the derived quarantine
// before it applies the operator's audited decision.
const status = slackTaskStartActivityStatus(row);
return {
id: row.id,
kind: "action" as const,
actionType: "slash_task_start" as const,
status,
summary: `Slack slash-command task start ${status.replaceAll("_", " ")}`,
detail:
status === "delivery_unknown"
? recovery
? "Slack may have accepted the task-start message, so Paperclip will not replay it automatically. Check Slack first; an explicit retry can create a duplicate starter message and task."
: "Slack may have accepted the task-start message. This older action lacks the context required for a safe explicit retry, so check Slack and cancel it here before submitting a new command."
: status === "provider_confirmed"
? "Slack accepted the task-start message. Paperclip is completing durable task admission without sending another Slack message."
: status === "admitting"
? "Slack accepted the task-start message. A Paperclip worker is admitting the task without replaying the Slack send."
: status === "failed"
? "Slack rejected the task-start message. Submit the command again to retry."
: status === "cancelled"
? "An operator cancelled this unconfirmed task start."
: null,
createdAt: row.createdAt.toISOString(),
replayable: false,
resolutionActions:
status === "delivery_unknown"
? recovery
? (["retry_anyway", "cancel"] as const)
: (["cancel"] as const)
: [],
};
}),
...providerEffects.map((row) => {
const payload = providerEffectPayload(row.payload);
return {
id: row.id,
kind: "action" as const,
actionType: "provider_effect" as const,
status: row.status,
summary: "Provider reply delivery unknown",
detail: payload?.completeConversationId
? "The provider may have accepted this reply, but Paperclip could not confirm it or close the task conversation. Check the provider first. Marking it delivered closes the conversation; retrying can create a duplicate message."
: "The provider may have accepted this reply, but Paperclip could not confirm it. Check the provider first. Retrying can create a duplicate message.",
createdAt: row.createdAt.toISOString(),
replayable: false,
resolutionActions: [
"mark_delivered",
"retry_anyway",
"cancel",
] as const,
};
}),
...slackSessionActions.map((row) => {
const isSync = row.kind === "slack_session_sync";
const retrying =
row.status === "received" &&
row.result?.code === "slack_session_sync_retry";
return {
id: row.id,
kind: "action" as const,
actionType: isSync
? ("slack_session_sync" as const)
: ("slack_session_stop" as const),
status: retrying ? "retry" : row.status,
summary: isSync ? "Slack session status" : "Slack Stop request",
detail: isSync
? row.result?.outcome === "unavailable"
? "Slack has not enabled native session status for this destination. Message delivery continues."
: row.result?.code === "slack_session_sync_rejected"
? "Slack rejected the session indicator update. Check app permissions and channel access; message delivery is tracked separately."
: retrying
? "The Slack session indicator is waiting to sync. Paperclip will not resend the response."
: row.result?.sessionStatus === "processing"
? "Working status is refreshed automatically while the run remains active."
: null
: row.status === "processed"
? "Paperclip stopped the work authorized by this request."
: row.status === "cancelled"
? "No work was stopped: this request was no longer authorized or its target was no longer current."
: row.status === "failed"
? row.result?.retryable === true
? "The Stop request could not finish yet. Paperclip will retry against the original work only."
: "The Stop request could not be completed. Check the task's current run before trying again."
: "Paperclip is processing this Stop request against its original task and run.",
createdAt: row.updatedAt.toISOString(),
replayable: false,
resolutionActions: [],
};
}),
...githubIngressFailures.map((row) => {
const deliveryId =
typeof row.payload.deliveryId === "string"
? row.payload.deliveryId
: null;
return {
id: row.id,
kind: "action" as const,
actionType: "github_webhook_ingress" as const,
status: row.status,
summary: "GitHub webhook could not be processed",
detail: deliveryId
? `GitHub delivery ${deliveryId} was authenticated, but Paperclip could not finish processing it. Fix the connection or destination, then redeliver this delivery from the GitHub App's Recent Deliveries page.`
: "An authenticated GitHub webhook could not be processed. Fix the connection or destination, then redeliver it from the GitHub App's Recent Deliveries page.",
createdAt: row.createdAt.toISOString(),
replayable: false,
resolutionActions: [],
};
}),
...githubRecoveries.map(({ action: row, ingressStatus }) => ({
id: row.id,
kind: "repair" as const,
status:
row.status === "cancelled"
? "cancelled"
: ingressStatus
? "received"
: "pending",
summary:
row.status === "cancelled"
? "GitHub webhook recovery skipped"
: ingressStatus
? "GitHub webhook received after recovery request"
: "GitHub webhook redelivery requested",
detail:
row.status === "cancelled"
? row.result?.code === "source_changed_or_unavailable"
? "The original comment changed or is no longer available. Paperclip did not replay its old contents."
: "This callback was not an eligible current user comment. Paperclip did not replay it."
: ingressStatus
? "Paperclip received this callback. Its normal access checks and processing still apply."
: "Paperclip asked GitHub to resend a recent missed message. This does not yet confirm receipt or a reply. Automatic requests are limited; if it remains unanswered, check the App's Recent Deliveries and send your request again in the current conversation.",
createdAt: row.updatedAt.toISOString(),
replayable: false,
resolutionActions: [],
})),
...githubRecoveryScans.flatMap((row) => {
const value = row.value as { outcome?: string } | null;
const detail =
value?.outcome === "history_limit_reached"
? "Recent webhook volume exceeded automatic recovery's bounded scan. No partial history was replayed. Check the App's Recent Deliveries for missed messages."
: value?.outcome === "callback_mismatch"
? "The GitHub App's callback no longer matches this connection. Reconnect the App to repair it; repository access will not change."
: value?.outcome === "scan_failed"
? "Paperclip could not check GitHub's failed deliveries. The check will retry with backoff; normal callbacks are still processed."
: null;
return detail
? [
{
id: row.id,
kind: "health" as const,
status: "attention",
summary: "GitHub webhook recovery needs attention",
detail,
createdAt: row.updatedAt.toISOString(),
replayable: false,
resolutionActions: [],
},
]
: [];
}),
]
.sort((a, b) => b.createdAt.localeCompare(a.createdAt) || b.id.localeCompare(a.id))
.slice(0, limit);
}
async function listActivityPage(endpointId: string, limit = 25, cursor?: string) {
if (!Number.isInteger(limit) || limit < 1 || limit > 100) throw badRequest("Activity limit must be between 1 and 100");
let before: { createdAt: string; id: string } | undefined;
if (cursor !== undefined) {
try {
if (cursor.length > 256) throw new Error("Invalid cursor");
const value = JSON.parse(Buffer.from(cursor, "base64url").toString("utf8"));
if (!Array.isArray(value) || value.length !== 2
|| typeof value[0] !== "string" || new Date(value[0]).toISOString() !== value[0]
|| typeof value[1] !== "string" || !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(value[1])) throw new Error("Invalid cursor");
before = { createdAt: value[0], id: value[1] };
} catch { throw badRequest("Invalid activity cursor"); }
}
const rows = await listActivity(endpointId, { limit, before });
const items = rows.slice(0, limit);
const last = items.at(-1);
return {
items,
nextCursor: rows.length > limit && last
? Buffer.from(JSON.stringify([last.createdAt, last.id])).toString("base64url")
: null,
};
}
return { listConversations, listActivity, listActivityPage };
}
@@ -0,0 +1,253 @@
import type { EndpointRow, DbTransaction, RuntimeContext, LifecycleRuntimeFence, InternalSetupState } from "../runtime-types.js";
import { notifyChatEndpointWork } from "../../chat-work-notifications.js";
import {
and,
eq,
inArray,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatConversations,
chatEndpointResources,
chatEndpoints,
toolConnections,
} from "@paperclipai/db";
import type {
ChatEndpointSetupState,
} from "@paperclipai/shared";
import {
type DiscordGatewayCallbackEvent,
} from "../../chat-sdk-runtime.js";
import {
type ChatProviderLifecycleEffect,
} from "../../chat-provider-lifecycle.js";
import type { EndpointRecord } from "../runtime-types.js";
type DiscordGatewayEventsOptions = {
ensureDiscordGatewayRuntimeIsCurrent: (endpointId: string, context: RuntimeContext) => Promise<boolean>;
discordGatewayRuntimeIsCurrent: (endpointId: string, context: RuntimeContext) => boolean;
runtimeCallbackRecord: (endpointId: string, context: RuntimeContext, statuses: EndpointRow["status"][]) => Promise<EndpointRecord | null>;
runtimeCallbackEndpoint: (tx: DbTransaction, endpointId: string, context: LifecycleRuntimeFence, statuses: EndpointRow["status"][]) => Promise<EndpointRow | null>;
applyProviderLifecycleEffect: (endpoint: EndpointRow, effect: ChatProviderLifecycleEffect, context?: RuntimeContext) => Promise<boolean>;
runtimeGeneration: (setup: ChatEndpointSetupState) => number;
discordGatewayEventBarrier?: (event: DiscordGatewayCallbackEvent) => Promise<void>;
};
export function createDiscordGatewayEvents(db: Db, options: DiscordGatewayEventsOptions) {
const { ensureDiscordGatewayRuntimeIsCurrent, discordGatewayRuntimeIsCurrent, runtimeCallbackRecord, runtimeCallbackEndpoint, applyProviderLifecycleEffect, runtimeGeneration } = options;
function discordGatewayFailureDetail(
event: DiscordGatewayCallbackEvent["event"],
): string {
if (event.type === "guild_removed") {
return "Discord bot was removed from the configured server";
}
if (event.type === "guild_unavailable") {
return "Discord server is temporarily unavailable; reconnecting";
}
if (event.type === "disconnected") {
return `Discord Gateway disconnected${event.code === undefined ? "" : ` (code ${event.code})`}; reconnecting`;
}
if (event.type === "failure") {
const code = event.error.code ?? event.error.status;
return `Discord Gateway failed${code === undefined ? "" : ` (${String(code).slice(0, 64)})`}; ${event.fatal ? "credentials or privileged intents need attention" : "reconnecting"}`;
}
return "Discord Gateway connection needs attention";
}
function handleDiscordGatewayEvent(
callback: DiscordGatewayCallbackEvent,
context: RuntimeContext,
): Promise<void> {
const pending = (context.discordGatewayEventQueue ?? Promise.resolve())
.catch(() => undefined)
.then(() => processDiscordGatewayEvent(callback, context));
context.discordGatewayEventQueue = pending;
return pending;
}
async function processDiscordGatewayEvent(
callback: DiscordGatewayCallbackEvent,
context: RuntimeContext,
): Promise<void> {
if (
!(await ensureDiscordGatewayRuntimeIsCurrent(
callback.endpointId,
context,
))
)
return;
if (callback.sequence <= (context.discordGatewaySequence ?? 0)) return;
context.discordGatewaySequence = callback.sequence;
await options.discordGatewayEventBarrier?.(callback);
const record = await runtimeCallbackRecord(callback.endpointId, context, [
"verifying",
"active",
"attention",
]);
if (!record || record.endpoint.provider !== "discord") return;
const event = callback.event;
if (event.type === "channel_removed") {
await applyProviderLifecycleEffect(
record.endpoint,
{
kind: "resource",
provider: "discord",
providerEventId: `gateway:channel_removed:${event.channelId}`,
providerResourceId: event.channelId,
resourceType: "channel",
label: event.label?.trim() || event.channelId,
availability: "removed",
metadata: { source: "discord_gateway" },
},
context,
);
return;
}
await db.transaction(async (tx) => {
const current = await runtimeCallbackEndpoint(
tx,
callback.endpointId,
context,
["verifying", "active", "attention"],
);
if (
!current ||
current.provider !== "discord" ||
!discordGatewayRuntimeIsCurrent(callback.endpointId, context)
) {
return;
}
const now = new Date();
if (event.type === "connecting") {
await tx
.update(chatEndpoints)
.set({
healthMessage: "Discord Gateway connecting",
lastError: null,
lastEventAt: now,
updatedAt: now,
})
.where(eq(chatEndpoints.id, current.id));
await tx
.update(toolConnections)
.set({
healthStatus: "unchecked",
healthMessage: "Discord Gateway connecting",
lastError: null,
healthCheckedAt: now,
updatedAt: now,
})
.where(eq(toolConnections.id, current.connectionId));
return;
}
if (event.type === "ready" || event.type === "guild_available") {
const gatewayOwnedAttention =
current.status === "attention" &&
current.healthMessage?.startsWith("Discord ") === true;
if (current.status === "attention" && !gatewayOwnedAttention) return;
const status =
current.setup.step === "complete" ? "active" : "verifying";
if (status !== current.status) await notifyChatEndpointWork(tx);
await tx
.update(chatEndpoints)
.set({
status,
healthMessage:
status === "active"
? "Connected"
: "Waiting for a test conversation",
lastError: null,
lastEventAt: now,
updatedAt: now,
})
.where(eq(chatEndpoints.id, current.id));
await tx
.update(toolConnections)
.set({
status: "active",
enabled: true,
healthStatus: "healthy",
healthMessage: "Connected",
lastError: null,
healthCheckedAt: now,
updatedAt: now,
})
.where(eq(toolConnections.id, current.connectionId));
return;
}
const detail = discordGatewayFailureDetail(event);
const transientGuildOutage = event.type === "guild_unavailable";
const terminal =
event.type === "guild_removed" ||
((event.type === "failure" || event.type === "disconnected") &&
event.fatal);
await tx
.update(chatEndpoints)
.set({
status: transientGuildOutage
? current.status
: event.type === "guild_removed"
? "revoked"
: "attention",
healthMessage: detail,
lastError: detail,
lastEventAt: now,
...(terminal
? {
setup: {
...current.setup,
runtimeGeneration: runtimeGeneration(current.setup) + 1,
} as InternalSetupState,
}
: {}),
updatedAt: now,
})
.where(eq(chatEndpoints.id, current.id));
await tx
.update(toolConnections)
.set({
status: terminal ? "disabled" : "active",
enabled: !terminal,
healthStatus: event.type === "guild_removed" ? "failed" : "degraded",
healthMessage: detail,
lastError: detail,
healthCheckedAt: now,
updatedAt: now,
})
.where(eq(toolConnections.id, current.connectionId));
if (event.type === "guild_removed") {
await tx
.update(chatEndpointResources)
.set({ availability: "unavailable", updatedAt: now })
.where(
and(
eq(chatEndpointResources.companyId, current.companyId),
eq(chatEndpointResources.endpointId, current.id),
),
);
await tx
.update(chatConversations)
.set({ state: "unavailable", updatedAt: now })
.where(
and(
eq(chatConversations.companyId, current.companyId),
eq(chatConversations.endpointId, current.id),
inArray(chatConversations.state, ["active", "waiting"]),
),
);
}
});
}
return { handleDiscordGatewayEvent };
}
@@ -1,3 +1,4 @@
import type { FileUpload } from "chat";
import { createHash } from "node:crypto";
@@ -28,3 +29,11 @@ export function discordLifecycleFilesDigest(raw: unknown): string {
: [];
return createHash("sha256").update(JSON.stringify(projection)).digest("hex");
}
export function discordMarkdownAttachment(text: string): FileUpload {
return {
data: Buffer.from(text, "utf8"),
filename: "paperclip-response.md",
mimeType: "text/markdown; charset=utf-8",
};
}
@@ -0,0 +1,570 @@
import type { EndpointRow, DbTransaction, DbOrTransaction, RuntimeContext, LifecycleRuntimeFence, InboundRuntimeContext } from "../runtime-types.js";
import { notifyChatDeliveryWork } from "../../chat-work-notifications.js";
import type { ChatEndpointRuntime } from "../../voice/voice-runtime.js";
import { chatCredentialMutationLease } from "../../chat-credential-mutation-lease.js";
import {
createHash,
} from "node:crypto";
import {
and,
eq,
inArray,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatConversations,
chatDeliveries,
chatEndpointResources,
chatEndpoints,
chatExternalPrincipals,
companyMemberships,
} from "@paperclipai/db";
import {
conflict,
} from "../../../errors.js";
import { logger } from "../../../middleware/logger.js";
import {
DiscordAdapterCompatibilityError,
type ChatSdkMessageCallbackEvent,
type DiscordRootMentionAdmissionEvent,
} from "../../chat-sdk-runtime.js";
import { classifyChatPublicationError } from "../../chat-publication-errors.js";
import type {
Message,
Thread,
} from "chat";
type DeliveryRow = typeof chatDeliveries.$inferSelect;
type DiscordRootThreadsOptions = {
ensureDiscordGatewayRuntimeIsCurrent: (endpointId: string, context: RuntimeContext) => Promise<boolean>;
withCredentialMutationLease: ReturnType<typeof chatCredentialMutationLease>;
runtimeCallbackEndpoint: (tx: DbTransaction, endpointId: string, context: LifecycleRuntimeFence, statuses: EndpointRow["status"][]) => Promise<EndpointRow | null>;
lockCurrentPrincipalAuthorization: (tx: DbOrTransaction, endpoint: EndpointRow, principalId: string) => Promise<{ allowed: boolean; linkedDenied: boolean }>;
processMessage: (endpoint: EndpointRow, thread: Thread, message: Message, trigger: ChatSdkMessageCallbackEvent["trigger"], ingressOnly?: boolean, recoveredProviderUrl?: string | null, runtimeContext?: InboundRuntimeContext, providerUpdateId?: number, admittedDeliveryId?: string | null, receiptReactionSupported?: boolean, preauthorizedUserId?: string | null, deferDrainUntilFollowup?: boolean) => Promise<void>;
redactError: (error: unknown) => string;
scheduleConversationDrain: (endpointId: string, threadId: string) => void;
lifecycleRuntimeFence: (delivery: DeliveryRow) => LifecycleRuntimeFence | null;
runtimeFor: (endpoint: EndpointRow) => Promise<ChatEndpointRuntime>;
DISCORD_ROOT_THREAD_CONFIRMATION_DELAY_MS: number;
discordRootThreadTransportBarrier?: () => Promise<void>;
};
export function createDiscordRootThreads(db: Db, options: DiscordRootThreadsOptions) {
const { ensureDiscordGatewayRuntimeIsCurrent, withCredentialMutationLease, runtimeCallbackEndpoint, lockCurrentPrincipalAuthorization, processMessage, redactError, scheduleConversationDrain, lifecycleRuntimeFence, runtimeFor, DISCORD_ROOT_THREAD_CONFIRMATION_DELAY_MS } = options;
async function admitDiscordRootMention(
configuredEndpoint: EndpointRow,
event: DiscordRootMentionAdmissionEvent,
context: RuntimeContext,
): Promise<boolean> {
if (
!(await ensureDiscordGatewayRuntimeIsCurrent(
configuredEndpoint.id,
context,
))
) {
return false;
}
return withCredentialMutationLease(
configuredEndpoint,
async (credentialLease) => {
const threadId = `discord:${event.guildId}:${event.channelId}:${event.messageId}`;
const providerEventId = `${threadId}:${event.messageId}`;
const allowed = await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const endpoint = await runtimeCallbackEndpoint(
tx,
configuredEndpoint.id,
context,
["verifying", "active"],
);
const existing = await tx
.select({ state: chatDeliveries.state })
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.endpointId, configuredEndpoint.id),
eq(chatDeliveries.providerEventId, providerEventId),
),
)
.then((rows) => rows[0] ?? null);
if (
existing &&
["processed", "filtered", "failed"].includes(existing.state)
)
return false;
let reason: string | null = null;
if (!endpoint || endpoint.providerAccountId !== event.guildId) {
reason = "Connection is not active";
}
const resource = endpoint
? await tx
.select()
.from(chatEndpointResources)
.where(
and(
eq(chatEndpointResources.endpointId, endpoint.id),
eq(chatEndpointResources.type, "channel"),
eq(
chatEndpointResources.providerResourceId,
event.channelId,
),
),
)
.for("update")
.then((rows) => rows[0] ?? null)
: null;
const enabledCount =
endpoint?.status === "verifying"
? await tx
.select({ count: sql<number>`count(*)::int` })
.from(chatEndpointResources)
.where(
and(
eq(chatEndpointResources.endpointId, endpoint.id),
eq(chatEndpointResources.type, "channel"),
eq(chatEndpointResources.enabled, true),
eq(chatEndpointResources.availability, "available"),
),
)
.then((rows) => rows[0]?.count ?? 0)
: 0;
const setupDestination =
endpoint?.status === "verifying" &&
enabledCount === 0 &&
resource?.availability === "available";
if (
!reason &&
!(
endpoint?.allowGroupChats &&
resource?.availability === "available" &&
(resource.enabled || setupDestination)
)
) {
reason = "Destination is not enabled in Paperclip";
}
if (!reason && endpoint) {
const principal = await tx
.select({ id: chatExternalPrincipals.id })
.from(chatExternalPrincipals)
.where(
and(
eq(chatExternalPrincipals.companyId, endpoint.companyId),
eq(chatExternalPrincipals.provider, "discord"),
eq(chatExternalPrincipals.providerAccountId, event.guildId),
eq(chatExternalPrincipals.externalId, event.userId),
),
)
.then((rows) => rows[0] ?? null);
let allowed: boolean;
let linkedDenied = false;
if (principal) {
const authorization = await lockCurrentPrincipalAuthorization(
tx,
endpoint,
principal.id,
);
allowed = authorization.allowed;
linkedDenied = authorization.linkedDenied;
} else if (!endpoint.allowUnlinkedPeople) {
allowed = false;
} else if (!endpoint.sponsorUserId) {
allowed = true;
} else {
const sponsor = await tx
.select({
status: companyMemberships.status,
membershipRole: companyMemberships.membershipRole,
})
.from(companyMemberships)
.where(
and(
eq(companyMemberships.companyId, endpoint.companyId),
eq(companyMemberships.principalType, "user"),
eq(companyMemberships.principalId, endpoint.sponsorUserId),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
allowed =
sponsor?.status === "active" &&
sponsor.membershipRole !== "viewer";
}
if (!allowed) {
reason = linkedDenied
? "Linked Paperclip account is not currently permitted"
: endpoint.allowUnlinkedPeople
? "Endpoint sponsor can no longer authorize external guests"
: "External identity must be linked to a Paperclip account";
}
}
if (!reason) {
await credentialLease.assertOwned(tx);
return true;
}
const filteredAt = new Date();
const normalizedEvent = {
providerEventId,
kind: "mention",
trigger: "mention",
resource: {
type: "channel",
providerResourceId: event.channelId,
},
conversation: { externalThreadId: threadId },
message: { providerMessageId: event.messageId },
filtering: { contentRetained: false, providerThreadCreated: false },
};
await notifyChatDeliveryWork(tx);
await tx
.insert(chatDeliveries)
.values({
companyId: configuredEndpoint.companyId,
endpointId: configuredEndpoint.id,
providerEventId,
deduplicationKey: createHash("sha256")
.update(providerEventId)
.digest("hex"),
eventKind: "mention",
normalizedEvent,
state: "filtered",
redactedError: reason,
processedAt: filteredAt,
})
.onConflictDoUpdate({
target: [
chatDeliveries.endpointId,
chatDeliveries.providerEventId,
],
set: {
normalizedEvent,
principalId: null,
state: "filtered",
nextAttemptAt: null,
redactedError: reason,
processedAt: filteredAt,
updatedAt: filteredAt,
},
});
await tx
.update(chatEndpoints)
.set({ lastEventAt: filteredAt, updatedAt: filteredAt })
.where(eq(chatEndpoints.id, configuredEndpoint.id));
await credentialLease.assertOwned(tx);
return false;
});
if (!allowed || !event.message || !context.endpointRuntime) {
return false;
}
const thread = context.endpointRuntime.thread(event.threadId);
await processMessage(
configuredEndpoint,
thread,
event.message,
"mention",
true,
null,
context,
undefined,
null,
true,
undefined,
true,
);
const staged = await db
.select({ state: chatDeliveries.state })
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.endpointId, configuredEndpoint.id),
eq(chatDeliveries.providerEventId, providerEventId),
),
)
.then((rows) => rows[0] ?? null);
if (
!staged ||
!["received", "retry", "processing"].includes(staged.state)
) {
return false;
}
await options.discordRootThreadTransportBarrier?.();
await credentialLease.assertOwned();
if (
!(await ensureDiscordGatewayRuntimeIsCurrent(
configuredEndpoint.id,
context,
))
) {
return false;
}
try {
await context.endpointRuntime.ensureDiscordRootThread({
channelId: event.channelId,
content: event.message.text,
messageId: event.messageId,
});
} catch (error) {
// The durable provisional receipt owns recovery. Do not let a
// transient REST failure escape the Gateway event handler or ask the
// adapter to create the same thread outside Paperclip's lifecycle
// fence.
logger.warn(
{
endpointId: configuredEndpoint.id,
error: redactError(error),
providerEventId,
},
"Discord root thread creation will be retried from the durable receipt",
);
return false;
}
const promoted = await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const current = await runtimeCallbackEndpoint(
tx,
configuredEndpoint.id,
context,
["verifying", "active"],
);
if (!current) return false;
await notifyChatDeliveryWork(tx);
const [row] = await tx
.update(chatDeliveries)
.set({
state: "received",
normalizedEvent: sql`${chatDeliveries.normalizedEvent} - 'providerThreadPending'`,
nextAttemptAt: null,
redactedError: null,
updatedAt: new Date(),
})
.where(
and(
eq(chatDeliveries.endpointId, configuredEndpoint.id),
eq(chatDeliveries.providerEventId, providerEventId),
eq(chatDeliveries.state, "retry"),
sql`${chatDeliveries.normalizedEvent}->>'providerThreadPending' = 'true'`,
),
)
.returning({ id: chatDeliveries.id });
await credentialLease.assertOwned(tx);
return Boolean(row);
});
if (promoted)
scheduleConversationDrain(configuredEndpoint.id, threadId);
// Paperclip created the provider thread inside its credential/reach
// lease. Returning false prevents the adapter from creating it again;
// the durable drain owns the remaining task mutation.
return false;
},
);
}
function discordRootThreadPending(
delivery: Pick<DeliveryRow, "normalizedEvent">,
): boolean {
return (
Boolean(delivery.normalizedEvent) &&
typeof delivery.normalizedEvent === "object" &&
!Array.isArray(delivery.normalizedEvent) &&
(delivery.normalizedEvent as { providerThreadPending?: unknown })
.providerThreadPending === true
);
}
async function reconcileDiscordRootThread(
endpoint: EndpointRow,
delivery: DeliveryRow,
threadId: string,
): Promise<boolean> {
try {
const normalized = delivery.normalizedEvent as {
message?: { providerMessageId?: unknown; text?: unknown };
resource?: { providerResourceId?: unknown };
};
const discordThreadParts = threadId.split(":");
const channelId =
discordThreadParts[0] === "discord" && discordThreadParts.length >= 4
? discordThreadParts[2]
: normalized.resource?.providerResourceId;
const messageId = normalized.message?.providerMessageId;
if (typeof channelId !== "string" || typeof messageId !== "string") {
throw new DiscordAdapterCompatibilityError(
"the provisional root receipt is missing its channel or message id",
);
}
const runtimeFence = lifecycleRuntimeFence(delivery);
if (!runtimeFence) {
throw new DiscordAdapterCompatibilityError(
"the provisional root receipt is missing its runtime fence",
);
}
return await withCredentialMutationLease(
endpoint,
async (credentialLease) => {
const currentEndpoint = await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const current = await runtimeCallbackEndpoint(
tx,
endpoint.id,
runtimeFence,
["verifying", "active"],
);
await credentialLease.assertOwned(tx);
return current;
});
if (!currentEndpoint) {
throw conflict(
"Discord root thread credentials or connection state changed before recovery",
{ code: "chat_endpoint_runtime_superseded" },
);
}
const endpointRuntime = await runtimeFor(currentEndpoint);
await credentialLease.assertOwned();
await endpointRuntime.ensureDiscordRootThread({
channelId,
messageId,
content:
typeof normalized.message?.text === "string"
? normalized.message.text
: "Paperclip task",
});
return await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const current = await runtimeCallbackEndpoint(
tx,
endpoint.id,
runtimeFence,
["verifying", "active"],
);
if (!current) return false;
await notifyChatDeliveryWork(tx);
const [promoted] = await tx
.update(chatDeliveries)
.set({
state: "received",
normalizedEvent: sql`${chatDeliveries.normalizedEvent} - 'providerThreadPending'`,
nextAttemptAt: null,
redactedError: null,
updatedAt: new Date(),
})
.where(
and(
eq(chatDeliveries.id, delivery.id),
eq(chatDeliveries.state, "retry"),
sql`${chatDeliveries.normalizedEvent}->>'providerThreadPending' = 'true'`,
),
)
.returning({ id: chatDeliveries.id });
await credentialLease.assertOwned(tx);
return Boolean(promoted);
});
},
);
} catch (error) {
const disposition = classifyChatPublicationError(
error,
delivery.attempts + 1,
);
const terminal =
disposition.kind === "resource_unavailable" ||
disposition.kind === "failed";
const retryAfterMs =
disposition.kind === "retry"
? disposition.retryAfterMs
: DISCORD_ROOT_THREAD_CONFIRMATION_DELAY_MS;
const now = new Date();
await db.transaction(async (tx) => {
if (disposition.kind === "resource_unavailable") {
const fence = lifecycleRuntimeFence(delivery);
const normalized = delivery.normalizedEvent as {
resource?: { providerResourceId?: unknown };
};
const providerResourceId = normalized.resource?.providerResourceId;
const current = fence
? await runtimeCallbackEndpoint(tx, endpoint.id, fence, [
"verifying",
"active",
"attention",
])
: null;
const resource =
current && typeof providerResourceId === "string"
? await tx
.select({ id: chatEndpointResources.id })
.from(chatEndpointResources)
.where(
and(
eq(chatEndpointResources.companyId, current.companyId),
eq(chatEndpointResources.endpointId, current.id),
eq(
chatEndpointResources.providerResourceId,
providerResourceId,
),
),
)
.for("update")
.then((rows) => rows[0] ?? null)
: null;
if (resource) {
await tx
.update(chatEndpointResources)
.set({ availability: "unavailable", updatedAt: now })
.where(eq(chatEndpointResources.id, resource.id));
await tx
.update(chatConversations)
.set({ state: "unavailable", updatedAt: now })
.where(
and(
eq(chatConversations.companyId, current!.companyId),
eq(chatConversations.endpointId, current!.id),
eq(chatConversations.resourceId, resource.id),
inArray(chatConversations.state, ["active", "waiting"]),
),
);
}
}
await notifyChatDeliveryWork(tx);
await tx
.update(chatDeliveries)
.set({
state: terminal ? "filtered" : "retry",
attempts: delivery.attempts + 1,
nextAttemptAt: terminal
? null
: new Date(now.getTime() + retryAfterMs),
processedAt: terminal ? now : null,
redactedError: terminal
? "Discord thread creation was not confirmed"
: redactError(error),
updatedAt: now,
})
.where(
and(
eq(chatDeliveries.id, delivery.id),
eq(chatDeliveries.state, "retry"),
sql`${chatDeliveries.normalizedEvent}->>'providerThreadPending' = 'true'`,
),
);
});
return false;
}
}
return { admitDiscordRootMention, discordRootThreadPending, reconcileDiscordRootThread };
}
@@ -0,0 +1,674 @@
import type { EndpointRow, DbTransaction, DbOrTransaction, LifecycleRuntimeFence } from "./runtime-types.js";
import type { ChatEndpointRuntime } from "../voice/voice-runtime.js";
import { takePhotonCompanion } from "../photon/attachments.js";
import { HEIF_CONTENT_TYPES, photonHeifPreview, validatePhotonImage } from "../photon/media.js";
import { PhotonError } from "../photon/cloud.js";
import {
createHash,
} from "node:crypto";
import {
and,
eq,
inArray,
isNull,
ne,
or,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
assets,
chatActions,
chatConversations,
chatDeliveries,
issueComments,
issueAttachments,
} from "@paperclipai/db";
import {
isAllowedContentType,
MAX_ATTACHMENT_BYTES,
normalizeContentType,
normalizeUploadAttachmentContentType,
} from "../../attachment-types.js";
import {
hasTelegramMediaProvenance,
identifyTelegramMedia,
telegramMediaNeedsIdentification,
} from "../chat-telegram-media-intake.js";
import {
forbidden,
} from "../../errors.js";
import { logger } from "../../middleware/logger.js";
import type { StorageService } from "../../storage/types.js";
import { issueService } from "../issues.js";
import { isExternalChatWaitAuthorizationContention } from "../native-runtime/chat-attachment-reuse.js";
import {
GITHUB_ATTACHMENT_BATCH_TIMEOUT_MS,
GitHubAttachmentUnavailableError,
prepareGitHubPublicAttachment,
} from "../chat-github-attachments.js";
import type {
Attachment,
} from "chat";
type DeliveryRow = typeof chatDeliveries.$inferSelect;
type InboundAttachmentsOptions = {
storage?: StorageService;
authorizeInboundWakeup: (tx: DbOrTransaction, action: typeof chatActions.$inferSelect) => Promise<{ delivery: DeliveryRow; conversation: typeof chatConversations.$inferSelect }>;
lifecycleRuntimeFence: (delivery: DeliveryRow) => LifecycleRuntimeFence | null;
sanitizeFilename: (value: string | undefined) => string | null;
issuesSvc: Pick<ReturnType<typeof issueService>, "createAttachment">;
isExternalActionAuthorizationChange: (error: unknown) => boolean;
redactError: (error: unknown) => string;
};
export function createInboundAttachments(db: Db, options: InboundAttachmentsOptions) {
const { authorizeInboundWakeup, lifecycleRuntimeFence, sanitizeFilename, issuesSvc, isExternalActionAuthorizationChange, redactError } = options;
async function ingestAttachments(input: {
endpoint: EndpointRow;
endpointRuntime: ChatEndpointRuntime;
deliveryId: string;
issueId: string;
issueCommentId: string;
attachments: Attachment[];
attachmentLimitOmissions?: number;
unavailableReferenceCount?: number;
actorUserId: string | null;
}): Promise<{
storedIds: string[];
omissionReasons: Record<string, number>;
}> {
const omissionReasons: Record<string, number> = {};
const omit = (reason: string, count = 1) => {
omissionReasons[reason] = (omissionReasons[reason] ?? 0) + count;
};
const boundedAttachments = input.attachments.slice(0, 20);
const photonDerivatives = new Map<Attachment, { source: Attachment; sourceHash: string; kind: "heif_jpeg_preview" | "live_photo_video" }>();
const originalPhotonAttachments = new Map<Attachment, string>();
if (
input.endpoint.provider === "microsoft-teams" &&
input.unavailableReferenceCount
) {
// A channel/group reference is not a download capability. Still tell
// the exact current-turn consumer that those bytes are unavailable,
// including when there are no ingestible attachments at all.
omit("download_unavailable", input.unavailableReferenceCount);
}
if (
["github", "imessage-photon"].includes(input.endpoint.provider) &&
input.attachmentLimitOmissions
) {
omit("attachment_limit", input.attachmentLimitOmissions);
}
if (input.attachments.length > boundedAttachments.length) {
omit(
"attachment_limit",
input.attachments.length - boundedAttachments.length,
);
}
if (!options.storage) {
if (boundedAttachments.length)
omit("storage_unavailable", boundedAttachments.length);
return { storedIds: [], omissionReasons };
}
const existingByFingerprint = new Map<string, string[]>();
const existingAttachments = await db
.select({
id: issueAttachments.id,
byteSize: assets.byteSize,
contentType: assets.contentType,
originalFilename: assets.originalFilename,
sha256: assets.sha256,
})
.from(issueAttachments)
.innerJoin(
assets,
and(
eq(assets.companyId, issueAttachments.companyId),
eq(assets.id, issueAttachments.assetId),
),
)
.where(
and(
eq(issueAttachments.companyId, input.endpoint.companyId),
eq(issueAttachments.issueId, input.issueId),
eq(issueAttachments.issueCommentId, input.issueCommentId),
),
);
for (const existing of existingAttachments) {
const fingerprint = JSON.stringify([
existing.sha256,
existing.byteSize,
existing.contentType,
existing.originalFilename,
]);
const ids = existingByFingerprint.get(fingerprint) ?? [];
ids.push(existing.id);
existingByFingerprint.set(fingerprint, ids);
}
const storedIds: string[] = [];
const withCurrentAttachmentAuthorization = async <T>(
work: (tx: DbTransaction) => Promise<T>,
enforceUnchangedSource = false,
telegramMedia?: Attachment,
): Promise<T> => {
for (let attempt = 0; ; attempt++) {
try {
return await db.transaction(async (tx) => {
const action = await tx
.select()
.from(chatActions)
.where(
and(
eq(chatActions.companyId, input.endpoint.companyId),
eq(chatActions.endpointId, input.endpoint.id),
eq(chatActions.deliveryId, input.deliveryId),
eq(chatActions.kind, "inbound_wakeup"),
),
)
.limit(1)
.then((rows) => rows[0]);
if (
!action ||
!["preparing", "issued", "processing", "processed"].includes(
action.status,
) ||
action.payload.issueId !== input.issueId ||
action.payload.commentId !== input.issueCommentId
)
throw forbidden(
"The attachment's current chat admission is unavailable",
{ code: "chat_action_authorization_changed" },
);
const current = await authorizeInboundWakeup(tx, action);
// Delivery settlement/replay can update these receipts without
// taking the issue lock. Lock in delivery -> action order only
// after policy authorization; NOWAIT breaks cross-worker lock
// inversions and the outer loop retries the whole transaction.
const lockedDelivery = await tx
.select()
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.companyId, input.endpoint.companyId),
eq(chatDeliveries.id, input.deliveryId),
),
)
.for("update", { noWait: true })
.limit(1)
.then((rows) => rows[0]);
const lockedAction = await tx
.select()
.from(chatActions)
.where(
and(
eq(chatActions.companyId, input.endpoint.companyId),
eq(chatActions.id, action.id),
),
)
.for("update", { noWait: true })
.limit(1)
.then((rows) => rows[0]);
const scopeKeys = [
"version",
"issueId",
"commentId",
"agentId",
"sessionGeneration",
"requestedByActorType",
"requestedByActorId",
] as const;
if (
current.delivery.state !== "processing" ||
!lockedDelivery ||
!lockedAction ||
lockedDelivery.state !== "processing" ||
lockedDelivery.endpointId !== action.endpointId ||
lockedDelivery.conversationId !== action.conversationId ||
lockedDelivery.principalId !== action.principalId ||
lockedDelivery.attempts !== current.delivery.attempts ||
JSON.stringify(lifecycleRuntimeFence(lockedDelivery)) !==
JSON.stringify(lifecycleRuntimeFence(current.delivery)) ||
lockedAction.endpointId !== action.endpointId ||
lockedAction.deliveryId !== action.deliveryId ||
lockedAction.conversationId !== action.conversationId ||
lockedAction.principalId !== action.principalId ||
lockedAction.kind !== "inbound_wakeup" ||
!["preparing", "issued", "processing", "processed"].includes(
lockedAction.status,
) ||
scopeKeys.some(
(key) => lockedAction.payload[key] !== action.payload[key],
)
)
throw forbidden(
"The attachment's current chat admission is unavailable",
{ code: "chat_action_authorization_changed" },
);
if (enforceUnchangedSource) {
if (telegramMedia) {
const normalized = lockedDelivery.normalizedEvent as {
principal?: { externalId?: string };
message?: {
providerMessageId?: string;
attachments?: Array<{ recovery?: unknown }>;
};
};
const fence = lifecycleRuntimeFence(lockedDelivery);
const descriptor =
input.endpointRuntime.attachmentRecoveryDescriptor(
telegramMedia,
{
threadId: current.conversation.externalThreadId,
messageId: normalized.message?.providerMessageId ?? "",
principalExternalId: normalized.principal?.externalId,
runtimeGeneration: fence?.generation,
credentialFingerprint: fence?.credentialFingerprint,
},
);
if (
descriptor?.locator.kind !== "telegram_media" ||
!normalized.message?.attachments?.some((item) => {
const retained = item.recovery as {
version?: unknown;
provider?: unknown;
locator?: Record<string, unknown>;
attachment?: Record<string, unknown>;
} | null;
return (
retained?.version === 1 &&
retained.provider === "telegram" &&
Object.entries(descriptor.locator).every(
([key, value]) => retained.locator?.[key] === value,
) &&
Object.entries(descriptor.attachment).every(
([key, value]) => retained.attachment?.[key] === value,
)
);
})
)
throw forbidden(
"The media's admitted source is unavailable",
{ code: "chat_action_authorization_changed" },
);
}
const [source] = await tx
.select({ id: issueComments.id })
.from(issueComments)
.where(
and(
eq(issueComments.companyId, input.endpoint.companyId),
eq(issueComments.issueId, input.issueId),
eq(issueComments.id, input.issueCommentId),
isNull(issueComments.deletedAt),
sql`${issueComments.updatedAt} = ${issueComments.createdAt}`,
),
)
.for("share", { noWait: true });
// Lifecycle admission precedes its conversation drain. Pending
// source changes already invalidate this image download, even
// when the append-only original comment is still untouched.
const [lifecycle] = await tx
.select({ id: chatDeliveries.id })
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.companyId, input.endpoint.companyId),
eq(chatDeliveries.endpointId, input.endpoint.id),
or(
isNull(chatDeliveries.conversationId),
eq(
chatDeliveries.conversationId,
current.conversation.id,
),
),
ne(chatDeliveries.state, "filtered"),
inArray(chatDeliveries.eventKind, [
"message_updated",
"message_deleted",
"message_restored",
]),
sql`${chatDeliveries.normalizedEvent}->'runtimeContext' = ${JSON.stringify(lifecycleRuntimeFence(lockedDelivery))}::jsonb`,
sql`${chatDeliveries.normalizedEvent}->'message'->>'targetProviderEventId' = ${lockedDelivery.providerEventId}`,
),
)
.limit(1);
if (!source || lifecycle)
throw forbidden(
"The image's admitted source changed before attachment registration",
{
code: "chat_action_authorization_changed",
},
);
}
return await work(tx);
});
} catch (error) {
if (!isExternalChatWaitAuthorizationContention(error) || attempt >= 4)
throw error;
// Retry the complete short transaction; no network/storage work or
// backoff occurs while governance rows are locked.
await new Promise((resolve) =>
setTimeout(resolve, 20 * (attempt + 1)),
);
}
}
};
// One shared deadline bounds the sequential batch; an expired batch never
// starts another GitHub request. Other providers retain their own policy.
const githubBatchSignal =
input.endpoint.provider === "github"
? AbortSignal.timeout(GITHUB_ATTACHMENT_BATCH_TIMEOUT_MS)
: undefined;
// This new image-only lane shares one budget across token acquisition and
// downloads. DB/storage commit semantics remain explicit and unchanged.
const teamsInlineBatchSignal =
input.endpoint.provider === "microsoft-teams"
? AbortSignal.timeout(10_000)
: undefined;
for (let attachment of boundedAttachments) {
const teamsInlineImage =
input.endpoint.provider === "microsoft-teams" &&
input.endpointRuntime.attachmentRecoveryDescriptor(attachment)?.locator
.kind === "teams_inline_image";
const telegramMedia =
input.endpoint.provider === "telegram" &&
hasTelegramMediaProvenance(attachment);
const sourceBoundMedia =
teamsInlineImage ||
telegramMedia ||
input.endpoint.provider === "imessage-photon";
const requireCurrentAttachmentAuthorization =
input.endpoint.provider === "github" || sourceBoundMedia;
try {
if (teamsInlineImage && teamsInlineBatchSignal?.aborted) {
omit("download_unavailable");
continue;
}
// GitHub's anonymized upload URLs carry no trustworthy MIME metadata.
// Resolve public bytes only here, after durable comment admission, and
// then apply the same storage/type policy as every native attachment.
if (input.endpoint.provider === "github") {
await withCurrentAttachmentAuthorization(async () => {});
attachment = await prepareGitHubPublicAttachment(
attachment,
githubBatchSignal,
async (request, signal) => {
await withCurrentAttachmentAuthorization(async () => {});
return (
(await input.endpointRuntime.resolveGitHubAttachmentComment?.(
request,
signal,
)) ?? null
);
},
);
}
if (
requireCurrentAttachmentAuthorization &&
input.endpoint.provider !== "github"
)
await withCurrentAttachmentAuthorization(
async () => {},
sourceBoundMedia,
telegramMedia ? attachment : undefined,
);
if (
attachment.size !== undefined &&
attachment.size > MAX_ATTACHMENT_BYTES
) {
omit("declared_too_large");
continue;
}
if (!attachment.fetchData) {
omit("download_unavailable");
continue;
}
const originalFilename = sanitizeFilename(attachment.name);
let contentType = normalizeUploadAttachmentContentType({
contentType: normalizeContentType(
attachment.mimeType ?? "application/octet-stream",
),
originalFilename,
isAllowedContentType,
});
// Reject a provider-declared type before allocating the downloaded
// payload. The actual byte length is checked again after the adapter's
// bounded fetch contract resolves.
// A filename may infer an Office MIME for ordinary documents, but it
// must not bypass byte identification for native media without MIME.
const identifyTelegram =
telegramMedia &&
telegramMediaNeedsIdentification(attachment) &&
normalizeContentType(
attachment.mimeType ?? "application/octet-stream",
) === "application/octet-stream";
if (!isAllowedContentType(contentType) && !identifyTelegram) {
omit("unsupported_type");
continue;
}
const fetched = teamsInlineImage
? await input.endpointRuntime.fetchTeamsInlineImage(
attachment,
teamsInlineBatchSignal!,
)
: await attachment.fetchData();
const body = Buffer.isBuffer(fetched) ? fetched : Buffer.from(fetched);
if (body.length === 0) {
omit("empty_download");
continue;
}
if (body.length > MAX_ATTACHMENT_BYTES) {
omit("downloaded_too_large");
continue;
}
if (identifyTelegram) {
contentType =
identifyTelegramMedia(attachment, body) ??
"application/octet-stream";
if (!isAllowedContentType(contentType)) {
omit("unsupported_type");
continue;
}
}
const photonSourceHash = createHash("sha256")
.update(body)
.digest("hex");
if (input.endpoint.provider === "imessage-photon") {
await validatePhotonImage(body, contentType);
const companion = takePhotonCompanion(body);
if (companion?.unavailable) omit("companion_unavailable");
else if (companion) {
const video: Attachment = { type: "video", name: companion.fileName, mimeType: companion.mimeType, size: companion.data.length, fetchData: async () => companion.data };
photonDerivatives.set(video, { source: attachment, sourceHash: photonSourceHash, kind: "live_photo_video" });
boundedAttachments.push(video);
}
if (
HEIF_CONTENT_TYPES.has(contentType) &&
isAllowedContentType("image/jpeg")
) {
try {
const preview = await photonHeifPreview(body);
const derivative: Attachment = {
type: "image",
name: `${originalFilename} (JPEG preview).jpg`,
mimeType: "image/jpeg",
size: preview.length,
fetchData: async () => preview,
};
photonDerivatives.set(derivative, { source: attachment, sourceHash: photonSourceHash, kind: "heif_jpeg_preview" });
boundedAttachments.push(derivative);
} catch {
omit("preview_unavailable");
}
}
}
const fingerprint = JSON.stringify([
photonSourceHash,
body.length,
contentType,
originalFilename,
]);
const existingIds = existingByFingerprint.get(fingerprint);
const existingId = existingIds?.shift();
if (requireCurrentAttachmentAuthorization) {
// Revoke during either API resolution or the anonymous byte download
// must prevent storage and attachment registration for that input.
await withCurrentAttachmentAuthorization(
async () => {},
sourceBoundMedia,
telegramMedia ? attachment : undefined,
);
}
const registerPhotonProvenance = async (attachmentId: string) => {
if (input.endpoint.provider !== "imessage-photon") return;
const relation = photonDerivatives.get(attachment);
if (!relation) {
originalPhotonAttachments.set(attachment, attachmentId);
return;
}
const originalAttachmentId =
originalPhotonAttachments.get(relation.source);
if (!originalAttachmentId)
throw new Error("Photon related attachment source is missing");
await db
.insert(chatActions)
.values({
companyId: input.endpoint.companyId,
endpointId: input.endpoint.id,
deliveryId: input.deliveryId,
kind: relation.kind === "heif_jpeg_preview" ? "attachment_derivative" : "attachment_companion",
providerActionId: `photon-${relation.kind}:${input.deliveryId}:${originalAttachmentId}`,
payload: {
originalAttachmentId,
derivativeAttachmentId: attachmentId,
originalSha256: relation.sourceHash,
derivativeSha256: photonSourceHash,
kind: relation.kind,
},
status: "processed",
})
.onConflictDoNothing();
};
if (existingId) {
storedIds.push(existingId);
await registerPhotonProvenance(existingId);
continue;
}
const stored = await options.storage.putFile({
companyId: input.endpoint.companyId,
namespace: `issues/${input.issueId}`,
originalFilename,
contentType,
body,
});
const registration = {
issueId: input.issueId,
issueCommentId: input.issueCommentId,
provider: stored.provider,
objectKey: stored.objectKey,
contentType: stored.contentType,
byteSize: stored.byteSize,
sha256: stored.sha256,
originalFilename: stored.originalFilename,
createdByUserId: input.actorUserId,
};
let row;
try {
row = requireCurrentAttachmentAuthorization
? await withCurrentAttachmentAuthorization(
(tx) =>
issueService(tx as unknown as Db).createAttachment(
registration,
),
sourceBoundMedia,
telegramMedia ? attachment : undefined,
)
: await issuesSvc.createAttachment(registration);
} catch (error) {
// An explicit authorization denial happens before registration. Do
// not delete on ambiguous database errors which may have committed.
if (
requireCurrentAttachmentAuthorization &&
isExternalActionAuthorizationChange(error)
)
await options.storage
.deleteObject(input.endpoint.companyId, stored.objectKey)
.catch(() => undefined);
throw error;
}
storedIds.push(row.id);
await registerPhotonProvenance(row.id);
} catch (error) {
if (isExternalActionAuthorizationChange(error)) throw error;
if (
error instanceof PhotonError &&
["attachment_not_ready", "network", "quota", "credentials"].includes(
error.code,
)
)
throw error;
// Use the closed current-input omission vocabulary consumed by native
// prompts; provider-specific diagnostics remain redacted log codes.
omit(
error instanceof GitHubAttachmentUnavailableError
? "download_unavailable"
: "processing_failed",
);
// A malformed or unavailable provider attachment must not strand the
// durable text delivery. The rejected file is intentionally omitted;
// the delivery remains auditable through its normalized attachment
// metadata without persisting provider credentials or download URLs.
logger.warn(
{
endpointId: input.endpoint.id,
issueId: input.issueId,
...(error instanceof GitHubAttachmentUnavailableError
? {
deliveryId: input.deliveryId,
attachmentDiagnosticCode: error.code,
}
: {}),
attachmentName: sanitizeFilename(attachment.name),
error: redactError(error),
},
"external chat attachment was rejected",
);
}
}
return { storedIds, omissionReasons };
}
function attachmentOmissionDetail(result: {
omissionReasons: Record<string, number>;
}): string | null {
const entries = Object.entries(result.omissionReasons).filter(
([, count]) => count > 0,
);
const omitted = entries.reduce((total, [, count]) => total + count, 0);
if (!omitted) return null;
const reasons = entries
.map(([reason, count]) => `${reason.replaceAll("_", " ")}: ${count}`)
.join(", ");
return `${omitted} external attachment${omitted === 1 ? " was" : "s were"} omitted (${reasons})`;
}
return { ingestAttachments, attachmentOmissionDetail };
}
@@ -0,0 +1,18 @@
import { sql } from "drizzle-orm";
import { chatTeamsFileTransfers } from "@paperclipai/db";
export const fileTransferProjectionColumns = {
publicationId: chatTeamsFileTransfers.publicationId,
phase: chatTeamsFileTransfers.phase,
version: chatTeamsFileTransfers.version,
filename: chatTeamsFileTransfers.filename,
expiresAt: chatTeamsFileTransfers.expiresAt,
consentMessageId: chatTeamsFileTransfers.consentMessageId,
responseActivityId: chatTeamsFileTransfers.responseActivityId,
fileInfoMessageId: chatTeamsFileTransfers.fileInfoMessageId,
operatorConfirmed: sql<boolean>`
coalesce(${chatTeamsFileTransfers.privateState}#>>'{resolution,schema}', '') = 'paperclip.teams.file-resolution.v1'
and coalesce(${chatTeamsFileTransfers.privateState}#>>'{resolution,action}', '') = 'mark_delivered'
and coalesce(${chatTeamsFileTransfers.privateState}#>>'{resolution,fromPhase}', '') = 'file_info_unknown'
`,
};
@@ -0,0 +1,294 @@
import {
createHash,
} from "node:crypto";
import {
and,
eq,
inArray,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
assets,
chatPublications,
issueAttachments,
} from "@paperclipai/db";
import type {
SafeChatPublicationPayload,
} from "@paperclipai/shared";
import {
isAllowedContentType,
MAX_ATTACHMENT_BYTES,
} from "../../attachment-types.js";
import type { StorageService } from "../../storage/types.js";
import {
isTeamsInlineImageContentType,
TEAMS_INLINE_IMAGE_MAX_BYTES,
} from "../chat-teams-inline-images.js";
import type {
FileUpload,
} from "chat";
const OUTBOUND_ATTACHMENT_STORAGE_TIMEOUT_MS = 10_000;
class OutboundAttachmentHydrationError extends Error {
readonly code = "CHAT_ATTACHMENT_HYDRATION_FAILED";
constructor(message: string, cause?: unknown) {
super(message, cause === undefined ? undefined : { cause });
this.name = "OutboundAttachmentHydrationError";
}
}
class OutboundAttachmentValidationError extends Error {
readonly code = "CHAT_ATTACHMENT_VALIDATION_FAILED";
constructor(message: string) {
super(message);
this.name = "OutboundAttachmentValidationError";
}
}
function isOutboundAttachmentHydrationError(
error: unknown,
): error is OutboundAttachmentHydrationError {
return error instanceof OutboundAttachmentHydrationError;
}
function isOutboundAttachmentValidationError(
error: unknown,
): error is OutboundAttachmentValidationError {
return error instanceof OutboundAttachmentValidationError;
}
/**
* Read one already-authorized attachment before any provider transport starts.
* The database digest, rather than mutable object-store metadata, is the
* integrity boundary. Both acquisition and streaming are bounded so a stuck
* storage backend cannot hold the publication lane indefinitely.
*/
export async function hydrateOutboundAttachment(input: {
storage: StorageService;
companyId: string;
objectKey: string;
byteSize: number;
sha256: string;
filename: string;
mimeType: string;
timeoutMs?: number;
}): Promise<FileUpload> {
const timeoutMs =
typeof input.timeoutMs === "number" &&
Number.isFinite(input.timeoutMs) &&
input.timeoutMs > 0
? input.timeoutMs
: OUTBOUND_ATTACHMENT_STORAGE_TIMEOUT_MS;
if (
!Number.isSafeInteger(input.byteSize) ||
input.byteSize <= 0 ||
input.byteSize > MAX_ATTACHMENT_BYTES ||
!/^[0-9a-f]{64}$/iu.test(input.sha256)
) {
throw new OutboundAttachmentHydrationError(
"Chat publication attachment metadata is invalid",
);
}
const acquisitionTimeoutError = new OutboundAttachmentHydrationError(
"Chat publication attachment storage read timed out",
);
let acquisitionTimedOut = false;
let rejectAcquisition!: (error: Error) => void;
const acquisitionTimeout = new Promise<never>((_resolve, reject) => {
rejectAcquisition = reject;
});
const acquisitionTimer = setTimeout(() => {
acquisitionTimedOut = true;
rejectAcquisition(acquisitionTimeoutError);
}, timeoutMs);
acquisitionTimer.unref?.();
const objectPromise = Promise.resolve()
.then(() => input.storage.getObject(input.companyId, input.objectKey))
.then((object) => {
if (acquisitionTimedOut) object.stream.destroy();
return object;
})
.catch((error) => {
throw isOutboundAttachmentHydrationError(error)
? error
: new OutboundAttachmentHydrationError(
"Chat publication attachment storage read failed",
error,
);
});
let object: Awaited<ReturnType<StorageService["getObject"]>>;
try {
object = await Promise.race([objectPromise, acquisitionTimeout]);
} finally {
clearTimeout(acquisitionTimer);
}
if (
object.contentLength !== undefined &&
object.contentLength !== input.byteSize
) {
object.stream.destroy();
throw new OutboundAttachmentHydrationError(
"Chat publication attachment size changed after registration",
);
}
const streamTimeoutError = new OutboundAttachmentHydrationError(
"Chat publication attachment storage read timed out",
);
const streamTimer = setTimeout(() => {
object.stream.destroy(streamTimeoutError);
}, timeoutMs);
streamTimer.unref?.();
const chunks: Buffer[] = [];
const digest = createHash("sha256");
let bytes = 0;
try {
for await (const chunk of object.stream) {
const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
bytes += buffer.length;
if (bytes > input.byteSize || bytes > MAX_ATTACHMENT_BYTES) {
object.stream.destroy();
throw new OutboundAttachmentHydrationError(
"Chat publication attachment exceeded its streaming byte limit",
);
}
digest.update(buffer);
chunks.push(buffer);
}
} catch (error) {
throw isOutboundAttachmentHydrationError(error)
? error
: new OutboundAttachmentHydrationError(
"Chat publication attachment storage read failed",
error,
);
} finally {
clearTimeout(streamTimer);
}
const actualDigest = digest.digest("hex");
if (bytes !== input.byteSize || actualDigest !== input.sha256.toLowerCase()) {
if (!object.stream.destroyed) object.stream.destroy();
throw new OutboundAttachmentHydrationError(
"Chat publication attachment integrity changed after registration",
);
}
return {
data: Buffer.concat(chunks, bytes),
filename: input.filename,
mimeType: input.mimeType,
};
}
export { isOutboundAttachmentHydrationError, isOutboundAttachmentValidationError };
type OutboundAttachmentsOptions = { storage?: StorageService; };
export function createOutboundAttachments(db: Db, options: OutboundAttachmentsOptions) {
async function publicationFiles(
publication: typeof chatPublications.$inferSelect,
payload: SafeChatPublicationPayload,
preparation: { teamsInlineImagesOnly?: boolean } = {},
): Promise<FileUpload[]> {
if (!payload.attachmentIds?.length) return [];
try {
const rows = await db
.select({
id: issueAttachments.id,
issueId: issueAttachments.issueId,
issueCommentId: issueAttachments.issueCommentId,
objectKey: assets.objectKey,
contentType: assets.contentType,
byteSize: assets.byteSize,
sha256: assets.sha256,
originalFilename: assets.originalFilename,
})
.from(issueAttachments)
.innerJoin(assets, eq(issueAttachments.assetId, assets.id))
.where(
and(
eq(issueAttachments.companyId, publication.companyId),
eq(issueAttachments.issueId, publication.issueId),
inArray(issueAttachments.id, payload.attachmentIds),
),
);
const byId = new Map(rows.map((row) => [row.id, row]));
const orderedRows = payload.attachmentIds.map((attachmentId) => {
const row = byId.get(attachmentId);
if (
!row ||
(publication.commentId &&
row.issueCommentId !== publication.commentId) ||
row.byteSize <= 0 ||
row.byteSize > MAX_ATTACHMENT_BYTES ||
!row.objectKey.trim() ||
!/^[0-9a-f]{64}$/iu.test(row.sha256) ||
!isAllowedContentType(row.contentType)
)
throw new OutboundAttachmentValidationError(
"Chat publication attachment is invalid or outside its authorized task comment",
);
return row;
});
// Picture messages have a different Teams contract from arbitrary files.
// Preserve the normal scope/metadata checks above, but never hydrate a
// document or oversized image just to discover it needs a task link.
if (
preparation.teamsInlineImagesOnly &&
orderedRows.some(
(row) =>
row.byteSize > TEAMS_INLINE_IMAGE_MAX_BYTES ||
!isTeamsInlineImageContentType(row.contentType),
)
) {
return [];
}
// Authorization and persisted metadata must fail definitively even when
// storage is unavailable; do not mask an invalid file as a transient outage.
if (!options.storage)
throw new OutboundAttachmentHydrationError(
"Attachment storage is unavailable for chat publication",
);
const files: FileUpload[] = [];
for (const row of orderedRows) {
files.push(
await hydrateOutboundAttachment({
storage: options.storage,
companyId: publication.companyId,
objectKey: row.objectKey,
byteSize: row.byteSize,
sha256: row.sha256,
filename: row.originalFilename ?? `attachment-${row.id}`,
mimeType: row.contentType,
}),
);
}
return files;
} catch (error) {
if (
isOutboundAttachmentHydrationError(error) ||
isOutboundAttachmentValidationError(error)
) {
throw error;
}
throw new OutboundAttachmentHydrationError(
"Chat publication attachment preparation failed",
error,
);
}
}
return { publicationFiles };
}
@@ -0,0 +1,957 @@
import { createTelegramCallbackNoticeAuthority } from "./telegram/callback-notices.js";
import { slackThreadChannelId } from "./slack/messages.js";
import type { EndpointRow, DbTransaction, DbOrTransaction, LifecycleRuntimeFence } from "./runtime-types.js";
import { baseTeamsConversationId } from "./microsoft-teams/resources.js";
import { dueChatAction, PROVIDER_EFFECT_STALE_MS } from "../chat-action-work.js";
import { notifyChatDeliveryWork, notifyChatActionWork } from "../chat-work-notifications.js";
import type { ChatEndpointRuntime } from "../voice/voice-runtime.js";
import { chatCredentialMutationLease } from "../chat-credential-mutation-lease.js";
import { projectSafeChatPublicationText } from "../chat-publication-projection.js";
import { PhotonChatAdapter } from "../photon/adapter.js";
import {
and,
asc,
eq,
inArray,
lte,
sql,
} from "drizzle-orm";
import {
teamsConversationId,
} from "../chat-control-chronology.js";
import type { Db } from "@paperclipai/db";
import {
chatActions,
chatConversations,
chatDeliveries,
chatEndpointResources,
chatEndpoints,
toolConnections,
} from "@paperclipai/db";
import type {
ChatProvider,
} from "@paperclipai/shared";
import {
parseTelegramCallbackReceipt,
telegramCallbackThreadId,
TELEGRAM_PRIVATE_ACTION_UNAVAILABLE,
type TelegramCallbackReceipt,
} from "../chat-telegram-ephemeral.js";
import { logger } from "../../middleware/logger.js";
import { redactSensitiveText } from "../../redaction.js";
import { classifyChatPublicationError } from "../chat-publication-errors.js";
import type {
Thread,
} from "chat";
import type { EndpointRecord } from "./runtime-types.js";
type ProviderEffectTarget = {
post: Thread["post"];
postEphemeral?: Thread["postEphemeral"];
};
type ProviderEffectPayload = {
version: 1;
effect: "thread_message" | "ephemeral_message" | "telegram_callback_notice";
telegramCallback?: TelegramCallbackReceipt;
telegramAuthorizationSha256?: string;
authorizationMode?: "principal" | "safe_notice";
threadId: string;
userId?: string;
text: string;
fallbackText?: string;
settleDelivery: boolean;
completeConversationId?: string;
resourceId?: string;
runtimeGeneration: number;
credentialFingerprint: string;
};
function providerEffectPayload(
payload: Record<string, unknown>,
): ProviderEffectPayload | null {
if (
payload.version !== 1 ||
(payload.effect !== "thread_message" &&
payload.effect !== "ephemeral_message" &&
payload.effect !== "telegram_callback_notice") ||
typeof payload.threadId !== "string" ||
!payload.threadId ||
typeof payload.text !== "string" ||
!payload.text ||
typeof payload.settleDelivery !== "boolean" ||
typeof payload.runtimeGeneration !== "number" ||
!Number.isSafeInteger(payload.runtimeGeneration) ||
typeof payload.credentialFingerprint !== "string"
) {
return null;
}
if (
payload.effect === "ephemeral_message" &&
(typeof payload.userId !== "string" || !payload.userId)
) {
return null;
}
for (const key of [
"authorizationMode",
"userId",
"fallbackText",
"completeConversationId",
"resourceId",
] as const) {
const value = payload[key];
if (value !== undefined && (typeof value !== "string" || !value)) {
return null;
}
}
if (
payload.authorizationMode !== undefined &&
payload.authorizationMode !== "principal" &&
payload.authorizationMode !== "safe_notice"
) {
return null;
}
if (payload.effect === "telegram_callback_notice") {
const receipt = parseTelegramCallbackReceipt(payload.telegramCallback);
if (
!receipt ||
payload.authorizationMode !== "safe_notice" ||
payload.text !== TELEGRAM_PRIVATE_ACTION_UNAVAILABLE ||
payload.threadId !== telegramCallbackThreadId(receipt) ||
payload.userId !== receipt.receiverUserId ||
payload.settleDelivery !== false ||
payload.fallbackText !== undefined ||
payload.completeConversationId !== undefined ||
typeof payload.telegramAuthorizationSha256 !== "string" ||
!/^[a-f0-9]{64}$/.test(payload.telegramAuthorizationSha256)
)
return null;
} else if (
payload.telegramCallback !== undefined ||
payload.telegramAuthorizationSha256 !== undefined
)
return null;
return payload as ProviderEffectPayload;
}
export type { ProviderEffectTarget, ProviderEffectPayload };
export { providerEffectPayload };
type ProviderEffectsOptions = {
endpointRecord: (endpointId: string) => Promise<EndpointRecord | null>;
runtimeCallbackEndpoint: (tx: DbTransaction, endpointId: string, context: LifecycleRuntimeFence, statuses: EndpointRow["status"][]) => Promise<EndpointRow | null>;
withCredentialMutationLease: ReturnType<typeof chatCredentialMutationLease>;
runtimeFor: (endpoint: EndpointRow) => Promise<ChatEndpointRuntime>;
invalidateRuntime: (endpointId: string) => Promise<boolean>;
lockCurrentPrincipalAuthorization: (tx: DbOrTransaction, endpoint: EndpointRow, principalId: string) => Promise<{ allowed: boolean }>;
redactError: (error: unknown) => string;
MAX_ERROR_TEXT: number;
nonDirectDestinationAllowed: (endpoint: EndpointRow, resource: typeof chatEndpointResources.$inferSelect | null | undefined) => boolean;
lockTelegramCallbackNoticeAuthority: ReturnType<typeof createTelegramCallbackNoticeAuthority>["lockTelegramCallbackNoticeAuthority"];
};
export function createProviderEffects(db: Db, options: ProviderEffectsOptions) {
const { endpointRecord, runtimeCallbackEndpoint, withCredentialMutationLease, runtimeFor, invalidateRuntime, lockCurrentPrincipalAuthorization, redactError, MAX_ERROR_TEXT, nonDirectDestinationAllowed, lockTelegramCallbackNoticeAuthority } = options;
async function stageProviderEffect(
database: DbOrTransaction,
input: {
endpoint: EndpointRow;
deliveryId?: string | null;
conversationId?: string | null;
principalId?: string | null;
providerActionId: string;
payload: Omit<
ProviderEffectPayload,
"runtimeGeneration" | "credentialFingerprint"
>;
runtimeContext: LifecycleRuntimeFence;
},
) {
await notifyChatActionWork(database, "provider_effect");
const [inserted] = await database
.insert(chatActions)
.values({
companyId: input.endpoint.companyId,
endpointId: input.endpoint.id,
deliveryId: input.deliveryId ?? null,
conversationId: input.conversationId ?? null,
principalId: input.principalId ?? null,
kind: "provider_effect",
providerActionId: input.providerActionId,
payload: {
...input.payload,
runtimeGeneration: input.runtimeContext.generation,
credentialFingerprint: input.runtimeContext.credentialFingerprint,
},
status: "received",
})
.onConflictDoNothing()
.returning();
if (inserted) return inserted;
return database
.select()
.from(chatActions)
.where(
and(
eq(chatActions.endpointId, input.endpoint.id),
eq(chatActions.providerActionId, input.providerActionId),
eq(chatActions.kind, "provider_effect"),
),
)
.then((rows) => rows[0] ?? null);
}
function providerEffectDisposition(
error: unknown,
attempt: number,
providerEffectCompleted: boolean,
) {
if (!providerEffectCompleted)
return classifyChatPublicationError(error, attempt);
return {
kind: "delivery_unknown" as const,
reason:
"Provider effect completed, but Paperclip could not confirm its durable result",
};
}
async function finalizeProviderEffectFailure(input: {
action: typeof chatActions.$inferSelect;
endpoint: EndpointRow;
payload: ProviderEffectPayload;
disposition: ReturnType<typeof providerEffectDisposition>;
attempt: number;
}): Promise<boolean> {
const failure = redactSensitiveText(input.disposition.reason).slice(
0,
MAX_ERROR_TEXT,
);
const retryable = input.disposition.kind === "retry";
const retryAt =
input.disposition.kind === "retry"
? new Date(Date.now() + input.disposition.retryAfterMs)
: null;
let invalidateCurrentRuntime = false;
const finalized = await db.transaction(async (tx) => {
// Admission locks the endpoint before its delivery. Keep that order
// when failure settlement also needs to change endpoint/resource health.
const currentEndpoint =
input.disposition.kind === "endpoint_attention" ||
input.disposition.kind === "resource_unavailable"
? await runtimeCallbackEndpoint(
tx,
input.endpoint.id,
{
generation: input.payload.runtimeGeneration,
credentialFingerprint: input.payload.credentialFingerprint,
},
["verifying", "active"],
)
: null;
await notifyChatActionWork(tx, "provider_effect");
const [ownedAction] = await tx
.update(chatActions)
.set({
status:
input.disposition.kind === "delivery_unknown"
? "delivery_unknown"
: "failed",
result: {
code: `provider_effect_${input.disposition.kind}`,
attempts: input.attempt,
redactedError: failure,
retryable,
retryAt: retryAt?.toISOString() ?? null,
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, input.action.id),
eq(chatActions.status, "processing"),
sql`(${chatActions.result}->>'attempts')::int = ${input.attempt}`,
),
)
.returning({ id: chatActions.id });
if (!ownedAction) return false;
if (input.payload.settleDelivery && input.action.deliveryId) {
await notifyChatDeliveryWork(tx);
await tx
.update(chatDeliveries)
.set({
state: retryable ? "retry" : "failed",
nextAttemptAt: retryAt,
redactedError:
input.disposition.kind === "delivery_unknown"
? `Provider response is unconfirmed: ${failure}`.slice(
0,
MAX_ERROR_TEXT,
)
: failure,
updatedAt: new Date(),
})
.where(
and(
eq(chatDeliveries.id, input.action.deliveryId),
eq(chatDeliveries.state, "processing"),
),
);
}
if (input.disposition.kind === "endpoint_attention" && currentEndpoint) {
await tx
.update(chatEndpoints)
.set({
status: "attention",
healthMessage: "Provider credentials or permissions need attention",
lastError: failure,
updatedAt: new Date(),
})
.where(eq(chatEndpoints.id, currentEndpoint.id));
await tx
.update(toolConnections)
.set({
status: "disabled",
enabled: false,
healthStatus: "degraded",
healthMessage: "Provider credentials or permissions need attention",
lastError: failure,
healthCheckedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(toolConnections.id, currentEndpoint.connectionId));
invalidateCurrentRuntime = true;
} else if (
input.disposition.kind === "resource_unavailable" &&
input.payload.resourceId &&
currentEndpoint
) {
await tx
.update(chatEndpointResources)
.set({ availability: "unavailable", updatedAt: new Date() })
.where(
and(
eq(chatEndpointResources.id, input.payload.resourceId),
eq(chatEndpointResources.endpointId, currentEndpoint.id),
),
);
}
return true;
});
if (finalized && invalidateCurrentRuntime) {
await invalidateRuntime(input.endpoint.id).catch(() => undefined);
}
return finalized;
}
async function quarantineStaleProviderEffect(
action: typeof chatActions.$inferSelect,
) {
const payload = providerEffectPayload(action.payload);
const failure =
"Provider delivery could not be confirmed after the worker stopped. The effect will not be replayed automatically.";
await db.transaction(async (tx) => {
await notifyChatActionWork(tx, "provider_effect");
const [quarantined] = await tx
.update(chatActions)
.set({
status: "delivery_unknown",
result: {
code: "provider_effect_delivery_unknown",
redactedError: failure,
retryable: false,
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "processing"),
eq(chatActions.updatedAt, action.updatedAt),
sql`(${chatActions.result}->>'attempts')::int = ${
typeof action.result?.attempts === "number"
? action.result.attempts
: -1
}`,
lte(
chatActions.updatedAt,
new Date(Date.now() - PROVIDER_EFFECT_STALE_MS),
),
),
)
.returning({ id: chatActions.id });
if (quarantined && payload?.settleDelivery && action.deliveryId) {
await notifyChatDeliveryWork(tx);
await tx
.update(chatDeliveries)
.set({
state: "failed",
nextAttemptAt: null,
redactedError: failure,
updatedAt: new Date(),
})
.where(
and(
eq(chatDeliveries.id, action.deliveryId),
eq(chatDeliveries.state, "processing"),
),
);
}
});
}
function providerEffectThreadResourceId(
provider: ChatProvider,
threadId: string,
): string | null {
if (provider === "slack") {
return (
slackThreadChannelId(threadId) ??
(!threadId.includes(":") ? threadId : null)
);
}
if (provider === "telegram") {
return /^telegram:([^:]+)/.exec(threadId)?.[1] ?? null;
}
if (provider === "github") {
return /^github:([^:]+)/.exec(threadId)?.[1]?.toLowerCase() ?? null;
}
if (provider === "discord") {
return /^discord:[^:]+:([^:]+)/.exec(threadId)?.[1] ?? null;
}
const conversationId = teamsConversationId(threadId);
return conversationId ? baseTeamsConversationId(conversationId) : null;
}
async function lockCurrentProviderEffectAuthorization(
tx: DbTransaction,
action: typeof chatActions.$inferSelect,
endpoint: EndpointRow,
payload: ProviderEffectPayload,
): Promise<boolean> {
if (payload.effect === "telegram_callback_notice") {
const receipt = parseTelegramCallbackReceipt(payload.telegramCallback);
if (!receipt || !action.deliveryId) return false;
const delivery = await tx
.select()
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.id, action.deliveryId),
eq(chatDeliveries.companyId, action.companyId),
eq(chatDeliveries.endpointId, action.endpointId),
eq(chatDeliveries.state, "filtered"),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
const retained =
delivery &&
parseTelegramCallbackReceipt(delivery.normalizedEvent.telegramCallback);
if (
!delivery ||
delivery.principalId !== action.principalId ||
retained?.sourceSha256 !== receipt.sourceSha256 ||
delivery.normalizedEvent.telegramAuthorizationSha256 !==
payload.telegramAuthorizationSha256
)
return false;
return (
(await lockTelegramCallbackNoticeAuthority(
tx,
endpoint,
receipt,
action.principalId,
)) === payload.telegramAuthorizationSha256
);
}
const conversation = action.conversationId
? await tx
.select()
.from(chatConversations)
.where(
and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
eq(chatConversations.id, action.conversationId),
inArray(chatConversations.state, ["active", "waiting"]),
),
)
.for("update")
.then((rows) => rows[0] ?? null)
: null;
if (action.conversationId && !conversation) return false;
const inferredResourceId = providerEffectThreadResourceId(
endpoint.provider,
payload.threadId,
);
const resource = await tx
.select()
.from(chatEndpointResources)
.where(
and(
eq(chatEndpointResources.companyId, endpoint.companyId),
eq(chatEndpointResources.endpointId, endpoint.id),
payload.resourceId || conversation?.resourceId
? eq(
chatEndpointResources.id,
payload.resourceId ?? conversation!.resourceId!,
)
: inferredResourceId
? eq(chatEndpointResources.providerResourceId, inferredResourceId)
: sql`false`,
),
)
.for("update")
.then((rows) => rows[0] ?? null);
const slackChannelId =
endpoint.provider === "slack"
? slackThreadChannelId(payload.threadId)
: null;
const isDirectMessage =
conversation?.isDirectMessage === true ||
resource?.type === "direct_message" ||
Boolean(slackChannelId && /^D[A-Z0-9]+$/i.test(slackChannelId));
const destinationAllowed = isDirectMessage
? endpoint.allowDirectMessages
: nonDirectDestinationAllowed(endpoint, resource);
if (payload.authorizationMode === "safe_notice") return true;
if (!destinationAllowed) return false;
if (!action.principalId) return false;
return (
await lockCurrentPrincipalAuthorization(tx, endpoint, action.principalId)
).allowed;
}
async function cancelUnauthorizedProviderEffect(
tx: DbTransaction,
action: typeof chatActions.$inferSelect,
attempt: number,
): Promise<boolean> {
await notifyChatActionWork(tx, "provider_effect");
const [cancelled] = await tx
.update(chatActions)
.set({
status: "cancelled",
result: {
attempts: attempt,
code: "provider_effect_no_longer_authorized",
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "processing"),
sql`(${chatActions.result}->>'attempts')::int = ${attempt}`,
),
)
.returning({ id: chatActions.id });
if (!cancelled) return false;
if (action.deliveryId) {
await notifyChatDeliveryWork(tx);
await tx
.update(chatDeliveries)
.set({
state: "filtered",
processedAt: new Date(),
nextAttemptAt: null,
redactedError:
"Provider reply was suppressed because current chat authorization changed",
updatedAt: new Date(),
})
.where(
and(
eq(chatDeliveries.id, action.deliveryId),
eq(chatDeliveries.state, "processing"),
),
);
}
return true;
}
async function processProviderEffect(
actionId: string,
liveTarget?: ProviderEffectTarget,
): Promise<"processed" | "pending" | "failed" | "delivery_unknown"> {
const initial = await db
.select()
.from(chatActions)
.where(
and(
eq(chatActions.id, actionId),
eq(chatActions.kind, "provider_effect"),
),
)
.then((rows) => rows[0] ?? null);
if (!initial) return "failed";
const initialEndpoint = await endpointRecord(initial.endpointId);
if (!initialEndpoint) return "failed";
const processClaimedEffect = async () => {
let action = await db
.select()
.from(chatActions)
.where(eq(chatActions.id, actionId))
.then((rows) => rows[0] ?? null);
if (!action) return "failed";
const payload = providerEffectPayload(action.payload);
if (!payload) {
await db.transaction(async (tx) => {
await notifyChatActionWork(tx, "provider_effect");
return tx
.update(chatActions)
.set({
status: "failed",
result: { code: "provider_effect_payload_invalid" },
updatedAt: new Date(),
})
.where(eq(chatActions.id, action.id));
});
return "failed";
}
if (action.status === "processed") return "processed";
if (action.status === "delivery_unknown") return "delivery_unknown";
if (action.status === "processing") {
if (action.updatedAt > new Date(Date.now() - PROVIDER_EFFECT_STALE_MS))
return "pending";
await quarantineStaleProviderEffect(action);
return "delivery_unknown";
}
if (action.status === "failed") {
if (action.result?.retryable !== true) return "failed";
const retryAt =
typeof action.result.retryAt === "string"
? new Date(action.result.retryAt)
: null;
if (retryAt && retryAt > new Date()) return "pending";
} else if (action.status !== "received") {
return "failed";
}
const attempt =
(typeof action.result?.attempts === "number"
? action.result.attempts
: 0) + 1;
const record = await endpointRecord(action.endpointId);
let providerEffectCompleted = false;
try {
if (!record) {
throw Object.assign(
new Error("Chat endpoint is unavailable for this provider effect"),
{ code: "CHAT_PROVIDER_PRETRANSPORT_REJECTED" },
);
}
return await withCredentialMutationLease(
record.endpoint,
async (credentialLease) => {
const claim = await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const currentAction = await tx
.select()
.from(chatActions)
.where(
and(
eq(chatActions.id, action!.id),
eq(chatActions.kind, "provider_effect"),
eq(chatActions.status, action!.status),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
if (!currentAction) return null;
const currentEndpoint = await runtimeCallbackEndpoint(
tx,
currentAction.endpointId,
{
generation: payload.runtimeGeneration,
credentialFingerprint: payload.credentialFingerprint,
},
["verifying", "active"],
);
const authorized = currentEndpoint
? await lockCurrentProviderEffectAuthorization(
tx,
currentAction,
currentEndpoint,
payload,
)
: false;
await notifyChatActionWork(tx, "provider_effect");
const [transportClaim] = await tx
.update(chatActions)
.set({
status: "processing",
result: { attempts: attempt },
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, currentAction.id),
eq(chatActions.status, currentAction.status),
),
)
.returning();
if (!transportClaim) return null;
if (!currentEndpoint || !authorized) {
const cancelled = await cancelUnauthorizedProviderEffect(
tx,
transportClaim,
attempt,
);
if (!cancelled) {
throw new Error(
"Provider effect ownership changed before authorization cancellation",
);
}
return { outcome: "cancelled" as const };
}
await credentialLease.assertOwned(tx);
return {
action: transportClaim,
endpoint: currentEndpoint,
outcome: "claimed" as const,
};
});
if (!claim) return "pending";
if (claim.outcome === "cancelled") return "failed";
action = claim.action;
// `processing` is the durable, single-owner transport claim and
// authorization linearization point. Its transaction has committed
// before provider I/O begins. The renewable credential lease keeps
// runtime identity stable without holding database row locks.
let sent: { id: string; threadId: string } | null = null;
if (payload.effect === "telegram_callback_notice") {
sent = await (
await runtimeFor(claim.endpoint)
).sendTelegramCallbackNotice(
payload.telegramCallback!,
payload.text,
);
} else {
const target =
liveTarget ??
(await runtimeFor(claim.endpoint)).thread(payload.threadId);
if (payload.effect === "ephemeral_message") {
if (!target.postEphemeral) {
if (!payload.fallbackText) {
throw Object.assign(
new Error("Provider does not support ephemeral messages"),
{ code: "CHAT_PROVIDER_PRETRANSPORT_REJECTED" },
);
}
sent = await target.post(payload.fallbackText);
} else {
try {
sent = await target.postEphemeral(
payload.userId!,
payload.text,
{ fallbackToDM: false },
);
} catch (error) {
if (
!payload.fallbackText ||
classifyChatPublicationError(error, attempt).kind !==
"failed"
) {
throw error;
}
}
if (!sent && payload.fallbackText) {
sent = await target.post(payload.fallbackText);
}
}
} else if (claim.endpoint.provider === "imessage-photon") {
const adapter = (await runtimeFor(claim.endpoint)).getProviderAdapter();
if (!(adapter instanceof PhotonChatAdapter)) throw new Error("Photon runtime unavailable");
const receipt = await adapter.publish(payload.threadId, `effect:${action.id}`, projectSafeChatPublicationText(payload.text), {
assertCurrent: async () => { await credentialLease.assertOwned(); },
});
sent = { id: receipt.id, threadId: payload.threadId };
} else {
sent = await target.post(payload.text);
}
}
if (!sent) {
throw Object.assign(
new Error("Provider did not accept the message"),
{ code: "CHAT_PROVIDER_PRETRANSPORT_REJECTED" },
);
}
providerEffectCompleted = true;
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
// A duplicate webhook holds the endpoint while waiting for the
// delivery's unique-index entry. Lock this endpoint first too,
// before settling the action/conversation/delivery, so neither
// transaction can wait on a row already held by the other.
const currentEndpoint = await runtimeCallbackEndpoint(
tx,
claim.endpoint.id,
{
generation: payload.runtimeGeneration,
credentialFingerprint: payload.credentialFingerprint,
},
["verifying", "active"],
);
await notifyChatActionWork(tx, "provider_effect");
const [completed] = await tx
.update(chatActions)
.set({
status: "processed",
result: {
attempts: attempt,
providerMessageId: sent.id,
threadId: sent.threadId,
...(payload.effect === "telegram_callback_notice"
? {
code:
payload.telegramCallback?.chatType === "private"
? "telegram_private_api_accepted"
: "telegram_ephemeral_api_accepted",
}
: {}),
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action!.id),
eq(chatActions.status, "processing"),
sql`(${chatActions.result}->>'attempts')::int = ${attempt}`,
),
)
.returning({ id: chatActions.id });
if (!completed) {
throw new Error(
"Provider effect ownership changed before commit",
);
}
if (payload.completeConversationId) {
await tx
.update(chatConversations)
.set({ state: "completed", updatedAt: new Date() })
.where(
and(
eq(chatConversations.id, payload.completeConversationId),
eq(chatConversations.endpointId, claim.endpoint.id),
inArray(chatConversations.state, ["active", "waiting"]),
),
);
}
if (payload.settleDelivery && action!.deliveryId) {
await notifyChatDeliveryWork(tx);
const [settled] = await tx
.update(chatDeliveries)
.set({
state: "processed",
processedAt: new Date(),
nextAttemptAt: null,
redactedError: null,
updatedAt: new Date(),
})
.where(
and(
eq(chatDeliveries.id, action!.deliveryId),
eq(chatDeliveries.state, "processing"),
),
)
.returning({ id: chatDeliveries.id });
if (!settled) {
throw new Error(
"Provider effect delivery ownership changed before commit",
);
}
}
if (currentEndpoint) {
await tx
.update(chatEndpoints)
.set({ lastEventAt: new Date(), updatedAt: new Date() })
.where(eq(chatEndpoints.id, currentEndpoint.id));
}
await credentialLease.assertOwned(tx);
});
return "processed";
},
);
} catch (error) {
const disposition = providerEffectDisposition(
error,
attempt,
providerEffectCompleted,
);
try {
await finalizeProviderEffectFailure({
action,
endpoint: record?.endpoint ?? initialEndpoint.endpoint,
payload,
disposition,
attempt,
});
} catch (finalizationError) {
if (disposition.kind === "delivery_unknown") {
throw Object.assign(
new Error(
"Provider effect is ambiguous and its durable quarantine failed",
{ cause: finalizationError },
),
{ code: "CHAT_PROVIDER_EFFECT_AMBIGUOUS" },
);
}
throw finalizationError;
}
throw error;
}
};
return processClaimedEffect();
}
async function processPendingProviderEffects(limit = 25) {
const now = new Date();
const staleBefore = new Date(now.getTime() - PROVIDER_EFFECT_STALE_MS);
const actions = await db
.select()
.from(chatActions)
.where(dueChatAction("provider_effect", now.getTime()))
.orderBy(asc(chatActions.createdAt))
.limit(limit);
for (const action of actions) {
if (action.status === "processing" && action.updatedAt <= staleBefore) {
await quarantineStaleProviderEffect(action);
continue;
}
try {
await processProviderEffect(action.id);
} catch (error) {
logger.warn(
{
endpointId: action.endpointId,
actionId: action.id,
error: redactError(error),
},
"chat provider effect reconciliation failed",
);
}
}
return actions.length;
}
return { stageProviderEffect, processProviderEffect, processPendingProviderEffects };
}
@@ -0,0 +1,891 @@
import type { EndpointRow, DbTransaction, DbOrTransaction, RuntimeContext, LifecycleRuntimeFence, InboundRuntimeContext } from "./runtime-types.js";
import {
stableExternalPrincipalId,
} from "./message-lifecycle.js";
import { notifyChatDeliveryWork } from "../chat-work-notifications.js";
import {
createHash,
} from "node:crypto";
import {
and,
asc,
desc,
eq,
gt,
gte,
inArray,
isNull,
lt,
lte,
or,
sql,
type SQL,
} from "drizzle-orm";
import {
externalThreadIdentityCondition,
} from "../chat-control-chronology.js";
import type { Db } from "@paperclipai/db";
import {
chatConversations,
chatDeliveries,
chatEndpointResources,
chatMessageLinks,
chatPublications,
} from "@paperclipai/db";
import type {
ChatEventKind,
ChatProvider,
} from "@paperclipai/shared";
import { logger } from "../../middleware/logger.js";
import {
type ChatSdkCallbackEvent,
} from "../chat-sdk-runtime.js";
import type {
Author,
ReactionEvent,
} from "chat";
import type { EndpointRecord } from "./runtime-types.js";
import { slackThreadChannelId } from "./slack/messages.js";
// Reactions can beat the transaction that persists a just-sent provider
// message link. Keep that narrow gap durable, but never retain an unbound
// reaction indefinitely or let it enter the ordinary inbound-message FIFO.
const REACTION_LINK_RETRY_DELAY_MS = 250;
const REACTION_LINK_MAX_DELAY_MS = 5_000;
const REACTION_LINK_MAX_ATTEMPTS = 20;
const REACTION_LINK_MAX_AGE_MS = 2 * 60_000;
type DeliveryRow = typeof chatDeliveries.$inferSelect;
type ConversationRow = typeof chatConversations.$inferSelect;
type InboundReactionsOptions = {
lockCurrentPrincipalAuthorization: (tx: DbOrTransaction, endpoint: EndpointRow, principalId: string) => Promise<{ allowed: boolean }>;
runtimeCallbackRecord: (endpointId: string, context: RuntimeContext, statuses: EndpointRow["status"][]) => Promise<EndpointRecord | null>;
runtimeCallbackEndpoint: (tx: DbTransaction, endpointId: string, context: LifecycleRuntimeFence, statuses: EndpointRow["status"][]) => Promise<EndpointRow | null>;
ensurePrincipal: (endpoint: EndpointRow, author: Author, raw?: unknown) => Promise<{ principal: { id: string } }>;
durableExternalThreadIdentity: (threadId: string) => string;
recordCurrentMicrosoftTeamsRoute: (endpoint: EndpointRow, context: InboundRuntimeContext | undefined, threadId: string, raw: unknown) => Promise<void>;
lifecycleRuntimeFence: (delivery: DeliveryRow) => LifecycleRuntimeFence | null;
deliveryReady: (delivery: DeliveryRow, now: Date) => boolean;
redactError: (error: unknown) => string;
nonDirectDestinationAllowed: (endpoint: EndpointRow, resource: typeof chatEndpointResources.$inferSelect | null | undefined) => boolean;
reactionLinkPreflightBarrier?: () => Promise<void>;
reactionReplayEndpointLockBarrier?: () => Promise<void>;
reactionReplayConversationLockBarrier?: () => Promise<void>;
};
export function createInboundReactions(db: Db, options: InboundReactionsOptions) {
const { lockCurrentPrincipalAuthorization, runtimeCallbackRecord, runtimeCallbackEndpoint, ensurePrincipal, durableExternalThreadIdentity, recordCurrentMicrosoftTeamsRoute, lifecycleRuntimeFence, deliveryReady, redactError, nonDirectDestinationAllowed } = options;
function reactionConversationThreadCondition(
provider: ChatProvider,
threadId: string,
): SQL | undefined {
const exact = externalThreadIdentityCondition(
sql<string>`${chatConversations.externalThreadId}`,
threadId,
);
const slackDmChannel =
provider === "slack" ? slackThreadChannelId(threadId) : null;
return slackDmChannel && /^D[A-Z0-9-]*$/i.test(slackDmChannel)
? or(
exact,
and(
eq(chatConversations.isDirectMessage, true),
eq(chatConversations.externalThreadId, `slack:${slackDmChannel}:`),
),
)
: exact;
}
async function reactionConversationForMessage(
database: DbOrTransaction,
endpoint: EndpointRow,
threadId: string,
providerMessageId: string,
direction?: "outbound",
): Promise<ConversationRow | null> {
return database
.select({ conversation: chatConversations })
.from(chatMessageLinks)
.innerJoin(
chatConversations,
and(
eq(chatConversations.companyId, chatMessageLinks.companyId),
eq(chatConversations.endpointId, chatMessageLinks.endpointId),
eq(chatConversations.id, chatMessageLinks.conversationId),
),
)
.where(
and(
eq(chatMessageLinks.companyId, endpoint.companyId),
eq(chatMessageLinks.endpointId, endpoint.id),
eq(chatMessageLinks.providerMessageId, providerMessageId),
direction ? eq(chatMessageLinks.direction, direction) : undefined,
reactionConversationThreadCondition(endpoint.provider, threadId),
),
)
.orderBy(desc(chatConversations.sessionGeneration))
.limit(1)
.then((rows) => rows[0]?.conversation ?? null);
}
async function lockReactionConversationForMessage(
tx: DbTransaction,
endpoint: EndpointRow,
threadId: string,
providerMessageId: string,
direction?: "outbound",
): Promise<ConversationRow | null> {
const candidate = await reactionConversationForMessage(
tx,
endpoint,
threadId,
providerMessageId,
direction,
);
if (!candidate) return null;
const current = await tx
.select()
.from(chatConversations)
.where(
and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
eq(chatConversations.id, candidate.id),
),
)
// Publication settlement inserts the outbound message link before it
// updates the endpoint. NO KEY UPDATE serializes state changes while
// remaining compatible with that link's conversation FK KEY SHARE.
.for("no key update")
.then((rows) => rows[0] ?? null);
if (
!current ||
!(current.isDirectMessage
? ["active", "waiting", "completed"].includes(current.state)
: ["active", "waiting"].includes(current.state))
) {
return null;
}
const linkStillExists = await tx
.select({ id: chatMessageLinks.id })
.from(chatMessageLinks)
.where(
and(
eq(chatMessageLinks.companyId, endpoint.companyId),
eq(chatMessageLinks.endpointId, endpoint.id),
eq(chatMessageLinks.conversationId, current.id),
eq(chatMessageLinks.providerMessageId, providerMessageId),
direction ? eq(chatMessageLinks.direction, direction) : undefined,
),
)
.limit(1)
.then((rows) => rows.length > 0);
return linkStillExists ? current : null;
}
async function streamingReactionConversation(
database: DbOrTransaction,
endpoint: EndpointRow,
threadId: string,
): Promise<ConversationRow | null> {
const candidates = await database
.selectDistinct({
conversation: chatConversations,
sessionGeneration: chatConversations.sessionGeneration,
})
.from(chatConversations)
.innerJoin(
chatPublications,
and(
eq(chatPublications.companyId, chatConversations.companyId),
eq(chatPublications.endpointId, chatConversations.endpointId),
eq(chatPublications.conversationId, chatConversations.id),
),
)
.where(
and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
or(
inArray(chatConversations.state, ["active", "waiting"]),
and(
eq(chatConversations.isDirectMessage, true),
eq(chatConversations.state, "completed"),
),
),
reactionConversationThreadCondition(endpoint.provider, threadId),
eq(chatPublications.state, "streaming"),
),
)
.orderBy(desc(chatConversations.sessionGeneration))
.limit(2);
if (candidates.length !== 1) return null;
return candidates[0]!.conversation;
}
async function lockStreamingReactionConversation(
tx: DbTransaction,
endpoint: EndpointRow,
threadId: string,
): Promise<ConversationRow | null> {
const candidate = await streamingReactionConversation(
tx,
endpoint,
threadId,
);
if (!candidate) return null;
const current = await tx
.select()
.from(chatConversations)
.where(
and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
eq(chatConversations.id, candidate.id),
or(
inArray(chatConversations.state, ["active", "waiting"]),
and(
eq(chatConversations.isDirectMessage, true),
eq(chatConversations.state, "completed"),
),
),
),
)
// Keep the same lock strength/order as the exact-link path above so an
// in-flight message-link insert cannot deadlock reaction admission.
.for("no key update")
.then((rows) => rows[0] ?? null);
if (!current) return null;
const stillStreaming = await tx
.select({ id: chatPublications.id })
.from(chatPublications)
.where(
and(
eq(chatPublications.companyId, endpoint.companyId),
eq(chatPublications.endpointId, endpoint.id),
eq(chatPublications.conversationId, current.id),
eq(chatPublications.state, "streaming"),
),
)
.limit(1)
.then((rows) => rows.length > 0);
return stillStreaming ? current : null;
}
async function authorizeReactionConversation(
tx: DbTransaction,
endpoint: EndpointRow,
conversation: ConversationRow,
principalId: string,
): Promise<boolean> {
const resource =
conversation.resourceId && !conversation.isDirectMessage
? await tx
.select()
.from(chatEndpointResources)
.where(
and(
eq(chatEndpointResources.companyId, endpoint.companyId),
eq(chatEndpointResources.endpointId, endpoint.id),
eq(chatEndpointResources.id, conversation.resourceId),
),
)
.for("update")
.then((rows) => rows[0] ?? null)
: null;
const destinationAllowed = conversation.isDirectMessage
? endpoint.allowDirectMessages
: nonDirectDestinationAllowed(endpoint, resource);
if (!destinationAllowed) return false;
return (await lockCurrentPrincipalAuthorization(tx, endpoint, principalId))
.allowed;
}
async function handleReaction(
event: ChatSdkCallbackEvent<ReactionEvent>,
runtimeContext: RuntimeContext,
) {
const record = await runtimeCallbackRecord(
event.endpointId,
runtimeContext,
["active"],
);
if (
!record ||
record.endpoint.status !== "active" ||
record.endpoint.provider !== event.provider ||
record.endpoint.capabilities.reactions !== true ||
event.event.user.isMe ||
event.event.user.isBot === true ||
event.event.user.isSystem
) {
return;
}
// A DM can have a newer task generation while this reaction still belongs
// to an older linked message. Resolve its exact lineage before choosing a
// generation; never attach it to the latest task merely sharing the chat.
// A just-sent provider message can be visible before its publication/link
// transaction commits. Only that exact thread's streaming publication is
// enough evidence to stage an otherwise-unbound reaction for replay.
let conversation = await reactionConversationForMessage(
db,
record.endpoint,
event.event.threadId,
event.event.messageId,
);
await options.reactionLinkPreflightBarrier?.();
const plausibleStreamingConversation = conversation
? null
: await streamingReactionConversation(
db,
record.endpoint,
event.event.threadId,
);
if (!conversation && !plausibleStreamingConversation) {
// Settlement can commit between the first exact-link lookup and the
// streaming-publication lookup. Recheck the durable link before treating
// the callback as an arbitrary unknown message.
conversation = await reactionConversationForMessage(
db,
record.endpoint,
event.event.threadId,
event.event.messageId,
);
if (!conversation) return;
}
const principal = await ensurePrincipal(
record.endpoint,
event.event.user,
event.event.raw,
);
let rawFingerprint: string;
try {
rawFingerprint = createHash("sha256")
.update(JSON.stringify(event.event.raw ?? null))
.digest("hex")
.slice(0, 24);
} catch {
rawFingerprint = "unavailable";
}
const eventKind: ChatEventKind = event.event.added
? "reaction_added"
: "reaction_removed";
const providerEventId = [
eventKind,
durableExternalThreadIdentity(event.event.threadId),
event.event.messageId,
stableExternalPrincipalId(
record.endpoint.provider,
event.event.user,
event.event.raw,
),
event.event.rawEmoji,
rawFingerprint,
].join(":");
const normalizedEvent = {
providerEventId,
kind: eventKind,
conversation: { externalThreadId: event.event.threadId },
message: { providerMessageId: event.event.messageId },
reaction: {
emoji: event.event.emoji.name,
rawEmoji: event.event.rawEmoji,
added: event.event.added,
},
runtimeContext: {
generation: runtimeContext.generation,
credentialFingerprint: runtimeContext.credentialFingerprint,
},
};
const admitted = await db.transaction(async (tx) => {
const currentEndpoint = await runtimeCallbackEndpoint(
tx,
event.endpointId,
runtimeContext,
["active"],
);
if (!currentEndpoint) return false;
const linkedConversation = await lockReactionConversationForMessage(
tx,
currentEndpoint,
event.event.threadId,
event.event.messageId,
);
const currentConversation =
linkedConversation ??
(await lockStreamingReactionConversation(
tx,
currentEndpoint,
event.event.threadId,
));
if (
!currentConversation ||
!(await authorizeReactionConversation(
tx,
currentEndpoint,
currentConversation,
principal.principal.id,
))
) {
return false;
}
const now = new Date();
if (linkedConversation) {
const expiryCutoff = new Date(now.getTime() - REACTION_LINK_MAX_AGE_MS);
const hasOutboundLink = await tx
.select({ id: chatMessageLinks.id })
.from(chatMessageLinks)
.where(
and(
eq(chatMessageLinks.companyId, currentEndpoint.companyId),
eq(chatMessageLinks.endpointId, currentEndpoint.id),
eq(chatMessageLinks.conversationId, linkedConversation.id),
eq(chatMessageLinks.providerMessageId, event.event.messageId),
eq(chatMessageLinks.direction, "outbound"),
),
)
.limit(1)
.then((rows) => rows.length > 0);
await notifyChatDeliveryWork(tx);
const reconciled = hasOutboundLink
? await tx
.update(chatDeliveries)
.set({
conversationId: linkedConversation.id,
state: "processed",
attempts: sql`${chatDeliveries.attempts} + 1`,
nextAttemptAt: null,
processedAt: now,
redactedError: null,
updatedAt: now,
})
.where(
and(
eq(chatDeliveries.endpointId, currentEndpoint.id),
eq(chatDeliveries.providerEventId, providerEventId),
inArray(chatDeliveries.state, ["received", "retry"]),
gt(chatDeliveries.receivedAt, expiryCutoff),
lt(chatDeliveries.attempts, REACTION_LINK_MAX_ATTEMPTS),
sql`${chatDeliveries.normalizedEvent}->'runtimeContext'->>'generation' = ${String(runtimeContext.generation)}`,
sql`${chatDeliveries.normalizedEvent}->'runtimeContext'->>'credentialFingerprint' = ${runtimeContext.credentialFingerprint}`,
),
)
.returning({ id: chatDeliveries.id })
: [];
if (reconciled.length > 0) return true;
await notifyChatDeliveryWork(tx);
const expired = await tx
.update(chatDeliveries)
.set({
state: "filtered",
nextAttemptAt: null,
processedAt: now,
redactedError:
"Reaction target did not become linked before replay expiry",
updatedAt: now,
})
.where(
and(
eq(chatDeliveries.endpointId, currentEndpoint.id),
eq(chatDeliveries.providerEventId, providerEventId),
inArray(chatDeliveries.state, ["received", "retry"]),
or(
lte(chatDeliveries.receivedAt, expiryCutoff),
gte(chatDeliveries.attempts, REACTION_LINK_MAX_ATTEMPTS),
),
sql`${chatDeliveries.normalizedEvent}->'runtimeContext'->>'generation' = ${String(runtimeContext.generation)}`,
sql`${chatDeliveries.normalizedEvent}->'runtimeContext'->>'credentialFingerprint' = ${runtimeContext.credentialFingerprint}`,
),
)
.returning({ id: chatDeliveries.id });
if (expired.length > 0) return false;
}
await notifyChatDeliveryWork(tx);
const inserted = await tx
.insert(chatDeliveries)
.values({
companyId: currentEndpoint.companyId,
endpointId: currentEndpoint.id,
conversationId: linkedConversation?.id ?? null,
principalId: principal.principal.id,
providerEventId,
deduplicationKey: createHash("sha256")
.update(providerEventId)
.digest("hex"),
eventKind,
normalizedEvent,
state: linkedConversation ? "processed" : "received",
attempts: linkedConversation ? 1 : 0,
nextAttemptAt: linkedConversation
? null
: new Date(now.getTime() + REACTION_LINK_RETRY_DELAY_MS),
processedAt: linkedConversation ? now : null,
})
.onConflictDoNothing()
.returning({ id: chatDeliveries.id });
if (inserted.length > 0) return true;
return tx
.select({ id: chatDeliveries.id })
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.endpointId, currentEndpoint.id),
eq(chatDeliveries.providerEventId, providerEventId),
inArray(chatDeliveries.state, ["received", "retry", "processed"]),
),
)
.limit(1)
.then((rows) => rows.length > 0);
});
if (admitted) {
await recordCurrentMicrosoftTeamsRoute(
record.endpoint,
runtimeContext,
event.event.threadId,
event.event.raw,
);
}
}
function normalizedPendingReaction(delivery: DeliveryRow): {
added: boolean;
emoji: string;
messageId: string;
rawEmoji: string;
runtimeContext: LifecycleRuntimeFence;
threadId: string;
} | null {
if (
delivery.eventKind !== "reaction_added" &&
delivery.eventKind !== "reaction_removed"
) {
return null;
}
const normalized = delivery.normalizedEvent;
if (
!normalized ||
typeof normalized !== "object" ||
Array.isArray(normalized)
) {
return null;
}
const value = normalized as Record<string, unknown>;
const conversation = value.conversation;
const message = value.message;
const reaction = value.reaction;
const runtimeContext = lifecycleRuntimeFence(delivery);
if (
!conversation ||
typeof conversation !== "object" ||
Array.isArray(conversation) ||
!message ||
typeof message !== "object" ||
Array.isArray(message) ||
!reaction ||
typeof reaction !== "object" ||
Array.isArray(reaction) ||
!runtimeContext
) {
return null;
}
const threadId = (conversation as Record<string, unknown>).externalThreadId;
const messageId = (message as Record<string, unknown>).providerMessageId;
const reactionRecord = reaction as Record<string, unknown>;
const added = reactionRecord.added;
const emoji = reactionRecord.emoji;
const rawEmoji = reactionRecord.rawEmoji;
return typeof threadId === "string" &&
threadId.length > 0 &&
typeof messageId === "string" &&
messageId.length > 0 &&
typeof added === "boolean" &&
added === (delivery.eventKind === "reaction_added") &&
typeof emoji === "string" &&
emoji.length > 0 &&
typeof rawEmoji === "string" &&
rawEmoji.length > 0
? { added, emoji, messageId, rawEmoji, runtimeContext, threadId }
: null;
}
async function processPendingReactionDelivery(
candidate: DeliveryRow,
): Promise<void> {
const reaction = normalizedPendingReaction(candidate);
if (!reaction || !candidate.principalId) {
const now = new Date();
await db.transaction(async (tx) => {
await notifyChatDeliveryWork(tx);
return tx
.update(chatDeliveries)
.set({
state: "failed",
nextAttemptAt: null,
processedAt: now,
redactedError: "Normalized reaction delivery is incomplete",
updatedAt: now,
})
.where(
and(
eq(chatDeliveries.id, candidate.id),
inArray(chatDeliveries.state, ["received", "retry"]),
),
);
});
return;
}
const principalId = candidate.principalId;
await db.transaction(async (tx) => {
const currentEndpoint = await runtimeCallbackEndpoint(
tx,
candidate.endpointId,
reaction.runtimeContext,
["active"],
);
let conversation: ConversationRow | null = null;
let disposition:
| { kind: "processed"; conversationId: string }
| { kind: "retry" }
| { kind: "filtered"; reason: string };
if (!currentEndpoint) {
disposition = {
kind: "filtered",
reason: "Reaction callback belonged to an unavailable runtime",
};
} else {
await options.reactionReplayEndpointLockBarrier?.();
conversation = await lockReactionConversationForMessage(
tx,
currentEndpoint,
reaction.threadId,
reaction.messageId,
"outbound",
);
if (conversation) {
await options.reactionReplayConversationLockBarrier?.();
}
if (conversation) {
disposition = (await authorizeReactionConversation(
tx,
currentEndpoint,
conversation,
principalId,
))
? { kind: "processed", conversationId: conversation.id }
: {
kind: "filtered",
reason:
"Reaction destination or principal is no longer authorized",
};
} else {
conversation = await lockStreamingReactionConversation(
tx,
currentEndpoint,
reaction.threadId,
);
if (conversation) {
await options.reactionReplayConversationLockBarrier?.();
}
if (!conversation) {
disposition = {
kind: "filtered",
reason: "Reaction target was not linked to a sent message",
};
} else {
disposition = (await authorizeReactionConversation(
tx,
currentEndpoint,
conversation,
principalId,
))
? { kind: "retry" }
: {
kind: "filtered",
reason:
"Reaction destination or principal is no longer authorized",
};
}
}
}
// Endpoint, conversation, destination and identity locks are acquired
// before the delivery row so pause/revoke and replay use one lock order.
const locked = await tx
.select()
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.id, candidate.id),
inArray(chatDeliveries.state, ["received", "retry"]),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
// Authorization locks may wait behind a concurrent lifecycle mutation.
// Sample the expiry clock only after this delivery is exclusively held.
const decisionAt = new Date();
if (!locked || !deliveryReady(locked, decisionAt)) return;
const lockedReaction = normalizedPendingReaction(locked);
if (
!lockedReaction ||
lockedReaction.runtimeContext.generation !==
reaction.runtimeContext.generation ||
lockedReaction.runtimeContext.credentialFingerprint !==
reaction.runtimeContext.credentialFingerprint
) {
disposition = {
kind: "filtered",
reason: "Reaction delivery identity changed before replay",
};
}
const nextAttempts = locked.attempts + 1;
const expired =
decisionAt.getTime() - locked.receivedAt.getTime() >=
REACTION_LINK_MAX_AGE_MS;
if (expired || locked.attempts >= REACTION_LINK_MAX_ATTEMPTS) {
disposition = {
kind: "filtered",
reason: "Reaction target did not become linked before replay expiry",
};
}
if (disposition.kind === "retry" && !expired) {
if (nextAttempts < REACTION_LINK_MAX_ATTEMPTS) {
const delay = Math.min(
REACTION_LINK_RETRY_DELAY_MS *
2 ** Math.min(Math.max(locked.attempts, 0), 5),
REACTION_LINK_MAX_DELAY_MS,
);
await notifyChatDeliveryWork(tx);
await tx
.update(chatDeliveries)
.set({
state: "retry",
attempts: nextAttempts,
nextAttemptAt: new Date(decisionAt.getTime() + delay),
redactedError: "Waiting for the sent message link",
updatedAt: decisionAt,
})
.where(
and(
eq(chatDeliveries.id, locked.id),
eq(chatDeliveries.state, locked.state),
eq(chatDeliveries.attempts, locked.attempts),
),
);
return;
}
disposition = {
kind: "filtered",
reason: "Reaction target did not become linked before retry expiry",
};
} else if (disposition.kind === "retry") {
disposition = {
kind: "filtered",
reason: "Reaction target did not become linked before age expiry",
};
}
const processedAt = new Date();
await notifyChatDeliveryWork(tx);
await tx
.update(chatDeliveries)
.set(
disposition.kind === "processed"
? {
conversationId: disposition.conversationId,
state: "processed",
attempts: nextAttempts,
nextAttemptAt: null,
processedAt,
redactedError: null,
updatedAt: processedAt,
}
: {
state: "filtered",
attempts: nextAttempts,
nextAttemptAt: null,
processedAt,
redactedError: disposition.reason,
updatedAt: processedAt,
},
)
.where(
and(
eq(chatDeliveries.id, locked.id),
eq(chatDeliveries.state, locked.state),
eq(chatDeliveries.attempts, locked.attempts),
),
);
});
}
async function processPendingReactionDeliveries(
limit: number,
onlyDeliveryId?: string,
): Promise<number> {
const now = new Date();
const candidates = await db
.select()
.from(chatDeliveries)
.where(
and(
sql`not exists (select 1 from chat_endpoints e where e.id = ${chatDeliveries.endpointId} and e.provider = 'agentmail')`,
onlyDeliveryId ? eq(chatDeliveries.id, onlyDeliveryId) : undefined,
inArray(chatDeliveries.eventKind, [
"reaction_added",
"reaction_removed",
]),
or(
and(
eq(chatDeliveries.state, "received"),
or(
isNull(chatDeliveries.nextAttemptAt),
lte(chatDeliveries.nextAttemptAt, now),
),
),
and(
eq(chatDeliveries.state, "retry"),
or(
isNull(chatDeliveries.nextAttemptAt),
lte(chatDeliveries.nextAttemptAt, now),
),
),
),
),
)
.orderBy(asc(chatDeliveries.receivedAt), asc(chatDeliveries.id))
.limit(limit);
for (const candidate of candidates) {
try {
await processPendingReactionDelivery(candidate);
} catch (error) {
logger.warn(
{
endpointId: candidate.endpointId,
deliveryId: candidate.id,
error: redactError(error),
},
"chat reaction-link reconciliation failed",
);
}
}
return candidates.length;
}
return { handleReaction, processPendingReactionDeliveries };
}
@@ -0,0 +1,586 @@
import type { ChatRuntime } from "../chat-runtime.js";
import type { EndpointRow, RuntimeContext, LifecycleRuntimeFence } from "./runtime-types.js";
import { dueChatAction, PROVIDER_EFFECT_STALE_MS } from "../chat-action-work.js";
import { notifyChatActionWork } from "../chat-work-notifications.js";
import type { ChatEndpointRuntime } from "../voice/voice-runtime.js";
import {
receiptReactionPayload,
type ReceiptReactionPayload,
} from "../chat-receipt-reactions.js";
import { chatCredentialMutationLease, type CredentialMutationLeaseGuard } from "../chat-credential-mutation-lease.js";
import {
and,
asc,
eq,
like,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatActions,
chatDeliveries,
chatEndpoints,
toolConnections,
} from "@paperclipai/db";
import type {
ChatEndpointSetupState,
ChatProvider,
ToolCredentialSecretRef,
} from "@paperclipai/shared";
import {
parseGitHubReceiptIdentity,
type GitHubReceiptIdentity,
} from "../chat-github-receipt-reactions.js";
import { logger } from "../../middleware/logger.js";
import { classifyChatPublicationError } from "../chat-publication-errors.js";
import type {
Message,
Thread,
} from "chat";
import type { EndpointRecord } from "./runtime-types.js";
import { isSlackReceiptReactionAlreadyApplied } from "./slack/messages.js";
type ReceiptReactionsOptions = {
endpointRecord: (endpointId: string) => Promise<EndpointRecord | null>;
runtimeContextForRecord: (record: EndpointRecord) => RuntimeContext;
withCredentialMutationLease: ReturnType<typeof chatCredentialMutationLease>;
runtimeFor: (endpoint: EndpointRow) => Promise<ChatEndpointRuntime>;
fetchImpl: typeof fetch;
runtime: Pick<ChatRuntime, "get">;
runtimeGeneration: (setup: ChatEndpointSetupState) => number;
credentialFingerprint: (refs: ToolCredentialSecretRef[]) => string;
redactError: (error: unknown) => string;
MAX_ERROR_TEXT: number;
receiptReactionTransportBarrier?: () => Promise<void>;
};
export function createReceiptReactions(db: Db, options: ReceiptReactionsOptions) {
const { endpointRecord, runtimeContextForRecord, withCredentialMutationLease, runtimeFor, fetchImpl, runtime, runtimeGeneration, credentialFingerprint, redactError, MAX_ERROR_TEXT } = options;
async function addReceiptReaction(input: {
deliveryId: string;
endpoint: EndpointRow;
message: Message;
runtimeContext?: LifecycleRuntimeFence;
thread: Thread;
}): Promise<void> {
// GitHub acknowledges through one editable response comment. Continue to
// drain legacy eyes removals, but never create new eyes for these requests.
if (input.endpoint.provider === "github") return;
const record = await endpointRecord(input.endpoint.id);
if (!record) return;
// A live SDK object remains bound to the credentials that authenticated
// its callback. Preserve that original fence rather than sampling the
// endpoint again after a rotation and accidentally blessing an obsolete
// adapter with the new generation.
const context = input.runtimeContext ?? runtimeContextForRecord(record);
const [inserted] = await db.transaction(async (tx) => {
await notifyChatActionWork(tx, "receipt_reaction");
return tx
.insert(chatActions)
.values({
companyId: input.endpoint.companyId,
endpointId: input.endpoint.id,
deliveryId: input.deliveryId,
kind: "receipt_reaction",
providerActionId: `receipt_reaction:${input.deliveryId}`,
payload: {
version: 1,
operation: "add",
threadId: input.thread.id,
messageId: input.message.id,
reaction: "eyes",
runtimeGeneration: context.generation,
credentialFingerprint: context.credentialFingerprint,
} satisfies ReceiptReactionPayload,
status: "received",
})
.onConflictDoNothing()
.returning({ id: chatActions.id });
});
const actionId =
inserted?.id ??
(await db
.select({ id: chatActions.id })
.from(chatActions)
.where(
and(
eq(chatActions.endpointId, input.endpoint.id),
eq(
chatActions.providerActionId,
`receipt_reaction:${input.deliveryId}`,
),
),
)
.then((rows) => rows[0]?.id ?? null));
if (!actionId) return;
await options.receiptReactionTransportBarrier?.();
await processReceiptReaction(actionId);
}
async function processReceiptReaction(actionId: string): Promise<void> {
const initialAction = await db
.select({ endpointId: chatActions.endpointId })
.from(chatActions)
.where(
and(
eq(chatActions.id, actionId),
eq(chatActions.kind, "receipt_reaction"),
),
)
.then((rows) => rows[0] ?? null);
if (!initialAction) return;
const initialEndpoint = await endpointRecord(initialAction.endpointId);
if (!initialEndpoint) return;
// Receipt reactions are GitHub repository mutations too. Serialize their
// final runtime-fence check and provider call with reconnect, secret
// rotation, pause, and removal so credentials cannot become obsolete
// between authorization and transport.
await withCredentialMutationLease(initialEndpoint.endpoint, (lease) =>
processReceiptReactionWithCredentialLease(actionId, lease),
);
}
async function processReceiptReactionWithCredentialLease(
actionId: string,
credentialLease: CredentialMutationLeaseGuard,
): Promise<void> {
let action = await db
.select()
.from(chatActions)
.where(
and(
eq(chatActions.id, actionId),
eq(chatActions.kind, "receipt_reaction"),
),
)
.then((rows) => rows[0] ?? null);
if (!action || ["processed", "cancelled"].includes(action.status)) return;
const payload = receiptReactionPayload(action.payload);
if (!payload) {
await db.transaction(async (tx) => {
await notifyChatActionWork(tx, "receipt_reaction");
return tx
.update(chatActions)
.set({
status: "failed",
result: {
code: "receipt_reaction_payload_invalid",
retryable: false,
},
updatedAt: new Date(),
})
.where(eq(chatActions.id, action.id));
});
return;
}
if (payload.operation === "add" && action.deliveryId) {
const terminalRemoval = await db
.select({ id: chatActions.id })
.from(chatActions)
.where(
and(
eq(chatActions.endpointId, action.endpointId),
eq(
chatActions.providerActionId,
`receipt_reaction_remove:${action.deliveryId}`,
),
),
)
.then((rows) => rows[0] ?? null);
if (terminalRemoval) {
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatActionWork(tx, "receipt_reaction");
await tx
.update(chatActions)
.set({
status: "cancelled",
result: {
...(typeof action!.result?.attempts === "number"
? { attempts: action!.result.attempts }
: {}),
code: "receipt_reaction_superseded_by_terminal_publication",
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action!.id),
eq(chatActions.status, action!.status),
),
);
await credentialLease.assertOwned(tx);
});
return;
}
}
if (action.status === "processing") {
if (action.updatedAt > new Date(Date.now() - PROVIDER_EFFECT_STALE_MS))
return;
// Adding or removing the same reaction is idempotent. A worker
// interruption can be retried safely without an ambiguous
// duplicate-message boundary.
await db.transaction(async (tx) => {
await notifyChatActionWork(tx, "receipt_reaction");
return tx
.update(chatActions)
.set({
status: "failed",
result: {
...(action.result ?? {}),
retryable: true,
retryAt: new Date().toISOString(),
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "processing"),
eq(chatActions.updatedAt, action.updatedAt),
),
);
});
action = await db
.select()
.from(chatActions)
.where(eq(chatActions.id, action.id))
.then((rows) => rows[0] ?? null);
if (!action) return;
}
if (action.status === "failed") {
if (action.result?.retryable !== true) return;
const retryAt =
typeof action.result.retryAt === "string"
? new Date(action.result.retryAt)
: null;
if (retryAt && retryAt > new Date()) return;
} else if (action.status !== "received") {
return;
}
const attempt =
(typeof action.result?.attempts === "number" ? action.result.attempts : 0) +
1;
const retainedGitHubReceipt = parseGitHubReceiptIdentity(
action.result?.githubReceipt,
);
const [claimed] = await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatActionWork(tx, "receipt_reaction");
const rows = await tx
.update(chatActions)
.set({
status: "processing",
result: {
attempts: attempt,
...(retainedGitHubReceipt
? { githubReceipt: retainedGitHubReceipt }
: {}),
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action!.id),
eq(chatActions.status, action!.status),
),
)
.returning();
await credentialLease.assertOwned(tx);
return rows;
});
if (!claimed) return;
let reactionProvider: ChatProvider | null = null;
let githubReceipt: GitHubReceiptIdentity | null = retainedGitHubReceipt;
try {
const record = await endpointRecord(action.endpointId);
if (
!record ||
!["verifying", "active"].includes(record.endpoint.status) ||
runtimeGeneration(record.endpoint.setup) !== payload.runtimeGeneration ||
credentialFingerprint(record.credentialSecretRefs) !==
payload.credentialFingerprint
) {
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatActionWork(tx, "receipt_reaction");
await tx
.update(chatActions)
.set({
status: "cancelled",
result: {
attempts: attempt,
code: "receipt_reaction_runtime_superseded",
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action!.id),
eq(chatActions.status, "processing"),
sql`(${chatActions.result}->>'attempts')::int = ${attempt}`,
),
);
await credentialLease.assertOwned(tx);
});
return;
}
reactionProvider = record.endpoint.provider;
// Resolve the adapter only after acquiring the credential lease and
// proving the action's durable generation/fingerprint is still current.
// Never reuse the live callback's adapter: it may hold an installation
// token minted by a superseded private key.
const endpointRuntime = await runtimeFor(record.endpoint);
const adapter = endpointRuntime.thread(payload.threadId).adapter;
await credentialLease.assertOwned();
if (record.endpoint.provider === "slack") {
await endpointRuntime.applySlackReceiptReaction(payload, fetchImpl);
} else if (record.endpoint.provider === "github") {
const assertCurrent = async () => {
await credentialLease.assertOwned();
const [current] = await db
.select({ endpoint: chatEndpoints, connection: toolConnections })
.from(chatEndpoints)
.innerJoin(
toolConnections,
and(
eq(toolConnections.id, chatEndpoints.connectionId),
eq(toolConnections.companyId, chatEndpoints.companyId),
),
)
.where(
and(
eq(chatEndpoints.id, action!.endpointId),
eq(chatEndpoints.companyId, action!.companyId),
),
)
.limit(1);
if (
!current ||
!["active", "verifying"].includes(current.endpoint.status) ||
!current.connection.enabled ||
current.connection.status !== "active" ||
runtimeGeneration(current.endpoint.setup) !==
payload.runtimeGeneration ||
credentialFingerprint(current.connection.credentialSecretRefs) !==
payload.credentialFingerprint ||
runtime.get(action!.endpointId) !== endpointRuntime
)
throw new Error("GitHub receipt authority is no longer current");
};
const applied = await endpointRuntime.applyGitHubReceiptReaction(
{ ...payload, ...(githubReceipt ? { githubReceipt } : {}) },
assertCurrent,
fetchImpl,
);
githubReceipt = parseGitHubReceiptIdentity(applied);
if (
!githubReceipt ||
(payload.operation === "add" && !githubReceipt.reactionId)
)
throw new Error("GitHub receipt identity was not confirmed");
} else if (payload.operation === "remove") {
await adapter.removeReaction(
payload.threadId,
payload.messageId,
payload.reaction,
);
} else {
await adapter.addReaction(
payload.threadId,
payload.messageId,
payload.reaction,
);
}
// The provider may already have accepted the idempotent reaction. Fence
// the local success receipt so a reclaimed lease cannot report success
// for an obsolete runtime.
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatActionWork(tx, "receipt_reaction");
const processed = await tx
.update(chatActions)
.set({
status: "processed",
result: {
attempts: attempt,
...(githubReceipt ? { githubReceipt } : {}),
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "processing"),
sql`(${chatActions.result}->>'attempts')::int = ${attempt}`,
),
)
.returning({ id: chatActions.id });
if (!processed.length) {
throw new Error(
"Receipt-reaction ownership changed before success settlement",
);
}
if (action.deliveryId) {
await tx
.update(chatDeliveries)
.set({ redactedError: null, updatedAt: new Date() })
.where(
and(
eq(chatDeliveries.id, action.deliveryId),
eq(chatDeliveries.state, "processed"),
like(chatDeliveries.redactedError, "Receipt reaction %"),
),
);
}
await credentialLease.assertOwned(tx);
});
} catch (error) {
if (
payload.operation === "add" &&
reactionProvider === "slack" &&
isSlackReceiptReactionAlreadyApplied(error)
) {
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatActionWork(tx, "receipt_reaction");
const processed = await tx
.update(chatActions)
.set({
status: "processed",
result: {
attempts: attempt,
code: "receipt_reaction_already_applied",
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action!.id),
eq(chatActions.status, "processing"),
sql`(${chatActions.result}->>'attempts')::int = ${attempt}`,
),
)
.returning({ id: chatActions.id });
if (!processed.length) {
throw new Error(
"Receipt-reaction ownership changed before idempotent settlement",
);
}
if (action.deliveryId) {
await tx
.update(chatDeliveries)
.set({ redactedError: null, updatedAt: new Date() })
.where(
and(
eq(chatDeliveries.id, action.deliveryId),
eq(chatDeliveries.state, "processed"),
like(chatDeliveries.redactedError, "Receipt reaction %"),
),
);
}
await credentialLease.assertOwned(tx);
});
return;
}
const disposition = classifyChatPublicationError(error, attempt);
const retryable =
disposition.kind === "retry" ||
(disposition.kind === "delivery_unknown" && attempt < 3);
const retryAfterMs =
disposition.kind === "retry"
? disposition.retryAfterMs
: 100 * 2 ** Math.max(0, attempt - 1);
const operationLabel =
payload.operation === "remove" ? " reaction removal" : " reaction";
const diagnostic =
`Receipt${operationLabel} ${retryable ? "deferred" : "failed"} after ${attempt} attempt${attempt === 1 ? "" : "s"} (${disposition.kind}): ${redactError(error)}`.slice(
0,
MAX_ERROR_TEXT,
);
const retryAt = retryable
? new Date(Date.now() + retryAfterMs).toISOString()
: null;
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatActionWork(tx, "receipt_reaction");
const [failed] = await tx
.update(chatActions)
.set({
status: "failed",
result: {
attempts: attempt,
...(githubReceipt ? { githubReceipt } : {}),
code:
payload.operation === "remove"
? `receipt_reaction_removal_${disposition.kind}`
: `receipt_reaction_${disposition.kind}`,
redactedError: diagnostic,
retryable,
retryAt,
},
updatedAt: new Date(),
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "processing"),
sql`(${chatActions.result}->>'attempts')::int = ${attempt}`,
),
)
.returning({ id: chatActions.id });
if (failed && action.deliveryId) {
await tx
.update(chatDeliveries)
.set({ redactedError: diagnostic, updatedAt: new Date() })
.where(
and(
eq(chatDeliveries.id, action.deliveryId),
eq(chatDeliveries.state, "processed"),
),
);
}
await credentialLease.assertOwned(tx);
});
logger.warn(
{
endpointId: action.endpointId,
deliveryId: action.deliveryId,
attempts: attempt,
disposition: disposition.kind,
operation: payload.operation,
retryAt,
},
"external chat receipt reaction deferred",
);
}
}
async function processPendingReceiptReactions(
limit = 25,
onlyActionId?: string,
) {
const actions = await db.select({ id: chatActions.id }).from(chatActions)
.where(and(dueChatAction("receipt_reaction"), onlyActionId ? eq(chatActions.id, onlyActionId) : undefined))
.orderBy(asc(chatActions.createdAt)).limit(limit);
for (const action of actions) await processReceiptReaction(action.id);
return actions.length;
}
return { addReceiptReaction, processReceiptReaction, processPendingReceiptReactions };
}
@@ -1,5 +1,5 @@
import type { Db, chatEndpoints } from "@paperclipai/db";
import type { ToolCredentialSecretRef } from "@paperclipai/shared";
import type { ToolCredentialSecretRef, ChatEndpointSetupState, ChatEndpointCallbackSurfaces } from "@paperclipai/shared";
import type { ChatEndpointRuntime } from "../voice/voice-runtime.js";
export type EndpointRow = typeof chatEndpoints.$inferSelect;
@@ -36,3 +36,17 @@ export type EndpointRecord = {
applicationId: string;
credentialSecretRefs: ToolCredentialSecretRef[];
};
export type SlackCallbackSurface = keyof ChatEndpointCallbackSurfaces;
export type SlackCallbackObservation = { url: string; observedAt: string };
export type InternalSetupState = ChatEndpointSetupState & {
photonIntakeAfter?: string;
runtimeGeneration?: number;
/** Binds pre-install URL verification to the signing secret later configured. */
slackVerificationSigningFingerprint?: string;
slackCallbackSurfaces?: Partial<
Record<SlackCallbackSurface, SlackCallbackObservation>
>;
};
@@ -0,0 +1,353 @@
import type { EndpointRow, DbOrTransaction } from "./runtime-types.js";
import { notifyChatEndpointWork } from "../chat-work-notifications.js";
import { chatCredentialMutationLease } from "../chat-credential-mutation-lease.js";
import { githubChatManagementService } from "../chat-github-management.js";
import {
and,
asc,
desc,
eq,
gte,
inArray,
isNotNull,
or,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatActions,
chatConversations,
chatDeliveries,
chatEndpoints,
chatIdentityLinks,
chatMessageLinks,
chatPublications,
heartbeatRuns,
issueComments,
toolConnections,
} from "@paperclipai/db";
import type {
ChatEndpoint,
ChatEndpointSetupState,
} from "@paperclipai/shared";
import {
conflict,
forbidden,
notFound,
} from "../../errors.js";
import {
logActivity,
} from "../activity-log.js";
import {
githubRunReplyState,
} from "../chat-run-publications.js";
import type { EndpointRecord } from "./runtime-types.js";
type SetupProbesOptions = {
endpointRecord: (endpointId: string) => Promise<EndpointRecord | null>;
withCredentialMutationLease: ReturnType<typeof chatCredentialMutationLease>;
fetchImpl: typeof fetch;
findAuthorizedIdentityLink: (endpoint: EndpointRow, userId: string) => Promise<{ id: string } | null>;
runtimeGeneration: (setup: ChatEndpointSetupState) => number;
lockCurrentPrincipalAuthorization: (tx: DbOrTransaction, endpoint: EndpointRow, principalId: string) => Promise<{ allowed: boolean }>;
get: (endpointId: string) => Promise<ChatEndpoint>;
setupTestActivationBarrier?: () => Promise<void>;
};
export function createSetupProbes(db: Db, options: SetupProbesOptions) {
const { endpointRecord, withCredentialMutationLease, fetchImpl, findAuthorizedIdentityLink, runtimeGeneration, lockCurrentPrincipalAuthorization, get } = options;
async function setupTestStatus(endpointId: string, userId: string) {
const record = await endpointRecord(endpointId);
if (!record) throw notFound("Chat endpoint not found");
const startedAt = record.endpoint.setup.testStartedAt;
if (!startedAt) return { messageReceivedAt: null };
const links = await db.select({ principalId: chatIdentityLinks.principalId }).from(chatIdentityLinks).where(and(
eq(chatIdentityLinks.companyId, record.endpoint.companyId), eq(chatIdentityLinks.endpointId, endpointId),
eq(chatIdentityLinks.paperclipUserId, userId), eq(chatIdentityLinks.status, "linked"),
));
if (!links.length) return { messageReceivedAt: null };
const principalIds = links.map((link) => link.principalId);
const [delivery, command] = await Promise.all([
db.select({ at: chatDeliveries.createdAt }).from(chatDeliveries).where(and(
eq(chatDeliveries.companyId, record.endpoint.companyId), eq(chatDeliveries.endpointId, endpointId),
inArray(chatDeliveries.principalId, principalIds), gte(chatDeliveries.createdAt, new Date(startedAt)),
inArray(chatDeliveries.eventKind, ["message", "mention"]),
)).orderBy(asc(chatDeliveries.createdAt)).limit(1).then((rows) => rows[0]),
db.select({ at: chatActions.createdAt }).from(chatActions).where(and(
eq(chatActions.companyId, record.endpoint.companyId), eq(chatActions.endpointId, endpointId),
inArray(chatActions.principalId, principalIds), eq(chatActions.kind, "slash_task_start"), gte(chatActions.createdAt, new Date(startedAt)),
)).orderBy(asc(chatActions.createdAt)).limit(1).then((rows) => rows[0]),
]);
const at = [delivery?.at, command?.at].filter((value): value is Date => Boolean(value)).sort((a, b) => a.getTime() - b.getTime())[0];
return { messageReceivedAt: at?.toISOString() ?? null };
}
async function test(endpointId: string, finishOptions?: { optionalSlackTestForUser: string }) {
const initial = await endpointRecord(endpointId);
if (!initial) throw notFound("Chat endpoint not found");
return withCredentialMutationLease(
initial.endpoint,
async (credentialLease) => {
// Reload only after owning the same lane as reconnect, pause, removal,
// and credential rotation. Qualification and activation must describe
// one exact provider runtime, never a setup snapshot that a concurrent
// reconnect has already superseded.
const record = await endpointRecord(endpointId);
if (!record) throw notFound("Chat endpoint not found");
const endpoint = record.endpoint;
if (endpoint.status !== "verifying" || endpoint.setup.step !== "test") {
throw conflict("This connection is not waiting for a setup test", {
code: "chat_endpoint_not_testing",
});
}
if (endpoint.provider === "github" && endpoint.setup.github) {
const verified = await githubChatManagementService(db, fetchImpl).verification(endpoint.id);
if (!verified.ready && !(finishOptions?.optionalSlackTestForUser && verified.connectionReady)) throw conflict("Finish verifying the App, repositories, and assigned agent's tools before activating this bot", { checks: verified.checks });
}
const testStartedAtValue = endpoint.setup.testStartedAt;
const testStartedAt = testStartedAtValue
? new Date(testStartedAtValue)
: null;
const optionalSlackTest = Boolean(finishOptions?.optionalSlackTestForUser);
if (optionalSlackTest) {
if (!["slack", "github"].includes(endpoint.provider) || !endpoint.setup.webhookVerifiedAt || !endpoint.providerAccountId) {
throw conflict("Verify the connection before finishing setup");
}
const linked = await findAuthorizedIdentityLink(endpoint, finishOptions!.optionalSlackTestForUser);
if (!linked) throw forbidden("Connect your account before finishing setup");
}
if (!optionalSlackTest) {
if (
!endpoint.lastEventAt ||
!testStartedAtValue ||
!testStartedAt ||
Number.isNaN(testStartedAt.getTime()) ||
endpoint.lastEventAt < testStartedAt
) {
throw conflict(
"Send the test message in the provider before completing setup",
{
code: "chat_test_message_missing",
},
);
}
const requiredTrigger =
["telegram", "imessage-photon"].includes(endpoint.provider)
? "direct_message"
: "subscribed_message";
const qualifyingDelivery = await db
.select({
id: chatDeliveries.id,
conversationId: chatDeliveries.conversationId,
processedAt: chatDeliveries.processedAt,
})
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.companyId, endpoint.companyId),
eq(chatDeliveries.endpointId, endpoint.id),
eq(chatDeliveries.state, "processed"),
gte(chatDeliveries.processedAt, testStartedAt),
sql`${chatDeliveries.normalizedEvent}->>'trigger' = ${requiredTrigger}`,
),
)
.orderBy(desc(chatDeliveries.processedAt))
.limit(1)
.then((rows) => rows[0] ?? null);
if (
!qualifyingDelivery?.conversationId ||
!qualifyingDelivery.processedAt
) {
throw conflict(
["telegram", "imessage-photon"].includes(endpoint.provider)
? "Send the test direct message before completing setup"
: "Reply once without mentioning the agent before completing setup",
{ code: "chat_test_follow_up_missing" },
);
}
const finalPublication = await db
.select({
commentId: chatPublications.commentId,
payload: chatPublications.payload,
})
.from(chatPublications)
.innerJoin(
issueComments,
and(
eq(issueComments.id, chatPublications.commentId),
eq(issueComments.companyId, chatPublications.companyId),
eq(issueComments.issueId, chatPublications.issueId),
eq(issueComments.authorType, "agent"),
eq(issueComments.authorAgentId, endpoint.assignedAgentId),
),
)
.innerJoin(
chatMessageLinks,
and(
eq(chatMessageLinks.companyId, chatPublications.companyId),
eq(chatMessageLinks.endpointId, chatPublications.endpointId),
eq(
chatMessageLinks.conversationId,
chatPublications.conversationId,
),
eq(chatMessageLinks.deliveryId, qualifyingDelivery.id),
eq(chatMessageLinks.direction, "inbound"),
isNotNull(chatMessageLinks.commentId),
),
)
.innerJoin(
heartbeatRuns,
and(
eq(heartbeatRuns.id, issueComments.createdByRunId),
eq(heartbeatRuns.companyId, chatPublications.companyId),
eq(heartbeatRuns.agentId, endpoint.assignedAgentId),
eq(heartbeatRuns.status, "succeeded"),
eq(
sql<string>`${heartbeatRuns.contextSnapshot} ->> 'issueId'`,
sql<string>`${chatPublications.issueId}::text`,
),
or(
sql`${chatMessageLinks.commentId}::text = ${heartbeatRuns.contextSnapshot} ->> 'wakeCommentId'`,
sql`${chatMessageLinks.commentId}::text = ${heartbeatRuns.contextSnapshot} ->> 'commentId'`,
sql`coalesce(${heartbeatRuns.contextSnapshot} -> 'wakeCommentIds', '[]'::jsonb) ? ${chatMessageLinks.commentId}::text`,
),
),
)
.where(
and(
eq(chatPublications.companyId, endpoint.companyId),
eq(chatPublications.endpointId, endpoint.id),
eq(
chatPublications.conversationId,
qualifyingDelivery.conversationId,
),
eq(chatPublications.state, "published"),
gte(chatPublications.publishedAt, qualifyingDelivery.processedAt),
),
)
.orderBy(desc(chatPublications.publishedAt))
.then((rows) =>
rows.find(
(row) =>
row.payload.interactionId === undefined &&
row.payload.progressState === undefined &&
row.commentId !== null,
),
);
let githubToolReply = false;
if (!finalPublication && endpoint.provider === "github") {
const setupRuns = await db.select({ id: heartbeatRuns.id, issueId: chatConversations.issueId })
.from(heartbeatRuns)
.innerJoin(chatMessageLinks, and(
eq(chatMessageLinks.companyId, endpoint.companyId),
eq(chatMessageLinks.endpointId, endpoint.id),
eq(chatMessageLinks.deliveryId, qualifyingDelivery.id),
eq(chatMessageLinks.direction, "inbound"),
or(
sql`${chatMessageLinks.commentId}::text = ${heartbeatRuns.contextSnapshot}->>'wakeCommentId'`,
sql`coalesce(${heartbeatRuns.contextSnapshot}->'wakeCommentIds', '[]'::jsonb) ? ${chatMessageLinks.commentId}::text`,
),
))
.innerJoin(chatConversations, and(
eq(chatConversations.id, qualifyingDelivery.conversationId),
eq(chatConversations.id, chatMessageLinks.conversationId),
eq(chatConversations.companyId, endpoint.companyId),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${chatConversations.issueId}::text`,
))
.where(and(eq(heartbeatRuns.companyId, endpoint.companyId),
eq(heartbeatRuns.agentId, endpoint.assignedAgentId), eq(heartbeatRuns.status, "succeeded")))
.orderBy(desc(heartbeatRuns.createdAt)).limit(20);
for (const run of setupRuns) {
if (await githubRunReplyState(db, { companyId: endpoint.companyId, endpointId: endpoint.id,
issueId: run.issueId, runId: run.id }) === "confirmed") {
githubToolReply = true;
break;
}
}
}
if (!finalPublication && !githubToolReply) {
throw conflict(
"Wait for the Paperclip agent to reply to the setup turn before completing setup",
{
code: "chat_test_round_trip_incomplete",
},
);
}
}
await options.setupTestActivationBarrier?.();
const expectedGeneration = runtimeGeneration(endpoint.setup);
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
if (optionalSlackTest) {
const linked = await tx.select({ principalId: chatIdentityLinks.principalId }).from(chatIdentityLinks).where(and(
eq(chatIdentityLinks.endpointId, endpoint.id), eq(chatIdentityLinks.paperclipUserId, finishOptions!.optionalSlackTestForUser), eq(chatIdentityLinks.status, "linked"),
)).limit(1).then((rows) => rows[0]);
if (!linked || !(await lockCurrentPrincipalAuthorization(tx, endpoint, linked.principalId)).allowed) throw forbidden("Your connected account is no longer authorized");
}
await notifyChatEndpointWork(tx);
const [activated] = await tx
.update(chatEndpoints)
.set({
status: "active",
setup: {
...endpoint.setup,
step: "complete",
testStartedAt: null,
testSkipped: optionalSlackTest,
},
healthMessage: optionalSlackTest ? "Connected; conversation test optional" : "Connected",
activatedAt: endpoint.activatedAt ?? new Date(),
updatedAt: new Date(),
})
.where(
and(
eq(chatEndpoints.id, endpointId),
eq(chatEndpoints.status, "verifying"),
sql`${chatEndpoints.setup}->>'step' = 'test'`,
sql`coalesce((${chatEndpoints.setup}->>'runtimeGeneration')::integer, 0) = ${expectedGeneration}`,
sql`${chatEndpoints.setup}->>'testStartedAt' = ${testStartedAtValue}`,
),
)
.returning({ id: chatEndpoints.id });
if (!activated) {
throw conflict("This setup test was superseded by a reconnect", {
code: "chat_endpoint_test_superseded",
});
}
await tx
.update(toolConnections)
.set({
status: "active",
enabled: true,
healthStatus: "healthy",
healthMessage: optionalSlackTest ? "Connected; conversation test optional" : "Connected",
lastError: null,
healthCheckedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(toolConnections.id, endpoint.connectionId));
if (optionalSlackTest) await logActivity(tx as unknown as Db, {
companyId: endpoint.companyId, actorType: "user", actorId: finishOptions!.optionalSlackTestForUser,
action: "chat.setup_completed", entityType: "chat_endpoint", entityId: endpointId,
details: { conversationTestOptional: true },
});
await credentialLease.assertOwned(tx);
});
return get(endpointId);
},
);
}
return { setupTestStatus, test };
}
@@ -0,0 +1,300 @@
import { isProvisionedSlackSetup } from "@paperclipai/shared";
import type { EndpointRow, DbOrTransaction, SlackCallbackSurface, InternalSetupState } from "../runtime-types.js";
import { notifyChatVerificationWork } from "../../chat-work-notifications.js";
import { slackChatRegistrationService } from "../../chat-slack-registration.js";
import { chatSlackRegistrations } from "@paperclipai/db";
import { runtimeCanonicalOrigin } from "../../cloud-runtime-identity.js";
import {
createHash,
createHmac,
timingSafeEqual,
} from "node:crypto";
import {
and,
eq,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
chatEndpoints,
toolConnections,
} from "@paperclipai/db";
import type {
ToolCredentialSecretRef,
} from "@paperclipai/shared";
import {
logActivity,
} from "../../activity-log.js";
import { secretService } from "../../secrets.js";
function slackRequestSignatureIsValid(
request: Request,
body: string,
signingSecret: string,
): boolean {
const timestamp = request.headers.get("x-slack-request-timestamp");
const signature = request.headers.get("x-slack-signature");
if (!timestamp || !signature) return false;
const timestampSeconds = Number(timestamp);
if (
!Number.isFinite(timestampSeconds) ||
Math.abs(Math.floor(Date.now() / 1000) - timestampSeconds) > 300
)
return false;
const expected = `v0=${createHmac("sha256", signingSecret).update(`v0:${timestamp}:${body}`).digest("hex")}`;
try {
return timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
} catch {
return false;
}
}
function slackRequestWorkspaceId(
body: string,
contentType: string,
): string | null {
let payload: unknown;
try {
if (contentType.includes("application/json")) {
payload = JSON.parse(body);
} else {
const form = new URLSearchParams(body);
const interactivePayload = form.get("payload");
payload = interactivePayload
? JSON.parse(interactivePayload)
: Object.fromEntries(form.entries());
}
} catch {
return null;
}
if (!payload || typeof payload !== "object") return null;
const record = payload as Record<string, unknown>;
const nestedId = (value: unknown): string | null => {
if (!value || typeof value !== "object") return null;
const id = (value as Record<string, unknown>).id;
return typeof id === "string" && id.length > 0 ? id : null;
};
const stringField = (key: string): string | null => {
const value = record[key];
return typeof value === "string" && value.length > 0 ? value : null;
};
const authorization = Array.isArray(record.authorizations)
? record.authorizations.find(
(value): value is Record<string, unknown> =>
Boolean(value) && typeof value === "object",
)
: null;
const authorizationField = (key: string): string | null => {
const value = authorization?.[key];
return typeof value === "string" && value.length > 0 ? value : null;
};
// A Grid event may carry both an enterprise id and the concrete workspace
// id. The bot identity is verified against auth.test's team_id, so prefer a
// team id and use the enterprise id only when Slack omitted team context.
return (
stringField("team_id") ??
nestedId(record.team) ??
authorizationField("team_id") ??
stringField("enterprise_id") ??
nestedId(record.enterprise) ??
authorizationField("enterprise_id")
);
}
function canonicalCallbackUrl(value: string): string | null {
try {
const url = new URL(value);
if (url.protocol !== "https:" && url.protocol !== "http:") return null;
url.hostname = url.hostname.replace(/\.$/, "");
url.username = "";
url.password = "";
url.search = "";
url.hash = "";
return url.toString();
} catch {
return null;
}
}
// TLS terminates at a reverse proxy in many self-hosted deployments. The
// transport scheme is not evidence of URL drift; authority and path still are.
// This is diagnostic only: it does not trust forwarded headers or change auth.
function slackCallbackMatchesPublicUrl(observed: string, current: string): boolean {
const canonical = canonicalCallbackUrl(observed);
if (canonical === current) return true;
if (!canonical) return false;
const observedUrl = new URL(canonical);
const currentUrl = new URL(current);
return observedUrl.protocol === "http:"
&& currentUrl.protocol === "https:"
&& observedUrl.host === currentUrl.host
&& observedUrl.pathname === currentUrl.pathname;
}
async function inspectSlackCallback(
request: Request,
): Promise<SlackCallbackInspection | null> {
const url = slackCallbackObservationUrl(request);
if (!url) return null;
const contentType = request.headers.get("content-type")?.toLowerCase() ?? "";
let body: string;
try {
body = await request.text();
} catch {
return null;
}
if (contentType.includes("application/json")) {
try {
const payload = JSON.parse(body) as {
type?: unknown;
challenge?: unknown;
};
const isUrlVerification =
payload.type === "url_verification" &&
typeof payload.challenge === "string" &&
payload.challenge.length > 0;
if (isUrlVerification || payload.type === "event_callback") {
return {
surface: "events",
url,
isUrlVerification,
};
}
} catch {
return null;
}
return null;
}
const form = new URLSearchParams(body);
if (form.get("command")) {
return { surface: "slashCommands", url, isUrlVerification: false };
}
const interactivePayload = form.get("payload");
if (!interactivePayload) return null;
try {
const payload = JSON.parse(interactivePayload) as { type?: unknown };
if (
typeof payload.type === "string" &&
[
"block_actions",
"block_suggestion",
"dialog_submission",
"message_action",
"shortcut",
"view_closed",
"view_submission",
"workflow_step_edit",
].includes(payload.type)
) {
return { surface: "interactivity", url, isUrlVerification: false };
}
} catch {
return null;
}
return null;
}
type SlackCallbackInspection = {
surface: SlackCallbackSurface;
url: string;
isUrlVerification: boolean;
};
// Cloud records the public request host in dedicated headers so provider edges
// cannot replace it with their upstream host. Only use callback-health evidence on a
// claimed Cloud instance, after provider authentication succeeded. It must not
// change the Request passed to the adapter, signature verification, routing,
// board identity, or the configured URL used to generate callbacks.
function slackCallbackObservationUrl(request: Request): string | null {
const directUrl = canonicalCallbackUrl(request.url);
if (!directUrl || !runtimeCanonicalOrigin()) return directUrl;
const hasCloudEvidence = request.headers.has("x-paperclip-cloud-forwarded-host");
const host = request.headers.get(hasCloudEvidence ? "x-paperclip-cloud-forwarded-host" : "x-forwarded-host")?.trim();
const protocol = request.headers.get(hasCloudEvidence ? "x-paperclip-cloud-forwarded-proto" : "x-forwarded-proto")?.trim();
if (!host || /[\s/@\\?#,]/.test(host) || (protocol !== "https" && protocol !== "http")) return directUrl;
try {
const origin = new URL(`${protocol}://${host}`);
// Accept one authority only, never credentials, paths, queries, or lists.
if (!origin.host || origin.username || origin.password || origin.pathname !== "/" || origin.search || origin.hash) return directUrl;
const observed = new URL(directUrl);
observed.protocol = origin.protocol;
observed.host = origin.host;
observed.port = origin.port;
return canonicalCallbackUrl(observed.toString());
} catch {
return directUrl;
}
}
export { slackRequestSignatureIsValid, slackRequestWorkspaceId, canonicalCallbackUrl, slackCallbackMatchesPublicUrl, inspectSlackCallback };
type SlackCallbackVerificationOptions = {
resolveCredentialRefs: (endpoint: EndpointRow, refs: ToolCredentialSecretRef[], database?: DbOrTransaction) => Promise<Record<string, string>>;
getWebhookPublicBaseUrl: () => string | null;
scheduleMessageProcessing: (task: () => Promise<void>) => void;
slackRegistration: Pick<ReturnType<typeof slackChatRegistrationService>, "notifyVerified">;
};
export function createSlackCallbackVerification(db: Db, options: SlackCallbackVerificationOptions) {
const { resolveCredentialRefs, getWebhookPublicBaseUrl, scheduleMessageProcessing, slackRegistration } = options;
async function verifySlackRequestUrl(endpoint: EndpointRow, request: Request): Promise<Response | null> {
const callback = await inspectSlackCallback(request.clone());
if (!callback?.isUrlVerification) return null;
const body = await request.clone().text();
let notifyAccount = false;
// URL verification needs only the signing secret. Do not initialize the
// SDK (which calls Slack auth.test), wait for OAuth, or acquire the creation
// lease: Slack can deliver this check while app setup still owns that lease.
const response = await db.transaction(async tx => {
const [current] = await tx.select().from(chatEndpoints).where(eq(chatEndpoints.id, endpoint.id)).for("no key update");
if (!current || ["archived", "paused", "revoked"].includes(current.status)) return new Response("ignored", { status: 200 });
const [connection] = await tx.select({ status: toolConnections.status, refs: toolConnections.credentialSecretRefs }).from(toolConnections)
.where(and(eq(toolConnections.id, current.connectionId), eq(toolConnections.companyId, current.companyId)));
if (!connection || connection.status === "archived") return new Response("ignored", { status: 200 });
let secret = (await resolveCredentialRefs(current, connection.refs.filter(ref => ref.configPath.replace(/^credentials\./, "") === "signingSecret"), tx)).signingSecret;
if (!secret && isProvisionedSlackSetup(current.setup.slackSetupMethod)) {
const [registration] = await tx.select().from(chatSlackRegistrations)
.where(and(eq(chatSlackRegistrations.endpointId, current.id), eq(chatSlackRegistrations.companyId, current.companyId)));
if (registration?.status !== "removed" && registration?.secretIds.signingSecret) {
secret = await secretService(tx as unknown as Db).resolveSecretValue(current.companyId, registration.secretIds.signingSecret, "latest", {
accessContext: { consumerType: "system", consumerId: `slack-registration:${current.id}`, configPath: "slack_registration.signingSecret", actorType: "system" },
});
}
}
if (!secret) return new Response("Slack signing secret is not ready yet", { status: 503 });
if (!slackRequestSignatureIsValid(request, body, secret)) return new Response("Invalid signature", { status: 401 });
const workspaceId = slackRequestWorkspaceId(body, request.headers.get("content-type") ?? "");
if (workspaceId && current.providerAccountId && workspaceId !== current.providerAccountId) return new Response("ignored", { status: 200 });
const observedAt = new Date();
const setup = current.setup as InternalSetupState;
const matchesCurrentUrl = slackCallbackMatchesPublicUrl(callback.url, `${getWebhookPublicBaseUrl()}/api/chat-webhooks/${current.publicId}/slack`);
await notifyChatVerificationWork(tx);
await tx.update(chatEndpoints).set({ setup: {
...setup,
...(matchesCurrentUrl ? { webhookVerifiedAt: setup.webhookVerifiedAt ?? observedAt.toISOString(),
slackVerificationSigningFingerprint: createHash("sha256").update(secret).digest("hex") } : {}),
slackCallbackSurfaces: { ...setup.slackCallbackSurfaces, events: { url: callback.url, observedAt: observedAt.toISOString() } },
...(matchesCurrentUrl && setup.slackAccount?.status === "linked" && !setup.slackAccount.verificationStatus
? { slackAccount: { ...setup.slackAccount, verificationStatus: "pending" as const } } : {}),
} as InternalSetupState, ...(matchesCurrentUrl ? { healthMessage: "Slack Events Request URL verified" } : {}), updatedAt: observedAt }).where(eq(chatEndpoints.id, current.id));
if (matchesCurrentUrl && !setup.webhookVerifiedAt) await logActivity(tx as unknown as Db, { companyId: current.companyId, actorType: "system", actorId: "slack-webhook",
action: "chat_endpoint.webhook_verified", entityType: "tool_connection", entityId: current.connectionId,
details: { endpointId: current.id, provider: "slack" } });
notifyAccount = matchesCurrentUrl && setup.slackAccount?.status === "linked" && !setup.slackAccount.verificationStatus;
return Response.json({ challenge: (JSON.parse(body) as { challenge: string }).challenge });
});
if (notifyAccount) scheduleMessageProcessing(() => slackRegistration.notifyVerified(endpoint.id));
return response;
}
return { verifySlackRequestUrl };
}
@@ -41,3 +41,49 @@ export function slackMessageSentAt(raw: unknown, messageId: string): Date | null
const date = new Date(Number(ts) * 1_000);
return Number.isFinite(date.getTime()) ? date : null;
}
export function isSlackReceiptReactionAlreadyApplied(error: unknown): boolean {
const pending: Array<{ depth: number; value: unknown }> = [
{ depth: 0, value: error },
];
const seen = new Set<object>();
while (pending.length > 0) {
const current = pending.shift();
if (
!current ||
current.depth > 4 ||
!current.value ||
typeof current.value !== "object" ||
seen.has(current.value)
) {
continue;
}
seen.add(current.value);
const record = current.value as {
cause?: unknown;
code?: unknown;
data?: { error?: unknown };
original?: unknown;
originalError?: unknown;
};
if (
record.code === "already_reacted" ||
record.data?.error === "already_reacted"
) {
return true;
}
for (const value of [record.cause, record.original, record.originalError]) {
pending.push({ depth: current.depth + 1, value });
}
}
return false;
}
export function slackThreadChannelId(threadId: string): string | null {
if (!threadId.startsWith("slack:")) return null;
const channelAndThread = threadId.slice("slack:".length);
const separator = channelAndThread.indexOf(":");
const channelId =
separator === -1 ? channelAndThread : channelAndThread.slice(0, separator);
return channelId || null;
}
@@ -1073,3 +1073,21 @@ export function createSlackTaskStart(db: Db, options: SlackTaskStartOptions) {
}
return { latestSlackDmControlThreadId, finalizeSlackTaskStartFailure, admitConfirmedSlackTaskStart, processSlackTaskStart, processPendingSlackTaskStarts };
}
export function slackTaskStartActivityStatus(action: {
status: string;
updatedAt: Date;
}): string {
const age = Date.now() - action.updatedAt.getTime();
if (action.status === "validating" && age >= SLACK_COMMAND_POST_STALE_MS) {
return "queued";
}
if (
(action.status === "received" && age >= SLACK_COMMAND_POST_STALE_MS) ||
(action.status === "resolving" &&
age >= SLACK_COMMAND_EXPLICIT_RETRY_STALE_MS)
) {
return "delivery_unknown";
}
return action.status;
}
@@ -0,0 +1,128 @@
import type { EndpointRow, DbTransaction, DbOrTransaction } from "../runtime-types.js";
import {
createHash,
} from "node:crypto";
import {
and,
eq,
} from "drizzle-orm";
import {
chatEndpointResources,
chatExternalPrincipals,
chatIdentityLinks,
} from "@paperclipai/db";
import {
type TelegramCallbackReceipt,
} from "../../chat-telegram-ephemeral.js";
type TelegramCallbackNoticeAuthorityOptions = {
nonDirectDestinationAllowed: (endpoint: EndpointRow, resource: typeof chatEndpointResources.$inferSelect | null | undefined) => boolean;
lockCurrentPrincipalAuthorization: (tx: DbOrTransaction, endpoint: EndpointRow, principalId: string) => Promise<{ allowed: boolean; linkedDenied: boolean; userId: string | null; sponsorUserId?: string | null }>;
};
export function createTelegramCallbackNoticeAuthority(options: TelegramCallbackNoticeAuthorityOptions) {
const { nonDirectDestinationAllowed, lockCurrentPrincipalAuthorization } = options;
// This is permission to return one fixed, neutral callback denial, not
// permission to execute the denied action or publish to the conversation.
// Retain the initial identity-policy snapshot so queued notices cannot cross
// a subsequent actor/link/reach change. No provider I/O occurs under locks.
async function lockTelegramCallbackNoticeAuthority(
tx: DbTransaction,
endpoint: EndpointRow,
receipt: TelegramCallbackReceipt,
principalId: string | null,
): Promise<string | null> {
if (
endpoint.provider !== "telegram" ||
endpoint.status !== "active" ||
endpoint.companyId !== receipt.companyId ||
endpoint.id !== receipt.endpointId ||
endpoint.botExternalId !== receipt.botUserId ||
!endpoint.providerAccountId ||
!principalId ||
Date.now() < receipt.receivedAtMs ||
Date.now() >= receipt.deadlineAtMs
)
return null;
const resource =
receipt.chatType === "private"
? null
: await tx
.select()
.from(chatEndpointResources)
.where(
and(
eq(chatEndpointResources.companyId, endpoint.companyId),
eq(chatEndpointResources.endpointId, endpoint.id),
eq(chatEndpointResources.providerResourceId, receipt.chatId),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
if (
receipt.chatType === "private"
? !endpoint.allowDirectMessages
: !resource || !nonDirectDestinationAllowed(endpoint, resource)
)
return null;
const principal = await tx
.select()
.from(chatExternalPrincipals)
.where(
and(
eq(chatExternalPrincipals.companyId, endpoint.companyId),
eq(chatExternalPrincipals.provider, endpoint.provider),
eq(
chatExternalPrincipals.providerAccountId,
endpoint.providerAccountId,
),
eq(chatExternalPrincipals.id, principalId),
eq(chatExternalPrincipals.externalId, receipt.receiverUserId),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
if (!principal || principal.kind !== "user" || principal.isBot) return null;
const authorization = await lockCurrentPrincipalAuthorization(
tx,
endpoint,
principal.id,
);
const link = await tx
.select({
status: chatIdentityLinks.status,
userId: chatIdentityLinks.paperclipUserId,
})
.from(chatIdentityLinks)
.where(
and(
eq(chatIdentityLinks.companyId, endpoint.companyId),
eq(chatIdentityLinks.endpointId, endpoint.id),
eq(chatIdentityLinks.principalId, principal.id),
),
)
.then((rows) => rows[0] ?? null);
return createHash("sha256")
.update(
JSON.stringify([
endpoint.companyId,
endpoint.id,
receipt.botUserId,
resource?.id ?? null,
principal.id,
principal.externalId,
link,
authorization,
endpoint.allowUnlinkedPeople,
endpoint.sponsorUserId,
]),
)
.digest("hex");
}
return { lockTelegramCallbackNoticeAuthority };
}
@@ -1,3 +1,4 @@
import type { Attachment } from "chat";
import { createHash } from "node:crypto";
import type { chatDeliveries } from "@paperclipai/db";
import type { ChatProvider } from "@paperclipai/shared";
@@ -221,3 +222,14 @@ export function telegramMessageSentAt(raw: unknown): Date | null {
const sentAt = new Date(value * 1_000);
return Number.isFinite(sentAt.getTime()) ? sentAt : null;
}
export function telegramMarkdownAttachment(text: string): Attachment {
const data = Buffer.from(text, "utf8");
return {
data,
mimeType: "text/markdown; charset=utf-8",
name: "paperclip-response.md",
size: data.byteLength,
type: "file",
};
}