From 3a213ab987080150387c16e45b993eaf1435dc30 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 12 Oct 2026 09:39:51 -0500 Subject: [PATCH 1/7] refactor(chat): extract resource reconciliation and availability (#15834) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages AI agents and their work. > - Chat resources control which provider destinations can admit that work. > - Inventory updates and lifecycle callbacks must preserve company scope and operator choices. > - These rules share a large service with runtime and credential ownership. > - This PR extracts resource reconciliation and its availability transaction. > - Reviewers can inspect resource changes together while the service retains runtime control. ## Linked Issues or Issue Description Refs #15823 (merged). Slice 2 of 5 now targets `master`. Remaining order: #15834 → #15836 → #15844 → #15845. ## What Changed - Extract shared resource inventory, audited selection and availability settlement from the large service. - Put GitHub repository parsing, rename/transfer repair, webhook recovery and listing in `github/resources.ts`. - Put Telegram group migration in `telegram/resources.ts`, using the coordinator's existing transaction and timestamp. - Put Slack labels, Teams IDs and Photon participant parsing in their channel folders. - Keep lease assertions, delivery settlement and the resource upsert between Telegram migration steps in the shared coordinator. - The shared module is 598 lines; provider resource modules total 409 lines. Update the dated plan with this reassessment. ## Verification - On updated master, 15 existing resource integration cases and one Telegram migration case passed before this folder move. The same 16 cases passed after. Other cases were excluded by the filter. - Eight moved declarations and three transaction blocks match the prior implementation. The GitHub provider guard remains in the caller. - Server TypeScript, module-boundary and whitespace checks passed. Rebuilt plugin SDK types after rebasing onto current master. - Historical browser shard 8 timed out on inbox retry; its unchanged [rerun passed](https://github.com/paperclipai/paperclip/actions/runs/38134430238/job/114457529315). [Original failure](https://github.com/paperclipai/paperclip/actions/runs/38134430238/job/114455332214). - Tests use local disposable databases and fixture provider transports. No live third-party execution is claimed. - Fresh Greptile review is 5/5 on `e914f532ca66c429125492c1723df38afc6e13d7`, with a successful check and no open review threads. - Final-stack `pnpm -r typecheck` and `pnpm build` passed at `57ec5ba94aecd860471f9b14db46f99120c7c790`. - All CI checks passed on `e914f532ca66c429125492c1723df38afc6e13d7`. - The final full-local-suite outcome for this cumulative stack is recorded in #15845; do not infer that gate from the focused tests above. ## Risks - Resource identity and migrations depend on lock and write order. Provider modules use the original transaction; statement order and credential fencing remain unchanged. - Shared selection and availability still contain provider dispatch predicates. They coordinate common state transitions and should remain shared. - Full local tests and current-revision CI remain required for a green handoff. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and repository tools. The exact runtime model ID and context-window size are not exposed in this session. No separate agent was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../2026-10-10-chat-channels-refactor.md | 47 + server/src/services/chat-channels.ts | 931 +----------------- .../chat-channels/github/resources.ts | 278 ++++++ .../imessage-photon/resources.ts | 14 + .../microsoft-teams/resources.ts | 3 + .../chat-channels/resource-reconciliation.ts | 598 +++++++++++ .../services/chat-channels/slack/resources.ts | 10 + .../chat-channels/telegram/resources.ts | 104 ++ 8 files changed, 1065 insertions(+), 920 deletions(-) create mode 100644 server/src/services/chat-channels/github/resources.ts create mode 100644 server/src/services/chat-channels/imessage-photon/resources.ts create mode 100644 server/src/services/chat-channels/microsoft-teams/resources.ts create mode 100644 server/src/services/chat-channels/resource-reconciliation.ts create mode 100644 server/src/services/chat-channels/slack/resources.ts create mode 100644 server/src/services/chat-channels/telegram/resources.ts diff --git a/doc/plans/2026-10-10-chat-channels-refactor.md b/doc/plans/2026-10-10-chat-channels-refactor.md index 711c6e6b125..2b6765a72ea 100644 --- a/doc/plans/2026-10-10-chat-channels-refactor.md +++ b/doc/plans/2026-10-10-chat-channels-refactor.md @@ -125,3 +125,50 @@ this slice to `paperclipai/paperclip`; subsequent extractions are not part of this PR. Review and final verification: [PR #15823](https://github.com/paperclipai/paperclip/pull/15823). + +## Authorized stacked continuation + +The user requested at least five stacked PRs on 2026-10-10. Continue the four +remaining boundaries in order, with each branch based on its predecessor. Keep +merge and deployment outside this series. Each PR carries its own current-head +verification and review results; do not transfer a prior head's pass to a new one. + +### Slice 2: resource reconciliation and availability + +The lifecycle handler also owns credential replacement, delivery claims and +runtime shutdown. Keep those responsibilities in the service. Extract the full +non-GitHub availability transaction, inventory reconciliation, repository rename +handling, provider ordering, resource listing and audited selection together. +The factory takes the database, endpoint lookup, existing credential lease +function, generation reader and error text limit. It creates no worker or cache. + +- Source: 38,633 lines / 1,491,422 bytes → 37,720 lines / 1,454,762 bytes. +- Destination: 947 lines / 39,584 bytes in `resource-reconciliation.ts`. +- Mechanical comparison: 14 declarations and the complete availability + transaction are unchanged; all other function bodies remain unchanged. +- Baseline: all three complete chat CI shards passed on predecessor `cbba30a526`. +- Local baseline and after selections could not start PostgreSQL and skipped + the suite. They are not passes. Server typecheck and module boundaries pass. +- Require complete chat CI on this slice before starting the next extraction. + +### 2026-10-12 channel-folder reassessment: slice 2 + +PR #15823 was merged at the user's request. Rebase this slice onto the merged +master, then retain the remaining four PRs for review in dependency order. + +- GitHub payload parsing, repository coordinate repair, webhook recovery, + repository pagination and bulk-selection entry points live in + `chat-channels/github/resources.ts`. +- Telegram migration preparation and conversation migration live in + `telegram/resources.ts`. They receive the existing transaction and timestamp; + the shared coordinator still upserts the destination between those steps. +- Slack fallback labels, Teams conversation IDs and Photon participant parsing + live in each provider's `resources.ts`. +- Shared inventory replacement, availability settlement and audited selection + retain their transaction and credential-lease ownership. Small provider + dispatch predicates remain next to the shared operation they select. +- All eight moved declarations and three transaction blocks retain their logic. + The GitHub provider guard stays in the caller. No factory starts background work. +- Existing public-service tests: 15 resource cases and the Telegram migration + case passed before; the same 16 cases passed after. The name filter excludes + unrelated cases. No live third-party provider calls were made. diff --git a/server/src/services/chat-channels.ts b/server/src/services/chat-channels.ts index 9716073c91f..a0b5c93324c 100644 --- a/server/src/services/chat-channels.ts +++ b/server/src/services/chat-channels.ts @@ -1,3 +1,8 @@ +import { createChatResourceReconciliation } from "./chat-channels/resource-reconciliation.js"; +import { githubRepositoryInventoryItemFromPayload } from "./chat-channels/github/resources.js"; +import { baseTeamsConversationId } from "./chat-channels/microsoft-teams/resources.js"; +import { slackResourceLabelIsFallback } from "./chat-channels/slack/resources.js"; +import { telegramResourceLabelIsFallback } from "./chat-channels/telegram/resources.js"; import { MAX_INBOUND_TEXT, stableExternalPrincipalId, @@ -53,7 +58,7 @@ import type { AgentAvatarRequest } from "./agent-avatars.js"; import { slackChatRegistrationService, slackRegistrationProjection } from "./chat-slack-registration.js"; import { chatSlackRegistrations } from "@paperclipai/db"; import { SLACK_CHAT_BOT_SCOPES } from "@paperclipai/shared"; -import { authorizationService, canActorReadIssuePrivacy, canPublishIssueToChatAudience } from "./authorization.js"; +import { canActorReadIssuePrivacy, canPublishIssueToChatAudience } from "./authorization.js"; import { withSlackBoardLease } from "./slack-board-lease.js"; import { mirrorSlackBoardComment, slackBoardReplyBindings } from "./slack-board-messages.js"; import { authorizeSlackBoardPublication } from "./slack-board-authority.js"; @@ -64,7 +69,7 @@ import { slackAuthorizationRevision } from "./connectors/slack-revision.js"; import { slackPublicationAllowed } from "./connectors/slack-access.js"; import { instanceSettingsService } from "./instance-settings.js"; import { registerSlackTaskAuthority, slackRunOrigin } from "./connectors/slack-authority.js"; -import { captureRunIdentity } from "./run-identity.js"; + import { buildChatCommunicationGuidance } from "./chat-communication-guidance.js"; function githubPolicyRecord(value: unknown): Record { return value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; } import { githubChatManagementService } from "./chat-github-management.js"; @@ -102,7 +107,7 @@ import { validateNativeQuestionResponseInput } from "./native-runtime/native-que import type { AskUserQuestionsAnswer, AskUserQuestionsInteraction, IssueThreadInteraction } from "@paperclipai/shared"; import { PhotonCloudClient, PhotonError, photonFailure, photonSharedIdentity, photonSharedScope } from "./photon/cloud.js"; import { PhotonChatAdapter, photonThreadId, photonReplyReference } from "./photon/adapter.js"; -import { photonChannelConfigurationSchema, type PhotonChannelConfiguration } from "@paperclipai/shared"; +import { photonChannelConfigurationSchema } from "@paperclipai/shared"; import type { LiveEvent as PhotonEvent } from "@photon-ai/advanced-imessage"; import { defaultSlackAppConfiguration } from "@paperclipai/shared"; import { @@ -346,14 +351,7 @@ import { restoreGitHubAttachmentLimitOmissions, } from "./chat-github-attachments.js"; import { getExternalChannelBindingSummary } from "./chat-channel-binding.js"; -import { - discoverDedicatedGitHubAppInstallation, - getSlackBotChannel, - listGitHubInstallationRepositories, - listSlackBotChannels, - type ChatProviderInventoryResult, - type ChatProviderResourceInventoryItem, -} from "./chat-provider-inventory.js"; +import { discoverDedicatedGitHubAppInstallation, getSlackBotChannel, listGitHubInstallationRepositories, listSlackBotChannels, type ChatProviderInventoryResult } from "./chat-provider-inventory.js"; import { discordMarkdownRequiresAttachment, listDiscordBotChannels, @@ -1623,10 +1621,6 @@ function microsoftTeamsServiceUrl(raw: unknown): string | null { return typeof value === "string" && value ? value : null; } -function baseTeamsConversationId(value: string): string { - return value.replace(/;messageid=[^;]+/i, ""); -} - function teamsThreadRootMessageId(threadId: string): string | null { const conversationId = teamsConversationId(threadId); if (!conversationId) return null; @@ -1656,28 +1650,6 @@ function canonicalProviderResourceId( return thread.channelId; } -function slackResourceLabelIsFallback( - providerResourceId: string, - label: string, -): boolean { - const candidate = label.trim(); - return ( - candidate === providerResourceId || - candidate === `slack:${providerResourceId}` - ); -} - -function telegramResourceLabelIsFallback( - providerResourceId: string, - label: string, -): boolean { - const candidate = label.trim(); - return ( - candidate === providerResourceId || - candidate === `telegram:${providerResourceId}` - ); -} - function providerResourceLabelFromThread( provider: ChatProvider, thread: Pick, @@ -2053,67 +2025,6 @@ function isSupportedTelegramWebhookBaseUrl(value: string | null): boolean { } } -function githubRepositoryInventoryItemFromPayload( - payload: unknown, -): ChatProviderResourceInventoryItem | null { - if (!payload || typeof payload !== "object" || Array.isArray(payload)) - return null; - const repository = (payload as { repository?: unknown }).repository; - if ( - !repository || - typeof repository !== "object" || - Array.isArray(repository) - ) - return null; - const value = repository as { - id?: unknown; - full_name?: unknown; - html_url?: unknown; - name?: unknown; - owner?: { id?: unknown; login?: unknown }; - private?: unknown; - }; - const repositoryId = - typeof value.id === "number" && Number.isSafeInteger(value.id) - ? String(value.id) - : typeof value.id === "string" && /^\d+$/.test(value.id) - ? value.id - : null; - const owner = - typeof value.owner?.login === "string" ? value.owner.login.trim() : ""; - const name = typeof value.name === "string" ? value.name.trim() : ""; - const fullName = - typeof value.full_name === "string" && value.full_name.includes("/") - ? value.full_name.trim() - : owner && name - ? `${owner}/${name}` - : ""; - if (!repositoryId || !fullName) return null; - const ownerId = - typeof value.owner?.id === "number" && Number.isSafeInteger(value.owner.id) - ? String(value.owner.id) - : typeof value.owner?.id === "string" && /^\d+$/.test(value.owner.id) - ? value.owner.id - : undefined; - return { - providerResourceId: fullName.toLowerCase(), - parentProviderResourceId: ownerId, - type: "repository", - label: fullName, - providerUrl: - typeof value.html_url === "string" && value.html_url.length > 0 - ? value.html_url - : `https://github.com/${fullName}`, - metadata: { - providerRepositoryId: repositoryId, - fullName, - ...(owner ? { owner } : {}), - private: value.private === true, - source: "provider_webhook", - }, - }; -} - function providerSetupState( endpoint: Pick< EndpointRow, @@ -6283,6 +6194,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { } const withCredentialMutationLease = chatCredentialMutationLease(db, options); + const { reconcileGitHubWebhookRepository, reconcileProviderResourceRows, listResources, listGitHubRepositories, replaceResources, toggleAllGitHubRepositories, hasNewerProcessedProviderLifecycleEffect, applyProviderAvailability } = createChatResourceReconciliation(db, { endpointRecord, withCredentialMutationLease, runtimeGeneration, maxErrorText: MAX_ERROR_TEXT }); async function generateSetupSecret( endpointId: string, @@ -6563,273 +6475,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { ); } - function githubRepositoryStableId( - item: ChatProviderResourceInventoryItem, - ): string | null { - const value = item.metadata?.providerRepositoryId; - return typeof value === "string" && /^\d+$/.test(value) ? value : null; - } - - async function upsertProviderResourceRow( - tx: DbTransaction, - endpoint: EndpointRow, - item: ChatProviderResourceInventoryItem, - ) { - const stableGitHubId = - endpoint.provider === "github" ? githubRepositoryStableId(item) : null; - if (stableGitHubId) { - await tx.execute( - sql`select pg_advisory_xact_lock(hashtextextended(${`chat-github-repository:${endpoint.id}:${stableGitHubId}`}, 0))`, - ); - const stableResource = await tx - .select() - .from(chatEndpointResources) - .where( - and( - eq(chatEndpointResources.companyId, endpoint.companyId), - eq(chatEndpointResources.endpointId, endpoint.id), - eq(chatEndpointResources.type, "repository"), - sql`${chatEndpointResources.metadata}->>'providerRepositoryId' = ${stableGitHubId}`, - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - if ( - stableResource && - stableResource.providerResourceId !== item.providerResourceId - ) { - const coordinateResource = await tx - .select() - .from(chatEndpointResources) - .where( - and( - eq(chatEndpointResources.companyId, endpoint.companyId), - eq(chatEndpointResources.endpointId, endpoint.id), - eq(chatEndpointResources.type, "repository"), - eq( - chatEndpointResources.providerResourceId, - item.providerResourceId, - ), - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - if (coordinateResource && coordinateResource.id !== stableResource.id) { - const coordinateConversation = await tx - .select({ id: chatConversations.id }) - .from(chatConversations) - .where( - and( - eq(chatConversations.endpointId, endpoint.id), - eq(chatConversations.resourceId, coordinateResource.id), - ), - ) - .limit(1) - .then((rows) => rows[0] ?? null); - if (coordinateConversation) { - throw conflict( - "GitHub repository identity is already bound to two coordinates", - { code: "chat_github_repository_identity_conflict" }, - ); - } - await tx - .delete(chatEndpointResources) - .where(eq(chatEndpointResources.id, coordinateResource.id)); - } - - const oldRepository = stableResource.providerResourceId; - const oldThreadPrefix = `github:${oldRepository}`; - const newThreadPrefix = `github:${item.providerResourceId}`; - const oldProviderUrl = `https://github.com/${oldRepository}`; - const newProviderUrl = `https://github.com/${item.providerResourceId}`; - await tx - .update(chatEndpointResources) - .set({ - providerResourceId: item.providerResourceId, - parentProviderResourceId: item.parentProviderResourceId ?? null, - label: item.label, - providerUrl: item.providerUrl ?? null, - availability: "available", - enabled: - stableResource.enabled || coordinateResource?.enabled === true, - metadata: item.metadata ?? {}, - updatedAt: new Date(), - }) - .where(eq(chatEndpointResources.id, stableResource.id)); - await tx - .update(chatConversations) - .set({ - externalConversationId: sql`case - when lower(${chatConversations.externalConversationId}) = ${oldRepository} then ${item.providerResourceId} - when lower(${chatConversations.externalConversationId}) = ${oldThreadPrefix} then ${newThreadPrefix} - else ${chatConversations.externalConversationId} - end`, - externalThreadId: sql`case - when lower(${chatConversations.externalThreadId}) = ${oldThreadPrefix} then ${newThreadPrefix} - when lower(${chatConversations.externalThreadId}) like ${`${oldThreadPrefix}:%`} - then ${newThreadPrefix} || substring( - ${chatConversations.externalThreadId} - from char_length(${oldThreadPrefix}) + 1 - ) - else ${chatConversations.externalThreadId} - end`, - externalLabel: item.label, - providerUrl: sql`case - when lower(${chatConversations.providerUrl}) like ${`${oldProviderUrl}/%`} - then ${newProviderUrl} || substring( - ${chatConversations.providerUrl} - from char_length(${oldProviderUrl}) + 1 - ) - else ${chatConversations.providerUrl} - end`, - updatedAt: new Date(), - }) - .where( - and( - eq(chatConversations.companyId, endpoint.companyId), - eq(chatConversations.endpointId, endpoint.id), - eq(chatConversations.resourceId, stableResource.id), - ), - ); - return { id: stableResource.id }; - } - } - - const [resource] = await tx - .insert(chatEndpointResources) - .values({ - companyId: endpoint.companyId, - endpointId: endpoint.id, - type: item.type, - providerResourceId: item.providerResourceId, - parentProviderResourceId: item.parentProviderResourceId ?? null, - label: item.label, - providerUrl: item.providerUrl ?? null, - availability: "available", - // Slack inventory contains only channels the bot has joined. Apply the - // invitation default on insert; conflict updates preserve operator choices. - enabled: - endpoint.provider === "slack" && - item.type === "channel" && - item.metadata?.creator !== endpoint.botExternalId, - metadata: item.metadata ?? {}, - }) - .onConflictDoUpdate({ - target: [ - chatEndpointResources.endpointId, - chatEndpointResources.type, - chatEndpointResources.providerResourceId, - ], - set: { - parentProviderResourceId: item.parentProviderResourceId ?? null, - label: item.label, - providerUrl: item.providerUrl ?? null, - availability: "available", - metadata: item.metadata ?? {}, - updatedAt: new Date(), - }, - }) - .returning({ id: chatEndpointResources.id }); - return resource ?? null; - } - - async function reconcileGitHubWebhookRepository( - endpoint: EndpointRow, - payload: unknown, - credentialLease: CredentialMutationLeaseGuard, - ) { - const item = githubRepositoryInventoryItemFromPayload(payload); - if (!item) return; - await db.transaction(async (tx) => { - await credentialLease.assertOwned(tx); - const resource = await upsertProviderResourceRow(tx, endpoint, item); - if (!resource) return; - // A correctly signed repository callback is current provider proof. Keep - // Paperclip's enabled choice, and reopen only conversations previously - // quarantined for provider availability loss. - await tx - .update(chatConversations) - .set({ state: "active", updatedAt: new Date() }) - .where( - and( - eq(chatConversations.companyId, endpoint.companyId), - eq(chatConversations.endpointId, endpoint.id), - eq(chatConversations.resourceId, resource.id), - eq(chatConversations.state, "unavailable"), - ), - ); - await credentialLease.assertOwned(tx); - }); - } - - async function reconcileProviderResourceRows( - endpoint: EndpointRow, - inventory: ChatProviderInventoryResult, - credentialLease: CredentialMutationLeaseGuard, - ) { - const resourceType = - endpoint.provider === "github" ? "repository" : "channel"; - const discovered = new Set( - inventory.resources.map((resource) => resource.providerResourceId), - ); - const absentAvailability = - endpoint.provider === "github" ? "removed" : "unavailable"; - await db.transaction(async (tx) => { - await credentialLease.assertOwned(tx); - for (const item of inventory.resources) { - const resource = await upsertProviderResourceRow(tx, endpoint, item); - if (resource) { - // Resource recovery reopens only bindings that Paperclip marked - // unavailable. Completed historical tasks stay completed. - await tx - .update(chatConversations) - .set({ state: "active", updatedAt: new Date() }) - .where( - and( - eq(chatConversations.companyId, endpoint.companyId), - eq(chatConversations.endpointId, endpoint.id), - eq(chatConversations.resourceId, resource.id), - eq(chatConversations.state, "unavailable"), - ), - ); - } - } - - const existing = await tx - .select({ - id: chatEndpointResources.id, - providerResourceId: chatEndpointResources.providerResourceId, - }) - .from(chatEndpointResources) - .where( - and( - eq(chatEndpointResources.companyId, endpoint.companyId), - eq(chatEndpointResources.endpointId, endpoint.id), - eq(chatEndpointResources.type, resourceType), - ), - ); - for (const resource of existing) { - if (discovered.has(resource.providerResourceId)) continue; - await tx - .update(chatEndpointResources) - .set({ availability: absentAvailability, updatedAt: new Date() }) - .where(eq(chatEndpointResources.id, resource.id)); - await tx - .update(chatConversations) - .set({ state: "unavailable", updatedAt: new Date() }) - .where( - and( - eq(chatConversations.companyId, endpoint.companyId), - eq(chatConversations.endpointId, endpoint.id), - eq(chatConversations.resourceId, resource.id), - inArray(chatConversations.state, ["active", "waiting"]), - ), - ); - } - await credentialLease.assertOwned(tx); - }); - } - async function hydrateSlackResourceLabel( endpointId: string, providerResourceId: string, @@ -23812,309 +23457,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { effect.kind === "endpoint" && (effect.availability === "attention" || effect.availability === "revoked"); - await db.transaction(async (tx) => { - await credentialLease.assertOwned(tx); - const now = new Date(); - if (staleResourceEffect) { - // Persisted provider-native ordering wins over request arrival or - // lease acquisition order. The older callback remains auditable, - // but cannot roll a resource or conversation back. - await tx - .update(chatEndpoints) - .set({ lastEventAt: now, updatedAt: now }) - .where(eq(chatEndpoints.id, currentEndpoint.id)); - await notifyChatDeliveryWork(tx); - await tx - .update(chatDeliveries) - .set({ - state: "processed", - processedAt: now, - redactedError: null, - nextAttemptAt: null, - updatedAt: now, - }) - .where(eq(chatDeliveries.id, claimed.id)); - return; - } - if (effect.kind === "resource") { - const providerResourceId = - currentEndpoint.provider === "microsoft-teams" - ? baseTeamsConversationId(effect.providerResourceId) - : effect.providerResourceId; - const previousProviderResourceId = - currentEndpoint.provider === "telegram" - ? effect.previousProviderResourceId - : undefined; - const previousResource = previousProviderResourceId - ? await tx - .select() - .from(chatEndpointResources) - .where( - and( - eq(chatEndpointResources.endpointId, currentEndpoint.id), - eq(chatEndpointResources.type, "chat"), - eq( - chatEndpointResources.providerResourceId, - previousProviderResourceId, - ), - ), - ) - .for("update") - .then((rows) => rows[0] ?? null) - : null; - const migratedLabel = - previousResource && - telegramResourceLabelIsFallback(providerResourceId, effect.label) - ? previousResource.label - : effect.label; - const migratedEnabled = previousResource?.enabled === true; - if ( - previousResource && - previousProviderResourceId !== providerResourceId - ) { - await tx - .update(chatEndpointResources) - .set({ - availability: "unavailable", - enabled: false, - metadata: { - ...previousResource.metadata, - source: "chat_migration", - migratedTo: providerResourceId, - }, - updatedAt: now, - }) - .where(eq(chatEndpointResources.id, previousResource.id)); - } - const preserveProviderLabel = - (currentEndpoint.provider === "slack" && - slackResourceLabelIsFallback( - providerResourceId, - migratedLabel, - )) || - (currentEndpoint.provider === "telegram" && - telegramResourceLabelIsFallback( - providerResourceId, - migratedLabel, - )); - const [resource] = await tx - .insert(chatEndpointResources) - .values({ - companyId: currentEndpoint.companyId, - endpointId: currentEndpoint.id, - type: effect.resourceType, - providerResourceId, - parentProviderResourceId: - effect.parentProviderResourceId ?? null, - label: migratedLabel, - providerUrl: effect.providerUrl ?? null, - availability: effect.availability, - enabled: - migratedEnabled || - (currentEndpoint.provider === "slack" && - effect.resourceType === "channel" && - effect.availability === "available"), - metadata: effect.metadata ?? {}, - }) - .onConflictDoUpdate({ - target: [ - chatEndpointResources.endpointId, - chatEndpointResources.type, - chatEndpointResources.providerResourceId, - ], - set: { - parentProviderResourceId: - effect.parentProviderResourceId ?? null, - ...(preserveProviderLabel ? {} : { label: migratedLabel }), - providerUrl: effect.providerUrl ?? null, - availability: effect.availability, - ...(previousProviderResourceId - ? { - enabled: sql`${chatEndpointResources.enabled} or ${migratedEnabled}`, - } - : {}), - metadata: effect.metadata ?? {}, - updatedAt: now, - }, - }) - .returning({ id: chatEndpointResources.id }); - if ( - resource && - previousResource && - previousProviderResourceId !== providerResourceId - ) { - const previousThreadPrefix = `telegram:${previousProviderResourceId}`; - const migratedThreadPrefix = `telegram:${providerResourceId}`; - await tx - .update(chatConversations) - .set({ - resourceId: resource.id, - // Telegram SDK threads use the namespaced channel id. Keep - // the migrated row in that canonical shape so the first - // reply from the replacement supergroup finds this same - // task instead of creating a second conversation. - externalConversationId: migratedThreadPrefix, - externalThreadId: sql`case - when ${chatConversations.externalThreadId} = ${previousThreadPrefix} then ${migratedThreadPrefix} - when ${chatConversations.externalThreadId} like ${`${previousThreadPrefix}:%`} - then ${migratedThreadPrefix} || substring( - ${chatConversations.externalThreadId} - from char_length(${previousThreadPrefix}) + 1 - ) - else ${chatConversations.externalThreadId} - end`, - externalLabel: migratedLabel, - state: "active", - updatedAt: now, - }) - .where( - and( - eq(chatConversations.companyId, currentEndpoint.companyId), - eq(chatConversations.endpointId, currentEndpoint.id), - eq(chatConversations.resourceId, previousResource.id), - ), - ); - } - if (resource) { - await tx - .update(chatConversations) - .set({ - state: - effect.availability === "available" - ? "active" - : "unavailable", - updatedAt: now, - }) - .where( - and( - eq(chatConversations.companyId, currentEndpoint.companyId), - eq(chatConversations.endpointId, currentEndpoint.id), - eq(chatConversations.resourceId, resource.id), - effect.availability === "available" - ? eq(chatConversations.state, "unavailable") - : inArray(chatConversations.state, ["active", "waiting"]), - ), - ); - } - await tx - .update(chatEndpoints) - .set({ lastEventAt: now, updatedAt: now }) - .where(eq(chatEndpoints.id, currentEndpoint.id)); - } else { - const reason = redactSensitiveText(effect.reason).slice( - 0, - MAX_ERROR_TEXT, - ); - if (effect.availability === "available") { - if ( - currentEndpoint.status !== "paused" && - currentEndpoint.status !== "archived" - ) { - const status = - currentEndpoint.setup.step === "complete" - ? "active" - : currentEndpoint.status === "attention" - ? "verifying" - : currentEndpoint.status; - if (status !== currentEndpoint.status) await notifyChatEndpointWork(tx); - await tx - .update(chatEndpoints) - .set({ - status, - healthMessage: - status === "active" - ? "Connected" - : "Waiting for a test conversation", - lastError: null, - lastEventAt: now, - updatedAt: now, - }) - .where(eq(chatEndpoints.id, currentEndpoint.id)); - await tx - .update(toolConnections) - .set({ - status: "active", - enabled: true, - healthStatus: "healthy", - healthMessage: "Connected", - lastError: null, - healthCheckedAt: now, - updatedAt: now, - }) - .where(eq(toolConnections.id, currentEndpoint.connectionId)); - } - } else { - await tx - .update(chatEndpoints) - .set({ - status: effect.availability, - healthMessage: reason, - lastError: reason, - lastEventAt: now, - setup: { - ...currentEndpoint.setup, - runtimeGeneration: - runtimeGeneration(currentEndpoint.setup) + 1, - } as InternalSetupState, - updatedAt: now, - }) - .where(eq(chatEndpoints.id, currentEndpoint.id)); - await tx - .update(toolConnections) - .set({ - status: "disabled", - enabled: false, - healthStatus: - effect.availability === "revoked" ? "failed" : "degraded", - healthMessage: reason, - lastError: reason, - healthCheckedAt: now, - updatedAt: now, - }) - .where(eq(toolConnections.id, currentEndpoint.connectionId)); - await tx - .update(chatEndpointResources) - .set({ availability: "unavailable", updatedAt: now }) - .where( - and( - eq( - chatEndpointResources.companyId, - currentEndpoint.companyId, - ), - eq(chatEndpointResources.endpointId, currentEndpoint.id), - ), - ); - await tx - .update(chatConversations) - .set({ state: "unavailable", updatedAt: now }) - .where( - and( - eq(chatConversations.companyId, currentEndpoint.companyId), - eq(chatConversations.endpointId, currentEndpoint.id), - inArray(chatConversations.state, ["active", "waiting"]), - ), - ); - } - } - await notifyChatDeliveryWork(tx); - await tx - .update(chatDeliveries) - .set({ - state: "processed", - processedAt: now, - redactedError: null, - nextAttemptAt: null, - updatedAt: now, - }) - .where( - and( - eq(chatDeliveries.id, claimed.id), - eq(chatDeliveries.state, "processing"), - eq(chatDeliveries.attempts, candidate.attempts + 1), - ), - ); - await credentialLease.assertOwned(tx); - }); + await applyProviderAvailability({ currentEndpoint, effect, credentialLease, staleResourceEffect, claimed, candidate }); stopRuntimeAfterLifecycle = endpointEffectStopsRuntime; }); } catch (error) { @@ -24210,43 +23553,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { return { ...normalized, runtimeContext }; } - async function hasNewerProcessedProviderLifecycleEffect( - endpoint: EndpointRow, - effect: Extract, - ): Promise { - const sequence = effect.providerOrder?.sequence; - const occurredAt = effect.providerOrder?.occurredAt; - if (!sequence && !occurredAt) return false; - const providerResourceId = - endpoint.provider === "microsoft-teams" - ? baseTeamsConversationId(effect.providerResourceId) - : effect.providerResourceId; - const newerOrder = sequence - ? sql`( - ${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerOrder,sequence}' ~ '^[0-9]+([.][0-9]+)?$' - and (${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerOrder,sequence}')::numeric > cast(${sequence} as numeric) - )` - : sql`( - nullif(${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerOrder,occurredAt}', '')::timestamptz > cast(${occurredAt!} as timestamptz) - )`; - const newer = await db - .select({ id: chatDeliveries.id }) - .from(chatDeliveries) - .where( - and( - eq(chatDeliveries.endpointId, endpoint.id), - eq(chatDeliveries.state, "processed"), - inArray(chatDeliveries.eventKind, ["installation", "uninstallation"]), - sql`${chatDeliveries.normalizedEvent}#>>'{lifecycle,kind}' = 'resource'`, - sql`${chatDeliveries.normalizedEvent}#>>'{lifecycle,provider}' = ${effect.provider}`, - sql`${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerResourceId}' = ${providerResourceId}`, - newerOrder, - ), - ) - .limit(1); - return newer.length > 0; - } - async function applyProviderLifecycleEffects( endpoint: EndpointRow, effects: ChatProviderLifecycleEffect[], @@ -27715,221 +27021,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { return reactionCount + rows.length; } - async function listResources(endpointId: string) { - const resources = await db - .select() - .from(chatEndpointResources) - .where( - and( - eq(chatEndpointResources.endpointId, endpointId), - ne(chatEndpointResources.type, "direct_message"), - ), - ) - .orderBy(asc(chatEndpointResources.label)); - return resources.map((resource) => ({ - ...resource, - participants: - resource.providerResourceId.startsWith("imessage-photon:") && - Array.isArray(resource.metadata.participants) - ? resource.metadata.participants.flatMap((participant) => { - const address = - participant && typeof participant === "object" - ? (participant as Record).address - : null; - return typeof address === "string" ? [address] : []; - }) - : undefined, - })); - } - - async function listGitHubRepositories( - endpointId: string, - query: { limit: number; offset: number; search: string }, - ) { - const record = await endpointRecord(endpointId); - if (!record || record.endpoint.provider !== "github") throw notFound("GitHub bot not found"); - const scope = and( - eq(chatEndpointResources.companyId, record.endpoint.companyId), - eq(chatEndpointResources.endpointId, endpointId), - eq(chatEndpointResources.type, "repository"), - ); - // Literal, case-insensitive substring search (%, _ and backslashes are - // ordinary characters). Counts always describe the entire connection. - const search = query.search - ? sql`strpos(lower(${chatEndpointResources.label}), lower(${query.search})) > 0` - : undefined; - const [rows, [counts]] = await Promise.all([ - db.select().from(chatEndpointResources).where(and(scope, search)) - .orderBy(asc(chatEndpointResources.label), asc(chatEndpointResources.id)) - .offset(query.offset).limit(query.limit + 1), - db.select({ - totalCount: sql`count(*)::integer`, - enabledCount: sql`count(*) filter (where ${chatEndpointResources.enabled})::integer`, - availableCount: sql`count(*) filter (where ${chatEndpointResources.availability} = 'available')::integer`, - }).from(chatEndpointResources).where(scope), - ]); - return { - items: rows.slice(0, query.limit), - nextOffset: rows.length > query.limit ? query.offset + query.limit : null, - totalCount: counts!.totalCount, - enabledCount: counts!.enabledCount, - availableCount: counts!.availableCount, - }; - } - - async function replaceResources( - endpointId: string, - updates: Array<{ id: string; enabled: boolean }>, - actorUserId?: string | null, - options?: { initialGitHubImport?: boolean }, - ) { - await updateResourceSelection(endpointId, updates, actorUserId, options); - return listResources(endpointId); - } - - async function toggleAllGitHubRepositories(endpointId: string, enabled: boolean, actorUserId?: string | null) { - await updateResourceSelection(endpointId, [], actorUserId, { allGitHubRepositoriesEnabled: enabled }); - return { success: true as const }; - } - - async function updateResourceSelection( - endpointId: string, - updates: Array<{ id: string; enabled: boolean }>, - actorUserId?: string | null, - options?: { initialGitHubImport?: boolean; allGitHubRepositoriesEnabled?: boolean }, - ) { - const initial = await endpointRecord(endpointId); - if (!initial) throw notFound("Chat endpoint not found"); - const bulk = options?.allGitHubRepositoriesEnabled !== undefined; - if (bulk && initial.endpoint.provider !== "github") throw badRequest("Only GitHub repositories support toggle all"); - if (updates.length === 0 && initial.endpoint.provider !== "github") return; - if (initial.endpoint.provider === "imessage-photon" && initial.endpoint.botExternalId?.startsWith("photon-project:") && updates.some((entry) => entry.enabled)) - throw unprocessable("Photon shared channels support direct messages only; groups cannot be enabled"); - await withCredentialMutationLease( - initial.endpoint, - async (credentialLease) => { - const ids = updates.map((entry) => entry.id); - const publications: ActivityPublication[] = []; - await db.transaction(async (tx) => { - await credentialLease.assertOwned(tx); - const [endpoint] = await tx - .select({ - companyId: chatEndpoints.companyId, - connectionId: chatEndpoints.connectionId, - provider: chatEndpoints.provider, - setup: chatEndpoints.setup, - }) - .from(chatEndpoints) - .where( - and( - eq(chatEndpoints.id, endpointId), - eq(chatEndpoints.companyId, initial.endpoint.companyId), - ), - ) - .for("no key update"); - if (!endpoint) throw notFound("Chat endpoint not found"); - if (options?.initialGitHubImport && endpoint.provider === "github" && (endpoint.setup.github?.repositorySelectionSaved || !endpoint.setup.github?.initialRepositoryImportPending)) return; - const rows = await tx - .select({ - id: chatEndpointResources.id, - availability: chatEndpointResources.availability, - enabled: chatEndpointResources.enabled, - }) - .from(chatEndpointResources) - .where( - and( - eq(chatEndpointResources.companyId, endpoint.companyId), - eq(chatEndpointResources.endpointId, endpointId), - bulk - ? and( - eq(chatEndpointResources.type, "repository"), - options!.allGitHubRepositoriesEnabled - ? eq(chatEndpointResources.availability, "available") - : undefined, - ) - : inArray(chatEndpointResources.id, ids), - ), - ) - .orderBy(asc(chatEndpointResources.id)) - .for("no key update"); - if (!bulk && rows.length !== new Set(ids).size) - throw unprocessable("Every resource must belong to this endpoint"); - const selectedUpdates = bulk - ? rows.map((row) => ({ id: row.id, enabled: options!.allGitHubRepositoriesEnabled! })) - : updates; - const availabilityById = new Map( - rows.map((row) => [row.id, row.availability]), - ); - // Validate every submitted grant, including intermediate duplicate - // entries. Netting below describes the audit, not new authority. - const unavailable = selectedUpdates.find( - (entry) => - entry.enabled && availabilityById.get(entry.id) !== "available", - ); - if (unavailable) - throw conflict( - "A destination must still be available from the provider before it can be enabled", - { code: "chat_resource_unavailable", resourceId: unavailable.id }, - ); - const finalEnabled = new Map( - selectedUpdates.map((entry) => [entry.id, entry.enabled]), - ); - const changes = rows - .filter((row) => row.enabled !== finalEnabled.get(row.id)) - .map((row) => ({ - resourceId: row.id, - before: { enabled: row.enabled }, - after: { enabled: finalEnabled.get(row.id)! }, - })); - if (bulk) { - // One atomic database update, independent of pagination/search and - // serialized with refresh, reconnect, and individual row changes. - await tx.update(chatEndpointResources) - .set({ enabled: options!.allGitHubRepositoriesEnabled!, updatedAt: new Date() }) - .where(and( - eq(chatEndpointResources.companyId, endpoint.companyId), - eq(chatEndpointResources.endpointId, endpointId), - eq(chatEndpointResources.type, "repository"), - options!.allGitHubRepositoriesEnabled - ? eq(chatEndpointResources.availability, "available") - : undefined, - )); - } else for (const entry of selectedUpdates) - await tx - .update(chatEndpointResources) - .set({ enabled: entry.enabled, updatedAt: new Date() }) - .where( - and( - eq(chatEndpointResources.companyId, endpoint.companyId), - eq(chatEndpointResources.endpointId, endpointId), - eq(chatEndpointResources.id, entry.id), - ), - ); - if (endpoint.provider === "github") - await tx.update(chatEndpoints).set({ setup: sql`jsonb_set(${chatEndpoints.setup}, '{github}', coalesce(${chatEndpoints.setup}->'github', '{}'::jsonb) || '{"repositorySelectionSaved":true,"initialRepositoryImportPending":false}'::jsonb)`, updatedAt: new Date() }).where(eq(chatEndpoints.id, endpointId)); - if (changes.length > 0 || (endpoint.provider === "github" && !endpoint.setup.github?.repositorySelectionSaved)) - await logActivity( - tx as unknown as Db, - { - companyId: endpoint.companyId, - actorType: "user", - actorId: actorUserId ?? "board", - action: "chat_endpoint.resources_updated", - entityType: "tool_connection", - entityId: endpoint.connectionId, - details: { endpointId, provider: endpoint.provider, changes }, - }, - publications, - ); - await credentialLease.assertOwned(tx); - }); - // A committed reach change is factual even if the outer lease's last - // check subsequently fails. Never publish an uncommitted activity. - for (const publication of publications) publishActivity(publication); - }, - ); - } - async function listPrincipals(endpointId: string) { const record = await endpointRecord(endpointId); if (!record) throw notFound("Chat endpoint not found"); diff --git a/server/src/services/chat-channels/github/resources.ts b/server/src/services/chat-channels/github/resources.ts new file mode 100644 index 00000000000..5abe2384cd2 --- /dev/null +++ b/server/src/services/chat-channels/github/resources.ts @@ -0,0 +1,278 @@ +import { and, asc, eq, sql } from "drizzle-orm"; +import { chatConversations, chatEndpointResources, chatEndpoints, type Db } from "@paperclipai/db"; +import { conflict, notFound } from "../../../errors.js"; +import type { CredentialMutationLeaseGuard } from "../../chat-credential-mutation-lease.js"; +import type { ChatProviderResourceInventoryItem } from "../../chat-provider-inventory.js"; +type EndpointRow = typeof chatEndpoints.$inferSelect; +type DbTransaction = Parameters[0]>[0]; + +export function githubRepositoryInventoryItemFromPayload( + payload: unknown, +): ChatProviderResourceInventoryItem | null { + if (!payload || typeof payload !== "object" || Array.isArray(payload)) + return null; + const repository = (payload as { repository?: unknown }).repository; + if ( + !repository || + typeof repository !== "object" || + Array.isArray(repository) + ) + return null; + const value = repository as { + id?: unknown; + full_name?: unknown; + html_url?: unknown; + name?: unknown; + owner?: { id?: unknown; login?: unknown }; + private?: unknown; + }; + const repositoryId = + typeof value.id === "number" && Number.isSafeInteger(value.id) + ? String(value.id) + : typeof value.id === "string" && /^\d+$/.test(value.id) + ? value.id + : null; + const owner = + typeof value.owner?.login === "string" ? value.owner.login.trim() : ""; + const name = typeof value.name === "string" ? value.name.trim() : ""; + const fullName = + typeof value.full_name === "string" && value.full_name.includes("/") + ? value.full_name.trim() + : owner && name + ? `${owner}/${name}` + : ""; + if (!repositoryId || !fullName) return null; + const ownerId = + typeof value.owner?.id === "number" && Number.isSafeInteger(value.owner.id) + ? String(value.owner.id) + : typeof value.owner?.id === "string" && /^\d+$/.test(value.owner.id) + ? value.owner.id + : undefined; + return { + providerResourceId: fullName.toLowerCase(), + parentProviderResourceId: ownerId, + type: "repository", + label: fullName, + providerUrl: + typeof value.html_url === "string" && value.html_url.length > 0 + ? value.html_url + : `https://github.com/${fullName}`, + metadata: { + providerRepositoryId: repositoryId, + fullName, + ...(owner ? { owner } : {}), + private: value.private === true, + source: "provider_webhook", + }, + }; +} + +function githubRepositoryStableId( + item: ChatProviderResourceInventoryItem, +): string | null { + const value = item.metadata?.providerRepositoryId; + return typeof value === "string" && /^\d+$/.test(value) ? value : null; +} + +export async function reconcileGitHubRepositoryCoordinates(tx: DbTransaction, endpoint: EndpointRow, item: ChatProviderResourceInventoryItem) { + const stableGitHubId = + githubRepositoryStableId(item); + if (stableGitHubId) { + await tx.execute( + sql`select pg_advisory_xact_lock(hashtextextended(${`chat-github-repository:${endpoint.id}:${stableGitHubId}`}, 0))`, + ); + const stableResource = await tx + .select() + .from(chatEndpointResources) + .where( + and( + eq(chatEndpointResources.companyId, endpoint.companyId), + eq(chatEndpointResources.endpointId, endpoint.id), + eq(chatEndpointResources.type, "repository"), + sql`${chatEndpointResources.metadata}->>'providerRepositoryId' = ${stableGitHubId}`, + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + if ( + stableResource && + stableResource.providerResourceId !== item.providerResourceId + ) { + const coordinateResource = await tx + .select() + .from(chatEndpointResources) + .where( + and( + eq(chatEndpointResources.companyId, endpoint.companyId), + eq(chatEndpointResources.endpointId, endpoint.id), + eq(chatEndpointResources.type, "repository"), + eq( + chatEndpointResources.providerResourceId, + item.providerResourceId, + ), + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + if (coordinateResource && coordinateResource.id !== stableResource.id) { + const coordinateConversation = await tx + .select({ id: chatConversations.id }) + .from(chatConversations) + .where( + and( + eq(chatConversations.endpointId, endpoint.id), + eq(chatConversations.resourceId, coordinateResource.id), + ), + ) + .limit(1) + .then((rows) => rows[0] ?? null); + if (coordinateConversation) { + throw conflict( + "GitHub repository identity is already bound to two coordinates", + { code: "chat_github_repository_identity_conflict" }, + ); + } + await tx + .delete(chatEndpointResources) + .where(eq(chatEndpointResources.id, coordinateResource.id)); + } + + const oldRepository = stableResource.providerResourceId; + const oldThreadPrefix = `github:${oldRepository}`; + const newThreadPrefix = `github:${item.providerResourceId}`; + const oldProviderUrl = `https://github.com/${oldRepository}`; + const newProviderUrl = `https://github.com/${item.providerResourceId}`; + await tx + .update(chatEndpointResources) + .set({ + providerResourceId: item.providerResourceId, + parentProviderResourceId: item.parentProviderResourceId ?? null, + label: item.label, + providerUrl: item.providerUrl ?? null, + availability: "available", + enabled: + stableResource.enabled || coordinateResource?.enabled === true, + metadata: item.metadata ?? {}, + updatedAt: new Date(), + }) + .where(eq(chatEndpointResources.id, stableResource.id)); + await tx + .update(chatConversations) + .set({ + externalConversationId: sql`case + when lower(${chatConversations.externalConversationId}) = ${oldRepository} then ${item.providerResourceId} + when lower(${chatConversations.externalConversationId}) = ${oldThreadPrefix} then ${newThreadPrefix} + else ${chatConversations.externalConversationId} + end`, + externalThreadId: sql`case + when lower(${chatConversations.externalThreadId}) = ${oldThreadPrefix} then ${newThreadPrefix} + when lower(${chatConversations.externalThreadId}) like ${`${oldThreadPrefix}:%`} + then ${newThreadPrefix} || substring( + ${chatConversations.externalThreadId} + from char_length(${oldThreadPrefix}) + 1 + ) + else ${chatConversations.externalThreadId} + end`, + externalLabel: item.label, + providerUrl: sql`case + when lower(${chatConversations.providerUrl}) like ${`${oldProviderUrl}/%`} + then ${newProviderUrl} || substring( + ${chatConversations.providerUrl} + from char_length(${oldProviderUrl}) + 1 + ) + else ${chatConversations.providerUrl} + end`, + updatedAt: new Date(), + }) + .where( + and( + eq(chatConversations.companyId, endpoint.companyId), + eq(chatConversations.endpointId, endpoint.id), + eq(chatConversations.resourceId, stableResource.id), + ), + ); + return { id: stableResource.id }; + } + } + + return null; +} + +type GitHubResourceOptions = { + endpointRecord: (endpointId: string) => Promise<{ endpoint: EndpointRow } | null>; + upsertProviderResourceRow: (tx: DbTransaction, endpoint: EndpointRow, item: ChatProviderResourceInventoryItem) => Promise<{ id: string } | null>; + updateResourceSelection: (endpointId: string, updates: Array<{ id: string; enabled: boolean }>, actorUserId?: string | null, options?: { allGitHubRepositoriesEnabled?: boolean }) => Promise; +}; + +export function createGitHubResources(db: Db, options: GitHubResourceOptions) { + const { endpointRecord, upsertProviderResourceRow, updateResourceSelection } = options; + async function reconcileGitHubWebhookRepository( + endpoint: EndpointRow, + payload: unknown, + credentialLease: CredentialMutationLeaseGuard, + ) { + const item = githubRepositoryInventoryItemFromPayload(payload); + if (!item) return; + await db.transaction(async (tx) => { + await credentialLease.assertOwned(tx); + const resource = await upsertProviderResourceRow(tx, endpoint, item); + if (!resource) return; + // A correctly signed repository callback is current provider proof. Keep + // Paperclip's enabled choice, and reopen only conversations previously + // quarantined for provider availability loss. + await tx + .update(chatConversations) + .set({ state: "active", updatedAt: new Date() }) + .where( + and( + eq(chatConversations.companyId, endpoint.companyId), + eq(chatConversations.endpointId, endpoint.id), + eq(chatConversations.resourceId, resource.id), + eq(chatConversations.state, "unavailable"), + ), + ); + await credentialLease.assertOwned(tx); + }); + } + + async function listGitHubRepositories( + endpointId: string, + query: { limit: number; offset: number; search: string }, + ) { + const record = await endpointRecord(endpointId); + if (!record || record.endpoint.provider !== "github") throw notFound("GitHub bot not found"); + const scope = and( + eq(chatEndpointResources.companyId, record.endpoint.companyId), + eq(chatEndpointResources.endpointId, endpointId), + eq(chatEndpointResources.type, "repository"), + ); + // Literal, case-insensitive substring search (%, _ and backslashes are + // ordinary characters). Counts always describe the entire connection. + const search = query.search + ? sql`strpos(lower(${chatEndpointResources.label}), lower(${query.search})) > 0` + : undefined; + const [rows, [counts]] = await Promise.all([ + db.select().from(chatEndpointResources).where(and(scope, search)) + .orderBy(asc(chatEndpointResources.label), asc(chatEndpointResources.id)) + .offset(query.offset).limit(query.limit + 1), + db.select({ + totalCount: sql`count(*)::integer`, + enabledCount: sql`count(*) filter (where ${chatEndpointResources.enabled})::integer`, + availableCount: sql`count(*) filter (where ${chatEndpointResources.availability} = 'available')::integer`, + }).from(chatEndpointResources).where(scope), + ]); + return { + items: rows.slice(0, query.limit), + nextOffset: rows.length > query.limit ? query.offset + query.limit : null, + totalCount: counts!.totalCount, + enabledCount: counts!.enabledCount, + availableCount: counts!.availableCount, + }; + } + + async function toggleAllGitHubRepositories(endpointId: string, enabled: boolean, actorUserId?: string | null) { + await updateResourceSelection(endpointId, [], actorUserId, { allGitHubRepositoriesEnabled: enabled }); + return { success: true as const }; + } + return { reconcileGitHubWebhookRepository, listGitHubRepositories, toggleAllGitHubRepositories }; +} diff --git a/server/src/services/chat-channels/imessage-photon/resources.ts b/server/src/services/chat-channels/imessage-photon/resources.ts new file mode 100644 index 00000000000..b86a15c3e42 --- /dev/null +++ b/server/src/services/chat-channels/imessage-photon/resources.ts @@ -0,0 +1,14 @@ +import type { chatEndpointResources } from "@paperclipai/db"; + +export function imessageResourceParticipants(resource: typeof chatEndpointResources.$inferSelect) { + return resource.providerResourceId.startsWith("imessage-photon:") && + Array.isArray(resource.metadata.participants) + ? resource.metadata.participants.flatMap((participant) => { + const address = + participant && typeof participant === "object" + ? (participant as Record).address + : null; + return typeof address === "string" ? [address] : []; + }) + : undefined; +} diff --git a/server/src/services/chat-channels/microsoft-teams/resources.ts b/server/src/services/chat-channels/microsoft-teams/resources.ts new file mode 100644 index 00000000000..675245b00e5 --- /dev/null +++ b/server/src/services/chat-channels/microsoft-teams/resources.ts @@ -0,0 +1,3 @@ +export function baseTeamsConversationId(value: string): string { + return value.replace(/;messageid=[^;]+/i, ""); +} diff --git a/server/src/services/chat-channels/resource-reconciliation.ts b/server/src/services/chat-channels/resource-reconciliation.ts new file mode 100644 index 00000000000..ac0d1069d73 --- /dev/null +++ b/server/src/services/chat-channels/resource-reconciliation.ts @@ -0,0 +1,598 @@ +import { imessageResourceParticipants } from "./imessage-photon/resources.js"; +import { createGitHubResources, reconcileGitHubRepositoryCoordinates } from "./github/resources.js"; +import { baseTeamsConversationId } from "./microsoft-teams/resources.js"; +import { slackResourceLabelIsFallback } from "./slack/resources.js"; +import { telegramResourceLabelIsFallback, prepareTelegramResourceMigration, migrateTelegramResourceConversations } from "./telegram/resources.js"; +import { and, asc, eq, inArray, ne, sql } from "drizzle-orm"; +import { chatConversations, chatDeliveries, chatEndpointResources, chatEndpoints, toolConnections, type Db } from "@paperclipai/db"; +import type { ChatEndpointSetupState } from "@paperclipai/shared"; +import { badRequest, conflict, notFound, unprocessable } from "../../errors.js"; +import { redactSensitiveText } from "../../redaction.js"; +import { notifyChatDeliveryWork, notifyChatEndpointWork } from "../chat-work-notifications.js"; +import { logActivity, publishActivity, type ActivityPublication } from "../activity-log.js"; +import type { CredentialMutationLeaseGuard, chatCredentialMutationLease } from "../chat-credential-mutation-lease.js"; +import type { ChatProviderInventoryResult, ChatProviderResourceInventoryItem } from "../chat-provider-inventory.js"; +import type { ChatProviderLifecycleEffect } from "../chat-provider-lifecycle.js"; + +type EndpointRow = typeof chatEndpoints.$inferSelect; +type DbTransaction = Parameters[0]>[0]; +type InternalSetupState = ChatEndpointSetupState & { runtimeGeneration?: number }; + +type ResourceReconciliationOptions = { + endpointRecord: (endpointId: string) => Promise<{ endpoint: EndpointRow } | null>; + withCredentialMutationLease: ReturnType; + runtimeGeneration: (setup: ChatEndpointSetupState) => number; + maxErrorText: number; +}; + +export function createChatResourceReconciliation(db: Db, options: ResourceReconciliationOptions) { + const { endpointRecord, withCredentialMutationLease, runtimeGeneration, maxErrorText: MAX_ERROR_TEXT } = options; + + async function upsertProviderResourceRow( + tx: DbTransaction, + endpoint: EndpointRow, + item: ChatProviderResourceInventoryItem, + ) { + if (endpoint.provider === "github") { + const renamed = await reconcileGitHubRepositoryCoordinates(tx, endpoint, item); + if (renamed) return renamed; + } + + const [resource] = await tx + .insert(chatEndpointResources) + .values({ + companyId: endpoint.companyId, + endpointId: endpoint.id, + type: item.type, + providerResourceId: item.providerResourceId, + parentProviderResourceId: item.parentProviderResourceId ?? null, + label: item.label, + providerUrl: item.providerUrl ?? null, + availability: "available", + // Slack inventory contains only channels the bot has joined. Apply the + // invitation default on insert; conflict updates preserve operator choices. + enabled: + endpoint.provider === "slack" && + item.type === "channel" && + item.metadata?.creator !== endpoint.botExternalId, + metadata: item.metadata ?? {}, + }) + .onConflictDoUpdate({ + target: [ + chatEndpointResources.endpointId, + chatEndpointResources.type, + chatEndpointResources.providerResourceId, + ], + set: { + parentProviderResourceId: item.parentProviderResourceId ?? null, + label: item.label, + providerUrl: item.providerUrl ?? null, + availability: "available", + metadata: item.metadata ?? {}, + updatedAt: new Date(), + }, + }) + .returning({ id: chatEndpointResources.id }); + return resource ?? null; + } + + async function reconcileProviderResourceRows( + endpoint: EndpointRow, + inventory: ChatProviderInventoryResult, + credentialLease: CredentialMutationLeaseGuard, + ) { + const resourceType = + endpoint.provider === "github" ? "repository" : "channel"; + const discovered = new Set( + inventory.resources.map((resource) => resource.providerResourceId), + ); + const absentAvailability = + endpoint.provider === "github" ? "removed" : "unavailable"; + await db.transaction(async (tx) => { + await credentialLease.assertOwned(tx); + for (const item of inventory.resources) { + const resource = await upsertProviderResourceRow(tx, endpoint, item); + if (resource) { + // Resource recovery reopens only bindings that Paperclip marked + // unavailable. Completed historical tasks stay completed. + await tx + .update(chatConversations) + .set({ state: "active", updatedAt: new Date() }) + .where( + and( + eq(chatConversations.companyId, endpoint.companyId), + eq(chatConversations.endpointId, endpoint.id), + eq(chatConversations.resourceId, resource.id), + eq(chatConversations.state, "unavailable"), + ), + ); + } + } + + const existing = await tx + .select({ + id: chatEndpointResources.id, + providerResourceId: chatEndpointResources.providerResourceId, + }) + .from(chatEndpointResources) + .where( + and( + eq(chatEndpointResources.companyId, endpoint.companyId), + eq(chatEndpointResources.endpointId, endpoint.id), + eq(chatEndpointResources.type, resourceType), + ), + ); + for (const resource of existing) { + if (discovered.has(resource.providerResourceId)) continue; + await tx + .update(chatEndpointResources) + .set({ availability: absentAvailability, updatedAt: new Date() }) + .where(eq(chatEndpointResources.id, resource.id)); + await tx + .update(chatConversations) + .set({ state: "unavailable", updatedAt: new Date() }) + .where( + and( + eq(chatConversations.companyId, endpoint.companyId), + eq(chatConversations.endpointId, endpoint.id), + eq(chatConversations.resourceId, resource.id), + inArray(chatConversations.state, ["active", "waiting"]), + ), + ); + } + await credentialLease.assertOwned(tx); + }); + } + + async function listResources(endpointId: string) { + const resources = await db + .select() + .from(chatEndpointResources) + .where( + and( + eq(chatEndpointResources.endpointId, endpointId), + ne(chatEndpointResources.type, "direct_message"), + ), + ) + .orderBy(asc(chatEndpointResources.label)); + return resources.map((resource) => ({ + ...resource, + participants: imessageResourceParticipants(resource), + })); + } + + async function replaceResources( + endpointId: string, + updates: Array<{ id: string; enabled: boolean }>, + actorUserId?: string | null, + options?: { initialGitHubImport?: boolean }, + ) { + await updateResourceSelection(endpointId, updates, actorUserId, options); + return listResources(endpointId); + } + + async function updateResourceSelection( + endpointId: string, + updates: Array<{ id: string; enabled: boolean }>, + actorUserId?: string | null, + options?: { initialGitHubImport?: boolean; allGitHubRepositoriesEnabled?: boolean }, + ) { + const initial = await endpointRecord(endpointId); + if (!initial) throw notFound("Chat endpoint not found"); + const bulk = options?.allGitHubRepositoriesEnabled !== undefined; + if (bulk && initial.endpoint.provider !== "github") throw badRequest("Only GitHub repositories support toggle all"); + if (updates.length === 0 && initial.endpoint.provider !== "github") return; + if (initial.endpoint.provider === "imessage-photon" && initial.endpoint.botExternalId?.startsWith("photon-project:") && updates.some((entry) => entry.enabled)) + throw unprocessable("Photon shared channels support direct messages only; groups cannot be enabled"); + await withCredentialMutationLease( + initial.endpoint, + async (credentialLease) => { + const ids = updates.map((entry) => entry.id); + const publications: ActivityPublication[] = []; + await db.transaction(async (tx) => { + await credentialLease.assertOwned(tx); + const [endpoint] = await tx + .select({ + companyId: chatEndpoints.companyId, + connectionId: chatEndpoints.connectionId, + provider: chatEndpoints.provider, + setup: chatEndpoints.setup, + }) + .from(chatEndpoints) + .where( + and( + eq(chatEndpoints.id, endpointId), + eq(chatEndpoints.companyId, initial.endpoint.companyId), + ), + ) + .for("no key update"); + if (!endpoint) throw notFound("Chat endpoint not found"); + if (options?.initialGitHubImport && endpoint.provider === "github" && (endpoint.setup.github?.repositorySelectionSaved || !endpoint.setup.github?.initialRepositoryImportPending)) return; + const rows = await tx + .select({ + id: chatEndpointResources.id, + availability: chatEndpointResources.availability, + enabled: chatEndpointResources.enabled, + }) + .from(chatEndpointResources) + .where( + and( + eq(chatEndpointResources.companyId, endpoint.companyId), + eq(chatEndpointResources.endpointId, endpointId), + bulk + ? and( + eq(chatEndpointResources.type, "repository"), + options!.allGitHubRepositoriesEnabled + ? eq(chatEndpointResources.availability, "available") + : undefined, + ) + : inArray(chatEndpointResources.id, ids), + ), + ) + .orderBy(asc(chatEndpointResources.id)) + .for("no key update"); + if (!bulk && rows.length !== new Set(ids).size) + throw unprocessable("Every resource must belong to this endpoint"); + const selectedUpdates = bulk + ? rows.map((row) => ({ id: row.id, enabled: options!.allGitHubRepositoriesEnabled! })) + : updates; + const availabilityById = new Map( + rows.map((row) => [row.id, row.availability]), + ); + // Validate every submitted grant, including intermediate duplicate + // entries. Netting below describes the audit, not new authority. + const unavailable = selectedUpdates.find( + (entry) => + entry.enabled && availabilityById.get(entry.id) !== "available", + ); + if (unavailable) + throw conflict( + "A destination must still be available from the provider before it can be enabled", + { code: "chat_resource_unavailable", resourceId: unavailable.id }, + ); + const finalEnabled = new Map( + selectedUpdates.map((entry) => [entry.id, entry.enabled]), + ); + const changes = rows + .filter((row) => row.enabled !== finalEnabled.get(row.id)) + .map((row) => ({ + resourceId: row.id, + before: { enabled: row.enabled }, + after: { enabled: finalEnabled.get(row.id)! }, + })); + if (bulk) { + // One atomic database update, independent of pagination/search and + // serialized with refresh, reconnect, and individual row changes. + await tx.update(chatEndpointResources) + .set({ enabled: options!.allGitHubRepositoriesEnabled!, updatedAt: new Date() }) + .where(and( + eq(chatEndpointResources.companyId, endpoint.companyId), + eq(chatEndpointResources.endpointId, endpointId), + eq(chatEndpointResources.type, "repository"), + options!.allGitHubRepositoriesEnabled + ? eq(chatEndpointResources.availability, "available") + : undefined, + )); + } else for (const entry of selectedUpdates) + await tx + .update(chatEndpointResources) + .set({ enabled: entry.enabled, updatedAt: new Date() }) + .where( + and( + eq(chatEndpointResources.companyId, endpoint.companyId), + eq(chatEndpointResources.endpointId, endpointId), + eq(chatEndpointResources.id, entry.id), + ), + ); + if (endpoint.provider === "github") + await tx.update(chatEndpoints).set({ setup: sql`jsonb_set(${chatEndpoints.setup}, '{github}', coalesce(${chatEndpoints.setup}->'github', '{}'::jsonb) || '{"repositorySelectionSaved":true,"initialRepositoryImportPending":false}'::jsonb)`, updatedAt: new Date() }).where(eq(chatEndpoints.id, endpointId)); + if (changes.length > 0 || (endpoint.provider === "github" && !endpoint.setup.github?.repositorySelectionSaved)) + await logActivity( + tx as unknown as Db, + { + companyId: endpoint.companyId, + actorType: "user", + actorId: actorUserId ?? "board", + action: "chat_endpoint.resources_updated", + entityType: "tool_connection", + entityId: endpoint.connectionId, + details: { endpointId, provider: endpoint.provider, changes }, + }, + publications, + ); + await credentialLease.assertOwned(tx); + }); + // A committed reach change is factual even if the outer lease's last + // check subsequently fails. Never publish an uncommitted activity. + for (const publication of publications) publishActivity(publication); + }, + ); + } + + async function hasNewerProcessedProviderLifecycleEffect( + endpoint: EndpointRow, + effect: Extract, + ): Promise { + const sequence = effect.providerOrder?.sequence; + const occurredAt = effect.providerOrder?.occurredAt; + if (!sequence && !occurredAt) return false; + const providerResourceId = + endpoint.provider === "microsoft-teams" + ? baseTeamsConversationId(effect.providerResourceId) + : effect.providerResourceId; + const newerOrder = sequence + ? sql`( + ${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerOrder,sequence}' ~ '^[0-9]+([.][0-9]+)?$' + and (${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerOrder,sequence}')::numeric > cast(${sequence} as numeric) + )` + : sql`( + nullif(${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerOrder,occurredAt}', '')::timestamptz > cast(${occurredAt!} as timestamptz) + )`; + const newer = await db + .select({ id: chatDeliveries.id }) + .from(chatDeliveries) + .where( + and( + eq(chatDeliveries.endpointId, endpoint.id), + eq(chatDeliveries.state, "processed"), + inArray(chatDeliveries.eventKind, ["installation", "uninstallation"]), + sql`${chatDeliveries.normalizedEvent}#>>'{lifecycle,kind}' = 'resource'`, + sql`${chatDeliveries.normalizedEvent}#>>'{lifecycle,provider}' = ${effect.provider}`, + sql`${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerResourceId}' = ${providerResourceId}`, + newerOrder, + ), + ) + .limit(1); + return newer.length > 0; + } + + async function applyProviderAvailability(input: { + currentEndpoint: EndpointRow; + effect: ChatProviderLifecycleEffect; + credentialLease: CredentialMutationLeaseGuard; + staleResourceEffect: boolean; + claimed: Pick; + candidate: Pick; + }): Promise { + const { currentEndpoint, effect, credentialLease, staleResourceEffect, claimed, candidate } = input; + await db.transaction(async (tx) => { + await credentialLease.assertOwned(tx); + const now = new Date(); + if (staleResourceEffect) { + // Persisted provider-native ordering wins over request arrival or + // lease acquisition order. The older callback remains auditable, + // but cannot roll a resource or conversation back. + await tx + .update(chatEndpoints) + .set({ lastEventAt: now, updatedAt: now }) + .where(eq(chatEndpoints.id, currentEndpoint.id)); + await notifyChatDeliveryWork(tx); + await tx + .update(chatDeliveries) + .set({ + state: "processed", + processedAt: now, + redactedError: null, + nextAttemptAt: null, + updatedAt: now, + }) + .where(eq(chatDeliveries.id, claimed.id)); + return; + } + if (effect.kind === "resource") { + const providerResourceId = + currentEndpoint.provider === "microsoft-teams" + ? baseTeamsConversationId(effect.providerResourceId) + : effect.providerResourceId; + const previousProviderResourceId = + currentEndpoint.provider === "telegram" + ? effect.previousProviderResourceId + : undefined; + const { previousResource, migratedLabel, migratedEnabled } = await prepareTelegramResourceMigration( + tx, currentEndpoint, effect, providerResourceId, previousProviderResourceId, now, + ); + const preserveProviderLabel = + (currentEndpoint.provider === "slack" && + slackResourceLabelIsFallback( + providerResourceId, + migratedLabel, + )) || + (currentEndpoint.provider === "telegram" && + telegramResourceLabelIsFallback( + providerResourceId, + migratedLabel, + )); + const [resource] = await tx + .insert(chatEndpointResources) + .values({ + companyId: currentEndpoint.companyId, + endpointId: currentEndpoint.id, + type: effect.resourceType, + providerResourceId, + parentProviderResourceId: + effect.parentProviderResourceId ?? null, + label: migratedLabel, + providerUrl: effect.providerUrl ?? null, + availability: effect.availability, + enabled: + migratedEnabled || + (currentEndpoint.provider === "slack" && + effect.resourceType === "channel" && + effect.availability === "available"), + metadata: effect.metadata ?? {}, + }) + .onConflictDoUpdate({ + target: [ + chatEndpointResources.endpointId, + chatEndpointResources.type, + chatEndpointResources.providerResourceId, + ], + set: { + parentProviderResourceId: + effect.parentProviderResourceId ?? null, + ...(preserveProviderLabel ? {} : { label: migratedLabel }), + providerUrl: effect.providerUrl ?? null, + availability: effect.availability, + ...(previousProviderResourceId + ? { + enabled: sql`${chatEndpointResources.enabled} or ${migratedEnabled}`, + } + : {}), + metadata: effect.metadata ?? {}, + updatedAt: now, + }, + }) + .returning({ id: chatEndpointResources.id }); + if ( + resource && + previousResource && + previousProviderResourceId !== providerResourceId + ) { + await migrateTelegramResourceConversations(tx, currentEndpoint, previousResource, resource, previousProviderResourceId, providerResourceId, migratedLabel, now); + } + if (resource) { + await tx + .update(chatConversations) + .set({ + state: + effect.availability === "available" + ? "active" + : "unavailable", + updatedAt: now, + }) + .where( + and( + eq(chatConversations.companyId, currentEndpoint.companyId), + eq(chatConversations.endpointId, currentEndpoint.id), + eq(chatConversations.resourceId, resource.id), + effect.availability === "available" + ? eq(chatConversations.state, "unavailable") + : inArray(chatConversations.state, ["active", "waiting"]), + ), + ); + } + await tx + .update(chatEndpoints) + .set({ lastEventAt: now, updatedAt: now }) + .where(eq(chatEndpoints.id, currentEndpoint.id)); + } else { + const reason = redactSensitiveText(effect.reason).slice( + 0, + MAX_ERROR_TEXT, + ); + if (effect.availability === "available") { + if ( + currentEndpoint.status !== "paused" && + currentEndpoint.status !== "archived" + ) { + const status = + currentEndpoint.setup.step === "complete" + ? "active" + : currentEndpoint.status === "attention" + ? "verifying" + : currentEndpoint.status; + if (status !== currentEndpoint.status) await notifyChatEndpointWork(tx); + await tx + .update(chatEndpoints) + .set({ + status, + healthMessage: + status === "active" + ? "Connected" + : "Waiting for a test conversation", + lastError: null, + lastEventAt: now, + updatedAt: now, + }) + .where(eq(chatEndpoints.id, currentEndpoint.id)); + await tx + .update(toolConnections) + .set({ + status: "active", + enabled: true, + healthStatus: "healthy", + healthMessage: "Connected", + lastError: null, + healthCheckedAt: now, + updatedAt: now, + }) + .where(eq(toolConnections.id, currentEndpoint.connectionId)); + } + } else { + await tx + .update(chatEndpoints) + .set({ + status: effect.availability, + healthMessage: reason, + lastError: reason, + lastEventAt: now, + setup: { + ...currentEndpoint.setup, + runtimeGeneration: + runtimeGeneration(currentEndpoint.setup) + 1, + } as InternalSetupState, + updatedAt: now, + }) + .where(eq(chatEndpoints.id, currentEndpoint.id)); + await tx + .update(toolConnections) + .set({ + status: "disabled", + enabled: false, + healthStatus: + effect.availability === "revoked" ? "failed" : "degraded", + healthMessage: reason, + lastError: reason, + healthCheckedAt: now, + updatedAt: now, + }) + .where(eq(toolConnections.id, currentEndpoint.connectionId)); + await tx + .update(chatEndpointResources) + .set({ availability: "unavailable", updatedAt: now }) + .where( + and( + eq( + chatEndpointResources.companyId, + currentEndpoint.companyId, + ), + eq(chatEndpointResources.endpointId, currentEndpoint.id), + ), + ); + await tx + .update(chatConversations) + .set({ state: "unavailable", updatedAt: now }) + .where( + and( + eq(chatConversations.companyId, currentEndpoint.companyId), + eq(chatConversations.endpointId, currentEndpoint.id), + inArray(chatConversations.state, ["active", "waiting"]), + ), + ); + } + } + await notifyChatDeliveryWork(tx); + await tx + .update(chatDeliveries) + .set({ + state: "processed", + processedAt: now, + redactedError: null, + nextAttemptAt: null, + updatedAt: now, + }) + .where( + and( + eq(chatDeliveries.id, claimed.id), + eq(chatDeliveries.state, "processing"), + eq(chatDeliveries.attempts, candidate.attempts + 1), + ), + ); + await credentialLease.assertOwned(tx); + }); + } + + const { reconcileGitHubWebhookRepository, listGitHubRepositories, toggleAllGitHubRepositories } = createGitHubResources(db, { endpointRecord, upsertProviderResourceRow, updateResourceSelection }); + + return { reconcileGitHubWebhookRepository, reconcileProviderResourceRows, listResources, listGitHubRepositories, replaceResources, toggleAllGitHubRepositories, hasNewerProcessedProviderLifecycleEffect, applyProviderAvailability }; +} diff --git a/server/src/services/chat-channels/slack/resources.ts b/server/src/services/chat-channels/slack/resources.ts new file mode 100644 index 00000000000..1d7773f750a --- /dev/null +++ b/server/src/services/chat-channels/slack/resources.ts @@ -0,0 +1,10 @@ +export function slackResourceLabelIsFallback( + providerResourceId: string, + label: string, +): boolean { + const candidate = label.trim(); + return ( + candidate === providerResourceId || + candidate === `slack:${providerResourceId}` + ); +} diff --git a/server/src/services/chat-channels/telegram/resources.ts b/server/src/services/chat-channels/telegram/resources.ts new file mode 100644 index 00000000000..17bc3b39b52 --- /dev/null +++ b/server/src/services/chat-channels/telegram/resources.ts @@ -0,0 +1,104 @@ +import { and, eq, sql } from "drizzle-orm"; +import { chatConversations, chatEndpointResources, chatEndpoints, type Db } from "@paperclipai/db"; +import type { ChatProviderLifecycleEffect } from "../../chat-provider-lifecycle.js"; +type DbTransaction = Parameters[0]>[0]; +type EndpointRow = typeof chatEndpoints.$inferSelect; + +export function telegramResourceLabelIsFallback( + providerResourceId: string, + label: string, +): boolean { + const candidate = label.trim(); + return ( + candidate === providerResourceId || + candidate === `telegram:${providerResourceId}` + ); +} + +export async function prepareTelegramResourceMigration( + tx: DbTransaction, currentEndpoint: EndpointRow, + effect: Extract, + providerResourceId: string, previousProviderResourceId: string | undefined, now: Date, +) { + const previousResource = previousProviderResourceId + ? await tx + .select() + .from(chatEndpointResources) + .where( + and( + eq(chatEndpointResources.endpointId, currentEndpoint.id), + eq(chatEndpointResources.type, "chat"), + eq( + chatEndpointResources.providerResourceId, + previousProviderResourceId, + ), + ), + ) + .for("update") + .then((rows) => rows[0] ?? null) + : null; + const migratedLabel = + previousResource && + telegramResourceLabelIsFallback(providerResourceId, effect.label) + ? previousResource.label + : effect.label; + const migratedEnabled = previousResource?.enabled === true; + if ( + previousResource && + previousProviderResourceId !== providerResourceId + ) { + await tx + .update(chatEndpointResources) + .set({ + availability: "unavailable", + enabled: false, + metadata: { + ...previousResource.metadata, + source: "chat_migration", + migratedTo: providerResourceId, + }, + updatedAt: now, + }) + .where(eq(chatEndpointResources.id, previousResource.id)); + } + return { previousResource, migratedLabel, migratedEnabled }; +} + +export async function migrateTelegramResourceConversations( + tx: DbTransaction, currentEndpoint: EndpointRow, + previousResource: typeof chatEndpointResources.$inferSelect, + resource: { id: string }, previousProviderResourceId: string | undefined, + providerResourceId: string, migratedLabel: string, now: Date, +) { + const previousThreadPrefix = `telegram:${previousProviderResourceId}`; + const migratedThreadPrefix = `telegram:${providerResourceId}`; + await tx + .update(chatConversations) + .set({ + resourceId: resource.id, + // Telegram SDK threads use the namespaced channel id. Keep + // the migrated row in that canonical shape so the first + // reply from the replacement supergroup finds this same + // task instead of creating a second conversation. + externalConversationId: migratedThreadPrefix, + externalThreadId: sql`case + when ${chatConversations.externalThreadId} = ${previousThreadPrefix} then ${migratedThreadPrefix} + when ${chatConversations.externalThreadId} like ${`${previousThreadPrefix}:%`} + then ${migratedThreadPrefix} || substring( + ${chatConversations.externalThreadId} + from char_length(${previousThreadPrefix}) + 1 + ) + else ${chatConversations.externalThreadId} + end`, + externalLabel: migratedLabel, + state: "active", + updatedAt: now, + }) + .where( + and( + eq(chatConversations.companyId, currentEndpoint.companyId), + eq(chatConversations.endpointId, currentEndpoint.id), + eq(chatConversations.resourceId, previousResource.id), + ), + ); +} From 848e659bc666b18fc05c990c0de8758dce901d67 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 12 Oct 2026 09:40:09 -0500 Subject: [PATCH 2/7] refactor(heartbeat): extract session preparation (#15966) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat execution prepares a provider session before dispatch. > - The same executor also owns workspace setup, admission, cleanup, and recovery. > - Session freshness and compaction form a separate responsibility with explicit inputs and results. > - This pull request moves those two session phases into one module and keeps their order around workspace preparation. > - The benefit is a shorter executor with unchanged session behavior. ## Linked Issues or Issue Description **What existing behavior does this improve?** Maintenance of heartbeat execution. This continues the extraction series after #15872. **Current behavior** `heartbeat.ts` has 4,989 lines and 206,650 bytes. Session config fingerprints, resume selection, credential compatibility, and compaction are embedded in the executor. **Proposed behavior** The executor calls `prepareHeartbeatSession` before workspace preparation and `resolveHeartbeatAdapterSession` after it. `heartbeat.ts` has 4,740 lines and 195,258 bytes. The new cohesive module has 385 lines and 19,518 bytes. **Reason and benefit** A reviewer can find session policy in one module. Runtime acquisition, admission gates, and cleanup remain in the executor. Related open work includes #15787 and #15142; this PR does not change either policy. **Breaking changes** None intended. Existing public helper exports remain available. ## What Changed - Move 267 lines into `server/src/services/heartbeat/session-preparation.ts` with typed task, config, workspace, and session inputs. - Keep config fingerprints and context mutations on the same objects. Return compaction and adapter-session results to existing consumers. - Add four real-PostgreSQL characterization tests through heartbeat wakeup and adapter dispatch. - Document the two session preparation phases. ## Verification - Before extraction: four new characterization tests pass; 180 existing workspace/session tests pass. - After extraction: 205 tests pass across session preparation, workspace/session policy, managed-account session compatibility, and accepted-plan refresh. No skips. - `pnpm -r typecheck` and `pnpm build` pass. - Normalized syntax comparison: both moved blocks and the reconstructed executor, including public exports, match the base. - Isolated browser acceptance: 2/2 pass. A real process worker completes after automatic recovery; persistent timeouts stop after three attempts. Both terminal states survive reload. - All hosted CI gates pass on `d33f8506a7`, including the complete test shards, typecheck, build, and browser suites. The duplicate local `pnpm test:run` was interrupted after hosted CI passed; it is not claimed as a full local pass. - Greptile: 5/5 on this head, with no review threads. ## Risks - Session freshness can alter provider context if a dependency is omitted. Typed inputs and passing-before/passing-after dispatch tests cover unchanged resumes, model changes, tool refresh, and compaction. - The module acquires no runtime resource. Workspace preparation still occurs between the two phases. - Live external providers are not exercised; browser acceptance uses a real process fixture in a disposable app. ## Model Used OpenAI Codex, based on GPT-6, assisted with reasoning, code edits, tool execution, and verification. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/DEVELOPING.md | 7 + .../heartbeat-session-preparation.test.ts | 114 ++++++ server/src/services/heartbeat.ts | 301 ++------------ .../services/heartbeat/session-preparation.ts | 385 ++++++++++++++++++ 4 files changed, 532 insertions(+), 275 deletions(-) create mode 100644 server/src/__tests__/heartbeat-session-preparation.test.ts create mode 100644 server/src/services/heartbeat/session-preparation.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 79d71fb4012..7cf3e8b1471 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1293,6 +1293,13 @@ cleanup. Public-path characterization tests are in browser fixture also checks the persisted prompt variants and description revision from a real process adapter before completing its task. +Session preparation is in `server/src/services/heartbeat/session-preparation.ts`. +It selects fresh or resumable sessions from effective config and managed-account +identity, then resolves adapter session identity and compaction after workspace +preparation. Both phases keep their original ordering in `heartbeat.ts`; they +acquire no runtime resources. Public-path characterization tests live in +`server/src/__tests__/heartbeat-session-preparation.test.ts`. + Run-log formatting is in `server/src/services/heartbeat/run-log.ts`. It bounds stored event payloads, redacts and shortens log chunks, and caps stdout/stderr excerpts. `heartbeat.ts` keeps event writes, current-user redaction, and live diff --git a/server/src/__tests__/heartbeat-session-preparation.test.ts b/server/src/__tests__/heartbeat-session-preparation.test.ts new file mode 100644 index 00000000000..7460a457338 --- /dev/null +++ b/server/src/__tests__/heartbeat-session-preparation.test.ts @@ -0,0 +1,114 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, agentTaskSessions, companies, createDb, heartbeatRuns, issues } from "@paperclipai/db"; +import { registerServerAdapter, unregisterServerAdapter, type AdapterExecutionContext } from "../adapters/index.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const describePostgres = support.supported ? describe : describe.skip; +if (!support.supported) console.warn(`Skipping session preparation tests: ${support.reason}`); + +describePostgres("heartbeat session preparation through adapter dispatch", () => { + const adapterType = "session_preparation_test"; + const captured = new Map(); + let temporary: Awaited>; + let db: ReturnType; + let heartbeat: ReturnType; + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("heartbeat-session-preparation-"); + db = createDb(temporary.connectionString); + heartbeat = heartbeatService(db); + registerServerAdapter({ + type: adapterType, + supportsLocalAgentJwt: false, + execute: async (input) => { + captured.set(input.runId, input); + // Settle the fixture's work so no automatic unfinished-task retry + // competes with the next explicit session-continuation wake. + await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, String(input.context.issueId))); + return { exitCode: 0, signal: null, timedOut: false, sessionId: "saved-session", sessionParams: { sessionId: "saved-session", adapterState: "preserved" }, resultJson: {} }; + }, + testEnvironment: async () => ({ adapterType, status: "pass", checks: [], testedAt: new Date().toISOString() }), + }); + }, 30_000); + + afterAll(async () => { + await heartbeat?.drainActiveRunExecutions(); + unregisterServerAdapter(adapterType); + await db?.$client.end({ timeout: 0 }); + await temporary?.cleanup(); + }); + + async function seed(runtimeConfig: Record = {}) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Session fixture", issuePrefix: `S${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Session reader", adapterType, status: "idle", adapterConfig: { model: "model-a" }, runtimeConfig }); + await db.insert(issues).values({ id: issueId, companyId, title: "Session fixture", status: "todo", assigneeAgentId: agentId, responsibleUserId: "session-user", issueNumber: 1 }); + return { companyId, agentId, issueId }; + } + + async function execute(f: Awaited>, context: Record = {}) { + const run = await heartbeat.wakeup(f.agentId, { + source: "on_demand", triggerDetail: "manual", reason: "manual", + payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId, ...context }, + }); + expect(run).not.toBeNull(); + await heartbeat.drainActiveRunExecutions(); + const finished = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)).then(rows => rows[0]); + expect(finished?.status, finished?.error ?? undefined).toBe("succeeded"); + expect(captured.has(run!.id)).toBe(true); + return { input: captured.get(run!.id)!, finished }; + } + + it("reuses unchanged task sessions and strips host freshness metadata before dispatch", async () => { + const f = await seed(); + const first = await execute(f); + expect(first.input.runtime.sessionId).toBeNull(); + const stored = await db.select().from(agentTaskSessions).where(eq(agentTaskSessions.agentId, f.agentId)).then(rows => rows[0]); + expect(stored?.sessionParamsJson).toHaveProperty("__paperclipConfigFingerprint"); + const resumed = await execute(f, { paperclipSessionHandoffMarkdown: "stale handoff", paperclipSessionRotationReason: "stale reason", paperclipPreviousSessionId: "stale session" }); + expect(resumed.input.runtime).toMatchObject({ sessionId: "saved-session", sessionDisplayId: "saved-session", taskKey: f.issueId }); + expect(resumed.input.runtime.sessionParams).toMatchObject({ sessionId: "saved-session", adapterState: "preserved" }); + expect(resumed.input.runtime.sessionParams).not.toHaveProperty("__paperclipConfigFingerprint"); + expect(resumed.input.context.paperclipSessionHandoffMarkdown).toBeUndefined(); + expect(resumed.input.context.paperclipSessionRotationReason).toBeUndefined(); + expect(resumed.input.context.paperclipPreviousSessionId).toBeUndefined(); + expect(resumed.finished.sessionIdBefore).toBe("saved-session"); + }); + + it("starts fresh when effective model configuration changes and persists the reason", async () => { + const f = await seed(); + await execute(f); + await db.update(agents).set({ adapterConfig: { model: "model-b" } }).where(eq(agents.id, f.agentId)); + const next = await execute(f); + expect(next.input.runtime.sessionId).toBeNull(); + expect(next.input.runtime.sessionParams).toBeNull(); + expect(next.finished.sessionIdBefore).toBeNull(); + expect(next.finished.resultJson).toMatchObject({ configFreshness: { session: { reset: true, taskSessionAvailable: true, taskSessionReused: false } } }); + }); + + it("refreshes tools with a fresh session when the selected adapter cannot refresh on resume", async () => { + const f = await seed(); + await execute(f); + const next = await execute(f, { refreshTools: true }); + expect(next.input.context.forceFreshSession).toBe(true); + expect(next.input.runtime.sessionId).toBeNull(); + expect(next.finished.resultJson).toMatchObject({ configFreshness: { session: { resetReasons: ["forceFreshSession was requested"] } } }); + }); + + it("rotates a session at the configured run threshold and supplies a handoff", async () => { + const f = await seed({ sessionCompaction: { enabled: true, maxSessionRuns: 1 } }); + await execute(f); + await execute(f); + const next = await execute(f); + expect(next.input.runtime.sessionId).toBeNull(); + expect(next.input.runtime.sessionParams).toBeNull(); + expect(next.input.context.paperclipPreviousSessionId).toBe("saved-session"); + expect(next.input.context.paperclipSessionRotationReason).toBe("session exceeded 1 runs"); + expect(next.input.context.paperclipSessionHandoffMarkdown).toEqual(expect.any(String)); + expect(next.finished.sessionIdBefore).toBeNull(); + }); +}); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index bcbcdb0a648..03274998d80 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { prepareHeartbeatSession, resolveHeartbeatAdapterSession } from "./heartbeat/session-preparation.js"; import { prepareHeartbeatTaskContext } from "./heartbeat/task-context.js"; import { selectHeartbeatEnvironment } from "./heartbeat/environment-selection.js"; import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js"; @@ -80,9 +81,6 @@ import { normalizeResumeParamsForAdapter, truncateDisplayId, resolveLedgerScopeForRun, - describeSessionResetReason, - isCanonicalSessionIdForAdapter, - requiresCanonicalSessionIds, resolveNextSessionState, resolveCacheAdjustedCostUsd, resolveLedgerCostStatus, @@ -139,16 +137,10 @@ export { import { buildExecutionWorkspaceConfigSnapshot, stripWorkspaceRuntimeFromExecutionRunConfig, - buildEffectiveRunSessionConfigMetadata, - readConfigFingerprintFromSessionParams, - readConfiguredModelFromAdapterConfig, - resolveTaskSessionConfigFreshness, - stripPaperclipSessionMetadataFromSessionParams, resolveRuntimeSessionParamsForWorkspace, buildRunWorkspaceHints, buildReferencedProjectRunObservability, assertGitSensitiveAdapterWorkspaceValid, - type EffectiveRunSessionConfigMetadata, createHeartbeatWorkspaceResolver, } from "./heartbeat/workspaces.js"; export { @@ -275,7 +267,7 @@ import { } from "./ai-connection-runtime.js"; import { aiConnectionBindingSchema, aiRuntimeConnectionBindingSchema, type AiConnectionRouterSelection } from "@paperclipai/shared"; import { aiConnectionRouterService, AiConnectionPoolExhausted, applyAiConnectionRouterTaskSettings } from "./ai-connection-router.js"; -import { aiConnectionSessionCompatibilityInputs, managedAiSessionIdentityCompatible } from "./ai-connection-session.js"; +import { managedAiSessionIdentityCompatible } from "./ai-connection-session.js"; import { getExecutionBlocker, } from "./execution-blocker.js"; @@ -319,7 +311,6 @@ import { } from "node:crypto"; import { and, - desc, eq, exists, inArray, @@ -336,7 +327,6 @@ import { } from "@paperclipai/shared"; import { agents, - agentConfigRevisions, agentWakeupRequests, chatActions, heartbeatRuns, @@ -384,7 +374,6 @@ import { type NativeExecutionInput, type NativeSessionBackend, } from "../vendor/paperclip-runner/index.js"; -import { createNativeSessionHandoffLoader } from "./native-runtime/native-session-handoff.js"; import { getRunLogStore, type RunLogHandle } from "./run-log-store.js"; import { @@ -2910,143 +2899,22 @@ export function heartbeatService( ...identityRedactor.values, ...collectRunFailureSecretValues(runtimeConfig.env, secretKeys), ]; - const latestAgentConfigRevision = await getLatestAgentConfigRevision( - agent.companyId, - agent.id, - ); - const sessionConfigMetadataInput = { - agentIdentityKeyId: agentIdentity?.keyId, - adapterType: agent.adapterType, - effectiveAdapterConfig: runtimeConfig, - managedAiHome: managedAiRuntime?.home, - agentRuntimeConfig: agent.runtimeConfig, - issueOverrides: issueAssigneeOverrides, - workspaceConfig: { - requestedMode: requestedExecutionWorkspaceMode, - effectiveMode: effectiveExecutionWorkspaceMode, - issueConfigRevisionAt: - issueContext?.updatedAt instanceof Date - ? issueContext.updatedAt.toISOString() - : (issueContext?.updatedAt ?? null), - projectConfigRevisionAt: - projectContext?.updatedAt instanceof Date - ? projectContext.updatedAt.toISOString() - : (projectContext?.updatedAt ?? null), - projectPolicy: projectExecutionWorkspacePolicy, - issueSettings: issueExecutionWorkspaceSettings, - reusableExecutionWorkspaceConfig: - requestedReusableExecutionWorkspaceConfig, - existingExecutionWorkspace: reusableExistingExecutionWorkspace - ? { - id: reusableExistingExecutionWorkspace.id, - mode: reusableExistingExecutionWorkspace.mode, - strategyType: reusableExistingExecutionWorkspace.strategyType, - projectWorkspaceId: - reusableExistingExecutionWorkspace.projectWorkspaceId, - repoUrl: reusableExistingExecutionWorkspace.repoUrl, - baseRef: reusableExistingExecutionWorkspace.baseRef, - branchName: reusableExistingExecutionWorkspace.branchName, - config: reusableExistingExecutionWorkspace.config, - } - : null, - }, - environment: { - selectionSource: environmentResolution.source, - selectedEnvironmentId, - selectedEnvironment: selectedEnvironmentForConfig - ? { - id: selectedEnvironmentForConfig.id, - driver: selectedEnvironmentForConfig.driver, - config: selectedEnvironmentForConfig.config, - configRevisionAt: - selectedEnvironmentForConfig.updatedAt instanceof Date - ? selectedEnvironmentForConfig.updatedAt.toISOString() - : (selectedEnvironmentForConfig.updatedAt ?? null), - } - : null, - executionPolicy, - }, - environmentEnv: selectedEnvironmentForConfig?.envVars ?? null, - projectEnv: projectContext?.env ?? null, - routineEnv: routineEnvContext.env, - secretManifest, - runtimeSkills: runtimeSkillEntries, - agentConfigRevision: latestAgentConfigRevision - ? { - id: latestAgentConfigRevision.id, - changedKeys: latestAgentConfigRevision.changedKeys, - configRevisionAt: - latestAgentConfigRevision.createdAt.toISOString(), - } - : null, - }; - const sessionConfigMetadata = await buildEffectiveRunSessionConfigMetadata(sessionConfigMetadataInput); - let compatibleConfigMetadata: EffectiveRunSessionConfigMetadata[] = []; - if (managedAiRuntime && aiBinding && taskSession && - readConfigFingerprintFromSessionParams(taskSession.sessionParamsJson)?.fingerprint !== sessionConfigMetadata.fingerprint) { - const revisions = requestedAiBinding?.mode === "router" - ? await db.select().from(agentConfigRevisions).where(and(eq(agentConfigRevisions.companyId, agent.companyId), eq(agentConfigRevisions.agentId, agent.id))).orderBy(desc(agentConfigRevisions.createdAt), desc(agentConfigRevisions.id)).limit(21) - : []; - const candidates = aiConnectionSessionCompatibilityInputs({ - effectiveAdapterConfig: runtimeConfig, agentRuntimeConfig: agent.runtimeConfig, - agentConfigRevision: sessionConfigMetadataInput.agentConfigRevision, - issueOverrides: issueAssigneeOverrides, originalIssueOverrides: originalAiIssueOverrides, - binding: aiBinding, router: requestedAiBinding?.mode === "router", - storedIdentity: taskSession.sessionParamsJson?.paperclipAiCredentialIdentity, - sessionIdentity: managedAiRuntime.sessionIdentity, credentialIdentity: managedAiRuntime.identity, revisions, - }); - compatibleConfigMetadata = await Promise.all(candidates.map(candidate => buildEffectiveRunSessionConfigMetadata({ ...sessionConfigMetadataInput, ...candidate }))); - } - const configuredModel = - readConfiguredModelFromAdapterConfig(runtimeConfig); - if (context.refreshTools === true && agent.adapterType !== "paperclip_runner") { - const capability = getServerAdapter(agent.adapterType).supportsToolRefreshOnResume; - const canRefresh = typeof capability === "function" ? capability(runtimeConfig) : capability === true; - if (!canRefresh) context.forceFreshSession = true; - } - const wakeSessionResetReason = describeSessionResetReason(context); - const sessionConfigFreshness = resolveTaskSessionConfigFreshness({ - hasTaskSession: taskSession != null, - configuredModel, - taskSessionParams: - taskSession?.sessionParamsJson ?? taskSessionDecodedParams, - configMetadata: sessionConfigMetadata, - compatibleConfigMetadata, - wakeResetReason: wakeSessionResetReason, - preserveLegacySessionWithoutConfigMetadata: - acceptedPlanContinuationWake && !acceptedPlanWakeRoutingDecision && !agentIdentity, + const preparedSession = await prepareHeartbeatSession(db, { + run, agent, + task: { context, issueRef, issueContext, projectContext, wakeCommentId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision }, + session: { taskSession, taskSessionDecodedParams, explicitResumeSessionParams, explicitResumeSessionDisplayId, sessionCodec }, + config: { + runtimeConfig, agentIdentity, issueAssigneeOverrides, originalAiIssueOverrides, + managedAiRuntime, aiBinding, requestedAiBinding, secretManifest, runtimeSkillEntries, routineEnvContext, + }, + workspace: { + requestedExecutionWorkspaceMode, effectiveExecutionWorkspaceMode, projectExecutionWorkspacePolicy, + issueExecutionWorkspaceSettings, executionPolicy, requestedReusableExecutionWorkspaceConfig, reusableExistingExecutionWorkspace, + }, + environment: { environmentResolution, selectedEnvironmentId, selectedEnvironmentForConfig }, + getLatestAgentConfigRevision, }); - const resetTaskSession = - shouldResetTaskSessionForWake(context) || sessionConfigFreshness.reset; - const sessionResetReason = - sessionConfigFreshness.reasons.join("; ") || null; - const taskSessionForRun = resetTaskSession ? null : taskSession; - const getFreshSessionHandoff = issueRef ? createNativeSessionHandoffLoader({ - db, companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id, before: run.createdAt, - throughCommentId: readNonEmptyString(context.conversationReplayThroughCommentId) ?? (context.interactionKind ? null : wakeCommentId), - }) : undefined; - const previousSessionParams = - explicitResumeSessionParams ?? - (isCanonicalSessionIdForAdapter( - agent.adapterType, - explicitResumeSessionDisplayId, - ) - ? { sessionId: explicitResumeSessionDisplayId } - : null) ?? - normalizeResumeParamsForAdapter( - agent.adapterType, - stripPaperclipSessionMetadataFromSessionParams( - sessionCodec.deserialize( - taskSessionForRun?.sessionParamsJson ?? null, - ), - ), - ); - // Legacy plugins can consume the existing context field on a known-fresh - // dispatch. Built-ins also load lazily if their resume attempt fails. - if (agent.adapterType !== "paperclip_runner" && !previousSessionParams && getFreshSessionHandoff) { - const handoff = await getFreshSessionHandoff(); - if (handoff) context.paperclipFreshSessionHandoffMarkdown = handoff; - } + const { sessionConfigMetadata, configuredModel, resetTaskSession, sessionResetReason, taskSessionForRun, getFreshSessionHandoff, previousSessionParams } = preparedSession; const { resolvedWorkspace, hostExecutionWorkspaceConfig, @@ -3500,135 +3368,18 @@ export function heartbeatService( ) { context.projectId = executionWorkspace.projectId; } - const runtimeSessionFallback = - taskKey || resetTaskSession - ? null - : isCanonicalSessionIdForAdapter(agent.adapterType, runtime.sessionId) - ? runtime.sessionId - : null; - const runtimeSessionDisplayId = truncateDisplayId( - explicitResumeSessionDisplayId ?? - taskSessionForRun?.sessionDisplayId ?? - (sessionCodec.getDisplayId - ? sessionCodec.getDisplayId(runtimeSessionParams) - : null) ?? - readNonEmptyString(runtimeSessionParams?.sessionId) ?? - runtimeSessionFallback, - ); - let previousSessionDisplayId = requiresCanonicalSessionIds( - agent.adapterType, - ) - ? truncateDisplayId( - readNonEmptyString(previousSessionParams?.sessionId) ?? - (isCanonicalSessionIdForAdapter( - agent.adapterType, - runtimeSessionDisplayId, - ) - ? runtimeSessionDisplayId - : null) ?? - runtimeSessionFallback, - ) - : runtimeSessionDisplayId; - let runtimeSessionIdForAdapter = - readNonEmptyString(runtimeSessionParams?.sessionId) ?? - runtimeSessionFallback; - let runtimeSessionParamsForAdapter = normalizeSessionParams( - stripPaperclipSessionMetadataFromSessionParams(runtimeSessionParams), - ); - - const sessionCompaction = await evaluateSessionCompaction({ + const { + previousSessionDisplayId, runtimeSessionParamsForAdapter, taskSessionCredentialCompatible, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction, + } = await resolveHeartbeatAdapterSession({ agent, - sessionId: previousSessionDisplayId ?? runtimeSessionIdForAdapter, - issueId, - continuationSummaryBody: continuationSummary?.body ?? null, - }); - if (sessionCompaction.rotate) { - context.paperclipSessionHandoffMarkdown = - sessionCompaction.handoffMarkdown; - context.paperclipSessionRotationReason = sessionCompaction.reason; - context.paperclipPreviousSessionId = - previousSessionDisplayId ?? runtimeSessionIdForAdapter; - runtimeSessionIdForAdapter = null; - runtimeSessionParamsForAdapter = null; - previousSessionDisplayId = null; - if (sessionCompaction.reason) { - runtimeWorkspaceWarnings.push( - `Starting a fresh session because ${sessionCompaction.reason}.`, - ); - } - } else { - delete context.paperclipSessionHandoffMarkdown; - delete context.paperclipSessionRotationReason; - delete context.paperclipPreviousSessionId; - } - - const taskSessionCredentialCompatible = !managedAiRuntime || managedAiSessionIdentityCompatible( - taskSession?.sessionParamsJson?.paperclipAiCredentialIdentity, - managedAiRuntime.sessionIdentity, - managedAiRuntime.identity, - ); - if (managedAiRuntime) { - sessionConfigMetadata.aiCredentialIdentity = managedAiRuntime.sessionIdentity; - if (!taskSessionCredentialCompatible) { - runtimeSessionIdForAdapter = null; - runtimeSessionParamsForAdapter = null; - previousSessionDisplayId = null; - delete executionContinuation?.resumeDelta; - } - } - const runtimeForAdapter = { - sessionId: runtimeSessionIdForAdapter, - sessionParams: runtimeSessionParamsForAdapter, - sessionDisplayId: previousSessionDisplayId, - taskKey, - }; - // A delta is safe only when the selected provider session is exactly the - // task session whose last dispatch supplied the baseline history. - if ( - executionContinuation?.resumeDelta && - (!taskSessionForRun || - !taskSession?.sessionDisplayId || - runtimeForAdapter.sessionDisplayId !== taskSession.sessionDisplayId || - resetTaskSession || - context.forceFreshSession === true) - ) { - delete executionContinuation.resumeDelta; - } - const configFreshnessResultMetadata = { - version: sessionConfigMetadata.version, - session: { - fingerprintVersion: sessionConfigMetadata.version, - categories: sessionConfigMetadata.categories, - reset: resetTaskSession, - resetReasons: sessionConfigFreshness.reasons, - changedCategories: sessionConfigFreshness.changedCategories, - taskSessionAvailable: taskSession != null, - taskSessionReused: taskSessionForRun != null, - storedFingerprintPresent: Boolean( - sessionConfigFreshness.storedFingerprint, - ), - nextFingerprint: sessionConfigFreshness.nextFingerprint, - }, + task: { context, taskKey, issueId, continuationSummary, executionContinuation }, + session: { runtime, taskSession, sessionCodec, explicitResumeSessionDisplayId, runtimeSessionParams, managedAiRuntime, prepared: preparedSession }, workspace: { - fingerprintVersion: latestWorkspaceConfigMetadata.version, - categories: latestWorkspaceConfigMetadata.categories, - action: workspaceConfigFreshness.action, - changedCategories: workspaceConfigFreshness.changedCategories, - reasons: workspaceConfigFreshness.reasons, - reuseRequested: requestedShouldReuseExisting, - workspaceReused: Boolean(reusedExecutionWorkspace), - configSnapshotRefreshed: - resolvedWorkspaceReusePolicy.shouldRefreshWorkspaceConfigSnapshot, - storedFingerprintPresent: - workspaceConfigFreshness.storedFingerprintPresent, - storedFingerprint: workspaceConfigFreshness.storedFingerprint, - inferredFingerprint: workspaceConfigFreshness.inferredFingerprint, - nextFingerprint: workspaceConfigFreshness.nextFingerprint, - previousWorkspaceId: - workspaceReuseRequest.requestedExecutionWorkspaceId, - activeWorkspaceId: persistedExecutionWorkspace?.id ?? null, + latestWorkspaceConfigMetadata, workspaceConfigFreshness, reusedExecutionWorkspace, resolvedWorkspaceReusePolicy, + persistedExecutionWorkspace, runtimeWorkspaceWarnings, requestedShouldReuseExisting, workspaceReuseRequest, }, - }; + evaluateSessionCompaction, + }); let handle: RunLogHandle | null = null; const goalCheckpointSession: { diff --git a/server/src/services/heartbeat/session-preparation.ts b/server/src/services/heartbeat/session-preparation.ts new file mode 100644 index 00000000000..a7845726dc0 --- /dev/null +++ b/server/src/services/heartbeat/session-preparation.ts @@ -0,0 +1,385 @@ +import { and, desc, eq } from "drizzle-orm"; +import { agentConfigRevisions, type Db, type agents, type agentTaskSessions, type heartbeatRuns, type issues, type projects } from "@paperclipai/db"; +import type { AiConnectionBinding, AiRuntimeConnectionBinding } from "@paperclipai/shared"; +import { getServerAdapter } from "../../adapters/index.js"; +import type { prepareManagedAiRuntime } from "../ai-connection-runtime.js"; +import { aiConnectionSessionCompatibilityInputs, managedAiSessionIdentityCompatible } from "../ai-connection-session.js"; +import { createNativeSessionHandoffLoader } from "../native-runtime/native-session-handoff.js"; +import { + describeSessionResetReason, shouldResetTaskSessionForWake, isCanonicalSessionIdForAdapter, + normalizeResumeParamsForAdapter, normalizeSessionParams, requiresCanonicalSessionIds, + truncateDisplayId, type createHeartbeatRunState, type getAdapterSessionCodec, +} from "./run-state.js"; +import { + buildEffectiveRunSessionConfigMetadata, readConfigFingerprintFromSessionParams, + readConfiguredModelFromAdapterConfig, resolveTaskSessionConfigFreshness, + stripPaperclipSessionMetadataFromSessionParams, type EffectiveRunSessionConfigMetadata, +} from "./workspaces.js"; +import type { HeartbeatWorkspacePreparationInput, prepareHeartbeatWorkspace } from "./workspace-preparation.js"; +import type { prepareHeartbeatTaskContext } from "./task-context.js"; + +type RunState = ReturnType; +type WorkspaceInput = HeartbeatWorkspacePreparationInput; +type PreparedWorkspace = Awaited>; +type TaskContext = Awaited>; +type SessionMetadataInput = Parameters[0]; +type ManagedAiRuntime = Awaited>; +type TaskSession = typeof agentTaskSessions.$inferSelect | null; +type SessionCodec = ReturnType; + +export interface HeartbeatSessionPreparationInput { + run: Pick; + agent: typeof agents.$inferSelect; + task: { + context: Record; + issueRef: Pick | null; + issueContext: Pick | null; + projectContext: Pick | null; + wakeCommentId: string | null; + acceptedPlanContinuationWake: boolean; + acceptedPlanWakeRoutingDecision: unknown; + }; + session: { + taskSession: TaskSession; + taskSessionDecodedParams: Record | null; + explicitResumeSessionParams: Record | null; + explicitResumeSessionDisplayId: string | null; + sessionCodec: SessionCodec; + }; + config: { + runtimeConfig: Record; + agentIdentity: { keyId: string } | undefined; + issueAssigneeOverrides: unknown; + originalAiIssueOverrides: unknown; + managedAiRuntime: ManagedAiRuntime | undefined; + aiBinding: AiConnectionBinding | undefined; + requestedAiBinding: AiRuntimeConnectionBinding | undefined; + secretManifest: SessionMetadataInput["secretManifest"]; + runtimeSkillEntries: SessionMetadataInput["runtimeSkills"]; + routineEnvContext: { env: unknown }; + }; + workspace: Pick & { + requestedReusableExecutionWorkspaceConfig: NonNullable["config"] | null; + reusableExistingExecutionWorkspace: WorkspaceInput["reuse"]["reusableExistingExecutionWorkspace"]; + }; + environment: Pick; + getLatestAgentConfigRevision: RunState["getLatestAgentConfigRevision"]; +} + +function readNonEmptyString(value: unknown): string | null { + return typeof value === "string" && value.trim().length > 0 ? value : null; +} + +/** Selects a resumable session before workspace preparation; acquires no runtime resources. */ +export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPreparationInput) { + const { run, agent, getLatestAgentConfigRevision } = input; + const { context, issueRef, issueContext, projectContext, wakeCommentId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision } = input.task; + const { taskSession, taskSessionDecodedParams, explicitResumeSessionParams, explicitResumeSessionDisplayId, sessionCodec } = input.session; + const { runtimeConfig, agentIdentity, issueAssigneeOverrides, originalAiIssueOverrides, managedAiRuntime, aiBinding, requestedAiBinding, secretManifest, runtimeSkillEntries, routineEnvContext } = input.config; + const { requestedExecutionWorkspaceMode, effectiveExecutionWorkspaceMode, projectExecutionWorkspacePolicy, issueExecutionWorkspaceSettings, executionPolicy, requestedReusableExecutionWorkspaceConfig, reusableExistingExecutionWorkspace } = input.workspace; + const { environmentResolution, selectedEnvironmentId, selectedEnvironmentForConfig } = input.environment; + const latestAgentConfigRevision = await getLatestAgentConfigRevision( + agent.companyId, + agent.id, + ); + const sessionConfigMetadataInput = { + agentIdentityKeyId: agentIdentity?.keyId, + adapterType: agent.adapterType, + effectiveAdapterConfig: runtimeConfig, + managedAiHome: managedAiRuntime?.home, + agentRuntimeConfig: agent.runtimeConfig, + issueOverrides: issueAssigneeOverrides, + workspaceConfig: { + requestedMode: requestedExecutionWorkspaceMode, + effectiveMode: effectiveExecutionWorkspaceMode, + issueConfigRevisionAt: + issueContext?.updatedAt instanceof Date + ? issueContext.updatedAt.toISOString() + : (issueContext?.updatedAt ?? null), + projectConfigRevisionAt: + projectContext?.updatedAt instanceof Date + ? projectContext.updatedAt.toISOString() + : (projectContext?.updatedAt ?? null), + projectPolicy: projectExecutionWorkspacePolicy, + issueSettings: issueExecutionWorkspaceSettings, + reusableExecutionWorkspaceConfig: + requestedReusableExecutionWorkspaceConfig, + existingExecutionWorkspace: reusableExistingExecutionWorkspace + ? { + id: reusableExistingExecutionWorkspace.id, + mode: reusableExistingExecutionWorkspace.mode, + strategyType: reusableExistingExecutionWorkspace.strategyType, + projectWorkspaceId: + reusableExistingExecutionWorkspace.projectWorkspaceId, + repoUrl: reusableExistingExecutionWorkspace.repoUrl, + baseRef: reusableExistingExecutionWorkspace.baseRef, + branchName: reusableExistingExecutionWorkspace.branchName, + config: reusableExistingExecutionWorkspace.config, + } + : null, + }, + environment: { + selectionSource: environmentResolution.source, + selectedEnvironmentId, + selectedEnvironment: selectedEnvironmentForConfig + ? { + id: selectedEnvironmentForConfig.id, + driver: selectedEnvironmentForConfig.driver, + config: selectedEnvironmentForConfig.config, + configRevisionAt: + selectedEnvironmentForConfig.updatedAt instanceof Date + ? selectedEnvironmentForConfig.updatedAt.toISOString() + : (selectedEnvironmentForConfig.updatedAt ?? null), + } + : null, + executionPolicy, + }, + environmentEnv: selectedEnvironmentForConfig?.envVars ?? null, + projectEnv: projectContext?.env ?? null, + routineEnv: routineEnvContext.env, + secretManifest, + runtimeSkills: runtimeSkillEntries, + agentConfigRevision: latestAgentConfigRevision + ? { + id: latestAgentConfigRevision.id, + changedKeys: latestAgentConfigRevision.changedKeys, + configRevisionAt: + latestAgentConfigRevision.createdAt.toISOString(), + } + : null, + }; + const sessionConfigMetadata = await buildEffectiveRunSessionConfigMetadata(sessionConfigMetadataInput); + let compatibleConfigMetadata: EffectiveRunSessionConfigMetadata[] = []; + if (managedAiRuntime && aiBinding && taskSession && + readConfigFingerprintFromSessionParams(taskSession.sessionParamsJson)?.fingerprint !== sessionConfigMetadata.fingerprint) { + const revisions = requestedAiBinding?.mode === "router" + ? await db.select().from(agentConfigRevisions).where(and(eq(agentConfigRevisions.companyId, agent.companyId), eq(agentConfigRevisions.agentId, agent.id))).orderBy(desc(agentConfigRevisions.createdAt), desc(agentConfigRevisions.id)).limit(21) + : []; + const candidates = aiConnectionSessionCompatibilityInputs({ + effectiveAdapterConfig: runtimeConfig, agentRuntimeConfig: agent.runtimeConfig, + agentConfigRevision: sessionConfigMetadataInput.agentConfigRevision, + issueOverrides: issueAssigneeOverrides, originalIssueOverrides: originalAiIssueOverrides, + binding: aiBinding, router: requestedAiBinding?.mode === "router", + storedIdentity: taskSession.sessionParamsJson?.paperclipAiCredentialIdentity, + sessionIdentity: managedAiRuntime.sessionIdentity, credentialIdentity: managedAiRuntime.identity, revisions, + }); + compatibleConfigMetadata = await Promise.all(candidates.map(candidate => buildEffectiveRunSessionConfigMetadata({ ...sessionConfigMetadataInput, ...candidate }))); + } + const configuredModel = + readConfiguredModelFromAdapterConfig(runtimeConfig); + if (context.refreshTools === true && agent.adapterType !== "paperclip_runner") { + const capability = getServerAdapter(agent.adapterType).supportsToolRefreshOnResume; + const canRefresh = typeof capability === "function" ? capability(runtimeConfig) : capability === true; + if (!canRefresh) context.forceFreshSession = true; + } + const wakeSessionResetReason = describeSessionResetReason(context); + const sessionConfigFreshness = resolveTaskSessionConfigFreshness({ + hasTaskSession: taskSession != null, + configuredModel, + taskSessionParams: + taskSession?.sessionParamsJson ?? taskSessionDecodedParams, + configMetadata: sessionConfigMetadata, + compatibleConfigMetadata, + wakeResetReason: wakeSessionResetReason, + preserveLegacySessionWithoutConfigMetadata: + acceptedPlanContinuationWake && !acceptedPlanWakeRoutingDecision && !agentIdentity, + }); + const resetTaskSession = + shouldResetTaskSessionForWake(context) || sessionConfigFreshness.reset; + const sessionResetReason = + sessionConfigFreshness.reasons.join("; ") || null; + const taskSessionForRun = resetTaskSession ? null : taskSession; + const getFreshSessionHandoff = issueRef ? createNativeSessionHandoffLoader({ + db, companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id, before: run.createdAt, + throughCommentId: readNonEmptyString(context.conversationReplayThroughCommentId) ?? (context.interactionKind ? null : wakeCommentId), + }) : undefined; + const previousSessionParams = + explicitResumeSessionParams ?? + (isCanonicalSessionIdForAdapter( + agent.adapterType, + explicitResumeSessionDisplayId, + ) + ? { sessionId: explicitResumeSessionDisplayId } + : null) ?? + normalizeResumeParamsForAdapter( + agent.adapterType, + stripPaperclipSessionMetadataFromSessionParams( + sessionCodec.deserialize( + taskSessionForRun?.sessionParamsJson ?? null, + ), + ), + ); + // Legacy plugins can consume the existing context field on a known-fresh + // dispatch. Built-ins also load lazily if their resume attempt fails. + if (agent.adapterType !== "paperclip_runner" && !previousSessionParams && getFreshSessionHandoff) { + const handoff = await getFreshSessionHandoff(); + if (handoff) context.paperclipFreshSessionHandoffMarkdown = handoff; + } + return { sessionConfigMetadata, configuredModel, sessionConfigFreshness, resetTaskSession, sessionResetReason, taskSessionForRun, getFreshSessionHandoff, previousSessionParams }; +} + +export interface HeartbeatAdapterSessionInput { + agent: typeof agents.$inferSelect; + task: { + context: Record; + taskKey: string | null; + issueId: string | null; + continuationSummary: TaskContext["continuationSummary"]; + executionContinuation: TaskContext["executionContinuation"]; + }; + session: { + runtime: { sessionId: string | null }; + taskSession: TaskSession; + sessionCodec: SessionCodec; + explicitResumeSessionDisplayId: string | null; + runtimeSessionParams: Record | null; + managedAiRuntime: ManagedAiRuntime | undefined; + prepared: Awaited>; + }; + workspace: Pick & { + runtimeWorkspaceWarnings: string[]; + requestedShouldReuseExisting: boolean; + workspaceReuseRequest: WorkspaceInput["reuse"]["workspaceReuseRequest"]; + }; + evaluateSessionCompaction: RunState["evaluateSessionCompaction"]; +} + +/** Resolves adapter identity after workspace preparation, retaining in-place context and delta updates. */ +export async function resolveHeartbeatAdapterSession(input: HeartbeatAdapterSessionInput) { + const { agent, evaluateSessionCompaction } = input; + const { context, taskKey, issueId, continuationSummary, executionContinuation } = input.task; + const { runtime, taskSession, sessionCodec, explicitResumeSessionDisplayId, runtimeSessionParams, managedAiRuntime, prepared } = input.session; + const { sessionConfigMetadata, sessionConfigFreshness, resetTaskSession, taskSessionForRun, previousSessionParams } = prepared; + const { latestWorkspaceConfigMetadata, workspaceConfigFreshness, reusedExecutionWorkspace, resolvedWorkspaceReusePolicy, persistedExecutionWorkspace, runtimeWorkspaceWarnings, requestedShouldReuseExisting, workspaceReuseRequest } = input.workspace; + const runtimeSessionFallback = + taskKey || resetTaskSession + ? null + : isCanonicalSessionIdForAdapter(agent.adapterType, runtime.sessionId) + ? runtime.sessionId + : null; + const runtimeSessionDisplayId = truncateDisplayId( + explicitResumeSessionDisplayId ?? + taskSessionForRun?.sessionDisplayId ?? + (sessionCodec.getDisplayId + ? sessionCodec.getDisplayId(runtimeSessionParams) + : null) ?? + readNonEmptyString(runtimeSessionParams?.sessionId) ?? + runtimeSessionFallback, + ); + let previousSessionDisplayId = requiresCanonicalSessionIds( + agent.adapterType, + ) + ? truncateDisplayId( + readNonEmptyString(previousSessionParams?.sessionId) ?? + (isCanonicalSessionIdForAdapter( + agent.adapterType, + runtimeSessionDisplayId, + ) + ? runtimeSessionDisplayId + : null) ?? + runtimeSessionFallback, + ) + : runtimeSessionDisplayId; + let runtimeSessionIdForAdapter = + readNonEmptyString(runtimeSessionParams?.sessionId) ?? + runtimeSessionFallback; + let runtimeSessionParamsForAdapter = normalizeSessionParams( + stripPaperclipSessionMetadataFromSessionParams(runtimeSessionParams), + ); + + const sessionCompaction = await evaluateSessionCompaction({ + agent, + sessionId: previousSessionDisplayId ?? runtimeSessionIdForAdapter, + issueId, + continuationSummaryBody: continuationSummary?.body ?? null, + }); + if (sessionCompaction.rotate) { + context.paperclipSessionHandoffMarkdown = + sessionCompaction.handoffMarkdown; + context.paperclipSessionRotationReason = sessionCompaction.reason; + context.paperclipPreviousSessionId = + previousSessionDisplayId ?? runtimeSessionIdForAdapter; + runtimeSessionIdForAdapter = null; + runtimeSessionParamsForAdapter = null; + previousSessionDisplayId = null; + if (sessionCompaction.reason) { + runtimeWorkspaceWarnings.push( + `Starting a fresh session because ${sessionCompaction.reason}.`, + ); + } + } else { + delete context.paperclipSessionHandoffMarkdown; + delete context.paperclipSessionRotationReason; + delete context.paperclipPreviousSessionId; + } + + const taskSessionCredentialCompatible = !managedAiRuntime || managedAiSessionIdentityCompatible( + taskSession?.sessionParamsJson?.paperclipAiCredentialIdentity, + managedAiRuntime.sessionIdentity, + managedAiRuntime.identity, + ); + if (managedAiRuntime) { + sessionConfigMetadata.aiCredentialIdentity = managedAiRuntime.sessionIdentity; + if (!taskSessionCredentialCompatible) { + runtimeSessionIdForAdapter = null; + runtimeSessionParamsForAdapter = null; + previousSessionDisplayId = null; + delete executionContinuation?.resumeDelta; + } + } + const runtimeForAdapter = { + sessionId: runtimeSessionIdForAdapter, + sessionParams: runtimeSessionParamsForAdapter, + sessionDisplayId: previousSessionDisplayId, + taskKey, + }; + // A delta is safe only when the selected provider session is exactly the + // task session whose last dispatch supplied the baseline history. + if ( + executionContinuation?.resumeDelta && + (!taskSessionForRun || + !taskSession?.sessionDisplayId || + runtimeForAdapter.sessionDisplayId !== taskSession.sessionDisplayId || + resetTaskSession || + context.forceFreshSession === true) + ) { + delete executionContinuation.resumeDelta; + } + const configFreshnessResultMetadata = { + version: sessionConfigMetadata.version, + session: { + fingerprintVersion: sessionConfigMetadata.version, + categories: sessionConfigMetadata.categories, + reset: resetTaskSession, + resetReasons: sessionConfigFreshness.reasons, + changedCategories: sessionConfigFreshness.changedCategories, + taskSessionAvailable: taskSession != null, + taskSessionReused: taskSessionForRun != null, + storedFingerprintPresent: Boolean( + sessionConfigFreshness.storedFingerprint, + ), + nextFingerprint: sessionConfigFreshness.nextFingerprint, + }, + workspace: { + fingerprintVersion: latestWorkspaceConfigMetadata.version, + categories: latestWorkspaceConfigMetadata.categories, + action: workspaceConfigFreshness.action, + changedCategories: workspaceConfigFreshness.changedCategories, + reasons: workspaceConfigFreshness.reasons, + reuseRequested: requestedShouldReuseExisting, + workspaceReused: Boolean(reusedExecutionWorkspace), + configSnapshotRefreshed: + resolvedWorkspaceReusePolicy.shouldRefreshWorkspaceConfigSnapshot, + storedFingerprintPresent: + workspaceConfigFreshness.storedFingerprintPresent, + storedFingerprint: workspaceConfigFreshness.storedFingerprint, + inferredFingerprint: workspaceConfigFreshness.inferredFingerprint, + nextFingerprint: workspaceConfigFreshness.nextFingerprint, + previousWorkspaceId: + workspaceReuseRequest.requestedExecutionWorkspaceId, + activeWorkspaceId: persistedExecutionWorkspace?.id ?? null, + }, + }; + + return { previousSessionDisplayId, runtimeSessionParamsForAdapter, taskSessionCredentialCompatible, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction }; +} From de79750939bac6b566fefce48a7268f29e3aea3f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 12 Oct 2026 09:49:33 -0500 Subject: [PATCH 3/7] refactor(chat): extract identity linking (#15836) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip connects provider identities to company members. > - A confirmed identity determines whose permissions apply to chat messages. > - Token consumption and membership checks must remain atomic. > - These operations currently share the large chat channel service. > - This PR moves identity linking into one module with explicit runtime callbacks. > - Reviewers can inspect the complete identity-link flow together. ## Linked Issues or Issue Description Refs #15834 and #15823. Stack slice 3 of 5, based on `codex/chat-resource-reconciliation`. Merge the predecessor first, then retarget this PR. Do not merge the stack out of order. Stack order: #15823 (provider messages) → #15834 (resource reconciliation) → #15836 (identity links) → #15844 (publication replacement) → #15845 (failed-run retries). GitHub native stack merging preserves this order and automatically restacks the remaining branches. ## What Changed - Extract company-scoped identity linking into shared `identity-links.ts`. - Move Slack connect receipt queries, access requests and confirmation-notice staging into `slack/identity-links.ts`. - Move GitHub identity fields into `github/identity-links.ts`. - Preserve shared token consumption, link locks, membership checks and revocation. - Stage the Slack notice inside the original confirmation transaction. Schedule delivery after commit. Keep runtime and public URL callbacks live. - Restack on the revised resource PR #15834 and record the architectural reassessment in the plan. ## Verification - Four existing service integration cases passed before and after. The Slack connect case covers discovery, preview, access requests, duplicate requests, member confirmation, provider notice, consumed tokens and endpoint isolation. - Receipt queries, the access-request transaction and notice payload match the previous implementation with explicit parameter substitutions. - Server typecheck, module-boundary and whitespace checks passed. - These are local database and fixture transport tests, not live Slack execution. - Fresh Greptile review is 5/5 on `7b44962f09fe8043c77ed40e05f761f31c64b33c`, with a successful check and no open review threads. - Final-stack `pnpm -r typecheck` and `pnpm build` passed at `57ec5ba94aecd860471f9b14db46f99120c7c790`. - All CI checks passed on `62ee045a4728da2d756be3d2e40d13e592a0b5a9`. - The [original workspace shard](https://github.com/paperclipai/paperclip/actions/runs/38256203686/job/114834873234) timed out waiting for the CLI import/export test server health endpoint. Its [single unchanged rerun passed](https://github.com/paperclipai/paperclip/actions/runs/38256203686/job/114838954081). The original timeout remains part of the evidence. - The final full-local-suite outcome for this cumulative stack is recorded in #15845; do not infer that gate from the focused tests above. - Merge verification on the restacked tree: 118 focused identity, publication, Photon and retry integration tests passed; server typecheck and module-boundary checks passed. GitHub’s rebased trees exactly match the previously reviewed file contents. - All current-head required checks passed on `7b44962f09fe8043c77ed40e05f761f31c64b33c`: full CI; Greptile 5/5 with no unresolved threads. ## Risks - Confirmation must retain transaction and token ownership. The Slack helper receives the existing transaction and stages the same effect; scheduling remains after commit in the shared coordinator. - This PR depends on #15834. Merge predecessors and retarget before merging this PR. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and repository tools. The exact runtime model ID and context-window size are not exposed in this session. No separate agent was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../2026-10-10-chat-channels-refactor.md | 31 ++ server/src/services/chat-channels.ts | 435 +----------------- .../chat-channels/github/identity-links.ts | 7 + .../services/chat-channels/identity-links.ts | 397 ++++++++++++++++ .../chat-channels/slack/identity-links.ts | 113 +++++ 5 files changed, 553 insertions(+), 430 deletions(-) create mode 100644 server/src/services/chat-channels/github/identity-links.ts create mode 100644 server/src/services/chat-channels/identity-links.ts create mode 100644 server/src/services/chat-channels/slack/identity-links.ts diff --git a/doc/plans/2026-10-10-chat-channels-refactor.md b/doc/plans/2026-10-10-chat-channels-refactor.md index 2b6765a72ea..ce51837f239 100644 --- a/doc/plans/2026-10-10-chat-channels-refactor.md +++ b/doc/plans/2026-10-10-chat-channels-refactor.md @@ -172,3 +172,34 @@ master, then retain the remaining four PRs for review in dependency order. - Existing public-service tests: 15 resource cases and the Telegram migration case passed before; the same 16 cases passed after. The name filter excludes unrelated cases. No live third-party provider calls were made. + +### Slice 3: identity linking + +Extract six identity-link operations into `identity-links.ts`. Keep the live +public URL getter. Fetch confirmation runtime authority through a callback that +reads the existing service map after awaiting runtime creation. Keep the same +transaction locks, token consumption, membership checks and post-commit notice. + +- Source: 37,720 lines / 1,454,762 bytes → 37,295 lines / 1,436,867 bytes. +- Destination: 459 lines / 19,970 bytes. +- Baseline: complete CI and 5/5 review on slice 2 (`c01f34ad2b`, PR #15834). +- Server typecheck, module boundaries and mechanical comparison pass. The only + body substitution moves the runtime lookup behind the supplied live callback. + All remaining statements are unchanged. Local PostgreSQL remains unavailable; + record this slice's complete CI and review in its PR before continuing. + +### 2026-10-12 channel-folder reassessment: slice 3 + +Keep identity-token consumption, principal scope, membership checks, link locks +and revocation in shared `identity-links.ts`. Move Slack receipt queries, +self-service access requests and confirmation-notice staging into +`slack/identity-links.ts`. GitHub identity fields live in `github/identity-links.ts`. +The shared confirmation transaction still stages the Slack notice before commit +and schedules it only after commit. The live runtime and public URL callbacks +remain live. The Slack factory holds no cache or worker state. + +Four existing public-service cases passed before and after, including the full +Slack connect journey: discovery, preview, outsider access request, duplicate +request, member confirmation, provider notice, consumed-token rejection and +endpoint isolation. Receipt queries, access-request transaction and notice +payload retain their statements with explicit parameter substitutions. diff --git a/server/src/services/chat-channels.ts b/server/src/services/chat-channels.ts index a0b5c93324c..5930982dd0b 100644 --- a/server/src/services/chat-channels.ts +++ b/server/src/services/chat-channels.ts @@ -1,3 +1,4 @@ +import { createChatIdentityLinks } from "./chat-channels/identity-links.js"; import { createChatResourceReconciliation } from "./chat-channels/resource-reconciliation.js"; import { githubRepositoryInventoryItemFromPayload } from "./chat-channels/github/resources.js"; import { baseTeamsConversationId } from "./chat-channels/microsoft-teams/resources.js"; @@ -161,7 +162,6 @@ import { agentWakeupRequests, agents, assets, - authUsers, chatActions, chatConversations, chatDeliveries, @@ -174,7 +174,6 @@ import { chatPublications, chatSdkState, chatTeamsFileTransfers, - companies, companyMemberships, companySecretBindings, heartbeatRunEvents, @@ -187,8 +186,6 @@ import { issueThreadInteractions, issueQuestionResponseDeliveries, issues, - invites, - joinRequests, toolApplications, toolConnections, } from "@paperclipai/db"; @@ -2406,6 +2403,10 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { // Resolve its live identity when producing URLs, not once at service startup. // An explicit webhook ingress remains separate from board/identity links. const getPublicBaseUrl = () => runtimeCanonicalOrigin() ?? configuredPublicBaseUrl; + const { listPrincipals, createLinkIntent, previewIdentityLink, requestIdentityAccess, confirmIdentityLink, revokeLink } = createChatIdentityLinks(db, { + endpointRecord, getPublicBaseUrl, stageProviderEffect, scheduleProviderEffect, + confirmationRuntimeFor: async (endpoint) => runtimeContexts.get((await runtimeFor(endpoint)) as object), + }); // Task links must validate the original configured URL before normalization // can remove credentials or other evidence that makes it unsafe to publish. const getTaskBaseUrl = () => runtimeCanonicalOrigin() ?? options.publicBaseUrl; @@ -27021,432 +27022,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { return reactionCount + rows.length; } - async function listPrincipals(endpointId: string) { - const record = await endpointRecord(endpointId); - if (!record) throw notFound("Chat endpoint not found"); - const principals = await db - .select() - .from(chatExternalPrincipals) - .where( - and( - eq(chatExternalPrincipals.companyId, record.endpoint.companyId), - eq(chatExternalPrincipals.provider, record.endpoint.provider), - eq( - chatExternalPrincipals.providerAccountId, - record.endpoint.providerAccountId ?? "unknown", - ), - ), - ) - .orderBy(asc(chatExternalPrincipals.displayName)); - const links = await db - .select() - .from(chatIdentityLinks) - .where(eq(chatIdentityLinks.endpointId, endpointId)); - const connects = record.endpoint.provider === "slack" ? await db - .select({ principalId: chatActions.principalId, lastConnectAt: sql`max(${chatActions.createdAt})::text` }) - .from(chatActions) - .where(and( - eq(chatActions.companyId, record.endpoint.companyId), - eq(chatActions.endpointId, endpointId), - eq(chatActions.kind, "slack_connect"), - eq(chatActions.status, "processed"), - )) - .groupBy(chatActions.principalId) : []; - const connectByPrincipal = new Map(connects.map((row) => [row.principalId, new Date(row.lastConnectAt).toISOString()])); - const userIds = links.flatMap((link) => - link.paperclipUserId ? [link.paperclipUserId] : [], - ); - const users = userIds.length - ? await db - .select({ - id: authUsers.id, - name: authUsers.name, - email: authUsers.email, - }) - .from(authUsers) - .where(inArray(authUsers.id, userIds)) - : []; - const linkByPrincipal = new Map( - links.map((link) => [link.principalId, link]), - ); - const userById = new Map(users.map((user) => [user.id, user])); - return principals.map((principal) => { - const link = linkByPrincipal.get(principal.id); - const user = link?.paperclipUserId - ? userById.get(link.paperclipUserId) - : null; - return { - id: link?.id ?? principal.id, - principalId: principal.id, - ...(record.endpoint.provider === "github" ? { githubUserId: principal.externalId, githubLogin: principal.handle } : {}), - externalLabel: - principal.displayName ?? principal.handle ?? principal.externalId, - externalDetail: principal.handle - ? `@${principal.handle.replace(/^@/, "")}` - : principal.externalId, - paperclipUserId: link?.paperclipUserId ?? null, - paperclipUserLabel: user?.name ?? user?.email ?? null, - status: link?.status ?? "pending", - lastConnectAt: connectByPrincipal.get(principal.id) ?? null, - }; - }); - } - - async function createLinkIntent( - endpointId: string, - principalId: string, - expiresInSeconds: number, - ) { - const record = await endpointRecord(endpointId); - if (!record) throw notFound("Chat endpoint not found"); - const principal = await db - .select() - .from(chatExternalPrincipals) - .where( - and( - eq(chatExternalPrincipals.companyId, record.endpoint.companyId), - eq(chatExternalPrincipals.id, principalId), - eq(chatExternalPrincipals.provider, record.endpoint.provider), - eq( - chatExternalPrincipals.providerAccountId, - record.endpoint.providerAccountId ?? "", - ), - ), - ) - .then((rows) => rows[0] ?? null); - if (!principal) throw notFound("External identity not found"); - if (principal.isBot || principal.kind !== "user") { - throw unprocessable("Only a human external identity can be linked"); - } - const token = randomBytes(32).toString("base64url"); - const tokenHash = createHash("sha256").update(token).digest("hex"); - const expiresAt = new Date(Date.now() + expiresInSeconds * 1000); - await db.transaction(async (tx) => { - await tx.execute( - sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${record.endpoint.companyId}:${principalId}`}, 0))`, - ); - const existingLink = await tx - .select({ status: chatIdentityLinks.status }) - .from(chatIdentityLinks) - .where( - and( - eq(chatIdentityLinks.endpointId, endpointId), - eq(chatIdentityLinks.principalId, principalId), - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - if (existingLink?.status === "linked") { - throw conflict("This external identity is already linked", { - code: "chat_identity_already_linked", - }); - } - await tx - .insert(chatIdentityLinks) - .values({ - companyId: record.endpoint.companyId, - endpointId, - principalId, - status: "pending", - confirmationTokenHash: tokenHash, - expiresAt, - }) - .onConflictDoUpdate({ - target: [chatIdentityLinks.endpointId, chatIdentityLinks.principalId], - set: { - paperclipUserId: null, - status: "pending", - confirmationTokenHash: tokenHash, - expiresAt, - confirmedAt: null, - revokedAt: null, - updatedAt: new Date(), - }, - }); - }); - const path = `/chat-identity/confirm?token=${encodeURIComponent(token)}`; - return { - confirmationUrl: getPublicBaseUrl() ? `${getPublicBaseUrl()}${path}` : path, - expiresAt: expiresAt.toISOString(), - }; - } - - async function previewIdentityLink(token: string, userId?: string | null) { - const tokenHash = createHash("sha256").update(token).digest("hex"); - const row = await db - .select({ - link: chatIdentityLinks, - principal: chatExternalPrincipals, - endpoint: chatEndpoints, - companyName: companies.name, - companyPrefix: companies.issuePrefix, - }) - .from(chatIdentityLinks) - .innerJoin( - chatExternalPrincipals, - and( - eq(chatExternalPrincipals.companyId, chatIdentityLinks.companyId), - eq(chatExternalPrincipals.id, chatIdentityLinks.principalId), - ), - ) - .innerJoin( - chatEndpoints, - and( - eq(chatEndpoints.companyId, chatIdentityLinks.companyId), - eq(chatEndpoints.id, chatIdentityLinks.endpointId), - ), - ) - .innerJoin(companies, eq(companies.id, chatIdentityLinks.companyId)) - .where( - and( - eq(chatIdentityLinks.confirmationTokenHash, tokenHash), - eq(chatIdentityLinks.status, "pending"), - ), - ) - .then((rows) => rows[0] ?? null); - if (!row || !row.link.expiresAt || row.link.expiresAt <= new Date()) { - throw unprocessable("This identity-link request is invalid or expired"); - } - if (!["active", "verifying"].includes(row.endpoint.status)) throw unprocessable("This connection is not available"); - const selfService = Boolean(await db.select({ id: chatActions.id }).from(chatActions).where(and( - eq(chatActions.companyId, row.link.companyId), eq(chatActions.endpointId, row.link.endpointId), - eq(chatActions.principalId, row.link.principalId), eq(chatActions.kind, "slack_connect"), - sql`${chatActions.payload}->>'identityLinkHash' = ${tokenHash}`, - )).limit(1).then((rows) => rows[0])); - const membership = userId ? await db.select({ status: companyMemberships.status }).from(companyMemberships).where(and( - eq(companyMemberships.companyId, row.link.companyId), eq(companyMemberships.principalType, "user"), eq(companyMemberships.principalId, userId), - )).then((rows) => rows[0]) : null; - return { - selfService, - canConfirm: membership?.status === "active", - endpointId: row.endpoint.id, - companyId: row.endpoint.companyId, - companyName: row.companyName, - companyPrefix: row.companyPrefix, - provider: row.endpoint.provider, - providerAccountLabel: row.endpoint.providerAccountLabel, - botLabel: row.endpoint.botDisplayName, - externalLabel: - row.principal.displayName ?? - row.principal.handle ?? - row.principal.externalId, - externalDetail: row.principal.handle - ? `@${row.principal.handle.replace(/^@/, "")}` - : row.principal.externalId, - expiresAt: row.link.expiresAt.toISOString(), - }; - } - - async function requestIdentityAccess(token: string, userId: string, requestIp: string) { - const preview = await previewIdentityLink(token, userId); - if (!preview.selfService) throw forbidden("This identity link cannot request access"); - if (preview.canConfirm) return { status: "member" as const }; - return db.transaction(async (tx) => { - await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`chat-join:${preview.companyId}:${userId}`}, 0))`); - const tokenHash = createHash("sha256").update(token).digest("hex"); - const validLink = await tx.select({ id: chatIdentityLinks.id }).from(chatIdentityLinks).where(and( - eq(chatIdentityLinks.companyId, preview.companyId), eq(chatIdentityLinks.endpointId, preview.endpointId), - eq(chatIdentityLinks.confirmationTokenHash, tokenHash), eq(chatIdentityLinks.status, "pending"), gt(chatIdentityLinks.expiresAt, new Date()), - )).for("update").then((rows) => rows[0]); - if (!validLink) throw unprocessable("This identity-link request is invalid or expired"); - const user = await tx.select({ email: authUsers.email }).from(authUsers).where(eq(authUsers.id, userId)).then((rows) => rows[0]); - if (!user) throw forbidden("Sign in to request company access"); - const existing = await tx.select({ id: joinRequests.id }).from(joinRequests).where(and( - eq(joinRequests.companyId, preview.companyId), eq(joinRequests.requestType, "human"), eq(joinRequests.status, "pending_approval"), - or(eq(joinRequests.requestingUserId, userId), sql`lower(${joinRequests.requestEmailSnapshot}) = ${user.email.toLowerCase()}`), - )).then((rows) => rows[0]); - if (existing) return { status: "pending_approval" as const }; - const now = new Date(); - const [invite] = await tx.insert(invites).values({ - companyId: preview.companyId, tokenHash: createHash("sha256").update(randomBytes(32)).digest("hex"), - inviteType: "company_join", allowedJoinTypes: "human", acceptedAt: now, expiresAt: now, - defaultsPayload: { human: { role: "operator" }, source: "slack_identity_link" }, - }).returning({ id: invites.id }); - const [request] = await tx.insert(joinRequests).values({ - inviteId: invite.id, companyId: preview.companyId, requestType: "human", status: "pending_approval", - requestIp, requestingUserId: userId, requestEmailSnapshot: user.email, - }).returning({ id: joinRequests.id }); - await logActivity(tx as unknown as Db, { - companyId: preview.companyId, actorType: "user", actorId: userId, action: "join.requested", - entityType: "join_request", entityId: request.id, details: { requestType: "human", source: "slack_identity_link", endpointId: preview.endpointId }, - }); - return { status: "pending_approval" as const }; - }); - } - - async function confirmIdentityLink(token: string, paperclipUserId: string) { - const tokenHash = createHash("sha256").update(token).digest("hex"); - const link = await db - .select() - .from(chatIdentityLinks) - .where( - and( - eq(chatIdentityLinks.confirmationTokenHash, tokenHash), - eq(chatIdentityLinks.status, "pending"), - ), - ) - .then((rows) => rows[0] ?? null); - if (!link || !link.expiresAt || link.expiresAt <= new Date()) - throw unprocessable("This identity-link request is invalid or expired"); - const endpointRecordForLink = await endpointRecord(link.endpointId); - if (!endpointRecordForLink || !["active", "verifying"].includes(endpointRecordForLink.endpoint.status)) throw unprocessable("This connection is not available"); - const connectReceipt = endpointRecordForLink.endpoint.provider === "slack" ? await db.select({ payload: chatActions.payload }).from(chatActions).where(and( - eq(chatActions.endpointId, link.endpointId), eq(chatActions.principalId, link.principalId), eq(chatActions.kind, "slack_connect"), - )).orderBy(desc(chatActions.createdAt)).limit(1).then((rows) => rows[0]) : null; - const confirmationRuntime = connectReceipt ? runtimeContexts.get((await runtimeFor(endpointRecordForLink.endpoint)) as object) : null; - let confirmationEffectId: string | null = null; - const confirmed = await db.transaction(async (tx) => { - await tx.execute( - sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${link.companyId}:${link.principalId}`}, 0))`, - ); - const currentLink = await tx - .select({ - id: chatIdentityLinks.id, - companyId: chatIdentityLinks.companyId, - endpointId: chatIdentityLinks.endpointId, - principalId: chatIdentityLinks.principalId, - expiresAt: chatIdentityLinks.expiresAt, - }) - .from(chatIdentityLinks) - .where( - and( - eq(chatIdentityLinks.id, link.id), - eq(chatIdentityLinks.status, "pending"), - eq(chatIdentityLinks.confirmationTokenHash, tokenHash), - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - const now = new Date(); - if ( - !currentLink || - !currentLink.expiresAt || - currentLink.expiresAt <= now - ) { - return null; - } - const membership = await tx - .select({ status: companyMemberships.status }) - .from(companyMemberships) - .where( - and( - eq(companyMemberships.companyId, currentLink.companyId), - eq(companyMemberships.principalType, "user"), - eq(companyMemberships.principalId, paperclipUserId), - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - if (membership?.status !== "active") { - throw forbidden( - "The signed-in Paperclip account is not a member of this company", - ); - } - const conflictingLink = await tx - .select({ - id: chatIdentityLinks.id, - paperclipUserId: chatIdentityLinks.paperclipUserId, - }) - .from(chatIdentityLinks) - .where( - and( - eq(chatIdentityLinks.companyId, link.companyId), - eq(chatIdentityLinks.principalId, link.principalId), - eq(chatIdentityLinks.status, "linked"), - ne(chatIdentityLinks.paperclipUserId, paperclipUserId), - ), - ) - .limit(1) - .then((rows) => rows[0] ?? null); - if (conflictingLink) { - throw conflict( - "This provider identity is linked to a different Paperclip account", - { - code: "chat_identity_link_conflict", - }, - ); - } - const confirmedLink = await tx - .update(chatIdentityLinks) - .set({ - paperclipUserId, - status: "linked", - confirmationTokenHash: null, - confirmedAt: now, - updatedAt: now, - }) - .where( - and( - eq(chatIdentityLinks.id, link.id), - eq(chatIdentityLinks.status, "pending"), - eq(chatIdentityLinks.confirmationTokenHash, tokenHash), - gt(chatIdentityLinks.expiresAt, now), - ), - ) - .returning({ endpointId: chatIdentityLinks.endpointId }) - .then((rows) => rows[0] ?? null); - if (confirmedLink && connectReceipt && confirmationRuntime && - typeof connectReceipt.payload.channelId === "string" && typeof connectReceipt.payload.userId === "string") { - const effect = await stageProviderEffect(tx, { - endpoint: endpointRecordForLink.endpoint, principalId: link.principalId, - providerActionId: `provider_effect:identity_linked:${link.id}:${tokenHash}`, - payload: { version: 1, effect: "ephemeral_message", authorizationMode: "safe_notice", - threadId: connectReceipt.payload.channelId, userId: connectReceipt.payload.userId, - text: "Your Slack account is connected to Paperclip. Future messages use your Paperclip permissions.", settleDelivery: false }, - runtimeContext: confirmationRuntime, - }); - if (!effect) throw conflict("The Slack connection changed; try confirming again"); - confirmationEffectId = effect.id; - } - return confirmedLink; - }); - if (!confirmed) { - throw conflict("This identity-link request was already used or expired", { - code: "chat_identity_link_consumed", - }); - } - if (confirmationEffectId) scheduleProviderEffect(confirmationEffectId); - return { ok: true, endpointId: confirmed.endpointId }; - } - - async function revokeLink(endpointId: string, principalId: string) { - const link = await db - .select({ companyId: chatIdentityLinks.companyId }) - .from(chatIdentityLinks) - .where( - and( - eq(chatIdentityLinks.endpointId, endpointId), - eq(chatIdentityLinks.principalId, principalId), - ), - ) - .then((rows) => rows[0] ?? null); - if (!link) throw notFound("Identity link not found"); - const revoked = await db.transaction(async (tx) => { - await tx.execute( - sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${link.companyId}:${principalId}`}, 0))`, - ); - return tx - .update(chatIdentityLinks) - .set({ - paperclipUserId: null, - status: "revoked", - confirmationTokenHash: null, - revokedAt: new Date(), - updatedAt: new Date(), - }) - .where( - and( - eq(chatIdentityLinks.companyId, link.companyId), - eq(chatIdentityLinks.endpointId, endpointId), - eq(chatIdentityLinks.principalId, principalId), - ), - ) - .returning({ id: chatIdentityLinks.id }) - .then((rows) => rows[0] ?? null); - }); - if (!revoked) throw notFound("Identity link not found"); - } - async function listConversations(endpointId: string) { const record = await endpointRecord(endpointId); if (!record) throw notFound("Chat endpoint not found"); diff --git a/server/src/services/chat-channels/github/identity-links.ts b/server/src/services/chat-channels/github/identity-links.ts new file mode 100644 index 00000000000..c5a60c87e83 --- /dev/null +++ b/server/src/services/chat-channels/github/identity-links.ts @@ -0,0 +1,7 @@ +import type { chatExternalPrincipals } from "@paperclipai/db"; + +export function githubIdentityFields( + principal: Pick, +) { + return { githubUserId: principal.externalId, githubLogin: principal.handle }; +} diff --git a/server/src/services/chat-channels/identity-links.ts b/server/src/services/chat-channels/identity-links.ts new file mode 100644 index 00000000000..47596405f3a --- /dev/null +++ b/server/src/services/chat-channels/identity-links.ts @@ -0,0 +1,397 @@ +import { githubIdentityFields } from "./github/identity-links.js"; +import { createSlackIdentityLinks, type SlackIdentityEffectStager } from "./slack/identity-links.js"; +import { createHash, randomBytes } from "node:crypto"; +import { and, asc, eq, gt, inArray, ne, sql } from "drizzle-orm"; + +import type { Db } from "@paperclipai/db"; + +import { authUsers, chatEndpoints, chatExternalPrincipals, chatIdentityLinks, companies, companyMemberships } from "@paperclipai/db"; + +import { conflict, forbidden, notFound, unprocessable } from "../../errors.js"; + +type EndpointRow = typeof chatEndpoints.$inferSelect; +type ConfirmationRuntime = { credentialFingerprint: string; generation: number }; + +type ChatIdentityLinkOptions = { + endpointRecord: (endpointId: string) => Promise<{ endpoint: EndpointRow } | null>; + getPublicBaseUrl: () => string | null; + confirmationRuntimeFor: (endpoint: EndpointRow) => Promise; + stageProviderEffect: SlackIdentityEffectStager; + scheduleProviderEffect: (actionId: string) => void; +}; + +export function createChatIdentityLinks(db: Db, options: ChatIdentityLinkOptions) { + const { endpointRecord, getPublicBaseUrl, confirmationRuntimeFor, stageProviderEffect, scheduleProviderEffect } = options; + const slackIdentityLinks = createSlackIdentityLinks(db, { previewIdentityLink, stageProviderEffect }); + + async function listPrincipals(endpointId: string) { + const record = await endpointRecord(endpointId); + if (!record) throw notFound("Chat endpoint not found"); + const principals = await db + .select() + .from(chatExternalPrincipals) + .where( + and( + eq(chatExternalPrincipals.companyId, record.endpoint.companyId), + eq(chatExternalPrincipals.provider, record.endpoint.provider), + eq( + chatExternalPrincipals.providerAccountId, + record.endpoint.providerAccountId ?? "unknown", + ), + ), + ) + .orderBy(asc(chatExternalPrincipals.displayName)); + const links = await db + .select() + .from(chatIdentityLinks) + .where(eq(chatIdentityLinks.endpointId, endpointId)); + const connects = record.endpoint.provider === "slack" ? await slackIdentityLinks.listConnects(record.endpoint) : []; + const connectByPrincipal = new Map(connects.map((row) => [row.principalId, new Date(row.lastConnectAt).toISOString()])); + const userIds = links.flatMap((link) => + link.paperclipUserId ? [link.paperclipUserId] : [], + ); + const users = userIds.length + ? await db + .select({ + id: authUsers.id, + name: authUsers.name, + email: authUsers.email, + }) + .from(authUsers) + .where(inArray(authUsers.id, userIds)) + : []; + const linkByPrincipal = new Map( + links.map((link) => [link.principalId, link]), + ); + const userById = new Map(users.map((user) => [user.id, user])); + return principals.map((principal) => { + const link = linkByPrincipal.get(principal.id); + const user = link?.paperclipUserId + ? userById.get(link.paperclipUserId) + : null; + return { + id: link?.id ?? principal.id, + principalId: principal.id, + ...(record.endpoint.provider === "github" ? githubIdentityFields(principal) : {}), + externalLabel: + principal.displayName ?? principal.handle ?? principal.externalId, + externalDetail: principal.handle + ? `@${principal.handle.replace(/^@/, "")}` + : principal.externalId, + paperclipUserId: link?.paperclipUserId ?? null, + paperclipUserLabel: user?.name ?? user?.email ?? null, + status: link?.status ?? "pending", + lastConnectAt: connectByPrincipal.get(principal.id) ?? null, + }; + }); + } + + async function createLinkIntent( + endpointId: string, + principalId: string, + expiresInSeconds: number, + ) { + const record = await endpointRecord(endpointId); + if (!record) throw notFound("Chat endpoint not found"); + const principal = await db + .select() + .from(chatExternalPrincipals) + .where( + and( + eq(chatExternalPrincipals.companyId, record.endpoint.companyId), + eq(chatExternalPrincipals.id, principalId), + eq(chatExternalPrincipals.provider, record.endpoint.provider), + eq( + chatExternalPrincipals.providerAccountId, + record.endpoint.providerAccountId ?? "", + ), + ), + ) + .then((rows) => rows[0] ?? null); + if (!principal) throw notFound("External identity not found"); + if (principal.isBot || principal.kind !== "user") { + throw unprocessable("Only a human external identity can be linked"); + } + const token = randomBytes(32).toString("base64url"); + const tokenHash = createHash("sha256").update(token).digest("hex"); + const expiresAt = new Date(Date.now() + expiresInSeconds * 1000); + await db.transaction(async (tx) => { + await tx.execute( + sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${record.endpoint.companyId}:${principalId}`}, 0))`, + ); + const existingLink = await tx + .select({ status: chatIdentityLinks.status }) + .from(chatIdentityLinks) + .where( + and( + eq(chatIdentityLinks.endpointId, endpointId), + eq(chatIdentityLinks.principalId, principalId), + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + if (existingLink?.status === "linked") { + throw conflict("This external identity is already linked", { + code: "chat_identity_already_linked", + }); + } + await tx + .insert(chatIdentityLinks) + .values({ + companyId: record.endpoint.companyId, + endpointId, + principalId, + status: "pending", + confirmationTokenHash: tokenHash, + expiresAt, + }) + .onConflictDoUpdate({ + target: [chatIdentityLinks.endpointId, chatIdentityLinks.principalId], + set: { + paperclipUserId: null, + status: "pending", + confirmationTokenHash: tokenHash, + expiresAt, + confirmedAt: null, + revokedAt: null, + updatedAt: new Date(), + }, + }); + }); + const path = `/chat-identity/confirm?token=${encodeURIComponent(token)}`; + return { + confirmationUrl: getPublicBaseUrl() ? `${getPublicBaseUrl()}${path}` : path, + expiresAt: expiresAt.toISOString(), + }; + } + + async function previewIdentityLink(token: string, userId?: string | null) { + const tokenHash = createHash("sha256").update(token).digest("hex"); + const row = await db + .select({ + link: chatIdentityLinks, + principal: chatExternalPrincipals, + endpoint: chatEndpoints, + companyName: companies.name, + companyPrefix: companies.issuePrefix, + }) + .from(chatIdentityLinks) + .innerJoin( + chatExternalPrincipals, + and( + eq(chatExternalPrincipals.companyId, chatIdentityLinks.companyId), + eq(chatExternalPrincipals.id, chatIdentityLinks.principalId), + ), + ) + .innerJoin( + chatEndpoints, + and( + eq(chatEndpoints.companyId, chatIdentityLinks.companyId), + eq(chatEndpoints.id, chatIdentityLinks.endpointId), + ), + ) + .innerJoin(companies, eq(companies.id, chatIdentityLinks.companyId)) + .where( + and( + eq(chatIdentityLinks.confirmationTokenHash, tokenHash), + eq(chatIdentityLinks.status, "pending"), + ), + ) + .then((rows) => rows[0] ?? null); + if (!row || !row.link.expiresAt || row.link.expiresAt <= new Date()) { + throw unprocessable("This identity-link request is invalid or expired"); + } + if (!["active", "verifying"].includes(row.endpoint.status)) throw unprocessable("This connection is not available"); + const selfService = await slackIdentityLinks.isSelfService(row.link, tokenHash); + const membership = userId ? await db.select({ status: companyMemberships.status }).from(companyMemberships).where(and( + eq(companyMemberships.companyId, row.link.companyId), eq(companyMemberships.principalType, "user"), eq(companyMemberships.principalId, userId), + )).then((rows) => rows[0]) : null; + return { + selfService, + canConfirm: membership?.status === "active", + endpointId: row.endpoint.id, + companyId: row.endpoint.companyId, + companyName: row.companyName, + companyPrefix: row.companyPrefix, + provider: row.endpoint.provider, + providerAccountLabel: row.endpoint.providerAccountLabel, + botLabel: row.endpoint.botDisplayName, + externalLabel: + row.principal.displayName ?? + row.principal.handle ?? + row.principal.externalId, + externalDetail: row.principal.handle + ? `@${row.principal.handle.replace(/^@/, "")}` + : row.principal.externalId, + expiresAt: row.link.expiresAt.toISOString(), + }; + } + + async function confirmIdentityLink(token: string, paperclipUserId: string) { + const tokenHash = createHash("sha256").update(token).digest("hex"); + const link = await db + .select() + .from(chatIdentityLinks) + .where( + and( + eq(chatIdentityLinks.confirmationTokenHash, tokenHash), + eq(chatIdentityLinks.status, "pending"), + ), + ) + .then((rows) => rows[0] ?? null); + if (!link || !link.expiresAt || link.expiresAt <= new Date()) + throw unprocessable("This identity-link request is invalid or expired"); + const endpointRecordForLink = await endpointRecord(link.endpointId); + if (!endpointRecordForLink || !["active", "verifying"].includes(endpointRecordForLink.endpoint.status)) throw unprocessable("This connection is not available"); + const connectReceipt = endpointRecordForLink.endpoint.provider === "slack" ? await slackIdentityLinks.latestConnect(link) : null; + const confirmationRuntime = connectReceipt ? await confirmationRuntimeFor(endpointRecordForLink.endpoint) : null; + let confirmationEffectId: string | null = null; + const confirmed = await db.transaction(async (tx) => { + await tx.execute( + sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${link.companyId}:${link.principalId}`}, 0))`, + ); + const currentLink = await tx + .select({ + id: chatIdentityLinks.id, + companyId: chatIdentityLinks.companyId, + endpointId: chatIdentityLinks.endpointId, + principalId: chatIdentityLinks.principalId, + expiresAt: chatIdentityLinks.expiresAt, + }) + .from(chatIdentityLinks) + .where( + and( + eq(chatIdentityLinks.id, link.id), + eq(chatIdentityLinks.status, "pending"), + eq(chatIdentityLinks.confirmationTokenHash, tokenHash), + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + const now = new Date(); + if ( + !currentLink || + !currentLink.expiresAt || + currentLink.expiresAt <= now + ) { + return null; + } + const membership = await tx + .select({ status: companyMemberships.status }) + .from(companyMemberships) + .where( + and( + eq(companyMemberships.companyId, currentLink.companyId), + eq(companyMemberships.principalType, "user"), + eq(companyMemberships.principalId, paperclipUserId), + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + if (membership?.status !== "active") { + throw forbidden( + "The signed-in Paperclip account is not a member of this company", + ); + } + const conflictingLink = await tx + .select({ + id: chatIdentityLinks.id, + paperclipUserId: chatIdentityLinks.paperclipUserId, + }) + .from(chatIdentityLinks) + .where( + and( + eq(chatIdentityLinks.companyId, link.companyId), + eq(chatIdentityLinks.principalId, link.principalId), + eq(chatIdentityLinks.status, "linked"), + ne(chatIdentityLinks.paperclipUserId, paperclipUserId), + ), + ) + .limit(1) + .then((rows) => rows[0] ?? null); + if (conflictingLink) { + throw conflict( + "This provider identity is linked to a different Paperclip account", + { + code: "chat_identity_link_conflict", + }, + ); + } + const confirmedLink = await tx + .update(chatIdentityLinks) + .set({ + paperclipUserId, + status: "linked", + confirmationTokenHash: null, + confirmedAt: now, + updatedAt: now, + }) + .where( + and( + eq(chatIdentityLinks.id, link.id), + eq(chatIdentityLinks.status, "pending"), + eq(chatIdentityLinks.confirmationTokenHash, tokenHash), + gt(chatIdentityLinks.expiresAt, now), + ), + ) + .returning({ endpointId: chatIdentityLinks.endpointId }) + .then((rows) => rows[0] ?? null); + if (confirmedLink && connectReceipt && confirmationRuntime && + typeof connectReceipt.payload.channelId === "string" && typeof connectReceipt.payload.userId === "string") { + confirmationEffectId = await slackIdentityLinks.stageConfirmation(tx, { + endpoint: endpointRecordForLink.endpoint, link, tokenHash, + channelId: connectReceipt.payload.channelId, userId: connectReceipt.payload.userId, + confirmationRuntime, + }); + } + return confirmedLink; + }); + if (!confirmed) { + throw conflict("This identity-link request was already used or expired", { + code: "chat_identity_link_consumed", + }); + } + if (confirmationEffectId) scheduleProviderEffect(confirmationEffectId); + return { ok: true, endpointId: confirmed.endpointId }; + } + + async function revokeLink(endpointId: string, principalId: string) { + const link = await db + .select({ companyId: chatIdentityLinks.companyId }) + .from(chatIdentityLinks) + .where( + and( + eq(chatIdentityLinks.endpointId, endpointId), + eq(chatIdentityLinks.principalId, principalId), + ), + ) + .then((rows) => rows[0] ?? null); + if (!link) throw notFound("Identity link not found"); + const revoked = await db.transaction(async (tx) => { + await tx.execute( + sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${link.companyId}:${principalId}`}, 0))`, + ); + return tx + .update(chatIdentityLinks) + .set({ + paperclipUserId: null, + status: "revoked", + confirmationTokenHash: null, + revokedAt: new Date(), + updatedAt: new Date(), + }) + .where( + and( + eq(chatIdentityLinks.companyId, link.companyId), + eq(chatIdentityLinks.endpointId, endpointId), + eq(chatIdentityLinks.principalId, principalId), + ), + ) + .returning({ id: chatIdentityLinks.id }) + .then((rows) => rows[0] ?? null); + }); + if (!revoked) throw notFound("Identity link not found"); + } + + return { listPrincipals, createLinkIntent, previewIdentityLink, requestIdentityAccess: slackIdentityLinks.requestIdentityAccess, confirmIdentityLink, revokeLink }; +} diff --git a/server/src/services/chat-channels/slack/identity-links.ts b/server/src/services/chat-channels/slack/identity-links.ts new file mode 100644 index 00000000000..1a37771fa01 --- /dev/null +++ b/server/src/services/chat-channels/slack/identity-links.ts @@ -0,0 +1,113 @@ +import { createHash, randomBytes } from "node:crypto"; +import { and, desc, eq, gt, or, sql } from "drizzle-orm"; + +import type { Db } from "@paperclipai/db"; + +import { authUsers, chatActions, chatEndpoints, chatIdentityLinks, invites, joinRequests } from "@paperclipai/db"; + +import { conflict, forbidden, unprocessable } from "../../../errors.js"; + +import { logActivity } from "../../activity-log.js"; + +type EndpointRow = typeof chatEndpoints.$inferSelect; +type DbOrTransaction = Db | Parameters[0]>[0]; +type ConfirmationRuntime = { credentialFingerprint: string; generation: number }; +export type SlackIdentityEffectStager = (database: DbOrTransaction, input: { + endpoint: EndpointRow; principalId: string; providerActionId: string; + payload: { version: 1; effect: "ephemeral_message"; authorizationMode: "safe_notice"; threadId: string; userId: string; text: string; settleDelivery: false }; + runtimeContext: ConfirmationRuntime; +}) => Promise<{ id: string } | null>; + +type SlackIdentityLinkOptions = { + previewIdentityLink: (token: string, userId: string) => Promise<{ selfService: boolean; canConfirm: boolean; companyId: string; endpointId: string }>; + stageProviderEffect: SlackIdentityEffectStager; +}; + +export function createSlackIdentityLinks(db: Db, { previewIdentityLink, stageProviderEffect }: SlackIdentityLinkOptions) { + function listConnects(endpoint: EndpointRow) { + const endpointId = endpoint.id; + return db + .select({ principalId: chatActions.principalId, lastConnectAt: sql`max(${chatActions.createdAt})::text` }) + .from(chatActions) + .where(and( + eq(chatActions.companyId, endpoint.companyId), + eq(chatActions.endpointId, endpointId), + eq(chatActions.kind, "slack_connect"), + eq(chatActions.status, "processed"), + )) + .groupBy(chatActions.principalId); + } + + async function isSelfService(link: typeof chatIdentityLinks.$inferSelect, tokenHash: string) { + const selfService = Boolean(await db.select({ id: chatActions.id }).from(chatActions).where(and( + eq(chatActions.companyId, link.companyId), eq(chatActions.endpointId, link.endpointId), + eq(chatActions.principalId, link.principalId), eq(chatActions.kind, "slack_connect"), + sql`${chatActions.payload}->>'identityLinkHash' = ${tokenHash}`, + )).limit(1).then((rows) => rows[0])); + return selfService; + } + + function latestConnect(link: typeof chatIdentityLinks.$inferSelect) { + return db.select({ payload: chatActions.payload }).from(chatActions).where(and( + eq(chatActions.endpointId, link.endpointId), eq(chatActions.principalId, link.principalId), eq(chatActions.kind, "slack_connect"), + )).orderBy(desc(chatActions.createdAt)).limit(1).then((rows) => rows[0]); + } + + async function stageConfirmation(tx: DbOrTransaction, input: { + endpoint: EndpointRow; link: typeof chatIdentityLinks.$inferSelect; tokenHash: string; + channelId: string; userId: string; confirmationRuntime: ConfirmationRuntime; + }) { + const { endpoint, link, tokenHash, channelId, userId, confirmationRuntime } = input; + const effect = await stageProviderEffect(tx, { + endpoint: endpoint, principalId: link.principalId, + providerActionId: `provider_effect:identity_linked:${link.id}:${tokenHash}`, + payload: { + version: 1, effect: "ephemeral_message", authorizationMode: "safe_notice", + threadId: channelId, userId: userId, + text: "Your Slack account is connected to Paperclip. Future messages use your Paperclip permissions.", settleDelivery: false + }, + runtimeContext: confirmationRuntime, + }); + if (!effect) throw conflict("The Slack connection changed; try confirming again"); + return effect.id; + } + + async function requestIdentityAccess(token: string, userId: string, requestIp: string) { + const preview = await previewIdentityLink(token, userId); + if (!preview.selfService) throw forbidden("This identity link cannot request access"); + if (preview.canConfirm) return { status: "member" as const }; + return db.transaction(async (tx) => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`chat-join:${preview.companyId}:${userId}`}, 0))`); + const tokenHash = createHash("sha256").update(token).digest("hex"); + const validLink = await tx.select({ id: chatIdentityLinks.id }).from(chatIdentityLinks).where(and( + eq(chatIdentityLinks.companyId, preview.companyId), eq(chatIdentityLinks.endpointId, preview.endpointId), + eq(chatIdentityLinks.confirmationTokenHash, tokenHash), eq(chatIdentityLinks.status, "pending"), gt(chatIdentityLinks.expiresAt, new Date()), + )).for("update").then((rows) => rows[0]); + if (!validLink) throw unprocessable("This identity-link request is invalid or expired"); + const user = await tx.select({ email: authUsers.email }).from(authUsers).where(eq(authUsers.id, userId)).then((rows) => rows[0]); + if (!user) throw forbidden("Sign in to request company access"); + const existing = await tx.select({ id: joinRequests.id }).from(joinRequests).where(and( + eq(joinRequests.companyId, preview.companyId), eq(joinRequests.requestType, "human"), eq(joinRequests.status, "pending_approval"), + or(eq(joinRequests.requestingUserId, userId), sql`lower(${joinRequests.requestEmailSnapshot}) = ${user.email.toLowerCase()}`), + )).then((rows) => rows[0]); + if (existing) return { status: "pending_approval" as const }; + const now = new Date(); + const [invite] = await tx.insert(invites).values({ + companyId: preview.companyId, tokenHash: createHash("sha256").update(randomBytes(32)).digest("hex"), + inviteType: "company_join", allowedJoinTypes: "human", acceptedAt: now, expiresAt: now, + defaultsPayload: { human: { role: "operator" }, source: "slack_identity_link" }, + }).returning({ id: invites.id }); + const [request] = await tx.insert(joinRequests).values({ + inviteId: invite.id, companyId: preview.companyId, requestType: "human", status: "pending_approval", + requestIp, requestingUserId: userId, requestEmailSnapshot: user.email, + }).returning({ id: joinRequests.id }); + await logActivity(tx as unknown as Db, { + companyId: preview.companyId, actorType: "user", actorId: userId, action: "join.requested", + entityType: "join_request", entityId: request.id, details: { requestType: "human", source: "slack_identity_link", endpointId: preview.endpointId }, + }); + return { status: "pending_approval" as const }; + }); + } + + return { listConnects, isSelfService, latestConnect, stageConfirmation, requestIdentityAccess }; +} From 5a72061ccc5ce6acf5524bb5c3b1cb8d10de5147 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 12 Oct 2026 09:49:34 -0500 Subject: [PATCH 4/7] refactor(chat): extract publication replacement decisions (#15844) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip publishes agent progress and results back to chat. > - Some updates replace an earlier message while others need a new message. > - The current outbound link and task generation determine that authority. > - These decisions are spread through the large chat service. > - This PR groups publication replacement and supersession decisions in one module. > - Reviewers can follow those decisions without reading provider transport code. ## Linked Issues or Issue Description Refs #15836, #15834 and #15823. Stack slice 4 of 5, based on `codex/chat-identity-links`. Merge the predecessor first, then retarget this PR. Do not merge the stack out of order. Stack order: #15823 (provider messages) → #15834 (resource reconciliation) → #15836 (identity links) → #15844 (publication replacement) → #15845 (failed-run retries). GitHub native stack merging preserves this order and automatically restacks the remaining branches. ## What Changed - Extract shared publication replacement and supersession queries from the large service. - Keep exact outbound ownership, run/wake scope, consumed progress, interaction replacement and retained-source authorization together. - Put the iMessage typing-versus-progress policy in `imessage-photon/publications.ts`. - Preserve endpoint lookup timing and all ownership query bodies. The multi-provider close allowlist remains shared dispatch metadata. - Restack onto the revised #15836 and record why this boundary remains mostly shared. ## Verification - The same 49 existing publication and Photon integration cases passed before and after the provider-policy move. Other cases were excluded by the filter. - Cases cover close-owned progress, ambiguous delivery, interleaved Slack control, Telegram confirmation, FIFO replacement and Photon channel behavior. - Server typecheck, module-boundary and whitespace checks passed. Only the iMessage predicate/reason moved; ownership queries are unchanged. - Historical server shard 2 hit a cleanup deadlock; its unchanged [isolated rerun passed](https://github.com/paperclipai/paperclip/actions/runs/38134431077/job/114457999409). [Original failure](https://github.com/paperclipai/paperclip/actions/runs/38134431077/job/114455332022). - Tests use disposable databases and fixture provider transports, not live third-party execution. - Fresh Greptile review is 5/5 on `8e2da97b6dc11eb446128b61c789edb529c50d5a`, with a successful check and no open review threads. - Final-stack `pnpm -r typecheck` and `pnpm build` passed at `57ec5ba94aecd860471f9b14db46f99120c7c790`. - Historical pre-restack CI failed on `ci / General tests (server (11/12))` and its aggregate `ci / verify`. All 1,255 tests passed, but worker-exit cleanup in unchanged `server/src/__tests__/setup-supertest.ts:32` failed with EACCES removing a temporary `paperclip-vitest-home-*` directory. The [original job](https://github.com/paperclipai/paperclip/actions/runs/38256549828/job/114836065382) and the [single unchanged rerun](https://github.com/paperclipai/paperclip/actions/runs/38256549828/job/114839735273) failed the same way. No further reruns were requested. After GitHub automatically restacked the branches, the full cumulative run on #15845 passed this server shard. This middle PR uses the repository’s policy-only stack CI scope; the top PR runs all suites. - The final full-local-suite outcome for this cumulative stack is recorded in #15845; do not infer that gate from the focused tests above. - Merge verification on the restacked tree: 118 focused identity, publication, Photon and retry integration tests passed; server typecheck and module-boundary checks passed. GitHub’s rebased trees exactly match the previously reviewed file contents. - All current-head required checks passed on `8e2da97b6dc11eb446128b61c789edb529c50d5a`: policy CI (middle-stack scope; full cumulative suites passed on #15845); Greptile 5/5 with no unresolved threads. ## Risks - Splitting shared ownership queries per channel would duplicate authorization rules. They remain in the shared module. - Preserve the issue-to-run lock order and retained-source callback. The factory creates no worker or mutable state. - This PR depends on #15836. Merge and retarget in stack order. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and repository tools. The exact runtime model ID and context-window size are not exposed in this session. No separate agent was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../2026-10-10-chat-channels-refactor.md | 30 + server/src/services/chat-channels.ts | 906 +--------------- .../imessage-photon/publications.ts | 9 + .../chat-channels/publication-replacement.ts | 965 ++++++++++++++++++ 4 files changed, 1006 insertions(+), 904 deletions(-) create mode 100644 server/src/services/chat-channels/imessage-photon/publications.ts create mode 100644 server/src/services/chat-channels/publication-replacement.ts diff --git a/doc/plans/2026-10-10-chat-channels-refactor.md b/doc/plans/2026-10-10-chat-channels-refactor.md index ce51837f239..bba089b0c2a 100644 --- a/doc/plans/2026-10-10-chat-channels-refactor.md +++ b/doc/plans/2026-10-10-chat-channels-refactor.md @@ -203,3 +203,33 @@ Slack connect journey: discovery, preview, outsider access request, duplicate request, member confirmation, provider notice, consumed-token rejection and endpoint isolation. Receipt queries, access-request transaction and notice payload retain their statements with explicit parameter substitutions. + +### Slice 4: publication replacement and supersession + +Move 13 functions into `publication-replacement.ts`. The module selects the exact +provider message a publication can replace and suppresses obsolete progress. It +uses the database, endpoint lookup and retained-source authorization callback. +Transport execution, publication claims and receipt reactions remain coordinated +by the service. No state or registration moves in this slice. + +- Source: 37,295 lines / 1,436,867 bytes → 36,393 lines / 1,400,636 bytes. +- Destination: 963 lines / 38,724 bytes. +- Baseline: all 51 checks and 5/5 review passed on slice 3 (`581dd656c1`, #15836). +- All 13 bodies and every remaining statement match the predecessor exactly. + Server typecheck, module boundaries and whitespace checks pass. +- Existing CI fixtures cover exact outbound ownership, interleaved control + messages, consumed progress, interactions, committed failure replacement and + issue-to-run lock order. Local database capacity remains blocked. + +### 2026-10-12 channel-folder reassessment: slice 4 + +Publication replacement is shared ownership logic: exact outbound links, run and +wake scope, consumed progress messages, interaction prompts and retained-source +authorization. Keep those queries together. The close-message provider allowlist +is shared dispatch metadata, not four separate implementations. + +Move the iMessage queued/working suppression policy and explanation to +`imessage-photon/publications.ts`. Preserve when endpoint lookup occurs and leave +all ownership queries unchanged. Baseline: 49 public-service publication and +Photon cases passed, including close-owned progress, interleaved Slack control, +Telegram confirmation and FIFO replacement. The same 49 cases passed after. Server typecheck and module boundaries pass. diff --git a/server/src/services/chat-channels.ts b/server/src/services/chat-channels.ts index 5930982dd0b..2e88f05b99c 100644 --- a/server/src/services/chat-channels.ts +++ b/server/src/services/chat-channels.ts @@ -1,3 +1,4 @@ +import { createChatPublicationReplacement } from "./chat-channels/publication-replacement.js"; import { createChatIdentityLinks } from "./chat-channels/identity-links.js"; import { createChatResourceReconciliation } from "./chat-channels/resource-reconciliation.js"; import { githubRepositoryInventoryItemFromPayload } from "./chat-channels/github/resources.js"; @@ -2427,6 +2428,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { }), }); const issuesSvc = issueService(db); + const { runIdFromMilestonePublication, runOwnershipMilestoneSupersessionReason, runPublicationToReplace, receiptReactionCompletionRunId, inboundWakePublicationToReplace, interactionResolutionPublicationToReplace, interactionPromptPublicationToReplace, progressSupersededByFastResponse, runProgressSupersededByPublishedInteraction, taskStatusPublicationToReplace, closeProgressToReplace } = createChatPublicationReplacement(db, { endpointRecord, authorizeRetainedChatSourcePublication }); const secrets = secretService(db); const questionResponses = questionResponseDeliveryService(db, { heartbeat: @@ -32677,541 +32679,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { }); } - function runIdFromMilestonePublication( - publication: typeof chatPublications.$inferSelect, - ): string | null { - if (!publication.payload.progressState) return null; - const match = - /^run:([^:]+):(?:queued|working|waiting_for_input|completed|failed):/.exec( - publication.idempotencyKey, - ); - const runId = match?.[1] ?? null; - // This identifier is also compared against a UUID column below. Old or - // manually repaired rows must not be allowed to turn the global - // publication sweep into a PostgreSQL cast error. - return isUuidLike(runId) ? runId : null; - } - - async function runOwnershipMilestoneSupersessionReason( - publication: typeof chatPublications.$inferSelect, - ): Promise { - const progress = publication.payload.progressState; - if (progress && ["queued", "working"].includes(progress)) { - const endpoint = await endpointRecord(publication.endpointId); - if (endpoint?.endpoint.provider === "imessage-photon") return "iMessage uses typing instead of progress bubbles"; - } - if ( - !progress || - !["queued", "working", "waiting_for_input"].includes(progress) - ) { - return null; - } - const runId = runIdFromMilestonePublication(publication); - if (!runId) return null; - const run = await db.transaction(async (tx) => { - // Use native finalization's issue -> run lock order. If terminalization - // won, the current status suppresses this obsolete provider update. If - // this short authorization snapshot wins, the progress was still true - // at its durable send boundary and the later final remains authoritative. - const currentIssue = await tx - .select({ id: issues.id }) - .from(issues) - .where( - and( - eq(issues.id, publication.issueId), - eq(issues.companyId, publication.companyId), - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - if (!currentIssue) return null; - return tx - .select({ - status: heartbeatRuns.status, - errorCode: heartbeatRuns.errorCode, - }) - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, runId), - eq(heartbeatRuns.companyId, publication.companyId), - sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${publication.issueId}`, - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - }); - const baseProgressKey = `run:${runId}:${progress}:${publication.endpointId}`; - const nativeProgressPrefix = `run:${runId}:working:${publication.endpointId}:native:`; - const isTerminalizableProgress = - (progress === "queued" || progress === "working") && - (publication.idempotencyKey === baseProgressKey || - publication.idempotencyKey.startsWith(nativeProgressPrefix)); - if ( - isTerminalizableProgress && - run && - ["succeeded", "interrupted", "failed", "cancelled", "timed_out"].includes( - run.status, - ) - ) { - return "Run reached a terminal state before progress delivery"; - } - // Match only the closed ownership-attention projection. Historical progress - // without an extant run retains its existing delivery semantics. - const ownershipBlocked = - run?.status === "running" && - [ - "native_execution_ownership_unverified", - "native_adopted_runner_authentication_timeout", - ].includes(run.errorCode ?? ""); - if (progress === "waiting_for_input") { - return ownershipBlocked - ? null - : "Run ownership attention no longer applies to the bound task"; - } - return ownershipBlocked - ? "Run progress was superseded by ownership recovery attention" - : null; - } - - async function providerProgressLaneConsumed( - publication: typeof chatPublications.$inferSelect, - providerMessageId: string, - ): Promise { - // Published progress rows retain their old provider ID after an edit. - // Consult that exact message's current outbound link, not the conversation - // tail: an authored answer or failure has consumed the lane, while an - // interleaved task-status update may still be replaced normally. - const [consumed] = await db - .select({ id: chatPublications.id }) - .from(chatMessageLinks) - .innerJoin( - chatPublications, - and( - eq(chatPublications.id, chatMessageLinks.publicationId), - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq(chatPublications.providerMessageId, providerMessageId), - eq(chatPublications.state, "published"), - ), - ) - .where( - and( - eq(chatMessageLinks.companyId, publication.companyId), - eq(chatMessageLinks.endpointId, publication.endpointId), - eq(chatMessageLinks.conversationId, publication.conversationId), - eq(chatMessageLinks.direction, "outbound"), - eq(chatMessageLinks.providerMessageId, providerMessageId), - or( - and( - isNotNull(chatPublications.commentId), - sql`${chatPublications.payload}->>'progressState' is null`, - ), - sql`${chatPublications.payload}->>'progressState' = 'failed'`, - ), - ), - ) - .limit(1); - return Boolean(consumed); - } - - async function committedResponseMayReplaceFailure( - publication: typeof chatPublications.$inferSelect, - runId: string, - providerMessageId: string, - ): Promise { - if ( - !publication.commentId || - publication.payload.progressState !== undefined - ) - return false; - return db - .transaction(async (tx) => { - const [run] = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, runId), - eq(heartbeatRuns.companyId, publication.companyId), - eq(heartbeatRuns.nativeIssueId, publication.issueId), - eq(heartbeatRuns.runtimeMode, "native"), - eq(heartbeatRuns.status, "succeeded"), - isNotNull(heartbeatRuns.finishedAt), - ), - ) - .for("share", { noWait: true }); - const presentation = run?.resultJson?.presentationDecision as - Record | undefined; - if ( - !run?.resultJson?.nativeCommittedChatResponse || - presentation?.commentId !== publication.commentId - ) - return false; - const [comment] = await tx - .select({ id: issueComments.id }) - .from(issueComments) - .where( - and( - eq(issueComments.id, publication.commentId!), - eq(issueComments.companyId, publication.companyId), - eq(issueComments.issueId, publication.issueId), - eq(issueComments.createdByRunId, runId), - eq(issueComments.authorAgentId, run.agentId), - isNull(issueComments.deletedAt), - ), - ) - .for("share", { noWait: true }); - if (!comment) return false; - const [priorAnswer] = await tx - .select({ id: chatPublications.id }) - .from(chatPublications) - .innerJoin( - issueComments, - and( - eq(issueComments.id, chatPublications.commentId), - eq(issueComments.companyId, publication.companyId), - eq(issueComments.createdByRunId, runId), - ), - ) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - ne(chatPublications.id, publication.id), - inArray(chatPublications.state, [ - "published", - "streaming", - "delivery_unknown", - ]), - sql`${chatPublications.payload}->>'progressState' is null`, - ), - ) - .limit(1); - if (priorAnswer) return false; - // A historical progress row can retain this ID after a later edit. - // Only its CURRENT outbound link may grant this narrow exception; an - // authored answer or another run's failure must remain consumed. - const [failure] = await tx - .select({ id: chatPublications.id }) - .from(chatMessageLinks) - .innerJoin( - chatPublications, - and( - eq(chatPublications.id, chatMessageLinks.publicationId), - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq(chatPublications.providerMessageId, providerMessageId), - eq( - chatPublications.idempotencyKey, - `run:${runId}:failed:${publication.endpointId}`, - ), - eq(chatPublications.state, "published"), - isNull(chatPublications.commentId), - sql`${chatPublications.payload}->>'progressState' = 'failed'`, - ), - ) - .where( - and( - eq(chatMessageLinks.companyId, publication.companyId), - eq(chatMessageLinks.endpointId, publication.endpointId), - eq(chatMessageLinks.conversationId, publication.conversationId), - eq(chatMessageLinks.providerMessageId, providerMessageId), - eq(chatMessageLinks.direction, "outbound"), - ), - ); - if (!failure) return false; - // The marker is read from the authoritative run, never the publication - // payload. Recheck accepted result/decision plus the complete current - // source batch, actor, access and epoch before borrowing its failure lane. - // The transport claim repeats those checks at the I/O boundary. - return authorizeRetainedChatSourcePublication(tx, publication); - }) - .catch((error: unknown) => { - if (isExternalChatWaitAuthorizationContention(error)) - throw new NativeChatReviewPresentationContentionError(); - throw error; - }); - } - - async function runPublicationToReplace( - publication: typeof chatPublications.$inferSelect, - payload: SafeChatPublicationPayload, - ): Promise { - if (payload.attachmentIds?.length) return null; - const currentRunId = - runIdFromMilestonePublication(publication) ?? - (publication.commentId - ? await db - .select({ runId: issueComments.createdByRunId }) - .from(issueComments) - .where(eq(issueComments.id, publication.commentId)) - .then((rows) => rows[0]?.runId ?? null) - : null); - if (!currentRunId) return null; - return db - .select({ - id: chatPublications.id, - commentId: chatPublications.commentId, - providerMessageId: chatPublications.providerMessageId, - payload: chatPublications.payload, - }) - .from(chatPublications) - .leftJoin(issueComments, eq(issueComments.id, chatPublications.commentId)) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.state, "published"), - isNotNull(chatPublications.providerMessageId), - or( - like(chatPublications.idempotencyKey, `run:${currentRunId}:%`), - eq(issueComments.createdByRunId, currentRunId), - ), - ), - ) - .orderBy(desc(chatPublications.createdAt), desc(chatPublications.id)) - .then(async (rows) => { - // Progress updates are one replaceable provider-message lane per run. - // The first durable agent comment may turn that placeholder into the - // terminal response, but later comments from the same run are distinct - // user-visible outputs and must be posted separately. Re-editing the - // placeholder for each comment silently erases the earlier replies. - if ( - publication.commentId && - rows.some( - (row) => - row.commentId !== null && row.payload.progressState === undefined, - ) - ) - return null; - const replacement = rows.find( - (row) => - Boolean(row.providerMessageId) && - row.payload.progressState !== undefined, - ); - if (!replacement?.providerMessageId) return null; - if ( - (await providerProgressLaneConsumed( - publication, - replacement.providerMessageId, - )) && - !(await committedResponseMayReplaceFailure( - publication, - currentRunId, - replacement.providerMessageId, - )) - ) - return null; - // Replacement identity belongs to the run, not to the provider-visible - // tail. A status/control reply may legitimately interleave while the run - // is active; making the tail the edit candidate would strand this run's - // working placeholder forever. The query is bounded by endpoint, - // conversation (the task generation), and run id, so an interleaved - // control or another run can never donate its provider message here. - return replacement.providerMessageId; - }); - } - - async function receiptReactionCompletionRunId( - tx: Db, - publication: typeof chatPublications.$inferSelect, - payload: SafeChatPublicationPayload, - ): Promise { - if ( - payload.progressState === "completed" || - payload.progressState === "failed" - ) { - return runIdFromMilestonePublication(publication); - } - if (publication.commentId) { - const runId = await tx - .select({ runId: issueComments.createdByRunId }) - .from(issueComments) - .where( - and( - eq(issueComments.id, publication.commentId), - eq(issueComments.companyId, publication.companyId), - ), - ) - .then((rows) => rows[0]?.runId ?? null); - return isUuidLike(runId) ? runId : null; - } - if ( - payload.interactionId && - publication.idempotencyKey === - `interaction:${payload.interactionId}:${publication.endpointId}` - ) { - const runId = await tx - .select({ runId: issueThreadInteractions.sourceRunId }) - .from(issueThreadInteractions) - .where( - and( - eq(issueThreadInteractions.id, payload.interactionId), - eq(issueThreadInteractions.companyId, publication.companyId), - eq(issueThreadInteractions.issueId, publication.issueId), - ), - ) - .then((rows) => rows[0]?.runId ?? null); - return isUuidLike(runId) ? runId : null; - } - return null; - } - - async function inboundWakePublicationToReplace( - publication: typeof chatPublications.$inferSelect, - payload: SafeChatPublicationPayload, - ): Promise { - if (payload.attachmentIds?.length) return null; - const notice = parseInboundWakePublicationKey(publication.idempotencyKey); - let wakeId: string | null = - notice && notice.state !== "queued" ? notice.wakeId : null; - let runId = runIdFromMilestonePublication(publication); - if ( - !notice && - !runId && - payload.interactionId && - publication.idempotencyKey === - `interaction:${payload.interactionId}:${publication.endpointId}` - ) { - const [interaction] = await db - .select({ runId: issueThreadInteractions.sourceRunId }) - .from(issueThreadInteractions) - .where( - and( - eq(issueThreadInteractions.id, payload.interactionId), - eq(issueThreadInteractions.companyId, publication.companyId), - eq(issueThreadInteractions.issueId, publication.issueId), - inArray(issueThreadInteractions.kind, [ - "ask_user_questions", - "request_confirmation", - ]), - ), - ); - runId = interaction?.runId ?? null; - } - if (!notice && !runId && publication.commentId) { - const [comment] = await db - .select({ runId: issueComments.createdByRunId }) - .from(issueComments) - .where( - and( - eq(issueComments.id, publication.commentId), - eq(issueComments.companyId, publication.companyId), - eq(issueComments.issueId, publication.issueId), - ), - ); - runId = comment?.runId ?? null; - } - let runContext: Record | null = null; - if (runId) { - if (publication.commentId) { - const [priorAnswer] = await db - .select({ id: chatPublications.id }) - .from(chatPublications) - .innerJoin( - issueComments, - eq(issueComments.id, chatPublications.commentId), - ) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.state, "published"), - eq(issueComments.createdByRunId, runId), - sql`${chatPublications.payload}->>'progressState' is null`, - ), - ) - .limit(1); - if (priorAnswer) return null; - } - const [run] = await db - .select({ - wakeId: heartbeatRuns.wakeupRequestId, - context: heartbeatRuns.contextSnapshot, - }) - .from(heartbeatRuns) - .innerJoin( - agentWakeupRequests, - and( - eq(agentWakeupRequests.id, heartbeatRuns.wakeupRequestId), - eq(agentWakeupRequests.runId, heartbeatRuns.id), - eq(agentWakeupRequests.companyId, heartbeatRuns.companyId), - eq(agentWakeupRequests.agentId, heartbeatRuns.agentId), - // Failed/cancelled is the admitted run's outcome, not loss of - // its original admission. Its terminal milestone still owns - // this exact lane even if no working update was sent first. - ne(agentWakeupRequests.status, "skipped"), - ), - ) - .where( - and( - eq(heartbeatRuns.id, runId), - eq(heartbeatRuns.companyId, publication.companyId), - sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${publication.issueId}`, - ), - ); - wakeId = run?.wakeId ?? null; - runContext = run?.context ?? null; - } - if (!wakeId) return null; - const [queued] = await db - .select({ - commentId: chatPublications.commentId, - providerMessageId: chatPublications.providerMessageId, - }) - .from(chatPublications) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq(chatPublications.state, "published"), - isNotNull(chatPublications.providerMessageId), - eq( - chatPublications.idempotencyKey, - inboundWakePublicationKey( - wakeId, - "queued", - publication.endpointId, - publication.conversationId, - ), - ), - ), - ); - if (!queued?.commentId || !queued.providerMessageId) return null; - // The run helper may have rejected this same provider ID because its - // queued/working placeholder is already an answer or terminal failure. - // Falling back to the older wake row must not erase that newer output. - if ( - await providerProgressLaneConsumed(publication, queued.providerMessageId) - ) - return null; - if (notice && notice.state !== "queued") - return queued.commentId === publication.commentId - ? queued.providerMessageId - : null; - // A deferred owner can be a Board wake into which the exact external - // comment coalesced. The run must actually contain that source comment; - // wake ID alone cannot authorize a different successor's edit lane. - return runContext && - (runContext.wakeCommentId === queued.commentId || - runContext.commentId === queued.commentId || - (Array.isArray(runContext.wakeCommentIds) && - runContext.wakeCommentIds.includes(queued.commentId))) - ? queued.providerMessageId - : null; - } - async function stageTerminalReceiptReactionRemovals( tx: Db, input: { @@ -33236,375 +32703,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { }); } - async function interactionResolutionPublicationToReplace( - publication: typeof chatPublications.$inferSelect, - payload: SafeChatPublicationPayload, - ): Promise { - if ( - !publication.idempotencyKey.startsWith("interaction-resolution:") || - !payload.interactionId - ) - return null; - return db - .select({ providerMessageId: chatPublications.providerMessageId }) - .from(chatPublications) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq( - chatPublications.idempotencyKey, - `interaction:${payload.interactionId}:${publication.endpointId}`, - ), - eq(chatPublications.state, "published"), - isNotNull(chatPublications.providerMessageId), - ), - ) - .then((rows) => rows[0]?.providerMessageId ?? null); - } - - async function interactionPromptPublicationToReplace( - publication: typeof chatPublications.$inferSelect, - payload: SafeChatPublicationPayload, - ): Promise { - if ( - !payload.interactionId || - publication.idempotencyKey !== - `interaction:${payload.interactionId}:${publication.endpointId}` - ) { - return null; - } - const interaction = await db - .select({ - kind: issueThreadInteractions.kind, - sourceRunId: issueThreadInteractions.sourceRunId, - }) - .from(issueThreadInteractions) - .where( - and( - eq(issueThreadInteractions.id, payload.interactionId), - eq(issueThreadInteractions.companyId, publication.companyId), - eq(issueThreadInteractions.issueId, publication.issueId), - inArray(issueThreadInteractions.kind, [ - "ask_user_questions", - "request_confirmation", - ]), - ), - ) - .then((rows) => rows[0] ?? null); - if (!interaction?.sourceRunId) return null; - - // A provider-visible interaction is this run's response, not an additional - // message beside its progress indicator. Move the durable interaction - // publication onto the exact source run's provider-message lane. This - // retires both the normal working placeholder and the queued placeholder - // when a very fast run asks before the working update is published. - for (const progressState of ["working", "queued"] as const) { - const providerMessageId = await db - .select({ providerMessageId: chatPublications.providerMessageId }) - .from(chatPublications) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq( - chatPublications.idempotencyKey, - `run:${interaction.sourceRunId}:${progressState}:${publication.endpointId}`, - ), - eq(chatPublications.state, "published"), - isNotNull(chatPublications.providerMessageId), - ), - ) - .then((rows) => rows[0]?.providerMessageId ?? null); - if (providerMessageId) return providerMessageId; - } - return null; - } - - async function progressSupersededByFastResponse(publication: typeof chatPublications.$inferSelect): Promise { - if (!["queued", "working"].includes(publication.payload.progressState ?? "")) return false; - const runId = runIdFromMilestonePublication(publication); - const [run] = runId ? await db.select({ context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, publication.companyId), eq(heartbeatRuns.id, runId))) : []; - const sourceCommentId = typeof run?.context?.wakeCommentId === "string" ? run.context.wakeCommentId : typeof run?.context?.commentId === "string" ? run.context.commentId : publication.commentId; - if (!sourceCommentId) return false; - const [receipt] = await db.select({ id: fastResponseRequests.id }).from(fastResponseRequests).innerJoin(chatPublications, - and(eq(chatPublications.companyId, publication.companyId), eq(chatPublications.commentId, fastResponseRequests.commentId), eq(chatPublications.state, "published"))) - .where(and(eq(fastResponseRequests.companyId, publication.companyId), eq(fastResponseRequests.sourceCommentId, sourceCommentId), eq(fastResponseRequests.endpointId, publication.endpointId), eq(fastResponseRequests.conversationId, publication.conversationId))).limit(1); - return Boolean(receipt); - } - - async function runProgressSupersededByPublishedInteraction( - publication: typeof chatPublications.$inferSelect, - ): Promise { - if ( - !["queued", "working"].includes(publication.payload.progressState ?? "") - ) { - return false; - } - const sourceRunId = runIdFromMilestonePublication(publication); - if (!sourceRunId) return false; - const interactionIds = await db - .select({ id: issueThreadInteractions.id }) - .from(issueThreadInteractions) - .where( - and( - eq(issueThreadInteractions.companyId, publication.companyId), - eq(issueThreadInteractions.issueId, publication.issueId), - eq(issueThreadInteractions.sourceRunId, sourceRunId), - inArray(issueThreadInteractions.kind, [ - "ask_user_questions", - "request_confirmation", - ]), - ), - ) - .then((rows) => rows.map((row) => row.id)); - if (interactionIds.length === 0) return false; - return db - .select({ id: chatPublications.id }) - .from(chatPublications) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq(chatPublications.state, "published"), - inArray( - chatPublications.idempotencyKey, - interactionIds.map( - (interactionId) => - `interaction:${interactionId}:${publication.endpointId}`, - ), - ), - ), - ) - .limit(1) - .then((rows) => rows.length > 0); - } - - async function taskStatusPublicationToReplace( - publication: typeof chatPublications.$inferSelect, - payload: SafeChatPublicationPayload, - ): Promise { - if ( - !publication.idempotencyKey.startsWith("control:status:") || - payload.attachmentIds?.length - ) - return null; - const rows = await db - .select({ - commentId: chatPublications.commentId, - commentRunId: issueComments.createdByRunId, - idempotencyKey: chatPublications.idempotencyKey, - payload: chatPublications.payload, - providerMessageId: chatPublications.providerMessageId, - }) - .from(chatPublications) - .leftJoin(issueComments, eq(issueComments.id, chatPublications.commentId)) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.state, "published"), - isNotNull(chatPublications.providerMessageId), - ), - ) - .orderBy(desc(chatPublications.createdAt), desc(chatPublications.id)); - const rowRunId = (row: (typeof rows)[number]) => { - const milestoneMatch = - /^run:([^:]+):(?:queued|working|waiting_for_input|completed|failed):/.exec( - row.idempotencyKey, - ); - return milestoneMatch?.[1] ?? row.commentRunId ?? null; - }; - for (const candidate of rows) { - if ( - !candidate.providerMessageId || - !["queued", "working"].includes(candidate.payload.progressState ?? "") - ) { - continue; - } - const runId = rowRunId(candidate); - if (!runId) continue; - const laneClosed = rows.some((row) => { - if (rowRunId(row) !== runId) return false; - return ( - ["waiting_for_input", "completed", "failed"].includes( - row.payload.progressState ?? "", - ) || - (row.commentId !== null && row.payload.progressState === undefined) - ); - }); - if (!laneClosed) return candidate.providerMessageId; - } - return null; - } - - async function closeProgressToReplace( - publication: typeof chatPublications.$inferSelect, - endpoint: EndpointRow, - ): Promise<{ providerMessageId: string; runId: string } | null> { - if ( - !["telegram", "slack", "discord", "microsoft-teams"].includes( - endpoint.provider, - ) || - !publication.idempotencyKey.startsWith("control:close:") - ) - return null; - const current = alias(chatPublications, "close_current_progress"); - const plainText = (table: typeof chatPublications | typeof current) => - and( - sql`jsonb_typeof(${table.payload}->'text') = 'string'`, - sql`${table.payload}->'card' is null`, - sql`${table.payload}->'interactionId' is null`, - sql`${table.payload}->'transportPart' is null`, - sql`(${table.payload}->'attachmentIds' is null or ${table.payload}->'attachmentIds' = '[]'::jsonb)`, - ); - const ownedMilestone = (table: typeof chatPublications | typeof current) => - or( - sql`${table.idempotencyKey} = 'run:' || ${heartbeatRuns.id}::text || ':queued:' || ${endpoint.id}`, - sql`${table.idempotencyKey} = 'run:' || ${heartbeatRuns.id}::text || ':working:' || ${endpoint.id}`, - sql`${table.idempotencyKey} like 'run:' || ${heartbeatRuns.id}::text || ':working:' || ${endpoint.id} || ':native:%'`, - ); - // A historical progress row alone is not edit authority: its exact current - // outbound link must still belong to that run's progress (or an interleaved - // status control). Read under the same credential lease as final delivery. - const lanes = await db - .selectDistinct({ - providerMessageId: chatMessageLinks.providerMessageId, - runId: heartbeatRuns.id, - }) - .from(chatPublications) - .innerJoin( - heartbeatRuns, - and( - eq(heartbeatRuns.companyId, publication.companyId), - eq(heartbeatRuns.agentId, endpoint.assignedAgentId), - sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${publication.issueId}`, - sql`split_part(${chatPublications.idempotencyKey}, ':', 2) = ${heartbeatRuns.id}::text`, - ), - ) - .innerJoin( - chatMessageLinks, - and( - eq(chatMessageLinks.companyId, publication.companyId), - eq(chatMessageLinks.endpointId, publication.endpointId), - eq(chatMessageLinks.conversationId, publication.conversationId), - eq(chatMessageLinks.direction, "outbound"), - eq( - chatMessageLinks.providerMessageId, - chatPublications.providerMessageId, - ), - ), - ) - .innerJoin( - current, - and( - eq(current.id, chatMessageLinks.publicationId), - eq(current.companyId, publication.companyId), - eq(current.endpointId, publication.endpointId), - eq(current.conversationId, publication.conversationId), - eq(current.issueId, publication.issueId), - eq(current.state, "published"), - eq(current.providerMessageId, chatMessageLinks.providerMessageId), - isNull(current.commentId), - plainText(current), - or( - and( - sql`${current.payload}->>'progressState' in ('queued', 'working')`, - ownedMilestone(current), - ), - like(current.idempotencyKey, "control:status:%"), - ), - ), - ) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq(chatPublications.state, "published"), - isNull(chatPublications.commentId), - plainText(chatPublications), - sql`${chatPublications.payload}->>'progressState' in ('queued', 'working')`, - ownedMilestone(chatPublications), - ), - ) - .limit(2); - // One close receipt owns one edit. Multiple independent lanes require a - // separate durable multi-effect protocol, never an arbitrary latest guess. - if (lanes.length !== 1) return null; - const lane = lanes[0]!; - const [possiblyConsumed] = await db - .select({ id: chatPublications.id }) - .from(chatPublications) - .leftJoin( - issueComments, - and( - eq(issueComments.id, chatPublications.commentId), - eq(issueComments.companyId, publication.companyId), - eq(issueComments.issueId, publication.issueId), - ), - ) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - or( - inArray(chatPublications.state, [ - "pending", - "retry", - "streaming", - "delivery_unknown", - "published", - ]), - // Stopping retries (including an operator's unknown-delivery - // cancellation) does not prove an attempted edit never arrived. - and( - inArray(chatPublications.state, ["cancelled", "failed"]), - gt(chatPublications.attempts, 0), - ), - ), - or( - eq(issueComments.createdByRunId, lane.runId), - // Question/confirmation prompts consume this same run's progress - // ID through interactionPromptPublicationToReplace, but have no - // commentId. A stale working link is not proof that their edit - // never reached the provider (including an unconfirmed receipt). - sql`exists ( - select 1 from issue_thread_interactions consuming_interaction - where consuming_interaction.company_id = ${publication.companyId}::uuid - and consuming_interaction.issue_id = ${publication.issueId}::uuid - and consuming_interaction.source_run_id = ${lane.runId}::uuid - and consuming_interaction.kind in ('ask_user_questions', 'request_confirmation') - and ${chatPublications.payload}->>'interactionId' = consuming_interaction.id::text - and ${chatPublications.idempotencyKey} = 'interaction:' || consuming_interaction.id::text || ':' || ${publication.endpointId} - )`, - eq( - chatPublications.idempotencyKey, - `run:${lane.runId}:completed:${endpoint.id}`, - ), - eq( - chatPublications.idempotencyKey, - `run:${lane.runId}:failed:${endpoint.id}`, - ), - ), - ), - ) - .limit(1); - // A final may have reached the provider before its database receipt. Even an - // unchanged working link cannot disprove that ambiguous provider effect. - return possiblyConsumed ? null : lane; - } - async function currentTaskControlPayload( publication: typeof chatPublications.$inferSelect, ): Promise { diff --git a/server/src/services/chat-channels/imessage-photon/publications.ts b/server/src/services/chat-channels/imessage-photon/publications.ts new file mode 100644 index 00000000000..d85b4410cb5 --- /dev/null +++ b/server/src/services/chat-channels/imessage-photon/publications.ts @@ -0,0 +1,9 @@ +import type { SafeChatPublicationPayload } from "@paperclipai/shared"; + +export function imessageProgressSupersessionReason( + progress: SafeChatPublicationPayload["progressState"], +): string | null { + return progress && ["queued", "working"].includes(progress) + ? "iMessage uses typing instead of progress bubbles" + : null; +} diff --git a/server/src/services/chat-channels/publication-replacement.ts b/server/src/services/chat-channels/publication-replacement.ts new file mode 100644 index 00000000000..626161201a2 --- /dev/null +++ b/server/src/services/chat-channels/publication-replacement.ts @@ -0,0 +1,965 @@ +import { imessageProgressSupersessionReason } from "./imessage-photon/publications.js"; +import { fastResponseRequests } from "@paperclipai/db"; + +import { + and, + desc, + eq, + gt, + inArray, + isNotNull, + isNull, + like, + ne, + or, + sql, +} from "drizzle-orm"; +import { alias } from "drizzle-orm/pg-core"; + +import type { Db } from "@paperclipai/db"; + +import { + agentWakeupRequests, + chatEndpoints, + chatMessageLinks, + chatPublications, + heartbeatRuns, + issueComments, + issueThreadInteractions, + issues, +} from "@paperclipai/db"; +import type { + SafeChatPublicationPayload, +} from "@paperclipai/shared"; +import { + isUuidLike, +} from "@paperclipai/shared"; + +import { + NativeChatReviewPresentationContentionError, +} from "../native-runtime/native-chat-review-presentation.js"; +import { isExternalChatWaitAuthorizationContention } from "../native-runtime/chat-attachment-reuse.js"; + +import { + inboundWakePublicationKey, + parseInboundWakePublicationKey, +} from "../chat-inbound-wakeup-publications.js"; + +type EndpointRow = typeof chatEndpoints.$inferSelect; +type DbOrTransaction = Db | Parameters[0]>[0]; +type ChatPublicationReplacementOptions = { + endpointRecord: (endpointId: string) => Promise<{ endpoint: EndpointRow } | null>; + authorizeRetainedChatSourcePublication: (database: DbOrTransaction, publication: typeof chatPublications.$inferSelect) => Promise; +}; + +export function createChatPublicationReplacement(db: Db, options: ChatPublicationReplacementOptions) { + const { endpointRecord, authorizeRetainedChatSourcePublication } = options; + + + function runIdFromMilestonePublication( + publication: typeof chatPublications.$inferSelect, + ): string | null { + if (!publication.payload.progressState) return null; + const match = + /^run:([^:]+):(?:queued|working|waiting_for_input|completed|failed):/.exec( + publication.idempotencyKey, + ); + const runId = match?.[1] ?? null; + // This identifier is also compared against a UUID column below. Old or + // manually repaired rows must not be allowed to turn the global + // publication sweep into a PostgreSQL cast error. + return isUuidLike(runId) ? runId : null; + } + + async function runOwnershipMilestoneSupersessionReason( + publication: typeof chatPublications.$inferSelect, + ): Promise { + const progress = publication.payload.progressState; + const typingReason = imessageProgressSupersessionReason(progress); + if (typingReason) { + const endpoint = await endpointRecord(publication.endpointId); + if (endpoint?.endpoint.provider === "imessage-photon") return typingReason; + } + if ( + !progress || + !["queued", "working", "waiting_for_input"].includes(progress) + ) { + return null; + } + const runId = runIdFromMilestonePublication(publication); + if (!runId) return null; + const run = await db.transaction(async (tx) => { + // Use native finalization's issue -> run lock order. If terminalization + // won, the current status suppresses this obsolete provider update. If + // this short authorization snapshot wins, the progress was still true + // at its durable send boundary and the later final remains authoritative. + const currentIssue = await tx + .select({ id: issues.id }) + .from(issues) + .where( + and( + eq(issues.id, publication.issueId), + eq(issues.companyId, publication.companyId), + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + if (!currentIssue) return null; + return tx + .select({ + status: heartbeatRuns.status, + errorCode: heartbeatRuns.errorCode, + }) + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, runId), + eq(heartbeatRuns.companyId, publication.companyId), + sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${publication.issueId}`, + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + }); + const baseProgressKey = `run:${runId}:${progress}:${publication.endpointId}`; + const nativeProgressPrefix = `run:${runId}:working:${publication.endpointId}:native:`; + const isTerminalizableProgress = + (progress === "queued" || progress === "working") && + (publication.idempotencyKey === baseProgressKey || + publication.idempotencyKey.startsWith(nativeProgressPrefix)); + if ( + isTerminalizableProgress && + run && + ["succeeded", "interrupted", "failed", "cancelled", "timed_out"].includes( + run.status, + ) + ) { + return "Run reached a terminal state before progress delivery"; + } + // Match only the closed ownership-attention projection. Historical progress + // without an extant run retains its existing delivery semantics. + const ownershipBlocked = + run?.status === "running" && + [ + "native_execution_ownership_unverified", + "native_adopted_runner_authentication_timeout", + ].includes(run.errorCode ?? ""); + if (progress === "waiting_for_input") { + return ownershipBlocked + ? null + : "Run ownership attention no longer applies to the bound task"; + } + return ownershipBlocked + ? "Run progress was superseded by ownership recovery attention" + : null; + } + + async function providerProgressLaneConsumed( + publication: typeof chatPublications.$inferSelect, + providerMessageId: string, + ): Promise { + // Published progress rows retain their old provider ID after an edit. + // Consult that exact message's current outbound link, not the conversation + // tail: an authored answer or failure has consumed the lane, while an + // interleaved task-status update may still be replaced normally. + const [consumed] = await db + .select({ id: chatPublications.id }) + .from(chatMessageLinks) + .innerJoin( + chatPublications, + and( + eq(chatPublications.id, chatMessageLinks.publicationId), + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq(chatPublications.providerMessageId, providerMessageId), + eq(chatPublications.state, "published"), + ), + ) + .where( + and( + eq(chatMessageLinks.companyId, publication.companyId), + eq(chatMessageLinks.endpointId, publication.endpointId), + eq(chatMessageLinks.conversationId, publication.conversationId), + eq(chatMessageLinks.direction, "outbound"), + eq(chatMessageLinks.providerMessageId, providerMessageId), + or( + and( + isNotNull(chatPublications.commentId), + sql`${chatPublications.payload}->>'progressState' is null`, + ), + sql`${chatPublications.payload}->>'progressState' = 'failed'`, + ), + ), + ) + .limit(1); + return Boolean(consumed); + } + + async function committedResponseMayReplaceFailure( + publication: typeof chatPublications.$inferSelect, + runId: string, + providerMessageId: string, + ): Promise { + if ( + !publication.commentId || + publication.payload.progressState !== undefined + ) + return false; + return db + .transaction(async (tx) => { + const [run] = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, runId), + eq(heartbeatRuns.companyId, publication.companyId), + eq(heartbeatRuns.nativeIssueId, publication.issueId), + eq(heartbeatRuns.runtimeMode, "native"), + eq(heartbeatRuns.status, "succeeded"), + isNotNull(heartbeatRuns.finishedAt), + ), + ) + .for("share", { noWait: true }); + const presentation = run?.resultJson?.presentationDecision as + Record | undefined; + if ( + !run?.resultJson?.nativeCommittedChatResponse || + presentation?.commentId !== publication.commentId + ) + return false; + const [comment] = await tx + .select({ id: issueComments.id }) + .from(issueComments) + .where( + and( + eq(issueComments.id, publication.commentId!), + eq(issueComments.companyId, publication.companyId), + eq(issueComments.issueId, publication.issueId), + eq(issueComments.createdByRunId, runId), + eq(issueComments.authorAgentId, run.agentId), + isNull(issueComments.deletedAt), + ), + ) + .for("share", { noWait: true }); + if (!comment) return false; + const [priorAnswer] = await tx + .select({ id: chatPublications.id }) + .from(chatPublications) + .innerJoin( + issueComments, + and( + eq(issueComments.id, chatPublications.commentId), + eq(issueComments.companyId, publication.companyId), + eq(issueComments.createdByRunId, runId), + ), + ) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + ne(chatPublications.id, publication.id), + inArray(chatPublications.state, [ + "published", + "streaming", + "delivery_unknown", + ]), + sql`${chatPublications.payload}->>'progressState' is null`, + ), + ) + .limit(1); + if (priorAnswer) return false; + // A historical progress row can retain this ID after a later edit. + // Only its CURRENT outbound link may grant this narrow exception; an + // authored answer or another run's failure must remain consumed. + const [failure] = await tx + .select({ id: chatPublications.id }) + .from(chatMessageLinks) + .innerJoin( + chatPublications, + and( + eq(chatPublications.id, chatMessageLinks.publicationId), + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq(chatPublications.providerMessageId, providerMessageId), + eq( + chatPublications.idempotencyKey, + `run:${runId}:failed:${publication.endpointId}`, + ), + eq(chatPublications.state, "published"), + isNull(chatPublications.commentId), + sql`${chatPublications.payload}->>'progressState' = 'failed'`, + ), + ) + .where( + and( + eq(chatMessageLinks.companyId, publication.companyId), + eq(chatMessageLinks.endpointId, publication.endpointId), + eq(chatMessageLinks.conversationId, publication.conversationId), + eq(chatMessageLinks.providerMessageId, providerMessageId), + eq(chatMessageLinks.direction, "outbound"), + ), + ); + if (!failure) return false; + // The marker is read from the authoritative run, never the publication + // payload. Recheck accepted result/decision plus the complete current + // source batch, actor, access and epoch before borrowing its failure lane. + // The transport claim repeats those checks at the I/O boundary. + return authorizeRetainedChatSourcePublication(tx, publication); + }) + .catch((error: unknown) => { + if (isExternalChatWaitAuthorizationContention(error)) + throw new NativeChatReviewPresentationContentionError(); + throw error; + }); + } + + async function runPublicationToReplace( + publication: typeof chatPublications.$inferSelect, + payload: SafeChatPublicationPayload, + ): Promise { + if (payload.attachmentIds?.length) return null; + const currentRunId = + runIdFromMilestonePublication(publication) ?? + (publication.commentId + ? await db + .select({ runId: issueComments.createdByRunId }) + .from(issueComments) + .where(eq(issueComments.id, publication.commentId)) + .then((rows) => rows[0]?.runId ?? null) + : null); + if (!currentRunId) return null; + return db + .select({ + id: chatPublications.id, + commentId: chatPublications.commentId, + providerMessageId: chatPublications.providerMessageId, + payload: chatPublications.payload, + }) + .from(chatPublications) + .leftJoin(issueComments, eq(issueComments.id, chatPublications.commentId)) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.state, "published"), + isNotNull(chatPublications.providerMessageId), + or( + like(chatPublications.idempotencyKey, `run:${currentRunId}:%`), + eq(issueComments.createdByRunId, currentRunId), + ), + ), + ) + .orderBy(desc(chatPublications.createdAt), desc(chatPublications.id)) + .then(async (rows) => { + // Progress updates are one replaceable provider-message lane per run. + // The first durable agent comment may turn that placeholder into the + // terminal response, but later comments from the same run are distinct + // user-visible outputs and must be posted separately. Re-editing the + // placeholder for each comment silently erases the earlier replies. + if ( + publication.commentId && + rows.some( + (row) => + row.commentId !== null && row.payload.progressState === undefined, + ) + ) + return null; + const replacement = rows.find( + (row) => + Boolean(row.providerMessageId) && + row.payload.progressState !== undefined, + ); + if (!replacement?.providerMessageId) return null; + if ( + (await providerProgressLaneConsumed( + publication, + replacement.providerMessageId, + )) && + !(await committedResponseMayReplaceFailure( + publication, + currentRunId, + replacement.providerMessageId, + )) + ) + return null; + // Replacement identity belongs to the run, not to the provider-visible + // tail. A status/control reply may legitimately interleave while the run + // is active; making the tail the edit candidate would strand this run's + // working placeholder forever. The query is bounded by endpoint, + // conversation (the task generation), and run id, so an interleaved + // control or another run can never donate its provider message here. + return replacement.providerMessageId; + }); + } + + async function receiptReactionCompletionRunId( + tx: Db, + publication: typeof chatPublications.$inferSelect, + payload: SafeChatPublicationPayload, + ): Promise { + if ( + payload.progressState === "completed" || + payload.progressState === "failed" + ) { + return runIdFromMilestonePublication(publication); + } + if (publication.commentId) { + const runId = await tx + .select({ runId: issueComments.createdByRunId }) + .from(issueComments) + .where( + and( + eq(issueComments.id, publication.commentId), + eq(issueComments.companyId, publication.companyId), + ), + ) + .then((rows) => rows[0]?.runId ?? null); + return isUuidLike(runId) ? runId : null; + } + if ( + payload.interactionId && + publication.idempotencyKey === + `interaction:${payload.interactionId}:${publication.endpointId}` + ) { + const runId = await tx + .select({ runId: issueThreadInteractions.sourceRunId }) + .from(issueThreadInteractions) + .where( + and( + eq(issueThreadInteractions.id, payload.interactionId), + eq(issueThreadInteractions.companyId, publication.companyId), + eq(issueThreadInteractions.issueId, publication.issueId), + ), + ) + .then((rows) => rows[0]?.runId ?? null); + return isUuidLike(runId) ? runId : null; + } + return null; + } + + async function inboundWakePublicationToReplace( + publication: typeof chatPublications.$inferSelect, + payload: SafeChatPublicationPayload, + ): Promise { + if (payload.attachmentIds?.length) return null; + const notice = parseInboundWakePublicationKey(publication.idempotencyKey); + let wakeId: string | null = + notice && notice.state !== "queued" ? notice.wakeId : null; + let runId = runIdFromMilestonePublication(publication); + if ( + !notice && + !runId && + payload.interactionId && + publication.idempotencyKey === + `interaction:${payload.interactionId}:${publication.endpointId}` + ) { + const [interaction] = await db + .select({ runId: issueThreadInteractions.sourceRunId }) + .from(issueThreadInteractions) + .where( + and( + eq(issueThreadInteractions.id, payload.interactionId), + eq(issueThreadInteractions.companyId, publication.companyId), + eq(issueThreadInteractions.issueId, publication.issueId), + inArray(issueThreadInteractions.kind, [ + "ask_user_questions", + "request_confirmation", + ]), + ), + ); + runId = interaction?.runId ?? null; + } + if (!notice && !runId && publication.commentId) { + const [comment] = await db + .select({ runId: issueComments.createdByRunId }) + .from(issueComments) + .where( + and( + eq(issueComments.id, publication.commentId), + eq(issueComments.companyId, publication.companyId), + eq(issueComments.issueId, publication.issueId), + ), + ); + runId = comment?.runId ?? null; + } + let runContext: Record | null = null; + if (runId) { + if (publication.commentId) { + const [priorAnswer] = await db + .select({ id: chatPublications.id }) + .from(chatPublications) + .innerJoin( + issueComments, + eq(issueComments.id, chatPublications.commentId), + ) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.state, "published"), + eq(issueComments.createdByRunId, runId), + sql`${chatPublications.payload}->>'progressState' is null`, + ), + ) + .limit(1); + if (priorAnswer) return null; + } + const [run] = await db + .select({ + wakeId: heartbeatRuns.wakeupRequestId, + context: heartbeatRuns.contextSnapshot, + }) + .from(heartbeatRuns) + .innerJoin( + agentWakeupRequests, + and( + eq(agentWakeupRequests.id, heartbeatRuns.wakeupRequestId), + eq(agentWakeupRequests.runId, heartbeatRuns.id), + eq(agentWakeupRequests.companyId, heartbeatRuns.companyId), + eq(agentWakeupRequests.agentId, heartbeatRuns.agentId), + // Failed/cancelled is the admitted run's outcome, not loss of + // its original admission. Its terminal milestone still owns + // this exact lane even if no working update was sent first. + ne(agentWakeupRequests.status, "skipped"), + ), + ) + .where( + and( + eq(heartbeatRuns.id, runId), + eq(heartbeatRuns.companyId, publication.companyId), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${publication.issueId}`, + ), + ); + wakeId = run?.wakeId ?? null; + runContext = run?.context ?? null; + } + if (!wakeId) return null; + const [queued] = await db + .select({ + commentId: chatPublications.commentId, + providerMessageId: chatPublications.providerMessageId, + }) + .from(chatPublications) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq(chatPublications.state, "published"), + isNotNull(chatPublications.providerMessageId), + eq( + chatPublications.idempotencyKey, + inboundWakePublicationKey( + wakeId, + "queued", + publication.endpointId, + publication.conversationId, + ), + ), + ), + ); + if (!queued?.commentId || !queued.providerMessageId) return null; + // The run helper may have rejected this same provider ID because its + // queued/working placeholder is already an answer or terminal failure. + // Falling back to the older wake row must not erase that newer output. + if ( + await providerProgressLaneConsumed(publication, queued.providerMessageId) + ) + return null; + if (notice && notice.state !== "queued") + return queued.commentId === publication.commentId + ? queued.providerMessageId + : null; + // A deferred owner can be a Board wake into which the exact external + // comment coalesced. The run must actually contain that source comment; + // wake ID alone cannot authorize a different successor's edit lane. + return runContext && + (runContext.wakeCommentId === queued.commentId || + runContext.commentId === queued.commentId || + (Array.isArray(runContext.wakeCommentIds) && + runContext.wakeCommentIds.includes(queued.commentId))) + ? queued.providerMessageId + : null; + } + + async function interactionResolutionPublicationToReplace( + publication: typeof chatPublications.$inferSelect, + payload: SafeChatPublicationPayload, + ): Promise { + if ( + !publication.idempotencyKey.startsWith("interaction-resolution:") || + !payload.interactionId + ) + return null; + return db + .select({ providerMessageId: chatPublications.providerMessageId }) + .from(chatPublications) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq( + chatPublications.idempotencyKey, + `interaction:${payload.interactionId}:${publication.endpointId}`, + ), + eq(chatPublications.state, "published"), + isNotNull(chatPublications.providerMessageId), + ), + ) + .then((rows) => rows[0]?.providerMessageId ?? null); + } + + async function interactionPromptPublicationToReplace( + publication: typeof chatPublications.$inferSelect, + payload: SafeChatPublicationPayload, + ): Promise { + if ( + !payload.interactionId || + publication.idempotencyKey !== + `interaction:${payload.interactionId}:${publication.endpointId}` + ) { + return null; + } + const interaction = await db + .select({ + kind: issueThreadInteractions.kind, + sourceRunId: issueThreadInteractions.sourceRunId, + }) + .from(issueThreadInteractions) + .where( + and( + eq(issueThreadInteractions.id, payload.interactionId), + eq(issueThreadInteractions.companyId, publication.companyId), + eq(issueThreadInteractions.issueId, publication.issueId), + inArray(issueThreadInteractions.kind, [ + "ask_user_questions", + "request_confirmation", + ]), + ), + ) + .then((rows) => rows[0] ?? null); + if (!interaction?.sourceRunId) return null; + + // A provider-visible interaction is this run's response, not an additional + // message beside its progress indicator. Move the durable interaction + // publication onto the exact source run's provider-message lane. This + // retires both the normal working placeholder and the queued placeholder + // when a very fast run asks before the working update is published. + for (const progressState of ["working", "queued"] as const) { + const providerMessageId = await db + .select({ providerMessageId: chatPublications.providerMessageId }) + .from(chatPublications) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq( + chatPublications.idempotencyKey, + `run:${interaction.sourceRunId}:${progressState}:${publication.endpointId}`, + ), + eq(chatPublications.state, "published"), + isNotNull(chatPublications.providerMessageId), + ), + ) + .then((rows) => rows[0]?.providerMessageId ?? null); + if (providerMessageId) return providerMessageId; + } + return null; + } + + async function progressSupersededByFastResponse(publication: typeof chatPublications.$inferSelect): Promise { + if (!["queued", "working"].includes(publication.payload.progressState ?? "")) return false; + const runId = runIdFromMilestonePublication(publication); + const [run] = runId ? await db.select({ context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, publication.companyId), eq(heartbeatRuns.id, runId))) : []; + const sourceCommentId = typeof run?.context?.wakeCommentId === "string" ? run.context.wakeCommentId : typeof run?.context?.commentId === "string" ? run.context.commentId : publication.commentId; + if (!sourceCommentId) return false; + const [receipt] = await db.select({ id: fastResponseRequests.id }).from(fastResponseRequests).innerJoin(chatPublications, + and(eq(chatPublications.companyId, publication.companyId), eq(chatPublications.commentId, fastResponseRequests.commentId), eq(chatPublications.state, "published"))) + .where(and(eq(fastResponseRequests.companyId, publication.companyId), eq(fastResponseRequests.sourceCommentId, sourceCommentId), eq(fastResponseRequests.endpointId, publication.endpointId), eq(fastResponseRequests.conversationId, publication.conversationId))).limit(1); + return Boolean(receipt); + } + + async function runProgressSupersededByPublishedInteraction( + publication: typeof chatPublications.$inferSelect, + ): Promise { + if ( + !["queued", "working"].includes(publication.payload.progressState ?? "") + ) { + return false; + } + const sourceRunId = runIdFromMilestonePublication(publication); + if (!sourceRunId) return false; + const interactionIds = await db + .select({ id: issueThreadInteractions.id }) + .from(issueThreadInteractions) + .where( + and( + eq(issueThreadInteractions.companyId, publication.companyId), + eq(issueThreadInteractions.issueId, publication.issueId), + eq(issueThreadInteractions.sourceRunId, sourceRunId), + inArray(issueThreadInteractions.kind, [ + "ask_user_questions", + "request_confirmation", + ]), + ), + ) + .then((rows) => rows.map((row) => row.id)); + if (interactionIds.length === 0) return false; + return db + .select({ id: chatPublications.id }) + .from(chatPublications) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq(chatPublications.state, "published"), + inArray( + chatPublications.idempotencyKey, + interactionIds.map( + (interactionId) => + `interaction:${interactionId}:${publication.endpointId}`, + ), + ), + ), + ) + .limit(1) + .then((rows) => rows.length > 0); + } + + async function taskStatusPublicationToReplace( + publication: typeof chatPublications.$inferSelect, + payload: SafeChatPublicationPayload, + ): Promise { + if ( + !publication.idempotencyKey.startsWith("control:status:") || + payload.attachmentIds?.length + ) + return null; + const rows = await db + .select({ + commentId: chatPublications.commentId, + commentRunId: issueComments.createdByRunId, + idempotencyKey: chatPublications.idempotencyKey, + payload: chatPublications.payload, + providerMessageId: chatPublications.providerMessageId, + }) + .from(chatPublications) + .leftJoin(issueComments, eq(issueComments.id, chatPublications.commentId)) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.state, "published"), + isNotNull(chatPublications.providerMessageId), + ), + ) + .orderBy(desc(chatPublications.createdAt), desc(chatPublications.id)); + const rowRunId = (row: (typeof rows)[number]) => { + const milestoneMatch = + /^run:([^:]+):(?:queued|working|waiting_for_input|completed|failed):/.exec( + row.idempotencyKey, + ); + return milestoneMatch?.[1] ?? row.commentRunId ?? null; + }; + for (const candidate of rows) { + if ( + !candidate.providerMessageId || + !["queued", "working"].includes(candidate.payload.progressState ?? "") + ) { + continue; + } + const runId = rowRunId(candidate); + if (!runId) continue; + const laneClosed = rows.some((row) => { + if (rowRunId(row) !== runId) return false; + return ( + ["waiting_for_input", "completed", "failed"].includes( + row.payload.progressState ?? "", + ) || + (row.commentId !== null && row.payload.progressState === undefined) + ); + }); + if (!laneClosed) return candidate.providerMessageId; + } + return null; + } + + async function closeProgressToReplace( + publication: typeof chatPublications.$inferSelect, + endpoint: EndpointRow, + ): Promise<{ providerMessageId: string; runId: string } | null> { + if ( + !["telegram", "slack", "discord", "microsoft-teams"].includes( + endpoint.provider, + ) || + !publication.idempotencyKey.startsWith("control:close:") + ) + return null; + const current = alias(chatPublications, "close_current_progress"); + const plainText = (table: typeof chatPublications | typeof current) => + and( + sql`jsonb_typeof(${table.payload}->'text') = 'string'`, + sql`${table.payload}->'card' is null`, + sql`${table.payload}->'interactionId' is null`, + sql`${table.payload}->'transportPart' is null`, + sql`(${table.payload}->'attachmentIds' is null or ${table.payload}->'attachmentIds' = '[]'::jsonb)`, + ); + const ownedMilestone = (table: typeof chatPublications | typeof current) => + or( + sql`${table.idempotencyKey} = 'run:' || ${heartbeatRuns.id}::text || ':queued:' || ${endpoint.id}`, + sql`${table.idempotencyKey} = 'run:' || ${heartbeatRuns.id}::text || ':working:' || ${endpoint.id}`, + sql`${table.idempotencyKey} like 'run:' || ${heartbeatRuns.id}::text || ':working:' || ${endpoint.id} || ':native:%'`, + ); + // A historical progress row alone is not edit authority: its exact current + // outbound link must still belong to that run's progress (or an interleaved + // status control). Read under the same credential lease as final delivery. + const lanes = await db + .selectDistinct({ + providerMessageId: chatMessageLinks.providerMessageId, + runId: heartbeatRuns.id, + }) + .from(chatPublications) + .innerJoin( + heartbeatRuns, + and( + eq(heartbeatRuns.companyId, publication.companyId), + eq(heartbeatRuns.agentId, endpoint.assignedAgentId), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${publication.issueId}`, + sql`split_part(${chatPublications.idempotencyKey}, ':', 2) = ${heartbeatRuns.id}::text`, + ), + ) + .innerJoin( + chatMessageLinks, + and( + eq(chatMessageLinks.companyId, publication.companyId), + eq(chatMessageLinks.endpointId, publication.endpointId), + eq(chatMessageLinks.conversationId, publication.conversationId), + eq(chatMessageLinks.direction, "outbound"), + eq( + chatMessageLinks.providerMessageId, + chatPublications.providerMessageId, + ), + ), + ) + .innerJoin( + current, + and( + eq(current.id, chatMessageLinks.publicationId), + eq(current.companyId, publication.companyId), + eq(current.endpointId, publication.endpointId), + eq(current.conversationId, publication.conversationId), + eq(current.issueId, publication.issueId), + eq(current.state, "published"), + eq(current.providerMessageId, chatMessageLinks.providerMessageId), + isNull(current.commentId), + plainText(current), + or( + and( + sql`${current.payload}->>'progressState' in ('queued', 'working')`, + ownedMilestone(current), + ), + like(current.idempotencyKey, "control:status:%"), + ), + ), + ) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq(chatPublications.state, "published"), + isNull(chatPublications.commentId), + plainText(chatPublications), + sql`${chatPublications.payload}->>'progressState' in ('queued', 'working')`, + ownedMilestone(chatPublications), + ), + ) + .limit(2); + // One close receipt owns one edit. Multiple independent lanes require a + // separate durable multi-effect protocol, never an arbitrary latest guess. + if (lanes.length !== 1) return null; + const lane = lanes[0]!; + const [possiblyConsumed] = await db + .select({ id: chatPublications.id }) + .from(chatPublications) + .leftJoin( + issueComments, + and( + eq(issueComments.id, chatPublications.commentId), + eq(issueComments.companyId, publication.companyId), + eq(issueComments.issueId, publication.issueId), + ), + ) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + or( + inArray(chatPublications.state, [ + "pending", + "retry", + "streaming", + "delivery_unknown", + "published", + ]), + // Stopping retries (including an operator's unknown-delivery + // cancellation) does not prove an attempted edit never arrived. + and( + inArray(chatPublications.state, ["cancelled", "failed"]), + gt(chatPublications.attempts, 0), + ), + ), + or( + eq(issueComments.createdByRunId, lane.runId), + // Question/confirmation prompts consume this same run's progress + // ID through interactionPromptPublicationToReplace, but have no + // commentId. A stale working link is not proof that their edit + // never reached the provider (including an unconfirmed receipt). + sql`exists ( + select 1 from issue_thread_interactions consuming_interaction + where consuming_interaction.company_id = ${publication.companyId}::uuid + and consuming_interaction.issue_id = ${publication.issueId}::uuid + and consuming_interaction.source_run_id = ${lane.runId}::uuid + and consuming_interaction.kind in ('ask_user_questions', 'request_confirmation') + and ${chatPublications.payload}->>'interactionId' = consuming_interaction.id::text + and ${chatPublications.idempotencyKey} = 'interaction:' || consuming_interaction.id::text || ':' || ${publication.endpointId} + )`, + eq( + chatPublications.idempotencyKey, + `run:${lane.runId}:completed:${endpoint.id}`, + ), + eq( + chatPublications.idempotencyKey, + `run:${lane.runId}:failed:${endpoint.id}`, + ), + ), + ), + ) + .limit(1); + // A final may have reached the provider before its database receipt. Even an + // unchanged working link cannot disprove that ambiguous provider effect. + return possiblyConsumed ? null : lane; + } + + return { runIdFromMilestonePublication, runOwnershipMilestoneSupersessionReason, runPublicationToReplace, receiptReactionCompletionRunId, inboundWakePublicationToReplace, interactionResolutionPublicationToReplace, interactionPromptPublicationToReplace, progressSupersededByFastResponse, runProgressSupersededByPublishedInteraction, taskStatusPublicationToReplace, closeProgressToReplace }; +} From d497d1ccecb8756c2219a72b279cfd1465156615 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 12 Oct 2026 09:49:34 -0500 Subject: [PATCH 5/7] refactor(chat): extract failed-run retry coordination (#15845) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages agent work and sends results back to chat. > - A failed chat run can be retried only while its original source remains authorized. > - That decision includes persisted source evidence and native runtime state. > - Retry claims also share in-flight promises and depend on service shutdown. > - This PR groups retry authorization, execution and notices in one module. > - The service retains the live admission, runtime and shutdown boundaries. ## Linked Issues or Issue Description Refs #15844, #15836, #15834 and #15823. Stack slice 5 of 5, based on `codex/chat-publication-replacement`. Merge each predecessor first, then retarget this PR. Do not merge the stack out of order. Stack order: #15823 (provider messages) → #15834 (resource reconciliation) → #15836 (identity links) → #15844 (publication replacement) → #15845 (failed-run retries). GitHub native stack merging preserves this order and automatically restacks the remaining branches. ## What Changed - Extract failed-run retry admission and coordination while keeping the in-flight map, cursor, error class and authority registrations scoped to one chat service. - Move the exact Teams uncertain-file continuation proof into `microsoft-teams/file-transfer-source.ts`. - Pass the existing transaction, explicit source identifiers and denial factory. Return only the proven publication ID. - Keep committed-result, complete source-batch, ancestor and other-uncertain-effect checks in shared `failed-run-retry.ts`. - Keep native execution recovery together; its provider references describe runner backends, not chat channels. - The retry coordinator is 1,825 lines and the Teams proof is 149 lines. Restack on revised #15844 and record the reassessment. ## Verification - Baseline: 59 retry cases and six negative Teams authority cases passed. The combined 65 cases passed after the move. - Cases cover retry ownership, service restart/lifetime, ambiguous Teams receipts, sibling unknown effects, changed source authority and active attempts. - Teams guard, joins, locks, attempt checks and selected ID match after explicit input substitutions. The shared coordinator retains its state and cleanup ownership. - Server typecheck, module-boundary and whitespace checks passed. - Integration uses disposable databases and fixture provider transports; no live third-party execution is claimed. - Fresh Greptile review is 5/5 on `4f7151409717458717633aa270441a8faf33664e`, with a successful check and no open review threads. - Final-stack `pnpm -r typecheck` and `pnpm build` passed at `57ec5ba94aecd860471f9b14db46f99120c7c790`. - All CI checks passed on `57ec5ba94aecd860471f9b14db46f99120c7c790`. - Full local `pnpm test:run` stopped in the general-server group: 18,763 passed, 105 skipped, and one test timed out after 20 seconds in unchanged `workspace-runtime.test.ts` ("does not accept an occupied allocated port when listener ownership is unavailable"). The same case passed in an isolated rerun without code, assertion or timeout changes (1 passed, 175 excluded). The original timeout is retained as a failed full run; later workspace and serialized groups were not reached. The full local gate remains unverified. - Merge verification on the restacked tree: 118 focused identity, publication, Photon and retry integration tests passed; server typecheck and module-boundary checks passed. GitHub’s rebased trees exactly match the previously reviewed file contents. - All current-head required checks passed on `4f7151409717458717633aa270441a8faf33664e`: full CI; Greptile 5/5 with no unresolved threads. ## Risks - Authorization must remain fail-closed across committed output and retry admission. The helper proves only the Teams transfer; the caller still checks the complete source and result in the same transaction. - In-flight promises, error-class identity, registration cleanup, cursor state and the live shutdown getter must remain per-service. Their code is unchanged by this channel-folder move. - This PR depends on #15844. Merge and retarget in stack order. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and repository tools. The exact runtime model ID and context-window size are not exposed in this session. No separate agent was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../2026-10-10-chat-channels-refactor.md | 71 + .../chat-failed-run-retry-boundary.test.ts | 83 + server/src/services/chat-channels.ts | 1837 +---------------- .../chat-channels/failed-run-retry.ts | 1825 ++++++++++++++++ .../microsoft-teams/file-transfer-source.ts | 149 ++ 5 files changed, 2153 insertions(+), 1812 deletions(-) create mode 100644 server/src/__tests__/chat-failed-run-retry-boundary.test.ts create mode 100644 server/src/services/chat-channels/failed-run-retry.ts create mode 100644 server/src/services/chat-channels/microsoft-teams/file-transfer-source.ts diff --git a/doc/plans/2026-10-10-chat-channels-refactor.md b/doc/plans/2026-10-10-chat-channels-refactor.md index bba089b0c2a..21d90f2740a 100644 --- a/doc/plans/2026-10-10-chat-channels-refactor.md +++ b/doc/plans/2026-10-10-chat-channels-refactor.md @@ -233,3 +233,74 @@ Move the iMessage queued/working suppression policy and explanation to all ownership queries unchanged. Baseline: 49 public-service publication and Photon cases passed, including close-owned progress, interleaved Slack control, Telegram confirmation and FIFO replacement. The same 49 cases passed after. Server typecheck and module boundaries pass. + + +### Slice 5: failed chat run retry coordination + +Extract source validation, native failure evidence, retry preparation, claims, +wake authorization and retry notices into `failed-run-retry.ts`. Keep the +in-flight promise map, notice cursor, error class and two authority registrations +inside the factory so each service retains its own lifetime. Keep the live +shutdown and heartbeat callbacks. The service still owns cleanup order and waits +for the module's pending retries before unregistering authorities. + +- Source: 36,393 lines / 1,400,636 bytes → 34,606 lines / 1,336,443 bytes. + Net reduction: 1,787 lines / 64,193 bytes. +- Destination: 1,936 lines / 68,329 bytes. +- Baseline: all 52 checks and 5/5 review passed on slice 4 (`166ea04ac6`, #15844). +- Added three tests through the existing public service before the move. All + three passed before and after: duplicate calls share a promise, shutdown + during a pending read prevents a new claim and waits for completion, rejected + work permits a later attempt, and old-service cleanup preserves replacement + authority. These tests use controlled reads and do not require PostgreSQL. +- All 21 moved declarations and every remaining statement match the predecessor + after restoring the documented live callbacks and pending-task accessor. + Server typecheck, module boundaries and whitespace checks pass. +- Existing CI database fixtures cover corrected/edited sources, lifecycle epochs, + native recovery evidence, recreated services and concurrent retries. Local + full database execution remains blocked by the host's shared-memory limit. + Current-head full CI, broad local gates and review results belong in the PR. + +## Reassessment after five slices + +The service falls from 39,155 lines / 1,509,090 bytes at the rebased baseline to +34,606 lines / 1,336,443 bytes: 4,549 lines and 172,647 bytes removed (11.6% of +source lines). The five modules total 4,857 lines / 185,635 bytes, including their +imports and typed boundaries. This is useful separation, not completion of the +larger decomposition. + +The service function still spans 32,254 lines. Its 244 direct nested functions +occupy 31,525 lines; 48 other statements occupy 435 lines of state, wiring, +registration and public API. The remaining lines are whitespace, comments and +syntax. The largest functions are `processMessage` (2,166), `handleAction` +(1,074), `configureWithCredentialLease` (868), `processSelectedPublication` (729), +`handleModalSubmit` (708), `handleWebhook` (685), `processLifecycleDelivery` (657) +and `applyProviderLifecycleEffect` (593). + +The best next planning candidate is inbound message admission and persistence, +starting from `processMessage` and its conversation/drain ownership. Define the +transaction, live-message retention and scheduling contracts before moving it; +do not move that entire function and claim its complexity is resolved. Action +resolution and endpoint configuration are later candidates. Keep runtime lease +acquisition, partial startup and shutdown together until their ownership contract +is explicit. This batch ends with five stacked PRs; merging is separate work. + +### 2026-10-12 channel-folder reassessment: slice 5 + +Keep retry admission, complete source-batch authorization, ancestor checks, +in-flight deduplication, authority registration and shutdown coordination shared. +References to native runner providers describe execution recovery, not chat +channel implementations, and remain with that shared proof. + +Move the exact Teams file-transfer continuation proof into +`microsoft-teams/file-transfer-source.ts`. It receives the caller's transaction, +explicit source identifiers and original denial factory; it returns only the +verified publication ID. The shared caller still validates the committed result, +all original inputs and other uncertain publications. All provider proof guards, +join predicates, row locks and attempt checks retain their statements. + +Baseline: 59 retry cases plus six negative Teams authority cases passed. These +include service lifetime checks and transfer ambiguity without duplicate uploads. +The same 65 cases passed after. Server typecheck and module boundaries pass. +The remaining retry coordinator +has 1,825 lines; the Teams proof module has 149 lines. diff --git a/server/src/__tests__/chat-failed-run-retry-boundary.test.ts b/server/src/__tests__/chat-failed-run-retry-boundary.test.ts new file mode 100644 index 00000000000..ad62169924d --- /dev/null +++ b/server/src/__tests__/chat-failed-run-retry-boundary.test.ts @@ -0,0 +1,83 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { chatActions, type Db } from "@paperclipai/db"; +import { chatChannelService, type ChatChannelService, type ChatChannelServiceOptions } from "../services/chat-channels.js"; +import { authorizeCommittedChatResponse, CommittedChatResponseAuthorizationError } from "../services/durable-chat-wakeup.js"; + +const services = new Set(); +afterEach(async () => { + for (const service of services) await service.shutdown(); + services.clear(); +}); + +function createService(db: Db) { + const wakeup = vi.fn().mockResolvedValue(undefined); + const runtimeShutdown = vi.fn().mockResolvedValue(undefined); + const service = chatChannelService(db, { + heartbeat: { wakeup }, + runtime: { shutdown: runtimeShutdown } as unknown as ChatChannelServiceOptions["runtime"], + }); + services.add(service); + return { service, wakeup, runtimeShutdown }; +} + +function deferred() { + let resolve!: (value: T) => void; + const promise = new Promise((done) => { resolve = done; }); + return { promise, resolve }; +} + +describe("failed chat retry service lifetime", () => { + it("shares an in-flight retry and observes shutdown after its candidate read", async () => { + const candidate = deferred(); + const reads = vi.fn((table: unknown) => table === chatActions ? candidate.promise : Promise.resolve([])); + const update = vi.fn(() => { throw new Error("A shutdown retry must not claim work"); }); + const db = { + select: () => ({ from: (table: unknown) => ({ where: () => reads(table) }) }), + update, + } as unknown as Db; + const { service, wakeup, runtimeShutdown } = createService(db); + const first = service.processFailedChatRunRetry("retry-action"); + const second = service.processFailedChatRunRetry("retry-action"); + expect(second).toBe(first); + expect(reads).toHaveBeenCalledTimes(1); + const shutdown = service.shutdown(); + expect(runtimeShutdown).not.toHaveBeenCalled(); + candidate.resolve([{ + id: "retry-action", companyId: "company", status: "issued", + payload: { issueId: "issue" }, result: {}, updatedAt: new Date(0), + }]); + await expect(first).resolves.toMatchObject({ actionId: "retry-action", status: "queued" }); + await shutdown; + services.delete(service); + expect(update).not.toHaveBeenCalled(); + expect(wakeup).not.toHaveBeenCalled(); + expect(runtimeShutdown).toHaveBeenCalledTimes(1); + }); + + it("releases a rejected in-flight retry so a later attempt can read again", async () => { + const where = vi.fn().mockResolvedValue([]); + const db = { select: () => ({ from: () => ({ where }) }) } as unknown as Db; + const { service } = createService(db); + const first = service.processFailedChatRunRetry("missing-action"); + await expect(first).rejects.toMatchObject({ status: 404 }); + const second = service.processFailedChatRunRetry("missing-action"); + expect(second).not.toBe(first); + await expect(second).rejects.toMatchObject({ status: 404 }); + expect(where).toHaveBeenCalledTimes(2); + }); + + it("does not unregister a replacement service's committed-response authority", async () => { + const db = {} as Db; + const first = createService(db).service; + const replacement = createService(db).service; + await first.shutdown(); + services.delete(first); + const readFailure = new Error("replacement authority reached current source reads"); + const tx = { select: () => { throw readFailure; } } as unknown as Db; + const input = { companyId: "company", issueId: "issue", agentId: "agent", runId: "run", resultId: "result" }; + await expect(authorizeCommittedChatResponse(db, tx, input)).rejects.toBe(readFailure); + await replacement.shutdown(); + services.delete(replacement); + await expect(authorizeCommittedChatResponse(db, tx, input)).rejects.toBeInstanceOf(CommittedChatResponseAuthorizationError); + }); +}); diff --git a/server/src/services/chat-channels.ts b/server/src/services/chat-channels.ts index 2e88f05b99c..fa5c85cea27 100644 --- a/server/src/services/chat-channels.ts +++ b/server/src/services/chat-channels.ts @@ -1,3 +1,4 @@ +import { createChatFailedRunRetry, type ChatFailedRetrySource as FailedChatRetrySource } from "./chat-channels/failed-run-retry.js"; import { createChatPublicationReplacement } from "./chat-channels/publication-replacement.js"; import { createChatIdentityLinks } from "./chat-channels/identity-links.js"; import { createChatResourceReconciliation } from "./chat-channels/resource-reconciliation.js"; @@ -155,9 +156,6 @@ import type { Db } from "@paperclipai/db"; import { createDurableChatWakeupRequest, assertDurableChatWakeupReceipt, - registerFailedChatRunRetryAuthority, - registerCommittedChatResponseAuthority, - CommittedChatResponseAuthorizationError, } from "./durable-chat-wakeup.js"; import { agentWakeupRequests, @@ -177,15 +175,11 @@ import { chatTeamsFileTransfers, companyMemberships, companySecretBindings, - heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, - nativeRunResults, - environmentLeases, issueComments, issueAttachments, issueThreadInteractions, - issueQuestionResponseDeliveries, issues, toolApplications, toolConnections, @@ -208,7 +202,6 @@ import type { } from "@paperclipai/shared"; import { isAgentStatusInvokable, - CHAT_PROVIDERS, isUuidLike, LOW_TRUST_REVIEW_PRESET, LOW_TRUST_REVIEW_PRESET_VERSION, @@ -269,11 +262,7 @@ import { telegramStopSubscriptionPlan, telegramStopSubscriptionConfirmed, } from "./chat-telegram-stop-subscription.js"; -import { - nativeFailedRunRetryStateIsSafe, - nativePreProviderRetryAfterCleanupStateIsSafe, - nativeProviderRecoveryEvidence, -} from "./native-runtime/native-session-executor.js"; + import { parseChatWebhookPublicBaseUrl } from "../chat-webhook-public-url.js"; import { parseGitHubReceiptIdentity, @@ -10937,1804 +10926,28 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { ]); } - const failedChatRetryDenied = () => - conflict( - "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.", - { code: "chat_failed_run_retry_not_authorized" }, - ); - class FailedChatRetryPublicationReadError extends Error { - constructor() { - super("Chat retry publication authorization is temporarily unavailable"); - } - } - - function retryCommentIds(context: Record): string[] { - const ids = - context.wakeCommentIds ?? - (context.wakeCommentId ? [context.wakeCommentId] : []); - if ( - !Array.isArray(ids) || - ids.length === 0 || - ids.length > 50 || - ids.some((id) => typeof id !== "string" || !isUuidLike(id)) || - new Set(ids).size !== ids.length || - (context.wakeCommentId !== undefined && - context.wakeCommentId !== ids.at(-1)) - ) - throw failedChatRetryDenied(); - return ids as string[]; - } - - type FailedChatRetrySource = { - failedRunId: string; - sourceWakeupRequestId: string; - issueId: string; - agentId: string; - taskKey: string; - provider: ChatProvider; - commentIds: string[]; - sources: Array>; - endpointId: string; - conversationId: string; - principalId: string; - sessionGeneration: number; - requestedByActorType: "user" | "system"; - requestedByActorId: string; - retryAncestors: Array<{ - actionId: string; - failedRunId: string; - sourceScopeSha256: string; - }>; - sourceScopeSha256: string; - }; - function hashFailedRetrySource( - scope: Omit, - ): FailedChatRetrySource { - return { - ...scope, - sourceScopeSha256: createHash("sha256") - .update(JSON.stringify(scope)) - .digest("hex"), - }; - } - - function assertFailedRetryReceipt( - ...args: Parameters - ) { - try { - assertDurableChatWakeupReceipt(...args); - } catch { - throw failedChatRetryDenied(); - } - } - - /** The reviewed-chat ownership check runs before runtime/profile resolution - * or provider startup. Recognize only that exact persisted failure together - * with its sole system diagnostic and absence of execution evidence. A - * generic setup error (or a prior-session display hint) is not this proof. */ - async function isPreProviderReviewedChatFailure( - tx: DbOrTransaction, - run: typeof heartbeatRuns.$inferSelect, - ): Promise { - const diagnostic = "reviewed_chat_execution_binding_not_authorized"; - if ( - run.runtimeMode !== "legacy" || - run.errorCode !== "setup_failed" || - run.error !== diagnostic || - [ - run.runtimeModeResolverVersion, - run.runtimeModeReason, - run.runtimeModeResolvedAt, - run.runnerProfileJson, - run.runnerInstanceId, - run.nativeSessionId, - run.nativeIssueId, - run.nativePhase, - run.driverKind, - run.driverVersion, - run.completionContractId, - run.completionContractSha256, - run.sessionIdAfter, - run.externalRunId, - run.processPid, - run.processGroupId, - run.processStartedAt, - run.logStore, - run.logRef, - run.logBytes, - run.logSha256, - run.stdoutExcerpt, - run.stderrExcerpt, - run.lastOutputAt, - run.lastOutputStream, - run.lastOutputBytes, - run.usageJson, - run.exitCode, - run.signal, - ].some((value) => value !== null) || - run.lastOutputSeq !== 0 || - run.logCompressed - ) - return false; - const events = await tx - .select() - .from(heartbeatRunEvents) - .where( - and( - eq(heartbeatRunEvents.companyId, run.companyId), - eq(heartbeatRunEvents.runId, run.id), - ), - ) - .limit(2) - .for("share", { noWait: true }); - const event = events[0]; - if ( - events.length !== 1 || - !event || - event.agentId !== run.agentId || - event.seq !== 1 || - event.eventType !== "error" || - event.stream !== "system" || - event.level !== "error" || - event.message !== diagnostic || - event.payload !== null || - event.sourceInstanceId !== null || - event.sourceEventId !== null || - event.sourceSeq !== null || - event.sourcePayloadSha256 !== null || - event.protocolSchemaVersion !== null - ) - return false; - const evidence = await tx.execute(sql`select 1 where - exists (select 1 from native_run_finalizations where company_id = ${run.companyId}::uuid and run_id = ${run.id}::uuid) - or exists (select 1 from native_run_results where company_id = ${run.companyId}::uuid and run_id = ${run.id}::uuid) - or exists (select 1 from environment_leases where company_id = ${run.companyId}::uuid and heartbeat_run_id = ${run.id}::uuid)`); - return evidence.length === 0; - } - - async function failedNativeRetryCoordinator( - tx: DbOrTransaction, - run: typeof heartbeatRuns.$inferSelect, - issueId: string, - ) { - const [coordinator] = await tx - .select() - .from(nativeRunFinalizations) - .where( - and( - eq(nativeRunFinalizations.runId, run.id), - eq(nativeRunFinalizations.companyId, run.companyId), - eq(nativeRunFinalizations.issueId, issueId), - ), - ) - .for("share", { noWait: true }); - // A queued request can fail before its first provider event because the - // predecessor's exact retained checkpoint has not yet been suspended. - // A later authenticated cleanup receipt can authorize retrying that - // request; never pretend it exhausted provider execution attempts. - if ( - coordinator && - run.nativeIssueId === issueId && - run.status === "failed" && - run.errorCode === "adapter_failed" && - // A diagnostic reason does not change this failure category. The exact - // checkpoint, cleanup receipt, and no-provider-work proofs below still - // decide whether the original request can be retried. - (run.error === "runner_state_identity_mismatch" || - run.error?.startsWith("runner_state_identity_mismatch: ")) && - run.nativePhase === "observed" && - coordinator.phase === "observed" && - coordinator.attempt === 0 && - !coordinator.leaseOwner && - !coordinator.leaseExpiresAt && - !coordinator.nextAttemptAt && - !coordinator.resultId && - !coordinator.assessmentId && - !coordinator.decisionId && - run.nativeSessionId && - run.processPid && - run.processGroupId === run.processPid - ) { - const [event] = await tx - .select({ id: heartbeatRunEvents.id }) - .from(heartbeatRunEvents) - .where( - and( - eq(heartbeatRunEvents.companyId, run.companyId), - eq(heartbeatRunEvents.runId, run.id), - or( - isNotNull(heartbeatRunEvents.sourceInstanceId), - sql`${heartbeatRunEvents.payload}->'prpEvent' is not null`, - ), - ), - ) - .limit(1); - const [result] = await tx - .select({ id: nativeRunResults.id }) - .from(nativeRunResults) - .where( - and( - eq(nativeRunResults.companyId, run.companyId), - eq(nativeRunResults.runId, run.id), - ), - ) - .limit(1); - const checkpoint = run.runnerProfileJson?.sessionCheckpoint as - | Record - | undefined; - const binding = checkpoint?.identity as - | Record - | undefined; - if ( - !event && - !result && - binding?.companyId === run.companyId && - binding?.issueId === issueId && - binding?.agentId === run.agentId && - binding?.runId === run.id && - binding?.sessionId === run.nativeSessionId && - typeof checkpoint?.sessionId === "string" && - checkpoint.sessionId.trim().length > 0 && - (checkpoint.providerSessionId == null || - (typeof checkpoint.providerSessionId === "string" && - checkpoint.providerSessionId.trim().length > 0)) - ) { - const predecessors = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.companyId, run.companyId), - eq(heartbeatRuns.agentId, run.agentId), - eq(heartbeatRuns.nativeIssueId, issueId), - eq(heartbeatRuns.nativeSessionId, run.nativeSessionId), - eq(heartbeatRuns.processPid, run.processPid), - eq(heartbeatRuns.processGroupId, run.processGroupId), - eq(heartbeatRuns.runtimeMode, "native"), - eq(heartbeatRuns.status, "succeeded"), - ne(heartbeatRuns.id, run.id), - // Warm runs can share a PID and session. Only a durable maintenance - // owner is a candidate; the physical check below must still prove - // that this exact run owns the canonical suspended checkpoint. - sql`exists (select 1 from native_run_finalizations retired - where retired.company_id = ${heartbeatRuns.companyId} - and retired.run_id = ${heartbeatRuns.id} - and retired.phase = 'committed' - and exists (select 1 from jsonb_array_elements(retired.recovery_history) receipt - where receipt->>'kind' = 'native_cleanup_maintenance' - and receipt->>'phase' = 'settled' and receipt->>'version' = '1'))`, - ), - ) - .for("share", { noWait: true }) - .limit(2); - if (predecessors.length === 1) { - const predecessor = predecessors[0]!; - const [retired] = await tx - .select() - .from(nativeRunFinalizations) - .where( - and( - eq(nativeRunFinalizations.companyId, run.companyId), - eq(nativeRunFinalizations.issueId, issueId), - eq(nativeRunFinalizations.runId, predecessor.id), - ), - ) - .for("share", { noWait: true }); - const receipt = retired?.recoveryHistory.findLast( - (entry) => entry.kind === "native_cleanup_maintenance", - ); - const leases = await tx - .select() - .from(environmentLeases) - .where( - and( - eq(environmentLeases.companyId, run.companyId), - eq(environmentLeases.heartbeatRunId, run.id), - ), - ) - .for("share", { noWait: true }); - if ( - retired?.phase === "committed" && - retired.resultId && - retired.assessmentId && - retired.decisionId && - !retired.leaseOwner && - !retired.leaseExpiresAt && - !retired.nextAttemptAt && - receipt?.version === 1 && - receipt.phase === "settled" && - typeof receipt.requestId === "string" && - receipt.requestId.length > 0 && - receipt.nativeSessionId === run.nativeSessionId && - receipt.runnerInstanceId === predecessor.runnerInstanceId && - // The cleanup receipt identifies the provider thread. The - // checkpoint's providerSessionId is the separate backend account. - receipt.providerSessionId === checkpoint.sessionId && - typeof receipt.settledFingerprint === "string" && - /^[a-f0-9]{64}$/.test(receipt.settledFingerprint) && - typeof receipt.sourceFingerprint === "string" && - /^[a-f0-9]{64}$/.test(receipt.sourceFingerprint) && - leases.every( - (lease) => - lease.provider === "local" && - lease.providerLeaseId === null && - lease.issueId === issueId && - ["failed", "released", "expired"].includes(lease.status) && - (lease.cleanupStatus === null || - lease.cleanupStatus === "success") && - lease.releasedAt !== null, - ) - ) { - return { - coordinator, - leases, - retiredOwner: { - predecessor, - receipt, - providerSessionId: checkpoint.sessionId, - providerBackendSessionId: checkpoint.providerSessionId ?? null, - }, - }; - } - } - } - } - const detail = coordinator?.failureDetail; - const ordinary = [ - "provider_process_exited", - "provider_stdout_closed", - "provider_process_output_closed", - "provider_process_status_failed", - "provider_initialize_timeout", - "provider_initialize_protocol_error", - "provider_request_timeout", - "provider_request_protocol_error", - "provider_transport_failed", - "native_runner_process_exited", - ]; - if ( - !coordinator || - run.nativeIssueId !== issueId || - run.nativePhase !== "terminal_failure" || - coordinator.phase !== "terminal_failure" || - coordinator.recoveryState !== "blocked" || - coordinator.failureCode !== "native_session_retry_exhausted" || - coordinator.attempt < 3 || - coordinator.leaseOwner || - coordinator.leaseExpiresAt || - coordinator.nextAttemptAt || - coordinator.resultId || - coordinator.assessmentId || - coordinator.decisionId || - !detail || - !ordinary.includes(String(detail.originalFailureCode)) || - !["bootstrap_retry", "exact_checkpoint_resume"].includes( - String(detail.recoveryMode), - ) || - ![ - "adapter_failed", - "native_session_retry_exhausted", - String(detail.originalFailureCode), - ].includes(run.errorCode ?? "") || - (detail.recoveryMode === "bootstrap_retry" && - (detail.providerSessionEstablished !== false || - detail.providerEventsExist !== false || - detail.checkpointExists !== false)) || - (detail.recoveryMode === "exact_checkpoint_resume" && - (detail.providerSessionEstablished !== true || - detail.checkpointExists !== true)) - ) - throw failedChatRetryDenied(); - const evidence = await nativeProviderRecoveryEvidence({ - db: tx as unknown as Db, - runId: run.id, - sourceFailureCode: detail.originalFailureCode as Parameters< - typeof nativeProviderRecoveryEvidence - >[0]["sourceFailureCode"], - }); - if ( - evidence.recoveryMode !== detail.recoveryMode || - evidence.providerSessionEstablished !== - detail.providerSessionEstablished || - evidence.providerEventsExist !== detail.providerEventsExist || - evidence.checkpointExists !== detail.checkpointExists - ) - throw failedChatRetryDenied(); - const leases = await tx - .select() - .from(environmentLeases) - .where( - and( - eq(environmentLeases.companyId, run.companyId), - eq(environmentLeases.heartbeatRunId, run.id), - ), - ) - .for("share", { noWait: true }); - if ( - leases.some( - (lease) => - !["released", "expired", "failed"].includes(lease.status) || - lease.cleanupStatus !== "success" || - !lease.releasedAt, - ) - ) - throw failedChatRetryDenied(); - return { coordinator, leases, retiredOwner: null }; - } - - async function assertFailedNativeRetryState( - tx: DbOrTransaction, - source: Pick, - companyId: string, - ) { - const [run] = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, source.failedRunId), - eq(heartbeatRuns.companyId, companyId), - eq(heartbeatRuns.agentId, source.agentId), - ), - ) - .for("share", { noWait: true }); - if (!run) throw failedChatRetryDenied(); - if (run.runtimeMode !== "native") return; - const { coordinator, leases, retiredOwner } = - await failedNativeRetryCoordinator(tx, run, source.issueId); - if (retiredOwner) { - if ( - !nativePreProviderRetryAfterCleanupStateIsSafe({ - failedExecution: run.runnerProfileJson?.nativeExecutionInput, - retiredExecution: - retiredOwner.predecessor.runnerProfileJson?.nativeExecutionInput, - companyId, - issueId: source.issueId, - agentId: source.agentId, - failedRunId: run.id, - retiredRunId: retiredOwner.predecessor.id, - nativeSessionId: run.nativeSessionId!, - runnerInstanceId: retiredOwner.predecessor.runnerInstanceId!, - providerSessionId: retiredOwner.providerSessionId, - providerBackendSessionId: retiredOwner.providerBackendSessionId, - processPid: run.processPid!, - processGroupId: run.processGroupId!, - receipt: retiredOwner.receipt, - }) - ) - throw failedChatRetryDenied(); - return; - } - const checkpoint = run.runnerProfileJson?.sessionCheckpoint as - | Record - | undefined; - if ( - !run.nativeSessionId || - !run.runnerInstanceId || - (checkpoint?.providerSessionId != null && - (typeof checkpoint.providerSessionId !== "string" || - checkpoint.providerSessionId.trim().length === 0)) || - !nativeFailedRunRetryStateIsSafe({ - execution: run.runnerProfileJson?.nativeExecutionInput, - companyId, - issueId: source.issueId, - agentId: source.agentId, - runId: run.id, - nativeSessionId: run.nativeSessionId, - runnerInstanceId: run.runnerInstanceId, - processPid: run.processPid, - processGroupId: run.processGroupId, - providerSessionId: - typeof checkpoint?.sessionId === "string" - ? checkpoint.sessionId - : null, - providerBackendSessionId: - typeof checkpoint?.providerSessionId === "string" - ? checkpoint.providerSessionId - : null, - recoveryMode: coordinator.failureDetail!.recoveryMode as - | "bootstrap_retry" - | "exact_checkpoint_resume", - allowVerifiedBackup: leases.length > 0, - }) - ) - throw failedChatRetryDenied(); - } - - /** Resolve only a complete, originally admitted direct-chat batch. A retry - * never turns caller JSON, task-description text, or a provider redelivery - * into new input authority. Answer continuations require their own proof. */ - async function failedChatRetrySource( - tx: DbOrTransaction, - input: { - companyId: string; - issueId: string; - agentId: string; - failedRunId: string; - retryActionId?: string; - runId?: string; - publication?: boolean; - /** Internal selector for already committed output, never retry admission. */ - committedResponse?: { - resultId: string; - canonicalSha256?: string; - decisionId?: string; - /** Internal source-proof continuation for one locked Teams transfer. - * Never permission to repeat its card/PUT or retry a model run. */ - continuingTeamsFileTransfer?: { - transferId: string; - publicationId: string; - version: number; - phase: "consent_unknown" | "file_info_unknown"; - }; - }; - }, - visited = new Set(), - ): Promise { - if (visited.has(input.failedRunId) || visited.size >= 16) - throw failedChatRetryDenied(); - const ancestors = new Set([...visited, input.failedRunId]); - const [issue] = await tx - .select() - .from(issues) - .where( - and( - eq(issues.id, input.issueId), - eq(issues.companyId, input.companyId), - ), - ) - .for( - "update", - input.publication || input.retryActionId ? { noWait: true } : undefined, - ); - const [actor] = await tx - .select({ status: agents.status }) - .from(agents) - .where( - and( - eq(agents.id, input.agentId), - eq(agents.companyId, input.companyId), - ), - ) - .for("share", { noWait: true }); - const [failedRun] = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, input.failedRunId), - eq(heartbeatRuns.companyId, input.companyId), - eq(heartbeatRuns.agentId, input.agentId), - ), - ) - .for("share", { noWait: true }); - if ( - !issue || - issue.assigneeAgentId !== input.agentId || - !actor || - !isAgentStatusInvokable(actor.status) || - !failedRun || - !failedRun.wakeupRequestId || - (!input.committedResponse && - (!["failed", "timed_out"].includes(failedRun.status) || - !failedRun.finishedAt)) || - (!input.committedResponse && - failedRun.runtimeMode !== "native" && - !( - [ - "adapter_failed", - "adapter_exit_code", - "process_exit", - "timeout", - ].includes(failedRun.errorCode ?? "") || - (await isPreProviderReviewedChatFailure(tx, failedRun)) - )) || - (!input.publication && - ["backlog", "done", "cancelled"].includes(issue.status)) - ) - throw failedChatRetryDenied(); - if (failedRun.runtimeMode === "native" && !input.committedResponse) - await failedNativeRetryCoordinator(tx, failedRun, issue.id); - const context = failedRun.contextSnapshot ?? {}; - const provider = CHAT_PROVIDERS.find( - (value) => context.source === `chat:${value}`, - ); - if ( - !provider || - context.issueId !== issue.id || - context.interactionId || - (context.taskKey !== issue.identifier && context.taskKey !== issue.id) - ) - throw failedChatRetryDenied(); - const commentIds = retryCommentIds(context); - const [accepted] = await tx - .select() - .from(nativeRunResults) - .where( - and( - eq(nativeRunResults.companyId, input.companyId), - eq(nativeRunResults.runId, failedRun.id), - eq(nativeRunResults.schemaStatus, "accepted"), - input.committedResponse - ? eq(nativeRunResults.id, input.committedResponse.resultId) - : undefined, - ), - ) - .limit(1) - .for("share", { noWait: true }); - if (input.committedResponse) { - const [coordinator] = await tx - .select() - .from(nativeRunFinalizations) - .where( - and( - eq(nativeRunFinalizations.companyId, input.companyId), - eq(nativeRunFinalizations.issueId, issue.id), - eq(nativeRunFinalizations.runId, failedRun.id), - ), - ) - .for("share", { noWait: true }); - const result = accepted?.resultJson.result as - | Record - | undefined; - const terminal = accepted?.resultJson.terminal as - | Record - | undefined; - const continuation = result?.continuation as - | Record - | undefined; - if ( - failedRun.runtimeMode !== "native" || - failedRun.nativeIssueId !== issue.id || - !accepted || - accepted.id !== input.committedResponse.resultId || - accepted.issueId !== issue.id || - accepted.completionContractId !== failedRun.completionContractId || - coordinator?.phase !== "committed" || - coordinator.resultId !== accepted.id || - !coordinator.decisionId || - !coordinator.assessmentId || - (input.committedResponse.canonicalSha256 !== undefined && - input.committedResponse.canonicalSha256 !== - accepted.canonicalSha256) || - (input.committedResponse.decisionId !== undefined && - input.committedResponse.decisionId !== coordinator.decisionId) || - result?.schema !== "paperclip.run_result.v1" || - result.reportedWorkDisposition !== "yielded" || - !Array.isArray(result.attentionRequests) || - result.attentionRequests.length !== 0 || - continuation?.kind !== "response_wake" || - terminal?.schema !== "paperclip.prp.terminal.v1" || - terminal.turnTerminalState !== "completed" || - terminal.runTerminalState !== "succeeded" || - terminal.reportedWorkDisposition !== "yielded" - ) - throw failedChatRetryDenied(); - } - let continuingUnknownPublicationId: string | undefined; - const continuingTransfer = - input.committedResponse?.continuingTeamsFileTransfer; - if (continuingTransfer !== undefined) { - if ( - input.publication !== true || - input.retryActionId !== undefined || - input.runId !== undefined || - provider !== "microsoft-teams" || - !continuingTransfer || - !isUuidLike(continuingTransfer.transferId) || - !isUuidLike(continuingTransfer.publicationId) || - !Number.isSafeInteger(continuingTransfer.version) || - continuingTransfer.version < 1 || - !["consent_unknown", "file_info_unknown"].includes( - continuingTransfer.phase, - ) - ) - throw failedChatRetryDenied(); - // The caller supplies a selector only. Independently prove the exact - // unknown publication is this native run's own selected file, not a - // sibling effect or another actor/generation. All ordinary source and - // committed-result authorization below still runs unchanged. - const [continuing] = await tx - .select({ - publicationId: chatPublications.id, - attemptId: chatTeamsFileTransfers.attemptId, - attemptExpiresAt: chatTeamsFileTransfers.attemptExpiresAt, - }) - .from(chatTeamsFileTransfers) - .innerJoin( - chatPublications, - and( - eq(chatPublications.id, chatTeamsFileTransfers.publicationId), - eq(chatPublications.companyId, chatTeamsFileTransfers.companyId), - eq(chatPublications.endpointId, chatTeamsFileTransfers.endpointId), - eq( - chatPublications.conversationId, - chatTeamsFileTransfers.conversationId, - ), - eq(chatPublications.issueId, chatTeamsFileTransfers.issueId), - eq(chatPublications.commentId, chatTeamsFileTransfers.commentId), - eq(chatPublications.state, "delivery_unknown"), - sql`${chatPublications.payload}->'attachmentIds' = jsonb_build_array(${chatTeamsFileTransfers.attachmentId}::text)`, - ), - ) - .innerJoin( - chatEndpoints, - and( - eq(chatEndpoints.id, chatTeamsFileTransfers.endpointId), - eq(chatEndpoints.companyId, input.companyId), - eq(chatEndpoints.provider, "microsoft-teams"), - ), - ) - .innerJoin( - chatConversations, - and( - eq(chatConversations.id, chatTeamsFileTransfers.conversationId), - eq(chatConversations.companyId, input.companyId), - eq(chatConversations.endpointId, chatTeamsFileTransfers.endpointId), - eq(chatConversations.issueId, issue.id), - eq( - chatConversations.sessionGeneration, - chatTeamsFileTransfers.conversationGeneration, - ), - ), - ) - .innerJoin( - issueComments, - and( - eq(issueComments.id, chatTeamsFileTransfers.commentId), - eq(issueComments.companyId, input.companyId), - eq(issueComments.issueId, issue.id), - eq(issueComments.createdByRunId, failedRun.id), - eq(issueComments.authorAgentId, input.agentId), - isNull(issueComments.deletedAt), - ), - ) - .innerJoin( - issueAttachments, - and( - eq(issueAttachments.id, chatTeamsFileTransfers.attachmentId), - eq(issueAttachments.companyId, input.companyId), - eq(issueAttachments.issueId, issue.id), - eq(issueAttachments.issueCommentId, issueComments.id), - eq(issueAttachments.originatingRunId, failedRun.id), - ), - ) - .innerJoin( - assets, - and( - eq(assets.id, issueAttachments.assetId), - eq(assets.companyId, input.companyId), - eq(assets.sha256, chatTeamsFileTransfers.sha256), - eq(assets.byteSize, chatTeamsFileTransfers.byteSize), - ), - ) - .where( - and( - eq(chatTeamsFileTransfers.companyId, input.companyId), - eq(chatTeamsFileTransfers.issueId, issue.id), - eq(chatTeamsFileTransfers.id, continuingTransfer.transferId), - eq( - chatTeamsFileTransfers.publicationId, - continuingTransfer.publicationId, - ), - eq(chatTeamsFileTransfers.version, continuingTransfer.version), - eq(chatTeamsFileTransfers.phase, continuingTransfer.phase), - ), - ) - .for("share", { noWait: true }); - if ( - !continuing || - (continuing.attemptId && - (!continuing.attemptExpiresAt || - continuing.attemptExpiresAt > new Date())) - ) - throw failedChatRetryDenied(); - continuingUnknownPublicationId = continuing.publicationId; - } - const [question] = await tx - .select({ id: issueThreadInteractions.id }) - .from(issueThreadInteractions) - .where( - and( - eq(issueThreadInteractions.companyId, input.companyId), - eq(issueThreadInteractions.issueId, issue.id), - eq(issueThreadInteractions.sourceRunId, failedRun.id), - ), - ) - .limit(1); - const [answerDelivery] = await tx - .select({ id: issueQuestionResponseDeliveries.id }) - .from(issueQuestionResponseDeliveries) - .where( - and( - eq(issueQuestionResponseDeliveries.companyId, input.companyId), - eq(issueQuestionResponseDeliveries.targetRunId, failedRun.id), - ), - ) - .limit(1); - const [resultPublication] = await tx - .select({ id: chatPublications.id }) - .from(chatPublications) - .where( - and( - eq(chatPublications.companyId, input.companyId), - eq(chatPublications.issueId, issue.id), - input.committedResponse - ? eq(chatPublications.state, "delivery_unknown") - : undefined, - continuingUnknownPublicationId - ? ne(chatPublications.id, continuingUnknownPublicationId) - : undefined, - or( - and( - like(chatPublications.idempotencyKey, `run:${failedRun.id}:%`), - eq(chatPublications.state, "delivery_unknown"), - ), - sql`exists (select 1 from issue_comments result_comment where result_comment.id = ${chatPublications.commentId} - and result_comment.company_id = ${input.companyId}::uuid and result_comment.created_by_run_id = ${failedRun.id}::uuid - )`, - ), - ), - ) - .limit(1); - if ( - (!input.committedResponse && accepted) || - resultPublication || - question || - answerDelivery - ) - throw failedChatRetryDenied(); - if (context.chatFailedRunRetry) { - const hint = context.chatFailedRunRetry as Record; - const [priorIntent] = await tx - .select() - .from(chatActions) - .where( - and( - eq(chatActions.id, failedRun.wakeupRequestId), - eq(chatActions.companyId, input.companyId), - ), - ) - .for("share", { noWait: true }); - const [priorReceipt] = await tx - .select() - .from(agentWakeupRequests) - .where( - and( - eq(agentWakeupRequests.id, failedRun.wakeupRequestId), - eq(agentWakeupRequests.companyId, input.companyId), - ), - ) - .for("share", { noWait: true }); - if ( - !priorIntent || - priorIntent.kind !== "failed_run_retry" || - priorIntent.status !== "processed" || - !priorReceipt || - priorReceipt.runId !== failedRun.id || - !["failed", "completed"].includes(priorReceipt.status) || - typeof priorIntent.payload.failedRunId !== "string" || - !isUuidLike(priorIntent.payload.failedRunId) || - hint.version !== 1 || - hint.actionId !== priorIntent.id || - hint.failedRunId !== priorIntent.payload.failedRunId || - failedRun.retryOfRunId !== priorIntent.payload.failedRunId || - context.retryOfRunId !== failedRun.retryOfRunId - ) - throw failedChatRetryDenied(); - const parent = await failedChatRetrySource( - tx, - { - ...input, - failedRunId: priorIntent.payload.failedRunId, - committedResponse: undefined, - }, - ancestors, - ); - if ( - priorIntent.payload.sourceScopeSha256 !== parent.sourceScopeSha256 || - priorIntent.endpointId !== parent.endpointId || - priorIntent.conversationId !== parent.conversationId || - priorIntent.principalId !== parent.principalId || - context.taskKey !== parent.taskKey || - provider !== parent.provider || - JSON.stringify(commentIds) !== JSON.stringify(parent.commentIds) - ) - throw failedChatRetryDenied(); - assertFailedRetryReceipt( - createDurableChatWakeupRequest({ - id: priorIntent.id, - companyId: input.companyId, - agentId: input.agentId, - issueId: input.issueId, - commentId: parent.commentIds.at(-1)!, - requestedByActorType: parent.requestedByActorType, - requestedByActorId: parent.requestedByActorId, - requestedAt: priorIntent.createdAt, - authorize: async () => {}, - }), - priorReceipt, - ); - const { sourceScopeSha256, ...parentScope } = parent; - return hashFailedRetrySource({ - ...parentScope, - failedRunId: failedRun.id, - retryAncestors: [ - ...parent.retryAncestors, - { - actionId: priorIntent.id, - failedRunId: parent.failedRunId, - sourceScopeSha256, - }, - ], - }); - } - // Include every receipt that fed this run, including a deferred owner's - // coalesced children. A mutable latest-comment hint cannot drop a sibling. - const admittedBatch = - await tx.execute(sql`select source.id, source.payload->>'commentId' as comment_id - from agent_wakeup_requests receipt join chat_actions source on source.id = receipt.id - where receipt.company_id = ${input.companyId}::uuid and receipt.agent_id = ${input.agentId}::uuid - and (receipt.run_id = ${failedRun.id}::uuid or receipt.id = ${failedRun.wakeupRequestId}::uuid - or receipt.payload->>'coalescedIntoWakeupRequestId' = ${failedRun.wakeupRequestId}) - and source.company_id = ${input.companyId}::uuid and source.kind = 'inbound_wakeup' limit 51`); - if ( - admittedBatch.length !== commentIds.length || - admittedBatch.some( - (entry) => !commentIds.includes(String(entry.comment_id)), - ) - ) - throw failedChatRetryDenied(); - const sources = await tx - .select() - .from(chatActions) - .where( - and( - eq(chatActions.companyId, input.companyId), - eq(chatActions.kind, "inbound_wakeup"), - sql`${chatActions.payload}->>'issueId' = ${issue.id}`, - inArray( - sql`${chatActions.payload}->>'commentId'`, - commentIds, - ), - ), - ); - if (sources.length !== commentIds.length) throw failedChatRetryDenied(); - const ordered = commentIds.map( - (id) => sources.find((action) => action.payload.commentId === id)!, - ); - const first = ordered[0]!; - if ( - !first || - !first.principalId || - !first.conversationId || - sources.some( - (action) => - action.endpointId !== first.endpointId || - action.conversationId !== first.conversationId || - action.principalId !== first.principalId || - action.payload.requestedByActorType !== - first.payload.requestedByActorType || - action.payload.requestedByActorId !== - first.payload.requestedByActorId, - ) - ) - throw failedChatRetryDenied(); - const identity = await tx.execute( - sql`select pg_try_advisory_xact_lock(hashtextextended(${`chat-identity:${input.companyId}:${first.principalId}`}, 0)) as locked`, - ); - if (!identity[0]?.locked) - throw new NativeChatReviewPresentationContentionError(); - const canonicalSources: Array> = []; - let destination: Awaited> | null = - null; - for (const action of ordered) { - destination = await authorizeInboundWakeup(tx, action, { - nonblocking: true, - terminal: input.publication === true, - }).catch((error) => { - if (isExternalActionAuthorizationChange(error)) - throw failedChatRetryDenied(); - throw error; - }); - const [delivery] = await tx - .select() - .from(chatDeliveries) - .where(eq(chatDeliveries.id, action.deliveryId!)) - .for("update", { noWait: true }); - const [current] = await tx - .select() - .from(chatActions) - .where(eq(chatActions.id, action.id)) - .for("update", { noWait: true }); - const [comment] = await tx - .select() - .from(issueComments) - .where( - and( - eq(issueComments.id, String(action.payload.commentId)), - eq(issueComments.companyId, input.companyId), - eq(issueComments.issueId, issue.id), - ), - ) - .for("share", { noWait: true }); - if ( - !current || - current.status !== "processed" || - inboundWakeScope(current) !== inboundWakeScope(action) || - !delivery || - delivery.state !== "processed" || - !comment || - comment.deletedAt || - comment.updatedAt.getTime() !== comment.createdAt.getTime() || - comment.updatedAt > - (input.committedResponse - ? accepted!.createdAt - : failedRun.finishedAt!) || - destination.endpoint.provider !== provider || - JSON.stringify(lifecycleRuntimeFence(delivery)) !== - JSON.stringify(lifecycleRuntimeFence(destination.delivery)) - ) - throw failedChatRetryDenied(); - const [lifecycle] = await tx - .select({ id: chatDeliveries.id }) - .from(chatDeliveries) - .where( - and( - eq(chatDeliveries.companyId, input.companyId), - eq(chatDeliveries.endpointId, first.endpointId), - or( - isNull(chatDeliveries.conversationId), - eq(chatDeliveries.conversationId, first.conversationId), - ), - ne(chatDeliveries.state, "filtered"), - sql`${chatDeliveries.normalizedEvent}->'runtimeContext' = ${JSON.stringify(lifecycleRuntimeFence(delivery))}::jsonb`, - inArray(chatDeliveries.eventKind, [ - "message_updated", - "message_deleted", - "message_restored", - ]), - sql`${chatDeliveries.normalizedEvent}->'message'->>'targetProviderEventId' = ${delivery.providerEventId}`, - ), - ) - .limit(1); - if (lifecycle) throw failedChatRetryDenied(); - const [receipt] = await tx - .select() - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, action.id)) - .for("share", { noWait: true }); - if (!receipt) throw failedChatRetryDenied(); - assertFailedRetryReceipt( - createDurableChatWakeupRequest({ - id: action.id, - companyId: input.companyId, - agentId: input.agentId, - issueId: issue.id, - commentId: comment.id, - requestedByActorType: action.payload.requestedByActorType as - | "user" - | "system", - requestedByActorId: String(action.payload.requestedByActorId), - requestedAt: action.createdAt, - authorize: async () => {}, - }), - receipt, - ); - let owner = receipt; - const seen = new Set(); - for ( - let depth = 0; - owner.payload?.coalescedIntoWakeupRequestId !== undefined; - depth++ - ) { - const ownerId = owner.payload.coalescedIntoWakeupRequestId; - // The scheduler writes a direct child -> canonical owner edge. Nested - // chains/cycles are not silently interpreted as additional authority. - if ( - depth >= 1 || - owner.status !== "coalesced" || - typeof ownerId !== "string" || - !isUuidLike(ownerId) || - seen.has(ownerId) - ) - throw failedChatRetryDenied(); - seen.add(ownerId); - const [next] = await tx - .select() - .from(agentWakeupRequests) - .where( - and( - eq(agentWakeupRequests.id, ownerId), - eq(agentWakeupRequests.companyId, input.companyId), - ), - ) - .for("share", { noWait: true }); - if (!next) throw failedChatRetryDenied(); - owner = next; - } - if ( - owner.id !== failedRun.wakeupRequestId || - owner.runId !== failedRun.id || - owner.agentId !== input.agentId || - owner.requestedByActorType !== receipt.requestedByActorType || - owner.requestedByActorId !== receipt.requestedByActorId || - ["skipped", "cancelled"].includes(receipt.status) - ) - throw failedChatRetryDenied(); - canonicalSources.push({ - actionId: action.id, - deliveryId: delivery.id, - commentId: comment.id, - principalId: action.principalId, - scope: inboundWakeScope(action), - fence: lifecycleRuntimeFence(delivery), - bodySha256: createHash("sha256").update(comment.body).digest("hex"), - omissionReasons: action.payload.attachmentOmissionReasons ?? {}, - receiptId: receipt.id, - ownerId: owner.id, - }); - } - if (!destination) throw failedChatRetryDenied(); - const newestSource = ordered.reduce( - (latest, action) => - action.createdAt > latest ? action.createdAt : latest, - first.createdAt, - ); - const [newer] = await tx - .select({ id: chatActions.id }) - .from(chatActions) - .where( - and( - eq(chatActions.companyId, input.companyId), - eq(chatActions.conversationId, first.conversationId), - eq(chatActions.kind, "inbound_wakeup"), - notInArray( - chatActions.id, - ordered.map((action) => action.id), - ), - gte(chatActions.createdAt, newestSource), - sql`exists (select 1 from agent_wakeup_requests newer_admission where newer_admission.id = ${chatActions.id} - and newer_admission.company_id = ${input.companyId}::uuid and newer_admission.status <> 'skipped')`, - ), - ) - .limit(1); - if (newer && !input.publication) throw failedChatRetryDenied(); - const scope = { - failedRunId: failedRun.id, - sourceWakeupRequestId: failedRun.wakeupRequestId, - issueId: issue.id, - agentId: input.agentId, - taskKey: context.taskKey as string, - provider, - commentIds, - sources: canonicalSources, - endpointId: first.endpointId, - conversationId: first.conversationId, - principalId: first.principalId, - sessionGeneration: destination.conversation.sessionGeneration, - requestedByActorType: first.payload.requestedByActorType as - | "user" - | "system", - requestedByActorId: String(first.payload.requestedByActorId), - retryAncestors: [], - }; - return hashFailedRetrySource(scope); - } - - const unregisterCommittedResponseAuthority = - registerCommittedChatResponseAuthority(db, async (tx, input) => { - try { - await failedChatRetrySource(tx, { - ...input, - failedRunId: input.runId, - publication: true, - committedResponse: { resultId: input.resultId }, - }); - } catch (error) { - if ( - error instanceof HttpError && - (error.details as { code?: string } | undefined)?.code === - "chat_failed_run_retry_not_authorized" - ) { - throw new CommittedChatResponseAuthorizationError(); - } - throw error; - } - }); - - async function prepareFailedChatRunRetry( - tx: DbOrTransaction, - input: { - companyId: string; - issueId: string; - agentId: string; - failedRunId: string; - initiatedByUserId: string; - }, - ): Promise<{ actionId: string; issueId: string }> { - const source = await failedChatRetrySource(tx, input); - const providerActionId = `failed-run-retry:${source.failedRunId}`; - const [existing] = await tx - .select() - .from(chatActions) - .where( - and( - eq(chatActions.endpointId, source.endpointId), - eq(chatActions.providerActionId, providerActionId), - ), - ) - .for("update", { noWait: true }); - if (existing) { - if ( - existing.kind !== "failed_run_retry" || - existing.companyId !== input.companyId || - existing.payload.sourceScopeSha256 !== source.sourceScopeSha256 - ) - throw failedChatRetryDenied(); - return { actionId: existing.id, issueId: input.issueId }; - } - await assertFailedNativeRetryState(tx, source, input.companyId); - const [busy] = await tx - .select({ id: heartbeatRuns.id }) - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.companyId, input.companyId), - inArray(heartbeatRuns.status, [ - "queued", - "running", - "scheduled_retry", - ]), - sql`coalesce(${heartbeatRuns.contextSnapshot}->>'issueId', ${heartbeatRuns.contextSnapshot}->>'taskId') = ${input.issueId}`, - ), - ) - .limit(1); - if (busy || !input.initiatedByUserId) throw failedChatRetryDenied(); - const now = new Date(); - const [intent] = await tx - .insert(chatActions) - .values({ - companyId: input.companyId, - endpointId: source.endpointId, - conversationId: source.conversationId, - principalId: source.principalId, - kind: "failed_run_retry", - providerActionId, - payload: { - version: 1, - ...source, - initiatedByUserId: input.initiatedByUserId, - }, - status: "issued", - result: { code: "failed_chat_retry_queued", attemptCount: 0 }, - createdAt: now, - updatedAt: now, - }) - .returning(); - await logActivity(tx as unknown as Db, { - companyId: input.companyId, - actorType: "user", - actorId: input.initiatedByUserId, - action: "chat.failed_run_retry_requested", - entityType: "heartbeat_run", - entityId: input.failedRunId, - details: { - retryActionId: intent.id, - issueId: input.issueId, - commentCount: source.commentIds.length, - }, - }); - return { actionId: intent.id, issueId: input.issueId }; - } - - async function authorizeFailedChatRunRetry( - tx: DbOrTransaction, - input: { - companyId: string; - issueId: string; - agentId: string; - retryActionId: string; - runId?: string; - publication?: boolean; - }, - ) { - const [snapshot] = await tx - .select() - .from(chatActions) - .where( - and( - eq(chatActions.id, input.retryActionId), - eq(chatActions.companyId, input.companyId), - ), - ); - if ( - !snapshot || - snapshot.kind !== "failed_run_retry" || - typeof snapshot.payload.failedRunId !== "string" || - snapshot.payload.issueId !== input.issueId || - snapshot.payload.agentId !== input.agentId - ) - throw failedChatRetryDenied(); - const source = await failedChatRetrySource(tx, { - ...input, - failedRunId: snapshot.payload.failedRunId, - }); - const [intent] = await tx - .select() - .from(chatActions) - .where(eq(chatActions.id, snapshot.id)) - .for("update", { noWait: true }); - if ( - !intent || - !( - input.publication - ? ["issued", "processing", "processed", "failed", "cancelled"] - : ["issued", "processing", "processed"] - ).includes(intent.status) || - intent.payload.version !== 1 || - intent.endpointId !== source.endpointId || - intent.conversationId !== source.conversationId || - intent.principalId !== source.principalId || - intent.payload.sourceScopeSha256 !== source.sourceScopeSha256 || - JSON.stringify(intent.payload) !== JSON.stringify(snapshot.payload) - ) - throw failedChatRetryDenied(); - const [receipt] = await tx - .select() - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, intent.id)) - .for("share", { noWait: true }); - if (receipt) { - assertFailedRetryReceipt( - createDurableChatWakeupRequest({ - id: intent.id, - companyId: input.companyId, - agentId: input.agentId, - issueId: input.issueId, - commentId: source.commentIds.at(-1)!, - requestedByActorType: source.requestedByActorType, - requestedByActorId: source.requestedByActorId, - requestedAt: intent.createdAt, - authorize: async () => {}, - }), - receipt, - ); - if ( - (!input.publication && - ["skipped", "cancelled"].includes(receipt.status)) || - (input.runId && receipt.runId !== input.runId) - ) - throw failedChatRetryDenied(); - } else if (input.runId) throw failedChatRetryDenied(); - if (input.runId) { - const [run] = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, input.runId), - eq(heartbeatRuns.companyId, input.companyId), - eq(heartbeatRuns.agentId, input.agentId), - ), - ) - .for("share", { noWait: true }); - if ( - !run || - run.wakeupRequestId !== intent.id || - run.retryOfRunId !== source.failedRunId - ) - throw failedChatRetryDenied(); - } - return { ...source, intent }; - } - - const unregisterFailedRetryAuthority = registerFailedChatRunRetryAuthority( - db, - async (tx, input) => { - const source = await authorizeFailedChatRunRetry(tx, { - ...input, - retryActionId: input.wakeupRequestId, - }); - await assertFailedNativeRetryState(tx, source, input.companyId); - const context = input.contextSnapshot; - const hint = context.chatFailedRunRetry as - | Record - | undefined; - if ( - context.issueId !== source.issueId || - context.source !== `chat:${source.provider}` || - context.taskKey !== source.taskKey || - context.retryOfRunId !== source.failedRunId || - JSON.stringify(retryCommentIds(context)) !== - JSON.stringify(source.commentIds) || - hint?.version !== 1 || - hint.actionId !== source.intent.id || - hint.failedRunId !== source.failedRunId || - !Array.isArray(context.externalAttachmentOmissions) || - JSON.stringify( - context.externalAttachmentOmissions.map((entry) => - entry && typeof entry === "object" - ? [entry.commentId, entry.reasons] - : null, - ), - ) !== - JSON.stringify( - source.sources.map((entry) => [ - entry.commentId, - entry.omissionReasons, - ]), - ) - ) - throw failedChatRetryDenied(); - }, - ); - - async function failedChatRetryReceipt( - action: typeof chatActions.$inferSelect, - ) { - const [receipt] = await db - .select() - .from(agentWakeupRequests) - .where( - and( - eq(agentWakeupRequests.id, action.id), - eq(agentWakeupRequests.companyId, action.companyId), - ), - ); - const [run] = receipt?.runId - ? await db - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, receipt.runId), - eq(heartbeatRuns.companyId, action.companyId), - ), - ) - : []; - const status = ["succeeded", "failed", "cancelled"].includes( - run?.status ?? "", - ) - ? (run!.status as "succeeded" | "failed" | "cancelled") - : run?.status === "timed_out" - ? "failed" - : run?.status === "running" - ? "running" - : receipt?.status === "deferred_issue_execution" - ? "deferred" - : ["failed", "cancelled"].includes(action.status) - ? (action.status as "failed" | "cancelled") - : ["skipped", "failed", "cancelled"].includes( - receipt?.status ?? "", - ) - ? "failed" - : "queued"; - return { - actionId: action.id, - issueId: String(action.payload.issueId), - runId: run?.id ?? null, - status, - }; - } - - async function claimFailedChatRunRetry(actionId: string) { - const [candidate] = await db - .select() - .from(chatActions) - .where( - and( - eq(chatActions.id, actionId), - eq(chatActions.kind, "failed_run_retry"), - ), - ); - if (!candidate) throw notFound("Chat retry request not found"); - const now = new Date(); - if ( - shuttingDown || - !["issued", "processing"].includes(candidate.status) || - (candidate.status === "processing" && - candidate.updatedAt.getTime() > - now.getTime() - DELIVERY_PROCESSING_STALE_MS) || - (typeof candidate.result?.retryAt === "string" && - Date.parse(candidate.result.retryAt) > now.getTime()) - ) - return failedChatRetryReceipt(candidate); - const [claimed] = await db - .update(chatActions) - .set({ status: "processing", updatedAt: now }) - .where( - and( - eq(chatActions.id, candidate.id), - eq(chatActions.status, candidate.status), - eq(chatActions.updatedAt, candidate.updatedAt), - ), - ) - .returning(); - if (!claimed) return failedChatRetryReceipt(candidate); - const attemptCount = Number(candidate.result?.attemptCount ?? 0) + 1; - const settle = async (status: string, result: Record) => { - const [current] = await db - .update(chatActions) - .set({ status, result, updatedAt: new Date() }) - .where( - and( - eq(chatActions.id, claimed.id), - eq(chatActions.status, "processing"), - eq(chatActions.updatedAt, now), - ), - ) - .returning(); - return failedChatRetryReceipt(current ?? claimed); - }; - try { - const scope = { - companyId: claimed.companyId, - issueId: String(claimed.payload.issueId), - agentId: String(claimed.payload.agentId), - retryActionId: claimed.id, - }; - const source = await db.transaction((tx) => - authorizeFailedChatRunRetry(tx, scope), - ); - const existing = await db - .select({ id: agentWakeupRequests.id }) - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, claimed.id)); - if (!existing.length) { - const context = { - issueId: source.issueId, - taskKey: source.taskKey, - source: `chat:${source.provider}`, - wakeCommentId: source.commentIds.at(-1)!, - wakeCommentIds: source.commentIds, - retryOfRunId: source.failedRunId, - chatFailedRunRetry: { - version: 1, - actionId: claimed.id, - failedRunId: source.failedRunId, - }, - externalAttachmentOmissions: source.sources.map((entry) => ({ - commentId: entry.commentId, - reasons: entry.omissionReasons, - })), - }; - await options.heartbeat.wakeup(source.agentId, { - source: "on_demand", - triggerDetail: "manual", - reason: "retry_failed_run", - payload: context, - contextSnapshot: context, - allowRunCoalescing: false, - requestedByActorType: source.requestedByActorType, - requestedByActorId: source.requestedByActorId, - durableChatRequest: createDurableChatWakeupRequest({ - id: claimed.id, - companyId: claimed.companyId, - agentId: source.agentId, - issueId: source.issueId, - commentId: source.commentIds.at(-1)!, - requestedByActorType: source.requestedByActorType, - requestedByActorId: source.requestedByActorId, - requestedAt: claimed.createdAt, - failedRunRetry: { failedRunId: source.failedRunId }, - authorize: async (tx) => { - const current = await authorizeFailedChatRunRetry(tx, scope); - await assertFailedNativeRetryState( - tx, - current, - claimed.companyId, - ); - }, - }), - }); - } - const [receipt] = await db - .select() - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, claimed.id)); - if (!receipt) throw new Error("chat_retry_receipt_not_committed"); - return settle( - ["skipped", "failed", "cancelled"].includes(receipt.status) - ? "failed" - : "processed", - { code: "failed_chat_retry_durable", attemptCount }, - ); - } catch (error) { - const [receipt] = await db - .select() - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, claimed.id)); - if (receipt) - return settle("processed", { - code: "failed_chat_retry_durable", - attemptCount, - }); - const denied = - error && - typeof error === "object" && - "details" in error && - error.details && - typeof error.details === "object" && - "code" in error.details && - [ - "chat_failed_run_retry_not_authorized", - "chat_action_authorization_changed", - ].includes(String(error.details.code)); - const retryable = !denied && attemptCount < 8; - return settle(retryable ? "issued" : "failed", { - code: retryable - ? "failed_chat_retry_contended" - : "failed_chat_retry_not_authorized", - attemptCount, - ...(retryable - ? { - retryAt: new Date( - Date.now() + Math.min(5_000, attemptCount * 250), - ).toISOString(), - } - : {}), - }); - } - } - - const failedRetryTasks = new Map< - string, - ReturnType - >(); - function processFailedChatRunRetry(actionId: string) { - const existing = failedRetryTasks.get(actionId); - if (existing) return existing; - const task = claimFailedChatRunRetry(actionId); - failedRetryTasks.set(actionId, task); - void task.then( - () => failedRetryTasks.delete(actionId), - () => failedRetryTasks.delete(actionId), - ); - return task; - } - - async function processFailedChatRunRetries(limit = 20) { - if (shuttingDown) return 0; - const now = new Date(); - const candidates = await db - .select({ id: chatActions.id }) - .from(chatActions) - .where( - and( - eq(chatActions.kind, "failed_run_retry"), - or( - eq(chatActions.status, "issued"), - and( - eq(chatActions.status, "processing"), - lte( - chatActions.updatedAt, - new Date(now.getTime() - DELIVERY_PROCESSING_STALE_MS), - ), - ), - ), - sql`(${chatActions.result}->>'retryAt' is null or (${chatActions.result}->>'retryAt')::timestamptz <= ${now.toISOString()}::timestamptz)`, - ), - ) - .orderBy(asc(chatActions.createdAt), asc(chatActions.id)) - .limit(Math.max(1, Math.min(50, limit))); - for (const candidate of candidates) - await processFailedChatRunRetry(candidate.id); - return candidates.length; - } - - let failedRetryNoticeCursor: { createdAt: Date; id: string } | null = null; - async function enqueueFailedChatRetryPublications( - limit: number, - ): Promise { - const cursor = failedRetryNoticeCursor; - const batchSize = Math.max(1, Math.min(50, limit)); - const candidates = await db - .select({ action: chatActions, receipt: agentWakeupRequests }) - .from(chatActions) - .innerJoin( - agentWakeupRequests, - and( - eq(agentWakeupRequests.id, chatActions.id), - eq(agentWakeupRequests.companyId, chatActions.companyId), - ), - ) - .where( - and( - eq(chatActions.kind, "failed_run_retry"), - isNull(agentWakeupRequests.runId), - cursor - ? sql`(${chatActions.createdAt}, ${chatActions.id}) > (${cursor.createdAt.toISOString()}::timestamptz, ${cursor.id}::uuid)` - : undefined, - inArray(agentWakeupRequests.status, [ - "deferred_issue_execution", - "cancelled", - "failed", - "skipped", - ]), - sql`not exists (select 1 from chat_publications retry_notice where retry_notice.company_id = ${chatActions.companyId} - and retry_notice.idempotency_key = 'wake:' || ${chatActions.id}::text || ':' || - case when ${agentWakeupRequests.status} = 'deferred_issue_execution' then 'queued' else 'not_started' end || ':' || - ${chatActions.endpointId}::text || ':' || ${chatActions.conversationId}::text)`, - ), - ) - .orderBy(asc(chatActions.createdAt), asc(chatActions.id)) - .limit(batchSize); - const last = candidates.at(-1)?.action; - failedRetryNoticeCursor = - candidates.length >= batchSize && last - ? { createdAt: last.createdAt, id: last.id } - : null; - let inserted = 0; - for (const { action, receipt } of candidates) { - try { - inserted += await db.transaction(async (tx) => { - const source = await authorizeFailedChatRunRetry(tx, { - companyId: action.companyId, - issueId: String(action.payload.issueId), - agentId: String(action.payload.agentId), - retryActionId: action.id, - publication: true, - }); - const [current] = await tx - .select() - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, receipt.id)) - .for("update", { noWait: true }); - if (!current || current.runId || current.status !== receipt.status) - return 0; - const state = - current.status === "deferred_issue_execution" - ? "queued" - : "not_started"; - await notifyChatPublicationWork(tx); - return ( - await tx - .insert(chatPublications) - .values({ - companyId: action.companyId, - endpointId: source.endpointId, - conversationId: source.conversationId, - issueId: source.issueId, - commentId: source.commentIds.at(-1)!, - idempotencyKey: inboundWakePublicationKey( - action.id, - state, - source.endpointId, - source.conversationId, - ), - payload: projectSafeChatPublication({ - classification: "external", - source: "safe_milestone", - text: - state === "queued" - ? "Your retry is queued." - : "This retry was not started. Open the task in Paperclip for details.", - progressState: state === "queued" ? "queued" : "failed", - }), - state: "pending", - }) - .onConflictDoNothing() - .returning({ id: chatPublications.id }) - ).length; - }); - } catch (error) { - if ( - isExternalChatWaitAuthorizationContention(error) || - error instanceof NativeChatReviewPresentationContentionError || - isExternalActionAuthorizationChange(error) || - (error && - typeof error === "object" && - "status" in error && - error.status === 409) - ) - continue; - throw error; - } - } - return inserted; - } - + const { + retryCommentIds, + assertFailedRetryReceipt, + FailedChatRetryPublicationReadError, + failedChatRetrySource, + unregisterCommittedResponseAuthority, + prepareFailedChatRunRetry, + authorizeFailedChatRunRetry, + unregisterFailedRetryAuthority, + processFailedChatRunRetry, + processFailedChatRunRetries, + enqueueFailedChatRetryPublications, + pendingFailedRetries, + } = createChatFailedRunRetry(db, { + authorizeInboundWakeup, + isExternalActionAuthorizationChange, + inboundWakeScope, + lifecycleRuntimeFence, + isShuttingDown: () => shuttingDown, + wakeup: (...args) => options.heartbeat.wakeup(...args), + staleProcessingMs: DELIVERY_PROCESSING_STALE_MS, + }); async function authorizeRetainedChatSourcePublication( tx: DbOrTransaction, publication: typeof chatPublications.$inferSelect, @@ -36349,7 +34562,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { getIssueBinding, shutdown: async () => { shuttingDown = true; - await Promise.allSettled([...failedRetryTasks.values()]); + await Promise.allSettled(pendingFailedRetries()); unregisterFailedRetryAuthority(); unregisterSlackTaskAuthority(); voice.unregisterAgentTools(); diff --git a/server/src/services/chat-channels/failed-run-retry.ts b/server/src/services/chat-channels/failed-run-retry.ts new file mode 100644 index 00000000000..c0b8d43988f --- /dev/null +++ b/server/src/services/chat-channels/failed-run-retry.ts @@ -0,0 +1,1825 @@ +import { continuingTeamsFilePublication, type ContinuingTeamsFileTransfer } from "./microsoft-teams/file-transfer-source.js"; +import { notifyChatPublicationWork } from "../chat-work-notifications.js"; + +import { + createHash, +} from "node:crypto"; +import { + and, + asc, + eq, + gte, + inArray, + isNotNull, + isNull, + like, + lte, + ne, + notInArray, + or, + sql, +} from "drizzle-orm"; + +import type { Db } from "@paperclipai/db"; +import { + createDurableChatWakeupRequest, + assertDurableChatWakeupReceipt, + registerFailedChatRunRetryAuthority, + registerCommittedChatResponseAuthority, + CommittedChatResponseAuthorizationError, +} from "../durable-chat-wakeup.js"; +import { + agentWakeupRequests, + agents, + chatActions, + chatConversations, + chatDeliveries, + chatEndpoints, + chatPublications, + heartbeatRunEvents, + heartbeatRuns, + nativeRunFinalizations, + nativeRunResults, + environmentLeases, + issueComments, + issueThreadInteractions, + issueQuestionResponseDeliveries, + issues, +} from "@paperclipai/db"; +import type { + ChatProvider, +} from "@paperclipai/shared"; +import { + isAgentStatusInvokable, + CHAT_PROVIDERS, + isUuidLike, +} from "@paperclipai/shared"; + +import { + nativeFailedRunRetryStateIsSafe, + nativePreProviderRetryAfterCleanupStateIsSafe, + nativeProviderRecoveryEvidence, +} from "../native-runtime/native-session-executor.js"; + +import { + HttpError, + conflict, + notFound, +} from "../../errors.js"; + +import { + logActivity, +} from "../activity-log.js"; +import { + type IssueAssignmentWakeupDeps, +} from "../issue-assignment-wakeup.js"; + +import { + NativeChatReviewPresentationContentionError, +} from "../native-runtime/native-chat-review-presentation.js"; +import { isExternalChatWaitAuthorizationContention } from "../native-runtime/chat-attachment-reuse.js"; +import { projectSafeChatPublication } from "../chat-publication-projection.js"; + +import { + inboundWakePublicationKey, +} from "../chat-inbound-wakeup-publications.js"; + +type DbOrTransaction = Db | Parameters[0]>[0]; +type DeliveryRow = typeof chatDeliveries.$inferSelect; +type RetryRuntimeFence = { credentialFingerprint: string; generation: number }; +export type ChatFailedRetrySource = Awaited["failedChatRetrySource"]>>; + +type ChatFailedRunRetryOptions = { + authorizeInboundWakeup: ( + database: DbOrTransaction, + action: typeof chatActions.$inferSelect, + notice?: { nonblocking: true; terminal: boolean }, + ) => Promise<{ + conversation: Pick; + endpoint: Pick; + delivery: DeliveryRow; + }>; + isExternalActionAuthorizationChange: (error: unknown) => boolean; + inboundWakeScope: (action: typeof chatActions.$inferSelect) => string; + lifecycleRuntimeFence: (delivery: DeliveryRow) => RetryRuntimeFence | null; + isShuttingDown: () => boolean; + wakeup: IssueAssignmentWakeupDeps["wakeup"]; + staleProcessingMs: number; +}; + +export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOptions) { + const { + authorizeInboundWakeup, + isExternalActionAuthorizationChange, + inboundWakeScope, + lifecycleRuntimeFence, + isShuttingDown, + wakeup, + staleProcessingMs: DELIVERY_PROCESSING_STALE_MS, + } = options; + + + const failedChatRetryDenied = () => + conflict( + "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.", + { code: "chat_failed_run_retry_not_authorized" }, + ); + class FailedChatRetryPublicationReadError extends Error { + constructor() { + super("Chat retry publication authorization is temporarily unavailable"); + } + } + + function retryCommentIds(context: Record): string[] { + const ids = + context.wakeCommentIds ?? + (context.wakeCommentId ? [context.wakeCommentId] : []); + if ( + !Array.isArray(ids) || + ids.length === 0 || + ids.length > 50 || + ids.some((id) => typeof id !== "string" || !isUuidLike(id)) || + new Set(ids).size !== ids.length || + (context.wakeCommentId !== undefined && + context.wakeCommentId !== ids.at(-1)) + ) + throw failedChatRetryDenied(); + return ids as string[]; + } + + type FailedChatRetrySource = { + failedRunId: string; + sourceWakeupRequestId: string; + issueId: string; + agentId: string; + taskKey: string; + provider: ChatProvider; + commentIds: string[]; + sources: Array>; + endpointId: string; + conversationId: string; + principalId: string; + sessionGeneration: number; + requestedByActorType: "user" | "system"; + requestedByActorId: string; + retryAncestors: Array<{ + actionId: string; + failedRunId: string; + sourceScopeSha256: string; + }>; + sourceScopeSha256: string; + }; + function hashFailedRetrySource( + scope: Omit, + ): FailedChatRetrySource { + return { + ...scope, + sourceScopeSha256: createHash("sha256") + .update(JSON.stringify(scope)) + .digest("hex"), + }; + } + + function assertFailedRetryReceipt( + ...args: Parameters + ) { + try { + assertDurableChatWakeupReceipt(...args); + } catch { + throw failedChatRetryDenied(); + } + } + + /** The reviewed-chat ownership check runs before runtime/profile resolution + * or provider startup. Recognize only that exact persisted failure together + * with its sole system diagnostic and absence of execution evidence. A + * generic setup error (or a prior-session display hint) is not this proof. */ + async function isPreProviderReviewedChatFailure( + tx: DbOrTransaction, + run: typeof heartbeatRuns.$inferSelect, + ): Promise { + const diagnostic = "reviewed_chat_execution_binding_not_authorized"; + if ( + run.runtimeMode !== "legacy" || + run.errorCode !== "setup_failed" || + run.error !== diagnostic || + [ + run.runtimeModeResolverVersion, + run.runtimeModeReason, + run.runtimeModeResolvedAt, + run.runnerProfileJson, + run.runnerInstanceId, + run.nativeSessionId, + run.nativeIssueId, + run.nativePhase, + run.driverKind, + run.driverVersion, + run.completionContractId, + run.completionContractSha256, + run.sessionIdAfter, + run.externalRunId, + run.processPid, + run.processGroupId, + run.processStartedAt, + run.logStore, + run.logRef, + run.logBytes, + run.logSha256, + run.stdoutExcerpt, + run.stderrExcerpt, + run.lastOutputAt, + run.lastOutputStream, + run.lastOutputBytes, + run.usageJson, + run.exitCode, + run.signal, + ].some((value) => value !== null) || + run.lastOutputSeq !== 0 || + run.logCompressed + ) + return false; + const events = await tx + .select() + .from(heartbeatRunEvents) + .where( + and( + eq(heartbeatRunEvents.companyId, run.companyId), + eq(heartbeatRunEvents.runId, run.id), + ), + ) + .limit(2) + .for("share", { noWait: true }); + const event = events[0]; + if ( + events.length !== 1 || + !event || + event.agentId !== run.agentId || + event.seq !== 1 || + event.eventType !== "error" || + event.stream !== "system" || + event.level !== "error" || + event.message !== diagnostic || + event.payload !== null || + event.sourceInstanceId !== null || + event.sourceEventId !== null || + event.sourceSeq !== null || + event.sourcePayloadSha256 !== null || + event.protocolSchemaVersion !== null + ) + return false; + const evidence = await tx.execute(sql`select 1 where + exists (select 1 from native_run_finalizations where company_id = ${run.companyId}::uuid and run_id = ${run.id}::uuid) + or exists (select 1 from native_run_results where company_id = ${run.companyId}::uuid and run_id = ${run.id}::uuid) + or exists (select 1 from environment_leases where company_id = ${run.companyId}::uuid and heartbeat_run_id = ${run.id}::uuid)`); + return evidence.length === 0; + } + + async function failedNativeRetryCoordinator( + tx: DbOrTransaction, + run: typeof heartbeatRuns.$inferSelect, + issueId: string, + ) { + const [coordinator] = await tx + .select() + .from(nativeRunFinalizations) + .where( + and( + eq(nativeRunFinalizations.runId, run.id), + eq(nativeRunFinalizations.companyId, run.companyId), + eq(nativeRunFinalizations.issueId, issueId), + ), + ) + .for("share", { noWait: true }); + // A queued request can fail before its first provider event because the + // predecessor's exact retained checkpoint has not yet been suspended. + // A later authenticated cleanup receipt can authorize retrying that + // request; never pretend it exhausted provider execution attempts. + if ( + coordinator && + run.nativeIssueId === issueId && + run.status === "failed" && + run.errorCode === "adapter_failed" && + // A diagnostic reason does not change this failure category. The exact + // checkpoint, cleanup receipt, and no-provider-work proofs below still + // decide whether the original request can be retried. + (run.error === "runner_state_identity_mismatch" || + run.error?.startsWith("runner_state_identity_mismatch: ")) && + run.nativePhase === "observed" && + coordinator.phase === "observed" && + coordinator.attempt === 0 && + !coordinator.leaseOwner && + !coordinator.leaseExpiresAt && + !coordinator.nextAttemptAt && + !coordinator.resultId && + !coordinator.assessmentId && + !coordinator.decisionId && + run.nativeSessionId && + run.processPid && + run.processGroupId === run.processPid + ) { + const [event] = await tx + .select({ id: heartbeatRunEvents.id }) + .from(heartbeatRunEvents) + .where( + and( + eq(heartbeatRunEvents.companyId, run.companyId), + eq(heartbeatRunEvents.runId, run.id), + or( + isNotNull(heartbeatRunEvents.sourceInstanceId), + sql`${heartbeatRunEvents.payload}->'prpEvent' is not null`, + ), + ), + ) + .limit(1); + const [result] = await tx + .select({ id: nativeRunResults.id }) + .from(nativeRunResults) + .where( + and( + eq(nativeRunResults.companyId, run.companyId), + eq(nativeRunResults.runId, run.id), + ), + ) + .limit(1); + const checkpoint = run.runnerProfileJson?.sessionCheckpoint as + | Record + | undefined; + const binding = checkpoint?.identity as + | Record + | undefined; + if ( + !event && + !result && + binding?.companyId === run.companyId && + binding?.issueId === issueId && + binding?.agentId === run.agentId && + binding?.runId === run.id && + binding?.sessionId === run.nativeSessionId && + typeof checkpoint?.sessionId === "string" && + checkpoint.sessionId.trim().length > 0 && + (checkpoint.providerSessionId == null || + (typeof checkpoint.providerSessionId === "string" && + checkpoint.providerSessionId.trim().length > 0)) + ) { + const predecessors = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.companyId, run.companyId), + eq(heartbeatRuns.agentId, run.agentId), + eq(heartbeatRuns.nativeIssueId, issueId), + eq(heartbeatRuns.nativeSessionId, run.nativeSessionId), + eq(heartbeatRuns.processPid, run.processPid), + eq(heartbeatRuns.processGroupId, run.processGroupId), + eq(heartbeatRuns.runtimeMode, "native"), + eq(heartbeatRuns.status, "succeeded"), + ne(heartbeatRuns.id, run.id), + // Warm runs can share a PID and session. Only a durable maintenance + // owner is a candidate; the physical check below must still prove + // that this exact run owns the canonical suspended checkpoint. + sql`exists (select 1 from native_run_finalizations retired + where retired.company_id = ${heartbeatRuns.companyId} + and retired.run_id = ${heartbeatRuns.id} + and retired.phase = 'committed' + and exists (select 1 from jsonb_array_elements(retired.recovery_history) receipt + where receipt->>'kind' = 'native_cleanup_maintenance' + and receipt->>'phase' = 'settled' and receipt->>'version' = '1'))`, + ), + ) + .for("share", { noWait: true }) + .limit(2); + if (predecessors.length === 1) { + const predecessor = predecessors[0]!; + const [retired] = await tx + .select() + .from(nativeRunFinalizations) + .where( + and( + eq(nativeRunFinalizations.companyId, run.companyId), + eq(nativeRunFinalizations.issueId, issueId), + eq(nativeRunFinalizations.runId, predecessor.id), + ), + ) + .for("share", { noWait: true }); + const receipt = retired?.recoveryHistory.findLast( + (entry) => entry.kind === "native_cleanup_maintenance", + ); + const leases = await tx + .select() + .from(environmentLeases) + .where( + and( + eq(environmentLeases.companyId, run.companyId), + eq(environmentLeases.heartbeatRunId, run.id), + ), + ) + .for("share", { noWait: true }); + if ( + retired?.phase === "committed" && + retired.resultId && + retired.assessmentId && + retired.decisionId && + !retired.leaseOwner && + !retired.leaseExpiresAt && + !retired.nextAttemptAt && + receipt?.version === 1 && + receipt.phase === "settled" && + typeof receipt.requestId === "string" && + receipt.requestId.length > 0 && + receipt.nativeSessionId === run.nativeSessionId && + receipt.runnerInstanceId === predecessor.runnerInstanceId && + // The cleanup receipt identifies the provider thread. The + // checkpoint's providerSessionId is the separate backend account. + receipt.providerSessionId === checkpoint.sessionId && + typeof receipt.settledFingerprint === "string" && + /^[a-f0-9]{64}$/.test(receipt.settledFingerprint) && + typeof receipt.sourceFingerprint === "string" && + /^[a-f0-9]{64}$/.test(receipt.sourceFingerprint) && + leases.every( + (lease) => + lease.provider === "local" && + lease.providerLeaseId === null && + lease.issueId === issueId && + ["failed", "released", "expired"].includes(lease.status) && + (lease.cleanupStatus === null || + lease.cleanupStatus === "success") && + lease.releasedAt !== null, + ) + ) { + return { + coordinator, + leases, + retiredOwner: { + predecessor, + receipt, + providerSessionId: checkpoint.sessionId, + providerBackendSessionId: checkpoint.providerSessionId ?? null, + }, + }; + } + } + } + } + const detail = coordinator?.failureDetail; + const ordinary = [ + "provider_process_exited", + "provider_stdout_closed", + "provider_process_output_closed", + "provider_process_status_failed", + "provider_initialize_timeout", + "provider_initialize_protocol_error", + "provider_request_timeout", + "provider_request_protocol_error", + "provider_transport_failed", + "native_runner_process_exited", + ]; + if ( + !coordinator || + run.nativeIssueId !== issueId || + run.nativePhase !== "terminal_failure" || + coordinator.phase !== "terminal_failure" || + coordinator.recoveryState !== "blocked" || + coordinator.failureCode !== "native_session_retry_exhausted" || + coordinator.attempt < 3 || + coordinator.leaseOwner || + coordinator.leaseExpiresAt || + coordinator.nextAttemptAt || + coordinator.resultId || + coordinator.assessmentId || + coordinator.decisionId || + !detail || + !ordinary.includes(String(detail.originalFailureCode)) || + !["bootstrap_retry", "exact_checkpoint_resume"].includes( + String(detail.recoveryMode), + ) || + ![ + "adapter_failed", + "native_session_retry_exhausted", + String(detail.originalFailureCode), + ].includes(run.errorCode ?? "") || + (detail.recoveryMode === "bootstrap_retry" && + (detail.providerSessionEstablished !== false || + detail.providerEventsExist !== false || + detail.checkpointExists !== false)) || + (detail.recoveryMode === "exact_checkpoint_resume" && + (detail.providerSessionEstablished !== true || + detail.checkpointExists !== true)) + ) + throw failedChatRetryDenied(); + const evidence = await nativeProviderRecoveryEvidence({ + db: tx as unknown as Db, + runId: run.id, + sourceFailureCode: detail.originalFailureCode as Parameters< + typeof nativeProviderRecoveryEvidence + >[0]["sourceFailureCode"], + }); + if ( + evidence.recoveryMode !== detail.recoveryMode || + evidence.providerSessionEstablished !== + detail.providerSessionEstablished || + evidence.providerEventsExist !== detail.providerEventsExist || + evidence.checkpointExists !== detail.checkpointExists + ) + throw failedChatRetryDenied(); + const leases = await tx + .select() + .from(environmentLeases) + .where( + and( + eq(environmentLeases.companyId, run.companyId), + eq(environmentLeases.heartbeatRunId, run.id), + ), + ) + .for("share", { noWait: true }); + if ( + leases.some( + (lease) => + !["released", "expired", "failed"].includes(lease.status) || + lease.cleanupStatus !== "success" || + !lease.releasedAt, + ) + ) + throw failedChatRetryDenied(); + return { coordinator, leases, retiredOwner: null }; + } + + async function assertFailedNativeRetryState( + tx: DbOrTransaction, + source: Pick, + companyId: string, + ) { + const [run] = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, source.failedRunId), + eq(heartbeatRuns.companyId, companyId), + eq(heartbeatRuns.agentId, source.agentId), + ), + ) + .for("share", { noWait: true }); + if (!run) throw failedChatRetryDenied(); + if (run.runtimeMode !== "native") return; + const { coordinator, leases, retiredOwner } = + await failedNativeRetryCoordinator(tx, run, source.issueId); + if (retiredOwner) { + if ( + !nativePreProviderRetryAfterCleanupStateIsSafe({ + failedExecution: run.runnerProfileJson?.nativeExecutionInput, + retiredExecution: + retiredOwner.predecessor.runnerProfileJson?.nativeExecutionInput, + companyId, + issueId: source.issueId, + agentId: source.agentId, + failedRunId: run.id, + retiredRunId: retiredOwner.predecessor.id, + nativeSessionId: run.nativeSessionId!, + runnerInstanceId: retiredOwner.predecessor.runnerInstanceId!, + providerSessionId: retiredOwner.providerSessionId, + providerBackendSessionId: retiredOwner.providerBackendSessionId, + processPid: run.processPid!, + processGroupId: run.processGroupId!, + receipt: retiredOwner.receipt, + }) + ) + throw failedChatRetryDenied(); + return; + } + const checkpoint = run.runnerProfileJson?.sessionCheckpoint as + | Record + | undefined; + if ( + !run.nativeSessionId || + !run.runnerInstanceId || + (checkpoint?.providerSessionId != null && + (typeof checkpoint.providerSessionId !== "string" || + checkpoint.providerSessionId.trim().length === 0)) || + !nativeFailedRunRetryStateIsSafe({ + execution: run.runnerProfileJson?.nativeExecutionInput, + companyId, + issueId: source.issueId, + agentId: source.agentId, + runId: run.id, + nativeSessionId: run.nativeSessionId, + runnerInstanceId: run.runnerInstanceId, + processPid: run.processPid, + processGroupId: run.processGroupId, + providerSessionId: + typeof checkpoint?.sessionId === "string" + ? checkpoint.sessionId + : null, + providerBackendSessionId: + typeof checkpoint?.providerSessionId === "string" + ? checkpoint.providerSessionId + : null, + recoveryMode: coordinator.failureDetail!.recoveryMode as + | "bootstrap_retry" + | "exact_checkpoint_resume", + allowVerifiedBackup: leases.length > 0, + }) + ) + throw failedChatRetryDenied(); + } + + /** Resolve only a complete, originally admitted direct-chat batch. A retry + * never turns caller JSON, task-description text, or a provider redelivery + * into new input authority. Answer continuations require their own proof. */ + async function failedChatRetrySource( + tx: DbOrTransaction, + input: { + companyId: string; + issueId: string; + agentId: string; + failedRunId: string; + retryActionId?: string; + runId?: string; + publication?: boolean; + /** Internal selector for already committed output, never retry admission. */ + committedResponse?: { + resultId: string; + canonicalSha256?: string; + decisionId?: string; + /** Internal source-proof continuation for one locked Teams transfer. + * Never permission to repeat its card/PUT or retry a model run. */ + continuingTeamsFileTransfer?: ContinuingTeamsFileTransfer; + }; + }, + visited = new Set(), + ): Promise { + if (visited.has(input.failedRunId) || visited.size >= 16) + throw failedChatRetryDenied(); + const ancestors = new Set([...visited, input.failedRunId]); + const [issue] = await tx + .select() + .from(issues) + .where( + and( + eq(issues.id, input.issueId), + eq(issues.companyId, input.companyId), + ), + ) + .for( + "update", + input.publication || input.retryActionId ? { noWait: true } : undefined, + ); + const [actor] = await tx + .select({ status: agents.status }) + .from(agents) + .where( + and( + eq(agents.id, input.agentId), + eq(agents.companyId, input.companyId), + ), + ) + .for("share", { noWait: true }); + const [failedRun] = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, input.failedRunId), + eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), + ), + ) + .for("share", { noWait: true }); + if ( + !issue || + issue.assigneeAgentId !== input.agentId || + !actor || + !isAgentStatusInvokable(actor.status) || + !failedRun || + !failedRun.wakeupRequestId || + (!input.committedResponse && + (!["failed", "timed_out"].includes(failedRun.status) || + !failedRun.finishedAt)) || + (!input.committedResponse && + failedRun.runtimeMode !== "native" && + !( + [ + "adapter_failed", + "adapter_exit_code", + "process_exit", + "timeout", + ].includes(failedRun.errorCode ?? "") || + (await isPreProviderReviewedChatFailure(tx, failedRun)) + )) || + (!input.publication && + ["backlog", "done", "cancelled"].includes(issue.status)) + ) + throw failedChatRetryDenied(); + if (failedRun.runtimeMode === "native" && !input.committedResponse) + await failedNativeRetryCoordinator(tx, failedRun, issue.id); + const context = failedRun.contextSnapshot ?? {}; + const provider = CHAT_PROVIDERS.find( + (value) => context.source === `chat:${value}`, + ); + if ( + !provider || + context.issueId !== issue.id || + context.interactionId || + (context.taskKey !== issue.identifier && context.taskKey !== issue.id) + ) + throw failedChatRetryDenied(); + const commentIds = retryCommentIds(context); + const [accepted] = await tx + .select() + .from(nativeRunResults) + .where( + and( + eq(nativeRunResults.companyId, input.companyId), + eq(nativeRunResults.runId, failedRun.id), + eq(nativeRunResults.schemaStatus, "accepted"), + input.committedResponse + ? eq(nativeRunResults.id, input.committedResponse.resultId) + : undefined, + ), + ) + .limit(1) + .for("share", { noWait: true }); + if (input.committedResponse) { + const [coordinator] = await tx + .select() + .from(nativeRunFinalizations) + .where( + and( + eq(nativeRunFinalizations.companyId, input.companyId), + eq(nativeRunFinalizations.issueId, issue.id), + eq(nativeRunFinalizations.runId, failedRun.id), + ), + ) + .for("share", { noWait: true }); + const result = accepted?.resultJson.result as + | Record + | undefined; + const terminal = accepted?.resultJson.terminal as + | Record + | undefined; + const continuation = result?.continuation as + | Record + | undefined; + if ( + failedRun.runtimeMode !== "native" || + failedRun.nativeIssueId !== issue.id || + !accepted || + accepted.id !== input.committedResponse.resultId || + accepted.issueId !== issue.id || + accepted.completionContractId !== failedRun.completionContractId || + coordinator?.phase !== "committed" || + coordinator.resultId !== accepted.id || + !coordinator.decisionId || + !coordinator.assessmentId || + (input.committedResponse.canonicalSha256 !== undefined && + input.committedResponse.canonicalSha256 !== + accepted.canonicalSha256) || + (input.committedResponse.decisionId !== undefined && + input.committedResponse.decisionId !== coordinator.decisionId) || + result?.schema !== "paperclip.run_result.v1" || + result.reportedWorkDisposition !== "yielded" || + !Array.isArray(result.attentionRequests) || + result.attentionRequests.length !== 0 || + continuation?.kind !== "response_wake" || + terminal?.schema !== "paperclip.prp.terminal.v1" || + terminal.turnTerminalState !== "completed" || + terminal.runTerminalState !== "succeeded" || + terminal.reportedWorkDisposition !== "yielded" + ) + throw failedChatRetryDenied(); + } + let continuingUnknownPublicationId: string | undefined; + const continuingTransfer = + input.committedResponse?.continuingTeamsFileTransfer; + if (continuingTransfer !== undefined) { + continuingUnknownPublicationId = await continuingTeamsFilePublication(tx, { + companyId: input.companyId, + agentId: input.agentId, + issueId: issue.id, + failedRunId: failedRun.id, + provider, + publication: input.publication, + retryActionId: input.retryActionId, + runId: input.runId, + continuingTransfer, + }, failedChatRetryDenied); + } + const [question] = await tx + .select({ id: issueThreadInteractions.id }) + .from(issueThreadInteractions) + .where( + and( + eq(issueThreadInteractions.companyId, input.companyId), + eq(issueThreadInteractions.issueId, issue.id), + eq(issueThreadInteractions.sourceRunId, failedRun.id), + ), + ) + .limit(1); + const [answerDelivery] = await tx + .select({ id: issueQuestionResponseDeliveries.id }) + .from(issueQuestionResponseDeliveries) + .where( + and( + eq(issueQuestionResponseDeliveries.companyId, input.companyId), + eq(issueQuestionResponseDeliveries.targetRunId, failedRun.id), + ), + ) + .limit(1); + const [resultPublication] = await tx + .select({ id: chatPublications.id }) + .from(chatPublications) + .where( + and( + eq(chatPublications.companyId, input.companyId), + eq(chatPublications.issueId, issue.id), + input.committedResponse + ? eq(chatPublications.state, "delivery_unknown") + : undefined, + continuingUnknownPublicationId + ? ne(chatPublications.id, continuingUnknownPublicationId) + : undefined, + or( + and( + like(chatPublications.idempotencyKey, `run:${failedRun.id}:%`), + eq(chatPublications.state, "delivery_unknown"), + ), + sql`exists (select 1 from issue_comments result_comment where result_comment.id = ${chatPublications.commentId} + and result_comment.company_id = ${input.companyId}::uuid and result_comment.created_by_run_id = ${failedRun.id}::uuid + )`, + ), + ), + ) + .limit(1); + if ( + (!input.committedResponse && accepted) || + resultPublication || + question || + answerDelivery + ) + throw failedChatRetryDenied(); + if (context.chatFailedRunRetry) { + const hint = context.chatFailedRunRetry as Record; + const [priorIntent] = await tx + .select() + .from(chatActions) + .where( + and( + eq(chatActions.id, failedRun.wakeupRequestId), + eq(chatActions.companyId, input.companyId), + ), + ) + .for("share", { noWait: true }); + const [priorReceipt] = await tx + .select() + .from(agentWakeupRequests) + .where( + and( + eq(agentWakeupRequests.id, failedRun.wakeupRequestId), + eq(agentWakeupRequests.companyId, input.companyId), + ), + ) + .for("share", { noWait: true }); + if ( + !priorIntent || + priorIntent.kind !== "failed_run_retry" || + priorIntent.status !== "processed" || + !priorReceipt || + priorReceipt.runId !== failedRun.id || + !["failed", "completed"].includes(priorReceipt.status) || + typeof priorIntent.payload.failedRunId !== "string" || + !isUuidLike(priorIntent.payload.failedRunId) || + hint.version !== 1 || + hint.actionId !== priorIntent.id || + hint.failedRunId !== priorIntent.payload.failedRunId || + failedRun.retryOfRunId !== priorIntent.payload.failedRunId || + context.retryOfRunId !== failedRun.retryOfRunId + ) + throw failedChatRetryDenied(); + const parent = await failedChatRetrySource( + tx, + { + ...input, + failedRunId: priorIntent.payload.failedRunId, + committedResponse: undefined, + }, + ancestors, + ); + if ( + priorIntent.payload.sourceScopeSha256 !== parent.sourceScopeSha256 || + priorIntent.endpointId !== parent.endpointId || + priorIntent.conversationId !== parent.conversationId || + priorIntent.principalId !== parent.principalId || + context.taskKey !== parent.taskKey || + provider !== parent.provider || + JSON.stringify(commentIds) !== JSON.stringify(parent.commentIds) + ) + throw failedChatRetryDenied(); + assertFailedRetryReceipt( + createDurableChatWakeupRequest({ + id: priorIntent.id, + companyId: input.companyId, + agentId: input.agentId, + issueId: input.issueId, + commentId: parent.commentIds.at(-1)!, + requestedByActorType: parent.requestedByActorType, + requestedByActorId: parent.requestedByActorId, + requestedAt: priorIntent.createdAt, + authorize: async () => {}, + }), + priorReceipt, + ); + const { sourceScopeSha256, ...parentScope } = parent; + return hashFailedRetrySource({ + ...parentScope, + failedRunId: failedRun.id, + retryAncestors: [ + ...parent.retryAncestors, + { + actionId: priorIntent.id, + failedRunId: parent.failedRunId, + sourceScopeSha256, + }, + ], + }); + } + // Include every receipt that fed this run, including a deferred owner's + // coalesced children. A mutable latest-comment hint cannot drop a sibling. + const admittedBatch = + await tx.execute(sql`select source.id, source.payload->>'commentId' as comment_id + from agent_wakeup_requests receipt join chat_actions source on source.id = receipt.id + where receipt.company_id = ${input.companyId}::uuid and receipt.agent_id = ${input.agentId}::uuid + and (receipt.run_id = ${failedRun.id}::uuid or receipt.id = ${failedRun.wakeupRequestId}::uuid + or receipt.payload->>'coalescedIntoWakeupRequestId' = ${failedRun.wakeupRequestId}) + and source.company_id = ${input.companyId}::uuid and source.kind = 'inbound_wakeup' limit 51`); + if ( + admittedBatch.length !== commentIds.length || + admittedBatch.some( + (entry) => !commentIds.includes(String(entry.comment_id)), + ) + ) + throw failedChatRetryDenied(); + const sources = await tx + .select() + .from(chatActions) + .where( + and( + eq(chatActions.companyId, input.companyId), + eq(chatActions.kind, "inbound_wakeup"), + sql`${chatActions.payload}->>'issueId' = ${issue.id}`, + inArray( + sql`${chatActions.payload}->>'commentId'`, + commentIds, + ), + ), + ); + if (sources.length !== commentIds.length) throw failedChatRetryDenied(); + const ordered = commentIds.map( + (id) => sources.find((action) => action.payload.commentId === id)!, + ); + const first = ordered[0]!; + if ( + !first || + !first.principalId || + !first.conversationId || + sources.some( + (action) => + action.endpointId !== first.endpointId || + action.conversationId !== first.conversationId || + action.principalId !== first.principalId || + action.payload.requestedByActorType !== + first.payload.requestedByActorType || + action.payload.requestedByActorId !== + first.payload.requestedByActorId, + ) + ) + throw failedChatRetryDenied(); + const identity = await tx.execute( + sql`select pg_try_advisory_xact_lock(hashtextextended(${`chat-identity:${input.companyId}:${first.principalId}`}, 0)) as locked`, + ); + if (!identity[0]?.locked) + throw new NativeChatReviewPresentationContentionError(); + const canonicalSources: Array> = []; + let destination: Awaited> | null = + null; + for (const action of ordered) { + destination = await authorizeInboundWakeup(tx, action, { + nonblocking: true, + terminal: input.publication === true, + }).catch((error) => { + if (isExternalActionAuthorizationChange(error)) + throw failedChatRetryDenied(); + throw error; + }); + const [delivery] = await tx + .select() + .from(chatDeliveries) + .where(eq(chatDeliveries.id, action.deliveryId!)) + .for("update", { noWait: true }); + const [current] = await tx + .select() + .from(chatActions) + .where(eq(chatActions.id, action.id)) + .for("update", { noWait: true }); + const [comment] = await tx + .select() + .from(issueComments) + .where( + and( + eq(issueComments.id, String(action.payload.commentId)), + eq(issueComments.companyId, input.companyId), + eq(issueComments.issueId, issue.id), + ), + ) + .for("share", { noWait: true }); + if ( + !current || + current.status !== "processed" || + inboundWakeScope(current) !== inboundWakeScope(action) || + !delivery || + delivery.state !== "processed" || + !comment || + comment.deletedAt || + comment.updatedAt.getTime() !== comment.createdAt.getTime() || + comment.updatedAt > + (input.committedResponse + ? accepted!.createdAt + : failedRun.finishedAt!) || + destination.endpoint.provider !== provider || + JSON.stringify(lifecycleRuntimeFence(delivery)) !== + JSON.stringify(lifecycleRuntimeFence(destination.delivery)) + ) + throw failedChatRetryDenied(); + const [lifecycle] = await tx + .select({ id: chatDeliveries.id }) + .from(chatDeliveries) + .where( + and( + eq(chatDeliveries.companyId, input.companyId), + eq(chatDeliveries.endpointId, first.endpointId), + or( + isNull(chatDeliveries.conversationId), + eq(chatDeliveries.conversationId, first.conversationId), + ), + ne(chatDeliveries.state, "filtered"), + sql`${chatDeliveries.normalizedEvent}->'runtimeContext' = ${JSON.stringify(lifecycleRuntimeFence(delivery))}::jsonb`, + inArray(chatDeliveries.eventKind, [ + "message_updated", + "message_deleted", + "message_restored", + ]), + sql`${chatDeliveries.normalizedEvent}->'message'->>'targetProviderEventId' = ${delivery.providerEventId}`, + ), + ) + .limit(1); + if (lifecycle) throw failedChatRetryDenied(); + const [receipt] = await tx + .select() + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.id, action.id)) + .for("share", { noWait: true }); + if (!receipt) throw failedChatRetryDenied(); + assertFailedRetryReceipt( + createDurableChatWakeupRequest({ + id: action.id, + companyId: input.companyId, + agentId: input.agentId, + issueId: issue.id, + commentId: comment.id, + requestedByActorType: action.payload.requestedByActorType as + | "user" + | "system", + requestedByActorId: String(action.payload.requestedByActorId), + requestedAt: action.createdAt, + authorize: async () => {}, + }), + receipt, + ); + let owner = receipt; + const seen = new Set(); + for ( + let depth = 0; + owner.payload?.coalescedIntoWakeupRequestId !== undefined; + depth++ + ) { + const ownerId = owner.payload.coalescedIntoWakeupRequestId; + // The scheduler writes a direct child -> canonical owner edge. Nested + // chains/cycles are not silently interpreted as additional authority. + if ( + depth >= 1 || + owner.status !== "coalesced" || + typeof ownerId !== "string" || + !isUuidLike(ownerId) || + seen.has(ownerId) + ) + throw failedChatRetryDenied(); + seen.add(ownerId); + const [next] = await tx + .select() + .from(agentWakeupRequests) + .where( + and( + eq(agentWakeupRequests.id, ownerId), + eq(agentWakeupRequests.companyId, input.companyId), + ), + ) + .for("share", { noWait: true }); + if (!next) throw failedChatRetryDenied(); + owner = next; + } + if ( + owner.id !== failedRun.wakeupRequestId || + owner.runId !== failedRun.id || + owner.agentId !== input.agentId || + owner.requestedByActorType !== receipt.requestedByActorType || + owner.requestedByActorId !== receipt.requestedByActorId || + ["skipped", "cancelled"].includes(receipt.status) + ) + throw failedChatRetryDenied(); + canonicalSources.push({ + actionId: action.id, + deliveryId: delivery.id, + commentId: comment.id, + principalId: action.principalId, + scope: inboundWakeScope(action), + fence: lifecycleRuntimeFence(delivery), + bodySha256: createHash("sha256").update(comment.body).digest("hex"), + omissionReasons: action.payload.attachmentOmissionReasons ?? {}, + receiptId: receipt.id, + ownerId: owner.id, + }); + } + if (!destination) throw failedChatRetryDenied(); + const newestSource = ordered.reduce( + (latest, action) => + action.createdAt > latest ? action.createdAt : latest, + first.createdAt, + ); + const [newer] = await tx + .select({ id: chatActions.id }) + .from(chatActions) + .where( + and( + eq(chatActions.companyId, input.companyId), + eq(chatActions.conversationId, first.conversationId), + eq(chatActions.kind, "inbound_wakeup"), + notInArray( + chatActions.id, + ordered.map((action) => action.id), + ), + gte(chatActions.createdAt, newestSource), + sql`exists (select 1 from agent_wakeup_requests newer_admission where newer_admission.id = ${chatActions.id} + and newer_admission.company_id = ${input.companyId}::uuid and newer_admission.status <> 'skipped')`, + ), + ) + .limit(1); + if (newer && !input.publication) throw failedChatRetryDenied(); + const scope = { + failedRunId: failedRun.id, + sourceWakeupRequestId: failedRun.wakeupRequestId, + issueId: issue.id, + agentId: input.agentId, + taskKey: context.taskKey as string, + provider, + commentIds, + sources: canonicalSources, + endpointId: first.endpointId, + conversationId: first.conversationId, + principalId: first.principalId, + sessionGeneration: destination.conversation.sessionGeneration, + requestedByActorType: first.payload.requestedByActorType as + | "user" + | "system", + requestedByActorId: String(first.payload.requestedByActorId), + retryAncestors: [], + }; + return hashFailedRetrySource(scope); + } + + const unregisterCommittedResponseAuthority = + registerCommittedChatResponseAuthority(db, async (tx, input) => { + try { + await failedChatRetrySource(tx, { + ...input, + failedRunId: input.runId, + publication: true, + committedResponse: { resultId: input.resultId }, + }); + } catch (error) { + if ( + error instanceof HttpError && + (error.details as { code?: string } | undefined)?.code === + "chat_failed_run_retry_not_authorized" + ) { + throw new CommittedChatResponseAuthorizationError(); + } + throw error; + } + }); + + async function prepareFailedChatRunRetry( + tx: DbOrTransaction, + input: { + companyId: string; + issueId: string; + agentId: string; + failedRunId: string; + initiatedByUserId: string; + }, + ): Promise<{ actionId: string; issueId: string }> { + const source = await failedChatRetrySource(tx, input); + const providerActionId = `failed-run-retry:${source.failedRunId}`; + const [existing] = await tx + .select() + .from(chatActions) + .where( + and( + eq(chatActions.endpointId, source.endpointId), + eq(chatActions.providerActionId, providerActionId), + ), + ) + .for("update", { noWait: true }); + if (existing) { + if ( + existing.kind !== "failed_run_retry" || + existing.companyId !== input.companyId || + existing.payload.sourceScopeSha256 !== source.sourceScopeSha256 + ) + throw failedChatRetryDenied(); + return { actionId: existing.id, issueId: input.issueId }; + } + await assertFailedNativeRetryState(tx, source, input.companyId); + const [busy] = await tx + .select({ id: heartbeatRuns.id }) + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.companyId, input.companyId), + inArray(heartbeatRuns.status, [ + "queued", + "running", + "scheduled_retry", + ]), + sql`coalesce(${heartbeatRuns.contextSnapshot}->>'issueId', ${heartbeatRuns.contextSnapshot}->>'taskId') = ${input.issueId}`, + ), + ) + .limit(1); + if (busy || !input.initiatedByUserId) throw failedChatRetryDenied(); + const now = new Date(); + const [intent] = await tx + .insert(chatActions) + .values({ + companyId: input.companyId, + endpointId: source.endpointId, + conversationId: source.conversationId, + principalId: source.principalId, + kind: "failed_run_retry", + providerActionId, + payload: { + version: 1, + ...source, + initiatedByUserId: input.initiatedByUserId, + }, + status: "issued", + result: { code: "failed_chat_retry_queued", attemptCount: 0 }, + createdAt: now, + updatedAt: now, + }) + .returning(); + await logActivity(tx as unknown as Db, { + companyId: input.companyId, + actorType: "user", + actorId: input.initiatedByUserId, + action: "chat.failed_run_retry_requested", + entityType: "heartbeat_run", + entityId: input.failedRunId, + details: { + retryActionId: intent.id, + issueId: input.issueId, + commentCount: source.commentIds.length, + }, + }); + return { actionId: intent.id, issueId: input.issueId }; + } + + async function authorizeFailedChatRunRetry( + tx: DbOrTransaction, + input: { + companyId: string; + issueId: string; + agentId: string; + retryActionId: string; + runId?: string; + publication?: boolean; + }, + ) { + const [snapshot] = await tx + .select() + .from(chatActions) + .where( + and( + eq(chatActions.id, input.retryActionId), + eq(chatActions.companyId, input.companyId), + ), + ); + if ( + !snapshot || + snapshot.kind !== "failed_run_retry" || + typeof snapshot.payload.failedRunId !== "string" || + snapshot.payload.issueId !== input.issueId || + snapshot.payload.agentId !== input.agentId + ) + throw failedChatRetryDenied(); + const source = await failedChatRetrySource(tx, { + ...input, + failedRunId: snapshot.payload.failedRunId, + }); + const [intent] = await tx + .select() + .from(chatActions) + .where(eq(chatActions.id, snapshot.id)) + .for("update", { noWait: true }); + if ( + !intent || + !( + input.publication + ? ["issued", "processing", "processed", "failed", "cancelled"] + : ["issued", "processing", "processed"] + ).includes(intent.status) || + intent.payload.version !== 1 || + intent.endpointId !== source.endpointId || + intent.conversationId !== source.conversationId || + intent.principalId !== source.principalId || + intent.payload.sourceScopeSha256 !== source.sourceScopeSha256 || + JSON.stringify(intent.payload) !== JSON.stringify(snapshot.payload) + ) + throw failedChatRetryDenied(); + const [receipt] = await tx + .select() + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.id, intent.id)) + .for("share", { noWait: true }); + if (receipt) { + assertFailedRetryReceipt( + createDurableChatWakeupRequest({ + id: intent.id, + companyId: input.companyId, + agentId: input.agentId, + issueId: input.issueId, + commentId: source.commentIds.at(-1)!, + requestedByActorType: source.requestedByActorType, + requestedByActorId: source.requestedByActorId, + requestedAt: intent.createdAt, + authorize: async () => {}, + }), + receipt, + ); + if ( + (!input.publication && + ["skipped", "cancelled"].includes(receipt.status)) || + (input.runId && receipt.runId !== input.runId) + ) + throw failedChatRetryDenied(); + } else if (input.runId) throw failedChatRetryDenied(); + if (input.runId) { + const [run] = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, input.runId), + eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), + ), + ) + .for("share", { noWait: true }); + if ( + !run || + run.wakeupRequestId !== intent.id || + run.retryOfRunId !== source.failedRunId + ) + throw failedChatRetryDenied(); + } + return { ...source, intent }; + } + + const unregisterFailedRetryAuthority = registerFailedChatRunRetryAuthority( + db, + async (tx, input) => { + const source = await authorizeFailedChatRunRetry(tx, { + ...input, + retryActionId: input.wakeupRequestId, + }); + await assertFailedNativeRetryState(tx, source, input.companyId); + const context = input.contextSnapshot; + const hint = context.chatFailedRunRetry as + | Record + | undefined; + if ( + context.issueId !== source.issueId || + context.source !== `chat:${source.provider}` || + context.taskKey !== source.taskKey || + context.retryOfRunId !== source.failedRunId || + JSON.stringify(retryCommentIds(context)) !== + JSON.stringify(source.commentIds) || + hint?.version !== 1 || + hint.actionId !== source.intent.id || + hint.failedRunId !== source.failedRunId || + !Array.isArray(context.externalAttachmentOmissions) || + JSON.stringify( + context.externalAttachmentOmissions.map((entry) => + entry && typeof entry === "object" + ? [entry.commentId, entry.reasons] + : null, + ), + ) !== + JSON.stringify( + source.sources.map((entry) => [ + entry.commentId, + entry.omissionReasons, + ]), + ) + ) + throw failedChatRetryDenied(); + }, + ); + + async function failedChatRetryReceipt( + action: typeof chatActions.$inferSelect, + ) { + const [receipt] = await db + .select() + .from(agentWakeupRequests) + .where( + and( + eq(agentWakeupRequests.id, action.id), + eq(agentWakeupRequests.companyId, action.companyId), + ), + ); + const [run] = receipt?.runId + ? await db + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, receipt.runId), + eq(heartbeatRuns.companyId, action.companyId), + ), + ) + : []; + const status = ["succeeded", "failed", "cancelled"].includes( + run?.status ?? "", + ) + ? (run!.status as "succeeded" | "failed" | "cancelled") + : run?.status === "timed_out" + ? "failed" + : run?.status === "running" + ? "running" + : receipt?.status === "deferred_issue_execution" + ? "deferred" + : ["failed", "cancelled"].includes(action.status) + ? (action.status as "failed" | "cancelled") + : ["skipped", "failed", "cancelled"].includes( + receipt?.status ?? "", + ) + ? "failed" + : "queued"; + return { + actionId: action.id, + issueId: String(action.payload.issueId), + runId: run?.id ?? null, + status, + }; + } + + async function claimFailedChatRunRetry(actionId: string) { + const [candidate] = await db + .select() + .from(chatActions) + .where( + and( + eq(chatActions.id, actionId), + eq(chatActions.kind, "failed_run_retry"), + ), + ); + if (!candidate) throw notFound("Chat retry request not found"); + const now = new Date(); + if ( + isShuttingDown() || + !["issued", "processing"].includes(candidate.status) || + (candidate.status === "processing" && + candidate.updatedAt.getTime() > + now.getTime() - DELIVERY_PROCESSING_STALE_MS) || + (typeof candidate.result?.retryAt === "string" && + Date.parse(candidate.result.retryAt) > now.getTime()) + ) + return failedChatRetryReceipt(candidate); + const [claimed] = await db + .update(chatActions) + .set({ status: "processing", updatedAt: now }) + .where( + and( + eq(chatActions.id, candidate.id), + eq(chatActions.status, candidate.status), + eq(chatActions.updatedAt, candidate.updatedAt), + ), + ) + .returning(); + if (!claimed) return failedChatRetryReceipt(candidate); + const attemptCount = Number(candidate.result?.attemptCount ?? 0) + 1; + const settle = async (status: string, result: Record) => { + const [current] = await db + .update(chatActions) + .set({ status, result, updatedAt: new Date() }) + .where( + and( + eq(chatActions.id, claimed.id), + eq(chatActions.status, "processing"), + eq(chatActions.updatedAt, now), + ), + ) + .returning(); + return failedChatRetryReceipt(current ?? claimed); + }; + try { + const scope = { + companyId: claimed.companyId, + issueId: String(claimed.payload.issueId), + agentId: String(claimed.payload.agentId), + retryActionId: claimed.id, + }; + const source = await db.transaction((tx) => + authorizeFailedChatRunRetry(tx, scope), + ); + const existing = await db + .select({ id: agentWakeupRequests.id }) + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.id, claimed.id)); + if (!existing.length) { + const context = { + issueId: source.issueId, + taskKey: source.taskKey, + source: `chat:${source.provider}`, + wakeCommentId: source.commentIds.at(-1)!, + wakeCommentIds: source.commentIds, + retryOfRunId: source.failedRunId, + chatFailedRunRetry: { + version: 1, + actionId: claimed.id, + failedRunId: source.failedRunId, + }, + externalAttachmentOmissions: source.sources.map((entry) => ({ + commentId: entry.commentId, + reasons: entry.omissionReasons, + })), + }; + await wakeup(source.agentId, { + source: "on_demand", + triggerDetail: "manual", + reason: "retry_failed_run", + payload: context, + contextSnapshot: context, + allowRunCoalescing: false, + requestedByActorType: source.requestedByActorType, + requestedByActorId: source.requestedByActorId, + durableChatRequest: createDurableChatWakeupRequest({ + id: claimed.id, + companyId: claimed.companyId, + agentId: source.agentId, + issueId: source.issueId, + commentId: source.commentIds.at(-1)!, + requestedByActorType: source.requestedByActorType, + requestedByActorId: source.requestedByActorId, + requestedAt: claimed.createdAt, + failedRunRetry: { failedRunId: source.failedRunId }, + authorize: async (tx) => { + const current = await authorizeFailedChatRunRetry(tx, scope); + await assertFailedNativeRetryState( + tx, + current, + claimed.companyId, + ); + }, + }), + }); + } + const [receipt] = await db + .select() + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.id, claimed.id)); + if (!receipt) throw new Error("chat_retry_receipt_not_committed"); + return settle( + ["skipped", "failed", "cancelled"].includes(receipt.status) + ? "failed" + : "processed", + { code: "failed_chat_retry_durable", attemptCount }, + ); + } catch (error) { + const [receipt] = await db + .select() + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.id, claimed.id)); + if (receipt) + return settle("processed", { + code: "failed_chat_retry_durable", + attemptCount, + }); + const denied = + error && + typeof error === "object" && + "details" in error && + error.details && + typeof error.details === "object" && + "code" in error.details && + [ + "chat_failed_run_retry_not_authorized", + "chat_action_authorization_changed", + ].includes(String(error.details.code)); + const retryable = !denied && attemptCount < 8; + return settle(retryable ? "issued" : "failed", { + code: retryable + ? "failed_chat_retry_contended" + : "failed_chat_retry_not_authorized", + attemptCount, + ...(retryable + ? { + retryAt: new Date( + Date.now() + Math.min(5_000, attemptCount * 250), + ).toISOString(), + } + : {}), + }); + } + } + + const failedRetryTasks = new Map< + string, + ReturnType + >(); + function processFailedChatRunRetry(actionId: string) { + const existing = failedRetryTasks.get(actionId); + if (existing) return existing; + const task = claimFailedChatRunRetry(actionId); + failedRetryTasks.set(actionId, task); + void task.then( + () => failedRetryTasks.delete(actionId), + () => failedRetryTasks.delete(actionId), + ); + return task; + } + + async function processFailedChatRunRetries(limit = 20) { + if (isShuttingDown()) return 0; + const now = new Date(); + const candidates = await db + .select({ id: chatActions.id }) + .from(chatActions) + .where( + and( + eq(chatActions.kind, "failed_run_retry"), + or( + eq(chatActions.status, "issued"), + and( + eq(chatActions.status, "processing"), + lte( + chatActions.updatedAt, + new Date(now.getTime() - DELIVERY_PROCESSING_STALE_MS), + ), + ), + ), + sql`(${chatActions.result}->>'retryAt' is null or (${chatActions.result}->>'retryAt')::timestamptz <= ${now.toISOString()}::timestamptz)`, + ), + ) + .orderBy(asc(chatActions.createdAt), asc(chatActions.id)) + .limit(Math.max(1, Math.min(50, limit))); + for (const candidate of candidates) + await processFailedChatRunRetry(candidate.id); + return candidates.length; + } + + let failedRetryNoticeCursor: { createdAt: Date; id: string } | null = null; + async function enqueueFailedChatRetryPublications( + limit: number, + ): Promise { + const cursor = failedRetryNoticeCursor; + const batchSize = Math.max(1, Math.min(50, limit)); + const candidates = await db + .select({ action: chatActions, receipt: agentWakeupRequests }) + .from(chatActions) + .innerJoin( + agentWakeupRequests, + and( + eq(agentWakeupRequests.id, chatActions.id), + eq(agentWakeupRequests.companyId, chatActions.companyId), + ), + ) + .where( + and( + eq(chatActions.kind, "failed_run_retry"), + isNull(agentWakeupRequests.runId), + cursor + ? sql`(${chatActions.createdAt}, ${chatActions.id}) > (${cursor.createdAt.toISOString()}::timestamptz, ${cursor.id}::uuid)` + : undefined, + inArray(agentWakeupRequests.status, [ + "deferred_issue_execution", + "cancelled", + "failed", + "skipped", + ]), + sql`not exists (select 1 from chat_publications retry_notice where retry_notice.company_id = ${chatActions.companyId} + and retry_notice.idempotency_key = 'wake:' || ${chatActions.id}::text || ':' || + case when ${agentWakeupRequests.status} = 'deferred_issue_execution' then 'queued' else 'not_started' end || ':' || + ${chatActions.endpointId}::text || ':' || ${chatActions.conversationId}::text)`, + ), + ) + .orderBy(asc(chatActions.createdAt), asc(chatActions.id)) + .limit(batchSize); + const last = candidates.at(-1)?.action; + failedRetryNoticeCursor = + candidates.length >= batchSize && last + ? { createdAt: last.createdAt, id: last.id } + : null; + let inserted = 0; + for (const { action, receipt } of candidates) { + try { + inserted += await db.transaction(async (tx) => { + const source = await authorizeFailedChatRunRetry(tx, { + companyId: action.companyId, + issueId: String(action.payload.issueId), + agentId: String(action.payload.agentId), + retryActionId: action.id, + publication: true, + }); + const [current] = await tx + .select() + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.id, receipt.id)) + .for("update", { noWait: true }); + if (!current || current.runId || current.status !== receipt.status) + return 0; + const state = + current.status === "deferred_issue_execution" + ? "queued" + : "not_started"; + await notifyChatPublicationWork(tx); + return ( + await tx + .insert(chatPublications) + .values({ + companyId: action.companyId, + endpointId: source.endpointId, + conversationId: source.conversationId, + issueId: source.issueId, + commentId: source.commentIds.at(-1)!, + idempotencyKey: inboundWakePublicationKey( + action.id, + state, + source.endpointId, + source.conversationId, + ), + payload: projectSafeChatPublication({ + classification: "external", + source: "safe_milestone", + text: + state === "queued" + ? "Your retry is queued." + : "This retry was not started. Open the task in Paperclip for details.", + progressState: state === "queued" ? "queued" : "failed", + }), + state: "pending", + }) + .onConflictDoNothing() + .returning({ id: chatPublications.id }) + ).length; + }); + } catch (error) { + if ( + isExternalChatWaitAuthorizationContention(error) || + error instanceof NativeChatReviewPresentationContentionError || + isExternalActionAuthorizationChange(error) || + (error && + typeof error === "object" && + "status" in error && + error.status === 409) + ) + continue; + throw error; + } + } + return inserted; + } + + return { + retryCommentIds, + assertFailedRetryReceipt, + FailedChatRetryPublicationReadError, + failedChatRetrySource, + unregisterCommittedResponseAuthority, + prepareFailedChatRunRetry, + authorizeFailedChatRunRetry, + unregisterFailedRetryAuthority, + processFailedChatRunRetry, + processFailedChatRunRetries, + enqueueFailedChatRetryPublications, + pendingFailedRetries: () => [...failedRetryTasks.values()], + }; +} diff --git a/server/src/services/chat-channels/microsoft-teams/file-transfer-source.ts b/server/src/services/chat-channels/microsoft-teams/file-transfer-source.ts new file mode 100644 index 00000000000..5990648fa8e --- /dev/null +++ b/server/src/services/chat-channels/microsoft-teams/file-transfer-source.ts @@ -0,0 +1,149 @@ +import { and, eq, isNull, sql } from "drizzle-orm"; +import { assets, chatConversations, chatEndpoints, chatPublications, chatTeamsFileTransfers, issueAttachments, issueComments, type Db } from "@paperclipai/db"; +import { isUuidLike, type ChatProvider } from "@paperclipai/shared"; + +type DbOrTransaction = Db | Parameters[0]>[0]; + +export type ContinuingTeamsFileTransfer = { + transferId: string; + publicationId: string; + version: number; + phase: "consent_unknown" | "file_info_unknown"; +}; + +type TeamsFileTransferSourceInput = { + companyId: string; + agentId: string; + issueId: string; + failedRunId: string; + provider: ChatProvider; + publication?: boolean; + retryActionId?: string; + runId?: string; + continuingTransfer: ContinuingTeamsFileTransfer; +}; + +// A selector proves only this transfer's source. The caller must still verify +// the committed result and complete original source batch in this transaction. +export async function continuingTeamsFilePublication( + tx: DbOrTransaction, + input: TeamsFileTransferSourceInput, + failedChatRetryDenied: () => Error, +): Promise { + const { provider, continuingTransfer } = input; + if ( + input.publication !== true || + input.retryActionId !== undefined || + input.runId !== undefined || + provider !== "microsoft-teams" || + !continuingTransfer || + !isUuidLike(continuingTransfer.transferId) || + !isUuidLike(continuingTransfer.publicationId) || + !Number.isSafeInteger(continuingTransfer.version) || + continuingTransfer.version < 1 || + !["consent_unknown", "file_info_unknown"].includes( + continuingTransfer.phase, + ) + ) + throw failedChatRetryDenied(); + // The caller supplies a selector only. Independently prove the exact + // unknown publication is this native run's own selected file, not a + // sibling effect or another actor/generation. All ordinary source and + // committed-result authorization below still runs unchanged. + const [continuing] = await tx + .select({ + publicationId: chatPublications.id, + attemptId: chatTeamsFileTransfers.attemptId, + attemptExpiresAt: chatTeamsFileTransfers.attemptExpiresAt, + }) + .from(chatTeamsFileTransfers) + .innerJoin( + chatPublications, + and( + eq(chatPublications.id, chatTeamsFileTransfers.publicationId), + eq(chatPublications.companyId, chatTeamsFileTransfers.companyId), + eq(chatPublications.endpointId, chatTeamsFileTransfers.endpointId), + eq( + chatPublications.conversationId, + chatTeamsFileTransfers.conversationId, + ), + eq(chatPublications.issueId, chatTeamsFileTransfers.issueId), + eq(chatPublications.commentId, chatTeamsFileTransfers.commentId), + eq(chatPublications.state, "delivery_unknown"), + sql`${chatPublications.payload}->'attachmentIds' = jsonb_build_array(${chatTeamsFileTransfers.attachmentId}::text)`, + ), + ) + .innerJoin( + chatEndpoints, + and( + eq(chatEndpoints.id, chatTeamsFileTransfers.endpointId), + eq(chatEndpoints.companyId, input.companyId), + eq(chatEndpoints.provider, "microsoft-teams"), + ), + ) + .innerJoin( + chatConversations, + and( + eq(chatConversations.id, chatTeamsFileTransfers.conversationId), + eq(chatConversations.companyId, input.companyId), + eq(chatConversations.endpointId, chatTeamsFileTransfers.endpointId), + eq(chatConversations.issueId, input.issueId), + eq( + chatConversations.sessionGeneration, + chatTeamsFileTransfers.conversationGeneration, + ), + ), + ) + .innerJoin( + issueComments, + and( + eq(issueComments.id, chatTeamsFileTransfers.commentId), + eq(issueComments.companyId, input.companyId), + eq(issueComments.issueId, input.issueId), + eq(issueComments.createdByRunId, input.failedRunId), + eq(issueComments.authorAgentId, input.agentId), + isNull(issueComments.deletedAt), + ), + ) + .innerJoin( + issueAttachments, + and( + eq(issueAttachments.id, chatTeamsFileTransfers.attachmentId), + eq(issueAttachments.companyId, input.companyId), + eq(issueAttachments.issueId, input.issueId), + eq(issueAttachments.issueCommentId, issueComments.id), + eq(issueAttachments.originatingRunId, input.failedRunId), + ), + ) + .innerJoin( + assets, + and( + eq(assets.id, issueAttachments.assetId), + eq(assets.companyId, input.companyId), + eq(assets.sha256, chatTeamsFileTransfers.sha256), + eq(assets.byteSize, chatTeamsFileTransfers.byteSize), + ), + ) + .where( + and( + eq(chatTeamsFileTransfers.companyId, input.companyId), + eq(chatTeamsFileTransfers.issueId, input.issueId), + eq(chatTeamsFileTransfers.id, continuingTransfer.transferId), + eq( + chatTeamsFileTransfers.publicationId, + continuingTransfer.publicationId, + ), + eq(chatTeamsFileTransfers.version, continuingTransfer.version), + eq(chatTeamsFileTransfers.phase, continuingTransfer.phase), + ), + ) + .for("share", { noWait: true }); + if ( + !continuing || + (continuing.attemptId && + (!continuing.attemptExpiresAt || + continuing.attemptExpiresAt > new Date())) + ) + throw failedChatRetryDenied(); + return continuing.publicationId; +} From 37913e4907324838ae2c8c94dc8155002b6d4223 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 12 Oct 2026 09:51:17 -0500 Subject: [PATCH 6/7] refactor(heartbeat): extract run output recording (#15969) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat runs stream adapter output and persist execution results. > - Output recording currently sits inside the executor beside admission and cleanup. > - Log handles, sequence numbers, progress writes, and excerpt buffers share one attempt lifetime. > - This pull request moves that state and its callbacks into a per-attempt recorder. > - The executor keeps dispatch and finalization visible, while finalization reads live output values. ## Linked Issues or Issue Description **What existing behavior does this improve?** Maintenance of heartbeat execution. This PR follows #15966, which is now merged. The branch is rebased onto `master`; `git range-diff` confirms that its extraction patch is unchanged. **Current behavior** `heartbeat.ts` has 4,740 lines and 195,258 bytes. It directly owns log handles, excerpt buffers, output sequence numbers, progress flushes, and adapter event callbacks. **Proposed behavior** `run-output.ts` owns the per-attempt recorder. The main file has 4,561 lines and 188,950 bytes; the recorder has 209 lines and 7,757 bytes. **Reason and benefit** Output state has one explicit lifetime and interface. The shared log store and finalization policy retain their original owners. Related open output work includes #12539 and #15524; this PR does not change redaction policy. **Breaking changes** None intended. Public flush constants and log formatting exports remain available from `heartbeat.ts`. ## What Changed - Move log initialization, redaction, excerpts, progress persistence, live delivery, and adapter callbacks into `createHeartbeatRunOutput`. - Preserve the acquired log handle if later setup fails. Bind callbacks to the execution-start run row. - Pass live excerpt getters to success and failure completion so output arriving during finalization remains visible. - Add three real-database public-path tests. Extend browser acceptance to inspect persisted worker stdout and output byte counts. - Update the module map. ## Verification - Before extraction: 23 tests pass, including all three new output characterization cases. - After extraction: 45 tests pass across output, log formatting, event sequencing, and run completion. No skips. - `pnpm -r typecheck` and `pnpm build` pass. - Normalized syntax comparison confirms the moved bodies and reconstructed executor match. Reviewed the factory ownership, callback binding, getter wiring, and constant re-exports separately. - Isolated browser acceptance: 2/2 pass, including persisted worker stdout and matching final byte counts. The process worker completes after recovery; persistent timeouts stop after three attempts. Both terminal states survive reload. - Hosted full-suite CI passes on `d32c8fc40e72b9766bdd9e191d777e8dc554e3f2`; all 55 reported checks are successful. Greptile is 5/5 with zero review threads. - The initial chat shard timed out waiting for GitHub mention admission. Its fixture mocks heartbeat wakeup and never calls the output recorder. Both parameterized cases passed locally; the unchanged shard passed on one rerun. No test or timeout changes. - Merge preparation: all seven session-preparation and run-output regression tests pass after rebasing onto `master`. Fresh full CI passes for `60e2a49ce2879645a9f618a801e0ba0903ede161`: 55 successful checks and two optional Storybook skips, including all eight browser shards, typecheck, tests, and build. Greptile is 5/5 on this exact commit with zero review threads. ## Risks - A stale excerpt snapshot could lose output that arrives while finalization awaits I/O. Passing-before/passing-after tests exercise this for both successful and failed runs. - Per-attempt state must not become service-wide. Each executor constructs its own recorder and keeps the existing shared store. - Browser acceptance uses a real process fixture and disposable data; it makes no live provider calls. ## Model Used OpenAI Codex, based on GPT-6, assisted with reasoning, code edits, tool execution, and verification. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/DEVELOPING.md | 7 +- .../__tests__/heartbeat-run-output.test.ts | 121 ++++++++++ server/src/services/heartbeat.ts | 203 +---------------- server/src/services/heartbeat/run-output.ts | 209 ++++++++++++++++++ .../e2e/workspace-bootstrap/recovery.spec.ts | 6 + 5 files changed, 353 insertions(+), 193 deletions(-) create mode 100644 server/src/__tests__/heartbeat-run-output.test.ts create mode 100644 server/src/services/heartbeat/run-output.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 7cf3e8b1471..da1bc58a960 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1302,8 +1302,11 @@ acquire no runtime resources. Public-path characterization tests live in Run-log formatting is in `server/src/services/heartbeat/run-log.ts`. It bounds stored event payloads, redacts and shortens log chunks, and caps stdout/stderr -excerpts. `heartbeat.ts` keeps event writes, current-user redaction, and live -event delivery. Existing public helpers remain available from `heartbeat.ts`. +excerpts. `run-output.ts` owns per-attempt log handles, current-user redaction, +output progress writes, live log delivery, and adapter event callbacks. The log +store remains service-scoped. Completion reads live excerpt getters so output +that arrives during finalization is retained. Existing public helpers and flush +interval constants remain available from `heartbeat.ts`. Workspace preparation is in `server/src/services/heartbeat/workspaces.ts`. It owns managed checkout materialization, workspace validation and reuse, referenced diff --git a/server/src/__tests__/heartbeat-run-output.test.ts b/server/src/__tests__/heartbeat-run-output.test.ts new file mode 100644 index 00000000000..a912d6e0e37 --- /dev/null +++ b/server/src/__tests__/heartbeat-run-output.test.ts @@ -0,0 +1,121 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issues } from "@paperclipai/db"; +import { registerServerAdapter, unregisterServerAdapter, type AdapterExecutionContext } from "../adapters/index.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import { getRunLogStore } from "../services/run-log-store.js"; +import { subscribeCompanyLiveEvents } from "../services/live-events.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const describePostgres = support.supported ? describe : describe.skip; +if (!support.supported) console.warn(`Skipping run output tests: ${support.reason}`); + +describePostgres("heartbeat output through adapter dispatch and finalization", () => { + const adapterType = "run_output_test"; + let temporary: Awaited>; + let db: ReturnType; + let heartbeat: ReturnType; + const executeByAgent = new Map Promise>(); + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("heartbeat-run-output-"); + db = createDb(temporary.connectionString); + heartbeat = heartbeatService(db); + registerServerAdapter({ + type: adapterType, supportsLocalAgentJwt: false, + execute: async (input) => { + await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, String(input.context.issueId))); + await executeByAgent.get(input.agent.id)!(input); + return { exitCode: 0, signal: null, timedOut: false, resultJson: {} }; + }, + testEnvironment: async () => ({ adapterType, status: "pass", checks: [], testedAt: new Date().toISOString() }), + }); + }, 30_000); + afterEach(() => vi.restoreAllMocks()); + afterAll(async () => { + await heartbeat?.drainActiveRunExecutions(); + unregisterServerAdapter(adapterType); + await db?.$client.end({ timeout: 0 }); + await temporary?.cleanup(); + }); + + async function seed(execute: (input: AdapterExecutionContext) => Promise) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Output fixture", issuePrefix: `O${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Output writer", adapterType, status: "idle" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Output fixture", status: "todo", assigneeAgentId: agentId, responsibleUserId: "output-user", issueNumber: 1 }); + executeByAgent.set(agentId, execute); + return { companyId, agentId, issueId }; + } + async function execute(f: Awaited>) { + const run = await heartbeat.wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "manual", payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId } }); + expect(run).not.toBeNull(); + await heartbeat.drainActiveRunExecutions(); + const finished = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)).then(rows => rows[0]); + expect(finished).toBeDefined(); + const log = await heartbeat.readLog(run!.id); + const entries = log.content.trim().split("\n").filter(Boolean).map(line => JSON.parse(line)); + return { finished, entries }; + } + + it("redacts stored and live chunks, preserves sequence, and flushes the last output at completion", async () => { + const secret = "output-test-bearer-canary"; + const f = await seed(async input => { + await input.onLog("stdout", `first: Authorization: Bearer ${secret}\n`); + await input.onLog("stdout", `${"x".repeat(10_000)}live-tail\n`); + await input.onLog("stderr", "last stderr\n"); + await input.onMeta?.({ adapterType, command: "fixture", env: { PAPERCLIP_API_KEY: secret }, prompt: "fixture prompt" }); + await input.onEvent?.({ eventType: " fixture.progress ", stream: "system", level: "info", message: "fixture event", payload: { step: 1 } }); + await input.onEvent?.({ eventType: " ", stream: "system", level: "info", message: "ignored blank event" }); + }); + const live: Record[] = []; + const unsubscribe = subscribeCompanyLiveEvents(f.companyId, event => { + if (event.type === "heartbeat.run.log") live.push(event.payload); + }); + try { + const { finished, entries } = await execute(f); + expect(finished.status, finished.error ?? undefined).toBe("succeeded"); + const ownLive = live.filter(event => event.runId === finished.id); + expect(entries.map(entry => entry.seq)).toEqual(entries.map((_, i) => i + 1)); + expect(ownLive.map(event => event.seq)).toEqual(entries.map(entry => entry.seq)); + expect(JSON.stringify(entries)).not.toContain(secret); + expect(JSON.stringify(ownLive)).not.toContain(secret); + expect(finished.stdoutExcerpt).not.toContain(secret); + expect(finished.stderrExcerpt).toContain("last stderr"); + expect(ownLive.find(event => String(event.chunk).endsWith("live-tail\n"))).toMatchObject({ truncated: true }); + expect(finished.lastOutputSeq).toBe(entries.at(-1).seq); + expect(finished.lastOutputStream).toBe("stderr"); + expect(finished.lastOutputBytes).toBe(finished.logBytes); + expect(finished.lastOutputAt).toEqual(new Date(entries.at(-1).ts)); + const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, finished.id)); + expect(JSON.stringify(events)).not.toContain(secret); + expect(events.find(event => event.eventType === "adapter.invoke")?.payload).toMatchObject({ env: { PAPERCLIP_API_KEY: "***REDACTED***" } }); + expect(events.find(event => event.eventType === "fixture.progress")).toMatchObject({ message: "fixture event", payload: { step: 1 } }); + expect(events.some(event => event.message === "ignored blank event")).toBe(false); + } finally { unsubscribe(); } + }); + + it.each([false, true])("retains output arriving during finalization (adapter throws: %s)", async fail => { + let onLog: AdapterExecutionContext["onLog"]; + const f = await seed(async input => { + onLog = input.onLog; + await onLog("stdout", "before finalization\n"); + if (fail) throw new Error("invalid fixture configuration"); + }); + const store = getRunLogStore(); + const finalize = store.finalize.bind(store); + vi.spyOn(store, "finalize").mockImplementationOnce(async handle => { + await onLog("stdout", "during finalization\n"); + return finalize(handle); + }); + const { finished, entries } = await execute(f); + expect(finished.status, finished.error ?? undefined).toBe(fail ? "failed" : "succeeded"); + expect(finished.stdoutExcerpt).toContain("before finalization"); + expect(finished.stdoutExcerpt).toContain("during finalization"); + expect(entries.at(-1).chunk).toBe("during finalization\n"); + expect(finished.lastOutputSeq).toBe(entries.at(-1).seq); + expect(finished.lastOutputBytes).toBe(finished.logBytes); + }); +}); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 03274998d80..54006b8b7ef 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { createHeartbeatRunOutput } from "./heartbeat/run-output.js"; import { prepareHeartbeatSession, resolveHeartbeatAdapterSession } from "./heartbeat/session-preparation.js"; import { prepareHeartbeatTaskContext } from "./heartbeat/task-context.js"; import { selectHeartbeatEnvironment } from "./heartbeat/environment-selection.js"; @@ -30,8 +31,6 @@ export { } from "./heartbeat/run-control.js"; import { isHeartbeatRunTerminalStatus, - isHeartbeatRunRuntimeStatusActive, - publishHeartbeatRunRuntimeProgress, decorateHeartbeatRunRuntimeStatus, createHeartbeatLifecycle, } from "./heartbeat/run-lifecycle.js"; @@ -202,11 +201,7 @@ export { stripPaperclipSessionMetadataFromSessionParams, resolveTaskSessionConfigFreshness, } from "./heartbeat/workspaces.js"; -import { - appendExcerpt, - boundHeartbeatRunEventPayloadForStorage, - compactRunLogChunk, -} from "./heartbeat/run-log.js"; +import { boundHeartbeatRunEventPayloadForStorage } from "./heartbeat/run-log.js"; export { boundHeartbeatRunEventPayloadForStorage, compactRunLogChunk, @@ -375,15 +370,11 @@ import { type NativeSessionBackend, } from "../vendor/paperclip-runner/index.js"; -import { getRunLogStore, type RunLogHandle } from "./run-log-store.js"; +import { getRunLogStore } from "./run-log-store.js"; import { providerTraceStore, } from "./provider-trace-store.js"; import { getServerAdapter, runningProcesses } from "../adapters/index.js"; -import type { - AdapterInvocationMeta, - AdapterRuntimeEvent, -} from "../adapters/index.js"; import { createRuntimeToolsToken } from "../runtime-tools-token.js"; import { @@ -522,9 +513,6 @@ import { DIRECT_NON_INVOKABLE_STATUSES, } from "./agent-invokability.js"; -import { - redactCurrentUserText, -} from "../log-redaction.js"; import { createRunSecretRedactionRegistry } from "./run-secret-redaction.js"; @@ -544,7 +532,6 @@ import { environmentRunOrchestrator } from "./environment-run-orchestrator.js"; import { clearHeartbeatRunRuntimeStatus, sweepExpiredHeartbeatRunRuntimeStatuses, - touchHeartbeatRunRuntimeStatus, } from "./heartbeat-run-runtime-status.js"; import { @@ -557,8 +544,6 @@ import type { PluginWorkerManager } from "./plugin-worker-manager.js"; import { computeTaskDrain, applyTaskDrain, startTaskDrain, stopTaskDrain, readTaskDrain } from "./task-admission.js"; -const MAX_LIVE_LOG_CHUNK_BYTES = 8 * 1024; - const ACCEPTED_PLAN_CONVERSION_SKILL_KEY = "paperclipai/paperclip/paperclip-converting-plans-to-tasks"; @@ -574,8 +559,7 @@ export { ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS, ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS, } from "./recovery/service.js"; -export const ACTIVE_RUN_OUTPUT_PROGRESS_FLUSH_INTERVAL_MS = 60 * 1000; -export const ACTIVE_RUN_LOG_RUNTIME_STATUS_REFRESH_INTERVAL_MS = 5 * 1000; +export { ACTIVE_RUN_OUTPUT_PROGRESS_FLUSH_INTERVAL_MS, ACTIVE_RUN_LOG_RUNTIME_STATUS_REFRESH_INTERVAL_MS } from "./heartbeat/run-output.js"; export { INTERACTION_CONTINUATION_INFRA_RETRY_REASON, @@ -3381,49 +3365,16 @@ export function heartbeatService( evaluateSessionCompaction, }); - let handle: RunLogHandle | null = null; + const runOutput = createHeartbeatRunOutput(db, { + run, runId, issueId, runLogStore, identityRedactor, secretKeys, + appendRunEvent, getCurrentUserRedactionOptions, + }); const goalCheckpointSession: { current: { params: Record; displayId: string; } | null; } = { current: null }; - let stdoutExcerpt = ""; - let stderrExcerpt = ""; - let outputSeq = Number(run.lastOutputSeq ?? 0); - let lastOutputFlushAt: Date | null = run.lastOutputAt ?? null; - let lastLogRuntimeStatusTouchMs = 0; - const outputProgressState: { - pending: { - at: Date; - seq: number; - stream: "stdout" | "stderr"; - bytes: number; - } | null; - } = { pending: null }; - let persistedLogBytes = Number(run.logBytes ?? 0); - const flushOutputProgress = async (opts?: { force?: boolean }) => { - const pendingOutputProgress = outputProgressState.pending; - if (!pendingOutputProgress) return; - const shouldFlush = - opts?.force === true || - !lastOutputFlushAt || - pendingOutputProgress.at.getTime() - lastOutputFlushAt.getTime() >= - ACTIVE_RUN_OUTPUT_PROGRESS_FLUSH_INTERVAL_MS; - if (!shouldFlush) return; - await db - .update(heartbeatRuns) - .set({ - lastOutputAt: pendingOutputProgress.at, - lastOutputSeq: pendingOutputProgress.seq, - lastOutputStream: pendingOutputProgress.stream, - lastOutputBytes: pendingOutputProgress.bytes, - updatedAt: new Date(), - }) - .where(eq(heartbeatRuns.id, run.id)); - lastOutputFlushAt = pendingOutputProgress.at; - outputProgressState.pending = null; - }; try { const startedAt = run.startedAt ?? new Date(); const runningWithSession = await db @@ -3502,100 +3453,8 @@ export function heartbeatService( message: "run started", }); - handle = await runLogStore.begin({ - companyId: run.companyId, - agentId: run.agentId, - runId, - }); - - await db - .update(heartbeatRuns) - .set({ - logStore: handle.store, - logRef: handle.logRef, - updatedAt: new Date(), - }) - .where(eq(heartbeatRuns.id, runId)); - - const currentUserRedactionOptions = - await getCurrentUserRedactionOptions(); - const appendIdentityRedactedLog = async (stream: "stdout" | "stderr", chunk: string) => { - const sanitizedChunk = compactRunLogChunk( - redactCurrentUserText(chunk, currentUserRedactionOptions), - ); - if (stream === "stdout") - stdoutExcerpt = appendExcerpt(stdoutExcerpt, sanitizedChunk); - if (stream === "stderr") - stderrExcerpt = appendExcerpt(stderrExcerpt, sanitizedChunk); - const ts = new Date().toISOString(); - - outputSeq += 1; - const chunkSeq = outputSeq; - let appendedBytes = 0; - if (handle) { - appendedBytes = await runLogStore.append(handle, { - stream, - chunk: sanitizedChunk, - ts, - seq: chunkSeq, - }); - persistedLogBytes += appendedBytes; - } - outputProgressState.pending = { - at: new Date(ts), - seq: chunkSeq, - stream, - bytes: persistedLogBytes, - }; - await flushOutputProgress(); - - // Streamed CLI output is real run activity: keep the in-memory - // runtime status ("Working... / X ago") fresh between structured - // events so sandbox runs with mid-run log streaming never show a - // minutes-stale timestamp. Throttled to avoid churning the live - // event stream on every 250ms tail chunk. - const logActivityAt = new Date(ts); - if ( - isHeartbeatRunRuntimeStatusActive(run.status) && - logActivityAt.getTime() - lastLogRuntimeStatusTouchMs >= - ACTIVE_RUN_LOG_RUNTIME_STATUS_REFRESH_INTERVAL_MS - ) { - lastLogRuntimeStatusTouchMs = logActivityAt.getTime(); - const touchedStatus = touchHeartbeatRunRuntimeStatus({ - companyId: run.companyId, - issueId, - agentId: run.agentId, - runId: run.id, - at: logActivityAt, - }); - if (touchedStatus) - publishHeartbeatRunRuntimeProgress(touchedStatus); - } - - const payloadChunk = - sanitizedChunk.length > MAX_LIVE_LOG_CHUNK_BYTES - ? sanitizedChunk.slice( - sanitizedChunk.length - MAX_LIVE_LOG_CHUNK_BYTES, - ) - : sanitizedChunk; - - publishLiveEvent({ - companyId: run.companyId, - type: "heartbeat.run.log", - payload: { - runId: run.id, - agentId: run.agentId, - issueId, - ts, - seq: chunkSeq, - stream, - chunk: payloadChunk, - truncated: payloadChunk.length !== sanitizedChunk.length, - }, - }); - }; - const onLog = (stream: "stdout" | "stderr", chunk: string) => - appendIdentityRedactedLog(stream, identityRedactor.chunk(stream, chunk)); + const { currentUserRedactionOptions, appendIdentityRedactedLog, onLog, onAdapterMeta, onAdapterEvent } = + await runOutput.begin(currentRun); if (runScopedMentionedSkillKeys.length > 0) { await onLog( "stdout", @@ -3683,36 +3542,6 @@ export function heartbeatService( ); } } - const onAdapterMeta = async (meta: AdapterInvocationMeta) => { - meta = identityRedactor.redact(meta); - if (meta.env && secretKeys.size > 0) { - for (const key of secretKeys) { - if (key in meta.env) meta.env[key] = "***REDACTED***"; - } - } - await appendRunEvent(currentRun, { - eventType: "adapter.invoke", - stream: "system", - level: "info", - message: "adapter invocation", - payload: meta as unknown as Record, - }); - }; - - const onAdapterEvent = async (event: AdapterRuntimeEvent) => { - event = identityRedactor.redact(event); - const eventType = event.eventType.trim(); - if (!eventType) return; - await appendRunEvent(currentRun, { - eventType: eventType.slice(0, 120), - stream: event.stream, - level: event.level, - color: event.color, - message: event.message, - payload: event.payload, - }); - }; - const adapter = getServerAdapter(agent.adapterType); const durableGoalControlRun = readNonEmptyString(context.goalControlRequestId) !== null || @@ -4101,11 +3930,7 @@ export function heartbeatService( } await runCompletion.completeRun({ run, agent, issueId, issueRef, signal: executionControl.controller.signal, - output: { - handle, outputProgressState, flushOutputProgress, - get stdoutExcerpt() { return stdoutExcerpt; }, - get stderrExcerpt() { return stderrExcerpt; }, - }, + output: runOutput, session: { runtimeForAdapter, previousSessionParams, taskKey, configuredModel, sessionConfigMetadata }, readFailureReportSecrets, adapterResult, sessionCodec, taskSessionForRun, sessionCompaction, @@ -4236,11 +4061,7 @@ export function heartbeatService( } await runCompletion.failRun({ run, agent, issueId, issueRef, signal: executionControl.controller.signal, - output: { - handle, outputProgressState, flushOutputProgress, - get stdoutExcerpt() { return stdoutExcerpt; }, - get stderrExcerpt() { return stderrExcerpt; }, - }, + output: runOutput, session: { runtimeForAdapter, previousSessionParams, taskKey, configuredModel, sessionConfigMetadata }, readFailureReportSecrets, runId, err, identityRedactor, requiredWorkspaceRestoreEvidence, legacyAdapterEntered, diff --git a/server/src/services/heartbeat/run-output.ts b/server/src/services/heartbeat/run-output.ts new file mode 100644 index 00000000000..249db52646e --- /dev/null +++ b/server/src/services/heartbeat/run-output.ts @@ -0,0 +1,209 @@ +import { eq } from "drizzle-orm"; +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import type { AdapterInvocationMeta, AdapterRuntimeEvent } from "../../adapters/index.js"; +import { redactCurrentUserText, type CurrentUserRedactionOptions } from "../../log-redaction.js"; +import type { createAgentIdentityRedactor } from "../agent-identity-redaction.js"; +import { touchHeartbeatRunRuntimeStatus } from "../heartbeat-run-runtime-status.js"; +import { publishLiveEvent } from "../live-events.js"; +import type { RunLogHandle, RunLogStore } from "../run-log-store.js"; +import { appendExcerpt, compactRunLogChunk } from "./run-log.js"; +import { isHeartbeatRunRuntimeStatusActive, publishHeartbeatRunRuntimeProgress, type createHeartbeatLifecycle } from "./run-lifecycle.js"; + +const MAX_LIVE_LOG_CHUNK_BYTES = 8 * 1024; +export const ACTIVE_RUN_OUTPUT_PROGRESS_FLUSH_INTERVAL_MS = 60 * 1000; +export const ACTIVE_RUN_LOG_RUNTIME_STATUS_REFRESH_INTERVAL_MS = 5 * 1000; + +type Run = typeof heartbeatRuns.$inferSelect; + +export interface HeartbeatRunOutputInput { + run: Run; + runId: string; + issueId: string | null; + runLogStore: Pick; + identityRedactor: Pick, "chunk" | "redact">; + secretKeys: Set; + appendRunEvent: ReturnType["appendRunEvent"]; + getCurrentUserRedactionOptions: () => Promise; +} + +/** One recorder per execution attempt; the log store itself stays service-scoped. */ +export function createHeartbeatRunOutput(db: Db, input: HeartbeatRunOutputInput) { + const { run, runId, issueId, runLogStore, identityRedactor, secretKeys, appendRunEvent, getCurrentUserRedactionOptions } = input; + let handle: RunLogHandle | null = null; + let stdoutExcerpt = ""; + let stderrExcerpt = ""; + let outputSeq = Number(run.lastOutputSeq ?? 0); + let lastOutputFlushAt: Date | null = run.lastOutputAt ?? null; + let lastLogRuntimeStatusTouchMs = 0; + const outputProgressState: { + pending: { + at: Date; + seq: number; + stream: "stdout" | "stderr"; + bytes: number; + } | null; + } = { pending: null }; + let persistedLogBytes = Number(run.logBytes ?? 0); + const flushOutputProgress = async (opts?: { force?: boolean }) => { + const pendingOutputProgress = outputProgressState.pending; + if (!pendingOutputProgress) return; + const shouldFlush = + opts?.force === true || + !lastOutputFlushAt || + pendingOutputProgress.at.getTime() - lastOutputFlushAt.getTime() >= + ACTIVE_RUN_OUTPUT_PROGRESS_FLUSH_INTERVAL_MS; + if (!shouldFlush) return; + await db + .update(heartbeatRuns) + .set({ + lastOutputAt: pendingOutputProgress.at, + lastOutputSeq: pendingOutputProgress.seq, + lastOutputStream: pendingOutputProgress.stream, + lastOutputBytes: pendingOutputProgress.bytes, + updatedAt: new Date(), + }) + .where(eq(heartbeatRuns.id, run.id)); + lastOutputFlushAt = pendingOutputProgress.at; + outputProgressState.pending = null; + }; + // Bind handlers to the run row persisted at execution start. Progress writes + // above use only the immutable run ID from the initial row. + async function begin(currentRun: Run) { + const run = currentRun; + handle = await runLogStore.begin({ + companyId: run.companyId, + agentId: run.agentId, + runId, + }); + + await db + .update(heartbeatRuns) + .set({ + logStore: handle.store, + logRef: handle.logRef, + updatedAt: new Date(), + }) + .where(eq(heartbeatRuns.id, runId)); + + const currentUserRedactionOptions = + await getCurrentUserRedactionOptions(); + const appendIdentityRedactedLog = async (stream: "stdout" | "stderr", chunk: string) => { + const sanitizedChunk = compactRunLogChunk( + redactCurrentUserText(chunk, currentUserRedactionOptions), + ); + if (stream === "stdout") + stdoutExcerpt = appendExcerpt(stdoutExcerpt, sanitizedChunk); + if (stream === "stderr") + stderrExcerpt = appendExcerpt(stderrExcerpt, sanitizedChunk); + const ts = new Date().toISOString(); + + outputSeq += 1; + const chunkSeq = outputSeq; + let appendedBytes = 0; + if (handle) { + appendedBytes = await runLogStore.append(handle, { + stream, + chunk: sanitizedChunk, + ts, + seq: chunkSeq, + }); + persistedLogBytes += appendedBytes; + } + outputProgressState.pending = { + at: new Date(ts), + seq: chunkSeq, + stream, + bytes: persistedLogBytes, + }; + await flushOutputProgress(); + + // Streamed CLI output is real run activity: keep the in-memory + // runtime status ("Working... / X ago") fresh between structured + // events so sandbox runs with mid-run log streaming never show a + // minutes-stale timestamp. Throttled to avoid churning the live + // event stream on every 250ms tail chunk. + const logActivityAt = new Date(ts); + if ( + isHeartbeatRunRuntimeStatusActive(run.status) && + logActivityAt.getTime() - lastLogRuntimeStatusTouchMs >= + ACTIVE_RUN_LOG_RUNTIME_STATUS_REFRESH_INTERVAL_MS + ) { + lastLogRuntimeStatusTouchMs = logActivityAt.getTime(); + const touchedStatus = touchHeartbeatRunRuntimeStatus({ + companyId: run.companyId, + issueId, + agentId: run.agentId, + runId: run.id, + at: logActivityAt, + }); + if (touchedStatus) + publishHeartbeatRunRuntimeProgress(touchedStatus); + } + + const payloadChunk = + sanitizedChunk.length > MAX_LIVE_LOG_CHUNK_BYTES + ? sanitizedChunk.slice( + sanitizedChunk.length - MAX_LIVE_LOG_CHUNK_BYTES, + ) + : sanitizedChunk; + + publishLiveEvent({ + companyId: run.companyId, + type: "heartbeat.run.log", + payload: { + runId: run.id, + agentId: run.agentId, + issueId, + ts, + seq: chunkSeq, + stream, + chunk: payloadChunk, + truncated: payloadChunk.length !== sanitizedChunk.length, + }, + }); + }; + const onLog = (stream: "stdout" | "stderr", chunk: string) => + appendIdentityRedactedLog(stream, identityRedactor.chunk(stream, chunk)); + const onAdapterMeta = async (meta: AdapterInvocationMeta) => { + meta = identityRedactor.redact(meta); + if (meta.env && secretKeys.size > 0) { + for (const key of secretKeys) { + if (key in meta.env) meta.env[key] = "***REDACTED***"; + } + } + await appendRunEvent(currentRun, { + eventType: "adapter.invoke", + stream: "system", + level: "info", + message: "adapter invocation", + payload: meta as unknown as Record, + }); + }; + + const onAdapterEvent = async (event: AdapterRuntimeEvent) => { + event = identityRedactor.redact(event); + const eventType = event.eventType.trim(); + if (!eventType) return; + await appendRunEvent(currentRun, { + eventType: eventType.slice(0, 120), + stream: event.stream, + level: event.level, + color: event.color, + message: event.message, + payload: event.payload, + }); + }; + + return { currentUserRedactionOptions, appendIdentityRedactedLog, onLog, onAdapterMeta, onAdapterEvent }; + } + + return { + begin, + get handle() { return handle; }, + outputProgressState, + flushOutputProgress, + // Finalization awaits I/O and can receive more output before persisting excerpts. + get stdoutExcerpt() { return stdoutExcerpt; }, + get stderrExcerpt() { return stderrExcerpt; }, + }; +} diff --git a/tests/e2e/workspace-bootstrap/recovery.spec.ts b/tests/e2e/workspace-bootstrap/recovery.spec.ts index 884a20a8bdc..e5136159860 100644 --- a/tests/e2e/workspace-bootstrap/recovery.spec.ts +++ b/tests/e2e/workspace-bootstrap/recovery.spec.ts @@ -75,6 +75,12 @@ for (const persistent of [false, true]) { // Independently confirm the configured source was never overwritten. expect(sourceCopy).toBeTruthy(); expect(completed.scheduledRetryAttempt).toBe(1); + const completedRun = await api(`/heartbeat-runs/${completed.id}`); + const completedLog = await api(`/heartbeat-runs/${completed.id}/log`); + expect(completedRun.stdoutExcerpt).toContain("Workspace recovery verification complete."); + expect(completedRun.lastOutputSeq).toBeGreaterThan(0); + expect(completedRun.lastOutputBytes).toBe(completedRun.logBytes); + expect(completedLog.content).toContain("Workspace recovery verification complete."); expect(await readFile(path.join(fixture.source, "README.md"), "utf8")).toBe("Existing uncommitted work\n"); } await page.reload(); From 914b70572c9a03348532e5e4ab9533927416cf83 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 12 Oct 2026 10:01:00 -0500 Subject: [PATCH 7/7] refactor(heartbeat): extract execution configuration (#15973) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat turns task, agent, environment, and account settings into an executable run. > - The executor still assembles scoped credentials and skill configuration inline between environment selection and session preparation. > - That phase has a clear input/output boundary, but credential acquisition must remain visible to cleanup even when later setup fails. > - This pull request extracts configuration assembly and immediately notifies the executor of acquired resources and redaction state. > - The executor retains admission, dispatch, and cleanup, with the same statement ordering and explicit busy-account deferral. ## Linked Issues or Issue Description **What existing behavior does this improve?** Maintenance of heartbeat execution. This PR follows #15969 and #15966, both now merged. The branch is rebased onto `master`; `git range-diff` confirms that the extraction patch is unchanged. **Current behavior** `heartbeat.ts` has 4,561 lines and 188,950 bytes. Configuration assembly mixes workspace overrides, secrets, managed AI preparation, and skill delivery into the executor. **Proposed behavior** `execution-configuration.ts` owns this preparation phase in 269 lines and 16,204 bytes. The executor calls it between environment and session preparation and retains immediate cleanup ownership. `heartbeat.ts` becomes 4,372 lines and 178,459 bytes. **Reason and benefit** The execution path exposes another named phase, with tests that exercise its effective configuration, resource lifetime, failure diagnostics, and busy-account result through public heartbeat dispatch. Related work: #11827 changes pre-dispatch secret refresh and #15355 changes GitHub delegation; this extraction preserves current policies. **Breaking changes** None intended. Public exports remain available, and the original helpers keep their modules. ## What Changed - Move configuration merging, scoped secret resolution, managed AI acquisition, runtime skills, connector instructions, and managed identity preparation into one typed module. - Notify the executor immediately when failure-redaction values, managed credentials, and identity redaction become available. Returning resources only after all preparation succeeds would lose cleanup on partial failure. - Preserve the live failure-secret getter for later runtime environment changes. Return an explicit deferred result for busy accounts. - Add four real-database characterization tests through `heartbeat.wakeup`, using actual credential files and cleanup with a fixture adapter. - Update the developer module map. ## Verification - Before extraction: all 45 tests pass across the new public-path tests and existing project-environment tests. - After extraction: 47 tests pass across execution configuration, project environment, and runtime skills. No skips. - Two additional real-process managed identity tests pass; nine unrelated cases are deliberately filtered out. - `pnpm -r typecheck` and `pnpm build` pass. - Normalized syntax comparison confirms the moved body and reconstructed executor match after accounting for immediate ownership callbacks and the explicit prepared/deferred result. - Isolated browser acceptance: 2/2 pass (3.9 minutes). A real process worker completes after Git recovery, with persisted stdout and matching final output byte counts. Persistent failures stop after three attempts. Both outcomes survive reload. The disposable server was shut down after verification. - All eleven new public-path characterization cases across the three-PR stack pass together on the final commit. - All 140 named public exports from the batch baseline remain available. - CI initially failed the legacy-continuation message browser case: periodic recovery escalated the issue, then queue admission cancelled the run with `execution_reconciliation_required` before configuration preparation. The unchanged predecessor shard passes, and the exact case passes locally (5 seconds in the browser). The original failing case passed on its single rerun (8.2 seconds), but GitHub then shut down the runner during a different test (`The runner has received a shutdown signal`). The separate infrastructure restart completed: 24 browser cases passed, with two existing optional cases skipped. Assertions and timeouts are unchanged. - Greptile: 5/5 on `27ab7e3df64c157ba1f00c5757db812ff7a4e5aa`, with no actionable findings. Hosted full-suite CI passes on this exact commit: all 54 check runs and the additional commit status are successful. - Merge preparation: all eleven characterization tests pass after rebasing onto `master`. Fresh full CI passes for `1758673f87eaa76a8b2c520eac4b631ada5d9ba1`: 55 successful checks and two optional Storybook skips, including all eight browser shards, typecheck, tests, and build. Greptile is 5/5 on this exact commit with zero review threads. All heartbeat source, tests, documentation, and browser assertions are identical to the previously reviewed final stack. ## Risks - A failure after AI acquisition must still remove the disposable credential home. The new public-path failure test proves actual filesystem cleanup and diagnostic redaction before the external reporting sink. - The baseline exposed an existing gap: arbitrary exception text in the local `run.error` field can contain a credential even though the failure reporter redacts it. This behavior is unchanged and merits a separate fix; the test does not claim the local error field is sanitized. - Account-busy deferral must return without dispatch while retaining its scheduled retry. The new test checks the durable run states. - Browser acceptance uses a real process fixture and disposable data, without live provider calls. ## Model Used OpenAI Codex, based on GPT-6, assisted with reasoning, code edits, tool execution, and verification. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/DEVELOPING.md | 8 + .../heartbeat-execution-configuration.test.ts | 160 +++++++++++ server/src/services/heartbeat.ts | 225 ++------------- .../heartbeat/execution-configuration.ts | 269 ++++++++++++++++++ 4 files changed, 455 insertions(+), 207 deletions(-) create mode 100644 server/src/__tests__/heartbeat-execution-configuration.test.ts create mode 100644 server/src/services/heartbeat/execution-configuration.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index da1bc58a960..90b2760794e 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1293,6 +1293,14 @@ cleanup. Public-path characterization tests are in browser fixture also checks the persisted prompt variants and description revision from a real process adapter before completing its task. +Execution configuration is assembled in +`server/src/services/heartbeat/execution-configuration.ts`: workspace and issue +overrides, scoped credentials, managed AI identity, and runtime skill delivery. +The executor receives credential ownership and failure-redaction updates as soon +as they are available, so later setup failures still clean up acquired resources. +Busy-account deferral returns explicitly before session/workspace preparation. +Public-path tests are in `server/src/__tests__/heartbeat-execution-configuration.test.ts`. + Session preparation is in `server/src/services/heartbeat/session-preparation.ts`. It selects fresh or resumable sessions from effective config and managed-account identity, then resolves adapter session identity and compaction after workspace diff --git a/server/src/__tests__/heartbeat-execution-configuration.test.ts b/server/src/__tests__/heartbeat-execution-configuration.test.ts new file mode 100644 index 00000000000..ee1dbd66298 --- /dev/null +++ b/server/src/__tests__/heartbeat-execution-configuration.test.ts @@ -0,0 +1,160 @@ +import { randomUUID } from "node:crypto"; +import { access, readFile } from "node:fs/promises"; +import path from "node:path"; +import { eq } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, companyMemberships, createDb, heartbeatRunEvents, heartbeatRuns, issues, projects } from "@paperclipai/db"; +import { registerServerAdapter, unregisterServerAdapter, type AdapterExecutionContext } from "../adapters/index.js"; +import { HttpError } from "../errors.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import { aiConnectionService } from "../services/ai-connections.js"; +import * as aiRuntime from "../services/ai-connection-runtime.js"; +import * as sentry from "../sentry.js"; +import { waitForPendingRunFailureReports } from "../services/run-failure-report.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const skillFailures = vi.hoisted(() => new Map()); +vi.mock("../services/company-skills.js", async importOriginal => { + const actual = await importOriginal(); + return { ...actual, companySkillService: (...args: Parameters) => { + const service = actual.companySkillService(...args); + return { ...service, listRuntimeSkillEntries: async (...input: Parameters) => { + const failure = skillFailures.get(input[0]); + if (failure) throw new Error(failure); + return service.listRuntimeSkillEntries(...input); + } }; + } }; +}); + +const support = await getEmbeddedPostgresTestSupport(); +const describePostgres = support.supported ? describe : describe.skip; +if (!support.supported) console.warn(`Skipping execution configuration tests: ${support.reason}`); + +describePostgres("heartbeat execution configuration through adapter dispatch", () => { + let temporary: Awaited>; + let db: ReturnType; + let heartbeat: ReturnType; + const captured = new Map(); + const inspect = new Map Promise>(); + const adapterType = "codex_local"; + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("heartbeat-execution-config-"); + db = createDb(temporary.connectionString); + heartbeat = heartbeatService(db); + // Exercise real preparation for a managed-capable adapter without calling a provider. + registerServerAdapter({ + type: adapterType, supportsLocalAgentJwt: false, + execute: async input => { + captured.set(input.runId, input); + await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, String(input.context.issueId))); + await inspect.get(input.agent.id)?.(input); + return { exitCode: 0, signal: null, timedOut: false, resultJson: {} }; + }, + testEnvironment: async () => ({ adapterType, status: "pass", checks: [], testedAt: new Date().toISOString() }), + }); + }, 30_000); + afterEach(() => { vi.restoreAllMocks(); skillFailures.clear(); }); + afterAll(async () => { + await heartbeat?.drainActiveRunExecutions(); + unregisterServerAdapter(adapterType); + await db?.$client.end({ timeout: 0 }); + await temporary?.cleanup(); + }); + + async function seed(managed = false) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), projectId = randomUUID(); + const userId = `configuration-user-${companyId}`; + const credential = `fixture-api-key-${companyId}`; + await db.insert(companies).values({ id: companyId, name: "Execution configuration fixture", issuePrefix: `C${companyId.slice(0, 6)}` }); + await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" }); + const account = managed ? await aiConnectionService(db).save(companyId, userId, { + provider: "openai", method: "api_key", ownership: "personal", name: "Fixture account", apiKey: credential, agentIds: [], allAgents: true, + }, credential) : null; + await db.insert(projects).values({ id: projectId, companyId, name: "Config project", env: { SHARED_VALUE: "project", PROJECT_ONLY: "project-only" } }); + await db.insert(agents).values({ id: agentId, companyId, name: "Configuration reader", adapterType, status: "idle", + adapterConfig: { model: "agent-model", env: { SHARED_VALUE: "agent", AGENT_ONLY: "agent-only", ...(managed ? { OPENAI_API_KEY: "stale-agent-key" } : {}) } }, + runtimeConfig: account ? { aiConnection: { provider: "openai", method: "api_key", mode: "delegated", connectionId: account.connectionId, grantId: account.grantId } } : {}, + }); + await db.insert(issues).values({ id: issueId, companyId, projectId, title: "Configuration fixture", status: "todo", assigneeAgentId: agentId, responsibleUserId: userId, issueNumber: 1, assigneeAdapterOverrides: { adapterConfig: { model: "issue-model" } } }); + return { companyId, agentId, issueId, credential }; + } + + async function execute(f: Awaited>) { + const run = await heartbeat.wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "manual", payload: { issueId: f.issueId }, contextSnapshot: { + issueId: f.issueId, paperclipSecrets: { manifest: [{ secretKey: "untrusted-wake-value" }] }, + paperclipWake: { connectorSkillInstructions: "untrusted wake instructions" }, + } }); + expect(run).not.toBeNull(); + await heartbeat.drainActiveRunExecutions(); + const [finished] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)); + expect(finished).toBeDefined(); + return { finished, input: captured.get(run!.id) }; + } + + it("applies project environment and issue overrides and replaces caller-supplied runtime fields", async () => { + const f = await seed(); + const { finished, input } = await execute(f); + expect(finished.status, finished.error ?? undefined).toBe("succeeded"); + expect(input?.config).toMatchObject({ model: "issue-model", env: { SHARED_VALUE: "project", AGENT_ONLY: "agent-only", PROJECT_ONLY: "project-only" } }); + expect(input?.context.paperclipSecrets).toBeUndefined(); + expect(JSON.stringify(input?.context.paperclipWake)).not.toContain("untrusted wake instructions"); + }); + + it("delivers the selected credential in an isolated home and removes it after completion", async () => { + const f = await seed(true); + let providerHome: string | undefined; + inspect.set(f.agentId, async input => { + const env = input.config.env as Record; + providerHome = env.CODEX_HOME; + expect(env.OPENAI_API_KEY).toBe(f.credential); + expect(env.CODEX_API_KEY).toBe(f.credential); + expect(JSON.parse(await readFile(path.join(providerHome, "auth.json"), "utf8"))).toEqual({ OPENAI_API_KEY: f.credential }); + await input.onMeta?.({ adapterType, command: "fixture", env, prompt: "fixture" }); + }); + const { finished, input } = await execute(f); + expect(finished.status, finished.error ?? undefined).toBe("succeeded"); + expect(input?.context.aiConnection).toMatchObject({ accountName: "Fixture account", provider: "openai" }); + expect(providerHome).toEqual(expect.any(String)); + await expect(access(providerHome!)).rejects.toMatchObject({ code: "ENOENT" }); + const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, finished.id)); + expect(events.some(event => event.eventType === "adapter.invoke")).toBe(true); + expect(JSON.stringify(events)).not.toContain(f.credential); + }); + + it("cleans acquired credentials and redacts diagnostics when later skill preparation fails", async () => { + const f = await seed(true); + skillFailures.set(f.companyId, `Invalid fixture skill configuration: ${f.credential}`); + const capturedFailure = vi.spyOn(sentry, "captureRunFailure").mockImplementation(() => undefined); + const prepare = aiRuntime.prepareManagedAiRuntime; + let acquiredHome: string | undefined; + vi.spyOn(aiRuntime, "prepareManagedAiRuntime").mockImplementation(async (...args) => { + const runtime = await prepare(...args); + acquiredHome = runtime.home; + await access(acquiredHome!); + return runtime; + }); + const { finished, input } = await execute(f); + expect(finished.status).toBe("failed"); + expect(input).toBeUndefined(); + expect(acquiredHome).toEqual(expect.any(String)); + await expect(access(acquiredHome!)).rejects.toMatchObject({ code: "ENOENT" }); + expect(finished.error).toContain("Invalid fixture skill configuration"); + // The diagnostic reporter redacts runtime values before the external sink. + // The legacy local run.error field does not apply this value-based redaction. + await waitForPendingRunFailureReports(); + expect(capturedFailure).toHaveBeenCalled(); + expect(JSON.stringify(capturedFailure.mock.calls)).toContain("Invalid fixture skill configuration"); + expect(JSON.stringify(capturedFailure.mock.calls)).not.toContain(f.credential); + }); + + it("defers a busy account without dispatch and preserves its scheduled retry", async () => { + const f = await seed(true); + vi.spyOn(aiRuntime, "prepareManagedAiRuntime").mockRejectedValueOnce(new HttpError(422, "AI credentials are being updated. Retry shortly.", { code: "ai_connection_busy" })); + const { finished, input } = await execute(f); + expect(input).toBeUndefined(); + expect(finished).toMatchObject({ status: "cancelled", errorCode: "ai_connection_busy", resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false } }, contextSnapshot: { aiConnectionBusyDeferredWhileAssignee: true } }); + const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agentId)); + expect(runs.filter(run => run.id !== finished.id)).toEqual([expect.objectContaining({ status: "scheduled_retry", scheduledRetryReason: "ai_connection_busy" })]); + }); +}); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 54006b8b7ef..8f562fb8627 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { prepareHeartbeatExecutionConfiguration } from "./heartbeat/execution-configuration.js"; import { createHeartbeatRunOutput } from "./heartbeat/run-output.js"; import { prepareHeartbeatSession, resolveHeartbeatAdapterSession } from "./heartbeat/session-preparation.js"; import { prepareHeartbeatTaskContext } from "./heartbeat/task-context.js"; @@ -113,9 +114,6 @@ import { resolveAcceptedPlanWakeRoutingDecision, clearInteractionContinuationWakeContext, PAPERCLIP_WAKE_PAYLOAD_KEY, - resolveRunScopedMentionedSkillKeys, - resolveExecutionRunAdapterConfig, - applyRunScopedMentionedSkillKeys, MANAGED_GITHUB_TOKEN_KEYS, configuredPaperclipApiBaseUrl, createHeartbeatRunPreparation, @@ -134,8 +132,6 @@ export { buildPaperclipWakePayload, } from "./heartbeat/run-preparation.js"; import { - buildExecutionWorkspaceConfigSnapshot, - stripWorkspaceRuntimeFromExecutionRunConfig, resolveRuntimeSessionParamsForWorkspace, buildRunWorkspaceHints, buildReferencedProjectRunObservability, @@ -211,10 +207,7 @@ export { buildPaperclipTaskMarkdown } from "./heartbeat/task-markdown.js"; import { decisionModelService } from "./decision-models.js"; import { TASK_QUESTION_GUIDANCE } from "./issue-question-context.js"; import { createAgentIdentityRedactor } from "./agent-identity-redaction.js"; -import { agentIdentityService, supportsManagedAgentIdentity } from "./agent-identity.js"; -import { prepareConnectionInstructionDelivery } from "./connection-instructions.js"; -import { resolveAssignedConnectionInstructionsForRun } from "./native-runtime/assigned-mcp-tools.js"; import { startupCancellationFence, @@ -250,19 +243,13 @@ import { watchLegacyControllerLease, } from "./legacy-controller-lease.js"; -import { applyConnectorSkills, prepareConnectorSkillDelivery, resolveConnectorAssignments } from "./connector-runtime.js"; -import { connectionIntentService } from "./connection-intents.js"; import { prepareManagedAiRuntime, assertManagedAiProjectAuth, - stripAiAuthBindings, - isAiConnectionBusy, - AI_AUTH_ENV_KEYS, } from "./ai-connection-runtime.js"; import { aiConnectionBindingSchema, aiRuntimeConnectionBindingSchema, type AiConnectionRouterSelection } from "@paperclipai/shared"; import { aiConnectionRouterService, AiConnectionPoolExhausted, applyAiConnectionRouterTaskSettings } from "./ai-connection-router.js"; -import { managedAiSessionIdentityCompatible } from "./ai-connection-session.js"; import { getExecutionBlocker, } from "./execution-blocker.js"; @@ -336,7 +323,6 @@ import { HttpError, notFound } from "../errors.js"; import { logger } from "../middleware/logger.js"; import { - resolveManagedGitHubIdentitySelection, scrubGitCredentialText, } from "./git-credentials.js"; // Re-exported because heartbeat's workspace surface exposed the scrubber before the @@ -360,10 +346,6 @@ import { resolveHeartbeatNativeRuntimeMode, } from "./native-runtime/index.js"; -import { - recordFailedSkillPreparation, - type NativeRunHistoricalSpan, -} from "./native-runtime/native-run-trace.js"; import { parseNativeExecutionInput, type NativeExecutionInput, @@ -396,7 +378,6 @@ import { emitAgentTaskRun, emitAgentTaskRunById, } from "./agent-task-run-telemetry.js"; -import { readAiConnectionConfigurationFailure, readAiCredentialAccessFailure } from "./ai-connection-configuration-failure.js"; import { reportRunFailure } from "./run-failure-report.js"; import { collectRunFailureSecretValues, type RunFailureReportOptions } from "./run-failure-diagnostics.js"; @@ -463,7 +444,6 @@ import { } from "./run-scratch.js"; import { applyDefaultIsolatedExecutionWorkspacePolicy, - buildExecutionWorkspaceAdapterConfig, gateProjectExecutionWorkspacePolicy, parseIssueExecutionWorkspaceSettings, parseProjectExecutionWorkspacePolicy, @@ -489,7 +469,6 @@ import { type PostCommitEffect, MAX_TURN_CONTINUATION_RETRY_REASON, WORKSPACE_BUSY_RETRY_REASON, - AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON, INTERACTION_CONTINUATION_INFRA_RETRY_REASON, INTERACTION_CONTINUATION_INFRA_WAKE_REASON, @@ -516,9 +495,6 @@ import { import { createRunSecretRedactionRegistry } from "./run-secret-redaction.js"; -import { - readPaperclipSkillSyncPreference, -} from "@paperclipai/adapter-utils/server-utils"; import { environmentService } from "./environments.js"; @@ -526,7 +502,6 @@ import { environmentRuntimeService, type ProviderResourceDisposition, } from "./environment-runtime.js"; -import { skillVersionSelectionMap } from "./runtime-skill-selections.js"; import { environmentRunOrchestrator } from "./environment-run-orchestrator.js"; import { @@ -544,9 +519,6 @@ import type { PluginWorkerManager } from "./plugin-worker-manager.js"; import { computeTaskDrain, applyTaskDrain, startTaskDrain, stopTaskDrain, readTaskDrain } from "./task-admission.js"; -const ACCEPTED_PLAN_CONVERSION_SKILL_KEY = - "paperclipai/paperclip/paperclip-converting-plans-to-tasks"; - const execFile = promisify(execFileCallback); const UNSUCCESSFUL_HEARTBEAT_RUN_TERMINAL_STATUSES = [ "failed", @@ -2698,185 +2670,24 @@ export function heartbeatService( config: { config, issueAssigneeOverrides }, services: { executionWorkspacesSvc, environmentsSvc, instanceSettings, findSharedWorkspaceHolder }, }); - const workspaceManagedConfig = buildExecutionWorkspaceAdapterConfig({ - agentConfig: config, - projectPolicy: projectExecutionWorkspacePolicy, - issueSettings: issueExecutionWorkspaceSettings, - mode: requestedExecutionWorkspaceMode, - legacyUseProjectWorkspace: - issueAssigneeOverrides?.useProjectWorkspace ?? null, + const executionConfiguration = await prepareHeartbeatExecutionConfiguration(db, { + run, agent, + task: { issueId, context, responsibleUserId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision, projectContext, routineEnvContext }, + config: { config, issueAssigneeOverrides, requestedAiBinding, aiBinding }, + policy: { trustPreset, projectExecutionWorkspacePolicy, issueExecutionWorkspaceSettings, requestedExecutionWorkspaceMode, useIsolatedTaskDirectory }, + environment: { selectedEnvironmentId, selectedEnvironmentForConfig, resolvedInstanceSettings, persistedNativeExecutionInput }, + services: { secretsSvc, companySkills, appendRunEvent, finalizeAiConnectionBusyDeferral }, + onFailureSecretsResolved: (readSecrets) => { readFailureReportSecrets = readSecrets; }, + onManagedAiRuntimeAcquired: (runtime) => { managedAiRuntime = runtime; }, + onIdentityRedactorPrepared: (redactor) => { identityRedactor = redactor; }, }); - const mergedConfig = { - ...workspaceManagedConfig, - ...Object.fromEntries(Object.entries(issueAssigneeOverrides?.adapterConfig ?? {}).filter(([key]) => requestedAiBinding?.mode !== "router" || !["provider", "acpxAgent", "model", "modelReasoningEffort", "reasoningEffort", "effort", "variant"].includes(key))), - ...(requestedAiBinding?.mode === "router" ? parseObject(parseObject(context.aiRouterSelection).runtimeConfig) : {}), - // The base below is already task-owned. Keep directory transport while - // preserving isolated mode and the mandatory sandbox preflight. - ...(useIsolatedTaskDirectory ? { workspaceStrategy: { type: "project_primary" } } : {}), - }; - const configSnapshot = buildExecutionWorkspaceConfigSnapshot( - mergedConfig, - selectedEnvironmentId, - ); - const executionRunConfig = - stripWorkspaceRuntimeFromExecutionRunConfig(mergedConfig); - const runScopedMentionedSkillKeys = - await resolveRunScopedMentionedSkillKeys({ - db, - companyId: agent.companyId, - issueId, - commentIds: extractWakeCommentIds(context), - }); - const runScopedSkillKeys = - acceptedPlanContinuationWake && - !acceptedPlanWakeRoutingDecision?.suppressAcceptedContinuation - ? [...runScopedMentionedSkillKeys, ACCEPTED_PLAN_CONVERSION_SKILL_KEY] - : runScopedMentionedSkillKeys; - const githubSelection = await resolveManagedGitHubIdentitySelection( - db, - agent.companyId, - { - agentId: agent.id, - responsibleUserId, - allowStandingDelegation: false, - }, - ); - const useHostGitHub = - !githubSelection.configured && - trustPreset.kind === "standard" && - ["local", "ssh"].includes( - selectedEnvironmentForConfig?.driver ?? "local", - ); - - const { resolvedConfig, configuredTaskEnvironment, secretKeys, secretManifest } = - await resolveExecutionRunAdapterConfig({ - managedAiCredentials: Boolean(aiBinding), - managedGitHubCredentials: !useHostGitHub, - companyId: agent.companyId, - agentId: agent.id, - adapterType: agent.adapterType, - issueId, - heartbeatRunId: run.id, - environmentId: selectedEnvironmentForConfig?.id ?? null, - environmentEnv: aiBinding ? stripAiAuthBindings(selectedEnvironmentForConfig?.envVars) : selectedEnvironmentForConfig?.envVars ?? null, - environmentDriver: selectedEnvironmentForConfig?.driver ?? null, - projectId: projectContext?.id ?? null, - routineId: routineEnvContext.routineId, - responsibleUserId, - executionRunConfig: aiBinding ? { ...executionRunConfig, env: stripAiAuthBindings(executionRunConfig.env) } : executionRunConfig, - projectEnv: aiBinding ? stripAiAuthBindings(projectContext?.env) : projectContext?.env ?? null, - routineEnv: aiBinding ? stripAiAuthBindings(routineEnvContext.env) : routineEnvContext.env, - secretsSvc, - trustPreset, - }); - readFailureReportSecrets = () => collectRunFailureSecretValues(resolvedConfig.env, secretKeys); - if (aiBinding) { - try { - managedAiRuntime = await prepareManagedAiRuntime(db, { companyId: agent.companyId, agentId: agent.id, responsibleUserId, adapterType: agent.adapterType, binding: aiBinding, config: resolvedConfig }); - } catch (error) { - // Only fresh executions can receive a pre-provider wait receipt. A - // persisted native input may already have provider effects to recover. - if (isAiConnectionBusy(error) && !persistedNativeExecutionInput) { - // Use the authority recorded by the locked admission gate, never - // the issue's mutable assignee observed during runtime preparation. - const authorizedNonAssigneeWake = - parseObject(run.runnerProfileJson).aiConnectionNonAssigneeCommentWake === true || - (run.scheduledRetryReason === AI_CONNECTION_BUSY_RETRY_REASON && - isNonAssigneeWorkspaceBusyRetry(run.scheduledRetryReason, parseObject(run.contextSnapshot))); - await finalizeAiConnectionBusyDeferral(run, error, !authorizedNonAssigneeWake); - return; - } - if (responsibleUserId && issueId) { - await connectionIntentService(db).request({ sub: agent.id, company_id: agent.companyId, run_id: run.id, responsible_user_id: responsibleUserId }, aiBinding.provider, { purpose: "ai" }).catch(() => { - logger.warn({ runId: run.id, agentId: agent.id }, "Could not attach AI connection request; runtime configuration action remains available"); - }); - } - throw new ConfigurationIncompleteFailure(error instanceof Error ? error.message : "Configure this agent’s AI connection", { - configurationIncomplete: { reason: "ai_connection_unavailable", companyId: agent.companyId, agentId: agent.id, responsibleUserId, - ...(!persistedNativeExecutionInput && readAiConnectionConfigurationFailure(error) - ? { selectionFailure: readAiConnectionConfigurationFailure(error) } : {}), - ...(readAiCredentialAccessFailure(error) - ? { credentialAccess: readAiCredentialAccessFailure(error) } : {}), - provider: aiBinding.provider, method: aiBinding.method, actionUrl: `/agents/${agent.id}/runtime`, - fingerprint: `ai:${agent.id}:${responsibleUserId}:${JSON.stringify(aiBinding)}` }, - }); - } - const savedAiAccount = parseObject(run.contextSnapshot?.aiConnection); - if (persistedNativeExecutionInput && !managedAiSessionIdentityCompatible( - savedAiAccount.sessionIdentity ?? savedAiAccount.identity, - managedAiRuntime.sessionIdentity, - managedAiRuntime.identity, - )) { - throw new ConfigurationIncompleteFailure("The AI account changed while this native run was suspended. Start a new execution.", { configurationIncomplete: { reason: "ai_connection_changed", actionUrl: `/agents/${agent.id}/runtime` } }); - } - Object.assign(resolvedConfig, managedAiRuntime.config); - for (const key of AI_AUTH_ENV_KEYS) secretKeys.add(key); - context.aiConnection = { ...managedAiRuntime.attribution, accountName: managedAiRuntime.accountName, identity: managedAiRuntime.identity, sessionIdentity: managedAiRuntime.sessionIdentity }; - await db.update(heartbeatRuns).set({ contextSnapshot: sql`coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) || ${JSON.stringify({ aiConnection: context.aiConnection, ...(context.aiRouterTaskKey ? { aiRouterTaskKey: context.aiRouterTaskKey } : {}), ...(context.aiRouterSelection ? { aiRouterSelection: context.aiRouterSelection } : {}) })}::jsonb` }).where(eq(heartbeatRuns.id, run.id)); - } - if (secretManifest.length > 0) { - context.paperclipSecrets = { - manifest: secretManifest, - }; - } else { - delete context.paperclipSecrets; - } - const effectiveResolvedConfig = applyRunScopedMentionedSkillKeys( - resolvedConfig, - runScopedSkillKeys, - ); - const runtimeSkillPreference = readPaperclipSkillSyncPreference( - effectiveResolvedConfig, - ); - const nativeRunnerPreparationSpans: NativeRunHistoricalSpan[] = []; - const skillsPrepareStartedAtMs = Date.now(); - const runtimeSkillEntries = await (async () => { - try { - return await companySkills.listRuntimeSkillEntries(agent.companyId, { - versionSelections: skillVersionSelectionMap( - runtimeSkillPreference.desiredSkillEntries, - { - versionPinsEnabled: - resolvedInstanceSettings.experimental.enableBetaSkills === - true, - }, - ), - }); - } catch (error) { - if (agent.adapterType === "paperclip_runner") { - await recordFailedSkillPreparation({ - runId: run.id, - startedAtMs: skillsPrepareStartedAtMs, - onEvent: async (event) => { - await appendRunEvent(run, event); - }, - }); - } - throw error; - } - })(); - nativeRunnerPreparationSpans.push({ - name: "skills.prepare", - parentName: "task.prepare", - startedAtMs: skillsPrepareStartedAtMs, - endedAtMs: Date.now(), - }); - const connectorAssignments = await resolveConnectorAssignments(db, { companyId: agent.companyId, agentId: agent.id, runId: run.id, issueId: typeof context.issueId === "string" ? context.issueId : undefined }); - const connectorSkillConfig = await applyConnectorSkills(effectiveResolvedConfig, runtimeSkillEntries, connectorAssignments); - // Both CLI adapters and native context materialization use the same resolved set. - runtimeSkillEntries.splice(0, runtimeSkillEntries.length, ...connectorSkillConfig.paperclipRuntimeSkills); - const connectorDelivery = await prepareConnectorSkillDelivery(connectorSkillConfig, agent.adapterType); - // Always replace this runtime-only field; caller wake data cannot supply skills. - context.paperclipWake = { ...parseObject(context.paperclipWake), connectorSkillInstructions: connectorDelivery.instructions }; - let runtimeConfig = await prepareConnectionInstructionDelivery({ - resolve: () => resolveAssignedConnectionInstructionsForRun(db, { companyId: agent.companyId, agentId: agent.id, runId: run.id }), - context, config: connectorDelivery.config, native: agent.adapterType === "paperclip_runner", - }); - const agentIdentity = supportsManagedAgentIdentity(agent.adapterType, agent.adapterConfig, run.runtimeMode === "native" ? run.driverKind : undefined) - ? await agentIdentityService(db).ensureAgentIdentity(agent.companyId, agent.id) - : undefined; - identityRedactor = createAgentIdentityRedactor(agentIdentity?.privateKeyPem); - if (agentIdentity) secretKeys.add("PAPERCLIP_AGENT_PRIVATE_KEY"); + if (!executionConfiguration.prepared) return; + const { + mergedConfig, configSnapshot, runScopedMentionedSkillKeys, githubSelection, useHostGitHub, + resolvedConfig, configuredTaskEnvironment, secretKeys, secretManifest, + runtimeSkillEntries, nativeRunnerPreparationSpans, agentIdentity, + } = executionConfiguration; + let { runtimeConfig } = executionConfiguration; const resolvedFailureSecrets = readFailureReportSecrets(); readFailureReportSecrets = () => [ ...resolvedFailureSecrets, diff --git a/server/src/services/heartbeat/execution-configuration.ts b/server/src/services/heartbeat/execution-configuration.ts new file mode 100644 index 00000000000..859371de9aa --- /dev/null +++ b/server/src/services/heartbeat/execution-configuration.ts @@ -0,0 +1,269 @@ +import { eq, sql } from "drizzle-orm"; +import { heartbeatRuns, type Db, type agents, type projects } from "@paperclipai/db"; +import type { AiConnectionBinding, AiRuntimeConnectionBinding } from "@paperclipai/shared"; +import { readPaperclipSkillSyncPreference } from "@paperclipai/adapter-utils/server-utils"; +import { parseObject } from "../../adapters/utils.js"; +import { logger } from "../../middleware/logger.js"; +import { AI_CONNECTION_BUSY_RETRY_REASON, extractWakeCommentIds, isNonAssigneeWorkspaceBusyRetry } from "../../modules/run-dispatch/index.js"; +import { createAgentIdentityRedactor } from "../agent-identity-redaction.js"; +import { agentIdentityService, supportsManagedAgentIdentity } from "../agent-identity.js"; +import { readAiConnectionConfigurationFailure, readAiCredentialAccessFailure } from "../ai-connection-configuration-failure.js"; +import { AI_AUTH_ENV_KEYS, isAiConnectionBusy, prepareManagedAiRuntime, stripAiAuthBindings } from "../ai-connection-runtime.js"; +import { managedAiSessionIdentityCompatible } from "../ai-connection-session.js"; +import type { companySkillService } from "../company-skills.js"; +import { connectionIntentService } from "../connection-intents.js"; +import { prepareConnectionInstructionDelivery } from "../connection-instructions.js"; +import { applyConnectorSkills, prepareConnectorSkillDelivery, resolveConnectorAssignments } from "../connector-runtime.js"; +import { buildExecutionWorkspaceAdapterConfig } from "../execution-workspace-policy.js"; +import { resolveManagedGitHubIdentitySelection } from "../git-credentials.js"; +import { resolveAssignedConnectionInstructionsForRun } from "../native-runtime/assigned-mcp-tools.js"; +import { recordFailedSkillPreparation, type NativeRunHistoricalSpan } from "../native-runtime/native-run-trace.js"; +import { collectRunFailureSecretValues } from "../run-failure-diagnostics.js"; +import { skillVersionSelectionMap } from "../runtime-skill-selections.js"; +import type { selectHeartbeatEnvironment } from "./environment-selection.js"; +import type { createHeartbeatRetries } from "./retries.js"; +import type { createHeartbeatLifecycle } from "./run-lifecycle.js"; +import { + ConfigurationIncompleteFailure, applyRunScopedMentionedSkillKeys, resolveExecutionRunAdapterConfig, + resolveRunScopedMentionedSkillKeys, type createHeartbeatRunPreparation, type resolveAcceptedPlanWakeRoutingDecision, +} from "./run-preparation.js"; +import { buildExecutionWorkspaceConfigSnapshot, stripWorkspaceRuntimeFromExecutionRunConfig } from "./workspaces.js"; +import type { HeartbeatWorkspacePreparationInput } from "./workspace-preparation.js"; + +const ACCEPTED_PLAN_CONVERSION_SKILL_KEY = + "paperclipai/paperclip/paperclip-converting-plans-to-tasks"; + +type ManagedAiRuntime = Awaited>; +type EnvironmentSelection = Awaited>; + +export interface HeartbeatExecutionConfigurationInput { + run: typeof heartbeatRuns.$inferSelect; + agent: typeof agents.$inferSelect; + task: { + issueId: string | null; + context: Record; + responsibleUserId: string | null; + acceptedPlanContinuationWake: boolean; + acceptedPlanWakeRoutingDecision: Awaited> | null; + projectContext: Pick | null; + routineEnvContext: Awaited["getRoutineEnvForExecutionIssue"]>>; + }; + config: { + config: Record; + issueAssigneeOverrides: { adapterConfig: Record | null; useProjectWorkspace: boolean | null } | null; + requestedAiBinding: AiRuntimeConnectionBinding | undefined; + aiBinding: AiConnectionBinding | undefined; + }; + policy: Pick; + environment: Pick; + services: { + secretsSvc: Parameters[0]["secretsSvc"]; + companySkills: Pick, "listRuntimeSkillEntries">; + appendRunEvent: ReturnType["appendRunEvent"]; + finalizeAiConnectionBusyDeferral: ReturnType["finalizeAiConnectionBusyDeferral"]; + }; + // The executor must observe these before a later preparation step can throw. + onFailureSecretsResolved: (readSecrets: () => string[]) => void; + onManagedAiRuntimeAcquired: (runtime: ManagedAiRuntime) => void; + onIdentityRedactorPrepared: (redactor: ReturnType) => void; +} + +/** Assembles effective run configuration; the executor retains cleanup ownership. */ +export async function prepareHeartbeatExecutionConfiguration(db: Db, input: HeartbeatExecutionConfigurationInput) { + const { run, agent, onFailureSecretsResolved, onManagedAiRuntimeAcquired, onIdentityRedactorPrepared } = input; + const { issueId, context, responsibleUserId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision, projectContext, routineEnvContext } = input.task; + const { config, issueAssigneeOverrides, requestedAiBinding, aiBinding } = input.config; + const { trustPreset, projectExecutionWorkspacePolicy, issueExecutionWorkspaceSettings, requestedExecutionWorkspaceMode, useIsolatedTaskDirectory } = input.policy; + const { selectedEnvironmentId, selectedEnvironmentForConfig, resolvedInstanceSettings, persistedNativeExecutionInput } = input.environment; + const { secretsSvc, companySkills, appendRunEvent, finalizeAiConnectionBusyDeferral } = input.services; + let managedAiRuntime: ManagedAiRuntime | undefined; + const workspaceManagedConfig = buildExecutionWorkspaceAdapterConfig({ + agentConfig: config, + projectPolicy: projectExecutionWorkspacePolicy, + issueSettings: issueExecutionWorkspaceSettings, + mode: requestedExecutionWorkspaceMode, + legacyUseProjectWorkspace: + issueAssigneeOverrides?.useProjectWorkspace ?? null, + }); + const mergedConfig = { + ...workspaceManagedConfig, + ...Object.fromEntries(Object.entries(issueAssigneeOverrides?.adapterConfig ?? {}).filter(([key]) => requestedAiBinding?.mode !== "router" || !["provider", "acpxAgent", "model", "modelReasoningEffort", "reasoningEffort", "effort", "variant"].includes(key))), + ...(requestedAiBinding?.mode === "router" ? parseObject(parseObject(context.aiRouterSelection).runtimeConfig) : {}), + // The base below is already task-owned. Keep directory transport while + // preserving isolated mode and the mandatory sandbox preflight. + ...(useIsolatedTaskDirectory ? { workspaceStrategy: { type: "project_primary" } } : {}), + }; + const configSnapshot = buildExecutionWorkspaceConfigSnapshot( + mergedConfig, + selectedEnvironmentId, + ); + const executionRunConfig = + stripWorkspaceRuntimeFromExecutionRunConfig(mergedConfig); + const runScopedMentionedSkillKeys = + await resolveRunScopedMentionedSkillKeys({ + db, + companyId: agent.companyId, + issueId, + commentIds: extractWakeCommentIds(context), + }); + const runScopedSkillKeys = + acceptedPlanContinuationWake && + !acceptedPlanWakeRoutingDecision?.suppressAcceptedContinuation + ? [...runScopedMentionedSkillKeys, ACCEPTED_PLAN_CONVERSION_SKILL_KEY] + : runScopedMentionedSkillKeys; + const githubSelection = await resolveManagedGitHubIdentitySelection( + db, + agent.companyId, + { + agentId: agent.id, + responsibleUserId, + allowStandingDelegation: false, + }, + ); + const useHostGitHub = + !githubSelection.configured && + trustPreset.kind === "standard" && + ["local", "ssh"].includes( + selectedEnvironmentForConfig?.driver ?? "local", + ); + + const { resolvedConfig, configuredTaskEnvironment, secretKeys, secretManifest } = + await resolveExecutionRunAdapterConfig({ + managedAiCredentials: Boolean(aiBinding), + managedGitHubCredentials: !useHostGitHub, + companyId: agent.companyId, + agentId: agent.id, + adapterType: agent.adapterType, + issueId, + heartbeatRunId: run.id, + environmentId: selectedEnvironmentForConfig?.id ?? null, + environmentEnv: aiBinding ? stripAiAuthBindings(selectedEnvironmentForConfig?.envVars) : selectedEnvironmentForConfig?.envVars ?? null, + environmentDriver: selectedEnvironmentForConfig?.driver ?? null, + projectId: projectContext?.id ?? null, + routineId: routineEnvContext.routineId, + responsibleUserId, + executionRunConfig: aiBinding ? { ...executionRunConfig, env: stripAiAuthBindings(executionRunConfig.env) } : executionRunConfig, + projectEnv: aiBinding ? stripAiAuthBindings(projectContext?.env) : projectContext?.env ?? null, + routineEnv: aiBinding ? stripAiAuthBindings(routineEnvContext.env) : routineEnvContext.env, + secretsSvc, + trustPreset, + }); + onFailureSecretsResolved(() => collectRunFailureSecretValues(resolvedConfig.env, secretKeys)); + if (aiBinding) { + try { + managedAiRuntime = await prepareManagedAiRuntime(db, { companyId: agent.companyId, agentId: agent.id, responsibleUserId, adapterType: agent.adapterType, binding: aiBinding, config: resolvedConfig }); + onManagedAiRuntimeAcquired(managedAiRuntime); + } catch (error) { + // Only fresh executions can receive a pre-provider wait receipt. A + // persisted native input may already have provider effects to recover. + if (isAiConnectionBusy(error) && !persistedNativeExecutionInput) { + // Use the authority recorded by the locked admission gate, never + // the issue's mutable assignee observed during runtime preparation. + const authorizedNonAssigneeWake = + parseObject(run.runnerProfileJson).aiConnectionNonAssigneeCommentWake === true || + (run.scheduledRetryReason === AI_CONNECTION_BUSY_RETRY_REASON && + isNonAssigneeWorkspaceBusyRetry(run.scheduledRetryReason, parseObject(run.contextSnapshot))); + await finalizeAiConnectionBusyDeferral(run, error, !authorizedNonAssigneeWake); + return { prepared: false as const }; + } + if (responsibleUserId && issueId) { + await connectionIntentService(db).request({ sub: agent.id, company_id: agent.companyId, run_id: run.id, responsible_user_id: responsibleUserId }, aiBinding.provider, { purpose: "ai" }).catch(() => { + logger.warn({ runId: run.id, agentId: agent.id }, "Could not attach AI connection request; runtime configuration action remains available"); + }); + } + throw new ConfigurationIncompleteFailure(error instanceof Error ? error.message : "Configure this agent’s AI connection", { + configurationIncomplete: { reason: "ai_connection_unavailable", companyId: agent.companyId, agentId: agent.id, responsibleUserId, + ...(!persistedNativeExecutionInput && readAiConnectionConfigurationFailure(error) + ? { selectionFailure: readAiConnectionConfigurationFailure(error) } : {}), + ...(readAiCredentialAccessFailure(error) + ? { credentialAccess: readAiCredentialAccessFailure(error) } : {}), + provider: aiBinding.provider, method: aiBinding.method, actionUrl: `/agents/${agent.id}/runtime`, + fingerprint: `ai:${agent.id}:${responsibleUserId}:${JSON.stringify(aiBinding)}` }, + }); + } + const savedAiAccount = parseObject(run.contextSnapshot?.aiConnection); + if (persistedNativeExecutionInput && !managedAiSessionIdentityCompatible( + savedAiAccount.sessionIdentity ?? savedAiAccount.identity, + managedAiRuntime.sessionIdentity, + managedAiRuntime.identity, + )) { + throw new ConfigurationIncompleteFailure("The AI account changed while this native run was suspended. Start a new execution.", { configurationIncomplete: { reason: "ai_connection_changed", actionUrl: `/agents/${agent.id}/runtime` } }); + } + Object.assign(resolvedConfig, managedAiRuntime.config); + for (const key of AI_AUTH_ENV_KEYS) secretKeys.add(key); + context.aiConnection = { ...managedAiRuntime.attribution, accountName: managedAiRuntime.accountName, identity: managedAiRuntime.identity, sessionIdentity: managedAiRuntime.sessionIdentity }; + await db.update(heartbeatRuns).set({ contextSnapshot: sql`coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) || ${JSON.stringify({ aiConnection: context.aiConnection, ...(context.aiRouterTaskKey ? { aiRouterTaskKey: context.aiRouterTaskKey } : {}), ...(context.aiRouterSelection ? { aiRouterSelection: context.aiRouterSelection } : {}) })}::jsonb` }).where(eq(heartbeatRuns.id, run.id)); + } + if (secretManifest.length > 0) { + context.paperclipSecrets = { + manifest: secretManifest, + }; + } else { + delete context.paperclipSecrets; + } + const effectiveResolvedConfig = applyRunScopedMentionedSkillKeys( + resolvedConfig, + runScopedSkillKeys, + ); + const runtimeSkillPreference = readPaperclipSkillSyncPreference( + effectiveResolvedConfig, + ); + const nativeRunnerPreparationSpans: NativeRunHistoricalSpan[] = []; + const skillsPrepareStartedAtMs = Date.now(); + const runtimeSkillEntries = await (async () => { + try { + return await companySkills.listRuntimeSkillEntries(agent.companyId, { + versionSelections: skillVersionSelectionMap( + runtimeSkillPreference.desiredSkillEntries, + { + versionPinsEnabled: + resolvedInstanceSettings.experimental.enableBetaSkills === + true, + }, + ), + }); + } catch (error) { + if (agent.adapterType === "paperclip_runner") { + await recordFailedSkillPreparation({ + runId: run.id, + startedAtMs: skillsPrepareStartedAtMs, + onEvent: async (event) => { + await appendRunEvent(run, event); + }, + }); + } + throw error; + } + })(); + nativeRunnerPreparationSpans.push({ + name: "skills.prepare", + parentName: "task.prepare", + startedAtMs: skillsPrepareStartedAtMs, + endedAtMs: Date.now(), + }); + const connectorAssignments = await resolveConnectorAssignments(db, { companyId: agent.companyId, agentId: agent.id, runId: run.id, issueId: typeof context.issueId === "string" ? context.issueId : undefined }); + const connectorSkillConfig = await applyConnectorSkills(effectiveResolvedConfig, runtimeSkillEntries, connectorAssignments); + // Both CLI adapters and native context materialization use the same resolved set. + runtimeSkillEntries.splice(0, runtimeSkillEntries.length, ...connectorSkillConfig.paperclipRuntimeSkills); + const connectorDelivery = await prepareConnectorSkillDelivery(connectorSkillConfig, agent.adapterType); + // Always replace this runtime-only field; caller wake data cannot supply skills. + context.paperclipWake = { ...parseObject(context.paperclipWake), connectorSkillInstructions: connectorDelivery.instructions }; + let runtimeConfig = await prepareConnectionInstructionDelivery({ + resolve: () => resolveAssignedConnectionInstructionsForRun(db, { companyId: agent.companyId, agentId: agent.id, runId: run.id }), + context, config: connectorDelivery.config, native: agent.adapterType === "paperclip_runner", + }); + const agentIdentity = supportsManagedAgentIdentity(agent.adapterType, agent.adapterConfig, run.runtimeMode === "native" ? run.driverKind : undefined) + ? await agentIdentityService(db).ensureAgentIdentity(agent.companyId, agent.id) + : undefined; + const identityRedactor = createAgentIdentityRedactor(agentIdentity?.privateKeyPem); + onIdentityRedactorPrepared(identityRedactor); + if (agentIdentity) secretKeys.add("PAPERCLIP_AGENT_PRIVATE_KEY"); + return { + prepared: true as const, + mergedConfig, configSnapshot, runScopedMentionedSkillKeys, githubSelection, useHostGitHub, + resolvedConfig, configuredTaskEnvironment, secretKeys, secretManifest, + runtimeSkillEntries, nativeRunnerPreparationSpans, runtimeConfig, agentIdentity, + }; +}