mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
## Thinking Path > - Paperclip manages AI agents and their work. > - Slack connectors turn authenticated messages into company work. > - Direct callbacks need an awake tenant. > - Cloud can retain authenticated asynchronous events until the tenant wakes. > - Replay must preserve the original callback proof and current endpoint authority. > - Configuration changes must retain registration responsibility through failures. > - This PR enrolls verified Slack identities and drains sealed events through the normal SDK owner. ## Linked Issues or Issue Description **Subsystem affected** Server chat connectors and signed Cloud connector delivery. **Problem or motivation** Direct Slack Events API delivery depends on an awake tenant. Broker outages, lost responses and configuration changes must preserve delivery and registration responsibility. Enrollment failures must also leave other connector events able to drain. **Proposed solution** Stage nonsecret enrollment or retirement intent inside the local configuration transaction. Let the existing connection-event owner perform broker requests and retries. Replay sealed events through the current Slack SDK with their original authentication evidence. **Alternatives considered** An early acknowledgment for every callback changes command and modal semantics. A second delivery ledger duplicates existing ownership. Reusing a new stack identity to retire old registrations crosses an authorization boundary. A companion Cloud broker change is required. Related setup fixes: Refs #18837, Refs #18878. Kickoff searches found no competing async enrollment consumer. The ROADMAP's connector setup and access controls stay in place. ## What Changed - Persist enrollment and credential-free retirement intent before local configuration commits. Broker failure does not make a completed pause or removal look unsuccessful. - Bind company, endpoint, connection, public ID, generation and authorized actor. Preserve lost-response responsibility and recover only the supported same-instance configuration outage. - Permit an explicit move to a new authorized stack only after confirmed old retirement. Keep the old identity until the new broker receipt matches the current intent and generation. - Show permanent signing-credential failures as attention. Preserve transient retries and reject uncertain inbound authentication before SDK admission. - Drain events and acknowledgments before enrollment reconciliation. Persist a bounded retry on the exact busy intent so other due endpoints can enroll or retire. Preserve current custody under concurrent changes. - Stage retirement in the Apps removal transaction too. Archive the endpoint and connection, retain the original subject and instance, then revoke credentials. Retirement does not need the deleted Slack credential. - Acknowledge a malformed sealed Slack receipt as invalid once. Keep actual broker outages on the retry path. - Run the real configuration, SDK ingress, generation fences and public Apps removal paths in the default SQL suite. Keep Vitest transport helpers outside production services. - Preserve original callback bytes, signature, time and public URL evidence. Use the existing transactional chat-delivery ledger for redelivery. - Provide a checked-in prewire guard and launcher using the current Node executable. Add broker, worker, SDK, SQL and controlled-process acceptance. - Stabilize AI account summaries by ID and grant ID. This fixes a CI migration-repeat equality failure without changing SQL, authorization or reconnect behavior. - Open the four configured database connections before the enrollment test freezes its clock. Keep all 35 original assertions and all timeouts. - Measure status-card dispatch entry before its mock does database work. Keep the original strict 500 ms bound and check any still-armed earlier deadline. Bound that fixture's pool to four connections. ## Verification - Candidate: `54fa701d41b4ced62f4ff338d39b946015670ae3`. Normal merges preserve master `47027e7` and all owned changes. A read-only merge assessment against fetched master `cb5d7e44` is conflict-free. The only overlapping file has separate AI test additions. The branch has not merged that later master; fresh remote checks must validate the current PR test tree. - At this head, normal server production typechecking and declaration emission passed in 35.24s. The affected actual CI test shard passed in 6.27s. Other unchanged CI layers, 92 affected pure controls and 12 guard controls retain their source-matched results. - All **15 default real-PostgreSQL cases passed in 34.66s**, with no skips or filtering. They cover normal configure and SDK ingress, rollback, stale generations, malformed receipts, busy-lease fairness, concurrent intent custody, unknown errors, and public Apps removal. Removal tests check actual credential revocation, retained retirement through outage and repeat removal, one ordinary retry and confirmed retirement, transaction rollback and foreign-company refusal. - The public removal counterfactual failed before the fix: archival and credential revocation passed, but the inactive intent was absent. The same test source passed after the staging call. Separate counterfactuals also reproduced busy-endpoint starvation and malformed-receipt retries. Original failures remain retained. - The paired broker/SDK/process journey passed in 31.98s at ancestor `0324c011`. It preserves callback proof, lost-ACK redelivery, exactly one comment, one delivery and one successful controlled process. Its test, producer, SDK, guard and runtime bytes are unchanged. The new public Apps archive branch is not called by that journey; its evidence carries on that checked basis. - The new SQL cases need no companion Cloud checkout. Normal CI runs them with the repository fixture. For an existing dedicated PostgreSQL fixture, set `PAPERCLIP_SLACK_OWNER_TEST_DATABASE_URL` and `PAPERCLIP_TEST_POSTGRES_ADMIN_URL`, then run `node server/scripts/test-slack-cloud-owner.mjs` with Node24. The optional paired launcher still requires the normally built companion Cloud checkout. - Every pool was bounded to four or fewer connections. The final 15-case run sampled three runtime clients plus one observer. The independent final audit found zero clients and all five recorded processes absent. Earlier closing-socket observations are kept separate from final cleanup. - Earlier unrelated CI failures are documented: [enrollment timeout](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483754) and [status-card observation bound](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483843). The enrollment test retains all 35 assertions and its original timeouts. A controlled connection probe established the frozen-clock mechanism; the original remote access history remains unknown. The status-card fixture measures dispatch entry with the original strict 500ms bound. A real delayed-dispatch control fails that bound. - Full local workspace typecheck, build and test inventory was not run because its embedded-cluster and wider resource paths exceed this local lease. Exact-head remote CI now passes: 54 successful checks, three skips and Ponytail's neutral check, with aggregate verification green and the legacy security context independently confirmed successful. Both complete bot reviews are 5/5, with all 13 paginated review threads resolved. Seven execution jobs originally reported runner shutdown; one diagnostic rerun recovered them and the dependent aggregate check. Original logs remain retained, and the underlying shutdown cause is unknown. This is recovery evidence, not a product-fix claim. - Provider responses are controlled. This proves local SQL, broker, SDK and process behavior. It does not prove live Slack, agent tools, synchronous callbacks or deployed suspension. ## Risks The companion broker is required. Outage or conflicting authority can leave explicit pending or attention responsibility. A move between stacks requires confirmation from the old authorized owner first. Synchronous commands, actions, modal validation, options and challenges retain their deadlines and endpoint blocker. This slice grants no full Slack endpoint sleep exemption. No deployed suspension or savings claim is made. ## Model Used OpenAI `gpt-6.1-sol` workers through Codex, with reasoning, repository tools and code execution. The served backend revision and context limit are not exposed. A GPT-6-family coordinator performed technical review and integration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>
646 lines
29 KiB
TypeScript
646 lines
29 KiB
TypeScript
import { SlackCloudReceiptError } from "./chat-channels/slack/cloud-event.js";
|
||
import { reconcilePendingSlackCloudEnrollments } from "./chat-channels/slack/cloud-enrollment.js";
|
||
import { notifyExternalObjectRefresh } from "./external-object-refresh-notifications.js";
|
||
import { notifyGitHubCredentialChange } from "./github-connection-notifications.js";
|
||
import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js";
|
||
import { dispatchGitHubBotCloudEvent } from "./chat-github-cloud-ingress.js";
|
||
import { dispatchSlackCloudEvent } from "./chat-channels/slack/cloud-ingress.js";
|
||
import { chatEndpoints } from "@paperclipai/db";
|
||
import { randomUUID } from "node:crypto";
|
||
import { isDeepStrictEqual } from "node:util";
|
||
import {
|
||
connectionEventDeliveries,
|
||
connectionGrants,
|
||
externalObjects,
|
||
toolConnections,
|
||
type Db,
|
||
} from "@paperclipai/db";
|
||
import { and, eq, ne, sql } from "drizzle-orm";
|
||
import {
|
||
logActivity,
|
||
publishActivity,
|
||
type ActivityPublication,
|
||
} from "./activity-log.js";
|
||
import {
|
||
createPaperclipCloudConnector,
|
||
paperclipCloudConnectorConfigFromEnv,
|
||
type PaperclipCloudConnector,
|
||
type SealedConnectorEvents,
|
||
} from "./paperclip-cloud-connector.js";
|
||
import { issueThreadInteractionService } from "./issue-thread-interactions.js";
|
||
import { logger } from "../middleware/logger.js";
|
||
|
||
type LeasedEvent = SealedConnectorEvents["events"][number];
|
||
type GitHubBinding = {
|
||
id: string;
|
||
companyId: string;
|
||
connectionId: string;
|
||
grantId: string;
|
||
subject: string;
|
||
installationId: string;
|
||
providerTenant: NonNullable<typeof connectionGrants.$inferSelect.providerTenant>;
|
||
};
|
||
|
||
export type GitHubConnectionEventPollResult = {
|
||
leased: number;
|
||
processed: number;
|
||
duplicate: number;
|
||
ignored: number;
|
||
failed: number;
|
||
};
|
||
|
||
function record(value: unknown): Record<string, unknown> {
|
||
return value && typeof value === "object" && !Array.isArray(value)
|
||
? value as Record<string, unknown>
|
||
: {};
|
||
}
|
||
|
||
function stringValue(value: unknown): string | null {
|
||
return typeof value === "string" && value.length > 0 ? value : null;
|
||
}
|
||
|
||
function positiveInteger(value: unknown): number | null {
|
||
return typeof value === "number" && Number.isSafeInteger(value) && value > 0 ? value : null;
|
||
}
|
||
|
||
function boundedString(value: unknown, maximum: number): string | null {
|
||
return typeof value === "string" && value.length > 0 && value.length <= maximum ? value : null;
|
||
}
|
||
|
||
function identifier(value: unknown): string | null {
|
||
return typeof value === "string" && /^[1-9][0-9]{0,30}$/.test(value) ? value : null;
|
||
}
|
||
|
||
function isoDate(value: unknown): string | null {
|
||
const candidate = boundedString(value, 100);
|
||
if (!candidate || Number.isNaN(Date.parse(candidate))) return null;
|
||
return new Date(candidate).toISOString();
|
||
}
|
||
|
||
function commitSha(value: unknown): string | null {
|
||
return typeof value === "string" && /^[0-9a-f]{40,64}$/i.test(value) ? value.toLowerCase() : null;
|
||
}
|
||
|
||
function githubUrl(value: unknown): string | null {
|
||
const candidate = boundedString(value, 2_000);
|
||
if (!candidate) return null;
|
||
try {
|
||
const url = new URL(candidate);
|
||
return url.protocol === "https:" && url.hostname.toLowerCase() === "github.com" ? url.toString() : null;
|
||
} catch {
|
||
return null;
|
||
}
|
||
}
|
||
|
||
function compact(values: Record<string, unknown>): Record<string, unknown> {
|
||
return Object.fromEntries(Object.entries(values).filter(([, value]) => value !== null && value !== undefined));
|
||
}
|
||
|
||
/**
|
||
* Treat the sealed Cloud batch as an untrusted boundary. Cloud already normalizes
|
||
* GitHub payloads, but the instance independently allowlists and bounds the small
|
||
* reconciliation record that it persists and processes.
|
||
*/
|
||
function normalizeLeasedPayload(event: LeasedEvent): Record<string, unknown> {
|
||
const payload = record(event.payload);
|
||
const base = {
|
||
event: boundedString(payload.event, 100),
|
||
action: boundedString(payload.action, 100),
|
||
installationId: identifier(payload.installationId),
|
||
repositoryId: identifier(payload.repositoryId),
|
||
repository: boundedString(payload.repository, 300),
|
||
senderId: identifier(payload.senderId),
|
||
senderLogin: boundedString(payload.senderLogin, 100),
|
||
};
|
||
if (event.event === "pull_request") {
|
||
return compact({
|
||
...base,
|
||
number: positiveInteger(payload.number),
|
||
url: githubUrl(payload.url),
|
||
state: boundedString(payload.state, 40),
|
||
merged: payload.merged === true,
|
||
mergedAt: isoDate(payload.mergedAt),
|
||
updatedAt: isoDate(payload.updatedAt),
|
||
headRef: boundedString(payload.headRef, 300),
|
||
headSha: commitSha(payload.headSha),
|
||
baseRef: boundedString(payload.baseRef, 300),
|
||
baseSha: commitSha(payload.baseSha),
|
||
});
|
||
}
|
||
if (event.event === "installation_repositories") {
|
||
const repositoryIds = (value: unknown) => Array.isArray(value)
|
||
? value.slice(0, 1_000).flatMap((item) => identifier(item) ?? [])
|
||
: [];
|
||
return compact({
|
||
...base,
|
||
repositorySelection: boundedString(payload.repositorySelection, 40),
|
||
repositoriesAdded: repositoryIds(payload.repositoriesAdded),
|
||
repositoriesRemoved: repositoryIds(payload.repositoriesRemoved),
|
||
});
|
||
}
|
||
if (event.event === "installation") {
|
||
return compact({
|
||
...base,
|
||
accountId: identifier(payload.accountId),
|
||
accountLogin: boundedString(payload.accountLogin, 100),
|
||
repositorySelection: boundedString(payload.repositorySelection, 40),
|
||
});
|
||
}
|
||
return compact(base);
|
||
}
|
||
|
||
export function githubConnectionEventBindings(rows: Array<{
|
||
grant: typeof connectionGrants.$inferSelect;
|
||
connection: typeof toolConnections.$inferSelect;
|
||
}>): GitHubBinding[] {
|
||
return rows.flatMap(({ grant, connection }) => {
|
||
const config = record(connection.config);
|
||
const oauth = record(config.oauth);
|
||
if (config.sourceTemplateKey !== "github" || oauth.connectorProfile !== "github.code") return [];
|
||
const github = grant.providerTenant?.github;
|
||
if (!github || grant.status !== "active") return [];
|
||
const subject = grant.kind === "agent" && grant.subjectAgentId
|
||
? `agent:${grant.subjectAgentId}`
|
||
: grant.kind === "user" && grant.subjectUserId
|
||
? grant.subjectUserId
|
||
: null;
|
||
if (!subject) return [];
|
||
return github.installationIds.map((installationId) => ({
|
||
id: `${grant.id}_${installationId}`,
|
||
companyId: grant.companyId,
|
||
connectionId: connection.id,
|
||
grantId: grant.id,
|
||
subject,
|
||
installationId,
|
||
providerTenant: grant.providerTenant!,
|
||
}));
|
||
});
|
||
}
|
||
|
||
function githubSnapshotUpdate(payload: Record<string, unknown>) {
|
||
const repository = stringValue(payload.repository);
|
||
const number = positiveInteger(payload.number);
|
||
if (!repository || !number) return null;
|
||
const state = stringValue(payload.state) ?? "unknown";
|
||
const merged = payload.merged === true;
|
||
const [owner, repo, ...extra] = repository.split("/");
|
||
if (!owner || !repo || extra.length > 0) return null;
|
||
return {
|
||
repository,
|
||
owner,
|
||
repo,
|
||
number,
|
||
externalId: `${repository}#pull/${number}`,
|
||
state,
|
||
merged,
|
||
statusKey: merged ? "merged" : state === "closed" ? "closed" : "open",
|
||
statusLabel: merged ? "Merged" : state === "closed" ? "Closed" : "Open",
|
||
statusCategory: merged ? "succeeded" : state === "closed" ? "closed" : "open",
|
||
statusTone: merged ? "success" : state === "closed" ? "muted" : "info",
|
||
statusIconKey: merged ? "git-merge" : state === "closed" ? "x-circle" : "git-pull-request",
|
||
data: {
|
||
provider: "github",
|
||
owner,
|
||
repo,
|
||
number,
|
||
state,
|
||
merged,
|
||
...(stringValue(payload.url) ? { url: stringValue(payload.url) } : {}),
|
||
...(stringValue(payload.mergedAt) ? { mergedAt: stringValue(payload.mergedAt) } : {}),
|
||
...(stringValue(payload.headRef) ? { headRef: stringValue(payload.headRef) } : {}),
|
||
...(stringValue(payload.headSha) ? { headSha: stringValue(payload.headSha) } : {}),
|
||
...(stringValue(payload.baseRef) ? { baseRef: stringValue(payload.baseRef) } : {}),
|
||
...(stringValue(payload.baseSha) ? { baseSha: stringValue(payload.baseSha) } : {}),
|
||
},
|
||
remoteVersion: stringValue(payload.updatedAt),
|
||
} as const;
|
||
}
|
||
|
||
export function githubConnectionEventService(
|
||
db: Db,
|
||
options: {
|
||
connector?: PaperclipCloudConnector;
|
||
env?: NodeJS.ProcessEnv;
|
||
now?: () => Date;
|
||
wakeup?: NonNullable<Parameters<typeof issueThreadInteractionService>[1]>["wakeup"];
|
||
} = {},
|
||
) {
|
||
const now = options.now ?? (() => new Date());
|
||
let nextPollAt = 0;
|
||
let nextEnrollmentAt: number | null = null;
|
||
let nextEnrollmentDiagnosticAt = 0;
|
||
const slackDiagnostics = new Map<string, number>();
|
||
let configuredConnector: {
|
||
config: NonNullable<ReturnType<typeof paperclipCloudConnectorConfigFromEnv>>;
|
||
connector: PaperclipCloudConnector;
|
||
} | undefined;
|
||
function currentConnector() {
|
||
if (options.connector) return options.connector;
|
||
// Enrollment and identity replacement can happen after this owner starts.
|
||
// Read only on an admitted pass; preserve negotiated delivery while unchanged.
|
||
const config = paperclipCloudConnectorConfigFromEnv(options.env);
|
||
if (!config) { configuredConnector = undefined; return null; }
|
||
if (!configuredConnector || !isDeepStrictEqual(configuredConnector.config, config)) {
|
||
configuredConnector = { config, connector: createPaperclipCloudConnector({ config }) };
|
||
}
|
||
return configuredConnector.connector;
|
||
}
|
||
const forcePolling = (options.env ?? process.env).PAPERCLIP_GITHUB_EVENT_POLLING === "1";
|
||
let emptyPolls = 0;
|
||
let nextSetupPollAt = 0;
|
||
let inFlight: Promise<GitHubConnectionEventPollResult> | null = null;
|
||
|
||
async function activeBindings() {
|
||
const rows = await db.select({ grant: connectionGrants, connection: toolConnections })
|
||
.from(connectionGrants)
|
||
.innerJoin(toolConnections, and(
|
||
eq(toolConnections.id, connectionGrants.connectionId),
|
||
eq(toolConnections.companyId, connectionGrants.companyId),
|
||
))
|
||
.where(and(
|
||
sql`${toolConnections.config}->>'sourceTemplateKey' = 'github'`,
|
||
sql`${toolConnections.config}->'oauth'->>'connectorProfile' = 'github.code'`,
|
||
eq(connectionGrants.status, "active"),
|
||
eq(toolConnections.status, "active"),
|
||
eq(toolConnections.enabled, true),
|
||
sql`exists (select 1 from companies where id = ${connectionGrants.companyId} and status <> 'archived')`,
|
||
));
|
||
return githubConnectionEventBindings(rows);
|
||
}
|
||
|
||
async function applyPullRequestEvent(database: Db, companyId: string, event: LeasedEvent, effects: Array<(committedDb: Db) => Promise<void>>) {
|
||
const snapshot = githubSnapshotUpdate(event.payload);
|
||
if (!snapshot) return;
|
||
const appliedAt = now();
|
||
await notifyExternalObjectRefresh(database, companyId, { providerKey: "github", objectType: "pull_request",
|
||
externalId: snapshot.externalId, ignoreExternalIdCase: true });
|
||
await database.update(externalObjects).set({
|
||
statusKey: snapshot.statusKey,
|
||
statusLabel: snapshot.statusLabel,
|
||
statusCategory: snapshot.statusCategory,
|
||
statusTone: snapshot.statusTone,
|
||
statusIconKey: snapshot.statusIconKey,
|
||
isTerminal: snapshot.merged || snapshot.state === "closed",
|
||
data: sql`${externalObjects.data} || ${JSON.stringify(snapshot.data)}::jsonb`,
|
||
remoteVersion: snapshot.remoteVersion,
|
||
lastResolvedAt: appliedAt,
|
||
lastChangedAt: appliedAt,
|
||
nextRefreshAt: appliedAt,
|
||
refreshStartedAt: null,
|
||
refreshToken: null,
|
||
updatedAt: appliedAt,
|
||
}).where(and(
|
||
eq(externalObjects.companyId, companyId),
|
||
eq(externalObjects.providerKey, "github"),
|
||
eq(externalObjects.objectType, "pull_request"),
|
||
sql`lower(${externalObjects.externalId}) = lower(${snapshot.externalId})`,
|
||
));
|
||
if (snapshot.merged && event.action === "closed") {
|
||
await issueThreadInteractionService(database, { wakeup: options.wakeup })
|
||
.sweepMergedPullRequestConfirmations([{
|
||
companyId,
|
||
owner: snapshot.owner,
|
||
repo: snapshot.repo,
|
||
number: snapshot.number,
|
||
}], { companyId }, undefined, { deferConfirmationCommitEffects: effect => effects.push(effect) });
|
||
}
|
||
}
|
||
|
||
async function applyInstallationEvent(database: Db, binding: GitHubBinding, event: LeasedEvent) {
|
||
// Bindings are loaded before the Cloud request. Lock and read the grant
|
||
// again so a refresh completed during that request cannot be overwritten.
|
||
const [currentGrant] = await database.select().from(connectionGrants).where(and(
|
||
eq(connectionGrants.id, binding.grantId),
|
||
eq(connectionGrants.companyId, binding.companyId),
|
||
eq(connectionGrants.status, "active"),
|
||
)).for("update").limit(1);
|
||
const currentProviderTenant = currentGrant?.providerTenant;
|
||
const github = currentProviderTenant?.github;
|
||
if (!github) return;
|
||
// A newly bound instance can receive installation events from before OAuth
|
||
// verified its repository list. Those events must not erase newer access.
|
||
if (Date.parse(github.lastAccessRefreshAt ?? "") > Date.parse(event.createdAt)) {
|
||
await database.update(connectionGrants).set({
|
||
providerTenant: {
|
||
...currentProviderTenant,
|
||
github: { ...github, lastWebhookAt: now().toISOString(), webhookHealth: "healthy" },
|
||
},
|
||
updatedAt: now(),
|
||
}).where(and(eq(connectionGrants.id, binding.grantId), eq(connectionGrants.companyId, binding.companyId)));
|
||
return;
|
||
}
|
||
const unavailable = event.event === "installation" && (event.action === "deleted" || event.action === "suspend");
|
||
const installationIds = unavailable
|
||
? github.installationIds.filter((id) => id !== binding.installationId)
|
||
: [...new Set([...github.installationIds, binding.installationId])];
|
||
const added = Array.isArray(event.payload.repositoriesAdded) ? event.payload.repositoriesAdded.length : 0;
|
||
const removed = Array.isArray(event.payload.repositoriesRemoved) ? event.payload.repositoriesRemoved.length : 0;
|
||
const repositoryCount = Math.max(
|
||
0,
|
||
unavailable && installationIds.length === 0
|
||
? 0
|
||
: unavailable
|
||
? github.repositoryCount
|
||
: github.repositoryCount + added - removed,
|
||
);
|
||
const providerTenant = {
|
||
...currentProviderTenant,
|
||
github: {
|
||
...github,
|
||
// Lifecycle webhooks carry IDs, not the user token’s complete repository view.
|
||
// Discard the snapshot until Refresh access verifies it again.
|
||
accessRevision: randomUUID(),
|
||
...(unavailable || github.bindingSync || !github.installationIds.includes(binding.installationId) ? { bindingSync: {
|
||
id: randomUUID(), removeInstallationIds: [...new Set([
|
||
...(github.bindingSync?.removeInstallationIds ?? []), ...(unavailable ? [binding.installationId] : []),
|
||
])].filter(id => !installationIds.includes(id)),
|
||
} } : {}),
|
||
repositories: undefined,
|
||
installationIds,
|
||
installationCount: installationIds.length,
|
||
repositoryCount,
|
||
repositorySelection: repositoryCount === 0 ? "none" as const : github.repositorySelection,
|
||
lastWebhookAt: now().toISOString(),
|
||
webhookHealth: unavailable ? "unhealthy" as const : "healthy" as const,
|
||
},
|
||
};
|
||
await notifyGitHubCredentialChange(database, { companyId: binding.companyId, config: { sourceTemplateKey: "github" } });
|
||
await database.update(connectionGrants).set({ providerTenant, updatedAt: now() })
|
||
.where(and(eq(connectionGrants.id, binding.grantId), eq(connectionGrants.companyId, binding.companyId)));
|
||
await notifyToolConnectionHealth(database, { companyId: binding.companyId, id: binding.connectionId });
|
||
await database.update(toolConnections).set({
|
||
healthStatus: unavailable ? "failed" : "ok",
|
||
healthMessage: unavailable
|
||
? "GitHub installation access was removed or suspended. Manage repository access on GitHub."
|
||
: "GitHub installation and repository access are available.",
|
||
healthCheckedAt: now(),
|
||
lastHealthAt: now(),
|
||
lastError: unavailable ? "GitHub installation unavailable" : null,
|
||
updatedAt: now(),
|
||
}).where(and(eq(toolConnections.id, binding.connectionId), eq(toolConnections.companyId, binding.companyId)));
|
||
}
|
||
|
||
async function processForCompany(companyId: string, bindings: GitHubBinding[], event: LeasedEvent) {
|
||
const receiptAt = now();
|
||
const normalizedEvent = { ...event, payload: normalizeLeasedPayload(event) };
|
||
const [receipt] = await db.insert(connectionEventDeliveries).values({
|
||
companyId,
|
||
provider: event.provider,
|
||
providerDeliveryId: event.id,
|
||
event: event.event,
|
||
action: event.action,
|
||
installationId: event.installationId,
|
||
repositoryId: event.repositoryId,
|
||
normalizedPayload: normalizedEvent.payload,
|
||
providerCreatedAt: new Date(event.createdAt),
|
||
status: "received",
|
||
attempts: 1,
|
||
updatedAt: receiptAt,
|
||
}).onConflictDoNothing().returning();
|
||
if (!receipt) {
|
||
const [existing] = await db.select().from(connectionEventDeliveries).where(and(
|
||
eq(connectionEventDeliveries.companyId, companyId),
|
||
eq(connectionEventDeliveries.provider, event.provider),
|
||
eq(connectionEventDeliveries.providerDeliveryId, event.id),
|
||
)).limit(1);
|
||
if (existing?.status === "processed") return "duplicate" as const;
|
||
|
||
}
|
||
const postCommitPublications: ActivityPublication[] = [];
|
||
const postCommitEffects: Array<(committedDb: Db) => Promise<void>> = [];
|
||
let applied = true;
|
||
try {
|
||
const applyAndFinalize = async (database: Db) => {
|
||
const [locked] = await database.select().from(connectionEventDeliveries).where(and(
|
||
eq(connectionEventDeliveries.companyId, companyId), eq(connectionEventDeliveries.provider, event.provider),
|
||
eq(connectionEventDeliveries.providerDeliveryId, event.id),
|
||
)).for("update").limit(1);
|
||
if (locked!.status === "processed") { applied = false; return; }
|
||
if (!receipt) await database.update(connectionEventDeliveries).set({
|
||
status: "received", attempts: sql`${connectionEventDeliveries.attempts} + 1`, lastError: null, updatedAt: receiptAt,
|
||
}).where(eq(connectionEventDeliveries.id, locked!.id));
|
||
if (event.event === "pull_request") await applyPullRequestEvent(database, companyId, normalizedEvent, postCommitEffects);
|
||
if (event.event === "installation" || event.event === "installation_repositories") {
|
||
for (const binding of bindings) await applyInstallationEvent(database, binding, normalizedEvent);
|
||
} else {
|
||
const touchedAt = now();
|
||
for (const binding of bindings) {
|
||
const github = binding.providerTenant.github;
|
||
if (!github) continue;
|
||
await database.update(connectionGrants).set({
|
||
// Update only webhook fields; a concurrent access/token refresh
|
||
// owns the remaining metadata and must not be replaced here.
|
||
providerTenant: sql`jsonb_set(${connectionGrants.providerTenant}, '{github}',
|
||
(${connectionGrants.providerTenant}->'github') || ${JSON.stringify({
|
||
lastWebhookAt: touchedAt.toISOString(), webhookHealth: "healthy",
|
||
})}::jsonb)`,
|
||
updatedAt: touchedAt,
|
||
}).where(and(
|
||
eq(connectionGrants.id, binding.grantId),
|
||
eq(connectionGrants.companyId, companyId),
|
||
eq(connectionGrants.status, "active"),
|
||
sql`${connectionGrants.providerTenant}->'github' is not null`,
|
||
));
|
||
}
|
||
}
|
||
const finishedAt = now();
|
||
await database.update(connectionEventDeliveries).set({
|
||
status: "processed",
|
||
processedAt: finishedAt,
|
||
lastError: null,
|
||
updatedAt: finishedAt,
|
||
}).where(and(
|
||
eq(connectionEventDeliveries.companyId, companyId),
|
||
eq(connectionEventDeliveries.provider, event.provider),
|
||
eq(connectionEventDeliveries.providerDeliveryId, event.id),
|
||
));
|
||
await logActivity(database, {
|
||
companyId,
|
||
actorType: "system",
|
||
actorId: "system:github-webhook",
|
||
action: "tool_connection.webhook_processed",
|
||
entityType: "tool_connection",
|
||
entityId: bindings[0]!.connectionId,
|
||
details: {
|
||
provider: "github",
|
||
event: event.event,
|
||
action: event.action,
|
||
deliveryId: event.id,
|
||
installationId: event.installationId,
|
||
repositoryId: event.repositoryId,
|
||
},
|
||
}, postCommitPublications);
|
||
};
|
||
await db.transaction(async (tx) => applyAndFinalize(tx as unknown as Db));
|
||
} catch (error) {
|
||
await db.update(connectionEventDeliveries).set({
|
||
status: "failed",
|
||
lastError: error instanceof Error ? error.message.slice(0, 500) : "GitHub event processing failed",
|
||
updatedAt: now(),
|
||
}).where(and(
|
||
eq(connectionEventDeliveries.companyId, companyId),
|
||
eq(connectionEventDeliveries.provider, event.provider),
|
||
eq(connectionEventDeliveries.providerDeliveryId, event.id),
|
||
ne(connectionEventDeliveries.status, "processed"),
|
||
));
|
||
throw error;
|
||
}
|
||
// Persistence is complete at this point (and installation deltas have
|
||
// committed). A synchronous live-event subscriber must not turn that
|
||
// durable success back into a retryable receipt and replay the delta.
|
||
for (const publication of postCommitPublications) {
|
||
try {
|
||
publishActivity(publication);
|
||
} catch (error) {
|
||
logger.warn({
|
||
err: error,
|
||
companyId,
|
||
providerDeliveryId: event.id,
|
||
}, "GitHub webhook activity publication failed after commit");
|
||
}
|
||
}
|
||
for (const effect of postCommitEffects) await effect(db);
|
||
return applied ? "processed" as const : "duplicate" as const;
|
||
}
|
||
|
||
async function runPoll(): Promise<GitHubConnectionEventPollResult> {
|
||
const connector = currentConnector();
|
||
if (!connector) {
|
||
nextPollAt = Number.POSITIVE_INFINITY;
|
||
return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 };
|
||
}
|
||
try {
|
||
return await drainConnectorEvents(connector);
|
||
} finally {
|
||
try {
|
||
nextEnrollmentAt = await reconcilePendingSlackCloudEnrollments(db, now());
|
||
} catch {
|
||
const time = now().getTime();
|
||
nextEnrollmentAt = time + 5_000;
|
||
if (time >= nextEnrollmentDiagnosticAt) {
|
||
nextEnrollmentDiagnosticAt = time + 300_000;
|
||
logger.warn({ reason: "enrollment_reconcile_unavailable" }, "Slack enrollment responsibility retained");
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
async function drainConnectorEvents(connector: PaperclipCloudConnector): Promise<GitHubConnectionEventPollResult> {
|
||
const bindings = await activeBindings();
|
||
const githubBots = await db.select().from(chatEndpoints).where(and(eq(chatEndpoints.provider, "github"), sql`${chatEndpoints.setup}->'github'->>'cloudRegistrationId' is not null`, sql`${chatEndpoints.status} not in ('archived', 'disabled')`, sql`exists (select 1 from companies where id = ${chatEndpoints.companyId} and status <> 'archived')`));
|
||
const slackBots = await db.select().from(chatEndpoints).where(and(eq(chatEndpoints.provider, "slack"), sql`${chatEndpoints.setup}->'slackCloudEvents' is not null`, sql`${chatEndpoints.status} in ('active','verifying','attention')`, sql`exists (select 1 from companies where id = ${chatEndpoints.companyId} and status <> 'archived')`));
|
||
const bots = [...githubBots, ...slackBots];
|
||
let identity: { subject: string; companyId: string } | null = bindings[0] ?? (bots[0] ? { subject: "github-bot-events", companyId: bots[0].companyId } : null);
|
||
const hasConsumer = bindings.length > 0 || bots.length > 0;
|
||
if (!identity) {
|
||
const [history] = await db.select({ grant: connectionGrants, connection: toolConnections }).from(connectionGrants)
|
||
.innerJoin(toolConnections, and(eq(toolConnections.id, connectionGrants.connectionId), eq(toolConnections.companyId, connectionGrants.companyId)))
|
||
.where(and(sql`${toolConnections.config}->>'sourceTemplateKey' = 'github'`, sql`${toolConnections.config}->'oauth'->>'connectorProfile' = 'github.code'`)).limit(1);
|
||
const grant = history?.grant;
|
||
const subject = grant?.kind === "agent" && grant.subjectAgentId ? `agent:${grant.subjectAgentId}`
|
||
: grant?.kind === "user" ? grant.subjectUserId : null;
|
||
if (grant && subject) identity = { subject, companyId: grant.companyId };
|
||
}
|
||
if (!identity) {
|
||
const [history] = await db.select().from(chatEndpoints).where(sql`(${chatEndpoints.provider} = 'github' AND ${chatEndpoints.setup}->'github'->>'cloudRegistrationId' is not null) OR (${chatEndpoints.provider} = 'slack' AND ${chatEndpoints.setup}->'slackCloudEvents' is not null)`).limit(1);
|
||
if (history) identity = { subject: "github-bot-events", companyId: history.companyId };
|
||
}
|
||
if (!identity) {
|
||
nextPollAt = Number.POSITIVE_INFINITY;
|
||
return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 };
|
||
}
|
||
const first = identity;
|
||
const lease = await connector.leaseEvents({ subject: first.subject, companyId: first.companyId, eventWakeVersion: forcePolling ? 0 : 1 });
|
||
if (!lease) {
|
||
emptyPolls += 1;
|
||
nextPollAt = !hasConsumer || (!forcePolling && connector.eventWakeVersion?.() === 1) ? Number.POSITIVE_INFINITY
|
||
: now().getTime() + Math.min(bots.length ? 15_000 : 5 * 60_000, 5_000 * (2 ** Math.min(emptyPolls, 6)));
|
||
return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 };
|
||
}
|
||
emptyPolls = 0;
|
||
nextPollAt = !hasConsumer || (!forcePolling && connector.eventWakeVersion?.() === 1) ? Number.POSITIVE_INFINITY : now().getTime() + 5_000;
|
||
const result: GitHubConnectionEventPollResult = {
|
||
leased: lease.events.length + (lease.failedEventIds?.length ?? 0),
|
||
processed: 0,
|
||
duplicate: 0,
|
||
ignored: 0,
|
||
failed: lease.failedEventIds?.length ?? 0,
|
||
};
|
||
const acknowledge: string[] = [];
|
||
for (const event of lease.events) {
|
||
if (event.provider === "slack") {
|
||
try { await dispatchSlackCloudEvent(db, event); acknowledge.push(event.id); result.processed += 1; }
|
||
catch (error) {
|
||
const code = error instanceof SlackCloudReceiptError ? error.code : "owner_or_admission_unavailable";
|
||
const disposition = error instanceof SlackCloudReceiptError ? error.disposition : "retry";
|
||
const time = now().getTime();
|
||
if (time - (slackDiagnostics.get(event.id) ?? Number.NEGATIVE_INFINITY) >= 300_000) {
|
||
if (slackDiagnostics.size >= 1000) slackDiagnostics.delete(slackDiagnostics.keys().next().value!);
|
||
slackDiagnostics.set(event.id, time);
|
||
logger.warn({ deliveryId: event.id.slice(0, 200), reason: code, disposition,
|
||
recoveryExpiresAt: Number.isFinite(Date.parse(event.createdAt)) ? new Date(Date.parse(event.createdAt) + 30 * 24 * 60 * 60_000).toISOString() : null }, "Slack connector receipt requires attention");
|
||
}
|
||
if (disposition === "invalid") { acknowledge.push(event.id); result.ignored += 1; }
|
||
else result.failed += 1;
|
||
}
|
||
continue;
|
||
}
|
||
if (event.payload.githubApp !== undefined) {
|
||
try { await dispatchGitHubBotCloudEvent(db, event); acknowledge.push(event.id); result.processed += 1; }
|
||
catch { result.failed += 1; }
|
||
continue;
|
||
}
|
||
const matched = bindings.filter((binding) => event.bindingIds.includes(binding.id));
|
||
if (matched.length === 0) {
|
||
result.ignored += 1;
|
||
acknowledge.push(event.id);
|
||
continue;
|
||
}
|
||
try {
|
||
const companies = new Map<string, GitHubBinding[]>();
|
||
for (const binding of matched) companies.set(binding.companyId, [...(companies.get(binding.companyId) ?? []), binding]);
|
||
for (const [companyId, companyBindings] of companies) {
|
||
const status = await processForCompany(companyId, companyBindings, event);
|
||
result[status] += 1;
|
||
}
|
||
acknowledge.push(event.id);
|
||
|
||
} catch {
|
||
result.failed += 1;
|
||
}
|
||
}
|
||
if (acknowledge.length > 0) {
|
||
await connector.acknowledgeEvents({
|
||
subject: first.subject,
|
||
companyId: first.companyId,
|
||
leaseId: lease.leaseId,
|
||
deliveryIds: acknowledge,
|
||
});
|
||
}
|
||
for (const deliveryId of lease.failedEventIds ?? []) {
|
||
logger.warn({ deliveryId, reason: "inner_event_authentication_failed" },
|
||
"GitHub connector event retained for retry");
|
||
}
|
||
return result;
|
||
}
|
||
|
||
return {
|
||
nextPollAt: () => { const next = Math.min(nextPollAt, nextEnrollmentAt ?? Number.POSITIVE_INFINITY); return Number.isFinite(next) ? next : null; },
|
||
pollOnce(input: { forSetup?: boolean; forDelivery?: boolean } = {}): Promise<GitHubConnectionEventPollResult> {
|
||
// A manager actively awaiting a dedicated App's signed ping must not
|
||
// inherit a personal connection's old empty-queue backoff. This is a
|
||
// bounded readiness probe, not an additional background polling loop.
|
||
if (inFlight) return inFlight;
|
||
const time = now().getTime();
|
||
if (!input.forDelivery && time < (input.forSetup ? nextSetupPollAt : Math.min(nextPollAt, nextEnrollmentAt ?? Number.POSITIVE_INFINITY))) {
|
||
return Promise.resolve({ leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 });
|
||
}
|
||
nextSetupPollAt = time + 5_000;
|
||
nextPollAt = Math.max(nextPollAt, nextSetupPollAt);
|
||
const work = runPoll().finally(() => { if (inFlight === work) inFlight = null; });
|
||
inFlight = work;
|
||
return work;
|
||
},
|
||
};
|
||
}
|