mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
feat(chat): receive sealed Slack events through the current SDK owner (#19019)
## Thinking Path > - Paperclip manages AI agents and their work. > - Slack connectors turn authenticated messages into company work. > - Direct callbacks need an awake tenant. > - Cloud can retain authenticated asynchronous events until the tenant wakes. > - Replay must preserve the original callback proof and current endpoint authority. > - Configuration changes must retain registration responsibility through failures. > - This PR enrolls verified Slack identities and drains sealed events through the normal SDK owner. ## Linked Issues or Issue Description **Subsystem affected** Server chat connectors and signed Cloud connector delivery. **Problem or motivation** Direct Slack Events API delivery depends on an awake tenant. Broker outages, lost responses and configuration changes must preserve delivery and registration responsibility. Enrollment failures must also leave other connector events able to drain. **Proposed solution** Stage nonsecret enrollment or retirement intent inside the local configuration transaction. Let the existing connection-event owner perform broker requests and retries. Replay sealed events through the current Slack SDK with their original authentication evidence. **Alternatives considered** An early acknowledgment for every callback changes command and modal semantics. A second delivery ledger duplicates existing ownership. Reusing a new stack identity to retire old registrations crosses an authorization boundary. A companion Cloud broker change is required. Related setup fixes: Refs #18837, Refs #18878. Kickoff searches found no competing async enrollment consumer. The ROADMAP's connector setup and access controls stay in place. ## What Changed - Persist enrollment and credential-free retirement intent before local configuration commits. Broker failure does not make a completed pause or removal look unsuccessful. - Bind company, endpoint, connection, public ID, generation and authorized actor. Preserve lost-response responsibility and recover only the supported same-instance configuration outage. - Permit an explicit move to a new authorized stack only after confirmed old retirement. Keep the old identity until the new broker receipt matches the current intent and generation. - Show permanent signing-credential failures as attention. Preserve transient retries and reject uncertain inbound authentication before SDK admission. - Drain events and acknowledgments before enrollment reconciliation. Persist a bounded retry on the exact busy intent so other due endpoints can enroll or retire. Preserve current custody under concurrent changes. - Stage retirement in the Apps removal transaction too. Archive the endpoint and connection, retain the original subject and instance, then revoke credentials. Retirement does not need the deleted Slack credential. - Acknowledge a malformed sealed Slack receipt as invalid once. Keep actual broker outages on the retry path. - Run the real configuration, SDK ingress, generation fences and public Apps removal paths in the default SQL suite. Keep Vitest transport helpers outside production services. - Preserve original callback bytes, signature, time and public URL evidence. Use the existing transactional chat-delivery ledger for redelivery. - Provide a checked-in prewire guard and launcher using the current Node executable. Add broker, worker, SDK, SQL and controlled-process acceptance. - Stabilize AI account summaries by ID and grant ID. This fixes a CI migration-repeat equality failure without changing SQL, authorization or reconnect behavior. - Open the four configured database connections before the enrollment test freezes its clock. Keep all 35 original assertions and all timeouts. - Measure status-card dispatch entry before its mock does database work. Keep the original strict 500 ms bound and check any still-armed earlier deadline. Bound that fixture's pool to four connections. ## Verification - Candidate: `54fa701d41b4ced62f4ff338d39b946015670ae3`. Normal merges preserve master `47027e7` and all owned changes. A read-only merge assessment against fetched master `cb5d7e44` is conflict-free. The only overlapping file has separate AI test additions. The branch has not merged that later master; fresh remote checks must validate the current PR test tree. - At this head, normal server production typechecking and declaration emission passed in 35.24s. The affected actual CI test shard passed in 6.27s. Other unchanged CI layers, 92 affected pure controls and 12 guard controls retain their source-matched results. - All **15 default real-PostgreSQL cases passed in 34.66s**, with no skips or filtering. They cover normal configure and SDK ingress, rollback, stale generations, malformed receipts, busy-lease fairness, concurrent intent custody, unknown errors, and public Apps removal. Removal tests check actual credential revocation, retained retirement through outage and repeat removal, one ordinary retry and confirmed retirement, transaction rollback and foreign-company refusal. - The public removal counterfactual failed before the fix: archival and credential revocation passed, but the inactive intent was absent. The same test source passed after the staging call. Separate counterfactuals also reproduced busy-endpoint starvation and malformed-receipt retries. Original failures remain retained. - The paired broker/SDK/process journey passed in 31.98s at ancestor `0324c011`. It preserves callback proof, lost-ACK redelivery, exactly one comment, one delivery and one successful controlled process. Its test, producer, SDK, guard and runtime bytes are unchanged. The new public Apps archive branch is not called by that journey; its evidence carries on that checked basis. - The new SQL cases need no companion Cloud checkout. Normal CI runs them with the repository fixture. For an existing dedicated PostgreSQL fixture, set `PAPERCLIP_SLACK_OWNER_TEST_DATABASE_URL` and `PAPERCLIP_TEST_POSTGRES_ADMIN_URL`, then run `node server/scripts/test-slack-cloud-owner.mjs` with Node24. The optional paired launcher still requires the normally built companion Cloud checkout. - Every pool was bounded to four or fewer connections. The final 15-case run sampled three runtime clients plus one observer. The independent final audit found zero clients and all five recorded processes absent. Earlier closing-socket observations are kept separate from final cleanup. - Earlier unrelated CI failures are documented: [enrollment timeout](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483754) and [status-card observation bound](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483843). The enrollment test retains all 35 assertions and its original timeouts. A controlled connection probe established the frozen-clock mechanism; the original remote access history remains unknown. The status-card fixture measures dispatch entry with the original strict 500ms bound. A real delayed-dispatch control fails that bound. - Full local workspace typecheck, build and test inventory was not run because its embedded-cluster and wider resource paths exceed this local lease. Exact-head remote CI now passes: 54 successful checks, three skips and Ponytail's neutral check, with aggregate verification green and the legacy security context independently confirmed successful. Both complete bot reviews are 5/5, with all 13 paginated review threads resolved. Seven execution jobs originally reported runner shutdown; one diagnostic rerun recovered them and the dependent aggregate check. Original logs remain retained, and the underlying shutdown cause is unknown. This is recovery evidence, not a product-fix claim. - Provider responses are controlled. This proves local SQL, broker, SDK and process behavior. It does not prove live Slack, agent tools, synchronous callbacks or deployed suspension. ## Risks The companion broker is required. Outage or conflicting authority can leave explicit pending or attention responsibility. A move between stacks requires confirmation from the old authorized owner first. Synchronous commands, actions, modal validation, options and challenges retain their deadlines and endpoint blocker. This slice grants no full Slack endpoint sleep exemption. No deployed suspension or savings claim is made. ## Model Used OpenAI `gpt-6.1-sol` workers through Codex, with reasoning, repository tools and code execution. The served backend revision and context limit are not exposed. A GPT-6-family coordinator performed technical review and integration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
32 files changed
+2714
-29
No files matched your search
@@ -17,7 +17,7 @@ import { mkdtemp, realpath, rm, access, readFile, writeFile, stat } from "node:f
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { createDb, companies, agents, agentTaskSessions, heartbeatRuns, companyMemberships, connectionGrants, toolApplications, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, plugins, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests, companySecrets, companySecretVersions, userSecretDefinitions, principalPermissionGrants } from "@paperclipai/db";
|
||||
import { createDb, type Db, companies, agents, agentTaskSessions, heartbeatRuns, companyMemberships, connectionGrants, toolApplications, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, plugins, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests, companySecrets, companySecretVersions, userSecretDefinitions, principalPermissionGrants } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres";
|
||||
import { fetchCompanyQuotaWindows } from "../services/quota-windows.js";
|
||||
import { quotaCredentialRecoveryPath, quotaCredentialRecovery, quotaCredentialHash } from "../services/quota-credential-recovery.js";
|
||||
@@ -2213,6 +2213,33 @@ console.log(JSON.stringify({ type: "result", subtype: "success", result: "hello"
|
||||
const [agent] = await db.select().from(agents).where(eq(agents.id, agentId));
|
||||
expect(agent.runtimeConfig.aiConnection).toBeUndefined();
|
||||
});
|
||||
it("keeps account summaries ordered across sequential scans and repeat migration", async () => {
|
||||
const orderedCompanyId = randomUUID(), userId = "stable-list-owner";
|
||||
await db.insert(companies).values({ id: orderedCompanyId, name: "Stable account list", issuePrefix: `O${orderedCompanyId.slice(0, 5)}` });
|
||||
await db.insert(companyMemberships).values({ companyId: orderedCompanyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });
|
||||
const accounts = [];
|
||||
for (const name of ["First account", "Second account", "Third account"]) {
|
||||
accounts.push(await service.save(orderedCompanyId, userId, { provider: "anthropic", method: "api_key", ownership: "personal", name, apiKey: "fixture-stable-list", agentIds: [], allAgents: false }, "fixture-stable-list"));
|
||||
}
|
||||
const expected = accounts.map(({ connectionId: id, grantId }) => ({ id, grantId }))
|
||||
.sort((a, b) => a.id.localeCompare(b.id) || a.grantId.localeCompare(b.grantId));
|
||||
await db.transaction(async tx => {
|
||||
await tx.execute(sql`SET LOCAL enable_indexscan = off`);
|
||||
await tx.execute(sql`SET LOCAL enable_bitmapscan = off`);
|
||||
await tx.execute(sql`SET LOCAL enable_hashjoin = off`);
|
||||
await tx.execute(sql`SET LOCAL enable_mergejoin = off`);
|
||||
for (const account of [...expected].reverse()) {
|
||||
await tx.update(toolConnections).set({ name: sql`${toolConnections.name}` }).where(eq(toolConnections.id, account.id));
|
||||
await tx.update(connectionGrants).set({ status: sql`${connectionGrants.status}` }).where(eq(connectionGrants.id, account.grantId));
|
||||
}
|
||||
const transactionalService = aiConnectionService(tx as unknown as Db);
|
||||
const before = await transactionalService.list(orderedCompanyId, userId);
|
||||
expect(before.map(({ id, grantId }) => ({ id, grantId }))).toEqual(expected);
|
||||
const migration = await readFile(new URL("../../../packages/db/src/migrations/0276_hard_mandroid.sql", import.meta.url), "utf8");
|
||||
for (const statement of migration.split("--> statement-breakpoint").filter(value => value.trim())) await tx.execute(sql.raw(statement));
|
||||
expect(await transactionalService.list(orderedCompanyId, userId)).toEqual(before);
|
||||
});
|
||||
});
|
||||
it("runs two Claude subscription executions for the same grant at the same time", async () => {
|
||||
// Claude writes no auth file back to the grant, so two runs share no
|
||||
// mutable state and must not wait for each other.
|
||||
|
||||
@@ -90,6 +90,13 @@ const support = await getEmbeddedPostgresTestSupport();
|
||||
grantId = grant!.id;
|
||||
const env = { PAPERCLIP_CLOUD_CONNECTOR_BASE_URL: origin, PAPERCLIP_CLOUD_CONNECTOR_ENVIRONMENT: "test" };
|
||||
const service = githubConnectionEventService(db, { env });
|
||||
const reserved: Awaited<ReturnType<typeof db.$client.reserve>>[] = [];
|
||||
try {
|
||||
for (let slot = 0; slot < 4; slot++) reserved.push(await db.$client.reserve());
|
||||
await Promise.all(reserved.map(connection => connection`SELECT 1`));
|
||||
} finally {
|
||||
for (const connection of reserved) connection.release();
|
||||
}
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: e => errors.push(e) });
|
||||
const selects = vi.spyOn(db, "select"), transactions = vi.spyOn(db, "transaction"), executes = vi.spyOn(db, "execute");
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
import { createCipheriv, createHash, createHmac, diffieHellman, generateKeyPairSync, hkdfSync, randomBytes, verify } from "node:crypto";
|
||||
import { expect, vi } from "vitest";
|
||||
import { createPaperclipCloudConnector, type PaperclipCloudConnectorConfig } from "../../services/paperclip-cloud-connector.js";
|
||||
|
||||
export function createSlackCloudConnectorFixture(instanceId: string, companyId: string, subject: string) {
|
||||
const signing = generateKeyPairSync("ed25519"), sealing = generateKeyPairSync("x25519");
|
||||
const config: PaperclipCloudConnectorConfig = { baseUrl: "http://127.0.0.1:1", instanceId, environment: "test",
|
||||
signPrivateKey: signing.privateKey.export({ format: "jwk" }).d!, sealPrivateKey: sealing.privateKey.export({ format: "jwk" }).d! };
|
||||
const calls: { operation: string; enrollment?: Record<string, unknown> }[] = [];
|
||||
const acknowledged: string[] = [];
|
||||
const events: Record<string, unknown>[] = [];
|
||||
let unavailable = false;
|
||||
let beforeEnrollmentReceipt: (() => Promise<void>) | undefined;
|
||||
function seal(payload: unknown, profile: "github.code" | "slack.events.v1") {
|
||||
const provider = profile === "github.code" ? "github" : "slack";
|
||||
const ephemeral = generateKeyPairSync("x25519");
|
||||
const epk = ephemeral.publicKey.export({ format: "jwk" }).x!;
|
||||
const recipient = sealing.publicKey.export({ format: "jwk" }).x!;
|
||||
const aad = Buffer.from([1, "X25519-HKDF-SHA256-A256GCM", "events", instanceId, "test", provider, profile, ""].join("\n"));
|
||||
const key = Buffer.from(hkdfSync("sha256", diffieHellman({ privateKey: ephemeral.privateKey, publicKey: sealing.publicKey }),
|
||||
Buffer.concat([Buffer.from(epk, "base64url"), Buffer.from(recipient, "base64url")]), aad, 32));
|
||||
const iv = randomBytes(12), cipher = createCipheriv("aes-256-gcm", key, iv); cipher.setAAD(aad);
|
||||
const ct = Buffer.concat([cipher.update(JSON.stringify(payload), "utf8"), cipher.final(), cipher.getAuthTag()]);
|
||||
return { v: 1, alg: "X25519-HKDF-SHA256-A256GCM", purpose: "events", provider, profile, epk,
|
||||
iv: iv.toString("base64url"), ct: ct.toString("base64url") };
|
||||
}
|
||||
const requestErrors: unknown[] = [];
|
||||
const request: typeof fetch = vi.fn(async (input: RequestInfo | URL, init?: RequestInit): Promise<Response> => {
|
||||
try {
|
||||
const url = new URL(input instanceof Request ? input.url : String(input));
|
||||
expect(url.origin).toBe(config.baseUrl);
|
||||
expect(init?.method).toBe("POST");
|
||||
const body = JSON.parse(String(init?.body)) as { request: string; enrollment?: string; acknowledgement?: string };
|
||||
const [header, payload, signature] = body.request.split(".");
|
||||
expect(verify(null, Buffer.from(`${header}.${payload}`), signing.publicKey, Buffer.from(signature!, "base64url"))).toBe(true);
|
||||
const claims = JSON.parse(Buffer.from(payload!, "base64url").toString("utf8")) as Record<string, unknown>;
|
||||
expect(claims).toMatchObject({ iss: instanceId, env: "test", aud: url.toString() });
|
||||
const operation = String(claims.op);
|
||||
if (operation !== "status") {
|
||||
const expectedSubject = operation === "slack-events" ? subject
|
||||
: ["event-lease", "event-ack"].includes(operation) ? "github-bot-events" : undefined;
|
||||
if (!expectedSubject) throw new Error(`Unexpected synthetic connector authority ${operation}`);
|
||||
expect(claims).toMatchObject({ sub: expectedSubject, cid: companyId });
|
||||
}
|
||||
if (unavailable) return Response.json({ error: "fixture_broker_unavailable" }, { status: 503 });
|
||||
if (operation === "status") return Response.json({ active: true, status: "active", profiles: [], slackEvents: { version: 1 } });
|
||||
if (operation === "slack-events") {
|
||||
expect(claims.sh).toBe(createHash("sha256").update(body.enrollment!).digest("base64url"));
|
||||
const enrollment = JSON.parse(body.enrollment!) as Record<string, unknown>;
|
||||
calls.push({ operation, enrollment: { ...enrollment, signingSecret: undefined } });
|
||||
await beforeEnrollmentReceipt?.();
|
||||
const { active, signingSecret: _secret, ...identity } = enrollment;
|
||||
const bindingId = `slack-event:${createHash("sha256").update(JSON.stringify(identity)).digest("hex")}`;
|
||||
return Response.json(active === false ? { retirement: { version: 1, instanceId, ...identity } }
|
||||
: { enrollment: { version: 1, instanceId, ...identity, bindingId } });
|
||||
}
|
||||
calls.push({ operation });
|
||||
if (operation === "event-lease") {
|
||||
if (!events.length) return Response.json({ events: [], eventWakeVersion: 1 });
|
||||
const leased = events.splice(0);
|
||||
return Response.json({ leaseId: "fixture-lease", eventWakeVersion: 1,
|
||||
sealed: seal({ v: 1, instanceId, environment: "test", leaseId: "fixture-lease", events: leased }, "github.code") });
|
||||
}
|
||||
if (operation === "event-ack") {
|
||||
expect(claims.sh).toBe(createHash("sha256").update(body.acknowledgement!).digest("base64url"));
|
||||
const value = JSON.parse(body.acknowledgement!) as { deliveryIds: string[] };
|
||||
acknowledged.push(...value.deliveryIds); return Response.json({ acknowledged: value.deliveryIds.length });
|
||||
}
|
||||
throw new Error(`Unexpected synthetic connector operation ${operation}`);
|
||||
} catch (error) {
|
||||
requestErrors.push(error);
|
||||
console.error("Synthetic Slack connector rejected its request", error);
|
||||
throw error;
|
||||
}
|
||||
});
|
||||
function install() {
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", config.baseUrl);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", instanceId);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_ENVIRONMENT", "test");
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SIGN_PRIVATE_KEY", config.signPrivateKey);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SEAL_PRIVATE_KEY", config.sealPrivateKey);
|
||||
}
|
||||
function enqueue(identity: Record<string, unknown>, signingSecret: string, eventId: string) {
|
||||
const acceptedAt = new Date().toISOString(), timestamp = String(Math.floor(Date.now() / 1000));
|
||||
const body = Buffer.from(JSON.stringify({ type: "event_callback", team_id: identity.teamId, api_app_id: identity.appId, event_id: eventId,
|
||||
event: { type: "app_mention", text: "<@UBOT> fixture callback", user: "UOPERATOR", channel: "CTEST", ts: `${Date.now() / 1000}`, event_ts: `${Date.now() / 1000}` } }));
|
||||
const id = `slack_${String(identity.bindingId).slice("slack-event:".length)}_${createHash("sha256").update(eventId).digest("hex")}`;
|
||||
const packet = { version: 1, instanceId, environment: "test", ...identity, eventId, acceptedAt, rawBody: body.toString("base64"), timestamp,
|
||||
signature: `v0=${createHmac("sha256", signingSecret).update(`v0:${timestamp}:`).update(body).digest("hex")}` };
|
||||
events.push({ id, provider: "slack", event: "event_callback", installationId: identity.bindingId, bindingIds: [identity.bindingId], createdAt: acceptedAt,
|
||||
payload: { slackEvent: { bindingId: identity.bindingId, sealed: seal(packet, "slack.events.v1") } } });
|
||||
return { id, packet, body };
|
||||
}
|
||||
return { config, request, requestErrors, calls, acknowledged, events, install, enqueue, connector: createPaperclipCloudConnector({ config, request }),
|
||||
setUnavailable(value: boolean) { unavailable = value; },
|
||||
beforeReceipt(work: (() => Promise<void>) | undefined) { beforeEnrollmentReceipt = work; } };
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
const key = Symbol.for('paperclip.slack.fixture.prewire');
|
||||
if (!process[key]) {
|
||||
const net = require('node:net');
|
||||
const tls = require('node:tls');
|
||||
const http = require('node:http');
|
||||
const https = require('node:https');
|
||||
const fs = require('node:fs');
|
||||
const loopback = new Set(['127.0.0.1', 'localhost', '::1', '[::1]']);
|
||||
const listeners = new Set();
|
||||
const pgTargets = new Set();
|
||||
const pgPorts = new Set([5432, 55448]);
|
||||
for (const name of ['PAPERCLIP_SLACK_TEST_DATABASE_URL', 'SLACK_EVENT_TEST_DATABASE_URL']) {
|
||||
if (!process.env[name]) continue;
|
||||
let url;
|
||||
try { url = new URL(process.env[name]); } catch { throw new Error('Invalid Slack fixture database target'); }
|
||||
if (!['postgres:', 'postgresql:'].includes(url.protocol) || !loopback.has(url.hostname)
|
||||
|| !/^[a-zA-Z0-9_]+_test$/.test(decodeURIComponent(url.pathname.slice(1)))
|
||||
|| ['options', 'database', 'dbname'].some(k => url.searchParams.has(k))) throw new Error('Invalid Slack fixture database target');
|
||||
const port = Number(url.port || 5432);
|
||||
pgPorts.add(port);
|
||||
pgTargets.add(`${url.hostname}:${port}`);
|
||||
}
|
||||
const sockets = new Set();
|
||||
const observed = new WeakSet();
|
||||
const attempts = [];
|
||||
let peakPgSockets = 0;
|
||||
const record = value => {
|
||||
if (process.env.SLACK_GATE_PROCESS_LOG) fs.appendFileSync(process.env.SLACK_GATE_PROCESS_LOG, JSON.stringify(value) + '\n');
|
||||
};
|
||||
const fail = (host, port) => {
|
||||
attempts.push({ host: String(host), port: Number(port) || null });
|
||||
process.exitCode = 98;
|
||||
throw new Error('Slack fixture blocked an unowned network target before transport');
|
||||
};
|
||||
const check = (host = 'localhost', port, protocol) => {
|
||||
host = String(host).toLowerCase();
|
||||
if (!loopback.has(host)) fail(host, port);
|
||||
if (protocol && !['http:', 'https:'].includes(protocol)) fail(host, port);
|
||||
if (port && pgPorts.has(Number(port)) && !pgTargets.has(`${host}:${Number(port)}`)) fail(host, port);
|
||||
if (port && !pgTargets.has(`${host}:${Number(port)}`) && !listeners.has(Number(port))) fail(host, port);
|
||||
};
|
||||
const checkArgs = args => {
|
||||
const first = args[0];
|
||||
if (first instanceof URL || typeof first === 'string' && /^https?:/.test(first)) {
|
||||
const url = first instanceof URL ? first : new URL(first);
|
||||
check(url.hostname, url.port || (url.protocol === 'https:' ? 443 : 80), url.protocol);
|
||||
} else if (first && typeof first === 'object') {
|
||||
if (first.path && !first.port && !first.host && !first.hostname) fail('unix-socket', 0);
|
||||
check(first.hostname || first.host, first.port, first.protocol);
|
||||
} else if (typeof first === 'number') check(typeof args[1] === 'string' ? args[1] : undefined, first);
|
||||
else fail('unknown-target', 0);
|
||||
};
|
||||
const listen = net.Server.prototype.listen;
|
||||
net.Server.prototype.listen = function (...args) {
|
||||
this.once('listening', () => {
|
||||
const address = this.address();
|
||||
if (address && typeof address === 'object' && loopback.has(address.address)) {
|
||||
listeners.add(address.port);
|
||||
this.once('close', () => listeners.delete(address.port));
|
||||
}
|
||||
});
|
||||
return Reflect.apply(listen, this, args);
|
||||
};
|
||||
for (const module of [http, https]) for (const name of ['request', 'get']) {
|
||||
const original = module[name];
|
||||
module[name] = function (...args) { checkArgs(args); return Reflect.apply(original, this, args); };
|
||||
}
|
||||
const connect = net.Socket.prototype.connect;
|
||||
net.Socket.prototype.connect = function (...args) {
|
||||
const normalized = Array.isArray(args[0]) ? args[0] : args;
|
||||
checkArgs(normalized);
|
||||
const first = normalized[0];
|
||||
const port = typeof first === 'object' ? first.port : first;
|
||||
if (pgPorts.has(Number(port)) && !observed.has(this)) {
|
||||
observed.add(this);
|
||||
this.once('connect', () => { sockets.add(this); peakPgSockets = Math.max(peakPgSockets, sockets.size); });
|
||||
this.once('close', () => sockets.delete(this));
|
||||
}
|
||||
return Reflect.apply(connect, this, args);
|
||||
};
|
||||
const tlsConnect = tls.connect;
|
||||
tls.connect = function (...args) { checkArgs(args); return Reflect.apply(tlsConnect, this, args); };
|
||||
const fetch = globalThis.fetch;
|
||||
if (fetch) globalThis.fetch = function (input, init) {
|
||||
checkArgs([input instanceof Request ? input.url : input]);
|
||||
return fetch(input, init);
|
||||
};
|
||||
const stats = () => ({ pid: process.pid, peakPgSockets, remainingPgSockets: sockets.size, unexpectedAttempts: attempts.slice() });
|
||||
process[key] = Object.freeze({ pid: process.pid, stats });
|
||||
globalThis[key] = process[key];
|
||||
process.env.SLACK_GATE_GUARD_PID = String(process.pid);
|
||||
record({ phase: 'repository-preload', pid: process.pid });
|
||||
process.on('exit', () => record({ phase: 'exit', ...stats(), exitCode: process.exitCode || 0 }));
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
import http from "node:http";
|
||||
import https from "node:https";
|
||||
import { expect, vi } from "vitest";
|
||||
|
||||
export function guardSlackFixtureNetwork(options: { requirePreload?: boolean } = {}) {
|
||||
const prewire = Reflect.get(process, Symbol.for("paperclip.slack.fixture.prewire")) as {
|
||||
pid: number;
|
||||
stats(): { pid: number; peakPgSockets: number; remainingPgSockets: number; unexpectedAttempts: unknown[] };
|
||||
} | undefined;
|
||||
if (options.requirePreload) {
|
||||
expect(prewire?.pid).toBe(process.pid);
|
||||
expect(process.env.SLACK_GATE_GUARD_PID).toBe(String(process.pid));
|
||||
}
|
||||
if (process.env.SLACK_GATE_EXPECT_PRELOAD === "1") {
|
||||
expect(process.env.SLACK_GATE_GUARD_PID).toBe(String(process.pid));
|
||||
console.info(JSON.stringify({ slackFixtureWorkerPid: process.pid, prewireGuardPid: process.env.SLACK_GATE_GUARD_PID }));
|
||||
}
|
||||
const blocked: string[] = [];
|
||||
const check = (value: unknown) => {
|
||||
const url = value instanceof URL ? value : typeof value === "string" ? new URL(value) : undefined;
|
||||
const options = value as { hostname?: string; host?: string } | undefined;
|
||||
const host = url?.hostname ?? options?.hostname ?? options?.host ?? "localhost";
|
||||
if (!["localhost", "127.0.0.1", "::1", "[::1]"].includes(host)) {
|
||||
blocked.push(host);
|
||||
throw new Error("Unexpected off-host Slack fixture request");
|
||||
}
|
||||
};
|
||||
const realHttp = http.request, realHttps = https.request, realFetch = globalThis.fetch;
|
||||
const httpSpy = vi.spyOn(http, "request").mockImplementation(((...args: unknown[]) => {
|
||||
check(args[0]); return Reflect.apply(realHttp, http, args);
|
||||
}) as typeof http.request);
|
||||
const httpsSpy = vi.spyOn(https, "request").mockImplementation(((...args: unknown[]) => {
|
||||
check(args[0]); return Reflect.apply(realHttps, https, args);
|
||||
}) as typeof https.request);
|
||||
const fetchSpy = vi.spyOn(globalThis, "fetch").mockImplementation((input, init) => {
|
||||
check(input instanceof Request ? input.url : input); return realFetch(input, init);
|
||||
});
|
||||
return { assertQuiet: () => {
|
||||
expect(blocked).toEqual([]);
|
||||
if (prewire) expect(prewire.stats().unexpectedAttempts).toEqual([]);
|
||||
const stats = prewire?.stats;
|
||||
if (process.env.SLACK_GATE_EXPECT_PRELOAD === "1") {
|
||||
expect(stats).toBeTypeOf("function");
|
||||
console.info(JSON.stringify({ slackFixtureWorkerPid: process.pid, networkStats: stats!() }));
|
||||
}
|
||||
}, restore: () => {
|
||||
fetchSpy.mockRestore(); httpsSpy.mockRestore(); httpSpy.mockRestore();
|
||||
} };
|
||||
}
|
||||
|
||||
export function stubSlackFixtureTransport(adapter: unknown) {
|
||||
const client = (adapter as { _client: {
|
||||
axios: { defaults: { adapter: unknown } };
|
||||
buildResult(response: unknown): Promise<unknown>;
|
||||
} })._client;
|
||||
const failures: unknown[] = [];
|
||||
const buildResult = client.buildResult.bind(client);
|
||||
client.buildResult = async response => {
|
||||
try { return await buildResult(response); }
|
||||
catch (error) { failures.push(error); throw error; }
|
||||
};
|
||||
const transport = vi.fn(async (config: { url: string; baseURL: string }) => {
|
||||
const url = new URL(config.url, config.baseURL);
|
||||
if (url.hostname !== "127.0.0.1") throw new Error("Slack fixture API origin changed");
|
||||
let data: unknown;
|
||||
if (url.pathname.endsWith("/users.info")) data = { ok: true, user: { id: "UOPERATOR", name: "operator", is_bot: false, profile: { display_name: "Operator" } } };
|
||||
else if (url.pathname.endsWith("/conversations.info")) data = { ok: true, channel: { id: "CTEST", name: "fixture" } };
|
||||
else throw new Error(`Unexpected Slack fixture API method ${url.pathname}`);
|
||||
return { data, status: 200, statusText: "OK", headers: {}, config, request: { path: url.pathname } };
|
||||
});
|
||||
client.axios.defaults.adapter = transport;
|
||||
return Object.assign(transport, { assertNoErrors: async () => {
|
||||
await Promise.all(transport.mock.results.map(result => result.value));
|
||||
expect(failures).toEqual([]);
|
||||
} });
|
||||
}
|
||||
@@ -0,0 +1,433 @@
|
||||
import { createHmac, randomUUID } from "node:crypto";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { SLACK_CHAT_BOT_SCOPES } from "@paperclipai/shared";
|
||||
import { activityLog, agentRuntimeState, agentTaskSessions, agentWakeupRequests, agents, applyPendingMigrations, authUsers,
|
||||
chatDeliveries, chatEndpoints, chatExternalPrincipals, companySecretBindings, chatIdentityLinks, chatSlackRegistrations, companies, companyMemberships,
|
||||
companySecrets, companySkills, costEvents, financeEvents, heartbeatRunEvents, heartbeatRuns, issueComments, issues, toolApplications, toolConnections,
|
||||
closeCheckpointDb, closeFileStorageDb, closeLivenessDb, createDb } from "@paperclipai/db";
|
||||
import { databaseWorkPending, reconcileDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db/work-signals";
|
||||
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { createSlackCloudConnectorFixture } from "./helpers/slack-cloud-connector-fixture.js";
|
||||
import { chatChannelService } from "../services/chat-channels.js";
|
||||
import { toolAccessService } from "../services/tool-access.js";
|
||||
import { secretService } from "../services/secrets.js";
|
||||
import { ChatRuntime } from "../services/chat-runtime.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { githubConnectionEventService } from "../services/github-connection-events.js";
|
||||
import { chatCredentialMutationLease } from "../services/chat-credential-mutation-lease.js";
|
||||
import { createSlackCloudEnrollment, registerSlackCloudEnrollmentOwner, reconcilePendingSlackCloudEnrollments } from "../services/chat-channels/slack/cloud-enrollment.js";
|
||||
import { conflict } from "../errors.js";
|
||||
import { guardSlackFixtureNetwork, stubSlackFixtureTransport } from "./helpers/slack-cloud/transport.js";
|
||||
import type { InternalSetupState } from "../services/chat-channels/runtime-types.js";
|
||||
import type { ChatSdkMessageCallbackEvent } from "../services/chat-sdk-runtime.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_SLACK_OWNER_TEST_DATABASE_URL");
|
||||
if (process.env.SLACK_GATE_EXPECT_PRELOAD === "1" && !external) throw new Error("Guarded local Slack owner tests require their exact external test database");
|
||||
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)("Slack Cloud default owner and SQL fences", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let service: ReturnType<typeof chatChannelService>;
|
||||
let heartbeat: ReturnType<typeof heartbeatService>;
|
||||
let cloud: ReturnType<typeof createSlackCloudConnectorFixture>;
|
||||
let network: ReturnType<typeof guardSlackFixtureNetwork>;
|
||||
let directory: string;
|
||||
let companyId: string, actorId: string, agentId: string, applicationId: string;
|
||||
let runtime: ChatRuntime;
|
||||
let providerFetch: typeof fetch;
|
||||
const callbacks: ChatSdkMessageCallbackEvent[] = [];
|
||||
const transports: ReturnType<typeof stubSlackFixtureTransport>[] = [];
|
||||
const releases: (() => void)[] = [];
|
||||
const heldLeases: Promise<unknown>[] = [];
|
||||
const deferred: (() => void | Promise<void>)[] = [];
|
||||
const secret = "0123456789abcdef0123456789abcdef";
|
||||
const row = async (id: string) => (await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, id)))[0]!;
|
||||
const setup = async (id: string) => (await row(id)).setup as InternalSetupState;
|
||||
beforeAll(async () => {
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("slack-cloud-owner-");
|
||||
const url = external ?? temporary!.connectionString;
|
||||
await applyPendingMigrations(url);
|
||||
db = createDb(url, { maxConnections: 2, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
|
||||
}, 60_000);
|
||||
beforeEach(async () => {
|
||||
callbacks.length = 0; transports.length = 0; releases.length = 0; heldLeases.length = 0; deferred.length = 0;
|
||||
network = guardSlackFixtureNetwork({ requirePreload: process.env.SLACK_GATE_EXPECT_PRELOAD === "1" });
|
||||
directory = await mkdtemp(path.join(os.tmpdir(), "slack-owner-default-"));
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", path.join(directory, "master.key"));
|
||||
companyId = randomUUID(); actorId = `slack-owner-${randomUUID()}`; agentId = randomUUID(); applicationId = randomUUID();
|
||||
cloud = createSlackCloudConnectorFixture(`instance-${randomUUID()}`, companyId, actorId); cloud.install();
|
||||
vi.spyOn(globalThis, "fetch").mockImplementation(cloud.request);
|
||||
const now = new Date();
|
||||
await db.insert(companies).values({ id: companyId, name: "Slack default owner", issuePrefix: `S${companyId.slice(0, 7)}` });
|
||||
await db.insert(authUsers).values({ id: actorId, name: "Slack owner", email: `${actorId}@example.test`, emailVerified: true, createdAt: now, updatedAt: now });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: actorId, status: "active", membershipRole: "operator" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Slack fixture", adapterType: "process", status: "idle",
|
||||
runtimeConfig: { heartbeat: { wakeOnDemand: false, intervalSec: 0 } } });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "agent", principalId: agentId, status: "active", membershipRole: "member" });
|
||||
await db.insert(toolApplications).values({ id: applicationId, companyId, name: "Slack", type: "chat" });
|
||||
runtime = makeRuntime();
|
||||
const slackFetch: typeof fetch = async (input, init) => {
|
||||
const url = new URL(input instanceof Request ? input.url : String(input));
|
||||
if (url.origin !== "https://slack.com") throw new Error(`Unexpected Slack fixture origin ${url.origin}`);
|
||||
if (url.pathname === "/api/auth.test") {
|
||||
expect(init?.method).toBe("POST");
|
||||
const token = new Headers(init?.headers).get("authorization") ?? "";
|
||||
const suffix = token.slice(token.lastIndexOf("-") + 1);
|
||||
return Response.json({ ok: true, team_id: `T${suffix}`, team: "Fixture", user_id: `U${suffix}`, user: "fixture" },
|
||||
{ headers: { "x-oauth-scopes": SLACK_CHAT_BOT_SCOPES.join(",") } });
|
||||
}
|
||||
if (url.pathname === "/api/conversations.list") return Response.json({ ok: true,
|
||||
channels: [{ id: "CTEST", name: "fixture", is_member: true }], response_metadata: { next_cursor: "" } });
|
||||
if (["/api/reactions.add", "/api/reactions.remove"].includes(url.pathname)) return Response.json({ ok: true });
|
||||
throw new Error(`Unexpected Slack fixture method ${url.pathname}`);
|
||||
};
|
||||
providerFetch = slackFetch;
|
||||
heartbeat = heartbeatService(db);
|
||||
service = chatChannelService(db, { runtime, heartbeat, fetch: slackFetch, publicBaseUrl: "https://tenant.example",
|
||||
deferWebhookProcessing: true, scheduleDeferredWork: work => { deferred.push(work); } });
|
||||
});
|
||||
afterEach(async () => {
|
||||
const errors: unknown[] = [];
|
||||
for (const release of releases) release();
|
||||
for (const work of heldLeases) { try { await work; } catch (error) { errors.push(error); } }
|
||||
try { await service?.shutdown(); } catch (error) { errors.push(error); }
|
||||
try { await heartbeat?.drainActiveRunExecutions(); } catch (error) { errors.push(error); }
|
||||
try {
|
||||
await db.delete(chatEndpoints).where(eq(chatEndpoints.companyId, companyId));
|
||||
await db.delete(financeEvents).where(eq(financeEvents.companyId, companyId));
|
||||
await db.delete(costEvents).where(eq(costEvents.companyId, companyId));
|
||||
await db.delete(activityLog).where(eq(activityLog.companyId, companyId));
|
||||
await db.delete(agentTaskSessions).where(eq(agentTaskSessions.companyId, companyId));
|
||||
await db.delete(agentRuntimeState).where(eq(agentRuntimeState.companyId, companyId));
|
||||
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, companyId));
|
||||
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId));
|
||||
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId));
|
||||
await db.delete(issueComments).where(eq(issueComments.companyId, companyId));
|
||||
await db.delete(issues).where(eq(issues.companyId, companyId));
|
||||
await db.delete(agents).where(eq(agents.companyId, companyId));
|
||||
await db.delete(companyMemberships).where(eq(companyMemberships.companyId, companyId));
|
||||
await db.delete(companySecrets).where(eq(companySecrets.companyId, companyId));
|
||||
await db.delete(companySkills).where(eq(companySkills.companyId, companyId));
|
||||
await db.delete(companies).where(eq(companies.id, companyId));
|
||||
await db.delete(authUsers).where(eq(authUsers.id, actorId));
|
||||
} catch (error) { errors.push(error); }
|
||||
try { network.assertQuiet(); } catch (error) { errors.push(error); }
|
||||
network.restore(); vi.restoreAllMocks(); vi.unstubAllEnvs();
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
if (errors.length) throw new AggregateError(errors, "Slack owner fixture cleanup failed; original body failure is reported separately by Vitest");
|
||||
});
|
||||
afterAll(async () => {
|
||||
const closed = await Promise.allSettled([closeCheckpointDb(db, 1), closeFileStorageDb(db, 1), closeLivenessDb(db, 1)]);
|
||||
await db?.$client.end({ timeout: 1 }); await temporary?.cleanup();
|
||||
const errors = closed.flatMap(value => value.status === "rejected" ? [value.reason] : []);
|
||||
if (errors.length) throw new AggregateError(errors, "Slack owner auxiliary shutdown failed");
|
||||
});
|
||||
function makeRuntime() {
|
||||
const runtime = new ChatRuntime();
|
||||
const replace = runtime.replaceEndpoint.bind(runtime);
|
||||
vi.spyOn(runtime, "replaceEndpoint").mockImplementation(async input => {
|
||||
if (input.providerConfig.provider !== "slack") throw new Error("Unexpected fixture provider");
|
||||
const value = await replace({ ...input, providerConfig: { ...input.providerConfig,
|
||||
credentials: { ...input.providerConfig.credentials, apiUrl: "http://127.0.0.1:1/api/" } },
|
||||
callbacks: { ...input.callbacks, onMessage: async event => { callbacks.push(event); await input.callbacks.onMessage(event); } } });
|
||||
transports.push(stubSlackFixtureTransport(value.getProviderAdapter())); return value;
|
||||
});
|
||||
return runtime;
|
||||
}
|
||||
async function createEndpoint(suffix: string) {
|
||||
const endpointId = randomUUID(), connectionId = randomUUID(), publicId = randomUUID().replaceAll("-", "").padEnd(43, "a");
|
||||
await db.insert(toolConnections).values({ id: connectionId, companyId, applicationId, name: `Slack ${suffix}`, uid: endpointId,
|
||||
transport: "chat_sdk", connectionPurpose: "channel", enabled: true, status: "active" });
|
||||
await db.insert(chatEndpoints).values({ id: endpointId, companyId, connectionId, assignedAgentId: agentId, publicId,
|
||||
provider: "slack", status: "draft", sponsorUserId: actorId, allowGroupChats: true, setup: { step: "provider_setup" } });
|
||||
await db.insert(chatSlackRegistrations).values({ companyId, endpointId, requestId: randomUUID(), status: "configured", appId: `A${suffix}`,
|
||||
manifest: {}, manifestHash: "fixture", callbackUri: `https://tenant.example/api/chat-webhooks/${publicId}/slack`, createdByUserId: actorId });
|
||||
await service.configure(endpointId, { action: "configure", credentials: { botToken: `xoxb-fixture-${suffix}`, signingSecret: secret } }, actorId);
|
||||
expect((await setup(endpointId)).slackCloudIntent).toMatchObject({ companyId, endpointId, connectionId,
|
||||
instanceId: cloud.config.instanceId, subject: actorId, active: true });
|
||||
return endpointId;
|
||||
}
|
||||
async function confirm(endpointId: string) {
|
||||
const owner = githubConnectionEventService(db, { connector: cloud.connector });
|
||||
await owner.pollOnce({ forDelivery: true });
|
||||
expect((await setup(endpointId)).slackCloudIntent).toBeUndefined();
|
||||
expect((await setup(endpointId)).slackCloudEvents).toBeDefined();
|
||||
return owner;
|
||||
}
|
||||
async function activate(endpointId: string) {
|
||||
const endpoint = await row(endpointId);
|
||||
const accountId = endpoint.providerAccountId;
|
||||
if (!accountId) throw new Error("Normal Slack configure did not confirm its workspace");
|
||||
const payload = JSON.stringify({ type: "url_verification", team_id: accountId, api_app_id: `A${accountId.slice(1)}`, challenge: "fixture-challenge" });
|
||||
const timestamp = String(Math.floor(Date.now() / 1000));
|
||||
const response = await service.handleWebhook(endpoint.publicId, "slack", new Request(`https://tenant.example/api/chat-webhooks/${endpoint.publicId}/slack`, {
|
||||
method: "POST", headers: { "content-type": "application/json", "x-slack-request-timestamp": timestamp,
|
||||
"x-slack-signature": `v0=${createHmac("sha256", secret).update(`v0:${timestamp}:${payload}`).digest("hex")}` }, body: payload }));
|
||||
expect(response.status).toBe(200); await response.body?.cancel();
|
||||
await service.configure(endpointId, { action: "verify" }, actorId);
|
||||
const [principal] = await db.insert(chatExternalPrincipals).values({ companyId, provider: "slack", providerAccountId: accountId,
|
||||
externalId: "UOPERATOR", kind: "user", displayName: "Operator" }).returning();
|
||||
await db.insert(chatIdentityLinks).values({ companyId, endpointId, principalId: principal!.id, paperclipUserId: actorId, status: "linked", confirmedAt: new Date() });
|
||||
expect((await service.test(endpointId, { optionalSlackTestForUser: actorId })).status).toBe("active");
|
||||
}
|
||||
async function holdCredentials(endpointId: string) {
|
||||
let acquired!: () => void;
|
||||
const ready = new Promise<void>(resolve => { acquired = resolve; });
|
||||
const release = new Promise<void>(resolve => { releases.push(resolve); });
|
||||
heldLeases.push(chatCredentialMutationLease(db)(await row(endpointId), async lease => { await lease.assertOwned(); acquired(); await release; }));
|
||||
await ready;
|
||||
}
|
||||
|
||||
it("normal configure stages SQL intent, worker unseals into registered SDK and old generations cannot reenter", async () => {
|
||||
const endpointId = await createEndpoint("BOT");
|
||||
await confirm(endpointId); await activate(endpointId); await confirm(endpointId);
|
||||
const endpoint = await row(endpointId), saved = await setup(endpointId), enrolled = saved.slackCloudEvents!;
|
||||
const callbackEvidence = structuredClone(saved.slackCallbackSurfaces);
|
||||
const delivered = cloud.enqueue({ companyId, endpointId, publicId: endpoint.publicId, connectionId: endpoint.connectionId,
|
||||
runtimeGeneration: saved.runtimeGeneration, teamId: enrolled.teamId, appId: enrolled.appId, bindingId: enrolled.bindingId }, secret, "EvDefaultOwner");
|
||||
const owner = githubConnectionEventService(db, { connector: cloud.connector });
|
||||
expect((await owner.pollOnce({ forDelivery: true })).processed).toBe(1);
|
||||
expect(callbacks).toHaveLength(1);
|
||||
expect(callbacks[0]!.message.author.userId).toBe("UOPERATOR");
|
||||
expect(cloud.acknowledged).toEqual([delivered.id]);
|
||||
expect((await setup(endpointId)).slackCallbackSurfaces).toEqual(callbackEvidence);
|
||||
expect(await db.select().from(chatDeliveries).where(eq(chatDeliveries.endpointId, endpointId))).toHaveLength(1);
|
||||
for (const transport of transports) await transport.assertNoErrors();
|
||||
await service.configure(endpointId, { action: "pause" }, actorId);
|
||||
const oldRequest = new Request("https://tenant.example/old-receipt", { method: "POST", body: new Uint8Array(delivered.body),
|
||||
headers: { "content-type": "application/json", "x-slack-request-timestamp": delivered.packet.timestamp, "x-slack-signature": delivered.packet.signature } });
|
||||
const rejected = await service.handleWebhook(endpoint.publicId, "slack", oldRequest, { slackCloudGeneration: Number(saved.runtimeGeneration) });
|
||||
expect(await rejected.text()).toBe("superseded");
|
||||
expect(callbacks).toHaveLength(1);
|
||||
await owner.pollOnce({ forDelivery: true });
|
||||
expect((await setup(endpointId)).slackCloudIntent).toBeUndefined();
|
||||
expect((await setup(endpointId)).slackCloudRetiredGeneration).toBeGreaterThan(Number(saved.runtimeGeneration));
|
||||
}, 60_000);
|
||||
|
||||
it("two due registrations settle B while A still owns a busy credential lease", async () => {
|
||||
const a = await createEndpoint("BUSY"); await confirm(a); await activate(a); await confirm(a);
|
||||
const b = await createEndpoint("READY"); await confirm(b); await activate(b); await confirm(b);
|
||||
await service.configure(a, { action: "pause" }, actorId);
|
||||
const before = structuredClone((await setup(a)).slackCloudIntent!);
|
||||
await service.configure(b, { action: "pause" }, actorId);
|
||||
await holdCredentials(a);
|
||||
const start = cloud.calls.length;
|
||||
const owner = githubConnectionEventService(db, { connector: cloud.connector });
|
||||
await owner.pollOnce({ forDelivery: true });
|
||||
await owner.pollOnce({ forDelivery: true });
|
||||
expect((await setup(b)).slackCloudIntent).toBeUndefined();
|
||||
expect((await setup(b)).slackCloudRetiredGeneration).toBeDefined();
|
||||
const retained = (await setup(a)).slackCloudIntent!;
|
||||
expect(retained).toMatchObject({ ...before, attempts: 1, failureCode: "credential_lease_busy", nextAttemptAt: expect.any(String) });
|
||||
expect(Date.parse(retained.nextAttemptAt!)).toBeGreaterThan(Date.now());
|
||||
expect(cloud.calls.slice(start).filter(call => call.enrollment?.endpointId === a)).toEqual([]);
|
||||
expect(owner.nextPollAt()).not.toBeNull();
|
||||
releases.splice(0).forEach(release => release()); await Promise.all(heldLeases);
|
||||
const delay = Math.max(0, Date.parse(retained.nextAttemptAt!) - Date.now());
|
||||
await new Promise(resolve => setTimeout(resolve, delay));
|
||||
await owner.pollOnce();
|
||||
expect((await setup(a)).slackCloudIntent).toBeUndefined();
|
||||
expect((await setup(a)).slackCloudRetiredGeneration).toBeDefined();
|
||||
}, 60_000);
|
||||
|
||||
it("normal configuration rollback leaves no registration debt or retirement request", async () => {
|
||||
const endpointId = await createEndpoint("ROLLBACK"); await confirm(endpointId); await activate(endpointId); await confirm(endpointId);
|
||||
const before = await row(endpointId), requests = cloud.calls.length;
|
||||
await service.shutdown(); runtime = makeRuntime();
|
||||
let rejectAfterIntent = false;
|
||||
const stop = subscribeDatabaseWork(db, DELIVERY_QUEUES.connectionEvents, event => { if (event === "intent") rejectAfterIntent = true; });
|
||||
service = chatChannelService(db, { runtime, heartbeat, fetch: providerFetch, publicBaseUrl: "https://tenant.example",
|
||||
deferWebhookProcessing: true, scheduleDeferredWork: work => { deferred.push(work); }, renewCredentialMutationLease: async () => !rejectAfterIntent });
|
||||
try {
|
||||
await expect(service.configure(endpointId, { action: "pause" }, actorId)).rejects.toMatchObject({ code: "CHAT_CREDENTIAL_LEASE_LOST" });
|
||||
expect((await row(endpointId)).status).toBe(before.status);
|
||||
expect(await setup(endpointId)).toEqual(before.setup);
|
||||
expect(cloud.calls).toHaveLength(requests);
|
||||
} finally { stop(); }
|
||||
const owner = githubConnectionEventService(db, { connector: cloud.connector });
|
||||
await owner.pollOnce({ forDelivery: true });
|
||||
expect(cloud.calls.filter(call => call.enrollment?.active === false)).toEqual([]);
|
||||
}, 60_000);
|
||||
|
||||
it.each(["intent", "generation"] as const)("real SQL refuses an enrollment response after newer %s custody wins", async change => {
|
||||
const endpointId = await createEndpoint(`CAS${change}`);
|
||||
const original = (await setup(endpointId)).slackCloudIntent!;
|
||||
const successor = { ...original, id: randomUUID(), runtimeGeneration: original.runtimeGeneration + (change === "generation" ? 1 : 0) };
|
||||
cloud.beforeReceipt(async () => {
|
||||
await db.update(chatEndpoints).set({ setup: sql`jsonb_set(jsonb_set(${chatEndpoints.setup}, '{slackCloudIntent}', ${JSON.stringify(successor)}::jsonb), '{runtimeGeneration}', ${JSON.stringify(successor.runtimeGeneration)}::jsonb)` })
|
||||
.where(and(eq(chatEndpoints.id, endpointId), eq(chatEndpoints.companyId, companyId)));
|
||||
});
|
||||
const producer = createSlackCloudEnrollment(db, { resolveCredentials: async () => ({ signingSecret: secret }),
|
||||
runtimeGeneration: value => Number(value.runtimeGeneration), connector: cloud.connector });
|
||||
await expect(chatCredentialMutationLease(db)(await row(endpointId), lease => producer(endpointId, undefined, lease))).rejects.toThrow("changed");
|
||||
expect((await setup(endpointId)).slackCloudIntent).toEqual(successor);
|
||||
expect((await setup(endpointId)).slackCloudEvents).toBeUndefined();
|
||||
}, 60_000);
|
||||
|
||||
it("malformed sealed Slack receipt is acknowledged once instead of retained as an owner outage", async () => {
|
||||
const endpointId = await createEndpoint("INVALID"); await confirm(endpointId);
|
||||
const endpoint = await row(endpointId), saved = await setup(endpointId), enrolled = saved.slackCloudEvents!;
|
||||
const malformed = cloud.enqueue({ companyId, endpointId: "bad", publicId: endpoint.publicId, connectionId: endpoint.connectionId,
|
||||
runtimeGeneration: saved.runtimeGeneration, teamId: enrolled.teamId, appId: enrolled.appId, bindingId: enrolled.bindingId }, secret, "EvMalformed");
|
||||
const owner = githubConnectionEventService(db, { connector: cloud.connector });
|
||||
const result = await owner.pollOnce({ forDelivery: true });
|
||||
expect(result.ignored).toBe(1); expect(result.failed).toBe(0);
|
||||
expect(cloud.acknowledged).toEqual([malformed.id]);
|
||||
expect(callbacks).toEqual([]);
|
||||
}, 60_000);
|
||||
|
||||
it.each(["intent", "generation", "malformed-generation"] as const)(
|
||||
"busy retry CAS preserves a concurrent %s winner and rethrows the exact owner error", async change => {
|
||||
const endpointId = await createEndpoint(`BUSYCAS${change}`);
|
||||
const before = await setup(endpointId), original = before.slackCloudIntent!;
|
||||
const successor = change === "intent" ? { ...original, id: randomUUID() } : original;
|
||||
const generation: unknown = change === "malformed-generation" ? { invalid: true }
|
||||
: Number(before.runtimeGeneration) + (change === "generation" ? 1 : 0);
|
||||
const busy = conflict("Synthetic owner credential lane is busy", { code: "chat_endpoint_credentials_busy" });
|
||||
const requests = cloud.calls.length;
|
||||
const pendingBefore = databaseWorkPending(db, DELIVERY_QUEUES.connectionEvents), inheritedSignals: string[] = [];
|
||||
const stopInherited = subscribeDatabaseWork(db, DELIVERY_QUEUES.connectionEvents, event => { inheritedSignals.push(event); });
|
||||
try {
|
||||
console.info(JSON.stringify({ slackRetrySignalProbe: "subscription", endpointId, pendingBefore, immediateSignals: [...inheritedSignals] }));
|
||||
expect(inheritedSignals).toEqual(pendingBefore ? ["intent"] : []);
|
||||
await reconcileDatabaseWork(db, DELIVERY_QUEUES.connectionEvents);
|
||||
const pendingAfter = databaseWorkPending(db, DELIVERY_QUEUES.connectionEvents);
|
||||
console.info(JSON.stringify({ slackRetrySignalProbe: "reconciled", endpointId, pendingBefore, pendingAfter, inheritedSignals: [...inheritedSignals] }));
|
||||
expect(pendingAfter).toBe(false);
|
||||
} finally { stopInherited(); }
|
||||
const signals: string[] = [];
|
||||
const stopSignals = subscribeDatabaseWork(db, DELIVERY_QUEUES.connectionEvents, event => { signals.push(event); });
|
||||
const unregister = registerSlackCloudEnrollmentOwner(db, async id => {
|
||||
expect(id).toBe(endpointId);
|
||||
await db.update(chatEndpoints).set({ setup: sql`jsonb_set(jsonb_set(${chatEndpoints.setup}, '{slackCloudIntent}', ${JSON.stringify(successor)}::jsonb), '{runtimeGeneration}', ${JSON.stringify(generation)}::jsonb)` })
|
||||
.where(and(eq(chatEndpoints.id, endpointId), eq(chatEndpoints.companyId, companyId)));
|
||||
throw busy;
|
||||
});
|
||||
try {
|
||||
await expect(reconcilePendingSlackCloudEnrollments(db, new Date())).rejects.toBe(busy);
|
||||
const after = await setup(endpointId);
|
||||
expect(after.slackCloudIntent).toEqual(successor);
|
||||
expect(after.runtimeGeneration).toEqual(generation);
|
||||
expect(after.slackCloudEvents).toBeUndefined();
|
||||
expect(cloud.calls).toHaveLength(requests);
|
||||
expect(signals).toEqual([]);
|
||||
} finally { unregister(); stopSignals(); }
|
||||
}, 60_000);
|
||||
|
||||
it.each(["unknown-error", "unrelated-conflict"] as const)(
|
||||
"the %s owner failure propagates without a busy retry rewrite", async kind => {
|
||||
const endpointId = await createEndpoint(`UNKNOWN${kind}`), before = await setup(endpointId);
|
||||
const original = kind === "unknown-error" ? new Error("Synthetic unknown owner failure")
|
||||
: conflict("Synthetic different authority failure", { code: "chat_endpoint_runtime_superseded" });
|
||||
const requests = cloud.calls.length;
|
||||
const unregister = registerSlackCloudEnrollmentOwner(db, async id => { expect(id).toBe(endpointId); throw original; });
|
||||
try {
|
||||
await expect(reconcilePendingSlackCloudEnrollments(db, new Date())).rejects.toBe(original);
|
||||
expect(await setup(endpointId)).toEqual(before);
|
||||
expect(cloud.calls).toHaveLength(requests);
|
||||
} finally { unregister(); }
|
||||
}, 60_000);
|
||||
|
||||
it("an injected retry transaction failure preserves the original busy error and persistence cause", async () => {
|
||||
const endpointId = await createEndpoint("RETRYWRITE"), before = await setup(endpointId);
|
||||
const busy = conflict("Synthetic owner credential lane is busy", { code: "chat_endpoint_credentials_busy" });
|
||||
const writeError = new Error("Synthetic retry transaction unavailable"), requests = cloud.calls.length;
|
||||
const unregister = registerSlackCloudEnrollmentOwner(db, async id => { expect(id).toBe(endpointId); throw busy; });
|
||||
const transaction = vi.spyOn(db, "transaction").mockRejectedValueOnce(writeError);
|
||||
try {
|
||||
const error = await reconcilePendingSlackCloudEnrollments(db, new Date()).then(() => undefined, error => error);
|
||||
expect(error).toBeInstanceOf(AggregateError);
|
||||
if (!(error instanceof AggregateError)) throw new Error("Expected retry failure to retain both causes", { cause: error });
|
||||
expect(error.errors).toHaveLength(2);
|
||||
expect(error.errors[0]).toBe(busy);
|
||||
expect(error.errors[1]).toBe(writeError);
|
||||
expect(transaction).toHaveBeenCalledOnce();
|
||||
expect(await setup(endpointId)).toEqual(before);
|
||||
expect(cloud.calls).toHaveLength(requests);
|
||||
} finally { transaction.mockRestore(); unregister(); }
|
||||
}, 60_000);
|
||||
|
||||
it("public tool removal retains Slack retirement custody through broker outage and ordinary owner recovery", async () => {
|
||||
const endpointId = await createEndpoint("TOOLREMOVE"); await confirm(endpointId); await activate(endpointId); await confirm(endpointId);
|
||||
const originalEndpoint = await row(endpointId), originalSetup = await setup(endpointId);
|
||||
const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.id, originalEndpoint.connectionId));
|
||||
const signingRef = connection!.credentialSecretRefs.find(ref => ref.configPath === "credentials.signingSecret");
|
||||
if (!signingRef) throw new Error("Normal Slack configure did not persist its signing credential binding");
|
||||
const credentialOwner = { consumerType: "tool_connection" as const, consumerId: connection!.id,
|
||||
configPath: "credentials.signingSecret", actorType: "system" as const };
|
||||
await expect(secretService(db).resolveSecretValue(companyId, signingRef.secretId, "latest", credentialOwner)).resolves.toBe(secret);
|
||||
cloud.setUnavailable(true);
|
||||
const requests = vi.mocked(cloud.request).mock.calls.length;
|
||||
const tools = toolAccessService(db);
|
||||
const removed = await tools.archiveConnection(connection!.id, companyId, { actorType: "user", actorId });
|
||||
expect(removed.connection).toMatchObject({ id: connection!.id, status: "archived", enabled: false });
|
||||
const archived = await row(endpointId);
|
||||
expect(archived.status).toBe("archived");
|
||||
const [revoked] = await db.select().from(toolConnections).where(eq(toolConnections.id, connection!.id));
|
||||
expect(revoked!.credentialSecretRefs).toEqual([]);
|
||||
expect(revoked!.credentialRefs).toEqual([]);
|
||||
expect(await db.select().from(companySecretBindings).where(and(eq(companySecretBindings.companyId, companyId),
|
||||
eq(companySecretBindings.targetType, "tool_connection"), eq(companySecretBindings.targetId, connection!.id)))).toEqual([]);
|
||||
await expect(secretService(db).resolveSecretValue(companyId, signingRef.secretId, "latest", credentialOwner)).rejects.toThrow();
|
||||
expect(vi.mocked(cloud.request).mock.calls).toHaveLength(requests);
|
||||
const intent = (await setup(endpointId)).slackCloudIntent;
|
||||
expect(intent).toMatchObject({ version: 1, active: false, companyId, endpointId, connectionId: connection!.id,
|
||||
publicId: originalEndpoint.publicId, runtimeGeneration: Number(originalSetup.runtimeGeneration),
|
||||
subject: originalSetup.slackCloudSubject, instanceId: originalSetup.slackCloudInstanceId, attempts: 0 });
|
||||
expect(JSON.stringify(intent)).not.toContain(secret);
|
||||
await tools.archiveConnection(connection!.id, companyId, { actorType: "user", actorId });
|
||||
expect((await setup(endpointId)).slackCloudIntent).toEqual(intent);
|
||||
const owner = githubConnectionEventService(db, { connector: cloud.connector });
|
||||
await expect(owner.pollOnce({ forDelivery: true })).rejects.toMatchObject({ status: 503 });
|
||||
const retry = (await setup(endpointId)).slackCloudIntent!;
|
||||
expect(retry).toMatchObject({ ...intent, attempts: 1, failureCode: "broker_unavailable", nextAttemptAt: expect.any(String) });
|
||||
const retryAt = Date.parse(retry.nextAttemptAt!);
|
||||
expect(retryAt).toBeGreaterThan(Date.now());
|
||||
cloud.setUnavailable(false); cloud.install();
|
||||
await new Promise(resolve => setTimeout(resolve, Math.max(0, retryAt - Date.now())));
|
||||
await owner.pollOnce();
|
||||
expect((await setup(endpointId)).slackCloudIntent).toBeUndefined();
|
||||
expect((await setup(endpointId)).slackCloudRetiredGeneration).toBe(Number(originalSetup.runtimeGeneration));
|
||||
expect(cloud.calls.filter(call => call.operation === "slack-events" && call.enrollment?.active === false
|
||||
&& call.enrollment.endpointId === endpointId)).toHaveLength(1);
|
||||
expect(callbacks).toEqual([]);
|
||||
}, 60_000);
|
||||
|
||||
it("public tool removal rolls back both archive writes without dropping current Slack custody or credentials", async () => {
|
||||
const endpointId = await createEndpoint("TOOLROLLBACK"); await confirm(endpointId); await activate(endpointId); await confirm(endpointId);
|
||||
const before = await row(endpointId);
|
||||
const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.id, before.connectionId));
|
||||
const failure = new Error("Synthetic removal transaction rollback"), requests = vi.mocked(cloud.request).mock.calls.length;
|
||||
const realTransaction = db.transaction.bind(db);
|
||||
const transaction = vi.spyOn(db, "transaction").mockImplementationOnce(async work => realTransaction(async tx => {
|
||||
await work(tx);
|
||||
const [archivedEndpoint] = await tx.select().from(chatEndpoints).where(eq(chatEndpoints.id, endpointId));
|
||||
const [archivedConnection] = await tx.select().from(toolConnections).where(eq(toolConnections.id, connection!.id));
|
||||
expect(archivedEndpoint!.status).toBe("archived");
|
||||
expect(archivedConnection).toMatchObject({ status: "archived", enabled: false });
|
||||
throw failure;
|
||||
}));
|
||||
try {
|
||||
await expect(toolAccessService(db).archiveConnection(connection!.id, companyId, { actorType: "user", actorId })).rejects.toBe(failure);
|
||||
expect(await row(endpointId)).toMatchObject({ status: before.status, setup: before.setup });
|
||||
const [after] = await db.select().from(toolConnections).where(eq(toolConnections.id, connection!.id));
|
||||
expect(after).toMatchObject({ status: connection!.status, enabled: connection!.enabled,
|
||||
credentialSecretRefs: connection!.credentialSecretRefs, credentialRefs: connection!.credentialRefs });
|
||||
expect(vi.mocked(cloud.request).mock.calls).toHaveLength(requests);
|
||||
} finally { transaction.mockRestore(); }
|
||||
}, 60_000);
|
||||
|
||||
it("public tool removal refuses a foreign company before Slack archival or retirement staging", async () => {
|
||||
const endpointId = await createEndpoint("TOOLFOREIGN"); await confirm(endpointId);
|
||||
const before = await row(endpointId), requests = vi.mocked(cloud.request).mock.calls.length;
|
||||
await expect(toolAccessService(db).archiveConnection(before.connectionId, randomUUID(), { actorType: "user", actorId })).rejects.toMatchObject({ status: 404 });
|
||||
expect(await row(endpointId)).toEqual(before);
|
||||
expect(vi.mocked(cloud.request).mock.calls).toHaveLength(requests);
|
||||
}, 60_000);
|
||||
});
|
||||
@@ -381,15 +381,27 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
|
||||
// window. A company hint must not make that real deadline wait one second.
|
||||
await coordinator!.stop(); coordinator = undefined;
|
||||
const dueAt = Date.now() + 500;
|
||||
const actualWakeup = wakeup.getMockImplementation()!;
|
||||
let otherWakeEnteredAt: number | null = null;
|
||||
wakeup.mockImplementation((id, options) => {
|
||||
// Capture dispatch ENTRY before the original mock performs any DB I/O.
|
||||
if (id === other.agentId) otherWakeEnteredAt ??= Date.now();
|
||||
return actualWakeup(id, options);
|
||||
});
|
||||
await db.update(statusCards).set({ nextEvalAt: new Date(dueAt), nextEvalReason: "refresh" }).where(eq(statusCards.id, other.id));
|
||||
await start(service); await flush();
|
||||
await issueService(db).addComment(otherIssue.id, "Another company's coalesced human update", { userId: "board-user" });
|
||||
await issueService(db).addComment((await db.select({ id: issues.id }).from(issues).where(eq(issues.companyId, originalCompany)))[0]!.id,
|
||||
"More changes while another company is due", { userId: "board-user" });
|
||||
await flush();
|
||||
const armedBeforeAdvance = coordinator!.nextWakeAt();
|
||||
// A null deadline can mean the due work is already running. Any still
|
||||
// armed deadline must preserve the known earlier obligation.
|
||||
if (armedBeforeAdvance !== null) expect(armedBeforeAdvance).toBeLessThanOrEqual(dueAt);
|
||||
await vi.advanceTimersByTimeAsync(500); await flush();
|
||||
await vi.waitFor(() => expect(wakeup).toHaveBeenCalledOnce());
|
||||
expect(Date.now()).toBeLessThan(dueAt + 500);
|
||||
expect(otherWakeEnteredAt).not.toBeNull();
|
||||
expect(otherWakeEnteredAt!).toBeLessThan(dueAt + 500);
|
||||
expect(wakeup.mock.calls[0]![0]).not.toBe(originalAgent);
|
||||
const generated = (await statusCardService(db).getById(other.id))!;
|
||||
expect(generated.generatingIssueId).toBeTruthy();
|
||||
|
||||
Reference in new issue
Block a user