mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
Merge master and adopt its scheduling regression fix
Master independently corrected the status-card dispatch timing assertion. Keep that exact test and remove the duplicate PR fixture change. Storage behavior remains unchanged. Preserve normal workflow inputs once, resolving the obsolete duplicate-property CI failure. Verified 72 affected regressions, ten workflow checks, the isolated workflow fixture typecheck, maintained preflight, module boundaries, whitespace and secrets/PII checks. Fresh CI and exact-head Apex review remain required. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
commit
d0a937768d
36 files changed
+2818
-40
No files matched your search
@@ -179,3 +179,101 @@ An explicit managed reconnect reinstalls the saved app and verifies the replacem
|
||||
bot credentials. Its request ID is saved before dispatch, so a duplicate completed
|
||||
request does not reinstall again. Credential-configuration recovery still resumes
|
||||
the saved token after a downstream failure. No reconnect creates another Slack app.
|
||||
|
||||
## Durable Cloud Events API delivery
|
||||
|
||||
A broker that advertises `slackEvents.version=1` can accept a signed Events API
|
||||
registration after ordinary configuration settles. The producer uses the saved
|
||||
child app ID, current workspace, signing key, endpoint/connection and runtime
|
||||
generation. It retains the existing public callback URL. Missing verified app
|
||||
identity or broker support leaves the existing setup in place; it does not guess
|
||||
an identity or substitute the Slack manager's signing key.
|
||||
|
||||
Cloud stores the child signing key through its existing encrypted stack vault.
|
||||
Registration metadata and receipts contain no plaintext credentials. An unchanged
|
||||
credential/endpoint identity reuses its immutable vault ref across generations;
|
||||
rotation and current registration authority remain separately fenced. Removal
|
||||
retires exact queue membership and prevents an old generation from reenrolling.
|
||||
The local configuration transaction first saves nonsecret enrollment or
|
||||
retirement intent and signals the existing connection-events owner. Broker work
|
||||
runs through that owner after commit; configuration APIs do not await an optional
|
||||
Cloud request. Configure, pause and removal can complete locally while Cloud is unavailable. The owner
|
||||
retries only retained due intent and reconstructs it after restart, including
|
||||
removal for archived endpoints after credential cleanup. Confirmation replaces
|
||||
only the same current intent and generation; a completed remote call with a
|
||||
lost response remains pending. Current authorization errors retain visible
|
||||
attention debt instead of claiming registration succeeded. Event draining and
|
||||
acknowledgments run before enrollment reconciliation. The owner then admits one
|
||||
due enrollment operation and schedules the earliest retained responsibility.
|
||||
A busy credential lease or enrollment failure retains a bounded retry without
|
||||
aborting that event drain. Rollback leaves no committed intent to run.
|
||||
|
||||
Retirement sends the original responsible subject and the exact instance,
|
||||
company, endpoint, public ID, connection and monotonic generation. Cloud can
|
||||
publish a nonreceiving generation floor without a signing key, live descriptor
|
||||
or previously returned binding ID. This fences delayed enrollment after local
|
||||
removal. Normal current broker authority still applies; the subject alone does
|
||||
not authorize an operation. An instance that never enrolled with Cloud creates
|
||||
no new intent or background retry when Cloud is absent. Previously configured
|
||||
retirement responsibility retains its original instance identity as visible
|
||||
attention if that identity becomes unavailable. Unsupported capability or disabled
|
||||
new enrollment retains bounded retry responsibility; a closed generation or
|
||||
identity conflict requires attention. Current authorized confirmation is required
|
||||
before clearing the exact pending intent.
|
||||
|
||||
Ordinary redeploys retain the same stack identity. To move enrollment to another
|
||||
stack, first confirm retirement under the old stack's authority, with no pending
|
||||
old intent. Then explicitly reconfigure under the new authorized stack at a newer
|
||||
runtime generation. The old stored identity remains until the new broker receipt
|
||||
matches the current intent and generation. An environment change cannot transfer
|
||||
live responsibility. Unavailable old authority remains visible attention.
|
||||
|
||||
A missing or disabled signing credential also requires attention. Enrollment does
|
||||
not retry it as a broker outage. Incoming receipts retain typed attention before
|
||||
SDK admission; uncertain authentication does not acknowledge the receipt.
|
||||
|
||||
Only authenticated `event_callback` messages use this durable lane. The sealed
|
||||
queue retains original bytes, signature, timestamp and acceptance time. The app
|
||||
checks current endpoint/generation authority before SDK handling. It verifies
|
||||
original freshness at Cloud acceptance; ordinary external requests still use
|
||||
current time. The credential snapshot lease releases before normal lifecycle
|
||||
callbacks reacquire their own leases.
|
||||
|
||||
Private replay provenance prevents internal replay URLs from replacing saved
|
||||
external callback verification evidence. The normal transactional chat delivery
|
||||
ledger remains the message admission authority. An ambiguous callback or ACK
|
||||
failure retains its receipt; malformed receipts rejected before useful admission
|
||||
can be explicitly ignored with bounded diagnostics. Credential-authentication
|
||||
uncertainty requires attention and is not silently acknowledged. The broker's
|
||||
existing retention horizon remains the disclosed recovery boundary.
|
||||
|
||||
This does not qualify a complete Slack endpoint for sleep. Slash commands,
|
||||
interactive actions, modal validation and challenges still need their full
|
||||
synchronous response semantics. The endpoint blocker remains. Qualification
|
||||
must cover normal signed enrollment, durable retry after removal, sealed drain,
|
||||
the actual event worker and SDK, lost-ACK redelivery and one domain effect,
|
||||
generation changes, and callback evidence. Synthetic protocol coverage does not
|
||||
prove live Slack, complete configuration UI or a deployed stop/restart/wake journey.
|
||||
|
||||
### Reproduce the paired Cloud acceptance
|
||||
|
||||
Use the repository's supported Node version and installed dependencies. Compile
|
||||
the companion Cloud checkout first. Set `PAPERCLIP_SLACK_TEST_DATABASE_URL` and
|
||||
`SLACK_EVENT_TEST_DATABASE_URL` to two distinct, owned loopback PostgreSQL
|
||||
`*_test` databases. Set `SLACK_TEST_CLOUD_DIST` to its absolute compiled `dist`
|
||||
directory. This test applies actual migrations and uses only those two databases.
|
||||
It does not create a PostgreSQL cluster or use a live Slack provider.
|
||||
|
||||
From the app repository root, run:
|
||||
|
||||
```sh
|
||||
node server/scripts/test-slack-cloud-postgres.mjs
|
||||
```
|
||||
|
||||
The launcher uses the current Node executable and one Vitest worker. It loads the
|
||||
checked-in prewire guard in that actual worker before client construction. The
|
||||
controlled process child explicitly loads the same guard before its script.
|
||||
The guard rejects off-host and unowned socket targets before transport. Caught
|
||||
unexpected attempts still fail the fixture. No private preload path or manual
|
||||
`SLACK_GATE_GUARD_PID` setting is needed. The fixture remains opt-in during the
|
||||
normal test suite; its default owner and guard regressions run without databases.
|
||||
@@ -1647,6 +1647,14 @@ On startup and on the periodic recovery loop, Paperclip performs the following r
|
||||
3. reconcile stranded assigned work
|
||||
4. scan silent active runs only for source-aware terminal folding and legacy cleanup; API reads classify ordinary output silence for the board UI
|
||||
|
||||
The orphan reap normally waits until a `running` run has been unchanged for five
|
||||
minutes. A legacy run whose controller lease belongs to an earlier server boot
|
||||
and has expired, which was still preparing (before adapter dispatch), and which
|
||||
has no recorded process is recovered when that lease expires instead. The
|
||||
expired lease proves that the earlier owner is gone, and no provider work
|
||||
started. A live lease, a lease of the current boot, a dispatched adapter, or a
|
||||
recorded process keeps the five-minute guard.
|
||||
|
||||
The stranded-work pass closes the gap where issue state survives a crash but the wake/run path does not. The silent-run scan covers the separate case where a live process exists but has stopped producing observable output.
|
||||
|
||||
Native finalization reconciliation (part of native recovery and the orphan
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
import { spawn } from "node:child_process";
|
||||
import { createRequire } from "node:module";
|
||||
import { existsSync, mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const patterns = { "--tool-removal-only": "public tool removal retains Slack retirement", "--fairness-only": "two due registrations", "--malformed-only": "malformed sealed Slack receipt",
|
||||
"--wiring-only": "normal configure stages SQL intent", "--cas-only": "real SQL refuses an enrollment response" };
|
||||
const args = process.argv.slice(2);
|
||||
if (!process.version.startsWith("v24.")) throw new Error("Slack owner fixture requires the current Node24 executable");
|
||||
if (args.length > 1 || args.length && !Object.hasOwn(patterns, args[0])) throw new Error("Unknown Slack owner fixture selector");
|
||||
const value = process.env.PAPERCLIP_SLACK_OWNER_TEST_DATABASE_URL;
|
||||
if (!value) throw new Error("Set PAPERCLIP_SLACK_OWNER_TEST_DATABASE_URL to the exact owned external test database");
|
||||
const fixture = new URL(value), admin = new URL(process.env.PAPERCLIP_TEST_POSTGRES_ADMIN_URL ?? "invalid:");
|
||||
if (!["postgres:", "postgresql:"].includes(fixture.protocol) || fixture.hostname !== "127.0.0.1" || fixture.port !== "55448"
|
||||
|| !/^[a-zA-Z0-9_]+_test$/.test(fixture.pathname.slice(1)) || fixture.search || fixture.hash
|
||||
|| admin.protocol !== fixture.protocol || admin.hostname !== fixture.hostname || admin.port !== fixture.port
|
||||
|| admin.pathname !== "/postgres" || admin.search || admin.hash) throw new Error("Slack owner fixture requires its explicit owned PG17 URL/admin grant");
|
||||
const log = process.env.SLACK_GATE_PROCESS_LOG;
|
||||
if (!log || !path.isAbsolute(log) || existsSync(log)) throw new Error("Slack owner fixture requires a fresh absolute guard log");
|
||||
const server = fileURLToPath(new URL("../", import.meta.url));
|
||||
const require = createRequire(new URL("../package.json", import.meta.url));
|
||||
const vitest = path.join(path.dirname(require.resolve("vitest/package.json")), "vitest.mjs");
|
||||
const guard = path.join(server, "src/__tests__/helpers/slack-cloud/prewire.cjs");
|
||||
const home = mkdtempSync(path.join(tmpdir(), "slack-owner-fixture-"));
|
||||
const env = { PATH: process.env.PATH, HOME: process.env.HOME, CODEX_HOME: process.env.CODEX_HOME, LANG: process.env.LANG,
|
||||
TMPDIR: home, PAPERCLIP_HOME: home, NODE_ENV: "test", NODE_OPTIONS: "", SLACK_GATE_EXPECT_PRELOAD: "1",
|
||||
PAPERCLIP_SLACK_OWNER_TEST_DATABASE_URL: value, PAPERCLIP_SLACK_TEST_DATABASE_URL: value,
|
||||
PAPERCLIP_TEST_POSTGRES_ADMIN_URL: process.env.PAPERCLIP_TEST_POSTGRES_ADMIN_URL, SLACK_GATE_PROCESS_LOG: log };
|
||||
const child = spawn(process.execPath, ["--require", guard, vitest, "run", "src/__tests__/slack-cloud-owner-work.test.ts",
|
||||
"--maxWorkers=1", "--no-file-parallelism", `--execArgv=--require=${guard}`,
|
||||
...(args.length ? [`--testNamePattern=${patterns[args[0]]}`] : [])], { cwd: server, env, stdio: "inherit" });
|
||||
for (const signal of ["SIGINT", "SIGTERM"]) process.on(signal, () => child.kill(signal));
|
||||
child.on("error", error => { console.error(error); process.exitCode = 1; });
|
||||
child.on("close", (code, signal) => { rmSync(home, { recursive: true, force: true }); process.exitCode = signal ? 1 : code ?? 1; });
|
||||
@@ -0,0 +1,47 @@
|
||||
import { spawn } from 'node:child_process';
|
||||
import { createRequire } from 'node:module';
|
||||
import { existsSync, mkdtempSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
if (process.argv.slice(2).length > 1 || process.argv.slice(2).some(arg => !['--instance-transition-only', '--normal-owner-only'].includes(arg))) throw new Error('Unknown Slack fixture selector');
|
||||
const transitionOnly = process.argv.includes('--instance-transition-only');
|
||||
const normalOwnerOnly = process.argv.includes('--normal-owner-only');
|
||||
const server = fileURLToPath(new URL('../', import.meta.url));
|
||||
const require = createRequire(new URL('../package.json', import.meta.url));
|
||||
const guard = path.join(server, 'src/__tests__/helpers/slack-cloud/prewire.cjs');
|
||||
const names = ['PAPERCLIP_SLACK_TEST_DATABASE_URL', 'SLACK_EVENT_TEST_DATABASE_URL'];
|
||||
const urls = names.map(name => {
|
||||
if (!process.env[name]) throw new Error(`Set ${name} to an owned dedicated PostgreSQL test database`);
|
||||
let url;
|
||||
try { url = new URL(process.env[name]); } catch { throw new Error(`${name} must name an explicit loopback *_test database`); }
|
||||
if (!['postgres:', 'postgresql:'].includes(url.protocol) || !['127.0.0.1', 'localhost', '::1', '[::1]'].includes(url.hostname)
|
||||
|| !/^[a-zA-Z0-9_]+_test$/.test(decodeURIComponent(url.pathname.slice(1)))
|
||||
|| ['options', 'database', 'dbname'].some(k => url.searchParams.has(k))) throw new Error(`${name} must name an explicit loopback *_test database`);
|
||||
return url;
|
||||
});
|
||||
if (urls[0].pathname === urls[1].pathname) throw new Error('Slack app and Cloud fixtures require distinct databases');
|
||||
if (!process.env.SLACK_TEST_CLOUD_DIST || !path.isAbsolute(process.env.SLACK_TEST_CLOUD_DIST)
|
||||
|| !existsSync(path.join(process.env.SLACK_TEST_CLOUD_DIST, 'connections/broker.js'))) throw new Error('SLACK_TEST_CLOUD_DIST must name the compiled companion Cloud dist directory');
|
||||
const vitest = path.join(path.dirname(require.resolve('vitest/package.json')), 'vitest.mjs');
|
||||
const ownedHome = mkdtempSync(path.join(tmpdir(), 'paperclip-slack-paired-'));
|
||||
const env = {
|
||||
PATH: process.env.PATH, HOME: process.env.HOME, CODEX_HOME: process.env.CODEX_HOME,
|
||||
LANG: process.env.LANG, TMPDIR: ownedHome, PAPERCLIP_HOME: ownedHome,
|
||||
NODE_ENV: 'test', NODE_OPTIONS: '', SLACK_GATE_EXPECT_PRELOAD: '1',
|
||||
PAPERCLIP_SLACK_TEST_DATABASE_URL: process.env.PAPERCLIP_SLACK_TEST_DATABASE_URL,
|
||||
SLACK_EVENT_TEST_DATABASE_URL: process.env.SLACK_EVENT_TEST_DATABASE_URL,
|
||||
SLACK_TEST_CLOUD_DIST: process.env.SLACK_TEST_CLOUD_DIST,
|
||||
SLACK_GATE_PROCESS_LOG: process.env.SLACK_GATE_PROCESS_LOG,
|
||||
};
|
||||
const child = spawn(process.execPath, ['--require', guard, vitest, 'run',
|
||||
...(transitionOnly ? ['src/services/chat-channels/slack/cloud-instance-transition.postgres.test.ts'] : normalOwnerOnly
|
||||
? ['src/services/chat-channels/slack/cloud-enrollment.postgres.test.ts'] : ['src/services/chat-channels/slack/cloud-enrollment.postgres.test.ts', 'src/services/chat-channels/slack/cloud-instance-transition.postgres.test.ts']), '--maxWorkers=1', '--no-file-parallelism', `--execArgv=--require=${guard}`],
|
||||
{ cwd: server, env, stdio: 'inherit' });
|
||||
for (const signal of ['SIGINT', 'SIGTERM']) process.on(signal, () => child.kill(signal));
|
||||
child.on('error', error => { console.error(error); process.exitCode = 1; });
|
||||
child.on('close', (code, signal) => {
|
||||
rmSync(ownedHome, { recursive: true, force: true });
|
||||
process.exitCode = signal ? 1 : code ?? 1;
|
||||
});
|
||||
@@ -17,7 +17,7 @@ import { mkdtemp, realpath, rm, access, readFile, writeFile, stat } from "node:f
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { createDb, companies, agents, agentTaskSessions, heartbeatRuns, companyMemberships, connectionGrants, toolApplications, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, plugins, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests, companySecrets, companySecretVersions, userSecretDefinitions, principalPermissionGrants } from "@paperclipai/db";
|
||||
import { createDb, type Db, companies, agents, agentTaskSessions, heartbeatRuns, companyMemberships, connectionGrants, toolApplications, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, plugins, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests, companySecrets, companySecretVersions, userSecretDefinitions, principalPermissionGrants } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres";
|
||||
import { fetchCompanyQuotaWindows } from "../services/quota-windows.js";
|
||||
import { quotaCredentialRecoveryPath, quotaCredentialRecovery, quotaCredentialHash } from "../services/quota-credential-recovery.js";
|
||||
@@ -2213,6 +2213,33 @@ console.log(JSON.stringify({ type: "result", subtype: "success", result: "hello"
|
||||
const [agent] = await db.select().from(agents).where(eq(agents.id, agentId));
|
||||
expect(agent.runtimeConfig.aiConnection).toBeUndefined();
|
||||
});
|
||||
it("keeps account summaries ordered across sequential scans and repeat migration", async () => {
|
||||
const orderedCompanyId = randomUUID(), userId = "stable-list-owner";
|
||||
await db.insert(companies).values({ id: orderedCompanyId, name: "Stable account list", issuePrefix: `O${orderedCompanyId.slice(0, 5)}` });
|
||||
await db.insert(companyMemberships).values({ companyId: orderedCompanyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });
|
||||
const accounts = [];
|
||||
for (const name of ["First account", "Second account", "Third account"]) {
|
||||
accounts.push(await service.save(orderedCompanyId, userId, { provider: "anthropic", method: "api_key", ownership: "personal", name, apiKey: "fixture-stable-list", agentIds: [], allAgents: false }, "fixture-stable-list"));
|
||||
}
|
||||
const expected = accounts.map(({ connectionId: id, grantId }) => ({ id, grantId }))
|
||||
.sort((a, b) => a.id.localeCompare(b.id) || a.grantId.localeCompare(b.grantId));
|
||||
await db.transaction(async tx => {
|
||||
await tx.execute(sql`SET LOCAL enable_indexscan = off`);
|
||||
await tx.execute(sql`SET LOCAL enable_bitmapscan = off`);
|
||||
await tx.execute(sql`SET LOCAL enable_hashjoin = off`);
|
||||
await tx.execute(sql`SET LOCAL enable_mergejoin = off`);
|
||||
for (const account of [...expected].reverse()) {
|
||||
await tx.update(toolConnections).set({ name: sql`${toolConnections.name}` }).where(eq(toolConnections.id, account.id));
|
||||
await tx.update(connectionGrants).set({ status: sql`${connectionGrants.status}` }).where(eq(connectionGrants.id, account.grantId));
|
||||
}
|
||||
const transactionalService = aiConnectionService(tx as unknown as Db);
|
||||
const before = await transactionalService.list(orderedCompanyId, userId);
|
||||
expect(before.map(({ id, grantId }) => ({ id, grantId }))).toEqual(expected);
|
||||
const migration = await readFile(new URL("../../../packages/db/src/migrations/0276_hard_mandroid.sql", import.meta.url), "utf8");
|
||||
for (const statement of migration.split("--> statement-breakpoint").filter(value => value.trim())) await tx.execute(sql.raw(statement));
|
||||
expect(await transactionalService.list(orderedCompanyId, userId)).toEqual(before);
|
||||
});
|
||||
});
|
||||
it("runs two Claude subscription executions for the same grant at the same time", async () => {
|
||||
// Claude writes no auth file back to the grant, so two runs share no
|
||||
// mutable state and must not wait for each other.
|
||||
|
||||
@@ -90,6 +90,13 @@ const support = await getEmbeddedPostgresTestSupport();
|
||||
grantId = grant!.id;
|
||||
const env = { PAPERCLIP_CLOUD_CONNECTOR_BASE_URL: origin, PAPERCLIP_CLOUD_CONNECTOR_ENVIRONMENT: "test" };
|
||||
const service = githubConnectionEventService(db, { env });
|
||||
const reserved: Awaited<ReturnType<typeof db.$client.reserve>>[] = [];
|
||||
try {
|
||||
for (let slot = 0; slot < 4; slot++) reserved.push(await db.$client.reserve());
|
||||
await Promise.all(reserved.map(connection => connection`SELECT 1`));
|
||||
} finally {
|
||||
for (const connection of reserved) connection.release();
|
||||
}
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: e => errors.push(e) });
|
||||
const selects = vi.spyOn(db, "select"), transactions = vi.spyOn(db, "transaction"), executes = vi.spyOn(db, "execute");
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
import { createCipheriv, createHash, createHmac, diffieHellman, generateKeyPairSync, hkdfSync, randomBytes, verify } from "node:crypto";
|
||||
import { expect, vi } from "vitest";
|
||||
import { createPaperclipCloudConnector, type PaperclipCloudConnectorConfig } from "../../services/paperclip-cloud-connector.js";
|
||||
|
||||
export function createSlackCloudConnectorFixture(instanceId: string, companyId: string, subject: string) {
|
||||
const signing = generateKeyPairSync("ed25519"), sealing = generateKeyPairSync("x25519");
|
||||
const config: PaperclipCloudConnectorConfig = { baseUrl: "http://127.0.0.1:1", instanceId, environment: "test",
|
||||
signPrivateKey: signing.privateKey.export({ format: "jwk" }).d!, sealPrivateKey: sealing.privateKey.export({ format: "jwk" }).d! };
|
||||
const calls: { operation: string; enrollment?: Record<string, unknown> }[] = [];
|
||||
const acknowledged: string[] = [];
|
||||
const events: Record<string, unknown>[] = [];
|
||||
let unavailable = false;
|
||||
let beforeEnrollmentReceipt: (() => Promise<void>) | undefined;
|
||||
function seal(payload: unknown, profile: "github.code" | "slack.events.v1") {
|
||||
const provider = profile === "github.code" ? "github" : "slack";
|
||||
const ephemeral = generateKeyPairSync("x25519");
|
||||
const epk = ephemeral.publicKey.export({ format: "jwk" }).x!;
|
||||
const recipient = sealing.publicKey.export({ format: "jwk" }).x!;
|
||||
const aad = Buffer.from([1, "X25519-HKDF-SHA256-A256GCM", "events", instanceId, "test", provider, profile, ""].join("\n"));
|
||||
const key = Buffer.from(hkdfSync("sha256", diffieHellman({ privateKey: ephemeral.privateKey, publicKey: sealing.publicKey }),
|
||||
Buffer.concat([Buffer.from(epk, "base64url"), Buffer.from(recipient, "base64url")]), aad, 32));
|
||||
const iv = randomBytes(12), cipher = createCipheriv("aes-256-gcm", key, iv); cipher.setAAD(aad);
|
||||
const ct = Buffer.concat([cipher.update(JSON.stringify(payload), "utf8"), cipher.final(), cipher.getAuthTag()]);
|
||||
return { v: 1, alg: "X25519-HKDF-SHA256-A256GCM", purpose: "events", provider, profile, epk,
|
||||
iv: iv.toString("base64url"), ct: ct.toString("base64url") };
|
||||
}
|
||||
const requestErrors: unknown[] = [];
|
||||
const request: typeof fetch = vi.fn(async (input: RequestInfo | URL, init?: RequestInit): Promise<Response> => {
|
||||
try {
|
||||
const url = new URL(input instanceof Request ? input.url : String(input));
|
||||
expect(url.origin).toBe(config.baseUrl);
|
||||
expect(init?.method).toBe("POST");
|
||||
const body = JSON.parse(String(init?.body)) as { request: string; enrollment?: string; acknowledgement?: string };
|
||||
const [header, payload, signature] = body.request.split(".");
|
||||
expect(verify(null, Buffer.from(`${header}.${payload}`), signing.publicKey, Buffer.from(signature!, "base64url"))).toBe(true);
|
||||
const claims = JSON.parse(Buffer.from(payload!, "base64url").toString("utf8")) as Record<string, unknown>;
|
||||
expect(claims).toMatchObject({ iss: instanceId, env: "test", aud: url.toString() });
|
||||
const operation = String(claims.op);
|
||||
if (operation !== "status") {
|
||||
const expectedSubject = operation === "slack-events" ? subject
|
||||
: ["event-lease", "event-ack"].includes(operation) ? "github-bot-events" : undefined;
|
||||
if (!expectedSubject) throw new Error(`Unexpected synthetic connector authority ${operation}`);
|
||||
expect(claims).toMatchObject({ sub: expectedSubject, cid: companyId });
|
||||
}
|
||||
if (unavailable) return Response.json({ error: "fixture_broker_unavailable" }, { status: 503 });
|
||||
if (operation === "status") return Response.json({ active: true, status: "active", profiles: [], slackEvents: { version: 1 } });
|
||||
if (operation === "slack-events") {
|
||||
expect(claims.sh).toBe(createHash("sha256").update(body.enrollment!).digest("base64url"));
|
||||
const enrollment = JSON.parse(body.enrollment!) as Record<string, unknown>;
|
||||
calls.push({ operation, enrollment: { ...enrollment, signingSecret: undefined } });
|
||||
await beforeEnrollmentReceipt?.();
|
||||
const { active, signingSecret: _secret, ...identity } = enrollment;
|
||||
const bindingId = `slack-event:${createHash("sha256").update(JSON.stringify(identity)).digest("hex")}`;
|
||||
return Response.json(active === false ? { retirement: { version: 1, instanceId, ...identity } }
|
||||
: { enrollment: { version: 1, instanceId, ...identity, bindingId } });
|
||||
}
|
||||
calls.push({ operation });
|
||||
if (operation === "event-lease") {
|
||||
if (!events.length) return Response.json({ events: [], eventWakeVersion: 1 });
|
||||
const leased = events.splice(0);
|
||||
return Response.json({ leaseId: "fixture-lease", eventWakeVersion: 1,
|
||||
sealed: seal({ v: 1, instanceId, environment: "test", leaseId: "fixture-lease", events: leased }, "github.code") });
|
||||
}
|
||||
if (operation === "event-ack") {
|
||||
expect(claims.sh).toBe(createHash("sha256").update(body.acknowledgement!).digest("base64url"));
|
||||
const value = JSON.parse(body.acknowledgement!) as { deliveryIds: string[] };
|
||||
acknowledged.push(...value.deliveryIds); return Response.json({ acknowledged: value.deliveryIds.length });
|
||||
}
|
||||
throw new Error(`Unexpected synthetic connector operation ${operation}`);
|
||||
} catch (error) {
|
||||
requestErrors.push(error);
|
||||
console.error("Synthetic Slack connector rejected its request", error);
|
||||
throw error;
|
||||
}
|
||||
});
|
||||
function install() {
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", config.baseUrl);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", instanceId);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_ENVIRONMENT", "test");
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SIGN_PRIVATE_KEY", config.signPrivateKey);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SEAL_PRIVATE_KEY", config.sealPrivateKey);
|
||||
}
|
||||
function enqueue(identity: Record<string, unknown>, signingSecret: string, eventId: string) {
|
||||
const acceptedAt = new Date().toISOString(), timestamp = String(Math.floor(Date.now() / 1000));
|
||||
const body = Buffer.from(JSON.stringify({ type: "event_callback", team_id: identity.teamId, api_app_id: identity.appId, event_id: eventId,
|
||||
event: { type: "app_mention", text: "<@UBOT> fixture callback", user: "UOPERATOR", channel: "CTEST", ts: `${Date.now() / 1000}`, event_ts: `${Date.now() / 1000}` } }));
|
||||
const id = `slack_${String(identity.bindingId).slice("slack-event:".length)}_${createHash("sha256").update(eventId).digest("hex")}`;
|
||||
const packet = { version: 1, instanceId, environment: "test", ...identity, eventId, acceptedAt, rawBody: body.toString("base64"), timestamp,
|
||||
signature: `v0=${createHmac("sha256", signingSecret).update(`v0:${timestamp}:`).update(body).digest("hex")}` };
|
||||
events.push({ id, provider: "slack", event: "event_callback", installationId: identity.bindingId, bindingIds: [identity.bindingId], createdAt: acceptedAt,
|
||||
payload: { slackEvent: { bindingId: identity.bindingId, sealed: seal(packet, "slack.events.v1") } } });
|
||||
return { id, packet, body };
|
||||
}
|
||||
return { config, request, requestErrors, calls, acknowledged, events, install, enqueue, connector: createPaperclipCloudConnector({ config, request }),
|
||||
setUnavailable(value: boolean) { unavailable = value; },
|
||||
beforeReceipt(work: (() => Promise<void>) | undefined) { beforeEnrollmentReceipt = work; } };
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
const key = Symbol.for('paperclip.slack.fixture.prewire');
|
||||
if (!process[key]) {
|
||||
const net = require('node:net');
|
||||
const tls = require('node:tls');
|
||||
const http = require('node:http');
|
||||
const https = require('node:https');
|
||||
const fs = require('node:fs');
|
||||
const loopback = new Set(['127.0.0.1', 'localhost', '::1', '[::1]']);
|
||||
const listeners = new Set();
|
||||
const pgTargets = new Set();
|
||||
const pgPorts = new Set([5432, 55448]);
|
||||
for (const name of ['PAPERCLIP_SLACK_TEST_DATABASE_URL', 'SLACK_EVENT_TEST_DATABASE_URL']) {
|
||||
if (!process.env[name]) continue;
|
||||
let url;
|
||||
try { url = new URL(process.env[name]); } catch { throw new Error('Invalid Slack fixture database target'); }
|
||||
if (!['postgres:', 'postgresql:'].includes(url.protocol) || !loopback.has(url.hostname)
|
||||
|| !/^[a-zA-Z0-9_]+_test$/.test(decodeURIComponent(url.pathname.slice(1)))
|
||||
|| ['options', 'database', 'dbname'].some(k => url.searchParams.has(k))) throw new Error('Invalid Slack fixture database target');
|
||||
const port = Number(url.port || 5432);
|
||||
pgPorts.add(port);
|
||||
pgTargets.add(`${url.hostname}:${port}`);
|
||||
}
|
||||
const sockets = new Set();
|
||||
const observed = new WeakSet();
|
||||
const attempts = [];
|
||||
let peakPgSockets = 0;
|
||||
const record = value => {
|
||||
if (process.env.SLACK_GATE_PROCESS_LOG) fs.appendFileSync(process.env.SLACK_GATE_PROCESS_LOG, JSON.stringify(value) + '\n');
|
||||
};
|
||||
const fail = (host, port) => {
|
||||
attempts.push({ host: String(host), port: Number(port) || null });
|
||||
process.exitCode = 98;
|
||||
throw new Error('Slack fixture blocked an unowned network target before transport');
|
||||
};
|
||||
const check = (host = 'localhost', port, protocol) => {
|
||||
host = String(host).toLowerCase();
|
||||
if (!loopback.has(host)) fail(host, port);
|
||||
if (protocol && !['http:', 'https:'].includes(protocol)) fail(host, port);
|
||||
if (port && pgPorts.has(Number(port)) && !pgTargets.has(`${host}:${Number(port)}`)) fail(host, port);
|
||||
if (port && !pgTargets.has(`${host}:${Number(port)}`) && !listeners.has(Number(port))) fail(host, port);
|
||||
};
|
||||
const checkArgs = args => {
|
||||
const first = args[0];
|
||||
if (first instanceof URL || typeof first === 'string' && /^https?:/.test(first)) {
|
||||
const url = first instanceof URL ? first : new URL(first);
|
||||
check(url.hostname, url.port || (url.protocol === 'https:' ? 443 : 80), url.protocol);
|
||||
} else if (first && typeof first === 'object') {
|
||||
if (first.path && !first.port && !first.host && !first.hostname) fail('unix-socket', 0);
|
||||
check(first.hostname || first.host, first.port, first.protocol);
|
||||
} else if (typeof first === 'number') check(typeof args[1] === 'string' ? args[1] : undefined, first);
|
||||
else fail('unknown-target', 0);
|
||||
};
|
||||
const listen = net.Server.prototype.listen;
|
||||
net.Server.prototype.listen = function (...args) {
|
||||
this.once('listening', () => {
|
||||
const address = this.address();
|
||||
if (address && typeof address === 'object' && loopback.has(address.address)) {
|
||||
listeners.add(address.port);
|
||||
this.once('close', () => listeners.delete(address.port));
|
||||
}
|
||||
});
|
||||
return Reflect.apply(listen, this, args);
|
||||
};
|
||||
for (const module of [http, https]) for (const name of ['request', 'get']) {
|
||||
const original = module[name];
|
||||
module[name] = function (...args) { checkArgs(args); return Reflect.apply(original, this, args); };
|
||||
}
|
||||
const connect = net.Socket.prototype.connect;
|
||||
net.Socket.prototype.connect = function (...args) {
|
||||
const normalized = Array.isArray(args[0]) ? args[0] : args;
|
||||
checkArgs(normalized);
|
||||
const first = normalized[0];
|
||||
const port = typeof first === 'object' ? first.port : first;
|
||||
if (pgPorts.has(Number(port)) && !observed.has(this)) {
|
||||
observed.add(this);
|
||||
this.once('connect', () => { sockets.add(this); peakPgSockets = Math.max(peakPgSockets, sockets.size); });
|
||||
this.once('close', () => sockets.delete(this));
|
||||
}
|
||||
return Reflect.apply(connect, this, args);
|
||||
};
|
||||
const tlsConnect = tls.connect;
|
||||
tls.connect = function (...args) { checkArgs(args); return Reflect.apply(tlsConnect, this, args); };
|
||||
const fetch = globalThis.fetch;
|
||||
if (fetch) globalThis.fetch = function (input, init) {
|
||||
checkArgs([input instanceof Request ? input.url : input]);
|
||||
return fetch(input, init);
|
||||
};
|
||||
const stats = () => ({ pid: process.pid, peakPgSockets, remainingPgSockets: sockets.size, unexpectedAttempts: attempts.slice() });
|
||||
process[key] = Object.freeze({ pid: process.pid, stats });
|
||||
globalThis[key] = process[key];
|
||||
process.env.SLACK_GATE_GUARD_PID = String(process.pid);
|
||||
record({ phase: 'repository-preload', pid: process.pid });
|
||||
process.on('exit', () => record({ phase: 'exit', ...stats(), exitCode: process.exitCode || 0 }));
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
import http from "node:http";
|
||||
import https from "node:https";
|
||||
import { expect, vi } from "vitest";
|
||||
|
||||
export function guardSlackFixtureNetwork(options: { requirePreload?: boolean } = {}) {
|
||||
const prewire = Reflect.get(process, Symbol.for("paperclip.slack.fixture.prewire")) as {
|
||||
pid: number;
|
||||
stats(): { pid: number; peakPgSockets: number; remainingPgSockets: number; unexpectedAttempts: unknown[] };
|
||||
} | undefined;
|
||||
if (options.requirePreload) {
|
||||
expect(prewire?.pid).toBe(process.pid);
|
||||
expect(process.env.SLACK_GATE_GUARD_PID).toBe(String(process.pid));
|
||||
}
|
||||
if (process.env.SLACK_GATE_EXPECT_PRELOAD === "1") {
|
||||
expect(process.env.SLACK_GATE_GUARD_PID).toBe(String(process.pid));
|
||||
console.info(JSON.stringify({ slackFixtureWorkerPid: process.pid, prewireGuardPid: process.env.SLACK_GATE_GUARD_PID }));
|
||||
}
|
||||
const blocked: string[] = [];
|
||||
const check = (value: unknown) => {
|
||||
const url = value instanceof URL ? value : typeof value === "string" ? new URL(value) : undefined;
|
||||
const options = value as { hostname?: string; host?: string } | undefined;
|
||||
const host = url?.hostname ?? options?.hostname ?? options?.host ?? "localhost";
|
||||
if (!["localhost", "127.0.0.1", "::1", "[::1]"].includes(host)) {
|
||||
blocked.push(host);
|
||||
throw new Error("Unexpected off-host Slack fixture request");
|
||||
}
|
||||
};
|
||||
const realHttp = http.request, realHttps = https.request, realFetch = globalThis.fetch;
|
||||
const httpSpy = vi.spyOn(http, "request").mockImplementation(((...args: unknown[]) => {
|
||||
check(args[0]); return Reflect.apply(realHttp, http, args);
|
||||
}) as typeof http.request);
|
||||
const httpsSpy = vi.spyOn(https, "request").mockImplementation(((...args: unknown[]) => {
|
||||
check(args[0]); return Reflect.apply(realHttps, https, args);
|
||||
}) as typeof https.request);
|
||||
const fetchSpy = vi.spyOn(globalThis, "fetch").mockImplementation((input, init) => {
|
||||
check(input instanceof Request ? input.url : input); return realFetch(input, init);
|
||||
});
|
||||
return { assertQuiet: () => {
|
||||
expect(blocked).toEqual([]);
|
||||
if (prewire) expect(prewire.stats().unexpectedAttempts).toEqual([]);
|
||||
const stats = prewire?.stats;
|
||||
if (process.env.SLACK_GATE_EXPECT_PRELOAD === "1") {
|
||||
expect(stats).toBeTypeOf("function");
|
||||
console.info(JSON.stringify({ slackFixtureWorkerPid: process.pid, networkStats: stats!() }));
|
||||
}
|
||||
}, restore: () => {
|
||||
fetchSpy.mockRestore(); httpsSpy.mockRestore(); httpSpy.mockRestore();
|
||||
} };
|
||||
}
|
||||
|
||||
export function stubSlackFixtureTransport(adapter: unknown) {
|
||||
const client = (adapter as { _client: {
|
||||
axios: { defaults: { adapter: unknown } };
|
||||
buildResult(response: unknown): Promise<unknown>;
|
||||
} })._client;
|
||||
const failures: unknown[] = [];
|
||||
const buildResult = client.buildResult.bind(client);
|
||||
client.buildResult = async response => {
|
||||
try { return await buildResult(response); }
|
||||
catch (error) { failures.push(error); throw error; }
|
||||
};
|
||||
const transport = vi.fn(async (config: { url: string; baseURL: string }) => {
|
||||
const url = new URL(config.url, config.baseURL);
|
||||
if (url.hostname !== "127.0.0.1") throw new Error("Slack fixture API origin changed");
|
||||
let data: unknown;
|
||||
if (url.pathname.endsWith("/users.info")) data = { ok: true, user: { id: "UOPERATOR", name: "operator", is_bot: false, profile: { display_name: "Operator" } } };
|
||||
else if (url.pathname.endsWith("/conversations.info")) data = { ok: true, channel: { id: "CTEST", name: "fixture" } };
|
||||
else throw new Error(`Unexpected Slack fixture API method ${url.pathname}`);
|
||||
return { data, status: 200, statusText: "OK", headers: {}, config, request: { path: url.pathname } };
|
||||
});
|
||||
client.axios.defaults.adapter = transport;
|
||||
return Object.assign(transport, { assertNoErrors: async () => {
|
||||
await Promise.all(transport.mock.results.map(result => result.value));
|
||||
expect(failures).toEqual([]);
|
||||
} });
|
||||
}
|
||||
@@ -301,6 +301,66 @@ describe("residual recovery production matrix", () => {
|
||||
}, { timeout: 45_000 });
|
||||
await competing.stop(); extraStops.pop(); await quiet();
|
||||
});
|
||||
it("recovers a fresh run from a prior boot at lease expiry instead of waiting for the stale-age threshold", async () => {
|
||||
// RC-27: a Retry admitted just before a kill records a controller lease but
|
||||
// no process. The prior boot's expired lease is the ownership proof.
|
||||
const f = await fixture(), expiry = new Date(Date.now() + 3_000);
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "running",
|
||||
runtimeMode: "legacy", responsibleUserId: f.actorId, controllerBootId: randomUUID(), controllerLeaseExpiresAt: expiry,
|
||||
executionStage: "preparing", updatedAt: new Date(), startedAt: new Date() }).returning();
|
||||
await start(true);
|
||||
expect((await getRun(run.id)).status).toBe("running");
|
||||
await vi.waitFor(async () => expect((await getRun(run.id)).status).toBe("failed"), { timeout: 15_000 });
|
||||
expect(Date.now() - expiry.getTime()).toBeLessThan(15_000);
|
||||
expect(await getRun(run.id)).toMatchObject({ errorCode: "process_lost",
|
||||
resultJson: expect.objectContaining({ executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }) });
|
||||
await vi.waitFor(async () => {
|
||||
const retries = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run.id));
|
||||
expect(retries).toHaveLength(1);
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
expect((await getRun(retries[0].id)).status).toBe("succeeded");
|
||||
}, { timeout: 45_000 });
|
||||
await quiet();
|
||||
});
|
||||
it("keeps the age guard when the prior boot dispatched between the scan and the revocation", async () => {
|
||||
// Stage the race: the scanned row says `preparing`, and the row committed
|
||||
// with the revocation says `dispatching` (the old controller's last renewal).
|
||||
const f = await fixture(), expiry = new Date(Date.now() + 2_000);
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "running",
|
||||
runtimeMode: "legacy", responsibleUserId: f.actorId, controllerBootId: randomUUID(), controllerLeaseExpiresAt: expiry,
|
||||
executionStage: "preparing", updatedAt: new Date(), startedAt: new Date() }).returning();
|
||||
await db.execute(sql.raw(`create or replace function rc27_dispatch_on_revoke() returns trigger language plpgsql as $$
|
||||
begin if new.id = '${run.id}' and new.controller_boot_id is distinct from old.controller_boot_id
|
||||
then new.execution_stage := 'dispatching'; end if; return new; end $$`));
|
||||
await db.execute(sql`create trigger rc27_dispatch_on_revoke before update on heartbeat_runs
|
||||
for each row execute function rc27_dispatch_on_revoke()`);
|
||||
extraStops.push(async () => {
|
||||
await db.execute(sql`drop trigger if exists rc27_dispatch_on_revoke on heartbeat_runs`);
|
||||
await db.execute(sql`drop function if exists rc27_dispatch_on_revoke()`);
|
||||
});
|
||||
await start(true);
|
||||
await vi.waitFor(async () => expect((await getRun(run.id)).controllerBootId).not.toBe(run.controllerBootId), { timeout: 15_000 });
|
||||
await realDelay(5_000); await Promise.allSettled([...passes]);
|
||||
expect(await getRun(run.id)).toMatchObject({ status: "running", executionStage: "dispatching", errorCode: null });
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run.id))).toEqual([]);
|
||||
});
|
||||
it("keeps the stale-age guard for this boot, a recorded process, or a dispatched adapter", async () => {
|
||||
const f = await fixture(), now = Date.now(), expired = new Date(now - 1_000);
|
||||
const [sameBoot] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "running",
|
||||
runtimeMode: "legacy", responsibleUserId: f.actorId, controllerBootId: legacyControllerBootId, controllerLeaseExpiresAt: expired,
|
||||
executionStage: "preparing", updatedAt: new Date(now), startedAt: new Date(now) }).returning();
|
||||
const [withProcess] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "running",
|
||||
runtimeMode: "legacy", responsibleUserId: f.actorId, controllerBootId: randomUUID(), controllerLeaseExpiresAt: expired,
|
||||
executionStage: "dispatching", processPid: 2 ** 22 + 17, updatedAt: new Date(now), startedAt: new Date(now) }).returning();
|
||||
// A dispatched adapter may hold remote work without a local process.
|
||||
const [dispatched] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, status: "running",
|
||||
runtimeMode: "legacy", responsibleUserId: f.actorId, controllerBootId: randomUUID(), controllerLeaseExpiresAt: expired,
|
||||
executionStage: "dispatching", updatedAt: new Date(now), startedAt: new Date(now) }).returning();
|
||||
await start(); await advance(4 * 60_000);
|
||||
expect((await getRun(sameBoot.id)).status).toBe("running");
|
||||
expect((await getRun(withProcess.id)).status).toBe("running");
|
||||
expect((await getRun(dispatched.id)).status).toBe("running");
|
||||
});
|
||||
it.each(["confirmed", "already_retired", "missing_terminal", "unproven_terminal", "ack_failure", "late_terminal"] as const)(
|
||||
"settles an API-only Stop after deferred controller retirement without inventing proof: %s", async mode => {
|
||||
const f = await fixture(), runId = randomUUID(), sessionId = randomUUID(), runnerId = randomUUID();
|
||||
|
||||
@@ -0,0 +1,433 @@
|
||||
import { createHmac, randomUUID } from "node:crypto";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { SLACK_CHAT_BOT_SCOPES } from "@paperclipai/shared";
|
||||
import { activityLog, agentRuntimeState, agentTaskSessions, agentWakeupRequests, agents, applyPendingMigrations, authUsers,
|
||||
chatDeliveries, chatEndpoints, chatExternalPrincipals, companySecretBindings, chatIdentityLinks, chatSlackRegistrations, companies, companyMemberships,
|
||||
companySecrets, companySkills, costEvents, financeEvents, heartbeatRunEvents, heartbeatRuns, issueComments, issues, toolApplications, toolConnections,
|
||||
closeCheckpointDb, closeFileStorageDb, closeLivenessDb, createDb } from "@paperclipai/db";
|
||||
import { databaseWorkPending, reconcileDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db/work-signals";
|
||||
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { createSlackCloudConnectorFixture } from "./helpers/slack-cloud-connector-fixture.js";
|
||||
import { chatChannelService } from "../services/chat-channels.js";
|
||||
import { toolAccessService } from "../services/tool-access.js";
|
||||
import { secretService } from "../services/secrets.js";
|
||||
import { ChatRuntime } from "../services/chat-runtime.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { githubConnectionEventService } from "../services/github-connection-events.js";
|
||||
import { chatCredentialMutationLease } from "../services/chat-credential-mutation-lease.js";
|
||||
import { createSlackCloudEnrollment, registerSlackCloudEnrollmentOwner, reconcilePendingSlackCloudEnrollments } from "../services/chat-channels/slack/cloud-enrollment.js";
|
||||
import { conflict } from "../errors.js";
|
||||
import { guardSlackFixtureNetwork, stubSlackFixtureTransport } from "./helpers/slack-cloud/transport.js";
|
||||
import type { InternalSetupState } from "../services/chat-channels/runtime-types.js";
|
||||
import type { ChatSdkMessageCallbackEvent } from "../services/chat-sdk-runtime.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_SLACK_OWNER_TEST_DATABASE_URL");
|
||||
if (process.env.SLACK_GATE_EXPECT_PRELOAD === "1" && !external) throw new Error("Guarded local Slack owner tests require their exact external test database");
|
||||
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)("Slack Cloud default owner and SQL fences", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let service: ReturnType<typeof chatChannelService>;
|
||||
let heartbeat: ReturnType<typeof heartbeatService>;
|
||||
let cloud: ReturnType<typeof createSlackCloudConnectorFixture>;
|
||||
let network: ReturnType<typeof guardSlackFixtureNetwork>;
|
||||
let directory: string;
|
||||
let companyId: string, actorId: string, agentId: string, applicationId: string;
|
||||
let runtime: ChatRuntime;
|
||||
let providerFetch: typeof fetch;
|
||||
const callbacks: ChatSdkMessageCallbackEvent[] = [];
|
||||
const transports: ReturnType<typeof stubSlackFixtureTransport>[] = [];
|
||||
const releases: (() => void)[] = [];
|
||||
const heldLeases: Promise<unknown>[] = [];
|
||||
const deferred: (() => void | Promise<void>)[] = [];
|
||||
const secret = "0123456789abcdef0123456789abcdef";
|
||||
const row = async (id: string) => (await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, id)))[0]!;
|
||||
const setup = async (id: string) => (await row(id)).setup as InternalSetupState;
|
||||
beforeAll(async () => {
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("slack-cloud-owner-");
|
||||
const url = external ?? temporary!.connectionString;
|
||||
await applyPendingMigrations(url);
|
||||
db = createDb(url, { maxConnections: 2, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
|
||||
}, 60_000);
|
||||
beforeEach(async () => {
|
||||
callbacks.length = 0; transports.length = 0; releases.length = 0; heldLeases.length = 0; deferred.length = 0;
|
||||
network = guardSlackFixtureNetwork({ requirePreload: process.env.SLACK_GATE_EXPECT_PRELOAD === "1" });
|
||||
directory = await mkdtemp(path.join(os.tmpdir(), "slack-owner-default-"));
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", path.join(directory, "master.key"));
|
||||
companyId = randomUUID(); actorId = `slack-owner-${randomUUID()}`; agentId = randomUUID(); applicationId = randomUUID();
|
||||
cloud = createSlackCloudConnectorFixture(`instance-${randomUUID()}`, companyId, actorId); cloud.install();
|
||||
vi.spyOn(globalThis, "fetch").mockImplementation(cloud.request);
|
||||
const now = new Date();
|
||||
await db.insert(companies).values({ id: companyId, name: "Slack default owner", issuePrefix: `S${companyId.slice(0, 7)}` });
|
||||
await db.insert(authUsers).values({ id: actorId, name: "Slack owner", email: `${actorId}@example.test`, emailVerified: true, createdAt: now, updatedAt: now });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: actorId, status: "active", membershipRole: "operator" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Slack fixture", adapterType: "process", status: "idle",
|
||||
runtimeConfig: { heartbeat: { wakeOnDemand: false, intervalSec: 0 } } });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "agent", principalId: agentId, status: "active", membershipRole: "member" });
|
||||
await db.insert(toolApplications).values({ id: applicationId, companyId, name: "Slack", type: "chat" });
|
||||
runtime = makeRuntime();
|
||||
const slackFetch: typeof fetch = async (input, init) => {
|
||||
const url = new URL(input instanceof Request ? input.url : String(input));
|
||||
if (url.origin !== "https://slack.com") throw new Error(`Unexpected Slack fixture origin ${url.origin}`);
|
||||
if (url.pathname === "/api/auth.test") {
|
||||
expect(init?.method).toBe("POST");
|
||||
const token = new Headers(init?.headers).get("authorization") ?? "";
|
||||
const suffix = token.slice(token.lastIndexOf("-") + 1);
|
||||
return Response.json({ ok: true, team_id: `T${suffix}`, team: "Fixture", user_id: `U${suffix}`, user: "fixture" },
|
||||
{ headers: { "x-oauth-scopes": SLACK_CHAT_BOT_SCOPES.join(",") } });
|
||||
}
|
||||
if (url.pathname === "/api/conversations.list") return Response.json({ ok: true,
|
||||
channels: [{ id: "CTEST", name: "fixture", is_member: true }], response_metadata: { next_cursor: "" } });
|
||||
if (["/api/reactions.add", "/api/reactions.remove"].includes(url.pathname)) return Response.json({ ok: true });
|
||||
throw new Error(`Unexpected Slack fixture method ${url.pathname}`);
|
||||
};
|
||||
providerFetch = slackFetch;
|
||||
heartbeat = heartbeatService(db);
|
||||
service = chatChannelService(db, { runtime, heartbeat, fetch: slackFetch, publicBaseUrl: "https://tenant.example",
|
||||
deferWebhookProcessing: true, scheduleDeferredWork: work => { deferred.push(work); } });
|
||||
});
|
||||
afterEach(async () => {
|
||||
const errors: unknown[] = [];
|
||||
for (const release of releases) release();
|
||||
for (const work of heldLeases) { try { await work; } catch (error) { errors.push(error); } }
|
||||
try { await service?.shutdown(); } catch (error) { errors.push(error); }
|
||||
try { await heartbeat?.drainActiveRunExecutions(); } catch (error) { errors.push(error); }
|
||||
try {
|
||||
await db.delete(chatEndpoints).where(eq(chatEndpoints.companyId, companyId));
|
||||
await db.delete(financeEvents).where(eq(financeEvents.companyId, companyId));
|
||||
await db.delete(costEvents).where(eq(costEvents.companyId, companyId));
|
||||
await db.delete(activityLog).where(eq(activityLog.companyId, companyId));
|
||||
await db.delete(agentTaskSessions).where(eq(agentTaskSessions.companyId, companyId));
|
||||
await db.delete(agentRuntimeState).where(eq(agentRuntimeState.companyId, companyId));
|
||||
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, companyId));
|
||||
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId));
|
||||
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId));
|
||||
await db.delete(issueComments).where(eq(issueComments.companyId, companyId));
|
||||
await db.delete(issues).where(eq(issues.companyId, companyId));
|
||||
await db.delete(agents).where(eq(agents.companyId, companyId));
|
||||
await db.delete(companyMemberships).where(eq(companyMemberships.companyId, companyId));
|
||||
await db.delete(companySecrets).where(eq(companySecrets.companyId, companyId));
|
||||
await db.delete(companySkills).where(eq(companySkills.companyId, companyId));
|
||||
await db.delete(companies).where(eq(companies.id, companyId));
|
||||
await db.delete(authUsers).where(eq(authUsers.id, actorId));
|
||||
} catch (error) { errors.push(error); }
|
||||
try { network.assertQuiet(); } catch (error) { errors.push(error); }
|
||||
network.restore(); vi.restoreAllMocks(); vi.unstubAllEnvs();
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
if (errors.length) throw new AggregateError(errors, "Slack owner fixture cleanup failed; original body failure is reported separately by Vitest");
|
||||
});
|
||||
afterAll(async () => {
|
||||
const closed = await Promise.allSettled([closeCheckpointDb(db, 1), closeFileStorageDb(db, 1), closeLivenessDb(db, 1)]);
|
||||
await db?.$client.end({ timeout: 1 }); await temporary?.cleanup();
|
||||
const errors = closed.flatMap(value => value.status === "rejected" ? [value.reason] : []);
|
||||
if (errors.length) throw new AggregateError(errors, "Slack owner auxiliary shutdown failed");
|
||||
});
|
||||
function makeRuntime() {
|
||||
const runtime = new ChatRuntime();
|
||||
const replace = runtime.replaceEndpoint.bind(runtime);
|
||||
vi.spyOn(runtime, "replaceEndpoint").mockImplementation(async input => {
|
||||
if (input.providerConfig.provider !== "slack") throw new Error("Unexpected fixture provider");
|
||||
const value = await replace({ ...input, providerConfig: { ...input.providerConfig,
|
||||
credentials: { ...input.providerConfig.credentials, apiUrl: "http://127.0.0.1:1/api/" } },
|
||||
callbacks: { ...input.callbacks, onMessage: async event => { callbacks.push(event); await input.callbacks.onMessage(event); } } });
|
||||
transports.push(stubSlackFixtureTransport(value.getProviderAdapter())); return value;
|
||||
});
|
||||
return runtime;
|
||||
}
|
||||
async function createEndpoint(suffix: string) {
|
||||
const endpointId = randomUUID(), connectionId = randomUUID(), publicId = randomUUID().replaceAll("-", "").padEnd(43, "a");
|
||||
await db.insert(toolConnections).values({ id: connectionId, companyId, applicationId, name: `Slack ${suffix}`, uid: endpointId,
|
||||
transport: "chat_sdk", connectionPurpose: "channel", enabled: true, status: "active" });
|
||||
await db.insert(chatEndpoints).values({ id: endpointId, companyId, connectionId, assignedAgentId: agentId, publicId,
|
||||
provider: "slack", status: "draft", sponsorUserId: actorId, allowGroupChats: true, setup: { step: "provider_setup" } });
|
||||
await db.insert(chatSlackRegistrations).values({ companyId, endpointId, requestId: randomUUID(), status: "configured", appId: `A${suffix}`,
|
||||
manifest: {}, manifestHash: "fixture", callbackUri: `https://tenant.example/api/chat-webhooks/${publicId}/slack`, createdByUserId: actorId });
|
||||
await service.configure(endpointId, { action: "configure", credentials: { botToken: `xoxb-fixture-${suffix}`, signingSecret: secret } }, actorId);
|
||||
expect((await setup(endpointId)).slackCloudIntent).toMatchObject({ companyId, endpointId, connectionId,
|
||||
instanceId: cloud.config.instanceId, subject: actorId, active: true });
|
||||
return endpointId;
|
||||
}
|
||||
async function confirm(endpointId: string) {
|
||||
const owner = githubConnectionEventService(db, { connector: cloud.connector });
|
||||
await owner.pollOnce({ forDelivery: true });
|
||||
expect((await setup(endpointId)).slackCloudIntent).toBeUndefined();
|
||||
expect((await setup(endpointId)).slackCloudEvents).toBeDefined();
|
||||
return owner;
|
||||
}
|
||||
async function activate(endpointId: string) {
|
||||
const endpoint = await row(endpointId);
|
||||
const accountId = endpoint.providerAccountId;
|
||||
if (!accountId) throw new Error("Normal Slack configure did not confirm its workspace");
|
||||
const payload = JSON.stringify({ type: "url_verification", team_id: accountId, api_app_id: `A${accountId.slice(1)}`, challenge: "fixture-challenge" });
|
||||
const timestamp = String(Math.floor(Date.now() / 1000));
|
||||
const response = await service.handleWebhook(endpoint.publicId, "slack", new Request(`https://tenant.example/api/chat-webhooks/${endpoint.publicId}/slack`, {
|
||||
method: "POST", headers: { "content-type": "application/json", "x-slack-request-timestamp": timestamp,
|
||||
"x-slack-signature": `v0=${createHmac("sha256", secret).update(`v0:${timestamp}:${payload}`).digest("hex")}` }, body: payload }));
|
||||
expect(response.status).toBe(200); await response.body?.cancel();
|
||||
await service.configure(endpointId, { action: "verify" }, actorId);
|
||||
const [principal] = await db.insert(chatExternalPrincipals).values({ companyId, provider: "slack", providerAccountId: accountId,
|
||||
externalId: "UOPERATOR", kind: "user", displayName: "Operator" }).returning();
|
||||
await db.insert(chatIdentityLinks).values({ companyId, endpointId, principalId: principal!.id, paperclipUserId: actorId, status: "linked", confirmedAt: new Date() });
|
||||
expect((await service.test(endpointId, { optionalSlackTestForUser: actorId })).status).toBe("active");
|
||||
}
|
||||
async function holdCredentials(endpointId: string) {
|
||||
let acquired!: () => void;
|
||||
const ready = new Promise<void>(resolve => { acquired = resolve; });
|
||||
const release = new Promise<void>(resolve => { releases.push(resolve); });
|
||||
heldLeases.push(chatCredentialMutationLease(db)(await row(endpointId), async lease => { await lease.assertOwned(); acquired(); await release; }));
|
||||
await ready;
|
||||
}
|
||||
|
||||
it("normal configure stages SQL intent, worker unseals into registered SDK and old generations cannot reenter", async () => {
|
||||
const endpointId = await createEndpoint("BOT");
|
||||
await confirm(endpointId); await activate(endpointId); await confirm(endpointId);
|
||||
const endpoint = await row(endpointId), saved = await setup(endpointId), enrolled = saved.slackCloudEvents!;
|
||||
const callbackEvidence = structuredClone(saved.slackCallbackSurfaces);
|
||||
const delivered = cloud.enqueue({ companyId, endpointId, publicId: endpoint.publicId, connectionId: endpoint.connectionId,
|
||||
runtimeGeneration: saved.runtimeGeneration, teamId: enrolled.teamId, appId: enrolled.appId, bindingId: enrolled.bindingId }, secret, "EvDefaultOwner");
|
||||
const owner = githubConnectionEventService(db, { connector: cloud.connector });
|
||||
expect((await owner.pollOnce({ forDelivery: true })).processed).toBe(1);
|
||||
expect(callbacks).toHaveLength(1);
|
||||
expect(callbacks[0]!.message.author.userId).toBe("UOPERATOR");
|
||||
expect(cloud.acknowledged).toEqual([delivered.id]);
|
||||
expect((await setup(endpointId)).slackCallbackSurfaces).toEqual(callbackEvidence);
|
||||
expect(await db.select().from(chatDeliveries).where(eq(chatDeliveries.endpointId, endpointId))).toHaveLength(1);
|
||||
for (const transport of transports) await transport.assertNoErrors();
|
||||
await service.configure(endpointId, { action: "pause" }, actorId);
|
||||
const oldRequest = new Request("https://tenant.example/old-receipt", { method: "POST", body: new Uint8Array(delivered.body),
|
||||
headers: { "content-type": "application/json", "x-slack-request-timestamp": delivered.packet.timestamp, "x-slack-signature": delivered.packet.signature } });
|
||||
const rejected = await service.handleWebhook(endpoint.publicId, "slack", oldRequest, { slackCloudGeneration: Number(saved.runtimeGeneration) });
|
||||
expect(await rejected.text()).toBe("superseded");
|
||||
expect(callbacks).toHaveLength(1);
|
||||
await owner.pollOnce({ forDelivery: true });
|
||||
expect((await setup(endpointId)).slackCloudIntent).toBeUndefined();
|
||||
expect((await setup(endpointId)).slackCloudRetiredGeneration).toBeGreaterThan(Number(saved.runtimeGeneration));
|
||||
}, 60_000);
|
||||
|
||||
it("two due registrations settle B while A still owns a busy credential lease", async () => {
|
||||
const a = await createEndpoint("BUSY"); await confirm(a); await activate(a); await confirm(a);
|
||||
const b = await createEndpoint("READY"); await confirm(b); await activate(b); await confirm(b);
|
||||
await service.configure(a, { action: "pause" }, actorId);
|
||||
const before = structuredClone((await setup(a)).slackCloudIntent!);
|
||||
await service.configure(b, { action: "pause" }, actorId);
|
||||
await holdCredentials(a);
|
||||
const start = cloud.calls.length;
|
||||
const owner = githubConnectionEventService(db, { connector: cloud.connector });
|
||||
await owner.pollOnce({ forDelivery: true });
|
||||
await owner.pollOnce({ forDelivery: true });
|
||||
expect((await setup(b)).slackCloudIntent).toBeUndefined();
|
||||
expect((await setup(b)).slackCloudRetiredGeneration).toBeDefined();
|
||||
const retained = (await setup(a)).slackCloudIntent!;
|
||||
expect(retained).toMatchObject({ ...before, attempts: 1, failureCode: "credential_lease_busy", nextAttemptAt: expect.any(String) });
|
||||
expect(Date.parse(retained.nextAttemptAt!)).toBeGreaterThan(Date.now());
|
||||
expect(cloud.calls.slice(start).filter(call => call.enrollment?.endpointId === a)).toEqual([]);
|
||||
expect(owner.nextPollAt()).not.toBeNull();
|
||||
releases.splice(0).forEach(release => release()); await Promise.all(heldLeases);
|
||||
const delay = Math.max(0, Date.parse(retained.nextAttemptAt!) - Date.now());
|
||||
await new Promise(resolve => setTimeout(resolve, delay));
|
||||
await owner.pollOnce();
|
||||
expect((await setup(a)).slackCloudIntent).toBeUndefined();
|
||||
expect((await setup(a)).slackCloudRetiredGeneration).toBeDefined();
|
||||
}, 60_000);
|
||||
|
||||
it("normal configuration rollback leaves no registration debt or retirement request", async () => {
|
||||
const endpointId = await createEndpoint("ROLLBACK"); await confirm(endpointId); await activate(endpointId); await confirm(endpointId);
|
||||
const before = await row(endpointId), requests = cloud.calls.length;
|
||||
await service.shutdown(); runtime = makeRuntime();
|
||||
let rejectAfterIntent = false;
|
||||
const stop = subscribeDatabaseWork(db, DELIVERY_QUEUES.connectionEvents, event => { if (event === "intent") rejectAfterIntent = true; });
|
||||
service = chatChannelService(db, { runtime, heartbeat, fetch: providerFetch, publicBaseUrl: "https://tenant.example",
|
||||
deferWebhookProcessing: true, scheduleDeferredWork: work => { deferred.push(work); }, renewCredentialMutationLease: async () => !rejectAfterIntent });
|
||||
try {
|
||||
await expect(service.configure(endpointId, { action: "pause" }, actorId)).rejects.toMatchObject({ code: "CHAT_CREDENTIAL_LEASE_LOST" });
|
||||
expect((await row(endpointId)).status).toBe(before.status);
|
||||
expect(await setup(endpointId)).toEqual(before.setup);
|
||||
expect(cloud.calls).toHaveLength(requests);
|
||||
} finally { stop(); }
|
||||
const owner = githubConnectionEventService(db, { connector: cloud.connector });
|
||||
await owner.pollOnce({ forDelivery: true });
|
||||
expect(cloud.calls.filter(call => call.enrollment?.active === false)).toEqual([]);
|
||||
}, 60_000);
|
||||
|
||||
it.each(["intent", "generation"] as const)("real SQL refuses an enrollment response after newer %s custody wins", async change => {
|
||||
const endpointId = await createEndpoint(`CAS${change}`);
|
||||
const original = (await setup(endpointId)).slackCloudIntent!;
|
||||
const successor = { ...original, id: randomUUID(), runtimeGeneration: original.runtimeGeneration + (change === "generation" ? 1 : 0) };
|
||||
cloud.beforeReceipt(async () => {
|
||||
await db.update(chatEndpoints).set({ setup: sql`jsonb_set(jsonb_set(${chatEndpoints.setup}, '{slackCloudIntent}', ${JSON.stringify(successor)}::jsonb), '{runtimeGeneration}', ${JSON.stringify(successor.runtimeGeneration)}::jsonb)` })
|
||||
.where(and(eq(chatEndpoints.id, endpointId), eq(chatEndpoints.companyId, companyId)));
|
||||
});
|
||||
const producer = createSlackCloudEnrollment(db, { resolveCredentials: async () => ({ signingSecret: secret }),
|
||||
runtimeGeneration: value => Number(value.runtimeGeneration), connector: cloud.connector });
|
||||
await expect(chatCredentialMutationLease(db)(await row(endpointId), lease => producer(endpointId, undefined, lease))).rejects.toThrow("changed");
|
||||
expect((await setup(endpointId)).slackCloudIntent).toEqual(successor);
|
||||
expect((await setup(endpointId)).slackCloudEvents).toBeUndefined();
|
||||
}, 60_000);
|
||||
|
||||
it("malformed sealed Slack receipt is acknowledged once instead of retained as an owner outage", async () => {
|
||||
const endpointId = await createEndpoint("INVALID"); await confirm(endpointId);
|
||||
const endpoint = await row(endpointId), saved = await setup(endpointId), enrolled = saved.slackCloudEvents!;
|
||||
const malformed = cloud.enqueue({ companyId, endpointId: "bad", publicId: endpoint.publicId, connectionId: endpoint.connectionId,
|
||||
runtimeGeneration: saved.runtimeGeneration, teamId: enrolled.teamId, appId: enrolled.appId, bindingId: enrolled.bindingId }, secret, "EvMalformed");
|
||||
const owner = githubConnectionEventService(db, { connector: cloud.connector });
|
||||
const result = await owner.pollOnce({ forDelivery: true });
|
||||
expect(result.ignored).toBe(1); expect(result.failed).toBe(0);
|
||||
expect(cloud.acknowledged).toEqual([malformed.id]);
|
||||
expect(callbacks).toEqual([]);
|
||||
}, 60_000);
|
||||
|
||||
it.each(["intent", "generation", "malformed-generation"] as const)(
|
||||
"busy retry CAS preserves a concurrent %s winner and rethrows the exact owner error", async change => {
|
||||
const endpointId = await createEndpoint(`BUSYCAS${change}`);
|
||||
const before = await setup(endpointId), original = before.slackCloudIntent!;
|
||||
const successor = change === "intent" ? { ...original, id: randomUUID() } : original;
|
||||
const generation: unknown = change === "malformed-generation" ? { invalid: true }
|
||||
: Number(before.runtimeGeneration) + (change === "generation" ? 1 : 0);
|
||||
const busy = conflict("Synthetic owner credential lane is busy", { code: "chat_endpoint_credentials_busy" });
|
||||
const requests = cloud.calls.length;
|
||||
const pendingBefore = databaseWorkPending(db, DELIVERY_QUEUES.connectionEvents), inheritedSignals: string[] = [];
|
||||
const stopInherited = subscribeDatabaseWork(db, DELIVERY_QUEUES.connectionEvents, event => { inheritedSignals.push(event); });
|
||||
try {
|
||||
console.info(JSON.stringify({ slackRetrySignalProbe: "subscription", endpointId, pendingBefore, immediateSignals: [...inheritedSignals] }));
|
||||
expect(inheritedSignals).toEqual(pendingBefore ? ["intent"] : []);
|
||||
await reconcileDatabaseWork(db, DELIVERY_QUEUES.connectionEvents);
|
||||
const pendingAfter = databaseWorkPending(db, DELIVERY_QUEUES.connectionEvents);
|
||||
console.info(JSON.stringify({ slackRetrySignalProbe: "reconciled", endpointId, pendingBefore, pendingAfter, inheritedSignals: [...inheritedSignals] }));
|
||||
expect(pendingAfter).toBe(false);
|
||||
} finally { stopInherited(); }
|
||||
const signals: string[] = [];
|
||||
const stopSignals = subscribeDatabaseWork(db, DELIVERY_QUEUES.connectionEvents, event => { signals.push(event); });
|
||||
const unregister = registerSlackCloudEnrollmentOwner(db, async id => {
|
||||
expect(id).toBe(endpointId);
|
||||
await db.update(chatEndpoints).set({ setup: sql`jsonb_set(jsonb_set(${chatEndpoints.setup}, '{slackCloudIntent}', ${JSON.stringify(successor)}::jsonb), '{runtimeGeneration}', ${JSON.stringify(generation)}::jsonb)` })
|
||||
.where(and(eq(chatEndpoints.id, endpointId), eq(chatEndpoints.companyId, companyId)));
|
||||
throw busy;
|
||||
});
|
||||
try {
|
||||
await expect(reconcilePendingSlackCloudEnrollments(db, new Date())).rejects.toBe(busy);
|
||||
const after = await setup(endpointId);
|
||||
expect(after.slackCloudIntent).toEqual(successor);
|
||||
expect(after.runtimeGeneration).toEqual(generation);
|
||||
expect(after.slackCloudEvents).toBeUndefined();
|
||||
expect(cloud.calls).toHaveLength(requests);
|
||||
expect(signals).toEqual([]);
|
||||
} finally { unregister(); stopSignals(); }
|
||||
}, 60_000);
|
||||
|
||||
it.each(["unknown-error", "unrelated-conflict"] as const)(
|
||||
"the %s owner failure propagates without a busy retry rewrite", async kind => {
|
||||
const endpointId = await createEndpoint(`UNKNOWN${kind}`), before = await setup(endpointId);
|
||||
const original = kind === "unknown-error" ? new Error("Synthetic unknown owner failure")
|
||||
: conflict("Synthetic different authority failure", { code: "chat_endpoint_runtime_superseded" });
|
||||
const requests = cloud.calls.length;
|
||||
const unregister = registerSlackCloudEnrollmentOwner(db, async id => { expect(id).toBe(endpointId); throw original; });
|
||||
try {
|
||||
await expect(reconcilePendingSlackCloudEnrollments(db, new Date())).rejects.toBe(original);
|
||||
expect(await setup(endpointId)).toEqual(before);
|
||||
expect(cloud.calls).toHaveLength(requests);
|
||||
} finally { unregister(); }
|
||||
}, 60_000);
|
||||
|
||||
it("an injected retry transaction failure preserves the original busy error and persistence cause", async () => {
|
||||
const endpointId = await createEndpoint("RETRYWRITE"), before = await setup(endpointId);
|
||||
const busy = conflict("Synthetic owner credential lane is busy", { code: "chat_endpoint_credentials_busy" });
|
||||
const writeError = new Error("Synthetic retry transaction unavailable"), requests = cloud.calls.length;
|
||||
const unregister = registerSlackCloudEnrollmentOwner(db, async id => { expect(id).toBe(endpointId); throw busy; });
|
||||
const transaction = vi.spyOn(db, "transaction").mockRejectedValueOnce(writeError);
|
||||
try {
|
||||
const error = await reconcilePendingSlackCloudEnrollments(db, new Date()).then(() => undefined, error => error);
|
||||
expect(error).toBeInstanceOf(AggregateError);
|
||||
if (!(error instanceof AggregateError)) throw new Error("Expected retry failure to retain both causes", { cause: error });
|
||||
expect(error.errors).toHaveLength(2);
|
||||
expect(error.errors[0]).toBe(busy);
|
||||
expect(error.errors[1]).toBe(writeError);
|
||||
expect(transaction).toHaveBeenCalledOnce();
|
||||
expect(await setup(endpointId)).toEqual(before);
|
||||
expect(cloud.calls).toHaveLength(requests);
|
||||
} finally { transaction.mockRestore(); unregister(); }
|
||||
}, 60_000);
|
||||
|
||||
it("public tool removal retains Slack retirement custody through broker outage and ordinary owner recovery", async () => {
|
||||
const endpointId = await createEndpoint("TOOLREMOVE"); await confirm(endpointId); await activate(endpointId); await confirm(endpointId);
|
||||
const originalEndpoint = await row(endpointId), originalSetup = await setup(endpointId);
|
||||
const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.id, originalEndpoint.connectionId));
|
||||
const signingRef = connection!.credentialSecretRefs.find(ref => ref.configPath === "credentials.signingSecret");
|
||||
if (!signingRef) throw new Error("Normal Slack configure did not persist its signing credential binding");
|
||||
const credentialOwner = { consumerType: "tool_connection" as const, consumerId: connection!.id,
|
||||
configPath: "credentials.signingSecret", actorType: "system" as const };
|
||||
await expect(secretService(db).resolveSecretValue(companyId, signingRef.secretId, "latest", credentialOwner)).resolves.toBe(secret);
|
||||
cloud.setUnavailable(true);
|
||||
const requests = vi.mocked(cloud.request).mock.calls.length;
|
||||
const tools = toolAccessService(db);
|
||||
const removed = await tools.archiveConnection(connection!.id, companyId, { actorType: "user", actorId });
|
||||
expect(removed.connection).toMatchObject({ id: connection!.id, status: "archived", enabled: false });
|
||||
const archived = await row(endpointId);
|
||||
expect(archived.status).toBe("archived");
|
||||
const [revoked] = await db.select().from(toolConnections).where(eq(toolConnections.id, connection!.id));
|
||||
expect(revoked!.credentialSecretRefs).toEqual([]);
|
||||
expect(revoked!.credentialRefs).toEqual([]);
|
||||
expect(await db.select().from(companySecretBindings).where(and(eq(companySecretBindings.companyId, companyId),
|
||||
eq(companySecretBindings.targetType, "tool_connection"), eq(companySecretBindings.targetId, connection!.id)))).toEqual([]);
|
||||
await expect(secretService(db).resolveSecretValue(companyId, signingRef.secretId, "latest", credentialOwner)).rejects.toThrow();
|
||||
expect(vi.mocked(cloud.request).mock.calls).toHaveLength(requests);
|
||||
const intent = (await setup(endpointId)).slackCloudIntent;
|
||||
expect(intent).toMatchObject({ version: 1, active: false, companyId, endpointId, connectionId: connection!.id,
|
||||
publicId: originalEndpoint.publicId, runtimeGeneration: Number(originalSetup.runtimeGeneration),
|
||||
subject: originalSetup.slackCloudSubject, instanceId: originalSetup.slackCloudInstanceId, attempts: 0 });
|
||||
expect(JSON.stringify(intent)).not.toContain(secret);
|
||||
await tools.archiveConnection(connection!.id, companyId, { actorType: "user", actorId });
|
||||
expect((await setup(endpointId)).slackCloudIntent).toEqual(intent);
|
||||
const owner = githubConnectionEventService(db, { connector: cloud.connector });
|
||||
await expect(owner.pollOnce({ forDelivery: true })).rejects.toMatchObject({ status: 503 });
|
||||
const retry = (await setup(endpointId)).slackCloudIntent!;
|
||||
expect(retry).toMatchObject({ ...intent, attempts: 1, failureCode: "broker_unavailable", nextAttemptAt: expect.any(String) });
|
||||
const retryAt = Date.parse(retry.nextAttemptAt!);
|
||||
expect(retryAt).toBeGreaterThan(Date.now());
|
||||
cloud.setUnavailable(false); cloud.install();
|
||||
await new Promise(resolve => setTimeout(resolve, Math.max(0, retryAt - Date.now())));
|
||||
await owner.pollOnce();
|
||||
expect((await setup(endpointId)).slackCloudIntent).toBeUndefined();
|
||||
expect((await setup(endpointId)).slackCloudRetiredGeneration).toBe(Number(originalSetup.runtimeGeneration));
|
||||
expect(cloud.calls.filter(call => call.operation === "slack-events" && call.enrollment?.active === false
|
||||
&& call.enrollment.endpointId === endpointId)).toHaveLength(1);
|
||||
expect(callbacks).toEqual([]);
|
||||
}, 60_000);
|
||||
|
||||
it("public tool removal rolls back both archive writes without dropping current Slack custody or credentials", async () => {
|
||||
const endpointId = await createEndpoint("TOOLROLLBACK"); await confirm(endpointId); await activate(endpointId); await confirm(endpointId);
|
||||
const before = await row(endpointId);
|
||||
const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.id, before.connectionId));
|
||||
const failure = new Error("Synthetic removal transaction rollback"), requests = vi.mocked(cloud.request).mock.calls.length;
|
||||
const realTransaction = db.transaction.bind(db);
|
||||
const transaction = vi.spyOn(db, "transaction").mockImplementationOnce(async work => realTransaction(async tx => {
|
||||
await work(tx);
|
||||
const [archivedEndpoint] = await tx.select().from(chatEndpoints).where(eq(chatEndpoints.id, endpointId));
|
||||
const [archivedConnection] = await tx.select().from(toolConnections).where(eq(toolConnections.id, connection!.id));
|
||||
expect(archivedEndpoint!.status).toBe("archived");
|
||||
expect(archivedConnection).toMatchObject({ status: "archived", enabled: false });
|
||||
throw failure;
|
||||
}));
|
||||
try {
|
||||
await expect(toolAccessService(db).archiveConnection(connection!.id, companyId, { actorType: "user", actorId })).rejects.toBe(failure);
|
||||
expect(await row(endpointId)).toMatchObject({ status: before.status, setup: before.setup });
|
||||
const [after] = await db.select().from(toolConnections).where(eq(toolConnections.id, connection!.id));
|
||||
expect(after).toMatchObject({ status: connection!.status, enabled: connection!.enabled,
|
||||
credentialSecretRefs: connection!.credentialSecretRefs, credentialRefs: connection!.credentialRefs });
|
||||
expect(vi.mocked(cloud.request).mock.calls).toHaveLength(requests);
|
||||
} finally { transaction.mockRestore(); }
|
||||
}, 60_000);
|
||||
|
||||
it("public tool removal refuses a foreign company before Slack archival or retirement staging", async () => {
|
||||
const endpointId = await createEndpoint("TOOLFOREIGN"); await confirm(endpointId);
|
||||
const before = await row(endpointId), requests = vi.mocked(cloud.request).mock.calls.length;
|
||||
await expect(toolAccessService(db).archiveConnection(before.connectionId, randomUUID(), { actorType: "user", actorId })).rejects.toMatchObject({ status: 404 });
|
||||
expect(await row(endpointId)).toEqual(before);
|
||||
expect(vi.mocked(cloud.request).mock.calls).toHaveLength(requests);
|
||||
}, 60_000);
|
||||
});
|
||||
@@ -380,23 +380,28 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
|
||||
// Re-register with an existing durable deadline inside the coalescing
|
||||
// window. A company hint must not make that real deadline wait one second.
|
||||
await coordinator!.stop(); coordinator = undefined;
|
||||
let deadlineWakeAt: number | undefined;
|
||||
const originalWakeup = wakeup.getMockImplementation()!;
|
||||
wakeup.mockImplementation(async (...args) => {
|
||||
deadlineWakeAt ??= Date.now();
|
||||
return originalWakeup(...args);
|
||||
});
|
||||
const dueAt = Date.now() + 500;
|
||||
const actualWakeup = wakeup.getMockImplementation()!;
|
||||
let otherWakeEnteredAt: number | null = null;
|
||||
wakeup.mockImplementation((id, options) => {
|
||||
// Capture dispatch ENTRY before the original mock performs any DB I/O.
|
||||
if (id === other.agentId) otherWakeEnteredAt ??= Date.now();
|
||||
return actualWakeup(id, options);
|
||||
});
|
||||
await db.update(statusCards).set({ nextEvalAt: new Date(dueAt), nextEvalReason: "refresh" }).where(eq(statusCards.id, other.id));
|
||||
await start(service); await flush();
|
||||
await issueService(db).addComment(otherIssue.id, "Another company's coalesced human update", { userId: "board-user" });
|
||||
await issueService(db).addComment((await db.select({ id: issues.id }).from(issues).where(eq(issues.companyId, originalCompany)))[0]!.id,
|
||||
"More changes while another company is due", { userId: "board-user" });
|
||||
await flush();
|
||||
await vi.advanceTimersByTimeAsync(Math.max(0, dueAt - Date.now())); await flush();
|
||||
const armedBeforeAdvance = coordinator!.nextWakeAt();
|
||||
// A null deadline can mean the due work is already running. Any still
|
||||
// armed deadline must preserve the known earlier obligation.
|
||||
if (armedBeforeAdvance !== null) expect(armedBeforeAdvance).toBeLessThanOrEqual(dueAt);
|
||||
await vi.advanceTimersByTimeAsync(500); await flush();
|
||||
await vi.waitFor(() => expect(wakeup).toHaveBeenCalledOnce());
|
||||
expect(deadlineWakeAt).toBeGreaterThanOrEqual(dueAt);
|
||||
expect(deadlineWakeAt).toBeLessThan(dueAt + 500);
|
||||
expect(otherWakeEnteredAt).not.toBeNull();
|
||||
expect(otherWakeEnteredAt!).toBeLessThan(dueAt + 500);
|
||||
expect(wakeup.mock.calls[0]![0]).not.toBe(originalAgent);
|
||||
const generated = (await statusCardService(db).getById(other.id))!;
|
||||
expect(generated.generatingIssueId).toBeTruthy();
|
||||
|
||||
@@ -174,7 +174,7 @@ export function aiConnectionService(db: Db) {
|
||||
: ("connected" as const),
|
||||
},
|
||||
];
|
||||
});
|
||||
}).sort((a, b) => a.id.localeCompare(b.id) || a.grantId.localeCompare(b.grantId));
|
||||
}
|
||||
async function setDefault(
|
||||
companyId: string,
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
import { githubReviewCheckService } from "./chat-github-checks.js";
|
||||
import { createSlackCloudEventIngress } from "./chat-channels/slack/cloud-ingress.js";
|
||||
import { createSlackCloudEnrollment, stageSlackCloudEnrollmentIntent, registerSlackCloudEnrollmentOwner } from "./chat-channels/slack/cloud-enrollment.js";
|
||||
import { leasedChatProvider } from "./chat-channels/runtime/receiver-policy.js";
|
||||
import { chatReceiverSchedule } from "./chat-channels/runtime/receiver-schedule.js";
|
||||
import { registerChatReceiverWork } from "./chat-channels/runtime/receiver-work.js";
|
||||
@@ -842,6 +844,10 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
||||
}
|
||||
|
||||
const withCredentialMutationLease = chatCredentialMutationLease(db, options);
|
||||
const unregisterSlackCloudIngress = createSlackCloudEventIngress(db, {
|
||||
withCredentialMutationLease, resolveCredentials, runtimeGeneration: endpoint => runtimeGeneration(endpoint.setup),
|
||||
handleWebhook: (publicId, provider, request, generation) => handleWebhook(publicId, provider, request, { slackCloudGeneration: generation }),
|
||||
});
|
||||
const { list, get, create, update } = createChatEndpoints(db, {
|
||||
endpointRecord, CAPABILITIES, PROVIDER_LABELS, withCredentialMutationLease,
|
||||
getPublicBaseUrl, getWebhookPublicBaseUrl,
|
||||
@@ -2936,10 +2942,11 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
||||
githubIngressActionId: string;
|
||||
githubIngressFence: LifecycleRuntimeFence;
|
||||
superseded?: boolean;
|
||||
},
|
||||
} | { slackCloudGeneration: number },
|
||||
) {
|
||||
const serviceReceivedAtMs = Date.now();
|
||||
const githubScope = createGitHubWebhookScope(provider, internalContext, options.githubWebhookResponseBudgetMs, GITHUB_WEBHOOK_RESPONSE_BUDGET_MS);
|
||||
const slackCloudGeneration = internalContext && "slackCloudGeneration" in internalContext ? internalContext.slackCloudGeneration : undefined;
|
||||
const githubScope = createGitHubWebhookScope(provider, internalContext && "githubIngressActionId" in internalContext ? internalContext : undefined, options.githubWebhookResponseBudgetMs, GITHUB_WEBHOOK_RESPONSE_BUDGET_MS);
|
||||
const { replayingDurableGitHubIngress, ignoreSupersededIngress, matchesGitHubIngressFence, githubResponseDeadlineAt } = githubScope;
|
||||
const endpoint = await db
|
||||
.select()
|
||||
@@ -2962,6 +2969,8 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
||||
throw notFound("Chat endpoint not found");
|
||||
}
|
||||
recordChatWebhookStage("endpoint_resolved", endpoint.id);
|
||||
if (provider === "slack" && slackCloudGeneration !== undefined && runtimeGeneration(endpoint.setup) !== slackCloudGeneration)
|
||||
return new Response("superseded", { status: 200 });
|
||||
// A pause is a durable ingress fence. Providers generally retry non-2xx
|
||||
// webhooks, so acknowledge late callbacks without recreating a runtime or
|
||||
// admitting any Paperclip mutation.
|
||||
@@ -2998,6 +3007,8 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
||||
code: "chat_endpoint_runtime_superseded",
|
||||
});
|
||||
}
|
||||
if (provider === "slack" && slackCloudGeneration !== undefined && runtimeContext.generation !== slackCloudGeneration)
|
||||
return new Response("superseded", { status: 200 });
|
||||
// runtimeFor can return a newer instance than the leased preflight saw.
|
||||
// Keep the exact staged epoch; callbacks on a matching instance retain
|
||||
// their normal transactional fences if a pause wins after this check.
|
||||
@@ -3403,7 +3414,14 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
||||
: null;
|
||||
}
|
||||
|
||||
const slackCloudEnrollment = createSlackCloudEnrollment(db, { resolveCredentials, runtimeGeneration });
|
||||
const unregisterSlackCloudEnrollment = registerSlackCloudEnrollmentOwner(db, async endpointId => {
|
||||
const record = await endpointRecord(endpointId);
|
||||
if (!record) throw new Error("Slack enrollment endpoint is absent");
|
||||
await withCredentialMutationLease(record.endpoint, lease => slackCloudEnrollment(endpointId, undefined, lease));
|
||||
});
|
||||
const { configure, configureWithCredentialLease, runtimeConfig } = createChatConfiguration(db, {
|
||||
stageConfigurationEffects: stageSlackCloudEnrollmentIntent,
|
||||
endpointRecord, get, withCredentialMutationLease, getWebhookPublicBaseUrl, invalidateRuntime,
|
||||
nativeBotIdentityMatches, assertNativeBotIdentityAvailable, isNativeBotIdentityUniqueViolation, nativeBotIdentityConflict,
|
||||
legacyGitHubLabelsMatchVerifiedCredentials, normalizedCredentials, verifyCredentials,
|
||||
@@ -3611,6 +3629,8 @@ return {
|
||||
unregisterSlackTaskAuthority();
|
||||
voice.unregisterAgentTools();
|
||||
unregisterGitHubCloudIngress();
|
||||
unregisterSlackCloudIngress();
|
||||
unregisterSlackCloudEnrollment();
|
||||
unregisterCommittedResponseAuthority();
|
||||
await Promise.allSettled([...publicationEndpointTasks.values()]);
|
||||
await Promise.allSettled([...backgroundMessageTasks]);
|
||||
|
||||
@@ -100,6 +100,7 @@ type ChatConfigurationOptions = Pick<ReturnType<typeof createChatCredentials>, "
|
||||
githubWizard: Pick<ReturnType<typeof createGitHubAppSetup>["githubWizard"], "repairCloud">;
|
||||
nativeBotIdentityClaimBarrier?: () => Promise<void>;
|
||||
allowLocalVoiceBoard?: boolean;
|
||||
stageConfigurationEffects?: (transaction: Db, endpointId: string, actorUserId: string | null | undefined) => Promise<void>;
|
||||
};
|
||||
export function createChatConfiguration(db: Db, options: ChatConfigurationOptions) {
|
||||
const { endpointRecord, get, withCredentialMutationLease, getWebhookPublicBaseUrl, invalidateRuntime, nativeBotIdentityMatches, assertNativeBotIdentityAvailable, isNativeBotIdentityUniqueViolation, nativeBotIdentityConflict, legacyGitHubLabelsMatchVerifiedCredentials, normalizedCredentials, verifyCredentials, persistCredentials, clearCredentials, resolveCredentials, prepareProviderInventory, reconcileProviderResourceRows, reconcileDiscordCommands, runtimeFor, runtimeContextForRecord, stageTelegramMaintenance, processTelegramMaintenance, recordTelegramStopSubscription, fetchImpl, redactError, runtimeGeneration, leasedChatProvider, PROVIDER_LABELS, SUPPLIED_CREDENTIAL_KEYS, PROVIDER_CREDENTIAL_CHECK_TIMEOUT_MS, isSupportedTelegramWebhookBaseUrl } = options;
|
||||
@@ -176,6 +177,14 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
|
||||
const record = await endpointRecord(endpointId);
|
||||
if (!record) throw notFound("Chat endpoint not found");
|
||||
const endpoint = record.endpoint;
|
||||
const mutate = <T>(work: (tx: Parameters<Parameters<Db["transaction"]>[0]>[0]) => Promise<T>) => db.transaction(async tx => {
|
||||
const result = await work(tx);
|
||||
if (endpoint.provider === "slack" && options.stageConfigurationEffects) {
|
||||
await options.stageConfigurationEffects(tx as unknown as Db, endpointId, actorUserId);
|
||||
await credentialLease.assertOwned(tx);
|
||||
}
|
||||
return result;
|
||||
});
|
||||
if (input.action === "pause") {
|
||||
if (endpoint.status !== "active") {
|
||||
throw conflict("Only an active chat connection can be paused", {
|
||||
@@ -183,7 +192,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
|
||||
});
|
||||
}
|
||||
await managementContext?.reauthorize();
|
||||
await db.transaction(async (tx) => {
|
||||
await mutate(async (tx) => {
|
||||
await credentialLease.assertOwned(tx);
|
||||
const pausedAt = new Date();
|
||||
const current = await tx
|
||||
@@ -299,7 +308,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
|
||||
let activatedEndpoint!: EndpointRow;
|
||||
try {
|
||||
await managementContext?.reauthorize();
|
||||
await db.transaction(async (tx) => {
|
||||
await mutate(async (tx) => {
|
||||
await credentialLease.assertOwned(tx);
|
||||
const resumedAt = new Date();
|
||||
const current = await tx
|
||||
@@ -384,7 +393,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
|
||||
const failure = redactError(error);
|
||||
const failedAt = new Date();
|
||||
await managementContext?.reauthorize();
|
||||
await db.transaction(async (tx) => {
|
||||
await mutate(async (tx) => {
|
||||
await credentialLease.assertOwned(tx);
|
||||
await notifyChatEndpointConfiguration(tx, endpoint);
|
||||
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
|
||||
@@ -478,7 +487,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
|
||||
});
|
||||
}
|
||||
await managementContext?.reauthorize();
|
||||
await db.transaction(async (tx) => {
|
||||
await mutate(async (tx) => {
|
||||
await credentialLease.assertOwned(tx);
|
||||
const current = await tx
|
||||
.select({ setup: chatEndpoints.setup, status: chatEndpoints.status })
|
||||
@@ -724,7 +733,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
|
||||
if (!endpoint.botExternalId) {
|
||||
try {
|
||||
await managementContext?.reauthorize();
|
||||
const claimed = await db.transaction(async (tx) => {
|
||||
const claimed = await mutate(async (tx) => {
|
||||
await credentialLease.assertOwned(tx);
|
||||
await notifyChatEndpointConfiguration(tx, endpoint);
|
||||
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
|
||||
@@ -793,7 +802,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
|
||||
(input.action === "configure" || input.action === "reconnect");
|
||||
try {
|
||||
await managementContext?.reauthorize();
|
||||
await db.transaction(async (tx) => {
|
||||
await mutate(async (tx) => {
|
||||
await credentialLease.assertOwned(tx);
|
||||
const current = await tx
|
||||
.select({ setup: chatEndpoints.setup })
|
||||
@@ -892,7 +901,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
|
||||
const failure = redactError(error);
|
||||
await invalidateRuntime(endpoint.id).catch(() => undefined);
|
||||
await managementContext?.reauthorize();
|
||||
await db.transaction(async (tx) => {
|
||||
await mutate(async (tx) => {
|
||||
await credentialLease.assertOwned(tx);
|
||||
await notifyChatEndpointConfiguration(tx, endpoint);
|
||||
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
|
||||
@@ -922,7 +931,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
|
||||
}
|
||||
|
||||
await managementContext?.reauthorize();
|
||||
await db.transaction(async (tx) => {
|
||||
await mutate(async (tx) => {
|
||||
await credentialLease.assertOwned(tx);
|
||||
await logActivity(tx as unknown as Db, {
|
||||
companyId: endpoint.companyId,
|
||||
|
||||
@@ -48,6 +48,34 @@ export type InternalSetupState = ChatEndpointSetupState & {
|
||||
runtimeGeneration?: number;
|
||||
/** Binds pre-install URL verification to the signing secret later configured. */
|
||||
slackVerificationSigningFingerprint?: string;
|
||||
|
||||
slackCloudSubject?: string;
|
||||
slackCloudInstanceId?: string;
|
||||
slackCloudRetiredGeneration?: number;
|
||||
slackCloudIntent?: {
|
||||
version: 1;
|
||||
id: string;
|
||||
subject: string;
|
||||
instanceId: string;
|
||||
companyId: string;
|
||||
endpointId: string;
|
||||
publicId: string;
|
||||
connectionId: string;
|
||||
runtimeGeneration: number;
|
||||
active: boolean;
|
||||
teamId?: string;
|
||||
appId?: string;
|
||||
attempts: number;
|
||||
nextAttemptAt: string | null;
|
||||
failureCode?: string;
|
||||
};
|
||||
slackCloudEvents?: {
|
||||
version: 1;
|
||||
bindingId: string;
|
||||
runtimeGeneration: number;
|
||||
teamId: string;
|
||||
appId: string;
|
||||
};
|
||||
slackCallbackSurfaces?: Partial<
|
||||
Record<SlackCallbackSurface, SlackCallbackObservation>
|
||||
>;
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
import { generateKeyPairSync } from "node:crypto";
|
||||
import { registerSlackCloudEnrollmentOwner } from "./cloud-enrollment.js";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { afterEach, expect, it, vi } from "vitest";
|
||||
import { githubConnectionEventService } from "../../github-connection-events.js";
|
||||
import type { PaperclipCloudConnector } from "../../paperclip-cloud-connector.js";
|
||||
import { registerSlackCloudEventIngress } from "./cloud-ingress.js";
|
||||
import { SlackCloudReceiptError, type SlackCloudEvent } from "./cloud-event.js";
|
||||
|
||||
afterEach(() => vi.unstubAllEnvs());
|
||||
function fixture(handler: (event: SlackCloudEvent) => Promise<void>, pendingEnrollment = false) {
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", "");
|
||||
let reads = 0;
|
||||
const db = { select: () => {
|
||||
const rows = [[], [], [{ companyId: "company" }], [{ id: "endpoint", setup: { slackCloudIntent: { nextAttemptAt: "2000-01-01T00:00:00Z" } } }]][reads++ % (pendingEnrollment ? 4 : 3)]!;
|
||||
const query = { from: () => query, innerJoin: () => query, where: () => query, orderBy: () => query, limit: async () => rows,
|
||||
then: (resolve: (rows: unknown[]) => unknown) => Promise.resolve(rows).then(resolve) };
|
||||
return query;
|
||||
} } as unknown as Db;
|
||||
const event: SlackCloudEvent = { id: "slack-receipt", provider: "slack", event: "event_callback", installationId: "binding", bindingIds: ["binding"],
|
||||
createdAt: new Date().toISOString(), payload: {} };
|
||||
const connector = { leaseEvents: vi.fn(async () => ({ leaseId: "lease", events: [event] })),
|
||||
acknowledgeEvents: vi.fn(async () => 1), eventWakeVersion: () => 1 as const };
|
||||
const unregister = registerSlackCloudEventIngress(db, handler);
|
||||
let time = Date.now();
|
||||
const service = githubConnectionEventService(db, { connector: connector as unknown as PaperclipCloudConnector, now: () => new Date(time) });
|
||||
return { db, service, connector, unregister, event, time: () => time, advance: (ms: number) => { time += ms; } };
|
||||
}
|
||||
|
||||
it("actual connection-event owner routes Slack and retries a lost ACK through the registered owner", async () => {
|
||||
const handler = vi.fn(async () => {}), f = fixture(handler);
|
||||
try {
|
||||
f.connector.acknowledgeEvents.mockRejectedValueOnce(new Error("ack response lost"));
|
||||
await expect(f.service.pollOnce({ forDelivery: true })).rejects.toThrow("ack response lost");
|
||||
expect((await f.service.pollOnce({ forDelivery: true })).processed).toBe(1);
|
||||
expect(handler).toHaveBeenCalledTimes(2);
|
||||
expect(f.connector.leaseEvents).toHaveBeenCalledWith({ subject: "github-bot-events", companyId: "company", eventWakeVersion: 1 });
|
||||
expect(f.connector.acknowledgeEvents).toHaveBeenLastCalledWith(expect.objectContaining({ deliveryIds: [f.event.id] }));
|
||||
} finally { f.unregister(); }
|
||||
});
|
||||
|
||||
it.each(["retry", "attention", "invalid"] as const)("actual owner distinguishes %s receipt failure before ACK", async disposition => {
|
||||
const f = fixture(async () => { throw new SlackCloudReceiptError("synthetic_code", disposition); });
|
||||
try {
|
||||
const result = await f.service.pollOnce({ forDelivery: true });
|
||||
expect(result.failed).toBe(disposition === "invalid" ? 0 : 1);
|
||||
expect(f.connector.acknowledgeEvents).toHaveBeenCalledTimes(disposition === "invalid" ? 1 : 0);
|
||||
} finally { f.unregister(); }
|
||||
});
|
||||
|
||||
it.each(["chat_endpoint_credentials_busy", "temporary enrollment owner failure"])("drains and ACKs before retaining %s on a bounded existing-worker retry", async message => {
|
||||
const handler = vi.fn(async () => {}), f = fixture(handler, true);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", "https://cloud.example.test");
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", "instance");
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_ENVIRONMENT", "test");
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SIGN_PRIVATE_KEY", generateKeyPairSync("ed25519").privateKey.export({ format: "jwk" }).d!);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SEAL_PRIVATE_KEY", generateKeyPairSync("x25519").privateKey.export({ format: "jwk" }).d!);
|
||||
const owner = vi.fn(async () => {
|
||||
expect(handler).toHaveBeenCalled();
|
||||
expect(f.connector.acknowledgeEvents).toHaveBeenCalled();
|
||||
throw new Error(message);
|
||||
});
|
||||
const unregister = registerSlackCloudEnrollmentOwner(f.db, owner);
|
||||
try {
|
||||
expect((await f.service.pollOnce({ forDelivery: true })).processed).toBe(1);
|
||||
expect(owner).toHaveBeenCalledTimes(1);
|
||||
expect(f.service.nextPollAt()).toBe(f.time() + 5_000);
|
||||
await f.service.pollOnce();
|
||||
expect(owner).toHaveBeenCalledTimes(1);
|
||||
expect(f.connector.leaseEvents).toHaveBeenCalledTimes(1);
|
||||
f.advance(5_000);
|
||||
expect((await f.service.pollOnce()).processed).toBe(1);
|
||||
expect(owner).toHaveBeenCalledTimes(2);
|
||||
} finally { unregister(); f.unregister(); }
|
||||
});
|
||||
@@ -0,0 +1,451 @@
|
||||
import { guardSlackFixtureNetwork, stubSlackFixtureTransport } from "../../../__tests__/helpers/slack-cloud/transport.js";
|
||||
import { createHmac, generateKeyPairSync, randomUUID } from "node:crypto";
|
||||
import { once } from "node:events";
|
||||
import { createServer } from "node:http";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import type { ToolCredentialSecretRef } from "@paperclipai/shared";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import { activityLog, agentRuntimeState, agentTaskSessions, agentWakeupRequests, agents, applyPendingMigrations, authUsers, chatConversations, chatDeliveries, chatExternalPrincipals, chatIdentityLinks, chatEndpointResources, chatEndpoints, chatMessageLinks, chatSlackRegistrations, heartbeatRunEvents, heartbeatRuns, companies, companyMemberships, companySecrets, companySkills, costEvents, financeEvents, closeCheckpointDb, closeFileStorageDb, closeLivenessDb, createDb, issueComments, issues, toolApplications, toolConnections } from "@paperclipai/db";
|
||||
import { resolveExternalTestDatabaseUrl } from "../../../lib/dedicated-test-database-url.js";
|
||||
import { subscribeDatabaseWork } from "@paperclipai/db/work-signals";
|
||||
import { DELIVERY_QUEUES } from "../../delivery-work-notifications.js";
|
||||
import { chatCredentialMutationLease } from "../../chat-credential-mutation-lease.js";
|
||||
import { createPaperclipCloudConnector, paperclipCloudConnectorConfigFromEnv } from "../../paperclip-cloud-connector.js";
|
||||
import { chatChannelService } from "../../chat-channels.js";
|
||||
import { ChatRuntime } from "../../chat-runtime.js";
|
||||
import { heartbeatService } from "../../heartbeat.js";
|
||||
import { githubConnectionEventService } from "../../github-connection-events.js";
|
||||
import { secretService } from "../../secrets.js";
|
||||
import type { ChatSdkMessageCallbackEvent } from "../../chat-sdk-runtime.js";
|
||||
import { createChatSdkEndpointRuntime } from "../../chat-sdk-runtime.js";
|
||||
import { createSlackCloudEnrollment, stageSlackCloudEnrollmentIntent } from "./cloud-enrollment.js";
|
||||
import { createSlackCloudEventIngress, dispatchSlackCloudEvent } from "./cloud-ingress.js";
|
||||
import type { InternalSetupState } from "../runtime-types.js";
|
||||
|
||||
const appUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_SLACK_TEST_DATABASE_URL");
|
||||
const cloudUrl = resolveExternalTestDatabaseUrl("SLACK_EVENT_TEST_DATABASE_URL");
|
||||
const cloudDist = process.env.SLACK_TEST_CLOUD_DIST;
|
||||
|
||||
it.skipIf(!appUrl || !cloudUrl || !cloudDist)("signed Slack enrollment and actual event owner preserve one domain reply through lost ACK and removal retries", async () => {
|
||||
const network = guardSlackFixtureNetwork({ requirePreload: true });
|
||||
const load = (file: string) => import(/* @vite-ignore */ pathToFileURL(path.resolve(cloudDist!, file)).href);
|
||||
const [{ PostgresCloudHarnessRegistry }, { PostgresConnectorStore }, { ConnectorBroker }, { InMemoryProviderSecretStore },
|
||||
{ providerRegistryFromEnvironment }, { CloudStackProvisioner }, { FakeCloudStackProvider }, { Pool }] = await Promise.all([
|
||||
load("registry/postgres.js"), load("connections/postgres.js"), load("connections/broker.js"), load("providers/neon.js"),
|
||||
load("connections/providers.js"), load("provisioner/provisioner.js"), load("providers/fake.js"),
|
||||
import(/* @vite-ignore */ pathToFileURL(path.resolve(cloudDist!, "../node_modules/pg/lib/index.js")).href),
|
||||
]);
|
||||
await applyPendingMigrations(appUrl!);
|
||||
const db = createDb(appUrl!, { maxConnections: 2, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
|
||||
const pool = new Pool({ connectionString: cloudUrl!, max: 2 });
|
||||
|
||||
const registry = Reflect.construct(PostgresCloudHarnessRegistry, [pool]) as {
|
||||
applyMigrations(folder: string, timeout: number): Promise<void>;
|
||||
hydrate(): Promise<void>;
|
||||
close(): Promise<void>;
|
||||
};
|
||||
try {
|
||||
await registry.applyMigrations(path.resolve(cloudDist!, "../migrations"), 30_000);
|
||||
await registry.hydrate();
|
||||
}
|
||||
catch (error) { await db.$client.end({ timeout: 1 }); await registry.close(); throw error; }
|
||||
const store = new PostgresConnectorStore(pool), secrets = new InMemoryProviderSecretStore();
|
||||
const id = randomUUID(), companyId = randomUUID(), endpointId = randomUUID(), agentId = randomUUID(), connectionId = randomUUID(), applicationId = randomUUID();
|
||||
const actorId = `slack-owner-${id}`;
|
||||
const instanceId = `instance-${id}`, stackId = `slack-${id}`, publicId = "a".repeat(43), signingSecret = "0123456789abcdef0123456789abcdef";
|
||||
const signing = generateKeyPairSync("ed25519"), sealing = generateKeyPairSync("x25519"), now = new Date();
|
||||
let broker: InstanceType<typeof ConnectorBroker>;
|
||||
let enrollmentRequests = 0;
|
||||
const server = createServer((req, res) => {
|
||||
if (req.url === "/v1/connector/slack-events") enrollmentRequests++;
|
||||
if (brokerUnavailable) { res.writeHead(503, { "content-type": "application/json" }); res.end(JSON.stringify({ error: "fixture_broker_unavailable" })); return; }
|
||||
if (loseEnrollmentResponse && req.url === "/v1/connector/slack-events") {
|
||||
loseEnrollmentResponse = false;
|
||||
res.end = (() => { res.destroy(); return res; }) as typeof res.end;
|
||||
}
|
||||
void broker.handle(req, res, new URL(req.url!, base)).catch((error: unknown) => {
|
||||
if (!res.headersSent) res.writeHead(500); res.end(String(error));
|
||||
}); });
|
||||
let base = "";
|
||||
let brokerUnavailable = false, loseEnrollmentResponse = false;
|
||||
let unregister: (() => void) | undefined;
|
||||
let runtime: ReturnType<typeof createChatSdkEndpointRuntime> | undefined;
|
||||
const deliveryIds: string[] = [];
|
||||
let normalService: ReturnType<typeof chatChannelService> | undefined;
|
||||
let secretDirectory: string | undefined;
|
||||
let normalHeartbeat: ReturnType<typeof heartbeatService> | undefined;
|
||||
let releaseEnrollmentLease: (() => void) | undefined;
|
||||
let heldEnrollmentLease: Promise<void> | undefined;
|
||||
try {
|
||||
await new CloudStackProvisioner({ provider: new FakeCloudStackProvider(), registry }).provisionDevelopmentStack({
|
||||
idempotencyKey: id, accountGroup: { id: `account-${id}`, slug: stackId, displayName: "Slack fixture" },
|
||||
owner: { id: actorId, email: `${id}@example.test` },
|
||||
stack: { stackId, accountGroupId: `account-${id}`, stackSlug: stackId, displayName: "Slack fixture", kind: "demo", primaryHost: `${stackId}.example.test`, region: "us-east-1" },
|
||||
versionPlan: { appVersion: "fixture", schemaVersion: "fixture", rolloutTrack: "stable" },
|
||||
sleepPolicy: { desiredState: "awake" }, primaryCompany: { paperclipCompanyId: companyId, displayName: "Fixture", defaultBudgetCents: 0 },
|
||||
});
|
||||
await store.putCloudStackInstance({ id: instanceId, kind: "cloud_stack", companyStackId: stackId, environment: "test",
|
||||
signPublicKey: signing.publicKey.export({ format: "jwk" }).x!, sealPublicKey: sealing.publicKey.export({ format: "jwk" }).x!,
|
||||
origins: ["https://tenant.example"], activeOrigin: "https://tenant.example", label: "Fixture", status: "active", createdAt: now, updatedAt: now });
|
||||
server.listen(0, "127.0.0.1"); await once(server, "listening");
|
||||
const address = server.address(); if (!address || typeof address === "string") throw new Error("Missing fixture address");
|
||||
base = `http://127.0.0.1:${address.port}`;
|
||||
const wake = vi.fn(async () => {});
|
||||
broker = new ConnectorBroker({ publicOrigin: base, environment: "test", store,
|
||||
providers: providerRegistryFromEnvironment({}, "test"), secrets: { resolve: async () => undefined },
|
||||
authenticate: async () => ({ globalUserId: actorId }), canApproveInstance: async (principal: { globalUserId: string }) => principal.globalUserId === actorId,
|
||||
slackEvents: { secrets, secretRefPrefix: "secret:///test/slack" },
|
||||
eventWakeEnrollmentEnabled: true, eventWakeDeliveryEnabled: true, wakeCloudStack: wake });
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", instanceId);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SIGN_PRIVATE_KEY", signing.privateKey.export({ format: "jwk" }).d!);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SEAL_PRIVATE_KEY", sealing.privateKey.export({ format: "jwk" }).d!);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_ENVIRONMENT", "test");
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", base);
|
||||
await db.insert(companies).values({ id: companyId, name: "Slack", issuePrefix: `S${id.slice(0, 7)}` });
|
||||
await db.insert(authUsers).values({ id: actorId, name: "Slack owner", email: `${id}@owner.example.test`, emailVerified: true, createdAt: now, updatedAt: now });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: actorId, status: "active", membershipRole: "operator" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Slack", adapterType: "process", status: "idle", runtimeConfig: { heartbeat: { wakeOnDemand: true, intervalSec: 0 } } });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "agent", principalId: agentId, status: "active", membershipRole: "member" });
|
||||
await db.insert(toolApplications).values({ id: applicationId, companyId, name: "Slack", type: "chat" });
|
||||
await db.insert(toolConnections).values({ id: connectionId, companyId, applicationId, name: "Slack", uid: id,
|
||||
transport: "chat_sdk", connectionPurpose: "channel", enabled: true, status: "active" });
|
||||
await db.insert(chatEndpoints).values({ id: endpointId, companyId, connectionId, assignedAgentId: agentId,
|
||||
publicId, provider: "slack", status: "active", providerAccountId: "TTEST", setup: { runtimeGeneration: 1 } as InternalSetupState });
|
||||
await db.insert(chatSlackRegistrations).values({ companyId, endpointId, requestId: randomUUID(), status: "configured",
|
||||
manifest: {}, manifestHash: "fixture", callbackUri: "https://tenant.example/callback", appId: "ATEST", createdByUserId: actorId });
|
||||
const withLease = chatCredentialMutationLease(db);
|
||||
const resolveCredentials = vi.fn(async () => ({ signingSecret }));
|
||||
const produce = createSlackCloudEnrollment(db, { resolveCredentials,
|
||||
runtimeGeneration: setup => Number(setup.runtimeGeneration ?? 0) });
|
||||
const endpoint = async () => (await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, endpointId)))[0]!;
|
||||
await db.transaction(tx => stageSlackCloudEnrollmentIntent(tx as unknown as typeof db, endpointId, actorId));
|
||||
await withLease(await endpoint(), lease => produce(endpointId, actorId, lease));
|
||||
const saved = (await endpoint()).setup as InternalSetupState;
|
||||
expect(saved.slackCloudEvents?.runtimeGeneration).toBe(1);
|
||||
expect(JSON.stringify(saved)).not.toContain(signingSecret);
|
||||
const rawBody = Buffer.from(JSON.stringify({ type: "event_callback", event_id: "EvPaired", team_id: "TTEST", api_app_id: "ATEST",
|
||||
event: { type: "app_mention", text: "<@UBOT> hello", user: "UOPERATOR", channel: "CTEST", ts: "1788.500", event_ts: "1788.500" } }));
|
||||
const timestamp = String(Math.floor(Date.now() / 1000));
|
||||
const request = { rawBody, timestamp, signature: `v0=${createHmac("sha256", signingSecret).update(`v0:${timestamp}:`).update(rawBody).digest("hex")}` };
|
||||
const current = await broker.slackEvents.current(instanceId, publicId);
|
||||
await expect(broker.slackEvents.receive(current, request)).rejects.toThrow("SLACK_DURABLE_WAKE_UNAVAILABLE");
|
||||
const connector = createPaperclipCloudConnector({ config: paperclipCloudConnectorConfigFromEnv()! });
|
||||
expect(await connector.leaseEvents({ subject: actorId, companyId, eventWakeVersion: 1 })).toBeNull();
|
||||
expect(await broker.slackEvents.receive(current, request)).toEqual({ status: 200, duplicate: false });
|
||||
await vi.waitFor(() => expect(wake).toHaveBeenCalledOnce());
|
||||
const leased = await connector.leaseEvents({ subject: actorId, companyId, eventWakeVersion: 1 });
|
||||
expect(leased?.failedEventIds).toBeUndefined(); expect(leased?.events).toHaveLength(1);
|
||||
const event = leased!.events[0]!; deliveryIds.push(event.id);
|
||||
if (event.provider !== "slack") throw new Error("Unexpected event owner");
|
||||
await expect(dispatchSlackCloudEvent(db, event)).rejects.toThrow("not ready");
|
||||
const onMessage = vi.fn(async (_event: ChatSdkMessageCallbackEvent) => {});
|
||||
runtime = createChatSdkEndpointRuntime({ companyId, endpointId, logger: "silent", callbacks: { onMessage },
|
||||
persistence: { read: async () => null, compareAndSet: async () => true, deleteIfVersion: async () => true },
|
||||
providerConfig: { provider: "slack", userName: "paperclip-agent", credentials: { apiUrl: "http://127.0.0.1:1/api/", botToken: "xoxb-fixture", botUserId: "UBOT", signingSecret } } });
|
||||
await runtime.initialize();
|
||||
const transport = stubSlackFixtureTransport(runtime.getProviderAdapter());
|
||||
unregister = createSlackCloudEventIngress(db, { withCredentialMutationLease: withLease, resolveCredentials,
|
||||
runtimeGeneration: value => Number((value.setup as InternalSetupState).runtimeGeneration ?? 0),
|
||||
handleWebhook: async (_publicId, _provider, req, generation) => {
|
||||
expect(generation).toBe(1);
|
||||
|
||||
await withLease(await endpoint(), async lease => { await lease.assertOwned(); });
|
||||
return runtime!.handleWebhook(req);
|
||||
} });
|
||||
await dispatchSlackCloudEvent(db, event);
|
||||
expect(onMessage).toHaveBeenCalledOnce();
|
||||
expect(onMessage).toHaveBeenCalledWith(expect.objectContaining({ message: expect.objectContaining({ author: expect.objectContaining({ userId: "UOPERATOR", userName: "Operator" }) }) }));
|
||||
expect(transport).toHaveBeenCalled();
|
||||
await transport.assertNoErrors();
|
||||
expect(await connector.acknowledgeEvents({ subject: actorId, companyId, leaseId: leased!.leaseId, deliveryIds: [event.id] })).toBe(1);
|
||||
expect(await connector.leaseEvents({ subject: actorId, companyId, eventWakeVersion: 1 })).toBeNull();
|
||||
await db.update(chatEndpoints).set({ setup: { ...saved, runtimeGeneration: 2 } as InternalSetupState }).where(eq(chatEndpoints.id, endpointId));
|
||||
await dispatchSlackCloudEvent(db, event);
|
||||
expect(onMessage).toHaveBeenCalledOnce();
|
||||
await db.transaction(tx => stageSlackCloudEnrollmentIntent(tx as unknown as typeof db, endpointId, actorId));
|
||||
await withLease(await endpoint(), lease => produce(endpointId, actorId, lease));
|
||||
const replacement = ((await endpoint()).setup as InternalSetupState).slackCloudEvents!;
|
||||
expect(replacement.runtimeGeneration).toBe(2); expect(replacement.bindingId).not.toBe(saved.slackCloudEvents!.bindingId);
|
||||
const firstMessage = onMessage.mock.calls[0]![0];
|
||||
const [task] = await db.insert(issues).values({ companyId, title: "Existing Slack conversation", status: "in_progress",
|
||||
assigneeAgentId: agentId, createdByUserId: actorId, identifier: `S${id.slice(0, 7)}-1`, issueNumber: 1 }).returning();
|
||||
const [resource] = await db.insert(chatEndpointResources).values({ companyId, endpointId, type: "channel",
|
||||
providerResourceId: firstMessage.thread.channelId, label: "Slack fixture", availability: "available", enabled: true }).returning();
|
||||
await db.insert(chatConversations).values({ companyId, endpointId, issueId: task!.id, resourceId: resource!.id,
|
||||
externalConversationId: firstMessage.thread.channelId, externalThreadId: firstMessage.thread.id,
|
||||
externalLabel: "Slack fixture", isDirectMessage: firstMessage.thread.isDM });
|
||||
secretDirectory = await mkdtemp(path.join(os.tmpdir(), "slack-owner-secrets-"));
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", path.join(secretDirectory, "master.key"));
|
||||
const markerPath = path.join(secretDirectory, "normal-process-marker.json");
|
||||
const markerText = `slack-domain-once-${id}`;
|
||||
await db.update(agents).set({ adapterConfig: { command: process.execPath, cwd: secretDirectory,
|
||||
args: ["--require", fileURLToPath(new URL("../../../__tests__/helpers/slack-cloud/prewire.cjs", import.meta.url)), "-e", `if(process.env.SLACK_GATE_GUARD_PID!==String(process.pid))throw Error("child prewire guard missing");require("node:fs").writeFileSync(${JSON.stringify(markerPath)},JSON.stringify({pid:process.pid,runId:process.env.PAPERCLIP_RUN_ID,marker:${JSON.stringify(markerText)}}),{flag:"ax"});console.log(${JSON.stringify(markerText)});`],
|
||||
timeoutSec: 10, graceSec: 1 } }).where(eq(agents.id, agentId));
|
||||
const vault = secretService(db), refs: ToolCredentialSecretRef[] = [];
|
||||
for (const [key, value] of Object.entries({ botToken: "xoxb-fixture", signingSecret })) {
|
||||
const secret = await vault.create(companyId, { name: `slack-${key}-${id}`, provider: "local_encrypted", value }, { userId: actorId });
|
||||
await vault.createBinding({ companyId, secretId: secret.id, targetType: "tool_connection", targetId: connectionId, configPath: `credentials.${key}` });
|
||||
refs.push({ secretId: secret.id, versionSelector: "latest", configPath: `credentials.${key}`, required: true, projectionClass: "unclassified" });
|
||||
}
|
||||
await db.update(toolConnections).set({ credentialSecretRefs: refs }).where(eq(toolConnections.id, connectionId));
|
||||
const [linkedPrincipal] = await db.insert(chatExternalPrincipals).values({ companyId, provider: "slack", providerAccountId: "TTEST",
|
||||
externalId: "UOPERATOR", kind: "user", displayName: "Operator" }).returning();
|
||||
await db.insert(chatIdentityLinks).values({ companyId, endpointId, principalId: linkedPrincipal!.id, paperclipUserId: actorId,
|
||||
status: "linked", confirmedAt: now });
|
||||
const canonicalCallback = "https://tenant.example/api/chat-webhooks/" + publicId + "/slack";
|
||||
await db.update(chatEndpoints).set({ botExternalId: "UBOT", sponsorUserId: actorId, allowGroupChats: true,
|
||||
setup: { ...((await endpoint()).setup as InternalSetupState), step: "complete", webhookVerifiedAt: now.toISOString(),
|
||||
slackCallbackSurfaces: { events: { url: canonicalCallback, observedAt: now.toISOString() } } } as InternalSetupState }).where(eq(chatEndpoints.id, endpointId));
|
||||
const callbackEvidence = structuredClone(((await endpoint()).setup as InternalSetupState).slackCallbackSurfaces);
|
||||
const actualRuntime = new ChatRuntime(), sdkCallbacks: ChatSdkMessageCallbackEvent[] = [];
|
||||
const transports: ReturnType<typeof stubSlackFixtureTransport>[] = [];
|
||||
const replaceEndpoint = actualRuntime.replaceEndpoint.bind(actualRuntime);
|
||||
vi.spyOn(actualRuntime, "replaceEndpoint").mockImplementation(async input => {
|
||||
if (input.providerConfig.provider !== "slack") throw new Error("Unexpected fixture provider");
|
||||
const value = await replaceEndpoint({ ...input, providerConfig: { ...input.providerConfig,
|
||||
credentials: { ...input.providerConfig.credentials, apiUrl: "http://127.0.0.1:1/api/" } },
|
||||
callbacks: { ...input.callbacks, onMessage: async message => { sdkCallbacks.push(message); await input.callbacks.onMessage(message); } } });
|
||||
transports.push(stubSlackFixtureTransport(value.getProviderAdapter()));
|
||||
return value;
|
||||
});
|
||||
normalHeartbeat = heartbeatService(db);
|
||||
const receiptFetch: typeof fetch = async (input, init) => {
|
||||
const url = input instanceof Request ? input.url : String(input);
|
||||
if (!["https://slack.com/api/reactions.add", "https://slack.com/api/reactions.remove"].includes(url)) throw new Error(`Unexpected normal Slack owner request ${url}`);
|
||||
expect(init?.method).toBe("POST");
|
||||
expect(JSON.parse(String(init?.body))).toMatchObject({ channel: "CTEST", name: "eyes" });
|
||||
return Response.json({ ok: true });
|
||||
};
|
||||
const deferred: (() => void | Promise<void>)[] = [];
|
||||
normalService = chatChannelService(db, { runtime: actualRuntime, publicBaseUrl: "https://tenant.example",
|
||||
deferWebhookProcessing: true, scheduleDeferredWork: work => { deferred.push(work); }, heartbeat: normalHeartbeat, fetch: receiptFetch });
|
||||
const ownerMessageTs = (Date.now() / 1000).toFixed(6);
|
||||
const ownerRawBody = Buffer.from(JSON.stringify({ type: "event_callback", event_id: "EvActualOwner", team_id: "TTEST", api_app_id: "ATEST",
|
||||
event: { type: "app_mention", text: "<@UBOT> one durable reply", user: "UOPERATOR", channel: "CTEST", ts: ownerMessageTs, event_ts: ownerMessageTs, thread_ts: "1788.500" } }));
|
||||
const ownerTimestamp = String(Math.floor(Date.now() / 1000));
|
||||
const ownerRequest = { rawBody: ownerRawBody, timestamp: ownerTimestamp,
|
||||
signature: `v0=${createHmac("sha256", signingSecret).update(`v0:${ownerTimestamp}:`).update(ownerRawBody).digest("hex")}` };
|
||||
await broker.slackEvents.receive(await broker.slackEvents.current(instanceId, publicId), ownerRequest);
|
||||
let lostAck = true;
|
||||
const captured: NonNullable<Awaited<ReturnType<typeof connector.leaseEvents>>>["events"] = [];
|
||||
const ownerConnector = { ...connector, leaseEvents: async (input: Parameters<typeof connector.leaseEvents>[0]) => {
|
||||
const batch = await connector.leaseEvents(input); if (batch) captured.push(...batch.events); return batch;
|
||||
}, acknowledgeEvents: async (input: Parameters<typeof connector.acknowledgeEvents>[0]) => {
|
||||
if (lostAck) { lostAck = false; throw new Error("fixture ACK request lost before broker settlement"); }
|
||||
return connector.acknowledgeEvents(input);
|
||||
} };
|
||||
const busyEndpointId = randomUUID(), busyConnectionId = randomUUID();
|
||||
await db.insert(toolConnections).values({ id: busyConnectionId, companyId, applicationId, name: "Retirement waiting for credential mutation", uid: busyEndpointId,
|
||||
transport: "chat_sdk", connectionPurpose: "channel", enabled: false, status: "disabled" });
|
||||
await db.insert(chatEndpoints).values({ id: busyEndpointId, companyId, connectionId: busyConnectionId, assignedAgentId: agentId,
|
||||
publicId: "f".repeat(43), provider: "slack", status: "archived", setup: { runtimeGeneration: 1 } as InternalSetupState });
|
||||
await db.transaction(tx => stageSlackCloudEnrollmentIntent(tx as unknown as typeof db, busyEndpointId, actorId));
|
||||
const [busyEndpoint] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, busyEndpointId));
|
||||
const originalBusyIntent = structuredClone((busyEndpoint!.setup as InternalSetupState).slackCloudIntent!);
|
||||
let acquired!: () => void;
|
||||
const acquiredPromise = new Promise<void>(resolve => { acquired = resolve; });
|
||||
const releasePromise = new Promise<void>(resolve => { releaseEnrollmentLease = resolve; });
|
||||
heldEnrollmentLease = withLease(busyEndpoint!, async lease => { await lease.assertOwned(); acquired(); await releasePromise; });
|
||||
await acquiredPromise;
|
||||
const eventOwner = githubConnectionEventService(db, { connector: ownerConnector });
|
||||
await expect(eventOwner.pollOnce({ forDelivery: true })).rejects.toThrow("ACK request lost");
|
||||
expect(captured).toHaveLength(1); const actualReceiptId = captured[0]!.id; deliveryIds.push(actualReceiptId);
|
||||
const beforeRedelivery = await db.select().from(chatDeliveries).where(eq(chatDeliveries.endpointId, endpointId));
|
||||
expect(beforeRedelivery).toHaveLength(1);
|
||||
const [retainedBusy] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, busyEndpointId));
|
||||
const busyIntent = (retainedBusy!.setup as InternalSetupState).slackCloudIntent!;
|
||||
expect(busyIntent).toEqual({ ...originalBusyIntent, attempts: 1, failureCode: "credential_lease_busy", nextAttemptAt: expect.any(String) });
|
||||
const originalRetryAt = Date.parse(busyIntent.nextAttemptAt!);
|
||||
expect(Number.isFinite(originalRetryAt)).toBe(true);
|
||||
expect(originalRetryAt).toBeGreaterThan(Date.now());
|
||||
expect(originalRetryAt).toBeLessThanOrEqual(Date.now() + 5_000);
|
||||
expect(eventOwner.nextPollAt()).not.toBeNull();
|
||||
expect(eventOwner.nextPollAt()!).toBeGreaterThan(Date.now());
|
||||
expect(eventOwner.nextPollAt()!).toBeLessThanOrEqual(Date.now() + 5_000);
|
||||
releaseEnrollmentLease!(); await heldEnrollmentLease;
|
||||
|
||||
const expired = await pool.query("UPDATE cloud_harness.connector_webhook_delivery_targets SET lease_expires_at=$3 WHERE connector_instance_id=$1 AND delivery_id=$2 AND status='leased' RETURNING delivery_id",
|
||||
[instanceId, actualReceiptId, new Date(Date.now() - 1)]);
|
||||
expect(expired.rowCount).toBe(1);
|
||||
expect((await eventOwner.pollOnce({ forDelivery: true })).processed).toBe(1);
|
||||
expect(captured.map(value => value.id)).toEqual([actualReceiptId, actualReceiptId]);
|
||||
const [beforeRetry] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, busyEndpointId));
|
||||
if (Date.now() < originalRetryAt) expect((beforeRetry!.setup as InternalSetupState).slackCloudIntent).toEqual(busyIntent);
|
||||
await new Promise(resolve => setTimeout(resolve, Math.max(0, originalRetryAt - Date.now())));
|
||||
await eventOwner.pollOnce();
|
||||
const [settledBusy] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, busyEndpointId));
|
||||
expect((settledBusy!.setup as InternalSetupState).slackCloudIntent).toBeUndefined();
|
||||
expect((settledBusy!.setup as InternalSetupState).slackCloudRetiredGeneration).toBe(1);
|
||||
expect((settledBusy!.setup as InternalSetupState).slackCloudRetiredGeneration).toBe(originalBusyIntent.runtimeGeneration);
|
||||
expect(captured.map(value => value.id)).toEqual([actualReceiptId, actualReceiptId]);
|
||||
|
||||
expect(captured[1]!.createdAt).toBe(captured[0]!.createdAt); expect(captured[1]!.payload).toEqual(captured[0]!.payload);
|
||||
expect(sdkCallbacks).toHaveLength(2); expect(sdkCallbacks[0]!.message.id).toBe(sdkCallbacks[1]!.message.id);
|
||||
expect(sdkCallbacks[0]!.thread.id).toBe(sdkCallbacks[1]!.thread.id);
|
||||
const afterRedelivery = await db.select().from(chatDeliveries).where(eq(chatDeliveries.endpointId, endpointId));
|
||||
expect(afterRedelivery).toHaveLength(1); expect(afterRedelivery[0]!.providerEventId).toBe(beforeRedelivery[0]!.providerEventId);
|
||||
expect(deferred.length).toBeGreaterThan(0);
|
||||
await deferred.shift()!();
|
||||
await normalService.processQueuedDeliveries();
|
||||
const links = await db.select().from(chatMessageLinks).where(and(eq(chatMessageLinks.endpointId, endpointId), eq(chatMessageLinks.direction, "inbound")));
|
||||
expect(links).toHaveLength(1); expect(links[0]!.deliveryId).toBe(afterRedelivery[0]!.id);
|
||||
const comments = await db.select().from(issueComments).where(eq(issueComments.issueId, task!.id));
|
||||
expect(comments.filter(value => value.id === links[0]!.commentId)).toHaveLength(1);
|
||||
expect(comments.find(value => value.id === links[0]!.commentId)?.body).toContain("one durable reply");
|
||||
const wakeReceipts = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, agentId));
|
||||
expect(wakeReceipts).toHaveLength(1);
|
||||
await normalHeartbeat.drainActiveRunExecutions();
|
||||
const completedRuns = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agentId));
|
||||
expect(completedRuns).toHaveLength(1);
|
||||
expect(completedRuns[0]).toMatchObject({ status: "succeeded", exitCode: 0, wakeupRequestId: wakeReceipts[0]!.id });
|
||||
expect(completedRuns[0]!.startedAt).not.toBeNull();
|
||||
expect(completedRuns[0]!.finishedAt).not.toBeNull();
|
||||
expect(completedRuns[0]!.stdoutExcerpt).toContain(markerText);
|
||||
const marker = JSON.parse(await readFile(markerPath, "utf8")) as { pid: number; runId: string; marker: string };
|
||||
expect(marker).toMatchObject({ runId: completedRuns[0]!.id, marker: markerText });
|
||||
expect(marker.pid).not.toBe(process.pid);
|
||||
expect(((await endpoint()).setup as InternalSetupState).slackCallbackSurfaces).toEqual(callbackEvidence);
|
||||
const signingRef = refs.find(ref => ref.configPath === "credentials.signingSecret")!;
|
||||
await vault.update(signingRef.secretId, { status: "disabled" });
|
||||
try {
|
||||
await expect(dispatchSlackCloudEvent(db, captured[1]!)).rejects.toMatchObject({
|
||||
code: "signing_credential_unavailable", disposition: "attention",
|
||||
});
|
||||
expect(sdkCallbacks).toHaveLength(2);
|
||||
} finally { await vault.update(signingRef.secretId, { status: "active" }); }
|
||||
|
||||
expect(await connector.leaseEvents({ subject: actorId, companyId, eventWakeVersion: 1 })).toBeNull();
|
||||
for (const transport of transports) await transport.assertNoErrors();
|
||||
await normalService.shutdown(); normalService = undefined;
|
||||
let rejectAfterIntent = false;
|
||||
const beforeRollbackRequests = enrollmentRequests;
|
||||
const stopSignals = subscribeDatabaseWork(db, DELIVERY_QUEUES.connectionEvents, event => { if (event === "intent") rejectAfterIntent = true; });
|
||||
normalService = chatChannelService(db, { heartbeat: normalHeartbeat, fetch: receiptFetch, publicBaseUrl: "https://tenant.example",
|
||||
scheduleDeferredWork: work => { deferred.push(work); }, renewCredentialMutationLease: async () => !rejectAfterIntent });
|
||||
try {
|
||||
await expect(normalService.configure(endpointId, { action: "pause" }, actorId)).rejects.toMatchObject({ code: "CHAT_CREDENTIAL_LEASE_LOST" });
|
||||
expect((await endpoint()).status).toBe("active");
|
||||
expect(((await endpoint()).setup as InternalSetupState).runtimeGeneration).toBe(2);
|
||||
expect(((await endpoint()).setup as InternalSetupState).slackCloudIntent).toBeUndefined();
|
||||
expect(enrollmentRequests).toBe(beforeRollbackRequests);
|
||||
} finally { stopSignals(); await normalService.shutdown(); normalService = undefined; }
|
||||
normalService = chatChannelService(db, { heartbeat: normalHeartbeat, fetch: receiptFetch, publicBaseUrl: "https://tenant.example",
|
||||
scheduleDeferredWork: work => { deferred.push(work); } });
|
||||
await db.transaction(async tx => {
|
||||
await tx.update(chatEndpoints).set({ setup: { ...((await endpoint()).setup as InternalSetupState), runtimeGeneration: 3 } as InternalSetupState }).where(eq(chatEndpoints.id, endpointId));
|
||||
await stageSlackCloudEnrollmentIntent(tx as unknown as typeof db, endpointId, actorId);
|
||||
});
|
||||
loseEnrollmentResponse = true;
|
||||
await withLease(await endpoint(), lease => produce(endpointId, actorId, lease));
|
||||
expect((await broker.slackEvents.current(instanceId, publicId)).runtimeGeneration).toBe(3);
|
||||
expect(((await endpoint()).setup as InternalSetupState).slackCloudEvents!.runtimeGeneration).toBe(2);
|
||||
expect(((await endpoint()).setup as InternalSetupState).slackCloudIntent).toMatchObject({ active: true, runtimeGeneration: 3, failureCode: "broker_unavailable" });
|
||||
await withLease(await endpoint(), async lease => {
|
||||
let transactionChecks = 0;
|
||||
await produce(endpointId, actorId, { assertOwned: async transaction => {
|
||||
await lease.assertOwned(transaction);
|
||||
if (transaction && ++transactionChecks === 2) throw new Error("fixture final enrollment guard lost");
|
||||
} });
|
||||
});
|
||||
expect(((await endpoint()).setup as InternalSetupState).slackCloudEvents!.runtimeGeneration).toBe(2);
|
||||
const casProducer = createSlackCloudEnrollment(db, { runtimeGeneration: setup => Number(setup.runtimeGeneration), resolveCredentials: async () => {
|
||||
await db.update(chatEndpoints).set({ setup: sql`jsonb_set(${chatEndpoints.setup}, '{runtimeGeneration}', '4'::jsonb)` }).where(eq(chatEndpoints.id, endpointId));
|
||||
return { signingSecret };
|
||||
} });
|
||||
await expect(withLease(await endpoint(), lease => casProducer(endpointId, actorId, lease))).rejects.toThrow("changed");
|
||||
expect(((await endpoint()).setup as InternalSetupState).slackCloudEvents!.runtimeGeneration).toBe(2);
|
||||
brokerUnavailable = true;
|
||||
expect((await normalService.configure(endpointId, { action: "pause" }, actorId)).status).toBe("paused");
|
||||
expect(((await endpoint()).setup as InternalSetupState).slackCloudIntent).toMatchObject({ active: false, runtimeGeneration: 5, attempts: 0 });
|
||||
await expect(eventOwner.pollOnce({ forDelivery: true })).rejects.toThrow("status=503");
|
||||
expect(((await endpoint()).setup as InternalSetupState).slackCloudIntent).toMatchObject({ failureCode: "broker_unavailable", attempts: 1 });
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", "");
|
||||
expect((await normalService.configure(endpointId, { action: "remove" }, actorId)).status).toBe("archived");
|
||||
expect(((await endpoint()).setup as InternalSetupState).slackCloudIntent).toMatchObject({ active: false, runtimeGeneration: 6, subject: actorId, attempts: 0 });
|
||||
expect(((await endpoint()).setup as InternalSetupState).slackCloudIntent).toMatchObject({ active: false, runtimeGeneration: 6, failureCode: "connector_configuration_unavailable", nextAttemptAt: null });
|
||||
expect((await db.select().from(toolConnections).where(eq(toolConnections.id, connectionId)))[0]!.credentialSecretRefs).toEqual([]);
|
||||
expect((await db.select().from(chatSlackRegistrations).where(eq(chatSlackRegistrations.endpointId, endpointId)))[0]!.status).toBe("removed");
|
||||
await normalService.shutdown(); normalService = undefined;
|
||||
brokerUnavailable = false;
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", base);
|
||||
normalService = chatChannelService(db, { heartbeat: normalHeartbeat, fetch: receiptFetch, publicBaseUrl: "https://tenant.example",
|
||||
scheduleDeferredWork: work => { deferred.push(work); } });
|
||||
const reconstructedOwner = githubConnectionEventService(db, { connector });
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", `replacement-${instanceId}`);
|
||||
await reconstructedOwner.pollOnce({ forDelivery: true });
|
||||
expect(((await endpoint()).setup as InternalSetupState).slackCloudIntent).toMatchObject({ instanceId, failureCode: "connector_configuration_unavailable", nextAttemptAt: null });
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", instanceId);
|
||||
await reconstructedOwner.pollOnce({ forDelivery: true });
|
||||
expect(reconstructedOwner.nextPollAt()).toBeNull();
|
||||
expect(((await endpoint()).setup as InternalSetupState).slackCloudIntent).toBeUndefined();
|
||||
expect(((await endpoint()).setup as InternalSetupState).slackCloudRetiredGeneration).toBe(6);
|
||||
expect(await broker.slackEvents.current(instanceId, publicId)).toBeUndefined();
|
||||
expect((await store.listWebhookBindings(instanceId)).find((value: { receivesEvents?: boolean }) => value.receivesEvents === false).providerMetadata.retirement.runtimeGeneration).toBe(6);
|
||||
await expect(broker.slackEvents.enroll(await store.getInstance(instanceId), actorId, { companyId, endpointId, publicId, connectionId,
|
||||
runtimeGeneration: 3, teamId: "TTEST", appId: "ATEST", signingSecret })).rejects.toThrow("GENERATION_CHANGED");
|
||||
|
||||
const pendingIds = [randomUUID(), randomUUID()];
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", "");
|
||||
for (const [index, pendingId] of pendingIds.entries()) {
|
||||
const pendingConnectionId = randomUUID();
|
||||
await db.insert(toolConnections).values({ id: pendingConnectionId, companyId, applicationId, name: "Retired Slack", uid: pendingId,
|
||||
transport: "chat_sdk", connectionPurpose: "channel", enabled: false, status: "disabled" });
|
||||
await db.insert(chatEndpoints).values({ id: pendingId, companyId, connectionId: pendingConnectionId, assignedAgentId: agentId,
|
||||
publicId: (index === 0 ? "d" : "e").repeat(43), provider: "slack", status: "archived", providerAccountId: "TTEST",
|
||||
setup: { runtimeGeneration: 1, slackCloudInstanceId: instanceId, slackCloudSubject: actorId } as InternalSetupState });
|
||||
await db.transaction(tx => stageSlackCloudEnrollmentIntent(tx as unknown as typeof db, pendingId, actorId));
|
||||
}
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", base);
|
||||
await reconstructedOwner.pollOnce({ forDelivery: true });
|
||||
const remaining = await db.select().from(chatEndpoints).where(and(eq(chatEndpoints.companyId, companyId),
|
||||
sql`${chatEndpoints.setup}->'slackCloudIntent' is not null`));
|
||||
expect(remaining).toHaveLength(1);
|
||||
expect(pendingIds).toContain(remaining[0]!.id);
|
||||
expect(reconstructedOwner.nextPollAt()).not.toBeNull();
|
||||
expect(reconstructedOwner.nextPollAt()!).toBeLessThanOrEqual(Date.now());
|
||||
await reconstructedOwner.pollOnce({ forDelivery: true });
|
||||
expect(await db.select().from(chatEndpoints).where(and(eq(chatEndpoints.companyId, companyId),
|
||||
sql`${chatEndpoints.setup}->'slackCloudIntent' is not null`))).toHaveLength(0);
|
||||
expect(reconstructedOwner.nextPollAt()).toBeNull();
|
||||
|
||||
} catch (error) {
|
||||
console.error("Slack paired fixture body failure", error);
|
||||
throw error;
|
||||
} finally {
|
||||
releaseEnrollmentLease?.(); await heldEnrollmentLease;
|
||||
await normalHeartbeat?.drainActiveRunExecutions();
|
||||
await normalService?.shutdown();
|
||||
unregister?.(); await runtime?.shutdown();
|
||||
if (server.listening) { server.close(); await once(server, "close"); }
|
||||
vi.unstubAllEnvs();
|
||||
try {
|
||||
await db.delete(chatEndpoints).where(eq(chatEndpoints.companyId, companyId));
|
||||
await db.delete(financeEvents).where(eq(financeEvents.companyId, companyId));
|
||||
await db.delete(costEvents).where(eq(costEvents.companyId, companyId));
|
||||
await db.delete(activityLog).where(eq(activityLog.companyId, companyId));
|
||||
await db.delete(agentTaskSessions).where(eq(agentTaskSessions.companyId, companyId));
|
||||
await db.delete(agentRuntimeState).where(eq(agentRuntimeState.companyId, companyId));
|
||||
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, companyId));
|
||||
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId));
|
||||
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId));
|
||||
await db.delete(issueComments).where(eq(issueComments.companyId, companyId));
|
||||
await db.delete(issues).where(eq(issues.companyId, companyId));
|
||||
await db.delete(agents).where(eq(agents.companyId, companyId));
|
||||
await db.delete(companyMemberships).where(eq(companyMemberships.companyId, companyId));
|
||||
await db.delete(companySecrets).where(eq(companySecrets.companyId, companyId));
|
||||
await db.delete(companySkills).where(eq(companySkills.companyId, companyId));
|
||||
await db.delete(companies).where(eq(companies.id, companyId));
|
||||
await db.delete(authUsers).where(eq(authUsers.id, actorId));
|
||||
await pool.query("DELETE FROM cloud_harness.connector_instances WHERE id=$1", [instanceId]);
|
||||
await pool.query("DELETE FROM cloud_harness.connector_webhook_deliveries WHERE id=ANY($1::text[])", [deliveryIds]);
|
||||
} finally { await closeCheckpointDb(db, 1); await closeFileStorageDb(db, 1); await closeLivenessDb(db, 1);
|
||||
await db.$client.end({ timeout: 1 }); await registry.close();
|
||||
if (secretDirectory) await rm(secretDirectory, { recursive: true, force: true });
|
||||
try { network.assertQuiet(); } finally { network.restore(); }
|
||||
}
|
||||
}
|
||||
}, 60_000);
|
||||
@@ -0,0 +1,214 @@
|
||||
import { HttpError } from "../../../errors.js";
|
||||
import { generateKeyPairSync } from "node:crypto";
|
||||
import { chatEndpoints, chatSlackRegistrations, type Db } from "@paperclipai/db";
|
||||
import { afterEach, expect, it, vi } from "vitest";
|
||||
import { PaperclipCloudConnectorError } from "../../paperclip-cloud-connector.js";
|
||||
import { createSlackCloudEnrollment, stageSlackCloudEnrollmentIntent } from "./cloud-enrollment.js";
|
||||
import type { InternalSetupState } from "../runtime-types.js";
|
||||
import type { CredentialMutationLeaseGuard } from "../../chat-credential-mutation-lease.js";
|
||||
|
||||
afterEach(() => vi.unstubAllEnvs());
|
||||
function configureCloud() {
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", "https://cloud.example.test");
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", "instance");
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_ENVIRONMENT", "test");
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SIGN_PRIVATE_KEY", generateKeyPairSync("ed25519").privateKey.export({ format: "jwk" }).d!);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SEAL_PRIVATE_KEY", generateKeyPairSync("x25519").privateKey.export({ format: "jwk" }).d!);
|
||||
}
|
||||
function jsonValues(value: unknown): unknown[] {
|
||||
if (typeof value === "string") { try { return [JSON.parse(value)]; } catch { return []; } }
|
||||
if (!value || typeof value !== "object") return [];
|
||||
const chunks = Reflect.get(value, "queryChunks");
|
||||
return Array.isArray(chunks) ? chunks.flatMap(jsonValues) : [];
|
||||
}
|
||||
function fixture() {
|
||||
configureCloud();
|
||||
const endpoint = { id: "00000000-0000-4000-8000-000000000001", companyId: "company", connectionId: "connection", publicId: "a".repeat(43),
|
||||
provider: "slack", status: "active", providerAccountId: "TTEST", setup: { runtimeGeneration: 1 } as InternalSetupState };
|
||||
const connection = { status: "active", enabled: true };
|
||||
let rejectWrite = false;
|
||||
const update = vi.fn((_table: object) => ({ set: (values: { setup: unknown }) => ({ where: () => {
|
||||
const write = () => {
|
||||
if (rejectWrite) return [];
|
||||
const priorIntent = endpoint.setup.slackCloudIntent;
|
||||
for (const value of jsonValues(values.setup)) {
|
||||
if (value && typeof value === "object" && "active" in value && "id" in value) {
|
||||
endpoint.setup.slackCloudIntent = value as NonNullable<InternalSetupState["slackCloudIntent"]>;
|
||||
|
||||
}
|
||||
else if (value && typeof value === "object" && "bindingId" in value) { delete endpoint.setup.slackCloudIntent; delete endpoint.setup.slackCloudRetiredGeneration; endpoint.setup.slackCloudEvents = value as NonNullable<InternalSetupState["slackCloudEvents"]>; endpoint.setup.slackCloudInstanceId = priorIntent!.instanceId; endpoint.setup.slackCloudSubject = priorIntent!.subject; }
|
||||
else if (typeof value === "number") { delete endpoint.setup.slackCloudIntent; endpoint.setup.slackCloudRetiredGeneration = value; endpoint.setup.slackCloudInstanceId = priorIntent!.instanceId; endpoint.setup.slackCloudSubject = priorIntent!.subject; }
|
||||
}
|
||||
return [{ id: endpoint.id }];
|
||||
};
|
||||
return { then: (resolve: (rows: unknown[]) => unknown) => Promise.resolve(write()).then(resolve), returning: async () => write() };
|
||||
} }) }));
|
||||
const db = { update, select: (fields?: Record<string, unknown>) => {
|
||||
let table: object, joined = false;
|
||||
const rows = () => table === chatSlackRegistrations ? [{ appId: "ATEST" }] : joined
|
||||
? [{ endpoint: structuredClone(endpoint), connection: structuredClone(connection) }]
|
||||
: fields?.setup ? [{ setup: structuredClone(endpoint.setup) }] : table === chatEndpoints ? [structuredClone(endpoint)] : [];
|
||||
const query = { from: (value: object) => { table = value; return query; }, innerJoin: () => { joined = true; return query; },
|
||||
where: () => query, limit: async () => rows(), then: (resolve: (rows: unknown[]) => unknown) => Promise.resolve(rows()).then(resolve) };
|
||||
return query;
|
||||
}, transaction: async (work: (tx: unknown) => Promise<unknown>) => {
|
||||
const previous = structuredClone(endpoint.setup);
|
||||
try { return await work(db); } catch (error) { endpoint.setup = previous; throw error; }
|
||||
} } as unknown as Db;
|
||||
const resolveCredentials = vi.fn(async () => ({ signingSecret: "synthetic" }));
|
||||
const connector = { slackEventsAvailable: vi.fn(async () => true), slackEventEnrollment: vi.fn(async (input: { enrollment: Record<string, unknown> }): Promise<unknown> => {
|
||||
const { active: _active, signingSecret: _secret, ...identity } = input.enrollment;
|
||||
return { version: 1, instanceId: "instance", ...identity, ...(input.enrollment.active ? { bindingId: `slack-event:${"b".repeat(64)}` } : {}) };
|
||||
}) };
|
||||
const lease = { assertOwned: vi.fn(async () => {}) } as unknown as CredentialMutationLeaseGuard;
|
||||
const run = createSlackCloudEnrollment(db, { resolveCredentials, runtimeGeneration: setup => Number(setup.runtimeGeneration), connector });
|
||||
const stage = () => db.transaction(tx => stageSlackCloudEnrollmentIntent(tx as unknown as Db, endpoint.id, "original-authorized-user"));
|
||||
return { db, endpoint, connection, update, connector, resolveCredentials, lease, stage, run,
|
||||
rejectWrites: () => { rejectWrite = true; } };
|
||||
}
|
||||
|
||||
it("stages nonsecret responsibility before network and clears only an exactly confirmed current intent", async () => {
|
||||
const f = fixture(); await f.stage();
|
||||
expect(f.connector.slackEventEnrollment).not.toHaveBeenCalled();
|
||||
expect(JSON.stringify(f.endpoint.setup.slackCloudIntent)).not.toContain("synthetic");
|
||||
await f.run(f.endpoint.id, undefined, f.lease);
|
||||
expect(f.endpoint.setup.slackCloudIntent).toBeUndefined();
|
||||
expect(f.endpoint.setup.slackCloudEvents?.runtimeGeneration).toBe(1);
|
||||
expect(f.connector.slackEventEnrollment).toHaveBeenCalledWith(expect.objectContaining({ subject: "original-authorized-user" }));
|
||||
});
|
||||
|
||||
it.each(["paused", "archived"])("retains credential-free %s removal responsibility after local cleanup", async status => {
|
||||
const f = fixture(); f.endpoint.status = status; f.connection.enabled = false; f.endpoint.setup.runtimeGeneration = 2;
|
||||
await f.stage(); expect(f.endpoint.setup.slackCloudIntent?.active).toBe(false);
|
||||
await f.run(f.endpoint.id, undefined, f.lease);
|
||||
expect(f.resolveCredentials).not.toHaveBeenCalled();
|
||||
expect(f.endpoint.setup.slackCloudRetiredGeneration).toBe(2);
|
||||
expect(f.endpoint.setup.slackCloudIntent).toBeUndefined();
|
||||
});
|
||||
|
||||
it("outage preserves a bounded pending retry; current broker denial preserves visible attention debt", async () => {
|
||||
const f = fixture(); await f.stage();
|
||||
f.connector.slackEventsAvailable.mockRejectedValueOnce(new PaperclipCloudConnectorError("offline", "CONNECTOR_UNAVAILABLE"));
|
||||
await f.run(f.endpoint.id, undefined, f.lease);
|
||||
expect(f.endpoint.setup.slackCloudIntent).toMatchObject({ attempts: 1, failureCode: "broker_unavailable" });
|
||||
expect(Date.parse(f.endpoint.setup.slackCloudIntent!.nextAttemptAt!)).toBeGreaterThan(Date.now());
|
||||
f.connector.slackEventEnrollment.mockRejectedValueOnce(new PaperclipCloudConnectorError("denied", "CONNECTOR_REQUEST_FAILED", 403));
|
||||
await f.run(f.endpoint.id, undefined, f.lease);
|
||||
expect(f.endpoint.setup.slackCloudIntent).toMatchObject({ failureCode: "authority_required", nextAttemptAt: null });
|
||||
});
|
||||
|
||||
it("receipt identity mismatch and failed transaction guard cannot publish a live descriptor", async () => {
|
||||
const f = fixture(); await f.stage();
|
||||
f.connector.slackEventEnrollment.mockResolvedValueOnce({ version: 1, runtimeGeneration: 999 });
|
||||
await f.run(f.endpoint.id, undefined, f.lease);
|
||||
expect(f.endpoint.setup.slackCloudEvents).toBeUndefined(); expect(f.endpoint.setup.slackCloudIntent).toBeDefined();
|
||||
f.rejectWrites(); await expect(f.run(f.endpoint.id, undefined, f.lease)).rejects.toThrow("changed");
|
||||
expect(f.endpoint.setup.slackCloudEvents).toBeUndefined(); expect(f.endpoint.setup.slackCloudIntent).toBeDefined();
|
||||
});
|
||||
|
||||
it("an absent Cloud configuration creates no new intent or background retry", async () => {
|
||||
const f = fixture(); for (const key of ["BASE_URL", "INSTANCE_ID", "SIGN_PRIVATE_KEY", "SEAL_PRIVATE_KEY", "ENVIRONMENT"]) vi.stubEnv(`PAPERCLIP_CLOUD_CONNECTOR_${key}`, "");
|
||||
await f.stage(); expect(f.update).not.toHaveBeenCalled(); expect(f.endpoint.setup.slackCloudIntent).toBeUndefined();
|
||||
});
|
||||
|
||||
it.each(["SLACK_ENROLLMENT_GENERATION_CHANGED", "SLACK_ENROLLMENT_DISABLED"])("closed broker reason %s distinguishes permanent authority from rollout retry", async reason => {
|
||||
const f = fixture(); await f.stage();
|
||||
f.connector.slackEventEnrollment.mockRejectedValueOnce(new PaperclipCloudConnectorError("rejected", "REAUTHORIZATION_REQUIRED", 409, undefined, reason));
|
||||
await f.run(f.endpoint.id, undefined, f.lease);
|
||||
expect(f.endpoint.setup.slackCloudEvents).toBeUndefined();
|
||||
expect(f.endpoint.setup.slackCloudIntent!.nextAttemptAt === null).toBe(reason === "SLACK_ENROLLMENT_GENERATION_CHANGED");
|
||||
expect(f.endpoint.setup.slackCloudIntent!.failureCode).toBe(reason === "SLACK_ENROLLMENT_DISABLED" ? "enrollment_disabled" : "authority_required");
|
||||
});
|
||||
|
||||
it("losing Cloud identity retains configured retirement responsibility as attention without an empty retry", async () => {
|
||||
const f = fixture(); await f.stage(); await f.run(f.endpoint.id, undefined, f.lease);
|
||||
f.endpoint.status = "archived"; f.connection.enabled = false; f.endpoint.setup.runtimeGeneration = 2;
|
||||
for (const key of ["BASE_URL", "INSTANCE_ID", "SIGN_PRIVATE_KEY", "SEAL_PRIVATE_KEY", "ENVIRONMENT"]) vi.stubEnv(`PAPERCLIP_CLOUD_CONNECTOR_${key}`, "");
|
||||
await f.stage();
|
||||
expect(f.endpoint.setup.slackCloudIntent).toMatchObject({ active: false, instanceId: "instance", runtimeGeneration: 2,
|
||||
failureCode: "connector_configuration_unavailable", nextAttemptAt: null });
|
||||
expect(f.connector.slackEventEnrollment).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it.each([404, 422])("unavailable signing credential %s becomes retained attention instead of broker retry", async status => {
|
||||
const f = fixture(); await f.stage();
|
||||
f.resolveCredentials.mockRejectedValueOnce(new HttpError(status, "synthetic credential failure"));
|
||||
await f.run(f.endpoint.id, undefined, f.lease);
|
||||
expect(f.endpoint.setup.slackCloudIntent).toMatchObject({ failureCode: "signing_credential_unavailable", nextAttemptAt: null });
|
||||
expect(f.connector.slackEventEnrollment).not.toHaveBeenCalled();
|
||||
expect(f.endpoint.setup.slackCloudEvents).toBeUndefined();
|
||||
});
|
||||
|
||||
it("temporary signing-credential store failure keeps the bounded retry", async () => {
|
||||
const f = fixture(); await f.stage();
|
||||
f.resolveCredentials.mockRejectedValueOnce(new Error("temporary database failure"));
|
||||
await f.run(f.endpoint.id, undefined, f.lease);
|
||||
expect(f.endpoint.setup.slackCloudIntent?.failureCode).toBe("broker_unavailable");
|
||||
expect(Date.parse(f.endpoint.setup.slackCloudIntent!.nextAttemptAt!)).toBeGreaterThan(Date.now());
|
||||
expect(f.connector.slackEventEnrollment).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("only positive old retirement and an explicit new generation can stage a replacement instance", async () => {
|
||||
const f = fixture(); await f.stage(); await f.run(f.endpoint.id, undefined, f.lease);
|
||||
const oldDescriptor = structuredClone(f.endpoint.setup.slackCloudEvents);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", "replacement");
|
||||
f.endpoint.setup.runtimeGeneration = 2;
|
||||
await f.stage();
|
||||
expect(f.endpoint.setup.slackCloudIntent).toMatchObject({ instanceId: "instance", subject: "original-authorized-user",
|
||||
failureCode: "instance_replacement_requires_retirement", nextAttemptAt: null });
|
||||
expect(f.endpoint.setup.slackCloudEvents).toEqual(oldDescriptor);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", "instance");
|
||||
f.endpoint.status = "paused"; f.connection.enabled = false; f.endpoint.setup.runtimeGeneration = 3;
|
||||
await f.stage(); await f.run(f.endpoint.id, undefined, f.lease);
|
||||
expect(f.endpoint.setup.slackCloudRetiredGeneration).toBe(3);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", "replacement");
|
||||
f.endpoint.status = "active"; f.connection.enabled = true; f.endpoint.setup.runtimeGeneration = 4;
|
||||
await f.stage();
|
||||
expect(f.endpoint.setup.slackCloudIntent).toMatchObject({ instanceId: "replacement", runtimeGeneration: 4 });
|
||||
expect(f.endpoint.setup.slackCloudInstanceId).toBe("instance");
|
||||
expect(f.endpoint.setup.slackCloudSubject).toBe("original-authorized-user");
|
||||
expect(f.endpoint.setup.slackCloudEvents).toEqual(oldDescriptor);
|
||||
f.connector.slackEventEnrollment.mockImplementationOnce(async input => {
|
||||
const { active: _active, signingSecret: _secret, ...identity } = input.enrollment;
|
||||
return { version: 1, instanceId: "replacement", ...identity, bindingId: `slack-event:${"c".repeat(64)}` };
|
||||
});
|
||||
await f.run(f.endpoint.id, undefined, f.lease);
|
||||
expect(f.endpoint.setup.slackCloudInstanceId).toBe("replacement");
|
||||
expect(f.endpoint.setup.slackCloudIntent).toBeUndefined();
|
||||
expect(f.endpoint.setup.slackCloudRetiredGeneration).toBeUndefined();
|
||||
expect(f.endpoint.setup.slackCloudEvents?.runtimeGeneration).toBe(4);
|
||||
});
|
||||
|
||||
it("a new config without an explicit actor cannot adopt even a retired old instance", async () => {
|
||||
const f = fixture(); await f.stage(); await f.run(f.endpoint.id, undefined, f.lease);
|
||||
f.endpoint.status = "paused"; f.connection.enabled = false; f.endpoint.setup.runtimeGeneration = 2;
|
||||
await f.stage(); await f.run(f.endpoint.id, undefined, f.lease);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", "replacement");
|
||||
f.endpoint.status = "active"; f.connection.enabled = true; f.endpoint.setup.runtimeGeneration = 3;
|
||||
await stageSlackCloudEnrollmentIntent(f.db, f.endpoint.id, undefined);
|
||||
expect(f.endpoint.setup.slackCloudIntent).toMatchObject({ instanceId: "instance", subject: "original-authorized-user",
|
||||
nextAttemptAt: null, failureCode: "instance_replacement_requires_retirement" });
|
||||
});
|
||||
|
||||
it("removal after an unconfirmed replacement targets its pending instance without new credentials", async () => {
|
||||
const f = fixture(); await f.stage(); await f.run(f.endpoint.id, undefined, f.lease);
|
||||
f.endpoint.status = "paused"; f.connection.enabled = false; f.endpoint.setup.runtimeGeneration = 2;
|
||||
await f.stage(); await f.run(f.endpoint.id, undefined, f.lease);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", "replacement");
|
||||
f.endpoint.status = "active"; f.connection.enabled = true; f.endpoint.setup.runtimeGeneration = 3;
|
||||
await f.stage();
|
||||
expect(f.endpoint.setup.slackCloudInstanceId).toBe("instance");
|
||||
f.endpoint.status = "archived"; f.connection.enabled = false; f.endpoint.setup.runtimeGeneration = 4;
|
||||
await f.stage();
|
||||
expect(f.endpoint.setup.slackCloudIntent).toMatchObject({ active: false, instanceId: "replacement", runtimeGeneration: 4 });
|
||||
f.resolveCredentials.mockClear();
|
||||
f.connector.slackEventEnrollment.mockImplementationOnce(async input => {
|
||||
const { active: _active, ...identity } = input.enrollment;
|
||||
return { version: 1, instanceId: "replacement", ...identity };
|
||||
});
|
||||
await f.run(f.endpoint.id, undefined, f.lease);
|
||||
expect(f.resolveCredentials).not.toHaveBeenCalled();
|
||||
expect(f.endpoint.setup.slackCloudIntent).toBeUndefined();
|
||||
expect(f.endpoint.setup.slackCloudInstanceId).toBe("replacement");
|
||||
expect(f.endpoint.setup.slackCloudRetiredGeneration).toBe(4);
|
||||
});
|
||||
@@ -0,0 +1,218 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq, or, sql } from "drizzle-orm";
|
||||
import { chatEndpoints, chatSlackRegistrations, toolConnections, type Db } from "@paperclipai/db";
|
||||
import { createPaperclipCloudConnector, paperclipCloudConnectorConfigFromEnv, PaperclipCloudConnectorError, type PaperclipCloudConnector } from "../../paperclip-cloud-connector.js";
|
||||
import type { CredentialMutationLeaseGuard } from "../../chat-credential-mutation-lease.js";
|
||||
import type { createChatCredentials } from "../credentials.js";
|
||||
import type { InternalSetupState } from "../runtime-types.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../delivery-work-notifications.js";
|
||||
import { logger } from "../../../middleware/logger.js";
|
||||
import { HttpError } from "../../../errors.js";
|
||||
|
||||
import { resolveSlackCloudSigningSecret } from "./cloud-signing-secret.js";
|
||||
import { SlackCloudReceiptError } from "./cloud-event.js";
|
||||
|
||||
type Intent = NonNullable<InternalSetupState["slackCloudIntent"]>;
|
||||
class SlackEnrollmentFailure extends Error {
|
||||
constructor(readonly code: string) { super(code); }
|
||||
}
|
||||
|
||||
type Owner = (endpointId: string) => Promise<void>;
|
||||
const owners = new WeakMap<Db, Owner>();
|
||||
|
||||
export function registerSlackCloudEnrollmentOwner(db: Db, owner: Owner) {
|
||||
owners.set(db, owner);
|
||||
return () => { if (owners.get(db) === owner) owners.delete(db); };
|
||||
}
|
||||
|
||||
function slackCloudEnrollmentIsActive(endpoint: { status: string }, connection: { status: string; enabled: boolean }) {
|
||||
return ["active", "verifying", "attention"].includes(endpoint.status) && connection.status === "active" && connection.enabled;
|
||||
}
|
||||
|
||||
export async function stageSlackCloudEnrollmentIntent(db: Db, endpointId: string, actorUserId: string | null | undefined) {
|
||||
let config: ReturnType<typeof paperclipCloudConnectorConfigFromEnv>;
|
||||
try { config = paperclipCloudConnectorConfigFromEnv(); } catch { config = null; }
|
||||
const [current] = await db.select({ endpoint: chatEndpoints, connection: toolConnections }).from(chatEndpoints)
|
||||
.innerJoin(toolConnections, and(eq(toolConnections.id, chatEndpoints.connectionId), eq(toolConnections.companyId, chatEndpoints.companyId)))
|
||||
.where(eq(chatEndpoints.id, endpointId)).limit(1);
|
||||
if (!current || current.endpoint.provider !== "slack") return;
|
||||
const endpoint = current.endpoint, setup = endpoint.setup as InternalSetupState;
|
||||
const generation = Number(setup.runtimeGeneration ?? 0);
|
||||
const active = slackCloudEnrollmentIsActive(endpoint, current.connection);
|
||||
const custodyInstance = setup.slackCloudIntent?.instanceId ?? setup.slackCloudInstanceId;
|
||||
const custodySubject = setup.slackCloudIntent?.subject ?? setup.slackCloudSubject;
|
||||
const oldRetired = !setup.slackCloudIntent && setup.slackCloudInstanceId
|
||||
&& Number.isSafeInteger(setup.slackCloudRetiredGeneration) && Number(setup.slackCloudRetiredGeneration) >= 1
|
||||
&& Number(setup.slackCloudRetiredGeneration) >= (setup.slackCloudEvents?.runtimeGeneration ?? 1);
|
||||
const changingInstance = !!custodyInstance && !!config && custodyInstance !== config.instanceId;
|
||||
const mayReplace = active && changingInstance && oldRetired && !!actorUserId
|
||||
&& generation > Number(setup.slackCloudRetiredGeneration);
|
||||
const blocked = changingInstance && !mayReplace;
|
||||
const instanceId = mayReplace ? config!.instanceId : custodyInstance ?? config?.instanceId;
|
||||
const subject = blocked || !active ? custodySubject ?? actorUserId : actorUserId ?? custodySubject;
|
||||
if (!instanceId || !subject || !Number.isSafeInteger(generation) || generation < 1) return;
|
||||
if (setup.slackCloudIntent?.runtimeGeneration === generation && setup.slackCloudIntent.active === active
|
||||
&& setup.slackCloudIntent.instanceId === instanceId && setup.slackCloudIntent.subject === subject) return;
|
||||
if (!blocked && setup.slackCloudInstanceId === instanceId && setup.slackCloudSubject === subject
|
||||
&& (active ? setup.slackCloudEvents?.runtimeGeneration === generation : setup.slackCloudRetiredGeneration === generation)) return;
|
||||
const [registration] = active ? await db.select({ appId: chatSlackRegistrations.appId }).from(chatSlackRegistrations)
|
||||
.where(and(eq(chatSlackRegistrations.endpointId, endpointId), eq(chatSlackRegistrations.companyId, endpoint.companyId))).limit(1) : [];
|
||||
const appId = registration?.appId ?? setup.slackCloudEvents?.appId;
|
||||
if (active && (!appId || !endpoint.providerAccountId)) return;
|
||||
const intent: Intent = { version: 1, id: randomUUID(), subject, instanceId, companyId: endpoint.companyId, endpointId,
|
||||
publicId: endpoint.publicId, connectionId: endpoint.connectionId, runtimeGeneration: generation, active,
|
||||
...(active ? { appId: appId!, teamId: endpoint.providerAccountId! } : {}), attempts: 0, nextAttemptAt: config && !blocked ? new Date().toISOString() : null,
|
||||
...(blocked ? { failureCode: "instance_replacement_requires_retirement" } : !config ? { failureCode: "connector_configuration_unavailable" } : {}) };
|
||||
await notifyDeliveryWork(db, DELIVERY_QUEUES.connectionEvents);
|
||||
await db.update(chatEndpoints).set({
|
||||
lastError: config && !blocked ? sql`${chatEndpoints.lastError}` : "Slack Cloud event registration requires attention",
|
||||
setup: sql`jsonb_set(${chatEndpoints.setup}, '{slackCloudIntent}', ${JSON.stringify(intent)}::jsonb)` })
|
||||
.where(and(eq(chatEndpoints.id, endpointId), eq(chatEndpoints.companyId, endpoint.companyId)));
|
||||
}
|
||||
|
||||
export async function reconcilePendingSlackCloudEnrollments(db: Db, now: Date): Promise<number | null> {
|
||||
const config = paperclipCloudConnectorConfigFromEnv();
|
||||
if (!config) return null;
|
||||
const recoverable = and(
|
||||
sql`${chatEndpoints.setup}->'slackCloudIntent'->>'nextAttemptAt' is null`,
|
||||
sql`${chatEndpoints.setup}->'slackCloudIntent'->>'failureCode' = 'connector_configuration_unavailable'`,
|
||||
sql`${chatEndpoints.setup}->'slackCloudIntent'->>'instanceId' = ${config.instanceId}`,
|
||||
);
|
||||
const scheduled = and(eq(chatEndpoints.provider, "slack"), or(
|
||||
sql`${chatEndpoints.setup}->'slackCloudIntent'->>'nextAttemptAt' is not null`, recoverable));
|
||||
const order = sql`coalesce(${chatEndpoints.setup}->'slackCloudIntent'->>'nextAttemptAt', '')`;
|
||||
const pending = await db.select().from(chatEndpoints).where(scheduled).orderBy(order).limit(1);
|
||||
const owner = owners.get(db);
|
||||
const deadline = (intent: Intent | undefined) => {
|
||||
if (!intent) return Number.POSITIVE_INFINITY;
|
||||
if (intent.nextAttemptAt === null) return intent.failureCode === "connector_configuration_unavailable" && intent.instanceId === config.instanceId
|
||||
? now.getTime() : Number.POSITIVE_INFINITY;
|
||||
const parsed = Date.parse(intent.nextAttemptAt);
|
||||
return Number.isFinite(parsed) ? parsed : now.getTime();
|
||||
};
|
||||
let next = Number.POSITIVE_INFINITY;
|
||||
for (const endpoint of pending) {
|
||||
const intent = (endpoint.setup as InternalSetupState).slackCloudIntent!;
|
||||
let due = deadline(intent);
|
||||
if (!Number.isFinite(due)) continue;
|
||||
if (due <= now.getTime()) {
|
||||
if (owner) {
|
||||
try { await owner(endpoint.id); }
|
||||
catch (error) {
|
||||
if (error instanceof HttpError && error.status === 409 && error.details && typeof error.details === "object"
|
||||
&& "code" in error.details && error.details.code === "chat_endpoint_credentials_busy") {
|
||||
try {
|
||||
if (intent.companyId === endpoint.companyId && intent.endpointId === endpoint.id
|
||||
&& intent.publicId === endpoint.publicId && intent.connectionId === endpoint.connectionId
|
||||
&& intent.runtimeGeneration === Number((endpoint.setup as InternalSetupState).runtimeGeneration ?? 0)) {
|
||||
await db.transaction(async tx => {
|
||||
const sameIntent = and(eq(chatEndpoints.id, endpoint.id), eq(chatEndpoints.companyId, intent.companyId),
|
||||
eq(chatEndpoints.provider, "slack"), eq(chatEndpoints.publicId, intent.publicId), eq(chatEndpoints.connectionId, intent.connectionId),
|
||||
sql`${chatEndpoints.setup}->'runtimeGeneration' = ${JSON.stringify(intent.runtimeGeneration)}::jsonb`,
|
||||
sql`${chatEndpoints.setup}->'slackCloudIntent' = ${JSON.stringify(intent)}::jsonb`);
|
||||
const [owned] = await tx.select({ id: chatEndpoints.id }).from(chatEndpoints).where(sameIntent).for("update").limit(1);
|
||||
if (!owned) return;
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.connectionEvents);
|
||||
const retry = { ...intent, attempts: Math.min(intent.attempts + 1, 1_000_000), failureCode: "credential_lease_busy",
|
||||
nextAttemptAt: new Date(Date.now() + Math.min(300_000, 5_000 * 2 ** Math.min(intent.attempts, 6))).toISOString() };
|
||||
const updated = await tx.update(chatEndpoints).set({
|
||||
setup: sql`jsonb_set(${chatEndpoints.setup}, '{slackCloudIntent}', ${JSON.stringify(retry)}::jsonb)`,
|
||||
}).where(sameIntent).returning({ id: chatEndpoints.id });
|
||||
if (updated.length !== 1) throw new Error("Slack enrollment busy retry custody changed");
|
||||
});
|
||||
}
|
||||
} catch (retryError) { throw new AggregateError([error, retryError], "Slack enrollment busy retry persistence failed"); }
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
else logger.warn({ endpointId: endpoint.id, reason: "slack_enrollment_owner_not_ready" }, "Slack enrollment responsibility retained");
|
||||
const [updated] = await db.select({ setup: chatEndpoints.setup }).from(chatEndpoints).where(eq(chatEndpoints.id, endpoint.id)).limit(1);
|
||||
const retained = (updated?.setup as InternalSetupState | undefined)?.slackCloudIntent;
|
||||
due = deadline(retained);
|
||||
if (!owner) due = now.getTime() + 5_000;
|
||||
}
|
||||
next = Math.min(next, due);
|
||||
}
|
||||
if (pending.length > 0) {
|
||||
const [earliest] = await db.select({ setup: chatEndpoints.setup }).from(chatEndpoints).where(scheduled).orderBy(order).limit(1);
|
||||
next = Math.min(next, deadline((earliest?.setup as InternalSetupState | undefined)?.slackCloudIntent));
|
||||
}
|
||||
return Number.isFinite(next) ? next : null;
|
||||
}
|
||||
|
||||
export function createSlackCloudEnrollment(db: Db, options: {
|
||||
resolveCredentials: ReturnType<typeof createChatCredentials>["resolveCredentials"];
|
||||
runtimeGeneration(setup: InternalSetupState): number;
|
||||
connector?: Pick<PaperclipCloudConnector, "slackEventsAvailable" | "slackEventEnrollment">;
|
||||
}) {
|
||||
return async (endpointId: string, _actorUserId: string | null | undefined, lease: CredentialMutationLeaseGuard) => {
|
||||
const [current] = await db.select({ endpoint: chatEndpoints, connection: toolConnections }).from(chatEndpoints)
|
||||
.innerJoin(toolConnections, and(eq(toolConnections.id, chatEndpoints.connectionId), eq(toolConnections.companyId, chatEndpoints.companyId)))
|
||||
.where(eq(chatEndpoints.id, endpointId)).limit(1);
|
||||
if (!current || current.endpoint.provider !== "slack") return;
|
||||
const endpoint = current.endpoint, setup = endpoint.setup as InternalSetupState, intent = setup.slackCloudIntent;
|
||||
const config = paperclipCloudConnectorConfigFromEnv();
|
||||
if (!config || !intent) return;
|
||||
const identity = { companyId: intent.companyId, endpointId: intent.endpointId, publicId: intent.publicId,
|
||||
connectionId: intent.connectionId, runtimeGeneration: intent.runtimeGeneration };
|
||||
const active = slackCloudEnrollmentIsActive(endpoint, current.connection);
|
||||
if (identity.companyId !== endpoint.companyId || identity.endpointId !== endpoint.id || identity.publicId !== endpoint.publicId
|
||||
|| identity.connectionId !== endpoint.connectionId || identity.runtimeGeneration !== options.runtimeGeneration(setup) || intent.active !== active) {
|
||||
await db.transaction(async tx => {
|
||||
await lease.assertOwned(tx);
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.connectionEvents);
|
||||
await tx.update(chatEndpoints).set({
|
||||
setup: sql`jsonb_set(${chatEndpoints.setup}, '{slackCloudIntent}', ${JSON.stringify({ ...intent, nextAttemptAt: null, failureCode: "intent_authority_changed" })}::jsonb)`,
|
||||
lastError: "Slack Cloud event registration requires attention",
|
||||
}).where(and(eq(chatEndpoints.id, endpoint.id), eq(chatEndpoints.companyId, endpoint.companyId), sql`${chatEndpoints.setup}->'slackCloudIntent'->>'id'=${intent.id}`));
|
||||
await lease.assertOwned(tx);
|
||||
});
|
||||
logger.warn({ endpointId, reason: "intent_authority_changed" }, "Slack enrollment responsibility retained");
|
||||
return;
|
||||
}
|
||||
const connector = options.connector ?? createPaperclipCloudConnector({ config });
|
||||
const commit = async (change: { receipt?: Record<string, unknown>; failure?: string; attention?: boolean; retryAt?: number }) => db.transaction(async tx => {
|
||||
await lease.assertOwned(tx);
|
||||
const next = change.failure ? { ...intent, attempts: Math.min(intent.attempts + 1, 1_000_000), failureCode: change.failure,
|
||||
nextAttemptAt: change.attention ? null : new Date(Math.max(Date.now() + Math.min(300_000, 5_000 * 2 ** Math.min(intent.attempts, 6)), change.retryAt ?? 0)).toISOString() } : null;
|
||||
const field = intent.active ? "slackCloudEvents" : "slackCloudRetiredGeneration";
|
||||
const enrolled = intent.active ? { version: 1, bindingId: change.receipt?.bindingId, runtimeGeneration: intent.runtimeGeneration, teamId: intent.teamId, appId: intent.appId } : intent.runtimeGeneration;
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.connectionEvents);
|
||||
const updated = await tx.update(chatEndpoints).set({
|
||||
lastError: change.attention ? "Slack Cloud event registration requires attention" : change.receipt
|
||||
? sql`case when ${chatEndpoints.lastError} = 'Slack Cloud event registration requires attention' then null else ${chatEndpoints.lastError} end` : sql`${chatEndpoints.lastError}`,
|
||||
setup: next
|
||||
? sql`jsonb_set(${chatEndpoints.setup}, '{slackCloudIntent}', ${JSON.stringify(next)}::jsonb)`
|
||||
: sql`jsonb_set(jsonb_set(jsonb_set(${chatEndpoints.setup} - 'slackCloudIntent' - 'slackCloudRetiredGeneration', ARRAY[${field}], ${JSON.stringify(enrolled)}::jsonb), '{slackCloudInstanceId}', ${JSON.stringify(intent.instanceId)}::jsonb), '{slackCloudSubject}', ${JSON.stringify(intent.subject)}::jsonb)` })
|
||||
.where(and(eq(chatEndpoints.id, endpointId), eq(chatEndpoints.companyId, endpoint.companyId),
|
||||
sql`${chatEndpoints.setup}->'slackCloudIntent'->>'id'=${intent.id}`,
|
||||
sql`coalesce((${chatEndpoints.setup}->>'runtimeGeneration')::integer,0)=${intent.runtimeGeneration}`)).returning({ id: chatEndpoints.id });
|
||||
if (updated.length !== 1) throw new Error("Slack configuration changed before event enrollment committed");
|
||||
await lease.assertOwned(tx);
|
||||
});
|
||||
try {
|
||||
await lease.assertOwned();
|
||||
if (config.instanceId !== intent.instanceId) { await commit({ failure: "instance_authority_changed", attention: true }); return; }
|
||||
if (!await connector.slackEventsAvailable()) { await commit({ failure: "unsupported_capability" }); return; }
|
||||
const signingSecret = intent.active ? await resolveSlackCloudSigningSecret(options.resolveCredentials, endpoint, db, ["signingSecret"]) : undefined;
|
||||
const receipt = await connector.slackEventEnrollment({ subject: intent.subject, companyId: intent.companyId,
|
||||
enrollment: { active: intent.active, ...identity, ...(intent.active ? { teamId: intent.teamId, appId: intent.appId, signingSecret: signingSecret! } : {}) } });
|
||||
if (!receipt || typeof receipt !== "object" || Array.isArray(receipt)) throw new SlackEnrollmentFailure("receipt_unconfirmed");
|
||||
const value = receipt as Record<string, unknown>;
|
||||
if (value.version !== 1 || value.instanceId !== intent.instanceId || Object.entries(identity).some(([key, expected]) => value[key] !== expected)
|
||||
|| (intent.active && (value.teamId !== intent.teamId || value.appId !== intent.appId || typeof value.bindingId !== "string" || !/^slack-event:[a-f0-9]{64}$/.test(value.bindingId))))
|
||||
throw new SlackEnrollmentFailure("receipt_identity_changed");
|
||||
await commit({ receipt: value });
|
||||
} catch (error) {
|
||||
const conflict = error instanceof PaperclipCloudConnectorError && error.status === 409
|
||||
&& ["SLACK_ENROLLMENT_GENERATION_CHANGED", "SLACK_ENROLLMENT_IDENTITY_CHANGED", "SLACK_ENROLLMENT_CHANGED", "SLACK_ENROLLMENT_UNSETTLED"].includes(error.brokerReason ?? "");
|
||||
const credentialAttention = error instanceof SlackCloudReceiptError && error.code === "signing_credential_unavailable" && error.disposition === "attention";
|
||||
const attention = credentialAttention || error instanceof SlackEnrollmentFailure || conflict || (error instanceof PaperclipCloudConnectorError && [400, 401, 403].includes(error.status ?? 0));
|
||||
const code = credentialAttention ? "signing_credential_unavailable" : error instanceof SlackEnrollmentFailure ? error.code : attention ? "authority_required"
|
||||
: error instanceof PaperclipCloudConnectorError && error.brokerReason === "SLACK_ENROLLMENT_DISABLED" ? "enrollment_disabled" : "broker_unavailable";
|
||||
await commit({ failure: code, attention, retryAt: error instanceof PaperclipCloudConnectorError ? error.retryAt : undefined });
|
||||
logger.warn({ endpointId, reason: code }, "Slack enrollment responsibility retained");
|
||||
}
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
import { guardSlackFixtureNetwork, stubSlackFixtureTransport } from "../../../__tests__/helpers/slack-cloud/transport.js";
|
||||
import { createHmac, createHash } from "node:crypto";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { verifySlackSignature } from "@chat-adapter/slack/webhook";
|
||||
import { createChatSdkEndpointRuntime } from "../../chat-sdk-runtime.js";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import type { EndpointRow } from "../runtime-types.js";
|
||||
import { createSlackWebhookCallbacks } from "./webhook-callbacks.js";
|
||||
import { receiveSlackCloudEvent, slackCloudAcceptedAt, type CurrentSlackCloudBinding, type SlackCloudEvent } from "./cloud-event.js";
|
||||
|
||||
const acceptedAt = Date.parse("2099-01-01T00:00:00Z");
|
||||
function fixture() {
|
||||
const binding: CurrentSlackCloudBinding = { companyId: "company", endpointId: "endpoint", publicId: "a".repeat(43),
|
||||
connectionId: "connection", runtimeGeneration: 1, teamId: "TTEST", appId: "ATEST", bindingId: `slack-event:${"b".repeat(64)}`,
|
||||
signingSecret: "0123456789abcdef0123456789abcdef" };
|
||||
const rawBody = Buffer.from(JSON.stringify({ type: "event_callback", event_id: "EvFIRST", team_id: "TTEST", api_app_id: "ATEST", event: { type: "app_mention", text: "<@UBOT> hello", user: "UOPERATOR", channel: "CTEST", ts: "1788.500", event_ts: "1788.500" } }));
|
||||
const timestamp = String(acceptedAt / 1000);
|
||||
const signature = `v0=${createHmac("sha256", binding.signingSecret).update(`v0:${timestamp}:`).update(rawBody).digest("hex")}`;
|
||||
const { signingSecret: _secret, ...identity } = binding;
|
||||
const event: SlackCloudEvent = { id: `slack_${"b".repeat(64)}_${createHash("sha256").update("EvFIRST").digest("hex")}`,
|
||||
provider: "slack", event: "event_callback", installationId: binding.bindingId, bindingIds: [binding.bindingId],
|
||||
createdAt: new Date(acceptedAt).toISOString(), payload: { slackEvent: { version: 1, ...identity,
|
||||
rawBody: rawBody.toString("base64"), timestamp, signature, eventId: "EvFIRST", acceptedAt: new Date(acceptedAt).toISOString() } } };
|
||||
return { binding, event, rawBody, timestamp, signature };
|
||||
}
|
||||
|
||||
describe("authenticated Slack Cloud receipt consumer", () => {
|
||||
it("crosses the real pinned Chat runtime after cold-start delay and preserves ordinary external freshness rejection", async () => {
|
||||
const network = guardSlackFixtureNetwork();
|
||||
const f = fixture(), onMessage = vi.fn(async () => {});
|
||||
const runtime = createChatSdkEndpointRuntime({ companyId: f.binding.companyId, endpointId: f.binding.endpointId,
|
||||
logger: "silent", callbacks: { onMessage }, persistence: { read: async () => null,
|
||||
compareAndSet: async () => true, deleteIfVersion: async () => true },
|
||||
providerConfig: { provider: "slack", userName: "paperclip-agent",
|
||||
credentials: { apiUrl: "http://127.0.0.1:1/api/", botToken: "xoxb-synthetic", botUserId: "UBOT", signingSecret: f.binding.signingSecret } } });
|
||||
try {
|
||||
await runtime.initialize();
|
||||
const transport = stubSlackFixtureTransport(runtime.getProviderAdapter());
|
||||
await receiveSlackCloudEvent(f.event, f.binding, request => runtime.handleWebhook(request), acceptedAt + 600_000);
|
||||
expect(onMessage).toHaveBeenCalledOnce();
|
||||
expect(onMessage).toHaveBeenCalledWith(expect.objectContaining({ message: expect.objectContaining({ author: expect.objectContaining({ userId: "UOPERATOR", userName: "Operator" }) }) }));
|
||||
expect(transport).toHaveBeenCalled();
|
||||
await transport.assertNoErrors();
|
||||
expect(onMessage).toHaveBeenCalledWith(expect.objectContaining({ provider: "slack", endpointId: f.binding.endpointId }));
|
||||
const response = await runtime.handleWebhook(new Request("https://external.invalid", { method: "POST",
|
||||
headers: { "content-type": "application/json", "x-slack-request-timestamp": f.timestamp, "x-slack-signature": f.signature },
|
||||
body: new Uint8Array(f.rawBody) }));
|
||||
expect(response.status).toBe(401);
|
||||
expect(onMessage).toHaveBeenCalledOnce();
|
||||
} finally {
|
||||
await runtime.shutdown();
|
||||
try { network.assertQuiet(); } finally { network.restore(); }
|
||||
}
|
||||
});
|
||||
it("replays original evidence after a cold-start delay without refreshing its timestamp", async () => {
|
||||
const f = fixture();
|
||||
let delivered: Request | undefined;
|
||||
await receiveSlackCloudEvent(f.event, f.binding, async request => {
|
||||
delivered = request;
|
||||
expect(slackCloudAcceptedAt(request)).toBe(acceptedAt);
|
||||
expect(request.headers.get("x-slack-request-timestamp")).toBe(f.timestamp);
|
||||
expect(request.headers.get("x-slack-signature")).toBe(f.signature);
|
||||
const body = await request.text();
|
||||
expect(body).toBe(f.rawBody.toString());
|
||||
|
||||
await verifySlackSignature(body, request.headers, { signingSecret: f.binding.signingSecret,
|
||||
now: () => slackCloudAcceptedAt(request) ?? acceptedAt + 600_000 });
|
||||
await expect(verifySlackSignature(body, request.headers, { signingSecret: f.binding.signingSecret,
|
||||
now: () => acceptedAt + 600_000 })).rejects.toThrow("too old");
|
||||
return new Response("durably accepted");
|
||||
}, acceptedAt + 600_000);
|
||||
expect(slackCloudAcceptedAt(delivered!)).toBeUndefined();
|
||||
expect(slackCloudAcceptedAt(new Request("https://external.invalid"))).toBeUndefined();
|
||||
});
|
||||
|
||||
it("ignores removed or superseded generations without invoking product admission", async () => {
|
||||
const f = fixture(), handler = vi.fn();
|
||||
await receiveSlackCloudEvent(f.event, null, handler, acceptedAt);
|
||||
await receiveSlackCloudEvent(f.event, { ...f.binding, runtimeGeneration: 2 }, handler, acceptedAt);
|
||||
expect(handler).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("retains current work on signature/body/receipt-time failure and on failed durable admission", async () => {
|
||||
const f = fixture(), handler = vi.fn(async () => new Response(null, { status: 503 }));
|
||||
await expect(receiveSlackCloudEvent(f.event, f.binding, handler, acceptedAt)).rejects.toThrow("callback_unaccepted");
|
||||
await expect(receiveSlackCloudEvent(f.event, { ...f.binding, signingSecret: "rotated" }, handler, acceptedAt)).rejects.toThrow("original_authentication_failed");
|
||||
await expect(receiveSlackCloudEvent({ ...f.event, createdAt: new Date(acceptedAt + 600_000).toISOString() }, f.binding, handler, acceptedAt)).rejects.toThrow("invalid_acceptance_time");
|
||||
await expect(receiveSlackCloudEvent(f.event, f.binding, handler, acceptedAt + 31 * 24 * 60 * 60_000)).rejects.toThrow("invalid_acceptance_time");
|
||||
expect(handler).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
it("private replay preserves canonical Slack callback evidence while ordinary external observation remains active", async () => {
|
||||
const f = fixture();
|
||||
const transaction = vi.fn(async () => { throw new Error("external callback observation"); });
|
||||
const db = { transaction } as unknown as Db;
|
||||
const callbacks = createSlackWebhookCallbacks(db, { resolveCredentials: async () => ({}), runtimeCallbackEndpoint: async () => null,
|
||||
getWebhookPublicBaseUrl: () => "https://canonical.example", scheduleMessageProcessing: () => {}, slackRegistration: { notifyVerified: async () => {} } });
|
||||
const endpoint = { setup: { slackCallbackSurfaces: { events: { url: "https://canonical.example/slack", observedAt: "2099-01-01T00:00:00Z" } } } } as unknown as EndpointRow;
|
||||
const before = structuredClone(endpoint.setup);
|
||||
const context = { credentialFingerprint: "fixture", generation: 1, localEpoch: 1, version: "fixture" };
|
||||
await receiveSlackCloudEvent(f.event, f.binding, async request => {
|
||||
await callbacks.recordSlackWebhookCallback(endpoint, request, request.clone(), context);
|
||||
return new Response(null, { status: 200 });
|
||||
}, acceptedAt);
|
||||
expect(endpoint.setup).toEqual(before); expect(transaction).not.toHaveBeenCalled();
|
||||
const external = new Request("https://canonical.example/slack", { method: "POST", headers: { "content-type": "application/json" }, body: new Uint8Array(f.rawBody) });
|
||||
await expect(callbacks.recordSlackWebhookCallback(endpoint, external, external.clone(), context)).rejects.toThrow("external callback observation");
|
||||
expect(transaction).toHaveBeenCalledOnce();
|
||||
});
|
||||
@@ -0,0 +1,77 @@
|
||||
import { createHash, createHmac, timingSafeEqual } from "node:crypto";
|
||||
|
||||
const acceptedTimes = new WeakMap<Request, number>();
|
||||
export const slackCloudAcceptedAt = (request: Request): number | undefined => acceptedTimes.get(request);
|
||||
|
||||
export class SlackCloudReceiptError extends Error {
|
||||
constructor(readonly code: string, readonly disposition: "invalid" | "attention" | "retry") { super(code); }
|
||||
}
|
||||
|
||||
export interface SlackCloudEvent {
|
||||
id: string;
|
||||
provider: string;
|
||||
event: string;
|
||||
installationId: string | null;
|
||||
bindingIds: string[];
|
||||
createdAt: string;
|
||||
payload: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface CurrentSlackCloudBinding {
|
||||
companyId: string;
|
||||
endpointId: string;
|
||||
publicId: string;
|
||||
connectionId: string;
|
||||
runtimeGeneration: number;
|
||||
teamId: string;
|
||||
appId: string;
|
||||
bindingId: string;
|
||||
signingSecret: string;
|
||||
}
|
||||
|
||||
export async function receiveSlackCloudEvent(event: SlackCloudEvent, current: CurrentSlackCloudBinding | null,
|
||||
handleWebhook: (request: Request) => Promise<Response>, now = Date.now()): Promise<void> {
|
||||
if (!current) return;
|
||||
const value = event.payload.slackEvent;
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) throw new SlackCloudReceiptError("invalid_receipt", "invalid");
|
||||
const packet = value as Record<string, unknown>;
|
||||
if (event.provider !== "slack" || event.event !== "event_callback" || packet.version !== 1)
|
||||
throw new SlackCloudReceiptError("unsupported_receipt", "attention");
|
||||
if (packet.bindingId !== current.bindingId || packet.companyId !== current.companyId
|
||||
|| packet.endpointId !== current.endpointId || packet.publicId !== current.publicId
|
||||
|| packet.connectionId !== current.connectionId || packet.runtimeGeneration !== current.runtimeGeneration
|
||||
|| packet.teamId !== current.teamId || packet.appId !== current.appId) return;
|
||||
if (event.installationId !== current.bindingId || !event.bindingIds.includes(current.bindingId)
|
||||
|| typeof packet.rawBody !== "string" || packet.rawBody.length > 1_398_104
|
||||
|| !/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/.test(packet.rawBody)
|
||||
|| typeof packet.timestamp !== "string" || !/^[0-9]{1,16}$/.test(packet.timestamp)
|
||||
|| typeof packet.signature !== "string" || !/^v0=[a-f0-9]{64}$/.test(packet.signature))
|
||||
throw new SlackCloudReceiptError("invalid_receipt", "invalid");
|
||||
const acceptedAt = Date.parse(event.createdAt);
|
||||
|
||||
if (packet.acceptedAt !== event.createdAt || !Number.isFinite(acceptedAt) || acceptedAt > now + 300_000 || now - acceptedAt >= 30 * 24 * 60 * 60_000
|
||||
|| Math.abs(Math.floor(acceptedAt / 1000) - Number(packet.timestamp)) > 300)
|
||||
throw new SlackCloudReceiptError("invalid_acceptance_time", "attention");
|
||||
const body = Buffer.from(packet.rawBody, "base64");
|
||||
if (body.byteLength > 1_048_576) throw new SlackCloudReceiptError("receipt_too_large", "invalid");
|
||||
const signature = createHmac("sha256", current.signingSecret).update(`v0:${packet.timestamp}:`).update(body).digest();
|
||||
if (!timingSafeEqual(signature, Buffer.from(packet.signature.slice(3), "hex")))
|
||||
throw new SlackCloudReceiptError("original_authentication_failed", "attention");
|
||||
let payload: unknown;
|
||||
try { payload = JSON.parse(body.toString("utf8")); } catch { throw new SlackCloudReceiptError("invalid_event", "invalid"); }
|
||||
if (!payload || typeof payload !== "object" || Array.isArray(payload)) throw new SlackCloudReceiptError("invalid_event", "invalid");
|
||||
const envelope = payload as Record<string, unknown>;
|
||||
if (envelope.type !== "event_callback" || envelope.team_id !== current.teamId || envelope.api_app_id !== current.appId
|
||||
|| typeof envelope.event_id !== "string" || !/^[A-Za-z0-9_-]{1,200}$/.test(envelope.event_id) || envelope.event_id !== packet.eventId
|
||||
|| event.id !== `slack_${current.bindingId.slice("slack-event:".length)}_${createHash("sha256").update(envelope.event_id).digest("hex")}`)
|
||||
throw new SlackCloudReceiptError("event_identity_changed", "invalid");
|
||||
const request = new Request("https://paperclip.invalid/slack-cloud-ingress", { method: "POST",
|
||||
headers: { "content-type": "application/json", "x-slack-request-timestamp": packet.timestamp,
|
||||
"x-slack-signature": packet.signature }, body: new Uint8Array(body) });
|
||||
acceptedTimes.set(request, acceptedAt);
|
||||
try {
|
||||
const response = await handleWebhook(request);
|
||||
await response.body?.cancel();
|
||||
if (!response.ok) throw new SlackCloudReceiptError("callback_unaccepted", "retry");
|
||||
} finally { acceptedTimes.delete(request); }
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
import { HttpError } from "../../../errors.js";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import { createSlackCloudEventIngress, dispatchSlackCloudEvent, registerSlackCloudEventIngress } from "./cloud-ingress.js";
|
||||
import type { SlackCloudEvent } from "./cloud-event.js";
|
||||
|
||||
it("missing or failed runtime owner retains the leased event; retired owner cleanup cannot remove its replacement", async () => {
|
||||
const db = {} as Db;
|
||||
const event: SlackCloudEvent = { id: "receipt", provider: "slack", event: "event_callback", installationId: "binding",
|
||||
bindingIds: ["binding"], createdAt: "2099-01-01T00:00:00Z", payload: {} };
|
||||
await expect(dispatchSlackCloudEvent(db, event)).rejects.toThrow("not ready");
|
||||
const stopped = vi.fn(async () => { throw new Error("admission stopped"); });
|
||||
const unregisterOld = registerSlackCloudEventIngress(db, stopped);
|
||||
await expect(dispatchSlackCloudEvent(db, event)).rejects.toThrow("admission stopped");
|
||||
const replacement = vi.fn(async () => {});
|
||||
const unregisterNew = registerSlackCloudEventIngress(db, replacement);
|
||||
unregisterOld();
|
||||
await dispatchSlackCloudEvent(db, event);
|
||||
expect(replacement).toHaveBeenCalledWith(event);
|
||||
unregisterNew();
|
||||
await expect(dispatchSlackCloudEvent(db, event)).rejects.toThrow("not ready");
|
||||
});
|
||||
|
||||
it.each([404, 422])("current ingress keeps credential failure %s as unacknowledged attention before callback admission", async status => {
|
||||
const endpoint = { id: "00000000-0000-4000-8000-000000000001", companyId: "company", publicId: "public", connectionId: "connection",
|
||||
provider: "slack", status: "active", providerAccountId: "TTEST", setup: { slackCloudEvents: {
|
||||
version: 1, bindingId: "binding", runtimeGeneration: 1, teamId: "TTEST", appId: "ATEST" } } };
|
||||
const db = { select: () => {
|
||||
let joined = false;
|
||||
const query = { from: () => query, where: () => query, innerJoin: () => { joined = true; return query; },
|
||||
limit: async () => joined ? [{ endpoint, connection: { enabled: true, status: "active" } }] : [endpoint] };
|
||||
return query;
|
||||
} } as unknown as Db;
|
||||
const handleWebhook = vi.fn(async () => new Response());
|
||||
const withCredentialMutationLease: Parameters<typeof createSlackCloudEventIngress>[1]["withCredentialMutationLease"] =
|
||||
async (_endpoint, work) => work({ assertOwned: async () => {} } as Parameters<typeof work>[0]);
|
||||
const unregister = createSlackCloudEventIngress(db, { withCredentialMutationLease, runtimeGeneration: () => 1, handleWebhook,
|
||||
resolveCredentials: async () => { throw new HttpError(status, "synthetic credential failure"); } });
|
||||
try {
|
||||
await expect(dispatchSlackCloudEvent(db, { id: "receipt", provider: "slack", event: "event_callback", installationId: "binding",
|
||||
bindingIds: ["binding"], createdAt: new Date().toISOString(), payload: { slackEvent: {
|
||||
endpointId: endpoint.id, companyId: endpoint.companyId, publicId: endpoint.publicId,
|
||||
connectionId: endpoint.connectionId, runtimeGeneration: 1 } } }))
|
||||
.rejects.toMatchObject({ code: "signing_credential_unavailable", disposition: "attention" });
|
||||
expect(handleWebhook).not.toHaveBeenCalled();
|
||||
} finally { unregister(); }
|
||||
});
|
||||
@@ -0,0 +1,63 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { chatEndpoints, toolConnections } from "@paperclipai/db";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import type { chatCredentialMutationLease } from "../../chat-credential-mutation-lease.js";
|
||||
import type { createChatCredentials } from "../credentials.js";
|
||||
import type { EndpointRow, InternalSetupState } from "../runtime-types.js";
|
||||
import { SlackCloudReceiptError, receiveSlackCloudEvent, type CurrentSlackCloudBinding, type SlackCloudEvent } from "./cloud-event.js";
|
||||
|
||||
import { resolveSlackCloudSigningSecret } from "./cloud-signing-secret.js";
|
||||
|
||||
type Handler = (event: SlackCloudEvent) => Promise<void>;
|
||||
const handlers = new WeakMap<Db, Handler>();
|
||||
|
||||
export function registerSlackCloudEventIngress(db: Db, handler: Handler) {
|
||||
handlers.set(db, handler);
|
||||
return () => { if (handlers.get(db) === handler) handlers.delete(db); };
|
||||
}
|
||||
|
||||
export async function dispatchSlackCloudEvent(db: Db, event: SlackCloudEvent): Promise<void> {
|
||||
const handler = handlers.get(db);
|
||||
if (!handler) throw new Error("Slack Cloud ingress is not ready");
|
||||
await handler(event);
|
||||
}
|
||||
|
||||
export function createSlackCloudEventIngress(db: Db, options: {
|
||||
withCredentialMutationLease: ReturnType<typeof chatCredentialMutationLease>;
|
||||
resolveCredentials: ReturnType<typeof createChatCredentials>["resolveCredentials"];
|
||||
runtimeGeneration(endpoint: EndpointRow): number;
|
||||
handleWebhook(publicId: string, provider: "slack", request: Request, expectedGeneration: number): Promise<Response>;
|
||||
}) {
|
||||
return registerSlackCloudEventIngress(db, async event => {
|
||||
const value = event.payload.slackEvent;
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) throw new SlackCloudReceiptError("invalid_receipt", "invalid");
|
||||
const packet = value as Record<string, unknown>;
|
||||
if (typeof packet.endpointId !== "string" || !/^[a-f0-9-]{36}$/i.test(packet.endpointId))
|
||||
throw new SlackCloudReceiptError("invalid_receipt", "invalid");
|
||||
const [initial] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, packet.endpointId)).limit(1);
|
||||
if (!initial || initial.provider !== "slack" || ["archived", "paused", "revoked"].includes(initial.status)) return;
|
||||
const binding = await options.withCredentialMutationLease(initial, async (lease): Promise<CurrentSlackCloudBinding | null> => {
|
||||
const [current] = await db.select({ endpoint: chatEndpoints, connection: toolConnections }).from(chatEndpoints)
|
||||
.innerJoin(toolConnections, and(eq(toolConnections.id, chatEndpoints.connectionId), eq(toolConnections.companyId, chatEndpoints.companyId)))
|
||||
.where(eq(chatEndpoints.id, initial.id)).limit(1);
|
||||
if (!current || current.endpoint.provider !== "slack" || ["archived", "paused", "revoked"].includes(current.endpoint.status)
|
||||
|| current.connection.status !== "active" || !current.connection.enabled) return null;
|
||||
const endpoint = current.endpoint;
|
||||
if (packet.companyId !== endpoint.companyId || packet.publicId !== endpoint.publicId
|
||||
|| packet.connectionId !== endpoint.connectionId || packet.runtimeGeneration !== options.runtimeGeneration(endpoint)) return null;
|
||||
const enrolled = (endpoint.setup as InternalSetupState).slackCloudEvents;
|
||||
|
||||
if (!enrolled) throw new Error("Slack Cloud enrollment is not locally committed");
|
||||
if (enrolled.version !== 1 || enrolled.runtimeGeneration !== options.runtimeGeneration(endpoint)
|
||||
|| enrolled.teamId !== endpoint.providerAccountId) throw new Error("Slack Cloud enrollment is unsettled");
|
||||
const signingSecret = await resolveSlackCloudSigningSecret(options.resolveCredentials, endpoint, db, ["signingSecret"]);
|
||||
await lease.assertOwned();
|
||||
return { companyId: endpoint.companyId, endpointId: endpoint.id,
|
||||
publicId: endpoint.publicId, connectionId: endpoint.connectionId, runtimeGeneration: options.runtimeGeneration(endpoint),
|
||||
bindingId: enrolled.bindingId, teamId: enrolled.teamId, appId: enrolled.appId, signingSecret };
|
||||
});
|
||||
if (!binding) return;
|
||||
await receiveSlackCloudEvent(event, binding,
|
||||
request => options.handleWebhook(binding.publicId, "slack", request, binding.runtimeGeneration));
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,202 @@
|
||||
import { createHmac, generateKeyPairSync, randomUUID } from "node:crypto";
|
||||
import { once } from "node:events";
|
||||
import { createServer } from "node:http";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import { SLACK_CHAT_BOT_SCOPES, type ToolCredentialSecretRef } from "@paperclipai/shared";
|
||||
import { activityLog, agents, applyPendingMigrations, authUsers, chatEndpoints, chatSlackRegistrations, closeCheckpointDb, closeFileStorageDb, closeLivenessDb, companies, companyMemberships, companySecrets, createDb, toolApplications, toolConnections } from "@paperclipai/db";
|
||||
import { resolveExternalTestDatabaseUrl } from "../../../lib/dedicated-test-database-url.js";
|
||||
import { heartbeatService } from "../../heartbeat.js";
|
||||
import { chatChannelService } from "../../chat-channels.js";
|
||||
import { createDeliveryWorkCoordinator } from "../../delivery-work-coordinator.js";
|
||||
import { githubConnectionEventService } from "../../github-connection-events.js";
|
||||
import { registerGitHubConnectionEventWork } from "../../github-connection-work.js";
|
||||
import { createPaperclipCloudConnector, paperclipCloudConnectorConfigFromEnv } from "../../paperclip-cloud-connector.js";
|
||||
import { secretService } from "../../secrets.js";
|
||||
import type { InternalSetupState } from "../runtime-types.js";
|
||||
import { guardSlackFixtureNetwork } from "../../../__tests__/helpers/slack-cloud/transport.js";
|
||||
|
||||
const appUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_SLACK_TEST_DATABASE_URL");
|
||||
const cloudUrl = resolveExternalTestDatabaseUrl("SLACK_EVENT_TEST_DATABASE_URL");
|
||||
const cloudDist = process.env.SLACK_TEST_CLOUD_DIST;
|
||||
|
||||
it.skipIf(!appUrl || !cloudUrl || !cloudDist)("normal notifications enroll the first consumer and authorized reconfigure replaces only positively retired instance custody", async () => {
|
||||
const network = guardSlackFixtureNetwork({ requirePreload: true });
|
||||
const load = (file: string) => import(/* @vite-ignore */ pathToFileURL(path.resolve(cloudDist!, file)).href);
|
||||
const [{ PostgresCloudHarnessRegistry }, { PostgresConnectorStore }, { ConnectorBroker }, { InMemoryProviderSecretStore },
|
||||
{ providerRegistryFromEnvironment }, { CloudStackProvisioner }, { FakeCloudStackProvider }, { Pool }] = await Promise.all([
|
||||
load("registry/postgres.js"), load("connections/postgres.js"), load("connections/broker.js"), load("providers/neon.js"),
|
||||
load("connections/providers.js"), load("provisioner/provisioner.js"), load("providers/fake.js"),
|
||||
import(/* @vite-ignore */ pathToFileURL(path.resolve(cloudDist!, "../node_modules/pg/lib/index.js")).href),
|
||||
]);
|
||||
await applyPendingMigrations(appUrl!);
|
||||
const db = createDb(appUrl!, { maxConnections: 2, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
|
||||
const pool = new Pool({ connectionString: cloudUrl!, max: 2 });
|
||||
const registry = Reflect.construct(PostgresCloudHarnessRegistry, [pool]) as {
|
||||
applyMigrations(folder: string, timeout: number): Promise<void>; hydrate(): Promise<void>; close(): Promise<void>;
|
||||
};
|
||||
try { await registry.applyMigrations(path.resolve(cloudDist!, "../migrations"), 30_000); await registry.hydrate(); }
|
||||
catch (error) { await db.$client.end({ timeout: 1 }); await registry.close(); throw error; }
|
||||
const store = new PostgresConnectorStore(pool), secrets = new InMemoryProviderSecretStore();
|
||||
const id = randomUUID(), companyId = randomUUID(), endpointId = randomUUID(), connectionId = randomUUID(), applicationId = randomUUID(), agentId = randomUUID();
|
||||
const actor = `owner-${id}`, instanceA = `a-${id}`, instanceB = `b-${id}`, publicId = "h".repeat(43);
|
||||
const stackIds = [instanceA, instanceB], signingSecret = "0123456789abcdef0123456789abcdef";
|
||||
const keys = stackIds.map(() => ({ sign: generateKeyPairSync("ed25519"), seal: generateKeyPairSync("x25519") }));
|
||||
let broker: InstanceType<typeof ConnectorBroker>, base = "", loseEnrollmentResponse = false;
|
||||
const requests: { path: string; issuer?: string }[] = [];
|
||||
const server = createServer((req, res) => {
|
||||
requests.push({ path: req.url! });
|
||||
if (loseEnrollmentResponse && req.url === "/v1/connector/slack-events") {
|
||||
loseEnrollmentResponse = false;
|
||||
res.end = (() => { res.destroy(); return res; }) as typeof res.end;
|
||||
}
|
||||
void broker.handle(req, res, new URL(req.url!, base)).catch((error: unknown) => {
|
||||
if (!res.headersSent) res.writeHead(500); res.end(String(error));
|
||||
});
|
||||
});
|
||||
const directory = await mkdtemp(path.join(os.tmpdir(), "slack-transition-"));
|
||||
let service: ReturnType<typeof chatChannelService> | undefined;
|
||||
let heartbeat: ReturnType<typeof heartbeatService> | undefined;
|
||||
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
let worker: ReturnType<typeof registerGitHubConnectionEventWork> | undefined;
|
||||
const errors: unknown[] = [];
|
||||
const deliveryIds: string[] = [];
|
||||
const endpoint = async () => (await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, endpointId)))[0]!;
|
||||
const setup = async () => (await endpoint()).setup as InternalSetupState;
|
||||
const configureIdentity = (index: number) => {
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", stackIds[index]!);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SIGN_PRIVATE_KEY", keys[index]!.sign.privateKey.export({ format: "jwk" }).d!);
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SEAL_PRIVATE_KEY", keys[index]!.seal.privateKey.export({ format: "jwk" }).d!);
|
||||
};
|
||||
try {
|
||||
for (const [index, stackId] of stackIds.entries()) {
|
||||
await new CloudStackProvisioner({ provider: new FakeCloudStackProvider(), registry }).provisionDevelopmentStack({
|
||||
idempotencyKey: stackId, accountGroup: { id: `account-${stackId}`, slug: stackId, displayName: "Transition" },
|
||||
owner: { id: actor, email: `${id}@example.test` },
|
||||
stack: { stackId, accountGroupId: `account-${stackId}`, stackSlug: stackId, displayName: "Transition", kind: "demo", primaryHost: `${stackId}.example.test`, region: "us-east-1" },
|
||||
versionPlan: { appVersion: "fixture", schemaVersion: "fixture", rolloutTrack: "stable" },
|
||||
sleepPolicy: { desiredState: "awake" }, primaryCompany: { paperclipCompanyId: index === 0 ? companyId : randomUUID(), displayName: "Transition", defaultBudgetCents: 0 },
|
||||
});
|
||||
await store.putCloudStackInstance({ id: stackId, kind: "cloud_stack", companyStackId: stackId, ownerUserId: actor, environment: "test",
|
||||
signPublicKey: keys[index]!.sign.publicKey.export({ format: "jwk" }).x!, sealPublicKey: keys[index]!.seal.publicKey.export({ format: "jwk" }).x!,
|
||||
origins: ["https://tenant.example"], activeOrigin: "https://tenant.example", label: "Transition", status: "active", createdAt: new Date(), updatedAt: new Date() });
|
||||
}
|
||||
server.listen(0, "127.0.0.1"); await once(server, "listening");
|
||||
const address = server.address(); if (!address || typeof address === "string") throw new Error("Missing fixture address");
|
||||
base = `http://127.0.0.1:${address.port}`;
|
||||
broker = new ConnectorBroker({ publicOrigin: base, environment: "test", store,
|
||||
providers: providerRegistryFromEnvironment({}, "test"), secrets: { resolve: async () => undefined },
|
||||
authenticate: async () => ({ globalUserId: actor }), slackEvents: { secrets, secretRefPrefix: "secret:///test/transition" },
|
||||
eventWakeEnrollmentEnabled: true, eventWakeDeliveryEnabled: true, wakeCloudStack: async () => {} });
|
||||
vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", base); vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_ENVIRONMENT", "test");
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", path.join(directory, "master.key")); configureIdentity(0);
|
||||
await db.insert(companies).values({ id: companyId, name: "Transition", issuePrefix: `T${id.slice(0, 7)}` });
|
||||
await db.insert(authUsers).values({ id: actor, name: "Owner", email: `${id}@owner.example.test`, emailVerified: true, createdAt: new Date(), updatedAt: new Date() });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: actor, status: "active", membershipRole: "operator" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Transition", adapterType: "process", status: "idle" });
|
||||
await db.insert(toolApplications).values({ id: applicationId, companyId, name: "Transition", type: "chat" });
|
||||
await db.insert(toolConnections).values({ id: connectionId, companyId, applicationId, name: "Transition", uid: id,
|
||||
transport: "chat_sdk", connectionPurpose: "channel", enabled: false, status: "disabled" });
|
||||
await db.insert(chatEndpoints).values({ id: endpointId, companyId, connectionId, assignedAgentId: agentId, publicId,
|
||||
provider: "slack", status: "paused", providerAccountId: "TTEST", botExternalId: "UBOT", setup: { runtimeGeneration: 0, step: "complete" } as InternalSetupState });
|
||||
await db.insert(chatSlackRegistrations).values({ companyId, endpointId, requestId: randomUUID(), status: "configured", appId: "ATEST",
|
||||
manifest: {}, manifestHash: "fixture", callbackUri: "https://tenant.example/callback", createdByUserId: actor });
|
||||
const vault = secretService(db), refs: ToolCredentialSecretRef[] = [];
|
||||
for (const [key, value] of Object.entries({ botToken: "xoxb-fixture", signingSecret })) {
|
||||
const secret = await vault.create(companyId, { name: `${key}-${id}`, provider: "local_encrypted", value }, { userId: actor });
|
||||
await vault.createBinding({ companyId, secretId: secret.id, targetType: "tool_connection", targetId: connectionId, configPath: `credentials.${key}` });
|
||||
refs.push({ secretId: secret.id, versionSelector: "latest", configPath: `credentials.${key}`, required: true, projectionClass: "unclassified" });
|
||||
}
|
||||
await db.update(toolConnections).set({ credentialSecretRefs: refs }).where(eq(toolConnections.id, connectionId));
|
||||
const fetchImpl: typeof fetch = async input => {
|
||||
const url = input instanceof Request ? new URL(input.url) : new URL(String(input));
|
||||
if (url.href === "https://slack.com/api/auth.test") return Response.json({ ok: true, team_id: "TTEST", user_id: "UBOT", user: "fixture" }, { headers: { "x-oauth-scopes": SLACK_CHAT_BOT_SCOPES.join(",") } });
|
||||
if (url.origin === "https://slack.com" && url.pathname === "/api/conversations.list") return Response.json({ ok: true, channels: [], response_metadata: {} });
|
||||
throw new Error(`Unexpected transition provider request ${url}`);
|
||||
};
|
||||
heartbeat = heartbeatService(db);
|
||||
service = chatChannelService(db, { heartbeat, fetch: fetchImpl, publicBaseUrl: "https://tenant.example" });
|
||||
const events = githubConnectionEventService(db);
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
worker = registerGitHubConnectionEventWork(db, coordinator, events, () => true); await worker.ready;
|
||||
await vi.waitFor(() => expect(coordinator!.nextWakeAt()).toBeNull());
|
||||
const leaseCount = () => requests.filter(value => value.path === "/v1/connector/events/lease").length;
|
||||
expect(leaseCount()).toBe(0);
|
||||
await service.configure(endpointId, { action: "resume" }, actor);
|
||||
await vi.waitFor(async () => expect((await setup()).slackCloudEvents?.runtimeGeneration).toBe(1), { timeout: 10_000 });
|
||||
await vi.waitFor(() => expect(leaseCount()).toBeGreaterThan(0), { timeout: 10_000 });
|
||||
expect((await store.getInstance(instanceA)).eventWakeVersion).toBe(1);
|
||||
expect((await setup()).slackCloudInstanceId).toBe(instanceA);
|
||||
await service.configure(endpointId, { action: "pause" }, actor);
|
||||
await vi.waitFor(async () => expect((await setup()).slackCloudRetiredGeneration).toBe(2), { timeout: 10_000 });
|
||||
expect(await broker.slackEvents.current(instanceA, publicId)).toBeUndefined();
|
||||
expect((await store.listWebhookBindings(instanceA)).every((binding: { receivesEvents: boolean }) => !binding.receivesEvents)).toBe(true);
|
||||
const retired = await setup(); configureIdentity(1);
|
||||
const foreign = createPaperclipCloudConnector({ config: paperclipCloudConnectorConfigFromEnv()! });
|
||||
await expect(foreign.slackEventEnrollment({ subject: `foreign-${actor}`, companyId, enrollment: {
|
||||
active: true, companyId, endpointId, publicId, connectionId, runtimeGeneration: 3, teamId: "TTEST", appId: "ATEST", signingSecret,
|
||||
} })).rejects.toMatchObject({ status: 403 });
|
||||
expect(await broker.slackEvents.current(instanceB, publicId)).toBeUndefined();
|
||||
expect((await setup()).slackCloudIntent).toBeUndefined();
|
||||
expect((await setup()).slackCloudInstanceId).toBe(instanceA);
|
||||
expect((await setup()).slackCloudSubject).toBe(actor);
|
||||
await service.configure(endpointId, { action: "resume" }, actor);
|
||||
const pending = await setup();
|
||||
expect(pending.slackCloudIntent).toMatchObject({ instanceId: instanceB, runtimeGeneration: 3, subject: actor });
|
||||
expect(pending.slackCloudInstanceId).toBe(instanceA);
|
||||
expect(pending.slackCloudEvents).toEqual(retired.slackCloudEvents);
|
||||
await vi.waitFor(async () => expect((await setup()).slackCloudInstanceId).toBe(instanceB), { timeout: 10_000 });
|
||||
expect((await setup()).slackCloudIntent).toBeUndefined();
|
||||
expect((await setup()).slackCloudRetiredGeneration).toBeUndefined();
|
||||
expect((await broker.slackEvents.current(instanceB, publicId)).runtimeGeneration).toBe(3);
|
||||
expect(await broker.slackEvents.current(instanceA, publicId)).toBeUndefined();
|
||||
await service.configure(endpointId, { action: "pause" }, actor);
|
||||
await vi.waitFor(async () => expect((await setup()).slackCloudRetiredGeneration).toBe(4), { timeout: 10_000 });
|
||||
loseEnrollmentResponse = true;
|
||||
await service.configure(endpointId, { action: "resume" }, actor);
|
||||
await vi.waitFor(async () => expect((await setup()).slackCloudIntent).toMatchObject({ instanceId: instanceB,
|
||||
runtimeGeneration: 5, failureCode: "broker_unavailable" }), { timeout: 10_000 });
|
||||
expect((await broker.slackEvents.current(instanceB, publicId)).runtimeGeneration).toBe(5);
|
||||
expect((await setup()).slackCloudEvents?.runtimeGeneration).toBe(3);
|
||||
const rawBody = Buffer.from(JSON.stringify({ type: "event_callback", event_id: `EvLostReplacement-${id}`, team_id: "TTEST", api_app_id: "ATEST",
|
||||
event: { type: "app_mention", text: "removed before admission", user: "UOPERATOR", channel: "CTEST", ts: "1788.600" } }));
|
||||
const timestamp = String(Math.floor(Date.now() / 1000));
|
||||
await broker.slackEvents.receive(await broker.slackEvents.current(instanceB, publicId), { rawBody, timestamp,
|
||||
signature: `v0=${createHmac("sha256", signingSecret).update(`v0:${timestamp}:`).update(rawBody).digest("hex")}` });
|
||||
deliveryIds.push(...(await pool.query("SELECT delivery_id FROM cloud_harness.connector_webhook_delivery_targets WHERE connector_instance_id=$1", [instanceB])).rows.map((row: { delivery_id: string }) => row.delivery_id));
|
||||
await service.configure(endpointId, { action: "remove" }, actor);
|
||||
expect((await db.select().from(toolConnections).where(eq(toolConnections.id, connectionId)))[0]!.credentialSecretRefs).toEqual([]);
|
||||
await vi.waitFor(async () => expect((await setup()).slackCloudRetiredGeneration).toBe(6), { timeout: 10_000 });
|
||||
expect((await setup()).slackCloudIntent).toBeUndefined();
|
||||
expect((await setup()).slackCloudInstanceId).toBe(instanceB);
|
||||
expect(await broker.slackEvents.current(instanceB, publicId)).toBeUndefined();
|
||||
expect((await pool.query("SELECT delivery_id FROM cloud_harness.connector_webhook_delivery_targets WHERE connector_instance_id=$1", [instanceB])).rows).toEqual([]);
|
||||
await expect(broker.slackEvents.enroll(await store.getInstance(instanceB), actor, { companyId, endpointId, publicId, connectionId,
|
||||
runtimeGeneration: 5, teamId: "TTEST", appId: "ATEST", signingSecret })).rejects.toThrow("GENERATION_CHANGED");
|
||||
expect(errors).toEqual([]);
|
||||
} catch (error) { console.error("Slack instance transition body failure", error); throw error; }
|
||||
finally {
|
||||
await worker?.stop(); await coordinator?.stop(); await service?.shutdown(); await heartbeat?.drainActiveRunExecutions();
|
||||
if (server.listening) { server.close(); await once(server, "close"); }
|
||||
vi.unstubAllEnvs();
|
||||
try {
|
||||
await db.delete(chatEndpoints).where(eq(chatEndpoints.companyId, companyId));
|
||||
await db.delete(activityLog).where(eq(activityLog.companyId, companyId));
|
||||
await db.delete(agents).where(eq(agents.companyId, companyId));
|
||||
await db.delete(companyMemberships).where(eq(companyMemberships.companyId, companyId));
|
||||
await db.delete(companySecrets).where(eq(companySecrets.companyId, companyId));
|
||||
await db.delete(companies).where(eq(companies.id, companyId));
|
||||
await db.delete(authUsers).where(eq(authUsers.id, actor));
|
||||
await pool.query("DELETE FROM cloud_harness.connector_instances WHERE id=ANY($1::text[])", [stackIds]);
|
||||
await pool.query("DELETE FROM cloud_harness.connector_webhook_deliveries WHERE id=ANY($1::text[])", [deliveryIds]);
|
||||
} finally {
|
||||
await closeCheckpointDb(db, 1); await closeFileStorageDb(db, 1); await closeLivenessDb(db, 1);
|
||||
await db.$client.end({ timeout: 1 }); await registry.close(); await rm(directory, { recursive: true, force: true });
|
||||
try { network.assertQuiet(); } finally { network.restore(); }
|
||||
}
|
||||
}
|
||||
}, 60_000);
|
||||
@@ -0,0 +1,19 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import { HttpError } from "../../../errors.js";
|
||||
import type { EndpointRow } from "../runtime-types.js";
|
||||
import { resolveSlackCloudSigningSecret } from "./cloud-signing-secret.js";
|
||||
|
||||
it.each([400, 401, 403, 404, 422])("credential resolution %s cannot authenticate or acknowledge a receipt", async status => {
|
||||
const resolve = vi.fn(async (): Promise<Record<string, string>> => { throw new HttpError(status, "synthetic"); });
|
||||
await expect(resolveSlackCloudSigningSecret(resolve, {} as EndpointRow, {} as Db, ["signingSecret"]))
|
||||
.rejects.toMatchObject({ code: "signing_credential_unavailable", disposition: "attention" });
|
||||
});
|
||||
|
||||
it("empty credential is attention, while transient failures retain their retry classification", async () => {
|
||||
await expect(resolveSlackCloudSigningSecret(async () => ({}), {} as EndpointRow))
|
||||
.rejects.toMatchObject({ code: "signing_credential_unavailable", disposition: "attention" });
|
||||
const transient = new Error("database temporarily unavailable");
|
||||
await expect(resolveSlackCloudSigningSecret(async () => { throw transient; }, {} as EndpointRow)).rejects.toBe(transient);
|
||||
await expect(resolveSlackCloudSigningSecret(async () => ({ signingSecret: "synthetic" }), {} as EndpointRow)).resolves.toBe("synthetic");
|
||||
});
|
||||
@@ -0,0 +1,18 @@
|
||||
import { HttpError } from "../../../errors.js";
|
||||
import type { createChatCredentials } from "../credentials.js";
|
||||
import { SlackCloudReceiptError } from "./cloud-event.js";
|
||||
|
||||
type Resolver = ReturnType<typeof createChatCredentials>["resolveCredentials"];
|
||||
|
||||
export async function resolveSlackCloudSigningSecret(resolve: Resolver, ...args: Parameters<Resolver>): Promise<string> {
|
||||
let credentials: Record<string, string>;
|
||||
try {
|
||||
credentials = await resolve(...args);
|
||||
} catch (error) {
|
||||
if (error instanceof HttpError && [400, 401, 403, 404, 422].includes(error.status))
|
||||
throw new SlackCloudReceiptError("signing_credential_unavailable", "attention");
|
||||
throw error;
|
||||
}
|
||||
if (!credentials.signingSecret) throw new SlackCloudReceiptError("signing_credential_unavailable", "attention");
|
||||
return credentials.signingSecret;
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { expect, it } from "vitest";
|
||||
|
||||
const guard = fileURLToPath(new URL("../../../__tests__/helpers/slack-cloud/prewire.cjs", import.meta.url));
|
||||
const launcher = fileURLToPath(new URL("../../../../scripts/test-slack-cloud-postgres.mjs", import.meta.url));
|
||||
const cleanEnv = () => ({ PATH: process.env.PATH, HOME: process.env.HOME, NODE_OPTIONS: "" });
|
||||
|
||||
it.each([
|
||||
"require('node:http').get('http://example.invalid')",
|
||||
"require('node:https').get('https://example.invalid')",
|
||||
"require('node:net').connect(443,'example.invalid')",
|
||||
"require('node:tls').connect(443,'example.invalid')",
|
||||
"fetch('https://example.invalid')",
|
||||
"require('node:net').connect(55448,'127.0.0.1')",
|
||||
"require('node:http').get('http://127.0.0.1:1/')",
|
||||
])("the checked-in guard fails caught unowned transport attempts: %s", operation => {
|
||||
const child = spawnSync(process.execPath, ["--require", guard, "-e",
|
||||
`let dnsCalls=0; require('node:dns').lookup=()=>{dnsCalls++;throw Error('DNS must not run');}; try { ${operation}; } catch {} console.log(JSON.stringify({...globalThis[Symbol.for('paperclip.slack.fixture.prewire')].stats(),dnsCalls}));`],
|
||||
{ env: cleanEnv(), encoding: "utf8", timeout: 5_000 });
|
||||
expect(child.error).toBeUndefined();
|
||||
expect(child.status).toBe(98);
|
||||
const stats = JSON.parse(child.stdout.trim());
|
||||
expect(stats.pid).toBe(child.pid);
|
||||
expect(stats.unexpectedAttempts).toHaveLength(1);
|
||||
expect(stats.peakPgSockets).toBe(0);
|
||||
expect(stats.dnsCalls).toBe(0);
|
||||
});
|
||||
|
||||
it("the same guard allows an actual owned loopback HTTP server and initializes before the child script", () => {
|
||||
const child = spawnSync(process.execPath, ["--require", guard, "-e", `
|
||||
const http=require('node:http');
|
||||
if(process.env.SLACK_GATE_GUARD_PID!==String(process.pid))throw Error('preload missing');
|
||||
const server=http.createServer((req,res)=>res.end('owned response'));
|
||||
server.listen(0,'127.0.0.1',()=>http.get('http://127.0.0.1:'+server.address().port,response=>{
|
||||
let body='';response.on('data',chunk=>body+=chunk);response.on('end',()=>server.close(()=>{
|
||||
if(body!=='owned response')throw Error('response missing');
|
||||
console.log(JSON.stringify(globalThis[Symbol.for('paperclip.slack.fixture.prewire')].stats()));
|
||||
}));
|
||||
}));`], { env: cleanEnv(), encoding: "utf8", timeout: 5_000 });
|
||||
expect(child.error).toBeUndefined();
|
||||
expect(child.status).toBe(0);
|
||||
expect(JSON.parse(child.stdout.trim())).toMatchObject({ pid: child.pid, peakPgSockets: 0, unexpectedAttempts: [] });
|
||||
});
|
||||
|
||||
it.each([
|
||||
{},
|
||||
{ PAPERCLIP_SLACK_TEST_DATABASE_URL: "postgres://user@127.0.0.1/live", SLACK_EVENT_TEST_DATABASE_URL: "postgres://user@127.0.0.1/cloud_test" },
|
||||
{ PAPERCLIP_SLACK_TEST_DATABASE_URL: "postgres://user@127.0.0.1/same_test", SLACK_EVENT_TEST_DATABASE_URL: "postgres://user@127.0.0.1/same_test" },
|
||||
{ PAPERCLIP_SLACK_TEST_DATABASE_URL: "postgres://user@example.invalid/app_test", SLACK_EVENT_TEST_DATABASE_URL: "postgres://user@127.0.0.1/cloud_test" },
|
||||
])("the repository launcher rejects unsafe or absent fixture settings before starting Vitest", settings => {
|
||||
const child = spawnSync(process.execPath, [launcher], { env: { ...cleanEnv(), ...settings }, encoding: "utf8", timeout: 5_000 });
|
||||
expect(child.error).toBeUndefined();
|
||||
expect(child.status).toBe(1);
|
||||
expect(child.stderr).toMatch(/test database|loopback|distinct/);
|
||||
expect(child.stdout).not.toContain("RUN");
|
||||
});
|
||||
@@ -1,3 +1,4 @@
|
||||
import { slackCloudAcceptedAt } from "./cloud-event.js";
|
||||
import { isProvisionedSlackSetup } from "@paperclipai/shared";
|
||||
import { slackRequestSignatureIsValid, slackRequestWorkspaceId, slackCallbackMatchesPublicUrl, inspectSlackCallback } from "./callback-verification.js";
|
||||
import type { InternalSetupState } from "../runtime-types.js";
|
||||
@@ -53,6 +54,7 @@ export function createSlackWebhookCallbacks(db: Db, options: SlackWebhookOptions
|
||||
return null;
|
||||
}
|
||||
async function recordSlackWebhookCallback(endpoint: EndpointRow, request: Request, slackCallbackInspection: Request, runtimeContext: RuntimeContext): Promise<void> {
|
||||
if (slackCloudAcceptedAt(request) !== undefined) return;
|
||||
const eventBody = await slackCallbackInspection.clone().text();
|
||||
const callback = await inspectSlackCallback(slackCallbackInspection);
|
||||
if (callback) {
|
||||
|
||||
@@ -17,6 +17,8 @@ import {
|
||||
} from "@chat-adapter/discord";
|
||||
import type { SlackAdapterConfig } from "@chat-adapter/slack";
|
||||
import { PaperclipSlackAdapter } from "./chat-channels/slack/adapter.js";
|
||||
import { verifySlackSignature } from "@chat-adapter/slack/webhook";
|
||||
import { slackCloudAcceptedAt } from "./chat-channels/slack/cloud-event.js";
|
||||
import {
|
||||
createTeamsAdapter,
|
||||
type TeamsAdapter,
|
||||
@@ -1361,6 +1363,13 @@ function createProviderAdapter(
|
||||
case "slack": {
|
||||
const adapterConfig: SlackAdapterConfig = {
|
||||
...config.credentials,
|
||||
webhookVerifier: async (request, body) => {
|
||||
await verifySlackSignature(body, request.headers, {
|
||||
signingSecret: config.credentials.signingSecret,
|
||||
now: () => slackCloudAcceptedAt(request) ?? Date.now(),
|
||||
});
|
||||
return true;
|
||||
},
|
||||
logger: resolvedLogger,
|
||||
mode: "webhook",
|
||||
nativeStreaming: true,
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
import { SlackCloudReceiptError } from "./chat-channels/slack/cloud-event.js";
|
||||
import { reconcilePendingSlackCloudEnrollments } from "./chat-channels/slack/cloud-enrollment.js";
|
||||
import { notifyExternalObjectRefresh } from "./external-object-refresh-notifications.js";
|
||||
import { notifyGitHubCredentialChange } from "./github-connection-notifications.js";
|
||||
import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js";
|
||||
import { dispatchGitHubBotCloudEvent } from "./chat-github-cloud-ingress.js";
|
||||
import { dispatchSlackCloudEvent } from "./chat-channels/slack/cloud-ingress.js";
|
||||
import { chatEndpoints } from "@paperclipai/db";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { isDeepStrictEqual } from "node:util";
|
||||
@@ -224,6 +227,9 @@ export function githubConnectionEventService(
|
||||
) {
|
||||
const now = options.now ?? (() => new Date());
|
||||
let nextPollAt = 0;
|
||||
let nextEnrollmentAt: number | null = null;
|
||||
let nextEnrollmentDiagnosticAt = 0;
|
||||
const slackDiagnostics = new Map<string, number>();
|
||||
let configuredConnector: {
|
||||
config: NonNullable<ReturnType<typeof paperclipCloudConnectorConfigFromEnv>>;
|
||||
connector: PaperclipCloudConnector;
|
||||
@@ -503,8 +509,27 @@ export function githubConnectionEventService(
|
||||
nextPollAt = Number.POSITIVE_INFINITY;
|
||||
return { leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 };
|
||||
}
|
||||
try {
|
||||
return await drainConnectorEvents(connector);
|
||||
} finally {
|
||||
try {
|
||||
nextEnrollmentAt = await reconcilePendingSlackCloudEnrollments(db, now());
|
||||
} catch {
|
||||
const time = now().getTime();
|
||||
nextEnrollmentAt = time + 5_000;
|
||||
if (time >= nextEnrollmentDiagnosticAt) {
|
||||
nextEnrollmentDiagnosticAt = time + 300_000;
|
||||
logger.warn({ reason: "enrollment_reconcile_unavailable" }, "Slack enrollment responsibility retained");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function drainConnectorEvents(connector: PaperclipCloudConnector): Promise<GitHubConnectionEventPollResult> {
|
||||
const bindings = await activeBindings();
|
||||
const bots = await db.select().from(chatEndpoints).where(and(eq(chatEndpoints.provider, "github"), sql`${chatEndpoints.setup}->'github'->>'cloudRegistrationId' is not null`, sql`${chatEndpoints.status} not in ('archived', 'disabled')`, sql`exists (select 1 from companies where id = ${chatEndpoints.companyId} and status <> 'archived')`));
|
||||
const githubBots = await db.select().from(chatEndpoints).where(and(eq(chatEndpoints.provider, "github"), sql`${chatEndpoints.setup}->'github'->>'cloudRegistrationId' is not null`, sql`${chatEndpoints.status} not in ('archived', 'disabled')`, sql`exists (select 1 from companies where id = ${chatEndpoints.companyId} and status <> 'archived')`));
|
||||
const slackBots = await db.select().from(chatEndpoints).where(and(eq(chatEndpoints.provider, "slack"), sql`${chatEndpoints.setup}->'slackCloudEvents' is not null`, sql`${chatEndpoints.status} in ('active','verifying','attention')`, sql`exists (select 1 from companies where id = ${chatEndpoints.companyId} and status <> 'archived')`));
|
||||
const bots = [...githubBots, ...slackBots];
|
||||
let identity: { subject: string; companyId: string } | null = bindings[0] ?? (bots[0] ? { subject: "github-bot-events", companyId: bots[0].companyId } : null);
|
||||
const hasConsumer = bindings.length > 0 || bots.length > 0;
|
||||
if (!identity) {
|
||||
@@ -517,7 +542,7 @@ export function githubConnectionEventService(
|
||||
if (grant && subject) identity = { subject, companyId: grant.companyId };
|
||||
}
|
||||
if (!identity) {
|
||||
const [history] = await db.select().from(chatEndpoints).where(and(eq(chatEndpoints.provider, "github"), sql`${chatEndpoints.setup}->'github'->>'cloudRegistrationId' is not null`)).limit(1);
|
||||
const [history] = await db.select().from(chatEndpoints).where(sql`(${chatEndpoints.provider} = 'github' AND ${chatEndpoints.setup}->'github'->>'cloudRegistrationId' is not null) OR (${chatEndpoints.provider} = 'slack' AND ${chatEndpoints.setup}->'slackCloudEvents' is not null)`).limit(1);
|
||||
if (history) identity = { subject: "github-bot-events", companyId: history.companyId };
|
||||
}
|
||||
if (!identity) {
|
||||
@@ -543,6 +568,23 @@ export function githubConnectionEventService(
|
||||
};
|
||||
const acknowledge: string[] = [];
|
||||
for (const event of lease.events) {
|
||||
if (event.provider === "slack") {
|
||||
try { await dispatchSlackCloudEvent(db, event); acknowledge.push(event.id); result.processed += 1; }
|
||||
catch (error) {
|
||||
const code = error instanceof SlackCloudReceiptError ? error.code : "owner_or_admission_unavailable";
|
||||
const disposition = error instanceof SlackCloudReceiptError ? error.disposition : "retry";
|
||||
const time = now().getTime();
|
||||
if (time - (slackDiagnostics.get(event.id) ?? Number.NEGATIVE_INFINITY) >= 300_000) {
|
||||
if (slackDiagnostics.size >= 1000) slackDiagnostics.delete(slackDiagnostics.keys().next().value!);
|
||||
slackDiagnostics.set(event.id, time);
|
||||
logger.warn({ deliveryId: event.id.slice(0, 200), reason: code, disposition,
|
||||
recoveryExpiresAt: Number.isFinite(Date.parse(event.createdAt)) ? new Date(Date.parse(event.createdAt) + 30 * 24 * 60 * 60_000).toISOString() : null }, "Slack connector receipt requires attention");
|
||||
}
|
||||
if (disposition === "invalid") { acknowledge.push(event.id); result.ignored += 1; }
|
||||
else result.failed += 1;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (event.payload.githubApp !== undefined) {
|
||||
try { await dispatchGitHubBotCloudEvent(db, event); acknowledge.push(event.id); result.processed += 1; }
|
||||
catch { result.failed += 1; }
|
||||
@@ -583,14 +625,14 @@ export function githubConnectionEventService(
|
||||
}
|
||||
|
||||
return {
|
||||
nextPollAt: () => Number.isFinite(nextPollAt) ? nextPollAt : null,
|
||||
nextPollAt: () => { const next = Math.min(nextPollAt, nextEnrollmentAt ?? Number.POSITIVE_INFINITY); return Number.isFinite(next) ? next : null; },
|
||||
pollOnce(input: { forSetup?: boolean; forDelivery?: boolean } = {}): Promise<GitHubConnectionEventPollResult> {
|
||||
// A manager actively awaiting a dedicated App's signed ping must not
|
||||
// inherit a personal connection's old empty-queue backoff. This is a
|
||||
// bounded readiness probe, not an additional background polling loop.
|
||||
if (inFlight) return inFlight;
|
||||
const time = now().getTime();
|
||||
if (!input.forDelivery && time < (input.forSetup ? nextSetupPollAt : nextPollAt)) {
|
||||
if (!input.forDelivery && time < (input.forSetup ? nextSetupPollAt : Math.min(nextPollAt, nextEnrollmentAt ?? Number.POSITIVE_INFINITY))) {
|
||||
return Promise.resolve({ leased: 0, processed: 0, duplicate: 0, ignored: 0, failed: 0 });
|
||||
}
|
||||
nextSetupPollAt = time + 5_000;
|
||||
|
||||
@@ -2471,8 +2471,23 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
const legacyLease = await liveLegacyControllerExpiresAt(db, run);
|
||||
if (legacyLease) { scope.onDeferredDeadline?.(legacyLease.getTime()); continue; }
|
||||
|
||||
// A legacy controller proves it is alive only by renewing its lease, and
|
||||
// the check above found no live lease (a missing expiry also lands here;
|
||||
// revocation below then refuses the row). When the lease belongs to a
|
||||
// previous boot, setup never reached adapter dispatch, and no process
|
||||
// was recorded, the owner is gone and no provider work can be running:
|
||||
// recover now instead of waiting for the age threshold. A dispatched
|
||||
// adapter (for example a remote HTTP request) keeps the age guard. The
|
||||
// stage is rechecked from the state committed with the revocation below.
|
||||
const priorBootSetupExpired =
|
||||
run.runtimeMode === "legacy" &&
|
||||
run.executionStage === "preparing" &&
|
||||
!!run.controllerBootId &&
|
||||
run.controllerBootId !== legacyControllerBootId &&
|
||||
!run.processPid &&
|
||||
!run.processGroupId;
|
||||
// Apply staleness threshold to avoid false positives
|
||||
if (staleThresholdMs > 0) {
|
||||
if (staleThresholdMs > 0 && !priorBootSetupExpired) {
|
||||
const refTime = run.updatedAt ? new Date(run.updatedAt).getTime() : 0;
|
||||
if (now.getTime() - refTime < staleThresholdMs) {
|
||||
scope.onDeferredDeadline?.(refTime + staleThresholdMs);
|
||||
@@ -2543,6 +2558,15 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
(!monitorNextCheckAt || monitorNextCheckAt.getTime() <= now.getTime());
|
||||
const revokedState = await revokeExpiredLegacyControllerWithState(db, run);
|
||||
if (!revokedState) continue;
|
||||
// The earlier snapshot may predate a final renewal that admitted the
|
||||
// adapter. Revocation leaves a fresh foreign lease, so a later pass
|
||||
// applies the age guard to that dispatched run.
|
||||
if (priorBootSetupExpired && staleThresholdMs > 0 && (revokedState.executionStage !== "preparing"
|
||||
|| revokedState.processPid || revokedState.processGroupId)) {
|
||||
const refTime = run.updatedAt ? new Date(run.updatedAt).getTime() : 0;
|
||||
scope.onDeferredDeadline?.(Math.max(refTime + staleThresholdMs, Date.now() + 60_000));
|
||||
continue;
|
||||
}
|
||||
// A legacy controller persists "dispatching" in the same lease renewal
|
||||
// that admits the adapter. An expired lease from another boot that is
|
||||
// still "preparing" proves a restart stopped setup before provider work.
|
||||
|
||||
@@ -584,11 +584,13 @@ describe("Paperclip Cloud connector", () => {
|
||||
return Response.json({
|
||||
active: true,
|
||||
status: "active",
|
||||
profiles: ["gmail.read", "drive.write", "unknown.profile", "gmail.read"],
|
||||
profiles: ["gmail.read", "drive.write", "unknown.profile", "slack.events.v1", "gmail.read"],
|
||||
slackEvents: { version: 1 },
|
||||
});
|
||||
});
|
||||
const connector = createPaperclipCloudConnector({ config: keys.config, request: request as typeof fetch });
|
||||
await expect(connector.getCapabilities()).resolves.toEqual(["gmail.read", "drive.write"]);
|
||||
await expect(connector.slackEventsAvailable()).resolves.toBe(true);
|
||||
});
|
||||
|
||||
it("fails capability discovery closed for inactive, legacy, malformed, or rejected status responses", async () => {
|
||||
@@ -699,10 +701,10 @@ function seal(
|
||||
recipientPublicKey: KeyObject,
|
||||
purpose: "initial" | "access" | "events",
|
||||
configValue: PaperclipCloudConnectorConfig,
|
||||
profile: keyof typeof GOOGLE_WORKSPACE_CONNECTOR_PROFILES | "slack.manager" | "github.bot" | "github.code",
|
||||
profile: keyof typeof GOOGLE_WORKSPACE_CONNECTOR_PROFILES | "slack.manager" | "slack.events.v1" | "github.bot" | "github.code",
|
||||
) {
|
||||
const provider = profile.startsWith("github.") ? "github" : profile === "slack.manager" ? "slack" : "google";
|
||||
const scopes = provider === "github" ? [] : profile === "slack.manager" ? SLACK_MANAGER_SCOPES : GOOGLE_WORKSPACE_CONNECTOR_PROFILES[profile as keyof typeof GOOGLE_WORKSPACE_CONNECTOR_PROFILES].scopes;
|
||||
const provider = profile.startsWith("github.") ? "github" : profile.startsWith("slack.") ? "slack" : "google";
|
||||
const scopes = provider === "github" || profile === "slack.events.v1" ? [] : profile === "slack.manager" ? SLACK_MANAGER_SCOPES : GOOGLE_WORKSPACE_CONNECTOR_PROFILES[profile as keyof typeof GOOGLE_WORKSPACE_CONNECTOR_PROFILES].scopes;
|
||||
const ephemeral = generateKeyPairSync("x25519");
|
||||
const ephemeralJwk = ephemeral.publicKey.export({ format: "jwk" }) as { x: string };
|
||||
const recipientJwk = recipientPublicKey.export({ format: "jwk" }) as { x: string };
|
||||
@@ -795,3 +797,66 @@ describe("dedicated App gateway delivery", () => {
|
||||
expect(JSON.stringify(result)).not.toContain('"ct"');
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
describe("sealed Slack event connector transport", () => {
|
||||
it("opens a Slack event through the normal outer lease without exposing ciphertext or changing callback bytes", async () => {
|
||||
const keys = config();
|
||||
const createdAt = "2026-11-05T10:00:00.000Z";
|
||||
const bindingId = `slack-event:${"a".repeat(64)}`;
|
||||
const packet = { version: 1, instanceId, environment: "staging", bindingId, acceptedAt: createdAt,
|
||||
rawBody: Buffer.from('{ "event_id": "EvFixture", "text": "original bytes" }').toString("base64"),
|
||||
timestamp: "1793872800", signature: `v0=${"b".repeat(64)}` };
|
||||
const inner = seal(packet, keys.sealPublicKey, "events", keys.config, "slack.events.v1");
|
||||
const outer = seal({ v: 1, instanceId, environment: "staging", leaseId: "lease-slack", events: [
|
||||
{ id: "receipt-slack", provider: "slack", event: "event_callback", createdAt, installationId: bindingId,
|
||||
bindingIds: [bindingId], payload: { slackEvent: { bindingId, sealed: inner } } },
|
||||
] }, keys.sealPublicKey, "events", keys.config, "github.code");
|
||||
const connector = createPaperclipCloudConnector({ config: keys.config,
|
||||
request: vi.fn(async () => Response.json({ leaseId: "lease-slack", sealed: outer })) as typeof fetch });
|
||||
const result = await connector.leaseEvents({ subject, companyId });
|
||||
expect(result?.failedEventIds).toBeUndefined();
|
||||
expect(result?.events).toHaveLength(1);
|
||||
expect(result?.events[0]?.payload.slackEvent).toEqual(packet);
|
||||
expect(JSON.stringify(inner)).not.toContain("original bytes");
|
||||
expect(JSON.stringify(result)).not.toContain('"ct"');
|
||||
});
|
||||
|
||||
it.each(["binding", "acceptedAt", "instance", "environment", "key", "profile"] as const)(
|
||||
"isolates an invalid inner Slack %s while retaining a healthy sibling", async failure => {
|
||||
const keys = config();
|
||||
const createdAt = "2026-11-05T10:00:00.000Z";
|
||||
const bindingId = `slack-event:${"a".repeat(64)}`;
|
||||
const packet = { version: 1, instanceId, environment: "staging", bindingId, acceptedAt: createdAt, rawBody: "e30=" };
|
||||
const changed = { ...packet, ...(failure === "binding" ? { bindingId: "other" } : {}),
|
||||
...(failure === "acceptedAt" ? { acceptedAt: "2026-11-05T10:00:00.001Z" } : {}),
|
||||
...(failure === "instance" ? { instanceId: "foreign" } : {}),
|
||||
...(failure === "environment" ? { environment: "production" } : {}) };
|
||||
const event = (id: string, inner: unknown) => ({ id, provider: "slack", event: "event_callback", createdAt,
|
||||
installationId: bindingId, bindingIds: [bindingId], payload: { slackEvent: { bindingId, sealed: inner } } });
|
||||
const bad = seal(changed, failure === "key" ? config().sealPublicKey : keys.sealPublicKey, "events", keys.config,
|
||||
failure === "profile" ? "slack.manager" : "slack.events.v1");
|
||||
const outer = seal({ v: 1, instanceId, environment: "staging", leaseId: "lease-slack", events: [
|
||||
event("bad", bad), event("healthy", seal(packet, keys.sealPublicKey, "events", keys.config, "slack.events.v1")),
|
||||
] }, keys.sealPublicKey, "events", keys.config, "github.code");
|
||||
const connector = createPaperclipCloudConnector({ config: keys.config,
|
||||
request: vi.fn(async () => Response.json({ leaseId: "lease-slack", sealed: outer })) as typeof fetch });
|
||||
await expect(connector.leaseEvents({ subject, companyId })).resolves.toMatchObject({
|
||||
leaseId: "lease-slack", failedEventIds: ["bad"], events: [{ id: "healthy", payload: { slackEvent: packet } }],
|
||||
});
|
||||
});
|
||||
|
||||
it.each([true, false])("selects the matching signed Slack operation receipt (active=%s)", async active => {
|
||||
const keys = config();
|
||||
const enrollment = { active, endpointId: "endpoint", runtimeGeneration: 7 };
|
||||
const enrolled = { version: 1, bindingId: "enrolled" }, retired = { version: 1, runtimeGeneration: 7 };
|
||||
const request = vi.fn(async () => Response.json({ enrollment: enrolled, retirement: retired }));
|
||||
const connector = createPaperclipCloudConnector({ config: keys.config, request: request as typeof fetch });
|
||||
await expect(connector.slackEventEnrollment({ subject, companyId, enrollment })).resolves.toEqual(active ? enrolled : retired);
|
||||
expect(request).toHaveBeenCalledOnce();
|
||||
const [url, init] = request.mock.calls[0] as unknown as [string | URL, RequestInit];
|
||||
expect(new URL(String(url)).pathname).toBe("/v1/connector/slack-events");
|
||||
expect(typeof init.body).toBe("string");
|
||||
expect(JSON.parse(String(init.body))).toMatchObject({ enrollment: JSON.stringify(enrollment) });
|
||||
});
|
||||
});
|
||||
@@ -35,7 +35,7 @@ export const GMAIL_CONNECTOR_SCOPES = [
|
||||
export { GOOGLE_WORKSPACE_CONNECTOR_PROFILES };
|
||||
|
||||
export type PaperclipCloudConnectorEnvironment = "development" | "test" | "staging" | "production";
|
||||
export type PaperclipCloudConnectorOperation = "status" | "session" | "claim" | "refresh" | "revoke" | "webhook-bind" | "event-lease" | "event-ack" | "github-app";
|
||||
export type PaperclipCloudConnectorOperation = "status" | "session" | "claim" | "refresh" | "revoke" | "webhook-bind" | "event-lease" | "event-ack" | "github-app" | "slack-events";
|
||||
export type PaperclipCloudConnectorProfileId = GoogleWorkspaceConnectorProfileId | GitHubConnectorProfileId | AsanaConnectorProfileId | "slack.manager" | "github.bot";
|
||||
export type PaperclipCloudConnectorProvider = "google" | "github" | "asana" | "slack";
|
||||
|
||||
@@ -75,7 +75,7 @@ export type SealedConnectorEvents = {
|
||||
leaseId: string;
|
||||
events: Array<{
|
||||
id: string;
|
||||
provider: "github";
|
||||
provider: "github" | "slack";
|
||||
event: string;
|
||||
action: string | null;
|
||||
installationId: string | null;
|
||||
@@ -86,12 +86,14 @@ export type SealedConnectorEvents = {
|
||||
}>;
|
||||
};
|
||||
|
||||
type SealedEnvelopeProfileId = PaperclipCloudConnectorProfileId | "slack.events.v1";
|
||||
|
||||
type SealedEnvelope = {
|
||||
v: 1;
|
||||
alg: "X25519-HKDF-SHA256-A256GCM";
|
||||
purpose: "initial" | "access" | "events";
|
||||
provider: PaperclipCloudConnectorProvider;
|
||||
profile: PaperclipCloudConnectorProfileId;
|
||||
profile: SealedEnvelopeProfileId;
|
||||
epk: string;
|
||||
iv: string;
|
||||
ct: string;
|
||||
@@ -115,10 +117,14 @@ type ConnectorResponse = {
|
||||
events?: unknown;
|
||||
acknowledged?: unknown;
|
||||
githubApps?: { version: number };
|
||||
slackEvents?: { version: number };
|
||||
enrollment?: unknown;
|
||||
retirement?: unknown;
|
||||
eventWakeVersion?: unknown;
|
||||
};
|
||||
|
||||
const ENDPOINTS: Record<PaperclipCloudConnectorOperation, string> = {
|
||||
"slack-events": "/v1/connector/slack-events",
|
||||
"github-app": "/v1/connector/github-apps",
|
||||
status: "/v1/connector/instance-status",
|
||||
session: "/v1/connector/sessions",
|
||||
@@ -147,6 +153,9 @@ const BROKER_REJECTION_REASONS = new Set([
|
||||
"AUDIENCE_MISMATCH", "OPERATION_MISMATCH", "PAYLOAD_MISMATCH",
|
||||
"MALFORMED_REQUEST", "UNSUPPORTED_ALGORITHM", "RATE_LIMITED",
|
||||
"PROVIDER_OPERATION_FAILED", "PROVIDER_REAUTHORIZATION_REQUIRED",
|
||||
"SLACK_ENROLLMENT_GENERATION_CHANGED", "SLACK_ENROLLMENT_IDENTITY_CHANGED",
|
||||
"SLACK_ENROLLMENT_CHANGED", "SLACK_ENROLLMENT_UNSETTLED", "SLACK_ENROLLMENT_DISABLED",
|
||||
"SLACK_COMPANY_MISMATCH", "SLACK_EVENTS_UNAVAILABLE",
|
||||
]);
|
||||
const UNKNOWN_BROKER_REASON = "UNKNOWN_BROKER_ERROR";
|
||||
|
||||
@@ -192,6 +201,7 @@ export class PaperclipCloudConnectorError extends Error {
|
||||
readonly code: string,
|
||||
readonly status?: number,
|
||||
readonly retryAt?: number,
|
||||
readonly brokerReason?: string,
|
||||
) {
|
||||
super(message);
|
||||
this.name = "PaperclipCloudConnectorError";
|
||||
@@ -280,7 +290,7 @@ export function createPaperclipCloudConnector(input: {
|
||||
async function call(
|
||||
operation: PaperclipCloudConnectorOperation,
|
||||
claims: { subject: string; companyId: string; profile?: PaperclipCloudConnectorProfileId; scopes?: string[]; returnUri?: string; returnState?: string; claimId?: string; redemptionId?: string; eventWakeVersion?: 0 | 1 },
|
||||
secret?: { field: "refreshToken" | "token" | "binding" | "acknowledgement"; value: string },
|
||||
secret?: { field: "refreshToken" | "token" | "binding" | "acknowledgement" | "enrollment"; value: string },
|
||||
): Promise<ConnectorResponse> {
|
||||
const endpoint = new URL(ENDPOINTS[operation], `${config.baseUrl}/`).toString();
|
||||
const issuedAt = Math.floor(now() / 1000);
|
||||
@@ -333,6 +343,7 @@ export function createPaperclipCloudConnector(input: {
|
||||
response.status === 409 ? "REAUTHORIZATION_REQUIRED" : "CONNECTOR_REQUEST_FAILED",
|
||||
response.status,
|
||||
providerRetryAt(response.headers.get("retry-after"), now()),
|
||||
reason,
|
||||
);
|
||||
}
|
||||
try {
|
||||
@@ -381,6 +392,14 @@ export function createPaperclipCloudConnector(input: {
|
||||
|
||||
return {
|
||||
eventWakeVersion: () => eventWakeVersion,
|
||||
async slackEventsAvailable() {
|
||||
const response = await call("status", { subject: "instance-capabilities", companyId: "instance-capabilities" });
|
||||
return response.active === true && response.slackEvents?.version === 1;
|
||||
},
|
||||
async slackEventEnrollment(values: { subject: string; companyId: string; enrollment: Record<string, unknown> }) {
|
||||
const response = await call("slack-events", values, { field: "enrollment", value: JSON.stringify(values.enrollment) });
|
||||
return values.enrollment.active === false ? response.retirement : response.enrollment;
|
||||
},
|
||||
async githubAppsAvailable() {
|
||||
const response = await call("status", { subject: "instance-capabilities", companyId: "instance-capabilities" });
|
||||
return response.active === true && response.githubApps?.version === 2;
|
||||
@@ -551,6 +570,16 @@ export function createPaperclipCloudConnector(input: {
|
||||
|| delivery.registrationId !== packet.registrationId) throw badEnvelope();
|
||||
event.payload = { githubApp: delivery };
|
||||
}
|
||||
if (event.provider === "slack") {
|
||||
const packet = event.payload.slackEvent;
|
||||
if (!isRecord(packet) || typeof packet.bindingId !== "string" || packet.sealed === undefined) throw badEnvelope();
|
||||
const sealed = parseEnvelope(packet.sealed, "events", "slack", "slack.events.v1");
|
||||
const delivery = decryptEnvelope(sealed, sealKey, config.instanceId, config.environment, "slack", "slack.events.v1", []);
|
||||
if (!isRecord(delivery) || delivery.version !== 1 || delivery.instanceId !== config.instanceId
|
||||
|| delivery.environment !== config.environment || delivery.bindingId !== packet.bindingId
|
||||
|| delivery.acceptedAt !== event.createdAt) throw badEnvelope();
|
||||
event.payload = { slackEvent: delivery };
|
||||
}
|
||||
events.push(event);
|
||||
} catch {
|
||||
failedEventIds.push(event.id);
|
||||
@@ -684,7 +713,7 @@ function parseEnvelope(
|
||||
value: unknown,
|
||||
purpose: SealedEnvelope["purpose"],
|
||||
provider: PaperclipCloudConnectorProvider,
|
||||
profile: PaperclipCloudConnectorProfileId,
|
||||
profile: SealedEnvelopeProfileId,
|
||||
): SealedEnvelope {
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) throw badEnvelope();
|
||||
const candidate = value as Partial<SealedEnvelope>;
|
||||
@@ -737,7 +766,7 @@ function unsealEvents(
|
||||
if (parsed.v !== 1 || parsed.instanceId !== instanceId || parsed.environment !== environment
|
||||
|| typeof parsed.leaseId !== "string" || !Array.isArray(parsed.events)) throw badEnvelope();
|
||||
for (const event of parsed.events) {
|
||||
if (!isRecord(event) || typeof event.id !== "string" || event.provider !== "github"
|
||||
if (!isRecord(event) || typeof event.id !== "string" || !["github", "slack"].includes(event.provider)
|
||||
|| typeof event.event !== "string" || !isRecord(event.payload) || !Array.isArray(event.bindingIds)
|
||||
|| !event.bindingIds.every((id) => typeof id === "string")) throw badEnvelope();
|
||||
}
|
||||
@@ -754,7 +783,7 @@ function decryptEnvelope(
|
||||
instanceId: string,
|
||||
environment: string,
|
||||
provider: PaperclipCloudConnectorProvider,
|
||||
profile: PaperclipCloudConnectorProfileId,
|
||||
profile: SealedEnvelopeProfileId,
|
||||
scopes: readonly string[],
|
||||
): unknown {
|
||||
const ephemeralRaw = Buffer.from(envelope.epk, "base64url");
|
||||
|
||||
@@ -3,7 +3,8 @@ import { notifyVoiceChange } from "../../voice/voice-recovery-notifications.js";
|
||||
import { notifyChatEndpointWork } from "../../chat-work-notifications.js";
|
||||
import { notifyToolConnectionHealth } from "../../tool-connection-health-notifications.js";
|
||||
import { toConnection } from "./projection.js";
|
||||
import { actorBinding } from "../actor.js";
|
||||
import { actorBinding, connectionActorUserId } from "../actor.js";
|
||||
import { stageSlackCloudEnrollmentIntent } from "../../chat-channels/slack/cloud-enrollment.js";
|
||||
import { vercelConnectResourcesFor } from "../vercel-credentials.js";
|
||||
import type { ActorInfo } from "../types.js";
|
||||
import { chatCredentialMutationLease } from "../../chat-credential-mutation-lease.js";
|
||||
@@ -241,6 +242,8 @@ async function removeConnection(
|
||||
await lease.assertOwned(tx);
|
||||
await tx.update(chatEndpoints).set({ status: "archived", updatedAt: new Date() }).where(eq(chatEndpoints.id, endpoint.id));
|
||||
await tx.update(toolConnections).set({ status: "archived", enabled: false, updatedAt: new Date() }).where(eq(toolConnections.id, connection.id));
|
||||
await stageSlackCloudEnrollmentIntent(tx as unknown as Db, endpoint.id, connectionActorUserId(actor));
|
||||
await lease.assertOwned(tx);
|
||||
});
|
||||
await removeSlackRegistration(db, endpoint.id, lease);
|
||||
return removeConnectionUnlocked(connectionId, companyId, actor);
|
||||
|
||||
@@ -37,8 +37,16 @@ export const reliabilityDefinitions = Object.freeze([
|
||||
// Until every case and the plan have an identity, selecting this plan fails
|
||||
// closed with `product_regression_testops_reliability_unregistered`.
|
||||
export const reliabilityRegistration = Object.freeze({
|
||||
planId: null,
|
||||
caseIds: Object.freeze({}),
|
||||
planId: 135,
|
||||
caseIds: Object.freeze({
|
||||
'interrupt-redirect': 885,
|
||||
'reassign-midrun': 886,
|
||||
'three-turn-revise': 887,
|
||||
'long-silent-command': 888,
|
||||
'tree-pause-release': 889,
|
||||
'queue-steer': 890,
|
||||
'queued-across-restart': 891,
|
||||
}),
|
||||
});
|
||||
|
||||
const fail = code => { throw new Error(`product_regression_testops_${code}`); };
|
||||
|
||||
Reference in new issue
Block a user