Files
PaperClipAI/server/src/services/ai-connections.ts
T
DottaandClaude Opus 5.5 4879a146fb fix(secrets): stop credential reads from aborting concurrent budget admissions (#21452)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paid model calls (company decisions, fast responses, browser use,
prompt-injection checks) pass a budget admission before they start
> - Admission runs in a REPEATABLE READ transaction and share-locks the
credential's secret row, so a key rotation cannot slip between admission
and the call
> - Every credential read also wrote "Last used" (`last_resolved_at`,
`updated_at`) to that same row
> - When two calls ran at once, one call's credential read aborted the
other's admission with a serialization failure (40001). The retry then
saw the first call's new budget hold and denied the work as
`budget_blocked`
> - This is the flaky `decision-models.test.ts` failure that held Cloud
readiness, and so releases, on 2026-11-13
> - This pull request stops credential reads from writing the secret
row. "Last used" now comes from the access event that every read already
records
> - The benefit is that concurrent paid calls are no longer wrongly
denied or retried, and the release gate no longer fails on this test

## Linked Issues or Issue Description

No public issue exists. The bug is described here.

**What happened?**
Cloud readiness `verify / General tests (server (4/10))` failed on
master commits 6917d0568d and 5ede64f247. The test "admits concurrent
work from the same budget observation and records both obligations"
reported `expected "vi.fn()" to be called 2 times, but got 1 times`.
#21387 replaced the 1-second poll with an unbounded wait. That does not
fix it: one decision really is denied, so the test now waits until the
test timeout.

**Root cause**
I logged the accounting retry loop under load. Each batch of 50 runs had
6 to 10 retries with `could not serialize access due to concurrent
update` on `select … from company_secrets … for share`. The other
decision's credential read had updated the row. In 4 of 5 batches, the
retried admission saw the other decision's committed reservation and
returned `budget_blocked`.

**Expected behavior**
A credential read does not change admission. Two decisions admitted from
the same budget observation both run and both record their obligation.

**Steps to reproduce**
Run the updated test on the old `secrets.ts`. The test makes the second
credential read land after the first admission's snapshot. It then
commits an unrelated hold after both observations. The old code fails
every run with `first settled before its provider call:
{"status":"unavailable","reason":"budget_blocked"}`.

## What Changed

- `secretService` no longer writes `last_resolved_at` / `updated_at`
when it resolves a value. It still records the `secret_access_events`
row, as before.
- `secretService.list` reports `lastResolvedAt` as the later of the
stored column (legacy values) and the newest successful access event. It
reads one row per secret, backwards on the existing `(secret_id,
created_at)` index.
- A read without a consumer context (for example a GitHub API credential
read) now records a `system` access event, so it still counts as "Last
used".
- The removed `UPDATE` also made a credential read wait for a concurrent
rotation to commit. `credential()` in `ai-connections.ts` depends on
that wait to re-read the rotated version instead of handing a run a
consumed single-use token. Its existing version re-read now takes `FOR
SHARE`, so it keeps that wait. `prepareManagedAiRuntime` has its own
before/after freshness read around `runtimeCredential()`; that read
takes `FOR SHARE` too. A share lock creates no new row version and
cannot abort an admission. It adds no statement.
- The first version of this PR throttled the write to once a minute.
Review found that a read crossing the minute could still abort an
admission, so the write is removed instead.
- The concurrent-admission test is now deterministic. It starts decision
A and waits until A is blocked, then starts decision B and waits again.
It commits an unrelated 1-cent hold after both observations, then checks
that both decisions are admitted. A decision that settles before its
provider call fails the test immediately with its result, instead of
hanging.

## Verification

- Red: the new test against the old `secrets.ts` fails on every run with
the `budget_blocked` result above.
- Green: the same test with `repeats: 49` (50 runs) passes 50/50 locally
on a machine at load average 20 to 60.
- Before the change, 3 parallel batches of 50 runs of the old test each
produced a false denial.
- `npx tsc --noEmit -p server`: no errors.
- New `secrets-service.test.ts` test: a resolution leaves the secret row
unchanged, and `list()` still shows "Last used".
- `ai-connections.test.ts` quota test now checks that a read leaves
`updated_at` unchanged and records a success access event.

## Risks

- `updated_at` on a secret no longer moves when the secret is read. It
still moves on every real change.
- "Last used" on the Secrets page costs one indexed lookup per listed
secret.
- Only `list()` derives "Last used". Other endpoints that return a
secret row show the stored column, which is now set only by older
releases.
- No migration. No API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:46:50 -06:00

1275 lines
65 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { notifyLoginSessionChanged } from "./login-cleanup-notifications.js";
import type { ConnectionManagementContext } from "./connection-management.js";
import { WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE } from "@paperclipai/adapter-utils/workspace-restore-merge";
import { withAccountHomeSecretMutationLock } from "@paperclipai/adapter-codex-local/server";
import { syncConnectionCredentialBindings } from "./connection-credential-bindings.js";
import { createHash, randomUUID } from "node:crypto";
import { isDeepStrictEqual } from "node:util";
import { and, eq, inArray, or, sql } from "drizzle-orm";
import {
type Db,
authUsers,
adapterAuthSessions,
aiConnectionDefaults,
aiProviderDefaults,
agents,
heartbeatRuns,
companyMemberships,
companySecrets,
userSecretDefinitions,
connectionGrants,
connectionGrantMembers,
toolApplications,
toolConnections,
toolConnectionInstalls,
} from "@paperclipai/db";
import {
AI_CONNECTION_CAPABILITIES,
decisionProviderForConnection,
aiConnectionCatalogSlug,
getAppStoreDefinition,
aiConnectionMetadataSchema,
aiSubscriptionNeedsIsolatedLogin,
isAiConnectionCompatible,
supportsAiConnectionUsage,
type AiConnectionBinding,
type AiConnectionAttribution,
type AiConnectionMetadata,
type AiManagedConnectionSummary,
type AiConnectionAccountIdentity,
type AiConnectionUsage,
type CreateAiConnection,
type AiConnectionLoginIntent,
} from "@paperclipai/shared";
import { forbidden, notFound, unprocessable } from "../errors.js";
import { aiConnectionConfigurationFailure, aiConnectionCredentialNotSharedFailure } from "./ai-connection-configuration-failure.js";
import { quotaCredentialRecovery, quotaCredentialHash } from "./quota-credential-recovery.js";
import { prepareAntigravityCredentialRecovery } from "./native-runtime/antigravity-runner-credentials.js";
import { logger } from "../middleware/logger.js";
import { logActivity } from "./activity-log.js";
import { secretService } from "./secrets.js";
import { probeAiConnectionUsage } from "./ai-connection-usage.js";
import { probeAiAccountIdentity } from "./ai-account-identity.js";
/** Same human audience displayed by the existing Connections identity controls. */
function canUseCredential(
grant: { kind: string; subjectUserId: string | null },
userId: string | null,
audience: { subjectType: string; subjectId: string }[],
) {
if (!userId) return false;
if (grant.kind === "user") return grant.subjectUserId === userId;
return grant.kind === "organization" && (
audience.length === 0 || audience.some((member) => member.subjectType === "user" && member.subjectId === userId)
);
}
export function aiConnectionService(db: Db) {
const secrets = secretService(db);
async function membership(companyId: string, userId: string | null) {
if (!userId) return false;
return Boolean(
(
await db
.select({ id: companyMemberships.id })
.from(companyMemberships)
.where(
and(
eq(companyMemberships.companyId, companyId),
eq(companyMemberships.principalType, "user"),
eq(companyMemberships.principalId, userId),
eq(companyMemberships.status, "active"),
),
)
.limit(1)
)[0],
);
}
async function rows(companyId: string) {
return db
.select({ connection: toolConnections, grant: connectionGrants })
.from(toolConnections)
.innerJoin(
connectionGrants,
and(
eq(connectionGrants.companyId, toolConnections.companyId),
eq(connectionGrants.connectionId, toolConnections.id),
),
)
.where(
and(
eq(toolConnections.companyId, companyId),
eq(toolConnections.connectionPurpose, "ai"),
),
);
}
async function list(
companyId: string,
userId: string,
_agentId?: string,
): Promise<AiManagedConnectionSummary[]> {
const [accounts, defaults, members, owners] =
await Promise.all([
rows(companyId),
db
.select()
.from(aiProviderDefaults)
.where(
and(
eq(aiProviderDefaults.companyId, companyId),
eq(aiProviderDefaults.userId, userId),
),
),
db
.select()
.from(connectionGrantMembers)
.where(eq(connectionGrantMembers.companyId, companyId)),
db
.select({ id: authUsers.id, name: authUsers.name })
.from(authUsers)
.innerJoin(
companyMemberships,
and(
eq(companyMemberships.principalId, authUsers.id),
eq(companyMemberships.companyId, companyId),
eq(companyMemberships.principalType, "user"),
),
),
]);
return accounts.flatMap(({ connection, grant }) => {
const metadata = aiConnectionMetadataSchema.safeParse(
connection.config.ai,
);
if (!metadata.success) return [];
const needsReconnect = aiSubscriptionNeedsIsolatedLogin(connection.config);
if (!canUseCredential(grant, userId, members.filter((m) => m.grantId === grant.id)))
return [];
return [
{
id: connection.id,
grantId: grant.id,
companyId,
...metadata.data,
usageProbeSupported: supportsAiConnectionUsage(metadata.data.provider, metadata.data.method),
name: connection.name,
accountLabel: grant.providerTenant?.name,
...(needsReconnect ? { unavailableReason: "Reconnect with a separate sign-in to protect your existing terminal login." } : {}),
ownership:
grant.kind === "user" ? ("personal" as const) : ("shared" as const),
ownerUserId: grant.subjectUserId ?? undefined,
ownerName:
owners.find((owner) => owner.id === grant.subjectUserId)?.name ??
(grant.subjectUserId === userId ? "You" : "Account owner"),
isDefault: defaults.some((d) => d.grantId === grant.id),
status:
grant.status === "revoked"
? ("revoked" as const)
: grant.status === "expired"
? ("expired" as const)
: grant.status !== "active" ||
!connection.enabled ||
connection.status !== "active" ||
connection.healthStatus !== "ok" || needsReconnect
? ("needs_attention" as const)
: ("connected" as const),
},
];
}).sort((a, b) => a.id.localeCompare(b.id) || a.grantId.localeCompare(b.grantId));
}
async function setDefault(
companyId: string,
userId: string,
grantId: string,
) {
return db.transaction(async (tx) => {
const [row] = await tx
.select({ connection: toolConnections, grant: connectionGrants })
.from(connectionGrants)
.innerJoin(
toolConnections,
eq(toolConnections.id, connectionGrants.connectionId),
)
.where(
and(
eq(connectionGrants.companyId, companyId),
eq(connectionGrants.id, grantId),
eq(connectionGrants.subjectUserId, userId),
eq(connectionGrants.kind, "user"),
),
)
.for("update");
if (!row)
throw forbidden("Only the owner can choose their personal default");
if (
row.connection.connectionPurpose !== "ai" ||
row.grant.status !== "active" ||
!row.connection.enabled ||
row.connection.healthStatus !== "ok"
)
throw unprocessable(
"Reconnect this account before making it your default",
);
const metadata = aiConnectionMetadataSchema.parse(row.connection.config.ai);
if (metadata.routing) throw unprocessable("Custom providers use an explicit connection selection on the agent.");
// Keep old servers' method preferences intact during an additive rollout.
await tx.insert(aiConnectionDefaults)
.values({ companyId, userId, ...metadata, grantId })
.onConflictDoUpdate({
target: [aiConnectionDefaults.companyId, aiConnectionDefaults.userId, aiConnectionDefaults.provider, aiConnectionDefaults.method],
set: { grantId, updatedAt: new Date() },
});
await tx.insert(aiProviderDefaults)
.values({ companyId, userId, provider: metadata.provider, grantId })
.onConflictDoUpdate({
target: [aiProviderDefaults.companyId, aiProviderDefaults.userId, aiProviderDefaults.provider],
set: { grantId, updatedAt: new Date() },
});
});
}
async function readableAccount(companyId: string, userId: string, connectionId: string, grantId?: string) {
if (!(await membership(companyId, userId))) throw forbidden("An active company member is required");
const candidates = (await rows(companyId)).filter((row) => row.connection.id === connectionId && (!grantId || row.grant.id === grantId));
const audience = await db.select().from(connectionGrantMembers).where(eq(connectionGrantMembers.companyId, companyId));
const visible = candidates.filter(({ grant }) => canUseCredential(grant, userId, audience.filter((member) => member.grantId === grant.id)));
if (!visible.length) throw notFound("AI connection not found");
if (visible.length > 1) throw unprocessable("Choose a credential grant for this account check");
const row = visible[0]!;
const metadata = aiConnectionMetadataSchema.safeParse(row.connection.config.ai);
if (!metadata.success) throw notFound("AI connection not found");
return { row, metadata: metadata.data };
}
/** On-demand account observation for either runner; never probes ambient CLI auth. */
async function probeUsage(companyId: string, userId: string, connectionId: string, grantId?: string): Promise<AiConnectionUsage> {
const { row, metadata } = await readableAccount(companyId, userId, connectionId, grantId);
const identity = { connectionId: row.connection.id, grantId: row.grant.id, ...metadata };
const empty = { ...identity, checkedAt: new Date().toISOString(), source: null, planType: null, limits: [], overage: null };
if (row.grant.status !== "active" || !row.connection.enabled || row.connection.status !== "active" ||
row.connection.healthStatus !== "ok" || aiSubscriptionNeedsIsolatedLogin(row.connection.config)) {
return { ...empty, status: "unavailable", errorCode: "connection_unavailable", message: "Reconnect or enable this AI connection before checking usage." };
}
if (!supportsAiConnectionUsage(metadata.provider, metadata.method)) {
return { ...identity, ...await probeAiConnectionUsage(metadata, "") };
}
const value = await credential(row);
return { ...identity, ...await probeAiConnectionUsage(metadata, value) };
}
async function accountIdentity(companyId: string, userId: string, connectionId: string, grantId?: string,
audit?: { actorType: "user" | "agent" | "system"; actorId: string; heartbeatRunId?: string | null }): Promise<AiConnectionAccountIdentity> {
const { row, metadata } = await readableAccount(companyId, userId, connectionId, grantId);
const identity = { connectionId: row.connection.id, grantId: row.grant.id };
if (row.grant.status !== "active" || !row.connection.enabled || row.connection.status !== "active" ||
row.connection.healthStatus !== "ok" || aiSubscriptionNeedsIsolatedLogin(row.connection.config)) {
return { ...identity, status: "unavailable", checkedAt: new Date().toISOString(), accountLabel: null, organizationLabel: null,
errorCode: "connection_unavailable", message: "Reconnect or enable this connection to check its account details." };
}
// Unsupported methods must not open their credentials just to report this limit.
const value = metadata.provider === "anthropic" && metadata.method === "subscription" && !metadata.routing
? await credential(row, 0, { actorType: "user", actorId: userId, ...audit, responsibleUserId: userId }) : "";
return { ...identity, ...await probeAiAccountIdentity(metadata, value) };
}
async function select(input: {
companyId: string;
userId: string | null;
agentId: string;
adapterType: string;
model?: unknown;
runnerProvider?: unknown;
acpxAgent?: unknown;
allowUninstalledPersonal?: boolean;
allowUninstalledShared?: boolean;
allowLegacyValidation?: boolean;
binding: AiConnectionBinding;
}) {
const { companyId, userId, agentId, binding } = input;
// The grant-kind checks below cover only these three modes. Callers parse
// the binding first, but an unknown mode must never skip those checks.
if (binding.mode !== "responsible_user" && binding.mode !== "shared" && binding.mode !== "delegated")
throw aiConnectionConfigurationFailure("ai_connection_incompatible", "Select a supported AI connection mode");
if (
!isAiConnectionCompatible(
binding,
input.adapterType,
input.model,
input.runnerProvider,
input.acpxAgent,
)
)
throw aiConnectionConfigurationFailure("ai_connection_incompatible", "Select an AI connection compatible with this harness and model");
if (binding.mode === "responsible_user" && !userId)
throw aiConnectionConfigurationFailure("ai_connection_responsible_user_missing", "This run needs a responsible user to select an AI connection");
if (userId && !(await membership(companyId, userId)))
throw forbidden("The responsible user is not an active company member");
const defaultRow =
binding.mode === "responsible_user"
? (
await db
.select()
.from(aiProviderDefaults)
.where(
and(
eq(aiProviderDefaults.companyId, companyId),
eq(aiProviderDefaults.userId, userId!),
eq(aiProviderDefaults.provider, binding.provider),
),
)
.limit(1)
)[0]
: null;
const grantId =
binding.mode === "responsible_user"
? defaultRow?.grantId
: binding.grantId;
if (!grantId)
throw aiConnectionConfigurationFailure("ai_connection_default_missing", "Connect an account and choose your personal default");
const row = (await rows(companyId)).find(
(r) =>
r.grant.id === grantId &&
(binding.mode === "responsible_user" ||
r.connection.id === binding.connectionId),
);
if (!row)
throw aiConnectionConfigurationFailure("ai_connection_missing", "The selected AI connection is unavailable");
const { connection, grant } = row;
const metadata = aiConnectionMetadataSchema.safeParse(connection.config.ai);
if (
!metadata.success ||
metadata.data.provider !== binding.provider ||
(binding.mode !== "responsible_user" && metadata.data.method !== binding.method) ||
!isAiConnectionCompatible(metadata.data, input.adapterType, input.model, input.runnerProvider, input.acpxAgent)
)
throw aiConnectionConfigurationFailure("ai_connection_incompatible", "The selected AI connection is incompatible");
if (aiSubscriptionNeedsIsolatedLogin(connection.config))
throw aiConnectionConfigurationFailure("ai_connection_unavailable", "Reconnect this subscription with a separate sign-in to protect your existing terminal login.", { connectionId: connection.id });
if (
grant.status !== "active" ||
!connection.enabled ||
connection.status !== "active" ||
(connection.healthStatus !== "ok" &&
!(
input.allowLegacyValidation &&
connection.config.aiLegacyAdoption === true
))
)
throw aiConnectionConfigurationFailure("ai_connection_unavailable", "Reconnect or validate the selected AI account", { connectionId: connection.id });
if (
grant.kind === "user" &&
!(await membership(companyId, grant.subjectUserId))
)
throw forbidden(
"The account owner is no longer an active company member",
);
if (
binding.mode === "responsible_user" &&
(grant.kind !== "user" || grant.subjectUserId !== userId)
)
throw forbidden("The default must belong to the responsible user");
if (binding.mode === "shared" && grant.kind !== "organization")
throw forbidden("Select a company-shared account");
if (binding.mode === "delegated" && grant.kind !== "user")
throw forbidden("Select a personal account");
const audience = await db
.select()
.from(connectionGrantMembers)
.where(and(
eq(connectionGrantMembers.companyId, companyId),
eq(connectionGrantMembers.grantId, grant.id),
));
// The existing human-access permission is authoritative for every binding,
// including old explicit personal selections. Agent delegation cannot bypass it.
if (!canUseCredential(grant, userId, audience)) {
// Membership was checked above. A known user's sharing choice is an
// operator action; absent identity must remain an unclassified denial.
if (userId) throw aiConnectionCredentialNotSharedFailure({ connectionName: connection.name, grantId: grant.id });
throw forbidden("This credential is not shared with the responsible user");
}
const installs = await db
.select()
.from(toolConnectionInstalls)
.where(
and(
eq(toolConnectionInstalls.companyId, companyId),
eq(toolConnectionInstalls.connectionId, connection.id),
or(
and(
eq(toolConnectionInstalls.targetType, "company"),
eq(toolConnectionInstalls.targetId, companyId),
),
and(
eq(toolConnectionInstalls.targetType, "agent"),
eq(toolConnectionInstalls.targetId, agentId),
),
),
),
);
if (
!installs.length &&
!(
(input.allowUninstalledPersonal &&
grant.kind === "user" && grant.subjectUserId === userId) ||
(input.allowUninstalledShared && binding.mode === "shared" && grant.kind === "organization")
)
)
throw forbidden("This connection is not permitted for this agent");
return {
...row,
attribution: {
connectionId: connection.id,
grantId: grant.id,
provider: binding.provider,
method: metadata.data.method,
mode: binding.mode,
responsibleUserId: userId,
} satisfies AiConnectionAttribution,
};
}
/** Internal decision use has its own capability check, never a pretend CLI harness. */
async function selectDecision(input: {
companyId: string; connectionId: string; grantId: string;
userId: string | null; agentId?: string | null; sponsoredBackground?: boolean;
}) {
const [row] = await db.select({ connection: toolConnections, grant: connectionGrants })
.from(toolConnections).innerJoin(connectionGrants, and(
eq(connectionGrants.connectionId, toolConnections.id), eq(connectionGrants.companyId, toolConnections.companyId),
)).where(and(eq(toolConnections.companyId, input.companyId), eq(toolConnections.id, input.connectionId),
eq(connectionGrants.id, input.grantId), eq(toolConnections.connectionPurpose, "ai"))).limit(1);
if (!row || row.grant.kind !== "organization") throw unprocessable("Shared AI connection unavailable", { code: "connection_unavailable" });
const metadata = aiConnectionMetadataSchema.safeParse(row.connection.config.ai);
const provider = metadata.success ? decisionProviderForConnection(metadata.data) : null;
if (!provider) throw unprocessable("Connection does not support decisions", { code: "incompatible_connection" });
if (row.grant.status !== "active" || !row.connection.enabled || row.connection.status !== "active" || row.connection.healthStatus !== "ok")
throw unprocessable("Reconnect or enable this connection", { code: "connection_unavailable" });
// Only the decision service may sponsor background use; ordinary AI selection never bypasses an audience.
if (!input.sponsoredBackground) {
if (!(await membership(input.companyId, input.userId))) throw forbidden("Active responsible user required");
const audience = await db.select().from(connectionGrantMembers).where(and(
eq(connectionGrantMembers.companyId, input.companyId), eq(connectionGrantMembers.grantId, input.grantId),
));
if (!canUseCredential(row.grant, input.userId, audience)) throw forbidden("Connection is not shared with this user");
} else if (input.userId || input.agentId) throw forbidden("Background sponsorship cannot replace caller permissions");
if (input.agentId) {
const [install] = await db.select({ id: toolConnectionInstalls.id }).from(toolConnectionInstalls).where(and(
eq(toolConnectionInstalls.companyId, input.companyId), eq(toolConnectionInstalls.connectionId, input.connectionId),
or(and(eq(toolConnectionInstalls.targetType, "company"), eq(toolConnectionInstalls.targetId, input.companyId)),
and(eq(toolConnectionInstalls.targetType, "agent"), eq(toolConnectionInstalls.targetId, input.agentId))),
)).limit(1);
if (!install) throw forbidden("Connection is not permitted for this agent");
}
return { ...row, provider };
}
/**
* A native run's image tool reuses the exact AI connection the run was
* admitted with. Recheck the same permissions as runtime selection on every
* call: an OpenAI API key sent directly to OpenAI, active and healthy, shared
* with the responsible user and installed for this agent.
*/
async function selectImageGeneration(input: {
companyId: string; connectionId: string; grantId: string;
userId: string | null; agentId: string;
}) {
const [row] = await db.select({ connection: toolConnections, grant: connectionGrants })
.from(toolConnections).innerJoin(connectionGrants, and(
eq(connectionGrants.connectionId, toolConnections.id), eq(connectionGrants.companyId, toolConnections.companyId),
)).where(and(eq(toolConnections.companyId, input.companyId), eq(toolConnections.id, input.connectionId),
eq(connectionGrants.id, input.grantId), eq(toolConnections.connectionPurpose, "ai"))).limit(1);
if (!row) throw unprocessable("AI connection unavailable", { code: "connection_unavailable" });
const metadata = aiConnectionMetadataSchema.safeParse(row.connection.config.ai);
if (!metadata.success || metadata.data.provider !== "openai" || metadata.data.method !== "api_key" || metadata.data.routing)
throw unprocessable("Image generation needs an OpenAI API-key connection", { code: "incompatible_connection" });
if (row.grant.status !== "active" || !row.connection.enabled || row.connection.status !== "active" || row.connection.healthStatus !== "ok")
throw unprocessable("Reconnect or enable this connection", { code: "connection_unavailable" });
if (!input.userId || !(await membership(input.companyId, input.userId))) throw forbidden("Active responsible user required");
if (row.grant.kind === "user" && !(await membership(input.companyId, row.grant.subjectUserId)))
throw forbidden("The account owner is no longer an active company member");
const audience = await db.select().from(connectionGrantMembers).where(and(
eq(connectionGrantMembers.companyId, input.companyId), eq(connectionGrantMembers.grantId, input.grantId),
));
if (!canUseCredential(row.grant, input.userId, audience)) throw forbidden("Connection is not shared with this user");
const [install] = await db.select({ id: toolConnectionInstalls.id }).from(toolConnectionInstalls).where(and(
eq(toolConnectionInstalls.companyId, input.companyId), eq(toolConnectionInstalls.connectionId, input.connectionId),
or(and(eq(toolConnectionInstalls.targetType, "company"), eq(toolConnectionInstalls.targetId, input.companyId)),
and(eq(toolConnectionInstalls.targetType, "agent"), eq(toolConnectionInstalls.targetId, input.agentId))),
)).limit(1);
if (!install) throw forbidden("Connection is not permitted for this agent");
return row;
}
/** Internal fast-response use has its own capability check, never a pretend CLI harness. */
async function selectFastResponse(input: {
companyId: string; connectionId: string; grantId: string;
userId: string | null; agentId?: string | null; sponsoredBackground?: boolean;
}) {
const [row] = await db.select({ connection: toolConnections, grant: connectionGrants })
.from(toolConnections).innerJoin(connectionGrants, and(
eq(connectionGrants.connectionId, toolConnections.id), eq(connectionGrants.companyId, toolConnections.companyId),
)).where(and(eq(toolConnections.companyId, input.companyId), eq(toolConnections.id, input.connectionId),
eq(connectionGrants.id, input.grantId), eq(toolConnections.connectionPurpose, "ai"))).limit(1);
if (!row || row.grant.kind !== "organization") throw unprocessable("Shared AI connection unavailable", { code: "connection_unavailable" });
const metadata = aiConnectionMetadataSchema.safeParse(row.connection.config.ai);
const provider = metadata.success && metadata.data.method === "api_key" ? metadata.data.provider : null;
if (!provider) throw unprocessable("Connection does not support fast responses", { code: "incompatible_connection" });
if (row.grant.status !== "active" || !row.connection.enabled || row.connection.status !== "active" || row.connection.healthStatus !== "ok")
throw unprocessable("Reconnect or enable this connection", { code: "connection_unavailable" });
// Only internal fast responses may sponsor background use; ordinary AI selection never bypasses an audience.
if (!input.sponsoredBackground) {
if (!(await membership(input.companyId, input.userId))) throw forbidden("Active responsible user required");
const audience = await db.select().from(connectionGrantMembers).where(and(
eq(connectionGrantMembers.companyId, input.companyId), eq(connectionGrantMembers.grantId, input.grantId),
));
if (!canUseCredential(row.grant, input.userId, audience)) throw forbidden("Connection is not shared with this user");
} else if (input.userId) throw forbidden("Background sponsorship cannot replace caller permissions");
if (input.agentId) {
const [install] = await db.select({ id: toolConnectionInstalls.id }).from(toolConnectionInstalls).where(and(
eq(toolConnectionInstalls.companyId, input.companyId), eq(toolConnectionInstalls.connectionId, input.connectionId),
or(and(eq(toolConnectionInstalls.targetType, "company"), eq(toolConnectionInstalls.targetId, input.companyId)),
and(eq(toolConnectionInstalls.targetType, "agent"), eq(toolConnectionInstalls.targetId, input.agentId))),
)).limit(1);
if (!install) throw forbidden("Connection is not permitted for this agent");
}
return { ...row, provider };
}
async function selectPromptInjectionPrevention(input: {
companyId: string; connectionId: string; grantId: string;
userId: string | null; agentId?: string | null; sponsoredBackground?: boolean;
}) {
const [row] = await db.select({ connection: toolConnections, grant: connectionGrants })
.from(toolConnections).innerJoin(connectionGrants, and(
eq(connectionGrants.connectionId, toolConnections.id), eq(connectionGrants.companyId, toolConnections.companyId),
)).where(and(eq(toolConnections.companyId, input.companyId), eq(toolConnections.id, input.connectionId),
eq(connectionGrants.id, input.grantId), eq(toolConnections.connectionPurpose, "ai"))).limit(1);
if (!row || row.grant.kind !== "organization") throw unprocessable("Shared AI connection unavailable", { code: "connection_unavailable" });
const metadata = aiConnectionMetadataSchema.safeParse(row.connection.config.ai);
const provider = metadata.success && decisionProviderForConnection(metadata.data) === "openrouter" ? "openrouter" as const : null;
if (!provider) throw unprocessable("Connection does not support prompt injection checks", { code: "incompatible_connection" });
if (row.grant.status !== "active" || !row.connection.enabled || row.connection.status !== "active" || row.connection.healthStatus !== "ok")
throw unprocessable("Reconnect or enable this connection", { code: "connection_unavailable" });
// Only internal prompt injection checks may sponsor background use; ordinary AI selection never bypasses an audience.
if (!input.sponsoredBackground) {
if (!(await membership(input.companyId, input.userId))) throw forbidden("Active responsible user required");
const audience = await db.select().from(connectionGrantMembers).where(and(
eq(connectionGrantMembers.companyId, input.companyId), eq(connectionGrantMembers.grantId, input.grantId),
));
if (!canUseCredential(row.grant, input.userId, audience)) throw forbidden("Connection is not shared with this user");
} else if (input.userId) throw forbidden("Background sponsorship cannot replace caller permissions");
if (input.agentId) {
const [install] = await db.select({ id: toolConnectionInstalls.id }).from(toolConnectionInstalls).where(and(
eq(toolConnectionInstalls.companyId, input.companyId), eq(toolConnectionInstalls.connectionId, input.connectionId),
or(and(eq(toolConnectionInstalls.targetType, "company"), eq(toolConnectionInstalls.targetId, input.companyId)),
and(eq(toolConnectionInstalls.targetType, "agent"), eq(toolConnectionInstalls.targetId, input.agentId))),
)).limit(1);
if (!install) throw forbidden("Connection is not permitted for this agent");
}
return { ...row, provider };
}
async function credential(row: Pick<Awaited<ReturnType<typeof select>>, "connection" | "grant">, retry = 0, audit?: { responsibleUserId: string | null; actorType: "user" | "agent" | "system"; actorId: string; issueId?: string | null; heartbeatRunId?: string | null }): Promise<string> {
const metadata = aiConnectionMetadataSchema.parse(row.connection.config.ai);
if (metadata.routing?.auth === "none") return "";
const ref = row.grant.credentialSecretRefs.find(
(r) => r.configPath === "ai.credential",
);
if (!ref)
throw unprocessable("Reconnect this AI account", {
code: "ai_connection_credential_missing",
});
const [secret] = await db
.select()
.from(companySecrets)
.where(
and(
eq(companySecrets.id, ref.secretId),
eq(companySecrets.companyId, row.connection.companyId),
),
);
if (!secret)
throw unprocessable("Reconnect this AI account", {
code: "ai_connection_credential_missing",
});
if (row.grant.kind === "user" && secret.scope !== "user")
throw forbidden("Credential ownership mismatch");
if (row.grant.kind === "organization" && secret.scope !== "company") throw forbidden("Credential ownership mismatch");
const context = {
consumerType: "tool_connection" as const,
consumerId: row.connection.id,
configPath: ref.configPath,
responsibleUserId: row.grant.subjectUserId,
actorType: "system" as const,
...audit,
};
let value: string;
if (secret.scope === "user") {
if (
secret.ownerUserId !== row.grant.subjectUserId ||
!secret.userSecretDefinitionId
)
throw forbidden("Credential ownership mismatch");
const result = await secrets.resolveUserSecretValue(
row.connection.companyId,
{
definitionId: secret.userSecretDefinitionId,
responsibleUserId: secret.ownerUserId,
required: true,
version: "latest",
},
context,
);
if (!result) throw unprocessable("Reconnect this AI account");
value = result.value;
} else {
value = await secrets.resolveSecretValue(row.connection.companyId, secret.id, "latest", context);
}
// Wait behind a rotating writer (the share lock), then re-read instead of
// handing a run the pre-rotation token. A lock takes no new row version,
// so it never aborts a concurrent REPEATABLE READ admission.
const [latest] = await db.select().from(companySecrets).where(and(
eq(companySecrets.id, secret.id), eq(companySecrets.companyId, row.connection.companyId),
)).for("share");
if (!latest || latest.status !== "active") throw unprocessable("Reconnect this AI account");
if (latest.latestVersion !== secret.latestVersion) {
if (retry >= 2) throw unprocessable("AI credentials are changing. Retry this execution.", { code: "ai_connection_busy" });
return credential(row, retry + 1, audit);
}
return value;
}
async function save(
companyId: string,
userId: string,
input: CreateAiConnection | AiConnectionLoginIntent,
verifiedCredential: string,
loginSession?: { id: string; complete?: (tx: Db) => Promise<void> },
attemptStartedAt = new Date(),
managementContext?: ConnectionManagementContext,
verifiedAccountLabel?: string,
) {
await managementContext?.reauthorize();
const routing = "routing" in input ? input.routing : undefined;
if (!(await membership(companyId, userId)))
throw forbidden("An active company member must own this connection");
const reconnect = input.connectionId
? (await rows(companyId)).find(
(r) => r.connection.id === input.connectionId,
)
: undefined;
if (input.connectionId && !reconnect)
throw notFound("AI connection not found");
if (
reconnect &&
(reconnect.grant.createdByUserId !== userId ||
(reconnect.grant.kind === "user" &&
reconnect.grant.subjectUserId !== userId))
)
throw forbidden("Only the account owner can reconnect it");
if (
reconnect &&
(reconnect.connection.config.ai as AiConnectionMetadata).provider !==
input.provider
)
throw unprocessable("Reconnect cannot change providers");
if (reconnect && !isDeepStrictEqual((reconnect.connection.config.ai as AiConnectionMetadata).routing, routing))
throw unprocessable("Reconnect must retain this connection’s routing. Create another connection to change its destination.");
if (
reconnect &&
((reconnect.connection.config.ai as AiConnectionMetadata).method !==
input.method ||
(reconnect.grant.kind === "user") !== (input.ownership === "personal"))
)
throw unprocessable(
"Reconnect cannot change the sign-in method or ownership",
);
const id = reconnect?.connection.id ?? randomUUID();
const grantId = reconnect?.grant.id ?? randomUUID();
const persist = () => withAccountHomeSecretMutationLock(undefined, companyId, () => db.transaction(async (tx) => {
await managementContext?.reauthorize();
const secrets = secretService(tx);
if (loginSession) {
const [session] = await tx
.select()
.from(adapterAuthSessions)
.where(
and(
eq(adapterAuthSessions.id, loginSession.id),
eq(adapterAuthSessions.companyId, companyId),
eq(adapterAuthSessions.startedByUserId, userId),
),
)
.for("update");
if (!session) throw forbidden("Login session ownership mismatch");
attemptStartedAt = session.createdAt;
if (session.connectionId && session.connectionGrantId)
return {
connectionId: session.connectionId,
grantId: session.connectionGrantId,
};
if (
!["promoting", "submitting", "awaiting_code"].includes(
session.status,
) ||
(session.expiresAt && session.expiresAt.getTime() <= Date.now())
)
throw unprocessable("The login attempt is no longer active");
if (
!session.aiConnection ||
session.aiConnection.provider !== input.provider ||
session.aiConnection.method !== input.method ||
session.aiConnection.connectionId !== input.connectionId ||
session.aiConnection.ownership !== input.ownership ||
session.aiConnection.allAgents !== input.allAgents ||
JSON.stringify(session.aiConnection.agentIds) !==
JSON.stringify(input.agentIds)
)
throw forbidden("Login target mismatch");
}
if (reconnect) {
const [current] = await tx
.select()
.from(connectionGrants)
.where(eq(connectionGrants.id, grantId))
.for("update");
if (
!current ||
current.updatedAt.getTime() !== reconnect.grant.updatedAt.getTime() ||
current.updatedAt.getTime() > attemptStartedAt.getTime()
)
throw unprocessable("The connection changed. Start reconnect again.");
}
let secretId = reconnect?.grant.credentialSecretRefs.find(
(r) => r.configPath === "ai.credential",
)?.secretId;
// Adoption indexes existing credentials without transferring ownership.
// Only rotate the private slot created for this grant. A reconnect of an
// indexed credential must leave every legacy consumer's value untouched,
// even after adoption has cleared the connection's validation marker.
if (secretId) {
const [source] = await tx.select({ name: companySecrets.name, key: userSecretDefinitions.key })
.from(companySecrets)
.leftJoin(userSecretDefinitions, eq(userSecretDefinitions.id, companySecrets.userSecretDefinitionId))
.where(and(eq(companySecrets.companyId, companyId), eq(companySecrets.id, secretId)));
const privateSlot = input.ownership === "personal"
? source?.key === `ai_${grantId.replaceAll("-", "_")}`
: source?.name === `ai-${grantId}`;
if (!privateSlot) secretId = undefined;
}
if (!verifiedCredential) { secretId = undefined; }
else if (secretId)
await secrets.rotate(
secretId,
{ value: verifiedCredential },
{ userId },
);
else if (input.ownership === "personal") {
const definition = await secrets.createUserSecretDefinition(
companyId,
{
key: `ai_${grantId.replaceAll("-", "_")}`,
name: input.name,
provider: "local_encrypted",
},
{ userId },
);
const secret = await secrets.createCurrentUserSecretValue(
companyId,
userId,
{ definitionId: definition.id, value: verifiedCredential },
{ userId },
);
secretId = secret.id;
} else
secretId = (
await secrets.create(
companyId,
{
name: `ai-${grantId}`,
provider: "local_encrypted",
value: verifiedCredential,
},
{ userId },
)
).id;
if (input.agentIds.length) {
const targets = await tx
.select({ id: agents.id })
.from(agents)
.where(
and(
eq(agents.companyId, companyId),
inArray(agents.id, input.agentIds),
),
);
if (targets.length !== new Set(input.agentIds).size)
throw forbidden("Agent does not belong to this company");
}
const source = aiConnectionCatalogSlug(input.provider, routing);
const providerName = getAppStoreDefinition(source)?.name ?? AI_CONNECTION_CAPABILITIES[input.provider].name;
const key = `app-gallery:${source}`;
await tx
.insert(toolApplications)
.values({
companyId,
applicationKey: key,
name: providerName,
type: "mcp_http",
metadata: { sourceTemplateKey: source },
ownerUserId: userId,
})
.onConflictDoNothing();
const [app] = await tx
.select()
.from(toolApplications)
.where(
and(
eq(toolApplications.companyId, companyId),
or(
eq(toolApplications.applicationKey, key),
eq(
toolApplications.name,
providerName,
),
),
),
);
if (!app) throw unprocessable("Could not find the provider application");
if (reconnect)
await tx
.update(toolConnections)
.set({
enabled: true,
status: "active",
healthStatus: "ok",
healthMessage: null,
config: { ...reconnect.connection.config, aiIsolatedSubscription: input.method === "subscription" && input.provider !== "anthropic" },
updatedAt: new Date(),
})
.where(eq(toolConnections.id, id));
else
await tx
.insert(toolConnections)
.values({
id,
companyId,
applicationId: app.id,
name: input.name,
uid: `ai-${id}`,
connectionPurpose: "ai",
transport: "runtime_auth",
authKind: input.method === "api_key" ? "api_key" : "oauth",
credentialPolicy:
input.ownership === "personal" ? "per_user" : "shared",
status: "active",
enabled: true,
healthStatus: "ok",
config: {
sourceTemplateKey: source,
ai: { provider: input.provider, method: input.method, ...(routing ? { routing } : {}) },
aiIsolatedSubscription: input.method === "subscription" && input.provider !== "anthropic",
},
createdByUserId: userId,
});
let accountLabel: string | undefined = input.provider === "google" ? verifiedAccountLabel : undefined;
if (input.method === "subscription" && input.provider !== "anthropic") {
try {
const credential = JSON.parse(verifiedCredential);
const claims = credential.tokens?.id_token
? JSON.parse(
Buffer.from(
credential.tokens.id_token.split(".")[1],
"base64url",
).toString(),
)
: credential;
const email = claims.email ?? claims.user?.email;
if (
typeof email === "string" &&
/^[^\s@/\\]{1,100}@[^\s@/\\]{1,100}\.[^\s@/\\]{2,40}$/.test(email)
)
accountLabel = email;
} catch {
/* Safe account identity is optional. */
}
}
const refs = secretId ? [
{
secretId: secretId!,
configPath: "ai.credential",
required: true,
versionSelector: "latest" as const,
},
] : [];
if (reconnect)
await tx
.update(connectionGrants)
.set({
status: "active",
providerTenant: accountLabel ? { name: accountLabel } : {},
credentialSecretRefs: refs,
revokedAt: null,
updatedAt: new Date(),
})
.where(eq(connectionGrants.id, grantId));
else
await tx
.insert(connectionGrants)
.values({
id: grantId,
companyId,
connectionId: id,
kind: input.ownership === "personal" ? "user" : "organization",
subjectUserId: input.ownership === "personal" ? userId : null,
isDefault: input.ownership === "shared",
providerTenant: accountLabel ? { name: accountLabel } : {},
credentialSecretRefs: refs,
createdByUserId: userId,
});
const [savedConnection] = await tx
.select()
.from(toolConnections)
.where(eq(toolConnections.id, id));
await syncConnectionCredentialBindings(tx, savedConnection, refs);
if (input.ownership === "personal" && !routing) {
await tx
.insert(aiConnectionDefaults)
.values({
companyId,
userId,
provider: input.provider,
method: input.method,
grantId,
})
.onConflictDoNothing();
await tx.insert(aiProviderDefaults).values({ companyId, userId, provider: input.provider, grantId }).onConflictDoNothing();
}
if (!reconnect) {
const installs = input.allAgents
? [{ targetType: "company" as const, targetId: companyId }]
: input.agentIds.map((targetId) => ({
targetType: "agent" as const,
targetId,
}));
if (installs.length)
await tx
.insert(toolConnectionInstalls)
.values(
installs.map((i) => ({
...i,
companyId,
connectionId: id,
createdByUserId: userId,
})),
);
}
if (loginSession) await notifyLoginSessionChanged(tx);
if (loginSession)
await tx
.update(adapterAuthSessions)
.set({
connectionId: id,
connectionGrantId: grantId,
connectionMethod: input.method,
})
.where(
eq(adapterAuthSessions.id, loginSession.id),
);
await loginSession?.complete?.(tx as unknown as Db);
await logActivity(tx as unknown as Db, {
companyId,
...(managementContext?.audit ?? { actorType: "user" as const, actorId: userId }),
action: reconnect
? "ai_connection.reconnected"
: "ai_connection.connected",
entityType: "tool_connection",
entityId: id,
details: { provider: input.provider, method: input.method, grantId },
});
return { connectionId: id, grantId };
}));
// Quota refresh can hold this lock through a 60-second token exchange.
// Keep the verified sign-in in memory across two 30-second acquisition
// timeouts instead of making its caller discard a successful login.
for (let attempt = 0; ; attempt++) {
try { return await persist(); }
catch (error) {
if (attempt >= 2 || (error as NodeJS.ErrnoException)?.code !== WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE) throw error;
}
}
}
async function recoverQuotaCredentialInTransaction(
tx: Db, row: Parameters<typeof credential>[0], current: string, onRecoveryStarted?: () => void,
) {
const metadata = aiConnectionMetadataSchema.safeParse(row.connection.config.ai);
if (!metadata.success || !["openai", "google"].includes(metadata.data.provider) || metadata.data.method !== "subscription") return null;
const recovery = quotaCredentialRecovery(row.connection.companyId, row.grant.id);
const pending = await recovery.read();
if (!pending) return null;
const ref = row.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential");
if (!ref || (pending.provider ?? "openai") !== metadata.data.provider
|| pending.secretId !== ref.secretId || pending.connectionId !== row.connection.id || pending.baseHash !== quotaCredentialHash(current)) {
await recovery.clear();
return null;
}
if (metadata.data.provider === "google") {
const [secret] = await tx.select({ epoch: companySecrets.aiSessionEpoch, version: companySecrets.latestVersion })
.from(companySecrets).where(and(eq(companySecrets.companyId, row.connection.companyId), eq(companySecrets.id, ref.secretId))).for("update");
if (!pending.google || pending.google.sessionEpoch !== secret?.epoch || pending.google.baseVersion !== secret?.version
|| pending.google.versionSelector !== (ref.versionSelector ?? "latest")
|| pending.google.accountEmail.toLowerCase() !== row.grant.providerTenant?.name?.toLowerCase()) {
await recovery.clear();
return null;
}
if (!pending.google.verified) {
onRecoveryStarted?.();
throw new Error("Google credential recovery needs account verification");
}
}
onRecoveryStarted?.();
await secretService(tx).rotate(ref.secretId, { value: pending.value, preserveAiSessionEpoch: true }, { userId: row.grant.subjectUserId });
// Automatic rotation changes secret revision, not the grant mutation marker
// used to fence reconnect attempts. Ownership and access are unchanged.
await logActivity(tx, { companyId: row.connection.companyId, actorType: "system", actorId: "quota_recovery",
action: "ai_connection.credential_refreshed", entityType: "tool_connection", entityId: row.connection.id,
details: { provider: metadata.data.provider, grantId: row.grant.id } });
return pending.value;
}
async function runtimeCredential(row: Parameters<typeof credential>[0]) {
const metadata = aiConnectionMetadataSchema.safeParse(row.connection.config.ai);
if (!metadata.success || !["openai", "google"].includes(metadata.data.provider) || metadata.data.method !== "subscription") return credential(row);
if (!(await quotaCredentialRecovery(row.connection.companyId, row.grant.id).read())) return credential(row);
let savingReplacement = false;
return withAccountHomeSecretMutationLock(undefined, row.connection.companyId, async () => {
if (metadata.data.provider === "google") {
const recovery = quotaCredentialRecovery(row.connection.companyId, row.grant.id);
const pending = await recovery.read();
if (pending?.provider === "google" && pending.google && !pending.google.verified
&& pending.connectionId === row.connection.id
&& pending.google.accountEmail.toLowerCase() === row.grant.providerTenant?.name?.toLowerCase()) {
const ref = row.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential");
const [secret] = await db.select({ epoch: companySecrets.aiSessionEpoch, version: companySecrets.latestVersion })
.from(companySecrets).where(and(eq(companySecrets.companyId, row.connection.companyId), eq(companySecrets.id, pending.secretId)));
const current = await credential(row);
if (ref?.secretId === pending.secretId && (ref.versionSelector ?? "latest") === pending.google.versionSelector
&& secret?.epoch === pending.google.sessionEpoch && secret?.version === pending.google.baseVersion
&& pending.baseHash === quotaCredentialHash(current)) {
savingReplacement = true;
// Network verification runs before acquiring database row locks.
// The transaction below rechecks the exact saved generation.
await prepareAntigravityCredentialRecovery({ originalCredential: current,
refreshedCredential: pending.value, accountEmail: pending.google.accountEmail,
scope: { companyId: row.connection.companyId, accountKey: row.grant.id },
persistVerified: value => recovery.write({ ...pending, value, google: { ...pending.google!, verified: true } }),
});
}
}
}
let recovered = false;
const value = await db.transaction(async tx => {
const [grant] = await tx.select().from(connectionGrants).where(and(
eq(connectionGrants.id, row.grant.id), eq(connectionGrants.companyId, row.connection.companyId),
eq(connectionGrants.connectionId, row.connection.id),
)).for("update", { noWait: true });
if (!grant || grant.status !== "active") throw new Error("credentials_unavailable");
const ref = grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential");
if (!ref) throw new Error("credentials_unavailable");
await tx.select({ id: companySecrets.id }).from(companySecrets).where(and(
eq(companySecrets.id, ref.secretId), eq(companySecrets.companyId, row.connection.companyId),
)).for("update", { noWait: true });
const selected = { connection: row.connection, grant };
const current = await aiConnectionService(tx as unknown as Db).credential(selected);
const pending = await recoverQuotaCredentialInTransaction(tx as unknown as Db, selected, current, () => {
savingReplacement = true;
});
recovered = pending !== null;
return pending ?? current;
});
if (recovered) await quotaCredentialRecovery(row.connection.companyId, row.grant.id).clear()
.catch(error => logger.warn({ err: error, grantId: row.grant.id }, "Saved runtime credential recovery cleanup pending"));
return value;
}).catch(error => {
// Once a matching replacement is found, a failed write or commit must
// defer execution. Keep its journal so the next attempt can save it.
if (savingReplacement) {
throw unprocessable("AI credentials are being updated. Retry shortly.", { code: "ai_connection_busy" });
}
// Drizzle wraps Postgres errors in `cause`. Contention is a temporary
// pre-provider wait, not evidence that this account needs reconnecting.
let current: unknown = error;
for (let depth = 0; depth < 4 && current && typeof current === "object"; depth += 1) {
const value = current as { code?: unknown; cause?: unknown };
if (value.code === WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE || value.code === "55P03") {
throw unprocessable("AI credentials are being updated. Retry shortly.", { code: "ai_connection_busy" });
}
current = value.cause;
}
throw error;
});
}
/** A late failure must never invalidate credentials that were refreshed or reconnected meanwhile. */
async function markAuthenticationFailed(input: {
companyId: string;
runId?: string;
agentId?: string;
runStartedAt: Date;
attribution: AiConnectionAttribution & { identity: string };
}) {
return withAccountHomeSecretMutationLock(undefined, input.companyId, async () => {
let recovered = false;
await db.transaction(async (tx) => {
const { attribution } = input;
const [grant] = await tx.select().from(connectionGrants).where(and(
eq(connectionGrants.companyId, input.companyId),
eq(connectionGrants.id, attribution.grantId),
eq(connectionGrants.connectionId, attribution.connectionId),
)).for("update");
if (!grant || grant.status !== "active" || grant.updatedAt > input.runStartedAt) return;
const [connection] = await tx.select().from(toolConnections).where(and(
eq(toolConnections.companyId, input.companyId),
eq(toolConnections.id, attribution.connectionId),
eq(toolConnections.connectionPurpose, "ai"),
));
if (!connection) return;
const metadata = aiConnectionMetadataSchema.safeParse(connection.config.ai);
if (!metadata.success || metadata.data.provider !== attribution.provider || metadata.data.method !== attribution.method) return;
const ref = grant.credentialSecretRefs.find((candidate) => candidate.configPath === "ai.credential");
if (!ref) return;
await tx.select({ id: companySecrets.id }).from(companySecrets).where(and(
eq(companySecrets.companyId, input.companyId), eq(companySecrets.id, ref.secretId),
)).for("update");
const value = await aiConnectionService(tx as unknown as Db).credential({ connection, grant });
const generation = createHash("sha256").update(value).digest("hex").slice(0, 16);
if (attribution.identity !== `${grant.id}:${attribution.responsibleUserId ?? "shared"}:${generation}`) return;
// A quota exchange may have consumed this run's token before a failed
// database commit. Save its working replacement instead of revoking it.
recovered = await recoverQuotaCredentialInTransaction(tx as unknown as Db, { connection, grant }, value) !== null;
if (recovered) return;
await tx.update(connectionGrants).set({ status: "needs_reauthorization", updatedAt: new Date() })
.where(eq(connectionGrants.id, grant.id));
await tx.update(toolConnections).set({ healthStatus: "error", healthMessage: "Sign in again to restore this AI connection.", updatedAt: new Date() })
.where(eq(toolConnections.id, connection.id));
await logActivity(tx as unknown as Db, {
companyId: input.companyId, actorType: "system", actorId: input.runId ? "heartbeat" : "adapter_test",
agentId: input.agentId, runId: input.runId, action: "ai_connection.authentication_failed",
entityType: "tool_connection", entityId: connection.id,
details: { provider: attribution.provider, grantId: grant.id },
});
});
if (recovered) await quotaCredentialRecovery(input.companyId, input.attribution.grantId).clear()
.catch(error => logger.warn({ err: error, grantId: input.attribution.grantId }, "Saved failed-run credential recovery cleanup pending"));
});
}
/** Refresh only the selected vaulted identity, serializing rotation and reconnect. */
async function refreshQuotaCredential(row: Parameters<typeof credential>[0], failedValue: string, signal: AbortSignal) {
return withAccountHomeSecretMutationLock(undefined, row.connection.companyId, async () => {
const recovery = quotaCredentialRecovery(row.connection.companyId, row.grant.id);
let replacement: { value: string; secretId: string; version: number; epoch: number; base: string; grantUpdatedAt: number; versionSelector: "latest" | number } | undefined;
const save = () => db.transaction(async (tx) => {
// Do not wait on a transaction that might itself be waiting for the file
// lock. All database locks must be available before consuming OAuth tokens.
const companyId = row.connection.companyId;
const [grant] = await tx.select().from(connectionGrants).where(and(
eq(connectionGrants.id, row.grant.id), eq(connectionGrants.companyId, companyId),
eq(connectionGrants.connectionId, row.connection.id),
)).for("update", { noWait: true });
if (!grant || grant.status !== "active") throw new Error("credentials_unavailable");
const ref = grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential");
if (!ref) throw new Error("credentials_unavailable");
const [secret] = await tx.select({ id: companySecrets.id, version: companySecrets.latestVersion, epoch: companySecrets.aiSessionEpoch }).from(companySecrets).where(and(
eq(companySecrets.id, ref.secretId), eq(companySecrets.companyId, companyId),
)).for("update", { noWait: true });
if (!secret) throw new Error("credentials_unavailable");
const current = await aiConnectionService(tx as unknown as Db).credential({ connection: row.connection, grant });
if (replacement) {
// A lost commit response is reconciled from the authoritative secret.
// Reconnect, revocation and a competing saved generation always win.
if (current === replacement.value) return current;
if (ref.secretId !== replacement.secretId || secret.version !== replacement.version
|| secret.epoch !== replacement.epoch || current !== replacement.base
|| grant.updatedAt.getTime() !== replacement.grantUpdatedAt
|| (ref.versionSelector ?? "latest") !== replacement.versionSelector) return current;
} else signal.throwIfAborted();
let pending = await recovery.read();
if (pending && (pending.secretId !== ref.secretId || pending.connectionId !== row.connection.id || pending.baseHash !== quotaCredentialHash(current))) {
// Reconnect may reuse this grant. Once the current authorized token
// itself expires, an older journal must not prevent refreshing it.
await recovery.clear();
pending = null;
}
// A delayed poll may have read A before B was saved and B's refresh
// produced an uncommitted C. Recover C before returning or clearing its
// journal; a newer database credential alone does not prove it is usable.
if (current !== failedValue && !pending) return current;
// An active provider may rotate this single-use token itself. Defer polling
// until its credential write-back instead of invalidating its live copy.
const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.innerJoin(agents, and(eq(agents.id, heartbeatRuns.agentId), eq(agents.companyId, companyId))).where(and(
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["queued", "running"]),
or(sql`${heartbeatRuns.contextSnapshot}->'aiConnection'->>'grantId' = ${grant.id}`,
sql`${agents.runtimeConfig}->'aiConnection'->>'provider' = 'openai'`, eq(agents.adapterType, "codex_local")),
)).limit(1);
if (active && !pending && !replacement) throw new Error("provider_unavailable");
let refreshed = replacement?.value ?? pending?.value;
if (!refreshed) {
const auth = JSON.parse(current);
const refreshToken = auth.tokens?.refresh_token;
if (typeof refreshToken !== "string" || !refreshToken) throw new Error("authentication_required");
// Once exchange starts, its lifetime is independent of the dashboard's
// shorter deadline. A consumed single-use token must still be read/saved.
signal.throwIfAborted();
const refreshSignal = AbortSignal.timeout(60_000);
// Matches the Codex OAuth client (openai/codex, login/src/auth/manager.rs).
const response = await fetch("https://auth.openai.com/oauth/token", {
method: "POST", redirect: "error", signal: refreshSignal,
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ client_id: "app_EMoamEEZ73f0CkXaXp7hrann", grant_type: "refresh_token", refresh_token: refreshToken }),
});
if (!response.ok) {
await response.body?.cancel();
throw new Error(response.status === 400 || response.status === 401 ? "authentication_required" : "provider_unavailable");
}
const tokens = await response.json() as Record<string, unknown>;
if (typeof tokens.access_token !== "string" || !tokens.access_token) throw new Error("provider_unavailable");
refreshed = JSON.stringify({ ...auth, tokens: { ...auth.tokens,
access_token: tokens.access_token,
...(typeof tokens.refresh_token === "string" && tokens.refresh_token ? { refresh_token: tokens.refresh_token } : {}),
...(typeof tokens.id_token === "string" && tokens.id_token ? { id_token: tokens.id_token } : {}),
}, last_refresh: new Date().toISOString() });
}
// Only this logical refresh owns the replacement. Save retries must not
// consume another single-use token. Process loss before commit can require
// signing in again in v1; no secondary credential vault is created.
replacement ??= { value: refreshed, secretId: ref.secretId, version: secret.version, epoch: secret.epoch, base: current,
grantUpdatedAt: grant.updatedAt.getTime(), versionSelector: ref.versionSelector ?? "latest" };
// Persist the rotated token even if the quota deadline expires after the
// successful exchange; losing it would strand all subsequent executions.
await secretService(tx).rotate(ref.secretId, { value: refreshed, preserveAiSessionEpoch: true }, { userId: grant.subjectUserId });
await logActivity(tx as unknown as Db, { companyId, actorType: "system", actorId: "quota",
action: "ai_connection.credential_refreshed", entityType: "tool_connection", entityId: row.connection.id,
details: { provider: "openai", grantId: grant.id } });
return refreshed;
});
for (let attempt = 0; ; attempt++) {
try {
const value = await save();
// Clear only after commit. A committed-but-lost response is detected
// by the current credential comparison on the next attempt.
await recovery.clear().catch((error) => logger.warn({ err: error, grantId: row.grant.id }, "Saved quota credential recovery cleanup pending"));
return value;
} catch (error) {
if (!replacement || attempt >= 2) throw error;
// Retry only the returned value. Required database reads still fail
// closed; exhaustion is an error, never a successful refresh.
}
}
});
}
// A quota read uses the same credential audience as execution. Operator status
// alone never grants access to another member's personal subscription.
async function quotaAccounts(companyId: string, userId: string, includeOtherProviders = false) {
if (!(await membership(companyId, userId))) return [];
const visible = await list(companyId, userId);
const accounts = await rows(companyId);
return accounts.flatMap(row => {
const summary = visible.find(item => item.id === row.connection.id && item.grantId === row.grant.id);
if (!summary || summary.method !== "subscription" || !(includeOtherProviders ? ["openai", "anthropic", "xai"] : ["openai", "anthropic"]).includes(summary.provider)) return [];
return [{ ...row, summary }];
});
}
return { list, selectDecision, selectFastResponse, selectImageGeneration, selectPromptInjectionPrevention, quotaAccounts, subscriptionAccounts: (companyId: string, userId: string) => quotaAccounts(companyId, userId, true), refreshQuotaCredential, select, credential, runtimeCredential, probeUsage, accountIdentity, save, setDefault, membership, markAuthenticationFailed };
}