Files
PaperClipAI/server/src/services/ai-connections.ts
T
DottaandPaperclip 2b14ec03a4 fix(ai-connections): remove account execution locks (#18778)
## Thinking Path

> - Paperclip manages AI agents and the accounts that supply their model
credentials.
> - Several runs can use the same AI connection at the same time.
> - Runtime preparation treated unfinished credential cleanup as
exclusive account ownership.
> - Manual account tests also held a company credential mutex for the
entire provider test.
> - A failed cleanup could therefore cancel fresh runs before the
provider started and keep them in automatic retries.
> - This pull request removes that execution gate and those read and
probe locks.
> - Each invocation keeps its own credential home. Credential updates
still use the existing freshness and authorization checks.

## Linked Issues or Issue Description

**What happened?**

Fresh runs stopped with `ai_connection_busy` and the message “The
selected AI account is still completing credential cleanup. Retry
shortly.” Pending cleanup from an earlier invocation blocked every new
invocation of that account. Concurrent manual account tests also waited
on a company mutex.

**Expected behavior**

AI connections support concurrent runs and account tests. Cleanup for
one private runtime must not prevent another private runtime from using
the same account.

**Steps to reproduce**

1. Prepare a managed OpenAI account and retain an earlier runtime whose
cleanup has not finished.
2. Prepare another run with that account. The old implementation rejects
it before provider work starts.
3. Alternatively, keep one manual account test open and start a second
test with the same account. The old implementation prevents the second
provider callback from starting.

**Paperclip version or commit**

Reproduced with two PostgreSQL regressions on
`17e8bd773543d62ab3f444c55255acfd82575efd`. Both fail before the fix and
pass after it.

**Deployment mode**

The reported deployment used authenticated Cloud. Local reproduction
uses disposable PostgreSQL and fixture credentials. No live provider
credentials are required for these regressions.

Related search found #15952, which concerns account removal and provider
application archival. It does not remove this runtime cleanup gate. This
fix does not introduce the account concurrency cap requested in #14564.
ROADMAP.md was checked; this change restores concurrent account use.

## What Changed

- Remove the account-wide cleanup assertion and cleanup mutex from run
admission.
- Read ordinary subscription credentials without acquiring the company
file mutex or grant and secret write locks. Only an actual durable
replacement recovery enters the existing write transaction.
- Remove the mutex held across manual account tests.
- Retry retained cleanup without making new runtime preparation wait for
it. A home-specific cleanup check considers only that exact private
home.
- Keep cleanup attempts for one runtime idempotent. Keep physical
retirement, credential freshness, reconnect, revocation and company
checks.
- Update concurrency, refresh and cleanup regressions, and the
implementation contract.

## Verification

- Before/after: the pending-cleanup and simultaneous-probe regressions
both fail on the original code and pass with the fix.
- Affected suites: 186 tests passed across `ai-connections.test.ts`,
`antigravity-ai-connections.integration.test.ts`, and
`retained-google-credential-cleanup.test.ts`. They use real PostgreSQL,
encrypted fixture credentials and controlled provider callbacks.
- `pnpm -r typecheck` passed.
- `node scripts/check-maps.mjs` and `git diff --check` passed.
- `pnpm build` passed on the rebased branch. The server typecheck and
production server build both passed again on final commit
`03bd3973bdf85108bbd82355abdb217f0e1c83bd` after the runtime-home
ownership changes.
- Caller coverage: all 166 cases pass across native credential handoff,
managed Google subscription writeback, model discovery, heartbeat
configuration and hired-agent execution. The full hiring file passed all
79 cases after its admission assertion was corrected. Together with the
affected suites, 352 targeted cases pass.
- The final home-binding check passed all 188 cases across the account,
managed Google writeback, model discovery and retained-cleanup suites.
- Review follow-up: two real-PostgreSQL cases pass for quota refresh and
concurrent Google account tests. The quota case observes a credential
row update in progress, verifies that the runtime takes no account file
mutex, and confirms that the private home receives the committed
replacement token. The Google case completes a second provider callback
while the first callback remains open in a separate private home.
- Greptile rated the runtime change 4/5 and found only misplaced map
patch notes. Those additions were removed. The implementation
documentation remains in `doc/connections/AI-CONNECTIONS.md` and the
implementation contract.
- `pnpm test:run` was attempted. Chat partition 1 passed 351 expanded
cases. Partition 2 passed 324 cases and failed one Teams file-receipt
API case at its 15-second deadline: `refuses generic resolution with
stale version transfer preconditions without changing receipts`. That
case exercises publication resolution, not AI account execution. Its
isolated rerun passed. The remaining local partitions did not run after
that failure. This is not a claim that the full local suite passed.
- Wider caller checks initially identified old serialization
expectations and two cleanup retry cases. Updated assertions now require
concurrent account use. The partial-removal correction lets a retained
home finish removal after a durable credential handoff. All affected
caller cases pass, including the full hiring file.
- Live provider concurrency and deployment recovery have not been
qualified. This PR does not claim that the reported Cloud deployment
contains the fix.

## Risks

- A new invocation can use the currently saved credential while another
runtime still owns a refreshed copy. The existing writeback checks
decide which credential can replace the saved value.
- Actual credential updates and durable replacement recovery retain
atomic storage transactions. This change removes execution
serialization, not the integrity checks on stored credentials.
- Unfinished cleanup retains its private home and its existing idle
hold. This change does not certify a previous provider process as
stopped.
- There is no database migration, credential format change, permission
expansion or ambient-account fallback.

## Model Used

- OpenAI GPT-6 through Codex desktop, with source editing, shell
execution, PostgreSQL fixture tests and read-only browser diagnosis. The
session does not expose the exact deployment ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 07:56:37 -06:00

1234 lines
61 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { notifyLoginSessionChanged } from "./login-cleanup-notifications.js";
import type { ConnectionManagementContext } from "./connection-management.js";
import { WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE } from "@paperclipai/adapter-utils/workspace-restore-merge";
import { withAccountHomeSecretMutationLock } from "@paperclipai/adapter-codex-local/server";
import { syncConnectionCredentialBindings } from "./connection-credential-bindings.js";
import { createHash, randomUUID } from "node:crypto";
import { isDeepStrictEqual } from "node:util";
import { and, eq, inArray, or, sql } from "drizzle-orm";
import {
type Db,
authUsers,
adapterAuthSessions,
aiConnectionDefaults,
aiProviderDefaults,
agents,
heartbeatRuns,
companyMemberships,
companySecrets,
userSecretDefinitions,
connectionGrants,
connectionGrantMembers,
toolApplications,
toolConnections,
toolConnectionInstalls,
} from "@paperclipai/db";
import {
AI_CONNECTION_CAPABILITIES,
decisionProviderForConnection,
aiConnectionCatalogSlug,
getAppStoreDefinition,
aiConnectionMetadataSchema,
aiSubscriptionNeedsIsolatedLogin,
isAiConnectionCompatible,
supportsAiConnectionUsage,
type AiConnectionBinding,
type AiConnectionAttribution,
type AiConnectionMetadata,
type AiManagedConnectionSummary,
type AiConnectionAccountIdentity,
type AiConnectionUsage,
type CreateAiConnection,
type AiConnectionLoginIntent,
} from "@paperclipai/shared";
import { forbidden, notFound, unprocessable } from "../errors.js";
import { aiConnectionConfigurationFailure, aiConnectionCredentialNotSharedFailure } from "./ai-connection-configuration-failure.js";
import { quotaCredentialRecovery, quotaCredentialHash } from "./quota-credential-recovery.js";
import { prepareAntigravityCredentialRecovery } from "./native-runtime/antigravity-runner-credentials.js";
import { logger } from "../middleware/logger.js";
import { logActivity } from "./activity-log.js";
import { secretService } from "./secrets.js";
import { probeAiConnectionUsage } from "./ai-connection-usage.js";
import { probeAiAccountIdentity } from "./ai-account-identity.js";
/** Same human audience displayed by the existing Connections identity controls. */
function canUseCredential(
grant: { kind: string; subjectUserId: string | null },
userId: string | null,
audience: { subjectType: string; subjectId: string }[],
) {
if (!userId) return false;
if (grant.kind === "user") return grant.subjectUserId === userId;
return grant.kind === "organization" && (
audience.length === 0 || audience.some((member) => member.subjectType === "user" && member.subjectId === userId)
);
}
export function aiConnectionService(db: Db) {
const secrets = secretService(db);
async function membership(companyId: string, userId: string | null) {
if (!userId) return false;
return Boolean(
(
await db
.select({ id: companyMemberships.id })
.from(companyMemberships)
.where(
and(
eq(companyMemberships.companyId, companyId),
eq(companyMemberships.principalType, "user"),
eq(companyMemberships.principalId, userId),
eq(companyMemberships.status, "active"),
),
)
.limit(1)
)[0],
);
}
async function rows(companyId: string) {
return db
.select({ connection: toolConnections, grant: connectionGrants })
.from(toolConnections)
.innerJoin(
connectionGrants,
and(
eq(connectionGrants.companyId, toolConnections.companyId),
eq(connectionGrants.connectionId, toolConnections.id),
),
)
.where(
and(
eq(toolConnections.companyId, companyId),
eq(toolConnections.connectionPurpose, "ai"),
),
);
}
async function list(
companyId: string,
userId: string,
_agentId?: string,
): Promise<AiManagedConnectionSummary[]> {
const [accounts, defaults, members, owners] =
await Promise.all([
rows(companyId),
db
.select()
.from(aiProviderDefaults)
.where(
and(
eq(aiProviderDefaults.companyId, companyId),
eq(aiProviderDefaults.userId, userId),
),
),
db
.select()
.from(connectionGrantMembers)
.where(eq(connectionGrantMembers.companyId, companyId)),
db
.select({ id: authUsers.id, name: authUsers.name })
.from(authUsers)
.innerJoin(
companyMemberships,
and(
eq(companyMemberships.principalId, authUsers.id),
eq(companyMemberships.companyId, companyId),
eq(companyMemberships.principalType, "user"),
),
),
]);
return accounts.flatMap(({ connection, grant }) => {
const metadata = aiConnectionMetadataSchema.safeParse(
connection.config.ai,
);
if (!metadata.success) return [];
const needsReconnect = aiSubscriptionNeedsIsolatedLogin(connection.config);
if (!canUseCredential(grant, userId, members.filter((m) => m.grantId === grant.id)))
return [];
return [
{
id: connection.id,
grantId: grant.id,
companyId,
...metadata.data,
usageProbeSupported: supportsAiConnectionUsage(metadata.data.provider, metadata.data.method),
name: connection.name,
accountLabel: grant.providerTenant?.name,
...(needsReconnect ? { unavailableReason: "Reconnect with a separate sign-in to protect your existing terminal login." } : {}),
ownership:
grant.kind === "user" ? ("personal" as const) : ("shared" as const),
ownerUserId: grant.subjectUserId ?? undefined,
ownerName:
owners.find((owner) => owner.id === grant.subjectUserId)?.name ??
(grant.subjectUserId === userId ? "You" : "Account owner"),
isDefault: defaults.some((d) => d.grantId === grant.id),
status:
grant.status === "revoked"
? ("revoked" as const)
: grant.status === "expired"
? ("expired" as const)
: grant.status !== "active" ||
!connection.enabled ||
connection.status !== "active" ||
connection.healthStatus !== "ok" || needsReconnect
? ("needs_attention" as const)
: ("connected" as const),
},
];
});
}
async function setDefault(
companyId: string,
userId: string,
grantId: string,
) {
return db.transaction(async (tx) => {
const [row] = await tx
.select({ connection: toolConnections, grant: connectionGrants })
.from(connectionGrants)
.innerJoin(
toolConnections,
eq(toolConnections.id, connectionGrants.connectionId),
)
.where(
and(
eq(connectionGrants.companyId, companyId),
eq(connectionGrants.id, grantId),
eq(connectionGrants.subjectUserId, userId),
eq(connectionGrants.kind, "user"),
),
)
.for("update");
if (!row)
throw forbidden("Only the owner can choose their personal default");
if (
row.connection.connectionPurpose !== "ai" ||
row.grant.status !== "active" ||
!row.connection.enabled ||
row.connection.healthStatus !== "ok"
)
throw unprocessable(
"Reconnect this account before making it your default",
);
const metadata = aiConnectionMetadataSchema.parse(row.connection.config.ai);
if (metadata.routing) throw unprocessable("Custom providers use an explicit connection selection on the agent.");
// Keep old servers' method preferences intact during an additive rollout.
await tx.insert(aiConnectionDefaults)
.values({ companyId, userId, ...metadata, grantId })
.onConflictDoUpdate({
target: [aiConnectionDefaults.companyId, aiConnectionDefaults.userId, aiConnectionDefaults.provider, aiConnectionDefaults.method],
set: { grantId, updatedAt: new Date() },
});
await tx.insert(aiProviderDefaults)
.values({ companyId, userId, provider: metadata.provider, grantId })
.onConflictDoUpdate({
target: [aiProviderDefaults.companyId, aiProviderDefaults.userId, aiProviderDefaults.provider],
set: { grantId, updatedAt: new Date() },
});
});
}
async function readableAccount(companyId: string, userId: string, connectionId: string, grantId?: string) {
if (!(await membership(companyId, userId))) throw forbidden("An active company member is required");
const candidates = (await rows(companyId)).filter((row) => row.connection.id === connectionId && (!grantId || row.grant.id === grantId));
const audience = await db.select().from(connectionGrantMembers).where(eq(connectionGrantMembers.companyId, companyId));
const visible = candidates.filter(({ grant }) => canUseCredential(grant, userId, audience.filter((member) => member.grantId === grant.id)));
if (!visible.length) throw notFound("AI connection not found");
if (visible.length > 1) throw unprocessable("Choose a credential grant for this account check");
const row = visible[0]!;
const metadata = aiConnectionMetadataSchema.safeParse(row.connection.config.ai);
if (!metadata.success) throw notFound("AI connection not found");
return { row, metadata: metadata.data };
}
/** On-demand account observation for either runner; never probes ambient CLI auth. */
async function probeUsage(companyId: string, userId: string, connectionId: string, grantId?: string): Promise<AiConnectionUsage> {
const { row, metadata } = await readableAccount(companyId, userId, connectionId, grantId);
const identity = { connectionId: row.connection.id, grantId: row.grant.id, ...metadata };
const empty = { ...identity, checkedAt: new Date().toISOString(), source: null, planType: null, limits: [], overage: null };
if (row.grant.status !== "active" || !row.connection.enabled || row.connection.status !== "active" ||
row.connection.healthStatus !== "ok" || aiSubscriptionNeedsIsolatedLogin(row.connection.config)) {
return { ...empty, status: "unavailable", errorCode: "connection_unavailable", message: "Reconnect or enable this AI connection before checking usage." };
}
if (!supportsAiConnectionUsage(metadata.provider, metadata.method)) {
return { ...identity, ...await probeAiConnectionUsage(metadata, "") };
}
const value = await credential(row);
return { ...identity, ...await probeAiConnectionUsage(metadata, value) };
}
async function accountIdentity(companyId: string, userId: string, connectionId: string, grantId?: string,
audit?: { actorType: "user" | "agent" | "system"; actorId: string; heartbeatRunId?: string | null }): Promise<AiConnectionAccountIdentity> {
const { row, metadata } = await readableAccount(companyId, userId, connectionId, grantId);
const identity = { connectionId: row.connection.id, grantId: row.grant.id };
if (row.grant.status !== "active" || !row.connection.enabled || row.connection.status !== "active" ||
row.connection.healthStatus !== "ok" || aiSubscriptionNeedsIsolatedLogin(row.connection.config)) {
return { ...identity, status: "unavailable", checkedAt: new Date().toISOString(), accountLabel: null, organizationLabel: null,
errorCode: "connection_unavailable", message: "Reconnect or enable this connection to check its account details." };
}
// Unsupported methods must not open their credentials just to report this limit.
const value = metadata.provider === "anthropic" && metadata.method === "subscription" && !metadata.routing
? await credential(row, 0, { actorType: "user", actorId: userId, ...audit, responsibleUserId: userId }) : "";
return { ...identity, ...await probeAiAccountIdentity(metadata, value) };
}
async function select(input: {
companyId: string;
userId: string | null;
agentId: string;
adapterType: string;
model?: unknown;
runnerProvider?: unknown;
acpxAgent?: unknown;
allowUninstalledPersonal?: boolean;
allowUninstalledShared?: boolean;
allowLegacyValidation?: boolean;
binding: AiConnectionBinding;
}) {
const { companyId, userId, agentId, binding } = input;
if (
!isAiConnectionCompatible(
binding,
input.adapterType,
input.model,
input.runnerProvider,
input.acpxAgent,
)
)
throw aiConnectionConfigurationFailure("ai_connection_incompatible", "Select an AI connection compatible with this harness and model");
if (binding.mode === "responsible_user" && !userId)
throw aiConnectionConfigurationFailure("ai_connection_responsible_user_missing", "This run needs a responsible user to select an AI connection");
if (userId && !(await membership(companyId, userId)))
throw forbidden("The responsible user is not an active company member");
const defaultRow =
binding.mode === "responsible_user"
? (
await db
.select()
.from(aiProviderDefaults)
.where(
and(
eq(aiProviderDefaults.companyId, companyId),
eq(aiProviderDefaults.userId, userId!),
eq(aiProviderDefaults.provider, binding.provider),
),
)
.limit(1)
)[0]
: null;
const grantId =
binding.mode === "responsible_user"
? defaultRow?.grantId
: binding.grantId;
if (!grantId)
throw aiConnectionConfigurationFailure("ai_connection_default_missing", "Connect an account and choose your personal default");
const row = (await rows(companyId)).find(
(r) =>
r.grant.id === grantId &&
(binding.mode === "responsible_user" ||
r.connection.id === binding.connectionId),
);
if (!row)
throw aiConnectionConfigurationFailure("ai_connection_missing", "The selected AI connection is unavailable");
const { connection, grant } = row;
const metadata = aiConnectionMetadataSchema.safeParse(connection.config.ai);
if (
!metadata.success ||
metadata.data.provider !== binding.provider ||
(binding.mode !== "responsible_user" && metadata.data.method !== binding.method) ||
!isAiConnectionCompatible(metadata.data, input.adapterType, input.model, input.runnerProvider, input.acpxAgent)
)
throw aiConnectionConfigurationFailure("ai_connection_incompatible", "The selected AI connection is incompatible");
if (aiSubscriptionNeedsIsolatedLogin(connection.config))
throw aiConnectionConfigurationFailure("ai_connection_unavailable", "Reconnect this subscription with a separate sign-in to protect your existing terminal login.", { connectionId: connection.id });
if (
grant.status !== "active" ||
!connection.enabled ||
connection.status !== "active" ||
(connection.healthStatus !== "ok" &&
!(
input.allowLegacyValidation &&
connection.config.aiLegacyAdoption === true
))
)
throw aiConnectionConfigurationFailure("ai_connection_unavailable", "Reconnect or validate the selected AI account", { connectionId: connection.id });
if (
grant.kind === "user" &&
!(await membership(companyId, grant.subjectUserId))
)
throw forbidden(
"The account owner is no longer an active company member",
);
if (
binding.mode === "responsible_user" &&
(grant.kind !== "user" || grant.subjectUserId !== userId)
)
throw forbidden("The default must belong to the responsible user");
if (binding.mode === "shared" && grant.kind !== "organization")
throw forbidden("Select a company-shared account");
if (binding.mode === "delegated" && grant.kind !== "user")
throw forbidden("Select a personal account");
const audience = await db
.select()
.from(connectionGrantMembers)
.where(and(
eq(connectionGrantMembers.companyId, companyId),
eq(connectionGrantMembers.grantId, grant.id),
));
// The existing human-access permission is authoritative for every binding,
// including old explicit personal selections. Agent delegation cannot bypass it.
if (!canUseCredential(grant, userId, audience)) {
// Membership was checked above. A known user's sharing choice is an
// operator action; absent identity must remain an unclassified denial.
if (userId) throw aiConnectionCredentialNotSharedFailure({ connectionName: connection.name, grantId: grant.id });
throw forbidden("This credential is not shared with the responsible user");
}
const installs = await db
.select()
.from(toolConnectionInstalls)
.where(
and(
eq(toolConnectionInstalls.companyId, companyId),
eq(toolConnectionInstalls.connectionId, connection.id),
or(
and(
eq(toolConnectionInstalls.targetType, "company"),
eq(toolConnectionInstalls.targetId, companyId),
),
and(
eq(toolConnectionInstalls.targetType, "agent"),
eq(toolConnectionInstalls.targetId, agentId),
),
),
),
);
if (
!installs.length &&
!(
(input.allowUninstalledPersonal &&
grant.kind === "user" && grant.subjectUserId === userId) ||
(input.allowUninstalledShared && binding.mode === "shared" && grant.kind === "organization")
)
)
throw forbidden("This connection is not permitted for this agent");
return {
...row,
attribution: {
connectionId: connection.id,
grantId: grant.id,
provider: binding.provider,
method: metadata.data.method,
mode: binding.mode,
responsibleUserId: userId,
} satisfies AiConnectionAttribution,
};
}
/** Internal decision use has its own capability check, never a pretend CLI harness. */
async function selectDecision(input: {
companyId: string; connectionId: string; grantId: string;
userId: string | null; agentId?: string | null; sponsoredBackground?: boolean;
}) {
const [row] = await db.select({ connection: toolConnections, grant: connectionGrants })
.from(toolConnections).innerJoin(connectionGrants, and(
eq(connectionGrants.connectionId, toolConnections.id), eq(connectionGrants.companyId, toolConnections.companyId),
)).where(and(eq(toolConnections.companyId, input.companyId), eq(toolConnections.id, input.connectionId),
eq(connectionGrants.id, input.grantId), eq(toolConnections.connectionPurpose, "ai"))).limit(1);
if (!row || row.grant.kind !== "organization") throw unprocessable("Shared AI connection unavailable", { code: "connection_unavailable" });
const metadata = aiConnectionMetadataSchema.safeParse(row.connection.config.ai);
const provider = metadata.success ? decisionProviderForConnection(metadata.data) : null;
if (!provider) throw unprocessable("Connection does not support decisions", { code: "incompatible_connection" });
if (row.grant.status !== "active" || !row.connection.enabled || row.connection.status !== "active" || row.connection.healthStatus !== "ok")
throw unprocessable("Reconnect or enable this connection", { code: "connection_unavailable" });
// Only the decision service may sponsor background use; ordinary AI selection never bypasses an audience.
if (!input.sponsoredBackground) {
if (!(await membership(input.companyId, input.userId))) throw forbidden("Active responsible user required");
const audience = await db.select().from(connectionGrantMembers).where(and(
eq(connectionGrantMembers.companyId, input.companyId), eq(connectionGrantMembers.grantId, input.grantId),
));
if (!canUseCredential(row.grant, input.userId, audience)) throw forbidden("Connection is not shared with this user");
} else if (input.userId || input.agentId) throw forbidden("Background sponsorship cannot replace caller permissions");
if (input.agentId) {
const [install] = await db.select({ id: toolConnectionInstalls.id }).from(toolConnectionInstalls).where(and(
eq(toolConnectionInstalls.companyId, input.companyId), eq(toolConnectionInstalls.connectionId, input.connectionId),
or(and(eq(toolConnectionInstalls.targetType, "company"), eq(toolConnectionInstalls.targetId, input.companyId)),
and(eq(toolConnectionInstalls.targetType, "agent"), eq(toolConnectionInstalls.targetId, input.agentId))),
)).limit(1);
if (!install) throw forbidden("Connection is not permitted for this agent");
}
return { ...row, provider };
}
/** Internal fast-response use has its own capability check, never a pretend CLI harness. */
async function selectFastResponse(input: {
companyId: string; connectionId: string; grantId: string;
userId: string | null; agentId?: string | null; sponsoredBackground?: boolean;
}) {
const [row] = await db.select({ connection: toolConnections, grant: connectionGrants })
.from(toolConnections).innerJoin(connectionGrants, and(
eq(connectionGrants.connectionId, toolConnections.id), eq(connectionGrants.companyId, toolConnections.companyId),
)).where(and(eq(toolConnections.companyId, input.companyId), eq(toolConnections.id, input.connectionId),
eq(connectionGrants.id, input.grantId), eq(toolConnections.connectionPurpose, "ai"))).limit(1);
if (!row || row.grant.kind !== "organization") throw unprocessable("Shared AI connection unavailable", { code: "connection_unavailable" });
const metadata = aiConnectionMetadataSchema.safeParse(row.connection.config.ai);
const provider = metadata.success && metadata.data.method === "api_key" ? metadata.data.provider : null;
if (!provider) throw unprocessable("Connection does not support fast responses", { code: "incompatible_connection" });
if (row.grant.status !== "active" || !row.connection.enabled || row.connection.status !== "active" || row.connection.healthStatus !== "ok")
throw unprocessable("Reconnect or enable this connection", { code: "connection_unavailable" });
// Only internal fast responses may sponsor background use; ordinary AI selection never bypasses an audience.
if (!input.sponsoredBackground) {
if (!(await membership(input.companyId, input.userId))) throw forbidden("Active responsible user required");
const audience = await db.select().from(connectionGrantMembers).where(and(
eq(connectionGrantMembers.companyId, input.companyId), eq(connectionGrantMembers.grantId, input.grantId),
));
if (!canUseCredential(row.grant, input.userId, audience)) throw forbidden("Connection is not shared with this user");
} else if (input.userId) throw forbidden("Background sponsorship cannot replace caller permissions");
if (input.agentId) {
const [install] = await db.select({ id: toolConnectionInstalls.id }).from(toolConnectionInstalls).where(and(
eq(toolConnectionInstalls.companyId, input.companyId), eq(toolConnectionInstalls.connectionId, input.connectionId),
or(and(eq(toolConnectionInstalls.targetType, "company"), eq(toolConnectionInstalls.targetId, input.companyId)),
and(eq(toolConnectionInstalls.targetType, "agent"), eq(toolConnectionInstalls.targetId, input.agentId))),
)).limit(1);
if (!install) throw forbidden("Connection is not permitted for this agent");
}
return { ...row, provider };
}
async function selectPromptInjectionPrevention(input: {
companyId: string; connectionId: string; grantId: string;
userId: string | null; agentId?: string | null; sponsoredBackground?: boolean;
}) {
const [row] = await db.select({ connection: toolConnections, grant: connectionGrants })
.from(toolConnections).innerJoin(connectionGrants, and(
eq(connectionGrants.connectionId, toolConnections.id), eq(connectionGrants.companyId, toolConnections.companyId),
)).where(and(eq(toolConnections.companyId, input.companyId), eq(toolConnections.id, input.connectionId),
eq(connectionGrants.id, input.grantId), eq(toolConnections.connectionPurpose, "ai"))).limit(1);
if (!row || row.grant.kind !== "organization") throw unprocessable("Shared AI connection unavailable", { code: "connection_unavailable" });
const metadata = aiConnectionMetadataSchema.safeParse(row.connection.config.ai);
const provider = metadata.success && decisionProviderForConnection(metadata.data) === "openrouter" ? "openrouter" as const : null;
if (!provider) throw unprocessable("Connection does not support prompt injection checks", { code: "incompatible_connection" });
if (row.grant.status !== "active" || !row.connection.enabled || row.connection.status !== "active" || row.connection.healthStatus !== "ok")
throw unprocessable("Reconnect or enable this connection", { code: "connection_unavailable" });
// Only internal prompt injection checks may sponsor background use; ordinary AI selection never bypasses an audience.
if (!input.sponsoredBackground) {
if (!(await membership(input.companyId, input.userId))) throw forbidden("Active responsible user required");
const audience = await db.select().from(connectionGrantMembers).where(and(
eq(connectionGrantMembers.companyId, input.companyId), eq(connectionGrantMembers.grantId, input.grantId),
));
if (!canUseCredential(row.grant, input.userId, audience)) throw forbidden("Connection is not shared with this user");
} else if (input.userId) throw forbidden("Background sponsorship cannot replace caller permissions");
if (input.agentId) {
const [install] = await db.select({ id: toolConnectionInstalls.id }).from(toolConnectionInstalls).where(and(
eq(toolConnectionInstalls.companyId, input.companyId), eq(toolConnectionInstalls.connectionId, input.connectionId),
or(and(eq(toolConnectionInstalls.targetType, "company"), eq(toolConnectionInstalls.targetId, input.companyId)),
and(eq(toolConnectionInstalls.targetType, "agent"), eq(toolConnectionInstalls.targetId, input.agentId))),
)).limit(1);
if (!install) throw forbidden("Connection is not permitted for this agent");
}
return { ...row, provider };
}
async function credential(row: Pick<Awaited<ReturnType<typeof select>>, "connection" | "grant">, retry = 0, audit?: { responsibleUserId: string | null; actorType: "user" | "agent" | "system"; actorId: string; issueId?: string | null; heartbeatRunId?: string | null }): Promise<string> {
const metadata = aiConnectionMetadataSchema.parse(row.connection.config.ai);
if (metadata.routing?.auth === "none") return "";
const ref = row.grant.credentialSecretRefs.find(
(r) => r.configPath === "ai.credential",
);
if (!ref)
throw unprocessable("Reconnect this AI account", {
code: "ai_connection_credential_missing",
});
const [secret] = await db
.select()
.from(companySecrets)
.where(
and(
eq(companySecrets.id, ref.secretId),
eq(companySecrets.companyId, row.connection.companyId),
),
);
if (!secret)
throw unprocessable("Reconnect this AI account", {
code: "ai_connection_credential_missing",
});
if (row.grant.kind === "user" && secret.scope !== "user")
throw forbidden("Credential ownership mismatch");
if (row.grant.kind === "organization" && secret.scope !== "company") throw forbidden("Credential ownership mismatch");
const context = {
consumerType: "tool_connection" as const,
consumerId: row.connection.id,
configPath: ref.configPath,
responsibleUserId: row.grant.subjectUserId,
actorType: "system" as const,
...audit,
};
let value: string;
if (secret.scope === "user") {
if (
secret.ownerUserId !== row.grant.subjectUserId ||
!secret.userSecretDefinitionId
)
throw forbidden("Credential ownership mismatch");
const result = await secrets.resolveUserSecretValue(
row.connection.companyId,
{
definitionId: secret.userSecretDefinitionId,
responsibleUserId: secret.ownerUserId,
required: true,
version: "latest",
},
context,
);
if (!result) throw unprocessable("Reconnect this AI account");
value = result.value;
} else {
value = await secrets.resolveSecretValue(row.connection.companyId, secret.id, "latest", context);
}
// Resolution records lastResolvedAt and can wait behind a rotating writer.
// Re-read after that wait instead of handing a run the pre-rotation token.
const [latest] = await db.select().from(companySecrets).where(and(
eq(companySecrets.id, secret.id), eq(companySecrets.companyId, row.connection.companyId),
));
if (!latest || latest.status !== "active") throw unprocessable("Reconnect this AI account");
if (latest.latestVersion !== secret.latestVersion) {
if (retry >= 2) throw unprocessable("AI credentials are changing. Retry this execution.", { code: "ai_connection_busy" });
return credential(row, retry + 1, audit);
}
return value;
}
async function save(
companyId: string,
userId: string,
input: CreateAiConnection | AiConnectionLoginIntent,
verifiedCredential: string,
loginSession?: { id: string; complete?: (tx: Db) => Promise<void> },
attemptStartedAt = new Date(),
managementContext?: ConnectionManagementContext,
verifiedAccountLabel?: string,
) {
await managementContext?.reauthorize();
const routing = "routing" in input ? input.routing : undefined;
if (!(await membership(companyId, userId)))
throw forbidden("An active company member must own this connection");
const reconnect = input.connectionId
? (await rows(companyId)).find(
(r) => r.connection.id === input.connectionId,
)
: undefined;
if (input.connectionId && !reconnect)
throw notFound("AI connection not found");
if (
reconnect &&
(reconnect.grant.createdByUserId !== userId ||
(reconnect.grant.kind === "user" &&
reconnect.grant.subjectUserId !== userId))
)
throw forbidden("Only the account owner can reconnect it");
if (
reconnect &&
(reconnect.connection.config.ai as AiConnectionMetadata).provider !==
input.provider
)
throw unprocessable("Reconnect cannot change providers");
if (reconnect && !isDeepStrictEqual((reconnect.connection.config.ai as AiConnectionMetadata).routing, routing))
throw unprocessable("Reconnect must retain this connection’s routing. Create another connection to change its destination.");
if (
reconnect &&
((reconnect.connection.config.ai as AiConnectionMetadata).method !==
input.method ||
(reconnect.grant.kind === "user") !== (input.ownership === "personal"))
)
throw unprocessable(
"Reconnect cannot change the sign-in method or ownership",
);
const id = reconnect?.connection.id ?? randomUUID();
const grantId = reconnect?.grant.id ?? randomUUID();
const persist = () => withAccountHomeSecretMutationLock(undefined, companyId, () => db.transaction(async (tx) => {
await managementContext?.reauthorize();
const secrets = secretService(tx);
if (loginSession) {
const [session] = await tx
.select()
.from(adapterAuthSessions)
.where(
and(
eq(adapterAuthSessions.id, loginSession.id),
eq(adapterAuthSessions.companyId, companyId),
eq(adapterAuthSessions.startedByUserId, userId),
),
)
.for("update");
if (!session) throw forbidden("Login session ownership mismatch");
attemptStartedAt = session.createdAt;
if (session.connectionId && session.connectionGrantId)
return {
connectionId: session.connectionId,
grantId: session.connectionGrantId,
};
if (
!["promoting", "submitting", "awaiting_code"].includes(
session.status,
) ||
(session.expiresAt && session.expiresAt.getTime() <= Date.now())
)
throw unprocessable("The login attempt is no longer active");
if (
!session.aiConnection ||
session.aiConnection.provider !== input.provider ||
session.aiConnection.method !== input.method ||
session.aiConnection.connectionId !== input.connectionId ||
session.aiConnection.ownership !== input.ownership ||
session.aiConnection.allAgents !== input.allAgents ||
JSON.stringify(session.aiConnection.agentIds) !==
JSON.stringify(input.agentIds)
)
throw forbidden("Login target mismatch");
}
if (reconnect) {
const [current] = await tx
.select()
.from(connectionGrants)
.where(eq(connectionGrants.id, grantId))
.for("update");
if (
!current ||
current.updatedAt.getTime() !== reconnect.grant.updatedAt.getTime() ||
current.updatedAt.getTime() > attemptStartedAt.getTime()
)
throw unprocessable("The connection changed. Start reconnect again.");
}
let secretId = reconnect?.grant.credentialSecretRefs.find(
(r) => r.configPath === "ai.credential",
)?.secretId;
// Adoption indexes existing credentials without transferring ownership.
// Only rotate the private slot created for this grant. A reconnect of an
// indexed credential must leave every legacy consumer's value untouched,
// even after adoption has cleared the connection's validation marker.
if (secretId) {
const [source] = await tx.select({ name: companySecrets.name, key: userSecretDefinitions.key })
.from(companySecrets)
.leftJoin(userSecretDefinitions, eq(userSecretDefinitions.id, companySecrets.userSecretDefinitionId))
.where(and(eq(companySecrets.companyId, companyId), eq(companySecrets.id, secretId)));
const privateSlot = input.ownership === "personal"
? source?.key === `ai_${grantId.replaceAll("-", "_")}`
: source?.name === `ai-${grantId}`;
if (!privateSlot) secretId = undefined;
}
if (!verifiedCredential) { secretId = undefined; }
else if (secretId)
await secrets.rotate(
secretId,
{ value: verifiedCredential },
{ userId },
);
else if (input.ownership === "personal") {
const definition = await secrets.createUserSecretDefinition(
companyId,
{
key: `ai_${grantId.replaceAll("-", "_")}`,
name: input.name,
provider: "local_encrypted",
},
{ userId },
);
const secret = await secrets.createCurrentUserSecretValue(
companyId,
userId,
{ definitionId: definition.id, value: verifiedCredential },
{ userId },
);
secretId = secret.id;
} else
secretId = (
await secrets.create(
companyId,
{
name: `ai-${grantId}`,
provider: "local_encrypted",
value: verifiedCredential,
},
{ userId },
)
).id;
if (input.agentIds.length) {
const targets = await tx
.select({ id: agents.id })
.from(agents)
.where(
and(
eq(agents.companyId, companyId),
inArray(agents.id, input.agentIds),
),
);
if (targets.length !== new Set(input.agentIds).size)
throw forbidden("Agent does not belong to this company");
}
const source = aiConnectionCatalogSlug(input.provider, routing);
const providerName = getAppStoreDefinition(source)?.name ?? AI_CONNECTION_CAPABILITIES[input.provider].name;
const key = `app-gallery:${source}`;
await tx
.insert(toolApplications)
.values({
companyId,
applicationKey: key,
name: providerName,
type: "mcp_http",
metadata: { sourceTemplateKey: source },
ownerUserId: userId,
})
.onConflictDoNothing();
const [app] = await tx
.select()
.from(toolApplications)
.where(
and(
eq(toolApplications.companyId, companyId),
or(
eq(toolApplications.applicationKey, key),
eq(
toolApplications.name,
providerName,
),
),
),
);
if (!app) throw unprocessable("Could not find the provider application");
if (reconnect)
await tx
.update(toolConnections)
.set({
enabled: true,
status: "active",
healthStatus: "ok",
healthMessage: null,
config: { ...reconnect.connection.config, aiIsolatedSubscription: input.method === "subscription" && input.provider !== "anthropic" },
updatedAt: new Date(),
})
.where(eq(toolConnections.id, id));
else
await tx
.insert(toolConnections)
.values({
id,
companyId,
applicationId: app.id,
name: input.name,
uid: `ai-${id}`,
connectionPurpose: "ai",
transport: "runtime_auth",
authKind: input.method === "api_key" ? "api_key" : "oauth",
credentialPolicy:
input.ownership === "personal" ? "per_user" : "shared",
status: "active",
enabled: true,
healthStatus: "ok",
config: {
sourceTemplateKey: source,
ai: { provider: input.provider, method: input.method, ...(routing ? { routing } : {}) },
aiIsolatedSubscription: input.method === "subscription" && input.provider !== "anthropic",
},
createdByUserId: userId,
});
let accountLabel: string | undefined = input.provider === "google" ? verifiedAccountLabel : undefined;
if (input.method === "subscription" && input.provider !== "anthropic") {
try {
const credential = JSON.parse(verifiedCredential);
const claims = credential.tokens?.id_token
? JSON.parse(
Buffer.from(
credential.tokens.id_token.split(".")[1],
"base64url",
).toString(),
)
: credential;
const email = claims.email ?? claims.user?.email;
if (
typeof email === "string" &&
/^[^\s@/\\]{1,100}@[^\s@/\\]{1,100}\.[^\s@/\\]{2,40}$/.test(email)
)
accountLabel = email;
} catch {
/* Safe account identity is optional. */
}
}
const refs = secretId ? [
{
secretId: secretId!,
configPath: "ai.credential",
required: true,
versionSelector: "latest" as const,
},
] : [];
if (reconnect)
await tx
.update(connectionGrants)
.set({
status: "active",
providerTenant: accountLabel ? { name: accountLabel } : {},
credentialSecretRefs: refs,
revokedAt: null,
updatedAt: new Date(),
})
.where(eq(connectionGrants.id, grantId));
else
await tx
.insert(connectionGrants)
.values({
id: grantId,
companyId,
connectionId: id,
kind: input.ownership === "personal" ? "user" : "organization",
subjectUserId: input.ownership === "personal" ? userId : null,
isDefault: input.ownership === "shared",
providerTenant: accountLabel ? { name: accountLabel } : {},
credentialSecretRefs: refs,
createdByUserId: userId,
});
const [savedConnection] = await tx
.select()
.from(toolConnections)
.where(eq(toolConnections.id, id));
await syncConnectionCredentialBindings(tx, savedConnection, refs);
if (input.ownership === "personal" && !routing) {
await tx
.insert(aiConnectionDefaults)
.values({
companyId,
userId,
provider: input.provider,
method: input.method,
grantId,
})
.onConflictDoNothing();
await tx.insert(aiProviderDefaults).values({ companyId, userId, provider: input.provider, grantId }).onConflictDoNothing();
}
if (!reconnect) {
const installs = input.allAgents
? [{ targetType: "company" as const, targetId: companyId }]
: input.agentIds.map((targetId) => ({
targetType: "agent" as const,
targetId,
}));
if (installs.length)
await tx
.insert(toolConnectionInstalls)
.values(
installs.map((i) => ({
...i,
companyId,
connectionId: id,
createdByUserId: userId,
})),
);
}
if (loginSession) await notifyLoginSessionChanged(tx);
if (loginSession)
await tx
.update(adapterAuthSessions)
.set({
connectionId: id,
connectionGrantId: grantId,
connectionMethod: input.method,
})
.where(
eq(adapterAuthSessions.id, loginSession.id),
);
await loginSession?.complete?.(tx as unknown as Db);
await logActivity(tx as unknown as Db, {
companyId,
...(managementContext?.audit ?? { actorType: "user" as const, actorId: userId }),
action: reconnect
? "ai_connection.reconnected"
: "ai_connection.connected",
entityType: "tool_connection",
entityId: id,
details: { provider: input.provider, method: input.method, grantId },
});
return { connectionId: id, grantId };
}));
// Quota refresh can hold this lock through a 60-second token exchange.
// Keep the verified sign-in in memory across two 30-second acquisition
// timeouts instead of making its caller discard a successful login.
for (let attempt = 0; ; attempt++) {
try { return await persist(); }
catch (error) {
if (attempt >= 2 || (error as NodeJS.ErrnoException)?.code !== WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE) throw error;
}
}
}
async function recoverQuotaCredentialInTransaction(
tx: Db, row: Parameters<typeof credential>[0], current: string, onRecoveryStarted?: () => void,
) {
const metadata = aiConnectionMetadataSchema.safeParse(row.connection.config.ai);
if (!metadata.success || !["openai", "google"].includes(metadata.data.provider) || metadata.data.method !== "subscription") return null;
const recovery = quotaCredentialRecovery(row.connection.companyId, row.grant.id);
const pending = await recovery.read();
if (!pending) return null;
const ref = row.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential");
if (!ref || (pending.provider ?? "openai") !== metadata.data.provider
|| pending.secretId !== ref.secretId || pending.connectionId !== row.connection.id || pending.baseHash !== quotaCredentialHash(current)) {
await recovery.clear();
return null;
}
if (metadata.data.provider === "google") {
const [secret] = await tx.select({ epoch: companySecrets.aiSessionEpoch, version: companySecrets.latestVersion })
.from(companySecrets).where(and(eq(companySecrets.companyId, row.connection.companyId), eq(companySecrets.id, ref.secretId))).for("update");
if (!pending.google || pending.google.sessionEpoch !== secret?.epoch || pending.google.baseVersion !== secret?.version
|| pending.google.versionSelector !== (ref.versionSelector ?? "latest")
|| pending.google.accountEmail.toLowerCase() !== row.grant.providerTenant?.name?.toLowerCase()) {
await recovery.clear();
return null;
}
if (!pending.google.verified) {
onRecoveryStarted?.();
throw new Error("Google credential recovery needs account verification");
}
}
onRecoveryStarted?.();
await secretService(tx).rotate(ref.secretId, { value: pending.value, preserveAiSessionEpoch: true }, { userId: row.grant.subjectUserId });
// Automatic rotation changes secret revision, not the grant mutation marker
// used to fence reconnect attempts. Ownership and access are unchanged.
await logActivity(tx, { companyId: row.connection.companyId, actorType: "system", actorId: "quota_recovery",
action: "ai_connection.credential_refreshed", entityType: "tool_connection", entityId: row.connection.id,
details: { provider: metadata.data.provider, grantId: row.grant.id } });
return pending.value;
}
async function runtimeCredential(row: Parameters<typeof credential>[0]) {
const metadata = aiConnectionMetadataSchema.safeParse(row.connection.config.ai);
if (!metadata.success || !["openai", "google"].includes(metadata.data.provider) || metadata.data.method !== "subscription") return credential(row);
if (!(await quotaCredentialRecovery(row.connection.companyId, row.grant.id).read())) return credential(row);
let savingReplacement = false;
return withAccountHomeSecretMutationLock(undefined, row.connection.companyId, async () => {
if (metadata.data.provider === "google") {
const recovery = quotaCredentialRecovery(row.connection.companyId, row.grant.id);
const pending = await recovery.read();
if (pending?.provider === "google" && pending.google && !pending.google.verified
&& pending.connectionId === row.connection.id
&& pending.google.accountEmail.toLowerCase() === row.grant.providerTenant?.name?.toLowerCase()) {
const ref = row.grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential");
const [secret] = await db.select({ epoch: companySecrets.aiSessionEpoch, version: companySecrets.latestVersion })
.from(companySecrets).where(and(eq(companySecrets.companyId, row.connection.companyId), eq(companySecrets.id, pending.secretId)));
const current = await credential(row);
if (ref?.secretId === pending.secretId && (ref.versionSelector ?? "latest") === pending.google.versionSelector
&& secret?.epoch === pending.google.sessionEpoch && secret?.version === pending.google.baseVersion
&& pending.baseHash === quotaCredentialHash(current)) {
savingReplacement = true;
// Network verification runs before acquiring database row locks.
// The transaction below rechecks the exact saved generation.
await prepareAntigravityCredentialRecovery({ originalCredential: current,
refreshedCredential: pending.value, accountEmail: pending.google.accountEmail,
scope: { companyId: row.connection.companyId, accountKey: row.grant.id },
persistVerified: value => recovery.write({ ...pending, value, google: { ...pending.google!, verified: true } }),
});
}
}
}
let recovered = false;
const value = await db.transaction(async tx => {
const [grant] = await tx.select().from(connectionGrants).where(and(
eq(connectionGrants.id, row.grant.id), eq(connectionGrants.companyId, row.connection.companyId),
eq(connectionGrants.connectionId, row.connection.id),
)).for("update", { noWait: true });
if (!grant || grant.status !== "active") throw new Error("credentials_unavailable");
const ref = grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential");
if (!ref) throw new Error("credentials_unavailable");
await tx.select({ id: companySecrets.id }).from(companySecrets).where(and(
eq(companySecrets.id, ref.secretId), eq(companySecrets.companyId, row.connection.companyId),
)).for("update", { noWait: true });
const selected = { connection: row.connection, grant };
const current = await aiConnectionService(tx as unknown as Db).credential(selected);
const pending = await recoverQuotaCredentialInTransaction(tx as unknown as Db, selected, current, () => {
savingReplacement = true;
});
recovered = pending !== null;
return pending ?? current;
});
if (recovered) await quotaCredentialRecovery(row.connection.companyId, row.grant.id).clear()
.catch(error => logger.warn({ err: error, grantId: row.grant.id }, "Saved runtime credential recovery cleanup pending"));
return value;
}).catch(error => {
// Once a matching replacement is found, a failed write or commit must
// defer execution. Keep its journal so the next attempt can save it.
if (savingReplacement) {
throw unprocessable("AI credentials are being updated. Retry shortly.", { code: "ai_connection_busy" });
}
// Drizzle wraps Postgres errors in `cause`. Contention is a temporary
// pre-provider wait, not evidence that this account needs reconnecting.
let current: unknown = error;
for (let depth = 0; depth < 4 && current && typeof current === "object"; depth += 1) {
const value = current as { code?: unknown; cause?: unknown };
if (value.code === WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE || value.code === "55P03") {
throw unprocessable("AI credentials are being updated. Retry shortly.", { code: "ai_connection_busy" });
}
current = value.cause;
}
throw error;
});
}
/** A late failure must never invalidate credentials that were refreshed or reconnected meanwhile. */
async function markAuthenticationFailed(input: {
companyId: string;
runId?: string;
agentId?: string;
runStartedAt: Date;
attribution: AiConnectionAttribution & { identity: string };
}) {
return withAccountHomeSecretMutationLock(undefined, input.companyId, async () => {
let recovered = false;
await db.transaction(async (tx) => {
const { attribution } = input;
const [grant] = await tx.select().from(connectionGrants).where(and(
eq(connectionGrants.companyId, input.companyId),
eq(connectionGrants.id, attribution.grantId),
eq(connectionGrants.connectionId, attribution.connectionId),
)).for("update");
if (!grant || grant.status !== "active" || grant.updatedAt > input.runStartedAt) return;
const [connection] = await tx.select().from(toolConnections).where(and(
eq(toolConnections.companyId, input.companyId),
eq(toolConnections.id, attribution.connectionId),
eq(toolConnections.connectionPurpose, "ai"),
));
if (!connection) return;
const metadata = aiConnectionMetadataSchema.safeParse(connection.config.ai);
if (!metadata.success || metadata.data.provider !== attribution.provider || metadata.data.method !== attribution.method) return;
const ref = grant.credentialSecretRefs.find((candidate) => candidate.configPath === "ai.credential");
if (!ref) return;
await tx.select({ id: companySecrets.id }).from(companySecrets).where(and(
eq(companySecrets.companyId, input.companyId), eq(companySecrets.id, ref.secretId),
)).for("update");
const value = await aiConnectionService(tx as unknown as Db).credential({ connection, grant });
const generation = createHash("sha256").update(value).digest("hex").slice(0, 16);
if (attribution.identity !== `${grant.id}:${attribution.responsibleUserId ?? "shared"}:${generation}`) return;
// A quota exchange may have consumed this run's token before a failed
// database commit. Save its working replacement instead of revoking it.
recovered = await recoverQuotaCredentialInTransaction(tx as unknown as Db, { connection, grant }, value) !== null;
if (recovered) return;
await tx.update(connectionGrants).set({ status: "needs_reauthorization", updatedAt: new Date() })
.where(eq(connectionGrants.id, grant.id));
await tx.update(toolConnections).set({ healthStatus: "error", healthMessage: "Sign in again to restore this AI connection.", updatedAt: new Date() })
.where(eq(toolConnections.id, connection.id));
await logActivity(tx as unknown as Db, {
companyId: input.companyId, actorType: "system", actorId: input.runId ? "heartbeat" : "adapter_test",
agentId: input.agentId, runId: input.runId, action: "ai_connection.authentication_failed",
entityType: "tool_connection", entityId: connection.id,
details: { provider: attribution.provider, grantId: grant.id },
});
});
if (recovered) await quotaCredentialRecovery(input.companyId, input.attribution.grantId).clear()
.catch(error => logger.warn({ err: error, grantId: input.attribution.grantId }, "Saved failed-run credential recovery cleanup pending"));
});
}
/** Refresh only the selected vaulted identity, serializing rotation and reconnect. */
async function refreshQuotaCredential(row: Parameters<typeof credential>[0], failedValue: string, signal: AbortSignal) {
return withAccountHomeSecretMutationLock(undefined, row.connection.companyId, async () => {
const recovery = quotaCredentialRecovery(row.connection.companyId, row.grant.id);
let replacement: { value: string; secretId: string; version: number; epoch: number; base: string; grantUpdatedAt: number; versionSelector: "latest" | number } | undefined;
const save = () => db.transaction(async (tx) => {
// Do not wait on a transaction that might itself be waiting for the file
// lock. All database locks must be available before consuming OAuth tokens.
const companyId = row.connection.companyId;
const [grant] = await tx.select().from(connectionGrants).where(and(
eq(connectionGrants.id, row.grant.id), eq(connectionGrants.companyId, companyId),
eq(connectionGrants.connectionId, row.connection.id),
)).for("update", { noWait: true });
if (!grant || grant.status !== "active") throw new Error("credentials_unavailable");
const ref = grant.credentialSecretRefs.find(ref => ref.configPath === "ai.credential");
if (!ref) throw new Error("credentials_unavailable");
const [secret] = await tx.select({ id: companySecrets.id, version: companySecrets.latestVersion, epoch: companySecrets.aiSessionEpoch }).from(companySecrets).where(and(
eq(companySecrets.id, ref.secretId), eq(companySecrets.companyId, companyId),
)).for("update", { noWait: true });
if (!secret) throw new Error("credentials_unavailable");
const current = await aiConnectionService(tx as unknown as Db).credential({ connection: row.connection, grant });
if (replacement) {
// A lost commit response is reconciled from the authoritative secret.
// Reconnect, revocation and a competing saved generation always win.
if (current === replacement.value) return current;
if (ref.secretId !== replacement.secretId || secret.version !== replacement.version
|| secret.epoch !== replacement.epoch || current !== replacement.base
|| grant.updatedAt.getTime() !== replacement.grantUpdatedAt
|| (ref.versionSelector ?? "latest") !== replacement.versionSelector) return current;
} else signal.throwIfAborted();
let pending = await recovery.read();
if (pending && (pending.secretId !== ref.secretId || pending.connectionId !== row.connection.id || pending.baseHash !== quotaCredentialHash(current))) {
// Reconnect may reuse this grant. Once the current authorized token
// itself expires, an older journal must not prevent refreshing it.
await recovery.clear();
pending = null;
}
// A delayed poll may have read A before B was saved and B's refresh
// produced an uncommitted C. Recover C before returning or clearing its
// journal; a newer database credential alone does not prove it is usable.
if (current !== failedValue && !pending) return current;
// An active provider may rotate this single-use token itself. Defer polling
// until its credential write-back instead of invalidating its live copy.
const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.innerJoin(agents, and(eq(agents.id, heartbeatRuns.agentId), eq(agents.companyId, companyId))).where(and(
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["queued", "running"]),
or(sql`${heartbeatRuns.contextSnapshot}->'aiConnection'->>'grantId' = ${grant.id}`,
sql`${agents.runtimeConfig}->'aiConnection'->>'provider' = 'openai'`, eq(agents.adapterType, "codex_local")),
)).limit(1);
if (active && !pending && !replacement) throw new Error("provider_unavailable");
let refreshed = replacement?.value ?? pending?.value;
if (!refreshed) {
const auth = JSON.parse(current);
const refreshToken = auth.tokens?.refresh_token;
if (typeof refreshToken !== "string" || !refreshToken) throw new Error("authentication_required");
// Once exchange starts, its lifetime is independent of the dashboard's
// shorter deadline. A consumed single-use token must still be read/saved.
signal.throwIfAborted();
const refreshSignal = AbortSignal.timeout(60_000);
// Matches the Codex OAuth client (openai/codex, login/src/auth/manager.rs).
const response = await fetch("https://auth.openai.com/oauth/token", {
method: "POST", redirect: "error", signal: refreshSignal,
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ client_id: "app_EMoamEEZ73f0CkXaXp7hrann", grant_type: "refresh_token", refresh_token: refreshToken }),
});
if (!response.ok) {
await response.body?.cancel();
throw new Error(response.status === 400 || response.status === 401 ? "authentication_required" : "provider_unavailable");
}
const tokens = await response.json() as Record<string, unknown>;
if (typeof tokens.access_token !== "string" || !tokens.access_token) throw new Error("provider_unavailable");
refreshed = JSON.stringify({ ...auth, tokens: { ...auth.tokens,
access_token: tokens.access_token,
...(typeof tokens.refresh_token === "string" && tokens.refresh_token ? { refresh_token: tokens.refresh_token } : {}),
...(typeof tokens.id_token === "string" && tokens.id_token ? { id_token: tokens.id_token } : {}),
}, last_refresh: new Date().toISOString() });
}
// Only this logical refresh owns the replacement. Save retries must not
// consume another single-use token. Process loss before commit can require
// signing in again in v1; no secondary credential vault is created.
replacement ??= { value: refreshed, secretId: ref.secretId, version: secret.version, epoch: secret.epoch, base: current,
grantUpdatedAt: grant.updatedAt.getTime(), versionSelector: ref.versionSelector ?? "latest" };
// Persist the rotated token even if the quota deadline expires after the
// successful exchange; losing it would strand all subsequent executions.
await secretService(tx).rotate(ref.secretId, { value: refreshed, preserveAiSessionEpoch: true }, { userId: grant.subjectUserId });
await logActivity(tx as unknown as Db, { companyId, actorType: "system", actorId: "quota",
action: "ai_connection.credential_refreshed", entityType: "tool_connection", entityId: row.connection.id,
details: { provider: "openai", grantId: grant.id } });
return refreshed;
});
for (let attempt = 0; ; attempt++) {
try {
const value = await save();
// Clear only after commit. A committed-but-lost response is detected
// by the current credential comparison on the next attempt.
await recovery.clear().catch((error) => logger.warn({ err: error, grantId: row.grant.id }, "Saved quota credential recovery cleanup pending"));
return value;
} catch (error) {
if (!replacement || attempt >= 2) throw error;
// Retry only the returned value. Required database reads still fail
// closed; exhaustion is an error, never a successful refresh.
}
}
});
}
// A quota read uses the same credential audience as execution. Operator status
// alone never grants access to another member's personal subscription.
async function quotaAccounts(companyId: string, userId: string, includeOtherProviders = false) {
if (!(await membership(companyId, userId))) return [];
const visible = await list(companyId, userId);
const accounts = await rows(companyId);
return accounts.flatMap(row => {
const summary = visible.find(item => item.id === row.connection.id && item.grantId === row.grant.id);
if (!summary || summary.method !== "subscription" || !(includeOtherProviders ? ["openai", "anthropic", "xai"] : ["openai", "anthropic"]).includes(summary.provider)) return [];
return [{ ...row, summary }];
});
}
return { list, selectDecision, selectFastResponse, selectPromptInjectionPrevention, quotaAccounts, subscriptionAccounts: (companyId: string, userId: string) => quotaAccounts(companyId, userId, true), refreshQuotaCredential, select, credential, runtimeCredential, probeUsage, accountIdentity, save, setDefault, membership, markAuthenticationFailed };
}