Files
PaperClipAI/server/src/services/ai-connection-runtime.ts
T
DottaandClaude Opus 5.5 4879a146fb fix(secrets): stop credential reads from aborting concurrent budget admissions (#21452)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paid model calls (company decisions, fast responses, browser use,
prompt-injection checks) pass a budget admission before they start
> - Admission runs in a REPEATABLE READ transaction and share-locks the
credential's secret row, so a key rotation cannot slip between admission
and the call
> - Every credential read also wrote "Last used" (`last_resolved_at`,
`updated_at`) to that same row
> - When two calls ran at once, one call's credential read aborted the
other's admission with a serialization failure (40001). The retry then
saw the first call's new budget hold and denied the work as
`budget_blocked`
> - This is the flaky `decision-models.test.ts` failure that held Cloud
readiness, and so releases, on 2026-11-13
> - This pull request stops credential reads from writing the secret
row. "Last used" now comes from the access event that every read already
records
> - The benefit is that concurrent paid calls are no longer wrongly
denied or retried, and the release gate no longer fails on this test

## Linked Issues or Issue Description

No public issue exists. The bug is described here.

**What happened?**
Cloud readiness `verify / General tests (server (4/10))` failed on
master commits 6917d0568d and 5ede64f247. The test "admits concurrent
work from the same budget observation and records both obligations"
reported `expected "vi.fn()" to be called 2 times, but got 1 times`.
#21387 replaced the 1-second poll with an unbounded wait. That does not
fix it: one decision really is denied, so the test now waits until the
test timeout.

**Root cause**
I logged the accounting retry loop under load. Each batch of 50 runs had
6 to 10 retries with `could not serialize access due to concurrent
update` on `select … from company_secrets … for share`. The other
decision's credential read had updated the row. In 4 of 5 batches, the
retried admission saw the other decision's committed reservation and
returned `budget_blocked`.

**Expected behavior**
A credential read does not change admission. Two decisions admitted from
the same budget observation both run and both record their obligation.

**Steps to reproduce**
Run the updated test on the old `secrets.ts`. The test makes the second
credential read land after the first admission's snapshot. It then
commits an unrelated hold after both observations. The old code fails
every run with `first settled before its provider call:
{"status":"unavailable","reason":"budget_blocked"}`.

## What Changed

- `secretService` no longer writes `last_resolved_at` / `updated_at`
when it resolves a value. It still records the `secret_access_events`
row, as before.
- `secretService.list` reports `lastResolvedAt` as the later of the
stored column (legacy values) and the newest successful access event. It
reads one row per secret, backwards on the existing `(secret_id,
created_at)` index.
- A read without a consumer context (for example a GitHub API credential
read) now records a `system` access event, so it still counts as "Last
used".
- The removed `UPDATE` also made a credential read wait for a concurrent
rotation to commit. `credential()` in `ai-connections.ts` depends on
that wait to re-read the rotated version instead of handing a run a
consumed single-use token. Its existing version re-read now takes `FOR
SHARE`, so it keeps that wait. `prepareManagedAiRuntime` has its own
before/after freshness read around `runtimeCredential()`; that read
takes `FOR SHARE` too. A share lock creates no new row version and
cannot abort an admission. It adds no statement.
- The first version of this PR throttled the write to once a minute.
Review found that a read crossing the minute could still abort an
admission, so the write is removed instead.
- The concurrent-admission test is now deterministic. It starts decision
A and waits until A is blocked, then starts decision B and waits again.
It commits an unrelated 1-cent hold after both observations, then checks
that both decisions are admitted. A decision that settles before its
provider call fails the test immediately with its result, instead of
hanging.

## Verification

- Red: the new test against the old `secrets.ts` fails on every run with
the `budget_blocked` result above.
- Green: the same test with `repeats: 49` (50 runs) passes 50/50 locally
on a machine at load average 20 to 60.
- Before the change, 3 parallel batches of 50 runs of the old test each
produced a false denial.
- `npx tsc --noEmit -p server`: no errors.
- New `secrets-service.test.ts` test: a resolution leaves the secret row
unchanged, and `list()` still shows "Last used".
- `ai-connections.test.ts` quota test now checks that a read leaves
`updated_at` unchanged and records a success access event.

## Risks

- `updated_at` on a secret no longer moves when the secret is read. It
still moves on every real change.
- "Last used" on the Secrets page costs one indexed lookup per listed
secret.
- Only `list()` derives "Last used". Other endpoints that return a
secret row show the stored column, which is now set only by older
releases.
- No migration. No API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:46:50 -06:00

640 lines
31 KiB
TypeScript

import { managedAiHomeEnvironment } from "@paperclipai/adapter-utils/managed-ai-runtime-env";
export { managedAiSessionFingerprintConfig } from "@paperclipai/adapter-utils/managed-ai-runtime-env";
import { environmentTaskResultSchema } from "@paperclipai/plugin-sdk";
import type { ManagedAiCredentialWritebackBinding } from "./managed-ai-credential-writeback.js";
import { OPENROUTER_ATTRIBUTION_HEADERS, openRouterClaudeHeaders } from "@paperclipai/shared/openrouter-attribution";
import { createHash } from "node:crypto";
import { HttpError, unprocessable } from "../errors.js";
import { mkdtemp, mkdir, writeFile, readFile, realpath, rm } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { and, eq } from "drizzle-orm";
import { type Db, companySecrets, connectionGrants } from "@paperclipai/db";
import {
AI_CONNECTION_CAPABILITIES,
aiConnectionMetadataSchema, aiRoutingHarness,
type AiConnectionBinding,
} from "@paperclipai/shared";
import { managedProviderRouting } from "./ai-provider-routing.js";
import { aiConnectionService } from "./ai-connections.js";
import { subscriptionService } from "./subscriptions.js";
import { refreshSubscriptionConnection } from "./subscription-refresh.js";
import { secretService } from "./secrets.js";
import { decideCodexAuthMerge, withAccountHomeSecretMutationLock } from "@paperclipai/adapter-codex-local/server";
import { WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE } from "@paperclipai/adapter-utils/workspace-restore-merge";
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import { runAdapterExecutionTargetProcess } from "@paperclipai/adapter-utils/execution-target";
import { decideGrokAuthMerge } from "@paperclipai/adapter-grok-local/server";
import { NativeEnvironmentProbeCleanupError, sanitizeAntigravityCredential } from "../vendor/paperclip-runner/index.js";
import { readLocalAiCredentialFile } from "./local-ai-credential-file.js";
import { prepareAntigravityCredentialWriteback } from "./native-runtime/antigravity-runner-credentials.js";
import { quotaCredentialHash, quotaCredentialRecovery, RetiredCredentialRecoveryError } from "./quota-credential-recovery.js";
import { bindRetainedAiCredentialHome, reconcileRetainedAiCleanup, retainAiCredentialCleanup } from "./retained-ai-credential-cleanup.js";
export function isAiConnectionBusy(error: unknown): error is HttpError {
return error instanceof HttpError && error.status === 422 &&
(error.details as { code?: unknown } | undefined)?.code === "ai_connection_busy";
}
/** A real conflict between the agent's AI connection and its project or arguments. */
export function isAiConnectionIncompatible(error: unknown): error is HttpError {
return error instanceof HttpError && error.status === 422 &&
(error.details as { code?: unknown } | undefined)?.code === "ai_connection_incompatible";
}
function projectAuthConflict(files: string[]) {
return unprocessable(
`This agent's managed AI connection conflicts with project authentication settings in ${files.join(" or ")}. Remove the API key, token or apiKeyHelper settings from that project file, or choose a different AI connection for this agent.`,
{ code: "ai_connection_incompatible" },
);
}
// Blank values intentionally override inherited credentials in CLI child environments.
export const AI_AUTH_ENV_KEYS = [
"COPILOT_GITHUB_TOKEN",
"PAPERCLIP_AI_PROVIDER_KEY", "PAPERCLIP_AI_PROVIDER_URL", "PAPERCLIP_CODEX_PROVIDERS",
"GEMINI_API_KEY", "GOOGLE_API_KEY", "GOOGLE_GEMINI_BASE_URL", "GOOGLE_GENAI_USE_VERTEXAI",
"GEMINI_HOME", "GOOGLE_APPLICATION_CREDENTIALS", "GOOGLE_CLOUD_ACCESS_TOKEN", "PAPERCLIP_ACPX_ANTIGRAVITY_AUTH_JSON_SECRET",
"HERMES_HOME", "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN", "AWS_BEARER_TOKEN_BEDROCK", "AWS_PROFILE", "AWS_DEFAULT_PROFILE", "AWS_SHARED_CREDENTIALS_FILE", "AWS_CONFIG_FILE", "AWS_WEB_IDENTITY_TOKEN_FILE", "AWS_ROLE_ARN", "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", "AWS_CONTAINER_CREDENTIALS_FULL_URI",
"ANTHROPIC_BEDROCK_BASE_URL", "ANTHROPIC_MODEL", "ANTHROPIC_DEFAULT_OPUS_MODEL", "ANTHROPIC_DEFAULT_SONNET_MODEL", "ANTHROPIC_DEFAULT_HAIKU_MODEL", "CLAUDE_CODE_SUBAGENT_MODEL",
"ANTHROPIC_API_KEY",
"ANTHROPIC_AUTH_TOKEN",
"CLAUDE_CODE_OAUTH_TOKEN",
"OPENAI_API_KEY",
"CODEX_API_KEY",
"OPENROUTER_API_KEY",
"XAI_API_KEY",
"GROK_API_KEY",
"CODEX_HOME",
"GROK_HOME",
"CLAUDE_CONFIG_DIR",
"OPENCODE_AUTH_JSON",
"OPENCODE_CONFIG_CONTENT",
"OPENCODE_CONFIG",
"OPENCODE_CONFIG_DIR",
"PAPERCLIP_OPENCODE_PROVIDERS",
"PAPERCLIP_PI_PROVIDERS",
"ANTHROPIC_BASE_URL",
"OPENAI_BASE_URL",
"XAI_BASE_URL",
"CLAUDE_CODE_USE_BEDROCK",
"CLAUDE_CODE_USE_VERTEX",
"CLAUDE_CODE_USE_FOUNDRY",
] as const;
export function stripAiAuthBindings(env: unknown): Record<string, unknown> {
const result = {
...(env && typeof env === "object" ? (env as Record<string, unknown>) : {}),
};
for (const key of AI_AUTH_ENV_KEYS)
if (
![
"ANTHROPIC_BASE_URL",
"OPENAI_BASE_URL",
"XAI_BASE_URL",
"CLAUDE_CODE_USE_BEDROCK",
"CLAUDE_CODE_USE_VERTEX",
"CLAUDE_CODE_USE_FOUNDRY",
"PAPERCLIP_OPENCODE_PROVIDERS",
].includes(key)
)
delete result[key];
return result;
}
export async function assertManagedAiProjectAuth(
config: Record<string, unknown>,
provider: AiConnectionBinding["provider"],
target?: AdapterExecutionTarget | null,
options: { deferRunnerTaskCapabilityCheck?: boolean } = {},
) {
const extraArgs = [
...(Array.isArray(config.extraArgs) ? config.extraArgs : []),
...(Array.isArray(config.args) ? config.args : []),
];
if (
extraArgs.some(
(arg) =>
typeof arg === "string" &&
/^(--config|-c|--settings|--setting-sources|--api-key|--auth-token)(=|$)/.test(
arg,
),
)
) {
throw unprocessable(
"Remove authentication/configuration overrides before selecting a managed AI connection",
{ code: "ai_connection_incompatible" },
);
}
const files =
provider === "anthropic"
? [".claude/settings.json", ".claude/settings.local.json"]
: provider === "openai"
? [".codex/config.toml"]
: [];
const pattern =
"apiKeyHelper|ANTHROPIC_API_KEY|ANTHROPIC_AUTH_TOKEN|CLAUDE_CODE_OAUTH_TOKEN|OPENAI_API_KEY|model_provider[[:space:]]*=|env_key[[:space:]]*=|experimental_bearer_token|cli_auth_credentials_store";
if (target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && files.length) {
if (options.deferRunnerTaskCapabilityCheck) return;
const description = environmentTaskResultSchema.parse(await target.runnerTask({ kind: "describe" }));
if (description.kind !== "description" || description.projectAuthCheckVersion !== 1) {
throw unprocessable("The plugin environment cannot validate managed project authentication", { code: "ai_connection_incompatible" });
}
// The process owner checks the exact prepared configuration before it can submit credentials.
return;
}
if (target?.kind === "remote" && files.length) {
// Only inspect for conflicting keys; never return configuration or credential values.
const result = await runAdapterExecutionTargetProcess(
`ai-auth-check-${Date.now()}`,
target,
"sh",
[
"-c",
`
directory=$1; pattern=$2; shift 2
while :; do
for relative in "$@"; do
file="$directory/$relative"
if test -f "$file"; then
grep -Eq "$pattern" "$file"
result=$?
if test "$result" -eq 0; then exit 42; fi
if test "$result" -ne 1; then exit 43; fi
fi
done
parent=$(dirname "$directory")
if test "$parent" = "$directory"; then break; fi
directory=$parent
done`,
"ai-auth-check",
target.remoteCwd,
pattern,
...files,
],
{
cwd: target.remoteCwd,
env: {},
timeoutSec: 15,
graceSec: 1,
onLog: async () => {},
},
);
if (result.exitCode === 42) throw projectAuthConflict(files);
// The check fails closed, but an unfinished check is not a found conflict.
if (result.exitCode !== 0)
throw new Error(
`Paperclip could not check the environment's project authentication settings (${result.timedOut ? "timed out" : `exit ${result.exitCode ?? result.signal ?? "unknown"}`}). The run did not start; it can be retried.`,
);
return;
}
let directory =
typeof config.cwd === "string" ? path.resolve(config.cwd) : process.cwd();
for (;;) {
for (const relative of files) {
try {
const content = await readFile(path.join(directory, relative), "utf8");
if (
/apiKeyHelper|ANTHROPIC_API_KEY|ANTHROPIC_AUTH_TOKEN|CLAUDE_CODE_OAUTH_TOKEN|OPENAI_API_KEY|model_provider\s*=|env_key\s*=|experimental_bearer_token|cli_auth_credentials_store/.test(
content,
)
) {
throw projectAuthConflict([relative]);
}
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
}
}
const parent = path.dirname(directory);
if (parent === directory) break;
directory = parent;
}
}
export async function prepareManagedAiRuntime(
db: Db,
input: {
companyId: string;
agentId: string;
responsibleUserId: string | null;
adapterType: string;
binding: AiConnectionBinding;
allowUninstalledPersonal?: boolean;
allowUninstalledShared?: boolean;
allowLegacyValidation?: boolean;
config: Record<string, unknown>;
},
) {
const configuredEnv =
input.config.env && typeof input.config.env === "object"
? (input.config.env as Record<string, unknown>)
: {};
for (const key of [
"ANTHROPIC_BASE_URL",
"OPENAI_BASE_URL",
"XAI_BASE_URL",
"CLAUDE_CODE_USE_BEDROCK",
"CLAUDE_CODE_USE_VERTEX",
"CLAUDE_CODE_USE_FOUNDRY",
"PAPERCLIP_OPENCODE_PROVIDERS",
"PAPERCLIP_AI_PROVIDER_URL", "PAPERCLIP_CODEX_PROVIDERS", "PAPERCLIP_PI_PROVIDERS", "OPENCODE_CONFIG_CONTENT", "OPENCODE_CONFIG", "OPENCODE_CONFIG_DIR", "GOOGLE_GEMINI_BASE_URL", "ANTHROPIC_BEDROCK_BASE_URL",
]) {
if (configuredEnv[key])
throw unprocessable(
"The configured provider routing is incompatible with this AI connection",
{ code: "ai_connection_incompatible" },
);
}
const harness = aiRoutingHarness(input.adapterType, input.config.provider, input.config.acpxAgent);
await assertManagedAiProjectAuth(input.config, harness === "claude_local" ? "anthropic" : harness === "codex_local" ? "openai" : input.binding.provider);
const service = aiConnectionService(db);
let selection = await service.select({
...input,
userId: input.responsibleUserId,
model: input.config.model,
runnerProvider: input.config.provider,
acpxAgent: input.config.acpxAgent,
});
const retainedCleanupScope = { companyId: input.companyId, accountKey: selection.grant.id };
void reconcileRetainedAiCleanup(retainedCleanupScope).catch(() => undefined);
const subscriptionFile =
selection.attribution.method === "subscription" &&
input.binding.provider !== "anthropic";
let home: string | undefined;
let releaseCredentialHome: (() => void) | undefined;
try {
const selectedGrantId = selection.grant.id;
selection = await service.select({
...input,
userId: input.responsibleUserId,
model: input.config.model,
runnerProvider: input.config.provider,
acpxAgent: input.config.acpxAgent,
});
if (selection.grant.id !== selectedGrantId)
throw unprocessable(
"The selected default changed. Retry this execution.",
);
const credentialRef = selection.grant.credentialSecretRefs.find((ref) => ref.configPath === "ai.credential");
const routing = aiConnectionMetadataSchema.parse(selection.connection.config.ai).routing;
const noAuth = routing?.auth === "none";
if (!noAuth && !credentialRef) throw unprocessable("The selected AI credential is unavailable");
const readFreshness = async () => {
if (!credentialRef) return undefined;
const [currentGrant] = await db.select({ refs: connectionGrants.credentialSecretRefs, status: connectionGrants.status })
.from(connectionGrants).where(and(eq(connectionGrants.companyId, input.companyId), eq(connectionGrants.id, selection.grant.id)));
const currentRef = currentGrant?.refs.find(ref => ref.configPath === "ai.credential");
if (currentGrant?.status !== "active" || currentRef?.secretId !== credentialRef.secretId || currentRef.versionSelector !== credentialRef.versionSelector) return undefined;
// The share lock waits behind a rotating writer, so the read after
// resolution sees a concurrent rotation instead of the pre-rotation
// version. A lock takes no new row version and aborts no admission.
return (await db.select({ epoch: companySecrets.aiSessionEpoch, version: companySecrets.latestVersion })
.from(companySecrets).where(and(eq(companySecrets.companyId, input.companyId), eq(companySecrets.id, credentialRef.secretId))).for("share"))[0];
};
const { value, freshness } = await (async () => {
if (noAuth) return { value: "", freshness: undefined };
// Recovering a rotated quota token can advance the secret version during
// the first read. Re-read the saved credential, retaining the epoch guard
// against a concurrent reconnect or explicit rotation.
for (let attempt = 0; attempt < 2; attempt += 1) {
const before = await readFreshness();
const value = await service.runtimeCredential(selection);
const after = await readFreshness();
if (before && after && before.version === after.version && before.epoch === after.epoch) {
return { value, freshness: after };
}
if (!before || before.epoch !== after?.epoch) break;
}
throw unprocessable("The AI credential changed during preparation; retry this execution");
})();
const antigravitySubscription = harness === "antigravity_runtime" && selection.attribution.method === "subscription";
const originalCredential = antigravitySubscription ? sanitizeAntigravityCredential(value) : value;
home = await mkdtemp(
path.join(
os.tmpdir(),
`paperclip-ai-${input.companyId}-${selection.grant.id}-`,
),
);
if (harness === "antigravity_runtime") home = await realpath(home);
const providerHome = path.join(home, "provider");
await mkdir(providerHome, { mode: 0o700 });
if (retainedCleanupScope) releaseCredentialHome = bindRetainedAiCredentialHome(providerHome, retainedCleanupScope);
const env: Record<string, unknown> = {
...stripAiAuthBindings(input.config.env),
...Object.fromEntries(AI_AUTH_ENV_KEYS.map((key) => [key, ""])),
...managedAiHomeEnvironment(home),
...(input.binding.provider === "github" ? { GH_TOKEN: "", GITHUB_TOKEN: "" } : {}),
};
const capability =
AI_CONNECTION_CAPABILITIES[input.binding.provider].methods[
selection.attribution.method
]!;
const authFile = antigravitySubscription
? path.join(providerHome, "antigravity-acp", "acp_token.json")
: path.join(providerHome, "auth.json");
if (harness === "codex_local")
await writeFile(
path.join(providerHome, "config.toml"),
'cli_auth_credentials_store = "file"\n',
{ mode: 0o600 },
);
if (!routing && subscriptionFile && !antigravitySubscription) await writeFile(authFile, value, { mode: 0o600 });
else if (!routing && !subscriptionFile) env[capability.envKey] = value;
if (
!routing && input.binding.provider === "openai" && harness === "codex_local" &&
selection.attribution.method === "api_key"
) {
env.CODEX_API_KEY = value;
await writeFile(authFile, JSON.stringify({ OPENAI_API_KEY: value }), {
mode: 0o600,
});
}
if (!routing && input.binding.provider === "openrouter" && harness === "opencode_local") {
env.OPENCODE_CONFIG_CONTENT = JSON.stringify({
provider: { openrouter: { options: { apiKey: value, headers: OPENROUTER_ATTRIBUTION_HEADERS } } },
});
env.OPENCODE_DISABLE_PROJECT_CONFIG = "true";
}
if (!routing && input.binding.provider === "google" && harness === "gemini_local") {
// Gemini headless CLI requires an explicit auth choice even with an API key.
// Seed only the disposable connection home, never the operator's settings.
const geminiHome = path.join(home, ".gemini");
await mkdir(geminiHome, { mode: 0o700 });
await writeFile(path.join(geminiHome, "settings.json"), JSON.stringify({
selectedAuthType: "gemini-api-key",
security: { auth: { selectedType: "gemini-api-key" } },
}), { mode: 0o600 });
}
if (harness === "antigravity_runtime") {
// Native Antigravity reads this explicit auth selection beneath GEMINI_HOME.
// The selected AI connection owns this disposable home, not a browser login.
env.GEMINI_HOME = providerHome;
const antigravityHome = path.join(providerHome, "antigravity-acp");
await mkdir(antigravityHome, { mode: 0o700 });
await writeFile(path.join(antigravityHome, "settings.json"), JSON.stringify({
auth: { type: antigravitySubscription ? "oauth-personal" : "gemini-api-key" },
}), { mode: 0o600 });
if (antigravitySubscription) await writeFile(authFile, originalCredential, { mode: 0o600 });
}
const projected = routing ? managedProviderRouting(routing, harness, value, typeof input.config.model === "string" ? input.config.model : "") : undefined;
if (projected) {
const customHeaders = env.ANTHROPIC_CUSTOM_HEADERS ?? process.env.ANTHROPIC_CUSTOM_HEADERS;
Object.assign(env, projected.env);
if (projected.env.ANTHROPIC_CUSTOM_HEADERS && customHeaders !== undefined) {
// Preserve secret references until adapter resolution; the adapter merges
// attribution after resolving them. Ordinary gateway headers survive too.
env.ANTHROPIC_CUSTOM_HEADERS = typeof customHeaders === "string"
? openRouterClaudeHeaders(customHeaders)
: customHeaders;
}
if (projected.hermesConfig) await writeFile(path.join(providerHome, "config.yaml"), projected.hermesConfig, { mode: 0o600 });
if (projected.codexConfig) await writeFile(path.join(providerHome, "config.toml"), 'cli_auth_credentials_store = "file"\n' + projected.codexConfig, { mode: 0o600 });
}
const attribution = { ...selection.attribution } as typeof selection.attribution & { subscriptionId?: string };
if (attribution.method === "subscription") {
const subscription = {
companyId: input.companyId, connectionId: selection.connection.id, grantId: selection.grant.id,
provider: selection.attribution.provider, name: selection.connection.name,
ownerUserId: selection.grant.subjectUserId, credential: value,
session: credentialRef && freshness ? { secretId: credentialRef.secretId, epoch: freshness.epoch } : null,
};
attribution.subscriptionId = await subscriptionService(db).register(subscription);
void refreshSubscriptionConnection(db, subscription, attribution.subscriptionId);
}
const generation = createHash("sha256")
.update(originalCredential)
.digest("hex")
.slice(0, 16);
const identity = `${selection.grant.id}:${input.responsibleUserId ?? "shared"}:${generation}`;
const sessionIdentity = `${selection.grant.id}:${input.responsibleUserId ?? "shared"}:${noAuth ? "no-auth" : `${credentialRef!.secretId}:${freshness!.epoch}`}`;
let homeRemoved = false;
let ownsGoogleRecovery = false;
let removalOnly = false;
let deferredWriteback: { cause: unknown } | undefined;
const removeHome = async () => {
if (home) await rm(home, { recursive: true, force: true });
homeRemoved = true;
releaseCredentialHome?.();
};
let cleanupRunning: Promise<void> | undefined;
const cleanupRuntime = (): Promise<void> => cleanupRunning ??= (async () => {
if (homeRemoved) return;
const cleanup = async () => {
if (removalOnly) {
await removeHome();
return;
}
let googleRecovery: ReturnType<typeof quotaCredentialRecovery> | undefined;
let durableGoogleRefresh = false;
try {
if (subscriptionFile) {
const incoming = antigravitySubscription
? sanitizeAntigravityCredential(await readLocalAiCredentialFile(authFile)) : undefined;
if (antigravitySubscription && incoming !== originalCredential) {
const accountEmail = selection.grant.providerTenant?.name;
if (!credentialRef || !freshness || !accountEmail) throw new Error("The selected Google credential recovery authority is unavailable");
googleRecovery = quotaCredentialRecovery(input.companyId, selection.grant.id);
const prior = await googleRecovery.read();
if (prior && prior.value !== incoming && !ownsGoogleRecovery) {
// A previous retired owner keeps its first recoverable refresh.
// Settle it before this owner can stage another generation.
await aiConnectionService(db).runtimeCredential(selection);
}
// Physical retirement is the caller's cleanup precondition. Save
// before network verification or a database write can fail; only
// this exact grant generation may later consume the encrypted copy.
await googleRecovery.write({ companyId: input.companyId, grantId: selection.grant.id,
connectionId: selection.connection.id, secretId: credentialRef.secretId,
baseHash: quotaCredentialHash(value), value: incoming!, provider: "google",
google: { sessionEpoch: freshness.epoch, baseVersion: freshness.version,
versionSelector: credentialRef.versionSelector ?? "latest", accountEmail, verified: false } });
ownsGoogleRecovery = true;
}
// Verification itself can consume a rotating refresh. Until its
// newest result is durably journaled, the private home owns it.
durableGoogleRefresh = false;
const refreshed = antigravitySubscription
? await prepareAntigravityCredentialWriteback({ originalCredential,
refreshedCredential: incoming!, managedHome: providerHome,
accountEmail: selection.grant.providerTenant?.name,
})
: await readFile(authFile, "utf8");
if (googleRecovery) {
const pending = await googleRecovery.read();
if (!pending?.google) throw new Error("The Google credential recovery journal is unavailable");
await googleRecovery.write({ ...pending, value: refreshed, google: { ...pending.google, verified: true } });
durableGoogleRefresh = true;
}
if (refreshed !== originalCredential) {
// A quota exchange can hold this company lock for 60 seconds.
// Retry its 30-second acquisition timeout without discarding the
// provider's only copy of a rotated, single-use refresh token.
const writeBack = () => withAccountHomeSecretMutationLock(undefined, input.companyId, () => db.transaction(async (tx) => {
const [grant] = await tx
.select()
.from(connectionGrants)
.where(
and(
eq(connectionGrants.id, selection.grant.id),
eq(connectionGrants.companyId, input.companyId),
),
)
.for("update");
// A missing or revoked grant blocks the write-back. Among
// active copies, the merge decision below keeps the
// credential with the newest provider freshness field.
if (!grant || grant.status !== "active") return;
const ref = grant.credentialSecretRefs.find(
(r) => r.configPath === "ai.credential",
);
if (!ref) return;
if (antigravitySubscription && (!credentialRef || ref.secretId !== credentialRef.secretId
|| ref.versionSelector !== credentialRef.versionSelector
|| grant.providerTenant?.name?.toLowerCase() !== selection.grant.providerTenant?.name?.toLowerCase())) return;
// Lock the referenced secret row for the rest of this
// transaction. The grant-row lock above does not cover it,
// so an authorized rotation of this secret could otherwise
// land between the read and the write below and be
// overwritten by this stale write-back.
const [currentSecret] = await tx
.select({ id: companySecrets.id, epoch: companySecrets.aiSessionEpoch, version: companySecrets.latestVersion })
.from(companySecrets)
.where(
and(
eq(companySecrets.id, ref.secretId),
eq(companySecrets.companyId, input.companyId),
),
)
.for("update");
if (antigravitySubscription && (!freshness || !currentSecret || currentSecret.epoch !== freshness.epoch
|| currentSecret.version !== freshness.version)) return;
const current = await aiConnectionService(
tx as unknown as Db,
).credential({ ...selection, grant });
if (antigravitySubscription) {
// Google refresh tokens have no provider freshness counter. Only
// the exact selected generation may replace itself; a reconnect,
// rotation, grant remap or concurrent refresh wins over this run.
if (sanitizeAntigravityCredential(current) !== originalCredential) return;
await secretService(tx).rotate(ref.secretId,
{ value: refreshed, preserveAiSessionEpoch: true }, { userId: grant.subjectUserId });
return;
}
const destination = path.join(
providerHome,
"current-auth.json",
);
await writeFile(destination, current, { mode: 0o600 });
const decision =
input.binding.provider === "openai"
? await decideCodexAuthMerge(authFile, destination, {
errorLabel: "AI account refresh",
})
: await decideGrokAuthMerge(authFile, destination, {
errorLabel: "AI account refresh",
});
if (decision !== 10) return;
await secretService(tx).rotate(
ref.secretId,
{ value: refreshed, preserveAiSessionEpoch: true },
{ userId: grant.subjectUserId },
);
// Keep grant.updatedAt for explicit account/access changes.
// The rotated secret revision invalidates quota caches without
// rejecting an otherwise valid in-flight reconnect.
}));
for (let attempt = 0; ; attempt++) {
try { await writeBack(); break; }
catch (error) {
if (attempt >= 2 || (error as NodeJS.ErrnoException)?.code !== WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE) throw error;
}
}
}
}
await googleRecovery?.clear();
// A committed save, intentional generation discard, or encrypted
// recovery handoff must own the refresh before deleting its home.
removalOnly = true;
await removeHome();
} catch (error) {
if (!durableGoogleRefresh && !removalOnly) throw error;
// Once native retirement and encrypted fsync transfer ownership, the
// original controller/home is unnecessary for next-use recovery.
if (!removalOnly) {
removalOnly = true;
deferredWriteback = { cause: error };
}
await removeHome();
if (deferredWriteback) throw new RetiredCredentialRecoveryError(deferredWriteback.cause);
}
};
// Journal writes and generation checks share the same reconnect fence.
try {
return await (antigravitySubscription
? withAccountHomeSecretMutationLock(undefined, input.companyId, cleanup) : cleanup());
} catch (error) {
if (retainedCleanupScope && !homeRemoved) retainAiCredentialCleanup({ scope: retainedCleanupScope, home: providerHome, cleanup: cleanupRuntime });
throw error;
}
})().finally(() => { cleanupRunning = undefined; });
return {
sessionIdentity,
...(retainedCleanupScope ? { retainedCleanupScope } : {}),
...(subscriptionFile && (input.binding.provider === "openai" || input.binding.provider === "xai") && credentialRef && freshness ? {
credentialWriteback: { grantId: selection.grant.id, secretId: credentialRef.secretId, sessionEpoch: freshness.epoch,
versionSelector: credentialRef.versionSelector ?? "latest" } satisfies ManagedAiCredentialWritebackBinding,
} : {}),
config: {
...input.config,
...projected?.config,
...(routing ? { managedAiRouting: routing } : {}),
env,
managedAiConnection: { ...attribution, identity, sessionIdentity },
},
attribution,
accountName: selection.connection.name,
accountOwnerUserId: selection.grant.subjectUserId,
identity,
home,
cleanup: cleanupRuntime,
};
} catch (error) {
if (home) await rm(home, { recursive: true, force: true });
releaseCredentialHome?.();
throw error;
}
}
export async function withManagedAiProbe<T>(
db: Db,
input: Parameters<typeof prepareManagedAiRuntime>[1],
probe: (runtime: Awaited<ReturnType<typeof prepareManagedAiRuntime>>) => Promise<T>,
): Promise<T> {
const runtime = await prepareManagedAiRuntime(db, input);
let retained = false;
try {
return await probe(runtime);
}
catch (error) {
if (error instanceof NativeEnvironmentProbeCleanupError) {
// Keep the selected credential generation until the native owner has
// retired and completed its refresh handoff. HTTP need not wait for it.
retained = true;
const retryCleanup = (error as { retryCleanup?: () => Promise<void> }).retryCleanup;
retainAiCredentialCleanup({ scope: runtime.retainedCleanupScope!, home: runtime.home && path.join(runtime.home, "provider"), cleanup: runtime.cleanup, retired: error.retired,
...(typeof retryCleanup === "function" ? { retryRetirement: () => retryCleanup.call(error) } : {}) });
throw unprocessable("The native account test is still cleaning up. Retry after runtime cleanup completes.", { code: "native_probe_cleanup_pending" });
}
throw error;
} finally {
if (!retained) {
try { await runtime.cleanup(); }
catch (error) {
if (runtime.retainedCleanupScope && !(error instanceof RetiredCredentialRecoveryError)) {
retainAiCredentialCleanup({ scope: runtime.retainedCleanupScope, home: runtime.home && path.join(runtime.home, "provider"), cleanup: runtime.cleanup });
}
throw error;
}
}
}
}