fix(secrets): stop credential reads from aborting concurrent budget admissions (#21452)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paid model calls (company decisions, fast responses, browser use,
prompt-injection checks) pass a budget admission before they start
> - Admission runs in a REPEATABLE READ transaction and share-locks the
credential's secret row, so a key rotation cannot slip between admission
and the call
> - Every credential read also wrote "Last used" (`last_resolved_at`,
`updated_at`) to that same row
> - When two calls ran at once, one call's credential read aborted the
other's admission with a serialization failure (40001). The retry then
saw the first call's new budget hold and denied the work as
`budget_blocked`
> - This is the flaky `decision-models.test.ts` failure that held Cloud
readiness, and so releases, on 2026-11-13
> - This pull request stops credential reads from writing the secret
row. "Last used" now comes from the access event that every read already
records
> - The benefit is that concurrent paid calls are no longer wrongly
denied or retried, and the release gate no longer fails on this test

## Linked Issues or Issue Description

No public issue exists. The bug is described here.

**What happened?**
Cloud readiness `verify / General tests (server (4/10))` failed on
master commits 6917d0568d and 5ede64f247. The test "admits concurrent
work from the same budget observation and records both obligations"
reported `expected "vi.fn()" to be called 2 times, but got 1 times`.
#21387 replaced the 1-second poll with an unbounded wait. That does not
fix it: one decision really is denied, so the test now waits until the
test timeout.

**Root cause**
I logged the accounting retry loop under load. Each batch of 50 runs had
6 to 10 retries with `could not serialize access due to concurrent
update` on `select … from company_secrets … for share`. The other
decision's credential read had updated the row. In 4 of 5 batches, the
retried admission saw the other decision's committed reservation and
returned `budget_blocked`.

**Expected behavior**
A credential read does not change admission. Two decisions admitted from
the same budget observation both run and both record their obligation.

**Steps to reproduce**
Run the updated test on the old `secrets.ts`. The test makes the second
credential read land after the first admission's snapshot. It then
commits an unrelated hold after both observations. The old code fails
every run with `first settled before its provider call:
{"status":"unavailable","reason":"budget_blocked"}`.

## What Changed

- `secretService` no longer writes `last_resolved_at` / `updated_at`
when it resolves a value. It still records the `secret_access_events`
row, as before.
- `secretService.list` reports `lastResolvedAt` as the later of the
stored column (legacy values) and the newest successful access event. It
reads one row per secret, backwards on the existing `(secret_id,
created_at)` index.
- A read without a consumer context (for example a GitHub API credential
read) now records a `system` access event, so it still counts as "Last
used".
- The removed `UPDATE` also made a credential read wait for a concurrent
rotation to commit. `credential()` in `ai-connections.ts` depends on
that wait to re-read the rotated version instead of handing a run a
consumed single-use token. Its existing version re-read now takes `FOR
SHARE`, so it keeps that wait. `prepareManagedAiRuntime` has its own
before/after freshness read around `runtimeCredential()`; that read
takes `FOR SHARE` too. A share lock creates no new row version and
cannot abort an admission. It adds no statement.
- The first version of this PR throttled the write to once a minute.
Review found that a read crossing the minute could still abort an
admission, so the write is removed instead.
- The concurrent-admission test is now deterministic. It starts decision
A and waits until A is blocked, then starts decision B and waits again.
It commits an unrelated 1-cent hold after both observations, then checks
that both decisions are admitted. A decision that settles before its
provider call fails the test immediately with its result, instead of
hanging.

## Verification

- Red: the new test against the old `secrets.ts` fails on every run with
the `budget_blocked` result above.
- Green: the same test with `repeats: 49` (50 runs) passes 50/50 locally
on a machine at load average 20 to 60.
- Before the change, 3 parallel batches of 50 runs of the old test each
produced a false denial.
- `npx tsc --noEmit -p server`: no errors.
- New `secrets-service.test.ts` test: a resolution leaves the secret row
unchanged, and `list()` still shows "Last used".
- `ai-connections.test.ts` quota test now checks that a read leaves
`updated_at` unchanged and records a success access event.

## Risks

- `updated_at` on a secret no longer moves when the secret is read. It
still moves on every real change.
- "Last used" on the Secrets page costs one indexed lookup per listed
secret.
- Only `list()` derives "Last used". Other endpoints that return a
secret row show the stored column, which is now set only by older
releases.
- No migration. No API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 19:46:50 -06:00
1 parent e67f3bd63a
commit 4879a146fb
8 files changed
+126 -16

No files matched your search

+5 -3
View File
@@ -17,7 +17,7 @@ import { mkdtemp, realpath, rm, access, readFile, writeFile, stat } from "node:f
import os from "node:os";
import path from "node:path";
import { and, eq, sql } from "drizzle-orm";
import { createDb, type Db, companies, agents, agentTaskSessions, heartbeatRuns, companyMemberships, connectionGrants, toolApplications, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, plugins, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests, companySecrets, companySecretVersions, userSecretDefinitions, principalPermissionGrants } from "@paperclipai/db";
import { createDb, type Db, companies, agents, agentTaskSessions, heartbeatRuns, companyMemberships, connectionGrants, toolApplications, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, plugins, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests, companySecrets, companySecretVersions, secretAccessEvents, userSecretDefinitions, principalPermissionGrants } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres";
import { fetchCompanyQuotaWindows } from "../services/quota-windows.js";
import { quotaCredentialRecoveryPath, quotaCredentialRecovery, quotaCredentialHash } from "../services/quota-credential-recovery.js";
@@ -717,8 +717,10 @@ describe("managed AI connections", () => {
try {
const first = await fetchCompanyQuotaWindows(db, companyId, owner);
const [after] = await db.select().from(companySecrets).where(eq(companySecrets.id, secretId));
expect(after.lastResolvedAt).not.toBeNull();
expect(after.updatedAt.getTime()).toBeGreaterThan(before.updatedAt.getTime());
// A read is recorded as an access event, never written to the row that
// admissions share-lock to fence rotation.
expect(after.updatedAt.getTime()).toBe(before.updatedAt.getTime());
expect(await db.select().from(secretAccessEvents).where(and(eq(secretAccessEvents.secretId, secretId), eq(secretAccessEvents.outcome, "success")))).not.toHaveLength(0);
expect(first[0].ok).toBe(true);
expect(await fetchCompanyQuotaWindows(db, companyId, owner)).toEqual(first);
expect(fetchSpy).toHaveBeenCalledTimes(1);
@@ -35,7 +35,7 @@ describe("Antigravity private managed AI home", () => {
const version = [{ epoch: 1, version: 1 }];
const db = { select: (fields: Record<string, unknown>) => ({ from: () => ({
where: () => Object.assign(Promise.resolve("refs" in fields ? [{ refs: account.refs, status: "active" }] : version),
{ limit: async () => version }),
{ limit: async () => version, for: async () => version }),
}) }) } as unknown as Db;
const input = {
companyId: "44444444-4444-4444-8444-444444444444", agentId: "55555555-5555-4555-8555-555555555555", responsibleUserId: "alice",
@@ -123,7 +123,8 @@ describe("managed Antigravity subscription projection and writeback", () => {
await expect(access(join(provider, "auth.json"))).rejects.toMatchObject({ code: "ENOENT" });
await writeFile(token, credential("resolved-project"));
await runtime.cleanup();
expect(account.locks).toEqual(["grant:update", "secret:update"]);
// Preparation reads the secret version under a share lock before and after resolving it.
expect(account.locks).toEqual(["secret:share", "secret:share", "grant:update", "secret:update"]);
expect(account.rotation).toHaveBeenCalledWith(account.refs[0].secretId,
{ value: credential("resolved-project"), preserveAiSessionEpoch: true }, { userId: "alice" });
await expect(access(runtime.home!)).rejects.toMatchObject({ code: "ENOENT" });
@@ -216,7 +217,7 @@ describe("managed Antigravity subscription projection and writeback", () => {
} else {
await expect(runtime.cleanup()).rejects.toThrow("The refreshed Google account does not match the selected connection.");
expect(account.rotation).not.toHaveBeenCalled();
expect(account.locks).toEqual([]);
expect(account.locks).toEqual(["secret:share", "secret:share"]);
expect(account.journal).toMatchObject({ provider: "google", google: { verified: false }, value: credential("resolved-project", "fixture-rotated-refresh") });
expect(await readFile(token, "utf8")).toBe(credential("resolved-project", "fixture-rotated-refresh"));
}
@@ -405,6 +405,54 @@ describe("company decision service", () => {
expect((await first).status).toBe("succeeded"); expect((await second).status).toBe("succeeded");
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId))).toHaveLength(2);
});
it("keeps an admission's budget observation when another decision reads the same credential mid-admission", async () => {
const f = await fixture();
await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId,
windowKind: "calendar_month_utc", amount: 1, reservationCents: "1", notifyEnabled: false });
let unlock!: () => void, locked!: (pid: number) => void, finishProviders!: () => void;
const lockGate = new Promise<void>(resolve => { unlock = resolve; });
const acquired = new Promise<number>(resolve => { locked = resolve; });
const providerGate = new Promise<void>(resolve => { finishProviders = resolve; });
const holding = db.transaction(async tx => {
await tx.select().from(companyDecisionModels).where(eq(companyDecisionModels.companyId, f.companyId)).for("no key update");
const [row] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
locked(row.pid); await lockGate;
});
const pid = await acquired;
const waitingAdmissions = (count: number) => vi.waitFor(async () => {
const [row] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity where ${pid} = any(pg_blocking_pids(pid))`);
expect(row.count).toBe(count);
}, { timeout: 10_000 });
let providersEntered = 0, bothEntered!: () => void;
const bothProvidersEntered = new Promise<void>(resolve => { bothEntered = resolve; });
f.provider.mockImplementation(async () => {
if (++providersEntered === 2) bothEntered();
await providerGate; return structuredClone(outcome);
});
// Order the two admissions so the second one's credential read happens
// after the first admission took its budget observation. Reading a
// credential must not invalidate that observation.
const first = f.service.decide(f.context, DECISION_TEST_REQUEST);
let second: ReturnType<typeof f.service.decide> | undefined;
try {
await waitingAdmissions(1);
second = f.service.decide(f.context, DECISION_TEST_REQUEST);
await waitingAdmissions(2);
// An obligation committed after both observations. An admission that
// keeps its observation ignores it; one that restarts is denied by it.
await db.insert(budgetReservations).values({ companyId: f.companyId, decisionInvocationId: randomUUID(), amountCents: "1" });
unlock(); await holding;
// Before the providers finish, a decision settles only if admission denied it.
const settledEarly = (name: string, pending: typeof first) => pending.then(result => `${name} settled before its provider call: ${JSON.stringify(result)}`);
expect(await Promise.race([bothProvidersEntered.then(() => "both admitted"), settledEarly("first", first), settledEarly("second", second)])).toBe("both admitted");
expect(f.provider).toHaveBeenCalledTimes(2);
const holds = await db.select().from(budgetReservations).where(and(eq(budgetReservations.companyId, f.companyId), sql`${budgetReservations.decisionInvocationId} in (select id from decision_invocations)`));
expect(holds).toHaveLength(2);
expect(holds.every(row => row.state === "held" && row.amountCents === "1.0000000")).toBe(true);
} finally { unlock(); finishProviders(); await holding; await Promise.allSettled([first, second]); }
expect((await first).status).toBe("succeeded"); expect((await second!).status).toBe("succeeded");
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId))).toHaveLength(2);
});
it("denies new work when an already committed reservation consumes the observed budget", async () => {
const f = await fixture();
await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId, windowKind: "calendar_month_utc", amount: 1, reservationCents: "1", notifyEnabled: false });
@@ -2660,6 +2660,37 @@ describeEmbeddedPostgres("secretService", () => {
).rejects.toThrow(/not bound/i);
});
it("records a read as \"Last used\" without writing the secret row that admissions lock", async () => {
const companyId = await seedCompany();
const svc = secretService(db);
const secret = await svc.create(companyId, {
name: `last-used-${randomUUID()}`,
provider: "local_encrypted",
value: "runtime-secret",
});
await svc.createBinding({ companyId, secretId: secret.id, targetType: "system", targetId: "system", configPath: "env.API_KEY" });
const [before] = await db.select().from(companySecrets).where(eq(companySecrets.id, secret.id));
expect((await svc.list(companyId)).find((row) => row.id === secret.id)?.lastResolvedAt).toBeNull();
await svc.resolveSecretValue(companyId, secret.id, "latest", { consumerType: "system", consumerId: "system", configPath: "env.API_KEY" });
// The row keeps its version: a concurrent REPEATABLE READ admission that
// share-locks it to fence rotation is not aborted by this read.
const [after] = await db.select().from(companySecrets).where(eq(companySecrets.id, secret.id));
expect(after).toEqual(before);
const listed = (await svc.list(companyId)).find((row) => row.id === secret.id);
expect(listed?.lastResolvedAt).toBeInstanceOf(Date);
expect(listed!.lastResolvedAt!.getTime()).toBeGreaterThanOrEqual(before!.createdAt.getTime());
// A read made without a consumer context also counts as use.
const unattributed = await svc.create(companyId, { name: `last-used-unattributed-${randomUUID()}`, provider: "local_encrypted", value: "runtime-secret" });
const [unattributedBefore] = await db.select().from(companySecrets).where(eq(companySecrets.id, unattributed.id));
await svc.resolveSecretValue(companyId, unattributed.id, "latest");
const [unattributedAfter] = await db.select().from(companySecrets).where(eq(companySecrets.id, unattributed.id));
expect(unattributedAfter).toEqual(unattributedBefore);
expect((await svc.list(companyId)).find((row) => row.id === unattributed.id)?.lastResolvedAt).toBeInstanceOf(Date);
});
it("preserves the original resolution error when failure access logging fails", async () => {
const companyId = await seedCompany();
const svc = secretService(db);
+4 -1
View File
@@ -284,8 +284,11 @@ export async function prepareManagedAiRuntime(
.from(connectionGrants).where(and(eq(connectionGrants.companyId, input.companyId), eq(connectionGrants.id, selection.grant.id)));
const currentRef = currentGrant?.refs.find(ref => ref.configPath === "ai.credential");
if (currentGrant?.status !== "active" || currentRef?.secretId !== credentialRef.secretId || currentRef.versionSelector !== credentialRef.versionSelector) return undefined;
// The share lock waits behind a rotating writer, so the read after
// resolution sees a concurrent rotation instead of the pre-rotation
// version. A lock takes no new row version and aborts no admission.
return (await db.select({ epoch: companySecrets.aiSessionEpoch, version: companySecrets.latestVersion })
.from(companySecrets).where(and(eq(companySecrets.companyId, input.companyId), eq(companySecrets.id, credentialRef.secretId))).limit(1))[0];
.from(companySecrets).where(and(eq(companySecrets.companyId, input.companyId), eq(companySecrets.id, credentialRef.secretId))).for("share"))[0];
};
const { value, freshness } = await (async () => {
if (noAuth) return { value: "", freshness: undefined };
+4 -3
View File
@@ -615,11 +615,12 @@ export function aiConnectionService(db: Db) {
} else {
value = await secrets.resolveSecretValue(row.connection.companyId, secret.id, "latest", context);
}
// Resolution records lastResolvedAt and can wait behind a rotating writer.
// Re-read after that wait instead of handing a run the pre-rotation token.
// Wait behind a rotating writer (the share lock), then re-read instead of
// handing a run the pre-rotation token. A lock takes no new row version,
// so it never aborts a concurrent REPEATABLE READ admission.
const [latest] = await db.select().from(companySecrets).where(and(
eq(companySecrets.id, secret.id), eq(companySecrets.companyId, row.connection.companyId),
));
)).for("share");
if (!latest || latest.status !== "active") throw unprocessable("Reconnect this AI account");
if (latest.latestVersion !== secret.latestVersion) {
if (retry >= 2) throw unprocessable("AI credentials are changing. Retry this execution.", { code: "ai_connection_busy" });
+30 -6
View File
@@ -88,6 +88,12 @@ import {
withAccountHomeSecretMutationLock,
} from "@paperclipai/adapter-codex-local/server";
function latestDate(a: Date | null, b: Date | null): Date | null {
if (!a) return b;
if (!b) return a;
return a.getTime() >= b.getTime() ? a : b;
}
const ENV_KEY_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;
const AGENT_ACCESS_CONFIG_PATH_PREFIX = "access.";
// System consumer id for a durable orphan-sandbox teardown. The cleanup sweep
@@ -642,6 +648,8 @@ type SecretConsumerContext = {
allowedBindingIds?: string[] | null;
};
const UNATTRIBUTED_SECRET_READ: SecretConsumerContext = { consumerType: "system", consumerId: "system", actorType: "system" };
type SecretBindingContext = Omit<SecretConsumerContext, "consumerType"> & {
consumerType: SecretBindingTargetType;
};
@@ -1365,6 +1373,19 @@ export function secretService(db: Db | DbTransaction) {
};
}
/** Newest successful resolution per secret, read backwards on (secret_id, created_at). */
async function lastSuccessfulResolutions(secretIds: string[]): Promise<Map<string, Date>> {
if (secretIds.length === 0) return new Map();
const rows = await db.execute<{ secret_id: string; used_at: string | Date | null }>(sql`
select ids.secret_id, (
select e.created_at from ${secretAccessEvents} e
where e.secret_id = ids.secret_id and e.outcome = 'success'
order by e.created_at desc limit 1
) as used_at
from unnest(array[${sql.join(secretIds.map((id) => sql`${id}`), sql`, `)}]::uuid[]) as ids(secret_id)`);
return new Map(rows.flatMap((row) => row.used_at ? [[row.secret_id, new Date(row.used_at)] as const] : []));
}
async function resolveSecretValueInternal(
companyId: string,
secretId: string,
@@ -1428,12 +1449,12 @@ export function secretService(db: Db | DbTransaction) {
if (isTailscaleNodeState(value) && systemPurpose !== "tailscale_network") {
throw forbidden("Network node state requires its server-only consumer", { code: "secret_system_only" });
}
// The access event is the usage record; "Last used" is derived from it
// in list(). A read must not write the secret row: admission paths
// (decisions, fast responses, browser use, prompt-injection checks)
// share-lock it under REPEATABLE READ to fence rotation, and a new row
// version would abort them with a serialization failure.
if (!readOnlyEnvironmentInspection) await Promise.all([
db
.update(companySecrets)
.set({ lastResolvedAt: new Date(), updatedAt: new Date() })
.where(eq(companySecrets.id, secret.id))
.catch(() => undefined),
recordAccessEvent({
companyId,
secretId: secret.id,
@@ -1441,7 +1462,8 @@ export function secretService(db: Db | DbTransaction) {
secretScope: secret.scope,
version: resolvedVersion,
provider: providerId,
context: accessContext,
// A read without a consumer still counts as "Last used".
context: accessContext ?? UNATTRIBUTED_SECRET_READ,
credentialOwnerUserId: secret.ownerUserId ?? null,
credentialSubjectType: secret.scope === "user" ? "user" : null,
credentialSubjectId: secret.ownerUserId ?? null,
@@ -3873,8 +3895,10 @@ export function secretService(db: Db | DbTransaction) {
.groupBy(companySecretBindings.secretId),
]);
const countsBySecretId = new Map(referenceCounts.map((row) => [row.secretId, row.count]));
const lastUsedBySecretId = await lastSuccessfulResolutions(secrets.map((secret) => secret.id));
return secrets.map((secret) => ({
...secret,
lastResolvedAt: latestDate(secret.lastResolvedAt, lastUsedBySecretId.get(secret.id) ?? null),
referenceCount: countsBySecretId.get(secret.id) ?? 0,
}));
},