test(decisions): let the concurrent admission test survive a legitimate serialization retry (#21416)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Decision models let features ask a company AI connection short
questions. Each decision reserves budget before it calls the provider.
> - The server test "admits concurrent work from the same budget
observation" fails or hangs intermittently in CI. This blocks Core image
publication from master.
> - The test assumed that both admission snapshots always survive. They
do not. A legitimate serialization retry can make one admission see the
other's committed hold.
> - This pull request changes the test so that it proves what the
product guarantees, with the same strength, and without the timing
assumption.
> - The benefit is a deterministic test and unblocked image publication.

## Linked Issues or Issue Description

Refs #21387

**What happened?**
The test "company decision service > admits concurrent work from the
same budget observation and records both obligations" in
`server/src/__tests__/decision-models.test.ts` hangs intermittently. CI
run 43451649374 (attempt 2, job "verify / General tests (server
(4/10))") failed with `Test timed out in 15000ms` at line 371. Before
#21387 the same race failed with `expected "vi.fn()" to be called 2
times, but got 1 times`.

**Root cause**
- `decide()` resolves the credential before admission.
`secrets.resolveSecretValue` writes `company_secrets.last_resolved_at`
and `updated_at` in its own statement
(`server/src/services/secrets.ts:1417-1421`).
- Admission runs in a REPEATABLE READ
`withOperationAccountingTransaction`
(`server/src/services/accounting-transaction.ts:31-53`). Its snapshot
starts before it waits on the test's lock.
- When decision B's `last_resolved_at` write commits after decision A's
snapshot, A's `SELECT ... FOR SHARE` on the same secret row
(`server/src/services/decision-models.ts:263-264`) raises 40001.
- The retry loop takes a fresh snapshot. If B has committed its 1c hold
by then, A sees 1c held against a 1c hard stop and correctly returns
`budget_blocked`. A never reaches the provider, so the test waits
forever.

**Expected behavior**
The test passes every time. It still fails if admission becomes a
company-wide writer mutex or if the hard stop stops blocking.

## What Changed

- Set the test budget to 2c with 1c reservations. Both decisions are now
admitted whether or not one admission retries.
- Kept the proof that both admissions are in flight at once (both wait
on the same row lock at the same time). This is the no-mutex guarantee.
- Kept the checks that both providers run concurrently, both holds are
`held` at 1c, both decisions succeed, and two cost events exist.
- Added a check that a third decision returns `budget_blocked` while
both holds are held.
- Renamed the test to "admits concurrent work without a company writer
mutex and records both obligations".
- No product code changed. The retry and the `budget_blocked` result
after the retry are correct hard-stop behavior.

## Verification

- Root cause reproduced with temporary instrumentation (not committed).
Starting decision B after decision A was parked gave 40001 on the
`company_secrets` FOR SHARE in 4/4 runs. With the retry delay forced to
300 ms, the old test failed with the exact CI message `expected
"vi.fn()" to be called 2 times, but got 1 times`, and the log showed
`admitted` then `budget_blocked`.
- The new test passes in that forced worst case (staggered starts plus a
300 ms retry delay): both admitted, the third `budget_blocked`.
- 50 repetitions (vitest `repeats: 49`, temporary) with 16 extra CPU
busy loops on an 18-core machine: 50/50 pass.
- Red check: changing the company read in
`withOperationAccountingTransaction` to `FOR NO KEY UPDATE` (a company
writer mutex) makes the test fail with `expected 1 to be 2`.
- `npx vitest run src/__tests__/decision-models.test.ts`: 37/37 in 2 of
3 runs. One run had a failure in a different test ("returns answers and
a committed charge before slow budget stop delivery"). That test passes
alone and is not changed here.

## Risks

- Low risk. Only one test changes.
- The test no longer proves that two admissions from the same snapshot
both pass a 1c hard stop. That overshoot is not a product guarantee: any
concurrent committed write to a share-locked row restarts the snapshot.
- Each concurrent decision on the same credential can force a sibling
admission to retry because of the `last_resolved_at` write. This is
extra latency, not a correctness problem. A follow-up can remove that
write from the decision hot path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 14:46:14 -06:00
1 parent c1ea9304f7
commit 4e2c149f2f
1 file changed
+3 -3
+3 -3
View File
@@ -368,10 +368,10 @@ describe("company decision service", () => {
await db.update(heartbeatRuns).set({ responsibleUserId: null }).where(eq(heartbeatRuns.id, runId));
expect(await f.service.availability(context)).toMatchObject({ available: false, reason: "responsible_user_missing" });
});
it("admits concurrent work from the same budget observation and records both obligations", async () => {
it("admits concurrent work without a company writer mutex and records both obligations", async () => {
const f = await fixture();
await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId,
windowKind: "calendar_month_utc", amount: 1, reservationCents: "1", notifyEnabled: false });
windowKind: "calendar_month_utc", amount: 2, reservationCents: "1", notifyEnabled: false });
let unlock!: () => void, locked!: (pid: number) => void, finishProviders!: () => void;
const lockGate = new Promise<void>(resolve => { unlock = resolve; });
const acquired = new Promise<number>(resolve => { locked = resolve; });
@@ -390,7 +390,6 @@ describe("company decision service", () => {
});
const first = f.service.decide(f.context, DECISION_TEST_REQUEST), second = f.service.decide(f.context, DECISION_TEST_REQUEST);
try {
// Both admission snapshots exist before either independent reservation commits.
await vi.waitFor(async () => {
const [row] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity where ${pid} = any(pg_blocking_pids(pid))`);
expect(row.count).toBe(2);
@@ -401,6 +400,7 @@ describe("company decision service", () => {
const holds = await db.select().from(budgetReservations).where(eq(budgetReservations.companyId, f.companyId));
expect(holds).toHaveLength(2);
expect(holds.every(row => row.state === "held" && row.amountCents === "1.0000000")).toBe(true);
expect(await f.service.decide(f.context, DECISION_TEST_REQUEST)).toMatchObject({ status: "unavailable", reason: "budget_blocked" });
} finally { unlock(); finishProviders(); await holding; await Promise.allSettled([first, second]); }
expect((await first).status).toBe("succeeded"); expect((await second).status).toBe("succeeded");
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId))).toHaveLength(2);