mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
test(decisions): let the concurrent admission test survive a legitimate serialization retry (#21416)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Decision models let features ask a company AI connection short questions. Each decision reserves budget before it calls the provider. > - The server test "admits concurrent work from the same budget observation" fails or hangs intermittently in CI. This blocks Core image publication from master. > - The test assumed that both admission snapshots always survive. They do not. A legitimate serialization retry can make one admission see the other's committed hold. > - This pull request changes the test so that it proves what the product guarantees, with the same strength, and without the timing assumption. > - The benefit is a deterministic test and unblocked image publication. ## Linked Issues or Issue Description Refs #21387 **What happened?** The test "company decision service > admits concurrent work from the same budget observation and records both obligations" in `server/src/__tests__/decision-models.test.ts` hangs intermittently. CI run 43451649374 (attempt 2, job "verify / General tests (server (4/10))") failed with `Test timed out in 15000ms` at line 371. Before #21387 the same race failed with `expected "vi.fn()" to be called 2 times, but got 1 times`. **Root cause** - `decide()` resolves the credential before admission. `secrets.resolveSecretValue` writes `company_secrets.last_resolved_at` and `updated_at` in its own statement (`server/src/services/secrets.ts:1417-1421`). - Admission runs in a REPEATABLE READ `withOperationAccountingTransaction` (`server/src/services/accounting-transaction.ts:31-53`). Its snapshot starts before it waits on the test's lock. - When decision B's `last_resolved_at` write commits after decision A's snapshot, A's `SELECT ... FOR SHARE` on the same secret row (`server/src/services/decision-models.ts:263-264`) raises 40001. - The retry loop takes a fresh snapshot. If B has committed its 1c hold by then, A sees 1c held against a 1c hard stop and correctly returns `budget_blocked`. A never reaches the provider, so the test waits forever. **Expected behavior** The test passes every time. It still fails if admission becomes a company-wide writer mutex or if the hard stop stops blocking. ## What Changed - Set the test budget to 2c with 1c reservations. Both decisions are now admitted whether or not one admission retries. - Kept the proof that both admissions are in flight at once (both wait on the same row lock at the same time). This is the no-mutex guarantee. - Kept the checks that both providers run concurrently, both holds are `held` at 1c, both decisions succeed, and two cost events exist. - Added a check that a third decision returns `budget_blocked` while both holds are held. - Renamed the test to "admits concurrent work without a company writer mutex and records both obligations". - No product code changed. The retry and the `budget_blocked` result after the retry are correct hard-stop behavior. ## Verification - Root cause reproduced with temporary instrumentation (not committed). Starting decision B after decision A was parked gave 40001 on the `company_secrets` FOR SHARE in 4/4 runs. With the retry delay forced to 300 ms, the old test failed with the exact CI message `expected "vi.fn()" to be called 2 times, but got 1 times`, and the log showed `admitted` then `budget_blocked`. - The new test passes in that forced worst case (staggered starts plus a 300 ms retry delay): both admitted, the third `budget_blocked`. - 50 repetitions (vitest `repeats: 49`, temporary) with 16 extra CPU busy loops on an 18-core machine: 50/50 pass. - Red check: changing the company read in `withOperationAccountingTransaction` to `FOR NO KEY UPDATE` (a company writer mutex) makes the test fail with `expected 1 to be 2`. - `npx vitest run src/__tests__/decision-models.test.ts`: 37/37 in 2 of 3 runs. One run had a failure in a different test ("returns answers and a committed charge before slow budget stop delivery"). That test passes alone and is not changed here. ## Risks - Low risk. Only one test changes. - The test no longer proves that two admissions from the same snapshot both pass a 1c hard stop. That overshoot is not a product guarantee: any concurrent committed write to a share-locked row restarts the snapshot. - Each concurrent decision on the same credential can force a sibling admission to retry because of the `last_resolved_at` write. This is extra latency, not a correctness problem. A follow-up can remove that write from the decision hot path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c1ea9304f7
commit
4e2c149f2f
1 file changed
+3
-3
@@ -368,10 +368,10 @@ describe("company decision service", () => {
|
||||
await db.update(heartbeatRuns).set({ responsibleUserId: null }).where(eq(heartbeatRuns.id, runId));
|
||||
expect(await f.service.availability(context)).toMatchObject({ available: false, reason: "responsible_user_missing" });
|
||||
});
|
||||
it("admits concurrent work from the same budget observation and records both obligations", async () => {
|
||||
it("admits concurrent work without a company writer mutex and records both obligations", async () => {
|
||||
const f = await fixture();
|
||||
await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId,
|
||||
windowKind: "calendar_month_utc", amount: 1, reservationCents: "1", notifyEnabled: false });
|
||||
windowKind: "calendar_month_utc", amount: 2, reservationCents: "1", notifyEnabled: false });
|
||||
let unlock!: () => void, locked!: (pid: number) => void, finishProviders!: () => void;
|
||||
const lockGate = new Promise<void>(resolve => { unlock = resolve; });
|
||||
const acquired = new Promise<number>(resolve => { locked = resolve; });
|
||||
@@ -390,7 +390,6 @@ describe("company decision service", () => {
|
||||
});
|
||||
const first = f.service.decide(f.context, DECISION_TEST_REQUEST), second = f.service.decide(f.context, DECISION_TEST_REQUEST);
|
||||
try {
|
||||
// Both admission snapshots exist before either independent reservation commits.
|
||||
await vi.waitFor(async () => {
|
||||
const [row] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity where ${pid} = any(pg_blocking_pids(pid))`);
|
||||
expect(row.count).toBe(2);
|
||||
@@ -401,6 +400,7 @@ describe("company decision service", () => {
|
||||
const holds = await db.select().from(budgetReservations).where(eq(budgetReservations.companyId, f.companyId));
|
||||
expect(holds).toHaveLength(2);
|
||||
expect(holds.every(row => row.state === "held" && row.amountCents === "1.0000000")).toBe(true);
|
||||
expect(await f.service.decide(f.context, DECISION_TEST_REQUEST)).toMatchObject({ status: "unavailable", reason: "budget_blocked" });
|
||||
} finally { unlock(); finishProviders(); await holding; await Promise.allSettled([first, second]); }
|
||||
expect((await first).status).toBe("succeeded"); expect((await second).status).toBe("succeeded");
|
||||
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId))).toHaveLength(2);
|
||||
|
||||
Reference in new issue
Block a user