mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget policies and usage records are written in serialized accounting transactions for each company > - When several writes for one company run at the same time, PostgreSQL rejects all but one and Paperclip retries the others > - The retries were few and short (4 attempts, 10 to 30 ms, the same delay for every writer), so concurrent writers collided again and the request failed with 503 > - The 503 message also said "this run's budget" for a policy write that has no run > - This pull request retries contended accounting writes with jittered exponential backoff and gives the error a correct message > - The benefit is that normal concurrent budget changes succeed, and a real failure tells the user what happened ## Linked Issues or Issue Description **What happened?** Several budget policy writes for one company at the same time returned 503 after 4 serialization retries. The same retry exhaustion appeared often in server logs of the chat feature evals. The message said "Paperclip could not save this run's budget or usage records … Retry the run" for a policy write. **Expected behavior** Concurrent budget policy writes succeed. If contention continues, the error describes budget or usage changes, not a run. **Steps to reproduce** Send 9 budget policy upserts for one company at the same time. Before this change, about half of them fail with 503. **Paperclip version or commit** master ## What Changed - `withOperationAccountingTransaction` retries up to `ACCOUNTING_TRANSACTION_ATTEMPTS` (8) times. The delay before each retry is jittered exponential backoff (`accountingRetryDelayMs`): 10 ms × 2^(attempt+1), capped at 400 ms, randomized between half and the full value. The worst case total wait is about 1.4 s. - `AccountingContentionError` uses a new generic message: "Paperclip could not save these budget or usage changes because other updates to the same company kept conflicting with them. Try again." - A failed run keeps its run-specific `ACCOUNTING_CONTENTION_MESSAGE` (run completion still uses it). - The existing exhausted-retry test uses the new attempt constant. ## Verification - `cd server && npx vitest run src/__tests__/budget-policy-write-contention.test.ts` - Red before the change: 5 of 9 concurrent policy writes failed with `AccountingContentionError`, and the message mentioned a run. - Green after: all writes succeed (5 of 5 repeated runs; also 3 of 3 runs with 17 concurrent writers). - `run-accounting-contention`, `run-completion`, `budget-control-publication`, `budget-evaluation-work` and `heartbeat-native-predispatch-retry` tests. - Paid runner E2E evidence for `goals.align-existing-work` is linked in a comment. ## Risks - Low. A write that keeps losing now waits up to about 1.4 s before it fails, instead of about 60 ms. Only transactions that PostgreSQL rolled back (40001 or 40P01) are retried, as before. - Moving policy writes out of the accounting transaction was considered. It is not done here, because the policy fence keeps budget evaluation consistent with policy changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>