mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents call connector tools through the tool gateway. The gateway writes a `tool_call_events` audit row for each step of a call. > - The audit insert has foreign keys to `companies`, `agents`, `heartbeat_runs` and `issues`. Postgres takes a key-share lock on each parent row, in trigger order: company, agent, run, issue. > - Other writers lock the issue row and then the run row. When such a transaction holds the issue and then asks for the run, while the audit insert holds the run and waits for the issue, Postgres detects a deadlock and aborts the audit insert (`40P01`). > - The gateway then returns the raw `Failed query: insert into "tool_call_events" ...` error to the agent. The agent does not get the tool result. The same call works when the agent sends it again. > - This pull request runs the audit insert through the shared bounded retry that #20971 added for the same lock cycle on `workspace_operations`. > - The benefit is that a short lock conflict no longer fails a connector call or loses its audit row. ## Linked Issues or Issue Description **What happened?** Some connector calls fail with `Failed query: insert into "tool_call_events" ...`. This happens most often when an agent sends several tool calls in parallel in one run. The failed call waits for some seconds, then fails. The other parallel calls succeed. The same call succeeds when the agent sends it again. The Postgres cause is `deadlock detected` (`40P01`). **Expected behavior** A transient lock conflict does not fail the tool call. The gateway writes the audit row and returns the tool result. **Steps to reproduce** 1. Transaction A: lock the issue row `FOR UPDATE`. 2. Call a connector tool for a run on that issue. Its `tool_call_events` insert takes key-share on the company, agent and run rows, then waits for the issue. 3. Transaction A: lock the run row `FOR UPDATE`. 4. Postgres reports a deadlock. Before this change the audit insert is the victim, and the tool call fails with the database error. **Additional context** The new test reproduces this with two real transactions on embedded Postgres. Transaction A gets a longer `deadlock_timeout`, so the audit insert is always the victim, as in production. The failing statement in the test has the same column list and the same placeholder shape as the production error. Refs #20971 Refs #21703 Open PRs #15830 and #16596 make the audit insert non-fatal. That approach drops the audit row. This change keeps the row. ## What Changed - `server/src/services/tool-gateway.ts`: `writeToolCallEvent` runs its `tool_call_events` insert through `retryTransientLockContention`. The helper retries only lock timeout, deadlock and serialization failure (`55P03`, `40P01`, `40001`): 5 attempts, with 100 ms × attempt backoff. Other errors fail as before. - `server/src/__tests__/tool-gateway-connector-telemetry.test.ts`: new test with a real two-transaction deadlock between an issue-then-run locker and the audit insert of a real `executeTool` call. The fixture now also returns the issue. ## Verification - Failing before the fix: `cd server && npx vitest run src/__tests__/tool-gateway-connector-telemetry.test.ts -t "loses a deadlock"` failed. `executeTool` rejected with `Failed query: insert into "tool_call_events" ...`, cause `deadlock detected`, code `40P01`. - Passing after the fix: the same test passes. I ran it 3 times. - The full `tool-gateway-connector-telemetry.test.ts` file passes (9 tests). - `tool-gateway-service.test.ts` passes (194 tests). - Not run: `pnpm verify` and the server typecheck. The local sandbox has not enough memory and disk for them. CI runs both. ## Risks - Low risk. Only the `tool_call_events` insert retries, and only on the three contention codes. - Each attempt is a new single-statement write. A deadlock victim's statement rolls back completely, so a retry cannot write a duplicate row. - The worst added delay is about 1 s of backoff across 5 attempts, and only when the lock conflict continues. - Authorization, screening and approval checks do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test execution on embedded Postgres). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>