Files
PaperClipAI/server
DottaandClaude Opus 5.5 264beefeff fix(tools): retry a tool audit write that loses a deadlock instead of failing the call (#22329)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents call connector tools through the tool gateway. The gateway
writes a `tool_call_events` audit row for each step of a call.
> - The audit insert has foreign keys to `companies`, `agents`,
`heartbeat_runs` and `issues`. Postgres takes a key-share lock on each
parent row, in trigger order: company, agent, run, issue.
> - Other writers lock the issue row and then the run row. When such a
transaction holds the issue and then asks for the run, while the audit
insert holds the run and waits for the issue, Postgres detects a
deadlock and aborts the audit insert (`40P01`).
> - The gateway then returns the raw `Failed query: insert into
"tool_call_events" ...` error to the agent. The agent does not get the
tool result. The same call works when the agent sends it again.
> - This pull request runs the audit insert through the shared bounded
retry that #20971 added for the same lock cycle on
`workspace_operations`.
> - The benefit is that a short lock conflict no longer fails a
connector call or loses its audit row.

## Linked Issues or Issue Description

**What happened?**
Some connector calls fail with `Failed query: insert into
"tool_call_events" ...`. This happens most often when an agent sends
several tool calls in parallel in one run. The failed call waits for
some seconds, then fails. The other parallel calls succeed. The same
call succeeds when the agent sends it again. The Postgres cause is
`deadlock detected` (`40P01`).

**Expected behavior**
A transient lock conflict does not fail the tool call. The gateway
writes the audit row and returns the tool result.

**Steps to reproduce**
1. Transaction A: lock the issue row `FOR UPDATE`.
2. Call a connector tool for a run on that issue. Its `tool_call_events`
insert takes key-share on the company, agent and run rows, then waits
for the issue.
3. Transaction A: lock the run row `FOR UPDATE`.
4. Postgres reports a deadlock. Before this change the audit insert is
the victim, and the tool call fails with the database error.

**Additional context**
The new test reproduces this with two real transactions on embedded
Postgres. Transaction A gets a longer `deadlock_timeout`, so the audit
insert is always the victim, as in production. The failing statement in
the test has the same column list and the same placeholder shape as the
production error.

Refs #20971
Refs #21703

Open PRs #15830 and #16596 make the audit insert non-fatal. That
approach drops the audit row. This change keeps the row.

## What Changed

- `server/src/services/tool-gateway.ts`: `writeToolCallEvent` runs its
`tool_call_events` insert through `retryTransientLockContention`. The
helper retries only lock timeout, deadlock and serialization failure
(`55P03`, `40P01`, `40001`): 5 attempts, with 100 ms × attempt backoff.
Other errors fail as before.
- `server/src/__tests__/tool-gateway-connector-telemetry.test.ts`: new
test with a real two-transaction deadlock between an issue-then-run
locker and the audit insert of a real `executeTool` call. The fixture
now also returns the issue.

## Verification

- Failing before the fix: `cd server && npx vitest run
src/__tests__/tool-gateway-connector-telemetry.test.ts -t "loses a
deadlock"` failed. `executeTool` rejected with `Failed query: insert
into "tool_call_events" ...`, cause `deadlock detected`, code `40P01`.
- Passing after the fix: the same test passes. I ran it 3 times.
- The full `tool-gateway-connector-telemetry.test.ts` file passes (9
tests).
- `tool-gateway-service.test.ts` passes (194 tests).
- Not run: `pnpm verify` and the server typecheck. The local sandbox has
not enough memory and disk for them. CI runs both.

## Risks

- Low risk. Only the `tool_call_events` insert retries, and only on the
three contention codes.
- Each attempt is a new single-statement write. A deadlock victim's
statement rolls back completely, so a retry cannot write a duplicate
row.
- The worst added delay is about 1 s of backoff across 5 attempts, and
only when the lock conflict continues.
- Authorization, screening and approval checks do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution on embedded
Postgres).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:23:21 -06:00
..