fix(tools): retry a named MCP gateway create that loses a deadlock (#22408)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs prepare a run-scoped MCP gateway for their agent before
the provider starts. `createNamedGateway` writes the `tool_mcp_gateways`
row and its profile binding in one transaction.
> - That transaction locks the agent row first (`FOR NO KEY UPDATE`).
Then its gateway insert takes a key-share lock on the company and
profile rows for the foreign keys.
> - Another writer can lock the company row first and then the agent
row. The two transactions then wait for each other. Postgres detects a
deadlock and aborts the gateway create (`40P01`).
> - The run then fails before the provider starts. The task stops on
`native_continuation_requires_reconciliation`, so a person must
reconcile a run that did no work.
> - This pull request runs the gateway create transaction through the
shared bounded retry that #22329 and #20971 use for the same type of
lock cycle.
> - The benefit is that a short lock conflict no longer fails the run
preparation.

## Linked Issues or Issue Description

**What happened?**
A native run failed while it prepared its MCP gateway. The error was
`Failed query: insert into "tool_mcp_gateways" (...)`. The Postgres
cause was `deadlock detected` (`40P01`). The task then stopped and
needed reconciliation.

**Expected behavior**
A transient lock conflict does not fail the run preparation. The gateway
create writes the gateway and its profile binding, and the run
continues.

**Steps to reproduce**
1. Transaction A: lock the company row `FOR UPDATE`.
2. Call `createNamedGateway` for an agent in that company. It locks the
agent row and then waits for the company key-share lock.
3. Transaction A: lock the agent row `FOR UPDATE`.
4. Postgres reports a deadlock. Before this change the gateway create is
the victim and `createNamedGateway` rejects with the database error.

**Additional context**
The new test reproduces this with two real transactions on embedded
Postgres. Transaction A gets a longer `deadlock_timeout`, so the gateway
create is always the victim, as in the observed failure. The failing
statement in the test is the same insert as in the observed error.

Refs #22329
Refs #20971

## What Changed

- `server/src/services/tool-gateway.ts`: `createNamedGateway` runs its
transaction through `retryTransientLockContention`. The helper retries
only lock timeout, deadlock and serialization failure (`55P03`, `40P01`,
`40001`): 5 attempts, with 100 ms × attempt backoff. Other errors fail
as before.
- `server/src/__tests__/tool-gateway-named-gateway-deadlock.test.ts`:
new test with a real two-transaction deadlock between a
company-then-agent locker and a real `createNamedGateway` call. The test
checks that exactly one gateway row and one profile binding exist after
the retry.

## Verification

- Failing before the fix: `cd server && npx vitest run
src/__tests__/tool-gateway-named-gateway-deadlock.test.ts` failed.
`createNamedGateway` rejected with `Failed query: insert into
"tool_mcp_gateways" ...`, cause `deadlock detected`, code `40P01`.
- Passing after the fix: the same test passes. I ran it 3 times.
- `heartbeat-runtime-mcp-servers.test.ts`,
`agent-runtime-assignments.test.ts`,
`agent-interaction-permissions.test.ts` and
`tool-gateway-service.test.ts` pass (4 files, 238 tests).
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low risk. Only the gateway create transaction retries, and only on the
three contention codes.
- Postgres rolls back the complete aborted transaction. A retry
therefore cannot write a duplicate gateway or profile binding.
- The interaction-permission check runs again in each attempt. A retry
cannot skip it.
- If a concurrent create with the same slug commits first, the retry
gets a unique violation. That error is not retried. The existing native
caller then reads the committed gateway by slug, as before.
- The worst added delay is about 1 s of backoff across 5 attempts, and
only when the lock conflict continues.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution on embedded
Postgres).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-20 05:47:47 -08:00
1 parent fa96716bb5
commit a792409d07
2 files changed
+109 -2

No files matched your search

@@ -0,0 +1,103 @@
import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, companies, createDb, toolMcpGateways, toolProfileBindings, toolProfiles } from "@paperclipai/db";
import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./helpers/embedded-postgres.js";
import { createToolGatewayService } from "../services/tool-gateway.js";
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
// A native run creates its agent's MCP gateway while it prepares. The create
// transaction locks the agent row, then its gateway insert takes key-share on
// the company row for the foreign key. A writer that locks the company row and
// then the agent row closes the cycle. Postgres aborts the gateway insert with
// 40P01, and the run fails before the provider starts.
describeEmbeddedPostgres("named MCP gateway creation under a deadlock", () => {
let db!: ReturnType<typeof createDb>;
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
beforeAll(async () => {
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-named-gateway-deadlock-");
db = createDb(tempDb.connectionString);
}, 20_000);
afterAll(async () => {
await tempDb?.cleanup();
});
it("creates the gateway after it loses a deadlock to a company-then-agent locker", async () => {
const [company] = await db.insert(companies).values({
name: `Gateway deadlock ${randomUUID()}`,
issuePrefix: `GD${randomUUID().slice(0, 5).toUpperCase()}`,
}).returning();
const [agent] = await db.insert(agents).values({
companyId: company!.id,
name: "Native agent",
role: "engineer",
adapterType: "paperclip_runner",
adapterConfig: {},
}).returning();
const [profile] = await db.insert(toolProfiles).values({
companyId: company!.id,
profileKey: `native:${randomUUID()}`,
name: "Native assignment",
defaultAction: "deny",
}).returning();
const slug = `native-${agent!.id.replaceAll("-", "").slice(0, 12)}-${randomUUID().slice(0, 8)}`;
const locker = createDb(tempDb!.connectionString);
let companyHeld!: () => void, lockAgent!: () => void;
const companyLocked = new Promise<void>(resolve => { companyHeld = resolve; });
const agentGate = new Promise<void>(resolve => { lockAgent = resolve; });
// The locker waits longer before it checks for a deadlock, so the gateway
// transaction is always the victim, as on staging.
const lockerDone = locker.transaction(async tx => {
await tx.execute(sql`set local deadlock_timeout = '20s'`);
await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, company!.id)).for("update");
companyHeld();
await agentGate;
await tx.select({ id: agents.id }).from(agents).where(eq(agents.id, agent!.id)).for("update");
});
try {
await companyLocked;
const create = createToolGatewayService(db).createNamedGateway({
companyId: company!.id,
body: {
name: "Native gateway",
slug,
description: "Run-scoped Paperclip Runner MCP gateway.",
profileId: profile!.id,
defaultProfileMode: "gateway_only",
agentId: agent!.id,
contextScopeType: "agent",
contextScopeId: agent!.id,
metadata: { agentId: agent!.id },
},
actor: { agentId: agent!.id },
}).then(value => ({ value }), error => ({ error }));
await vi.waitFor(async () => {
const [waiting] = await locker.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity
where wait_event_type = 'Lock' and query like 'insert into "tool_mcp_gateways"%'`);
expect(waiting?.count).toBe(1);
}, { timeout: 10_000, interval: 50 });
lockAgent();
await lockerDone;
const outcome = await create;
expect(outcome).not.toHaveProperty("error");
expect(outcome).toMatchObject({ value: { slug, agentId: agent!.id, profileId: profile!.id } });
expect(await db.select({ id: toolMcpGateways.id }).from(toolMcpGateways)
.where(eq(toolMcpGateways.slug, slug))).toHaveLength(1);
expect(await db.select({ id: toolProfileBindings.id }).from(toolProfileBindings)
.where(eq(toolProfileBindings.targetId, (outcome as { value: { id: string } }).value.id))).toHaveLength(1);
} finally {
companyHeld();
lockAgent();
await lockerDone.catch(() => undefined);
await locker.$client.end({ timeout: 1 });
}
}, 30_000);
});