From a792409d07e984e7bf8382e295bdaebd2aa18dda Mon Sep 17 00:00:00 2001 From: paperclipbugbot Date: Fri, 20 Nov 2026 05:47:47 -0800 Subject: [PATCH] fix(tools): retry a named MCP gateway create that loses a deadlock (#22408) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs prepare a run-scoped MCP gateway for their agent before the provider starts. `createNamedGateway` writes the `tool_mcp_gateways` row and its profile binding in one transaction. > - That transaction locks the agent row first (`FOR NO KEY UPDATE`). Then its gateway insert takes a key-share lock on the company and profile rows for the foreign keys. > - Another writer can lock the company row first and then the agent row. The two transactions then wait for each other. Postgres detects a deadlock and aborts the gateway create (`40P01`). > - The run then fails before the provider starts. The task stops on `native_continuation_requires_reconciliation`, so a person must reconcile a run that did no work. > - This pull request runs the gateway create transaction through the shared bounded retry that #22329 and #20971 use for the same type of lock cycle. > - The benefit is that a short lock conflict no longer fails the run preparation. ## Linked Issues or Issue Description **What happened?** A native run failed while it prepared its MCP gateway. The error was `Failed query: insert into "tool_mcp_gateways" (...)`. The Postgres cause was `deadlock detected` (`40P01`). The task then stopped and needed reconciliation. **Expected behavior** A transient lock conflict does not fail the run preparation. The gateway create writes the gateway and its profile binding, and the run continues. **Steps to reproduce** 1. Transaction A: lock the company row `FOR UPDATE`. 2. Call `createNamedGateway` for an agent in that company. It locks the agent row and then waits for the company key-share lock. 3. Transaction A: lock the agent row `FOR UPDATE`. 4. Postgres reports a deadlock. Before this change the gateway create is the victim and `createNamedGateway` rejects with the database error. **Additional context** The new test reproduces this with two real transactions on embedded Postgres. Transaction A gets a longer `deadlock_timeout`, so the gateway create is always the victim, as in the observed failure. The failing statement in the test is the same insert as in the observed error. Refs #22329 Refs #20971 ## What Changed - `server/src/services/tool-gateway.ts`: `createNamedGateway` runs its transaction through `retryTransientLockContention`. The helper retries only lock timeout, deadlock and serialization failure (`55P03`, `40P01`, `40001`): 5 attempts, with 100 ms × attempt backoff. Other errors fail as before. - `server/src/__tests__/tool-gateway-named-gateway-deadlock.test.ts`: new test with a real two-transaction deadlock between a company-then-agent locker and a real `createNamedGateway` call. The test checks that exactly one gateway row and one profile binding exist after the retry. ## Verification - Failing before the fix: `cd server && npx vitest run src/__tests__/tool-gateway-named-gateway-deadlock.test.ts` failed. `createNamedGateway` rejected with `Failed query: insert into "tool_mcp_gateways" ...`, cause `deadlock detected`, code `40P01`. - Passing after the fix: the same test passes. I ran it 3 times. - `heartbeat-runtime-mcp-servers.test.ts`, `agent-runtime-assignments.test.ts`, `agent-interaction-permissions.test.ts` and `tool-gateway-service.test.ts` pass (4 files, 238 tests). - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. Only the gateway create transaction retries, and only on the three contention codes. - Postgres rolls back the complete aborted transaction. A retry therefore cannot write a duplicate gateway or profile binding. - The interaction-permission check runs again in each attempt. A retry cannot skip it. - If a concurrent create with the same slug commits first, the retry gets a unique violation. That error is not retried. The existing native caller then reads the committed gateway by slug, as before. - The worst added delay is about 1 s of backoff across 5 attempts, and only when the lock conflict continues. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test execution on embedded Postgres). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 --- ...ool-gateway-named-gateway-deadlock.test.ts | 103 ++++++++++++++++++ server/src/services/tool-gateway.ts | 8 +- 2 files changed, 109 insertions(+), 2 deletions(-) create mode 100644 server/src/__tests__/tool-gateway-named-gateway-deadlock.test.ts diff --git a/server/src/__tests__/tool-gateway-named-gateway-deadlock.test.ts b/server/src/__tests__/tool-gateway-named-gateway-deadlock.test.ts new file mode 100644 index 00000000000..70391b5322b --- /dev/null +++ b/server/src/__tests__/tool-gateway-named-gateway-deadlock.test.ts @@ -0,0 +1,103 @@ +import { randomUUID } from "node:crypto"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, createDb, toolMcpGateways, toolProfileBindings, toolProfiles } from "@paperclipai/db"; +import { + getEmbeddedPostgresTestSupport, + startEmbeddedPostgresTestDatabase, +} from "./helpers/embedded-postgres.js"; +import { createToolGatewayService } from "../services/tool-gateway.js"; + +const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; + +// A native run creates its agent's MCP gateway while it prepares. The create +// transaction locks the agent row, then its gateway insert takes key-share on +// the company row for the foreign key. A writer that locks the company row and +// then the agent row closes the cycle. Postgres aborts the gateway insert with +// 40P01, and the run fails before the provider starts. +describeEmbeddedPostgres("named MCP gateway creation under a deadlock", () => { + let db!: ReturnType; + let tempDb: Awaited> | null = null; + + beforeAll(async () => { + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-named-gateway-deadlock-"); + db = createDb(tempDb.connectionString); + }, 20_000); + + afterAll(async () => { + await tempDb?.cleanup(); + }); + + it("creates the gateway after it loses a deadlock to a company-then-agent locker", async () => { + const [company] = await db.insert(companies).values({ + name: `Gateway deadlock ${randomUUID()}`, + issuePrefix: `GD${randomUUID().slice(0, 5).toUpperCase()}`, + }).returning(); + const [agent] = await db.insert(agents).values({ + companyId: company!.id, + name: "Native agent", + role: "engineer", + adapterType: "paperclip_runner", + adapterConfig: {}, + }).returning(); + const [profile] = await db.insert(toolProfiles).values({ + companyId: company!.id, + profileKey: `native:${randomUUID()}`, + name: "Native assignment", + defaultAction: "deny", + }).returning(); + const slug = `native-${agent!.id.replaceAll("-", "").slice(0, 12)}-${randomUUID().slice(0, 8)}`; + + const locker = createDb(tempDb!.connectionString); + let companyHeld!: () => void, lockAgent!: () => void; + const companyLocked = new Promise(resolve => { companyHeld = resolve; }); + const agentGate = new Promise(resolve => { lockAgent = resolve; }); + // The locker waits longer before it checks for a deadlock, so the gateway + // transaction is always the victim, as on staging. + const lockerDone = locker.transaction(async tx => { + await tx.execute(sql`set local deadlock_timeout = '20s'`); + await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, company!.id)).for("update"); + companyHeld(); + await agentGate; + await tx.select({ id: agents.id }).from(agents).where(eq(agents.id, agent!.id)).for("update"); + }); + try { + await companyLocked; + const create = createToolGatewayService(db).createNamedGateway({ + companyId: company!.id, + body: { + name: "Native gateway", + slug, + description: "Run-scoped Paperclip Runner MCP gateway.", + profileId: profile!.id, + defaultProfileMode: "gateway_only", + agentId: agent!.id, + contextScopeType: "agent", + contextScopeId: agent!.id, + metadata: { agentId: agent!.id }, + }, + actor: { agentId: agent!.id }, + }).then(value => ({ value }), error => ({ error })); + await vi.waitFor(async () => { + const [waiting] = await locker.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity + where wait_event_type = 'Lock' and query like 'insert into "tool_mcp_gateways"%'`); + expect(waiting?.count).toBe(1); + }, { timeout: 10_000, interval: 50 }); + lockAgent(); + await lockerDone; + const outcome = await create; + expect(outcome).not.toHaveProperty("error"); + expect(outcome).toMatchObject({ value: { slug, agentId: agent!.id, profileId: profile!.id } }); + expect(await db.select({ id: toolMcpGateways.id }).from(toolMcpGateways) + .where(eq(toolMcpGateways.slug, slug))).toHaveLength(1); + expect(await db.select({ id: toolProfileBindings.id }).from(toolProfileBindings) + .where(eq(toolProfileBindings.targetId, (outcome as { value: { id: string } }).value.id))).toHaveLength(1); + } finally { + companyHeld(); + lockAgent(); + await lockerDone.catch(() => undefined); + await locker.$client.end({ timeout: 1 }); + } + }, 30_000); +}); diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index 8660892d5df..ee08fcb44f1 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -11429,7 +11429,11 @@ export function createToolGatewayService( input.body.displaySlug ?? input.body.slug ?? slugSegment(input.body.name, "gateway"); - const gateway = await db.transaction(async tx => { + // The create locks the agent row, then its insert takes key-share on the + // company and profile rows. A writer that locks those in the other order + // can make this transaction the deadlock victim. The rolled-back attempt + // wrote nothing, so a bounded retry of the whole transaction is safe. + const gateway = await retryTransientLockContention(() => db.transaction(async tx => { await assertAgentToolAssignmentInteractions(tx as unknown as Db, input.managementContext?.actor.type === "agent" ? input.managementContext.actor.agentId : input.actor?.agentId, input.companyId, input.body.agentId ? [input.body.agentId] : []); @@ -11482,7 +11486,7 @@ export function createToolGatewayService( }) .onConflictDoNothing(); return gateway; - }); + })); await writeAudit({ session: { id: `gateway:${gateway.id}`,