From 6bf312270c3ed740d71cc1b5e064530930a6d473 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 20 Nov 2026 22:52:05 -0800 Subject: [PATCH] fix(ci): stabilize database-backed readiness checks (#22476) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - CI checks its recovery workers, attention routes, and external chat admission against real PostgreSQL. > - Large fixtures start from an empty database without statistics for their new rows. > - Slow query plans can exhaust test deadlines before the behavior checks finish. > - Route responses can also leave wakeups writing rows while teardown deletes their agents. > - Setup admission takes an endpoint lock that can deadlock with resource foreign-key checks and agent invalidation. > - This pull request prepares fixture statistics, drains pending work, and uses a compatible endpoint lock. > - The benefit is reliable readiness checks with the same fixtures, assertions, and deadlines. ## Linked Issues or Issue Description Refs: #21941, #21831, #21132. The compiler-memory mitigation in #19383 is already merged and is not repeated here. **What happened?** The recovery census and review-attention equivalence tests repeatedly timed out on unrelated PRs. Recovery-action teardown also failed on an agent-wakeup foreign key, and concurrent first GitHub mentions could deadlock during resource admission. Examples: [census timeout](https://github.com/paperclipai/paperclip/actions/runs/44868742060/job/136738633187), [review timeout](https://github.com/paperclipai/paperclip/actions/runs/44856500775/job/136732143246), [cleanup failure](https://github.com/paperclipai/paperclip/actions/runs/44866635624/job/136731817497), and [chat deadlock](https://github.com/paperclipai/paperclip/actions/runs/44845946341/job/136667344125). **Expected behavior** The complete fixtures retain their regression coverage, pending wakeups finish before deletion, and concurrent chat admission completes with exactly one setup destination enabled. **Steps to reproduce** 1. Run the two timeout suites against a freshly seeded database before its statistics refresh. 2. Complete an issue-update request that schedules an assignee wake, then delete its fixture immediately. 3. Run the new resource-admission regression on the base revision. It forces the endpoint/agent lock order and fails with PostgreSQL SQLSTATE `40P01`. **Paperclip version or commit** Base: `33ffaa5b9a2e0954c2afc063572346cb1be534a7`. ## What Changed - Run `ANALYZE` on the populated fixture tables after seeding the existing census and review-attention populations. No fixture reduction, assertion removal, retry increase, or deadline increase. - Await `drainActiveRunExecutions()` before recovery-action fixture cleanup. This includes pending wake admission after HTTP responses. - Use `NO KEY UPDATE` for setup endpoint admission. It still serializes competing admission transactions and endpoint mutations while allowing resource foreign-key `KEY SHARE` locks. - Add a real PostgreSQL regression using the real `ensureResource` write, with barriers that force concurrent endpoint admission and an agent-locked resource insert. Both transactions must finish and exactly one destination must become enabled. - Document fixture statistics, teardown ownership, and admission locking in `doc/test-infrastructure.md`. ## Verification - The new concurrency regression fails on the original endpoint lock with SQLSTATE `40P01`, then passes with `NO KEY UPDATE`. - Both timeout suites pass with their full populations and original deadlines: 8 tests. - The full recovery-action suite passes: 120 tests. - Existing GitHub first-repository and disabled-repository integration checks pass: 7 tests. - The new PostgreSQL concurrency regression passes: 1 test. - The local baseline timeout suites passed slowly; the review equivalence case took 35.8 seconds before statistics and 0.63 seconds afterward. These are local observations, not a CI speed guarantee. - The first broad typecheck caught an unsupported `Promise.withResolvers` type in the new test. The barriers now use ordinary promises compatible with the repository target. The final `pnpm -r typecheck` passes. - `git diff --check` passes. - `pnpm -r typecheck` and `pnpm build` pass. The full local `pnpm test:run` attempt stopped after all six chat shards: 1,969 tests passed and two Slack tests failed on skill-root discovery. The remaining local groups did not run. Both failures reproduce with the production change reverted to `origin/master`: the resolver selects an unrelated ancestor `skills/` directory without `slack/SKILL.md`, which also prevents the mocked Slack run from starting. This is a confirmed baseline local-layout issue, not a passing full-suite result. The [complete CI run](https://github.com/paperclipai/paperclip/actions/runs/44873350026) passes on `66c146b177b50e052a21921644dfd626c96c010d` on attempt 1. The PR check rollup has 58 successes, three expected skips, and no pending or failing checks. [Greptile reviewed the current head `66c146b`](https://github.com/paperclipai/paperclip/pull/22476#issuecomment-7058185817) at 5/5 with no actionable findings. Review follow-up verification: - Replaced the resource-read stub with the real `createChatInboundAuthority(...).ensureResource` persistence path. The revised regression fails with `FOR UPDATE` and PostgreSQL `40P01`, then passes with `NO KEY UPDATE`. - Scoped `ANALYZE` to the fixture tables and moved the behavior note from `DEVELOPING.md` to `test-infrastructure.md`. - All 9 affected tests pass after these edits; the complete server typecheck also passes. The production lock change is unchanged from the earlier full workspace typecheck and build passes. - The previous CI attempt had two interrupted jobs: [Runner static checks](https://github.com/paperclipai/paperclip/actions/runs/44872465981/job/136749952142) and [serialized server 9/9](https://github.com/paperclipai/paperclip/actions/runs/44872465981/job/136749952782). Both logs record a runner shutdown. The static lane reported 2,371 passing tests before cancellation; the interruption is not counted as a pass. The final head passes the complete fresh CI run linked above, including both lanes. ## Risks Low scope. Production code changes only the endpoint row-lock strength. The regression uses real foreign keys and connector invalidation triggers; the existing integration checks retain first-destination serialization and disabled-destination behavior. No schema, API, or authorization changes. `ANALYZE` updates only test-database statistics. Drain failures still fail teardown. All current-head remote checks now pass, including typecheck, build, tests, and e2e. The baseline local full-suite failure and unrun remaining local groups are recorded above; focused local checks pass. The already merged compiler-input fix does not establish the cause of every past runner shutdown. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh reasoning and shell/code execution. The session does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/test-infrastructure.md | 19 +++++ .../__tests__/issue-recovery-actions.test.ts | 1 + ...sue-recovery-live-run-agent-census.test.ts | 1 + ...tention-classification-equivalence.test.ts | 6 ++ .../inbound/resource-admission.test.ts | 73 +++++++++++++++++++ .../inbound/resource-admission.ts | 2 +- 6 files changed, 101 insertions(+), 1 deletion(-) create mode 100644 server/src/services/chat-channels/inbound/resource-admission.test.ts diff --git a/doc/test-infrastructure.md b/doc/test-infrastructure.md index 60c663fa66f..16b11bb6cbe 100644 --- a/doc/test-infrastructure.md +++ b/doc/test-infrastructure.md @@ -96,6 +96,25 @@ falls back to the fresh path automatically, so the cache can make a suite slower but never fail it or skip it: a migration that cannot apply still fails the suite on the fresh path. +## Seeded database statistics and teardown + +After seeding a large population, run `ANALYZE` on the populated tables before +measuring or exercising it. The empty template has no statistics for those rows, +and waiting for automatic analysis can cause slow query plans and test timeouts. +The census and review-attention equivalence suites retain their full populations +and original deadlines while analyzing their fixture tables before the checks. + +Tests that exercise route-triggered heartbeats must await +`drainActiveRunExecutions()` before deleting fixture rows. A completed HTTP +response can still have wake admission in flight; draining only run executions +would leave that writer racing with deletion. + +Setup destination admission uses `NO KEY UPDATE` on the endpoint. It serializes +competing admissions while permitting `KEY SHARE` from resource foreign-key +checks. The concurrency regression uses the real `ensureResource` write and +connector invalidation triggers, with barriers to force the endpoint/agent lock +order. Restoring `FOR UPDATE` produces PostgreSQL deadlock code `40P01`. + ## Typecheck memory TypeScript 7 is a native binary, so `NODE_OPTIONS=--max-old-space-size` does not diff --git a/server/src/__tests__/issue-recovery-actions.test.ts b/server/src/__tests__/issue-recovery-actions.test.ts index 18dbd995d89..6408a0e33d1 100644 --- a/server/src/__tests__/issue-recovery-actions.test.ts +++ b/server/src/__tests__/issue-recovery-actions.test.ts @@ -189,6 +189,7 @@ describeEmbeddedPostgres("issue recovery actions", () => { }, 30_000); afterEach(async () => { + await heartbeatService(db).drainActiveRunExecutions(); await db.delete(caseHandoffs); await db.delete(cases); await db.delete(documentRevisions); diff --git a/server/src/__tests__/issue-recovery-live-run-agent-census.test.ts b/server/src/__tests__/issue-recovery-live-run-agent-census.test.ts index e727a08eaf6..8da7b884595 100644 --- a/server/src/__tests__/issue-recovery-live-run-agent-census.test.ts +++ b/server/src/__tests__/issue-recovery-live-run-agent-census.test.ts @@ -137,6 +137,7 @@ describeIssueRecovery("stranded-task checks beside one long live run (census)", async function boot() { const f = await bounded(seed(), () => "the fixture to be seeded"); + await state.db.execute(sql`ANALYZE agents, issues, heartbeat_runs`); const events = record(); useCensusClock(); await bounded(state.start({}), () => "the recovery workers to start"); diff --git a/server/src/__tests__/review-attention-classification-equivalence.test.ts b/server/src/__tests__/review-attention-classification-equivalence.test.ts index 305e88d30dd..f38c8a913ca 100644 --- a/server/src/__tests__/review-attention-classification-equivalence.test.ts +++ b/server/src/__tests__/review-attention-classification-equivalence.test.ts @@ -1,5 +1,6 @@ import express from "express"; import request from "supertest"; +import { sql } from "drizzle-orm"; import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { createDb, inboxDismissals } from "@paperclipai/db"; import type { AttentionFeed, AttentionSummary, Issue } from "@paperclipai/shared"; @@ -76,6 +77,11 @@ describeEmbeddedPostgres("review attention through the routes: indexed classific // Two companies of the same shapes. Each request reads one of them. fixture = await seedReviewAttentionFixture(db, { tasks: 640, agents: 24, prefix: "RAE" }); other = await seedReviewAttentionFixture(db, { tasks: 320, agents: 12, prefix: "OTH" }); + await db.execute(sql`ANALYZE agents, issues, heartbeat_runs, issue_relations, + issue_thread_interactions, issue_recovery_actions, agent_wakeup_requests, + approvals, issue_approvals, chat_endpoints, chat_endpoint_resources, + chat_conversations, chat_github_reviews, chat_message_links, chat_deliveries, + chat_actions, run_identity_contexts, status_decisions`); }, 120_000); beforeEach(() => { diff --git a/server/src/services/chat-channels/inbound/resource-admission.test.ts b/server/src/services/chat-channels/inbound/resource-admission.test.ts new file mode 100644 index 00000000000..e4656676573 --- /dev/null +++ b/server/src/services/chat-channels/inbound/resource-admission.test.ts @@ -0,0 +1,73 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, chatEndpointResources, chatEndpoints, companies, createDb, toolApplications, toolConnections } from "@paperclipai/db"; +import type { Thread } from "chat"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js"; +import { createMessageResourceAdmission } from "./resource-admission.js"; +import { createChatInboundAuthority } from "./authorization.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const describePostgres = support.supported ? describe : describe.skip; + +describePostgres("setup destination admission with concurrent runtime invalidation", () => { + let database: Awaited>; + let db: ReturnType; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("chat-resource-admission-"); + db = createDb(database.connectionString); + }, 90_000); + + afterAll(async () => { + await db?.$client.end({ timeout: 1 }); + await database?.cleanup(); + }); + + it("admits a destination while an agent-locked resource insert checks its endpoint foreign key", async () => { + const [company] = await db.insert(companies).values({ name: "Admission" }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company!.id, name: "Worker", role: "engineer", + status: "idle", adapterType: "process" }).returning(); + const [application] = await db.insert(toolApplications).values({ companyId: company!.id, name: "GitHub", type: "chat" }).returning(); + const [connection] = await db.insert(toolConnections).values({ companyId: company!.id, applicationId: application!.id, + name: "GitHub", uid: randomUUID(), transport: "chat_sdk", connectionPurpose: "channel", status: "active" }).returning(); + const [endpoint] = await db.insert(chatEndpoints).values({ companyId: company!.id, connectionId: connection!.id, + provider: "github", publicId: randomUUID(), assignedAgentId: agent!.id, status: "verifying" }).returning(); + const [resource] = await db.insert(chatEndpointResources).values({ companyId: company!.id, endpointId: endpoint!.id, + type: "repository", providerResourceId: "example/first", label: "First", enabled: false }).returning(); + let releaseEndpoint!: () => void; + let releaseAgent!: () => void; + const endpointLocked = new Promise(resolve => { releaseEndpoint = resolve; }); + const agentLocked = new Promise(resolve => { releaseAgent = resolve; }); + const authority = createChatInboundAuthority(db, { + voiceStore: { authorizePrincipal: async () => { throw new Error("Unexpected voice authorization"); } }, + runtimeCallbackEndpoint: async () => { throw new Error("Unexpected runtime callback"); }, + }); + const admission = createMessageResourceAdmission(db, { + ensureResource: async (...args) => { + releaseEndpoint(); + await agentLocked; + return authority.ensureResource(...args); + }, + }); + + const thread = { isDM: false, id: "github:example/first:1", channelId: "example/first", + channel: { name: "example/first" } } as Thread; + const admitting = admission.admitMessageResource({ endpoint: endpoint!, thread, + addressed: true, suppressSetupDestinationActivation: false, surfaceKind: "native_thread" }); + const inserting = (async () => { + await endpointLocked; + await db.transaction(async tx => { + await tx.select({ id: agents.id }).from(agents).where(eq(agents.id, agent!.id)).for("no key update"); + releaseAgent(); + await tx.insert(chatEndpointResources).values({ companyId: company!.id, endpointId: endpoint!.id, + type: "repository", providerResourceId: "example/second", label: "Second", enabled: false }); + }); + })(); + const results = await Promise.allSettled([admitting, inserting]); + for (const result of results) if (result.status === "rejected") throw result.reason; + const resources = await db.select().from(chatEndpointResources).where(eq(chatEndpointResources.endpointId, endpoint!.id)); + expect(resources).toHaveLength(2); + expect(resources.filter(row => row.enabled).map(row => row.id)).toEqual([resource!.id]); + }); +}); diff --git a/server/src/services/chat-channels/inbound/resource-admission.ts b/server/src/services/chat-channels/inbound/resource-admission.ts index e83ba1c9749..df67f932e8f 100644 --- a/server/src/services/chat-channels/inbound/resource-admission.ts +++ b/server/src/services/chat-channels/inbound/resource-admission.ts @@ -56,7 +56,7 @@ export function createMessageResourceAdmission(db: Db, options: { ensureResource .select({ status: chatEndpoints.status }) .from(chatEndpoints) .where(eq(chatEndpoints.id, endpoint.id)) - .for("update") + .for("no key update") .then((rows) => rows[0] ?? null); if (!currentEndpoint) throw notFound("Chat endpoint not found"); const enabledResourceCount = await tx