fix(ci): stabilize database-backed readiness checks (#22476)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - CI checks its recovery workers, attention routes, and external chat
admission against real PostgreSQL.
> - Large fixtures start from an empty database without statistics for
their new rows.
> - Slow query plans can exhaust test deadlines before the behavior
checks finish.
> - Route responses can also leave wakeups writing rows while teardown
deletes their agents.
> - Setup admission takes an endpoint lock that can deadlock with
resource foreign-key checks and agent invalidation.
> - This pull request prepares fixture statistics, drains pending work,
and uses a compatible endpoint lock.
> - The benefit is reliable readiness checks with the same fixtures,
assertions, and deadlines.

## Linked Issues or Issue Description

Refs: #21941, #21831, #21132. The compiler-memory mitigation in #19383
is already merged and is not repeated here.

**What happened?**

The recovery census and review-attention equivalence tests repeatedly
timed out on unrelated PRs. Recovery-action teardown also failed on an
agent-wakeup foreign key, and concurrent first GitHub mentions could
deadlock during resource admission.

Examples: [census
timeout](https://github.com/paperclipai/paperclip/actions/runs/44868742060/job/136738633187),
[review
timeout](https://github.com/paperclipai/paperclip/actions/runs/44856500775/job/136732143246),
[cleanup
failure](https://github.com/paperclipai/paperclip/actions/runs/44866635624/job/136731817497),
and [chat
deadlock](https://github.com/paperclipai/paperclip/actions/runs/44845946341/job/136667344125).

**Expected behavior**

The complete fixtures retain their regression coverage, pending wakeups
finish before deletion, and concurrent chat admission completes with
exactly one setup destination enabled.

**Steps to reproduce**

1. Run the two timeout suites against a freshly seeded database before
its statistics refresh.
2. Complete an issue-update request that schedules an assignee wake,
then delete its fixture immediately.
3. Run the new resource-admission regression on the base revision. It
forces the endpoint/agent lock order and fails with PostgreSQL SQLSTATE
`40P01`.

**Paperclip version or commit**

Base: `33ffaa5b9a2e0954c2afc063572346cb1be534a7`.

## What Changed

- Run `ANALYZE` on the populated fixture tables after seeding the
existing census and review-attention populations. No fixture reduction,
assertion removal, retry increase, or deadline increase.
- Await `drainActiveRunExecutions()` before recovery-action fixture
cleanup. This includes pending wake admission after HTTP responses.
- Use `NO KEY UPDATE` for setup endpoint admission. It still serializes
competing admission transactions and endpoint mutations while allowing
resource foreign-key `KEY SHARE` locks.
- Add a real PostgreSQL regression using the real `ensureResource`
write, with barriers that force concurrent endpoint admission and an
agent-locked resource insert. Both transactions must finish and exactly
one destination must become enabled.
- Document fixture statistics, teardown ownership, and admission locking
in `doc/test-infrastructure.md`.

## Verification

- The new concurrency regression fails on the original endpoint lock
with SQLSTATE `40P01`, then passes with `NO KEY UPDATE`.
- Both timeout suites pass with their full populations and original
deadlines: 8 tests.
- The full recovery-action suite passes: 120 tests.
- Existing GitHub first-repository and disabled-repository integration
checks pass: 7 tests.
- The new PostgreSQL concurrency regression passes: 1 test.
- The local baseline timeout suites passed slowly; the review
equivalence case took 35.8 seconds before statistics and 0.63 seconds
afterward. These are local observations, not a CI speed guarantee.
- The first broad typecheck caught an unsupported
`Promise.withResolvers` type in the new test. The barriers now use
ordinary promises compatible with the repository target. The final `pnpm
-r typecheck` passes.
- `git diff --check` passes.
- `pnpm -r typecheck` and `pnpm build` pass. The full local `pnpm
test:run` attempt stopped after all six chat shards: 1,969 tests passed
and two Slack tests failed on skill-root discovery. The remaining local
groups did not run. Both failures reproduce with the production change
reverted to `origin/master`: the resolver selects an unrelated ancestor
`skills/` directory without `slack/SKILL.md`, which also prevents the
mocked Slack run from starting. This is a confirmed baseline
local-layout issue, not a passing full-suite result. The [complete CI
run](https://github.com/paperclipai/paperclip/actions/runs/44873350026)
passes on `66c146b177b50e052a21921644dfd626c96c010d` on attempt 1. The
PR check rollup has 58 successes, three expected skips, and no pending
or failing checks. [Greptile reviewed the current head
`66c146b`](https://github.com/paperclipai/paperclip/pull/22476#issuecomment-7058185817)
at 5/5 with no actionable findings.

Review follow-up verification:

- Replaced the resource-read stub with the real
`createChatInboundAuthority(...).ensureResource` persistence path. The
revised regression fails with `FOR UPDATE` and PostgreSQL `40P01`, then
passes with `NO KEY UPDATE`.
- Scoped `ANALYZE` to the fixture tables and moved the behavior note
from `DEVELOPING.md` to `test-infrastructure.md`.
- All 9 affected tests pass after these edits; the complete server
typecheck also passes. The production lock change is unchanged from the
earlier full workspace typecheck and build passes.
- The previous CI attempt had two interrupted jobs: [Runner static
checks](https://github.com/paperclipai/paperclip/actions/runs/44872465981/job/136749952142)
and [serialized server
9/9](https://github.com/paperclipai/paperclip/actions/runs/44872465981/job/136749952782).
Both logs record a runner shutdown. The static lane reported 2,371
passing tests before cancellation; the interruption is not counted as a
pass. The final head passes the complete fresh CI run linked above,
including both lanes.

## Risks

Low scope. Production code changes only the endpoint row-lock strength.
The regression uses real foreign keys and connector invalidation
triggers; the existing integration checks retain first-destination
serialization and disabled-destination behavior. No schema, API, or
authorization changes. `ANALYZE` updates only test-database statistics.
Drain failures still fail teardown.

All current-head remote checks now pass, including typecheck, build,
tests, and e2e. The baseline local full-suite failure and unrun
remaining local groups are recorded above; focused local checks pass.
The already merged compiler-input fix does not establish the cause of
every past runner shutdown.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh
reasoning and shell/code execution. The session does not expose its
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed 2026-11-20 22:52:05 -08:00
1 parent 33ffaa5b9a
commit 6bf312270c
6 files changed
+101 -1

No files matched your search

+19
View File
@@ -96,6 +96,25 @@ falls back to the fresh path automatically, so the cache can make a suite
slower but never fail it or skip it: a migration that cannot apply still fails
the suite on the fresh path.
## Seeded database statistics and teardown
After seeding a large population, run `ANALYZE` on the populated tables before
measuring or exercising it. The empty template has no statistics for those rows,
and waiting for automatic analysis can cause slow query plans and test timeouts.
The census and review-attention equivalence suites retain their full populations
and original deadlines while analyzing their fixture tables before the checks.
Tests that exercise route-triggered heartbeats must await
`drainActiveRunExecutions()` before deleting fixture rows. A completed HTTP
response can still have wake admission in flight; draining only run executions
would leave that writer racing with deletion.
Setup destination admission uses `NO KEY UPDATE` on the endpoint. It serializes
competing admissions while permitting `KEY SHARE` from resource foreign-key
checks. The concurrency regression uses the real `ensureResource` write and
connector invalidation triggers, with barriers to force the endpoint/agent lock
order. Restoring `FOR UPDATE` produces PostgreSQL deadlock code `40P01`.
## Typecheck memory
TypeScript 7 is a native binary, so `NODE_OPTIONS=--max-old-space-size` does not
@@ -189,6 +189,7 @@ describeEmbeddedPostgres("issue recovery actions", () => {
}, 30_000);
afterEach(async () => {
await heartbeatService(db).drainActiveRunExecutions();
await db.delete(caseHandoffs);
await db.delete(cases);
await db.delete(documentRevisions);
@@ -137,6 +137,7 @@ describeIssueRecovery("stranded-task checks beside one long live run (census)",
async function boot() {
const f = await bounded(seed(), () => "the fixture to be seeded");
await state.db.execute(sql`ANALYZE agents, issues, heartbeat_runs`);
const events = record();
useCensusClock();
await bounded(state.start({}), () => "the recovery workers to start");
@@ -1,5 +1,6 @@
import express from "express";
import request from "supertest";
import { sql } from "drizzle-orm";
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { createDb, inboxDismissals } from "@paperclipai/db";
import type { AttentionFeed, AttentionSummary, Issue } from "@paperclipai/shared";
@@ -76,6 +77,11 @@ describeEmbeddedPostgres("review attention through the routes: indexed classific
// Two companies of the same shapes. Each request reads one of them.
fixture = await seedReviewAttentionFixture(db, { tasks: 640, agents: 24, prefix: "RAE" });
other = await seedReviewAttentionFixture(db, { tasks: 320, agents: 12, prefix: "OTH" });
await db.execute(sql`ANALYZE agents, issues, heartbeat_runs, issue_relations,
issue_thread_interactions, issue_recovery_actions, agent_wakeup_requests,
approvals, issue_approvals, chat_endpoints, chat_endpoint_resources,
chat_conversations, chat_github_reviews, chat_message_links, chat_deliveries,
chat_actions, run_identity_contexts, status_decisions`);
}, 120_000);
beforeEach(() => {
@@ -0,0 +1,73 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { agents, chatEndpointResources, chatEndpoints, companies, createDb, toolApplications, toolConnections } from "@paperclipai/db";
import type { Thread } from "chat";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js";
import { createMessageResourceAdmission } from "./resource-admission.js";
import { createChatInboundAuthority } from "./authorization.js";
const support = await getEmbeddedPostgresTestSupport();
const describePostgres = support.supported ? describe : describe.skip;
describePostgres("setup destination admission with concurrent runtime invalidation", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("chat-resource-admission-");
db = createDb(database.connectionString);
}, 90_000);
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await database?.cleanup();
});
it("admits a destination while an agent-locked resource insert checks its endpoint foreign key", async () => {
const [company] = await db.insert(companies).values({ name: "Admission" }).returning();
const [agent] = await db.insert(agents).values({ companyId: company!.id, name: "Worker", role: "engineer",
status: "idle", adapterType: "process" }).returning();
const [application] = await db.insert(toolApplications).values({ companyId: company!.id, name: "GitHub", type: "chat" }).returning();
const [connection] = await db.insert(toolConnections).values({ companyId: company!.id, applicationId: application!.id,
name: "GitHub", uid: randomUUID(), transport: "chat_sdk", connectionPurpose: "channel", status: "active" }).returning();
const [endpoint] = await db.insert(chatEndpoints).values({ companyId: company!.id, connectionId: connection!.id,
provider: "github", publicId: randomUUID(), assignedAgentId: agent!.id, status: "verifying" }).returning();
const [resource] = await db.insert(chatEndpointResources).values({ companyId: company!.id, endpointId: endpoint!.id,
type: "repository", providerResourceId: "example/first", label: "First", enabled: false }).returning();
let releaseEndpoint!: () => void;
let releaseAgent!: () => void;
const endpointLocked = new Promise<void>(resolve => { releaseEndpoint = resolve; });
const agentLocked = new Promise<void>(resolve => { releaseAgent = resolve; });
const authority = createChatInboundAuthority(db, {
voiceStore: { authorizePrincipal: async () => { throw new Error("Unexpected voice authorization"); } },
runtimeCallbackEndpoint: async () => { throw new Error("Unexpected runtime callback"); },
});
const admission = createMessageResourceAdmission(db, {
ensureResource: async (...args) => {
releaseEndpoint();
await agentLocked;
return authority.ensureResource(...args);
},
});
const thread = { isDM: false, id: "github:example/first:1", channelId: "example/first",
channel: { name: "example/first" } } as Thread;
const admitting = admission.admitMessageResource({ endpoint: endpoint!, thread,
addressed: true, suppressSetupDestinationActivation: false, surfaceKind: "native_thread" });
const inserting = (async () => {
await endpointLocked;
await db.transaction(async tx => {
await tx.select({ id: agents.id }).from(agents).where(eq(agents.id, agent!.id)).for("no key update");
releaseAgent();
await tx.insert(chatEndpointResources).values({ companyId: company!.id, endpointId: endpoint!.id,
type: "repository", providerResourceId: "example/second", label: "Second", enabled: false });
});
})();
const results = await Promise.allSettled([admitting, inserting]);
for (const result of results) if (result.status === "rejected") throw result.reason;
const resources = await db.select().from(chatEndpointResources).where(eq(chatEndpointResources.endpointId, endpoint!.id));
expect(resources).toHaveLength(2);
expect(resources.filter(row => row.enabled).map(row => row.id)).toEqual([resource!.id]);
});
});
@@ -56,7 +56,7 @@ export function createMessageResourceAdmission(db: Db, options: { ensureResource
.select({ status: chatEndpoints.status })
.from(chatEndpoints)
.where(eq(chatEndpoints.id, endpoint.id))
.for("update")
.for("no key update")
.then((rows) => rows[0] ?? null);
if (!currentEndpoint) throw notFound("Chat endpoint not found");
const enabledResourceCount = await tx