mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
fix(ci): stabilize database-backed readiness checks (#22476)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - CI checks its recovery workers, attention routes, and external chat admission against real PostgreSQL. > - Large fixtures start from an empty database without statistics for their new rows. > - Slow query plans can exhaust test deadlines before the behavior checks finish. > - Route responses can also leave wakeups writing rows while teardown deletes their agents. > - Setup admission takes an endpoint lock that can deadlock with resource foreign-key checks and agent invalidation. > - This pull request prepares fixture statistics, drains pending work, and uses a compatible endpoint lock. > - The benefit is reliable readiness checks with the same fixtures, assertions, and deadlines. ## Linked Issues or Issue Description Refs: #21941, #21831, #21132. The compiler-memory mitigation in #19383 is already merged and is not repeated here. **What happened?** The recovery census and review-attention equivalence tests repeatedly timed out on unrelated PRs. Recovery-action teardown also failed on an agent-wakeup foreign key, and concurrent first GitHub mentions could deadlock during resource admission. Examples: [census timeout](https://github.com/paperclipai/paperclip/actions/runs/44868742060/job/136738633187), [review timeout](https://github.com/paperclipai/paperclip/actions/runs/44856500775/job/136732143246), [cleanup failure](https://github.com/paperclipai/paperclip/actions/runs/44866635624/job/136731817497), and [chat deadlock](https://github.com/paperclipai/paperclip/actions/runs/44845946341/job/136667344125). **Expected behavior** The complete fixtures retain their regression coverage, pending wakeups finish before deletion, and concurrent chat admission completes with exactly one setup destination enabled. **Steps to reproduce** 1. Run the two timeout suites against a freshly seeded database before its statistics refresh. 2. Complete an issue-update request that schedules an assignee wake, then delete its fixture immediately. 3. Run the new resource-admission regression on the base revision. It forces the endpoint/agent lock order and fails with PostgreSQL SQLSTATE `40P01`. **Paperclip version or commit** Base: `33ffaa5b9a2e0954c2afc063572346cb1be534a7`. ## What Changed - Run `ANALYZE` on the populated fixture tables after seeding the existing census and review-attention populations. No fixture reduction, assertion removal, retry increase, or deadline increase. - Await `drainActiveRunExecutions()` before recovery-action fixture cleanup. This includes pending wake admission after HTTP responses. - Use `NO KEY UPDATE` for setup endpoint admission. It still serializes competing admission transactions and endpoint mutations while allowing resource foreign-key `KEY SHARE` locks. - Add a real PostgreSQL regression using the real `ensureResource` write, with barriers that force concurrent endpoint admission and an agent-locked resource insert. Both transactions must finish and exactly one destination must become enabled. - Document fixture statistics, teardown ownership, and admission locking in `doc/test-infrastructure.md`. ## Verification - The new concurrency regression fails on the original endpoint lock with SQLSTATE `40P01`, then passes with `NO KEY UPDATE`. - Both timeout suites pass with their full populations and original deadlines: 8 tests. - The full recovery-action suite passes: 120 tests. - Existing GitHub first-repository and disabled-repository integration checks pass: 7 tests. - The new PostgreSQL concurrency regression passes: 1 test. - The local baseline timeout suites passed slowly; the review equivalence case took 35.8 seconds before statistics and 0.63 seconds afterward. These are local observations, not a CI speed guarantee. - The first broad typecheck caught an unsupported `Promise.withResolvers` type in the new test. The barriers now use ordinary promises compatible with the repository target. The final `pnpm -r typecheck` passes. - `git diff --check` passes. - `pnpm -r typecheck` and `pnpm build` pass. The full local `pnpm test:run` attempt stopped after all six chat shards: 1,969 tests passed and two Slack tests failed on skill-root discovery. The remaining local groups did not run. Both failures reproduce with the production change reverted to `origin/master`: the resolver selects an unrelated ancestor `skills/` directory without `slack/SKILL.md`, which also prevents the mocked Slack run from starting. This is a confirmed baseline local-layout issue, not a passing full-suite result. The [complete CI run](https://github.com/paperclipai/paperclip/actions/runs/44873350026) passes on `66c146b177b50e052a21921644dfd626c96c010d` on attempt 1. The PR check rollup has 58 successes, three expected skips, and no pending or failing checks. [Greptile reviewed the current head `66c146b`](https://github.com/paperclipai/paperclip/pull/22476#issuecomment-7058185817) at 5/5 with no actionable findings. Review follow-up verification: - Replaced the resource-read stub with the real `createChatInboundAuthority(...).ensureResource` persistence path. The revised regression fails with `FOR UPDATE` and PostgreSQL `40P01`, then passes with `NO KEY UPDATE`. - Scoped `ANALYZE` to the fixture tables and moved the behavior note from `DEVELOPING.md` to `test-infrastructure.md`. - All 9 affected tests pass after these edits; the complete server typecheck also passes. The production lock change is unchanged from the earlier full workspace typecheck and build passes. - The previous CI attempt had two interrupted jobs: [Runner static checks](https://github.com/paperclipai/paperclip/actions/runs/44872465981/job/136749952142) and [serialized server 9/9](https://github.com/paperclipai/paperclip/actions/runs/44872465981/job/136749952782). Both logs record a runner shutdown. The static lane reported 2,371 passing tests before cancellation; the interruption is not counted as a pass. The final head passes the complete fresh CI run linked above, including both lanes. ## Risks Low scope. Production code changes only the endpoint row-lock strength. The regression uses real foreign keys and connector invalidation triggers; the existing integration checks retain first-destination serialization and disabled-destination behavior. No schema, API, or authorization changes. `ANALYZE` updates only test-database statistics. Drain failures still fail teardown. All current-head remote checks now pass, including typecheck, build, tests, and e2e. The baseline local full-suite failure and unrun remaining local groups are recorded above; focused local checks pass. The already merged compiler-input fix does not establish the cause of every past runner shutdown. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh reasoning and shell/code execution. The session does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
33ffaa5b9a
commit
6bf312270c
6 files changed
+101
-1
No files matched your search
@@ -96,6 +96,25 @@ falls back to the fresh path automatically, so the cache can make a suite
|
||||
slower but never fail it or skip it: a migration that cannot apply still fails
|
||||
the suite on the fresh path.
|
||||
|
||||
## Seeded database statistics and teardown
|
||||
|
||||
After seeding a large population, run `ANALYZE` on the populated tables before
|
||||
measuring or exercising it. The empty template has no statistics for those rows,
|
||||
and waiting for automatic analysis can cause slow query plans and test timeouts.
|
||||
The census and review-attention equivalence suites retain their full populations
|
||||
and original deadlines while analyzing their fixture tables before the checks.
|
||||
|
||||
Tests that exercise route-triggered heartbeats must await
|
||||
`drainActiveRunExecutions()` before deleting fixture rows. A completed HTTP
|
||||
response can still have wake admission in flight; draining only run executions
|
||||
would leave that writer racing with deletion.
|
||||
|
||||
Setup destination admission uses `NO KEY UPDATE` on the endpoint. It serializes
|
||||
competing admissions while permitting `KEY SHARE` from resource foreign-key
|
||||
checks. The concurrency regression uses the real `ensureResource` write and
|
||||
connector invalidation triggers, with barriers to force the endpoint/agent lock
|
||||
order. Restoring `FOR UPDATE` produces PostgreSQL deadlock code `40P01`.
|
||||
|
||||
## Typecheck memory
|
||||
|
||||
TypeScript 7 is a native binary, so `NODE_OPTIONS=--max-old-space-size` does not
|
||||
|
||||
@@ -189,6 +189,7 @@ describeEmbeddedPostgres("issue recovery actions", () => {
|
||||
}, 30_000);
|
||||
|
||||
afterEach(async () => {
|
||||
await heartbeatService(db).drainActiveRunExecutions();
|
||||
await db.delete(caseHandoffs);
|
||||
await db.delete(cases);
|
||||
await db.delete(documentRevisions);
|
||||
|
||||
@@ -137,6 +137,7 @@ describeIssueRecovery("stranded-task checks beside one long live run (census)",
|
||||
|
||||
async function boot() {
|
||||
const f = await bounded(seed(), () => "the fixture to be seeded");
|
||||
await state.db.execute(sql`ANALYZE agents, issues, heartbeat_runs`);
|
||||
const events = record();
|
||||
useCensusClock();
|
||||
await bounded(state.start({}), () => "the recovery workers to start");
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { createDb, inboxDismissals } from "@paperclipai/db";
|
||||
import type { AttentionFeed, AttentionSummary, Issue } from "@paperclipai/shared";
|
||||
@@ -76,6 +77,11 @@ describeEmbeddedPostgres("review attention through the routes: indexed classific
|
||||
// Two companies of the same shapes. Each request reads one of them.
|
||||
fixture = await seedReviewAttentionFixture(db, { tasks: 640, agents: 24, prefix: "RAE" });
|
||||
other = await seedReviewAttentionFixture(db, { tasks: 320, agents: 12, prefix: "OTH" });
|
||||
await db.execute(sql`ANALYZE agents, issues, heartbeat_runs, issue_relations,
|
||||
issue_thread_interactions, issue_recovery_actions, agent_wakeup_requests,
|
||||
approvals, issue_approvals, chat_endpoints, chat_endpoint_resources,
|
||||
chat_conversations, chat_github_reviews, chat_message_links, chat_deliveries,
|
||||
chat_actions, run_identity_contexts, status_decisions`);
|
||||
}, 120_000);
|
||||
|
||||
beforeEach(() => {
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { agents, chatEndpointResources, chatEndpoints, companies, createDb, toolApplications, toolConnections } from "@paperclipai/db";
|
||||
import type { Thread } from "chat";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js";
|
||||
import { createMessageResourceAdmission } from "./resource-admission.js";
|
||||
import { createChatInboundAuthority } from "./authorization.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const describePostgres = support.supported ? describe : describe.skip;
|
||||
|
||||
describePostgres("setup destination admission with concurrent runtime invalidation", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
|
||||
beforeAll(async () => {
|
||||
database = await startEmbeddedPostgresTestDatabase("chat-resource-admission-");
|
||||
db = createDb(database.connectionString);
|
||||
}, 90_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await database?.cleanup();
|
||||
});
|
||||
|
||||
it("admits a destination while an agent-locked resource insert checks its endpoint foreign key", async () => {
|
||||
const [company] = await db.insert(companies).values({ name: "Admission" }).returning();
|
||||
const [agent] = await db.insert(agents).values({ companyId: company!.id, name: "Worker", role: "engineer",
|
||||
status: "idle", adapterType: "process" }).returning();
|
||||
const [application] = await db.insert(toolApplications).values({ companyId: company!.id, name: "GitHub", type: "chat" }).returning();
|
||||
const [connection] = await db.insert(toolConnections).values({ companyId: company!.id, applicationId: application!.id,
|
||||
name: "GitHub", uid: randomUUID(), transport: "chat_sdk", connectionPurpose: "channel", status: "active" }).returning();
|
||||
const [endpoint] = await db.insert(chatEndpoints).values({ companyId: company!.id, connectionId: connection!.id,
|
||||
provider: "github", publicId: randomUUID(), assignedAgentId: agent!.id, status: "verifying" }).returning();
|
||||
const [resource] = await db.insert(chatEndpointResources).values({ companyId: company!.id, endpointId: endpoint!.id,
|
||||
type: "repository", providerResourceId: "example/first", label: "First", enabled: false }).returning();
|
||||
let releaseEndpoint!: () => void;
|
||||
let releaseAgent!: () => void;
|
||||
const endpointLocked = new Promise<void>(resolve => { releaseEndpoint = resolve; });
|
||||
const agentLocked = new Promise<void>(resolve => { releaseAgent = resolve; });
|
||||
const authority = createChatInboundAuthority(db, {
|
||||
voiceStore: { authorizePrincipal: async () => { throw new Error("Unexpected voice authorization"); } },
|
||||
runtimeCallbackEndpoint: async () => { throw new Error("Unexpected runtime callback"); },
|
||||
});
|
||||
const admission = createMessageResourceAdmission(db, {
|
||||
ensureResource: async (...args) => {
|
||||
releaseEndpoint();
|
||||
await agentLocked;
|
||||
return authority.ensureResource(...args);
|
||||
},
|
||||
});
|
||||
|
||||
const thread = { isDM: false, id: "github:example/first:1", channelId: "example/first",
|
||||
channel: { name: "example/first" } } as Thread;
|
||||
const admitting = admission.admitMessageResource({ endpoint: endpoint!, thread,
|
||||
addressed: true, suppressSetupDestinationActivation: false, surfaceKind: "native_thread" });
|
||||
const inserting = (async () => {
|
||||
await endpointLocked;
|
||||
await db.transaction(async tx => {
|
||||
await tx.select({ id: agents.id }).from(agents).where(eq(agents.id, agent!.id)).for("no key update");
|
||||
releaseAgent();
|
||||
await tx.insert(chatEndpointResources).values({ companyId: company!.id, endpointId: endpoint!.id,
|
||||
type: "repository", providerResourceId: "example/second", label: "Second", enabled: false });
|
||||
});
|
||||
})();
|
||||
const results = await Promise.allSettled([admitting, inserting]);
|
||||
for (const result of results) if (result.status === "rejected") throw result.reason;
|
||||
const resources = await db.select().from(chatEndpointResources).where(eq(chatEndpointResources.endpointId, endpoint!.id));
|
||||
expect(resources).toHaveLength(2);
|
||||
expect(resources.filter(row => row.enabled).map(row => row.id)).toEqual([resource!.id]);
|
||||
});
|
||||
});
|
||||
@@ -56,7 +56,7 @@ export function createMessageResourceAdmission(db: Db, options: { ensureResource
|
||||
.select({ status: chatEndpoints.status })
|
||||
.from(chatEndpoints)
|
||||
.where(eq(chatEndpoints.id, endpoint.id))
|
||||
.for("update")
|
||||
.for("no key update")
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!currentEndpoint) throw notFound("Chat endpoint not found");
|
||||
const enabledResourceCount = await tx
|
||||
|
||||
Reference in new issue
Block a user