From e3faf579a1552b764d6095efe981d56550838efa Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 06:46:18 -0600 Subject: [PATCH] test(recovery): drain background runs before the fresh-continuation test cleanup (#21375) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server continues an interrupted task in a new run from saved work > - `fresh-continuation-fallback.test.ts` tests that behavior against a real PostgreSQL database > - Some tests start runs that continue in the background after the test body returns > - The cleanup truncated its tables while those runs still held locks, and PostgreSQL sometimes reported a deadlock > - The failure appeared on unrelated pull requests in server shard 9/12 > - This pull request makes the cleanup wait for in-flight runs and retry the truncate on a deadlock > - The benefit is that this file no longer fails CI at random ## Linked Issues or Issue Description **What happened?** The `afterEach` in `server/src/services/fresh-continuation-fallback.test.ts` sometimes fails with `PostgresError: deadlock detected` (code 40P01). The truncate waits for an `AccessExclusiveLock`. A background run that a test started holds an `AccessShareLock` and waits for another table. No assertion fails. The test that receives the failure changes from run to run. **Expected behavior** The cleanup completes after each test. The file passes on each run. **Steps to reproduce** 1. Check out master at 18fb6ee571. 2. Run `npx vitest run src/services/fresh-continuation-fallback.test.ts` in `server/` on a loaded machine, several times. 3. One test fails in `afterEach` with the deadlock error in some runs. **Relevant logs or output** `truncate table issue_recovery_actions, ... heartbeat_runs, issues cascade` → `PostgresError: deadlock detected (40P01)` Refs #21348, #21135. ## What Changed - The `afterEach` awaits `heartbeatService(db).drainActiveRunExecutions()` before the truncate. The registry of in-flight runs is shared by all service instances. - The `afterEach` truncates through the existing `truncateTablesWithDeadlockRetry` helper. - No product code changed. ## Verification - `npx vitest run src/services/fresh-continuation-fallback.test.ts` in `server/`: 72 of 72 passed, two times in sequence. - `npx tsc --noEmit -p .` in `server/`: no errors. - The deadlock is not deterministic. Two passing runs do not prove that it cannot occur. The retry helper covers a late writer that the drain cannot see. ## Risks Low risk. The change is in one test file. The drain adds a short wait to each test cleanup. ## Model Used Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 --- server/src/services/fresh-continuation-fallback.test.ts | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/server/src/services/fresh-continuation-fallback.test.ts b/server/src/services/fresh-continuation-fallback.test.ts index 390b322b19f..a0351244cf7 100644 --- a/server/src/services/fresh-continuation-fallback.test.ts +++ b/server/src/services/fresh-continuation-fallback.test.ts @@ -6,6 +6,7 @@ import { heartbeatRunEvents, issueComments, issues, nativeRunFinalizations, toolInvocations, } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { truncateTablesWithDeadlockRetry } from "../__tests__/helpers/truncate-with-deadlock-retry.js"; import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./execution-recovery-resolution.js"; import { PENDING_CONTINUATION_BOUND_MS } from "./recovery-continuation-pacing.js"; import { getExecutionBlocker } from "./execution-blocker.js"; @@ -62,7 +63,8 @@ suite("fresh-continuation fallback", () => { afterEach(async () => { resetBacklogContinuationPacingForTests(db); - await db.execute(sql`truncate table issue_recovery_actions, native_run_finalizations, tool_invocations, + await heartbeatService(db).drainActiveRunExecutions(); + await truncateTablesWithDeadlockRetry(db, `truncate table issue_recovery_actions, native_run_finalizations, tool_invocations, issue_approvals, approvals, issue_thread_interactions, activity_log, agent_wakeup_requests, environment_leases, execution_workspaces, heartbeat_runs, issues cascade`); });