Files
PaperClipAI/server
Nicky LeachandClaude Sonnet 5 41c1431d66 fix(server): remove the fixture-cleanup deadlock in the stale queued-run test (#14968)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server heartbeat tests use shared database fixtures and
concurrent run execution.
> - The stale queued-run test checked the database before all active
executions finished.
> - A late run writer could then hold a row lock while fixture cleanup
held another table lock.
> - PostgreSQL could report a deadlock, and the test cleanup could fail.
> - This pull request waits for active executions and shares the
deadlock retry helper.
> - The benefit is reliable test cleanup without a production code
change.

## Linked Issues or Issue Description

Refs: #14198

**What happened?**

The stale queued-run test could fail during fixture cleanup. Its cleanup
poll could miss a run that had not written its database row. A later
writer could then deadlock with the cleanup `TRUNCATE` operation.

**Expected behavior**

Fixture cleanup must wait for active run executions. It must retry
transient PostgreSQL deadlocks and foreign-key races.

**Steps to reproduce**

1. Run the server heartbeat tests with concurrent test activity.
2. Let an active run write its database row after the cleanup poll
starts.
3. Observe a PostgreSQL `40P01` deadlock during fixture cleanup.

**Paperclip version or commit**

`master` at the base commit for this pull request.

**Deployment mode**

Test-only change against the server test database.

## What Changed

- Replace the local cleanup poll with `drainHeartbeatRunsToQuiescence`.
- Add the shared `truncateTablesWithDeadlockRetry` test helper.
- Retry PostgreSQL `40P01` deadlocks and the existing foreign-key race.
- Use the shared helper in both heartbeat test files.
- Remove the private duplicate helper.
- Change no production code.

## Verification

- `vitest run
server/src/__tests__/helpers/truncate-with-deadlock-retry.test.ts`
passes.
- `vitest run
server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts`
passes.
- The target stale queued-run test passes in the engineer validation
run.
- CI must confirm the full stale queued-run test file and the required
server checks.

## Risks

Low risk. This change affects test cleanup only. The helper keeps a
bounded retry count and rethrows persistent errors.

The full stale queued-run test file needs CI because the local sandbox
lacks the managed Codex home required by an existing pre-dispatch
credential gate.

## Model Used

OpenAI Codex, GPT-5, with tool use and code review assistance.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
documentation change is needed for this test-only change)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

## Test plan

- [x] CI `General tests` passes for the `server` shards.
- [x] CI `verify` passes.
- [x] `vitest run
server/src/__tests__/helpers/truncate-with-deadlock-retry.test.ts`
passes.
- [x] `vitest run
server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts`
passes.
- [x] `vitest run
server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts` passes.

## Note on the sandbox test gap

The engineer could not run the full
`heartbeat-stale-queue-invalidation.test.ts` file green in the
development sandbox. A pre-dispatch credential gate in `heartbeat.ts`
needs a usable Codex home for `codex_local` agents, and that sandbox has
no managed home. The engineer proved the failure is pre-existing: they
stashed both test-file changes, reran the unmodified file, and got the
identical 25 failures with the identical error. CI supplies the
full-file run. Treat the CI result for this file as the authoritative
check.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-05 13:50:26 -07:00
..