Files
PaperClipAI/doc
Devin FoleyandPaperclip 6bf312270c fix(ci): stabilize database-backed readiness checks (#22476)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - CI checks its recovery workers, attention routes, and external chat
admission against real PostgreSQL.
> - Large fixtures start from an empty database without statistics for
their new rows.
> - Slow query plans can exhaust test deadlines before the behavior
checks finish.
> - Route responses can also leave wakeups writing rows while teardown
deletes their agents.
> - Setup admission takes an endpoint lock that can deadlock with
resource foreign-key checks and agent invalidation.
> - This pull request prepares fixture statistics, drains pending work,
and uses a compatible endpoint lock.
> - The benefit is reliable readiness checks with the same fixtures,
assertions, and deadlines.

## Linked Issues or Issue Description

Refs: #21941, #21831, #21132. The compiler-memory mitigation in #19383
is already merged and is not repeated here.

**What happened?**

The recovery census and review-attention equivalence tests repeatedly
timed out on unrelated PRs. Recovery-action teardown also failed on an
agent-wakeup foreign key, and concurrent first GitHub mentions could
deadlock during resource admission.

Examples: [census
timeout](https://github.com/paperclipai/paperclip/actions/runs/44868742060/job/136738633187),
[review
timeout](https://github.com/paperclipai/paperclip/actions/runs/44856500775/job/136732143246),
[cleanup
failure](https://github.com/paperclipai/paperclip/actions/runs/44866635624/job/136731817497),
and [chat
deadlock](https://github.com/paperclipai/paperclip/actions/runs/44845946341/job/136667344125).

**Expected behavior**

The complete fixtures retain their regression coverage, pending wakeups
finish before deletion, and concurrent chat admission completes with
exactly one setup destination enabled.

**Steps to reproduce**

1. Run the two timeout suites against a freshly seeded database before
its statistics refresh.
2. Complete an issue-update request that schedules an assignee wake,
then delete its fixture immediately.
3. Run the new resource-admission regression on the base revision. It
forces the endpoint/agent lock order and fails with PostgreSQL SQLSTATE
`40P01`.

**Paperclip version or commit**

Base: `33ffaa5b9a2e0954c2afc063572346cb1be534a7`.

## What Changed

- Run `ANALYZE` on the populated fixture tables after seeding the
existing census and review-attention populations. No fixture reduction,
assertion removal, retry increase, or deadline increase.
- Await `drainActiveRunExecutions()` before recovery-action fixture
cleanup. This includes pending wake admission after HTTP responses.
- Use `NO KEY UPDATE` for setup endpoint admission. It still serializes
competing admission transactions and endpoint mutations while allowing
resource foreign-key `KEY SHARE` locks.
- Add a real PostgreSQL regression using the real `ensureResource`
write, with barriers that force concurrent endpoint admission and an
agent-locked resource insert. Both transactions must finish and exactly
one destination must become enabled.
- Document fixture statistics, teardown ownership, and admission locking
in `doc/test-infrastructure.md`.

## Verification

- The new concurrency regression fails on the original endpoint lock
with SQLSTATE `40P01`, then passes with `NO KEY UPDATE`.
- Both timeout suites pass with their full populations and original
deadlines: 8 tests.
- The full recovery-action suite passes: 120 tests.
- Existing GitHub first-repository and disabled-repository integration
checks pass: 7 tests.
- The new PostgreSQL concurrency regression passes: 1 test.
- The local baseline timeout suites passed slowly; the review
equivalence case took 35.8 seconds before statistics and 0.63 seconds
afterward. These are local observations, not a CI speed guarantee.
- The first broad typecheck caught an unsupported
`Promise.withResolvers` type in the new test. The barriers now use
ordinary promises compatible with the repository target. The final `pnpm
-r typecheck` passes.
- `git diff --check` passes.
- `pnpm -r typecheck` and `pnpm build` pass. The full local `pnpm
test:run` attempt stopped after all six chat shards: 1,969 tests passed
and two Slack tests failed on skill-root discovery. The remaining local
groups did not run. Both failures reproduce with the production change
reverted to `origin/master`: the resolver selects an unrelated ancestor
`skills/` directory without `slack/SKILL.md`, which also prevents the
mocked Slack run from starting. This is a confirmed baseline
local-layout issue, not a passing full-suite result. The [complete CI
run](https://github.com/paperclipai/paperclip/actions/runs/44873350026)
passes on `66c146b177b50e052a21921644dfd626c96c010d` on attempt 1. The
PR check rollup has 58 successes, three expected skips, and no pending
or failing checks. [Greptile reviewed the current head
`66c146b`](https://github.com/paperclipai/paperclip/pull/22476#issuecomment-7058185817)
at 5/5 with no actionable findings.

Review follow-up verification:

- Replaced the resource-read stub with the real
`createChatInboundAuthority(...).ensureResource` persistence path. The
revised regression fails with `FOR UPDATE` and PostgreSQL `40P01`, then
passes with `NO KEY UPDATE`.
- Scoped `ANALYZE` to the fixture tables and moved the behavior note
from `DEVELOPING.md` to `test-infrastructure.md`.
- All 9 affected tests pass after these edits; the complete server
typecheck also passes. The production lock change is unchanged from the
earlier full workspace typecheck and build passes.
- The previous CI attempt had two interrupted jobs: [Runner static
checks](https://github.com/paperclipai/paperclip/actions/runs/44872465981/job/136749952142)
and [serialized server
9/9](https://github.com/paperclipai/paperclip/actions/runs/44872465981/job/136749952782).
Both logs record a runner shutdown. The static lane reported 2,371
passing tests before cancellation; the interruption is not counted as a
pass. The final head passes the complete fresh CI run linked above,
including both lanes.

## Risks

Low scope. Production code changes only the endpoint row-lock strength.
The regression uses real foreign keys and connector invalidation
triggers; the existing integration checks retain first-destination
serialization and disabled-destination behavior. No schema, API, or
authorization changes. `ANALYZE` updates only test-database statistics.
Drain failures still fail teardown.

All current-head remote checks now pass, including typecheck, build,
tests, and e2e. The baseline local full-suite failure and unrun
remaining local groups are recorded above; focused local checks pass.
The already merged compiler-input fix does not establish the cause of
every past runner shutdown.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh
reasoning and shell/code execution. The session does not expose its
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-20 22:52:05 -08:00
..
…
…
…
…