mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
DottaandClaude Opus 5.5
7644045bc5
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server Vitest shards gate every pull request > - Three open pull requests (#19289, #19267, #19266) failed server tests in code that they do not change > - Each failure had a different cause. One cause is a real product deadlock, and three causes are test races > - Random CI failures slow every contributor and hide real regressions > - This pull request removes each root cause. It does not add retries or skips > - The benefit is fewer false CI failures and one less deadlock in external-agent settlement ## Linked Issues or Issue Description Refs #19289, #19267, #19266 **What happened?** Server tests failed at random on pull requests that did not touch them: - #19289, job 127412331924: `dot-runner.test.ts` > "signed readiness, native tools and finalization through Dot local Rust" failed with `PostgresError: deadlock detected` on the run-event append (`select ... from heartbeat_runs ... for update`). - #19267, job 127427384499: `tool-access-service.test.ts` > "keeps an organization audience fail-closed when empty replacement races with member archive" received a widened grant instead of the 409 error. - #19267, job 127410158948: `heartbeat-process-recovery.test.ts` > "does not reap active adapter executions started by another heartbeat service instance" failed with "Timed out waiting for adapter execution to start". - #19266, job 127404184242: `voice-session-store.integration.test.ts` > "schedules phone reply retries at the saved rate-limit deadline without early database scans" failed with a duplicate key on `companies_issue_prefix_idx`. **Root causes** 1. Dot deadlock (product bug). `settle` in `native-broker.ts` takes the external admission guard. Then it updates the same assignment row in two or more statements. PostgreSQL re-runs the foreign-key checks when a transaction updates a row version that it wrote earlier, so the second UPDATE takes `FOR KEY SHARE` on the heartbeat run. Execution admission locks the issue and the run `FOR UPDATE` before it takes the guard. Thus settle holds the guard and waits for the run, and admission holds the run and waits for the guard. The run-event append can queue behind them as a third participant, and the database can kill it. A local reproduction with `log_lock_waits` showed the RI check: `SELECT 1 FROM ONLY heartbeat_runs x WHERE id = $1 FOR KEY SHARE OF x`, during `update dot_runner_assignments set status='settled'`. 2. Tool-access race (test bug). The empty replacement and the member cleanup both waited for the first replacement to commit. Both waiters woke at the same time and raced for the membership row. A replacement that wins that race widens access before the cleanup, and the product code documents that outcome as valid ("Whichever transaction wins is then authoritative"). The test only expected the cleanup to win. 3. Heartbeat timeout (test bug). The run start includes real runtime skill preparation for a new company. I measured 3.3 s to 5.5 s locally before the adapter starts. The test gave this step a fixed 3 s budget. 4. Voice prefix collision (test bug). Each of about 110 fixture companies used `V` plus seven random hex digits as its unique issue prefix, so two companies could get the same prefix. **Master:** These tests do not fail on recent master runs (the master workflows do not run the server shards). The failures are timing-dependent and are not specific to a branch. ## What Changed - `server/src/modules/external-agents/native-broker.ts`: `settle` writes the assignment in one UPDATE. This applies to both Dot and Muse. The update sets the same columns under the same conditions as before. - `server/src/__tests__/dot-runner.test.ts`: added "settles accept/finish under the admission guard without waiting on a run that admission holds". The test holds the run `FOR UPDATE` like admission does, then runs settle, then takes the guard. Before the fix, settle waits and PostgreSQL reports a deadlock. After the fix, settle completes. - `server/src/__tests__/tool-access-service.test.ts`: start the empty replacement after the cleanup holds the membership row. Then wait until the replacement is blocked by the cleanup. The test still proves that the replacement re-checks the member status after the cleanup and fails closed. - `server/src/__tests__/heartbeat-process-recovery.test.ts`: wait for the adapter to start. Fail at once if the run settles before the adapter starts. The test timeout is now 30 s, because the start (3 s to 6 s) and the existing 5 s settle wait came close to the 15 s default. - `server/src/__tests__/voice-session-store.integration.test.ts`: each fixture prefix includes a per-run sequence, so prefixes in one run are always distinct and stay at 10 characters. ## Verification - Red/green for the deadlock: with the `native-broker.ts` change removed, both new tests fail with `deadlock detected`. With the change, both pass. - `npx vitest run src/__tests__/dot-runner.test.ts` (server): 57 passed. - Before the fix, I ran the Dot Rust test 8 times with `deadlock_timeout=50ms` and got the deadlock in about 1 of 8 runs. Two further runs of 16 iterations each passed. - `npx vitest run src/__tests__/tool-access-service.test.ts -t "fail-closed when empty replacement races"`: 3 passed, run 4 times. - `npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "does not reap active adapter executions"`: this failed locally before the change (start took 3.3 s to 5.5 s). It passes after the change. - `npx vitest run src/__tests__/voice-session-store.integration.test.ts`: 132 passed. - `npx vitest run src/__tests__/muse-control.test.ts`: 256 passed. This file covers the Muse settle path. - Not run locally: the full `pnpm -r typecheck` / `pnpm test:run` / `pnpm build`. CI runs them. A server `tsc --noEmit` shows no errors in the changed lines. The existing errors come from the unbuilt runner package in this worktree. ## Risks - Low risk. The settle change keeps the same column writes and conditions, now in one statement. It removes a lock wait. It does not add a lock. - The other changes are only in tests. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, and GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>