mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
DottaandClaude Opus 5.5
7213f64962
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat records the provider process identity when a native or legacy run spawns its process > - This write updated the run row and then reserved the event sequence with a second UPDATE of the same row > - PostgreSQL re-checks every foreign key on a row that the current transaction already changed, so the second UPDATE took KEY SHARE on the issue, the predecessor run, and the other parent rows > - A transaction that held one of those rows FOR UPDATE blocked spawn persistence, and the failed write ended a user's Retry as `native_session_interrupted` > - This pull request writes the identity and reserves the sequence in one UPDATE, and it retries deadlock and serialization failures of the spawn write > - The benefit is that a user's Retry no longer fails because an unrelated transaction holds the issue or the failed predecessor run ## Linked Issues or Issue Description **What happened** A user clicked Retry on a failed native run in the `runs-adapters.inspect-failed-run` chat eval (embedded PostgreSQL, one server process). The retry run failed with `native_session_interrupted`. Its error was `Failed query: update "heartbeat_runs" set "next_event_seq" = "heartbeat_runs"."next_event_seq" + 1 ...`. The query started at 01:51:04 and failed at 01:51:16. The OpenCode driver awaits `onSpawn` before its health check, so `native.session.execute` failed after 12002 ms. The stack showed `allocateHeartbeatRunEventSeq` inside `appendHeartbeatRunEvent`, called from `persistHeartbeatRunProcessMetadata`. The process identity was lost (`processPid` stayed null). The server log does not include the PostgreSQL cause code. **Expected behavior** Spawn persistence must not wait on locks for unrelated parent rows. A transient deadlock or serialization failure must be retried. It must not end the run. **Steps to reproduce** 1. Create a run whose `issue_id` and `retry_of_run_id` point at an issue and a predecessor run. 2. In a second connection, begin a transaction and run `SELECT ... FROM issues WHERE id = <issue> FOR UPDATE` (or lock the predecessor run). Keep the transaction open. 3. Call `persistHeartbeatRunProcessMetadata` for the run. 4. Before this change, the call blocks on the `next_event_seq` UPDATE until the second transaction ends. If that transaction then waits on the run row, PostgreSQL reports a deadlock. `server/src/services/heartbeat/process-identity-persistence.test.ts` reproduces step 4. Root cause: `server/src/services/heartbeat/run-lifecycle.ts` (`persistHeartbeatRunProcessMetadata`) updated the run row, then `appendHeartbeatRunEvent` called `allocateHeartbeatRunEventSeq` (`server/src/services/heartbeat-run-events.ts`), which updated the same row again. PostgreSQL's `RI_FKey_fk_upd_check_required` always fires the foreign-key checks when the old row version was written by the current transaction. The checks take KEY SHARE on each referenced row, and KEY SHARE conflicts with FOR UPDATE. ## What Changed - `appendHeartbeatRunEvent` accepts a narrow `runUpdate` option (process identity columns only). `allocateHeartbeatRunEventSeq` applies these columns in the same UPDATE that reserves the sequence. Deduplicated receipts (native source, retry exhaustion) reject `runUpdate`, because they do not allocate. - `persistHeartbeatRunProcessMetadata` reads the run binding, appends `native.process_identity_recorded` with the identity as `runUpdate`, and returns the updated row. The run row now gets one UPDATE in this transaction. - New `retryTransientRunWrite` retries SQLSTATE `40P01` and `40001` up to four times. The lifecycle `persistRunProcessMetadata` and the legacy cleanup-intent wrapper in `heartbeat.ts` use it. The legacy wrapper still updates the run row a second time when it records the cleanup intent, so the retry covers a deadlock there. - New regression test `server/src/services/heartbeat/process-identity-persistence.test.ts`. ## Verification - `npx vitest run src/services/heartbeat/process-identity-persistence.test.ts` (from `server/`): 3 passed. Without the `run-lifecycle.ts` change, both lock cases fail with `expected 'blocked' not to be 'blocked'` after 3 s. - `npx vitest run src/__tests__/heartbeat-run-event-sequencing.test.ts src/services/heartbeat/run-lifecycle.test.ts`: 42 passed. - `npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "<the four tests that call persistHeartbeatRunProcessMetadata>"`: 5 passed. - `tsc --noEmit -p server`: no errors in the changed files. The only error comes from a local workspace-link mix in `packages/adapter-utils`, which this change does not touch. - I did not run `pnpm verify` or the full suite locally. CI supplies that coverage. ## Risks - Low risk. The written columns, the event type, the payload, and the sequence order do not change. - The event payload `processStartedAt` now comes from the value that the transaction writes, not from a re-read of the row. The value is the same. - An event insert still checks its own `agent_id` and `company_id` foreign keys. A transaction that holds the agent row FOR UPDATE can still delay the spawn write. That wait is the same for every event writer, and this change does not alter it. - Other paths that update a run row and then append an event in the same transaction keep the second-UPDATE foreign-key checks. This change fixes the spawn identity path that lost the run. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, embedded PostgreSQL experiments). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>