fix(heartbeat): retry operator Stop after a transaction deadlock (#17895)

<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/69/changes/paperclip/69
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operator Stop must revoke task authority before it stops provider
execution.
> - Its transaction locks the task and then the run. A concurrent writer
can hold the run while it waits for the task.
> - PostgreSQL can abort the Stop transaction with a confirmed deadlock.
The cancel API then returns 500.
> - This pull request retries only that aborted database transaction, at
most three times.
> - Provider Stop and activity publication stay after commit, so the
retry cannot repeat those effects.

## Linked Issues or Issue Description

Refs #17639. Its agent-chat project browser test completed the product
assertions, then failed during cleanup when the cancel API returned 500.
A real PostgreSQL lock-cycle test reproduces the same `40P01` error in
the Stop transaction.

Related: #13911 addresses audit foreign-key lock contention. This change
handles a different concurrent run-writer path without changing the
schema or task/run lock order.

Stack base: #17639. This is a separate correctness repair for the sleep
migration stack, not another polling migration.

## What Changed

- Control the runner timeout regression fixture's clock so it reaches
the pending append before expiry; retain production cancellation and
late-append assertions.

- Retry the Stop authority transaction only after PostgreSQL reports
`40P01`. Make at most three attempts, with short bounded backoff.
- Discard rolled-back local run state, claim flags, and queued activity
publications before each retry. Re-read current authority under fresh
locks.
- Keep provider cancellation and activity publication outside the
retried transaction. Preserve other errors and the existing native
lock-conflict response.
- Add a real two-transaction deadlock test. Add server-side failure
tests for retry exhaustion and errors that must not retry.
- Let run-control tests use a validated disposable PostgreSQL database
URL.

## Verification

Published head: `60ba892ae9fc5faaf8cd0833694b4a66c13323c3`, targeting
master after #17639 merged at
`1ed0af41867ba40fd1d23292d395588d18c2fcbe`. The normal ancestry merge
preserves the complete five-path tested cancellation patch
byte-for-byte, including the review repair, and all four unrelated
current master paths. The production retry logic remains unchanged from
this slice's reviewed implementation; the follow-up adds coverage and
its audit.

- **228 distinct complete cases pass:** all 48 run-control cases and all
180 native-session runtime cases. The two new targeted cases are
included in 48 and are not counted again. Server typechecking, build and
module-boundary checks also pass.
- The new real PostgreSQL cases reach a failed native run's correlated
Stop claim, cancellation fence, board handoff retirement and activity
write before raising `40P01`. One case preserves retry authority and
commits exactly one cancellation activity/live notice. The other changes
coordinator authority between attempts, returns 409, preserves the
failed run and handoff, and performs no provider/workspace cleanup.
- A negative control removes only the publication-list reset. It fails
with two live notices for one durable activity. Production was restored
byte-for-byte before final validation. The existing actual
two-connection deadlock, bounded attempts, non-deadlock rejection and
post-commit provider behavior remain covered.
- Preserve the initial new-assertion typecheck failure (optional
live-event payload) and the corrected final pass. Service-local test
files are typechecked; `src/__tests__` is excluded. The negative-control
harness initially expected an untruncated Vitest assertion line; the
actual named 1-versus-2 failure and restoration were inspected before
correcting its result matcher.
- [Current-parent acceptance and failure
ledger](https://github.com/paperclipai/paperclip/blob/60ba892ae9fc5faaf8cd0833694b4a66c13323c3/doc/plans/2026-10-28-cancellation-deadlock-acceptance.md)
records current scope, commands, isolated fixtures, mutation evidence
and limits. Earlier combined-stack counts retain their original revision
scope. The existing controlled runner timeout/late-append fixture is
retained and passes within the complete 180-case runtime suite.
- This addresses the prior Ponytail request to exercise native claimed
state before deadlock rollback and re-evaluate authority afterward. The
advisory loop-invariant snapshot/named-attempt-bound note remains a
maintainability suggestion, not a runtime defect. No production loop
refactor or new worker duty is introduced.
- **All 58 current checks are clean; Greptile and Ponytail both rate
this exact head 5/5.** Ponytail explicitly confirms that the
native-claim coverage request is resolved and says Ship. The formerly
queued dependency/quality review workflow has completed successfully.
Old `fffc`/parent checks and reviews are not current passes; the former
pending Snyk status is not called successful. Actual failing Ponytail
findings must be fixed. Missing/stopped/paused turns without a verdict
are authorized nonblockers. Active unfinished reviews retain their
original per-head timer, with a 25-minute monitor cutoff respecting the
30-minute maximum.
- No full local repository matrix, live provider execution, external
alarm, physical host/database suspension or savings claim is made. No
schema, dependency, workflow, shared scheduler contract or shared OS
resource changes are introduced.

## Risks

- Persistent contention still returns an error after the third attempt.
This change does not promise to eliminate every possible lock cycle.
- Retrying a write after an ambiguous connection failure would be
unsafe. The retry accepts only PostgreSQL's confirmed deadlock abort
code; it does not accept transport errors.
- No schema or public API contract changes. Current task/run authority
and cancellation eligibility are checked again on each attempt.

## Model Used

OpenAI GPT-6 via Codex, with shell, GitHub CLI, TypeScript, PostgreSQL,
and Playwright tools. The exact serving model ID and context window are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/plans/2026-10-28-cancellation-deadlock-acceptance.md

@@ -0,0 +1,77 @@
+# Cancellation deadlock retry acceptance
+
+This existing correctness slice (#17895) retries only the operator Stop database
+transaction after PostgreSQL reports `40P01`, at most three times. Provider Stop,
+workspace settlement, and activity publication remain after commit. This is not
+a new polling or scheduling duty.
+
+## Current parent and outstanding review
+
+The previous published head is
+`fffc8d0c46fbbeb3da4a23a36e638fefd3b8fe3d`, originally based on native progress
+`08d07ead105dc89c2af9b62f4d90bff8cc93540c`. The slice is prepared by a normal
+merge of verified native progress `6f365285f5b233a89acc714eadfead5ab145100e`.
+All four original changed-file hunks retain their exact content and context.
+Only the run-control blob index hashes differ because the parent also changed
+that file. The automatic composition is clean and preserves those parent edits.
+
+The prior Ponytail review said Ship with an actionable coverage request: existing
+deadlock tests did not reach a correlated native failed run's claimed state before
+rollback. The follow-up adds two real PostgreSQL cases in the existing run-control
+suite, with no new production behavior:
+
+1. Keep retry authority: claim the correlated Stop, fence the failed native run,
+   cancel its real board terminal handoff, and persist cancellation activity inside
+   the first transaction. Raise PostgreSQL `40P01` before commit. Retry succeeds;
+   exactly one cancellation activity is durable and exactly one live notice is
+   published. The correlation belongs to the committed attempt.
+2. Revoke retry authority: after that aborted transaction rolls back, an independent
+   coordinator update retires retry eligibility. The second attempt returns 409,
+   preserves the failed run and uncancelled handoff, publishes no cancellation
+   activity, and performs no provider or workspace cleanup.
+
+The existing actual two-connection lock-cycle regression and the bounded retry/
+non-deadlock cases remain in the complete suite. The new tests exercise claims
+and activity produced before a real PostgreSQL transaction abort; they do not
+infer correctness from a mocked retry callback alone.
+
+## Evidence and failure accounting
+
+- Both new cases pass against the retained production reset logic.
+- Removing only `publications.length = 0` makes the preserved-authority case fail:
+  two live cancellation notices appear for one committed activity. Production was
+  restored byte-for-byte to its integrated index before further validation.
+  `/tmp/sleep-aux-native-retry-mutation-proof.json` records that comparison.
+- The negative-control harness initially expected an untruncated assertion line;
+  Vitest abbreviates that line. Its actual named failure and 1-versus-2 result were
+  inspected, and the matcher was corrected without rerunning or altering production.
+- Typechecking caught an optional live-event payload in the new assertion. The
+  assertion now uses optional access. Preserve the original failed typecheck at
+  `/tmp/sleep-aux-current-parent-types.log`; the final typecheck passes at
+  `/tmp/sleep-aux-current-parent-types-final.log`.
+- **228 distinct complete cases pass:** all 48 run-control cases, including the two
+  additions, and all 180 native-session runtime cases, including the existing
+  controlled timeout/late-append fixture. The two targeted probes are included in
+  48 and are not counted again.
+- Server typechecking, build, and module-boundary checks pass. `server/tsconfig.json`
+  excludes `src/__tests__`, but includes service-local test files such as
+  `run-control.test.ts`; typechecking does not replace test execution. Earlier
+  statements about tests omitted by production typechecking must be read with
+  that path boundary.
+
+The three PostgreSQL probes use separate new disposable databases on the owned
+local cluster. Full-run results and exit receipts are in
+`/tmp/sleep-aux-current-parent-{types-final,build,boundaries,control,runner}.log`.
+The targeted positive and mutation logs are
+`/tmp/sleep-aux-native-retry-regression.log` and
+`/tmp/sleep-aux-native-retry-mutant-red.log`.
+
+## Remaining gates and limits
+
+This is local preparation. The dependency must merge, the normal merge onto its
+actual landed master must preserve this tested patch and all master changes, and
+fresh exact-head CI plus Greptile 5/5 must pass before protected merge. Prior
+review scores and checks on older heads are not current passes. The previous
+Snyk status was pending and is not called successful here. No shared OS resource
+cleanup, schema change, whole-repository local matrix, live provider execution,
+external alarm, physical suspension, or savings qualification is claimed.