Files
PaperClipAI/doc
DottaandClaude Opus 5.5 e620ce3678 fix(heartbeat): deliver each accepted follow-up exactly once through restarts and admission holds (#19741)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can comment, click Retry, or rely on automatic recovery while
an agent works on a task
> - Restart-chaos waves showed accepted follow-ups being dropped, or
delivered twice, around restarts and temporary company admission holds
(accounting still settling after a restart)
> - A dropped comment means the user's request is never acted on. A
duplicate reopens work that is already done
> - This pull request makes accepted work wait through a temporary hold
and resume when it clears. It also stops a comment from being promoted
again after a completed run already received it
> - The benefit is that each accepted follow-up is delivered exactly
once, eventually, unless the task is cancelled or a reached hard stop
applies

## Linked Issues or Issue Description

Refs #19702 (never-started tasks during a temporary hold), #19700
(bounded accounting hold), #14910 (stale reopen from a deferred wake; no
file overlap).

**What happened?**
Restart-chaos waves found these shapes:
1. **Comment wake cancelled.** A comment was saved behind a run and
promoted when the run ended. A temporary company hold refused the
unstarted queued run, and claim cancelled it terminally. The recovery
wake that followed was cancelled the same way, and the task rested
`blocked` without the comment.
2. **Dead Retry.** A Retry click during the hold stayed
`deferred_issue_execution` after the hold cleared. Nothing woke it, and
the issue held no execution lock.
3. **Refused assignment.** A new task's first wake was refused during
the hold. Nothing woke stranded-work recovery when the hold cleared, so
the task stayed `todo` with no run.
4. **Silent comment after recovery.** A comment was saved behind a run
that restart recovery finalized. A company-scoped hint waited out the 30
s deferred-receipt rescan, so the task looked done with the message
unread.
5. **Comment delivered twice.** A run whose provider process was lost
had a saved comment. Its retry started after the comment, received it in
its prompt (task history), did the work, and finished the task. The
saved wake was then promoted anyway, which reopened the done task.

**Expected behavior**
An accepted follow-up (comment, Retry, recovery) is delivered exactly
once, eventually, unless the task is cancelled. A temporary refusal
never ends in a terminal cancel, skip, or block. Reaching a hard stop
remains the board's decision.

## What Changed

- `heartbeat/queue.ts`: on a budget block at claim, an unstarted queued
run stays queued when the block is temporary. A reached hard stop or
blocked unpriced usage still cancels it.
- `recovery/service.ts`: stranded-work recovery waits out a temporary
hold for any task. It does not escalate to `blocked`. This extends
#19702 beyond never-started tasks.
- `budget-block-ownership.ts`: a shared `isBoardOwnedBudgetBlock`, moved
out of the recovery service. `budgets.ts` is unchanged.
- `run-cost-accounting.ts`: settling a run's accounting wakes queued-run
dispatch, the deferred-receipt worker, and stranded-work recovery. Held
work waits passively; this is what resumes it.
- `deferred-execution-notifications.ts`: new
`notifyDeferredWakesForIssue`. `native-run-projection.ts` (a run
reaching a terminal status) and `environments.ts` (lease release) now
hint each saved wake of the issue by id, so delivery does not wait out
the company rescan interval.
- `wake-queue/adapters/postgres.ts`: release treats a saved comment as
delivered only when a succeeded run of the wake's agent on that issue
started after the comment's latest revision, and either received the
comment as its own wake input (`wakeCommentIds`) or was a retry of the
interrupted turn whose task history included it. In that case it cancels
the saved wake with "Queued messages were already delivered to a
completed run" and leaves the task as it is. Task history alone never
consumes a message saved for its own turn. One query checks the whole
batch.
- `run-cost-accounting.ts`: settling late receipt debt also sends the
hold-release notifications.
- `doc/execution-semantics.md`: documents the rule.

## Verification

Red on `master` and green with this change, on embedded Postgres:
- `server/src/__tests__/heartbeat-held-comment-wake.test.ts`:
- A promoted comment refused by the hold stays pending, with no
cancelled run and no `blocked` task. It is delivered once after the hold
clears through real `accountRunCost`.
- A Retry clicked during the hold starts exactly once after accounting
settles, which wakes its owner.
- A new task whose assignment was refused during the hold starts once
accounting settles, which wakes stranded-work recovery.
- A comment that a later succeeded retry already received as history, or
that a later run received as wake input, is consumed without reopening
the task.
- Guard tests (green on both):
- A comment is still delivered in each of these cases: the later run
never received it, it was edited after that run started, it was posted
while the run was already running, or a non-retry run only saw it as
history.
  - A task whose continuation was refused waits out the hold.
  - A reached hard stop is still refused.
- `server/src/__tests__/native-run-projection-work.test.ts`: projecting
a recovered run terminal hints the saved comment's own wake (red on
`master`).
- Related suites pass locally (14 files, 482 tests): wake-queue, queued
comments routes, refused assignment wake, native run projection.
- Live replays (local, unmasked, kept instances):
- `--replay w-20261107T031005-s7501/t04` (Claude C1, K1): PASS. The
comment's run was created at 03:56:24 during the hold, stayed queued,
started at 04:06:24 when the hold cleared, and succeeded. It is the only
run carrying the comment.
- `--replay w-20261107T002911-s7401/t01` (before the per-wake hint): it
reproduced shape 4. Run 592d772f was killed 250 ms after
`run.result.accepted`, released and finalized at restart, and the task
rested `done` with the comment still saved. Calling the existing sweep
on the kept database promoted it, which confirmed that the missing piece
was the trigger.

## Risks

- A queued run can now sit queued for the length of a temporary hold,
which #19700 bounds. It does not hold the issue's execution lock, so
later comments still coalesce into it.
- The delivered-comment rule depends on the run's recorded prompt
context. A succeeded run that received the comment in its history counts
as delivery, even if the agent did not act on it. This matches
exactly-once delivery semantics.
- Settling accounting now emits three extra notifications per run. They
are coalesced per company.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in
Claude Code with tool use (shell, file editing, local test execution,
live restart-chaos replays).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 00:24:09 -06:00
..
…
…
…
…
…