Files
PaperClipAI/doc
DottaandPaperclip b928c6698f fix(heartbeat): explain deferred monitor dispatch (#16891)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - One-shot task monitors request future work through the normal
heartbeat queue.
> - Queue admission can hold execution until recovery or ownership is
reconciled.
> - The scheduler retained those monitor schedules but recorded no
monitor diagnostic.
> - Agents could repeatedly reschedule the same blocked check without
seeing why it did not run.
> - This pull request records safe deferral and failure activity while
retaining all execution gates.
> - Agents and operators can inspect the actual blocker and use its
existing recovery path.

## Linked Issues or Issue Description

Related: #15556 handles transient monitor client errors. #14590 handles
parked monitor wakes. This change records the diagnosis when queue
admission defers or fails; it does not change their admission or monitor
consumption policies.

**What happened?**

A due monitor on a task with an unresolved execution-recovery hold was
claimed, passed to queue admission, and released without monitor
activity. The saved monitor continued to look scheduled. Unexpected
dispatch errors had the same audit gap. Repeated rearming could not
clear the recovery hold.

**Expected behavior**

Keep the monitor and the recovery hold. Record the reason that prevented
its wake. Do not execute uncertain work or consume an attempt. Limit
repeated identical diagnostics. If audit storage fails, release the
claim so a later tick can retry.

**Steps to reproduce**

1. Schedule a due monitor on an agent-owned task in progress.
2. Add an active recovery action for an interrupted legacy execution.
3. Run the monitor scheduler twice. No provider may start while the hold
remains.
4. Before this fix, neither attempt writes monitor deferral activity.
5. Resolve the hold through its existing recovery path. The saved
monitor can then request one wake.

**Paperclip version or commit**

Reproduced on master 5739a53f5f with real PostgreSQL and the assembled
heartbeat service.

**Deployment mode**

Built from source. This affects the server monitor scheduler.

## What Changed

- Pass the existing execution blocker cause to the monitor scheduler
through a server-owned diagnostic callback.
- Record issue.monitor_deferred for held admission and
issue.monitor_dispatch_failed for unexpected admission failures.
- Keep activity to known reason codes and allowlisted blocker causes. Do
not copy raw provider messages.
- Fence claim release and activity against replacement, reassignment,
completion and competing scheduler instances.
- Limit identical activity for one schedule and cause to one record per
five minutes.
- Keep claim release functional when diagnostic storage fails.
- Update execution documentation and the task verification maps.

## Verification

- Red: explains held monitor dispatch without replaying uncertain
execution failed because it expected one deferral activity row and
received none. The fixture used the real queue, persisted source run and
recovery action.
- Red: records bounded dispatch failure diagnostics while retaining the
schedule failed because no failure activity was recorded.
- Green: both regressions pass after the fix. The held-source test
proves no successor while held, no monitor attempt consumed, and one
wake after explicit hold resolution.
- Passed: all 43 tests in heartbeat/scheduling.test.ts and
issue-monitor-scheduler.test.ts, with at most two workers.
- Additional coverage uses real PostgreSQL for diagnostic rate limits,
replacement races and an audit-insert trigger failure. Audit failure
must not retain the monitor claim. Private error and unknown blocker
text do not reach activity.
- Passed: node scripts/check-maps.mjs and git diff --check.
- Greptile: 5/5 with no actionable issues on
1d0aec738c.
- Latest-head checks: 55 successful, two skipped, and one neutral. No
check remains pending or failed. The branch has no merge conflict.
Paperclip Review ended neutral because its agent did not return a
complete assessment; it is not counted as a validated review pass.
- The dependency added on deferral is one bounded recent activity query
and at most one audit insert. Successful admission adds no new database
read. Existing queue authority checks still decide admission.
- The process fixtures validate queue admission and persisted state.
They log workspace setup errors after admission; this report does not
claim a live provider turn succeeded or that an existing deployment was
repaired.
- Passed: pnpm --dir server exec tsc --noEmit. The wrapper typecheck was
stopped because it started a redundant cold native Runner build. Full
repository checks run in PR CI; no local full-suite or full-build pass
is claimed.

## Risks

- Audit work must not strand a monitor. The storage-failure regression
verifies that rollback is followed by a claim-fenced release.
- Recovery holds still require their current reconciliation path. This
change makes their effect visible and does not bypass ownership, budget,
pause, Stop or company checks.
- Activity can be stale after a later repair, so each entry includes the
checked time and the exact saved deadline. A replacement cannot inherit
an old dispatch diagnostic.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution.
The exact runtime model suffix and context window are not exposed to
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-19 16:11:31 -05:00
..
2026-02-16 14:25:00 -06:00