Files
PaperClipAI/server/src
DottaandClaude Opus 5.5 940d681974 fix(heartbeat): hold a queued native run while its task's previous session finishes cleanup (#21031)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner runs keep a provider session per task. When a run
stops, its session close continues in the server process
> - The runner refuses every new session of the task until that close
settles. A close that cannot prove the old runner suspended needs a
stopped-process proof before it is released
> - A reassignment stops the old assignee's run mid-turn. The new
assignee's run was dispatched one second later into that gate, and it
failed with `native_session_cleanup_quarantined`
> - Only a Retry continued the task. Recovery must work without a
person, and the needed-retry rate must be 0%
> - This pull request keeps such a run queued until the close settles or
its old owner is proven stopped
> - The benefit is that the new run starts by itself after cleanup,
without a failed run or a Retry

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `tasks.edit-properties` (and
`tasks.assignment-conflict` in an earlier cohort), the CEO reassigned a
task while the old assignee's native run was mid-turn, with a Paperclip
tool call in flight. The reassignment cancelled that run. Its session
close then failed with `native_session_close_unrecoverable: runner did
not durably suspend before checkpoint`, and the runner kept an
operator-required cleanup gate for the task. The new assignee's run had
started 1 second after the cancel. The runner's admission waited for the
in-flight close. When the close escalated, admission refused the run
with `native_session_cleanup_quarantined`. No run was queued again.

**Steps to reproduce**
1. Run a native task and stop its run while its session close cannot
settle at once.
2. Queue a run of another agent for the same task.
3. The run is dispatched into the cleanup gate and fails before it
launches.

**Expected behavior**
The run waits for the cleanup, and starts once the close settles or the
old owner is proven stopped. It never fails, and no Retry is needed.

## What Changed

- `server/src/services/native-runtime/native-cleanup-admission-hold.ts`:
`nativeCleanupHoldsAdmission` reads the task's cleanup gate. For an
operator-required gate it first tries
`retireStoppedSessionCleanupGates`, which retires the gate when the old
owner's runner and provider processes are proven stopped. This is the
same proof that Retry uses.
- `server/src/services/heartbeat/queue.ts`: `claimQueuedRun` keeps an
unstarted `paperclip_runner` run queued while its task has a settling or
unproven close. This matches the existing
`predecessor_settling_in_process` hold, which covers only the same
agent. A reassignment changes the agent.
- `server/src/services/native-runtime/native-cleanup-release-wake.ts`: a
released close now also wakes queued-run dispatch. A close that
escalates to operator-required also wakes it, so admission tries the
stopped-process proof at once. The queued-run liveness redrive covers
any later proof.
-
`server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`:
an assembled heartbeat test. It is red on master (the run is claimed
into a pending close and into an operator-required close). It is green
here (the run stays queued, then is admitted after the close settles). A
control case shows that a run for another task is not held.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts` (3
passed; 2 red on master)
- `cd server && npx vitest run
src/services/quarantined-task-retry.test.ts
src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-workspace-busy.test.ts
src/services/queued-run-dispatch-liveness.test.ts` (180 passed)
- `cd server && npx tsc --noEmit` passes.
- Paid E2E run on a combined branch:
https://github.com/paperclipai/paperclip/actions/runs/43161593494
(results in a comment below).

## Risks

- A run whose task has an operator-required gate that is never proven
stopped now stays queued instead of failing. Before, that run failed,
and Retry was blocked by the same proof. The liveness redrive re-checks
it with back-off.
- The hold applies only to unstarted `paperclip_runner` runs of the
gated task. Other tasks and agents are not affected.
- Previous-release saved state: no persisted shape changes. The gate
lives in process memory, and runs saved by the previous release are
claimed by the same rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:53:54 -06:00
..
…
…
…
…