fix(heartbeat): hold the new owner's run while the reassigned owner's run settles (#21301)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person reassigns a task, the old owner's native run is
cancelled and the new owner's run is queued
> - The old run's executor keeps closing its provider session for
several seconds after the run is marked cancelled
> - #21031 holds a queued run while the task's previous session close is
pending. That hold reads the runner's cleanup gate, which exists only
after the close begins
> - The stale-lock sweep cleared the task lock as soon as the old run
was terminal. The new owner's run was then claimed before the close
began, met the close as it escalated, and failed with
`native_session_cleanup_quarantined`
> - This pull request keeps the new owner's run queued while the old
owner's stopped run still settles in this process, and wakes it when
that executor finishes
> - The benefit is that a reassignment never fails the new owner's run
and needs no Retry

## Linked Issues or Issue Description

Refs #21031

**What happened?**
In the paid chat eval `tasks.edit-properties`, the old owner's run
`6cf2ee38` was cancelled for reassignment at 04:03:47.611. Its process
stopped only at 04:03:57. The stale-lock sweep cleared the task lock at
04:03:47. The new owner's run was claimed at 04:03:48.626. The runner
admission waited for the old close, which escalated to operator-required
at 04:03:56, and the new run failed with
`native_session_cleanup_quarantined`.

Why #21031 did not apply: its hold reads the runner's in-memory cleanup
gate. At claim time the old executor had not started its session close,
so no gate existed. The existing `predecessor_settling_in_process` hold
covers only the same agent, and a reassignment changes the agent. #21260
does not touch this path: its new checkpoint runs only before a turn in
a recovered session, and both sessions here were new.

**Steps to reproduce**
1. Run a native task for agent A.
2. Reassign it to agent B. A's run is cancelled, but its executor is
still settling.
3. The stale-lock sweep clears the task lock.
4. B's queued run is claimed before A's session close begins.

**Expected behavior**
B's run stays queued until A's executor has finished. It starts by
itself after that, then #21031's gate covers any close that is still
settling.

## What Changed

- `server/src/services/heartbeat/queue.ts`: the
`predecessor_settling_in_process` hold in `claimQueuedRun` also holds
for another agent's native run on the same task when that run is
terminal but its executor is still active in this process. A live run of
another agent (for example a reviewer) does not hold it.
- `server/src/services/heartbeat/queue.ts`:
`notifyOtherAgentsQueuedOnSettledIssue` sends a queued-run dispatch
signal when a native executor settles and another agent has a queued run
on the same task. It does nothing when no run waits.
- `server/src/services/heartbeat.ts`: the executor calls it after its
cleanup, next to the existing same-agent `startNextQueuedRunForAgent`.
- `server/src/services/heartbeat/queue.test.ts`: the exact ordering
(cancelled old-owner run still in the executor set, lock cleared, new
owner's run queued), a control for a live run of another agent, and the
dispatch wake.

## Verification

- `cd server && npx vitest run src/services/heartbeat/queue.test.ts -t
"reassigned owner|another agent's live run|next owner once"` — the hold
and the wake tests fail on master and pass here.
- `cd server && npx vitest run src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/__tests__/heartbeat-workspace-busy.test.ts` (all passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- A queued run on a task waits while another agent's terminal native run
on that task is still settling in this process. That wait ends when the
executor finishes, and the new wake re-drives dispatch. The queued-run
liveness redrive still covers a lost signal.
- The wake is event-driven: one indexed query when a native executor
settles, and one notification only when another agent waits on that
task. It adds no polling.
- Previous-release saved state: no persisted shape changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 23:53:49 -06:00
1 parent f82ec65800
commit 99d9e608f4
3 files changed
+76 -3

No files matched your search

+6
View File
@@ -110,6 +110,7 @@ import {
shouldQueueFollowupForRunningIssueWake,
DEFERRED_WAKE_CONTEXT_KEY,
createHeartbeatQueue,
notifyOtherAgentsQueuedOnSettledIssue,
unsavedWakeAdmissionCount,
} from "./heartbeat/queue.js";
export {
@@ -4321,6 +4322,11 @@ export function heartbeatService(
});
}
await startNextQueuedRunForAgent(run.agentId);
if (latestRun && isHeartbeatRunTerminalStatus(latestRun.status)) {
await notifyOtherAgentsQueuedOnSettledIssue(db, latestRun).catch(err => {
logger.warn({ err, runId: run.id }, "failed to wake the task's next owner after execution cleanup");
});
}
}
}
}
+52 -1
View File
@@ -10,7 +10,7 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro
import { createPostgresWakeQueueAdapter, createWakeAdmissionWriter } from "../../modules/wake-queue/adapters/postgres.js";
import { createReleaseIssueExecution } from "../../modules/wake-queue/application/use-cases.js";
import { closeProcessRunAdmission, resetProcessRunAdmissionForTests } from "./run-admission.js";
import { createHeartbeatQueue, DEFERRED_WAKE_CONTEXT_KEY, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake, type HeartbeatQueueDependencies } from "./queue.js";
import { createHeartbeatQueue, DEFERRED_WAKE_CONTEXT_KEY, notifyOtherAgentsQueuedOnSettledIssue, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake, type HeartbeatQueueDependencies } from "./queue.js";
import { createWakeQueue } from "../../modules/wake-queue/index.js";
import { TransactionScope } from "../../modules/wake-queue/application/ports.js";
import { enrichWakeContextSnapshot, mergeCoalescedContextSnapshot } from "./run-preparation.js";
@@ -22,6 +22,7 @@ import { createHeartbeatRunControl } from "./run-control.js";
import { callbacks as runControlCallbacks } from "./__tests__/run-control-fixtures.js";
import { createHeartbeatQueueTestDependencies as callbacks } from "../../__tests__/helpers/heartbeat-queue.js";
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../delivery-work-notifications.js";
function guardedDatabase() {
const access = vi.fn(() => { throw new Error("Unexpected queue database access"); });
@@ -634,6 +635,56 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
expect(await queue.claimQueuedRun(successor)).toBeNull();
});
async function reassignedPredecessorFixture(previousStatus = "cancelled") {
const f = await fixture("queued");
await db.update(issues).set({ executionRunId: null, executionAgentNameKey: null, executionLockedAt: null })
.where(eq(issues.id, f.issue.id));
const [previousAgent] = await db.insert(agents).values({ companyId: f.company.id, name: "Previous owner",
role: "engineer", status: "idle", adapterType: "paperclip_runner", adapterConfig: {} }).returning();
const [previous] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: previousAgent.id,
status: previousStatus, runtimeMode: "native", invocationSource: "assignment", startedAt: new Date(),
finishedAt: previousStatus === "running" ? null : new Date(), nativeIssueId: f.issue.id,
errorCode: previousStatus === "cancelled" ? "issue_reassigned" : null,
contextSnapshot: { issueId: f.issue.id, wakeReason: "issue_assigned" } }).returning();
const [next] = await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user",
contextSnapshot: { issueId: f.issue.id, wakeReason: "issue_assigned" } })
.where(eq(heartbeatRuns.id, f.run.id)).returning();
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(f.agent);
deps.activeRunExecutions.add(previous.id);
return { ...f, previousAgent, previous, next, deps, queue: createHeartbeatQueue(db, deps) };
}
it("keeps the new owner's run queued while the reassigned owner's stopped run still settles", async () => {
const f = await reassignedPredecessorFixture();
expect(await f.queue.claimQueuedRun(f.next)).toBeNull();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.next.id)))[0])
.toMatchObject({ status: "queued", startedAt: null });
expect(f.deps.executeRun).not.toHaveBeenCalled();
f.deps.activeRunExecutions.delete(f.previous.id);
expect(await f.queue.claimQueuedRun(f.next)).toMatchObject({ id: f.next.id, status: "running" });
});
it("does not hold a run behind another agent's live run on the same task", async () => {
const f = await reassignedPredecessorFixture("running");
expect(await f.queue.claimQueuedRun(f.next)).toMatchObject({ id: f.next.id, status: "running" });
});
it("wakes queued dispatch for the task's next owner once the previous owner's execution settles", async () => {
const f = await reassignedPredecessorFixture();
let wakes = 0;
const unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.queuedRunDispatch, () => { wakes += 1; });
try {
expect(await notifyOtherAgentsQueuedOnSettledIssue(db, f.previous)).toBe(true);
await vi.waitFor(() => expect(wakes).toBeGreaterThan(0));
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.next.id));
expect(await notifyOtherAgentsQueuedOnSettledIssue(db, f.previous)).toBe(false);
} finally {
unsubscribe();
}
});
it("keeps a retry queued after failed cleanup and until its local executor releases", async () => {
const f = await pendingPredecessorFixture();
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "failed" })
+18 -2
View File
@@ -15,7 +15,7 @@ import { notifyExecutionRunWork } from "../execution-control-notifications.js";
import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../deferred-execution-wait.js";
import { assignmentWakePayloadSql, isAssignmentWakePayload, notifyDeferredExecutionWork } from "../deferred-execution-notifications.js";
import type { DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js";
import { notifyRunDispatch } from "../run-dispatch-notifications.js";
import { notifyQueuedRunDispatch, notifyRunDispatch } from "../run-dispatch-notifications.js";
import { notifyStatusCardIssueChange } from "../status-card-notifications.js";
import { notifyRoutineExecutionStatus } from "../routine-execution-status-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
@@ -200,6 +200,7 @@ import type { createHeartbeatRunState } from "./run-state.js";
import type { createHeartbeatRunPreparation } from "./run-preparation.js";
import type { HeartbeatRetryDependencies } from "./retries.js";
import type { HeartbeatRecoveryDependencies, createHeartbeatRecovery } from "./recovery.js";
import { HEARTBEAT_RUN_TERMINAL_STATUSES } from "./recovery.js";
import { executeIssuePostCommitActions, type IssuePostCommitAction, type issueService } from "../issues.js";
import type { issueTreeControlService } from "../issue-tree-control.js";
import type { budgetService } from "../budgets.js";
@@ -418,6 +419,21 @@ class RunAdmissionClosedDuringClaim extends Error {}
const unsavedWakeAdmissions = new Set<object>();
export function unsavedWakeAdmissionCount() { return unsavedWakeAdmissions.size; }
export async function notifyOtherAgentsQueuedOnSettledIssue(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect,
"companyId" | "agentId" | "runtimeMode" | "contextSnapshot">): Promise<boolean> {
const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
if (run.runtimeMode !== "native" || !issueId) return false;
const [waiting] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, run.companyId),
eq(heartbeatRuns.status, "queued"),
ne(heartbeatRuns.agentId, run.agentId),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,
)).limit(1);
if (!waiting) return false;
await db.transaction(tx => notifyQueuedRunDispatch(tx, { companyId: run.companyId }));
return true;
}
export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDependencies) {
const {
setRunStatusFromLive,
@@ -1171,7 +1187,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
.from(heartbeatRuns)
.where(and(
eq(heartbeatRuns.companyId, run.companyId),
eq(heartbeatRuns.agentId, run.agentId),
or(eq(heartbeatRuns.agentId, run.agentId), inArray(heartbeatRuns.status, [...HEARTBEAT_RUN_TERMINAL_STATUSES])),
eq(heartbeatRuns.runtimeMode, "native"),
inArray(heartbeatRuns.id, [...activeRunExecutions]),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,