From 99d9e608f4cfa605bfa2052fa57e77cb27a5c83b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 12 Nov 2026 23:53:49 -0600 Subject: [PATCH] fix(heartbeat): hold the new owner's run while the reassigned owner's run settles (#21301) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a person reassigns a task, the old owner's native run is cancelled and the new owner's run is queued > - The old run's executor keeps closing its provider session for several seconds after the run is marked cancelled > - #21031 holds a queued run while the task's previous session close is pending. That hold reads the runner's cleanup gate, which exists only after the close begins > - The stale-lock sweep cleared the task lock as soon as the old run was terminal. The new owner's run was then claimed before the close began, met the close as it escalated, and failed with `native_session_cleanup_quarantined` > - This pull request keeps the new owner's run queued while the old owner's stopped run still settles in this process, and wakes it when that executor finishes > - The benefit is that a reassignment never fails the new owner's run and needs no Retry ## Linked Issues or Issue Description Refs #21031 **What happened?** In the paid chat eval `tasks.edit-properties`, the old owner's run `6cf2ee38` was cancelled for reassignment at 04:03:47.611. Its process stopped only at 04:03:57. The stale-lock sweep cleared the task lock at 04:03:47. The new owner's run was claimed at 04:03:48.626. The runner admission waited for the old close, which escalated to operator-required at 04:03:56, and the new run failed with `native_session_cleanup_quarantined`. Why #21031 did not apply: its hold reads the runner's in-memory cleanup gate. At claim time the old executor had not started its session close, so no gate existed. The existing `predecessor_settling_in_process` hold covers only the same agent, and a reassignment changes the agent. #21260 does not touch this path: its new checkpoint runs only before a turn in a recovered session, and both sessions here were new. **Steps to reproduce** 1. Run a native task for agent A. 2. Reassign it to agent B. A's run is cancelled, but its executor is still settling. 3. The stale-lock sweep clears the task lock. 4. B's queued run is claimed before A's session close begins. **Expected behavior** B's run stays queued until A's executor has finished. It starts by itself after that, then #21031's gate covers any close that is still settling. ## What Changed - `server/src/services/heartbeat/queue.ts`: the `predecessor_settling_in_process` hold in `claimQueuedRun` also holds for another agent's native run on the same task when that run is terminal but its executor is still active in this process. A live run of another agent (for example a reviewer) does not hold it. - `server/src/services/heartbeat/queue.ts`: `notifyOtherAgentsQueuedOnSettledIssue` sends a queued-run dispatch signal when a native executor settles and another agent has a queued run on the same task. It does nothing when no run waits. - `server/src/services/heartbeat.ts`: the executor calls it after its cleanup, next to the existing same-agent `startNextQueuedRunForAgent`. - `server/src/services/heartbeat/queue.test.ts`: the exact ordering (cancelled old-owner run still in the executor set, lock cleared, new owner's run queued), a control for a live run of another agent, and the dispatch wake. ## Verification - `cd server && npx vitest run src/services/heartbeat/queue.test.ts -t "reassigned owner|another agent's live run|next owner once"` — the hold and the wake tests fail on master and pass here. - `cd server && npx vitest run src/services/heartbeat/queue.test.ts src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts src/services/queued-run-dispatch-liveness.test.ts src/__tests__/heartbeat-workspace-busy.test.ts` (all passed). - `cd server && npx tsc --noEmit` passes. - Paid E2E: see the combined-branch run in the comments. ## Risks - A queued run on a task waits while another agent's terminal native run on that task is still settling in this process. That wait ends when the executor finishes, and the new wake re-drives dispatch. The queued-run liveness redrive still covers a lost signal. - The wake is event-driven: one indexed query when a native executor settles, and one notification only when another agent waits on that task. It adds no polling. - Previous-release saved state: no persisted shape changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- server/src/services/heartbeat.ts | 6 +++ server/src/services/heartbeat/queue.test.ts | 53 ++++++++++++++++++++- server/src/services/heartbeat/queue.ts | 20 +++++++- 3 files changed, 76 insertions(+), 3 deletions(-) diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index a860dae7cc9..bd43b03ce1c 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -110,6 +110,7 @@ import { shouldQueueFollowupForRunningIssueWake, DEFERRED_WAKE_CONTEXT_KEY, createHeartbeatQueue, + notifyOtherAgentsQueuedOnSettledIssue, unsavedWakeAdmissionCount, } from "./heartbeat/queue.js"; export { @@ -4321,6 +4322,11 @@ export function heartbeatService( }); } await startNextQueuedRunForAgent(run.agentId); + if (latestRun && isHeartbeatRunTerminalStatus(latestRun.status)) { + await notifyOtherAgentsQueuedOnSettledIssue(db, latestRun).catch(err => { + logger.warn({ err, runId: run.id }, "failed to wake the task's next owner after execution cleanup"); + }); + } } } } diff --git a/server/src/services/heartbeat/queue.test.ts b/server/src/services/heartbeat/queue.test.ts index 168b0558bf6..e54dd59825f 100644 --- a/server/src/services/heartbeat/queue.test.ts +++ b/server/src/services/heartbeat/queue.test.ts @@ -10,7 +10,7 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro import { createPostgresWakeQueueAdapter, createWakeAdmissionWriter } from "../../modules/wake-queue/adapters/postgres.js"; import { createReleaseIssueExecution } from "../../modules/wake-queue/application/use-cases.js"; import { closeProcessRunAdmission, resetProcessRunAdmissionForTests } from "./run-admission.js"; -import { createHeartbeatQueue, DEFERRED_WAKE_CONTEXT_KEY, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake, type HeartbeatQueueDependencies } from "./queue.js"; +import { createHeartbeatQueue, DEFERRED_WAKE_CONTEXT_KEY, notifyOtherAgentsQueuedOnSettledIssue, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake, type HeartbeatQueueDependencies } from "./queue.js"; import { createWakeQueue } from "../../modules/wake-queue/index.js"; import { TransactionScope } from "../../modules/wake-queue/application/ports.js"; import { enrichWakeContextSnapshot, mergeCoalescedContextSnapshot } from "./run-preparation.js"; @@ -22,6 +22,7 @@ import { createHeartbeatRunControl } from "./run-control.js"; import { callbacks as runControlCallbacks } from "./__tests__/run-control-fixtures.js"; import { createHeartbeatQueueTestDependencies as callbacks } from "../../__tests__/helpers/heartbeat-queue.js"; +import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../delivery-work-notifications.js"; function guardedDatabase() { const access = vi.fn(() => { throw new Error("Unexpected queue database access"); }); @@ -634,6 +635,56 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => { expect(await queue.claimQueuedRun(successor)).toBeNull(); }); + async function reassignedPredecessorFixture(previousStatus = "cancelled") { + const f = await fixture("queued"); + await db.update(issues).set({ executionRunId: null, executionAgentNameKey: null, executionLockedAt: null }) + .where(eq(issues.id, f.issue.id)); + const [previousAgent] = await db.insert(agents).values({ companyId: f.company.id, name: "Previous owner", + role: "engineer", status: "idle", adapterType: "paperclip_runner", adapterConfig: {} }).returning(); + const [previous] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: previousAgent.id, + status: previousStatus, runtimeMode: "native", invocationSource: "assignment", startedAt: new Date(), + finishedAt: previousStatus === "running" ? null : new Date(), nativeIssueId: f.issue.id, + errorCode: previousStatus === "cancelled" ? "issue_reassigned" : null, + contextSnapshot: { issueId: f.issue.id, wakeReason: "issue_assigned" } }).returning(); + const [next] = await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user", + contextSnapshot: { issueId: f.issue.id, wakeReason: "issue_assigned" } }) + .where(eq(heartbeatRuns.id, f.run.id)).returning(); + const deps = callbacks(db); + deps.getAgent.mockResolvedValue(f.agent); + deps.activeRunExecutions.add(previous.id); + return { ...f, previousAgent, previous, next, deps, queue: createHeartbeatQueue(db, deps) }; + } + + it("keeps the new owner's run queued while the reassigned owner's stopped run still settles", async () => { + const f = await reassignedPredecessorFixture(); + expect(await f.queue.claimQueuedRun(f.next)).toBeNull(); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.next.id)))[0]) + .toMatchObject({ status: "queued", startedAt: null }); + expect(f.deps.executeRun).not.toHaveBeenCalled(); + + f.deps.activeRunExecutions.delete(f.previous.id); + expect(await f.queue.claimQueuedRun(f.next)).toMatchObject({ id: f.next.id, status: "running" }); + }); + + it("does not hold a run behind another agent's live run on the same task", async () => { + const f = await reassignedPredecessorFixture("running"); + expect(await f.queue.claimQueuedRun(f.next)).toMatchObject({ id: f.next.id, status: "running" }); + }); + + it("wakes queued dispatch for the task's next owner once the previous owner's execution settles", async () => { + const f = await reassignedPredecessorFixture(); + let wakes = 0; + const unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.queuedRunDispatch, () => { wakes += 1; }); + try { + expect(await notifyOtherAgentsQueuedOnSettledIssue(db, f.previous)).toBe(true); + await vi.waitFor(() => expect(wakes).toBeGreaterThan(0)); + await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.next.id)); + expect(await notifyOtherAgentsQueuedOnSettledIssue(db, f.previous)).toBe(false); + } finally { + unsubscribe(); + } + }); + it("keeps a retry queued after failed cleanup and until its local executor releases", async () => { const f = await pendingPredecessorFixture(); await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "failed" }) diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index e55b996a0aa..fe4adc984b1 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -15,7 +15,7 @@ import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../deferred-execution-wait.js"; import { assignmentWakePayloadSql, isAssignmentWakePayload, notifyDeferredExecutionWork } from "../deferred-execution-notifications.js"; import type { DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js"; -import { notifyRunDispatch } from "../run-dispatch-notifications.js"; +import { notifyQueuedRunDispatch, notifyRunDispatch } from "../run-dispatch-notifications.js"; import { notifyStatusCardIssueChange } from "../status-card-notifications.js"; import { notifyRoutineExecutionStatus } from "../routine-execution-status-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; @@ -200,6 +200,7 @@ import type { createHeartbeatRunState } from "./run-state.js"; import type { createHeartbeatRunPreparation } from "./run-preparation.js"; import type { HeartbeatRetryDependencies } from "./retries.js"; import type { HeartbeatRecoveryDependencies, createHeartbeatRecovery } from "./recovery.js"; +import { HEARTBEAT_RUN_TERMINAL_STATUSES } from "./recovery.js"; import { executeIssuePostCommitActions, type IssuePostCommitAction, type issueService } from "../issues.js"; import type { issueTreeControlService } from "../issue-tree-control.js"; import type { budgetService } from "../budgets.js"; @@ -418,6 +419,21 @@ class RunAdmissionClosedDuringClaim extends Error {} const unsavedWakeAdmissions = new Set(); export function unsavedWakeAdmissionCount() { return unsavedWakeAdmissions.size; } +export async function notifyOtherAgentsQueuedOnSettledIssue(db: Db, run: Pick): Promise { + const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId); + if (run.runtimeMode !== "native" || !issueId) return false; + const [waiting] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, run.companyId), + eq(heartbeatRuns.status, "queued"), + ne(heartbeatRuns.agentId, run.agentId), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`, + )).limit(1); + if (!waiting) return false; + await db.transaction(tx => notifyQueuedRunDispatch(tx, { companyId: run.companyId })); + return true; +} + export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDependencies) { const { setRunStatusFromLive, @@ -1171,7 +1187,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende .from(heartbeatRuns) .where(and( eq(heartbeatRuns.companyId, run.companyId), - eq(heartbeatRuns.agentId, run.agentId), + or(eq(heartbeatRuns.agentId, run.agentId), inArray(heartbeatRuns.status, [...HEARTBEAT_RUN_TERMINAL_STATUSES])), eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.id, [...activeRunExecutions]), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,