From 89f486cdabe95628e698c594154194f7a36f1920 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 13:55:38 -0600 Subject: [PATCH] fix(heartbeat): run one dispatch pass per agent, back off refused claims, and enforce the run limit in the claim (#21433) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts an agent's queued runs. Each trigger (a run finishes, a wake arrives, a sweep runs) asks for one dispatch pass of that agent > - An agent could run more runs than its `maxConcurrentRuns`, dispatch passes could stack up without a limit, and every pass offered the same unclaimable runs again > - On hosted staging (e23549cd9c), one agent with limit 20 had 20–35 runs `running` and 170–270 queued. The logs showed 375 "offered a free slot but could not be claimed" lines and 117 "claim was rejected" lines in 10 minutes > - This pull request runs one pass per agent at a time, folds triggers into one follow-up pass, backs off refused claims, enforces the limit in the claim transaction, and reads only the columns the pass needs > - The benefit is that an agent never runs past its limit (also across two servers during a deploy), and a burst of triggers costs one or two passes, not one pass per trigger ## Linked Issues or Issue Description Refs #21391, #21242, #21234, #21277, #21324, #21349 **What happened?** On hosted staging (Core e23549cd9c), one agent ("Codie", `maxConcurrentRuns` 20) had 170–270 queued runs and 20–35 runs `running`, so it was over its limit. The logs showed a refused claim offered again on every pass. This pull request is about correctness and load. It is not the cause of the staging out-of-memory crashes: a separate leak (`sessionToolAuthorityEpochs` in `native-session-executor.ts`, about 3 MB per finished native run) explains those, and another change fixes it. Live again at 17:02Z on staging (Core 04f12fd2be, which includes #21391): the same agent had 52 runs `running` against `maxConcurrentRuns` 40. All 52 started between 16:55Z and 17:02Z, right after a deploy released the backlog, with 112 runs still queued. So overlapping passes exceed the limit with claimable runs, not only with unclaimable ones. **Why did it happen?** 1. `withAgentStartLock` chained each trigger behind the previous one and measured its 30 s stale limit from when the predecessor was chained, not from when it started. When triggers arrived faster than one per pass, every trigger after about 30 s of backlog started its own pass next to the others. 2. Each pass counted free slots by itself, outside any lock that other passes or another server see. Overlapping passes and two servers during a deploy swap each claimed up to the limit. 3. Each pass read every queued run with `select *`, including the large `contextSnapshot`, and offered every refused run again with no backoff. A local reproduction of the staging shape by another session (250 unclaimable queued runs with 8 KB snapshots, one trigger per second, on e23549cd9c and again on master e7e28ba241): passes that run at the same time went 1 → 12 → 39 → 79 → 139 at t = 30/45/75/116/178 s, only 4 passes finished in 178 s, and post-GC heap grew by about 2.8 MB per stacked pass (519 MB → 1,061 MB at 190 passes, mostly the parsed `heartbeat_runs` rows and the dispatcher facts each pass holds). 242 s after the triggers stopped, 216 passes were still outstanding. **Expected behavior** At most one dispatch pass per agent runs at a time. Triggers that arrive during a pass cause one follow-up pass. The `running` count never goes above `maxConcurrentRuns`. A refused run is offered again after a bounded wait, and no queued run is stranded. **Steps to reproduce** 1. Give an agent a limit of 4, 12 aged queued runs that the chat-control gate refuses, and 12 claimable runs. 2. Start a pass. While its first claim waits, send 40 triggers, each after more than 30 s of the old lock's clock. 3. Count the passes, the refused offers, and the highest `running` count. ## What Changed - `server/src/services/agent-start-lock.ts`: replaces the stale-bypass lock with `dispatchForAgent`. One pass runs per agent. Triggers during a pass merge into one follow-up pass that all their callers share. A trigger from inside the agent's own pass does not wait on itself. A pass has a 60 s deadline on a monotonic clock, measured from when the pass started (not from when a waiter was chained). At the deadline the pass stops claiming (an abort signal) and the follow-up may start. A trigger that arrives after the deadline also ends the pass, so a stalled timer cannot keep an agent blocked. Post-pass effects run after the agent is free, as before. - `server/src/services/heartbeat/run-slot-admission.ts` (new): `assertAgentRunSlotFree` counts the agent's `running` runs and compares them with `maxConcurrentRunsSql()` inside the agent-scoped advisory lock that every claim already takes (`withExternalAdmissionGuard`). It throws `AgentRunSlotsFull` when the agent is full. The count leaves out the run that the claimed run replaces (`retryOfRunId` or `previousRunId`), so a successor that starts while its predecessor finishes is not refused. - `server/src/services/heartbeat/queue.ts`: - Every claim path (ordinary, queued comment, native review through `native-review-dispatch.ts`) checks the slot inside its claim transaction. A full agent leaves the run queued, and the pass stops. - The pass reads only the columns the dispatcher needs, with `issueId`, `wakeReason` and `previousRunId` taken from `contextSnapshot` in SQL. It reads the full row only for the run it is about to claim. - A refused claim (declined or 4xx) of a run that has waited more than an hour (the existing aging limit) is not offered again for 5 s, then 10, 20, 40, and at most 60 s. A person's run of that age waits 2 s. A run queued for less than an hour is never backed off, so a successor or retry that waits for its predecessor starts as soon as the predecessor ends. A backoff also ends when the run's task changes (status, `updatedAt` or execution run) or when a run of that task ends in this process. A backed-off run does not take the oldest-run share. The queued-run sweep (`resumeQueuedRuns`, `resumeQueuedRunsAfterDrain`, which the queued-run redrive calls) offers refused runs again regardless of the backoff. - A pass makes at most 24 claim attempts. When it reaches that limit it asks for one follow-up pass that skips every run the chain already tried, so the follow-up reaches runs further down the line. The chain ends when a pass does not reach the limit, so refused runs cannot cause a continuous chain of passes. Only this automatic follow-up carries the tried list. A wake, a completion or another fresh trigger that merges into it resets the list, so a run whose blocker cleared during the chain is offered again. - Admission, suppression and cutoff are read once per pass, not once per trigger. - Seam for a skip before the claim: in `runQueuedDispatchPass` (`claimQueuedRunsForAgent`), right after the skip line (`if (triedRunIds.has(candidate.id) || (!retryRefused && claimBackedOff(candidate.id, new Date(), taskStamp))) continue;`), before the claim-budget check, `claimAttempts += 1` and the full-row read. A `continue` there spends no claim attempt, reads no row, does not enter the claim backoff or the aged-decline note, and does not call `dispatcher.claimed`, so the run keeps its place and its oldest-run share. The dispatcher exposes `dispatcher.personOrigin(run)` (the person-origin fact it already derives) and `dispatcher.personRun(run)`. The candidate rows keep `id`, `companyId`, `agentId`, `status`, `invocationSource`, `retryOfRunId`, `wakeupRequestId`, `scheduledRetryAt`, `scheduledRetryAttempt`, `scheduledRetryReason`, `createdAt`, `startedAt`, and `contextSnapshot.issueId/wakeReason/previousRunId`. - Each pass is registered with the execution drain (`drainActiveRunExecutions`) until it ends or reaches its deadline. - Tests: new `server/src/services/agent-start-lock.test.ts` and `server/src/__tests__/queued-run-dispatch-stacking.test.ts`. `server/src/__tests__/heartbeat-start-lock.test.ts` is removed: it asserted the 30 s stale bypass that this pull request removes. `workspace-recovery-preparation-dispatch.test.ts` holds and counts passes through the new function. The chat-control fallback lookup from #21391 (`issueId = X OR nativeIssueId = X OR contextSnapshot->>'issueId' = X`) was measured on embedded Postgres with 200,000 runs in one company. Postgres uses a BitmapOr of three existing indexes (`heartbeat_runs_company_issue_created_idx`, `heartbeat_runs_company_native_issue_contract_id_uq`, `heartbeat_runs_company_ctx_issue_created_idx`), not a sequential scan: 0.33–0.45 ms per call for a recent previous run, a 30-day-old previous run, and no previous run. It is not changed. ## Verification Red/green, `server/src/__tests__/queued-run-dispatch-stacking.test.ts` (embedded Postgres, real heartbeat queue, limit 4, 12 refused aged runs, 12 claimable runs, 41 triggers): | Measure | master | this pull request | |---|---|---| | Passes | 42 | 2 | | Peak passes at the same time | 40 | 1 | | Peak claims in flight | 9 | 1 | | Refused-claim offers per trigger | 12 | 0.29 (12 in total) | | Peak `running` (limit 4) | 8 | 4 | | Passes outstanding after the burst | — | 0 | - "waits a bounded time before offering a declined/rejected run again": on master 10 triggers offer the refused run 10 times. With the fix they offer it once, and once the backoff passes it is offered again. The rejected case uses the 422 "Unable to resolve responsible user" path from the staging reproduction. - "starts a run that becomes ready during a chain of passes when a fresh trigger arrives": red at 3d48c4bed2, green now. - "stops chaining passes once every refused run was tried": 60 refused runs and one trigger cause exactly 3 passes, then no more. - "starts runnable work queued behind more refused person runs than one pass tries": 30 refused person runs ahead of 3 runnable runs, with free slots. The runnable runs start after one trigger, in at most 3 passes. At 68e64a2de9 the same 24 person runs were offered on every pass and the runnable runs never started. - `server/src/services/agent-start-lock.test.ts`: one pass for a burst of 50 triggers plus one follow-up, no second pass after 45 s, a hung pass stopped at its deadline without overlap, no self-wait, settle after release, and errors passed to the caller. - `node server/scripts/typecheck-layers.mjs`: pass. - Vitest (embedded Postgres): `heartbeat/queue.test.ts` (102), `recovery-wake-herd.test.ts` (35), `heartbeat-queued-run-claim-isolation.test.ts`, `workspace-recovery-preparation-dispatch.test.ts`, `heartbeat-restart-claim-lapse.test.ts`, `issue-reassignment-queued-run.test.ts`, `queued-comment-consumption.test.ts`, `issue-queued-comment-queue.test.ts`, `chat-run-milestone-work.test.ts`. ## Risks - Refusals of runs queued for less than an hour are not backed off. A backoff there delayed successors that wait for in-process predecessor ownership, which can end without a task change (two CI tests failed with it). Per trigger, such a run costs at most one claim attempt. - A refused run that has waited more than an hour now waits up to 60 s (a person's run 2 s) before an event-driven pass offers it again, unless its task changes. The sweep and the redrive ignore the backoff. - A successor may start while the run it replaces is still `running`. The limit check leaves that one predecessor out of the count. - Claims of an agent that is at its limit now fail inside the transaction. A run that `executeRun` claims directly (for example a native resume) stays queued when the agent is full, and the next pass starts it. - A pass that passes its 60 s deadline can still finish one in-flight claim next to the follow-up pass. The limit check in the transaction keeps the `running` count within the limit. - No schema or migration change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../__tests__/heartbeat-start-lock.test.ts | 30 -- .../queued-run-dispatch-stacking.test.ts | 324 +++++++++++++++++ server/src/services/agent-start-lock.test.ts | 155 +++++++++ server/src/services/agent-start-lock.ts | 127 ++++--- server/src/services/heartbeat.ts | 1 + server/src/services/heartbeat/queue.ts | 329 ++++++++++++------ .../services/heartbeat/run-slot-admission.ts | 28 ++ .../native-runtime/native-review-dispatch.ts | 6 +- ...pace-recovery-preparation-dispatch.test.ts | 5 +- 9 files changed, 831 insertions(+), 174 deletions(-) delete mode 100644 server/src/__tests__/heartbeat-start-lock.test.ts create mode 100644 server/src/__tests__/queued-run-dispatch-stacking.test.ts create mode 100644 server/src/services/agent-start-lock.test.ts create mode 100644 server/src/services/heartbeat/run-slot-admission.ts diff --git a/server/src/__tests__/heartbeat-start-lock.test.ts b/server/src/__tests__/heartbeat-start-lock.test.ts deleted file mode 100644 index 29aefdfb6da..00000000000 --- a/server/src/__tests__/heartbeat-start-lock.test.ts +++ /dev/null @@ -1,30 +0,0 @@ -import { randomUUID } from "node:crypto"; -import { afterEach, describe, expect, it, vi } from "vitest"; -import { withAgentStartLock } from "../services/agent-start-lock.ts"; - -describe("heartbeat agent start lock", () => { - afterEach(() => { - vi.useRealTimers(); - }); - - it("does not let a stale start lock freeze later queued-run starts", async () => { - vi.useFakeTimers(); - - const agentId = randomUUID(); - const firstStart = vi.fn(() => new Promise(() => undefined)); - const secondStart = vi.fn(async () => "started"); - - void withAgentStartLock(agentId, firstStart); - await Promise.resolve(); - expect(firstStart).toHaveBeenCalledTimes(1); - - const secondStartResult = withAgentStartLock(agentId, secondStart); - await Promise.resolve(); - expect(secondStart).not.toHaveBeenCalled(); - - await vi.advanceTimersByTimeAsync(30_000); - - await expect(secondStartResult).resolves.toBe("started"); - expect(secondStart).toHaveBeenCalledTimes(1); - }); -}); diff --git a/server/src/__tests__/queued-run-dispatch-stacking.test.ts b/server/src/__tests__/queued-run-dispatch-stacking.test.ts new file mode 100644 index 00000000000..db0744d53f8 --- /dev/null +++ b/server/src/__tests__/queued-run-dispatch-stacking.test.ts @@ -0,0 +1,324 @@ +import { randomUUID } from "node:crypto"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { + agentWakeupRequests, agents, authUsers, companies, companyMemberships, createDb, heartbeatRuns, issues, +} from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { heartbeatService } from "../services/heartbeat.ts"; +import { runningProcesses } from "../adapters/index.ts"; +import { logger } from "../middleware/logger.js"; +import { isAgentDispatchActive } from "../services/agent-start-lock.js"; + +const held = vi.hoisted(() => { + const state = { + release: () => undefined as void, + released: Promise.resolve(), + reset() { + state.released = new Promise((resolve) => { state.release = resolve; }); + }, + }; + return state; +}) + +const passes = vi.hoisted(() => ({ started: 0, active: 0, peak: 0 })); + +vi.mock("../adapters/index.ts", async () => { + const actual = await vi.importActual("../adapters/index.ts"); + return { + ...actual, + getServerAdapter: vi.fn(() => ({ + supportsLocalAgentJwt: false, + execute: async () => { + await held.released; + return { exitCode: 0, signal: null, timedOut: false, errorMessage: null, summary: "done", provider: "test", model: "test" }; + }, + })), + }; +}); + +vi.mock("../services/agent-start-lock.js", async () => { + const actual = await vi.importActual("../services/agent-start-lock.js"); + const measure = async (pass: () => Promise) => { + passes.started += 1; + passes.active += 1; + passes.peak = Math.max(passes.peak, passes.active); + try { + return await pass(); + } finally { + passes.active -= 1; + } + }; + return { + ...actual, + dispatchForAgent: (agentId: string, spec: Parameters[1]) => + actual.dispatchForAgent(agentId, { ...spec, run: (request, signal) => measure(() => spec.run(request, signal)) }), + }; +}); + +const support = await getEmbeddedPostgresTestSupport(); +const suite = support.supported ? describe : describe.skip; + +const CAP = 4; +const UNCLAIMABLE = 12; +const CLAIMABLE = 3 * CAP; +const TRIGGERS = 40; + +suite("queued-run dispatch under a trigger burst", () => { + let db!: ReturnType; + let temporary: Awaited> | null = null; + let companyId = "", userId = ""; + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("paperclip-queued-dispatch-stacking-"); + db = createDb(temporary.connectionString); + await db.execute(sql`create table running_peaks (agent_id uuid not null, running int not null)`); + await db.execute(sql`create function record_running_peak() returns trigger language plpgsql as $$ + begin + insert into running_peaks select new.agent_id, count(*) from heartbeat_runs where agent_id = new.agent_id and status = 'running'; + return new; + end $$`); + await db.execute(sql`create trigger record_running_peak after update of status on heartbeat_runs + for each row when (new.status = 'running') execute function record_running_peak()`); + }, 60_000); + + afterEach(async () => { + vi.restoreAllMocks(); + held.release(); + runningProcesses.clear(); + await db.execute(sql`truncate table ${companies} cascade`); + }); + + afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); }); + + async function seedAgent() { + userId = randomUUID(); + companyId = randomUUID(); + const agentId = randomUUID(); + await db.insert(authUsers).values({ id: userId, name: "Board", email: `${userId}@example.test`, + emailVerified: true, createdAt: new Date(), updatedAt: new Date() }); + await db.insert(companies).values({ id: companyId, name: "Burst", issuePrefix: `B${companyId.replaceAll("-", "").slice(0, 6)}`, + requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: userId }); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "owner" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Codie", role: "engineer", status: "idle", + adapterType: "codex_local", adapterConfig: {}, + runtimeConfig: { heartbeat: { enabled: true, wakeOnDemand: true, maxConcurrentRuns: CAP } }, permissions: {} }); + return agentId; + } + + async function task(agentId: string) { + const [row] = await db.insert(issues).values({ companyId, title: "Continue the work", status: "in_progress", assigneeAgentId: agentId }).returning(); + return row!; + } + + async function finishedRun(agentId: string, issueId: string, minutesAgo: number) { + const finishedAt = new Date(Date.now() - minutesAgo * 60_000); + const [row] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId, invocationSource: "assignment", + status: "failed", errorCode: "server_shutdown_interrupted", startedAt: new Date(finishedAt.getTime() - 60_000), finishedAt, + createdAt: new Date(finishedAt.getTime() - 120_000), contextSnapshot: { issueId, wakeReason: "issue_assigned" } }).returning(); + return row!; + } + + async function queuedContinuation(agentId: string, input: { minutesAgo: number; wakeReason: string; sourceIssueId?: string }) { + const issue = await task(agentId); + const source = await finishedRun(agentId, input.sourceIssueId ?? issue.id, input.minutesAgo + 5); + const [wake] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", triggerDetail: "system", + reason: input.wakeReason, status: "queued", requestedByActorType: "system", requestedByActorId: null, + payload: { issueId: issue.id } }).returning(); + const [row] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: issue.id, invocationSource: "automation", + triggerDetail: "system", status: "queued", retryOfRunId: source.id, wakeupRequestId: wake!.id, + createdAt: new Date(Date.now() - input.minutesAgo * 60_000), + contextSnapshot: { issueId: issue.id, taskId: issue.id, wakeReason: input.wakeReason, previousRunId: source.id, + source: "issue.assigned_work_invariant", paperclipWorkspace: { notes: "x".repeat(16_000) } } }).returning(); + await db.update(agentWakeupRequests).set({ runId: row!.id }).where(eq(agentWakeupRequests.id, wake!.id)); + return row!; + } + + const peakRunning = async (agentId: string) => Number((await db.execute(sql`select coalesce(max(running), 0) as peak + from running_peaks where agent_id = ${agentId}::uuid`))[0]!.peak); + + it("runs one pass at a time for a burst of triggers, offers a refused run once, and never runs past the agent's limit", async () => { + held.reset(); + passes.started = 0; passes.active = 0; passes.peak = 0; + const agentId = await seedAgent(); + const otherTask = await task(agentId); + const unclaimable = new Set(); + for (let index = 0; index < UNCLAIMABLE; index += 1) { + unclaimable.add((await queuedContinuation(agentId, { minutesAgo: 180 - index, wakeReason: "issue_continuation_needed", + sourceIssueId: otherTask.id })).id); + } + for (let index = 0; index < CLAIMABLE; index += 1) { + await queuedContinuation(agentId, { minutesAgo: 30 - index, wakeReason: "issue_assigned" }); + } + + const realNow = Date.now.bind(Date); + let skewMs = 0; + vi.spyOn(Date, "now").mockImplementation(() => realNow() + skewMs); + const offers = new Map(); + let inFlightClaims = 0; + let peakInFlightClaims = 0; + const triggers: Array> = []; + let burstFired = false; + const heartbeat = heartbeatService(db, { + beforeChatControlRecoveryCheck: async ({ runId, stage }) => { + if (stage !== "claim") return; + offers.set(runId, (offers.get(runId) ?? 0) + 1); + inFlightClaims += 1; + peakInFlightClaims = Math.max(peakInFlightClaims, inFlightClaims); + try { + if (!burstFired) { + burstFired = true; + for (let index = 0; index < TRIGGERS; index += 1) { + skewMs += 31_000; + triggers.push(heartbeat.resumeQueuedRunsForAgent(companyId, agentId)); + await new Promise((resolve) => setTimeout(resolve, 100)); + } + skewMs = 0; + } + await new Promise((resolve) => setTimeout(resolve, 25)); + } finally { + inFlightClaims -= 1; + } + }, + }); + + try { + await heartbeat.resumeQueuedRunsForAgent(companyId, agentId); + await Promise.all(triggers); + const refusedOffers = [...unclaimable].reduce((sum, id) => sum + (offers.get(id) ?? 0), 0); + expect({ + passes: passes.started, + peakConcurrentPasses: passes.peak, + peakInFlightClaims, + refusedOffersPerTrigger: refusedOffers / (TRIGGERS + 1), + peakRunning: await peakRunning(agentId), + }).toEqual({ + passes: 2, + peakConcurrentPasses: 1, + peakInFlightClaims: 1, + refusedOffersPerTrigger: UNCLAIMABLE / (TRIGGERS + 1), + peakRunning: CAP, + }); + await vi.waitFor(() => { + expect(passes.active).toBe(0); + expect(isAgentDispatchActive(agentId)).toBe(false); + }, { timeout: 10_000 }); + + } finally { + held.release(); + await heartbeat.drainActiveRunExecutions(); + } + }, 120_000); + + it.each(["declined", "rejected"] as const)("waits a bounded time before offering a %s run again, and offers it again once that time passes", async (refusal) => { + const agentId = await seedAgent(); + if (refusal === "rejected") { + await db.update(companies).set({ defaultResponsibleUserId: null }).where(eq(companies.id, companyId)); + await db.delete(companyMemberships).where(eq(companyMemberships.companyId, companyId)); + } + const otherTask = await task(agentId); + const refused = await queuedContinuation(agentId, { minutesAgo: 180, wakeReason: "issue_continuation_needed", sourceIssueId: otherTask.id }); + const warn = vi.spyOn(logger, "warn"); + const offersOf = () => warn.mock.calls.filter(([fields, message]) => typeof message === "string" + && message.startsWith("long-queued heartbeat run was offered a free slot") + && (fields as { runId?: string }).runId === refused.id).length; + const heartbeat = heartbeatService(db); + try { + for (let index = 0; index < 10; index += 1) await heartbeat.resumeQueuedRunsForAgent(companyId, agentId); + expect(offersOf()).toBe(1); + await vi.waitFor(async () => { + await heartbeat.resumeQueuedRunsForAgent(companyId, agentId); + expect(offersOf()).toBe(2); + }, { timeout: 20_000, interval: 1_000 }); + } finally { + await heartbeat.drainActiveRunExecutions(); + } + }, 60_000); + + async function passesSettle() { + let last = -1; + await vi.waitFor(() => { + const settled = passes.started === last && !passes.active; + last = passes.started; + expect(settled).toBe(true); + }, { timeout: 30_000, interval: 1_000 }); + return passes.started; + } + + it("stops chaining passes once every refused run was tried", async () => { + passes.started = 0; passes.active = 0; passes.peak = 0; + const agentId = await seedAgent(); + const otherTask = await task(agentId); + for (let index = 0; index < 60; index += 1) { + await queuedContinuation(agentId, { minutesAgo: 180 - index, wakeReason: "issue_continuation_needed", sourceIssueId: otherTask.id }); + } + const heartbeat = heartbeatService(db); + try { + await heartbeat.resumeQueuedRunsForAgent(companyId, agentId); + expect(await passesSettle()).toBe(3); + expect(passes.peak).toBe(1); + } finally { + await heartbeat.drainActiveRunExecutions(); + } + }, 120_000); + + it("starts runnable work queued behind more refused person runs than one pass tries", async () => { + held.reset(); + passes.started = 0; passes.active = 0; passes.peak = 0; + const agentId = await seedAgent(); + const otherTask = await task(agentId); + for (let index = 0; index < 30; index += 1) { + await queuedContinuation(agentId, { minutesAgo: 90 - index, wakeReason: "task_retry", sourceIssueId: otherTask.id }); + } + const runnable: string[] = []; + for (let index = 0; index < 3; index += 1) { + runnable.push((await queuedContinuation(agentId, { minutesAgo: 30 - index, wakeReason: "issue_assigned" })).id); + } + const heartbeat = heartbeatService(db); + try { + await heartbeat.resumeQueuedRunsForAgent(companyId, agentId); + expect(await passesSettle()).toBeLessThanOrEqual(3); + const started = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns) + .where(sql`${heartbeatRuns.id} in (${sql.join(runnable.map(id => sql`${id}::uuid`), sql`, `)}) and ${heartbeatRuns.startedAt} is not null`); + expect(started).toHaveLength(runnable.length); + } finally { + held.release(); + await heartbeat.drainActiveRunExecutions(); + } + }, 120_000); + + it("starts a run that becomes ready during a chain of passes when a fresh trigger arrives", async () => { + held.reset(); + passes.started = 0; passes.active = 0; passes.peak = 0; + const agentId = await seedAgent(); + const otherTask = await task(agentId); + const ready = await queuedContinuation(agentId, { minutesAgo: 200, wakeReason: "issue_continuation_needed", sourceIssueId: otherTask.id }); + for (let index = 0; index < 30; index += 1) { + await queuedContinuation(agentId, { minutesAgo: 180 - index, wakeReason: "issue_continuation_needed", sourceIssueId: otherTask.id }); + } + let refusedOffers = 0; + let freshTrigger: Promise | null = null; + const heartbeat = heartbeatService(db, { + beforeChatControlRecoveryCheck: async ({ runId, stage }) => { + if (stage !== "claim" || runId === ready.id) return; + refusedOffers += 1; + if (refusedOffers !== 5) return; + const source = await finishedRun(agentId, ready.issueId!, 205); + await db.update(heartbeatRuns).set({ retryOfRunId: source.id, + contextSnapshot: { ...ready.contextSnapshot, previousRunId: source.id } }).where(eq(heartbeatRuns.id, ready.id)); + await db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, ready.issueId!)); + freshTrigger = heartbeat.resumeQueuedRunsForAgent(companyId, agentId); + }, + }); + try { + await heartbeat.resumeQueuedRunsForAgent(companyId, agentId); + await passesSettle(); + await freshTrigger; + const [row] = await db.select({ startedAt: heartbeatRuns.startedAt }).from(heartbeatRuns).where(eq(heartbeatRuns.id, ready.id)); + expect(row!.startedAt).not.toBeNull(); + } finally { + held.release(); + await heartbeat.drainActiveRunExecutions(); + } + }, 120_000); +}); diff --git a/server/src/services/agent-start-lock.test.ts b/server/src/services/agent-start-lock.test.ts new file mode 100644 index 00000000000..578967249f1 --- /dev/null +++ b/server/src/services/agent-start-lock.test.ts @@ -0,0 +1,155 @@ +import { randomUUID } from "node:crypto"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { + AGENT_DISPATCH_PASS_DEADLINE_MS, + dispatchForAgent, + isAgentDispatchActive, + type AgentDispatchPass, +} from "./agent-start-lock.js"; + +function gate() { + let open!: () => void; + const opened = new Promise((resolve) => { open = resolve; }); + return { opened, open }; +} + +function meter() { + const stats = { passes: 0, active: 0, peak: 0, requests: [] as boolean[] }; + const spec = (attach: boolean, body: (signal: AbortSignal) => Promise): AgentDispatchPass => ({ + request: attach, + idle: -1, + merge: (current, next) => current || next, + async run(request, signal) { + stats.passes += 1; + stats.active += 1; + stats.peak = Math.max(stats.peak, stats.active); + stats.requests.push(request); + const pass = stats.passes; + try { + await body(signal); + } finally { + stats.active -= 1; + } + return { value: pass }; + }, + }); + return { stats, spec }; +} + +describe("agent dispatch passes", () => { + afterEach(() => { + vi.useRealTimers(); + }); + + it("runs one pass per agent and folds every trigger that arrives during it into one follow-up pass", async () => { + const agentId = randomUUID(); + const { stats, spec } = meter(); + const first = gate(); + const firstPass = dispatchForAgent(agentId, spec(false, () => first.opened)); + const burst = Array.from({ length: 50 }, (_, index) => + dispatchForAgent(agentId, spec(index === 7, async () => undefined))); + expect(stats).toMatchObject({ passes: 1, peak: 1 }); + first.open(); + expect(await firstPass).toBe(1); + expect(new Set(await Promise.all(burst))).toEqual(new Set([2])); + expect(stats).toMatchObject({ passes: 2, peak: 1, requests: [false, true] }); + expect(isAgentDispatchActive(agentId)).toBe(false); + }); + + it("keeps a second pass out while a slow pass runs past the old 30 second stale limit", async () => { + vi.useFakeTimers(); + const agentId = randomUUID(); + const { stats, spec } = meter(); + const slow = gate(); + const firstPass = dispatchForAgent(agentId, spec(false, () => slow.opened)); + const triggers: Array> = []; + for (let second = 0; second < 45; second += 1) { + await vi.advanceTimersByTimeAsync(1_000); + triggers.push(dispatchForAgent(agentId, spec(false, async () => undefined))); + } + expect(stats).toMatchObject({ passes: 1, peak: 1 }); + slow.open(); + await firstPass; + await Promise.all(triggers); + expect(stats).toMatchObject({ passes: 2, peak: 1 }); + }); + + it("stops a hung pass at its deadline and lets the waiting trigger run without overlapping it", async () => { + vi.useFakeTimers(); + const agentId = randomUUID(); + const { stats, spec } = meter(); + let hungSignal: AbortSignal | null = null; + void dispatchForAgent(agentId, spec(false, (signal) => { + hungSignal = signal; + return new Promise(() => undefined); + })); + const waiting = dispatchForAgent(agentId, spec(false, async () => undefined)); + await vi.advanceTimersByTimeAsync(AGENT_DISPATCH_PASS_DEADLINE_MS - 1); + expect(stats.passes).toBe(1); + expect(hungSignal!.aborted).toBe(false); + await vi.advanceTimersByTimeAsync(1); + expect(hungSignal!.aborted).toBe(true); + expect(await waiting).toBe(2); + expect(stats.passes).toBe(2); + expect(isAgentDispatchActive(agentId)).toBe(false); + expect(await dispatchForAgent(agentId, spec(false, async () => undefined))).toBe(3); + }); + + it("lets a trigger stop a hung pass by the pass's start time when its deadline timer has not run", async () => { + vi.useFakeTimers({ toFake: ["performance"] }); + const agentId = randomUUID(); + const { stats, spec } = meter(); + let hungSignal: AbortSignal | null = null; + void dispatchForAgent(agentId, spec(false, (signal) => { + hungSignal = signal; + return new Promise(() => undefined); + })); + vi.advanceTimersByTime(AGENT_DISPATCH_PASS_DEADLINE_MS); + expect(await dispatchForAgent(agentId, spec(false, async () => undefined))).toBe(2); + expect(hungSignal!.aborted).toBe(true); + expect(stats.passes).toBe(2); + }); + + it("does not wait on itself when a pass triggers its own agent", async () => { + const agentId = randomUUID(); + const { stats, spec } = meter(); + let nested: Promise | null = null; + const pass = dispatchForAgent(agentId, spec(false, async () => { + nested = dispatchForAgent(agentId, spec(false, async () => undefined)); + expect(await nested).toBe(-1); + })); + expect(await pass).toBe(1); + await vi.waitFor(() => expect(stats.passes).toBe(2)); + expect(stats.peak).toBe(1); + }); + + it("settles a pass's effects after the next pass may start", async () => { + const agentId = randomUUID(); + const order: string[] = []; + const release = gate(); + const pass = dispatchForAgent(agentId, { + request: null, idle: null, merge: () => null, + async run() { + await release.opened; + return { + value: null, + settle: async () => { + order.push(`settle:${isAgentDispatchActive(agentId)}`); + await dispatchForAgent(agentId, { request: null, idle: null, merge: () => null, + async run() { order.push("nested"); return { value: null }; } }); + }, + }; + }, + }); + release.open(); + await pass; + expect(order).toEqual(["settle:false", "nested"]); + }); + + it("passes a failed pass's error to its caller and frees the agent", async () => { + const agentId = randomUUID(); + const { spec } = meter(); + await expect(dispatchForAgent(agentId, spec(false, async () => { throw new Error("boom"); }))).rejects.toThrow("boom"); + expect(isAgentDispatchActive(agentId)).toBe(false); + }); +}); diff --git a/server/src/services/agent-start-lock.ts b/server/src/services/agent-start-lock.ts index 38cbaf74c6e..70f5db23284 100644 --- a/server/src/services/agent-start-lock.ts +++ b/server/src/services/agent-start-lock.ts @@ -1,48 +1,97 @@ +import { AsyncLocalStorage } from "node:async_hooks"; import { logger } from "../middleware/logger.js"; -const AGENT_START_LOCK_STALE_MS = 30_000; -const startLocksByAgent = new Map; startedAtMs: number }>(); +export const AGENT_DISPATCH_PASS_DEADLINE_MS = 60_000; -async function waitForAgentStartLock(agentId: string, lock: { promise: Promise; startedAtMs: number }) { - const elapsedMs = Date.now() - lock.startedAtMs; - const remainingMs = AGENT_START_LOCK_STALE_MS - elapsedMs; - if (remainingMs <= 0) { - logger.warn({ agentId, staleMs: elapsedMs }, "agent start lock stale; continuing queued-run start"); - return; - } +export type AgentDispatchPass = { + request: R; + idle: T; + merge(current: R, next: R): R; + run(request: R, signal: AbortSignal): Promise<{ value: T; settle?: () => Promise }>; + track?(pass: Promise): void; +}; - let timedOut = false; - let timeout: ReturnType | null = null; - await Promise.race([ - lock.promise, - new Promise((resolve) => { - timeout = setTimeout(() => { - timedOut = true; - resolve(); - }, remainingMs); - }), - ]); - if (timeout) clearTimeout(timeout); +type Deferred = { promise: Promise; resolve(value: T): void; reject(err: unknown): void }; +type FollowUp = { request: unknown; spec: AgentDispatchPass; done: Deferred }; +type AgentDispatchState = { followUp: FollowUp | null; startedAtMs: number; expire(): void }; - if (timedOut) { - logger.warn({ agentId, staleMs: AGENT_START_LOCK_STALE_MS }, "agent start lock timed out; continuing queued-run start"); - } +const dispatchStates = new Map(); +const dispatchingAgent = new AsyncLocalStorage(); + +function deferred(): Deferred { + let resolve!: (value: T) => void; + let reject!: (err: unknown) => void; + const promise = new Promise((res, rej) => { + resolve = res; + reject = rej; + }); + promise.catch(() => undefined); + return { promise, resolve, reject }; } -export async function withAgentStartLock(agentId: string, fn: () => Promise) { - const previous = startLocksByAgent.get(agentId); - const waitForPrevious = previous ? waitForAgentStartLock(agentId, previous) : Promise.resolve(); - const run = waitForPrevious.then(fn); - const marker = run.then( - () => undefined, - () => undefined, +export function isAgentDispatchActive(agentId: string) { + return dispatchStates.has(agentId); +} + +export function outsideAgentDispatch(fn: () => T): T { + return dispatchingAgent.exit(fn); +} + +export function dispatchForAgent(agentId: string, spec: AgentDispatchPass): Promise { + let state = dispatchStates.get(agentId); + if (state && performance.now() - state.startedAtMs >= AGENT_DISPATCH_PASS_DEADLINE_MS) { + state.expire(); + state = dispatchStates.get(agentId); + } + if (!state) return startPass(agentId, spec, spec.request); + if (state.followUp) { + state.followUp.request = spec.merge(state.followUp.request as R, spec.request); + state.followUp.spec = spec as AgentDispatchPass; + } else { + state.followUp = { request: spec.request, spec: spec as AgentDispatchPass, done: deferred() }; + } + if (dispatchingAgent.getStore() === agentId) return Promise.resolve(spec.idle); + return state.followUp.done.promise as Promise; +} + +function startPass(agentId: string, spec: AgentDispatchPass, request: R): Promise { + const controller = new AbortController(); + const expired = deferred(); + const state: AgentDispatchState = { followUp: null, startedAtMs: performance.now(), expire: () => expire() }; + dispatchStates.set(agentId, state); + let released = false; + const release = () => { + if (released) return; + released = true; + if (dispatchStates.get(agentId) === state) dispatchStates.delete(agentId); + const next = state.followUp; + state.followUp = null; + if (next) startPass(agentId, next.spec, next.request).then(next.done.resolve, next.done.reject); + }; + const expire = () => { + if (released) return; + clearTimeout(deadline); + logger.warn({ agentId, deadlineMs: AGENT_DISPATCH_PASS_DEADLINE_MS }, + "queued-run dispatch pass passed its deadline; it stops claiming and the next pass may start"); + controller.abort(); + release(); + expired.resolve(); + }; + const deadline = setTimeout(expire, AGENT_DISPATCH_PASS_DEADLINE_MS); + deadline.unref?.(); + const pass = dispatchingAgent.run(agentId, () => spec.run(request, controller.signal)).then( + async ({ value, settle }) => { + clearTimeout(deadline); + release(); + if (settle) await outsideAgentDispatch(settle); + return value; + }, + (err: unknown) => { + clearTimeout(deadline); + release(); + throw err; + }, ); - startLocksByAgent.set(agentId, { promise: marker, startedAtMs: Date.now() }); - try { - return await run; - } finally { - if (startLocksByAgent.get(agentId)?.promise === marker) { - startLocksByAgent.delete(agentId); - } - } + spec.track?.(Promise.race([pass.then(() => undefined, () => undefined), expired.promise])); + return pass; } diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 23bfa9766c1..684f41782ed 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -4312,6 +4312,7 @@ export function heartbeatService( } finally { controllerLease.stop(); activeRunExecutions.delete(run.id); + heartbeatQueue.releaseClaimBackoffsForTask(run); if (restartClaimLapsed || detachedWithoutHandoff) scheduleNativeSessionResumeDispatch(run.id, new Date(), "restart"); // A failed owned Stop remains visible until this exact executor settles, // including a graceful exit result arriving after the cancellation error. diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index d3f1d08b284..622827d66c3 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -12,7 +12,7 @@ import { hasAutomaticInteractionDeliveryHold, INTERACTION_DELIVERY_DENIED, reche import { deferredWorkIntentWait } from "../work-intent-outbox.js"; import { notifyIssueRecovery } from "../issue-recovery-notifications.js"; import { createHeartbeatTimerPolicy } from "./timer-policy.js"; -import { QUEUED_RUN_AGING_MS, createQueuedRunDispatcher, hasQueuedRunAged } from "./queued-run-dispatch-order.js"; +import { QUEUED_RUN_AGING_MS, createQueuedRunDispatcher, hasQueuedRunAged, queuedRunDispatchClass } from "./queued-run-dispatch-order.js"; import { queuedRunDispatchFacts } from "./queued-run-line.js"; import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../deferred-execution-wait.js"; @@ -188,7 +188,8 @@ import { } from "../execution-workspace-policy.js"; import { readContinuationAttempt } from "../recovery/index.js"; import { allowsDependencyBlockedWake, extractWakeCommentIds, wakeChatConversationsConflict } from "../../modules/run-dispatch/index.js"; -import { withAgentStartLock } from "../agent-start-lock.js"; +import { dispatchForAgent, outsideAgentDispatch } from "../agent-start-lock.js"; +import { AgentRunSlotsFull, assertAgentRunSlotFree } from "./run-slot-admission.js"; import { evaluateAgentInvokability, shouldCancelRunsForNonInvokableAgent, @@ -214,6 +215,10 @@ import { hasRetainedExternalChatRetry, type PostCommitEffect as WakeQueuePostCom import { crashPoint } from "@paperclipai/shared/crash-points"; type HeartbeatRun = typeof heartbeatRuns.$inferSelect; +type QueuedRunCandidate = Pick; +type QueuedDispatchRequest = { attachSetupRequest: boolean; retryRefused: boolean; continuesChain: boolean; triedRunIds: ReadonlySet }; type HeartbeatRunState = ReturnType; type RunPreparation = ReturnType; @@ -375,6 +380,10 @@ export function shouldQueueFollowupForRunningIssueWake(input: { /** Agent Chat outbox receipts. The outbox, not a skipped receipt, owns a capped message. */ const CONVERSATION_COMMENT_KEY_PREFIX = "conversation-comment:"; +const QUEUED_RUN_CLAIM_ATTEMPTS_PER_PASS = 24; +const QUEUED_RUN_CLAIM_BACKOFF_BASE_MS = 5_000; +const QUEUED_RUN_CLAIM_BACKOFF_MAX_MS = 60_000; +const QUEUED_PERSON_RUN_CLAIM_BACKOFF_MS = 2_000; function dailyLimitConversationRefusal(cap: { reason: string; observed: number; limit: number }) { // No receipt is written, so the person's message stays owed in the outbox // until the limit resets (agent-conversations.ts conversationDeliveryHold). @@ -597,7 +606,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende async function listQueuedRunDependencyReadiness( companyId: string, - queuedRuns: Array, + queuedRuns: Array>, ) { const issueIds = [ ...new Set( @@ -960,6 +969,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende companyAgents?: AgentOrgRow[], deferredPostCommitEffects?: WakeQueuePostCommitEffect[], deferredCancellations?: Array<() => Promise>, + slots?: { full: boolean }, ) { if (run.status !== "queued") return run; // Every start path ends here (dispatch passes, executeRun of a queued run, @@ -1583,7 +1593,10 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende // envelope. Preserve their established claim path; only an // explicitly bound queued-message envelope is subject to the // live-comment discard gate below. - await withExternalAdmissionGuard(tx, run.companyId, run.agentId, () => assertNoExternalOverlap(tx, run.companyId, run.agentId)); + await withExternalAdmissionGuard(tx, run.companyId, run.agentId, async () => { + await assertNoExternalOverlap(tx, run.companyId, run.agentId); + await assertAgentRunSlotFree(tx, run); + }); await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId }); const [claimedRun] = await tx .update(heartbeatRuns) @@ -1679,7 +1692,10 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende }; } - await withExternalAdmissionGuard(tx, run.companyId, run.agentId, () => assertNoExternalOverlap(tx, run.companyId, run.agentId)); + await withExternalAdmissionGuard(tx, run.companyId, run.agentId, async () => { + await assertNoExternalOverlap(tx, run.companyId, run.agentId); + await assertAgentRunSlotFree(tx, run); + }); await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId }); await tx .update(agentWakeupRequests) @@ -1723,6 +1739,10 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende .catch((error) => { if (isExternalChatWaitAuthorizationContention(error) || error instanceof RunAdmissionClosedDuringClaim) return { kind: "stale" as const, run: null }; + if (error instanceof AgentRunSlotsFull) { + if (slots) slots.full = true; + return { kind: "stale" as const, run: null }; + } throw error; }) : null; @@ -1795,7 +1815,10 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.agentId, run.agentId), )).for("update"); if (!lockedRun || lockedRun.status !== "queued") return null; - await withExternalAdmissionGuard(claimTx, run.companyId, run.agentId, () => assertNoExternalOverlap(claimTx, run.companyId, run.agentId)); + await withExternalAdmissionGuard(claimTx, run.companyId, run.agentId, async () => { + await assertNoExternalOverlap(claimTx, run.companyId, run.agentId); + await assertAgentRunSlotFree(claimTx, run); + }); await notifyChatRunMilestones(claimTx, { companyId: run.companyId, runId: run.id, issueId: issueId ?? undefined }); const claimedRun = await claimTx.update(heartbeatRuns).set(claimValues).where(and( eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "queued"), @@ -1806,6 +1829,10 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende }); }).catch((error: unknown) => { if (error instanceof RunAdmissionClosedDuringClaim) return null; + if (error instanceof AgentRunSlotsFull) { + if (slots) slots.full = true; + return null; + } throw error; }); if (!claimed) return null; @@ -2252,7 +2279,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende const agentIds = [...new Set(queuedRuns.map((r) => r.agentId))]; for (const agentId of agentIds) { - await startNextQueuedRunForAgent(agentId, true); + await startNextQueuedRunForAgent(agentId, true, true); } } @@ -2336,7 +2363,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende const declinedAgedClaimAt = new Map(); - function noteDeclinedAgedClaim(run: typeof heartbeatRuns.$inferSelect, reason: string) { + function noteDeclinedAgedClaim(run: QueuedRunCandidate, reason: string) { const now = Date.now(); if (!hasQueuedRunAged(run, new Date(now))) return; declinedAgedClaimAt.set(run.id, now); @@ -2345,6 +2372,45 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende "long-queued heartbeat run was offered a free slot but could not be claimed; it stays queued and the next oldest run takes its turn"); } + const claimBackoffs = new Map(); + + function claimBackedOff(runId: string, now: Date, taskStamp: string | null) { + const backoff = claimBackoffs.get(runId); + return Boolean(backoff && backoff.until > now.getTime() && backoff.taskStamp === taskStamp); + } + + function backOffClaim(run: QueuedRunCandidate, personRun: boolean, taskStamp: string | null) { + if (!hasQueuedRunAged(run, new Date())) return; + const attempts = (claimBackoffs.get(run.id)?.attempts ?? 0) + 1; + const delayMs = personRun ? QUEUED_PERSON_RUN_CLAIM_BACKOFF_MS + : Math.min(QUEUED_RUN_CLAIM_BACKOFF_BASE_MS * 2 ** (attempts - 1), QUEUED_RUN_CLAIM_BACKOFF_MAX_MS); + claimBackoffs.set(run.id, { agentId: run.agentId, issueId: readNonEmptyString(parseObject(run.contextSnapshot).issueId), + attempts, until: Date.now() + delayMs, taskStamp }); + } + + function releaseClaimBackoffsForTask(run: Pick) { + const issueId = run.issueId ?? readNonEmptyString(parseObject(run.contextSnapshot).issueId); + if (!issueId) return; + for (const [runId, backoff] of claimBackoffs) { + if (backoff.issueId === issueId) claimBackoffs.delete(runId); + } + } + + function queuedRunTaskStamp(run: QueuedRunCandidate, + issueById: Map) { + const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId); + const issue = issueId ? issueById.get(issueId) : null; + return issue ? `${issue.status}:${issue.updatedAt.getTime()}:${issue.executionRunId ?? ""}` : null; + } + + function forgetSettledClaimBackoffs(agentId: string, queuedRuns: QueuedRunCandidate[]) { + const queued = new Set(queuedRuns.map(run => run.id)); + for (const [runId, backoff] of claimBackoffs) { + if (backoff.agentId === agentId && !queued.has(runId)) claimBackoffs.delete(runId); + } + } + function pruneDeclinedAgedClaims(now: Date) { for (const [id, at] of declinedAgedClaimAt) { if (now.getTime() - at >= QUEUED_RUN_AGING_MS) declinedAgedClaimAt.delete(id); @@ -2353,36 +2419,87 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende async function queuedRunDispatcherForAgent( agent: typeof agents.$inferSelect, - queuedRuns: Array, + queuedRuns: QueuedRunCandidate[], dependencyReadiness: Map, - issueById: Map, + issueById: Map, ) { const now = new Date(); pruneDeclinedAgedClaims(now); + const runById = new Map(queuedRuns.map(run => [run.id, run])); const { facts, dispatchInput } = await queuedRunDispatchFacts(db, { - agent, queuedRuns, dependencyReadiness, issueById, now, shareDeferred: runId => declinedAgedClaimAt.has(runId) }); + agent, queuedRuns, dependencyReadiness, issueById, now, + shareDeferred: runId => declinedAgedClaimAt.has(runId) || claimBackedOff(runId, now, queuedRunTaskStamp(runById.get(runId)!, issueById)) }); const dispatcher = createQueuedRunDispatcher(facts, dispatchInput); const byRun = new Map(facts.map(entry => [entry.run, entry])); return { next: () => dispatcher.next()?.run ?? null, - claimed: (run: typeof heartbeatRuns.$inferSelect) => { const entry = byRun.get(run); if (entry) dispatcher.claimed(entry); }, + claimed: (run: QueuedRunCandidate) => { const entry = byRun.get(run); if (entry) dispatcher.claimed(entry); }, + personRun: (run: QueuedRunCandidate) => { + const entry = byRun.get(run); + return Boolean(entry && queuedRunDispatchClass(entry, now) === "person"); + }, + personOrigin: (run: QueuedRunCandidate) => byRun.get(run)?.requestedByUser ?? false, }; } - async function startNextQueuedRunForAgent(agentId: string, attachSetupRequest = false) { - // Null unless this server has not read the run-start hand-over recently. - const refreshing = refreshRunAdmission(); - if (refreshing) await refreshing; - if (isRunAdmissionClosed() || (await getSchedulingSuppression()).suppressed) return []; - const cutoff = await getWorktreeExecutionCutoff(); - // Cancelled after the start lock is released: cancelRunInternal promotes the - // agent's next queued run, which takes this same lock. + function startNextQueuedRunForAgent(agentId: string, attachSetupRequest = false, retryRefused = false) { + return requestQueuedDispatch(agentId, { attachSetupRequest, retryRefused, continuesChain: false, triedRunIds: new Set() }); + } + + function requestQueuedDispatch(agentId: string, request: QueuedDispatchRequest) { + return dispatchForAgent>(agentId, { + request, + idle: [], + merge: (current, next) => ({ + attachSetupRequest: current.attachSetupRequest || next.attachSetupRequest, + retryRefused: current.retryRefused || next.retryRefused, + continuesChain: current.continuesChain && next.continuesChain, + triedRunIds: current.continuesChain && next.continuesChain + ? new Set([...current.triedRunIds, ...next.triedRunIds]) : new Set(), + }), + run: (request, signal) => runQueuedDispatchPass(agentId, request, signal), + track: (pass) => { + const tracked = pass.then(() => undefined); + activeRunExecutionPromises.add(tracked); + void tracked.finally(() => activeRunExecutionPromises.delete(tracked)); + }, + }); + } + + async function runQueuedDispatchPass(agentId: string, { attachSetupRequest, retryRefused, triedRunIds }: QueuedDispatchRequest, + signal: AbortSignal) { + // Cancelled after the pass ends: cancelRunInternal promotes the agent's + // next queued run, which asks for another pass of this agent. const rejectedClaims: Array<{ run: typeof heartbeatRuns.$inferSelect; err: HttpError }> = []; const deferredPostCommitEffects: WakeQueuePostCommitEffect[] = []; const deferredCancellations: Array<() => Promise> = []; let cancellationReason: string | undefined; + const settle = async () => { + // Dispatch promoted inputs after the pass has ended. + try { + await applyWakeQueuePostCommitEffects(deferredPostCommitEffects); + } finally { + if (cancellationReason) await cancelActiveForAgentInternal(agentId, cancellationReason); + await cancelRejectedQueuedRuns(rejectedClaims); + for (const cancel of deferredCancellations) { + await cancel().catch((err) => logger.warn({ err, agentId }, + "queued heartbeat run refused by a pause hold could not be cancelled; the next claim retries it")); + } + } + }; + try { + return { value: await claimQueuedRunsForAgent(), settle }; + } catch (err) { + await settle(); + throw err; + } - return withAgentStartLock(agentId, async () => { + async function claimQueuedRunsForAgent(): Promise> { + // Null unless this server has not read the run-start hand-over recently. + const refreshing = refreshRunAdmission(); + if (refreshing) await refreshing; + if (isRunAdmissionClosed() || (await getSchedulingSuppression()).suppressed) return []; + const cutoff = await getWorktreeExecutionCutoff(); const agent = await getAgent(agentId); if (!agent) return []; const invokability = await getAgentInvokability(agent); @@ -2395,64 +2512,49 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende } const policy = parseHeartbeatPolicy(agent); const runningCount = await countRunningRunsForAgent(agentId); - const availableSlots = Math.max( - 0, - policy.maxConcurrentRuns - runningCount, - ); + const availableSlots = Math.max(0, policy.maxConcurrentRuns - runningCount); if (availableSlots <= 0) return []; - const queuedRuns = await db - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.agentId, agentId), - eq(heartbeatRuns.status, "queued"), - cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined, - ), - ) - .orderBy(asc(heartbeatRuns.createdAt)); + const queuedRuns = await listQueuedRunCandidates(agentId, cutoff); + forgetSettledClaimBackoffs(agentId, queuedRuns); if (queuedRuns.length === 0) return []; if (queuedRuns.every(run => !run.startedAt) && !await canStartNewExternalBotWork(agent)) return []; - const dependencyReadiness = await listQueuedRunDependencyReadiness( - agent.companyId, - queuedRuns, - ); - const queuedIssueIds = [ - ...new Set( - queuedRuns - .map((run) => - readNonEmptyString(parseObject(run.contextSnapshot).issueId), - ) - .filter((issueId): issueId is string => Boolean(issueId)), - ), - ]; - const issueRows = await db - .select({ - id: issues.id, - status: issues.status, - priority: issues.priority, - }) - .from(issues) - .where( - queuedIssueIds.length > 0 - ? and( - eq(issues.companyId, agent.companyId), - inArray(issues.id, queuedIssueIds), - ) - : sql`false`, - ); + const dependencyReadiness = await listQueuedRunDependencyReadiness(agent.companyId, queuedRuns); + const queuedIssueIds = [...new Set(queuedRuns.flatMap((run) => { + const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId); + return issueId ? [issueId] : []; + }))]; + const issueRows = queuedIssueIds.length > 0 + ? await db.select({ id: issues.id, status: issues.status, priority: issues.priority, updatedAt: issues.updatedAt, + executionRunId: issues.executionRunId }).from(issues) + .where(and(eq(issues.companyId, agent.companyId), inArray(issues.id, queuedIssueIds))) + : []; const issueById = new Map(issueRows.map((row) => [row.id, row])); const companyAgents = await listCompanyAgentOrgRows(agent.companyId); const dispatcher = await queuedRunDispatcherForAgent(agent, queuedRuns, dependencyReadiness, issueById); const claimedRuns: Array = []; - for (let queuedRun = dispatcher.next(); queuedRun; queuedRun = dispatcher.next()) { - if (claimedRuns.length >= availableSlots || isRunAdmissionClosed()) break; + const slots = { full: false }; + let claimAttempts = 0; + const triedThisPass: string[] = []; + for (let candidate = dispatcher.next(); candidate; candidate = dispatcher.next()) { + if (claimedRuns.length >= availableSlots || slots.full || isRunAdmissionClosed() || signal.aborted) break; + const taskStamp = queuedRunTaskStamp(candidate, issueById); + if (triedRunIds.has(candidate.id) || (!retryRefused && claimBackedOff(candidate.id, new Date(), taskStamp))) continue; + if (claimAttempts >= QUEUED_RUN_CLAIM_ATTEMPTS_PER_PASS) { + void requestQueuedDispatch(agentId, { attachSetupRequest: false, retryRefused, + continuesChain: true, triedRunIds: new Set([...triedRunIds, ...triedThisPass]) }); + break; + } + claimAttempts += 1; + triedThisPass.push(candidate.id); + const [queuedRun] = await db.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, candidate.id), eq(heartbeatRuns.companyId, candidate.companyId), eq(heartbeatRuns.status, "queued"))); + if (!queuedRun) continue; let claimed: typeof heartbeatRuns.$inferSelect | null; try { - claimed = await claimQueuedRun(queuedRun, companyAgents, deferredPostCommitEffects, deferredCancellations); + claimed = await claimQueuedRun(queuedRun, companyAgents, deferredPostCommitEffects, deferredCancellations, slots); } catch (err) { if (isPermanentClaimRejection(err)) { rejectedClaims.push({ run: queuedRun, err }); @@ -2463,54 +2565,76 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende { err, runId: queuedRun.id, agentId: queuedRun.agentId, companyId: queuedRun.companyId }, "queued heartbeat run claim was rejected; leaving it queued for the next recovery pass", ); - noteDeclinedAgedClaim(queuedRun, err.message); + noteDeclinedAgedClaim(candidate, err.message); + backOffClaim(candidate, dispatcher.personRun(candidate), taskStamp); continue; } if (!claimed) { - noteDeclinedAgedClaim(queuedRun, "claim_declined"); + if (slots.full) break; + noteDeclinedAgedClaim(candidate, "claim_declined"); + backOffClaim(candidate, dispatcher.personRun(candidate), taskStamp); continue; } claimedRuns.push(claimed); - dispatcher.claimed(queuedRun); - declinedAgedClaimAt.delete(queuedRun.id); + dispatcher.claimed(candidate); + declinedAgedClaimAt.delete(candidate.id); + claimBackoffs.delete(candidate.id); // Start each run as soon as it is claimed. The claim committed a legacy // controller lease that only the run's own execution renews; waiting // for the rest of a slow batch to be claimed let that lease lapse. startClaimedRun(claimed); } return claimedRuns; + } - function startClaimedRun(claimedRun: typeof heartbeatRuns.$inferSelect) { - const execution = executeRun(claimedRun.id).catch((err) => { - logger.error( - { err, runId: claimedRun.id }, - "queued heartbeat execution failed", - ); - }); - // Register the in-flight execution so drainActiveRunExecutions() can await - // it. executeRun resolves only after its finally block finishes flushing - // run rows/events, so awaiting this promise guarantees the run's writes - // have landed before a caller (e.g. a test's afterEach) mutates the DB. - activeRunExecutionPromises.add(execution); - void execution.finally(() => { - // drainActiveRunExecutions loops on activeRunExecutionPromises.size, - // so an entry that never clears here would hang it forever. - activeRunExecutionPromises.delete(execution); - }); - } - }).finally(async () => { - // Dispatch promoted inputs after the agent start lock is released. - try { - await applyWakeQueuePostCommitEffects(deferredPostCommitEffects); - } finally { - if (cancellationReason) await cancelActiveForAgentInternal(agentId, cancellationReason); - await cancelRejectedQueuedRuns(rejectedClaims); - for (const cancel of deferredCancellations) { - await cancel().catch((err) => logger.warn({ err, agentId }, - "queued heartbeat run refused by a pause hold could not be cancelled; the next claim retries it")); - } - } - }); + function startClaimedRun(claimedRun: typeof heartbeatRuns.$inferSelect) { + const execution = outsideAgentDispatch(() => executeRun(claimedRun.id)).catch((err) => { + logger.error( + { err, runId: claimedRun.id }, + "queued heartbeat execution failed", + ); + }); + // Register the in-flight execution so drainActiveRunExecutions() can await + // it. executeRun resolves only after its finally block finishes flushing + // run rows/events, so awaiting this promise guarantees the run's writes + // have landed before a caller (e.g. a test's afterEach) mutates the DB. + activeRunExecutionPromises.add(execution); + void execution.finally(() => { + // drainActiveRunExecutions loops on activeRunExecutionPromises.size, + // so an entry that never clears here would hang it forever. + activeRunExecutionPromises.delete(execution); + }); + } + } + + async function listQueuedRunCandidates(agentId: string, cutoff: Date | null): Promise { + const contextText = (key: string) => sql`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}->${key}) = 'string' + then ${heartbeatRuns.contextSnapshot}->>${key} end`; + const rows = await db.select({ + id: heartbeatRuns.id, + companyId: heartbeatRuns.companyId, + agentId: heartbeatRuns.agentId, + status: heartbeatRuns.status, + invocationSource: heartbeatRuns.invocationSource, + createdAt: heartbeatRuns.createdAt, + startedAt: heartbeatRuns.startedAt, + wakeupRequestId: heartbeatRuns.wakeupRequestId, + retryOfRunId: heartbeatRuns.retryOfRunId, + scheduledRetryAt: heartbeatRuns.scheduledRetryAt, + scheduledRetryAttempt: heartbeatRuns.scheduledRetryAttempt, + scheduledRetryReason: heartbeatRuns.scheduledRetryReason, + issueId: contextText("issueId"), + wakeReason: contextText("wakeReason"), + previousRunId: contextText("previousRunId"), + }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.agentId, agentId), + eq(heartbeatRuns.status, "queued"), + cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined, + )).orderBy(asc(heartbeatRuns.createdAt)); + return rows.map(({ issueId, wakeReason, previousRunId, ...run }) => ({ + ...run, + contextSnapshot: Object.fromEntries(Object.entries({ issueId, wakeReason, previousRunId }).filter(([, value]) => value !== null)), + })); } // Public wakeup entry point. Callers dispatch it fire-and-forget, so register @@ -5654,6 +5778,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende createNativeStatusWakeBatch, startNextQueuedRunForAgent, countRunningRunsForAgent, + releaseClaimBackoffsForTask, releaseRunClaimedJustBeforeSuppression, claimQueuedRun, withChatControlRecoveryGate, diff --git a/server/src/services/heartbeat/run-slot-admission.ts b/server/src/services/heartbeat/run-slot-admission.ts new file mode 100644 index 00000000000..ec70cc2ec64 --- /dev/null +++ b/server/src/services/heartbeat/run-slot-admission.ts @@ -0,0 +1,28 @@ +import { agents, heartbeatRuns, type Db } from "@paperclipai/db"; +import { isUuidLike } from "@paperclipai/shared"; +import { and, eq, notInArray, sql } from "drizzle-orm"; +import { maxConcurrentRunsSql } from "./timer-policy.js"; + +export class AgentRunSlotsFull extends Error { + constructor(readonly agentId: string, readonly running: number, readonly limit: number) { + super("The agent already runs as many runs at once as it allows."); + this.name = "AgentRunSlotsFull"; + } +} + +type ClaimingRun = Pick; + +export async function assertAgentRunSlotFree(tx: Pick, run: ClaimingRun) { + const previousRunId = (run.contextSnapshot as Record | null)?.previousRunId; + const replaced = [run.id, run.retryOfRunId, typeof previousRunId === "string" ? previousRunId : null] + .filter((id): id is string => typeof id === "string" && isUuidLike(id)); + const [running] = await tx.select({ count: sql`count(*)::int` }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.agentId, run.agentId), eq(heartbeatRuns.status, "running"), + notInArray(heartbeatRuns.id, replaced))); + const [agent] = await tx.select({ limit: maxConcurrentRunsSql() }).from(agents) + .where(and(eq(agents.id, run.agentId), eq(agents.companyId, run.companyId))); + if (!agent) return; + const count = Number(running?.count ?? 0); + const limit = Number(agent.limit); + if (count >= limit) throw new AgentRunSlotsFull(run.agentId, count, limit); +} diff --git a/server/src/services/native-runtime/native-review-dispatch.ts b/server/src/services/native-runtime/native-review-dispatch.ts index 76799a7ebbd..13eda8f9fc7 100644 --- a/server/src/services/native-runtime/native-review-dispatch.ts +++ b/server/src/services/native-runtime/native-review-dispatch.ts @@ -3,6 +3,7 @@ import { withExternalAdmissionGuard, assertNoExternalOverlap } from "../../modul import { agentWakeupRequests, heartbeatRuns, issues, type Db } from "@paperclipai/db"; import { and, eq } from "drizzle-orm"; import type { PgUpdateSetSource } from "drizzle-orm/pg-core"; +import { assertAgentRunSlotFree } from "../heartbeat/run-slot-admission.js"; import { claimNativeReviewExecutionLock, readNativeReviewAssignmentContext } from "./native-review-participant.js"; /** The run, wake, and issue lock form one reviewer admission claim. */ @@ -35,7 +36,10 @@ export async function claimQueuedNativeReviewRun(db: Db, input: { )).for("update"); if (!run || run.status !== "queued" || run.wakeupRequestId !== input.run.wakeupRequestId) return null; if ((run.contextSnapshot as Record | null)?.issueId !== issueId) return null; - await withExternalAdmissionGuard(tx, run.companyId, run.agentId, () => assertNoExternalOverlap(tx, run.companyId, run.agentId)); + await withExternalAdmissionGuard(tx, run.companyId, run.agentId, async () => { + await assertNoExternalOverlap(tx, run.companyId, run.agentId); + await assertAgentRunSlotFree(tx, run); + }); const locked = await claimNativeReviewExecutionLock(tx as unknown as Db, { companyId: run.companyId, issueId, agentId: run.agentId, runId: run.id, contextSnapshot: context, agentNameKey: input.agentNameKey, claimedAt: input.claimedAt, diff --git a/server/src/services/workspace-recovery-preparation-dispatch.test.ts b/server/src/services/workspace-recovery-preparation-dispatch.test.ts index e29b2485a84..dd7121d8929 100644 --- a/server/src/services/workspace-recovery-preparation-dispatch.test.ts +++ b/server/src/services/workspace-recovery-preparation-dispatch.test.ts @@ -48,7 +48,7 @@ describe("automatic retained workspace recovery", () => { const experimental = await instanceSettingsService(db).getExperimental(); let stoppedWorker: Awaited> | undefined; let storage: { mockRestore(): void } | undefined; - const starts = vi.spyOn(agentStarts, "withAgentStartLock"); + const starts = vi.spyOn(agentStarts, "dispatchForAgent"); const retryResumes: Promise[] = []; let resumeSpy: { mockRestore(): void } | undefined; let ownedHeartbeat: ReturnType | undefined; @@ -178,7 +178,8 @@ describe("automatic retained workspace recovery", () => { }); const app = boardRecoveryApp(undefined, heartbeat); const gate = new Promise(resolve => { releaseStart = resolve; }); - heldStart = agentStarts.withAgentStartLock(source.agentId, () => gate); + heldStart = agentStarts.dispatchForAgent(source.agentId, { request: null, idle: undefined, merge: () => null, + run: async () => ({ value: await gate }) }); const receipt = mode === "automatic_lost" ? null : (await request(app).post(`/api/issues/${f.issueId}/retry`) .send({ observedRunId: f.runId, idempotencyKey: randomUUID() }).expect(202)).body; if (mode === "automatic_lost") {