fix(dispatch): bound how long a queued run waits and keep a person's interrupted Retry person-first (#21349)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue decides which queued run an agent starts when a
slot frees up
> - Recent changes start person-initiated work first and give recovery
runs one start in four
> - A system continuation of a person's interrupted Retry ranked as
old-backlog recovery, and the recovery share always went to fresher
recovery runs
> - With a steady stream of new work, that continuation never started,
so the person's task sat queued for hours
> - This pull request adds aging and a longest-waiting share, keeps a
person's interrupted request person-first, and passes the turn on when a
long-queued run cannot be claimed
> - The benefit is that every ready queued run starts within a bounded
number of starts, and a person's Retry is not lost behind system work

## Linked Issues or Issue Description

**What happened?**

On a hosted staging instance, one agent had 28 runs running and kept
starting newer runs. The company had 67 queued runs, and 54 of them were
older than one hour. A person pressed Retry on a task. The Retry run was
interrupted by a server restart two minutes later. The system queued an
automatic continuation (`issue_continuation_needed`) with
`previousRunId` set to the interrupted Retry run. That continuation
stayed queued for more than 2.5 hours. The task showed as "working".
Other queued runs of the same person (some queued for more than 10
hours) also did not start.

Root cause:
- The dispatcher ranked the continuation as `recovery_backlog`, the last
tier, because it was a system wake and its source run ended more than 30
minutes ago. The person-first check only looked at wake requests
attached to the queued run itself. It did not look at the source run,
which was the person's `task_retry`.
- The 1-in-4 recovery share picked the first recovery run in sort order.
Fresh recovery runs sort before backlog runs, and fresh recovery runs
kept arriving, so the backlog run never got the share.
- Nothing aged a queued run. A run in a lower tier or with a lower
priority could wait forever.

**Expected behavior**

A system continuation of a person's Retry ranks as person-initiated.
Every ready queued run starts within a bounded number of starts, even
when fresh person and system work keeps arriving. A run that keeps
failing to claim is reported and does not block the next-oldest run.

**Steps to reproduce**

1. Give an agent one free slot and several queued runs.
2. Queue a person's `task_retry` run, start it, and interrupt it.
3. Queue a system continuation with `previousRunId` set to the
interrupted run, and let its source age past 30 minutes.
4. Keep adding fresh system and person work.
5. See that the continuation never starts.

**Paperclip version or commit**

Staging Core 7e7452e88f (contains #21234 and #21277). Fixed on top of
origin/master cb94108484.

**Deployment mode**

Hosted (authenticated, public exposure, managed cloud).

**Agent adapter(s) involved**

Any adapter. The bug is in queued-run dispatch order, before the adapter
starts.

Refs #21234, #21277

## What Changed

- `queued-run-dispatch-order.ts`: a ready run that has waited for one
hour (`QUEUED_RUN_AGING_MS`) moves up to at least the work tier.
- `queued-run-dispatch-order.ts`: every fourth start
(`OLDEST_DISPATCH_SHARE`) goes to the longest-waiting ready run that has
aged, whatever its class or priority. The dispatcher tracks this share
inside one pass.
- `queue.ts`: the dispatcher reads the agent's recent starts to know how
many starts passed over the oldest run. One query now serves both the
recovery share and the oldest-run share.
- `queue.ts`: a queued run is person-initiated when any run in its
source chain (`previousRunId`, then `retryOfRunId`, up to 8 runs, in one
recursive query) was a person's request: a `task_retry` or chat wake, a
run with a `taskRetryRequestId`, or a run whose wake request a user
made. A person's Retry stays first after repeated interruptions.
- `queue.ts`: when a run that has waited past the aging bound is offered
a slot and cannot be claimed, the server logs a warning with the reason
and the wait time. That run skips the oldest-run share for one aging
period, so the next-oldest run takes the turn. It is still offered a
slot in normal order on every pass.

## Verification

- `npx vitest run
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/recovery-wake-herd.test.ts` (38 passed).
- Red/green: with the old dispatcher and queue, these new tests fail and
the fix makes them pass:
- "starts an old queued continuation within a bounded number of starts
while fresh person and system work keeps arriving" (unit, continuous
stream of fresh person, work and recovery runs)
- "serves the longest-waiting run once every share even when fresher
work outranks it" (unit)
  - "lets an old-backlog recovery run age into the work tier" (unit)
- "starts the system continuation of a person's interrupted Retry ahead
of fresher system work" (embedded Postgres, the staging case)
- "gives the longest-waiting queued run a free slot after three fresher
starts passed it over" (embedded Postgres)
- "keeps a person's Retry first after its system continuation is
interrupted again" (embedded Postgres, two successive interruptions)
- "starts the oldest continuations of a long backlog within a bound
while later assignments and fresh recovery keep arriving" (unit, the
staging shape: 3 continuations 4-5 hours old behind 80 later
assignments)
- Added coverage that passes before and after: "starts a system
continuation that a person's Retry coalesced into ahead of fresher
system work", "passes the oldest-run share to the next oldest run while
the oldest one keeps failing to claim", and "does not use the oldest-run
share for a run that has not waited past the aging bound".
- Related suites pass: heartbeat-run-dispatch-work,
heartbeat-run-dispatch-lifecycle, heartbeat-queued-run-claim-isolation,
issue-reassignment-queued-run, heartbeat-startup-fresh-dispatch,
heartbeat-stale-queue-invalidation,
heartbeat-ephemeral-queued-migration, queued-comment-consumption (116
tests).
- `tsc --noEmit -p server` passes.
- Not related to this change: `heartbeat-stale-queue-invalidation`
"resumes a rejected connection_intent outcome and promotes the claimed
review task" also fails locally on the base commit without this change
(one queued run, so the new code does not run).

## Risks

- Behavior change: when runs have waited more than one hour, one start
in four goes to the oldest of them, even when fresher person work is
queued. A person's fresh work still gets the other three starts.
- More runs can rank as person-initiated: system continuations and
retries of a person's run. This is the intended outcome.
- One more small recursive query per dispatch pass (the source chain, at
most 8 hops per run, by primary key and the wake-request run index). It
only runs when more than one run is queued and some have a source run.
- The record of runs that failed to claim is in memory. After a restart,
a stuck run gets the oldest-run share again once, then is skipped again.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, browser for read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 04:59:56 -06:00
1 parent 2b69cdfcd6
commit ff971149c2
4 files changed
+314 -22

No files matched your search

@@ -158,6 +158,106 @@ suite("recovery wake herd", () => {
}
}, 120_000);
async function continuationOfInterruptedPersonRetry(agentId: string, createdMinutesAgo: number) {
const issue = await task(agentId);
const finishedAt = new Date(Date.now() - (createdMinutesAgo + 2) * 60_000);
const [retry] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", triggerDetail: "manual",
reason: "task_retry", status: "queued", requestedByActorType: "user", requestedByActorId: userId,
payload: { issueId: issue.id } }).returning();
const [source] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: issue.id, invocationSource: "on_demand",
triggerDetail: "manual", status: "interrupted", errorCode: "server_shutdown_interrupted", wakeupRequestId: retry!.id,
responsibleUserId: userId, startedAt: new Date(finishedAt.getTime() - 60_000), finishedAt,
createdAt: new Date(finishedAt.getTime() - 120_000),
contextSnapshot: { issueId: issue.id, wakeReason: "task_retry", taskRetryRequestId: retry!.id } }).returning();
await db.update(agentWakeupRequests).set({ runId: source!.id, status: "completed", finishedAt })
.where(eq(agentWakeupRequests.id, retry!.id));
const [continuation] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: issue.id,
invocationSource: "automation", triggerDetail: "system", status: "queued", responsibleUserId: userId,
createdAt: new Date(Date.now() - createdMinutesAgo * 60_000),
contextSnapshot: { issueId: issue.id, taskId: issue.id, wakeReason: "issue_continuation_needed", previousRunId: source!.id,
source: "issue.assigned_work_invariant", assignedWorkViolation: "retry_offered" } }).returning();
return continuation!;
}
it("starts the system continuation of a person's interrupted Retry ahead of fresher system work", async () => {
const agentId = await seedCompany(1);
const continuation = await continuationOfInterruptedPersonRetry(agentId, 40);
const fresher: Array<typeof heartbeatRuns.$inferSelect> = [];
for (let index = 0; index < 3; index += 1) fresher.push(await queuedSystemRun(agentId, 30 - index, "issue_assigned"));
fresher.push(await queuedRecoveryRun(agentId, 2, 3));
const heartbeat = heartbeatService(db);
try {
await heartbeat.resumeQueuedRuns();
expect((await readRun(continuation.id)).startedAt).not.toBeNull();
for (const run of fresher) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null });
} finally {
await heartbeat.drainActiveRunExecutions();
}
}, 120_000);
it("keeps a person's Retry first after its system continuation is interrupted again", async () => {
const agentId = await seedCompany(1);
const first = await continuationOfInterruptedPersonRetry(agentId, 90);
const firstEndedAt = new Date(Date.now() - 45 * 60_000);
await db.update(heartbeatRuns).set({ status: "interrupted", errorCode: "server_shutdown_interrupted",
startedAt: new Date(firstEndedAt.getTime() - 60_000), finishedAt: firstEndedAt }).where(eq(heartbeatRuns.id, first.id));
const [second] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: first.issueId,
invocationSource: "automation", triggerDetail: "system", status: "queued", responsibleUserId: userId,
createdAt: new Date(Date.now() - 40 * 60_000),
contextSnapshot: { issueId: first.issueId, taskId: first.issueId, wakeReason: "issue_continuation_needed",
previousRunId: first.id, source: "issue.assigned_work_invariant" } }).returning();
const fresher: Array<typeof heartbeatRuns.$inferSelect> = [];
for (let index = 0; index < 3; index += 1) fresher.push(await queuedSystemRun(agentId, 30 - index, "issue_assigned"));
const heartbeat = heartbeatService(db);
try {
await heartbeat.resumeQueuedRuns();
expect((await readRun(second!.id)).startedAt).not.toBeNull();
for (const run of fresher) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null });
} finally {
await heartbeat.drainActiveRunExecutions();
}
}, 120_000);
it("starts a system continuation that a person's Retry coalesced into ahead of fresher system work", async () => {
const agentId = await seedCompany(1);
const continuation = await queuedSystemRun(agentId, 40, "issue_continuation_needed");
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", triggerDetail: "manual",
reason: "task_retry", status: "coalesced", coalescedCount: 1, requestedByActorType: "user", requestedByActorId: userId,
runId: continuation.id, finishedAt: new Date(), payload: { issueId: continuation.issueId } });
const fresher: Array<typeof heartbeatRuns.$inferSelect> = [];
for (let index = 0; index < 3; index += 1) fresher.push(await queuedSystemRun(agentId, 30 - index, "issue_assigned"));
const heartbeat = heartbeatService(db);
try {
await heartbeat.resumeQueuedRuns();
expect((await readRun(continuation.id)).startedAt).not.toBeNull();
for (const run of fresher) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null });
} finally {
await heartbeat.drainActiveRunExecutions();
}
}, 120_000);
it("gives the longest-waiting queued run a free slot after three fresher starts passed it over", async () => {
const agentId = await seedCompany(1);
const old = await queuedRecoveryRun(agentId, 150, 152);
const people: Array<typeof heartbeatRuns.$inferSelect> = [];
for (let index = 0; index < 3; index += 1) people.push(await queuedPersonRetry(agentId));
const issue = await task(agentId);
for (let index = 0; index < 3; index += 1) {
const startedAt = new Date(Date.now() - (3 - index) * 60_000);
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: issue.id, invocationSource: "on_demand",
triggerDetail: "manual", status: "succeeded", createdAt: new Date(startedAt.getTime() - 1_000), startedAt,
finishedAt: new Date(startedAt.getTime() + 1_000), contextSnapshot: { issueId: issue.id, wakeReason: "issue_recovery_action_restored" } });
}
const heartbeat = heartbeatService(db);
try {
await heartbeat.resumeQueuedRuns();
expect((await readRun(old.id)).startedAt).not.toBeNull();
for (const run of people) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null });
} finally {
await heartbeat.drainActiveRunExecutions();
}
}, 120_000);
async function pendingContinuation(agentId: string, endedMinutesAgo: number) {
const issue = await task(agentId, "todo");
const source = await sourceRun(agentId, issue.id, endedMinutesAgo);
+97 -19
View File
@@ -10,7 +10,8 @@ import { hasAutomaticInteractionDeliveryHold, INTERACTION_DELIVERY_DENIED, reche
import { deferredWorkIntentWait } from "../work-intent-outbox.js";
import { notifyIssueRecovery } from "../issue-recovery-notifications.js";
import { createHeartbeatTimerPolicy } from "./timer-policy.js";
import { RECOVERY_DISPATCH_SHARE, isRecoveryWakeReason, isSystemRecoveryRun, createQueuedRunDispatcher } from "./queued-run-dispatch-order.js";
import { OLDEST_DISPATCH_SHARE, PERSON_WAKE_REASONS, QUEUED_RUN_AGING_MS, RECOVERY_DISPATCH_SHARE, createQueuedRunDispatcher,
hasQueuedRunAged, isRecoveryWakeReason, isSystemRecoveryRun, oldestReadyQueuedRun } from "./queued-run-dispatch-order.js";
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../deferred-execution-wait.js";
import { assignmentWakePayloadSql, isAssignmentWakePayload, notifyDeferredExecutionWork } from "../deferred-execution-notifications.js";
@@ -372,6 +373,8 @@ export function shouldQueueFollowupForRunningIssueWake(input: {
/** Agent Chat outbox receipts. The outbox, not a skipped receipt, owns a capped message. */
const CONVERSATION_COMMENT_KEY_PREFIX = "conversation-comment:";
const RUN_ID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
const PERSON_ORIGIN_CHAIN_DEPTH = 8;
function dailyLimitConversationRefusal(cap: { reason: string; observed: number; limit: number }) {
// No receipt is written, so the person's message stays owed in the outbox
// until the limit resets (agent-conversations.ts conversationDeliveryHold).
@@ -2346,6 +2349,48 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
.catch(() => logger.warn({ agentId: agent.id }, "Could not attach queued agent setup connection request"));
}
const declinedAgedClaimAt = new Map<string, number>();
function noteDeclinedAgedClaim(run: typeof heartbeatRuns.$inferSelect, reason: string) {
const now = Date.now();
if (!hasQueuedRunAged(run, new Date(now))) return;
declinedAgedClaimAt.set(run.id, now);
logger.warn({ runId: run.id, agentId: run.agentId, companyId: run.companyId, reason,
queuedForMs: now - run.createdAt.getTime() },
"long-queued heartbeat run was offered a free slot but could not be claimed; it stays queued and the next oldest run takes its turn");
}
function pruneDeclinedAgedClaims(now: Date) {
for (const [id, at] of declinedAgedClaimAt) {
if (now.getTime() - at >= QUEUED_RUN_AGING_MS) declinedAgedClaimAt.delete(id);
}
}
async function listPersonOriginSourceRunIds(companyId: string, sourceRunIds: string[]) {
const uuidPattern = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$";
const result = await db.execute<{ root_id: string }>(sql`
with recursive chain(root_id, run_id, depth) as (
select source.id, source.id, 0 from ${heartbeatRuns} source
where source.company_id = ${companyId} and source.id in (${sql.join(sourceRunIds.map(id => sql`${id}::uuid`), sql`, `)})
union all
select chain.root_id, parent.id, chain.depth + 1
from chain
join ${heartbeatRuns} child on child.id = chain.run_id
join ${heartbeatRuns} parent on parent.company_id = ${companyId} and parent.id = coalesce(
case when child.context_snapshot->>'previousRunId' ~* ${uuidPattern}
then (child.context_snapshot->>'previousRunId')::uuid end,
child.retry_of_run_id)
where chain.depth < ${PERSON_ORIGIN_CHAIN_DEPTH}
)
select distinct chain.root_id from chain
join ${heartbeatRuns} run on run.id = chain.run_id
where run.context_snapshot->>'wakeReason' in (${sql.join([...PERSON_WAKE_REASONS].map(reason => sql`${reason}`), sql`, `)})
or run.context_snapshot->>'taskRetryRequestId' is not null
or exists (select 1 from ${agentWakeupRequests} wake where wake.company_id = ${companyId}
and wake.requested_by_actor_type = 'user' and (wake.run_id = run.id or wake.id = run.wakeup_request_id))`);
return [...result].map(row => String(row.root_id));
}
async function queuedRunDispatcherForAgent(
agent: typeof agents.$inferSelect,
queuedRuns: Array<typeof heartbeatRuns.$inferSelect>,
@@ -2353,6 +2398,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
issueById: Map<string, { status: string; priority: string | null }>,
) {
const now = new Date();
pruneDeclinedAgedClaims(now);
const wakeReasonOf = (run: typeof heartbeatRuns.$inferSelect) =>
readNonEmptyString(parseObject(run.contextSnapshot).wakeReason);
const requestIds = [...new Set(queuedRuns.flatMap(run => run.wakeupRequestId ? [run.wakeupRequestId] : []))];
@@ -2368,30 +2414,29 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
eq(agentWakeupRequests.agentId, agent.id), eq(agentWakeupRequests.requestedByActorType, "user")))
: [];
const userRunIds = new Set(joinedUserRequests.flatMap(row => row.runId ? [row.runId] : []));
const requestedByUser = (run: typeof heartbeatRuns.$inferSelect) =>
const joinedByUser = (run: typeof heartbeatRuns.$inferSelect) =>
userRunIds.has(run.id) || Boolean(run.wakeupRequestId && userRequestIds.has(run.wakeupRequestId));
const sourceRunIdOf = (run: typeof heartbeatRuns.$inferSelect) =>
readNonEmptyString(parseObject(run.contextSnapshot).previousRunId) ?? run.retryOfRunId ?? null;
const recoveryRuns = queuedRuns.filter(run => isSystemRecoveryRun({ wakeReason: wakeReasonOf(run),
requestedByUser: requestedByUser(run) }));
const sourceRunIds = [...new Set(recoveryRuns.flatMap(run => sourceRunIdOf(run) ?? []))];
const sourceRows = queuedRuns.length > 1 && sourceRunIds.length
const sourceRunIds = queuedRuns.length > 1
? [...new Set(queuedRuns.flatMap(run => {
const sourceRunId = joinedByUser(run) ? null : sourceRunIdOf(run);
return sourceRunId && RUN_ID_PATTERN.test(sourceRunId) ? [sourceRunId] : [];
}))]
: [];
const sourceRows = sourceRunIds.length
? await db.select({ id: heartbeatRuns.id, finishedAt: heartbeatRuns.finishedAt }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, agent.companyId), inArray(heartbeatRuns.id, sourceRunIds)))
: [];
const sourceEndedAt = new Map(sourceRows.map(row => [row.id, row.finishedAt]));
let startsSinceRecovery = 0;
if (recoveryRuns.length && recoveryRuns.length < queuedRuns.length) {
const oldestRecoveryAt = new Date(Math.min(...recoveryRuns.map(run => run.createdAt.getTime())));
const recentStarts = await db.select({ wakeReason: sql<string | null>`${heartbeatRuns.contextSnapshot}->>'wakeReason'` })
.from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id),
isNotNull(heartbeatRuns.startedAt), gte(heartbeatRuns.startedAt, oldestRecoveryAt)))
.orderBy(desc(heartbeatRuns.startedAt))
.limit(RECOVERY_DISPATCH_SHARE);
const lastRecovery = recentStarts.findIndex(row => isRecoveryWakeReason(row.wakeReason));
startsSinceRecovery = lastRecovery < 0 ? recentStarts.length : lastRecovery;
}
const personSourceRunIds = new Set(sourceRows.length
? await listPersonOriginSourceRunIds(agent.companyId, sourceRows.map(row => row.id))
: []);
const requestedByUser = (run: typeof heartbeatRuns.$inferSelect) => {
if (joinedByUser(run)) return true;
const sourceRunId = sourceRunIdOf(run);
return Boolean(sourceRunId && personSourceRunIds.has(sourceRunId));
};
const facts = queuedRuns.map(run => {
const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
const ready = issueId ? (dependencyReadiness.get(issueId)?.isDependencyReady ?? true) : true;
@@ -2406,9 +2451,38 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
sourceEndedAt: sourceRunId ? sourceEndedAt.get(sourceRunId) ?? null : null,
readinessRank: issueId ? (ready ? (issue?.status === "in_progress" ? 0 : 1) : 3) : 2,
priorityRank: issueRunPriorityRank(issue?.priority),
shareDeferred: declinedAgedClaimAt.has(run.id),
};
});
const dispatcher = createQueuedRunDispatcher(facts, { now, startsSinceRecovery });
const recoveryRuns = facts.filter(entry => isSystemRecoveryRun(entry));
const oldestReady = oldestReadyQueuedRun(facts);
const agedOldest = oldestReady && hasQueuedRunAged(oldestReady, now) ? oldestReady : null;
const countsRecovery = recoveryRuns.length > 0 && recoveryRuns.length < facts.length;
let startsSinceRecovery = 0;
let startsSinceOldest = 0;
if (countsRecovery || (agedOldest && facts.length > 1)) {
const oldestRecoveryAt = countsRecovery ? Math.min(...recoveryRuns.map(entry => entry.createdAt.getTime())) : Infinity;
const since = new Date(Math.min(oldestRecoveryAt, agedOldest?.createdAt.getTime() ?? Infinity));
const recentStarts = await db.select({ wakeReason: sql<string | null>`${heartbeatRuns.contextSnapshot}->>'wakeReason'`,
startedAt: heartbeatRuns.startedAt, createdAt: heartbeatRuns.createdAt })
.from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id),
isNotNull(heartbeatRuns.startedAt), gte(heartbeatRuns.startedAt, since)))
.orderBy(desc(heartbeatRuns.startedAt))
.limit(Math.max(RECOVERY_DISPATCH_SHARE, OLDEST_DISPATCH_SHARE));
const startsSince = (at: number) => recentStarts.filter(row => row.startedAt && row.startedAt.getTime() >= at);
if (countsRecovery) {
const starts = startsSince(oldestRecoveryAt).slice(0, RECOVERY_DISPATCH_SHARE);
const lastRecovery = starts.findIndex(row => isRecoveryWakeReason(row.wakeReason));
startsSinceRecovery = lastRecovery < 0 ? starts.length : lastRecovery;
}
if (agedOldest && facts.length > 1) {
const starts = startsSince(agedOldest.createdAt.getTime()).slice(0, OLDEST_DISPATCH_SHARE);
const lastOldest = starts.findIndex(row => row.createdAt.getTime() <= agedOldest.createdAt.getTime());
startsSinceOldest = lastOldest < 0 ? starts.length : lastOldest;
}
}
const dispatcher = createQueuedRunDispatcher(facts, { now, startsSinceRecovery, startsSinceOldest });
const byRun = new Map(facts.map(entry => [entry.run, entry]));
return {
next: () => dispatcher.next()?.run ?? null,
@@ -2510,11 +2584,15 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
{ err, runId: queuedRun.id, agentId: queuedRun.agentId, companyId: queuedRun.companyId },
"queued heartbeat run claim was rejected; leaving it queued for the next recovery pass",
);
noteDeclinedAgedClaim(queuedRun, err.message);
continue;
}
if (claimed) {
claimedRuns.push(claimed);
dispatcher.claimed(queuedRun);
declinedAgedClaimAt.delete(queuedRun.id);
} else {
noteDeclinedAgedClaim(queuedRun, "claim_declined");
}
}
if (claimedRuns.length === 0) return [];
@@ -1,5 +1,7 @@
import { describe, expect, it } from "vitest";
import {
OLDEST_DISPATCH_SHARE,
QUEUED_RUN_AGING_MS,
RECOVERY_BACKLOG_AGE_MS,
createQueuedRunDispatcher,
RECOVERY_DISPATCH_SHARE,
@@ -122,4 +124,87 @@ describe("queued run dispatch order", () => {
const ordered = orderQueuedRunsForDispatch([todo, later, earlier, urgent, inProgress], { now, startsSinceRecovery: 0 });
expect(ordered.map(entry => entry.id)).toEqual([urgent.id, earlier.id, later.id, inProgress.id, todo.id]);
});
it("starts an old queued continuation within a bounded number of starts while fresh person and system work keeps arriving", () => {
const continuation = run({ wakeReason: "issue_continuation_needed", createdAt: minutesAgo(150), sourceEndedAt: minutesAgo(152) });
const started: QueuedRunDispatchFacts[] = [];
let pending: QueuedRunDispatchFacts[] = [continuation];
const steps = 40;
for (let step = 0; step < steps; step += 1) {
pending.push(personRetry(), run({ wakeReason: "issue_assigned", createdAt: minutesAgo(1) }),
run({ wakeReason: "issue_continuation_needed", createdAt: minutesAgo(1), sourceEndedAt: minutesAgo(2) }));
const recent = started.slice(-RECOVERY_DISPATCH_SHARE).reverse();
const lastRecovery = recent.findIndex(entry => isRecoveryWakeReason(entry.wakeReason));
const oldest = pending.reduce((left, right) => left.createdAt <= right.createdAt ? left : right);
const recentForOldest = started.slice(-OLDEST_DISPATCH_SHARE).reverse();
const lastOldest = recentForOldest.findIndex(entry => entry.createdAt <= oldest.createdAt);
const [next] = orderQueuedRunsForDispatch(pending, { now,
startsSinceRecovery: lastRecovery < 0 ? recent.length : lastRecovery,
startsSinceOldest: lastOldest < 0 ? recentForOldest.length : lastOldest });
started.push(next!);
pending = pending.filter(entry => entry.id !== next!.id);
}
const position = started.findIndex(entry => entry.id === continuation.id);
expect(position).toBeGreaterThanOrEqual(0);
expect(position).toBeLessThan(OLDEST_DISPATCH_SHARE);
expect(started.slice(0, position).every(entry => entry.wakeReason === "task_retry" || entry.wakeReason === "issue_continuation_needed")).toBe(true);
});
it("serves the longest-waiting run once every share even when fresher work outranks it", () => {
const people = Array.from({ length: 8 }, () => personRetry());
const old = run({ wakeReason: "issue_monitor_due", priorityRank: 3, createdAt: minutesAgo(200) });
const ordered = orderQueuedRunsForDispatch([...people, old], { now, startsSinceRecovery: 0, startsSinceOldest: 0 });
expect(ordered.findIndex(entry => entry.id === old.id)).toBe(OLDEST_DISPATCH_SHARE - 1);
const due = orderQueuedRunsForDispatch([...people, old], { now, startsSinceRecovery: 0, startsSinceOldest: OLDEST_DISPATCH_SHARE - 1 });
expect(due[0]!.id).toBe(old.id);
});
it("does not use the oldest-run share for a run that has not waited past the aging bound", () => {
const people = Array.from({ length: 3 }, () => personRetry());
const recent = run({ wakeReason: "issue_assigned", createdAt: new Date(now.getTime() - QUEUED_RUN_AGING_MS + 60_000) });
const ordered = orderQueuedRunsForDispatch([...people, recent], { now, startsSinceRecovery: 0, startsSinceOldest: OLDEST_DISPATCH_SHARE - 1 });
expect(ordered[0]!.wakeReason).toBe("task_retry");
});
it("lets an old-backlog recovery run age into the work tier", () => {
const aged = run({ wakeReason: "issue_continuation_needed", createdAt: new Date(now.getTime() - QUEUED_RUN_AGING_MS),
sourceEndedAt: minutesAgo(200) });
const fresh = run({ wakeReason: "issue_assigned", createdAt: minutesAgo(1) });
const ordered = orderQueuedRunsForDispatch([fresh, aged], { now, startsSinceRecovery: 0 });
expect(ordered.map(entry => entry.id)).toEqual([aged.id, fresh.id]);
});
it("passes the oldest-run share to the next oldest run while the oldest one keeps failing to claim", () => {
const people = Array.from({ length: 3 }, () => personRetry());
const stuck = run({ wakeReason: "issue_continuation_needed", createdAt: minutesAgo(300), shareDeferred: true });
const next = run({ wakeReason: "issue_monitor_due", priorityRank: 3, createdAt: minutesAgo(200) });
const ordered = orderQueuedRunsForDispatch([...people, stuck, next], { now, startsSinceRecovery: 0,
startsSinceOldest: OLDEST_DISPATCH_SHARE - 1 });
expect(ordered[0]!.id).toBe(next.id);
expect(ordered.slice(1, 3).every(entry => entry.wakeReason === "task_retry")).toBe(true);
});
it("starts the oldest continuations of a long backlog within a bound while later assignments and fresh recovery keep arriving", () => {
const continuations = [278, 245, 243].map(age => run({ wakeReason: "issue_continuation_needed", createdAt: minutesAgo(age),
sourceEndedAt: minutesAgo(age + 2) }));
const assignments = Array.from({ length: 80 }, (_, index) => run({ wakeReason: "issue_assigned", readinessRank: 1,
createdAt: minutesAgo(230 - index * 2) }));
const started: QueuedRunDispatchFacts[] = [];
let pending: QueuedRunDispatchFacts[] = [...continuations, ...assignments];
for (let step = 0; step < 3 * OLDEST_DISPATCH_SHARE; step += 1) {
pending.push(run({ wakeReason: "issue_recovery_action_restored", createdAt: minutesAgo(1), sourceEndedAt: minutesAgo(2) }),
run({ wakeReason: "issue_assigned", readinessRank: 1, createdAt: minutesAgo(1) }));
const recent = started.slice(-RECOVERY_DISPATCH_SHARE).reverse();
const lastRecovery = recent.findIndex(entry => isRecoveryWakeReason(entry.wakeReason));
const oldest = pending.reduce((left, right) => left.createdAt <= right.createdAt ? left : right);
const recentForOldest = started.slice(-OLDEST_DISPATCH_SHARE).reverse();
const lastOldest = recentForOldest.findIndex(entry => entry.createdAt <= oldest.createdAt);
const [next] = orderQueuedRunsForDispatch(pending, { now,
startsSinceRecovery: lastRecovery < 0 ? recent.length : lastRecovery,
startsSinceOldest: lastOldest < 0 ? recentForOldest.length : lastOldest });
started.push(next!);
pending = pending.filter(entry => entry.id !== next!.id);
}
for (const continuation of continuations) expect(started.map(entry => entry.id)).toContain(continuation.id);
});
});
@@ -14,6 +14,10 @@ export const RECOVERY_BACKLOG_AGE_MS = 30 * 60_000;
export const RECOVERY_DISPATCH_SHARE = 4;
export const QUEUED_RUN_AGING_MS = 60 * 60_000;
export const OLDEST_DISPATCH_SHARE = 4;
export type QueuedRunDispatchFacts = {
id: string;
createdAt: Date;
@@ -22,6 +26,7 @@ export type QueuedRunDispatchFacts = {
sourceEndedAt: Date | null;
readinessRank: number;
priorityRank: number;
shareDeferred?: boolean;
};
export type QueuedRunDispatchClass = "person" | "work" | "recovery" | "recovery_backlog";
@@ -46,20 +51,42 @@ export function queuedRunDispatchClass(run: QueuedRunDispatchFacts, now: Date):
const CLASS_TIER: Record<QueuedRunDispatchClass, number> = { person: 0, work: 1, recovery: 1, recovery_backlog: 2 };
const NOT_READY_RANK = 3;
export type QueuedRunDispatchInput = { now: Date; startsSinceRecovery: number; startsSinceOldest?: number };
export function hasQueuedRunAged(run: Pick<QueuedRunDispatchFacts, "createdAt">, now: Date) {
return now.getTime() - run.createdAt.getTime() >= QUEUED_RUN_AGING_MS;
}
export function oldestReadyQueuedRun<T extends QueuedRunDispatchFacts>(runs: readonly T[]): T | null {
let oldest: T | null = null;
for (const run of runs) {
if (run.readinessRank >= NOT_READY_RANK || run.shareDeferred) continue;
if (!oldest || run.createdAt.getTime() < oldest.createdAt.getTime()) oldest = run;
}
return oldest;
}
export function createQueuedRunDispatcher<T extends QueuedRunDispatchFacts>(
runs: readonly T[],
input: { now: Date; startsSinceRecovery: number },
input: QueuedRunDispatchInput,
) {
const tier = (run: T) => run.readinessRank >= NOT_READY_RANK ? 3 : CLASS_TIER[queuedRunDispatchClass(run, input.now)];
const tier = (run: T) => {
if (run.readinessRank >= NOT_READY_RANK) return 3;
const classTier = CLASS_TIER[queuedRunDispatchClass(run, input.now)];
return hasQueuedRunAged(run, input.now) ? Math.min(classTier, CLASS_TIER.work) : classTier;
};
const remaining = [...runs].sort((left, right) =>
tier(left) - tier(right)
|| left.readinessRank - right.readinessRank
|| left.priorityRank - right.priorityRank
|| left.createdAt.getTime() - right.createdAt.getTime());
let sinceRecovery = input.startsSinceRecovery;
let sinceOldest = input.startsSinceOldest ?? 0;
return {
next(): T | null {
if (!remaining.length) return null;
const oldest = sinceOldest >= OLDEST_DISPATCH_SHARE - 1 ? oldestReadyQueuedRun(remaining) : null;
if (oldest && hasQueuedRunAged(oldest, input.now)) return remaining.splice(remaining.indexOf(oldest), 1)[0]!;
const recoveryIndex = sinceRecovery >= RECOVERY_DISPATCH_SHARE - 1
? remaining.findIndex(run => run.readinessRank < NOT_READY_RANK && isSystemRecoveryRun(run))
: -1;
@@ -67,13 +94,15 @@ export function createQueuedRunDispatcher<T extends QueuedRunDispatchFacts>(
},
claimed(run: T) {
sinceRecovery = isSystemRecoveryRun(run) ? 0 : sinceRecovery + 1;
const oldest = oldestReadyQueuedRun(remaining);
sinceOldest = !oldest || oldest.createdAt.getTime() >= run.createdAt.getTime() ? 0 : sinceOldest + 1;
},
};
}
export function orderQueuedRunsForDispatch<T extends QueuedRunDispatchFacts>(
runs: readonly T[],
input: { now: Date; startsSinceRecovery: number },
input: QueuedRunDispatchInput,
): T[] {
const dispatcher = createQueuedRunDispatcher(runs, input);
const result: T[] = [];