mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
feat(queue): show tasks waiting for an agent's slot as waiting, with their place in line (#21405)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each agent runs at most `runtimeConfig.heartbeat.maxConcurrentRuns` tasks at once. Other runs wait in the agent's queue with status `queued` > - On hosted staging, one agent had all 20 slots full and 295 queued runs. Some tasks waited 3 to 7 hours and never started > - The UI showed these tasks as live or working: a pulsing "Live" badge in the task header and in task rows, a spinning "Working … for 4 hours" in the task chat, and "N live" counts that included every queued run > - People could not tell that work was waiting, why it waited, or what to change > - This pull request makes a deep queue an honest, visible state: "Waiting for Codie · 20 slots max · 41 of 295 in line · Configure" > - The benefit is that operators see the real state and the one setting that changes it ## Linked Issues or Issue Description Refs #21391, #21349, #21242 **What happened?** A queued run that never started looked the same as a running run. The task header, the task rows in lists and the inbox, the task chat and the sidebar counts all showed queued runs as live or working. **Expected behavior** A run that waits for one of its agent's concurrency slots shows that it waits. It names the agent, the agent's limit and the run's place in line, and it links to where the limit is set. The place in line is the order in which the server starts the agent's queued runs. **Why does it matter?** With a deep queue, the UI said "working" for hours while nothing ran. Operators could not see the cause or the fix. ## What Changed - Server: a new module `server/src/services/heartbeat/queued-run-line.ts` now holds the facts that the queued-run dispatcher orders by (`queuedRunDispatchFacts`). The dispatch path in `queue.ts` and the new "place in line" read use this one builder, so the line the UI shows is the dispatch order. The logic moved without change, including the recovery-share fix from #21391. - Server: `executionProjectionsForRuns` adds `agentQueue` (agent, limit, running count, position, queued count) and the label "Waiting for <agent>" with cause `agent_at_capacity` to a queued, never-started run when its agent has no free slot. It reads one batch of agents and running counts. Then it builds one line per full agent (at most 8 agents per read), using only the three context keys that ordering needs. No new polling. - Server: the place in line uses `queuedRunDispatchLine`, an O(n log n) replay of the dispatcher. The plain replay rescans the queue after each step, which is quadratic (measured: 3.6 ms at 300 runs, 76 ms at 3,000, 600 ms at 10,000, all on the event loop). Dispatch itself is unchanged: it stops after the free slots. Each agent's line is memoized for 5 s and shared by concurrent reads. In a preview worktree, the line applies the same creation-time cutoff as dispatch. - Server: the task list summary adds `runningDescendantCount` (descendants whose run started) next to `liveDescendantCount`. `liveDescendantCount` still decides whether a blocked parent is covered. The "n live below" chip uses the new count. - Shared: the new `ExecutionAgentQueue` type and the optional field `ExecutionProjection.agentQueue`. - Server: `/companies/:companyId/live-runs` lists running runs before queued runs, so a deep queue cannot push running work out of the 50-row limit. - UI: a new `AgentQueueWait` component shows "Waiting for Codie · 20 slots max · 41 of 295 in line · Configure", with no spinner and no elapsed timer. Configure opens the agent's Advanced view, where the run policy (`maxConcurrentRuns`) is edited. The task chat (both renderers), the run ledger and the Storybook story use it. - UI: the task header and the task rows (task list, inbox, legacy views) show a quiet "Waiting" instead of the pulsing "Live" when every live run of the task has not started. The full sentence shows on hover. "n live below" counts only started sub-tasks. A waiting sub-task still covers its blocked parent. - UI: the sidebar "N live" counts (Dashboard, agents) count only started runs. - UI: the task chat pill now names other server holds on a queued run (for example "Waiting for maintenance") instead of "Working". - Docs: DESIGN.md records the waiting-for-slot hold. - Storybook: `runs-agent-queue-wait` stories (full line, task chat, task row, one slot, mobile), with play checks. ## Verification - Red/green server test: `server/src/services/execution-projection-agent-queue.test.ts` (embedded Postgres). Before this change, the projection has no `agentQueue`. Now it checks the agent name, limit, running count, positions 1 to 4 in dispatch order (a person's retry first, then priority, not creation order), no wait while a slot is free, and isolation from other companies' runs. It also checks that the positions equal the line that the dispatch builder produces. - `pnpm exec vitest run` on the new test, `queued-run-dispatch-order`, `recovery-wake-herd`, `heartbeat-queued-run-claim-isolation` and `agent-live-run-routes` in `server/`. - Runtime cost (rule `runtime-dependency-boundaries`): the assumption is that the line stays cheap for a deep queue. `queued-run-line.test.ts` checks that `queuedRunDispatchLine` equals the dispatcher's claim-every-run order over 400 random queues (ages around the aging boundary, recovery and person runs, blocked runs, deferred share runs, all share counters). It also orders 10,000 runs in under 250 ms (the quadratic replay takes about 600 ms). The DB test projects a 300-run queue end to end with correct positions 1 to 300. Remaining limit: the line is up to 5 s stale. - Red/green: `issue-list-assignee-filter-routes.test.ts` now asserts `runningDescendantCount: 0` for a parent whose only live descendant is queued and never started. - UI tests: `liveIssueIds.test.ts` (waiting tasks, started runs keep "Live", no "running below" count for waiting sub-tasks) and `TaskChatLiveRunPill.test.tsx` (waiting line with no spinner and the Configure link, other holds named). The suites for the run ledger, sidebar, sidebar agents, task detail, task columns and task chat also pass. - The Storybook stories render in headless Chromium and their play checks pass. - After deploy: open a queued task on staging and confirm the waiting state. ## Risks - Cost on read: a projection read that includes a queued run of a full agent now reads that agent's queued runs (only small columns) and the dispatch facts. This is at most one line per full agent every 5 s, for at most 8 agents per read. Reads with no queued, never-started runs do no more work. - The place in line is the server's prediction of dispatch order. A claim that is refused at start time (a gate, a blocker) can let the next run start first. The position updates on the next refresh. - The process-local "declined aged claim" hint that the dispatcher uses is not visible to other servers, so the projection does not use it. This can change only the aged-oldest share step. - Configure opens the agent's Advanced view (the YAML run policy). The agent's Runtime tab hides the run-policy fields today. - Low risk otherwise: no schema change and no migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, headless browser) and subagents for the UI survey. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
bef2aac905
commit
e7e28ba241
36 files changed
+1095
-217
No files matched your search
@@ -1087,6 +1087,9 @@ describeEmbeddedPostgres("issue list routes assigneeAgentId filter", () => {
|
||||
expect(withSummary.body[0]).toMatchObject({
|
||||
id: rootIssueId,
|
||||
liveDescendantCount: 1,
|
||||
// The live grandchild's run is queued and never started: it covers the
|
||||
// blocked parent but is not running below it.
|
||||
runningDescendantCount: 0,
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -7262,7 +7262,9 @@ export function agentRoutes(
|
||||
inArray(heartbeatRuns.status, ["queued", "running"]),
|
||||
),
|
||||
)
|
||||
.orderBy(desc(heartbeatRuns.createdAt));
|
||||
// Running runs first: a deep queue must not push the work actually
|
||||
// running past the limit.
|
||||
.orderBy(sql`case when ${heartbeatRuns.status} = 'running' then 0 else 1 end`, desc(heartbeatRuns.createdAt));
|
||||
|
||||
const liveRuns = distinctTasks ? [] : await liveRunsQuery.limit(limit);
|
||||
let rows = liveRuns;
|
||||
|
||||
@@ -3113,6 +3113,9 @@ function toCompactIssue(issue: any): CompactIssue {
|
||||
...(issue.liveDescendantCount !== undefined
|
||||
? { liveDescendantCount: issue.liveDescendantCount }
|
||||
: {}),
|
||||
...(issue.runningDescendantCount !== undefined
|
||||
? { runningDescendantCount: issue.runningDescendantCount }
|
||||
: {}),
|
||||
...(issue.myLastTouchAt !== undefined
|
||||
? { myLastTouchAt: issue.myLastTouchAt }
|
||||
: {}),
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
import { agents, companies, createDb, heartbeatRuns, issues } from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
||||
import { executionProjectionsForRuns } from "./execution-projection.js";
|
||||
import { agentQueuedRunLine, clearAgentQueueLineCache, worktreeExecutionCutoff } from "./heartbeat/queued-run-line.js";
|
||||
import { issueService } from "./issues.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const now = new Date("2026-11-13T12:00:00Z");
|
||||
const minutesAgo = (minutes: number) => new Date(now.getTime() - minutes * 60_000);
|
||||
|
||||
(support.supported ? describe : describe.skip)("execution projection: waiting behind the agent's limit", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
beforeAll(async () => {
|
||||
database = await startEmbeddedPostgresTestDatabase("agent-queue-projection-");
|
||||
db = createDb(database.connectionString);
|
||||
}, 60_000);
|
||||
afterAll(async () => { await database?.cleanup(); });
|
||||
beforeEach(() => clearAgentQueueLineCache());
|
||||
|
||||
async function fixture(options: { maxConcurrentRuns: number; running: number }) {
|
||||
const companyId = randomUUID(), agentId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Queue test", issuePrefix: `Q${companyId.slice(0, 8)}` });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Codie", adapterType: "claude_local",
|
||||
runtimeConfig: { heartbeat: { maxConcurrentRuns: options.maxConcurrentRuns } } });
|
||||
for (let index = 0; index < options.running; index += 1) {
|
||||
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running",
|
||||
startedAt: minutesAgo(90), createdAt: minutesAgo(90) });
|
||||
}
|
||||
async function queue(input: { priority: "critical" | "high" | "medium" | "low"; createdMinutesAgo: number; wakeReason?: string }) {
|
||||
const issueId = randomUUID(), runId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: `Task ${input.priority}`, assigneeAgentId: agentId,
|
||||
status: "in_progress", priority: input.priority });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "queued", issueId,
|
||||
createdAt: minutesAgo(input.createdMinutesAgo),
|
||||
contextSnapshot: { issueId, ...(input.wakeReason ? { wakeReason: input.wakeReason } : {}) } });
|
||||
return runId;
|
||||
}
|
||||
return { companyId, agentId, queue };
|
||||
}
|
||||
|
||||
it("names the agent, its limit and each run's place in the order dispatch starts them", async () => {
|
||||
const f = await fixture({ maxConcurrentRuns: 2, running: 2 });
|
||||
// Created oldest first, but dispatch starts a person's retry first, then by priority.
|
||||
const lowOld = await f.queue({ priority: "low", createdMinutesAgo: 30 });
|
||||
const medium = await f.queue({ priority: "medium", createdMinutesAgo: 20 });
|
||||
const critical = await f.queue({ priority: "critical", createdMinutesAgo: 10 });
|
||||
const personRetry = await f.queue({ priority: "low", createdMinutesAgo: 5, wakeReason: "task_retry" });
|
||||
|
||||
const projections = await executionProjectionsForRuns(db, f.companyId, [lowOld, medium, critical, personRetry], now);
|
||||
const queueOf = (runId: string) => projections.get(runId)?.agentQueue;
|
||||
expect(projections.get(critical)).toMatchObject({
|
||||
phase: "queued",
|
||||
label: "Waiting for Codie",
|
||||
cause: "agent_at_capacity",
|
||||
agentQueue: { agentId: f.agentId, agentName: "Codie", maxConcurrentRuns: 2, runningCount: 2, position: 2, queuedCount: 4 },
|
||||
});
|
||||
expect([personRetry, critical, medium, lowOld].map(runId => queueOf(runId)?.position)).toEqual([1, 2, 3, 4]);
|
||||
|
||||
// The position is the dispatch line itself, not a client-side sort.
|
||||
const [agent] = await db.select().from(agents).where(eq(agents.id, f.agentId));
|
||||
expect(await agentQueuedRunLine(db, issueService(db), agent!, now)).toEqual([personRetry, critical, medium, lowOld]);
|
||||
});
|
||||
|
||||
it("does not claim a run waits on the limit while the agent has a free slot", async () => {
|
||||
const f = await fixture({ maxConcurrentRuns: 3, running: 2 });
|
||||
const runId = await f.queue({ priority: "high", createdMinutesAgo: 5 });
|
||||
const projection = (await executionProjectionsForRuns(db, f.companyId, [runId], now)).get(runId);
|
||||
expect(projection).toMatchObject({ phase: "queued", label: "Queued", cause: null });
|
||||
expect(projection?.agentQueue).toBeUndefined();
|
||||
|
||||
// Once the limit is full the same run explains the wait.
|
||||
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 2 } } }).where(eq(agents.id, f.agentId));
|
||||
expect((await executionProjectionsForRuns(db, f.companyId, [runId], now)).get(runId)).toMatchObject({
|
||||
label: "Waiting for Codie",
|
||||
agentQueue: { maxConcurrentRuns: 2, runningCount: 2, position: 1, queuedCount: 1 },
|
||||
});
|
||||
});
|
||||
|
||||
it("leaves out queued runs that a preview worktree's cutoff keeps from starting", async () => {
|
||||
const f = await fixture({ maxConcurrentRuns: 1, running: 1 });
|
||||
const copiedOld = await f.queue({ priority: "critical", createdMinutesAgo: 120 });
|
||||
const fresh = await f.queue({ priority: "low", createdMinutesAgo: 5 });
|
||||
const [agent] = await db.select().from(agents).where(eq(agents.id, f.agentId));
|
||||
expect(await agentQueuedRunLine(db, issueService(db), agent!, now, { cutoff: null })).toEqual([copiedOld, fresh]);
|
||||
expect(await agentQueuedRunLine(db, issueService(db), agent!, now, { cutoff: minutesAgo(60) })).toEqual([fresh]);
|
||||
// Outside a worktree there is no cutoff and no settings read.
|
||||
expect(await worktreeExecutionCutoff(db, {})).toBeNull();
|
||||
});
|
||||
|
||||
it("keeps a deep queue's place in line to a few reads, not one per run", async () => {
|
||||
const f = await fixture({ maxConcurrentRuns: 20, running: 20 });
|
||||
const runIds: string[] = [];
|
||||
for (let index = 0; index < 300; index += 1) {
|
||||
runIds.push(await f.queue({ priority: (["critical", "high", "medium", "low"] as const)[index % 4], createdMinutesAgo: 300 - index,
|
||||
...(index % 9 === 0 ? { wakeReason: "issue_continuation_needed" } : {}) }));
|
||||
}
|
||||
const started = performance.now();
|
||||
const projections = await executionProjectionsForRuns(db, f.companyId, runIds, now);
|
||||
const elapsedMs = performance.now() - started;
|
||||
const positions = runIds.map(runId => projections.get(runId)?.agentQueue?.position).sort((a, b) => a! - b!);
|
||||
expect(positions).toEqual(runIds.map((_, index) => index + 1));
|
||||
expect(projections.get(runIds[0]!)?.agentQueue?.queuedCount).toBe(300);
|
||||
// Generous bound for slow CI; the ordering itself is O(n log n).
|
||||
expect(elapsedMs).toBeLessThan(5_000);
|
||||
});
|
||||
|
||||
it("leaves running runs and other companies' runs out of the line", async () => {
|
||||
const f = await fixture({ maxConcurrentRuns: 1, running: 1 });
|
||||
const mine = await f.queue({ priority: "medium", createdMinutesAgo: 5 });
|
||||
const other = await fixture({ maxConcurrentRuns: 1, running: 1 });
|
||||
await other.queue({ priority: "critical", createdMinutesAgo: 50 });
|
||||
const projection = (await executionProjectionsForRuns(db, f.companyId, [mine], now)).get(mine);
|
||||
expect(projection?.agentQueue).toMatchObject({ position: 1, queuedCount: 1, runningCount: 1 });
|
||||
});
|
||||
});
|
||||
@@ -22,6 +22,7 @@ import { getStartupRecoveryState, startupQueuedRunPassOutstanding } from "../sta
|
||||
import { hasPendingRunCleanup } from "./execution-blocker.js";
|
||||
import { isAgentAwaitingSetup } from "../modules/agent-lifecycle/index.js";
|
||||
import { NATIVE_RESPONSE_WAKE_WAITING_REASON } from "./native-runtime/status-arbiter.js";
|
||||
import { agentQueueLinesForRuns } from "./heartbeat/queued-run-line.js";
|
||||
const MAX_PREDECESSOR_CLEANUP_CHECKS = 16;
|
||||
const text = (v: unknown) => (typeof v === "string" ? v : null);
|
||||
const executionRunColumns = {
|
||||
@@ -203,6 +204,11 @@ export async function executionProjectionsForRuns(
|
||||
retryRequired: sql<boolean>`coalesce(${agents.lifecycleOperation}->>'retryRequired', 'false') = 'true'` })
|
||||
.from(agents).where(and(eq(agents.companyId, companyId), inArray(agents.id, setupAgentIds)))) : [];
|
||||
const setupByAgent = new Map(setupAgents.map(agent => [agent.id, agent]));
|
||||
// Only never-started runs wait on the agent's limit; their place in line is
|
||||
// read once per full agent, not once per run.
|
||||
const agentQueueByRun = queuedRuns.some(run => !run.startedAt)
|
||||
? await read(() => agentQueueLinesForRuns(db, companyId, queuedRuns.filter(run => !run.startedAt), now))
|
||||
: new Map<string, NonNullable<ExecutionProjection["agentQueue"]>>();
|
||||
const issueIdOf = (run: Run) => run.nativeIssueId ?? text(run.contextSnapshot?.issueId);
|
||||
const queuedIssueIds = [...new Set(queuedRuns.map(issueIdOf).filter((id): id is string => !!id))];
|
||||
if (queuedIssueIds.length) {
|
||||
@@ -278,6 +284,7 @@ export async function executionProjectionsForRuns(
|
||||
settlingPredecessorKeys.has(run.id),
|
||||
setupByAgent.get(run.agentId),
|
||||
currentResponseWait(run),
|
||||
agentQueueByRun.get(run.id),
|
||||
),
|
||||
);
|
||||
}
|
||||
@@ -305,6 +312,7 @@ export function projectExecution(
|
||||
waitingForPredecessor = false,
|
||||
setupAgent?: { lifecycleState: string; lifecycleHolds: readonly string[]; retryRequired?: boolean },
|
||||
responseWaitCurrent = false,
|
||||
agentQueue?: ExecutionProjection["agentQueue"],
|
||||
): ExecutionProjection {
|
||||
const detail = coordinator?.failureDetail ?? {};
|
||||
const successorRunId = text(detail.successorRunId);
|
||||
@@ -553,5 +561,11 @@ export function projectExecution(
|
||||
projection.nextAction = "The previous run on this task is still releasing its runner. This turn starts automatically when it settles.";
|
||||
return set("queued", "Waiting for previous run to stop");
|
||||
}
|
||||
if (run.status === "queued" && !run.startedAt && agentQueue) {
|
||||
projection.cause = "agent_at_capacity";
|
||||
projection.agentQueue = agentQueue;
|
||||
projection.nextAction = `${agentQueue.agentName} is running ${agentQueue.runningCount} of ${agentQueue.maxConcurrentRuns} tasks at once. This task is ${agentQueue.position} of ${agentQueue.queuedCount} in line and starts automatically when a slot frees.`;
|
||||
return set("queued", `Waiting for ${agentQueue.agentName}`);
|
||||
}
|
||||
return projection;
|
||||
}
|
||||
@@ -12,8 +12,8 @@ import { hasAutomaticInteractionDeliveryHold, INTERACTION_DELIVERY_DENIED, reche
|
||||
import { deferredWorkIntentWait } from "../work-intent-outbox.js";
|
||||
import { notifyIssueRecovery } from "../issue-recovery-notifications.js";
|
||||
import { createHeartbeatTimerPolicy } from "./timer-policy.js";
|
||||
import { OLDEST_DISPATCH_SHARE, PERSON_WAKE_REASONS, QUEUED_RUN_AGING_MS, RECOVERY_DISPATCH_SHARE, createQueuedRunDispatcher,
|
||||
hasQueuedRunAged, isRecoveryWakeReason, isSystemRecoveryRun, oldestReadyQueuedRun } from "./queued-run-dispatch-order.js";
|
||||
import { QUEUED_RUN_AGING_MS, createQueuedRunDispatcher, hasQueuedRunAged } from "./queued-run-dispatch-order.js";
|
||||
import { queuedRunDispatchFacts } from "./queued-run-line.js";
|
||||
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
|
||||
import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../deferred-execution-wait.js";
|
||||
import { assignmentWakePayloadSql, isAssignmentWakePayload, notifyDeferredExecutionWork } from "../deferred-execution-notifications.js";
|
||||
@@ -375,8 +375,6 @@ export function shouldQueueFollowupForRunningIssueWake(input: {
|
||||
|
||||
/** Agent Chat outbox receipts. The outbox, not a skipped receipt, owns a capped message. */
|
||||
const CONVERSATION_COMMENT_KEY_PREFIX = "conversation-comment:";
|
||||
const RUN_ID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
|
||||
const PERSON_ORIGIN_CHAIN_DEPTH = 8;
|
||||
function dailyLimitConversationRefusal(cap: { reason: string; observed: number; limit: number }) {
|
||||
// No receipt is written, so the person's message stays owed in the outbox
|
||||
// until the limit resets (agent-conversations.ts conversationDeliveryHold).
|
||||
@@ -597,21 +595,6 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
const claimDueTimerHeartbeat = (agent: typeof agents.$inferSelect, now: Date, intervalSec: number) =>
|
||||
claimDueTimerHeartbeatForAgent(db, agent, now, intervalSec);
|
||||
|
||||
function issueRunPriorityRank(priority: string | null | undefined) {
|
||||
switch (priority) {
|
||||
case "critical":
|
||||
return 0;
|
||||
case "high":
|
||||
return 1;
|
||||
case "medium":
|
||||
return 2;
|
||||
case "low":
|
||||
return 3;
|
||||
default:
|
||||
return 4;
|
||||
}
|
||||
}
|
||||
|
||||
async function listQueuedRunDependencyReadiness(
|
||||
companyId: string,
|
||||
queuedRuns: Array<typeof heartbeatRuns.$inferSelect>,
|
||||
@@ -2368,31 +2351,6 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
}
|
||||
}
|
||||
|
||||
async function listPersonOriginSourceRunIds(companyId: string, sourceRunIds: string[]) {
|
||||
const uuidPattern = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$";
|
||||
const result = await db.execute<{ root_id: string }>(sql`
|
||||
with recursive chain(root_id, run_id, depth) as (
|
||||
select source.id, source.id, 0 from ${heartbeatRuns} source
|
||||
where source.company_id = ${companyId} and source.id in (${sql.join(sourceRunIds.map(id => sql`${id}::uuid`), sql`, `)})
|
||||
union all
|
||||
select chain.root_id, parent.id, chain.depth + 1
|
||||
from chain
|
||||
join ${heartbeatRuns} child on child.id = chain.run_id
|
||||
join ${heartbeatRuns} parent on parent.company_id = ${companyId} and parent.id = coalesce(
|
||||
case when child.context_snapshot->>'previousRunId' ~* ${uuidPattern}
|
||||
then (child.context_snapshot->>'previousRunId')::uuid end,
|
||||
child.retry_of_run_id)
|
||||
where chain.depth < ${PERSON_ORIGIN_CHAIN_DEPTH}
|
||||
)
|
||||
select distinct chain.root_id from chain
|
||||
join ${heartbeatRuns} run on run.id = chain.run_id
|
||||
where run.context_snapshot->>'wakeReason' in (${sql.join([...PERSON_WAKE_REASONS].map(reason => sql`${reason}`), sql`, `)})
|
||||
or run.context_snapshot->>'taskRetryRequestId' is not null
|
||||
or exists (select 1 from ${agentWakeupRequests} wake where wake.company_id = ${companyId}
|
||||
and wake.requested_by_actor_type = 'user' and (wake.run_id = run.id or wake.id = run.wakeup_request_id))`);
|
||||
return [...result].map(row => String(row.root_id));
|
||||
}
|
||||
|
||||
async function queuedRunDispatcherForAgent(
|
||||
agent: typeof agents.$inferSelect,
|
||||
queuedRuns: Array<typeof heartbeatRuns.$inferSelect>,
|
||||
@@ -2401,93 +2359,9 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
) {
|
||||
const now = new Date();
|
||||
pruneDeclinedAgedClaims(now);
|
||||
const wakeReasonOf = (run: typeof heartbeatRuns.$inferSelect) =>
|
||||
readNonEmptyString(parseObject(run.contextSnapshot).wakeReason);
|
||||
const requestIds = [...new Set(queuedRuns.flatMap(run => run.wakeupRequestId ? [run.wakeupRequestId] : []))];
|
||||
const userRequests = queuedRuns.length > 1 && requestIds.length
|
||||
? await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, agent.companyId), inArray(agentWakeupRequests.id, requestIds),
|
||||
eq(agentWakeupRequests.requestedByActorType, "user")))
|
||||
: [];
|
||||
const userRequestIds = new Set(userRequests.map(row => row.id));
|
||||
const joinedUserRequests = queuedRuns.length > 1
|
||||
? await db.select({ runId: agentWakeupRequests.runId }).from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, agent.companyId), inArray(agentWakeupRequests.runId, queuedRuns.map(run => run.id)),
|
||||
eq(agentWakeupRequests.agentId, agent.id), eq(agentWakeupRequests.requestedByActorType, "user")))
|
||||
: [];
|
||||
const userRunIds = new Set(joinedUserRequests.flatMap(row => row.runId ? [row.runId] : []));
|
||||
const joinedByUser = (run: typeof heartbeatRuns.$inferSelect) =>
|
||||
userRunIds.has(run.id) || Boolean(run.wakeupRequestId && userRequestIds.has(run.wakeupRequestId));
|
||||
const sourceRunIdOf = (run: typeof heartbeatRuns.$inferSelect) =>
|
||||
readNonEmptyString(parseObject(run.contextSnapshot).previousRunId) ?? run.retryOfRunId ?? null;
|
||||
const sourceRunIds = queuedRuns.length > 1
|
||||
? [...new Set(queuedRuns.flatMap(run => {
|
||||
const sourceRunId = joinedByUser(run) ? null : sourceRunIdOf(run);
|
||||
return sourceRunId && RUN_ID_PATTERN.test(sourceRunId) ? [sourceRunId] : [];
|
||||
}))]
|
||||
: [];
|
||||
const sourceRows = sourceRunIds.length
|
||||
? await db.select({ id: heartbeatRuns.id, finishedAt: heartbeatRuns.finishedAt }).from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, agent.companyId), inArray(heartbeatRuns.id, sourceRunIds)))
|
||||
: [];
|
||||
const sourceEndedAt = new Map(sourceRows.map(row => [row.id, row.finishedAt]));
|
||||
const personSourceRunIds = new Set(sourceRows.length
|
||||
? await listPersonOriginSourceRunIds(agent.companyId, sourceRows.map(row => row.id))
|
||||
: []);
|
||||
const requestedByUser = (run: typeof heartbeatRuns.$inferSelect) => {
|
||||
if (joinedByUser(run)) return true;
|
||||
const sourceRunId = sourceRunIdOf(run);
|
||||
return Boolean(sourceRunId && personSourceRunIds.has(sourceRunId));
|
||||
};
|
||||
const facts = queuedRuns.map(run => {
|
||||
const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
|
||||
const ready = issueId ? (dependencyReadiness.get(issueId)?.isDependencyReady ?? true) : true;
|
||||
const issue = issueId ? issueById.get(issueId) : null;
|
||||
const sourceRunId = sourceRunIdOf(run);
|
||||
return {
|
||||
run,
|
||||
id: run.id,
|
||||
createdAt: run.createdAt,
|
||||
wakeReason: wakeReasonOf(run),
|
||||
requestedByUser: requestedByUser(run),
|
||||
sourceEndedAt: sourceRunId ? sourceEndedAt.get(sourceRunId) ?? null : null,
|
||||
readinessRank: issueId ? (ready ? (issue?.status === "in_progress" ? 0 : 1) : 3) : 2,
|
||||
priorityRank: issueRunPriorityRank(issue?.priority),
|
||||
shareDeferred: declinedAgedClaimAt.has(run.id),
|
||||
};
|
||||
});
|
||||
const recoveryRuns = facts.filter(entry => isSystemRecoveryRun(entry));
|
||||
const oldestReady = oldestReadyQueuedRun(facts);
|
||||
const agedOldest = oldestReady && hasQueuedRunAged(oldestReady, now) ? oldestReady : null;
|
||||
const countsRecovery = recoveryRuns.length > 0 && recoveryRuns.length < facts.length;
|
||||
let startsSinceRecovery = 0;
|
||||
let startsSinceOldest = 0;
|
||||
if (countsRecovery || (agedOldest && facts.length > 1)) {
|
||||
const oldestRecoveryAt = countsRecovery ? Math.min(...recoveryRuns.map(entry => entry.createdAt.getTime())) : Infinity;
|
||||
const since = new Date(Math.min(oldestRecoveryAt, agedOldest?.createdAt.getTime() ?? Infinity));
|
||||
const recentStarts = await db.select({ id: heartbeatRuns.id, wakeReason: sql<string | null>`${heartbeatRuns.contextSnapshot}->>'wakeReason'`,
|
||||
startedAt: heartbeatRuns.startedAt, createdAt: heartbeatRuns.createdAt })
|
||||
.from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id),
|
||||
isNotNull(heartbeatRuns.startedAt), gte(heartbeatRuns.startedAt, since)))
|
||||
.orderBy(desc(heartbeatRuns.startedAt))
|
||||
.limit(Math.max(RECOVERY_DISPATCH_SHARE, OLDEST_DISPATCH_SHARE));
|
||||
const startsSince = (at: number) => recentStarts.filter(row => row.startedAt && row.startedAt.getTime() >= at);
|
||||
if (countsRecovery) {
|
||||
const starts = startsSince(oldestRecoveryAt).slice(0, RECOVERY_DISPATCH_SHARE);
|
||||
const recoveryStartIds = starts.flatMap(row => isRecoveryWakeReason(row.wakeReason) ? [row.id] : []);
|
||||
const personStarts = new Set(recoveryStartIds.length
|
||||
? await listPersonOriginSourceRunIds(agent.companyId, recoveryStartIds) : []);
|
||||
const lastRecovery = starts.findIndex(row => isRecoveryWakeReason(row.wakeReason) && !personStarts.has(row.id));
|
||||
startsSinceRecovery = lastRecovery < 0 ? starts.length : lastRecovery;
|
||||
}
|
||||
if (agedOldest && facts.length > 1) {
|
||||
const starts = startsSince(agedOldest.createdAt.getTime()).slice(0, OLDEST_DISPATCH_SHARE);
|
||||
const lastOldest = starts.findIndex(row => row.createdAt.getTime() <= agedOldest.createdAt.getTime());
|
||||
startsSinceOldest = lastOldest < 0 ? starts.length : lastOldest;
|
||||
}
|
||||
}
|
||||
const dispatcher = createQueuedRunDispatcher(facts, { now, startsSinceRecovery, startsSinceOldest });
|
||||
const { facts, dispatchInput } = await queuedRunDispatchFacts(db, {
|
||||
agent, queuedRuns, dependencyReadiness, issueById, now, shareDeferred: runId => declinedAgedClaimAt.has(runId) });
|
||||
const dispatcher = createQueuedRunDispatcher(facts, dispatchInput);
|
||||
const byRun = new Map(facts.map(entry => [entry.run, entry]));
|
||||
return {
|
||||
next: () => dispatcher.next()?.run ?? null,
|
||||
|
||||
@@ -66,20 +66,24 @@ export function oldestReadyQueuedRun<T extends QueuedRunDispatchFacts>(runs: rea
|
||||
return oldest;
|
||||
}
|
||||
|
||||
export function createQueuedRunDispatcher<T extends QueuedRunDispatchFacts>(
|
||||
runs: readonly T[],
|
||||
input: QueuedRunDispatchInput,
|
||||
) {
|
||||
function sortForDispatch<T extends QueuedRunDispatchFacts>(runs: readonly T[], now: Date): T[] {
|
||||
const tier = (run: T) => {
|
||||
if (run.readinessRank >= NOT_READY_RANK) return 3;
|
||||
const classTier = CLASS_TIER[queuedRunDispatchClass(run, input.now)];
|
||||
return hasQueuedRunAged(run, input.now) ? Math.min(classTier, CLASS_TIER.work) : classTier;
|
||||
const classTier = CLASS_TIER[queuedRunDispatchClass(run, now)];
|
||||
return hasQueuedRunAged(run, now) ? Math.min(classTier, CLASS_TIER.work) : classTier;
|
||||
};
|
||||
const remaining = [...runs].sort((left, right) =>
|
||||
return [...runs].sort((left, right) =>
|
||||
tier(left) - tier(right)
|
||||
|| left.readinessRank - right.readinessRank
|
||||
|| left.priorityRank - right.priorityRank
|
||||
|| left.createdAt.getTime() - right.createdAt.getTime());
|
||||
}
|
||||
|
||||
export function createQueuedRunDispatcher<T extends QueuedRunDispatchFacts>(
|
||||
runs: readonly T[],
|
||||
input: QueuedRunDispatchInput,
|
||||
) {
|
||||
const remaining = sortForDispatch(runs, input.now);
|
||||
let sinceRecovery = input.startsSinceRecovery;
|
||||
let sinceOldest = input.startsSinceOldest ?? 0;
|
||||
return {
|
||||
@@ -112,3 +116,62 @@ export function orderQueuedRunsForDispatch<T extends QueuedRunDispatchFacts>(
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* The whole dispatch order in O(n log n), for showing each queued run's place
|
||||
* in line. It replays exactly what {@link createQueuedRunDispatcher} does when
|
||||
* every offered run is claimed ({@link orderQueuedRunsForDispatch}), without
|
||||
* that replay's rescans of the remaining queue after each step. Dispatch itself
|
||||
* stops after the free slots, so it keeps the simpler dispatcher.
|
||||
*/
|
||||
export function queuedRunDispatchLine<T extends QueuedRunDispatchFacts>(
|
||||
runs: readonly T[],
|
||||
input: QueuedRunDispatchInput,
|
||||
): T[] {
|
||||
const order = sortForDispatch(runs, input.now);
|
||||
const removed = new Uint8Array(order.length);
|
||||
// Eligible for the oldest share, oldest first; ties keep dispatch order,
|
||||
// as oldestReadyQueuedRun keeps the first of equal ages.
|
||||
const byAge = order.map((_, index) => index)
|
||||
.filter(index => order[index]!.readinessRank < NOT_READY_RANK && !order[index]!.shareDeferred)
|
||||
.sort((left, right) => order[left]!.createdAt.getTime() - order[right]!.createdAt.getTime() || left - right);
|
||||
const recovery = order.map((_, index) => index)
|
||||
.filter(index => order[index]!.readinessRank < NOT_READY_RANK && isSystemRecoveryRun(order[index]!));
|
||||
let head = 0;
|
||||
let agePointer = 0;
|
||||
let recoveryPointer = 0;
|
||||
const firstOf = (indexes: number[] | null, pointer: number) => {
|
||||
if (!indexes) {
|
||||
while (head < order.length && removed[head]) head += 1;
|
||||
return head < order.length ? head : -1;
|
||||
}
|
||||
while (pointer < indexes.length && removed[indexes[pointer]!]) pointer += 1;
|
||||
return pointer;
|
||||
};
|
||||
const oldestIndex = () => {
|
||||
agePointer = firstOf(byAge, agePointer);
|
||||
return agePointer < byAge.length ? byAge[agePointer]! : -1;
|
||||
};
|
||||
let sinceRecovery = input.startsSinceRecovery;
|
||||
let sinceOldest = input.startsSinceOldest ?? 0;
|
||||
const result: T[] = [];
|
||||
while (result.length < order.length) {
|
||||
let pick = -1;
|
||||
if (sinceOldest >= OLDEST_DISPATCH_SHARE - 1) {
|
||||
const oldest = oldestIndex();
|
||||
if (oldest >= 0 && hasQueuedRunAged(order[oldest]!, input.now)) pick = oldest;
|
||||
}
|
||||
if (pick < 0 && sinceRecovery >= RECOVERY_DISPATCH_SHARE - 1) {
|
||||
recoveryPointer = firstOf(recovery, recoveryPointer);
|
||||
if (recoveryPointer < recovery.length) pick = recovery[recoveryPointer]!;
|
||||
}
|
||||
if (pick < 0) pick = firstOf(null, 0);
|
||||
removed[pick] = 1;
|
||||
const run = order[pick]!;
|
||||
result.push(run);
|
||||
sinceRecovery = isSystemRecoveryRun(run) ? 0 : sinceRecovery + 1;
|
||||
const oldest = oldestIndex();
|
||||
sinceOldest = oldest < 0 || order[oldest]!.createdAt.getTime() >= run.createdAt.getTime() ? 0 : sinceOldest + 1;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
OLDEST_DISPATCH_SHARE,
|
||||
QUEUED_RUN_AGING_MS,
|
||||
RECOVERY_DISPATCH_SHARE,
|
||||
orderQueuedRunsForDispatch,
|
||||
queuedRunDispatchLine,
|
||||
type QueuedRunDispatchFacts,
|
||||
} from "./queued-run-dispatch-order.js";
|
||||
|
||||
const now = new Date("2026-11-13T12:00:00Z");
|
||||
const reasons = [null, null, "issue_continuation_needed", "issue_recovery_action_restored", "task_retry", "issue_assigned"];
|
||||
|
||||
// Small deterministic PRNG so failures reproduce.
|
||||
function random(seed: number) {
|
||||
let state = seed >>> 0;
|
||||
return () => {
|
||||
state = (state * 1664525 + 1013904223) >>> 0;
|
||||
return state / 2 ** 32;
|
||||
};
|
||||
}
|
||||
|
||||
function queue(seed: number, size: number): QueuedRunDispatchFacts[] {
|
||||
const next = random(seed);
|
||||
const pick = <T>(values: readonly T[]) => values[Math.floor(next() * values.length)]!;
|
||||
return Array.from({ length: size }, (_, index) => {
|
||||
// Coarse ages so equal creation times and the aging boundary both occur.
|
||||
const createdAt = new Date(now.getTime() - Math.floor(next() * 12) * (QUEUED_RUN_AGING_MS / 4));
|
||||
return {
|
||||
id: `run-${index}`,
|
||||
createdAt,
|
||||
wakeReason: pick(reasons),
|
||||
requestedByUser: next() < 0.15,
|
||||
sourceEndedAt: next() < 0.5 ? new Date(createdAt.getTime() - Math.floor(next() * 4) * 20 * 60_000) : null,
|
||||
readinessRank: pick([0, 0, 1, 2, 3]),
|
||||
priorityRank: pick([0, 1, 2, 3, 4]),
|
||||
shareDeferred: next() < 0.1,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
describe("queuedRunDispatchLine", () => {
|
||||
it("matches the dispatcher's own claim-every-run order", () => {
|
||||
for (let seed = 1; seed <= 400; seed += 1) {
|
||||
const runs = queue(seed, 1 + (seed % 60));
|
||||
const input = {
|
||||
now,
|
||||
startsSinceRecovery: seed % (RECOVERY_DISPATCH_SHARE + 1),
|
||||
startsSinceOldest: seed % (OLDEST_DISPATCH_SHARE + 1),
|
||||
};
|
||||
expect(queuedRunDispatchLine(runs, input).map(run => run.id), `seed ${seed}`)
|
||||
.toEqual(orderQueuedRunsForDispatch(runs, input).map(run => run.id));
|
||||
}
|
||||
});
|
||||
|
||||
it("orders a 10,000-run queue without rescanning it per step", () => {
|
||||
const runs = queue(7, 10_000);
|
||||
const input = { now, startsSinceRecovery: 0, startsSinceOldest: 0 };
|
||||
const started = performance.now();
|
||||
const line = queuedRunDispatchLine(runs, input);
|
||||
const elapsedMs = performance.now() - started;
|
||||
expect(line).toHaveLength(10_000);
|
||||
expect(new Set(line.map(run => run.id)).size).toBe(10_000);
|
||||
// The quadratic replay takes about 600 ms here; the line takes a few ms.
|
||||
expect(elapsedMs).toBeLessThan(250);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,316 @@
|
||||
import { and, asc, desc, eq, gte, inArray, isNotNull, sql } from "drizzle-orm";
|
||||
import { agentWakeupRequests, agents, heartbeatRuns, issues, type Db } from "@paperclipai/db";
|
||||
import type { ExecutionAgentQueue } from "@paperclipai/shared";
|
||||
import { parseObject } from "../../adapters/utils.js";
|
||||
import {
|
||||
OLDEST_DISPATCH_SHARE,
|
||||
PERSON_WAKE_REASONS,
|
||||
RECOVERY_DISPATCH_SHARE,
|
||||
hasQueuedRunAged,
|
||||
isRecoveryWakeReason,
|
||||
isSystemRecoveryRun,
|
||||
oldestReadyQueuedRun,
|
||||
queuedRunDispatchLine,
|
||||
type QueuedRunDispatchFacts,
|
||||
type QueuedRunDispatchInput,
|
||||
} from "./queued-run-dispatch-order.js";
|
||||
import { parseHeartbeatPolicy } from "./timer-policy.js";
|
||||
import { instanceSettingsService, isTruthyRuntimeEnvValue, resolveWorktreeRunExecutionActivation } from "../instance-settings.js";
|
||||
|
||||
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
|
||||
type Agent = typeof agents.$inferSelect;
|
||||
|
||||
const RUN_ID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
|
||||
const PERSON_ORIGIN_CHAIN_DEPTH = 8;
|
||||
|
||||
/** The run fields dispatch ordering reads. */
|
||||
export type QueuedRunDispatchRun = Pick<HeartbeatRun, "id" | "createdAt" | "wakeupRequestId" | "retryOfRunId" | "contextSnapshot">;
|
||||
|
||||
function readNonEmptyString(value: unknown): string | null {
|
||||
return typeof value === "string" && value.trim().length > 0 ? value : null;
|
||||
}
|
||||
|
||||
export function issueRunPriorityRank(priority: string | null | undefined) {
|
||||
switch (priority) {
|
||||
case "critical": return 0;
|
||||
case "high": return 1;
|
||||
case "medium": return 2;
|
||||
case "low": return 3;
|
||||
default: return 4;
|
||||
}
|
||||
}
|
||||
|
||||
async function listPersonOriginSourceRunIds(db: Db, companyId: string, sourceRunIds: string[]) {
|
||||
const uuidPattern = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$";
|
||||
const result = await db.execute<{ root_id: string }>(sql`
|
||||
with recursive chain(root_id, run_id, depth) as (
|
||||
select source.id, source.id, 0 from ${heartbeatRuns} source
|
||||
where source.company_id = ${companyId} and source.id in (${sql.join(sourceRunIds.map(id => sql`${id}::uuid`), sql`, `)})
|
||||
union all
|
||||
select chain.root_id, parent.id, chain.depth + 1
|
||||
from chain
|
||||
join ${heartbeatRuns} child on child.id = chain.run_id
|
||||
join ${heartbeatRuns} parent on parent.company_id = ${companyId} and parent.id = coalesce(
|
||||
case when child.context_snapshot->>'previousRunId' ~* ${uuidPattern}
|
||||
then (child.context_snapshot->>'previousRunId')::uuid end,
|
||||
child.retry_of_run_id)
|
||||
where chain.depth < ${PERSON_ORIGIN_CHAIN_DEPTH}
|
||||
)
|
||||
select distinct chain.root_id from chain
|
||||
join ${heartbeatRuns} run on run.id = chain.run_id
|
||||
where run.context_snapshot->>'wakeReason' in (${sql.join([...PERSON_WAKE_REASONS].map(reason => sql`${reason}`), sql`, `)})
|
||||
or run.context_snapshot->>'taskRetryRequestId' is not null
|
||||
or exists (select 1 from ${agentWakeupRequests} wake where wake.company_id = ${companyId}
|
||||
and wake.requested_by_actor_type = 'user' and (wake.run_id = run.id or wake.id = run.wakeup_request_id))`);
|
||||
return [...result].map(row => String(row.root_id));
|
||||
}
|
||||
|
||||
/**
|
||||
* The facts and share counters the queued-run dispatcher orders an agent's
|
||||
* queued runs by. The dispatch path and the "position in line" a waiting task
|
||||
* shows both read this one builder, so the line the board sees is the order
|
||||
* the server starts runs in.
|
||||
*/
|
||||
export async function queuedRunDispatchFacts<R extends QueuedRunDispatchRun>(
|
||||
db: Db,
|
||||
input: {
|
||||
agent: Pick<Agent, "id" | "companyId">;
|
||||
queuedRuns: R[];
|
||||
dependencyReadiness: Map<string, { isDependencyReady: boolean }>;
|
||||
issueById: Map<string, { status: string; priority: string | null }>;
|
||||
shareDeferred?: (runId: string) => boolean;
|
||||
now: Date;
|
||||
},
|
||||
): Promise<{ facts: Array<QueuedRunDispatchFacts & { run: R }>; dispatchInput: QueuedRunDispatchInput }> {
|
||||
const { agent, queuedRuns, dependencyReadiness, issueById, now } = input;
|
||||
const wakeReasonOf = (run: R) => readNonEmptyString(parseObject(run.contextSnapshot).wakeReason);
|
||||
const requestIds = [...new Set(queuedRuns.flatMap(run => run.wakeupRequestId ? [run.wakeupRequestId] : []))];
|
||||
const userRequests = queuedRuns.length > 1 && requestIds.length
|
||||
? await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, agent.companyId), inArray(agentWakeupRequests.id, requestIds),
|
||||
eq(agentWakeupRequests.requestedByActorType, "user")))
|
||||
: [];
|
||||
const userRequestIds = new Set(userRequests.map(row => row.id));
|
||||
const joinedUserRequests = queuedRuns.length > 1
|
||||
? await db.select({ runId: agentWakeupRequests.runId }).from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, agent.companyId), inArray(agentWakeupRequests.runId, queuedRuns.map(run => run.id)),
|
||||
eq(agentWakeupRequests.agentId, agent.id), eq(agentWakeupRequests.requestedByActorType, "user")))
|
||||
: [];
|
||||
const userRunIds = new Set(joinedUserRequests.flatMap(row => row.runId ? [row.runId] : []));
|
||||
const joinedByUser = (run: R) =>
|
||||
userRunIds.has(run.id) || Boolean(run.wakeupRequestId && userRequestIds.has(run.wakeupRequestId));
|
||||
const sourceRunIdOf = (run: R) =>
|
||||
readNonEmptyString(parseObject(run.contextSnapshot).previousRunId) ?? run.retryOfRunId ?? null;
|
||||
const sourceRunIds = queuedRuns.length > 1
|
||||
? [...new Set(queuedRuns.flatMap(run => {
|
||||
const sourceRunId = joinedByUser(run) ? null : sourceRunIdOf(run);
|
||||
return sourceRunId && RUN_ID_PATTERN.test(sourceRunId) ? [sourceRunId] : [];
|
||||
}))]
|
||||
: [];
|
||||
const sourceRows = sourceRunIds.length
|
||||
? await db.select({ id: heartbeatRuns.id, finishedAt: heartbeatRuns.finishedAt }).from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, agent.companyId), inArray(heartbeatRuns.id, sourceRunIds)))
|
||||
: [];
|
||||
const sourceEndedAt = new Map(sourceRows.map(row => [row.id, row.finishedAt]));
|
||||
const personSourceRunIds = new Set(sourceRows.length
|
||||
? await listPersonOriginSourceRunIds(db, agent.companyId, sourceRows.map(row => row.id))
|
||||
: []);
|
||||
const requestedByUser = (run: R) => {
|
||||
if (joinedByUser(run)) return true;
|
||||
const sourceRunId = sourceRunIdOf(run);
|
||||
return Boolean(sourceRunId && personSourceRunIds.has(sourceRunId));
|
||||
};
|
||||
const facts = queuedRuns.map(run => {
|
||||
const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
|
||||
const ready = issueId ? (dependencyReadiness.get(issueId)?.isDependencyReady ?? true) : true;
|
||||
const issue = issueId ? issueById.get(issueId) : null;
|
||||
const sourceRunId = sourceRunIdOf(run);
|
||||
return {
|
||||
run,
|
||||
id: run.id,
|
||||
createdAt: run.createdAt,
|
||||
wakeReason: wakeReasonOf(run),
|
||||
requestedByUser: requestedByUser(run),
|
||||
sourceEndedAt: sourceRunId ? sourceEndedAt.get(sourceRunId) ?? null : null,
|
||||
readinessRank: issueId ? (ready ? (issue?.status === "in_progress" ? 0 : 1) : 3) : 2,
|
||||
priorityRank: issueRunPriorityRank(issue?.priority),
|
||||
shareDeferred: input.shareDeferred?.(run.id) ?? false,
|
||||
};
|
||||
});
|
||||
const recoveryRuns = facts.filter(entry => isSystemRecoveryRun(entry));
|
||||
const oldestReady = oldestReadyQueuedRun(facts);
|
||||
const agedOldest = oldestReady && hasQueuedRunAged(oldestReady, now) ? oldestReady : null;
|
||||
const countsRecovery = recoveryRuns.length > 0 && recoveryRuns.length < facts.length;
|
||||
let startsSinceRecovery = 0;
|
||||
let startsSinceOldest = 0;
|
||||
if (countsRecovery || (agedOldest && facts.length > 1)) {
|
||||
const oldestRecoveryAt = countsRecovery ? Math.min(...recoveryRuns.map(entry => entry.createdAt.getTime())) : Infinity;
|
||||
const since = new Date(Math.min(oldestRecoveryAt, agedOldest?.createdAt.getTime() ?? Infinity));
|
||||
const recentStarts = await db.select({ id: heartbeatRuns.id, wakeReason: sql<string | null>`${heartbeatRuns.contextSnapshot}->>'wakeReason'`,
|
||||
startedAt: heartbeatRuns.startedAt, createdAt: heartbeatRuns.createdAt })
|
||||
.from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id),
|
||||
isNotNull(heartbeatRuns.startedAt), gte(heartbeatRuns.startedAt, since)))
|
||||
.orderBy(desc(heartbeatRuns.startedAt))
|
||||
.limit(Math.max(RECOVERY_DISPATCH_SHARE, OLDEST_DISPATCH_SHARE));
|
||||
const startsSince = (at: number) => recentStarts.filter(row => row.startedAt && row.startedAt.getTime() >= at);
|
||||
if (countsRecovery) {
|
||||
const starts = startsSince(oldestRecoveryAt).slice(0, RECOVERY_DISPATCH_SHARE);
|
||||
const recoveryStartIds = starts.flatMap(row => isRecoveryWakeReason(row.wakeReason) ? [row.id] : []);
|
||||
const personStarts = new Set(recoveryStartIds.length
|
||||
? await listPersonOriginSourceRunIds(db, agent.companyId, recoveryStartIds) : []);
|
||||
const lastRecovery = starts.findIndex(row => isRecoveryWakeReason(row.wakeReason) && !personStarts.has(row.id));
|
||||
startsSinceRecovery = lastRecovery < 0 ? starts.length : lastRecovery;
|
||||
}
|
||||
if (agedOldest && facts.length > 1) {
|
||||
const starts = startsSince(agedOldest.createdAt.getTime()).slice(0, OLDEST_DISPATCH_SHARE);
|
||||
const lastOldest = starts.findIndex(row => row.createdAt.getTime() <= agedOldest.createdAt.getTime());
|
||||
startsSinceOldest = lastOldest < 0 ? starts.length : lastOldest;
|
||||
}
|
||||
}
|
||||
return { facts, dispatchInput: { now, startsSinceRecovery, startsSinceOldest } };
|
||||
}
|
||||
|
||||
/** At most this many agents' lines are computed for one projection read. */
|
||||
export const MAX_AGENT_QUEUE_LINES_PER_READ = 8;
|
||||
|
||||
/**
|
||||
* Where each queued, never-started run waits in its agent's line, for agents
|
||||
* whose concurrency limit is full. An agent with a free slot is omitted: its
|
||||
* runs are not waiting on the limit. One read of the agents and their running
|
||||
* counts, then one line per full agent.
|
||||
*/
|
||||
export async function agentQueueLinesForRuns(
|
||||
db: Db,
|
||||
companyId: string,
|
||||
queuedRuns: Array<{ id: string; agentId: string }>,
|
||||
now = new Date(),
|
||||
): Promise<Map<string, ExecutionAgentQueue>> {
|
||||
const result = new Map<string, ExecutionAgentQueue>();
|
||||
const agentIds = [...new Set(queuedRuns.map(run => run.agentId))];
|
||||
if (!agentIds.length) return result;
|
||||
const [agentRows, runningRows] = await Promise.all([
|
||||
db.select().from(agents).where(and(eq(agents.companyId, companyId), inArray(agents.id, agentIds))),
|
||||
db.select({ agentId: heartbeatRuns.agentId, count: sql<number>`count(*)::int` })
|
||||
.from(heartbeatRuns)
|
||||
.where(and(inArray(heartbeatRuns.agentId, agentIds), eq(heartbeatRuns.status, "running")))
|
||||
.groupBy(heartbeatRuns.agentId),
|
||||
]);
|
||||
const runningByAgent = new Map(runningRows.map(row => [row.agentId, Number(row.count)]));
|
||||
const fullAgents = agentRows.flatMap(agent => {
|
||||
const maxConcurrentRuns = parseHeartbeatPolicy(agent).maxConcurrentRuns;
|
||||
const runningCount = runningByAgent.get(agent.id) ?? 0;
|
||||
return runningCount >= maxConcurrentRuns ? [{ agent, maxConcurrentRuns, runningCount }] : [];
|
||||
}).slice(0, MAX_AGENT_QUEUE_LINES_PER_READ);
|
||||
if (!fullAgents.length) return result;
|
||||
// Lazy: the issue service imports the execution projection that calls this.
|
||||
const { issueService } = await import("../issues.js");
|
||||
const issuesSvc = issueService(db);
|
||||
const wanted = new Set(queuedRuns.map(run => run.id));
|
||||
await Promise.all(fullAgents.map(async ({ agent, maxConcurrentRuns, runningCount }) => {
|
||||
const line = await cachedAgentQueuedRunLine(db, issuesSvc, agent, now);
|
||||
line.forEach((runId, index) => {
|
||||
if (!wanted.has(runId)) return;
|
||||
result.set(runId, {
|
||||
agentId: agent.id,
|
||||
agentName: agent.name,
|
||||
maxConcurrentRuns,
|
||||
runningCount,
|
||||
position: index + 1,
|
||||
queuedCount: line.length,
|
||||
});
|
||||
});
|
||||
}));
|
||||
return result;
|
||||
}
|
||||
|
||||
/** The agent's queued runs, in the order dispatch starts them (the same set and order as startNextQueuedRunForAgent). */
|
||||
export async function agentQueuedRunLine(
|
||||
db: Db,
|
||||
issuesSvc: { listDependencyReadiness: (companyId: string, issueIds: string[]) => Promise<Map<string, { isDependencyReady: boolean }>> },
|
||||
agent: Agent,
|
||||
now = new Date(),
|
||||
options: { cutoff?: Date | null } = {},
|
||||
): Promise<string[]> {
|
||||
const cutoff = options.cutoff !== undefined ? options.cutoff : await worktreeExecutionCutoff(db);
|
||||
const queuedRuns = await db.select({
|
||||
id: heartbeatRuns.id,
|
||||
createdAt: heartbeatRuns.createdAt,
|
||||
wakeupRequestId: heartbeatRuns.wakeupRequestId,
|
||||
retryOfRunId: heartbeatRuns.retryOfRunId,
|
||||
// Only the keys ordering reads; full snapshots are large.
|
||||
contextSnapshot: sql<Record<string, unknown>>`jsonb_build_object(
|
||||
'issueId', ${heartbeatRuns.contextSnapshot}->'issueId',
|
||||
'wakeReason', ${heartbeatRuns.contextSnapshot}->'wakeReason',
|
||||
'previousRunId', ${heartbeatRuns.contextSnapshot}->'previousRunId')`,
|
||||
}).from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id),
|
||||
eq(heartbeatRuns.status, "queued"), cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined))
|
||||
.orderBy(asc(heartbeatRuns.createdAt));
|
||||
if (!queuedRuns.length) return [];
|
||||
const issueIds = [...new Set(queuedRuns.flatMap(run => {
|
||||
const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
|
||||
return issueId ? [issueId] : [];
|
||||
}))];
|
||||
const [dependencyReadiness, issueRows] = issueIds.length
|
||||
? await Promise.all([
|
||||
issuesSvc.listDependencyReadiness(agent.companyId, issueIds),
|
||||
db.select({ id: issues.id, status: issues.status, priority: issues.priority }).from(issues)
|
||||
.where(and(eq(issues.companyId, agent.companyId), inArray(issues.id, issueIds))),
|
||||
])
|
||||
: [new Map<string, { isDependencyReady: boolean }>(), []];
|
||||
const { facts, dispatchInput } = await queuedRunDispatchFacts(db, {
|
||||
agent,
|
||||
queuedRuns,
|
||||
dependencyReadiness,
|
||||
issueById: new Map(issueRows.map(row => [row.id, row])),
|
||||
now,
|
||||
});
|
||||
return queuedRunDispatchLine(facts, dispatchInput).map(entry => entry.id);
|
||||
}
|
||||
|
||||
/**
|
||||
* Dispatch skips queued runs created before an enabled preview worktree's
|
||||
* cutoff, so the line skips them too. Outside a worktree this reads nothing.
|
||||
*/
|
||||
export async function worktreeExecutionCutoff(db: Db, env: NodeJS.ProcessEnv = process.env): Promise<Date | null> {
|
||||
if (!isTruthyRuntimeEnvValue(env.PAPERCLIP_IN_WORKTREE)) return null;
|
||||
const activation = resolveWorktreeRunExecutionActivation(
|
||||
await instanceSettingsService(db).getExperimental(),
|
||||
env.PAPERCLIP_INSTANCE_ID?.trim() || null,
|
||||
);
|
||||
const cutoff = activation.armed ? new Date(activation.cutoff) : null;
|
||||
return cutoff && !Number.isNaN(cutoff.getTime()) ? cutoff : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Board reads repeat while the line changes only when a run starts, finishes or
|
||||
* joins. A short per-agent memo, shared by concurrent reads, keeps a deep queue
|
||||
* from being re-read on every task and run read.
|
||||
*/
|
||||
export const AGENT_QUEUE_LINE_TTL_MS = 5_000;
|
||||
const lineCache = new Map<string, { at: number; line: Promise<string[]> }>();
|
||||
|
||||
function cachedAgentQueuedRunLine(
|
||||
db: Db,
|
||||
issuesSvc: Parameters<typeof agentQueuedRunLine>[1],
|
||||
agent: Agent,
|
||||
now: Date,
|
||||
): Promise<string[]> {
|
||||
const key = `${agent.companyId}:${agent.id}`;
|
||||
const at = Date.now();
|
||||
const cached = lineCache.get(key);
|
||||
if (cached && at - cached.at < AGENT_QUEUE_LINE_TTL_MS) return cached.line;
|
||||
for (const [entryKey, entry] of lineCache) {
|
||||
if (at - entry.at >= AGENT_QUEUE_LINE_TTL_MS) lineCache.delete(entryKey);
|
||||
}
|
||||
const line = agentQueuedRunLine(db, issuesSvc, agent, now);
|
||||
lineCache.set(key, { at, line });
|
||||
line.catch(() => { if (lineCache.get(key)?.line === line) lineCache.delete(key); });
|
||||
return line;
|
||||
}
|
||||
|
||||
/** Test hook: forget memoized lines. */
|
||||
export function clearAgentQueueLineCache() {
|
||||
lineCache.clear();
|
||||
}
|
||||
@@ -3593,13 +3593,20 @@ async function activeRunMapForIssues(
|
||||
return map;
|
||||
}
|
||||
|
||||
/**
|
||||
* `live` counts descendants with a queued or running run: work that covers a
|
||||
* blocked parent. `running` counts only descendants whose run started; a run
|
||||
* still waiting for its agent's slot is not running below.
|
||||
*/
|
||||
type LiveDescendantCounts = { live: number; running: number };
|
||||
|
||||
async function liveDescendantCountMapForIssues(
|
||||
dbOrTx: any,
|
||||
companyId: string,
|
||||
issueIds: string[],
|
||||
): Promise<Map<string, number>> {
|
||||
): Promise<Map<string, LiveDescendantCounts>> {
|
||||
const uniqueIssueIds = [...new Set(issueIds)];
|
||||
const map = new Map<string, number>();
|
||||
const map = new Map<string, LiveDescendantCounts>();
|
||||
if (uniqueIssueIds.length === 0) return map;
|
||||
|
||||
for (const issueIdChunk of chunkList(
|
||||
@@ -3615,8 +3622,8 @@ async function liveDescendantCountMapForIssues(
|
||||
target_issues(issue_id) AS (
|
||||
VALUES ${sql.join(targetRows, sql`, `)}
|
||||
),
|
||||
live_issues(live_issue_id, parent_id) AS (
|
||||
SELECT DISTINCT live_issue.id, live_issue.parent_id
|
||||
live_issues(live_issue_id, parent_id, started) AS (
|
||||
SELECT DISTINCT live_issue.id, live_issue.parent_id, (live_run.status = 'running' OR live_run.started_at IS NOT NULL)
|
||||
FROM issues live_issue
|
||||
JOIN heartbeat_runs live_run ON live_run.id = live_issue.execution_run_id
|
||||
WHERE live_issue.company_id = ${companyId}
|
||||
@@ -3625,7 +3632,7 @@ async function liveDescendantCountMapForIssues(
|
||||
AND live_run.company_id = ${companyId}
|
||||
AND live_run.status IN ('queued', 'running')
|
||||
UNION
|
||||
SELECT DISTINCT live_issue.id, live_issue.parent_id
|
||||
SELECT DISTINCT live_issue.id, live_issue.parent_id, (live_run.status = 'running' OR live_run.started_at IS NOT NULL)
|
||||
FROM heartbeat_runs live_run
|
||||
JOIN issues live_issue ON live_issue.id::text = (live_run.context_snapshot ->> 'issueId')
|
||||
WHERE live_issue.company_id = ${companyId}
|
||||
@@ -3634,8 +3641,8 @@ async function liveDescendantCountMapForIssues(
|
||||
AND live_run.company_id = ${companyId}
|
||||
AND live_run.status IN ('queued', 'running')
|
||||
),
|
||||
live_ancestors(live_issue_id, ancestor_id, next_parent_id, visited_issue_ids) AS (
|
||||
SELECT live_issues.live_issue_id, parent.id, parent.parent_id, ARRAY[live_issues.live_issue_id, parent.id]
|
||||
live_ancestors(live_issue_id, started, ancestor_id, next_parent_id, visited_issue_ids) AS (
|
||||
SELECT live_issues.live_issue_id, live_issues.started, parent.id, parent.parent_id, ARRAY[live_issues.live_issue_id, parent.id]
|
||||
FROM live_issues
|
||||
JOIN issues parent ON parent.id = live_issues.parent_id
|
||||
WHERE parent.company_id = ${companyId}
|
||||
@@ -3644,6 +3651,7 @@ async function liveDescendantCountMapForIssues(
|
||||
UNION ALL
|
||||
SELECT
|
||||
live_ancestors.live_issue_id,
|
||||
live_ancestors.started,
|
||||
parent.id,
|
||||
parent.parent_id,
|
||||
live_ancestors.visited_issue_ids || parent.id
|
||||
@@ -3656,7 +3664,8 @@ async function liveDescendantCountMapForIssues(
|
||||
)
|
||||
SELECT
|
||||
live_ancestors.ancestor_id::text AS "issueId",
|
||||
count(DISTINCT live_ancestors.live_issue_id)::int AS "liveDescendantCount"
|
||||
count(DISTINCT live_ancestors.live_issue_id)::int AS "liveDescendantCount",
|
||||
count(DISTINCT live_ancestors.live_issue_id) FILTER (WHERE live_ancestors.started)::int AS "runningDescendantCount"
|
||||
FROM live_ancestors
|
||||
JOIN target_issues ON target_issues.issue_id = live_ancestors.ancestor_id
|
||||
WHERE live_ancestors.ancestor_id <> live_ancestors.live_issue_id
|
||||
@@ -3669,14 +3678,10 @@ async function liveDescendantCountMapForIssues(
|
||||
for (const row of resultRows) {
|
||||
if (typeof row !== "object" || row === null) continue;
|
||||
const issueId = (row as { issueId?: unknown }).issueId;
|
||||
const liveDescendantCount = (row as { liveDescendantCount?: unknown })
|
||||
.liveDescendantCount;
|
||||
if (typeof issueId !== "string") continue;
|
||||
const count =
|
||||
typeof liveDescendantCount === "number"
|
||||
? liveDescendantCount
|
||||
: Number(liveDescendantCount);
|
||||
if (Number.isFinite(count)) map.set(issueId, count);
|
||||
const live = Number((row as { liveDescendantCount?: unknown }).liveDescendantCount);
|
||||
const running = Number((row as { runningDescendantCount?: unknown }).runningDescendantCount);
|
||||
if (Number.isFinite(live)) map.set(issueId, { live, running: Number.isFinite(running) ? running : 0 });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6864,7 +6869,7 @@ async function listBlockedInboxIssues(
|
||||
listIssueBlockedInboxAttentionMap(dbOrTx, companyId, withRuns, readExternalReviewWaits),
|
||||
includeLiveDescendantSummary
|
||||
? liveDescendantCountMapForIssues(dbOrTx, companyId, issueIds)
|
||||
: Promise.resolve(new Map<string, number>()),
|
||||
: Promise.resolve(new Map<string, LiveDescendantCounts>()),
|
||||
]);
|
||||
|
||||
const rawSearchInput = filters?.q?.trim() ?? "";
|
||||
@@ -6937,7 +6942,9 @@ async function listBlockedInboxIssues(
|
||||
...(includeLiveDescendantSummary
|
||||
? {
|
||||
liveDescendantCount:
|
||||
liveDescendantCountByIssueId.get(row.id) ?? 0,
|
||||
liveDescendantCountByIssueId.get(row.id)?.live ?? 0,
|
||||
runningDescendantCount:
|
||||
liveDescendantCountByIssueId.get(row.id)?.running ?? 0,
|
||||
}
|
||||
: {}),
|
||||
...(contextUserId
|
||||
@@ -8801,7 +8808,7 @@ export function issueService(db: Db) {
|
||||
: Promise.resolve(new Map<string, IssueRelationIssueSummary[]>()),
|
||||
includeLiveDescendantSummary
|
||||
? timePhase("live_descendants", () => liveDescendantCountMapForIssues(db, companyId, issueIds))
|
||||
: Promise.resolve(new Map<string, number>()),
|
||||
: Promise.resolve(new Map<string, LiveDescendantCounts>()),
|
||||
]);
|
||||
const statsByIssueId = new Map(
|
||||
statsRows.map((row) => [row.issueId, row]),
|
||||
@@ -8857,7 +8864,9 @@ export function issueService(db: Db) {
|
||||
...(includeLiveDescendantSummary
|
||||
? {
|
||||
liveDescendantCount:
|
||||
liveDescendantCountByIssueId.get(row.id) ?? 0,
|
||||
liveDescendantCountByIssueId.get(row.id)?.live ?? 0,
|
||||
runningDescendantCount:
|
||||
liveDescendantCountByIssueId.get(row.id)?.running ?? 0,
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
@@ -8900,7 +8909,9 @@ export function issueService(db: Db) {
|
||||
...(includeLiveDescendantSummary
|
||||
? {
|
||||
liveDescendantCount:
|
||||
liveDescendantCountByIssueId.get(row.id) ?? 0,
|
||||
liveDescendantCountByIssueId.get(row.id)?.live ?? 0,
|
||||
runningDescendantCount:
|
||||
liveDescendantCountByIssueId.get(row.id)?.running ?? 0,
|
||||
}
|
||||
: {}),
|
||||
...deriveIssueUserContext(row, contextUserId, {
|
||||
|
||||
Reference in new issue
Block a user