Files
PaperClipAI/server/src/services/heartbeat/queue.ts
T
DottaandClaude Opus 5.5 4da6902cb3 fix(dispatch): start work on a task tree a person created in the person tier (#21916)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue chooses which queued run starts when an agent at
its concurrency limit frees a slot
> - The queue puts runs a person asked for in a "person" tier ahead of
agent "work"
> - Subtasks of a person's task, and the parent's resume after its
children complete, ranked as agent work
> - Under load these runs started only through the 60-minute oldest-run
share, so a half-finished task waited about 55 minutes per step behind
newer tasks
> - This pull request puts every non-recovery run on a task tree a
person created in the person tier
> - The benefit is that a person's started work finishes before newer
work starts, and the fairness bounds stay in place

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The order in which an agent at its concurrency limit starts queued runs.

**Subsystem affected**
Server heartbeat queue:
`server/src/services/heartbeat/queued-run-dispatch-order.ts` and
`queued-run-line.ts`.

**Current behavior**
A board user's new assignment ranks as person work. An agent's subtask
of that person's task, and the parent's `issue_children_completed`
resume, rank as agent work. In a restart chaos run with agents at
capacity, the median queue wait of a parent resume was about 1,640 s.
The median wait of a new assignment was about 190 to 290 s. One parent
waited while 14 newer tasks started first. Subtasks and parent resumes
started only near the 60-minute aging bound.

**Proposed behavior**
A non-recovery run on a task tree that a person created goes in the
person tier. A task is in such a tree when the task, or any task above
it, has `createdByUserId`. In the tier, runs keep the existing order:
readiness, then priority, then age (oldest first).

**Reason and benefit**
A person waits on the whole tree, not only on the top task. Work that a
person started now finishes before newer tasks take its slots.

**Breaking changes**
None. There is no schema or API change.

## What Changed

- `queuedRunDispatchClass` puts a run with `personTask` in the person
tier. System recovery stays in the recovery tiers, also on a person's
tree.
- `personTask` now means "the task, or a task above it, was created by a
person". The existing `createdByUserId` marker is the source.
- New `listPersonTreeIssueIds`: one recursive query per dispatch pass.
It walks each queued task's parent chain through the `(company_id, id)`
unique index. It stops at the first person-created task, at most 32
levels. It runs only when the agent has more than one queued run. The
start-history read uses the same query in place of its old direct
`createdByUserId` read, so the query count does not change there.
- Recovery's 1-in-4 share is held only while a run a person asked for
(`isPersonRequestedRun`) is ready, not for the rest of a person's tree.
So a deep tree cannot hold recovery indefinitely. The dispatcher and the
place-in-line replay use the same rule.
- The claim backoff for a declined claim keeps the short person backoff
only for runs a person asked for (unchanged behavior for tree runs).
- `doc/heartbeat-internals.md`: new "Queued run dispatch order" section
with the tiers, the order and the fairness bounds.

Fairness bound: the oldest-run share is unchanged. Every 4th start goes
to the longest-waiting ready run past 60 minutes, preferring runs people
wait on. So another person's new task starts within 60 minutes plus 4 x
(aged person-tier runs queued before it + 1) starts, even while a deep
tree keeps adding in-progress or higher-priority work. A newer tree run
never passes an older, equally ranked person task. Queues are per agent,
and an agent belongs to one company, so no agent or company gets ahead
of another.

## Verification

- Red on master, green with this change:
- `queued-run-dispatch-order.test.ts`: an older parent resume
(`issue_children_completed`, person tree) starts before a newer person
assignment. The place-in-line replay agrees.
- `queued-run-dispatch-order.test.ts`: an older subtask of a person's
task starts before a newer person assignment.
- `queued-run-dispatch-order.test.ts`: fairness simulation. A deep tree
adds two in-progress, higher-priority runs per start. Another person's
new task starts within `QUEUED_RUN_AGING_MS` + `OLDEST_DISPATCH_SHARE`
starts.
- `execution-projection-agent-queue.test.ts` (embedded Postgres, real
SQL): line is subtask (grandchild of a person's task), parent resume,
newer person assignment, then work on an agent-created tree. Master
gives the person assignment first and the tree runs last.
- Negative and guard tests (green on both): work on an agent-created
tree stays in the work tier behind a newer person assignment; system
recovery on a person's tree stays in the recovery tier; newer tree work
does not pass an older equally ranked person task; recovery still gets
its share while only tree work waits.
- Query plan on embedded Postgres with 300,000 issues and 200 queued
tasks six levels deep: every step is an index scan on
`issues_company_id_uq`, 1.5 ms execution.
- `pnpm verify` exited 2 (a doc is a global CI input). Focused run:
`queued-run-dispatch-order`, `queued-run-line`,
`execution-projection-agent-queue`, `fresh-continuation-retry-delivery`,
`assigned-work-stalled-queued-run`, `automatic-run-start-gate`,
`automatic-run-brake-recovery-passes`,
`heartbeat-stale-queue-invalidation`: 8 files, 173 tests passed. `tsc
--noEmit` for the server passed.

## Risks

- Behavior shift: in a company where people create most top-level tasks,
most issue runs are now person tier. The "work" tier then holds
agent-started trees, routines and runs without a task. Recovery keeps
its share because only runs a person asked for hold it.
- An in-progress run on a person's tree now ranks ahead of a new person
task that is not in progress (readiness rank). The oldest-run share
bounds that wait at about 60 minutes plus a few starts, the same bound
agent work had before.
- A tree deeper than 32 levels above a queued task is treated as not a
person tree. It falls back to the work tier and the aging share.
- The extra query runs only when an agent has more than one queued run,
and only for queued tasks without `createdByUserId`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 10:35:42 -06:00

5896 lines
281 KiB
TypeScript

import { QUEUED_MESSAGES_DELIVERED_REASON } from "../../modules/wake-queue/index.js";
import { connectionIntentService } from "../connection-intents.js";
import { verifyBlockerStallWake } from "../recovery/blocker-stall-wakeups.js";
import { CONTEXT_COMPACTION_WAKE_REASON, ContextCompactionError, isContextCompactionRun, readContextCompactionRequest } from "../context-compaction.js";
import { isProcessRunAdmissionClosed, isReattachPhaseActive as isProcessReattachPhaseActive } from "./run-admission.js";
import { continueFreshInTransaction, hasPendingFreshContinuation, isFreshIsolatedContinuationWake, settleCoalescedFreshContinuationHolds } from "../fresh-continuation-fallback.js";
import { deliverContinuationAsRun, foldWaitingRetriesIntoContinuation, foldWaitingRetriesIntoRecordedContinuation, undeliveredContinuation,
undeliveredContinuationContext } from "../fresh-continuation-retry-fold.js";
import { refreshRunStartYieldIfStale } from "../run-start-handover.js";
import { deliveredQueuedCommentIds } from "../queued-comment-delivery.js";
import { hasAutomaticInteractionDeliveryHold, INTERACTION_DELIVERY_DENIED, recheckQueuedRunInteractions, withInteractionDeliveryOrigin } from "../agent-interaction-delivery.js";
import { deferredWorkIntentWait } from "../work-intent-outbox.js";
import { notifyIssueGraphRecovery } from "../issue-recovery-notifications.js";
import { createHeartbeatTimerPolicy } from "./timer-policy.js";
import { QUEUED_RUN_AGING_MS, RECOVERY_WAKE_REASONS, createQueuedRunDispatcher, hasQueuedRunAged, isPersonRequestedRun, queuedRunDispatchClass } from "./queued-run-dispatch-order.js";
import { reserveAutomaticRunStart } from "../recovery/automatic-run-brake.js";
import { queuedRunDispatchFacts } from "./queued-run-line.js";
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../deferred-execution-wait.js";
import { assignmentWakePayloadSql, isAssignmentWakePayload, notifyDeferredExecutionWork } from "../deferred-execution-notifications.js";
import type { DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js";
import { notifyQueuedRunDispatch, notifyRunDispatch } from "../run-dispatch-notifications.js";
import { notifyStatusCardIssueChange } from "../status-card-notifications.js";
import { notifyRoutineExecutionStatus } from "../routine-execution-status-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js";
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
import { notifyUnsafeWorkspaceExportRecovery, notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
import { notifyScheduledRetryCancellation } from "../scheduled-retry-cancellation-notifications.js";
import { notifyChatResponseAvailability } from "../chat-response-availability.js";
import { notifyChatInboundWakeNotices } from "../chat-inbound-wake-notice-notifications.js";
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { assertTaskRetryConversationAllowed, readTaskRetryIntent, readTaskRetryBoardMessageOrigin, retireSupersededTaskRetry, taskRetryMutationAllowed } from "../task-retry.js";
import { insertIssueComment } from "../issue-comment-writes.js";
import { withExternalAdmissionGuard, assertNoExternalOverlap } from "../../modules/external-agents/index.js";
import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js";
import { normalizeAgentNameKey } from "./retries.js";
import { publishActiveDotComment } from "../dot-assignment-follow-up.js";
import { publishActiveMuseComment } from "../muse-assignment-follow-up.js";
import {
type WakeupOptions,
mergeCoalescedContextSnapshot,
enrichWakeContextSnapshot,
hasInteractionContinuationWakeContext,
isInteractionResolutionWakePayload,
} from "./run-preparation.js";
import { resolveExecutionWorkspaceReuseRequestForIssue } from "./workspaces.js";
import {
CHAT_COMPLETION_WAKE_REASON,
isCompletedOnboardingHandoffWake,
} from "../chat-completion-delivery.js";
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
import { isVerifiedDependencyInteractionResponse, readQueuedInteractionResponse } from "../queued-interaction-response.js";
import {
isConversation,
isConversationReset,
isConversationExecutionWake,
isWaitingConversation,
} from "../agent-conversations.js";
import { legacyControllerClaim } from "../legacy-controller-lease.js";
import {
admitExplicitNativeContinuation,
undeliveredLegacyUserCommentIds,
} from "../explicit-native-continuation.js";
import { canRetryStoppedRun } from "../cancelled-native-startup.js";
import { getExecutionBlocker, hasPendingRunCleanup } from "../execution-blocker.js";
import { nativeManualAdmissionHoldsExecution } from "../native-runtime/native-runner-ownership.js";
import { taskChatRetrySourceCleanupPending } from "../chat-channels/task-retry-authority.js";
import { validateNativeWorkspaceReconstruction } from "../native-runtime/native-workspace-reconstruction.js";
import { isConversationAdapter } from "../conversation-continuation.js";
import { recordExecutionWait } from "../execution-wait.js";
import {
getNativeReviewAssignment,
readNativeReviewAssignmentContext,
} from "../native-runtime/native-review-participant.js";
import { claimQueuedNativeReviewRun } from "../native-runtime/native-review-dispatch.js";
import { nativeCleanupHoldsAdmission } from "../native-runtime/native-cleanup-admission-hold.js";
import { adapterExecutionControls } from "../adapter-execution-control.js";
import { preserveTaskRemoteWorkspaceWriter } from "../native-runtime/native-failed-workspace-recovery.js";
import { executionFailureRetryCount } from "../execution-recovery-attempt.js";
import {
slackMentionAllowsRunStart,
resumeDeferredChatWake,
recordDeferredChatWakeDecline,
clearDeferredChatWakeRetry,
deferredChatWakeDeclineForError,
DEFERRED_CHAT_WAKE_RETRY_KEY,
assertDurableChatWakeupReceipt,
assertDurableChatWakeupRequest,
withDurableChatAdmission,
authorizeFailedChatRunRetryWake,
FailedChatRunRetryAuthorizationError,
unadmittedChatWakeupCondition,
} from "../durable-chat-wakeup.js";
import { agentService } from "../agents.js";
import { normalizeLegacyRunnerProvider } from "@paperclipai/adapter-utils";
import { randomUUID } from "node:crypto";
import {
and,
asc,
desc,
eq,
gt,
gte,
inArray,
isNotNull,
isNull,
lte,
ne,
notInArray,
or,
sql,
} from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
agents,
caseHandoffs,
cases,
agentWakeupRequests,
activityLog,
budgetIncidents,
chatConversations,
chatEndpoints,
companies,
executionWorkspaces,
heartbeatRunEvents,
heartbeatRuns,
issueComments,
issueRecoveryActions,
issueRelations,
issues,
} from "@paperclipai/db";
import {
conflict,
HttpError,
notFound,
} from "../../errors.js";
import { logger } from "../../middleware/logger.js";
import { isUniqueViolation } from "../../db-errors.js";
import { publishLiveEvent } from "../live-events.js";
import { allocateHeartbeatRunEventSeq } from "../heartbeat-run-events.js";
import {
queuedCommentIdsFromWakePayload,
queuedCommentIdsFromRunContext,
withQueuedCommentIdsInWakePayload,
withQueuedCommentIdsInRunContext,
} from "../issue-queued-comment-queue.js";
import { parseObject } from "../../adapters/utils.js";
import { isExternalChatWaitAuthorizationContention } from "../native-runtime/chat-attachment-reuse.js";
import { emitAgentTaskRun } from "../agent-task-run-telemetry.js";
import {
CHAT_CONTROL_RECOVERY_ADMISSION_KEY,
CHAT_CONTROL_RECOVERY_STOP_CODE,
CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE,
chatControlRecoveryAdmission,
readChatControlRecoveryAdmission,
readChatControlNativeParent,
readChatControlRecoveryStop,
} from "../chat-control-recovery-stop.js";
import {
ISSUE_NEW_INPUT_ACTIVITY_ACTIONS,
ISSUE_PROGRESS_ACTIVITY_ACTIONS,
ISSUE_REWAKE_LOOKBACK_MS,
ISSUE_REWAKE_RUN_SAMPLE_LIMIT,
evaluateIssueRewakeThrottle,
isThrottleCandidateIssueRewake,
NATIVE_STATUS_COMMITTER_ACTOR_ID,
} from "../issue-rewake-throttle.js";
import { logActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
import {
ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
isDependencyResolvedWakePayload,
dependencyResolvedWakePayloadSql,
} from "../issue-dependency-wakeups.js";
import { resumeIfPauseAlreadyReleased } from "../issue-tree-resume.js";
import {
ISSUE_PAUSED_ERROR_CODE,
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
isVerifiedIssueTreeControlInteractionWake,
pauseSuppressionResult,
} from "../issue-tree-control.js";
import {
buildExecutionWorkspaceAdapterConfig,
isUnrunnableWorktreeCombo,
parseIssueExecutionWorkspaceSettings,
resolveEffectiveWorkspaceStrategyType,
resolveExecutionWorkspaceMode,
WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE,
WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION,
} from "../execution-workspace-policy.js";
import { readContinuationAttempt } from "../recovery/index.js";
import { allowsDependencyBlockedWake, extractWakeCommentIds, wakeChatConversationsConflict } from "../../modules/run-dispatch/index.js";
import { dispatchForAgent, outsideAgentDispatch } from "../agent-start-lock.js";
import { AgentRunSlotsFull, assertAgentRunSlotFree } from "./run-slot-admission.js";
import {
evaluateAgentInvokability,
shouldCancelRunsForNonInvokableAgent,
type AgentOrgRow,
} from "../agent-invokability.js";
import { supportsUnscopedTimerHeartbeat } from "../heartbeat-policy.js";
import { ISSUE_BLOCKER_STALLED_WAKE_REASON, isUuidLike } from "@paperclipai/shared";
import { retryChatControlAdmission } from "../chat-control-admission-retry.js";
import type { HeartbeatServiceOptions } from "../heartbeat.js";
import type { createHeartbeatRunState } from "./run-state.js";
import type { createHeartbeatRunPreparation } from "./run-preparation.js";
import type { HeartbeatRetryDependencies } from "./retries.js";
import type { HeartbeatRecoveryDependencies, createHeartbeatRecovery } from "./recovery.js";
import { HEARTBEAT_RUN_TERMINAL_STATUSES } from "./recovery.js";
import { executeIssuePostCommitActions, type IssuePostCommitAction, type issueService } from "../issues.js";
import type { issueTreeControlService } from "../issue-tree-control.js";
import type { budgetService } from "../budgets.js";
import { isBoardOwnedBudgetBlock } from "../budget-block-ownership.js";
import type { instanceSettingsService } from "../instance-settings.js";
import type { createRunDispatch } from "../../modules/run-dispatch/index.js";
import { hasRetainedExternalChatRetry, type PostCommitEffect as WakeQueuePostCommitEffect, type createWakeQueue } from "../../modules/wake-queue/index.js";
import { crashPoint } from "@paperclipai/shared/crash-points";
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
type QueuedRunCandidate = Pick<HeartbeatRun, "id" | "companyId" | "agentId" | "status" | "invocationSource" | "createdAt"
| "startedAt" | "wakeupRequestId" | "retryOfRunId" | "scheduledRetryAt" | "scheduledRetryAttempt" | "scheduledRetryReason"
| "contextSnapshot">;
type QueuedDispatchRequest = { attachSetupRequest: boolean; retryRefused: boolean; continuesChain: boolean; triedRunIds: ReadonlySet<string> };
type HeartbeatRunState = ReturnType<typeof createHeartbeatRunState>;
type RunPreparation = ReturnType<typeof createHeartbeatRunPreparation>;
/** Queue policy uses the service's lifecycle effects and process-wide ownership sets. */
export interface HeartbeatQueueDependencies extends Pick<HeartbeatRetryDependencies,
"setWakeupStatus" | "appendRunEvent" | "finalizeAgentStatus" |
"getAgentInvokability" | "getWorktreeExecutionCutoff" | "applyRunDispatchPostCommitEffects">,
Pick<RunPreparation, "toAgentOrgRow" | "listCompanyAgentOrgRows" |
"resolveResponsibleUserIdForRun" | "getIssueExecutionContext" |
"getRoutineEnvForExecutionIssue" | "resolveResponsibleUserIdForRunSeed">,
Pick<HeartbeatRunState, "getAgent" | "getRun" | "resolveExplicitResumeSessionOverride" |
"resolveSessionBeforeForWakeup" | "hasResolvablePriorSessionWorkspaceForWake">,
Pick<HeartbeatRecoveryDependencies, "setRunStatusFromLive"> {
sweepPendingCleanupLeases: ReturnType<typeof createHeartbeatRecovery>["sweepPendingCleanupLeases"];
options: Pick<HeartbeatServiceOptions, "beforeChatControlRecoveryCheck">;
issuesSvc: Pick<ReturnType<typeof issueService>, "getDependencyReadiness" | "listDependencyReadiness" | "updateForCompany">;
treeControlSvc: Pick<ReturnType<typeof issueTreeControlService>, "getActivePauseHoldGate">;
budgets: Pick<ReturnType<typeof budgetService>, "getInvocationBlock">;
instanceSettings: Pick<ReturnType<typeof instanceSettingsService>, "getExperimental">;
runDispatch: Pick<ReturnType<typeof createRunDispatch>, "cancelStaleQueuedRun">;
wakeQueue: Pick<ReturnType<typeof createWakeQueue>, "createAdmissionTransactionScope" | "admitWakeBehindIssueExecution">;
activeRunExecutions: Set<string>;
activeRunExecutionPromises: Set<Promise<void>>;
activeWakeupPromises: Set<Promise<unknown>>;
liveRunExecutions: { has(id: string): boolean };
isHeartbeatRunTerminalStatus: (status: string | null | undefined) => boolean;
isSameTaskScope: (left: string | null, right: string | null) => boolean;
runTaskKey: (run: HeartbeatRun) => string | null;
filterZombieCoalesceTarget: <T extends { id: string; status: string }>(run: T | null, live: { has(id: string): boolean }) => T | null;
getSchedulingSuppression: () => Promise<{ suppressed: boolean; reason: "worktree_instance" | "database_restore_in_progress" | "task_drain" | null }>;
isRunAdmissionClosed?: () => boolean;
isReattachPhaseActive?: () => boolean;
/** Before a dispatch pass, refresh admission inputs that have gone stale; null when current. */
refreshRunAdmission?: () => Promise<void> | null;
canStartNewExternalBotWork: (agent: typeof agents.$inferSelect) => Promise<boolean>;
executeRun: (runId: string) => Promise<void>;
releaseIssueExecutionAndPromote: (run: Pick<HeartbeatRun, "id" | "companyId">, options?: { suppressImmediateRecovery?: boolean; resumeStrandedQueue?: boolean; deferredPostCommitEffects?: WakeQueuePostCommitEffect[] }) => Promise<unknown>;
applyWakeQueuePostCommitEffects: (effects: WakeQueuePostCommitEffect[]) => Promise<void>;
publishRunLifecyclePluginEvent: (run: HeartbeatRun) => void;
cancelRunInternal: (runId: string, reason?: string,
options?: { errorCode?: string; resultJson?: Record<string, unknown> }) => Promise<unknown>;
cancelActiveForAgentInternal: (agentId: string, reason: string) => Promise<unknown>;
resumeExecutionWaitComments: (scope?: DeferredReceiptRecoveryOptions) => Promise<unknown>;
resumeQueuedCommentInterrupt: (companyId: string, queueId: string) => Promise<unknown>;
resumeSavedLegacyComments: (companyId: string, queueId: string) => Promise<unknown>;
formatIssueIdentifierLink: (identifier: string | null, fallback: string) => string;
}
export const DEFERRED_WAKE_CONTEXT_KEY = "_paperclipWakeContext";
export const EXECUTION_PATH_HEARTBEAT_RUN_STATUSES = [
"queued",
"running",
"scheduled_retry",
] as const;
export class ChatControlRecoveryUnresolvedError extends Error {
constructor() {
super(
"Run admission could not acquire its database locks after bounded retries. No provider work started. Review database contention and send a fresh request; this attempt will not automatically retry.",
);
}
}
const RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP = new Set([
CHAT_COMPLETION_WAKE_REASON,
"approval_approved",
ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
"issue_recovery_action_restored",
]);
export function shouldRequireIssueCommentForWake(
contextSnapshot: Record<string, unknown> | null | undefined,
) {
if (contextSnapshot?.skipIssueComment === true) return false;
const wakeReason = readNonEmptyString(contextSnapshot?.wakeReason);
return (
wakeReason === "issue_assigned" ||
wakeReason === "execution_review_requested" ||
wakeReason === "execution_approval_requested" ||
wakeReason === "execution_changes_requested"
);
}
async function listUnresolvedBlockerSummaries(
dbOrTx: Pick<Db, "select">,
companyId: string,
issueId: string,
unresolvedBlockerIssueIds: string[],
) {
const ids = [...new Set(unresolvedBlockerIssueIds.filter(Boolean))];
if (ids.length === 0) return [];
return dbOrTx
.select({
id: issues.id,
identifier: issues.identifier,
title: issues.title,
status: issues.status,
priority: issues.priority,
assigneeAgentId: issues.assigneeAgentId,
assigneeUserId: issues.assigneeUserId,
})
.from(issueRelations)
.innerJoin(issues, eq(issueRelations.issueId, issues.id))
.where(
and(
eq(issueRelations.companyId, companyId),
eq(issueRelations.type, "blocks"),
eq(issueRelations.relatedIssueId, issueId),
inArray(issues.id, ids),
),
)
.orderBy(asc(issues.title));
}
export function shouldDeferFollowupWakeForSameIssue(input: {
activeRunStatus: string | null | undefined;
isSameExecutionAgent: boolean;
wakeCommentId: string | null | undefined;
forceFreshSession: boolean;
}) {
// A comment follow-up or explicit fresh-session wake needs a new run boundary.
if (!input.isSameExecutionAgent) return false;
if (input.activeRunStatus !== "running") return false;
if (input.wakeCommentId) return true;
if (input.forceFreshSession) return true;
return false;
}
export function shouldQueueFollowupForRunningIssueWake(input: {
contextSnapshot: Record<string, unknown> | null | undefined;
wakeCommentId: string | null;
}) {
if (input.wakeCommentId) return true;
// A structured interaction response is new input just like a comment. It
// must run after the turn that created the interaction instead of being
// merged into that still-running turn.
if (
readNonEmptyString(input.contextSnapshot?.interactionId) &&
readNonEmptyString(input.contextSnapshot?.interactionStatus)
) {
return true;
}
const wakeReason = readNonEmptyString(input.contextSnapshot?.wakeReason);
if (wakeReason === "issue_children_completed" && (
input.contextSnapshot?.onboardingCompletion === true ||
(input.contextSnapshot?.statusDecisionSource === "native_status_decision" &&
readNonEmptyString(input.contextSnapshot?.nativeChildCompletionDecisionId))
)) return true;
return Boolean(
wakeReason && RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP.has(wakeReason),
);
}
/** Agent Chat outbox receipts. The outbox, not a skipped receipt, owns a capped message. */
const CONVERSATION_COMMENT_KEY_PREFIX = "conversation-comment:";
const QUEUED_RUN_CLAIM_ATTEMPTS_PER_PASS = 24;
const QUEUED_RUN_CLAIM_BACKOFF_BASE_MS = 5_000;
const QUEUED_RUN_CLAIM_BACKOFF_MAX_MS = 60_000;
const QUEUED_PERSON_RUN_CLAIM_BACKOFF_MS = 2_000;
function dailyLimitConversationRefusal(cap: { reason: string; observed: number; limit: number }) {
// No receipt is written, so the person's message stays owed in the outbox
// until the limit resets (agent-conversations.ts conversationDeliveryHold).
return conflict("Agent reached its daily limit", { reason: cap.reason, observed: cap.observed, limit: cap.limit });
}
function readNonEmptyString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value : null;
}
export async function claimDueTimerHeartbeatForAgent(
db: Db,
agent: typeof agents.$inferSelect,
now: Date,
intervalSec: number,
) {
const dueBefore = new Date(now.getTime() - intervalSec * 1000);
const claimed = await db.transaction(async tx => {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
return tx
.update(agents)
.set({ lastHeartbeatAt: now, updatedAt: now })
.where(and(
eq(agents.id, agent.id),
eq(agents.companyId, agent.companyId),
or(
lte(agents.lastHeartbeatAt, dueBefore),
and(isNull(agents.lastHeartbeatAt), lte(agents.createdAt, dueBefore)),
),
))
.returning({ id: agents.id })
.then((rows) => rows[0] ?? null);
});
if (!claimed) return null;
return { wasFirstHeartbeat: !agent.lastHeartbeatAt };
}
/** Queue admission and dispatch share the service lifecycle callbacks and execution ownership. */
// Wake admissions that have not committed their wake and run yet. Process-wide, like
// the pending-wake set: a server replacement loses these, and only these, pending wakes.
/** Rolls back a claim whose server closed run admission while it was being taken. */
class RunAdmissionClosedDuringClaim extends Error {}
const unsavedWakeAdmissions = new Set<object>();
export function unsavedWakeAdmissionCount() { return unsavedWakeAdmissions.size; }
export async function notifyOtherAgentsQueuedOnSettledIssue(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect,
"companyId" | "agentId" | "runtimeMode" | "contextSnapshot">): Promise<boolean> {
const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
if (run.runtimeMode !== "native" || !issueId) return false;
const [waiting] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, run.companyId),
eq(heartbeatRuns.status, "queued"),
ne(heartbeatRuns.agentId, run.agentId),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,
)).limit(1);
if (!waiting) return false;
await db.transaction(tx => notifyQueuedRunDispatch(tx, { companyId: run.companyId }));
return true;
}
export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDependencies) {
const {
setRunStatusFromLive,
setWakeupStatus,
appendRunEvent,
releaseIssueExecutionAndPromote,
applyWakeQueuePostCommitEffects,
issuesSvc,
options,
publishRunLifecyclePluginEvent,
finalizeAgentStatus,
getAgent,
cancelRunInternal,
toAgentOrgRow,
getAgentInvokability,
budgets,
activeRunExecutions,
treeControlSvc,
runDispatch,
applyRunDispatchPostCommitEffects,
resolveResponsibleUserIdForRun,
getIssueExecutionContext,
isHeartbeatRunTerminalStatus,
liveRunExecutions,
getSchedulingSuppression,
isRunAdmissionClosed: isStartAdmissionClosed = isProcessRunAdmissionClosed,
isReattachPhaseActive = isProcessReattachPhaseActive,
refreshRunAdmission = refreshRunStartYieldIfStale,
resumeExecutionWaitComments,
getWorktreeExecutionCutoff,
resumeQueuedCommentInterrupt,
resumeSavedLegacyComments,
cancelActiveForAgentInternal,
listCompanyAgentOrgRows,
executeRun,
canStartNewExternalBotWork,
activeRunExecutionPromises,
activeWakeupPromises,
instanceSettings,
getRun,
sweepPendingCleanupLeases,
resolveExplicitResumeSessionOverride,
resolveSessionBeforeForWakeup,
hasResolvablePriorSessionWorkspaceForWake,
getRoutineEnvForExecutionIssue,
resolveResponsibleUserIdForRunSeed,
formatIssueIdentifierLink,
wakeQueue,
isSameTaskScope,
runTaskKey,
filterZombieCoalesceTarget,
} = dependencies;
const isRunAdmissionClosed = () => isStartAdmissionClosed() || isReattachPhaseActive();
const { parseHeartbeatPolicy, getHeartbeatDailyCapBlock, hasActionableTimerWork, nextTimerHeartbeatAt } = createHeartbeatTimerPolicy(db);
// claimQueuedRun works from the caller's copy of a queued run. Another
// dispatcher (an overlapping server during a swap) may have started and even
// finished that run since, so every cancellation it makes is a
// compare-and-set from "queued": it never overwrites a newer outcome.
async function cancelRunIfStillQueued(runId: string, patch: Partial<typeof heartbeatRuns.$inferInsert>) {
const write = await setRunStatusFromLive(runId, "cancelled", ["queued"], patch);
return write.updated ? write.run : null;
}
async function cancelQueuedRunForHeartbeatDailyCap(
run: typeof heartbeatRuns.$inferSelect,
dailyCapBlock: NonNullable<
Awaited<ReturnType<typeof getHeartbeatDailyCapBlock>>
>,
) {
const now = new Date();
const reason =
"Cancelled because the agent reached a per-day heartbeat budget cap before adapter invocation";
const cancelled = await cancelRunIfStillQueued(run.id, {
finishedAt: now,
error: reason,
errorCode: dailyCapBlock.reason,
resultJson: {
...parseObject(run.resultJson),
stopReason: dailyCapBlock.reason,
observed: dailyCapBlock.observed,
limit: dailyCapBlock.limit,
effectiveTimeoutSec: 0,
timeoutConfigured: false,
timeoutSource: "heartbeat_daily_cap_gate",
timeoutFired: false,
},
});
if (!cancelled) return null;
await setWakeupStatus(run.wakeupRequestId, "skipped", {
finishedAt: now,
error: reason,
});
await releaseConversationCommentsForDailyCap(cancelled);
await appendRunEvent(cancelled, {
eventType: "lifecycle",
stream: "system",
level: "warn",
message: reason,
payload: {
reason: dailyCapBlock.reason,
observed: dailyCapBlock.observed,
limit: dailyCapBlock.limit,
},
});
await releaseIssueExecutionAndPromote(cancelled, {
suppressImmediateRecovery: true,
});
return cancelled;
}
/**
* A chat message accepted before the limit was reached must not be consumed
* when dispatch then cancels its run at the limit. Releasing the outbox key
* on its receipts (the run's own and any coalesced into it) makes the
* message owed again; the outbox holds it until the limit resets.
*/
async function releaseConversationCommentsForDailyCap(run: typeof heartbeatRuns.$inferSelect) {
const owned = and(
eq(agentWakeupRequests.companyId, run.companyId),
eq(agentWakeupRequests.agentId, run.agentId),
or(eq(agentWakeupRequests.runId, run.id),
run.wakeupRequestId ? eq(agentWakeupRequests.id, run.wakeupRequestId) : undefined),
sql`${agentWakeupRequests.idempotencyKey} like ${CONVERSATION_COMMENT_KEY_PREFIX + "%"}`,
);
await db.transaction(async tx => {
const receipts = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(owned).for("update");
if (!receipts.length) return;
// Register the recovery signal before the write it announces.
await notifyIssueGraphRecovery(tx, { companyId: run.companyId, runId: run.id });
await tx.update(agentWakeupRequests).set({
idempotencyKey: null,
payload: sql`coalesce(${agentWakeupRequests.payload}, '{}'::jsonb)
|| jsonb_build_object('releasedConversationCommentKey', ${agentWakeupRequests.idempotencyKey})`,
updatedAt: new Date(),
}).where(and(owned, inArray(agentWakeupRequests.id, receipts.map(row => row.id))));
});
}
async function markTimerHeartbeatChecked(
agentId: string,
source: WakeupOptions["source"],
) {
if (source !== "timer") return;
await db.transaction(async tx => {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
const now = new Date();
await tx.update(agents).set({ lastHeartbeatAt: now, updatedAt: now })
.where(eq(agents.id, agentId));
});
}
const claimDueTimerHeartbeat = (agent: typeof agents.$inferSelect, now: Date, intervalSec: number) =>
claimDueTimerHeartbeatForAgent(db, agent, now, intervalSec);
async function listQueuedRunDependencyReadiness(
companyId: string,
queuedRuns: Array<Pick<HeartbeatRun, "contextSnapshot">>,
) {
const issueIds = [
...new Set(
queuedRuns
.map((run) =>
readNonEmptyString(parseObject(run.contextSnapshot).issueId),
)
.filter((issueId): issueId is string => Boolean(issueId)),
),
];
if (issueIds.length === 0) {
return new Map<
string,
Awaited<ReturnType<typeof issuesSvc.getDependencyReadiness>>
>();
}
return issuesSvc.listDependencyReadiness(companyId, issueIds);
}
async function countRunningRunsForAgent(agentId: string) {
const [{ count }] = await db
.select({ count: sql<number>`count(*)` })
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.agentId, agentId),
eq(heartbeatRuns.status, "running"),
),
);
return Number(count ?? 0);
}
async function withChatControlRecoveryGate(
run: typeof heartbeatRuns.$inferSelect,
stage: "claim" | "dispatch",
onClear: (tx: Db) => Promise<typeof heartbeatRuns.$inferSelect | null>,
nativeRecovery = false,
) {
const issueId = readNonEmptyString(
parseObject(run.contextSnapshot).issueId,
);
if (!issueId || run.invocationSource !== "automation") return onClear(db);
await options.beforeChatControlRecoveryCheck?.({
runId: run.id,
issueId,
stage,
});
if (stage === "dispatch" && nativeRecovery) {
// Protected historical/already-admitted owners did not previously take
// this fence's locks. A fresh exact read keeps unrelated Board contention
// from turning their recovery into a new admission failure.
const [current] = await db
.select()
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.id, run.id),
eq(heartbeatRuns.companyId, run.companyId),
eq(heartbeatRuns.agentId, run.agentId),
eq(heartbeatRuns.status, "running"),
),
)
.limit(1);
if (
current &&
current.wakeupRequestId === run.wakeupRequestId &&
parseObject(current.contextSnapshot).issueId === issueId
) {
const admission = readChatControlRecoveryAdmission(current);
const expected = readChatControlRecoveryAdmission(run);
if (
expected !== "invalid" &&
(admission === "admitted" ||
(admission === "historical" && expected === "historical"))
)
return onClear(db);
}
}
let terminal: typeof heartbeatRuns.$inferSelect | null = null;
try {
const attempt = () => db.transaction(async (tx) => {
terminal = null;
// Same queue-edit lock order, then the close committer's conversation
// row. NOWAIT releases partial locks on contention. Claim defers to the
// queue; dispatch retries this transaction before considering failure.
const [issue] = await tx
.select({ id: issues.id })
.from(issues)
.where(
and(eq(issues.id, issueId), eq(issues.companyId, run.companyId)),
)
.for("update", { noWait: true })
.limit(1);
if (!issue) return null;
if (run.wakeupRequestId)
await tx
.select({ id: agentWakeupRequests.id })
.from(agentWakeupRequests)
.where(
and(
eq(agentWakeupRequests.id, run.wakeupRequestId),
eq(agentWakeupRequests.companyId, run.companyId),
eq(agentWakeupRequests.agentId, run.agentId),
),
)
.for("update", { noWait: true })
.limit(1);
const [current] = await tx
.select()
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.id, run.id),
eq(heartbeatRuns.companyId, run.companyId),
eq(heartbeatRuns.agentId, run.agentId),
),
)
.for("update", { noWait: true })
.limit(1);
if (
!current ||
current.status !== (stage === "claim" ? "queued" : "running")
)
return null;
const admission = readChatControlRecoveryAdmission(current);
const expectedAdmission = readChatControlRecoveryAdmission(run);
const admissionLost =
expectedAdmission !== "historical" && admission === "historical";
if (
stage === "dispatch" &&
nativeRecovery &&
!admissionLost &&
(admission === "historical" || admission === "admitted")
)
return onClear(tx as unknown as Db);
// A new warm child can inherit a prior runner's PID during preparation.
// Those fields do not prove this run's turn was dispatched.
// Exact admitted/historical recovery retains its existing ownership;
// a marked, unadmitted bootstrap must earn admission even after crash.
const proof =
admission === "invalid" || admissionLost
? { kind: "unresolved" as const }
: await readChatControlRecoveryStop(
tx as unknown as Db,
{
companyId: run.companyId,
issueId,
agentId: run.agentId,
sourceRunId: run.id,
},
true,
);
if (proof.kind === "clear") {
if (stage === "claim" || admission === "required")
await tx
.update(heartbeatRuns)
.set({
runnerProfileJson: {
...parseObject(current.runnerProfileJson),
[CHAT_CONTROL_RECOVERY_ADMISSION_KEY]:
chatControlRecoveryAdmission(
current,
stage === "dispatch"
? "admitted"
: admission === "admitted"
? "admitted"
: "required",
),
},
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, current.id));
return onClear(tx as unknown as Db);
}
if (proof.kind === "unresolved" && stage === "claim") return null;
const now = new Date();
const stopped = proof.kind === "stopped";
const code = stopped
? CHAT_CONTROL_RECOVERY_STOP_CODE
: CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE;
const error = stopped
? "Automatic continuation stopped by the committed chat conversation close. Send a new request in chat or on the Board to start fresh work."
: "Automatic continuation source could not be verified before provider admission. Review the task and send a fresh request; this attempt will not automatically retry.";
await notifyChatRunMilestones(tx, { companyId: current.companyId, issueId });
await notifyInstructionCollection(tx, { companyId: current.companyId, runId: current.id });
await notifyExternalAgentSettlement(tx, { companyId: current.companyId, runId: current.id });
await notifyCancelledWorkspacePreservation(tx, { companyId: current.companyId, runId: current.id });
await notifyExecutionRunWork(tx, { companyId: current.companyId, agentId: current.agentId, runId: current.id });
await notifyUnsafeWorkspaceExportRunChange(tx, current);
[terminal] = await tx
.update(heartbeatRuns)
.set({
status: stopped ? "cancelled" : "failed",
errorCode: code,
error,
finishedAt: now,
resultJson: {
...parseObject(current.resultJson),
automaticRecovery: {
code,
providerDispatched: false,
...(stopped
? {
sourceRunId: proof.sourceRunId,
conversationId: proof.conversationId,
publicationId: proof.publicationId,
}
: {}),
},
},
updatedAt: now,
})
.where(
and(
eq(heartbeatRuns.id, current.id),
eq(heartbeatRuns.status, current.status),
),
)
.returning();
if (!terminal) return null;
if (current.wakeupRequestId)
await tx
.update(agentWakeupRequests)
.set({
status: stopped ? "skipped" : "failed",
error,
finishedAt: now,
updatedAt: now,
})
.where(
and(
eq(agentWakeupRequests.id, current.wakeupRequestId),
eq(agentWakeupRequests.companyId, current.companyId),
eq(agentWakeupRequests.agentId, current.agentId),
eq(agentWakeupRequests.runId, current.id),
ne(agentWakeupRequests.status, "cancelled"),
),
);
await tx
.update(issues)
.set({
executionRunId: null,
executionAgentNameKey: null,
executionLockedAt: null,
updatedAt: now,
})
.where(
and(
eq(issues.companyId, current.companyId),
eq(issues.id, issueId),
eq(issues.executionRunId, current.id),
),
);
return null;
});
const result = stage === "dispatch"
? await retryChatControlAdmission(attempt)
: await attempt();
if (terminal) {
const settled = terminal as typeof heartbeatRuns.$inferSelect;
publishLiveEvent({
companyId: settled.companyId,
type: "heartbeat.run.status",
payload: {
runId: settled.id,
agentId: settled.agentId,
status: settled.status,
errorCode: settled.errorCode,
error: settled.error,
},
});
publishRunLifecyclePluginEvent(settled);
if (stage === "dispatch")
await finalizeAgentStatus(settled.agentId, "cancelled");
}
return result;
} catch (error) {
if (!isExternalChatWaitAuthorizationContention(error)) throw error;
if (stage === "dispatch") {
// No effect was admitted. Let the existing setup-failure path settle
// this attempt distinctly; it must not become a successful close.
throw new ChatControlRecoveryUnresolvedError();
}
return null;
}
}
// Run admission can close (SIGTERM drain, or the hand-over yield) while
// claimQueuedRun awaits its checks, after its first admission check passed.
// Checked again inside the claim's transaction, right after the write that
// moves the run to running: throwing rolls the whole claim back, so the run,
// its wake and the task's execution lock stay exactly as they were, queued
// for the serving server.
function confirmRunAdmissionStillOpen(claimedRun: unknown) {
if (claimedRun && isRunAdmissionClosed()) throw new RunAdmissionClosedDuringClaim();
}
const CLOSED_TASK_CONTINUATION = { error: "The task was closed before this continuation started.",
errorCode: "issue_closed_before_start" };
const RECOVERY_NO_LONGER_NEEDED = { error: "The task no longer needed this recovery run when its turn came.",
errorCode: "recovery_no_longer_needed" };
/**
* A system recovery run waits behind people's work, so by its turn the task
* may have moved on. It is still needed only while its agent still owns the
* task and no other run continues it: none started after it was queued,
* none running, and no queued non-recovery run (a person's or an
* assignment's). Of two queued recovery runs for one task, the newer one stays.
*/
async function recoveryRunNoLongerNeeded(run: typeof heartbeatRuns.$inferSelect, issueId: string,
task: { assigneeAgentId: string | null }) {
if (run.wakeupRequestId) {
const [request] = await db.select({ actor: agentWakeupRequests.requestedByActorType }).from(agentWakeupRequests)
.where(and(eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.id, run.wakeupRequestId)));
// A person asked for this one (a board restore): it is theirs to keep.
if (request?.actor === "user") return false;
}
if (task.assigneeAgentId !== run.agentId) return true;
const recoveryReasons = sql.join([...RECOVERY_WAKE_REASONS].map(reason => sql`${reason}`), sql`, `);
// The run it recovers is never newer work, whatever the two rows' timestamps say.
const notOwnRuns = [run.id, run.retryOfRunId, readNonEmptyString(parseObject(run.contextSnapshot).previousRunId)]
.filter((id): id is string => Boolean(id && isUuidLike(id)));
const [other] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, run.companyId),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,
notInArray(heartbeatRuns.id, notOwnRuns),
or(
// Ran after this one was queued: the task already continued.
gt(heartbeatRuns.startedAt, run.createdAt),
eq(heartbeatRuns.status, "running"),
and(eq(heartbeatRuns.status, "queued"), or(
sql`coalesce(${heartbeatRuns.contextSnapshot}->>'wakeReason', '') not in (${recoveryReasons})`,
gt(heartbeatRuns.createdAt, run.createdAt))),
),
)).limit(1);
return Boolean(other);
}
/** System continuations that only make sense for an open task. */
const SYSTEM_CONTINUATION_WAKE_REASONS = new Set(["issue_recovery_action_restored", "issue_continuation_needed",
"issue_assignment_recovery", ISSUE_BLOCKER_STALLED_WAKE_REASON, "issue_children_completed"]);
/**
* Cancels a system continuation whose task closed while it was queued,
* unless it carries a person's message that no run of this agent has
* received yet (admission can attach deferred comments to a queued system
* continuation without changing its wake reason). Its wake and run are
* locked in the order admission's coalescing uses while the message check
* reads them. The cancel itself goes through the run lifecycle's
* compare-and-set, so it saves the same never-started proof as every other
* queued-run cancel (task release needs it to promote messages saved behind
* this run), and it is fenced on the row version the check saw: a message
* attached after the check changes the run, the cancel matches nothing, and
* the check runs again.
*/
async function cancelClosedTaskContinuation(run: typeof heartbeatRuns.$inferSelect, issueId: string,
outcome: { error: string; errorCode: string } = CLOSED_TASK_CONTINUATION) {
for (let attempt = 0; attempt < 3; attempt += 1) {
const decision = await checkClosedTaskContinuation(run, issueId);
if (decision.kind !== "cancel") return decision;
const write = await setRunStatusFromLive(run.id, "cancelled", ["queued"], {
finishedAt: new Date(), ...outcome,
}, undefined, sql`${heartbeatRuns}.xmin::text = ${decision.rowVersion}`);
if (write.updated && write.run) return { kind: "cancelled" as const, run: write.run };
if (write.run?.status !== "queued") return { kind: "not_queued" as const };
}
// Still changing under us: leave it queued; the next dispatch pass checks again.
return { kind: "still_queued" as const };
}
async function checkClosedTaskContinuation(run: typeof heartbeatRuns.$inferSelect, issueId: string) {
return db.transaction(async (tx) => {
if (run.wakeupRequestId) await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.id, run.wakeupRequestId), eq(agentWakeupRequests.companyId, run.companyId),
)).for("update");
const [locked] = await tx.select({ run: heartbeatRuns, rowVersion: sql<string>`${heartbeatRuns}.xmin::text` })
.from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued"),
)).for("update");
if (!locked) return { kind: "not_queued" as const };
const current = locked.run;
const context = parseObject(current.contextSnapshot);
const commentIds = [...new Set([...extractWakeCommentIds(context), ...queuedCommentIdsFromRunContext(context),
readNonEmptyString(context.wakeCommentId), readNonEmptyString(context.commentId)]
.filter((id): id is string => Boolean(id)))];
if (commentIds.length > 0) {
const personComments = await tx.select({ id: issueComments.id }).from(issueComments).where(and(
eq(issueComments.companyId, current.companyId), eq(issueComments.issueId, issueId),
inArray(issueComments.id, commentIds), eq(issueComments.authorType, "user"),
isNull(issueComments.createdByRunId), isNull(issueComments.deletedAt)));
const delivered = await deliveredQueuedCommentIds(tx as unknown as Db, { companyId: current.companyId, issueId,
agentId: current.agentId, commentIds: personComments.map(comment => comment.id) });
if (personComments.some(comment => !delivered.has(comment.id))) return { kind: "carries_person_message" as const };
}
return { kind: "cancel" as const, rowVersion: locked.rowVersion };
});
}
async function claimQueuedRun(
run: typeof heartbeatRuns.$inferSelect,
companyAgents?: AgentOrgRow[],
deferredPostCommitEffects?: WakeQueuePostCommitEffect[],
deferredCancellations?: Array<() => Promise<void>>,
slots?: { full: boolean },
) {
if (run.status !== "queued") return run;
// Every start path ends here (dispatch passes, executeRun of a queued run,
// a delivered continuation). A server whose run admission is closed (it
// received SIGTERM and is draining) or that yields new starts to a
// replacement already dispatching leaves the run queued for that server:
// anything it started now would be cut off by its own exit.
const refreshing = refreshRunAdmission();
if (refreshing) await refreshing;
if (isRunAdmissionClosed()) return null;
// A system continuation queued while its task was open has nothing left to
// do once another run finished or cancelled the task before this one
// started. A person's message on a done task is not one of these reasons.
const runContext = parseObject(run.contextSnapshot);
const systemContinuationReason = readNonEmptyString(runContext.wakeReason);
const continuedIssueId = readNonEmptyString(runContext.issueId);
if (systemContinuationReason && SYSTEM_CONTINUATION_WAKE_REASONS.has(systemContinuationReason) && continuedIssueId
&& runContext.onboardingCompletion !== true) {
const [target] = await db.select({ status: issues.status, assigneeAgentId: issues.assigneeAgentId }).from(issues)
.where(and(eq(issues.companyId, run.companyId), eq(issues.id, continuedIssueId)));
const closedTask = Boolean(target && ["done", "cancelled"].includes(target.status));
// Dropped quietly: the run is cancelled before it started and its wake is
// skipped, the same as a continuation whose task closed. Nothing failed.
const outcome = closedTask ? CLOSED_TASK_CONTINUATION
: target && RECOVERY_WAKE_REASONS.has(systemContinuationReason)
&& await recoveryRunNoLongerNeeded(run, continuedIssueId, target) ? RECOVERY_NO_LONGER_NEEDED : null;
if (outcome) {
const closed = await cancelClosedTaskContinuation(run, continuedIssueId, outcome);
if (closed.kind === "cancelled") {
const cancelled = closed.run;
await setWakeupStatus(run.wakeupRequestId, "skipped", { finishedAt: cancelled.finishedAt ?? new Date(), error: outcome.errorCode });
await releaseIssueExecutionAndPromote(cancelled, { suppressImmediateRecovery: true, deferredPostCommitEffects });
}
// A person's undelivered message rides on this run: it runs as their
// follow-up, which may reopen the task through the normal path.
if (closed.kind !== "carries_person_message") return null;
}
}
const interactionDelivery = await recheckQueuedRunInteractions(db, run);
if (interactionDelivery.run.status !== "queued") return null;
run = { ...run, contextSnapshot: interactionDelivery.run.contextSnapshot,
resultJson: interactionDelivery.run.resultJson, updatedAt: interactionDelivery.run.updatedAt };
if (interactionDelivery.denied) {
const cancelled = await cancelRunIfStillQueued(run.id, {
finishedAt: new Date(), error: "Queued agent input is no longer permitted.",
errorCode: INTERACTION_DELIVERY_DENIED,
});
if (cancelled) {
await setWakeupStatus(run.wakeupRequestId, "skipped", { finishedAt: new Date(), error: INTERACTION_DELIVERY_DENIED });
await appendRunEvent(cancelled, { eventType: "lifecycle", stream: "system", level: "warn",
message: "Queued agent input was refused after its interaction permission changed.", payload: { code: INTERACTION_DELIVERY_DENIED, denials: interactionDelivery.denials } });
await releaseIssueExecutionAndPromote(cancelled, { suppressImmediateRecovery: true, deferredPostCommitEffects });
}
return null;
}
const [caseHandoffReceipt] = run.wakeupRequestId ? await db.select().from(caseHandoffs).where(and(
eq(caseHandoffs.wakeupRequestId, run.wakeupRequestId), eq(caseHandoffs.companyId, run.companyId),
)).limit(1) : [];
if (caseHandoffReceipt) {
const [current] = await db.select({ version: cases.handoffVersion, issueId: cases.workIssueId,
agentId: issues.assigneeAgentId, statusVersion: issues.statusVersion }).from(cases)
.innerJoin(issues, eq(issues.id, cases.workIssueId)).where(and(eq(cases.id, caseHandoffReceipt.caseId),
eq(cases.companyId, run.companyId), eq(issues.companyId, run.companyId))).limit(1);
if (!["pending_dispatch", "delivered"].includes(caseHandoffReceipt.state) || caseHandoffReceipt.toAgentId !== run.agentId ||
!current || current.version !== caseHandoffReceipt.version || current.issueId !== caseHandoffReceipt.issueId ||
current.agentId !== run.agentId || (!run.startedAt && current.statusVersion !== caseHandoffReceipt.appliedIssueStatusVersion)) {
const cancelled = await db.transaction(async tx => {
const [currentRun] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued")))
.for("update");
if (!currentRun) return null;
await notifyChatRunMilestones(tx, { companyId: run.companyId, runId: run.id,
issueId: run.nativeIssueId ?? run.issueId ?? readNonEmptyString(parseObject(run.contextSnapshot).issueId) ?? undefined });
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
await notifyExternalAgentSettlement(tx, { companyId: run.companyId, runId: run.id });
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
await notifyExecutionRunWork(tx, { companyId: run.companyId, agentId: run.agentId, runId: run.id });
await notifyUnsafeWorkspaceExportRunChange(tx, run);
const [row] = await tx.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date(),
errorCode: "case_handoff_superseded", error: "This case handoff was cancelled or superseded.", updatedAt: new Date() })
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued"))).returning();
return row;
});
if (cancelled) await setWakeupStatus(run.wakeupRequestId, "cancelled", { finishedAt: new Date(), error: cancelled.error });
return null;
}
if (!(await instanceSettings.getExperimental()).enableCases) return null;
}
const caseIssueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
if (caseIssueId) {
const [pending] = await db.select().from(caseHandoffs).where(and(eq(caseHandoffs.companyId, run.companyId), eq(caseHandoffs.issueId, caseIssueId),
inArray(caseHandoffs.state, ["awaiting_release", "pending_dispatch"]))).limit(1);
if (pending && (pending.wakeupRequestId !== run.wakeupRequestId || pending.state === "awaiting_release")) return null;
}
// Claim is the first execution boundary. startedAt survives same-run
// retries; restart recovery may also enter executeRun already running.
// Neither may reinterpret an admitted, already-started message as new work.
if (!run.startedAt && !(await slackMentionAllowsRunStart(db, run))) {
const now = new Date();
const reason = "Message did not @mention the bot";
// This runs under the agent start lock. Do not use cancelRunInternal,
// which recursively starts the next run and reacquires that lock.
const cancelled = await cancelRunIfStillQueued(run.id, {
finishedAt: now,
error: reason,
errorCode: "chat_mention_required",
});
if (cancelled) {
await setWakeupStatus(run.wakeupRequestId, "skipped", { finishedAt: now, error: reason });
await appendRunEvent(cancelled, {
eventType: "lifecycle", stream: "system", level: "warn", message: reason,
});
await releaseIssueExecutionAndPromote(cancelled, {
suppressImmediateRecovery: true, deferredPostCommitEffects,
});
}
return null;
}
const agent = await getAgent(run.agentId);
if (!agent) {
await cancelRunInternal(
run.id,
"Cancelled because the agent no longer exists",
);
return null;
}
// Keep accepted work in the durable queue until setup completes.
if (isAgentAwaitingSetup(agent)) return null;
const invokability = companyAgents
? evaluateAgentInvokability(toAgentOrgRow(agent), companyAgents)
: await getAgentInvokability(agent);
if (!invokability.invokable) {
if (caseHandoffReceipt) return null;
await cancelRunInternal(
run.id,
`Cancelled because the agent is not invokable: ${invokability.reason}`,
);
return null;
}
if (!run.startedAt && !await canStartNewExternalBotWork(agent)) return null;
const context = parseObject(run.contextSnapshot);
const budgetBlock = await budgets.getInvocationBlock(
run.companyId,
run.agentId,
{
issueId: readNonEmptyString(context.issueId),
projectId: readNonEmptyString(context.projectId),
},
);
if (budgetBlock) {
if (caseHandoffReceipt) return null;
// A temporary hold (accounting that is still settling) only delays
// accepted work. Cancelling the unstarted run would drop the user's
// message, Retry, or recovery it carries; keep it queued. Settling
// accounting re-notifies run dispatch. A reached hard stop is the board's.
if (!run.startedAt && !(await isBoardOwnedBudgetBlock(db, run.companyId, budgetBlock))) {
logger.info({ runId: run.id, agentId: run.agentId, companyId: run.companyId, reason: "budget_hold_settling",
scopeType: budgetBlock.scopeType }, "claimQueuedRun: queued run kept queued behind a temporary budget hold");
return null;
}
// Record which budget stops this queued run, so Raise budget and resume
// continues it (budget-resume-continuation.ts). Saved before the cancel
// so a crash between them cannot lose it; the resume matches it only
// against a recorded cancellation with this same reason.
const budgetAdmissionStop = { scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId,
reason: budgetBlock.reason.slice(0, 512) };
const stamped = await db.transaction(async tx => {
// The same commit touches the scope's open incident. A Raise budget
// and resume whose repeatable-read snapshot predates this commit then
// conflicts on that row and retries, so its resume sees the stamp
// (a still-queued stamped run is claimed and waits for this cancel).
const open = await tx.update(budgetIncidents).set({ updatedAt: new Date() }).where(and(
eq(budgetIncidents.companyId, run.companyId), eq(budgetIncidents.scopeType, budgetBlock.scopeType),
eq(budgetIncidents.scopeId, budgetBlock.scopeId), eq(budgetIncidents.status, "open")))
.returning({ id: budgetIncidents.id });
if (!open.length) return false;
await tx.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb)
|| jsonb_build_object('budgetAdmissionStop', jsonb_build_object('scopeType', ${budgetAdmissionStop.scopeType}::text,
'scopeId', ${budgetAdmissionStop.scopeId}::text, 'reason', ${budgetAdmissionStop.reason}::text))` })
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, run.status)));
return true;
});
if (!stamped) {
// No open incident: a resume may have just closed it and raised the
// budget. Read the block again; resumed work stays queued.
const current = await budgets.getInvocationBlock(run.companyId, run.agentId, {
issueId: readNonEmptyString(context.issueId), projectId: readNonEmptyString(context.projectId) });
if (!current) return null;
if (!run.startedAt && !(await isBoardOwnedBudgetBlock(db, run.companyId, current))) return null;
await cancelRunInternal(run.id, current.reason);
return null;
}
await cancelRunInternal(run.id, budgetBlock.reason, { resultJson: { budgetAdmissionStop } });
return null;
}
const dailyCapBlock = await getHeartbeatDailyCapBlock(
agent,
parseHeartbeatPolicy(agent),
{
excludeRunId: run.id,
checkRunCap: true,
checkCostCap: true,
},
);
if (dailyCapBlock) {
if (caseHandoffReceipt) return null;
await cancelQueuedRunForHeartbeatDailyCap(run, dailyCapBlock);
return null;
}
const issueId = readNonEmptyString(context.issueId);
if (issueId && activeRunExecutions.size > 0) {
// Native finalization publishes success before workspace synchronization,
// provider suspension, and lease release finish. A queued comment must
// not acquire a fresh sandbox while its predecessor still owns that work.
// The executor's finally block retries this agent after removing its owner.
const [settlingOwner] = await db.select({ id: heartbeatRuns.id })
.from(heartbeatRuns)
.where(and(
eq(heartbeatRuns.companyId, run.companyId),
or(eq(heartbeatRuns.agentId, run.agentId), inArray(heartbeatRuns.status, [...HEARTBEAT_RUN_TERMINAL_STATUSES])),
eq(heartbeatRuns.runtimeMode, "native"),
inArray(heartbeatRuns.id, [...activeRunExecutions]),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,
))
.limit(1);
if (settlingOwner) {
logger.info({ runId: run.id, agentId: run.agentId, companyId: run.companyId, issueId,
reason: "predecessor_settling_in_process", predecessorRunId: settlingOwner.id },
"claimQueuedRun: queued run declined at the issue execution claim");
return null;
}
}
if (issueId && !run.startedAt && agent.adapterType === "paperclip_runner") {
const cleanupBlock = await nativeCleanupHoldsAdmission(db, { companyId: run.companyId, issueId });
if (cleanupBlock) {
logger.info({ runId: run.id, agentId: run.agentId, companyId: run.companyId, issueId,
reason: "native_session_cleanup_settling", cleanupBlock },
"claimQueuedRun: queued run kept queued while the task's previous session finishes cleanup");
return null;
}
}
if (issueId) {
const activePauseHold = await treeControlSvc.getActivePauseHoldGate(
run.companyId,
issueId,
);
const treeHoldInteractionWake =
activePauseHold &&
(await isVerifiedIssueTreeControlInteractionWake(db, {
companyId: run.companyId,
issueId,
agentId: run.agentId,
runId: run.id,
wakeupRequestId: run.wakeupRequestId,
contextSnapshot: context,
}));
if (activePauseHold && !treeHoldInteractionWake) {
const refuse = async () => {
// Record the covering hold so its release re-dispatches this work once.
await cancelRunInternal(
run.id,
"Cancelled because issue is held by an active subtree pause hold",
{ errorCode: ISSUE_PAUSED_ERROR_CODE, resultJson: pauseSuppressionResult(activePauseHold) },
);
await logActivity(db, {
companyId: run.companyId,
actorType: "system",
actorId: "system",
agentId: run.agentId,
runId: run.id,
action: "issue.tree_hold_run_interrupted",
entityType: "heartbeat_run",
entityId: run.id,
issueId: issueId,
details: {
issueId,
holdId: activePauseHold.holdId,
rootIssueId: activePauseHold.rootIssueId,
source: "heartbeat.claim_queued_run",
securityPrinciples: [
"Complete Mediation",
"Fail Securely",
"Secure Defaults",
],
},
});
// A release that committed before this refusal did not see it.
await resumeIfPauseAlreadyReleased(db, { companyId: run.companyId, issueId,
holdId: activePauseHold.holdId, wakeup: (agentId, opts) => trackWakeup(agentId, opts) });
};
// Cancellation promotes the agent's next queued run under the agent
// start lock, so a caller holding that lock settles it after release.
if (deferredCancellations) deferredCancellations.push(refuse);
else await refuse();
return null;
}
const dependencyReadiness = await issuesSvc.listDependencyReadiness(
run.companyId,
[issueId],
);
const readiness = dependencyReadiness.get(issueId);
const unresolvedBlockerCount = readiness?.unresolvedBlockerCount ?? 0;
// The run's wake request records who asked for it.
const blockerStallVerified = unresolvedBlockerCount > 0 && await verifyBlockerStallWake(db, {
companyId: run.companyId, issueId, contextSnapshot: context,
requestedByActorType: run.wakeupRequestId
? (await db.select({ actor: agentWakeupRequests.requestedByActorType }).from(agentWakeupRequests)
.where(eq(agentWakeupRequests.id, run.wakeupRequestId)).then(rows => rows[0]?.actor ?? null))
: null,
});
if (
unresolvedBlockerCount > 0 &&
!allowsDependencyBlockedWake(
context,
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
blockerStallVerified,
await isVerifiedDependencyInteractionResponse(db, { companyId: run.companyId,
issueId, agentId: run.agentId, context }),
)
) {
await cancelQueuedRunForBlockedDependencies(
run,
issueId,
readiness?.unresolvedBlockerIssueIds ?? [],
);
logger.info(
{ runId: run.id, issueId, unresolvedBlockerCount },
"claimQueuedRun: cancelled blocked queued run",
);
return null;
}
const staleness = await runDispatch.cancelStaleQueuedRun({
runId: run.id,
companyId: run.companyId,
expectedStatus: "queued",
});
if (staleness.outcome === "cancelled" || staleness.outcome === "deferred") {
applyRunDispatchPostCommitEffects(staleness.postCommitEffects);
if (staleness.outcome === "cancelled" && staleness.errorCode === "issue_assignee_changed") {
await releaseIssueExecutionAndPromote(run, { suppressImmediateRecovery: true, deferredPostCommitEffects }).catch((err) => {
logger.warn({ err, runId: run.id, issueId }, "claimQueuedRun: new owner's wake will be promoted by recovery");
});
}
logger.info(
{
runId: run.id,
issueId,
outcome: staleness.outcome,
errorCode: staleness.outcome === "cancelled" ? staleness.errorCode : undefined,
},
"claimQueuedRun: withheld queued run at the execution gate",
);
return null;
}
}
const claimedAt = new Date();
const responsibleUserId = await resolveResponsibleUserIdForRun({
run,
contextSnapshot: context,
issueContext: issueId
? await getIssueExecutionContext(run.companyId, issueId)
: null,
routineEnvContext: {
routineId: null,
env: null,
responsibleUserId: null,
},
});
// All ordinary and comment claims use the same company-scoped issue
// lock. A batch may claim several runs before executeRun tracks any owner.
async function lockIssueExecutionClaim(tx: Db) {
const [owner] = issueId ? await tx.select({
assigneeAgentId: issues.assigneeAgentId,
statusVersion: issues.statusVersion,
executionRunId: issues.executionRunId,
checkoutRunId: issues.checkoutRunId,
}).from(issues).where(and(
eq(issues.id, issueId), eq(issues.companyId, run.companyId),
)).for("update") : [];
const ownsIssue = owner?.assigneeAgentId === run.agentId &&
context.wakeReason !== "source_scoped_recovery_action";
const [handoff] = issueId ? await tx.select().from(caseHandoffs).where(and(eq(caseHandoffs.companyId, run.companyId), eq(caseHandoffs.issueId, issueId),
inArray(caseHandoffs.state, ["awaiting_release", "pending_dispatch"]))).limit(1) : [];
const declined = (reason: string, predecessorRunId?: string) => {
logger.info({ runId: run.id, agentId: run.agentId, companyId: run.companyId, issueId, reason,
...(predecessorRunId ? { predecessorRunId } : {}) }, "claimQueuedRun: queued run declined at the issue execution claim");
return { ownsIssue, blocked: true };
};
if (handoff && (handoff.wakeupRequestId !== run.wakeupRequestId || handoff.state === "awaiting_release"))
return declined("case_handoff_pending");
if (caseHandoffReceipt) {
const [current] = await tx.select({ state: caseHandoffs.state, version: caseHandoffs.version,
currentVersion: cases.handoffVersion, workIssueId: cases.workIssueId,
appliedIssueStatusVersion: caseHandoffs.appliedIssueStatusVersion,
toAgentId: caseHandoffs.toAgentId, wakeupRequestId: caseHandoffs.wakeupRequestId }).from(caseHandoffs)
.innerJoin(cases, eq(cases.id, caseHandoffs.caseId)).where(and(eq(caseHandoffs.id, caseHandoffReceipt.id),
eq(caseHandoffs.companyId, run.companyId))).limit(1);
if (!current || !["pending_dispatch", "delivered"].includes(current.state) || current.version !== current.currentVersion ||
current.workIssueId !== issueId || current.toAgentId !== run.agentId || current.wakeupRequestId !== run.wakeupRequestId ||
!ownsIssue || (!run.startedAt && current.appliedIssueStatusVersion !== owner?.statusVersion))
return declined("case_handoff_superseded");
}
if (ownsIssue && ["native_safe_replacement", "native_provider_overloaded"].includes(run.scheduledRetryReason ?? "") &&
owner.checkoutRunId && owner.checkoutRunId !== run.id) {
return declined("issue_checked_out_by_another_run", owner.checkoutRunId);
}
if (run.scheduledRetryReason === "native_provider_overloaded" && owner?.executionRunId &&
owner.executionRunId !== run.id && owner.executionRunId !== run.retryOfRunId) {
return declined("issue_execution_owned_by_another_run", owner.executionRunId);
}
if (ownsIssue && issueId && await taskChatRetrySourceCleanupPending(tx, run, issueId)) {
return declined("task_chat_retry_source_cleanup_pending");
}
const executionOwnerId = run.scheduledRetryReason === "native_provider_overloaded"
? run.retryOfRunId : ownsIssue ? owner?.executionRunId : null;
for (const previousRunId of new Set([executionOwnerId, ownsIssue ? run.retryOfRunId : null])) {
if (!previousRunId || previousRunId === run.id) continue;
const [previous] = await tx.select({ status: heartbeatRuns.status, agentId: heartbeatRuns.agentId,
issueId: heartbeatRuns.issueId, nativeIssueId: heartbeatRuns.nativeIssueId,
supersededByFreshContinuation: sql<boolean>`${heartbeatRuns.resultJson} ? 'freshContinuationSupersededAt'`,
contextIssueId: sql<unknown>`coalesce(${heartbeatRuns.contextSnapshot}->>'issueId', ${heartbeatRuns.contextSnapshot}->>'taskId')`,
})
.from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, previousRunId),
eq(heartbeatRuns.companyId, run.companyId),
));
if (previousRunId !== executionOwnerId && (previous?.agentId !== run.agentId ||
(previous.nativeIssueId ?? previous.issueId ?? previous.contextIssueId) !== issueId)) continue;
// A terminal result can precede workspace/lease cleanup on this or
// another controller. Local absence alone is not a release receipt.
if (!isHeartbeatRunTerminalStatus(previous?.status) ||
liveRunExecutions.has(previousRunId)) {
return declined("predecessor_still_executing", previousRunId);
}
// A fresh continuation replaced the predecessor: its writes are revoked
// and it shares no workspace with this run (a fenced remote epoch or a
// new isolated workspace), so its cleanup continues without holding it.
if (!previous?.supersededByFreshContinuation && await hasPendingRunCleanup(tx, run.companyId, previousRunId)) {
return declined("predecessor_cleanup_pending", previousRunId);
}
}
return { ownsIssue, blocked: false };
}
async function bindClaimedIssueExecution(tx: Db, ownsIssue: boolean, claimedRun: typeof heartbeatRuns.$inferSelect | null | undefined) {
if (!claimedRun || !issueId || !ownsIssue) return;
await notifyStatusCardIssueChange(tx, run.companyId);
await tx.update(issues).set({
executionRunId: claimedRun.id,
executionAgentNameKey: normalizeAgentNameKey(agent.name),
executionLockedAt: claimedAt,
updatedAt: claimedAt,
}).where(and(eq(issues.id, issueId), eq(issues.companyId, run.companyId))).catch((error: unknown) => {
// Another open task of the same routine holds its execution slot, for
// example a failed run that waits for its scheduled resume. Roll back
// this claim and keep the run queued until that slot is released.
if (isUniqueViolation(error, "issues_open_routine_execution_uq")) {
throw conflict("Another execution for this routine is already in progress");
}
throw error;
});
}
const nativeReviewContext = readNativeReviewAssignmentContext(context);
const queuedCommentIds = queuedCommentIdsFromRunContext(context);
if (
issueId &&
run.invocationSource === "automation" &&
queuedCommentIds.length > 0
)
await options.beforeChatControlRecoveryCheck?.({
runId: run.id,
issueId,
stage: "claim",
});
const queuedCommentClaim =
!nativeReviewContext && issueId && run.wakeupRequestId && queuedCommentIds.length > 0
? await db
.transaction(async (tx) => {
// Match the queue-edit lock order: issue, wake, then run. Once the
// run becomes running, a concurrent discard must observe the
// claimed wake and return an explicit conflict; if discard wins,
// this claim observes the cancelled queue and does no work.
const issueClaim = await lockIssueExecutionClaim(tx as unknown as Db);
if (issueClaim.blocked) return { kind: "stale" as const, run: null };
const wake = await tx
.select()
.from(agentWakeupRequests)
.where(
and(
eq(agentWakeupRequests.id, run.wakeupRequestId!),
eq(agentWakeupRequests.companyId, run.companyId),
eq(agentWakeupRequests.agentId, run.agentId),
),
)
.for("update")
.limit(1)
.then((rows) => rows[0] ?? null);
const lockedRun = await tx
.select()
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.id, run.id),
eq(heartbeatRuns.companyId, run.companyId),
eq(heartbeatRuns.agentId, run.agentId),
),
)
.for("update")
.limit(1)
.then((rows) => rows[0] ?? null);
if (
!wake ||
wake.status !== "queued" ||
wake.runId !== run.id ||
!lockedRun ||
lockedRun.status !== "queued" ||
lockedRun.wakeupRequestId !== wake.id
) {
return { kind: "stale" as const, run: null };
}
if (lockedRun.invocationSource === "automation") {
const admission = readChatControlRecoveryAdmission(lockedRun);
if (admission === "invalid")
return { kind: "stale" as const, run: null };
const proof = await readChatControlRecoveryStop(
tx as unknown as Db,
{
companyId: lockedRun.companyId,
issueId,
agentId: lockedRun.agentId,
sourceRunId: lockedRun.id,
},
true,
);
if (proof.kind === "unresolved")
return { kind: "stale" as const, run: null };
if (proof.kind === "stopped") {
const error =
"Automatic continuation stopped by the committed chat conversation close. Send a fresh request in chat or on the Board.";
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
await notifyExternalAgentSettlement(tx, { companyId: run.companyId, runId: run.id });
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
await notifyExecutionRunWork(tx, { companyId: run.companyId, agentId: run.agentId, runId: run.id });
await notifyUnsafeWorkspaceExportRunChange(tx, run);
const [cancelled] = await tx
.update(heartbeatRuns)
.set({
status: "cancelled",
errorCode: CHAT_CONTROL_RECOVERY_STOP_CODE,
error,
finishedAt: claimedAt,
resultJson: {
...parseObject(lockedRun.resultJson),
automaticRecovery: {
code: CHAT_CONTROL_RECOVERY_STOP_CODE,
providerDispatched: false,
sourceRunId: proof.sourceRunId,
conversationId: proof.conversationId,
publicationId: proof.publicationId,
},
},
updatedAt: claimedAt,
})
.where(
and(
eq(heartbeatRuns.id, lockedRun.id),
eq(heartbeatRuns.status, "queued"),
),
)
.returning();
if (!cancelled) return { kind: "stale" as const, run: null };
await tx
.update(agentWakeupRequests)
.set({
status: "skipped",
error,
finishedAt: claimedAt,
updatedAt: claimedAt,
})
.where(eq(agentWakeupRequests.id, wake.id));
await tx
.update(issues)
.set({
executionRunId: null,
executionAgentNameKey: null,
executionLockedAt: null,
updatedAt: claimedAt,
})
.where(
and(
eq(issues.id, issueId),
eq(issues.companyId, run.companyId),
eq(issues.executionRunId, run.id),
),
);
return { kind: "cancelled" as const, run: cancelled };
}
await tx
.update(heartbeatRuns)
.set({
runnerProfileJson: {
...parseObject(lockedRun.runnerProfileJson),
[CHAT_CONTROL_RECOVERY_ADMISSION_KEY]:
chatControlRecoveryAdmission(
lockedRun,
admission === "admitted" ? "admitted" : "required",
),
},
updatedAt: claimedAt,
})
.where(eq(heartbeatRuns.id, lockedRun.id));
}
const authoritativeIds = queuedCommentIdsFromWakePayload(
wake.payload,
);
if (authoritativeIds.length === 0) {
// Legacy/direct comment wakes carry comment ids in their ordinary
// payload and context, not in the authoritative queued-message
// envelope. Preserve their established claim path; only an
// explicitly bound queued-message envelope is subject to the
// live-comment discard gate below.
await withExternalAdmissionGuard(tx, run.companyId, run.agentId, async () => {
await assertNoExternalOverlap(tx, run.companyId, run.agentId);
await assertAgentRunSlotFree(tx, run);
});
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
const [claimedRun] = await tx
.update(heartbeatRuns)
.set({
status: "running",
runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`,
...legacyControllerClaim(run.runtimeMode),
responsibleUserId,
startedAt: lockedRun.startedAt ?? claimedAt,
updatedAt: claimedAt,
})
.where(
and(
eq(heartbeatRuns.id, lockedRun.id),
eq(heartbeatRuns.status, "queued"),
),
)
.returning();
await bindClaimedIssueExecution(tx as unknown as Db, issueClaim.ownsIssue, claimedRun);
confirmRunAdmissionStillOpen(claimedRun);
return claimedRun
? { kind: "claimed" as const, run: claimedRun }
: { kind: "stale" as const, run: null };
}
const commentRows = await tx
.select({
id: issueComments.id,
deletedAt: issueComments.deletedAt,
})
.from(issueComments)
.where(
and(
eq(issueComments.companyId, run.companyId),
eq(issueComments.issueId, issueId),
inArray(issueComments.id, authoritativeIds),
),
);
const liveIds = authoritativeIds.filter((commentId) => {
const comment = commentRows.find((row) => row.id === commentId);
return Boolean(comment && !comment.deletedAt);
});
if (liveIds.length === 0) {
const reason = "Queued messages were discarded before dispatch";
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
await notifyExternalAgentSettlement(tx, { companyId: run.companyId, runId: run.id });
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
await notifyExecutionRunWork(tx, { companyId: run.companyId, agentId: run.agentId, runId: run.id });
await notifyUnsafeWorkspaceExportRunChange(tx, run);
const [cancelled] = await tx
.update(heartbeatRuns)
.set({
status: "cancelled",
finishedAt: claimedAt,
error: reason,
errorCode: "queued_comment_discarded",
updatedAt: claimedAt,
})
.where(
and(
eq(heartbeatRuns.id, lockedRun.id),
eq(heartbeatRuns.status, "queued"),
),
)
.returning();
await tx
.update(agentWakeupRequests)
.set({
status: "cancelled",
finishedAt: claimedAt,
error: reason,
updatedAt: claimedAt,
})
.where(eq(agentWakeupRequests.id, wake.id));
await tx
.update(issues)
.set({
executionRunId: null,
executionAgentNameKey: null,
executionLockedAt: null,
updatedAt: claimedAt,
})
.where(
and(
eq(issues.id, issueId),
eq(issues.companyId, run.companyId),
eq(issues.executionRunId, run.id),
),
);
return {
kind: "cancelled" as const,
run: cancelled ?? lockedRun,
};
}
await withExternalAdmissionGuard(tx, run.companyId, run.agentId, async () => {
await assertNoExternalOverlap(tx, run.companyId, run.agentId);
await assertAgentRunSlotFree(tx, run);
});
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
await tx
.update(agentWakeupRequests)
.set({
status: "claimed",
claimedAt,
payload: withQueuedCommentIdsInWakePayload(
wake.payload,
liveIds,
),
updatedAt: claimedAt,
})
.where(eq(agentWakeupRequests.id, wake.id));
const [claimedRun] = await tx
.update(heartbeatRuns)
.set({
status: "running",
runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`,
...legacyControllerClaim(run.runtimeMode),
responsibleUserId,
startedAt: lockedRun.startedAt ?? claimedAt,
contextSnapshot: withQueuedCommentIdsInRunContext(
lockedRun.contextSnapshot,
liveIds,
),
updatedAt: claimedAt,
})
.where(
and(
eq(heartbeatRuns.id, lockedRun.id),
eq(heartbeatRuns.status, "queued"),
),
)
.returning();
await bindClaimedIssueExecution(tx as unknown as Db, issueClaim.ownsIssue, claimedRun);
confirmRunAdmissionStillOpen(claimedRun);
return claimedRun
? { kind: "claimed" as const, run: claimedRun }
: { kind: "stale" as const, run: null };
})
.catch((error) => {
if (isExternalChatWaitAuthorizationContention(error) || error instanceof RunAdmissionClosedDuringClaim)
return { kind: "stale" as const, run: null };
if (error instanceof AgentRunSlotsFull) {
if (slots) slots.full = true;
return { kind: "stale" as const, run: null };
}
throw error;
})
: null;
if (queuedCommentClaim?.kind === "cancelled") {
await appendRunEvent(queuedCommentClaim.run, {
eventType: "lifecycle",
stream: "system",
level: "warn",
message:
queuedCommentClaim.run.error ??
"Queued messages were discarded before dispatch",
});
publishLiveEvent({
companyId: queuedCommentClaim.run.companyId,
type: "heartbeat.run.status",
payload: {
runId: queuedCommentClaim.run.id,
agentId: queuedCommentClaim.run.agentId,
status: queuedCommentClaim.run.status,
invocationSource: queuedCommentClaim.run.invocationSource,
triggerDetail: queuedCommentClaim.run.triggerDetail,
error: queuedCommentClaim.run.error ?? null,
errorCode: queuedCommentClaim.run.errorCode ?? null,
startedAt: queuedCommentClaim.run.startedAt
? new Date(queuedCommentClaim.run.startedAt).toISOString()
: null,
finishedAt: queuedCommentClaim.run.finishedAt
? new Date(queuedCommentClaim.run.finishedAt).toISOString()
: null,
},
});
publishRunLifecyclePluginEvent(queuedCommentClaim.run);
// Fire-and-forget: nothing else in this path depends on the emission,
// so it must not delay the return.
void emitAgentTaskRun(db, queuedCommentClaim.run);
return null;
}
const claimed = queuedCommentClaim
? queuedCommentClaim.run
: await withChatControlRecoveryGate(run, "claim", async (tx) => {
const claimValues = {
status: "running",
runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`,
...legacyControllerClaim(run.runtimeMode),
responsibleUserId,
startedAt: run.startedAt ?? claimedAt,
updatedAt: claimedAt,
};
if (nativeReviewContext) {
if (run.scheduledRetryReason === "native_provider_overloaded") {
const predecessor = await lockIssueExecutionClaim(tx);
if (predecessor.blocked) return null;
}
return tx.transaction(async (reviewTx) => {
const reviewClaim = await claimQueuedNativeReviewRun(reviewTx as unknown as Db, {
run, claimedAt, claimValues,
agentNameKey: normalizeAgentNameKey(agent.name),
});
confirmRunAdmissionStillOpen(reviewClaim);
return reviewClaim;
});
}
return tx.transaction(async (claimTx) => {
const issueClaim = await lockIssueExecutionClaim(claimTx as unknown as Db);
if (issueClaim.blocked) return null;
if (run.wakeupRequestId) await claimTx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.id, run.wakeupRequestId), eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.agentId, run.agentId),
)).for("update");
const [lockedRun] = await claimTx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.agentId, run.agentId),
)).for("update");
if (!lockedRun || lockedRun.status !== "queued") return null;
await withExternalAdmissionGuard(claimTx, run.companyId, run.agentId, async () => {
await assertNoExternalOverlap(claimTx, run.companyId, run.agentId);
await assertAgentRunSlotFree(claimTx, run);
});
await notifyChatRunMilestones(claimTx, { companyId: run.companyId, runId: run.id, issueId: issueId ?? undefined });
const claimedRun = await claimTx.update(heartbeatRuns).set(claimValues).where(and(
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "queued"),
)).returning().then((rows) => rows[0] ?? null);
await bindClaimedIssueExecution(claimTx as unknown as Db, issueClaim.ownsIssue, claimedRun);
confirmRunAdmissionStillOpen(claimedRun);
return claimedRun;
});
}).catch((error: unknown) => {
if (error instanceof RunAdmissionClosedDuringClaim) return null;
if (error instanceof AgentRunSlotsFull) {
if (slots) slots.full = true;
return null;
}
throw error;
});
if (!claimed) return null;
publishLiveEvent({
companyId: claimed.companyId,
type: "heartbeat.run.status",
payload: {
runId: claimed.id,
agentId: claimed.agentId,
status: claimed.status,
invocationSource: claimed.invocationSource,
triggerDetail: claimed.triggerDetail,
error: claimed.error ?? null,
errorCode: claimed.errorCode ?? null,
startedAt: claimed.startedAt
? new Date(claimed.startedAt).toISOString()
: null,
finishedAt: claimed.finishedAt
? new Date(claimed.finishedAt).toISOString()
: null,
},
});
publishRunLifecyclePluginEvent(claimed);
if (!nativeReviewContext) {
crashPoint("wake.claim.after_run_running_before_wakeup_claimed");
await setWakeupStatus(claimed.wakeupRequestId, "claimed", { claimedAt });
}
// Fix A (lazy locking): stamp executionRunId now that the run is actually running,
// not at queue time. Guard is idempotent — safe if called more than once.
const claimedContext = parseObject(claimed.contextSnapshot);
const claimedIssueId = readNonEmptyString(claimedContext.issueId);
const claimedWakeReason = readNonEmptyString(claimedContext.wakeReason);
if (
!nativeReviewContext && claimedIssueId &&
claimedWakeReason !== "source_scoped_recovery_action"
) {
const claimedAgent = await getAgent(claimed.agentId);
await db
.update(issues)
.set({
executionRunId: claimed.id,
executionAgentNameKey: normalizeAgentNameKey(claimedAgent?.name),
executionLockedAt: claimedAt,
updatedAt: claimedAt,
})
.where(
and(
eq(issues.id, claimedIssueId),
eq(issues.companyId, claimed.companyId),
// Mention/context runs can touch an issue, but only the current assignee
// owns the issue execution lock shown as the active run.
eq(issues.assigneeAgentId, claimed.agentId),
["native_safe_replacement", "native_provider_overloaded"].includes(claimed.scheduledRetryReason ?? "")
? or(
isNull(issues.checkoutRunId),
eq(issues.checkoutRunId, claimed.id),
)
: undefined,
or(
isNull(issues.executionRunId),
eq(issues.executionRunId, claimed.id),
),
),
);
}
return claimed;
}
// startNextQueuedRunForAgent checks admission suppression once, then claims
// runs (sets status "running"), then dispatches each to executeRun, which
// checks suppression again before it does any work. Suppression (task
// drain, worktree mode, a database restore) can start in the gap between
// those two checks. When executeRun's check catches that, the run is
// already claimed — release it back to "queued" so it does not keep a
// running execution lock that nothing will ever process. This runs inside
// the same promise startNextQueuedRunForAgent already tracks in
// activeRunExecutionPromises, so getTaskDrainStatus() keeps reporting the
// run as active until the release finishes.
//
// The run row, the wakeup request, and the issue execution lock all guard
// the same claim, so one transaction commits all three writes together. A
// partial write (for example the run flips to "queued" but the wakeup or
// issue update then fails) would let the execution promise clear from
// activeRunExecutionPromises while the wakeup stayed "claimed" or the
// issue stayed locked to a queued run — task-drain status would then read
// quiescent while the database still held part of the old claim.
async function releaseRunClaimedJustBeforeSuppression(runId: string) {
const now = new Date();
await db.transaction(async (tx) => {
const [candidate] = await tx.select({ companyId: heartbeatRuns.companyId, contextSnapshot: heartbeatRuns.contextSnapshot })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running"))).for("update");
if (!candidate) return;
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, runId,
issueId: readNonEmptyString(parseObject(candidate.contextSnapshot).issueId) ?? undefined });
const released = await tx
.update(heartbeatRuns)
.set({
status: "queued",
startedAt: null,
responsibleUserId: null,
updatedAt: now,
})
.where(
and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running")),
)
.returning()
.then((rows) => rows[0] ?? null);
if (!released) return;
if (released.wakeupRequestId) {
await tx
.update(agentWakeupRequests)
.set({ status: "queued", claimedAt: null, updatedAt: now })
.where(and(eq(agentWakeupRequests.id, released.wakeupRequestId), ne(agentWakeupRequests.status, "cancelled")));
}
const context = parseObject(released.contextSnapshot);
const issueId = readNonEmptyString(context.issueId);
if (issueId) {
await tx
.update(issues)
.set({
executionRunId: null,
executionAgentNameKey: null,
executionLockedAt: null,
updatedAt: now,
})
.where(
and(
eq(issues.id, issueId),
eq(issues.companyId, released.companyId),
eq(issues.executionRunId, released.id),
),
);
}
});
}
async function cancelQueuedRunForBlockedDependencies(
run: typeof heartbeatRuns.$inferSelect,
issueId: string,
unresolvedBlockerIssueIds: string[],
) {
const now = new Date();
const reason =
"Cancelled because issue dependencies are still blocked; Paperclip will wake the assignee when blockers resolve";
const cancelled = await cancelRunIfStillQueued(run.id, {
finishedAt: now,
error: reason,
errorCode: "issue_dependencies_blocked",
resultJson: {
...parseObject(run.resultJson),
stopReason: "issue_dependencies_blocked",
effectiveTimeoutSec: 0,
timeoutConfigured: false,
timeoutSource: "dependency_gate",
timeoutFired: false,
},
});
if (!cancelled) return null;
await setWakeupStatus(run.wakeupRequestId, "skipped", {
finishedAt: now,
error: reason,
});
await db.transaction(async tx => {
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId });
await notifyNativeStatusReassessment(tx, { companyId: run.companyId, issueId });
await tx
.update(issues)
.set({
executionRunId: null,
executionAgentNameKey: null,
executionLockedAt: null,
updatedAt: now,
})
.where(
and(
eq(issues.companyId, run.companyId),
eq(issues.id, issueId),
eq(issues.executionRunId, run.id),
),
);
});
await appendRunEvent(cancelled, {
eventType: "lifecycle",
stream: "system",
level: "warn",
message: reason,
payload: {
issueId,
unresolvedBlockerIssueIds,
},
});
return cancelled;
}
async function resumeDeferredExecutionReceipts(scope: DeferredReceiptRecoveryOptions = {}) {
if ((await getSchedulingSuppression()).suppressed) return;
await resumeExecutionWaitComments(scope);
const cutoff = await getWorktreeExecutionCutoff();
const pendingInterrupts = await db.select({ id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId })
.from(agentWakeupRequests).innerJoin(companies, eq(companies.id, agentWakeupRequests.companyId))
.where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
eq(agentWakeupRequests.status, "deferred_issue_execution"),
eq(companies.status, "active"),
sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is not null`,
lte(agentWakeupRequests.updatedAt, new Date(Date.now() - 30_000)),
cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined))
.orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
for (const wake of pendingInterrupts) {
await db.update(agentWakeupRequests).set({ updatedAt: new Date() }).where(and(
eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"),
));
await resumeQueuedCommentInterrupt(wake.companyId, wake.id).catch(err => {
scope.onError?.(err);
logger.warn({ err, queueId: wake.id }, "failed to resume interrupted comment queue");
});
}
// Work-intent outbox: a message refused by a pause waits for the pause to
// lift, not behind the run. Lifting the pause is the operator's explicit
// authorization to send what the pause refused, as Send is after a Stop:
// when the pause stopped the task's run, the release drain never promotes
// queued work behind that acknowledged stop, so the lifted pause issues the
// task's next explicit wake here. It reconsiders the exact saved receipt
// through normal admission (a refusal rewrites the receipt in place) and
// adopts it atomically, so it is delivered once. It runs before the
// stranded-queue pass, which touches each saved wake.
const pauseLifted = await db.select({ wake: agentWakeupRequests })
.from(agentWakeupRequests)
.innerJoin(issues, and(eq(issues.companyId, agentWakeupRequests.companyId),
sql`${issues.id}::text = ${agentWakeupRequests.payload}->>'issueId'`,
eq(issues.assigneeAgentId, agentWakeupRequests.agentId), isNull(issues.executionRunId),
notInArray(issues.status, ["done", "cancelled"])))
.innerJoin(companies, and(eq(companies.id, agentWakeupRequests.companyId), eq(companies.status, "active")))
.where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId),
// Only messages the pause path saved (never a chat-inbound receipt,
// which resumes through its own live authorization), and only pauses
// this query can see lifted. A refusal by any other gate rewrites the
// receipt's reason, so it never retries in a loop here.
// A message the reconciler saved after its wake was lost waits on no
// hold, so the same explicit wake sends it.
sql`${agentWakeupRequests.payload}->'executionWait'->>'trigger' in ('pause_lifted', 'reconciler_backstop')`,
sql`${agentWakeupRequests.payload}->'executionWait'->>'reason' in ('company.inactive', 'agent.not_invokable', 'wake_lost')`,
eq(agentWakeupRequests.requestedByActorType, "user"),
sql`coalesce(${agentWakeupRequests.idempotencyKey}, '') not like 'chat-inbound:%'`,
sql`exists (select 1 from agents pause_agent where pause_agent.id = ${agentWakeupRequests.agentId}
and pause_agent.status not in ('paused', 'terminated', 'pending_approval'))`,
sql`NOT EXISTS (SELECT 1 FROM heartbeat_runs live_issue_run
WHERE live_issue_run.company_id = ${issues.companyId}
AND (live_issue_run.context_snapshot->>'issueId' = ${issues.id}::text
OR live_issue_run.native_issue_id = ${issues.id})
AND live_issue_run.status IN ('queued', 'running', 'scheduled_retry'))`,
cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined))
.orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
const pauseLiftedIssues = new Set<string>();
for (const { wake } of pauseLifted) {
const issueKey = `${wake.companyId}:${String(wake.payload?.issueId)}`;
if (pauseLiftedIssues.has(issueKey)) continue;
pauseLiftedIssues.add(issueKey);
// Consumption is per comment. The wake adopts every saved message of
// the task for this agent, so each receipt drops the comments a
// completed run already received; a receipt left empty is consumed.
const issueKeyId = String(wake.payload?.issueId);
const saved = await db.select().from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, wake.companyId), eq(agentWakeupRequests.agentId, wake.agentId),
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueKeyId}`));
for (const receipt of saved) {
const commentIds = queuedCommentIdsFromWakePayload(receipt.payload);
const delivered = await deliveredQueuedCommentIds(db, { companyId: wake.companyId, issueId: issueKeyId,
agentId: wake.agentId, commentIds: commentIds.filter(isUuidLike) });
if (delivered.size === 0) continue;
const remaining = commentIds.filter(commentId => !delivered.has(commentId));
const now = new Date();
await db.update(agentWakeupRequests).set(remaining.length === 0
? { status: "cancelled", finishedAt: now, updatedAt: now, error: QUEUED_MESSAGES_DELIVERED_REASON }
: { payload: withQueuedCommentIdsInWakePayload(parseObject(receipt.payload), remaining), updatedAt: now })
.where(and(eq(agentWakeupRequests.id, receipt.id), eq(agentWakeupRequests.status, "deferred_issue_execution")));
}
const [current] = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.id, wake.id),
eq(agentWakeupRequests.status, "deferred_issue_execution")));
if (!current) continue;
wake.payload = current.payload;
const context = parseObject(parseObject(wake.payload)[DEFERRED_WAKE_CONTEXT_KEY]);
const { executionWait: _wait, [DEFERRED_WAKE_CONTEXT_KEY]: _context, ...payload } = parseObject(wake.payload);
await enqueueWakeup(wake.agentId, {
source: wake.source as WakeupOptions["source"], triggerDetail: wake.triggerDetail as WakeupOptions["triggerDetail"],
reason: wake.reason ?? undefined, payload, contextSnapshot: context,
requestedByActorType: wake.requestedByActorType as WakeupOptions["requestedByActorType"],
requestedByActorId: wake.requestedByActorId ?? undefined,
}, wake.id, wake.payload?.interactionOrigin).catch(err => {
if (err instanceof PersistedExecutionWaitRefusal) return;
scope.onError?.(err);
logger.warn({ err, requestId: wake.id }, "a message saved during a pause could not resume");
});
}
// A server restart or a message/cleanup race can leave a deferred wake
// after its owner has released the issue lock. Revisit it through the same
// release admission, so recovery holds and operator Stops still apply.
const orphanedChatWakes = await db.select({ id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId,
payload: agentWakeupRequests.payload }).from(agentWakeupRequests)
.where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId),
sql`${agentWakeupRequests.idempotencyKey} like 'chat-inbound:%'`,
sql`EXISTS (SELECT 1 FROM chat_actions chat_wake_source WHERE chat_wake_source.company_id = ${agentWakeupRequests.companyId}
AND chat_wake_source.id = ${agentWakeupRequests.id} AND chat_wake_source.kind = 'inbound_wakeup')`,
sql`NOT EXISTS (SELECT 1 FROM issues chat_wake_issue WHERE chat_wake_issue.company_id = ${agentWakeupRequests.companyId}
AND chat_wake_issue.id::text = ${agentWakeupRequests.payload}->>'issueId')`))
.limit(50);
for (const wake of orphanedChatWakes) {
await recordDeferredChatWakeDecline(db, { companyId: wake.companyId, issueId: String(wake.payload?.issueId),
requestId: wake.id }, { settle: "task_missing" }).then(recorded => {
if (recorded) logger.warn({ requestId: wake.id, reason: recorded.reason, outcome: recorded.outcome, attempts: recorded.attempts },
"saved chat source could not resume after recovery");
}).catch(err => {
scope.onError?.(err);
logger.warn({ err, requestId: wake.id }, "saved chat source could not record its resume outcome");
});
}
const strandedQueues = await db.select({ wake: agentWakeupRequests })
.from(agentWakeupRequests)
.innerJoin(issues, and(eq(issues.companyId, agentWakeupRequests.companyId),
sql`${issues.id}::text = ${agentWakeupRequests.payload}->>'issueId'`,
or(eq(issues.assigneeAgentId, agentWakeupRequests.agentId),
sql`${agentWakeupRequests.payload} #>> '{_paperclipWakeContext,wakeReason}' = 'issue_comment_mentioned'`)))
.innerJoin(companies, and(eq(companies.id, issues.companyId), eq(companies.status, "active")))
.where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
eq(agentWakeupRequests.status, "deferred_issue_execution"),
isNull(issues.executionRunId),
or(and(
sql`jsonb_typeof(${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}') = 'array'`,
sql`${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`,
), sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`,
dependencyResolvedWakePayloadSql, assignmentWakePayloadSql),
sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is null`,
scope.requestId ? undefined : sql`coalesce((${agentWakeupRequests.payload} #>> ${`{${DEFERRED_CHAT_WAKE_RETRY_KEY},nextAttemptAt}`})::timestamptz <= now(), true)`,
sql`NOT EXISTS (SELECT 1 FROM heartbeat_runs live_issue_run
WHERE live_issue_run.company_id = ${issues.companyId}
AND (live_issue_run.context_snapshot->>'issueId' = ${issues.id}::text
OR live_issue_run.native_issue_id = ${issues.id})
AND live_issue_run.status IN ('queued', 'running', 'scheduled_retry'))`,
cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined))
.orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
const releasedNativeSources = new Set<string>();
for (const { wake } of strandedQueues) {
const ownerWake = isDependencyResolvedWakePayload(wake.payload) || isAssignmentWakePayload(wake.payload);
if (!ownerWake && !queuedCommentIdsFromWakePayload(wake.payload).length &&
!await readQueuedInteractionResponse(db, wake.companyId, String(wake.payload?.issueId), wake.payload)) continue;
const [latest] = await db.select({
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
status: heartbeatRuns.status,
finishedAt: heartbeatRuns.finishedAt,
startedAt: heartbeatRuns.startedAt,
runtimeMode: heartbeatRuns.runtimeMode,
errorCode: heartbeatRuns.errorCode,
sourceVersion: sql<string>`${heartbeatRuns}.xmin::text`,
}).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, wake.companyId),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${String(wake.payload?.issueId)}`,
or(isNull(heartbeatRuns.nativeIssueId), sql`${heartbeatRuns.nativeIssueId}::text = ${String(wake.payload?.issueId)}`),
)).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
if (!wake.idempotencyKey?.startsWith("chat-inbound:")) await db.update(agentWakeupRequests).set({ updatedAt: new Date() }).where(and(
eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"),
));
if (!latest || !latest.finishedAt || !isHeartbeatRunTerminalStatus(latest.status)) continue;
if (latest.runtimeMode !== "legacy" && latest.runtimeMode !== "native") continue;
if (wake.idempotencyKey?.startsWith("chat-inbound:")) {
const chatScope = { companyId: wake.companyId, issueId: String(wake.payload?.issueId), requestId: wake.id };
await resumeDeferredChatWake(db, { ...chatScope, agentId: wake.agentId, recoverySweep: true })
.then(decline => decline ?? null, err => err instanceof PersistedExecutionWaitRefusal ? null : deferredChatWakeDeclineForError(err))
.then(async decline => {
if (!decline) {
if (wake.payload?.[DEFERRED_CHAT_WAKE_RETRY_KEY]) await clearDeferredChatWakeRetry(db, chatScope);
return;
}
const recorded = await recordDeferredChatWakeDecline(db, chatScope, decline);
if (!recorded) return;
logger.warn({ requestId: wake.id, reason: recorded.reason, outcome: recorded.outcome, attempts: recorded.attempts,
silent: "silent" in recorded ? recorded.silent : undefined }, "saved chat source could not resume after recovery");
if (recorded.outcome === "retry") scope.onDeferredDeadline?.(recorded.nextAttemptAt);
}).catch(err => {
scope.onError?.(err);
logger.warn({ err, requestId: wake.id }, "saved chat source could not record its resume outcome");
});
continue;
}
const releaseSource = `${latest.companyId}:${String(wake.payload?.issueId)}:${latest.id}:${latest.sourceVersion}`;
if (latest.runtimeMode === "native" && releasedNativeSources.has(releaseSource)) continue;
if (latest.runtimeMode === "legacy") {
const cancelledAdmission = latest.status === "cancelled" && !latest.startedAt &&
latest.errorCode === "execution_reconciliation_required";
if ((latest.status !== "cancelled" || cancelledAdmission) && await getExecutionBlocker(db, wake.companyId, String(wake.payload?.issueId))) {
await resumeSavedLegacyComments(wake.companyId, wake.id).catch(err => {
scope.onError?.(err);
logger.warn({ err, queueId: wake.id }, "failed to deliver saved legacy comment after recovery stopped");
});
}
}
await releaseIssueExecutionAndPromote(latest, { suppressImmediateRecovery: true, resumeStrandedQueue: true }).then(() => {
if (latest.runtimeMode === "native") releasedNativeSources.add(releaseSource);
}).catch(err => {
scope.onError?.(err);
logger.warn({ err, queueId: wake.id }, "failed to promote stranded comments");
});
}
// The cancellation marker is durable intent. Retry while its exact queue
// is still deferred, including after a failed cleanup promotion or restart.
// Normal admission still checks process ownership, leases, pauses, and scope.
const interruptedQueues = await db
.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId })
.from(agentWakeupRequests)
.innerJoin(heartbeatRuns, and(
sql`${heartbeatRuns.resultJson}->>'queuedCommentInterruptQueueId' = ${agentWakeupRequests.id}::text`,
eq(heartbeatRuns.companyId, agentWakeupRequests.companyId),
eq(heartbeatRuns.agentId, agentWakeupRequests.agentId),
))
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
.where(and(
scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
eq(agentWakeupRequests.status, "deferred_issue_execution"),
eq(heartbeatRuns.status, "cancelled"),
eq(heartbeatRuns.runtimeMode, "legacy"),
eq(companies.status, "active"),
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
));
for (const run of interruptedQueues) {
await releaseIssueExecutionAndPromote(run, { suppressImmediateRecovery: true }).catch((err) => {
scope.onError?.(err);
logger.error({ err, runId: run.id }, "failed to retry interrupted comment queue");
});
}
}
async function resumeQueuedRuns() {
if (isRunAdmissionClosed()) return;
await resumeDeferredExecutionReceipts();
await resumeQueuedRunsAfterDrain();
}
// Release admission promptly without first scanning unrelated recovery work.
// Every run still passes the ordinary per-agent claim and dispatch gates.
async function resumeQueuedRunsAfterDrain(scope?: { companyId: string }) {
const refreshing = refreshRunAdmission();
if (refreshing) await refreshing;
if (isRunAdmissionClosed() || (await getSchedulingSuppression()).suppressed) return;
const cutoff = await getWorktreeExecutionCutoff();
const queuedRuns = await db
.select({ agentId: heartbeatRuns.agentId })
.from(heartbeatRuns)
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
.where(
and(
eq(heartbeatRuns.status, "queued"),
scope ? eq(heartbeatRuns.companyId, scope.companyId) : undefined,
eq(companies.status, "active"),
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
),
);
const agentIds = [...new Set(queuedRuns.map((r) => r.agentId))];
for (const agentId of agentIds) {
await startNextQueuedRunForAgent(agentId, true, true);
}
}
// Startup fast path: start queued turns that never executed, on agents with
// no run left `running` by the previous process and no queued retry or
// continuation. Those agents' runs depend on restart recovery and wait for
// the full queued-run pass. Every run still passes the ordinary per-agent
// claim and dispatch gates, exactly as a new wakeup during startup would.
async function resumeFreshQueuedRunsAtStartup() {
if (isRunAdmissionClosed() || (await getSchedulingSuppression()).suppressed) return { agents: 0 };
const cutoff = await getWorktreeExecutionCutoff();
const rows = await db
.selectDistinct({ agentId: heartbeatRuns.agentId })
.from(heartbeatRuns)
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
.where(
and(
eq(heartbeatRuns.status, "queued"),
eq(companies.status, "active"),
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
sql`not exists (
select 1 from heartbeat_runs prior
where prior.agent_id = ${heartbeatRuns.agentId}
and (prior.status = 'running' or (prior.status = 'queued' and (
prior.started_at is not null or prior.retry_of_run_id is not null
or coalesce(prior.process_loss_retry_count, 0) > 0
or coalesce(prior.continuation_attempt, 0) > 0)))
)`,
),
);
for (const { agentId } of rows) {
await startNextQueuedRunForAgent(agentId, true);
}
return { agents: rows.length };
}
// A 403 from claimQueuedRun comes from the run's own persisted identity
// (an unverifiable interrupt receipt, a manual wake with no user). Those rows
// do not change, so the claim fails the same way on every pass and restart.
function isPermanentClaimRejection(err: unknown): err is HttpError {
return err instanceof HttpError && err.status === 403;
}
// Other 4xx rejections can clear later (a responsible user gets assigned, a
// conflicting claim finishes). Keep the run queued, but do not let it stop
// the rest of the queue or startup recovery.
function isDeferrableClaimRejection(err: unknown): err is HttpError {
return err instanceof HttpError && err.status >= 400 && err.status < 500;
}
// Settle runs that can never be claimed. Letting the error escape stalls the
// agent's queue and, during startup recovery, stops the server from booting.
async function cancelRejectedQueuedRuns(
rejected: Array<{ run: typeof heartbeatRuns.$inferSelect; err: HttpError }>,
) {
for (const { run, err } of rejected) {
logger.warn(
{ err, runId: run.id, agentId: run.agentId, companyId: run.companyId },
"cancelling queued heartbeat run whose claim was rejected",
);
try {
await cancelRunInternal(
run.id,
`Cancelled because the queued run cannot be claimed: ${err.message}`,
{ errorCode: "queued_run_claim_rejected" },
);
} catch (cancelErr) {
logger.error(
{ err: cancelErr, runId: run.id },
"failed to cancel queued heartbeat run whose claim was rejected; it stays queued for the next recovery pass",
);
}
}
}
async function attachQueuedSetupRequest(agent: typeof agents.$inferSelect) {
if (isAgentAwaitingSetup(agent)) await connectionIntentService(db)
.requestForQueuedAgentSetup(agent.companyId, agent.id)
.catch(() => logger.warn({ agentId: agent.id }, "Could not attach queued agent setup connection request"));
}
const declinedAgedClaimAt = new Map<string, number>();
function noteDeclinedAgedClaim(run: QueuedRunCandidate, reason: string) {
const now = Date.now();
if (!hasQueuedRunAged(run, new Date(now))) return;
declinedAgedClaimAt.set(run.id, now);
logger.warn({ runId: run.id, agentId: run.agentId, companyId: run.companyId, reason,
queuedForMs: now - run.createdAt.getTime() },
"long-queued heartbeat run was offered a free slot but could not be claimed; it stays queued and the next oldest run takes its turn");
}
const claimBackoffs = new Map<string, { agentId: string; issueId: string | null; attempts: number; until: number;
taskStamp: string | null }>();
function claimBackedOff(runId: string, now: Date, taskStamp: string | null) {
const backoff = claimBackoffs.get(runId);
return Boolean(backoff && backoff.until > now.getTime() && backoff.taskStamp === taskStamp);
}
function backOffClaim(run: QueuedRunCandidate, personRun: boolean, taskStamp: string | null) {
if (!hasQueuedRunAged(run, new Date())) return;
const attempts = (claimBackoffs.get(run.id)?.attempts ?? 0) + 1;
const delayMs = personRun ? QUEUED_PERSON_RUN_CLAIM_BACKOFF_MS
: Math.min(QUEUED_RUN_CLAIM_BACKOFF_BASE_MS * 2 ** (attempts - 1), QUEUED_RUN_CLAIM_BACKOFF_MAX_MS);
claimBackoffs.set(run.id, { agentId: run.agentId, issueId: readNonEmptyString(parseObject(run.contextSnapshot).issueId),
attempts, until: Date.now() + delayMs, taskStamp });
}
function releaseClaimBackoffsForTask(run: Pick<HeartbeatRun, "issueId" | "contextSnapshot">) {
const issueId = run.issueId ?? readNonEmptyString(parseObject(run.contextSnapshot).issueId);
if (!issueId) return;
for (const [runId, backoff] of claimBackoffs) {
if (backoff.issueId === issueId) claimBackoffs.delete(runId);
}
}
function queuedRunTaskStamp(run: QueuedRunCandidate,
issueById: Map<string, { status: string; updatedAt: Date; executionRunId: string | null }>) {
const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
const issue = issueId ? issueById.get(issueId) : null;
return issue ? `${issue.status}:${issue.updatedAt.getTime()}:${issue.executionRunId ?? ""}` : null;
}
function forgetSettledClaimBackoffs(agentId: string, queuedRuns: QueuedRunCandidate[]) {
const queued = new Set(queuedRuns.map(run => run.id));
for (const [runId, backoff] of claimBackoffs) {
if (backoff.agentId === agentId && !queued.has(runId)) claimBackoffs.delete(runId);
}
}
function pruneDeclinedAgedClaims(now: Date) {
for (const [id, at] of declinedAgedClaimAt) {
if (now.getTime() - at >= QUEUED_RUN_AGING_MS) declinedAgedClaimAt.delete(id);
}
}
async function queuedRunDispatcherForAgent(
agent: typeof agents.$inferSelect,
queuedRuns: QueuedRunCandidate[],
dependencyReadiness: Map<string, { isDependencyReady: boolean }>,
issueById: Map<string, { status: string; priority: string | null; updatedAt: Date; executionRunId: string | null;
createdByUserId: string | null }>,
) {
const now = new Date();
pruneDeclinedAgedClaims(now);
const runById = new Map(queuedRuns.map(run => [run.id, run]));
const { facts, dispatchInput, personOriginChecked } = await queuedRunDispatchFacts(db, {
agent, queuedRuns, dependencyReadiness, issueById, now,
shareDeferred: runId => declinedAgedClaimAt.has(runId) || claimBackedOff(runId, now, queuedRunTaskStamp(runById.get(runId)!, issueById)) });
const dispatcher = createQueuedRunDispatcher(facts, dispatchInput);
const byRun = new Map(facts.map(entry => [entry.run, entry]));
return {
next: () => dispatcher.next()?.run ?? null,
claimed: (run: QueuedRunCandidate) => { const entry = byRun.get(run); if (entry) dispatcher.claimed(entry); },
personRun: (run: QueuedRunCandidate) => {
const entry = byRun.get(run);
return Boolean(entry && queuedRunDispatchClass(entry, now) === "person" && isPersonRequestedRun(entry));
},
personOrigin: (run: QueuedRunCandidate) => personOriginChecked ? byRun.get(run)?.requestedByUser ?? false : undefined,
};
}
function startNextQueuedRunForAgent(agentId: string, attachSetupRequest = false, retryRefused = false) {
return requestQueuedDispatch(agentId, { attachSetupRequest, retryRefused, continuesChain: false, triedRunIds: new Set() });
}
function requestQueuedDispatch(agentId: string, request: QueuedDispatchRequest) {
return dispatchForAgent<QueuedDispatchRequest, Array<typeof heartbeatRuns.$inferSelect>>(agentId, {
request,
idle: [],
merge: (current, next) => ({
attachSetupRequest: current.attachSetupRequest || next.attachSetupRequest,
retryRefused: current.retryRefused || next.retryRefused,
continuesChain: current.continuesChain && next.continuesChain,
triedRunIds: current.continuesChain && next.continuesChain
? new Set([...current.triedRunIds, ...next.triedRunIds]) : new Set<string>(),
}),
run: (request, signal) => runQueuedDispatchPass(agentId, request, signal),
track: (pass) => {
const tracked = pass.then(() => undefined);
activeRunExecutionPromises.add(tracked);
void tracked.finally(() => activeRunExecutionPromises.delete(tracked));
},
});
}
async function runQueuedDispatchPass(agentId: string, { attachSetupRequest, retryRefused, triedRunIds }: QueuedDispatchRequest,
signal: AbortSignal) {
// Cancelled after the pass ends: cancelRunInternal promotes the agent's
// next queued run, which asks for another pass of this agent.
const rejectedClaims: Array<{ run: typeof heartbeatRuns.$inferSelect; err: HttpError }> = [];
const deferredPostCommitEffects: WakeQueuePostCommitEffect[] = [];
const deferredCancellations: Array<() => Promise<void>> = [];
let cancellationReason: string | undefined;
const settle = async () => {
// Dispatch promoted inputs after the pass has ended.
try {
await applyWakeQueuePostCommitEffects(deferredPostCommitEffects);
} finally {
if (cancellationReason) await cancelActiveForAgentInternal(agentId, cancellationReason);
await cancelRejectedQueuedRuns(rejectedClaims);
for (const cancel of deferredCancellations) {
await cancel().catch((err) => logger.warn({ err, agentId },
"queued heartbeat run refused by a pause hold could not be cancelled; the next claim retries it"));
}
}
};
try {
return { value: await claimQueuedRunsForAgent(), settle };
} catch (err) {
await settle();
throw err;
}
async function claimQueuedRunsForAgent(): Promise<Array<typeof heartbeatRuns.$inferSelect>> {
// Null unless this server has not read the run-start hand-over recently.
const refreshing = refreshRunAdmission();
if (refreshing) await refreshing;
if (isRunAdmissionClosed() || (await getSchedulingSuppression()).suppressed) return [];
const cutoff = await getWorktreeExecutionCutoff();
const agent = await getAgent(agentId);
if (!agent) return [];
const invokability = await getAgentInvokability(agent);
if (!invokability.invokable) {
if (attachSetupRequest) await attachQueuedSetupRequest(agent);
if (shouldCancelRunsForNonInvokableAgent(invokability)) {
cancellationReason = `Cancelled because the agent is not invokable: ${invokability.reason}`;
}
return [];
}
const policy = parseHeartbeatPolicy(agent);
const runningCount = await countRunningRunsForAgent(agentId);
const availableSlots = Math.max(0, policy.maxConcurrentRuns - runningCount);
if (availableSlots <= 0) return [];
const queuedRuns = await listQueuedRunCandidates(agentId, cutoff);
forgetSettledClaimBackoffs(agentId, queuedRuns);
if (queuedRuns.length === 0) return [];
if (queuedRuns.every(run => !run.startedAt) && !await canStartNewExternalBotWork(agent)) return [];
const dependencyReadiness = await listQueuedRunDependencyReadiness(agent.companyId, queuedRuns);
const queuedIssueIds = [...new Set(queuedRuns.flatMap((run) => {
const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
return issueId ? [issueId] : [];
}))];
const issueRows = queuedIssueIds.length > 0
? await db.select({ id: issues.id, status: issues.status, priority: issues.priority, updatedAt: issues.updatedAt,
executionRunId: issues.executionRunId, createdByUserId: issues.createdByUserId }).from(issues)
.where(and(eq(issues.companyId, agent.companyId), inArray(issues.id, queuedIssueIds)))
: [];
const issueById = new Map(issueRows.map((row) => [row.id, row]));
const companyAgents = await listCompanyAgentOrgRows(agent.companyId);
const dispatcher = await queuedRunDispatcherForAgent(agent, queuedRuns, dependencyReadiness, issueById);
const claimedRuns: Array<typeof heartbeatRuns.$inferSelect> = [];
const slots = { full: false };
let claimAttempts = 0;
const triedThisPass: string[] = [];
for (let candidate = dispatcher.next(); candidate; candidate = dispatcher.next()) {
if (claimedRuns.length >= availableSlots || slots.full || isRunAdmissionClosed() || signal.aborted) break;
const taskStamp = queuedRunTaskStamp(candidate, issueById);
if (triedRunIds.has(candidate.id) || (!retryRefused && claimBackedOff(candidate.id, new Date(), taskStamp))) continue;
if (claimAttempts >= QUEUED_RUN_CLAIM_ATTEMPTS_PER_PASS) {
void requestQueuedDispatch(agentId, { attachSetupRequest: false, retryRefused,
continuesChain: true, triedRunIds: new Set([...triedRunIds, ...triedThisPass]) });
break;
}
const start = await reserveAutomaticRunStart(db, candidate, { personOrigin: dispatcher.personOrigin(candidate) });
if (start.held) continue;
claimAttempts += 1;
triedThisPass.push(candidate.id);
const [queuedRun] = await db.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, candidate.id), eq(heartbeatRuns.companyId, candidate.companyId), eq(heartbeatRuns.status, "queued")));
if (!queuedRun) {
await start.release();
continue;
}
let claimed: typeof heartbeatRuns.$inferSelect | null;
try {
claimed = await claimQueuedRun(queuedRun, companyAgents, deferredPostCommitEffects, deferredCancellations, slots);
} catch (err) {
await start.release();
if (isPermanentClaimRejection(err)) {
rejectedClaims.push({ run: queuedRun, err });
continue;
}
if (!isDeferrableClaimRejection(err)) throw err;
logger.warn(
{ err, runId: queuedRun.id, agentId: queuedRun.agentId, companyId: queuedRun.companyId },
"queued heartbeat run claim was rejected; leaving it queued for the next recovery pass",
);
noteDeclinedAgedClaim(candidate, err.message);
backOffClaim(candidate, dispatcher.personRun(candidate), taskStamp);
continue;
}
if (!claimed) {
await start.release();
if (slots.full) break;
noteDeclinedAgedClaim(candidate, "claim_declined");
backOffClaim(candidate, dispatcher.personRun(candidate), taskStamp);
continue;
}
claimedRuns.push(claimed);
dispatcher.claimed(candidate);
declinedAgedClaimAt.delete(candidate.id);
claimBackoffs.delete(candidate.id);
// Start each run as soon as it is claimed. The claim committed a legacy
// controller lease that only the run's own execution renews; waiting
// for the rest of a slow batch to be claimed let that lease lapse.
startClaimedRun(claimed);
}
return claimedRuns;
}
function startClaimedRun(claimedRun: typeof heartbeatRuns.$inferSelect) {
const execution = outsideAgentDispatch(() => executeRun(claimedRun.id)).catch((err) => {
logger.error(
{ err, runId: claimedRun.id },
"queued heartbeat execution failed",
);
});
// Register the in-flight execution so drainActiveRunExecutions() can await
// it. executeRun resolves only after its finally block finishes flushing
// run rows/events, so awaiting this promise guarantees the run's writes
// have landed before a caller (e.g. a test's afterEach) mutates the DB.
activeRunExecutionPromises.add(execution);
void execution.finally(() => {
// drainActiveRunExecutions loops on activeRunExecutionPromises.size,
// so an entry that never clears here would hang it forever.
activeRunExecutionPromises.delete(execution);
});
}
}
async function listQueuedRunCandidates(agentId: string, cutoff: Date | null): Promise<QueuedRunCandidate[]> {
const contextText = (key: string) => sql<string | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}->${key}) = 'string'
then ${heartbeatRuns.contextSnapshot}->>${key} end`;
const rows = await db.select({
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
status: heartbeatRuns.status,
invocationSource: heartbeatRuns.invocationSource,
createdAt: heartbeatRuns.createdAt,
startedAt: heartbeatRuns.startedAt,
wakeupRequestId: heartbeatRuns.wakeupRequestId,
retryOfRunId: heartbeatRuns.retryOfRunId,
scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
scheduledRetryAttempt: heartbeatRuns.scheduledRetryAttempt,
scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
issueId: contextText("issueId"),
wakeReason: contextText("wakeReason"),
previousRunId: contextText("previousRunId"),
}).from(heartbeatRuns).where(and(
eq(heartbeatRuns.agentId, agentId),
eq(heartbeatRuns.status, "queued"),
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
)).orderBy(asc(heartbeatRuns.createdAt));
return rows.map(({ issueId, wakeReason, previousRunId, ...run }) => ({
...run,
contextSnapshot: Object.fromEntries(Object.entries({ issueId, wakeReason, previousRunId }).filter(([, value]) => value !== null)),
}));
}
// Public wakeup entry point. Callers dispatch it fire-and-forget, so register
// the promise in activeWakeupPromises before it starts its asynchronous
// prologue. drainActiveRunExecutions can then await a wake that is still before
// run registration. Internal callers reference enqueueWakeup directly and
// already await it, so they do not need this registration.
function trackWakeup(
agentId: string,
opts: WakeupOptions = {},
): ReturnType<typeof enqueueWakeup> {
// Until its wake and run commit, this admission exists only in this process.
const unsaved = {};
unsavedWakeAdmissions.add(unsaved);
const promise = enqueueWakeup(agentId, opts, undefined, undefined, { saved: () => { unsavedWakeAdmissions.delete(unsaved); } });
activeWakeupPromises.add(promise);
void promise
.catch(() => {})
.finally(() => {
unsavedWakeAdmissions.delete(unsaved);
activeWakeupPromises.delete(promise);
});
return promise;
}
async function enqueueWakeup(agentId: string, opts: WakeupOptions = {}, executionWaitRequestId?: string, savedInteractionOrigin?: unknown,
admission?: { saved(): void }) {
const source = opts.source ?? "on_demand";
const triggerDetail = opts.triggerDetail ?? null;
const contextSnapshot: Record<string, unknown> = {
...(opts.contextSnapshot ?? {}),
};
const reason = opts.reason ?? null;
let payload = opts.payload ? { ...opts.payload } : null;
// Only the board queue route can record interruption authority on an
// existing receipt. Never accept this internal marker from a wake caller.
if (payload) {
delete payload.queuedCommentInterrupt;
delete payload.manualUserWake;
delete payload.taskRetry;
}
if (opts.manualUserWake) {
if (opts.requestedByActorType !== "user" || !opts.requestedByActorId || opts.failedRunId) {
throw new HttpError(403, "Manual wake requires an authenticated user");
}
payload = { ...payload, manualUserWake: true };
}
const executionReconciliationWake =
contextSnapshot.source === "execution.reconciled" ||
opts.idempotencyKey?.startsWith("execution-reconciliation:") === true;
const {
contextSnapshot: enrichedContextSnapshot,
issueIdFromPayload,
taskKey,
wakeCommentId,
} = enrichWakeContextSnapshot({
contextSnapshot,
reason,
source,
triggerDetail,
payload,
});
const isManualContextCompactionWake = () => Boolean(
reason === CONTEXT_COMPACTION_WAKE_REASON && source === "on_demand" && triggerDetail === "manual"
&& opts.manualUserWake === true && opts.requestedByActorType === "user"
&& Boolean(opts.requestedByActorId) && opts.allowRunCoalescing === false
&& payload?.intent === CONTEXT_COMPACTION_WAKE_REASON
&& readNonEmptyString(payload.requestId) !== null
&& readContextCompactionRequest(enrichedContextSnapshot)?.requestId === payload.requestId,
);
const hasIndependentConversationWakeCause = () => Boolean(
wakeCommentId || opts.taskRetryRequestId ||
hasInteractionContinuationWakeContext(enrichedContextSnapshot) ||
reason === CHAT_COMPLETION_WAKE_REASON || isManualContextCompactionWake(),
);
// Keep each request's own server-derived origin, including coalesced wakes.
// A run's merged context cannot establish which caller authored one receipt.
// Overwrite caller-supplied nested context rather than trusting it.
payload = withInteractionDeliveryOrigin(
{ ...payload, [DEFERRED_WAKE_CONTEXT_KEY]: { ...enrichedContextSnapshot } },
opts.requestedByActorType, opts.requestedByActorId, savedInteractionOrigin, opts.requestedByActorCompanyId,
);
let issueId =
readNonEmptyString(enrichedContextSnapshot.issueId) ?? issueIdFromPayload;
if (executionReconciliationWake && !issueId) return null;
const durableRequest = opts.durableChatRequest;
let taskRetryActorId: string | null = null;
delete enrichedContextSnapshot.boardMessageRetry;
if (opts.taskRetryRequestId) {
const [receipt] = await db.select().from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.id, opts.taskRetryRequestId), eq(agentWakeupRequests.agentId, agentId),
eq(agentWakeupRequests.status, "deferred_issue_execution"),
eq(agentWakeupRequests.requestedByActorType, "user"),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId ?? ""}`,
));
const retry = readTaskRetryIntent(receipt?.payload);
if (!receipt || retry?.generationId !== receipt.id || retry.canonicalIntentId || retry.queuedChatRequestId ||
retry.actorId !== receipt.requestedByActorId) return null;
const chatRetry = retry.chatActionId && durableRequest?.id === retry.chatActionId &&
durableRequest.failedRunRetry?.failedRunId === retry.sourceRunId;
if (retry.chatActionId ? !chatRetry :
opts.requestedByActorType !== "user" || retry.actorId !== opts.requestedByActorId) return null;
taskRetryActorId = retry.actorId;
enrichedContextSnapshot.forceFreshSession = true;
enrichedContextSnapshot.previousRunId = retry.sourceRunId;
enrichedContextSnapshot.taskRetryRequestId = receipt.id;
if (retry.boardMessageOrigin) enrichedContextSnapshot.boardMessageRetry = true;
}
let agent = await getAgent(agentId);
if (!agent) throw notFound("Agent not found");
if (issueId && await hasAutomaticInteractionDeliveryHold(db, {
companyId: agent.companyId, issueId, agentId, payload, context: enrichedContextSnapshot,
actorType: opts.requestedByActorType, actorId: opts.requestedByActorId,
})) return null;
if (issueId) {
const [pending] = await db.select().from(caseHandoffs).where(and(eq(caseHandoffs.companyId, agent.companyId), eq(caseHandoffs.issueId, issueId),
inArray(caseHandoffs.state, ["awaiting_release", "pending_dispatch"]))).limit(1);
if (pending && opts.durableCaseHandoffRequest?.handoffId !== pending.id) return null;
}
// Mentions only annotate comments. Ignore legacy callers before creating
// a run or deferred request; assignment and review have their own wakes.
if (reason === "issue_comment_mentioned" || enrichedContextSnapshot.wakeReason === "issue_comment_mentioned") return null;
if (issueId) {
const conversation = await getIssueExecutionContext(agent.companyId, issueId);
if (taskRetryActorId && conversation) await assertTaskRetryConversationAllowed(db, conversation, taskRetryActorId);
if (reason === "issue_children_completed" && conversation?.originKind === "onboarding_first_task") enrichedContextSnapshot.onboardingCompletion = true;
if (isConversation(conversation)) {
if (opts.manualUserWake && conversation!.conversationUserId !== opts.requestedByActorId) {
throw new HttpError(403, "Only the conversation owner can start a chat run");
}
if (isConversationExecutionWake(conversation, reason ?? readNonEmptyString(enrichedContextSnapshot.wakeReason))) return null;
if (agent.id !== conversation!.conversationAgentId) return null;
if (!(await instanceSettings.getExperimental()).enableAgentChat) return null;
if (isWaitingConversation(conversation) && !hasIndependentConversationWakeCause()) return null;
}
}
if (agent.adapterType === "paperclip_runner") {
const oldConfig = parseObject(agent.adapterConfig);
const nextConfig = normalizeLegacyRunnerProvider(oldConfig);
if (nextConfig !== oldConfig) {
await agentService(db).update(
agent.id,
{ adapterConfig: nextConfig },
{
recordRevision: {
source: "normalize_runner_provider",
createdByAgentId: null,
createdByUserId: null,
},
},
);
await logActivity(db, {
companyId: agent.companyId,
actorType: "system",
actorId: "heartbeat",
action: "agent.updated",
entityType: "agent",
entityId: agent.id,
details: { provider: "codex", reason: "native_codex_provider" },
});
agent = (await getAgent(agentId))!;
}
}
if (opts.failedRunId) {
const failed = await getRun(opts.failedRunId, { includeExecutionEvidence: true });
if (opts.requestedByActorType !== "user" || !opts.requestedByActorId ||
reason !== "retry_failed_run" || source !== "on_demand" || triggerDetail !== "manual" ||
!failed || failed.companyId !== agent.companyId || failed.agentId !== agentId ||
!(await canRetryStoppedRun(db, failed)) ||
(failed.nativeIssueId ?? readNonEmptyString(failed.contextSnapshot?.issueId)) !== issueId) {
throw conflict("The selected failed run cannot be retried for this task.");
}
if (!activeRunExecutions.has(failed.id) && !adapterExecutionControls.has(failed.id)) {
await sweepPendingCleanupLeases({ explicitRetry: {
companyId: failed.companyId, runId: failed.id, actorId: opts.requestedByActorId,
} });
}
if (isConversationAdapter(agent.adapterType) || agent.adapterType === "paperclip_runner") {
enrichedContextSnapshot.previousRunId = failed.id;
enrichedContextSnapshot.forceFreshSession = true;
}
}
const caseRequest = opts.durableCaseHandoffRequest;
const waitForCaseAdmission = (reason: string, message: string): never => {
// The case ledger is the outbox. Roll back this admission attempt rather
// than consume its one receipt ID in a generic skipped/deferred queue.
throw conflict(message, { code: "case_handoff_admission_wait", reason });
};
let caseReceipt: { id: string; requestedAt: Date } | undefined;
if (caseRequest) {
if (opts.taskRetryRequestId || opts.durableChatRequest || opts.durableDotRequest || opts.durableMuseRequest || source !== "assignment" || reason !== "case_handoff")
throw conflict("Invalid case handoff wake authority");
const [handoff] = await db.select().from(caseHandoffs).where(and(eq(caseHandoffs.id, caseRequest.handoffId),
eq(caseHandoffs.companyId, agent.companyId), eq(caseHandoffs.issueId, caseRequest.issueId),
eq(caseHandoffs.toAgentId, agentId), eq(caseHandoffs.wakeupRequestId, caseRequest.wakeupRequestId),
inArray(caseHandoffs.state, ["pending_dispatch", "delivered"]))).limit(1);
if (!handoff || issueId !== handoff.issueId || caseRequest.expectedIssueStatusVersion !== handoff.appliedIssueStatusVersion)
throw conflict("Case handoff no longer matches its task assignment");
caseReceipt = { id: handoff.wakeupRequestId, requestedAt: handoff.createdAt };
opts = { ...opts, allowRunCoalescing: false, idempotencyKey: `case-handoff:${handoff.id}`,
issueStateGuard: { assigneeAgentId: agentId, statuses: ["todo", "in_progress"], statusVersion: caseRequest.expectedIssueStatusVersion } };
}
if (durableRequest) {
assertDurableChatWakeupRequest(durableRequest, {
agentId,
companyId: agent.companyId,
issueId,
commentId: wakeCommentId ?? null,
requestedByActorType: opts.requestedByActorType,
requestedByActorId: opts.requestedByActorId,
});
opts = { ...opts, idempotencyKey: durableRequest.idempotencyKey };
}
if (
Object.hasOwn(enrichedContextSnapshot, "chatFailedRunRetry") &&
!durableRequest?.failedRunRetry
) {
throw new FailedChatRunRetryAuthorizationError();
}
if (durableRequest?.failedRunRetry) {
opts = { ...opts, allowRunCoalescing: false };
}
const dotRequest = opts.durableDotRequest;
const museRequest = opts.durableMuseRequest;
if (dotRequest && (durableRequest || museRequest || dotRequest.agentId !== agentId ||
dotRequest.companyId !== agent.companyId || dotRequest.issueId !== issueId ||
dotRequest.requestId !== payload?.dotRequestId || source !== "assignment" ||
opts.requestedByActorType !== "agent" || opts.requestedByActorId !== agentId ||
agent.adapterType !== "paperclip_runner" || !["openai_dot", "grok_bot"].includes(String(parseObject(agent.adapterConfig).provider)) ||
opts.idempotencyKey !== dotRequest.idempotencyKey)) {
throw conflict("Dot work request does not match its admission authority.");
}
if (museRequest && (durableRequest || dotRequest || museRequest.agentId !== agentId ||
museRequest.companyId !== agent.companyId || museRequest.issueId !== issueId ||
museRequest.requestId !== payload?.museRequestId || source !== "assignment" ||
opts.requestedByActorType !== "agent" || opts.requestedByActorId !== agentId ||
agent.adapterType !== "paperclip_runner" || parseObject(agent.adapterConfig).provider !== "muse" ||
parseObject(agent.adapterConfig).museBindingId !== museRequest.bindingId ||
!Number.isSafeInteger(museRequest.bindingGeneration) || museRequest.bindingGeneration < 1 ||
payload?.museBindingId !== museRequest.bindingId || payload?.museBindingGeneration !== museRequest.bindingGeneration ||
payload?.museRequestDigest !== museRequest.requestDigest || !/^sha256:[a-f0-9]{64}$/.test(museRequest.requestDigest) ||
opts.idempotencyKey !== museRequest.idempotencyKey)) {
throw conflict("Muse work request does not match its admission authority.");
}
const receiptRequest = durableRequest ?? dotRequest ?? museRequest ?? caseReceipt;
const durableReceiptFields = receiptRequest
? { id: receiptRequest.id, requestedAt: receiptRequest.requestedAt }
: {};
const existingDurableReceipt = async (queryDb: Db) => {
if (!receiptRequest) return null;
const receipt = await queryDb
.select()
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.id, receiptRequest.id))
.limit(1)
.then((rows) => rows[0] ?? null);
if (receipt && durableRequest) assertDurableChatWakeupReceipt(durableRequest, receipt);
if (receipt && caseRequest && (receipt.companyId !== agent.companyId || receipt.agentId !== agentId ||
receipt.payload?.issueId !== caseRequest.issueId || receipt.idempotencyKey !== `case-handoff:${caseRequest.handoffId}`))
throw conflict("Case handoff wake receipt identity changed");
if (receipt && dotRequest && (receipt.companyId !== dotRequest.companyId ||
receipt.agentId !== dotRequest.agentId || receipt.source !== "assignment" ||
receipt.requestedByActorType !== "agent" || receipt.requestedByActorId !== dotRequest.agentId ||
receipt.idempotencyKey !== dotRequest.idempotencyKey || receipt.payload?.issueId !== dotRequest.issueId ||
receipt.payload?.dotRequestId !== dotRequest.requestId)) {
throw conflict("requestId was reused for another task.");
}
if (receipt && museRequest && (receipt.companyId !== museRequest.companyId ||
receipt.agentId !== museRequest.agentId || receipt.source !== "assignment" ||
receipt.requestedByActorType !== "agent" || receipt.requestedByActorId !== museRequest.agentId ||
receipt.idempotencyKey !== museRequest.idempotencyKey || receipt.payload?.issueId !== museRequest.issueId ||
receipt.payload?.museRequestId !== museRequest.requestId || receipt.payload?.museBindingId !== museRequest.bindingId ||
receipt.payload?.museBindingGeneration !== museRequest.bindingGeneration || receipt.payload?.museRequestDigest !== museRequest.requestDigest)) {
throw conflict("Request ID reused with changed input.");
}
if (receipt && durableRequest && receipt.status === "deferred_issue_execution" && !receipt.runId &&
((durableRequest.failedRunRetry && opts.taskRetryRequestId &&
receipt.payload?.taskRetryRequestId === opts.taskRetryRequestId) ||
(durableRequest.resumeDeferred && !durableRequest.failedRunRetry &&
receipt.payload?.issueId === durableRequest.issueId &&
queuedCommentIdsFromWakePayload(receipt.payload).includes(durableRequest.commentId)))) {
// The live inbound owner or a bound Retry may reconsider the same saved receipt.
// The issue lock, live chat authority and execution holds below still
// decide whether it can dispatch; this never creates another wake.
executionWaitRequestId = receipt.id;
return null;
}
return receipt;
};
const priorReceipt = await existingDurableReceipt(db);
if (priorReceipt) {
// Replaying an admission receipt is not fresh authority to dispatch it.
// The normal queue owns dispatch and its current execution-policy checks.
return priorReceipt.runId ? getRun(priorReceipt.runId) : null;
}
const agentDebug = parseObject(parseObject(agent.runtimeConfig).debug);
const runDebug = parseObject(enrichedContextSnapshot.debug);
if (
agentDebug.providerTrace === "raw" &&
runDebug.providerTrace !== "raw"
) {
enrichedContextSnapshot.debug = {
...runDebug,
providerTrace: "raw",
};
enrichedContextSnapshot.providerTraceRequestedBy = `agent:${agent.id}:debug-setting`;
enrichedContextSnapshot.providerTraceRequestSource =
"agent_debug_setting";
}
// Automatic signals are replaceable; user input and interaction delivery
// retain distinct durable receipts even when the same gate blocks them.
const coalesceExecutionWait =
opts.requestedByActorType === "system" &&
opts.allowRunCoalescing !== false &&
!receiptRequest &&
!wakeCommentId &&
queuedCommentIdsFromRunContext(enrichedContextSnapshot).length === 0 &&
!isInteractionResolutionWakePayload(payload ?? {}) &&
!hasInteractionContinuationWakeContext(enrichedContextSnapshot);
const writeSkippedRequest = async (
skipReason: string,
patch: Partial<typeof agentWakeupRequests.$inferInsert> = {},
waitCondition?: Record<string, unknown>,
): Promise<{ created: boolean; waitSaved?: boolean }> => {
// A temporary refusal must leave the handoff retryable, without burning
// its one durable receipt ID on a skipped wake.
if (caseRequest) return { created: false };
if (executionWaitRequestId) {
const waitSaved = await writeDeferredExecutionWait(db, { companyId: agent.companyId, requestId: executionWaitRequestId }, {
reason: skipReason, message: patch.error ?? (skipReason === "issue_tree_hold_active"
? "This task is paused. Resume it to send your saved message."
: "Waiting for task execution to be enabled. Your message is saved."),
});
return { created: false, waitSaved };
}
const request = {
...durableReceiptFields,
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason: skipReason,
payload,
status: "skipped",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
finishedAt: new Date(),
...patch,
};
if (waitCondition && issueId && isUuidLike(issueId)) {
const waitIssueId = issueId;
return db.transaction(async (tx) => {
await tx.execute(sql`select id from issues where id = ${waitIssueId} and company_id = ${agent.companyId} for update`);
if (durableRequest) await notifyChatInboundWakeNotices(tx, {
companyId: durableRequest.companyId, issueId: durableRequest.issueId,
});
return recordExecutionWait(tx as unknown as Db, {
issueId: waitIssueId, request, condition: waitCondition, coalesce: coalesceExecutionWait,
});
});
}
if (durableRequest) {
await db.transaction(async tx => {
await notifyChatInboundWakeNotices(tx, {
companyId: durableRequest.companyId, issueId: durableRequest.issueId,
});
await tx.insert(agentWakeupRequests).values(request);
});
} else {
await db.insert(agentWakeupRequests).values(request);
}
return { created: true };
};
const writeSkippedHeartbeatRequest = async (
skipReason: string,
details: Record<string, unknown>,
) => {
await writeSkippedRequest(skipReason, {
payload: {
...(payload ?? {}),
heartbeatSkip: details,
},
});
};
// Work-intent outbox (invariant 4): a user's message refused by a
// temporary hold is saved as a deferred intent that names the refusal and
// the hold-release event that promotes it, never written as a terminal
// skip. Promotion runs normal admission again, once.
// The release drain promotes a saved message behind the task's latest run,
// once that run is terminal; a run still stopping after a pause commits is
// that run. A task that never ran has none: its message stays in the task,
// and stranded-work recovery starts the task once the hold lifts.
const canSaveDeferredMessage = async () => {
if (executionWaitRequestId || caseRequest || durableRequest || !issueId || !isUuidLike(issueId) ||
opts.requestedByActorType !== "user" || !wakeCommentId) return false;
const [latest] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, agent.companyId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`))
.limit(1);
return Boolean(latest);
};
const saveDeferredMessage = async (refusal: string, message: string, details: Record<string, unknown> = {}) => {
const savedIssueId = issueId!;
const condition = deferredWorkIntentWait(refusal, `${message} Your message is saved.`, details);
await db.transaction(async tx => {
await tx.execute(sql`select id from issues where id = ${savedIssueId} and company_id = ${agent.companyId} for update`);
await notifyDeferredExecutionWork(tx, { companyId: agent.companyId });
await tx.insert(agentWakeupRequests).values({
companyId: agent.companyId, agentId, source, triggerDetail, reason,
payload: withQueuedCommentIdsInWakePayload({
...payload, issueId: savedIssueId, [DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot, executionWait: condition,
}, [...new Set([...queuedCommentIdsFromRunContext(enrichedContextSnapshot), wakeCommentId!])]),
status: "deferred_issue_execution",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
});
});
return new PersistedExecutionWaitRefusal(message, details);
};
const schedulingSuppression = await getSchedulingSuppression();
// A task drain holds ADMISSION, not the request. The drain is a
// process-local pre-restart hold, so a wake that arrives while it is
// active still names real work that must run once the process comes
// back: leave it in the durable queue and let the dispatch-side checks
// (startNextQueuedRunForAgent / executeRun) keep it from starting until
// the drain lifts or the restart clears it. Writing it as `skipped`
// here dropped the wake permanently — an accepted plan whose
// continuation wake landed mid-drain left its issue in `todo` with no
// run and no path until a person noticed.
if (
schedulingSuppression.suppressed &&
schedulingSuppression.reason !== "task_drain"
) {
await writeSkippedHeartbeatRequest("heartbeat.scheduling_suppressed", {
reason: schedulingSuppression.reason,
});
return null;
}
const worktreeExecutionCutoff =
opts.requestedByActorType === "user"
? null
: await getWorktreeExecutionCutoff();
const company = await db
.select({ status: companies.status })
.from(companies)
.where(eq(companies.id, agent.companyId))
.then((rows) => rows[0] ?? null);
if (!company || company.status !== "active") {
if (caseRequest) return null;
const companyStatus = company?.status ?? "missing";
if (opts.requestedByActorType === "user") {
// A paused company is a temporary hold; an archived one is not.
if (companyStatus === "paused" && (await canSaveDeferredMessage()))
throw await saveDeferredMessage("company.inactive", "Company is not active", { status: companyStatus });
throw conflict("Company is not active", { status: companyStatus });
}
await writeSkippedRequest("company.inactive", {
error: `Wake suppressed because company status is ${companyStatus}`,
}, { companyStatus });
return null;
}
const explicitResumeSession = await resolveExplicitResumeSessionOverride(
agent,
payload,
taskKey,
);
if (explicitResumeSession) {
enrichedContextSnapshot.resumeFromRunId =
explicitResumeSession.resumeFromRunId;
enrichedContextSnapshot.resumeSessionDisplayId =
explicitResumeSession.sessionDisplayId;
enrichedContextSnapshot.resumeSessionParams =
explicitResumeSession.sessionParams;
if (
!readNonEmptyString(enrichedContextSnapshot.issueId) &&
explicitResumeSession.issueId
) {
enrichedContextSnapshot.issueId = explicitResumeSession.issueId;
}
if (
!readNonEmptyString(enrichedContextSnapshot.taskId) &&
explicitResumeSession.taskId
) {
enrichedContextSnapshot.taskId = explicitResumeSession.taskId;
}
if (
!readNonEmptyString(enrichedContextSnapshot.taskKey) &&
explicitResumeSession.taskKey
) {
enrichedContextSnapshot.taskKey = explicitResumeSession.taskKey;
}
issueId = readNonEmptyString(enrichedContextSnapshot.issueId) ?? issueId;
}
const effectiveTaskKey =
readNonEmptyString(enrichedContextSnapshot.taskKey) ?? taskKey;
const genericTimerWake =
source === "timer" &&
!issueId &&
!wakeCommentId &&
!readNonEmptyString(enrichedContextSnapshot.taskId) &&
!effectiveTaskKey;
// Old or direct timer producers cannot create a doomed taskless Runner
// run. Explicit session restoration above can supply a real task scope.
if (genericTimerWake && !supportsUnscopedTimerHeartbeat(agent)) return null;
const sessionBefore =
explicitResumeSession?.sessionDisplayId ??
(await resolveSessionBeforeForWakeup(agent, effectiveTaskKey));
let hasResolvablePriorSessionWorkspace: boolean | null = null;
const resolveHasResolvablePriorSessionWorkspace = async () => {
if (hasResolvablePriorSessionWorkspace !== null)
return hasResolvablePriorSessionWorkspace;
hasResolvablePriorSessionWorkspace = issueId
? await hasResolvablePriorSessionWorkspaceForWake({
agent,
contextSnapshot: enrichedContextSnapshot,
taskKey: effectiveTaskKey,
explicitResumeSession,
})
: false;
return hasResolvablePriorSessionWorkspace;
};
const continuationAttempt = readContinuationAttempt(
enrichedContextSnapshot.livenessContinuationAttempt,
);
let projectId = readNonEmptyString(enrichedContextSnapshot.projectId);
if (!projectId && issueId) {
// Look up by either UUID or identifier (e.g. "ENV-13"), but always scope
// by companyId so a row from another tenant can never be returned even
// when identifiers collide across companies. Guard the UUID arm because
// issues.id is a Postgres uuid column — passing "ENV-13" into eq(issues.id, …)
// would fail with an invalid-input-syntax cast error before the OR is
// evaluated.
const lookupIsUuid = isUuidLike(issueId);
const idMatch = lookupIsUuid
? or(
eq(issues.id, issueId),
eq(issues.identifier, issueId.toUpperCase()),
)
: eq(issues.identifier, issueId.toUpperCase());
const resolvedIssue = await db
.select({
id: issues.id,
projectId: issues.projectId,
createdAt: issues.createdAt,
})
.from(issues)
.where(and(eq(issues.companyId, agent.companyId), idMatch))
.then((rows) => rows[0] ?? null);
if (resolvedIssue) {
if (
worktreeExecutionCutoff &&
resolvedIssue.createdAt < worktreeExecutionCutoff
) {
await writeSkippedHeartbeatRequest(
"heartbeat.worktree_execution_cutoff",
{
reason: "worktree_execution_cutoff",
cutoff: worktreeExecutionCutoff.toISOString(),
issueId: resolvedIssue.id,
},
);
return null;
}
projectId = resolvedIssue.projectId ?? null;
// Canonicalize context to the UUID so downstream lookups always use UUID
if (resolvedIssue.id !== issueId) {
issueId = resolvedIssue.id;
enrichedContextSnapshot.issueId = issueId;
if (readNonEmptyString(enrichedContextSnapshot.taskId)) {
enrichedContextSnapshot.taskId = issueId;
}
}
}
}
// Propagate projectId into context so resolveWorkspaceForRun can bind the
// project workspace even when context.projectId wasn't set by the caller.
if (projectId && !readNonEmptyString(enrichedContextSnapshot.projectId)) {
enrichedContextSnapshot.projectId = projectId;
}
const isolatedWorkspacesEnabled = issueId
? (await instanceSettings.getExperimental()).enableIsolatedWorkspaces
: false;
let operatorResponsibleUserId: string | null = opts.manualUserWake ? opts.requestedByActorId! : null;
let queuedResponsibleUserIdPromise: Promise<string> | null = null;
const resolveQueuedResponsibleUserId = () => {
if (operatorResponsibleUserId) return Promise.resolve(operatorResponsibleUserId);
queuedResponsibleUserIdPromise ??= (async () => {
const queuedIssueContext = issueId
? await getIssueExecutionContext(agent.companyId, issueId)
: null;
const queuedRoutineEnvContext = await getRoutineEnvForExecutionIssue(
agent.companyId,
queuedIssueContext,
);
const queuedResponsibleUserId =
await resolveResponsibleUserIdForRunSeed({
companyId: agent.companyId,
contextSnapshot: enrichedContextSnapshot,
issueContext: queuedIssueContext,
routineEnvContext: queuedRoutineEnvContext,
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
source,
triggerDetail,
});
if (!queuedResponsibleUserId) {
throw new HttpError(
422,
"Unable to resolve responsible user for heartbeat run dispatch",
{
code: "responsible_user_unresolved",
agentId,
companyId: agent.companyId,
issueId: issueId ?? null,
source,
triggerDetail,
wakeReason: readNonEmptyString(
enrichedContextSnapshot.wakeReason,
),
},
);
}
return queuedResponsibleUserId;
})();
return queuedResponsibleUserIdPromise;
};
const budgetBlock = await budgets.getInvocationBlock(
agent.companyId,
agentId,
{
issueId,
projectId,
},
);
if (budgetBlock) {
// A budget block that is not a reached hard stop is an explicit pause.
if ((await canSaveDeferredMessage()) && !(await isBoardOwnedBudgetBlock(db, agent.companyId, budgetBlock)))
throw await saveDeferredMessage("budget.blocked", budgetBlock.reason,
{ scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId });
const wait = await writeSkippedRequest("budget.blocked", { error: budgetBlock.reason }, {
scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId,
});
if (caseRequest) return null;
const details = { scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId };
throw wait.waitSaved ? new PersistedExecutionWaitRefusal(budgetBlock.reason, details)
: conflict(budgetBlock.reason, details);
}
// Setup delays execution, but must not discard work accepted by a caller.
const invokability = await getAgentInvokability(isAgentAwaitingSetup(agent) ? { ...agent, status: "idle" } : agent);
if (!invokability.invokable) {
if (caseRequest) return null;
// Only a pause defers through the release drain (wake-queue policy); a
// pending approval still fails a promoted wake, so it is not saved here.
if (invokability.reason === "paused" && (await canSaveDeferredMessage()))
throw await saveDeferredMessage("agent.not_invokable", invokability.message,
{ status: agent.status, reason: invokability.reason });
let waitSaved = false;
if (opts.requestedByActorType !== "user" || executionWaitRequestId) {
const wait = await writeSkippedRequest("agent.not_invokable", {
error: invokability.message,
}, { status: agent.status, reason: invokability.reason });
waitSaved = wait.waitSaved === true;
}
const details = { status: agent.status, reason: invokability.reason,
invalidOrgChain: invokability.invalidOrgChain, ...invokability.details };
throw waitSaved ? new PersistedExecutionWaitRefusal(invokability.message, details)
: conflict(invokability.message, details);
}
const policy = parseHeartbeatPolicy(agent);
if (source === "timer" && !policy.enabled) {
await writeSkippedRequest("heartbeat.disabled", {}, { enabled: false });
return null;
}
if (source !== "timer" && !policy.wakeOnDemand) {
await writeSkippedRequest("heartbeat.wakeOnDemand.disabled", {}, { wakeOnDemand: false });
return null;
}
if (
policy.skipTimerWhenNoActionableWork &&
genericTimerWake &&
!(await hasActionableTimerWork(agent))
) {
await writeSkippedHeartbeatRequest("heartbeat.timer.no_actionable_work", {
reason:
"No assigned todo or in_progress issue requires this agent before timer adapter invocation.",
});
await markTimerHeartbeatChecked(agentId, source);
return null;
}
if (issueId) {
const activePauseHold = await treeControlSvc.getActivePauseHoldGate(
agent.companyId,
issueId,
);
if (activePauseHold) {
const treeHoldInteractionWake =
await isVerifiedIssueTreeControlInteractionWake(db, {
companyId: agent.companyId,
issueId,
agentId,
contextSnapshot: enrichedContextSnapshot,
requestedByActorType: opts.requestedByActorType,
requestedByActorId: opts.requestedByActorId,
});
if (!treeHoldInteractionWake) {
const wait = await writeSkippedRequest("issue_tree_hold_active", {}, {
holdId: activePauseHold.holdId,
});
if (wait.created) await logActivity(db, {
companyId: agent.companyId,
actorType: "system",
actorId: "system",
agentId,
runId: null,
action: "issue.tree_hold_wakeup_deferred",
entityType: "issue",
entityId: issueId,
details: {
holdId: activePauseHold.holdId,
rootIssueId: activePauseHold.rootIssueId,
requestedReason: reason,
source,
triggerDetail,
securityPrinciples: [
"Complete Mediation",
"Fail Securely",
"Secure Defaults",
],
},
});
return null;
}
enrichedContextSnapshot.treeHoldInteraction = true;
enrichedContextSnapshot.activeTreeHold = {
holdId: activePauseHold.holdId,
rootIssueId: activePauseHold.rootIssueId,
mode: activePauseHold.mode,
reason: activePauseHold.reason,
releasePolicy: activePauseHold.releasePolicy,
interaction: true,
};
}
}
if (issueId) {
// Mention-triggered wakes can request input from another agent, but they must
// still respect the issue execution lock so a second agent cannot start on the
// same issue workspace while the assignee already has a live run.
const agentNameKey = normalizeAgentNameKey(agent.name);
const cancelledRunsToEmit: (typeof heartbeatRuns.$inferSelect)[] = [];
const issueActivityPublications: ActivityPublication[] = [];
const issuePostCommitActions: IssuePostCommitAction[] = [];
const outcome = await db
.transaction(withDurableChatAdmission(durableRequest, async (tx) => {
await tx.execute(
sql`select id from issues where id = ${issueId} and company_id = ${agent.companyId} for update`,
);
if (executionWaitRequestId) {
const [pending] = await tx.select().from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.id, executionWaitRequestId), eq(agentWakeupRequests.companyId, agent.companyId),
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
// A user message can join a queue originally created by a
// system wake. Admission validates the saved user comment or board click.
(opts.queuedCommentInterruptId ?? opts.queuedCommentRequestId) === executionWaitRequestId
? undefined : eq(agentWakeupRequests.requestedByActorType,
durableRequest ? opts.requestedByActorType ?? "" : "user"),
opts.queuedCommentInterruptId === executionWaitRequestId
? sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt'->>'actorId' = ${opts.requestedByActorId ?? ""}`
: opts.queuedCommentRequestId === executionWaitRequestId ? undefined
: eq(agentWakeupRequests.requestedByActorId, opts.requestedByActorId ?? ""),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
));
// The issue lock serializes cleanup callbacks and periodic workers.
// An adopted, discarded, or edited receipt is no longer authority.
if (!pending || (!opts.taskRetryRequestId && !(wakeCommentId && queuedCommentIdsFromWakePayload(pending.payload).includes(wakeCommentId)) &&
!(opts.queuedCommentInterruptId && await readQueuedInteractionResponse(tx as unknown as Db,
agent.companyId, issueId, pending.payload)))) {
return { kind: "deferred" as const };
}
if (opts.taskRetryRequestId && durableRequest?.failedRunRetry) {
if (pending.id !== durableRequest.id || pending.payload?.taskRetryRequestId !== opts.taskRetryRequestId)
return { kind: "deferred" as const };
// The parent board Retry is checked separately below after the
// connected conversation reauthorizes this exact child receipt.
} else if (opts.taskRetryRequestId) {
const retry = readTaskRetryIntent(pending.payload);
if (pending.id !== opts.taskRetryRequestId || retry?.generationId !== pending.id || retry.chatActionId ||
retry.actorId !== opts.requestedByActorId || retry.sourceRunId !== enrichedContextSnapshot.previousRunId)
return { kind: "deferred" as const };
// A declined Retry stays saved, but never silently: name the gate
// so the task explains it and a new click can replace this intent.
const declineRetry = async (reason: string, message: string) => {
await writeDeferredExecutionWait(tx as unknown as Db, { companyId: agent.companyId,
requestId: pending.id }, { reason, message }, { replace: true, publications: issueActivityPublications });
logger.warn({ requestId: pending.id, issueId, reason }, "Task Retry could not start");
return { kind: "deferred" as const };
};
if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: agent.companyId, issueId,
actorId: retry.actorId, actorSource: retry.actorSource }))
return declineRetry("retry_authority_denied", "The user who requested Retry can no longer change this task.");
if (retry.boardMessageOrigin) {
const origin = await readTaskRetryBoardMessageOrigin(tx as unknown as Db, { companyId: agent.companyId,
issueId, agentId, sourceRunId: retry.sourceRunId, actorId: retry.actorId });
if (!origin || origin.commentId !== retry.boardMessageOrigin.commentId ||
origin.identityContextId !== retry.boardMessageOrigin.identityContextId)
return declineRetry("retry_source_changed", "The saved message this Retry continues has changed.");
}
}
if (opts.queuedCommentRequestId) {
const ids = await undeliveredLegacyUserCommentIds(tx as unknown as Db,
agent.companyId, issueId, agentId, queuedCommentIdsFromWakePayload(pending.payload));
if (!wakeCommentId || !ids.includes(wakeCommentId)) return { kind: "deferred" as const };
pending.payload = withQueuedCommentIdsInWakePayload(parseObject(pending.payload), ids);
await tx.update(agentWakeupRequests).set({ payload: pending.payload }).where(and(
eq(agentWakeupRequests.id, pending.id), eq(agentWakeupRequests.companyId, agent.companyId),
));
}
if (!opts.queuedCommentInterruptId && !opts.queuedCommentRequestId && pending.payload?.manualUserWake === true) {
// A persisted manual wake keeps its actor when an execution wait
// resumes. The locked receipt above has revalidated that actor.
payload = { ...payload, manualUserWake: true };
operatorResponsibleUserId = opts.requestedByActorId!;
}
if (opts.queuedCommentInterruptId) {
// The locked board receipt supplies execution authority even when
// another user authored the messages. Dispatch revalidates the receipt.
operatorResponsibleUserId = opts.requestedByActorId!;
}
if (opts.queuedCommentInterruptId || opts.queuedCommentRequestId) {
// Edits/discards between the click and dispatch remain authoritative.
Object.assign(enrichedContextSnapshot, withQueuedCommentIdsInRunContext(
enrichedContextSnapshot, queuedCommentIdsFromWakePayload(pending.payload),
));
}
}
let automaticParentRunId: string | null = null;
if (
source === "automation" &&
opts.requestedByActorType === "system"
) {
const nativeParent = await readChatControlNativeParent(
tx as unknown as Db,
{ companyId: agent.companyId, issueId, agentId },
opts.requestedByActorId ?? null,
);
const genericParent =
!opts.requestedByActorId && reason === "issue_continuation_needed"
? readNonEmptyString(enrichedContextSnapshot.retryOfRunId)
: null;
automaticParentRunId =
nativeParent.kind === "parent"
? nativeParent.runId
: genericParent;
if (automaticParentRunId) {
const [parent] = await tx
.select({
agentId: heartbeatRuns.agentId,
context: heartbeatRuns.contextSnapshot,
})
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.companyId, agent.companyId),
eq(heartbeatRuns.id, automaticParentRunId),
),
)
.limit(1);
if (
parent &&
(parseObject(parent.context).issueId ??
parseObject(parent.context).taskId) === issueId &&
parent.agentId !== agentId
)
automaticParentRunId = null;
}
const proof =
nativeParent.kind === "unresolved"
? { kind: "unresolved" as const }
: automaticParentRunId
? await readChatControlRecoveryStop(
tx as unknown as Db,
{
companyId: agent.companyId,
issueId,
agentId,
sourceRunId: automaticParentRunId,
},
true,
)
: { kind: "clear" as const };
if (proof.kind !== "clear") {
await tx
.insert(agentWakeupRequests)
.values({
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason:
proof.kind === "stopped"
? CHAT_CONTROL_RECOVERY_STOP_CODE
: CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE,
payload: { issueId },
status: "skipped",
requestedByActorType: "system",
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
finishedAt: new Date(),
});
return { kind: "skipped" as const };
}
}
const durableReceipt = await existingDurableReceipt(
tx as unknown as Db,
);
if (durableReceipt)
return { kind: "durable" as const, receipt: durableReceipt };
if (caseRequest) {
const [handoff] = await tx.select().from(caseHandoffs).where(and(eq(caseHandoffs.id, caseRequest.handoffId),
eq(caseHandoffs.companyId, agent.companyId),
eq(caseHandoffs.issueId, caseRequest.issueId), eq(caseHandoffs.state, "pending_dispatch"),
eq(caseHandoffs.toAgentId, agentId), eq(caseHandoffs.wakeupRequestId, caseRequest.wakeupRequestId))).for("update");
if (!handoff || handoff.appliedIssueStatusVersion !== caseRequest.expectedIssueStatusVersion)
throw conflict("Case handoff delivery was superseded");
}
const failedChatRetry = await authorizeFailedChatRunRetryWake(
db,
tx as unknown as Db,
{
phase: "admission",
wakeupRequestId: durableRequest?.id ?? null,
companyId: agent.companyId,
agentId,
issueId,
contextSnapshot: enrichedContextSnapshot,
},
);
if (
failedChatRetry !== Boolean(durableRequest?.failedRunRetry) ||
(failedChatRetry &&
durableRequest?.failedRunRetry?.failedRunId !==
readNonEmptyString(enrichedContextSnapshot.retryOfRunId))
) {
throw new FailedChatRunRetryAuthorizationError();
}
if (durableRequest) {
if (issueId !== durableRequest.issueId)
throw new Error("chat_inbound_wakeup_binding_denied");
await durableRequest.authorize(tx as unknown as Db);
if (opts.taskRetryRequestId) {
const [retryWake] = await tx.select().from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, agent.companyId), eq(agentWakeupRequests.id, opts.taskRetryRequestId),
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
)).for("update");
const retry = readTaskRetryIntent(retryWake?.payload);
if (!failedChatRetry || retry?.chatActionId !== durableRequest.id ||
retry.actorId !== taskRetryActorId || retry.sourceRunId !== durableRequest.failedRunRetry?.failedRunId)
throw new FailedChatRunRetryAuthorizationError();
if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: agent.companyId, issueId,
actorId: retry.actorId, actorSource: retry.actorSource })) return { kind: "deferred" as const };
}
} else {
if (
source === "on_demand" &&
triggerDetail === "manual" &&
reason === "retry_failed_run"
) {
// Generic Board retry does not carry server-authorized failed-run
// lineage. Caller-supplied source/comment/retry markers cannot
// restore it, including for a retired conversation generation.
const [chatBinding] = await tx
.select({ id: chatConversations.id })
.from(chatConversations)
.innerJoin(chatEndpoints, eq(chatEndpoints.id, chatConversations.endpointId))
.where(
and(
eq(chatEndpoints.externalExecutionPolicy, "restricted"),
eq(chatConversations.companyId, agent.companyId),
eq(chatConversations.issueId, issueId),
),
)
.limit(1);
if (chatBinding) {
throw conflict(
"Retry needs the exact failed chat request and current access. Send the request again in the current connected conversation.",
{ code: "chat_failed_run_retry_requires_authorized_context" },
);
}
}
const [held] = await tx
.select({ id: issues.id })
.from(issues)
.where(
and(
eq(issues.id, issueId),
eq(issues.companyId, agent.companyId),
unadmittedChatWakeupCondition(issues.id, issues.companyId),
),
)
.limit(1);
if (held) {
throw conflict(
"This task has external chat input that has not been admitted. Check the connection's Activity; let pending input finish, or restore access and send a new authorized message before starting the task.",
{ code: "chat_inbound_wakeup_unadmitted", issueId },
);
}
}
const issue = await tx
.select({
id: issues.id,
companyId: issues.companyId,
identifier: issues.identifier,
conversationAgentId: issues.conversationAgentId,
conversationUserId: issues.conversationUserId,
conversationState: issues.conversationState,
status: issues.status,
statusVersion: issues.statusVersion,
projectId: issues.projectId,
projectWorkspaceId: issues.projectWorkspaceId,
executionWorkspaceId: issues.executionWorkspaceId,
executionWorkspacePreference: issues.executionWorkspacePreference,
executionWorkspaceSettings: issues.executionWorkspaceSettings,
assigneeAgentId: issues.assigneeAgentId,
assigneeUserId: issues.assigneeUserId,
monitorNextCheckAt: issues.monitorNextCheckAt,
monitorWakeRequestedAt: issues.monitorWakeRequestedAt,
executionRunId: issues.executionRunId,
executionAgentNameKey: issues.executionAgentNameKey,
createdAt: issues.createdAt,
})
.from(issues)
.where(
and(
eq(issues.id, issueId),
eq(issues.companyId, agent.companyId),
),
)
.then((rows) => rows[0] ?? null);
if (!issue) {
if (caseRequest) throw conflict("Case handoff task no longer exists");
await tx.insert(agentWakeupRequests).values({
...durableReceiptFields,
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason: "issue_execution_issue_not_found",
payload,
status: "skipped",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
finishedAt: new Date(),
});
return { kind: "skipped" as const };
}
// Completion can settle the conversation while admission is outside
// the issue lock. Recheck the current state before persisting a wake.
if (isWaitingConversation(issue) && !hasIndependentConversationWakeCause()) {
return { kind: "skipped" as const };
}
if (taskRetryActorId) {
await assertTaskRetryConversationAllowed(tx as unknown as Db, issue, taskRetryActorId);
if (await retireSupersededTaskRetry(tx as unknown as Db, {
companyId: issue.companyId, issueId: issue.id, requestId: opts.taskRetryRequestId!,
sourceRunId: readNonEmptyString(enrichedContextSnapshot.previousRunId)!,
})) return { kind: "deferred" as const };
}
const issueStateGuard = opts.issueStateGuard;
const activeMonitorRun = issueStateGuard?.monitorNextCheckAt === undefined ? null
: await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.nativeIssueId, issue.id),
eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, ["queued", "running", "scheduled_retry"]),
)).limit(1).then(rows => rows[0] ?? null);
if (
issueStateGuard &&
(!issueStateGuard.statuses.includes(issue.status) ||
issue.assigneeAgentId !== issueStateGuard.assigneeAgentId ||
(issueStateGuard.statusVersion !== undefined && issue.statusVersion !== issueStateGuard.statusVersion) ||
(issueStateGuard.monitorNextCheckAt !== undefined && (
activeMonitorRun !== null || issue.assigneeUserId !== null ||
issue.monitorNextCheckAt?.toISOString() !== issueStateGuard.monitorNextCheckAt ||
issue.monitorWakeRequestedAt?.toISOString() !== issueStateGuard.monitorWakeRequestedAt
)))
) {
if (caseRequest) throw conflict("Case handoff task changed before admission");
// A deferred monitor retains its schedule; do not create a receipt
// that could suppress its next admission attempt.
if (issueStateGuard.monitorNextCheckAt !== undefined) return { kind: "skipped" as const };
await tx.insert(agentWakeupRequests).values({
...durableReceiptFields,
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason: "issue_state_guard_mismatch",
payload: {
...(payload ?? {}),
heartbeatSkip: {
reason:
"Issue status or assignee changed before the wake could be queued.",
issueId: issue.id,
expectedStatuses: issueStateGuard.statuses,
actualStatus: issue.status,
expectedAssigneeAgentId: issueStateGuard.assigneeAgentId,
actualAssigneeAgentId: issue.assigneeAgentId,
},
},
status: "skipped",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
finishedAt: new Date(),
});
return { kind: "skipped" as const };
}
if (opts.failedRunId) {
// The issue lock makes double-clicks and network retries adopt the
// same successor, including after it has already finished.
const [previousRetry] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.agentId, agentId),
eq(heartbeatRuns.retryOfRunId, opts.failedRunId),
sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'retry_failed_run'`,
)).orderBy(desc(heartbeatRuns.createdAt)).limit(1);
if (previousRetry) return { kind: "replayed" as const, run: previousRetry };
}
let reconciledSourceRunId: string | null = null;
let reconciledRestoreRetryCount: number | null = null;
if (executionReconciliationWake) {
const actionId = readNonEmptyString(
enrichedContextSnapshot.recoveryActionId,
);
if (
!actionId ||
!isUuidLike(actionId) ||
source !== "automation" ||
triggerDetail !== "system" ||
reason !== "issue_recovery_action_restored" ||
opts.requestedByActorType !== "system" ||
opts.requestedByActorId !== "execution-recovery" ||
opts.idempotencyKey !== `execution-reconciliation:${actionId}` ||
enrichedContextSnapshot.source !== "execution.reconciled" ||
enrichedContextSnapshot.forceFreshSession !== true ||
payload?.issueId !== issue.id ||
payload?.recoveryActionId !== actionId ||
issue.assigneeAgentId !== agentId ||
["done", "cancelled"].includes(issue.status)
)
return { kind: "skipped" as const };
// The issue lock serializes all admissions for this source. Validate
// the durable operator decision, then reconcile a prior queue commit
// before considering a new wake (including a now-terminal successor).
const [action] = await tx
.select()
.from(issueRecoveryActions)
.where(
and(
eq(issueRecoveryActions.companyId, issue.companyId),
eq(issueRecoveryActions.sourceIssueId, issue.id),
eq(issueRecoveryActions.id, actionId),
),
)
.for("update");
const decision = parseObject(
action?.evidence.executionReconciliation,
);
const sourceRunId = readNonEmptyString(decision.runId);
if (
!action ||
action.status !== "resolved" ||
action.kind !== "active_run_watchdog" ||
action.returnOwnerAgentId !== agentId ||
!sourceRunId ||
!isUuidLike(sourceRunId) ||
decision.providerStopped !== true ||
!["completed", "not_performed", "mixed"].includes(
String(decision.actionOutcome),
) ||
!readNonEmptyString(decision.outcomeEvidence) ||
enrichedContextSnapshot.previousRunId !== sourceRunId ||
enrichedContextSnapshot.retryOfRunId !== sourceRunId ||
!["pending", "delivered"].includes(
String(action.evidence.continuationDelivery),
)
)
return { kind: "skipped" as const };
const [existingWake] = await tx
.select()
.from(agentWakeupRequests)
.where(
and(
eq(agentWakeupRequests.companyId, issue.companyId),
eq(agentWakeupRequests.agentId, agentId),
eq(agentWakeupRequests.idempotencyKey, opts.idempotencyKey),
ne(agentWakeupRequests.status, "skipped"),
),
)
.orderBy(asc(agentWakeupRequests.requestedAt))
.limit(1);
if (existingWake) {
if (
existingWake.payload?.issueId !== issue.id ||
existingWake.payload?.recoveryActionId !== action.id ||
existingWake.requestedByActorType !== "system" ||
existingWake.requestedByActorId !== "execution-recovery" ||
!existingWake.runId
)
return { kind: "deferred" as const };
const [existingRun] = await tx
.select()
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.companyId, issue.companyId),
eq(heartbeatRuns.agentId, agentId),
eq(heartbeatRuns.id, existingWake.runId),
),
);
if (
!existingRun ||
existingRun.contextSnapshot?.issueId !== issue.id ||
existingRun.contextSnapshot?.recoveryActionId !== action.id ||
existingRun.contextSnapshot?.previousRunId !== sourceRunId
)
return { kind: "deferred" as const };
return { kind: "replayed" as const, run: existingRun };
}
if (action.evidence.continuationDelivery !== "pending")
return { kind: "skipped" as const };
// Under the task lock: a fresh continuation is the task's next run
// only while nothing else continued it. A run started since it was
// recorded (active or already finished, from a person's Retry or
// saved input) consumes it, and a task that moved to review gets none.
const freshRecordedAt = readNonEmptyString(parseObject(action.evidence.freshContinuation).recordedAt);
if (freshRecordedAt) {
const [intervening] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, issue.companyId), sql`${heartbeatRuns.id}::text <> ${sourceRunId}`,
or(eq(heartbeatRuns.nativeIssueId, issue.id), eq(heartbeatRuns.issueId, issue.id),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}::text`),
sql`${heartbeatRuns.createdAt} >= ${freshRecordedAt}::timestamptz`)).limit(1);
if (intervening || issue.status === "in_review") {
await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify(
intervening ? { continuationDelivery: "delivered", continuationRunId: intervening.id, continuationSupersededByRun: true }
: { continuationDelivery: "invalidated", continuationInvalidatedReason: "in_review" })}::jsonb` })
.where(and(eq(issueRecoveryActions.companyId, issue.companyId), eq(issueRecoveryActions.id, action.id)));
await settleCoalescedFreshContinuationHolds(tx as unknown as Db, { companyId: issue.companyId, primaryId: action.id,
delivery: intervening ? "delivered" : "invalidated", continuationRunId: intervening?.id ?? null });
return { kind: "skipped" as const };
}
}
const [reconciledRun] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.id, sourceRunId),
));
if (decision.workspaceReconstruction) {
await validateNativeWorkspaceReconstruction(tx as unknown as Db, {
companyId: issue.companyId, issueId: issue.id, runId: sourceRunId, actionId,
});
reconciledRestoreRetryCount = executionFailureRetryCount(reconciledRun!);
enrichedContextSnapshot.workspaceReconstructionActionId = action.id;
}
if (hasWorkspaceRestoreFailure(reconciledRun?.resultJson)) {
if ((readNonEmptyString(decision.workspaceRepairEvidence)?.length ?? 0) < 20)
return { kind: "skipped" as const };
// Repair does not reset the remaining automatic retry budget.
reconciledRestoreRetryCount = executionFailureRetryCount(reconciledRun!);
}
reconciledSourceRunId = sourceRunId;
}
let continuationWait = { reason: "execution_recovery", message: "Waiting for execution recovery. Your message is saved." };
let continuationBlocked = false;
const deferBlockedExecution = async (
executionBlocker: Pick<NonNullable<Awaited<ReturnType<typeof getExecutionBlocker>>>, "recoveryActionId" | "cause" | "nextAction">,
) => {
if (caseRequest) waitForCaseAdmission("execution_reconciliation_required", executionBlocker.nextAction);
opts.issueStateGuard?.onExecutionDeferred?.(executionBlocker.cause);
const condition = { recoveryActionId: executionBlocker.recoveryActionId, ...continuationWait };
if (executionWaitRequestId) {
await writeDeferredExecutionWait(tx as unknown as Db, { companyId: agent.companyId,
requestId: executionWaitRequestId }, condition, { replace: true, publications: issueActivityPublications });
return { kind: "deferred" as const, admitted: true };
}
if (durableRequest || wakeCommentId ||
hasInteractionContinuationWakeContext(enrichedContextSnapshot) ||
readNonEmptyString(enrichedContextSnapshot.nativeStatusWakeIntentId)) {
await notifyDeferredExecutionWork(tx, { companyId: agent.companyId });
await tx.insert(agentWakeupRequests).values({
...durableReceiptFields,
companyId: agent.companyId, agentId, source, triggerDetail, reason,
payload: withQueuedCommentIdsInWakePayload({
...payload,
issueId: issue.id,
[DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot,
executionWait: condition,
}, [...new Set([
...queuedCommentIdsFromRunContext(enrichedContextSnapshot),
...(wakeCommentId ? [wakeCommentId] : []),
])]),
status: "deferred_issue_execution",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
});
} else {
await recordExecutionWait(tx as unknown as Db, {
issueId: issue.id, condition, coalesce: coalesceExecutionWait,
request: {
...durableReceiptFields,
companyId: agent.companyId, agentId, source, triggerDetail,
reason: "execution_reconciliation_required",
error: executionBlocker.nextAction,
payload,
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
},
});
}
return { kind: "deferred" as const, admitted: true };
};
const explicitContinuationRunId = randomUUID();
const [resetComment] = isConversation(issue) && opts.requestedByActorType === "user" && opts.requestedByActorId && wakeCommentId
? await tx.select({ body: issueComments.body }).from(issueComments).where(and(
eq(issueComments.companyId, issue.companyId), eq(issueComments.issueId, issue.id),
eq(issueComments.id, wakeCommentId), eq(issueComments.authorUserId, opts.requestedByActorId),
eq(issueComments.authorType, "user"), isNull(issueComments.deletedAt), isNull(issueComments.createdByRunId),
)).limit(1)
: [];
const verifiedConversationReset = Boolean(resetComment && isConversationReset(resetComment.body));
if (wakeCommentId && opts.requestedByActorType === "user")
await preserveTaskRemoteWorkspaceWriter(tx as unknown as Db, issue.companyId, issue.id);
const executionBlocker = await getExecutionBlocker(
tx as unknown as Db, issue.companyId, issue.id,
{ conversationResetCommentId: verifiedConversationReset ? wakeCommentId : null },
);
// Prove eligibility without retiring the hold. Later gates can still
// decline this wake; hold retirement and successor creation stay atomic.
// A bound chat request has already rechecked its current principal
// above. Treat its new user message like a board comment, but keep
// failed-run retry actions on their separate exact-request path.
const ownsReconstructionContinuation = reconciledSourceRunId && executionBlocker?.runId === reconciledSourceRunId
&& executionBlocker.recoveryActionId === enrichedContextSnapshot.recoveryActionId;
const continuationHold = () => executionBlocker ?? {
recoveryActionId: null, cause: "native_continuation_requires_reconciliation" as const, nextAction: continuationWait.message,
};
// Only system continuation wakes (sweeps, monitors, timers) reach
// the fresh-continuation fallback. A person's Retry, message or
// queued send keeps its own exact admission; the disposition worker
// settles the hold through the same fallback, and the saved input
// then continues the task.
const systemWake = opts.requestedByActorType !== "user" && !taskRetryActorId && !wakeCommentId
&& !durableRequest && !opts.queuedCommentInterruptId && !opts.queuedCommentRequestId && !opts.failedRunId
&& !executionReconciliationWake;
// A recorded fresh continuation that has no run yet is this task's
// next run: another system wake is coalesced into it, never a second run.
if (systemWake && !executionBlocker && await hasPendingFreshContinuation(tx as unknown as Db, issue.companyId, issue.id)) {
await tx.insert(agentWakeupRequests).values({
...durableReceiptFields, companyId: agent.companyId, agentId, source, triggerDetail,
reason: "issue_continuation_pending", payload, status: "skipped",
requestedByActorType: opts.requestedByActorType ?? null, requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null, finishedAt: new Date(),
});
return { kind: "skipped" as const };
}
const continuationToDeliver = !systemWake && !executionReconciliationWake && !executionBlocker
? await undeliveredContinuation(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id }) : null;
if (continuationToDeliver)
Object.assign(enrichedContextSnapshot, undeliveredContinuationContext(continuationToDeliver, enrichedContextSnapshot));
if ((executionBlocker || opts.queuedCommentInterruptId) && !ownsReconstructionContinuation && !(await admitExplicitNativeContinuation({
db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id,
agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId,
reason: taskRetryActorId ? "task_retry" : durableRequest && !failedChatRetry ? "issue_commented" : reason,
commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId,
queuedCommentInterruptId: opts.queuedCommentInterruptId,
queuedCommentRequestId: opts.queuedCommentRequestId,
taskRetryRequestId: opts.taskRetryRequestId,
...(durableRequest?.resumeDeferred && !failedChatRetry ? { durableChatRequest: durableRequest } : {}),
interactionPayload: payload,
dryRun: true,
onBlocked: (reason, message) => { continuationBlocked = true; continuationWait = { reason, message }; },
})) && (executionBlocker || (continuationBlocked && !verifiedConversationReset))) {
// A hold left by an unfinished run never refuses an ordinary
// continuation: unless its cause is the person's, the fallback
// settles it in this transaction and this wake becomes the fresh
// continuation from the task's saved work.
// Agent Chat conversations keep their own owner-bound continuation.
const freshSourceRunId = systemWake && executionBlocker?.recoveryActionId && executionBlocker.runId
&& !issue.conversationAgentId ? executionBlocker.runId : null;
// The continuation is recorded as pending and owned by the
// execution-continuation worker until a run for it exists: a later
// gate (daily limit, slots, pause, budget), a skip or a crash can
// never consume it. This system wake is coalesced into it.
const fresh = freshSourceRunId ? await continueFreshInTransaction(tx as unknown as Db, {
companyId: issue.companyId, runId: freshSourceRunId, trigger: "continuation_wake", delivery: "record" }) : null;
if (fresh?.kind !== "continued" && fresh?.kind !== "already_continued") return deferBlockedExecution(continuationHold());
return { kind: "skipped" as const };
}
if (
worktreeExecutionCutoff &&
issue.createdAt < worktreeExecutionCutoff
) {
if (caseRequest) waitForCaseAdmission("heartbeat.worktree_execution_cutoff", "Case handoff is waiting for workspace execution to be enabled");
await tx.insert(agentWakeupRequests).values({
...durableReceiptFields,
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason: "heartbeat.worktree_execution_cutoff",
payload: {
...(payload ?? {}),
heartbeatSkip: {
reason: "worktree_execution_cutoff",
cutoff: worktreeExecutionCutoff.toISOString(),
issueId: issue.id,
},
},
status: "skipped",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
finishedAt: new Date(),
});
return { kind: "skipped" as const };
}
const cancelStaleScheduledRetry = async (
scheduledRun: typeof heartbeatRuns.$inferSelect,
) => {
const issueCancelled = issue.status === "cancelled";
if (
scheduledRun.status !== "scheduled_retry" ||
(scheduledRun.agentId === issue.assigneeAgentId &&
!issueCancelled)
) {
return false;
}
const now = new Date();
const reason = issueCancelled
? "Cancelled because the issue was cancelled before the scheduled retry became due"
: "Cancelled because the issue was reassigned before the scheduled retry became due";
await notifyInstructionCollection(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
await notifyExternalAgentSettlement(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
await notifyCancelledWorkspacePreservation(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
await notifyUnsafeWorkspaceExportRunChange(tx, scheduledRun);
await notifyScheduledRetryCancellation(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
await notifyChatRunMilestones(tx, { companyId: scheduledRun.companyId, issueId: issue.id });
const cancelled = await tx
.update(heartbeatRuns)
.set({
status: "cancelled",
finishedAt: now,
error: reason,
errorCode: issueCancelled
? "issue_cancelled"
: "issue_reassigned",
updatedAt: now,
})
.where(
and(
eq(heartbeatRuns.id, scheduledRun.id),
eq(heartbeatRuns.status, "scheduled_retry"),
),
)
.returning()
.then((rows) => rows[0] ?? null);
if (!cancelled) return false;
if (scheduledRun.wakeupRequestId) {
await tx
.update(agentWakeupRequests)
.set({
status: "cancelled",
finishedAt: now,
error: reason,
updatedAt: now,
})
.where(
eq(agentWakeupRequests.id, scheduledRun.wakeupRequestId),
);
}
if (issue.executionRunId === scheduledRun.id) {
await tx
.update(issues)
.set({
executionRunId: null,
executionAgentNameKey: null,
executionLockedAt: null,
updatedAt: now,
})
.where(
and(
eq(issues.id, issue.id),
eq(issues.executionRunId, scheduledRun.id),
),
);
}
const eventSeq = await allocateHeartbeatRunEventSeq(
tx as unknown as Db,
cancelled.id,
);
await tx.insert(heartbeatRunEvents).values({
companyId: cancelled.companyId,
runId: cancelled.id,
agentId: cancelled.agentId,
seq: eventSeq,
eventType: "lifecycle",
stream: "system",
level: "warn",
message: issueCancelled
? "Scheduled retry cancelled because issue was cancelled before it became due"
: "Scheduled retry cancelled because issue ownership changed before it became due",
payload: {
issueId: issue.id,
issueStatus: issue.status,
scheduledRetryAttempt: cancelled.scheduledRetryAttempt,
scheduledRetryAt: cancelled.scheduledRetryAt
? new Date(cancelled.scheduledRetryAt).toISOString()
: null,
scheduledRetryReason: cancelled.scheduledRetryReason,
previousRetryAgentId: cancelled.agentId,
currentAssigneeAgentId: issue.assigneeAgentId,
},
});
await tx
.update(heartbeatRuns)
.set({ nextEventSeq: eventSeq + 1, updatedAt: now })
.where(eq(heartbeatRuns.id, cancelled.id));
cancelledRunsToEmit.push(cancelled);
return true;
};
let activeExecutionRun = issue.executionRunId
? await tx
.select()
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, issue.executionRunId))
.then((rows) => rows[0] ?? null)
: null;
const manualExecutionHold = activeExecutionRun && await nativeManualAdmissionHoldsExecution(tx as unknown as Db,
{ companyId: issue.companyId, runId: activeExecutionRun.id, issueId: issue.id });
if (
activeExecutionRun &&
!manualExecutionHold &&
!EXECUTION_PATH_HEARTBEAT_RUN_STATUSES.includes(
activeExecutionRun.status as (typeof EXECUTION_PATH_HEARTBEAT_RUN_STATUSES)[number],
)
) {
activeExecutionRun = null;
}
if (
activeExecutionRun &&
(await cancelStaleScheduledRetry(activeExecutionRun))
) {
activeExecutionRun = null;
}
// A queued/scheduled run holding the lock for an agent that is
// no longer the issue's assignee is stale by design — the issue
// has been re-routed (e.g. blocked → in_review with a different
// assignee). Cancel it and release the lock; otherwise the new
// assignee's wake gets parked in `deferred_issue_execution`
// forever, because the original queued holder will never run
// (the issue's status / target now belongs to someone else).
//
// Race guard: pin the cancel UPDATE to the exact non-running
// status we read above. A worker could transition the holder
// from `queued` → `running` between the SELECT and this UPDATE;
// the status predicate ensures we never clobber a freshly-
// claimed running run. If zero rows matched, leave
// `activeExecutionRun` populated so the defer path runs
// normally against the now-running holder.
if (
activeExecutionRun &&
!manualExecutionHold &&
activeExecutionRun.status !== "running" &&
issue.assigneeAgentId &&
activeExecutionRun.agentId !== issue.assigneeAgentId
) {
await notifyInstructionCollection(tx, { companyId: activeExecutionRun.companyId, runId: activeExecutionRun.id });
await notifyExternalAgentSettlement(tx, { companyId: activeExecutionRun.companyId, runId: activeExecutionRun.id });
await notifyCancelledWorkspacePreservation(tx, { companyId: activeExecutionRun.companyId, runId: activeExecutionRun.id });
await notifyUnsafeWorkspaceExportRunChange(tx, activeExecutionRun);
await notifyChatRunMilestones(tx, { companyId: activeExecutionRun.companyId, issueId: issue.id });
const cancelled = await tx
.update(heartbeatRuns)
.set({
status: "cancelled",
finishedAt: new Date(),
error:
"Execution lock released after issue reassigned to a different agent",
errorCode: "lock_released_on_reassignment",
updatedAt: new Date(),
})
.where(
and(
eq(heartbeatRuns.id, activeExecutionRun.id),
eq(heartbeatRuns.status, activeExecutionRun.status),
),
)
.returning();
if (cancelled.length > 0) {
cancelledRunsToEmit.push(cancelled[0]);
if (activeExecutionRun.wakeupRequestId) {
await tx
.update(agentWakeupRequests)
.set({
status: "cancelled",
finishedAt: new Date(),
error:
"Execution lock released after issue reassigned to a different agent",
updatedAt: new Date(),
})
.where(
eq(
agentWakeupRequests.id,
activeExecutionRun.wakeupRequestId,
),
);
}
activeExecutionRun = null;
}
}
if (!activeExecutionRun && issue.executionRunId) {
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
await tx
.update(issues)
.set({
executionRunId: null,
executionAgentNameKey: null,
executionLockedAt: null,
updatedAt: new Date(),
})
.where(eq(issues.id, issue.id));
}
if (!activeExecutionRun) {
const legacyRun = await tx
.select()
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.companyId, issue.companyId),
inArray(heartbeatRuns.status, [
...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES,
]),
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`,
),
)
.orderBy(
sql`case when ${heartbeatRuns.status} = 'running' then 0 else 1 end`,
asc(heartbeatRuns.createdAt),
)
.limit(1)
.then((rows) => rows[0] ?? null);
if (legacyRun) {
if (await cancelStaleScheduledRetry(legacyRun)) {
activeExecutionRun = null;
} else {
activeExecutionRun = legacyRun;
const legacyAgent = await tx
.select({ name: agents.name })
.from(agents)
.where(eq(agents.id, legacyRun.agentId))
.then((rows) => rows[0] ?? null);
await notifyStatusCardIssueChange(tx, issue.companyId);
await tx
.update(issues)
.set({
executionRunId: legacyRun.id,
executionAgentNameKey: normalizeAgentNameKey(
legacyAgent?.name,
),
executionLockedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(issues.id, issue.id));
}
}
}
// Availability is advisory: competing clicks may both read it before
// admission. The existing issue lock serializes the run and deferred
// receipt checks with the admission below, without another lock owner.
if (isManualContextCompactionWake()) {
if (activeExecutionRun) {
const compacting = isContextCompactionRun(activeExecutionRun.contextSnapshot);
throw new ContextCompactionError(compacting ? "already_pending" : "agent_busy",
compacting ? "The agent is already compacting its context." : "Available when the agent finishes its current run.");
}
const [pending] = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, issue.companyId), eq(agentWakeupRequests.agentId, agentId),
inArray(agentWakeupRequests.status, ["queued", "deferred_issue_execution"]), isNull(agentWakeupRequests.runId),
executionWaitRequestId ? ne(agentWakeupRequests.id, executionWaitRequestId) : undefined,
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
sql`${agentWakeupRequests.payload}->>'intent' = ${CONTEXT_COMPACTION_WAKE_REASON}`,
sql`jsonb_typeof(${agentWakeupRequests.payload}->'requestId') = 'string'`,
sql`nullif(btrim(${agentWakeupRequests.payload}->>'requestId'), '') is not null`,
sql`${agentWakeupRequests.payload}->${DEFERRED_WAKE_CONTEXT_KEY}->>'contextCompactionRequestId' = ${agentWakeupRequests.payload}->>'requestId'`,
)).limit(1);
if (pending) throw new ContextCompactionError("already_pending", "The agent is already compacting its context.");
}
const dependencyReadiness = await issuesSvc
.listDependencyReadiness(issue.companyId, [issue.id], tx)
.then((rows) => rows.get(issue.id) ?? null);
// Blocked descendants should stay idle until the final blocker resolves.
// Human comment/mention wakes are the exception: they may run in a
// bounded interaction mode so the assignee can answer or triage.
// A stalled child or blocker (issue_blocker_stalled, once per stall)
// is the other exception: the assignee must be able to re-delegate,
// escalate or block with a reason while that dependency is stuck.
const blockedInteractionWake =
dependencyReadiness &&
!dependencyReadiness.isDependencyReady &&
allowsDependencyBlockedWake(
enrichedContextSnapshot,
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
await verifyBlockerStallWake(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id,
contextSnapshot: enrichedContextSnapshot, requestedByActorType: opts.requestedByActorType }),
await isVerifiedDependencyInteractionResponse(tx, { companyId: issue.companyId,
issueId: issue.id, agentId, context: enrichedContextSnapshot }),
);
if (blockedInteractionWake) {
enrichedContextSnapshot.dependencyBlockedInteraction = true;
enrichedContextSnapshot.unresolvedBlockerIssueIds =
dependencyReadiness.unresolvedBlockerIssueIds;
enrichedContextSnapshot.unresolvedBlockerCount =
dependencyReadiness.unresolvedBlockerCount;
enrichedContextSnapshot.unresolvedBlockerSummaries =
await listUnresolvedBlockerSummaries(
tx,
issue.companyId,
issue.id,
dependencyReadiness.unresolvedBlockerIssueIds,
);
}
if (
!activeExecutionRun &&
dependencyReadiness &&
!dependencyReadiness.isDependencyReady &&
!blockedInteractionWake
) {
if (caseRequest) waitForCaseAdmission("issue_dependencies_blocked", "Case handoff is waiting for task dependencies to resolve");
await recordExecutionWait(tx as unknown as Db, {
issueId: issue.id,
coalesce: coalesceExecutionWait,
condition: { unresolvedBlockerIssueIds: [...dependencyReadiness.unresolvedBlockerIssueIds].sort() },
request: {
...durableReceiptFields,
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason: "issue_dependencies_blocked",
payload: {
...(payload ?? {}),
issueId,
unresolvedBlockerIssueIds:
dependencyReadiness.unresolvedBlockerIssueIds,
},
status: "skipped",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
finishedAt: new Date(),
},
});
return { kind: "skipped" as const };
}
if (
isolatedWorkspacesEnabled &&
!activeExecutionRun &&
(issue.status !== "done" || Boolean(opts.queuedCommentInterruptId)) &&
issue.status !== "cancelled"
) {
const issueSettings = parseIssueExecutionWorkspaceSettings(
issue.executionWorkspaceSettings,
);
const resolvedMode = resolveExecutionWorkspaceMode({
projectPolicy: null,
issueSettings,
legacyUseProjectWorkspace: null,
});
const workspaceManagedConfig = buildExecutionWorkspaceAdapterConfig(
{
agentConfig: parseObject(agent.adapterConfig),
projectPolicy: null,
issueSettings,
mode: resolvedMode,
legacyUseProjectWorkspace: null,
},
);
const resolvedStrategy = resolveEffectiveWorkspaceStrategyType(
resolvedMode,
workspaceManagedConfig,
);
const existingExecutionWorkspaceStatus = issue.executionWorkspaceId
? await tx
.select({ status: executionWorkspaces.status })
.from(executionWorkspaces)
.where(
and(
eq(executionWorkspaces.id, issue.executionWorkspaceId),
eq(executionWorkspaces.companyId, issue.companyId),
),
)
.then((rows) => rows[0]?.status ?? null)
: null;
const reuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({
issueExecutionWorkspaceId: issue.executionWorkspaceId,
issueExecutionWorkspacePreference:
issue.executionWorkspacePreference,
existingExecutionWorkspaceStatus,
});
const hasResolvablePriorSessionWorkspace =
await resolveHasResolvablePriorSessionWorkspace();
if (
isUnrunnableWorktreeCombo({
issue: {
projectId: issue.projectId ?? projectId ?? null,
projectWorkspaceId: issue.projectWorkspaceId,
executionWorkspaceId: issue.executionWorkspaceId,
executionWorkspacePreference:
issue.executionWorkspacePreference,
},
resolvedMode,
resolvedStrategy,
reusableExecutionWorkspaceAvailable:
reuseRequest.existingExecutionWorkspaceAvailable,
hasResolvablePriorSessionWorkspace,
freshIsolatedContinuation: await isFreshIsolatedContinuationWake(tx as unknown as Db, {
companyId: issue.companyId, issueId: issue.id, contextSnapshot: enrichedContextSnapshot }),
})
) {
if (caseRequest) waitForCaseAdmission(WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE, WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE);
const now = new Date();
const issueLabel = formatIssueIdentifierLink(
issue.identifier,
issue.id,
);
const blockedComment = [
`Paperclip blocked ${issueLabel} before dispatch because its workspace settings are not runnable.`,
"",
`- Code: \`${WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE}\``,
`- Reason: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE}`,
`- Next action: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION}`,
].join("\n");
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
if (issue.status !== "blocked") await notifyChatResponseAvailability(tx, { companyId: issue.companyId, issueId: issue.id });
await notifyRoutineExecutionStatus(tx, issue.id, issue.companyId);
if (issue.status === "done") {
const blocked = await issuesSvc.updateForCompany(issue.id, issue.companyId,
{ status: "blocked", executionState: null }, tx, issueActivityPublications, issuePostCommitActions);
if (!blocked) throw conflict("The task changed before its workspace settings could be checked");
} else await tx
.update(issues)
.set({
status: "blocked",
checkoutRunId: null,
executionRunId: null,
executionAgentNameKey: null,
executionLockedAt: null,
updatedAt: now,
})
.where(eq(issues.id, issue.id));
await insertIssueComment(tx, {
companyId: issue.companyId,
issueId: issue.id,
body: blockedComment,
createdAt: now,
updatedAt: now,
});
await tx.insert(agentWakeupRequests).values({
...durableReceiptFields,
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
payload: {
...(payload ?? {}),
issueId,
heartbeatSkip: {
code: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE,
remediation:
WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION,
},
},
status: "skipped",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
finishedAt: now,
});
await logActivity(tx as unknown as Db, {
companyId: issue.companyId,
actorType: "system",
actorId: "system",
agentId,
runId: null,
action: "issue.workspace_preflight_blocked",
entityType: "issue",
entityId: issue.id,
details: {
code: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE,
remediation: WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION,
requestedReason: reason,
source,
triggerDetail,
resolvedMode,
resolvedStrategy,
hasResolvablePriorSessionWorkspace,
},
});
return { kind: "skipped" as const };
}
}
if (activeExecutionRun) {
// The case ledger owns retry. Do not place its exact receipt in a
// generic deferred queue that cannot retain handoff authority.
if (caseRequest) return { kind: "deferred" as const };
// The resolved action is already a durable retry outbox. Do not merge
// its fresh-session contract into unrelated work or create a second
// deferred wake that could later replay the same reconciliation.
if (reconciledSourceRunId) return { kind: "deferred" as const };
// Offer the input to the active external agent, but keep a durable queued wake
// until it reads the comment. A finish racing the event tick must
// leave unread input available for the next assignment.
const followUpProvider = parseObject(agent.adapterConfig).provider;
const museFollowUp = followUpProvider === "muse";
const bindingId = readNonEmptyString(parseObject(agent.adapterConfig)[museFollowUp ? "museBindingId" : followUpProvider === "grok_bot" ? "externalBindingId" : "dotBindingId"]);
const followUpPayloadKey = museFollowUp ? "museAssignmentFollowUp" : "dotAssignmentFollowUp";
let followUp: false | { assignmentId: string; consumed: boolean } = false;
let pendingFollowUps: typeof agentWakeupRequests.$inferSelect[] = [];
if (agent.adapterType === "paperclip_runner" && ["openai_dot", "grok_bot", "muse"].includes(String(followUpProvider))
&& bindingId && activeExecutionRun.agentId === agentId && issue.assigneeAgentId === agentId
&& reason === "issue_commented" && wakeCommentId && opts.allowRunCoalescing !== false
&& enrichedContextSnapshot.forceFreshSession !== true && !explicitResumeSession && !opts.manualUserWake
&& !enrichedContextSnapshot.interactionId && !payload?.interactionId && !receiptRequest) {
// Receipt consumption uses the same issue -> wake -> run -> guard ->
// binding order. Lock before publishing, then reuse these wake rows.
pendingFollowUps = await tx.select().from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, agent.companyId), eq(agentWakeupRequests.agentId, agentId),
isNull(agentWakeupRequests.runId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
sql`${agentWakeupRequests.payload}->>${followUpPayloadKey} is not null`,
)).orderBy(asc(agentWakeupRequests.createdAt), asc(agentWakeupRequests.id)).for("update");
const [currentRun] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, activeExecutionRun.id), eq(heartbeatRuns.companyId, agent.companyId),
eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.status, "running"),
eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.nativeIssueId, issue.id),
)).for("update");
if (currentRun) followUp = await withExternalAdmissionGuard(tx, agent.companyId, agentId,
() => (museFollowUp ? publishActiveMuseComment : publishActiveDotComment)(tx as unknown as Db, {
companyId: agent.companyId, agentId, bindingId, runId: currentRun.id,
issueId: issue.id, commentId: wakeCommentId,
}));
}
if (followUp) {
if (!followUp.consumed) await notifyDeferredExecutionWork(tx, { companyId: agent.companyId });
const assignmentId = followUp.assignmentId;
const pending = followUp.consumed ? undefined : pendingFollowUps.find(wake =>
wake.payload?.[followUpPayloadKey] === assignmentId &&
wake.requestedByActorType === (opts.requestedByActorType ?? null) &&
wake.requestedByActorId === (opts.requestedByActorId ?? null));
if (pending) await tx.update(agentWakeupRequests).set({
payload: withQueuedCommentIdsInWakePayload(pending.payload,
[...new Set([...queuedCommentIdsFromWakePayload(pending.payload), wakeCommentId!])]),
coalescedCount: pending.coalescedCount + 1, updatedAt: new Date(),
}).where(and(eq(agentWakeupRequests.id, pending.id), eq(agentWakeupRequests.companyId, agent.companyId),
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution")));
else await tx.insert(agentWakeupRequests).values({ companyId: agent.companyId, agentId, source, triggerDetail,
reason, payload: withQueuedCommentIdsInWakePayload({ ...payload, issueId: issue.id,
[followUpPayloadKey]: followUp.assignmentId,
[DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot }, [wakeCommentId!]),
status: followUp.consumed ? "coalesced" : "deferred_issue_execution",
runId: followUp.consumed ? activeExecutionRun.id : null,
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null, idempotencyKey: opts.idempotencyKey ?? null,
finishedAt: followUp.consumed ? new Date() : null });
return { kind: "coalesced" as const, run: activeExecutionRun };
}
const admissionScope = wakeQueue.createAdmissionTransactionScope(
agent.companyId,
tx as unknown as Db,
);
const admission = await wakeQueue.admitWakeBehindIssueExecution(
admissionScope,
{
companyId: agent.companyId,
issueId: issue.id,
agentId,
agentNameKey,
issueExecutionAgentNameKey: issue.executionAgentNameKey,
activeExecutionRun: {
id: activeExecutionRun.id,
agentId: activeExecutionRun.agentId,
status: activeExecutionRun.status,
contextSnapshot: activeExecutionRun.contextSnapshot,
wakeupRequestId: activeExecutionRun.wakeupRequestId,
},
allowRunCoalescing: isConversation(issue) ? false : opts.allowRunCoalescing,
durableReceipt: receiptRequest
? {
id: receiptRequest.id,
requestedAt: receiptRequest.requestedAt,
}
: undefined,
reason,
liveRunExecutions,
wakeCommentId,
forceFreshSession:
enrichedContextSnapshot.forceFreshSession === true,
contextSnapshot: enrichedContextSnapshot,
source,
triggerDetail,
payload,
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
},
);
if (admission.kind === "coalesced") {
return {
kind: "coalesced" as const,
run: admission.run as typeof heartbeatRuns.$inferSelect,
};
}
if (admission.kind === "deferred") {
return { kind: "deferred" as const, admitted: true };
}
// admission.kind === "proceed": no active run absorbed this wake,
// so fall through to the ordinary queue path below.
}
// PAP-13775: no live run holds the lock, so this wake would start a
// fresh adapter session. If this agent's recent runs on this issue
// keep succeeding without any issue-visible progress and the wake
// carries no new information, hold it back for an escalating cooldown
// so external pollers/reconcilers can't storm full-price sessions.
// Server-side recovery retries insert runs directly and never reach
// this gate. A verified durable case receipt carries a new phase;
// its agent-authored instruction comment is not an event-free wake.
if (
!caseRequest &&
isThrottleCandidateIssueRewake({
reason,
wakeCommentId: wakeCommentId ?? null,
requestedByActorType: opts.requestedByActorType ?? null,
forceFreshSession:
enrichedContextSnapshot.forceFreshSession === true,
hasExplicitResume: Boolean(explicitResumeSession),
declaredContinuationKind: readNonEmptyString(parseObject(payload).continuationKind),
})
) {
const throttleNow = new Date();
const notNativeStatusCommitterActivity = sql`not (${activityLog.actorType} = 'system' and ${activityLog.actorId} = ${NATIVE_STATUS_COMMITTER_ACTOR_ID})`;
const recentTerminalRuns = await tx
.select({
id: heartbeatRuns.id,
status: heartbeatRuns.status,
finishedAt: heartbeatRuns.finishedAt,
})
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.companyId, agent.companyId),
eq(heartbeatRuns.agentId, agentId),
sql`${heartbeatRuns.finishedAt} is not null`,
gte(
heartbeatRuns.finishedAt,
new Date(throttleNow.getTime() - ISSUE_REWAKE_LOOKBACK_MS),
),
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`,
),
)
.orderBy(desc(heartbeatRuns.finishedAt))
.limit(ISSUE_REWAKE_RUN_SAMPLE_LIMIT);
if (recentTerminalRuns.length > 0) {
const sampleRunIds = recentTerminalRuns.map(
(sampleRun) => sampleRun.id,
);
const progressRows = await tx
.select({ runId: activityLog.runId })
.from(activityLog)
.where(
and(
eq(activityLog.companyId, agent.companyId),
eq(activityLog.entityType, "issue"),
eq(activityLog.entityId, issue.id),
inArray(activityLog.runId, sampleRunIds),
inArray(
activityLog.action,
ISSUE_PROGRESS_ACTIVITY_ACTIONS,
),
notNativeStatusCommitterActivity,
),
);
const lastRunFinishedAt =
recentTerminalRuns[0]?.finishedAt ?? null;
const newInputRows = lastRunFinishedAt
? await tx
.select({ id: activityLog.id })
.from(activityLog)
.where(
and(
eq(activityLog.companyId, agent.companyId),
eq(activityLog.entityType, "issue"),
eq(activityLog.entityId, issue.id),
gt(activityLog.createdAt, lastRunFinishedAt),
sql`coalesce(${activityLog.details}->>'origin', '') <> 'fast_response'`,
notNativeStatusCommitterActivity,
inArray(
activityLog.action,
ISSUE_NEW_INPUT_ACTIVITY_ACTIONS,
),
wakeCommentId && opts.requestedByActorType === "agent"
? ne(activityLog.actorType, "agent")
: undefined,
),
)
.limit(1)
: [];
const throttleDecision = evaluateIssueRewakeThrottle({
now: throttleNow,
recentTerminalRuns,
runIdsWithIssueProgress: new Set(
progressRows
.map((row) => row.runId)
.filter((runId): runId is string => Boolean(runId)),
),
// For an agent comment wake, the query excludes agent-authored
// activity while preserving genuinely new user/system input.
// Presentation/author metadata therefore cannot smuggle human
// wake privilege, nor can it mask an actual human response.
hasNewIssueInputSinceLastRun: newInputRows.length > 0,
});
if (throttleDecision.blocked) {
await tx.insert(agentWakeupRequests).values({
...durableReceiptFields,
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason: "issue_rewake_throttled",
payload: {
...(payload ?? {}),
issueId,
heartbeatSkip: {
reason: "issue_rewake_throttled",
requestedReason: reason,
noProgressStreak: throttleDecision.noProgressStreak,
cooldownMs: throttleDecision.cooldownMs,
lastRunFinishedAt:
throttleDecision.lastRunFinishedAt.toISOString(),
nextAllowedAt:
throttleDecision.nextAllowedAt.toISOString(),
},
},
status: "skipped",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
finishedAt: throttleNow,
});
return { kind: "skipped" as const };
}
}
}
const dailyCapBlock = await getHeartbeatDailyCapBlock(
agent,
policy,
{},
tx,
);
if (dailyCapBlock) {
if (caseRequest) waitForCaseAdmission(dailyCapBlock.reason, "Case handoff is waiting for the agent's daily limit to clear");
if (executionWaitRequestId && executionBlocker) {
continuationWait = { reason: dailyCapBlock.reason,
message: "The agent has reached its daily limit. Your message is saved until work can resume." };
return deferBlockedExecution(executionBlocker);
}
if (opts.idempotencyKey?.startsWith(CONVERSATION_COMMENT_KEY_PREFIX)) throw dailyLimitConversationRefusal(dailyCapBlock);
const now = new Date();
await tx.insert(agentWakeupRequests).values({
...durableReceiptFields,
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason: dailyCapBlock.reason,
payload: {
...(payload ?? {}),
heartbeatSkip: {
reason:
"Per-agent heartbeat daily cap reached before adapter invocation.",
observed: dailyCapBlock.observed,
limit: dailyCapBlock.limit,
},
},
status: "skipped",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
finishedAt: now,
});
if (source === "timer") {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
await tx
.update(agents)
.set({
lastHeartbeatAt: now,
updatedAt: now,
})
.where(eq(agents.id, agentId));
}
return { kind: "skipped" as const };
}
const explicitContinuation = await admitExplicitNativeContinuation({
db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id,
agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId,
reason: taskRetryActorId ? "task_retry" : durableRequest && !failedChatRetry ? "issue_commented" : reason,
commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId,
queuedCommentInterruptId: opts.queuedCommentInterruptId,
queuedCommentRequestId: opts.queuedCommentRequestId,
taskRetryRequestId: opts.taskRetryRequestId,
...(durableRequest?.resumeDeferred && !failedChatRetry ? { durableChatRequest: durableRequest } : {}),
interactionPayload: payload,
onBlocked: (reason, message) => { continuationBlocked = true; continuationWait = { reason, message }; },
});
if (!explicitContinuation && (executionBlocker || (continuationBlocked && !verifiedConversationReset)) && !ownsReconstructionContinuation)
return deferBlockedExecution(continuationHold());
if (explicitContinuation) {
enrichedContextSnapshot.forceFreshSession = true;
enrichedContextSnapshot.previousRunId = explicitContinuation.previousRunId;
enrichedContextSnapshot.explicitUserContinuation = explicitContinuation;
}
if (opts.queuedCommentInterruptId && issue.status === "done") {
const reopened = await issuesSvc.updateForCompany(issue.id, issue.companyId,
{ status: "todo", executionState: null }, tx, issueActivityPublications, issuePostCommitActions);
if (!reopened) throw conflict("The task changed before its saved message could start");
await logActivity(tx as unknown as Db, {
companyId: issue.companyId, actorType: "user", actorId: opts.requestedByActorId!,
action: "issue.updated", entityType: "issue", entityId: issue.id,
details: { status: "todo", previousStatus: "done", source: "queued_comment_send", queueId: opts.queuedCommentInterruptId },
}, issueActivityPublications);
}
const wakeupValues = {
...durableReceiptFields,
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason,
payload,
status: "queued" as const,
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
};
// A deferred durable chat wake already owns this ID and source.
// Promote that receipt in the same transaction as its successor.
const wakeupRequest = durableRequest && executionWaitRequestId === durableRequest.id
? await tx.update(agentWakeupRequests).set({
payload, status: "queued", error: null, finishedAt: null, updatedAt: new Date(),
}).where(and(eq(agentWakeupRequests.id, durableRequest.id),
eq(agentWakeupRequests.companyId, agent.companyId),
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId)))
.returning().then(rows => rows[0])
: await tx.insert(agentWakeupRequests).values(wakeupValues).returning().then(rows => rows[0]);
if (!wakeupRequest) return { kind: "deferred" as const };
// A handoff changes the executor, not the owner of saved user input.
// Validate its exact stopped source while the issue row is locked;
// unrelated agents and dedicated continuations keep their own wakes.
const interruptedRunId = readNonEmptyString(enrichedContextSnapshot.interruptedRunId);
const handoffSource = source === "assignment" && reason === "issue_assigned" &&
issue.assigneeAgentId === agentId && interruptedRunId && isUuidLike(interruptedRunId)
? await tx.select({ agentId: heartbeatRuns.agentId }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, interruptedRunId), eq(heartbeatRuns.companyId, issue.companyId),
eq(heartbeatRuns.status, "cancelled"), eq(heartbeatRuns.errorCode, "issue_reassigned"),
ne(heartbeatRuns.agentId, agentId),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`,
or(isNull(heartbeatRuns.nativeIssueId), eq(heartbeatRuns.nativeIssueId, issue.id)),
)).then(rows => rows[0] ?? null)
: null;
// External input owns an exact authorized source batch. Board input
// must retain its separate receipt, including during normal resume.
const canCoalesceComments = !durableRequest && !isConversation(issue) && opts.allowRunCoalescing !== false;
// A resumed receipt already passed admission under this issue lock.
// Consume it with its successor even when chat keeps other messages
// in separate turns, or finalization will deliver it a second time.
const pendingComments =
(executionWaitRequestId || canCoalesceComments) &&
(ownsReconstructionContinuation || !(await getExecutionBlocker(tx as unknown as Db, issue.companyId, issue.id)))
? await tx
.select()
.from(agentWakeupRequests)
.where(
and(
eq(agentWakeupRequests.companyId, issue.companyId),
inArray(agentWakeupRequests.agentId, handoffSource ? [agentId, handoffSource.agentId] : [agentId]),
eq(agentWakeupRequests.status, "deferred_issue_execution"),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
// Chat Retry already pins its complete authorized source
// batch. Keep separate board input for its own admission.
canCoalesceComments || (opts.taskRetryRequestId && !failedChatRetry)
? undefined : eq(agentWakeupRequests.id, executionWaitRequestId!),
),
)
.orderBy(asc(agentWakeupRequests.requestedAt))
: [];
const adoptedComments = pendingComments.filter((wake) => {
if (wake.id === executionWaitRequestId) return true;
if (wake.id === opts.queuedCommentInterruptId || wake.id === opts.queuedCommentRequestId) return true;
const deferredPayload = parseObject(wake.payload);
const deferredContext = parseObject(
deferredPayload[DEFERRED_WAKE_CONTEXT_KEY],
);
// Dedicated interaction wakes carry their own source and session
// contract. ID-only adoption must not erase that continuation.
return (
// Durable chat work must keep its receipt, actor, source, and
// session contract through normal promotion and authorization.
!wake.idempotencyKey?.startsWith("chat-inbound:") &&
!isInteractionResolutionWakePayload(deferredPayload) &&
!hasInteractionContinuationWakeContext(deferredContext) &&
["issue_commented", "issue_reopened_via_comment"].includes(String(deferredContext.wakeReason ?? wake.reason)) &&
queuedCommentIdsFromWakePayload(wake.payload).length > 0
);
});
let adoptedCommentIds = [
...new Set([
...adoptedComments.flatMap((wake) =>
queuedCommentIdsFromWakePayload(wake.payload),
),
...queuedCommentIdsFromRunContext(enrichedContextSnapshot),
]),
];
if (opts.queuedCommentRequestId || (opts.taskRetryRequestId && !failedChatRetry) || handoffSource) {
adoptedCommentIds = await undeliveredLegacyUserCommentIds(tx as unknown as Db,
agent.companyId, issueId, agentId, adoptedCommentIds);
}
await notifyChatRunMilestones(tx, { companyId: agent.companyId, issueId: issueId ?? undefined });
const newRun = await tx
.insert(heartbeatRuns)
.values({
...(explicitContinuation ? { id: explicitContinuationRunId } : {}),
companyId: agent.companyId,
agentId,
scopeKind: readNonEmptyString(enrichedContextSnapshot.issueId) ? "issue" : "company",
issueId: readNonEmptyString(enrichedContextSnapshot.issueId),
invocationSource: source,
triggerDetail,
status: "queued",
responsibleUserId: await resolveQueuedResponsibleUserId(),
wakeupRequestId: wakeupRequest.id,
retryOfRunId: failedChatRetry
? durableRequest!.failedRunRetry!.failedRunId
: opts.failedRunId ?? automaticParentRunId,
contextSnapshot: adoptedComments.length
? withQueuedCommentIdsInRunContext(
enrichedContextSnapshot,
adoptedCommentIds,
adoptedComments.map((wake) => wake.id),
)
: enrichedContextSnapshot,
sessionIdBefore: explicitContinuation ? null : sessionBefore,
continuationAttempt,
...(reconciledSourceRunId
? { retryOfRunId: reconciledSourceRunId }
: {}),
...(reconciledRestoreRetryCount !== null ? {
scheduledRetryAttempt: reconciledRestoreRetryCount,
scheduledRetryReason: "transient_failure",
} : {}),
})
.returning()
.then((rows) => rows[0]);
if (ownsReconstructionContinuation) {
await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
continuationDelivery: "delivered", continuationRunId: newRun.id,
})}::jsonb` }).where(and(eq(issueRecoveryActions.companyId, issue.companyId),
eq(issueRecoveryActions.id, String(enrichedContextSnapshot.recoveryActionId)),
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`));
}
if (continuationToDeliver)
await deliverContinuationAsRun(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id, runId: newRun.id,
continuation: continuationToDeliver });
else if (reconciledSourceRunId)
await foldWaitingRetriesIntoRecordedContinuation(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id,
runId: newRun.id, recoveryActionId: readNonEmptyString(enrichedContextSnapshot.recoveryActionId), sourceRunId: reconciledSourceRunId });
await tx
.update(agentWakeupRequests)
.set({
runId: newRun.id,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
if (failedChatRetry && opts.taskRetryRequestId) {
await tx.update(agentWakeupRequests).set({
status: "coalesced", runId: newRun.id, finishedAt: new Date(), updatedAt: new Date(),
}).where(and(eq(agentWakeupRequests.companyId, issue.companyId),
eq(agentWakeupRequests.id, opts.taskRetryRequestId),
eq(agentWakeupRequests.status, "deferred_issue_execution")));
}
if (adoptedComments.length) {
await tx
.update(agentWakeupRequests)
.set({
status: "coalesced",
runId: newRun.id,
finishedAt: new Date(),
updatedAt: new Date(),
})
.where(
inArray(
agentWakeupRequests.id,
adoptedComments.map((wake) => wake.id),
),
);
await tx
.update(agentWakeupRequests)
.set({
payload: withQueuedCommentIdsInWakePayload(payload, adoptedCommentIds),
})
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
}
// executionRunId is NOT stamped here (enqueueWakeup queues the run but
// doesn't start it). It will be stamped in claimQueuedRun() once the run
// transitions to "running" — Fix A (lazy locking).
return { kind: "queued" as const, run: newRun };
}))
.catch((error) => {
if (isExternalChatWaitAuthorizationContention(error))
return { kind: "deferred" as const };
throw error;
});
// The admission has committed. What follows is dispatch of saved rows.
if (outcome.kind !== "deferred" && outcome.kind !== "skipped") admission?.saved();
for (const publication of issueActivityPublications) publishActivity(publication);
await executeIssuePostCommitActions(db, issuePostCommitActions);
// Telemetry for the cancelled runs is best-effort background work.
// Fire it here and never await it: none of the lifecycle work below,
// nor this function's return, depends on it, so a slow telemetry
// lookup must not delay them.
for (const cancelledRun of cancelledRunsToEmit) {
void emitAgentTaskRun(db, cancelledRun);
}
if (outcome.kind === "durable") {
return outcome.receipt.runId ? getRun(outcome.receipt.runId) : null;
}
if (outcome.kind === "deferred" || outcome.kind === "skipped") {
return null;
}
if (outcome.kind === "coalesced") {
await attachQueuedSetupRequest(agent);
await startNextQueuedRunForAgent(agent.id);
return outcome.run;
}
if (outcome.kind === "replayed") {
if (outcome.run.status === "queued") {
await attachQueuedSetupRequest(agent);
await startNextQueuedRunForAgent(agent.id);
}
return outcome.run;
}
const newRun = outcome.run;
publishLiveEvent({
companyId: newRun.companyId,
type: "heartbeat.run.queued",
payload: {
runId: newRun.id,
agentId: newRun.agentId,
invocationSource: newRun.invocationSource,
triggerDetail: newRun.triggerDetail,
wakeupRequestId: newRun.wakeupRequestId,
},
});
await attachQueuedSetupRequest(agent);
await startNextQueuedRunForAgent(agent.id);
return newRun;
}
if (durableRequest) throw new Error("chat_inbound_wakeup_binding_denied");
const activeRuns = await db
.select()
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.agentId, agentId),
inArray(heartbeatRuns.status, [
...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES,
]),
),
)
.orderBy(desc(heartbeatRuns.createdAt));
const sameScopeQueuedRun = activeRuns.find(
(candidate) =>
candidate.status === "queued" &&
isSameTaskScope(runTaskKey(candidate), taskKey),
);
const sameScopeScheduledRetryRun = activeRuns.find(
(candidate) =>
candidate.status === "scheduled_retry" &&
isSameTaskScope(runTaskKey(candidate), taskKey),
);
const sameScopeRunningRun = activeRuns.find(
(candidate) =>
candidate.status === "running" &&
isSameTaskScope(runTaskKey(candidate), taskKey),
);
const shouldQueueFollowupForRunningWake =
Boolean(sameScopeRunningRun) &&
!sameScopeQueuedRun &&
shouldQueueFollowupForRunningIssueWake({
contextSnapshot: enrichedContextSnapshot,
wakeCommentId,
});
// Unscoped manual wakes need their own receipt and execution identity too.
const rawCoalescedTarget =
opts.allowRunCoalescing === false || opts.manualUserWake
? null
: (sameScopeQueuedRun ??
sameScopeScheduledRetryRun ??
(shouldQueueFollowupForRunningWake
? null
: (sameScopeRunningRun ?? null)));
// A reviewed chat run attests one conversation, so never fold another
// conversation's reply into it; that reply queues its own run instead.
const coalescedTargetRun = rawCoalescedTarget &&
(hasRetainedExternalChatRetry(rawCoalescedTarget.contextSnapshot) ||
wakeChatConversationsConflict(rawCoalescedTarget.contextSnapshot, enrichedContextSnapshot))
? null
: filterZombieCoalesceTarget(rawCoalescedTarget, liveRunExecutions);
if (coalescedTargetRun) {
const mergedContextSnapshot = mergeCoalescedContextSnapshot(
coalescedTargetRun.contextSnapshot,
enrichedContextSnapshot,
{
preserveExistingInteractionContinuation:
coalescedTargetRun.status === "queued" ||
coalescedTargetRun.status === "scheduled_retry",
},
);
const mergedRun = await db
.update(heartbeatRuns)
.set({
contextSnapshot: mergedContextSnapshot,
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, coalescedTargetRun.id))
.returning()
.then((rows) => rows[0] ?? coalescedTargetRun);
await db.insert(agentWakeupRequests).values({
...durableReceiptFields,
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason,
payload,
status: "coalesced",
coalescedCount: 1,
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
runId: mergedRun.id,
finishedAt: new Date(),
});
return mergedRun;
}
const queuedRunId = randomUUID();
const queueOutcome = await db.transaction(async (tx) => {
await notifyRunDispatch(tx, { companyId: agent.companyId, issueId: readNonEmptyString(enrichedContextSnapshot.issueId),
runId: readNonEmptyString(enrichedContextSnapshot.issueId) ? undefined : queuedRunId });
await tx.execute(
sql`select id from agents where id = ${agentId} and company_id = ${agent.companyId} for update`,
);
const dailyCapBlock = await getHeartbeatDailyCapBlock(
agent,
policy,
{},
tx,
);
if (dailyCapBlock) {
if (opts.idempotencyKey?.startsWith(CONVERSATION_COMMENT_KEY_PREFIX)) throw dailyLimitConversationRefusal(dailyCapBlock);
const now = new Date();
await tx.insert(agentWakeupRequests).values({
...durableReceiptFields,
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason: dailyCapBlock.reason,
payload: {
...(payload ?? {}),
heartbeatSkip: {
reason:
"Per-agent heartbeat daily cap reached before adapter invocation.",
observed: dailyCapBlock.observed,
limit: dailyCapBlock.limit,
},
},
status: "skipped",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
finishedAt: now,
});
if (source === "timer") {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
await tx
.update(agents)
.set({
lastHeartbeatAt: now,
updatedAt: now,
})
.where(eq(agents.id, agentId));
}
return { kind: "skipped" as const };
}
const wakeupRequest = await tx
.insert(agentWakeupRequests)
.values({
...durableReceiptFields,
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason,
payload,
status: "queued",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
})
.returning()
.then((rows) => rows[0]);
const newRun = await tx
.insert(heartbeatRuns)
.values({
id: queuedRunId,
companyId: agent.companyId,
agentId,
scopeKind: readNonEmptyString(enrichedContextSnapshot.issueId) ? "issue" : "company",
issueId: readNonEmptyString(enrichedContextSnapshot.issueId),
invocationSource: source,
triggerDetail,
status: "queued",
responsibleUserId: await resolveQueuedResponsibleUserId(),
wakeupRequestId: wakeupRequest.id,
contextSnapshot: enrichedContextSnapshot,
sessionIdBefore: sessionBefore,
continuationAttempt,
})
.returning()
.then((rows) => rows[0]);
const continuedIssueId = readNonEmptyString(enrichedContextSnapshot.issueId);
if (continuedIssueId && readNonEmptyString(enrichedContextSnapshot.freshContinuationIsolation)) {
await tx.execute(sql`select id from issues where company_id = ${agent.companyId} and id = ${continuedIssueId} for update`);
await foldWaitingRetriesIntoContinuation(tx as unknown as Db, { companyId: agent.companyId, issueId: continuedIssueId, runId: newRun.id,
sourceRunId: readNonEmptyString(enrichedContextSnapshot.previousRunId) });
}
await tx
.update(agentWakeupRequests)
.set({
runId: newRun.id,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
return { kind: "queued" as const, run: newRun };
});
if (queueOutcome.kind === "skipped") return null;
admission?.saved();
const newRun = queueOutcome.run;
publishLiveEvent({
companyId: newRun.companyId,
type: "heartbeat.run.queued",
payload: {
runId: newRun.id,
agentId: newRun.agentId,
invocationSource: newRun.invocationSource,
triggerDetail: newRun.triggerDetail,
wakeupRequestId: newRun.wakeupRequestId,
},
});
await attachQueuedSetupRequest(agent);
await startNextQueuedRunForAgent(agent.id);
return newRun;
}
/**
* Native status commitment deliberately persists dependency/parent wake
* intents in the same transaction as the authoritative status projection.
* Those rows are not runnable until the heartbeat scheduler has applied its
* normal policy, workspace, concurrency, and responsible-user checks. Bridge
* the durable intent into that scheduler here instead of treating a bare
* `agent_wakeup_requests` row as if it were already a queued heartbeat run.
*
* The intent is claimed before dispatch. A deterministic dispatcher actor id
* lets a later sweep recover the narrow process-crash window after the real
* wake was inserted but before the intent was linked to it. Dispatch failure
* only requeues the intent; it never changes the provider run outcome.
*/
type NativeStatusWakeDispatchInput = {
companyId?: string;
limit?: number;
staleClaimMs?: number;
requestId?: string;
onDeferredDeadline?: (at: number) => void;
onError?: (error: unknown) => void;
};
type NativeStatusWakeDelivery = { runId: string | null; status: string };
function createNativeStatusWakeBatch() {
const deliveredByIssueScope = new Map<string, NativeStatusWakeDelivery>();
return (input: NativeStatusWakeDispatchInput = {}) => dispatchNativeStatusWakeBatch(input, deliveredByIssueScope);
}
function throttledDispatchUntil(dispatch: typeof agentWakeupRequests.$inferSelect): number | null {
if (dispatch.status !== "skipped" || dispatch.reason !== "issue_rewake_throttled") return null;
const at = Date.parse(String(parseObject(parseObject(dispatch.payload).heartbeatSkip).nextAllowedAt ?? ""));
return Number.isFinite(at) ? at : null;
}
function dispatchPendingNativeStatusWakeups(input: NativeStatusWakeDispatchInput = {}) {
return createNativeStatusWakeBatch()(input);
}
async function dispatchNativeStatusWakeBatch(input: NativeStatusWakeDispatchInput,
deliveredByIssueScope: Map<string, NativeStatusWakeDelivery>,
) {
const now = new Date();
const staleClaimMs = Math.max(1_000, input.staleClaimMs ?? 60_000);
const candidates = await db
.select()
.from(agentWakeupRequests)
.where(
and(
input.companyId
? eq(agentWakeupRequests.companyId, input.companyId)
: undefined,
input.requestId ? eq(agentWakeupRequests.id, input.requestId) : undefined,
eq(agentWakeupRequests.requestedByActorType, "system"),
eq(agentWakeupRequests.requestedByActorId, "native-status-committer"),
inArray(agentWakeupRequests.status, ["queued", "claimed"]),
isNull(agentWakeupRequests.runId),
),
)
.orderBy(asc(agentWakeupRequests.requestedAt))
.limit(Math.max(1, Math.min(input.limit ?? 100, 500)));
let dispatched = 0;
let recovered = 0;
let deferred = 0;
for (const candidate of candidates) {
const payload = parseObject(candidate.payload);
const wakeContext = parseObject(payload._paperclipWakeContext);
const issueId =
readNonEmptyString(payload.issueId) ??
readNonEmptyString(payload.taskId) ??
readNonEmptyString(wakeContext.issueId) ??
null;
const dispatchActorId = `native-status-wake-dispatch:${candidate.id}`;
const existingDispatch = await db
.select()
.from(agentWakeupRequests)
.where(
and(
eq(agentWakeupRequests.companyId, candidate.companyId),
eq(agentWakeupRequests.requestedByActorType, "system"),
eq(agentWakeupRequests.requestedByActorId, dispatchActorId),
),
)
.orderBy(desc(agentWakeupRequests.requestedAt))
.limit(1)
.then((rows) => rows[0] ?? null);
const throttledUntil = existingDispatch ? throttledDispatchUntil(existingDispatch) : null;
if (throttledUntil !== null && throttledUntil > now.getTime()) {
input.onDeferredDeadline?.(throttledUntil);
deferred += 1;
continue;
}
if (existingDispatch && throttledUntil === null) {
const recoveredStatus = existingDispatch.runId
? "coalesced"
: existingDispatch.status === "deferred_issue_execution"
? "coalesced"
: existingDispatch.status;
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
await tx
.update(agentWakeupRequests)
.set({
status: recoveredStatus,
runId: existingDispatch.runId,
finishedAt:
(existingDispatch.runId ?? existingDispatch.finishedAt)
? (existingDispatch.finishedAt ?? now)
: null,
error: existingDispatch.error,
updatedAt: now,
})
.where(
and(
eq(agentWakeupRequests.id, candidate.id),
isNull(agentWakeupRequests.runId),
),
);
});
recovered += 1;
continue;
}
if (
candidate.status === "claimed" &&
candidate.claimedAt &&
now.getTime() - candidate.claimedAt.getTime() < staleClaimMs
) {
input.onDeferredDeadline?.(candidate.claimedAt.getTime() + staleClaimMs);
deferred += 1;
continue;
}
const claimed = await db
.update(agentWakeupRequests)
.set({
status: "claimed",
claimedAt: now,
error: null,
updatedAt: now,
})
.where(
and(
eq(agentWakeupRequests.id, candidate.id),
isNull(agentWakeupRequests.runId),
candidate.status === "claimed"
? eq(agentWakeupRequests.status, "claimed")
: eq(agentWakeupRequests.status, "queued"),
),
)
.returning({ id: agentWakeupRequests.id });
if (claimed.length === 0) continue;
const scopeKey = issueId
? `${candidate.companyId}:${candidate.agentId}:${issueId}:${readNativeReviewAssignmentContext(wakeContext)?.nativeReviewInteractionId ?? ""}`
: null;
const priorDelivery = scopeKey
? deliveredByIssueScope.get(scopeKey)
: null;
if (priorDelivery) {
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
await tx
.update(agentWakeupRequests)
.set({
status: "coalesced",
runId: priorDelivery.runId,
finishedAt: new Date(),
error: null,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, candidate.id));
});
recovered += 1;
continue;
}
let completedOnboardingGuard: WakeupOptions["issueStateGuard"];
if (issueId) {
const targetIssue = await db
.select({
status: issues.status,
statusVersion: issues.statusVersion,
assigneeAgentId: issues.assigneeAgentId,
})
.from(issues)
.where(
and(
eq(issues.id, issueId),
eq(issues.companyId, candidate.companyId),
),
)
.limit(1)
.then((rows) => rows[0] ?? null);
const nativeReview = candidate.reason === "native_completion_review"
? await getNativeReviewAssignment(db, {
companyId: candidate.companyId, issueId, agentId: candidate.agentId,
contextSnapshot: wakeContext,
})
: null;
const onboardingResultReport = targetIssue?.status === "done" && await isCompletedOnboardingHandoffWake(db, {
companyId: candidate.companyId, issueId, agentId: candidate.agentId,
reason: candidate.reason, contextSnapshot: wakeContext,
});
if (onboardingResultReport) completedOnboardingGuard = { assigneeAgentId: candidate.agentId, statuses: ["done"], statusVersion: targetIssue!.statusVersion };
if (
!targetIssue ||
(["done", "cancelled"].includes(targetIssue.status) && !onboardingResultReport) ||
(targetIssue.assigneeAgentId !== candidate.agentId && !nativeReview) ||
(candidate.reason === "native_completion_review" && !nativeReview)
) {
await db.transaction(async tx => {
await notifyChatInboundWakeNotices(tx, { companyId: candidate.companyId, issueId });
await tx
.update(agentWakeupRequests)
.set({
status: "skipped",
finishedAt: new Date(),
error: !targetIssue
? "Native status wake target no longer exists"
: ["done", "cancelled"].includes(targetIssue.status)
? `Native status wake target is already ${targetIssue.status}`
: "Native status wake target has a different assignee",
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, candidate.id));
});
recovered += 1;
continue;
}
}
try {
const wakeRun = await enqueueWakeup(candidate.agentId, {
allowRunCoalescing: false,
source: candidate.source as WakeupOptions["source"],
triggerDetail: (candidate.triggerDetail ??
"system") as WakeupOptions["triggerDetail"],
reason: candidate.reason,
payload,
idempotencyKey: candidate.idempotencyKey,
requestedByActorType: "system",
requestedByActorId: dispatchActorId,
...(completedOnboardingGuard ? { issueStateGuard: completedOnboardingGuard } : {}),
contextSnapshot: {
...wakeContext,
...(issueId ? { issueId, taskId: issueId } : {}),
wakeReason: candidate.reason,
source: "native_status_decision",
statusDecisionSource: "native_status_decision",
nativeStatusWakeIntentId: candidate.id,
},
}, undefined, candidate.payload?.interactionOrigin);
const delivered = await db
.select()
.from(agentWakeupRequests)
.where(
and(
eq(agentWakeupRequests.companyId, candidate.companyId),
eq(agentWakeupRequests.requestedByActorType, "system"),
eq(agentWakeupRequests.requestedByActorId, dispatchActorId),
),
)
.orderBy(desc(agentWakeupRequests.requestedAt))
.limit(1)
.then((rows) => rows[0] ?? null);
const throttledDeliveryUntil = !wakeRun && delivered ? throttledDispatchUntil(delivered) : null;
if (throttledDeliveryUntil !== null) {
await db
.update(agentWakeupRequests)
.set({ status: "queued", claimedAt: null, finishedAt: null, error: null, updatedAt: new Date() })
.where(and(eq(agentWakeupRequests.id, candidate.id), isNull(agentWakeupRequests.runId)));
input.onDeferredDeadline?.(throttledDeliveryUntil);
deferred += 1;
continue;
}
// The committer intent is the durable outbox entry. When admission is
// blocked, enqueueWakeup creates a separate deferred dispatch receipt;
// leave the original intent coalesced so the release drain cannot
// promote both rows (the original has no nativeStatusWakeIntentId
// provenance and would otherwise run once before the dispatch receipt).
const deliveredStatus = delivered?.status ?? "queued";
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
await tx
.update(agentWakeupRequests)
.set({
status: wakeRun || deliveredStatus === "deferred_issue_execution"
? "coalesced"
: deliveredStatus,
runId: wakeRun?.id ?? delivered?.runId ?? null,
finishedAt:
wakeRun || delivered?.finishedAt
? (delivered?.finishedAt ?? new Date())
: null,
error: delivered?.error ?? null,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, candidate.id));
});
if (scopeKey) {
deliveredByIssueScope.set(scopeKey, {
runId: wakeRun?.id ?? delivered?.runId ?? null,
status: wakeRun ? "coalesced" : (delivered?.status ?? "queued"),
});
}
if (wakeRun) dispatched += 1;
else deferred += 1;
} catch (error) {
input.onError?.(error);
await db
.update(agentWakeupRequests)
.set({
status: "queued",
claimedAt: null,
error:
error instanceof Error
? error.message.slice(0, 1_000)
: String(error).slice(0, 1_000),
updatedAt: new Date(),
})
.where(
and(
eq(agentWakeupRequests.id, candidate.id),
eq(agentWakeupRequests.status, "claimed"),
isNull(agentWakeupRequests.runId),
),
);
logger.warn(
{
err: error,
wakeupRequestId: candidate.id,
agentId: candidate.agentId,
},
"failed to dispatch persisted native status wake intent",
);
deferred += 1;
}
}
return { scanned: candidates.length, dispatched, recovered, deferred };
}
return {
enqueueWakeup,
dispatchPendingNativeStatusWakeups,
createNativeStatusWakeBatch,
startNextQueuedRunForAgent,
countRunningRunsForAgent,
releaseClaimBackoffsForTask,
releaseRunClaimedJustBeforeSuppression,
claimQueuedRun,
withChatControlRecoveryGate,
trackWakeup,
resumeQueuedRuns,
resumeDeferredExecutionReceipts,
resumeQueuedRunsAfterDrain,
resumeFreshQueuedRunsAtStartup,
parseHeartbeatPolicy,
nextTimerHeartbeatAt,
claimDueTimerHeartbeat,
};
}