mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 22:56:58 +01:00
DottaandClaude Opus 5.5
8475826d2d
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user's follow-up comment on a task must reach the agent exactly once, through restarts (invariant 4) > - The final unmasked local chaos storm (96 tasks, 31 restarts) passed 94 tasks. The 2 failures were both lost user comments > - In t69, a restart killed the server between saving the comment and writing its wake. In t72, a comment's own run was cancelled as stale, because the earlier run completed the task first without reading the comment > - This pull request adds a reconciler rule that gives a lost comment its wake, and saves the comment before a stale cancellation > - The benefit is that a user's message is never silently dropped when a restart or a completion race hits at the wrong moment ## Linked Issues or Issue Description Refs #19956 (work-intent outbox), #19944, #19831 (plan, invariant 4). **What happened?** In chaos wave `w-20261108T055444-s9401` (local, unmasked, codex and claude, 30-min K13 storm, master `46bf913fb9`), 94 of 96 tasks passed. Two failed: - **t69, R1 `rested_not_done`.** The user's comment was saved at 06:14:59.358 and reopened the task to `todo`. A K1 SIGKILL fired 28 ms later, before the wake was written. No wake row exists. Recovery skips a `todo` task whose last run succeeded, so the task sat with no run until the wave ended. - **t72, R4 `concurrent_runs`.** A comment posted during a run queued its own run. That run was claimed at 06:16:04.874. The first run's completion marked the task `done` 2 ms later. The comment's run was then cancelled with `continuation_task_ownership_changed` before dispatch, and the comment was lost. **Expected behavior** Each accepted user comment is delivered once, unless the task is cancelled or reassigned, or a person closes it. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay w-20261108T055444-s9401/t69` (or `.../t72`). ## What Changed - `recovery/ownership-invariant-report.ts`: a new reconciler-owned I4 rule, `user_comment_without_wake`. It finds a user comment that meets all of these conditions: - it is newer than every run of the task; - the task is assigned, is not a conversation, has no live run or lock, and has at least one earlier run; - no wake request carries the comment. A comment younger than 30 s is not reported yet, because it may still get its normal wake. Its end-of-grace time is returned as `nextDueAt`. - `native-runtime/native-finalization-reconciler.ts`: the backstop step is now exported as `applyOwnershipReportBackstops`. For each such comment it saves the missing outbox receipt (`executionWait: wake_lost`) and promotes it. - `work-intent-outbox.ts`: new `saveUndeliveredUserComments`, which the reconciler rule and the stale-run path both use. It saves only live user comments that no completed run received. It writes nothing for a task that is reassigned, is a conversation, or has a status the caller does not allow. - `heartbeat/queue.ts`: the saved-receipt pass that already sends pause-saved messages also sends `wake_lost` receipts. It uses the same per-comment consumption check and the same chat-input exclusion. - `heartbeat/stale-comment-continuation.ts` and `heartbeat.ts`: before a stale continuation is cancelled, its undelivered user comments are saved as a deferred receipt. The release drain then reopens the `done` task and delivers them once. This applies only when an agent run that started before the comment completed the task after it. A task that a person marks done is left closed. - `tests/restart-chaos/grader.ts`: R4 ignores runs cancelled before dispatch (`continuation_task_ownership_changed`) when it looks for overlapping execution. Those runs never executed. Comment consumption still counts them. ## Verification Each focused file ran alone in one process (`--maxWorkers=1`): - `heartbeat-held-comment-wake.test.ts` (22 pass). This includes two new cases: - t69 shape: the reconciler rule fires, the backstop delivers the comment once, and a second pass does nothing; - t72 shape: the stale run is cancelled, the task reopens, and the comment is delivered once. Both fail with the rule or the save disabled, and pass with them. - `ownership-reconciliation-report.test.ts` (23 pass). It includes the rule's boundaries: a carried comment, a too-young comment (deadline only) and an already-read comment. - `tests/restart-chaos/grader.test.ts` (17 pass). The new R4 case fails with the old grader. A cancelled run that is not a pre-dispatch cancellation still fails R4. - Unchanged and passing: comment wake batching (27), refused assignment wake (3), wake-queue postgres adapter (103), stale queue invalidation (68), deferred receipt work (45). The first version also reopened a task that a person had marked done, which broke one deferred-receipt test. That case is now excluded. - The server package typecheck and the restart-chaos harness typecheck pass. - Same-session chaos context on master: K6 crash loop `w-20261108T063326-s9402`: 6 of 6 pass. The K4 long-hold wave is still running. After merge, I'll replay only the targeted fault shapes (t69 and t72). ## Risks - The reconciler rule runs on each report pass. It is bounded by the 1-hour lookback and the per-kind row cap, and it excludes conversation tasks, so chat input keeps its own authorization. - A comment younger than 30 s waits up to the end of its grace period before the backstop acts. - The stale-run save reopens a `done` task only for a comment that the completing run could not have read. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
5241 lines
237 KiB
TypeScript
5241 lines
237 KiB
TypeScript
import { CONTEXT_COMPACTION_WAKE_REASON, ContextCompactionError, isContextCompactionRun, readContextCompactionRequest } from "../context-compaction.js";
|
|
import { deliveredQueuedCommentIds } from "../queued-comment-delivery.js";
|
|
import { deferredWorkIntentWait } from "../work-intent-outbox.js";
|
|
import { createHeartbeatTimerPolicy } from "./timer-policy.js";
|
|
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
|
|
import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../deferred-execution-wait.js";
|
|
import { notifyDeferredExecutionWork } from "../deferred-execution-notifications.js";
|
|
import type { DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js";
|
|
import { notifyRunDispatch } from "../run-dispatch-notifications.js";
|
|
import { notifyStatusCardIssueChange } from "../status-card-notifications.js";
|
|
import { notifyRoutineExecutionStatus } from "../routine-execution-status-notifications.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
|
|
import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js";
|
|
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
|
|
import { notifyUnsafeWorkspaceExportRecovery, notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
|
|
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
|
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
|
|
import { notifyScheduledRetryCancellation } from "../scheduled-retry-cancellation-notifications.js";
|
|
import { notifyChatResponseAvailability } from "../chat-response-availability.js";
|
|
import { notifyChatInboundWakeNotices } from "../chat-inbound-wake-notice-notifications.js";
|
|
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
|
|
import { assertTaskRetryConversationAllowed, readTaskRetryIntent, readTaskRetryBoardMessageOrigin, retireSupersededTaskRetry, taskRetryMutationAllowed } from "../task-retry.js";
|
|
import { insertIssueComment } from "../issue-comment-writes.js";
|
|
import { withExternalAdmissionGuard, assertNoExternalOverlap } from "../../modules/external-agents/index.js";
|
|
import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js";
|
|
import { normalizeAgentNameKey } from "./retries.js";
|
|
import { publishActiveDotComment } from "../dot-assignment-follow-up.js";
|
|
import { publishActiveMuseComment } from "../muse-assignment-follow-up.js";
|
|
import {
|
|
type WakeupOptions,
|
|
mergeCoalescedContextSnapshot,
|
|
enrichWakeContextSnapshot,
|
|
hasInteractionContinuationWakeContext,
|
|
isInteractionResolutionWakePayload,
|
|
} from "./run-preparation.js";
|
|
import { resolveExecutionWorkspaceReuseRequestForIssue } from "./workspaces.js";
|
|
import {
|
|
CHAT_COMPLETION_WAKE_REASON,
|
|
isCompletedOnboardingHandoffWake,
|
|
} from "../chat-completion-delivery.js";
|
|
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
|
|
import { readQueuedInteractionResponse } from "../queued-interaction-response.js";
|
|
import {
|
|
isConversation,
|
|
isConversationReset,
|
|
isConversationExecutionWake,
|
|
isWaitingConversation,
|
|
} from "../agent-conversations.js";
|
|
import { legacyControllerClaim } from "../legacy-controller-lease.js";
|
|
import {
|
|
admitExplicitNativeContinuation,
|
|
undeliveredLegacyUserCommentIds,
|
|
} from "../explicit-native-continuation.js";
|
|
import { canRetryStoppedRun } from "../cancelled-native-startup.js";
|
|
import { getExecutionBlocker, hasPendingRunCleanup } from "../execution-blocker.js";
|
|
import { nativeManualAdmissionHoldsExecution } from "../native-runtime/native-runner-ownership.js";
|
|
import { taskChatRetrySourceCleanupPending } from "../chat-channels/task-retry-authority.js";
|
|
import { validateNativeWorkspaceReconstruction } from "../native-runtime/native-workspace-reconstruction.js";
|
|
import { isConversationAdapter } from "../conversation-continuation.js";
|
|
import { recordExecutionWait } from "../execution-wait.js";
|
|
import {
|
|
getNativeReviewAssignment,
|
|
readNativeReviewAssignmentContext,
|
|
} from "../native-runtime/native-review-participant.js";
|
|
import { claimQueuedNativeReviewRun } from "../native-runtime/native-review-dispatch.js";
|
|
import { adapterExecutionControls } from "../adapter-execution-control.js";
|
|
import { preserveTaskRemoteWorkspaceWriter } from "../native-runtime/native-failed-workspace-recovery.js";
|
|
import { executionFailureRetryCount } from "../execution-recovery-attempt.js";
|
|
import {
|
|
slackMentionAllowsRunStart,
|
|
resumeDeferredChatWake,
|
|
assertDurableChatWakeupReceipt,
|
|
assertDurableChatWakeupRequest,
|
|
withDurableChatAdmission,
|
|
authorizeFailedChatRunRetryWake,
|
|
FailedChatRunRetryAuthorizationError,
|
|
unadmittedChatWakeupCondition,
|
|
} from "../durable-chat-wakeup.js";
|
|
import { agentService } from "../agents.js";
|
|
import { normalizeLegacyRunnerProvider } from "@paperclipai/adapter-utils";
|
|
import { randomUUID } from "node:crypto";
|
|
import {
|
|
and,
|
|
asc,
|
|
desc,
|
|
eq,
|
|
gt,
|
|
gte,
|
|
inArray,
|
|
isNull,
|
|
lte,
|
|
ne,
|
|
notInArray,
|
|
or,
|
|
sql,
|
|
} from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
agents,
|
|
caseHandoffs,
|
|
cases,
|
|
agentWakeupRequests,
|
|
activityLog,
|
|
chatConversations,
|
|
chatEndpoints,
|
|
companies,
|
|
executionWorkspaces,
|
|
heartbeatRunEvents,
|
|
heartbeatRuns,
|
|
issueComments,
|
|
issueRecoveryActions,
|
|
issueRelations,
|
|
issues,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
conflict,
|
|
HttpError,
|
|
notFound,
|
|
} from "../../errors.js";
|
|
import { logger } from "../../middleware/logger.js";
|
|
import { isUniqueViolation } from "../../db-errors.js";
|
|
import { publishLiveEvent } from "../live-events.js";
|
|
import { allocateHeartbeatRunEventSeq } from "../heartbeat-run-events.js";
|
|
import {
|
|
queuedCommentIdsFromWakePayload,
|
|
queuedCommentIdsFromRunContext,
|
|
withQueuedCommentIdsInWakePayload,
|
|
withQueuedCommentIdsInRunContext,
|
|
} from "../issue-queued-comment-queue.js";
|
|
import { parseObject } from "../../adapters/utils.js";
|
|
import { isExternalChatWaitAuthorizationContention } from "../native-runtime/chat-attachment-reuse.js";
|
|
import { emitAgentTaskRun } from "../agent-task-run-telemetry.js";
|
|
import {
|
|
CHAT_CONTROL_RECOVERY_ADMISSION_KEY,
|
|
CHAT_CONTROL_RECOVERY_STOP_CODE,
|
|
CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE,
|
|
chatControlRecoveryAdmission,
|
|
readChatControlRecoveryAdmission,
|
|
readChatControlNativeParent,
|
|
readChatControlRecoveryStop,
|
|
} from "../chat-control-recovery-stop.js";
|
|
import {
|
|
ISSUE_NEW_INPUT_ACTIVITY_ACTIONS,
|
|
ISSUE_PROGRESS_ACTIVITY_ACTIONS,
|
|
ISSUE_REWAKE_LOOKBACK_MS,
|
|
ISSUE_REWAKE_RUN_SAMPLE_LIMIT,
|
|
evaluateIssueRewakeThrottle,
|
|
isThrottleCandidateIssueRewake,
|
|
} from "../issue-rewake-throttle.js";
|
|
import { logActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
|
|
import {
|
|
ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
|
|
isIssueBlockersResolvedWakePayload,
|
|
issueBlockersResolvedWakePayloadSql,
|
|
} from "../issue-dependency-wakeups.js";
|
|
import { resumeIfPauseAlreadyReleased } from "../issue-tree-resume.js";
|
|
import {
|
|
ISSUE_PAUSED_ERROR_CODE,
|
|
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
|
|
isVerifiedIssueTreeControlInteractionWake,
|
|
pauseSuppressionResult,
|
|
} from "../issue-tree-control.js";
|
|
import {
|
|
buildExecutionWorkspaceAdapterConfig,
|
|
isUnrunnableWorktreeCombo,
|
|
parseIssueExecutionWorkspaceSettings,
|
|
resolveEffectiveWorkspaceStrategyType,
|
|
resolveExecutionWorkspaceMode,
|
|
WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
|
|
WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE,
|
|
WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION,
|
|
} from "../execution-workspace-policy.js";
|
|
import { readContinuationAttempt } from "../recovery/index.js";
|
|
import { allowsIssueInteractionWake, wakeChatConversationsConflict } from "../../modules/run-dispatch/index.js";
|
|
import { withAgentStartLock } from "../agent-start-lock.js";
|
|
import {
|
|
evaluateAgentInvokability,
|
|
shouldCancelRunsForNonInvokableAgent,
|
|
type AgentOrgRow,
|
|
} from "../agent-invokability.js";
|
|
import { supportsUnscopedTimerHeartbeat } from "../heartbeat-policy.js";
|
|
import { isUuidLike } from "@paperclipai/shared";
|
|
import { retryChatControlAdmission } from "../chat-control-admission-retry.js";
|
|
|
|
import type { HeartbeatServiceOptions } from "../heartbeat.js";
|
|
import type { createHeartbeatRunState } from "./run-state.js";
|
|
import type { createHeartbeatRunPreparation } from "./run-preparation.js";
|
|
import type { HeartbeatRetryDependencies } from "./retries.js";
|
|
import type { HeartbeatRecoveryDependencies, createHeartbeatRecovery } from "./recovery.js";
|
|
import { executeIssuePostCommitActions, type IssuePostCommitAction, type issueService } from "../issues.js";
|
|
import type { issueTreeControlService } from "../issue-tree-control.js";
|
|
import type { budgetService } from "../budgets.js";
|
|
import { isBoardOwnedBudgetBlock } from "../budget-block-ownership.js";
|
|
import type { instanceSettingsService } from "../instance-settings.js";
|
|
import type { createRunDispatch } from "../../modules/run-dispatch/index.js";
|
|
import { hasRetainedExternalChatRetry, type PostCommitEffect as WakeQueuePostCommitEffect, type createWakeQueue } from "../../modules/wake-queue/index.js";
|
|
import { crashPoint } from "@paperclipai/shared/crash-points";
|
|
|
|
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
|
|
type HeartbeatRunState = ReturnType<typeof createHeartbeatRunState>;
|
|
type RunPreparation = ReturnType<typeof createHeartbeatRunPreparation>;
|
|
|
|
/** Queue policy uses the service's lifecycle effects and process-wide ownership sets. */
|
|
export interface HeartbeatQueueDependencies extends Pick<HeartbeatRetryDependencies,
|
|
"setWakeupStatus" | "appendRunEvent" | "finalizeAgentStatus" |
|
|
"getAgentInvokability" | "getWorktreeExecutionCutoff" | "applyRunDispatchPostCommitEffects">,
|
|
Pick<RunPreparation, "toAgentOrgRow" | "listCompanyAgentOrgRows" |
|
|
"resolveResponsibleUserIdForRun" | "getIssueExecutionContext" |
|
|
"getRoutineEnvForExecutionIssue" | "resolveResponsibleUserIdForRunSeed">,
|
|
Pick<HeartbeatRunState, "getAgent" | "getRun" | "resolveExplicitResumeSessionOverride" |
|
|
"resolveSessionBeforeForWakeup" | "hasResolvablePriorSessionWorkspaceForWake">,
|
|
Pick<HeartbeatRecoveryDependencies, "setRunStatus"> {
|
|
sweepPendingCleanupLeases: ReturnType<typeof createHeartbeatRecovery>["sweepPendingCleanupLeases"];
|
|
options: Pick<HeartbeatServiceOptions, "beforeChatControlRecoveryCheck">;
|
|
issuesSvc: Pick<ReturnType<typeof issueService>, "getDependencyReadiness" | "listDependencyReadiness" | "updateForCompany">;
|
|
treeControlSvc: Pick<ReturnType<typeof issueTreeControlService>, "getActivePauseHoldGate">;
|
|
budgets: Pick<ReturnType<typeof budgetService>, "getInvocationBlock">;
|
|
instanceSettings: Pick<ReturnType<typeof instanceSettingsService>, "getExperimental">;
|
|
runDispatch: Pick<ReturnType<typeof createRunDispatch>, "cancelStaleQueuedRun">;
|
|
wakeQueue: Pick<ReturnType<typeof createWakeQueue>, "createAdmissionTransactionScope" | "admitWakeBehindIssueExecution">;
|
|
activeRunExecutions: Set<string>;
|
|
activeRunExecutionPromises: Set<Promise<void>>;
|
|
activeWakeupPromises: Set<Promise<unknown>>;
|
|
liveRunExecutions: { has(id: string): boolean };
|
|
isHeartbeatRunTerminalStatus: (status: string | null | undefined) => boolean;
|
|
isSameTaskScope: (left: string | null, right: string | null) => boolean;
|
|
runTaskKey: (run: HeartbeatRun) => string | null;
|
|
filterZombieCoalesceTarget: <T extends { id: string; status: string }>(run: T | null, live: { has(id: string): boolean }) => T | null;
|
|
getSchedulingSuppression: () => Promise<{ suppressed: boolean; reason: "worktree_instance" | "database_restore_in_progress" | "task_drain" | null }>;
|
|
canStartNewExternalBotWork: (agent: typeof agents.$inferSelect) => Promise<boolean>;
|
|
executeRun: (runId: string) => Promise<void>;
|
|
releaseIssueExecutionAndPromote: (run: Pick<HeartbeatRun, "id" | "companyId">, options?: { suppressImmediateRecovery?: boolean; resumeStrandedQueue?: boolean; deferredPostCommitEffects?: WakeQueuePostCommitEffect[] }) => Promise<unknown>;
|
|
applyWakeQueuePostCommitEffects: (effects: WakeQueuePostCommitEffect[]) => Promise<void>;
|
|
publishRunLifecyclePluginEvent: (run: HeartbeatRun) => void;
|
|
cancelRunInternal: (runId: string, reason?: string,
|
|
options?: { errorCode?: string; resultJson?: Record<string, unknown> }) => Promise<unknown>;
|
|
cancelActiveForAgentInternal: (agentId: string, reason: string) => Promise<unknown>;
|
|
resumeExecutionWaitComments: (scope?: DeferredReceiptRecoveryOptions) => Promise<unknown>;
|
|
resumeQueuedCommentInterrupt: (companyId: string, queueId: string) => Promise<unknown>;
|
|
resumeSavedLegacyComments: (companyId: string, queueId: string) => Promise<unknown>;
|
|
formatIssueIdentifierLink: (identifier: string | null, fallback: string) => string;
|
|
}
|
|
|
|
|
|
|
|
|
|
export const DEFERRED_WAKE_CONTEXT_KEY = "_paperclipWakeContext";
|
|
|
|
export const EXECUTION_PATH_HEARTBEAT_RUN_STATUSES = [
|
|
"queued",
|
|
"running",
|
|
"scheduled_retry",
|
|
] as const;
|
|
|
|
|
|
export class ChatControlRecoveryUnresolvedError extends Error {
|
|
constructor() {
|
|
super(
|
|
"Run admission could not acquire its database locks after bounded retries. No provider work started. Review database contention and send a fresh request; this attempt will not automatically retry.",
|
|
);
|
|
}
|
|
}
|
|
|
|
const RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP = new Set([
|
|
CHAT_COMPLETION_WAKE_REASON,
|
|
"approval_approved",
|
|
ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
|
|
"issue_recovery_action_restored",
|
|
]);
|
|
|
|
|
|
export function shouldRequireIssueCommentForWake(
|
|
contextSnapshot: Record<string, unknown> | null | undefined,
|
|
) {
|
|
if (contextSnapshot?.skipIssueComment === true) return false;
|
|
|
|
const wakeReason = readNonEmptyString(contextSnapshot?.wakeReason);
|
|
return (
|
|
wakeReason === "issue_assigned" ||
|
|
wakeReason === "execution_review_requested" ||
|
|
wakeReason === "execution_approval_requested" ||
|
|
wakeReason === "execution_changes_requested"
|
|
);
|
|
}
|
|
|
|
async function listUnresolvedBlockerSummaries(
|
|
dbOrTx: Pick<Db, "select">,
|
|
companyId: string,
|
|
issueId: string,
|
|
unresolvedBlockerIssueIds: string[],
|
|
) {
|
|
const ids = [...new Set(unresolvedBlockerIssueIds.filter(Boolean))];
|
|
if (ids.length === 0) return [];
|
|
return dbOrTx
|
|
.select({
|
|
id: issues.id,
|
|
identifier: issues.identifier,
|
|
title: issues.title,
|
|
status: issues.status,
|
|
priority: issues.priority,
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
assigneeUserId: issues.assigneeUserId,
|
|
})
|
|
.from(issueRelations)
|
|
.innerJoin(issues, eq(issueRelations.issueId, issues.id))
|
|
.where(
|
|
and(
|
|
eq(issueRelations.companyId, companyId),
|
|
eq(issueRelations.type, "blocks"),
|
|
eq(issueRelations.relatedIssueId, issueId),
|
|
inArray(issues.id, ids),
|
|
),
|
|
)
|
|
.orderBy(asc(issues.title));
|
|
}
|
|
|
|
export function shouldDeferFollowupWakeForSameIssue(input: {
|
|
activeRunStatus: string | null | undefined;
|
|
isSameExecutionAgent: boolean;
|
|
wakeCommentId: string | null | undefined;
|
|
forceFreshSession: boolean;
|
|
}) {
|
|
// A comment follow-up or explicit fresh-session wake needs a new run boundary.
|
|
if (!input.isSameExecutionAgent) return false;
|
|
if (input.activeRunStatus !== "running") return false;
|
|
if (input.wakeCommentId) return true;
|
|
if (input.forceFreshSession) return true;
|
|
return false;
|
|
}
|
|
|
|
export function shouldQueueFollowupForRunningIssueWake(input: {
|
|
contextSnapshot: Record<string, unknown> | null | undefined;
|
|
wakeCommentId: string | null;
|
|
}) {
|
|
if (input.wakeCommentId) return true;
|
|
// A structured interaction response is new input just like a comment. It
|
|
// must run after the turn that created the interaction instead of being
|
|
// merged into that still-running turn.
|
|
if (
|
|
readNonEmptyString(input.contextSnapshot?.interactionId) &&
|
|
readNonEmptyString(input.contextSnapshot?.interactionStatus)
|
|
) {
|
|
return true;
|
|
}
|
|
const wakeReason = readNonEmptyString(input.contextSnapshot?.wakeReason);
|
|
if (wakeReason === "issue_children_completed" && (
|
|
input.contextSnapshot?.onboardingCompletion === true ||
|
|
(input.contextSnapshot?.statusDecisionSource === "native_status_decision" &&
|
|
readNonEmptyString(input.contextSnapshot?.nativeChildCompletionDecisionId))
|
|
)) return true;
|
|
return Boolean(
|
|
wakeReason && RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP.has(wakeReason),
|
|
);
|
|
}
|
|
|
|
function readNonEmptyString(value: unknown): string | null {
|
|
return typeof value === "string" && value.trim().length > 0 ? value : null;
|
|
}
|
|
|
|
export async function claimDueTimerHeartbeatForAgent(
|
|
db: Db,
|
|
agent: typeof agents.$inferSelect,
|
|
now: Date,
|
|
intervalSec: number,
|
|
) {
|
|
const dueBefore = new Date(now.getTime() - intervalSec * 1000);
|
|
const claimed = await db.transaction(async tx => {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
return tx
|
|
.update(agents)
|
|
.set({ lastHeartbeatAt: now, updatedAt: now })
|
|
.where(and(
|
|
eq(agents.id, agent.id),
|
|
eq(agents.companyId, agent.companyId),
|
|
or(
|
|
lte(agents.lastHeartbeatAt, dueBefore),
|
|
and(isNull(agents.lastHeartbeatAt), lte(agents.createdAt, dueBefore)),
|
|
),
|
|
))
|
|
.returning({ id: agents.id })
|
|
.then((rows) => rows[0] ?? null);
|
|
});
|
|
if (!claimed) return null;
|
|
return { wasFirstHeartbeat: !agent.lastHeartbeatAt };
|
|
}
|
|
|
|
|
|
/** Queue admission and dispatch share the service lifecycle callbacks and execution ownership. */
|
|
export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDependencies) {
|
|
const {
|
|
setRunStatus,
|
|
setWakeupStatus,
|
|
appendRunEvent,
|
|
releaseIssueExecutionAndPromote,
|
|
applyWakeQueuePostCommitEffects,
|
|
issuesSvc,
|
|
options,
|
|
publishRunLifecyclePluginEvent,
|
|
finalizeAgentStatus,
|
|
getAgent,
|
|
cancelRunInternal,
|
|
toAgentOrgRow,
|
|
getAgentInvokability,
|
|
budgets,
|
|
activeRunExecutions,
|
|
treeControlSvc,
|
|
runDispatch,
|
|
applyRunDispatchPostCommitEffects,
|
|
resolveResponsibleUserIdForRun,
|
|
getIssueExecutionContext,
|
|
isHeartbeatRunTerminalStatus,
|
|
liveRunExecutions,
|
|
getSchedulingSuppression,
|
|
resumeExecutionWaitComments,
|
|
getWorktreeExecutionCutoff,
|
|
resumeQueuedCommentInterrupt,
|
|
resumeSavedLegacyComments,
|
|
cancelActiveForAgentInternal,
|
|
listCompanyAgentOrgRows,
|
|
executeRun,
|
|
canStartNewExternalBotWork,
|
|
activeRunExecutionPromises,
|
|
activeWakeupPromises,
|
|
instanceSettings,
|
|
getRun,
|
|
sweepPendingCleanupLeases,
|
|
resolveExplicitResumeSessionOverride,
|
|
resolveSessionBeforeForWakeup,
|
|
hasResolvablePriorSessionWorkspaceForWake,
|
|
getRoutineEnvForExecutionIssue,
|
|
resolveResponsibleUserIdForRunSeed,
|
|
formatIssueIdentifierLink,
|
|
wakeQueue,
|
|
isSameTaskScope,
|
|
runTaskKey,
|
|
filterZombieCoalesceTarget,
|
|
} = dependencies;
|
|
|
|
const { parseHeartbeatPolicy, getHeartbeatDailyCapBlock, hasActionableTimerWork, nextTimerHeartbeatAt } = createHeartbeatTimerPolicy(db);
|
|
|
|
async function cancelQueuedRunForHeartbeatDailyCap(
|
|
run: typeof heartbeatRuns.$inferSelect,
|
|
dailyCapBlock: NonNullable<
|
|
Awaited<ReturnType<typeof getHeartbeatDailyCapBlock>>
|
|
>,
|
|
) {
|
|
const now = new Date();
|
|
const reason =
|
|
"Cancelled because the agent reached a per-day heartbeat budget cap before adapter invocation";
|
|
const cancelled = await setRunStatus(run.id, "cancelled", {
|
|
finishedAt: now,
|
|
error: reason,
|
|
errorCode: dailyCapBlock.reason,
|
|
resultJson: {
|
|
...parseObject(run.resultJson),
|
|
stopReason: dailyCapBlock.reason,
|
|
observed: dailyCapBlock.observed,
|
|
limit: dailyCapBlock.limit,
|
|
effectiveTimeoutSec: 0,
|
|
timeoutConfigured: false,
|
|
timeoutSource: "heartbeat_daily_cap_gate",
|
|
timeoutFired: false,
|
|
},
|
|
});
|
|
if (!cancelled) return null;
|
|
|
|
await setWakeupStatus(run.wakeupRequestId, "skipped", {
|
|
finishedAt: now,
|
|
error: reason,
|
|
});
|
|
|
|
await appendRunEvent(cancelled, {
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message: reason,
|
|
payload: {
|
|
reason: dailyCapBlock.reason,
|
|
observed: dailyCapBlock.observed,
|
|
limit: dailyCapBlock.limit,
|
|
},
|
|
});
|
|
|
|
await releaseIssueExecutionAndPromote(cancelled, {
|
|
suppressImmediateRecovery: true,
|
|
});
|
|
|
|
return cancelled;
|
|
}
|
|
|
|
async function markTimerHeartbeatChecked(
|
|
agentId: string,
|
|
source: WakeupOptions["source"],
|
|
) {
|
|
if (source !== "timer") return;
|
|
await db.transaction(async tx => {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
const now = new Date();
|
|
await tx.update(agents).set({ lastHeartbeatAt: now, updatedAt: now })
|
|
.where(eq(agents.id, agentId));
|
|
});
|
|
}
|
|
|
|
const claimDueTimerHeartbeat = (agent: typeof agents.$inferSelect, now: Date, intervalSec: number) =>
|
|
claimDueTimerHeartbeatForAgent(db, agent, now, intervalSec);
|
|
|
|
function issueRunPriorityRank(priority: string | null | undefined) {
|
|
switch (priority) {
|
|
case "critical":
|
|
return 0;
|
|
case "high":
|
|
return 1;
|
|
case "medium":
|
|
return 2;
|
|
case "low":
|
|
return 3;
|
|
default:
|
|
return 4;
|
|
}
|
|
}
|
|
|
|
async function listQueuedRunDependencyReadiness(
|
|
companyId: string,
|
|
queuedRuns: Array<typeof heartbeatRuns.$inferSelect>,
|
|
) {
|
|
const issueIds = [
|
|
...new Set(
|
|
queuedRuns
|
|
.map((run) =>
|
|
readNonEmptyString(parseObject(run.contextSnapshot).issueId),
|
|
)
|
|
.filter((issueId): issueId is string => Boolean(issueId)),
|
|
),
|
|
];
|
|
if (issueIds.length === 0) {
|
|
return new Map<
|
|
string,
|
|
Awaited<ReturnType<typeof issuesSvc.getDependencyReadiness>>
|
|
>();
|
|
}
|
|
return issuesSvc.listDependencyReadiness(companyId, issueIds);
|
|
}
|
|
|
|
async function countRunningRunsForAgent(agentId: string) {
|
|
const [{ count }] = await db
|
|
.select({ count: sql<number>`count(*)` })
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
eq(heartbeatRuns.status, "running"),
|
|
),
|
|
);
|
|
return Number(count ?? 0);
|
|
}
|
|
|
|
async function withChatControlRecoveryGate(
|
|
run: typeof heartbeatRuns.$inferSelect,
|
|
stage: "claim" | "dispatch",
|
|
onClear: (tx: Db) => Promise<typeof heartbeatRuns.$inferSelect | null>,
|
|
nativeRecovery = false,
|
|
) {
|
|
const issueId = readNonEmptyString(
|
|
parseObject(run.contextSnapshot).issueId,
|
|
);
|
|
if (!issueId || run.invocationSource !== "automation") return onClear(db);
|
|
await options.beforeChatControlRecoveryCheck?.({
|
|
runId: run.id,
|
|
issueId,
|
|
stage,
|
|
});
|
|
if (stage === "dispatch" && nativeRecovery) {
|
|
// Protected historical/already-admitted owners did not previously take
|
|
// this fence's locks. A fresh exact read keeps unrelated Board contention
|
|
// from turning their recovery into a new admission failure.
|
|
const [current] = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, run.id),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.agentId, run.agentId),
|
|
eq(heartbeatRuns.status, "running"),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (
|
|
current &&
|
|
current.wakeupRequestId === run.wakeupRequestId &&
|
|
parseObject(current.contextSnapshot).issueId === issueId
|
|
) {
|
|
const admission = readChatControlRecoveryAdmission(current);
|
|
const expected = readChatControlRecoveryAdmission(run);
|
|
if (
|
|
expected !== "invalid" &&
|
|
(admission === "admitted" ||
|
|
(admission === "historical" && expected === "historical"))
|
|
)
|
|
return onClear(db);
|
|
}
|
|
}
|
|
let terminal: typeof heartbeatRuns.$inferSelect | null = null;
|
|
try {
|
|
const attempt = () => db.transaction(async (tx) => {
|
|
terminal = null;
|
|
// Same queue-edit lock order, then the close committer's conversation
|
|
// row. NOWAIT releases partial locks on contention. Claim defers to the
|
|
// queue; dispatch retries this transaction before considering failure.
|
|
const [issue] = await tx
|
|
.select({ id: issues.id })
|
|
.from(issues)
|
|
.where(
|
|
and(eq(issues.id, issueId), eq(issues.companyId, run.companyId)),
|
|
)
|
|
.for("update", { noWait: true })
|
|
.limit(1);
|
|
if (!issue) return null;
|
|
if (run.wakeupRequestId)
|
|
await tx
|
|
.select({ id: agentWakeupRequests.id })
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, run.wakeupRequestId),
|
|
eq(agentWakeupRequests.companyId, run.companyId),
|
|
eq(agentWakeupRequests.agentId, run.agentId),
|
|
),
|
|
)
|
|
.for("update", { noWait: true })
|
|
.limit(1);
|
|
const [current] = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, run.id),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.agentId, run.agentId),
|
|
),
|
|
)
|
|
.for("update", { noWait: true })
|
|
.limit(1);
|
|
if (
|
|
!current ||
|
|
current.status !== (stage === "claim" ? "queued" : "running")
|
|
)
|
|
return null;
|
|
const admission = readChatControlRecoveryAdmission(current);
|
|
const expectedAdmission = readChatControlRecoveryAdmission(run);
|
|
const admissionLost =
|
|
expectedAdmission !== "historical" && admission === "historical";
|
|
if (
|
|
stage === "dispatch" &&
|
|
nativeRecovery &&
|
|
!admissionLost &&
|
|
(admission === "historical" || admission === "admitted")
|
|
)
|
|
return onClear(tx as unknown as Db);
|
|
// A new warm child can inherit a prior runner's PID during preparation.
|
|
// Those fields do not prove this run's turn was dispatched.
|
|
// Exact admitted/historical recovery retains its existing ownership;
|
|
// a marked, unadmitted bootstrap must earn admission even after crash.
|
|
const proof =
|
|
admission === "invalid" || admissionLost
|
|
? { kind: "unresolved" as const }
|
|
: await readChatControlRecoveryStop(
|
|
tx as unknown as Db,
|
|
{
|
|
companyId: run.companyId,
|
|
issueId,
|
|
agentId: run.agentId,
|
|
sourceRunId: run.id,
|
|
},
|
|
true,
|
|
);
|
|
if (proof.kind === "clear") {
|
|
if (stage === "claim" || admission === "required")
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
runnerProfileJson: {
|
|
...parseObject(current.runnerProfileJson),
|
|
[CHAT_CONTROL_RECOVERY_ADMISSION_KEY]:
|
|
chatControlRecoveryAdmission(
|
|
current,
|
|
stage === "dispatch"
|
|
? "admitted"
|
|
: admission === "admitted"
|
|
? "admitted"
|
|
: "required",
|
|
),
|
|
},
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(heartbeatRuns.id, current.id));
|
|
return onClear(tx as unknown as Db);
|
|
}
|
|
if (proof.kind === "unresolved" && stage === "claim") return null;
|
|
const now = new Date();
|
|
const stopped = proof.kind === "stopped";
|
|
const code = stopped
|
|
? CHAT_CONTROL_RECOVERY_STOP_CODE
|
|
: CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE;
|
|
const error = stopped
|
|
? "Automatic continuation stopped by the committed chat conversation close. Send a new request in chat or on the Board to start fresh work."
|
|
: "Automatic continuation source could not be verified before provider admission. Review the task and send a fresh request; this attempt will not automatically retry.";
|
|
await notifyChatRunMilestones(tx, { companyId: current.companyId, issueId });
|
|
await notifyInstructionCollection(tx, { companyId: current.companyId, runId: current.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: current.companyId, runId: current.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: current.companyId, runId: current.id });
|
|
await notifyExecutionRunWork(tx, { companyId: current.companyId, agentId: current.agentId, runId: current.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, current);
|
|
[terminal] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: stopped ? "cancelled" : "failed",
|
|
errorCode: code,
|
|
error,
|
|
finishedAt: now,
|
|
resultJson: {
|
|
...parseObject(current.resultJson),
|
|
automaticRecovery: {
|
|
code,
|
|
providerDispatched: false,
|
|
...(stopped
|
|
? {
|
|
sourceRunId: proof.sourceRunId,
|
|
conversationId: proof.conversationId,
|
|
publicationId: proof.publicationId,
|
|
}
|
|
: {}),
|
|
},
|
|
},
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, current.id),
|
|
eq(heartbeatRuns.status, current.status),
|
|
),
|
|
)
|
|
.returning();
|
|
if (!terminal) return null;
|
|
if (current.wakeupRequestId)
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: stopped ? "skipped" : "failed",
|
|
error,
|
|
finishedAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, current.wakeupRequestId),
|
|
eq(agentWakeupRequests.companyId, current.companyId),
|
|
eq(agentWakeupRequests.agentId, current.agentId),
|
|
eq(agentWakeupRequests.runId, current.id),
|
|
ne(agentWakeupRequests.status, "cancelled"),
|
|
),
|
|
);
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, current.companyId),
|
|
eq(issues.id, issueId),
|
|
eq(issues.executionRunId, current.id),
|
|
),
|
|
);
|
|
return null;
|
|
});
|
|
const result = stage === "dispatch"
|
|
? await retryChatControlAdmission(attempt)
|
|
: await attempt();
|
|
if (terminal) {
|
|
const settled = terminal as typeof heartbeatRuns.$inferSelect;
|
|
publishLiveEvent({
|
|
companyId: settled.companyId,
|
|
type: "heartbeat.run.status",
|
|
payload: {
|
|
runId: settled.id,
|
|
agentId: settled.agentId,
|
|
status: settled.status,
|
|
errorCode: settled.errorCode,
|
|
error: settled.error,
|
|
},
|
|
});
|
|
publishRunLifecyclePluginEvent(settled);
|
|
if (stage === "dispatch")
|
|
await finalizeAgentStatus(settled.agentId, "cancelled");
|
|
}
|
|
return result;
|
|
} catch (error) {
|
|
if (!isExternalChatWaitAuthorizationContention(error)) throw error;
|
|
if (stage === "dispatch") {
|
|
// No effect was admitted. Let the existing setup-failure path settle
|
|
// this attempt distinctly; it must not become a successful close.
|
|
throw new ChatControlRecoveryUnresolvedError();
|
|
}
|
|
return null;
|
|
}
|
|
}
|
|
|
|
async function claimQueuedRun(
|
|
run: typeof heartbeatRuns.$inferSelect,
|
|
companyAgents?: AgentOrgRow[],
|
|
deferredPostCommitEffects?: WakeQueuePostCommitEffect[],
|
|
deferredCancellations?: Array<() => Promise<void>>,
|
|
) {
|
|
if (run.status !== "queued") return run;
|
|
const [caseHandoffReceipt] = run.wakeupRequestId ? await db.select().from(caseHandoffs).where(and(
|
|
eq(caseHandoffs.wakeupRequestId, run.wakeupRequestId), eq(caseHandoffs.companyId, run.companyId),
|
|
)).limit(1) : [];
|
|
if (caseHandoffReceipt) {
|
|
const [current] = await db.select({ version: cases.handoffVersion, issueId: cases.workIssueId,
|
|
agentId: issues.assigneeAgentId, statusVersion: issues.statusVersion }).from(cases)
|
|
.innerJoin(issues, eq(issues.id, cases.workIssueId)).where(and(eq(cases.id, caseHandoffReceipt.caseId),
|
|
eq(cases.companyId, run.companyId), eq(issues.companyId, run.companyId))).limit(1);
|
|
if (!["pending_dispatch", "delivered"].includes(caseHandoffReceipt.state) || caseHandoffReceipt.toAgentId !== run.agentId ||
|
|
!current || current.version !== caseHandoffReceipt.version || current.issueId !== caseHandoffReceipt.issueId ||
|
|
current.agentId !== run.agentId || (!run.startedAt && current.statusVersion !== caseHandoffReceipt.appliedIssueStatusVersion)) {
|
|
const cancelled = await db.transaction(async tx => {
|
|
const [currentRun] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued")))
|
|
.for("update");
|
|
if (!currentRun) return null;
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, runId: run.id,
|
|
issueId: run.nativeIssueId ?? run.issueId ?? readNonEmptyString(parseObject(run.contextSnapshot).issueId) ?? undefined });
|
|
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExecutionRunWork(tx, { companyId: run.companyId, agentId: run.agentId, runId: run.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, run);
|
|
const [row] = await tx.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date(),
|
|
errorCode: "case_handoff_superseded", error: "This case handoff was cancelled or superseded.", updatedAt: new Date() })
|
|
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued"))).returning();
|
|
return row;
|
|
});
|
|
if (cancelled) await setWakeupStatus(run.wakeupRequestId, "cancelled", { finishedAt: new Date(), error: cancelled.error });
|
|
return null;
|
|
}
|
|
if (!(await instanceSettings.getExperimental()).enableCases) return null;
|
|
}
|
|
const caseIssueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
|
|
if (caseIssueId) {
|
|
const [pending] = await db.select().from(caseHandoffs).where(and(eq(caseHandoffs.companyId, run.companyId), eq(caseHandoffs.issueId, caseIssueId),
|
|
inArray(caseHandoffs.state, ["awaiting_release", "pending_dispatch"]))).limit(1);
|
|
if (pending && (pending.wakeupRequestId !== run.wakeupRequestId || pending.state === "awaiting_release")) return null;
|
|
}
|
|
// Claim is the first execution boundary. startedAt survives same-run
|
|
// retries; restart recovery may also enter executeRun already running.
|
|
// Neither may reinterpret an admitted, already-started message as new work.
|
|
if (!run.startedAt && !(await slackMentionAllowsRunStart(db, run))) {
|
|
const now = new Date();
|
|
const reason = "Message did not @mention the bot";
|
|
// This runs under the agent start lock. Do not use cancelRunInternal,
|
|
// which recursively starts the next run and reacquires that lock.
|
|
const cancelled = await setRunStatus(run.id, "cancelled", {
|
|
finishedAt: now,
|
|
error: reason,
|
|
errorCode: "chat_mention_required",
|
|
});
|
|
if (cancelled) {
|
|
await setWakeupStatus(run.wakeupRequestId, "skipped", { finishedAt: now, error: reason });
|
|
await appendRunEvent(cancelled, {
|
|
eventType: "lifecycle", stream: "system", level: "warn", message: reason,
|
|
});
|
|
await releaseIssueExecutionAndPromote(cancelled, {
|
|
suppressImmediateRecovery: true, deferredPostCommitEffects,
|
|
});
|
|
}
|
|
return null;
|
|
}
|
|
|
|
const agent = await getAgent(run.agentId);
|
|
if (!agent) {
|
|
await cancelRunInternal(
|
|
run.id,
|
|
"Cancelled because the agent no longer exists",
|
|
);
|
|
return null;
|
|
}
|
|
// Keep accepted work in the durable queue until setup completes.
|
|
if (isAgentAwaitingSetup(agent)) return null;
|
|
const invokability = companyAgents
|
|
? evaluateAgentInvokability(toAgentOrgRow(agent), companyAgents)
|
|
: await getAgentInvokability(agent);
|
|
if (!invokability.invokable) {
|
|
if (caseHandoffReceipt) return null;
|
|
await cancelRunInternal(
|
|
run.id,
|
|
`Cancelled because the agent is not invokable: ${invokability.reason}`,
|
|
);
|
|
return null;
|
|
}
|
|
if (!run.startedAt && !await canStartNewExternalBotWork(agent)) return null;
|
|
|
|
const context = parseObject(run.contextSnapshot);
|
|
const budgetBlock = await budgets.getInvocationBlock(
|
|
run.companyId,
|
|
run.agentId,
|
|
{
|
|
issueId: readNonEmptyString(context.issueId),
|
|
projectId: readNonEmptyString(context.projectId),
|
|
},
|
|
);
|
|
if (budgetBlock) {
|
|
if (caseHandoffReceipt) return null;
|
|
// A temporary hold (accounting that is still settling) only delays
|
|
// accepted work. Cancelling the unstarted run would drop the user's
|
|
// message, Retry, or recovery it carries; keep it queued. Settling
|
|
// accounting re-notifies run dispatch. A reached hard stop is the board's.
|
|
if (!run.startedAt && !(await isBoardOwnedBudgetBlock(db, run.companyId, budgetBlock))) return null;
|
|
await cancelRunInternal(run.id, budgetBlock.reason);
|
|
return null;
|
|
}
|
|
|
|
const dailyCapBlock = await getHeartbeatDailyCapBlock(
|
|
agent,
|
|
parseHeartbeatPolicy(agent),
|
|
{
|
|
excludeRunId: run.id,
|
|
checkRunCap: true,
|
|
checkCostCap: true,
|
|
},
|
|
);
|
|
if (dailyCapBlock) {
|
|
if (caseHandoffReceipt) return null;
|
|
await cancelQueuedRunForHeartbeatDailyCap(run, dailyCapBlock);
|
|
return null;
|
|
}
|
|
|
|
const issueId = readNonEmptyString(context.issueId);
|
|
if (issueId && activeRunExecutions.size > 0) {
|
|
// Native finalization publishes success before workspace synchronization,
|
|
// provider suspension, and lease release finish. A queued comment must
|
|
// not acquire a fresh sandbox while its predecessor still owns that work.
|
|
// The executor's finally block retries this agent after removing its owner.
|
|
const [settlingOwner] = await db.select({ id: heartbeatRuns.id })
|
|
.from(heartbeatRuns)
|
|
.where(and(
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.agentId, run.agentId),
|
|
eq(heartbeatRuns.runtimeMode, "native"),
|
|
inArray(heartbeatRuns.id, [...activeRunExecutions]),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,
|
|
))
|
|
.limit(1);
|
|
if (settlingOwner) return null;
|
|
}
|
|
if (issueId) {
|
|
const activePauseHold = await treeControlSvc.getActivePauseHoldGate(
|
|
run.companyId,
|
|
issueId,
|
|
);
|
|
const treeHoldInteractionWake =
|
|
activePauseHold &&
|
|
(await isVerifiedIssueTreeControlInteractionWake(db, {
|
|
companyId: run.companyId,
|
|
issueId,
|
|
agentId: run.agentId,
|
|
runId: run.id,
|
|
wakeupRequestId: run.wakeupRequestId,
|
|
contextSnapshot: context,
|
|
}));
|
|
if (activePauseHold && !treeHoldInteractionWake) {
|
|
const refuse = async () => {
|
|
// Record the covering hold so its release re-dispatches this work once.
|
|
await cancelRunInternal(
|
|
run.id,
|
|
"Cancelled because issue is held by an active subtree pause hold",
|
|
{ errorCode: ISSUE_PAUSED_ERROR_CODE, resultJson: pauseSuppressionResult(activePauseHold) },
|
|
);
|
|
await logActivity(db, {
|
|
companyId: run.companyId,
|
|
actorType: "system",
|
|
actorId: "system",
|
|
agentId: run.agentId,
|
|
runId: run.id,
|
|
action: "issue.tree_hold_run_interrupted",
|
|
entityType: "heartbeat_run",
|
|
entityId: run.id,
|
|
issueId: issueId,
|
|
details: {
|
|
issueId,
|
|
holdId: activePauseHold.holdId,
|
|
rootIssueId: activePauseHold.rootIssueId,
|
|
source: "heartbeat.claim_queued_run",
|
|
securityPrinciples: [
|
|
"Complete Mediation",
|
|
"Fail Securely",
|
|
"Secure Defaults",
|
|
],
|
|
},
|
|
});
|
|
// A release that committed before this refusal did not see it.
|
|
await resumeIfPauseAlreadyReleased(db, { companyId: run.companyId, issueId,
|
|
holdId: activePauseHold.holdId, wakeup: (agentId, opts) => trackWakeup(agentId, opts) });
|
|
};
|
|
// Cancellation promotes the agent's next queued run under the agent
|
|
// start lock, so a caller holding that lock settles it after release.
|
|
if (deferredCancellations) deferredCancellations.push(refuse);
|
|
else await refuse();
|
|
return null;
|
|
}
|
|
|
|
const dependencyReadiness = await issuesSvc.listDependencyReadiness(
|
|
run.companyId,
|
|
[issueId],
|
|
);
|
|
const readiness = dependencyReadiness.get(issueId);
|
|
const unresolvedBlockerCount = readiness?.unresolvedBlockerCount ?? 0;
|
|
if (
|
|
unresolvedBlockerCount > 0 &&
|
|
!allowsIssueInteractionWake(
|
|
context,
|
|
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
|
|
)
|
|
) {
|
|
await cancelQueuedRunForBlockedDependencies(
|
|
run,
|
|
issueId,
|
|
readiness?.unresolvedBlockerIssueIds ?? [],
|
|
);
|
|
logger.info(
|
|
{ runId: run.id, issueId, unresolvedBlockerCount },
|
|
"claimQueuedRun: cancelled blocked queued run",
|
|
);
|
|
return null;
|
|
}
|
|
|
|
const staleness = await runDispatch.cancelStaleQueuedRun({
|
|
runId: run.id,
|
|
companyId: run.companyId,
|
|
expectedStatus: "queued",
|
|
});
|
|
if (staleness.outcome === "cancelled" || staleness.outcome === "deferred") {
|
|
applyRunDispatchPostCommitEffects(staleness.postCommitEffects);
|
|
logger.info(
|
|
{
|
|
runId: run.id,
|
|
issueId,
|
|
outcome: staleness.outcome,
|
|
errorCode: staleness.outcome === "cancelled" ? staleness.errorCode : undefined,
|
|
},
|
|
"claimQueuedRun: withheld queued run at the execution gate",
|
|
);
|
|
return null;
|
|
}
|
|
}
|
|
|
|
const claimedAt = new Date();
|
|
const responsibleUserId = await resolveResponsibleUserIdForRun({
|
|
run,
|
|
contextSnapshot: context,
|
|
issueContext: issueId
|
|
? await getIssueExecutionContext(run.companyId, issueId)
|
|
: null,
|
|
routineEnvContext: {
|
|
routineId: null,
|
|
env: null,
|
|
responsibleUserId: null,
|
|
},
|
|
});
|
|
// All ordinary and comment claims use the same company-scoped issue
|
|
// lock. A batch may claim several runs before executeRun tracks any owner.
|
|
async function lockIssueExecutionClaim(tx: Db) {
|
|
const [owner] = issueId ? await tx.select({
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
statusVersion: issues.statusVersion,
|
|
executionRunId: issues.executionRunId,
|
|
checkoutRunId: issues.checkoutRunId,
|
|
}).from(issues).where(and(
|
|
eq(issues.id, issueId), eq(issues.companyId, run.companyId),
|
|
)).for("update") : [];
|
|
const ownsIssue = owner?.assigneeAgentId === run.agentId &&
|
|
context.wakeReason !== "source_scoped_recovery_action";
|
|
const [handoff] = issueId ? await tx.select().from(caseHandoffs).where(and(eq(caseHandoffs.companyId, run.companyId), eq(caseHandoffs.issueId, issueId),
|
|
inArray(caseHandoffs.state, ["awaiting_release", "pending_dispatch"]))).limit(1) : [];
|
|
if (handoff && (handoff.wakeupRequestId !== run.wakeupRequestId || handoff.state === "awaiting_release"))
|
|
return { ownsIssue, blocked: true };
|
|
if (caseHandoffReceipt) {
|
|
const [current] = await tx.select({ state: caseHandoffs.state, version: caseHandoffs.version,
|
|
currentVersion: cases.handoffVersion, workIssueId: cases.workIssueId,
|
|
appliedIssueStatusVersion: caseHandoffs.appliedIssueStatusVersion,
|
|
toAgentId: caseHandoffs.toAgentId, wakeupRequestId: caseHandoffs.wakeupRequestId }).from(caseHandoffs)
|
|
.innerJoin(cases, eq(cases.id, caseHandoffs.caseId)).where(and(eq(caseHandoffs.id, caseHandoffReceipt.id),
|
|
eq(caseHandoffs.companyId, run.companyId))).limit(1);
|
|
if (!current || !["pending_dispatch", "delivered"].includes(current.state) || current.version !== current.currentVersion ||
|
|
current.workIssueId !== issueId || current.toAgentId !== run.agentId || current.wakeupRequestId !== run.wakeupRequestId ||
|
|
!ownsIssue || (!run.startedAt && current.appliedIssueStatusVersion !== owner?.statusVersion))
|
|
return { ownsIssue, blocked: true };
|
|
}
|
|
if (ownsIssue && ["native_safe_replacement", "native_provider_overloaded"].includes(run.scheduledRetryReason ?? "") &&
|
|
owner.checkoutRunId && owner.checkoutRunId !== run.id) {
|
|
return { ownsIssue, blocked: true };
|
|
}
|
|
if (run.scheduledRetryReason === "native_provider_overloaded" && owner?.executionRunId &&
|
|
owner.executionRunId !== run.id && owner.executionRunId !== run.retryOfRunId) {
|
|
return { ownsIssue, blocked: true };
|
|
}
|
|
if (ownsIssue && issueId && await taskChatRetrySourceCleanupPending(tx, run, issueId)) {
|
|
return { ownsIssue, blocked: true };
|
|
}
|
|
const executionOwnerId = run.scheduledRetryReason === "native_provider_overloaded"
|
|
? run.retryOfRunId : ownsIssue ? owner?.executionRunId : null;
|
|
for (const previousRunId of new Set([executionOwnerId, ownsIssue ? run.retryOfRunId : null])) {
|
|
if (!previousRunId || previousRunId === run.id) continue;
|
|
const [previous] = await tx.select({ status: heartbeatRuns.status, agentId: heartbeatRuns.agentId,
|
|
issueId: heartbeatRuns.issueId, nativeIssueId: heartbeatRuns.nativeIssueId,
|
|
contextIssueId: sql<unknown>`coalesce(${heartbeatRuns.contextSnapshot}->>'issueId', ${heartbeatRuns.contextSnapshot}->>'taskId')`,
|
|
})
|
|
.from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, previousRunId),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
));
|
|
if (previousRunId !== executionOwnerId && (previous?.agentId !== run.agentId ||
|
|
(previous.nativeIssueId ?? previous.issueId ?? previous.contextIssueId) !== issueId)) continue;
|
|
// A terminal result can precede workspace/lease cleanup on this or
|
|
// another controller. Local absence alone is not a release receipt.
|
|
if (!isHeartbeatRunTerminalStatus(previous?.status) ||
|
|
liveRunExecutions.has(previousRunId)) {
|
|
return { ownsIssue, blocked: true };
|
|
}
|
|
if (await hasPendingRunCleanup(tx, run.companyId, previousRunId)) {
|
|
return { ownsIssue, blocked: true };
|
|
}
|
|
}
|
|
return { ownsIssue, blocked: false };
|
|
}
|
|
async function bindClaimedIssueExecution(tx: Db, ownsIssue: boolean, claimedRun: typeof heartbeatRuns.$inferSelect | null | undefined) {
|
|
if (!claimedRun || !issueId || !ownsIssue) return;
|
|
await notifyStatusCardIssueChange(tx, run.companyId);
|
|
await tx.update(issues).set({
|
|
executionRunId: claimedRun.id,
|
|
executionAgentNameKey: normalizeAgentNameKey(agent.name),
|
|
executionLockedAt: claimedAt,
|
|
updatedAt: claimedAt,
|
|
}).where(and(eq(issues.id, issueId), eq(issues.companyId, run.companyId))).catch((error: unknown) => {
|
|
// Another open task of the same routine holds its execution slot, for
|
|
// example a failed run that waits for its scheduled resume. Roll back
|
|
// this claim and keep the run queued until that slot is released.
|
|
if (isUniqueViolation(error, "issues_open_routine_execution_uq")) {
|
|
throw conflict("Another execution for this routine is already in progress");
|
|
}
|
|
throw error;
|
|
});
|
|
}
|
|
const nativeReviewContext = readNativeReviewAssignmentContext(context);
|
|
const queuedCommentIds = queuedCommentIdsFromRunContext(context);
|
|
if (
|
|
issueId &&
|
|
run.invocationSource === "automation" &&
|
|
queuedCommentIds.length > 0
|
|
)
|
|
await options.beforeChatControlRecoveryCheck?.({
|
|
runId: run.id,
|
|
issueId,
|
|
stage: "claim",
|
|
});
|
|
const queuedCommentClaim =
|
|
!nativeReviewContext && issueId && run.wakeupRequestId && queuedCommentIds.length > 0
|
|
? await db
|
|
.transaction(async (tx) => {
|
|
// Match the queue-edit lock order: issue, wake, then run. Once the
|
|
// run becomes running, a concurrent discard must observe the
|
|
// claimed wake and return an explicit conflict; if discard wins,
|
|
// this claim observes the cancelled queue and does no work.
|
|
const issueClaim = await lockIssueExecutionClaim(tx as unknown as Db);
|
|
if (issueClaim.blocked) return { kind: "stale" as const, run: null };
|
|
const wake = await tx
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, run.wakeupRequestId!),
|
|
eq(agentWakeupRequests.companyId, run.companyId),
|
|
eq(agentWakeupRequests.agentId, run.agentId),
|
|
),
|
|
)
|
|
.for("update")
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
const lockedRun = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, run.id),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.agentId, run.agentId),
|
|
),
|
|
)
|
|
.for("update")
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
if (
|
|
!wake ||
|
|
wake.status !== "queued" ||
|
|
wake.runId !== run.id ||
|
|
!lockedRun ||
|
|
lockedRun.status !== "queued" ||
|
|
lockedRun.wakeupRequestId !== wake.id
|
|
) {
|
|
return { kind: "stale" as const, run: null };
|
|
}
|
|
|
|
if (lockedRun.invocationSource === "automation") {
|
|
const admission = readChatControlRecoveryAdmission(lockedRun);
|
|
if (admission === "invalid")
|
|
return { kind: "stale" as const, run: null };
|
|
const proof = await readChatControlRecoveryStop(
|
|
tx as unknown as Db,
|
|
{
|
|
companyId: lockedRun.companyId,
|
|
issueId,
|
|
agentId: lockedRun.agentId,
|
|
sourceRunId: lockedRun.id,
|
|
},
|
|
true,
|
|
);
|
|
if (proof.kind === "unresolved")
|
|
return { kind: "stale" as const, run: null };
|
|
if (proof.kind === "stopped") {
|
|
const error =
|
|
"Automatic continuation stopped by the committed chat conversation close. Send a fresh request in chat or on the Board.";
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
|
|
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExecutionRunWork(tx, { companyId: run.companyId, agentId: run.agentId, runId: run.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, run);
|
|
const [cancelled] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
errorCode: CHAT_CONTROL_RECOVERY_STOP_CODE,
|
|
error,
|
|
finishedAt: claimedAt,
|
|
resultJson: {
|
|
...parseObject(lockedRun.resultJson),
|
|
automaticRecovery: {
|
|
code: CHAT_CONTROL_RECOVERY_STOP_CODE,
|
|
providerDispatched: false,
|
|
sourceRunId: proof.sourceRunId,
|
|
conversationId: proof.conversationId,
|
|
publicationId: proof.publicationId,
|
|
},
|
|
},
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, lockedRun.id),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
),
|
|
)
|
|
.returning();
|
|
if (!cancelled) return { kind: "stale" as const, run: null };
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "skipped",
|
|
error,
|
|
finishedAt: claimedAt,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wake.id));
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, run.companyId),
|
|
eq(issues.executionRunId, run.id),
|
|
),
|
|
);
|
|
return { kind: "cancelled" as const, run: cancelled };
|
|
}
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
runnerProfileJson: {
|
|
...parseObject(lockedRun.runnerProfileJson),
|
|
[CHAT_CONTROL_RECOVERY_ADMISSION_KEY]:
|
|
chatControlRecoveryAdmission(
|
|
lockedRun,
|
|
admission === "admitted" ? "admitted" : "required",
|
|
),
|
|
},
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(eq(heartbeatRuns.id, lockedRun.id));
|
|
}
|
|
const authoritativeIds = queuedCommentIdsFromWakePayload(
|
|
wake.payload,
|
|
);
|
|
if (authoritativeIds.length === 0) {
|
|
// Legacy/direct comment wakes carry comment ids in their ordinary
|
|
// payload and context, not in the authoritative queued-message
|
|
// envelope. Preserve their established claim path; only an
|
|
// explicitly bound queued-message envelope is subject to the
|
|
// live-comment discard gate below.
|
|
await withExternalAdmissionGuard(tx, run.companyId, run.agentId, () => assertNoExternalOverlap(tx, run.companyId, run.agentId));
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
|
|
const [claimedRun] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "running",
|
|
runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`,
|
|
...legacyControllerClaim(run.runtimeMode),
|
|
responsibleUserId,
|
|
startedAt: lockedRun.startedAt ?? claimedAt,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, lockedRun.id),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
),
|
|
)
|
|
.returning();
|
|
await bindClaimedIssueExecution(tx as unknown as Db, issueClaim.ownsIssue, claimedRun);
|
|
return claimedRun
|
|
? { kind: "claimed" as const, run: claimedRun }
|
|
: { kind: "stale" as const, run: null };
|
|
}
|
|
const commentRows = await tx
|
|
.select({
|
|
id: issueComments.id,
|
|
deletedAt: issueComments.deletedAt,
|
|
})
|
|
.from(issueComments)
|
|
.where(
|
|
and(
|
|
eq(issueComments.companyId, run.companyId),
|
|
eq(issueComments.issueId, issueId),
|
|
inArray(issueComments.id, authoritativeIds),
|
|
),
|
|
);
|
|
const liveIds = authoritativeIds.filter((commentId) => {
|
|
const comment = commentRows.find((row) => row.id === commentId);
|
|
return Boolean(comment && !comment.deletedAt);
|
|
});
|
|
if (liveIds.length === 0) {
|
|
const reason = "Queued messages were discarded before dispatch";
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
|
|
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExecutionRunWork(tx, { companyId: run.companyId, agentId: run.agentId, runId: run.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, run);
|
|
const [cancelled] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: claimedAt,
|
|
error: reason,
|
|
errorCode: "queued_comment_discarded",
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, lockedRun.id),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
),
|
|
)
|
|
.returning();
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: claimedAt,
|
|
error: reason,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wake.id));
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, run.companyId),
|
|
eq(issues.executionRunId, run.id),
|
|
),
|
|
);
|
|
return {
|
|
kind: "cancelled" as const,
|
|
run: cancelled ?? lockedRun,
|
|
};
|
|
}
|
|
|
|
await withExternalAdmissionGuard(tx, run.companyId, run.agentId, () => assertNoExternalOverlap(tx, run.companyId, run.agentId));
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "claimed",
|
|
claimedAt,
|
|
payload: withQueuedCommentIdsInWakePayload(
|
|
wake.payload,
|
|
liveIds,
|
|
),
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wake.id));
|
|
const [claimedRun] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "running",
|
|
runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`,
|
|
...legacyControllerClaim(run.runtimeMode),
|
|
responsibleUserId,
|
|
startedAt: lockedRun.startedAt ?? claimedAt,
|
|
contextSnapshot: withQueuedCommentIdsInRunContext(
|
|
lockedRun.contextSnapshot,
|
|
liveIds,
|
|
),
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, lockedRun.id),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
),
|
|
)
|
|
.returning();
|
|
await bindClaimedIssueExecution(tx as unknown as Db, issueClaim.ownsIssue, claimedRun);
|
|
return claimedRun
|
|
? { kind: "claimed" as const, run: claimedRun }
|
|
: { kind: "stale" as const, run: null };
|
|
})
|
|
.catch((error) => {
|
|
if (isExternalChatWaitAuthorizationContention(error))
|
|
return { kind: "stale" as const, run: null };
|
|
throw error;
|
|
})
|
|
: null;
|
|
if (queuedCommentClaim?.kind === "cancelled") {
|
|
await appendRunEvent(queuedCommentClaim.run, {
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message:
|
|
queuedCommentClaim.run.error ??
|
|
"Queued messages were discarded before dispatch",
|
|
});
|
|
publishLiveEvent({
|
|
companyId: queuedCommentClaim.run.companyId,
|
|
type: "heartbeat.run.status",
|
|
payload: {
|
|
runId: queuedCommentClaim.run.id,
|
|
agentId: queuedCommentClaim.run.agentId,
|
|
status: queuedCommentClaim.run.status,
|
|
invocationSource: queuedCommentClaim.run.invocationSource,
|
|
triggerDetail: queuedCommentClaim.run.triggerDetail,
|
|
error: queuedCommentClaim.run.error ?? null,
|
|
errorCode: queuedCommentClaim.run.errorCode ?? null,
|
|
startedAt: queuedCommentClaim.run.startedAt
|
|
? new Date(queuedCommentClaim.run.startedAt).toISOString()
|
|
: null,
|
|
finishedAt: queuedCommentClaim.run.finishedAt
|
|
? new Date(queuedCommentClaim.run.finishedAt).toISOString()
|
|
: null,
|
|
},
|
|
});
|
|
publishRunLifecyclePluginEvent(queuedCommentClaim.run);
|
|
// Fire-and-forget: nothing else in this path depends on the emission,
|
|
// so it must not delay the return.
|
|
void emitAgentTaskRun(db, queuedCommentClaim.run);
|
|
return null;
|
|
}
|
|
const claimed = queuedCommentClaim
|
|
? queuedCommentClaim.run
|
|
: await withChatControlRecoveryGate(run, "claim", async (tx) => {
|
|
const claimValues = {
|
|
status: "running",
|
|
runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`,
|
|
...legacyControllerClaim(run.runtimeMode),
|
|
responsibleUserId,
|
|
startedAt: run.startedAt ?? claimedAt,
|
|
updatedAt: claimedAt,
|
|
};
|
|
if (nativeReviewContext) {
|
|
if (run.scheduledRetryReason === "native_provider_overloaded") {
|
|
const predecessor = await lockIssueExecutionClaim(tx);
|
|
if (predecessor.blocked) return null;
|
|
}
|
|
return claimQueuedNativeReviewRun(tx, {
|
|
run, claimedAt, claimValues,
|
|
agentNameKey: normalizeAgentNameKey(agent.name),
|
|
});
|
|
}
|
|
return tx.transaction(async (claimTx) => {
|
|
const issueClaim = await lockIssueExecutionClaim(claimTx as unknown as Db);
|
|
if (issueClaim.blocked) return null;
|
|
if (run.wakeupRequestId) await claimTx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, run.wakeupRequestId), eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.agentId, run.agentId),
|
|
)).for("update");
|
|
const [lockedRun] = await claimTx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.agentId, run.agentId),
|
|
)).for("update");
|
|
if (!lockedRun || lockedRun.status !== "queued") return null;
|
|
await withExternalAdmissionGuard(claimTx, run.companyId, run.agentId, () => assertNoExternalOverlap(claimTx, run.companyId, run.agentId));
|
|
await notifyChatRunMilestones(claimTx, { companyId: run.companyId, runId: run.id, issueId: issueId ?? undefined });
|
|
const claimedRun = await claimTx.update(heartbeatRuns).set(claimValues).where(and(
|
|
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "queued"),
|
|
)).returning().then((rows) => rows[0] ?? null);
|
|
await bindClaimedIssueExecution(claimTx as unknown as Db, issueClaim.ownsIssue, claimedRun);
|
|
return claimedRun;
|
|
});
|
|
});
|
|
if (!claimed) return null;
|
|
|
|
publishLiveEvent({
|
|
companyId: claimed.companyId,
|
|
type: "heartbeat.run.status",
|
|
payload: {
|
|
runId: claimed.id,
|
|
agentId: claimed.agentId,
|
|
status: claimed.status,
|
|
invocationSource: claimed.invocationSource,
|
|
triggerDetail: claimed.triggerDetail,
|
|
error: claimed.error ?? null,
|
|
errorCode: claimed.errorCode ?? null,
|
|
startedAt: claimed.startedAt
|
|
? new Date(claimed.startedAt).toISOString()
|
|
: null,
|
|
finishedAt: claimed.finishedAt
|
|
? new Date(claimed.finishedAt).toISOString()
|
|
: null,
|
|
},
|
|
});
|
|
publishRunLifecyclePluginEvent(claimed);
|
|
|
|
if (!nativeReviewContext) {
|
|
crashPoint("wake.claim.after_run_running_before_wakeup_claimed");
|
|
await setWakeupStatus(claimed.wakeupRequestId, "claimed", { claimedAt });
|
|
}
|
|
|
|
// Fix A (lazy locking): stamp executionRunId now that the run is actually running,
|
|
// not at queue time. Guard is idempotent — safe if called more than once.
|
|
const claimedContext = parseObject(claimed.contextSnapshot);
|
|
const claimedIssueId = readNonEmptyString(claimedContext.issueId);
|
|
const claimedWakeReason = readNonEmptyString(claimedContext.wakeReason);
|
|
if (
|
|
!nativeReviewContext && claimedIssueId &&
|
|
claimedWakeReason !== "source_scoped_recovery_action"
|
|
) {
|
|
const claimedAgent = await getAgent(claimed.agentId);
|
|
await db
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: claimed.id,
|
|
executionAgentNameKey: normalizeAgentNameKey(claimedAgent?.name),
|
|
executionLockedAt: claimedAt,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, claimedIssueId),
|
|
eq(issues.companyId, claimed.companyId),
|
|
// Mention/context runs can touch an issue, but only the current assignee
|
|
// owns the issue execution lock shown as the active run.
|
|
eq(issues.assigneeAgentId, claimed.agentId),
|
|
["native_safe_replacement", "native_provider_overloaded"].includes(claimed.scheduledRetryReason ?? "")
|
|
? or(
|
|
isNull(issues.checkoutRunId),
|
|
eq(issues.checkoutRunId, claimed.id),
|
|
)
|
|
: undefined,
|
|
or(
|
|
isNull(issues.executionRunId),
|
|
eq(issues.executionRunId, claimed.id),
|
|
),
|
|
),
|
|
);
|
|
}
|
|
|
|
return claimed;
|
|
}
|
|
|
|
// startNextQueuedRunForAgent checks admission suppression once, then claims
|
|
// runs (sets status "running"), then dispatches each to executeRun, which
|
|
// checks suppression again before it does any work. Suppression (task
|
|
// drain, worktree mode, a database restore) can start in the gap between
|
|
// those two checks. When executeRun's check catches that, the run is
|
|
// already claimed — release it back to "queued" so it does not keep a
|
|
// running execution lock that nothing will ever process. This runs inside
|
|
// the same promise startNextQueuedRunForAgent already tracks in
|
|
// activeRunExecutionPromises, so getTaskDrainStatus() keeps reporting the
|
|
// run as active until the release finishes.
|
|
//
|
|
// The run row, the wakeup request, and the issue execution lock all guard
|
|
// the same claim, so one transaction commits all three writes together. A
|
|
// partial write (for example the run flips to "queued" but the wakeup or
|
|
// issue update then fails) would let the execution promise clear from
|
|
// activeRunExecutionPromises while the wakeup stayed "claimed" or the
|
|
// issue stayed locked to a queued run — task-drain status would then read
|
|
// quiescent while the database still held part of the old claim.
|
|
async function releaseRunClaimedJustBeforeSuppression(runId: string) {
|
|
const now = new Date();
|
|
await db.transaction(async (tx) => {
|
|
const [candidate] = await tx.select({ companyId: heartbeatRuns.companyId, contextSnapshot: heartbeatRuns.contextSnapshot })
|
|
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running"))).for("update");
|
|
if (!candidate) return;
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, runId,
|
|
issueId: readNonEmptyString(parseObject(candidate.contextSnapshot).issueId) ?? undefined });
|
|
const released = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "queued",
|
|
startedAt: null,
|
|
responsibleUserId: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running")),
|
|
)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!released) return;
|
|
|
|
if (released.wakeupRequestId) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({ status: "queued", claimedAt: null, updatedAt: now })
|
|
.where(and(eq(agentWakeupRequests.id, released.wakeupRequestId), ne(agentWakeupRequests.status, "cancelled")));
|
|
}
|
|
|
|
const context = parseObject(released.contextSnapshot);
|
|
const issueId = readNonEmptyString(context.issueId);
|
|
if (issueId) {
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, released.companyId),
|
|
eq(issues.executionRunId, released.id),
|
|
),
|
|
);
|
|
}
|
|
});
|
|
}
|
|
|
|
async function cancelQueuedRunForBlockedDependencies(
|
|
run: typeof heartbeatRuns.$inferSelect,
|
|
issueId: string,
|
|
unresolvedBlockerIssueIds: string[],
|
|
) {
|
|
const now = new Date();
|
|
const reason =
|
|
"Cancelled because issue dependencies are still blocked; Paperclip will wake the assignee when blockers resolve";
|
|
const cancelled = await setRunStatus(run.id, "cancelled", {
|
|
finishedAt: now,
|
|
error: reason,
|
|
errorCode: "issue_dependencies_blocked",
|
|
resultJson: {
|
|
...parseObject(run.resultJson),
|
|
stopReason: "issue_dependencies_blocked",
|
|
effectiveTimeoutSec: 0,
|
|
timeoutConfigured: false,
|
|
timeoutSource: "dependency_gate",
|
|
timeoutFired: false,
|
|
},
|
|
});
|
|
if (!cancelled) return null;
|
|
|
|
await setWakeupStatus(run.wakeupRequestId, "skipped", {
|
|
finishedAt: now,
|
|
error: reason,
|
|
});
|
|
|
|
await db.transaction(async tx => {
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId });
|
|
await notifyNativeStatusReassessment(tx, { companyId: run.companyId, issueId });
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, run.companyId),
|
|
eq(issues.id, issueId),
|
|
eq(issues.executionRunId, run.id),
|
|
),
|
|
);
|
|
});
|
|
|
|
await appendRunEvent(cancelled, {
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message: reason,
|
|
payload: {
|
|
issueId,
|
|
unresolvedBlockerIssueIds,
|
|
},
|
|
});
|
|
|
|
return cancelled;
|
|
}
|
|
|
|
async function resumeDeferredExecutionReceipts(scope: DeferredReceiptRecoveryOptions = {}) {
|
|
if ((await getSchedulingSuppression()).suppressed) return;
|
|
await resumeExecutionWaitComments(scope);
|
|
const cutoff = await getWorktreeExecutionCutoff();
|
|
const pendingInterrupts = await db.select({ id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId })
|
|
.from(agentWakeupRequests).innerJoin(companies, eq(companies.id, agentWakeupRequests.companyId))
|
|
.where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
|
|
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
eq(companies.status, "active"),
|
|
sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is not null`,
|
|
lte(agentWakeupRequests.updatedAt, new Date(Date.now() - 30_000)),
|
|
cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined))
|
|
.orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
|
|
for (const wake of pendingInterrupts) {
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date() }).where(and(
|
|
eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
));
|
|
await resumeQueuedCommentInterrupt(wake.companyId, wake.id).catch(err => {
|
|
scope.onError?.(err);
|
|
logger.warn({ err, queueId: wake.id }, "failed to resume interrupted comment queue");
|
|
});
|
|
}
|
|
// Work-intent outbox: a message refused by a pause waits for the pause to
|
|
// lift, not behind the run. Lifting the pause is the operator's explicit
|
|
// authorization to send what the pause refused, as Send is after a Stop:
|
|
// when the pause stopped the task's run, the release drain never promotes
|
|
// queued work behind that acknowledged stop, so the lifted pause issues the
|
|
// task's next explicit wake here. It reconsiders the exact saved receipt
|
|
// through normal admission (a refusal rewrites the receipt in place) and
|
|
// adopts it atomically, so it is delivered once. It runs before the
|
|
// stranded-queue pass, which touches each saved wake.
|
|
const pauseLifted = await db.select({ wake: agentWakeupRequests })
|
|
.from(agentWakeupRequests)
|
|
.innerJoin(issues, and(eq(issues.companyId, agentWakeupRequests.companyId),
|
|
sql`${issues.id}::text = ${agentWakeupRequests.payload}->>'issueId'`,
|
|
eq(issues.assigneeAgentId, agentWakeupRequests.agentId), isNull(issues.executionRunId),
|
|
notInArray(issues.status, ["done", "cancelled"])))
|
|
.innerJoin(companies, and(eq(companies.id, agentWakeupRequests.companyId), eq(companies.status, "active")))
|
|
.where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
|
|
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId),
|
|
// Only messages the pause path saved (never a chat-inbound receipt,
|
|
// which resumes through its own live authorization), and only pauses
|
|
// this query can see lifted. A refusal by any other gate rewrites the
|
|
// receipt's reason, so it never retries in a loop here.
|
|
// A message the reconciler saved after its wake was lost waits on no
|
|
// hold, so the same explicit wake sends it.
|
|
sql`${agentWakeupRequests.payload}->'executionWait'->>'trigger' in ('pause_lifted', 'reconciler_backstop')`,
|
|
sql`${agentWakeupRequests.payload}->'executionWait'->>'reason' in ('company.inactive', 'agent.not_invokable', 'wake_lost')`,
|
|
eq(agentWakeupRequests.requestedByActorType, "user"),
|
|
sql`coalesce(${agentWakeupRequests.idempotencyKey}, '') not like 'chat-inbound:%'`,
|
|
sql`exists (select 1 from agents pause_agent where pause_agent.id = ${agentWakeupRequests.agentId}
|
|
and pause_agent.status not in ('paused', 'terminated', 'pending_approval'))`,
|
|
sql`NOT EXISTS (SELECT 1 FROM heartbeat_runs live_issue_run
|
|
WHERE live_issue_run.company_id = ${issues.companyId}
|
|
AND (live_issue_run.context_snapshot->>'issueId' = ${issues.id}::text
|
|
OR live_issue_run.native_issue_id = ${issues.id})
|
|
AND live_issue_run.status IN ('queued', 'running', 'scheduled_retry'))`,
|
|
cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined))
|
|
.orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
|
|
const pauseLiftedIssues = new Set<string>();
|
|
for (const { wake } of pauseLifted) {
|
|
const issueKey = `${wake.companyId}:${String(wake.payload?.issueId)}`;
|
|
if (pauseLiftedIssues.has(issueKey)) continue;
|
|
pauseLiftedIssues.add(issueKey);
|
|
// Consumption is per comment. The wake adopts every saved message of
|
|
// the task for this agent, so each receipt drops the comments a
|
|
// completed run already received; a receipt left empty is consumed.
|
|
const issueKeyId = String(wake.payload?.issueId);
|
|
const saved = await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, wake.companyId), eq(agentWakeupRequests.agentId, wake.agentId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueKeyId}`));
|
|
for (const receipt of saved) {
|
|
const commentIds = queuedCommentIdsFromWakePayload(receipt.payload);
|
|
const delivered = await deliveredQueuedCommentIds(db, { companyId: wake.companyId, issueId: issueKeyId,
|
|
agentId: wake.agentId, commentIds: commentIds.filter(isUuidLike) });
|
|
if (delivered.size === 0) continue;
|
|
const remaining = commentIds.filter(commentId => !delivered.has(commentId));
|
|
const now = new Date();
|
|
await db.update(agentWakeupRequests).set(remaining.length === 0
|
|
? { status: "cancelled", finishedAt: now, updatedAt: now, error: "Queued messages were already delivered to a completed run" }
|
|
: { payload: withQueuedCommentIdsInWakePayload(parseObject(receipt.payload), remaining), updatedAt: now })
|
|
.where(and(eq(agentWakeupRequests.id, receipt.id), eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
}
|
|
const [current] = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.id, wake.id),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
if (!current) continue;
|
|
wake.payload = current.payload;
|
|
const context = parseObject(parseObject(wake.payload)[DEFERRED_WAKE_CONTEXT_KEY]);
|
|
const { executionWait: _wait, [DEFERRED_WAKE_CONTEXT_KEY]: _context, ...payload } = parseObject(wake.payload);
|
|
await enqueueWakeup(wake.agentId, {
|
|
source: wake.source as WakeupOptions["source"], triggerDetail: wake.triggerDetail as WakeupOptions["triggerDetail"],
|
|
reason: wake.reason ?? undefined, payload, contextSnapshot: context,
|
|
requestedByActorType: wake.requestedByActorType as WakeupOptions["requestedByActorType"],
|
|
requestedByActorId: wake.requestedByActorId ?? undefined,
|
|
}, wake.id).catch(err => {
|
|
if (err instanceof PersistedExecutionWaitRefusal) return;
|
|
scope.onError?.(err);
|
|
logger.warn({ err, requestId: wake.id }, "a message saved during a pause could not resume");
|
|
});
|
|
}
|
|
|
|
// A server restart or a message/cleanup race can leave a deferred wake
|
|
// after its owner has released the issue lock. Revisit it through the same
|
|
// release admission, so recovery holds and operator Stops still apply.
|
|
const strandedQueues = await db.select({ wake: agentWakeupRequests })
|
|
.from(agentWakeupRequests)
|
|
.innerJoin(issues, and(eq(issues.companyId, agentWakeupRequests.companyId),
|
|
sql`${issues.id}::text = ${agentWakeupRequests.payload}->>'issueId'`,
|
|
or(eq(issues.assigneeAgentId, agentWakeupRequests.agentId),
|
|
sql`${agentWakeupRequests.payload} #>> '{_paperclipWakeContext,wakeReason}' = 'issue_comment_mentioned'`)))
|
|
.innerJoin(companies, and(eq(companies.id, issues.companyId), eq(companies.status, "active")))
|
|
.where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
|
|
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
isNull(issues.executionRunId),
|
|
or(and(
|
|
sql`jsonb_typeof(${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}') = 'array'`,
|
|
sql`${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`,
|
|
), sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`,
|
|
issueBlockersResolvedWakePayloadSql),
|
|
sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is null`,
|
|
sql`NOT EXISTS (SELECT 1 FROM heartbeat_runs live_issue_run
|
|
WHERE live_issue_run.company_id = ${issues.companyId}
|
|
AND (live_issue_run.context_snapshot->>'issueId' = ${issues.id}::text
|
|
OR live_issue_run.native_issue_id = ${issues.id})
|
|
AND live_issue_run.status IN ('queued', 'running', 'scheduled_retry'))`,
|
|
cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined))
|
|
.orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
|
|
const releasedNativeSources = new Set<string>();
|
|
for (const { wake } of strandedQueues) {
|
|
const dependencyWake = isIssueBlockersResolvedWakePayload(wake.payload);
|
|
if (!dependencyWake && !queuedCommentIdsFromWakePayload(wake.payload).length &&
|
|
!await readQueuedInteractionResponse(db, wake.companyId, String(wake.payload?.issueId), wake.payload)) continue;
|
|
const [latest] = await db.select({
|
|
id: heartbeatRuns.id,
|
|
companyId: heartbeatRuns.companyId,
|
|
status: heartbeatRuns.status,
|
|
finishedAt: heartbeatRuns.finishedAt,
|
|
startedAt: heartbeatRuns.startedAt,
|
|
runtimeMode: heartbeatRuns.runtimeMode,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
sourceVersion: sql<string>`${heartbeatRuns}.xmin::text`,
|
|
}).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, wake.companyId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${String(wake.payload?.issueId)}`,
|
|
or(isNull(heartbeatRuns.nativeIssueId), sql`${heartbeatRuns.nativeIssueId}::text = ${String(wake.payload?.issueId)}`),
|
|
)).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date() }).where(and(
|
|
eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
));
|
|
if (!latest || !latest.finishedAt || !isHeartbeatRunTerminalStatus(latest.status)) continue;
|
|
if (latest.runtimeMode !== "legacy" && latest.runtimeMode !== "native") continue;
|
|
if (wake.idempotencyKey?.startsWith("chat-inbound:")) {
|
|
await resumeDeferredChatWake(db, { companyId: wake.companyId, agentId: wake.agentId,
|
|
issueId: String(wake.payload?.issueId), requestId: wake.id }).catch(err => {
|
|
scope.onError?.(err);
|
|
logger.warn({ requestId: wake.id }, "saved chat source could not resume after recovery");
|
|
});
|
|
continue;
|
|
}
|
|
const releaseSource = `${latest.companyId}:${String(wake.payload?.issueId)}:${latest.id}:${latest.sourceVersion}`;
|
|
if (latest.runtimeMode === "native" && releasedNativeSources.has(releaseSource)) continue;
|
|
if (latest.runtimeMode === "legacy") {
|
|
const cancelledAdmission = latest.status === "cancelled" && !latest.startedAt &&
|
|
latest.errorCode === "execution_reconciliation_required";
|
|
if ((latest.status !== "cancelled" || cancelledAdmission) && await getExecutionBlocker(db, wake.companyId, String(wake.payload?.issueId))) {
|
|
await resumeSavedLegacyComments(wake.companyId, wake.id).catch(err => {
|
|
scope.onError?.(err);
|
|
logger.warn({ err, queueId: wake.id }, "failed to deliver saved legacy comment after recovery stopped");
|
|
});
|
|
}
|
|
}
|
|
await releaseIssueExecutionAndPromote(latest, { suppressImmediateRecovery: true, resumeStrandedQueue: true }).then(() => {
|
|
if (latest.runtimeMode === "native") releasedNativeSources.add(releaseSource);
|
|
}).catch(err => {
|
|
scope.onError?.(err);
|
|
logger.warn({ err, queueId: wake.id }, "failed to promote stranded comments");
|
|
});
|
|
}
|
|
|
|
// The cancellation marker is durable intent. Retry while its exact queue
|
|
// is still deferred, including after a failed cleanup promotion or restart.
|
|
// Normal admission still checks process ownership, leases, pauses, and scope.
|
|
const interruptedQueues = await db
|
|
.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId })
|
|
.from(agentWakeupRequests)
|
|
.innerJoin(heartbeatRuns, and(
|
|
sql`${heartbeatRuns.resultJson}->>'queuedCommentInterruptQueueId' = ${agentWakeupRequests.id}::text`,
|
|
eq(heartbeatRuns.companyId, agentWakeupRequests.companyId),
|
|
eq(heartbeatRuns.agentId, agentWakeupRequests.agentId),
|
|
))
|
|
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
|
|
.where(and(
|
|
scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
|
|
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
eq(heartbeatRuns.status, "cancelled"),
|
|
eq(heartbeatRuns.runtimeMode, "legacy"),
|
|
eq(companies.status, "active"),
|
|
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
|
|
));
|
|
for (const run of interruptedQueues) {
|
|
await releaseIssueExecutionAndPromote(run, { suppressImmediateRecovery: true }).catch((err) => {
|
|
scope.onError?.(err);
|
|
logger.error({ err, runId: run.id }, "failed to retry interrupted comment queue");
|
|
});
|
|
}
|
|
|
|
}
|
|
|
|
async function resumeQueuedRuns() {
|
|
await resumeDeferredExecutionReceipts();
|
|
await resumeQueuedRunsAfterDrain();
|
|
}
|
|
|
|
// Release admission promptly without first scanning unrelated recovery work.
|
|
// Every run still passes the ordinary per-agent claim and dispatch gates.
|
|
async function resumeQueuedRunsAfterDrain(scope?: { companyId: string }) {
|
|
if ((await getSchedulingSuppression()).suppressed) return;
|
|
const cutoff = await getWorktreeExecutionCutoff();
|
|
const queuedRuns = await db
|
|
.select({ agentId: heartbeatRuns.agentId })
|
|
.from(heartbeatRuns)
|
|
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.status, "queued"),
|
|
scope ? eq(heartbeatRuns.companyId, scope.companyId) : undefined,
|
|
eq(companies.status, "active"),
|
|
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
|
|
),
|
|
);
|
|
|
|
const agentIds = [...new Set(queuedRuns.map((r) => r.agentId))];
|
|
for (const agentId of agentIds) {
|
|
await startNextQueuedRunForAgent(agentId);
|
|
}
|
|
}
|
|
|
|
// Startup fast path: start queued turns that never executed, on agents with
|
|
// no run left `running` by the previous process and no queued retry or
|
|
// continuation. Those agents' runs depend on restart recovery and wait for
|
|
// the full queued-run pass. Every run still passes the ordinary per-agent
|
|
// claim and dispatch gates, exactly as a new wakeup during startup would.
|
|
async function resumeFreshQueuedRunsAtStartup() {
|
|
if ((await getSchedulingSuppression()).suppressed) return { agents: 0 };
|
|
const cutoff = await getWorktreeExecutionCutoff();
|
|
const rows = await db
|
|
.selectDistinct({ agentId: heartbeatRuns.agentId })
|
|
.from(heartbeatRuns)
|
|
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.status, "queued"),
|
|
eq(companies.status, "active"),
|
|
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
|
|
sql`not exists (
|
|
select 1 from heartbeat_runs prior
|
|
where prior.agent_id = ${heartbeatRuns.agentId}
|
|
and (prior.status = 'running' or (prior.status = 'queued' and (
|
|
prior.started_at is not null or prior.retry_of_run_id is not null
|
|
or coalesce(prior.process_loss_retry_count, 0) > 0
|
|
or coalesce(prior.continuation_attempt, 0) > 0)))
|
|
)`,
|
|
),
|
|
);
|
|
for (const { agentId } of rows) {
|
|
await startNextQueuedRunForAgent(agentId);
|
|
}
|
|
return { agents: rows.length };
|
|
}
|
|
|
|
// A 403 from claimQueuedRun comes from the run's own persisted identity
|
|
// (an unverifiable interrupt receipt, a manual wake with no user). Those rows
|
|
// do not change, so the claim fails the same way on every pass and restart.
|
|
function isPermanentClaimRejection(err: unknown): err is HttpError {
|
|
return err instanceof HttpError && err.status === 403;
|
|
}
|
|
|
|
// Other 4xx rejections can clear later (a responsible user gets assigned, a
|
|
// conflicting claim finishes). Keep the run queued, but do not let it stop
|
|
// the rest of the queue or startup recovery.
|
|
function isDeferrableClaimRejection(err: unknown): err is HttpError {
|
|
return err instanceof HttpError && err.status >= 400 && err.status < 500;
|
|
}
|
|
|
|
// Settle runs that can never be claimed. Letting the error escape stalls the
|
|
// agent's queue and, during startup recovery, stops the server from booting.
|
|
async function cancelRejectedQueuedRuns(
|
|
rejected: Array<{ run: typeof heartbeatRuns.$inferSelect; err: HttpError }>,
|
|
) {
|
|
for (const { run, err } of rejected) {
|
|
logger.warn(
|
|
{ err, runId: run.id, agentId: run.agentId, companyId: run.companyId },
|
|
"cancelling queued heartbeat run whose claim was rejected",
|
|
);
|
|
try {
|
|
await cancelRunInternal(
|
|
run.id,
|
|
`Cancelled because the queued run cannot be claimed: ${err.message}`,
|
|
{ errorCode: "queued_run_claim_rejected" },
|
|
);
|
|
} catch (cancelErr) {
|
|
logger.error(
|
|
{ err: cancelErr, runId: run.id },
|
|
"failed to cancel queued heartbeat run whose claim was rejected; it stays queued for the next recovery pass",
|
|
);
|
|
}
|
|
}
|
|
}
|
|
|
|
async function startNextQueuedRunForAgent(agentId: string) {
|
|
if ((await getSchedulingSuppression()).suppressed) return [];
|
|
const cutoff = await getWorktreeExecutionCutoff();
|
|
// Cancelled after the start lock is released: cancelRunInternal promotes the
|
|
// agent's next queued run, which takes this same lock.
|
|
const rejectedClaims: Array<{ run: typeof heartbeatRuns.$inferSelect; err: HttpError }> = [];
|
|
const deferredPostCommitEffects: WakeQueuePostCommitEffect[] = [];
|
|
const deferredCancellations: Array<() => Promise<void>> = [];
|
|
let cancellationReason: string | undefined;
|
|
|
|
return withAgentStartLock(agentId, async () => {
|
|
const agent = await getAgent(agentId);
|
|
if (!agent) return [];
|
|
const invokability = await getAgentInvokability(agent);
|
|
if (!invokability.invokable) {
|
|
if (shouldCancelRunsForNonInvokableAgent(invokability)) {
|
|
cancellationReason = `Cancelled because the agent is not invokable: ${invokability.reason}`;
|
|
}
|
|
return [];
|
|
}
|
|
const policy = parseHeartbeatPolicy(agent);
|
|
const runningCount = await countRunningRunsForAgent(agentId);
|
|
const availableSlots = Math.max(
|
|
0,
|
|
policy.maxConcurrentRuns - runningCount,
|
|
);
|
|
if (availableSlots <= 0) return [];
|
|
|
|
const queuedRuns = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
|
|
),
|
|
)
|
|
.orderBy(asc(heartbeatRuns.createdAt));
|
|
if (queuedRuns.length === 0) return [];
|
|
if (queuedRuns.every(run => !run.startedAt) && !await canStartNewExternalBotWork(agent)) return [];
|
|
|
|
const dependencyReadiness = await listQueuedRunDependencyReadiness(
|
|
agent.companyId,
|
|
queuedRuns,
|
|
);
|
|
const queuedIssueIds = [
|
|
...new Set(
|
|
queuedRuns
|
|
.map((run) =>
|
|
readNonEmptyString(parseObject(run.contextSnapshot).issueId),
|
|
)
|
|
.filter((issueId): issueId is string => Boolean(issueId)),
|
|
),
|
|
];
|
|
const issueRows = await db
|
|
.select({
|
|
id: issues.id,
|
|
status: issues.status,
|
|
priority: issues.priority,
|
|
})
|
|
.from(issues)
|
|
.where(
|
|
queuedIssueIds.length > 0
|
|
? and(
|
|
eq(issues.companyId, agent.companyId),
|
|
inArray(issues.id, queuedIssueIds),
|
|
)
|
|
: sql`false`,
|
|
);
|
|
const issueById = new Map(issueRows.map((row) => [row.id, row]));
|
|
const companyAgents = await listCompanyAgentOrgRows(agent.companyId);
|
|
const prioritizedRuns = [...queuedRuns].sort((left, right) => {
|
|
const leftIssueId = readNonEmptyString(
|
|
parseObject(left.contextSnapshot).issueId,
|
|
);
|
|
const rightIssueId = readNonEmptyString(
|
|
parseObject(right.contextSnapshot).issueId,
|
|
);
|
|
const leftReadiness = leftIssueId
|
|
? dependencyReadiness.get(leftIssueId)
|
|
: null;
|
|
const rightReadiness = rightIssueId
|
|
? dependencyReadiness.get(rightIssueId)
|
|
: null;
|
|
const leftReady = leftIssueId
|
|
? (leftReadiness?.isDependencyReady ?? true)
|
|
: true;
|
|
const rightReady = rightIssueId
|
|
? (rightReadiness?.isDependencyReady ?? true)
|
|
: true;
|
|
const leftIssue = leftIssueId ? issueById.get(leftIssueId) : null;
|
|
const rightIssue = rightIssueId ? issueById.get(rightIssueId) : null;
|
|
const leftRank = leftIssueId
|
|
? leftReady
|
|
? leftIssue?.status === "in_progress"
|
|
? 0
|
|
: 1
|
|
: 3
|
|
: 2;
|
|
const rightRank = rightIssueId
|
|
? rightReady
|
|
? rightIssue?.status === "in_progress"
|
|
? 0
|
|
: 1
|
|
: 3
|
|
: 2;
|
|
if (leftRank !== rightRank) return leftRank - rightRank;
|
|
const leftPriorityRank = issueRunPriorityRank(leftIssue?.priority);
|
|
const rightPriorityRank = issueRunPriorityRank(rightIssue?.priority);
|
|
if (leftPriorityRank !== rightPriorityRank)
|
|
return leftPriorityRank - rightPriorityRank;
|
|
return left.createdAt.getTime() - right.createdAt.getTime();
|
|
});
|
|
|
|
const claimedRuns: Array<typeof heartbeatRuns.$inferSelect> = [];
|
|
for (const queuedRun of prioritizedRuns) {
|
|
if (claimedRuns.length >= availableSlots) break;
|
|
let claimed: typeof heartbeatRuns.$inferSelect | null;
|
|
try {
|
|
claimed = await claimQueuedRun(queuedRun, companyAgents, deferredPostCommitEffects, deferredCancellations);
|
|
} catch (err) {
|
|
if (isPermanentClaimRejection(err)) {
|
|
rejectedClaims.push({ run: queuedRun, err });
|
|
continue;
|
|
}
|
|
if (!isDeferrableClaimRejection(err)) throw err;
|
|
logger.warn(
|
|
{ err, runId: queuedRun.id, agentId: queuedRun.agentId, companyId: queuedRun.companyId },
|
|
"queued heartbeat run claim was rejected; leaving it queued for the next recovery pass",
|
|
);
|
|
continue;
|
|
}
|
|
if (claimed) claimedRuns.push(claimed);
|
|
}
|
|
if (claimedRuns.length === 0) return [];
|
|
|
|
for (const claimedRun of claimedRuns) {
|
|
const execution = executeRun(claimedRun.id).catch((err) => {
|
|
logger.error(
|
|
{ err, runId: claimedRun.id },
|
|
"queued heartbeat execution failed",
|
|
);
|
|
});
|
|
// Register the in-flight execution so drainActiveRunExecutions() can await
|
|
// it. executeRun resolves only after its finally block finishes flushing
|
|
// run rows/events, so awaiting this promise guarantees the run's writes
|
|
// have landed before a caller (e.g. a test's afterEach) mutates the DB.
|
|
activeRunExecutionPromises.add(execution);
|
|
void execution.finally(() => {
|
|
// drainActiveRunExecutions loops on activeRunExecutionPromises.size,
|
|
// so an entry that never clears here would hang it forever.
|
|
activeRunExecutionPromises.delete(execution);
|
|
});
|
|
}
|
|
return claimedRuns;
|
|
}).finally(async () => {
|
|
// Dispatch promoted inputs after the agent start lock is released.
|
|
try {
|
|
await applyWakeQueuePostCommitEffects(deferredPostCommitEffects);
|
|
} finally {
|
|
if (cancellationReason) await cancelActiveForAgentInternal(agentId, cancellationReason);
|
|
await cancelRejectedQueuedRuns(rejectedClaims);
|
|
for (const cancel of deferredCancellations) {
|
|
await cancel().catch((err) => logger.warn({ err, agentId },
|
|
"queued heartbeat run refused by a pause hold could not be cancelled; the next claim retries it"));
|
|
}
|
|
}
|
|
});
|
|
}
|
|
|
|
// Public wakeup entry point. Callers dispatch it fire-and-forget, so register
|
|
// the promise in activeWakeupPromises before it starts its asynchronous
|
|
// prologue. drainActiveRunExecutions can then await a wake that is still before
|
|
// run registration. Internal callers reference enqueueWakeup directly and
|
|
// already await it, so they do not need this registration.
|
|
function trackWakeup(
|
|
agentId: string,
|
|
opts: WakeupOptions = {},
|
|
): ReturnType<typeof enqueueWakeup> {
|
|
const promise = enqueueWakeup(agentId, opts);
|
|
activeWakeupPromises.add(promise);
|
|
void promise
|
|
.catch(() => {})
|
|
.finally(() => {
|
|
activeWakeupPromises.delete(promise);
|
|
});
|
|
return promise;
|
|
}
|
|
|
|
async function enqueueWakeup(agentId: string, opts: WakeupOptions = {}, executionWaitRequestId?: string) {
|
|
const source = opts.source ?? "on_demand";
|
|
const triggerDetail = opts.triggerDetail ?? null;
|
|
const contextSnapshot: Record<string, unknown> = {
|
|
...(opts.contextSnapshot ?? {}),
|
|
};
|
|
const reason = opts.reason ?? null;
|
|
let payload = opts.payload ? { ...opts.payload } : null;
|
|
// Only the board queue route can record interruption authority on an
|
|
// existing receipt. Never accept this internal marker from a wake caller.
|
|
if (payload) {
|
|
delete payload.queuedCommentInterrupt;
|
|
delete payload.manualUserWake;
|
|
delete payload.taskRetry;
|
|
}
|
|
if (opts.manualUserWake) {
|
|
if (opts.requestedByActorType !== "user" || !opts.requestedByActorId || opts.failedRunId) {
|
|
throw new HttpError(403, "Manual wake requires an authenticated user");
|
|
}
|
|
payload = { ...payload, manualUserWake: true };
|
|
}
|
|
const executionReconciliationWake =
|
|
contextSnapshot.source === "execution.reconciled" ||
|
|
opts.idempotencyKey?.startsWith("execution-reconciliation:") === true;
|
|
const {
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
issueIdFromPayload,
|
|
taskKey,
|
|
wakeCommentId,
|
|
} = enrichWakeContextSnapshot({
|
|
contextSnapshot,
|
|
reason,
|
|
source,
|
|
triggerDetail,
|
|
payload,
|
|
});
|
|
const isManualContextCompactionWake = () => Boolean(
|
|
reason === CONTEXT_COMPACTION_WAKE_REASON && source === "on_demand" && triggerDetail === "manual"
|
|
&& opts.manualUserWake === true && opts.requestedByActorType === "user"
|
|
&& Boolean(opts.requestedByActorId) && opts.allowRunCoalescing === false
|
|
&& payload?.intent === CONTEXT_COMPACTION_WAKE_REASON
|
|
&& readNonEmptyString(payload.requestId) !== null
|
|
&& readContextCompactionRequest(enrichedContextSnapshot)?.requestId === payload.requestId,
|
|
);
|
|
const hasIndependentConversationWakeCause = () => Boolean(
|
|
wakeCommentId || opts.taskRetryRequestId ||
|
|
hasInteractionContinuationWakeContext(enrichedContextSnapshot) ||
|
|
reason === CHAT_COMPLETION_WAKE_REASON || isManualContextCompactionWake(),
|
|
);
|
|
// Keep each request's own server-derived origin, including coalesced wakes.
|
|
// A run's merged context cannot establish which caller authored one receipt.
|
|
// Overwrite caller-supplied nested context rather than trusting it.
|
|
payload = { ...payload, [DEFERRED_WAKE_CONTEXT_KEY]: { ...enrichedContextSnapshot } };
|
|
let issueId =
|
|
readNonEmptyString(enrichedContextSnapshot.issueId) ?? issueIdFromPayload;
|
|
if (executionReconciliationWake && !issueId) return null;
|
|
const durableRequest = opts.durableChatRequest;
|
|
let taskRetryActorId: string | null = null;
|
|
delete enrichedContextSnapshot.boardMessageRetry;
|
|
if (opts.taskRetryRequestId) {
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, opts.taskRetryRequestId), eq(agentWakeupRequests.agentId, agentId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
eq(agentWakeupRequests.requestedByActorType, "user"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId ?? ""}`,
|
|
));
|
|
const retry = readTaskRetryIntent(receipt?.payload);
|
|
if (!receipt || retry?.generationId !== receipt.id || retry.canonicalIntentId || retry.queuedChatRequestId ||
|
|
retry.actorId !== receipt.requestedByActorId) return null;
|
|
const chatRetry = retry.chatActionId && durableRequest?.id === retry.chatActionId &&
|
|
durableRequest.failedRunRetry?.failedRunId === retry.sourceRunId;
|
|
if (retry.chatActionId ? !chatRetry :
|
|
opts.requestedByActorType !== "user" || retry.actorId !== opts.requestedByActorId) return null;
|
|
taskRetryActorId = retry.actorId;
|
|
enrichedContextSnapshot.forceFreshSession = true;
|
|
enrichedContextSnapshot.previousRunId = retry.sourceRunId;
|
|
enrichedContextSnapshot.taskRetryRequestId = receipt.id;
|
|
if (retry.boardMessageOrigin) enrichedContextSnapshot.boardMessageRetry = true;
|
|
}
|
|
|
|
|
|
let agent = await getAgent(agentId);
|
|
if (!agent) throw notFound("Agent not found");
|
|
if (issueId) {
|
|
const [pending] = await db.select().from(caseHandoffs).where(and(eq(caseHandoffs.companyId, agent.companyId), eq(caseHandoffs.issueId, issueId),
|
|
inArray(caseHandoffs.state, ["awaiting_release", "pending_dispatch"]))).limit(1);
|
|
if (pending && opts.durableCaseHandoffRequest?.handoffId !== pending.id) return null;
|
|
}
|
|
// Mentions only annotate comments. Ignore legacy callers before creating
|
|
// a run or deferred request; assignment and review have their own wakes.
|
|
if (reason === "issue_comment_mentioned" || enrichedContextSnapshot.wakeReason === "issue_comment_mentioned") return null;
|
|
if (issueId) {
|
|
const conversation = await getIssueExecutionContext(agent.companyId, issueId);
|
|
if (taskRetryActorId && conversation) await assertTaskRetryConversationAllowed(db, conversation, taskRetryActorId);
|
|
if (reason === "issue_children_completed" && conversation?.originKind === "onboarding_first_task") enrichedContextSnapshot.onboardingCompletion = true;
|
|
if (isConversation(conversation)) {
|
|
if (opts.manualUserWake && conversation!.conversationUserId !== opts.requestedByActorId) {
|
|
throw new HttpError(403, "Only the conversation owner can start a chat run");
|
|
}
|
|
if (isConversationExecutionWake(conversation, reason ?? readNonEmptyString(enrichedContextSnapshot.wakeReason))) return null;
|
|
if (agent.id !== conversation!.conversationAgentId) return null;
|
|
if (!(await instanceSettings.getExperimental()).enableAgentChat) return null;
|
|
if (isWaitingConversation(conversation) && !hasIndependentConversationWakeCause()) return null;
|
|
}
|
|
}
|
|
if (agent.adapterType === "paperclip_runner") {
|
|
const oldConfig = parseObject(agent.adapterConfig);
|
|
const nextConfig = normalizeLegacyRunnerProvider(oldConfig);
|
|
if (nextConfig !== oldConfig) {
|
|
await agentService(db).update(
|
|
agent.id,
|
|
{ adapterConfig: nextConfig },
|
|
{
|
|
recordRevision: {
|
|
source: "normalize_runner_provider",
|
|
createdByAgentId: null,
|
|
createdByUserId: null,
|
|
},
|
|
},
|
|
);
|
|
await logActivity(db, {
|
|
companyId: agent.companyId,
|
|
actorType: "system",
|
|
actorId: "heartbeat",
|
|
action: "agent.updated",
|
|
entityType: "agent",
|
|
entityId: agent.id,
|
|
details: { provider: "codex", reason: "native_codex_provider" },
|
|
});
|
|
agent = (await getAgent(agentId))!;
|
|
}
|
|
}
|
|
|
|
if (opts.failedRunId) {
|
|
const failed = await getRun(opts.failedRunId, { includeExecutionEvidence: true });
|
|
if (opts.requestedByActorType !== "user" || !opts.requestedByActorId ||
|
|
reason !== "retry_failed_run" || source !== "on_demand" || triggerDetail !== "manual" ||
|
|
!failed || failed.companyId !== agent.companyId || failed.agentId !== agentId ||
|
|
!(await canRetryStoppedRun(db, failed)) ||
|
|
(failed.nativeIssueId ?? readNonEmptyString(failed.contextSnapshot?.issueId)) !== issueId) {
|
|
throw conflict("The selected failed run cannot be retried for this task.");
|
|
}
|
|
if (!activeRunExecutions.has(failed.id) && !adapterExecutionControls.has(failed.id)) {
|
|
await sweepPendingCleanupLeases({ explicitRetry: {
|
|
companyId: failed.companyId, runId: failed.id, actorId: opts.requestedByActorId,
|
|
} });
|
|
}
|
|
if (isConversationAdapter(agent.adapterType) || agent.adapterType === "paperclip_runner") {
|
|
enrichedContextSnapshot.previousRunId = failed.id;
|
|
enrichedContextSnapshot.forceFreshSession = true;
|
|
}
|
|
}
|
|
|
|
const caseRequest = opts.durableCaseHandoffRequest;
|
|
const waitForCaseAdmission = (reason: string, message: string): never => {
|
|
// The case ledger is the outbox. Roll back this admission attempt rather
|
|
// than consume its one receipt ID in a generic skipped/deferred queue.
|
|
throw conflict(message, { code: "case_handoff_admission_wait", reason });
|
|
};
|
|
let caseReceipt: { id: string; requestedAt: Date } | undefined;
|
|
if (caseRequest) {
|
|
if (opts.taskRetryRequestId || opts.durableChatRequest || opts.durableDotRequest || opts.durableMuseRequest || source !== "assignment" || reason !== "case_handoff")
|
|
throw conflict("Invalid case handoff wake authority");
|
|
const [handoff] = await db.select().from(caseHandoffs).where(and(eq(caseHandoffs.id, caseRequest.handoffId),
|
|
eq(caseHandoffs.companyId, agent.companyId), eq(caseHandoffs.issueId, caseRequest.issueId),
|
|
eq(caseHandoffs.toAgentId, agentId), eq(caseHandoffs.wakeupRequestId, caseRequest.wakeupRequestId),
|
|
inArray(caseHandoffs.state, ["pending_dispatch", "delivered"]))).limit(1);
|
|
if (!handoff || issueId !== handoff.issueId || caseRequest.expectedIssueStatusVersion !== handoff.appliedIssueStatusVersion)
|
|
throw conflict("Case handoff no longer matches its task assignment");
|
|
caseReceipt = { id: handoff.wakeupRequestId, requestedAt: handoff.createdAt };
|
|
opts = { ...opts, allowRunCoalescing: false, idempotencyKey: `case-handoff:${handoff.id}`,
|
|
issueStateGuard: { assigneeAgentId: agentId, statuses: ["todo", "in_progress"], statusVersion: caseRequest.expectedIssueStatusVersion } };
|
|
}
|
|
if (durableRequest) {
|
|
assertDurableChatWakeupRequest(durableRequest, {
|
|
agentId,
|
|
companyId: agent.companyId,
|
|
issueId,
|
|
commentId: wakeCommentId ?? null,
|
|
requestedByActorType: opts.requestedByActorType,
|
|
requestedByActorId: opts.requestedByActorId,
|
|
});
|
|
opts = { ...opts, idempotencyKey: durableRequest.idempotencyKey };
|
|
}
|
|
if (
|
|
Object.hasOwn(enrichedContextSnapshot, "chatFailedRunRetry") &&
|
|
!durableRequest?.failedRunRetry
|
|
) {
|
|
throw new FailedChatRunRetryAuthorizationError();
|
|
}
|
|
if (durableRequest?.failedRunRetry) {
|
|
opts = { ...opts, allowRunCoalescing: false };
|
|
}
|
|
const dotRequest = opts.durableDotRequest;
|
|
const museRequest = opts.durableMuseRequest;
|
|
if (dotRequest && (durableRequest || museRequest || dotRequest.agentId !== agentId ||
|
|
dotRequest.companyId !== agent.companyId || dotRequest.issueId !== issueId ||
|
|
dotRequest.requestId !== payload?.dotRequestId || source !== "assignment" ||
|
|
opts.requestedByActorType !== "agent" || opts.requestedByActorId !== agentId ||
|
|
agent.adapterType !== "paperclip_runner" || !["openai_dot", "grok_bot"].includes(String(parseObject(agent.adapterConfig).provider)) ||
|
|
opts.idempotencyKey !== dotRequest.idempotencyKey)) {
|
|
throw conflict("Dot work request does not match its admission authority.");
|
|
}
|
|
if (museRequest && (durableRequest || dotRequest || museRequest.agentId !== agentId ||
|
|
museRequest.companyId !== agent.companyId || museRequest.issueId !== issueId ||
|
|
museRequest.requestId !== payload?.museRequestId || source !== "assignment" ||
|
|
opts.requestedByActorType !== "agent" || opts.requestedByActorId !== agentId ||
|
|
agent.adapterType !== "paperclip_runner" || parseObject(agent.adapterConfig).provider !== "muse" ||
|
|
parseObject(agent.adapterConfig).museBindingId !== museRequest.bindingId ||
|
|
!Number.isSafeInteger(museRequest.bindingGeneration) || museRequest.bindingGeneration < 1 ||
|
|
payload?.museBindingId !== museRequest.bindingId || payload?.museBindingGeneration !== museRequest.bindingGeneration ||
|
|
payload?.museRequestDigest !== museRequest.requestDigest || !/^sha256:[a-f0-9]{64}$/.test(museRequest.requestDigest) ||
|
|
opts.idempotencyKey !== museRequest.idempotencyKey)) {
|
|
throw conflict("Muse work request does not match its admission authority.");
|
|
}
|
|
const receiptRequest = durableRequest ?? dotRequest ?? museRequest ?? caseReceipt;
|
|
const durableReceiptFields = receiptRequest
|
|
? { id: receiptRequest.id, requestedAt: receiptRequest.requestedAt }
|
|
: {};
|
|
const existingDurableReceipt = async (queryDb: Db) => {
|
|
if (!receiptRequest) return null;
|
|
const receipt = await queryDb
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, receiptRequest.id))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
if (receipt && durableRequest) assertDurableChatWakeupReceipt(durableRequest, receipt);
|
|
if (receipt && caseRequest && (receipt.companyId !== agent.companyId || receipt.agentId !== agentId ||
|
|
receipt.payload?.issueId !== caseRequest.issueId || receipt.idempotencyKey !== `case-handoff:${caseRequest.handoffId}`))
|
|
throw conflict("Case handoff wake receipt identity changed");
|
|
if (receipt && dotRequest && (receipt.companyId !== dotRequest.companyId ||
|
|
receipt.agentId !== dotRequest.agentId || receipt.source !== "assignment" ||
|
|
receipt.requestedByActorType !== "agent" || receipt.requestedByActorId !== dotRequest.agentId ||
|
|
receipt.idempotencyKey !== dotRequest.idempotencyKey || receipt.payload?.issueId !== dotRequest.issueId ||
|
|
receipt.payload?.dotRequestId !== dotRequest.requestId)) {
|
|
throw conflict("requestId was reused for another task.");
|
|
}
|
|
if (receipt && museRequest && (receipt.companyId !== museRequest.companyId ||
|
|
receipt.agentId !== museRequest.agentId || receipt.source !== "assignment" ||
|
|
receipt.requestedByActorType !== "agent" || receipt.requestedByActorId !== museRequest.agentId ||
|
|
receipt.idempotencyKey !== museRequest.idempotencyKey || receipt.payload?.issueId !== museRequest.issueId ||
|
|
receipt.payload?.museRequestId !== museRequest.requestId || receipt.payload?.museBindingId !== museRequest.bindingId ||
|
|
receipt.payload?.museBindingGeneration !== museRequest.bindingGeneration || receipt.payload?.museRequestDigest !== museRequest.requestDigest)) {
|
|
throw conflict("Request ID reused with changed input.");
|
|
}
|
|
if (receipt && durableRequest && receipt.status === "deferred_issue_execution" && !receipt.runId &&
|
|
((durableRequest.failedRunRetry && opts.taskRetryRequestId &&
|
|
receipt.payload?.taskRetryRequestId === opts.taskRetryRequestId) ||
|
|
(durableRequest.resumeDeferred && !durableRequest.failedRunRetry &&
|
|
receipt.payload?.issueId === durableRequest.issueId &&
|
|
queuedCommentIdsFromWakePayload(receipt.payload).includes(durableRequest.commentId)))) {
|
|
// The live inbound owner or a bound Retry may reconsider the same saved receipt.
|
|
// The issue lock, live chat authority and execution holds below still
|
|
// decide whether it can dispatch; this never creates another wake.
|
|
executionWaitRequestId = receipt.id;
|
|
return null;
|
|
}
|
|
return receipt;
|
|
};
|
|
const priorReceipt = await existingDurableReceipt(db);
|
|
if (priorReceipt) {
|
|
// Replaying an admission receipt is not fresh authority to dispatch it.
|
|
// The normal queue owns dispatch and its current execution-policy checks.
|
|
return priorReceipt.runId ? getRun(priorReceipt.runId) : null;
|
|
}
|
|
|
|
const agentDebug = parseObject(parseObject(agent.runtimeConfig).debug);
|
|
const runDebug = parseObject(enrichedContextSnapshot.debug);
|
|
if (
|
|
agentDebug.providerTrace === "raw" &&
|
|
runDebug.providerTrace !== "raw"
|
|
) {
|
|
enrichedContextSnapshot.debug = {
|
|
...runDebug,
|
|
providerTrace: "raw",
|
|
};
|
|
enrichedContextSnapshot.providerTraceRequestedBy = `agent:${agent.id}:debug-setting`;
|
|
enrichedContextSnapshot.providerTraceRequestSource =
|
|
"agent_debug_setting";
|
|
}
|
|
|
|
// Automatic signals are replaceable; user input and interaction delivery
|
|
// retain distinct durable receipts even when the same gate blocks them.
|
|
const coalesceExecutionWait =
|
|
opts.requestedByActorType === "system" &&
|
|
opts.allowRunCoalescing !== false &&
|
|
!receiptRequest &&
|
|
!wakeCommentId &&
|
|
queuedCommentIdsFromRunContext(enrichedContextSnapshot).length === 0 &&
|
|
!isInteractionResolutionWakePayload(payload ?? {}) &&
|
|
!hasInteractionContinuationWakeContext(enrichedContextSnapshot);
|
|
const writeSkippedRequest = async (
|
|
skipReason: string,
|
|
patch: Partial<typeof agentWakeupRequests.$inferInsert> = {},
|
|
waitCondition?: Record<string, unknown>,
|
|
): Promise<{ created: boolean; waitSaved?: boolean }> => {
|
|
// A temporary refusal must leave the handoff retryable, without burning
|
|
// its one durable receipt ID on a skipped wake.
|
|
if (caseRequest) return { created: false };
|
|
if (executionWaitRequestId) {
|
|
const waitSaved = await writeDeferredExecutionWait(db, { companyId: agent.companyId, requestId: executionWaitRequestId }, {
|
|
reason: skipReason, message: patch.error ?? (skipReason === "issue_tree_hold_active"
|
|
? "This task is paused. Resume it to send your saved message."
|
|
: "Waiting for task execution to be enabled. Your message is saved."),
|
|
});
|
|
return { created: false, waitSaved };
|
|
}
|
|
const request = {
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: skipReason,
|
|
payload,
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
...patch,
|
|
};
|
|
if (waitCondition && issueId && isUuidLike(issueId)) {
|
|
const waitIssueId = issueId;
|
|
return db.transaction(async (tx) => {
|
|
await tx.execute(sql`select id from issues where id = ${waitIssueId} and company_id = ${agent.companyId} for update`);
|
|
if (durableRequest) await notifyChatInboundWakeNotices(tx, {
|
|
companyId: durableRequest.companyId, issueId: durableRequest.issueId,
|
|
});
|
|
return recordExecutionWait(tx as unknown as Db, {
|
|
issueId: waitIssueId, request, condition: waitCondition, coalesce: coalesceExecutionWait,
|
|
});
|
|
});
|
|
}
|
|
if (durableRequest) {
|
|
await db.transaction(async tx => {
|
|
await notifyChatInboundWakeNotices(tx, {
|
|
companyId: durableRequest.companyId, issueId: durableRequest.issueId,
|
|
});
|
|
await tx.insert(agentWakeupRequests).values(request);
|
|
});
|
|
} else {
|
|
await db.insert(agentWakeupRequests).values(request);
|
|
}
|
|
return { created: true };
|
|
};
|
|
const writeSkippedHeartbeatRequest = async (
|
|
skipReason: string,
|
|
details: Record<string, unknown>,
|
|
) => {
|
|
await writeSkippedRequest(skipReason, {
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: details,
|
|
},
|
|
});
|
|
};
|
|
|
|
// Work-intent outbox (invariant 4): a user's message refused by a
|
|
// temporary hold is saved as a deferred intent that names the refusal and
|
|
// the hold-release event that promotes it, never written as a terminal
|
|
// skip. Promotion runs normal admission again, once.
|
|
// The release drain promotes a saved message behind the task's latest run,
|
|
// once that run is terminal; a run still stopping after a pause commits is
|
|
// that run. A task that never ran has none: its message stays in the task,
|
|
// and stranded-work recovery starts the task once the hold lifts.
|
|
const canSaveDeferredMessage = async () => {
|
|
if (executionWaitRequestId || caseRequest || durableRequest || !issueId || !isUuidLike(issueId) ||
|
|
opts.requestedByActorType !== "user" || !wakeCommentId) return false;
|
|
const [latest] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.companyId, agent.companyId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`))
|
|
.limit(1);
|
|
return Boolean(latest);
|
|
};
|
|
const saveDeferredMessage = async (refusal: string, message: string, details: Record<string, unknown> = {}) => {
|
|
const savedIssueId = issueId!;
|
|
const condition = deferredWorkIntentWait(refusal, `${message} Your message is saved.`, details);
|
|
await db.transaction(async tx => {
|
|
await tx.execute(sql`select id from issues where id = ${savedIssueId} and company_id = ${agent.companyId} for update`);
|
|
await notifyDeferredExecutionWork(tx, { companyId: agent.companyId });
|
|
await tx.insert(agentWakeupRequests).values({
|
|
companyId: agent.companyId, agentId, source, triggerDetail, reason,
|
|
payload: withQueuedCommentIdsInWakePayload({
|
|
...payload, issueId: savedIssueId, [DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot, executionWait: condition,
|
|
}, [...new Set([...queuedCommentIdsFromRunContext(enrichedContextSnapshot), wakeCommentId!])]),
|
|
status: "deferred_issue_execution",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
});
|
|
});
|
|
return new PersistedExecutionWaitRefusal(message, details);
|
|
};
|
|
|
|
const schedulingSuppression = await getSchedulingSuppression();
|
|
// A task drain holds ADMISSION, not the request. The drain is a
|
|
// process-local pre-restart hold, so a wake that arrives while it is
|
|
// active still names real work that must run once the process comes
|
|
// back: leave it in the durable queue and let the dispatch-side checks
|
|
// (startNextQueuedRunForAgent / executeRun) keep it from starting until
|
|
// the drain lifts or the restart clears it. Writing it as `skipped`
|
|
// here dropped the wake permanently — an accepted plan whose
|
|
// continuation wake landed mid-drain left its issue in `todo` with no
|
|
// run and no path until a person noticed.
|
|
if (
|
|
schedulingSuppression.suppressed &&
|
|
schedulingSuppression.reason !== "task_drain"
|
|
) {
|
|
await writeSkippedHeartbeatRequest("heartbeat.scheduling_suppressed", {
|
|
reason: schedulingSuppression.reason,
|
|
});
|
|
return null;
|
|
}
|
|
|
|
const worktreeExecutionCutoff =
|
|
opts.requestedByActorType === "user"
|
|
? null
|
|
: await getWorktreeExecutionCutoff();
|
|
|
|
const company = await db
|
|
.select({ status: companies.status })
|
|
.from(companies)
|
|
.where(eq(companies.id, agent.companyId))
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!company || company.status !== "active") {
|
|
if (caseRequest) return null;
|
|
const companyStatus = company?.status ?? "missing";
|
|
if (opts.requestedByActorType === "user") {
|
|
// A paused company is a temporary hold; an archived one is not.
|
|
if (companyStatus === "paused" && (await canSaveDeferredMessage()))
|
|
throw await saveDeferredMessage("company.inactive", "Company is not active", { status: companyStatus });
|
|
throw conflict("Company is not active", { status: companyStatus });
|
|
}
|
|
await writeSkippedRequest("company.inactive", {
|
|
error: `Wake suppressed because company status is ${companyStatus}`,
|
|
}, { companyStatus });
|
|
return null;
|
|
}
|
|
|
|
const explicitResumeSession = await resolveExplicitResumeSessionOverride(
|
|
agent,
|
|
payload,
|
|
taskKey,
|
|
);
|
|
if (explicitResumeSession) {
|
|
enrichedContextSnapshot.resumeFromRunId =
|
|
explicitResumeSession.resumeFromRunId;
|
|
enrichedContextSnapshot.resumeSessionDisplayId =
|
|
explicitResumeSession.sessionDisplayId;
|
|
enrichedContextSnapshot.resumeSessionParams =
|
|
explicitResumeSession.sessionParams;
|
|
if (
|
|
!readNonEmptyString(enrichedContextSnapshot.issueId) &&
|
|
explicitResumeSession.issueId
|
|
) {
|
|
enrichedContextSnapshot.issueId = explicitResumeSession.issueId;
|
|
}
|
|
if (
|
|
!readNonEmptyString(enrichedContextSnapshot.taskId) &&
|
|
explicitResumeSession.taskId
|
|
) {
|
|
enrichedContextSnapshot.taskId = explicitResumeSession.taskId;
|
|
}
|
|
if (
|
|
!readNonEmptyString(enrichedContextSnapshot.taskKey) &&
|
|
explicitResumeSession.taskKey
|
|
) {
|
|
enrichedContextSnapshot.taskKey = explicitResumeSession.taskKey;
|
|
}
|
|
issueId = readNonEmptyString(enrichedContextSnapshot.issueId) ?? issueId;
|
|
}
|
|
const effectiveTaskKey =
|
|
readNonEmptyString(enrichedContextSnapshot.taskKey) ?? taskKey;
|
|
const genericTimerWake =
|
|
source === "timer" &&
|
|
!issueId &&
|
|
!wakeCommentId &&
|
|
!readNonEmptyString(enrichedContextSnapshot.taskId) &&
|
|
!effectiveTaskKey;
|
|
// Old or direct timer producers cannot create a doomed taskless Runner
|
|
// run. Explicit session restoration above can supply a real task scope.
|
|
if (genericTimerWake && !supportsUnscopedTimerHeartbeat(agent)) return null;
|
|
const sessionBefore =
|
|
explicitResumeSession?.sessionDisplayId ??
|
|
(await resolveSessionBeforeForWakeup(agent, effectiveTaskKey));
|
|
let hasResolvablePriorSessionWorkspace: boolean | null = null;
|
|
const resolveHasResolvablePriorSessionWorkspace = async () => {
|
|
if (hasResolvablePriorSessionWorkspace !== null)
|
|
return hasResolvablePriorSessionWorkspace;
|
|
hasResolvablePriorSessionWorkspace = issueId
|
|
? await hasResolvablePriorSessionWorkspaceForWake({
|
|
agent,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
taskKey: effectiveTaskKey,
|
|
explicitResumeSession,
|
|
})
|
|
: false;
|
|
return hasResolvablePriorSessionWorkspace;
|
|
};
|
|
const continuationAttempt = readContinuationAttempt(
|
|
enrichedContextSnapshot.livenessContinuationAttempt,
|
|
);
|
|
|
|
let projectId = readNonEmptyString(enrichedContextSnapshot.projectId);
|
|
if (!projectId && issueId) {
|
|
// Look up by either UUID or identifier (e.g. "ENV-13"), but always scope
|
|
// by companyId so a row from another tenant can never be returned even
|
|
// when identifiers collide across companies. Guard the UUID arm because
|
|
// issues.id is a Postgres uuid column — passing "ENV-13" into eq(issues.id, …)
|
|
// would fail with an invalid-input-syntax cast error before the OR is
|
|
// evaluated.
|
|
const lookupIsUuid = isUuidLike(issueId);
|
|
const idMatch = lookupIsUuid
|
|
? or(
|
|
eq(issues.id, issueId),
|
|
eq(issues.identifier, issueId.toUpperCase()),
|
|
)
|
|
: eq(issues.identifier, issueId.toUpperCase());
|
|
const resolvedIssue = await db
|
|
.select({
|
|
id: issues.id,
|
|
projectId: issues.projectId,
|
|
createdAt: issues.createdAt,
|
|
})
|
|
.from(issues)
|
|
.where(and(eq(issues.companyId, agent.companyId), idMatch))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (resolvedIssue) {
|
|
if (
|
|
worktreeExecutionCutoff &&
|
|
resolvedIssue.createdAt < worktreeExecutionCutoff
|
|
) {
|
|
await writeSkippedHeartbeatRequest(
|
|
"heartbeat.worktree_execution_cutoff",
|
|
{
|
|
reason: "worktree_execution_cutoff",
|
|
cutoff: worktreeExecutionCutoff.toISOString(),
|
|
issueId: resolvedIssue.id,
|
|
},
|
|
);
|
|
return null;
|
|
}
|
|
projectId = resolvedIssue.projectId ?? null;
|
|
// Canonicalize context to the UUID so downstream lookups always use UUID
|
|
if (resolvedIssue.id !== issueId) {
|
|
issueId = resolvedIssue.id;
|
|
enrichedContextSnapshot.issueId = issueId;
|
|
if (readNonEmptyString(enrichedContextSnapshot.taskId)) {
|
|
enrichedContextSnapshot.taskId = issueId;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
// Propagate projectId into context so resolveWorkspaceForRun can bind the
|
|
// project workspace even when context.projectId wasn't set by the caller.
|
|
if (projectId && !readNonEmptyString(enrichedContextSnapshot.projectId)) {
|
|
enrichedContextSnapshot.projectId = projectId;
|
|
}
|
|
const isolatedWorkspacesEnabled = issueId
|
|
? (await instanceSettings.getExperimental()).enableIsolatedWorkspaces
|
|
: false;
|
|
let operatorResponsibleUserId: string | null = opts.manualUserWake ? opts.requestedByActorId! : null;
|
|
let queuedResponsibleUserIdPromise: Promise<string> | null = null;
|
|
const resolveQueuedResponsibleUserId = () => {
|
|
if (operatorResponsibleUserId) return Promise.resolve(operatorResponsibleUserId);
|
|
queuedResponsibleUserIdPromise ??= (async () => {
|
|
const queuedIssueContext = issueId
|
|
? await getIssueExecutionContext(agent.companyId, issueId)
|
|
: null;
|
|
const queuedRoutineEnvContext = await getRoutineEnvForExecutionIssue(
|
|
agent.companyId,
|
|
queuedIssueContext,
|
|
);
|
|
const queuedResponsibleUserId =
|
|
await resolveResponsibleUserIdForRunSeed({
|
|
companyId: agent.companyId,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
issueContext: queuedIssueContext,
|
|
routineEnvContext: queuedRoutineEnvContext,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
source,
|
|
triggerDetail,
|
|
});
|
|
if (!queuedResponsibleUserId) {
|
|
throw new HttpError(
|
|
422,
|
|
"Unable to resolve responsible user for heartbeat run dispatch",
|
|
{
|
|
code: "responsible_user_unresolved",
|
|
agentId,
|
|
companyId: agent.companyId,
|
|
issueId: issueId ?? null,
|
|
source,
|
|
triggerDetail,
|
|
wakeReason: readNonEmptyString(
|
|
enrichedContextSnapshot.wakeReason,
|
|
),
|
|
},
|
|
);
|
|
}
|
|
return queuedResponsibleUserId;
|
|
})();
|
|
return queuedResponsibleUserIdPromise;
|
|
};
|
|
|
|
const budgetBlock = await budgets.getInvocationBlock(
|
|
agent.companyId,
|
|
agentId,
|
|
{
|
|
issueId,
|
|
projectId,
|
|
},
|
|
);
|
|
if (budgetBlock) {
|
|
// A budget block that is not a reached hard stop is an explicit pause.
|
|
if ((await canSaveDeferredMessage()) && !(await isBoardOwnedBudgetBlock(db, agent.companyId, budgetBlock)))
|
|
throw await saveDeferredMessage("budget.blocked", budgetBlock.reason,
|
|
{ scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId });
|
|
const wait = await writeSkippedRequest("budget.blocked", { error: budgetBlock.reason }, {
|
|
scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId,
|
|
});
|
|
if (caseRequest) return null;
|
|
const details = { scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId };
|
|
throw wait.waitSaved ? new PersistedExecutionWaitRefusal(budgetBlock.reason, details)
|
|
: conflict(budgetBlock.reason, details);
|
|
}
|
|
|
|
// Setup delays execution, but must not discard work accepted by a caller.
|
|
const invokability = await getAgentInvokability(isAgentAwaitingSetup(agent) ? { ...agent, status: "idle" } : agent);
|
|
if (!invokability.invokable) {
|
|
if (caseRequest) return null;
|
|
// Only a pause defers through the release drain (wake-queue policy); a
|
|
// pending approval still fails a promoted wake, so it is not saved here.
|
|
if (invokability.reason === "paused" && (await canSaveDeferredMessage()))
|
|
throw await saveDeferredMessage("agent.not_invokable", invokability.message,
|
|
{ status: agent.status, reason: invokability.reason });
|
|
let waitSaved = false;
|
|
if (opts.requestedByActorType !== "user" || executionWaitRequestId) {
|
|
const wait = await writeSkippedRequest("agent.not_invokable", {
|
|
error: invokability.message,
|
|
}, { status: agent.status, reason: invokability.reason });
|
|
waitSaved = wait.waitSaved === true;
|
|
}
|
|
const details = { status: agent.status, reason: invokability.reason,
|
|
invalidOrgChain: invokability.invalidOrgChain, ...invokability.details };
|
|
throw waitSaved ? new PersistedExecutionWaitRefusal(invokability.message, details)
|
|
: conflict(invokability.message, details);
|
|
}
|
|
|
|
const policy = parseHeartbeatPolicy(agent);
|
|
|
|
if (source === "timer" && !policy.enabled) {
|
|
await writeSkippedRequest("heartbeat.disabled", {}, { enabled: false });
|
|
return null;
|
|
}
|
|
if (source !== "timer" && !policy.wakeOnDemand) {
|
|
await writeSkippedRequest("heartbeat.wakeOnDemand.disabled", {}, { wakeOnDemand: false });
|
|
return null;
|
|
}
|
|
|
|
if (
|
|
policy.skipTimerWhenNoActionableWork &&
|
|
genericTimerWake &&
|
|
!(await hasActionableTimerWork(agent))
|
|
) {
|
|
await writeSkippedHeartbeatRequest("heartbeat.timer.no_actionable_work", {
|
|
reason:
|
|
"No assigned todo or in_progress issue requires this agent before timer adapter invocation.",
|
|
});
|
|
await markTimerHeartbeatChecked(agentId, source);
|
|
return null;
|
|
}
|
|
|
|
if (issueId) {
|
|
const activePauseHold = await treeControlSvc.getActivePauseHoldGate(
|
|
agent.companyId,
|
|
issueId,
|
|
);
|
|
if (activePauseHold) {
|
|
const treeHoldInteractionWake =
|
|
await isVerifiedIssueTreeControlInteractionWake(db, {
|
|
companyId: agent.companyId,
|
|
issueId,
|
|
agentId,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
requestedByActorType: opts.requestedByActorType,
|
|
requestedByActorId: opts.requestedByActorId,
|
|
});
|
|
|
|
if (!treeHoldInteractionWake) {
|
|
const wait = await writeSkippedRequest("issue_tree_hold_active", {}, {
|
|
holdId: activePauseHold.holdId,
|
|
});
|
|
if (wait.created) await logActivity(db, {
|
|
companyId: agent.companyId,
|
|
actorType: "system",
|
|
actorId: "system",
|
|
agentId,
|
|
runId: null,
|
|
action: "issue.tree_hold_wakeup_deferred",
|
|
entityType: "issue",
|
|
entityId: issueId,
|
|
details: {
|
|
holdId: activePauseHold.holdId,
|
|
rootIssueId: activePauseHold.rootIssueId,
|
|
requestedReason: reason,
|
|
source,
|
|
triggerDetail,
|
|
securityPrinciples: [
|
|
"Complete Mediation",
|
|
"Fail Securely",
|
|
"Secure Defaults",
|
|
],
|
|
},
|
|
});
|
|
return null;
|
|
}
|
|
|
|
enrichedContextSnapshot.treeHoldInteraction = true;
|
|
enrichedContextSnapshot.activeTreeHold = {
|
|
holdId: activePauseHold.holdId,
|
|
rootIssueId: activePauseHold.rootIssueId,
|
|
mode: activePauseHold.mode,
|
|
reason: activePauseHold.reason,
|
|
releasePolicy: activePauseHold.releasePolicy,
|
|
interaction: true,
|
|
};
|
|
}
|
|
}
|
|
|
|
if (issueId) {
|
|
// Mention-triggered wakes can request input from another agent, but they must
|
|
// still respect the issue execution lock so a second agent cannot start on the
|
|
// same issue workspace while the assignee already has a live run.
|
|
const agentNameKey = normalizeAgentNameKey(agent.name);
|
|
|
|
const cancelledRunsToEmit: (typeof heartbeatRuns.$inferSelect)[] = [];
|
|
const issueActivityPublications: ActivityPublication[] = [];
|
|
const issuePostCommitActions: IssuePostCommitAction[] = [];
|
|
|
|
const outcome = await db
|
|
.transaction(withDurableChatAdmission(durableRequest, async (tx) => {
|
|
await tx.execute(
|
|
sql`select id from issues where id = ${issueId} and company_id = ${agent.companyId} for update`,
|
|
);
|
|
|
|
if (executionWaitRequestId) {
|
|
const [pending] = await tx.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, executionWaitRequestId), eq(agentWakeupRequests.companyId, agent.companyId),
|
|
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
// A user message can join a queue originally created by a
|
|
// system wake. Admission validates the saved user comment or board click.
|
|
(opts.queuedCommentInterruptId ?? opts.queuedCommentRequestId) === executionWaitRequestId
|
|
? undefined : eq(agentWakeupRequests.requestedByActorType,
|
|
durableRequest ? opts.requestedByActorType ?? "" : "user"),
|
|
opts.queuedCommentInterruptId === executionWaitRequestId
|
|
? sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt'->>'actorId' = ${opts.requestedByActorId ?? ""}`
|
|
: opts.queuedCommentRequestId === executionWaitRequestId ? undefined
|
|
: eq(agentWakeupRequests.requestedByActorId, opts.requestedByActorId ?? ""),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
|
|
));
|
|
// The issue lock serializes cleanup callbacks and periodic workers.
|
|
// An adopted, discarded, or edited receipt is no longer authority.
|
|
if (!pending || (!opts.taskRetryRequestId && !(wakeCommentId && queuedCommentIdsFromWakePayload(pending.payload).includes(wakeCommentId)) &&
|
|
!(opts.queuedCommentInterruptId && await readQueuedInteractionResponse(tx as unknown as Db,
|
|
agent.companyId, issueId, pending.payload)))) {
|
|
return { kind: "deferred" as const };
|
|
}
|
|
if (opts.taskRetryRequestId && durableRequest?.failedRunRetry) {
|
|
if (pending.id !== durableRequest.id || pending.payload?.taskRetryRequestId !== opts.taskRetryRequestId)
|
|
return { kind: "deferred" as const };
|
|
// The parent board Retry is checked separately below after the
|
|
// connected conversation reauthorizes this exact child receipt.
|
|
} else if (opts.taskRetryRequestId) {
|
|
const retry = readTaskRetryIntent(pending.payload);
|
|
if (pending.id !== opts.taskRetryRequestId || retry?.generationId !== pending.id || retry.chatActionId ||
|
|
retry.actorId !== opts.requestedByActorId || retry.sourceRunId !== enrichedContextSnapshot.previousRunId)
|
|
return { kind: "deferred" as const };
|
|
// A declined Retry stays saved, but never silently: name the gate
|
|
// so the task explains it and a new click can replace this intent.
|
|
const declineRetry = async (reason: string, message: string) => {
|
|
await writeDeferredExecutionWait(tx as unknown as Db, { companyId: agent.companyId,
|
|
requestId: pending.id }, { reason, message }, { replace: true });
|
|
logger.warn({ requestId: pending.id, issueId, reason }, "Task Retry could not start");
|
|
return { kind: "deferred" as const };
|
|
};
|
|
if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: agent.companyId, issueId,
|
|
actorId: retry.actorId, actorSource: retry.actorSource }))
|
|
return declineRetry("retry_authority_denied", "The user who requested Retry can no longer change this task.");
|
|
if (retry.boardMessageOrigin) {
|
|
const origin = await readTaskRetryBoardMessageOrigin(tx as unknown as Db, { companyId: agent.companyId,
|
|
issueId, agentId, sourceRunId: retry.sourceRunId, actorId: retry.actorId });
|
|
if (!origin || origin.commentId !== retry.boardMessageOrigin.commentId ||
|
|
origin.identityContextId !== retry.boardMessageOrigin.identityContextId)
|
|
return declineRetry("retry_source_changed", "The saved message this Retry continues has changed.");
|
|
}
|
|
}
|
|
if (opts.queuedCommentRequestId) {
|
|
const ids = await undeliveredLegacyUserCommentIds(tx as unknown as Db,
|
|
agent.companyId, issueId, agentId, queuedCommentIdsFromWakePayload(pending.payload));
|
|
if (!wakeCommentId || !ids.includes(wakeCommentId)) return { kind: "deferred" as const };
|
|
pending.payload = withQueuedCommentIdsInWakePayload(parseObject(pending.payload), ids);
|
|
await tx.update(agentWakeupRequests).set({ payload: pending.payload }).where(and(
|
|
eq(agentWakeupRequests.id, pending.id), eq(agentWakeupRequests.companyId, agent.companyId),
|
|
));
|
|
}
|
|
if (!opts.queuedCommentInterruptId && !opts.queuedCommentRequestId && pending.payload?.manualUserWake === true) {
|
|
// A persisted manual wake keeps its actor when an execution wait
|
|
// resumes. The locked receipt above has revalidated that actor.
|
|
payload = { ...payload, manualUserWake: true };
|
|
operatorResponsibleUserId = opts.requestedByActorId!;
|
|
}
|
|
if (opts.queuedCommentInterruptId) {
|
|
// The locked board receipt supplies execution authority even when
|
|
// another user authored the messages. Dispatch revalidates the receipt.
|
|
operatorResponsibleUserId = opts.requestedByActorId!;
|
|
}
|
|
if (opts.queuedCommentInterruptId || opts.queuedCommentRequestId) {
|
|
// Edits/discards between the click and dispatch remain authoritative.
|
|
Object.assign(enrichedContextSnapshot, withQueuedCommentIdsInRunContext(
|
|
enrichedContextSnapshot, queuedCommentIdsFromWakePayload(pending.payload),
|
|
));
|
|
}
|
|
}
|
|
let automaticParentRunId: string | null = null;
|
|
if (
|
|
source === "automation" &&
|
|
opts.requestedByActorType === "system"
|
|
) {
|
|
const nativeParent = await readChatControlNativeParent(
|
|
tx as unknown as Db,
|
|
{ companyId: agent.companyId, issueId, agentId },
|
|
opts.requestedByActorId ?? null,
|
|
);
|
|
const genericParent =
|
|
!opts.requestedByActorId && reason === "issue_continuation_needed"
|
|
? readNonEmptyString(enrichedContextSnapshot.retryOfRunId)
|
|
: null;
|
|
automaticParentRunId =
|
|
nativeParent.kind === "parent"
|
|
? nativeParent.runId
|
|
: genericParent;
|
|
if (automaticParentRunId) {
|
|
const [parent] = await tx
|
|
.select({
|
|
agentId: heartbeatRuns.agentId,
|
|
context: heartbeatRuns.contextSnapshot,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, agent.companyId),
|
|
eq(heartbeatRuns.id, automaticParentRunId),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (
|
|
parent &&
|
|
(parseObject(parent.context).issueId ??
|
|
parseObject(parent.context).taskId) === issueId &&
|
|
parent.agentId !== agentId
|
|
)
|
|
automaticParentRunId = null;
|
|
}
|
|
const proof =
|
|
nativeParent.kind === "unresolved"
|
|
? { kind: "unresolved" as const }
|
|
: automaticParentRunId
|
|
? await readChatControlRecoveryStop(
|
|
tx as unknown as Db,
|
|
{
|
|
companyId: agent.companyId,
|
|
issueId,
|
|
agentId,
|
|
sourceRunId: automaticParentRunId,
|
|
},
|
|
true,
|
|
)
|
|
: { kind: "clear" as const };
|
|
if (proof.kind !== "clear") {
|
|
await tx
|
|
.insert(agentWakeupRequests)
|
|
.values({
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason:
|
|
proof.kind === "stopped"
|
|
? CHAT_CONTROL_RECOVERY_STOP_CODE
|
|
: CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE,
|
|
payload: { issueId },
|
|
status: "skipped",
|
|
requestedByActorType: "system",
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
}
|
|
|
|
const durableReceipt = await existingDurableReceipt(
|
|
tx as unknown as Db,
|
|
);
|
|
if (durableReceipt)
|
|
return { kind: "durable" as const, receipt: durableReceipt };
|
|
if (caseRequest) {
|
|
const [handoff] = await tx.select().from(caseHandoffs).where(and(eq(caseHandoffs.id, caseRequest.handoffId),
|
|
eq(caseHandoffs.companyId, agent.companyId),
|
|
eq(caseHandoffs.issueId, caseRequest.issueId), eq(caseHandoffs.state, "pending_dispatch"),
|
|
eq(caseHandoffs.toAgentId, agentId), eq(caseHandoffs.wakeupRequestId, caseRequest.wakeupRequestId))).for("update");
|
|
if (!handoff || handoff.appliedIssueStatusVersion !== caseRequest.expectedIssueStatusVersion)
|
|
throw conflict("Case handoff delivery was superseded");
|
|
}
|
|
const failedChatRetry = await authorizeFailedChatRunRetryWake(
|
|
db,
|
|
tx as unknown as Db,
|
|
{
|
|
phase: "admission",
|
|
wakeupRequestId: durableRequest?.id ?? null,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
issueId,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
},
|
|
);
|
|
if (
|
|
failedChatRetry !== Boolean(durableRequest?.failedRunRetry) ||
|
|
(failedChatRetry &&
|
|
durableRequest?.failedRunRetry?.failedRunId !==
|
|
readNonEmptyString(enrichedContextSnapshot.retryOfRunId))
|
|
) {
|
|
throw new FailedChatRunRetryAuthorizationError();
|
|
}
|
|
if (durableRequest) {
|
|
if (issueId !== durableRequest.issueId)
|
|
throw new Error("chat_inbound_wakeup_binding_denied");
|
|
await durableRequest.authorize(tx as unknown as Db);
|
|
if (opts.taskRetryRequestId) {
|
|
const [retryWake] = await tx.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, agent.companyId), eq(agentWakeupRequests.id, opts.taskRetryRequestId),
|
|
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
)).for("update");
|
|
const retry = readTaskRetryIntent(retryWake?.payload);
|
|
if (!failedChatRetry || retry?.chatActionId !== durableRequest.id ||
|
|
retry.actorId !== taskRetryActorId || retry.sourceRunId !== durableRequest.failedRunRetry?.failedRunId)
|
|
throw new FailedChatRunRetryAuthorizationError();
|
|
if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: agent.companyId, issueId,
|
|
actorId: retry.actorId, actorSource: retry.actorSource })) return { kind: "deferred" as const };
|
|
}
|
|
} else {
|
|
if (
|
|
source === "on_demand" &&
|
|
triggerDetail === "manual" &&
|
|
reason === "retry_failed_run"
|
|
) {
|
|
// Generic Board retry does not carry server-authorized failed-run
|
|
// lineage. Caller-supplied source/comment/retry markers cannot
|
|
// restore it, including for a retired conversation generation.
|
|
const [chatBinding] = await tx
|
|
.select({ id: chatConversations.id })
|
|
.from(chatConversations)
|
|
.innerJoin(chatEndpoints, eq(chatEndpoints.id, chatConversations.endpointId))
|
|
.where(
|
|
and(
|
|
eq(chatEndpoints.externalExecutionPolicy, "restricted"),
|
|
eq(chatConversations.companyId, agent.companyId),
|
|
eq(chatConversations.issueId, issueId),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (chatBinding) {
|
|
throw conflict(
|
|
"Retry needs the exact failed chat request and current access. Send the request again in the current connected conversation.",
|
|
{ code: "chat_failed_run_retry_requires_authorized_context" },
|
|
);
|
|
}
|
|
}
|
|
const [held] = await tx
|
|
.select({ id: issues.id })
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, agent.companyId),
|
|
unadmittedChatWakeupCondition(issues.id, issues.companyId),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (held) {
|
|
throw conflict(
|
|
"This task has external chat input that has not been admitted. Check the connection's Activity; let pending input finish, or restore access and send a new authorized message before starting the task.",
|
|
{ code: "chat_inbound_wakeup_unadmitted", issueId },
|
|
);
|
|
}
|
|
}
|
|
|
|
const issue = await tx
|
|
.select({
|
|
id: issues.id,
|
|
companyId: issues.companyId,
|
|
identifier: issues.identifier,
|
|
conversationAgentId: issues.conversationAgentId,
|
|
conversationUserId: issues.conversationUserId,
|
|
conversationState: issues.conversationState,
|
|
status: issues.status,
|
|
statusVersion: issues.statusVersion,
|
|
projectId: issues.projectId,
|
|
projectWorkspaceId: issues.projectWorkspaceId,
|
|
executionWorkspaceId: issues.executionWorkspaceId,
|
|
executionWorkspacePreference: issues.executionWorkspacePreference,
|
|
executionWorkspaceSettings: issues.executionWorkspaceSettings,
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
assigneeUserId: issues.assigneeUserId,
|
|
monitorNextCheckAt: issues.monitorNextCheckAt,
|
|
monitorWakeRequestedAt: issues.monitorWakeRequestedAt,
|
|
executionRunId: issues.executionRunId,
|
|
executionAgentNameKey: issues.executionAgentNameKey,
|
|
createdAt: issues.createdAt,
|
|
})
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, agent.companyId),
|
|
),
|
|
)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!issue) {
|
|
if (caseRequest) throw conflict("Case handoff task no longer exists");
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "issue_execution_issue_not_found",
|
|
payload,
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
// Completion can settle the conversation while admission is outside
|
|
// the issue lock. Recheck the current state before persisting a wake.
|
|
if (isWaitingConversation(issue) && !hasIndependentConversationWakeCause()) {
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
if (taskRetryActorId) {
|
|
await assertTaskRetryConversationAllowed(tx as unknown as Db, issue, taskRetryActorId);
|
|
if (await retireSupersededTaskRetry(tx as unknown as Db, {
|
|
companyId: issue.companyId, issueId: issue.id, requestId: opts.taskRetryRequestId!,
|
|
sourceRunId: readNonEmptyString(enrichedContextSnapshot.previousRunId)!,
|
|
})) return { kind: "deferred" as const };
|
|
}
|
|
const issueStateGuard = opts.issueStateGuard;
|
|
const activeMonitorRun = issueStateGuard?.monitorNextCheckAt === undefined ? null
|
|
: await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.nativeIssueId, issue.id),
|
|
eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, ["queued", "running", "scheduled_retry"]),
|
|
)).limit(1).then(rows => rows[0] ?? null);
|
|
if (
|
|
issueStateGuard &&
|
|
(!issueStateGuard.statuses.includes(issue.status) ||
|
|
issue.assigneeAgentId !== issueStateGuard.assigneeAgentId ||
|
|
(issueStateGuard.statusVersion !== undefined && issue.statusVersion !== issueStateGuard.statusVersion) ||
|
|
(issueStateGuard.monitorNextCheckAt !== undefined && (
|
|
activeMonitorRun !== null || issue.assigneeUserId !== null ||
|
|
issue.monitorNextCheckAt?.toISOString() !== issueStateGuard.monitorNextCheckAt ||
|
|
issue.monitorWakeRequestedAt?.toISOString() !== issueStateGuard.monitorWakeRequestedAt
|
|
)))
|
|
) {
|
|
if (caseRequest) throw conflict("Case handoff task changed before admission");
|
|
// A deferred monitor retains its schedule; do not create a receipt
|
|
// that could suppress its next admission attempt.
|
|
if (issueStateGuard.monitorNextCheckAt !== undefined) return { kind: "skipped" as const };
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "issue_state_guard_mismatch",
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: {
|
|
reason:
|
|
"Issue status or assignee changed before the wake could be queued.",
|
|
issueId: issue.id,
|
|
expectedStatuses: issueStateGuard.statuses,
|
|
actualStatus: issue.status,
|
|
expectedAssigneeAgentId: issueStateGuard.assigneeAgentId,
|
|
actualAssigneeAgentId: issue.assigneeAgentId,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
if (opts.failedRunId) {
|
|
// The issue lock makes double-clicks and network retries adopt the
|
|
// same successor, including after it has already finished.
|
|
const [previousRetry] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.agentId, agentId),
|
|
eq(heartbeatRuns.retryOfRunId, opts.failedRunId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'retry_failed_run'`,
|
|
)).orderBy(desc(heartbeatRuns.createdAt)).limit(1);
|
|
if (previousRetry) return { kind: "replayed" as const, run: previousRetry };
|
|
}
|
|
|
|
let reconciledSourceRunId: string | null = null;
|
|
let reconciledRestoreRetryCount: number | null = null;
|
|
if (executionReconciliationWake) {
|
|
const actionId = readNonEmptyString(
|
|
enrichedContextSnapshot.recoveryActionId,
|
|
);
|
|
if (
|
|
!actionId ||
|
|
!isUuidLike(actionId) ||
|
|
source !== "automation" ||
|
|
triggerDetail !== "system" ||
|
|
reason !== "issue_recovery_action_restored" ||
|
|
opts.requestedByActorType !== "system" ||
|
|
opts.requestedByActorId !== "execution-recovery" ||
|
|
opts.idempotencyKey !== `execution-reconciliation:${actionId}` ||
|
|
enrichedContextSnapshot.source !== "execution.reconciled" ||
|
|
enrichedContextSnapshot.forceFreshSession !== true ||
|
|
payload?.issueId !== issue.id ||
|
|
payload?.recoveryActionId !== actionId ||
|
|
issue.assigneeAgentId !== agentId ||
|
|
["done", "cancelled"].includes(issue.status)
|
|
)
|
|
return { kind: "skipped" as const };
|
|
|
|
// The issue lock serializes all admissions for this source. Validate
|
|
// the durable operator decision, then reconcile a prior queue commit
|
|
// before considering a new wake (including a now-terminal successor).
|
|
const [action] = await tx
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(
|
|
and(
|
|
eq(issueRecoveryActions.companyId, issue.companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, issue.id),
|
|
eq(issueRecoveryActions.id, actionId),
|
|
),
|
|
)
|
|
.for("update");
|
|
const decision = parseObject(
|
|
action?.evidence.executionReconciliation,
|
|
);
|
|
const sourceRunId = readNonEmptyString(decision.runId);
|
|
if (
|
|
!action ||
|
|
action.status !== "resolved" ||
|
|
action.kind !== "active_run_watchdog" ||
|
|
action.returnOwnerAgentId !== agentId ||
|
|
!sourceRunId ||
|
|
!isUuidLike(sourceRunId) ||
|
|
decision.providerStopped !== true ||
|
|
!["completed", "not_performed", "mixed"].includes(
|
|
String(decision.actionOutcome),
|
|
) ||
|
|
!readNonEmptyString(decision.outcomeEvidence) ||
|
|
enrichedContextSnapshot.previousRunId !== sourceRunId ||
|
|
enrichedContextSnapshot.retryOfRunId !== sourceRunId ||
|
|
!["pending", "delivered"].includes(
|
|
String(action.evidence.continuationDelivery),
|
|
)
|
|
)
|
|
return { kind: "skipped" as const };
|
|
|
|
const [existingWake] = await tx
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, issue.companyId),
|
|
eq(agentWakeupRequests.agentId, agentId),
|
|
eq(agentWakeupRequests.idempotencyKey, opts.idempotencyKey),
|
|
ne(agentWakeupRequests.status, "skipped"),
|
|
),
|
|
)
|
|
.orderBy(asc(agentWakeupRequests.requestedAt))
|
|
.limit(1);
|
|
if (existingWake) {
|
|
if (
|
|
existingWake.payload?.issueId !== issue.id ||
|
|
existingWake.payload?.recoveryActionId !== action.id ||
|
|
existingWake.requestedByActorType !== "system" ||
|
|
existingWake.requestedByActorId !== "execution-recovery" ||
|
|
!existingWake.runId
|
|
)
|
|
return { kind: "deferred" as const };
|
|
const [existingRun] = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, issue.companyId),
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
eq(heartbeatRuns.id, existingWake.runId),
|
|
),
|
|
);
|
|
if (
|
|
!existingRun ||
|
|
existingRun.contextSnapshot?.issueId !== issue.id ||
|
|
existingRun.contextSnapshot?.recoveryActionId !== action.id ||
|
|
existingRun.contextSnapshot?.previousRunId !== sourceRunId
|
|
)
|
|
return { kind: "deferred" as const };
|
|
return { kind: "replayed" as const, run: existingRun };
|
|
}
|
|
if (action.evidence.continuationDelivery !== "pending")
|
|
return { kind: "skipped" as const };
|
|
const [reconciledRun] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.id, sourceRunId),
|
|
));
|
|
if (decision.workspaceReconstruction) {
|
|
await validateNativeWorkspaceReconstruction(tx as unknown as Db, {
|
|
companyId: issue.companyId, issueId: issue.id, runId: sourceRunId, actionId,
|
|
});
|
|
reconciledRestoreRetryCount = executionFailureRetryCount(reconciledRun!);
|
|
enrichedContextSnapshot.workspaceReconstructionActionId = action.id;
|
|
}
|
|
if (hasWorkspaceRestoreFailure(reconciledRun?.resultJson)) {
|
|
if ((readNonEmptyString(decision.workspaceRepairEvidence)?.length ?? 0) < 20)
|
|
return { kind: "skipped" as const };
|
|
// Repair does not reset the remaining automatic retry budget.
|
|
reconciledRestoreRetryCount = executionFailureRetryCount(reconciledRun!);
|
|
}
|
|
reconciledSourceRunId = sourceRunId;
|
|
}
|
|
|
|
let continuationWait = { reason: "execution_recovery", message: "Waiting for execution recovery. Your message is saved." };
|
|
let continuationBlocked = false;
|
|
const deferBlockedExecution = async (
|
|
executionBlocker: Pick<NonNullable<Awaited<ReturnType<typeof getExecutionBlocker>>>, "recoveryActionId" | "cause" | "nextAction">,
|
|
) => {
|
|
if (caseRequest) waitForCaseAdmission("execution_reconciliation_required", executionBlocker.nextAction);
|
|
opts.issueStateGuard?.onExecutionDeferred?.(executionBlocker.cause);
|
|
const condition = { recoveryActionId: executionBlocker.recoveryActionId, ...continuationWait };
|
|
if (executionWaitRequestId) {
|
|
await writeDeferredExecutionWait(tx as unknown as Db, { companyId: agent.companyId,
|
|
requestId: executionWaitRequestId }, condition, { replace: true });
|
|
return { kind: "deferred" as const, admitted: true };
|
|
}
|
|
if (durableRequest || wakeCommentId ||
|
|
hasInteractionContinuationWakeContext(enrichedContextSnapshot) ||
|
|
readNonEmptyString(enrichedContextSnapshot.nativeStatusWakeIntentId)) {
|
|
await notifyDeferredExecutionWork(tx, { companyId: agent.companyId });
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId, agentId, source, triggerDetail, reason,
|
|
payload: withQueuedCommentIdsInWakePayload({
|
|
...payload,
|
|
issueId: issue.id,
|
|
[DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot,
|
|
executionWait: condition,
|
|
}, [...new Set([
|
|
...queuedCommentIdsFromRunContext(enrichedContextSnapshot),
|
|
...(wakeCommentId ? [wakeCommentId] : []),
|
|
])]),
|
|
status: "deferred_issue_execution",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
});
|
|
} else {
|
|
await recordExecutionWait(tx as unknown as Db, {
|
|
issueId: issue.id, condition, coalesce: coalesceExecutionWait,
|
|
request: {
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId, agentId, source, triggerDetail,
|
|
reason: "execution_reconciliation_required",
|
|
error: executionBlocker.nextAction,
|
|
payload,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
},
|
|
});
|
|
}
|
|
return { kind: "deferred" as const, admitted: true };
|
|
};
|
|
const explicitContinuationRunId = randomUUID();
|
|
const [resetComment] = isConversation(issue) && opts.requestedByActorType === "user" && opts.requestedByActorId && wakeCommentId
|
|
? await tx.select({ body: issueComments.body }).from(issueComments).where(and(
|
|
eq(issueComments.companyId, issue.companyId), eq(issueComments.issueId, issue.id),
|
|
eq(issueComments.id, wakeCommentId), eq(issueComments.authorUserId, opts.requestedByActorId),
|
|
eq(issueComments.authorType, "user"), isNull(issueComments.deletedAt), isNull(issueComments.createdByRunId),
|
|
)).limit(1)
|
|
: [];
|
|
const verifiedConversationReset = Boolean(resetComment && isConversationReset(resetComment.body));
|
|
if (wakeCommentId && opts.requestedByActorType === "user")
|
|
await preserveTaskRemoteWorkspaceWriter(tx as unknown as Db, issue.companyId, issue.id);
|
|
const executionBlocker = await getExecutionBlocker(
|
|
tx as unknown as Db, issue.companyId, issue.id,
|
|
{ conversationResetCommentId: verifiedConversationReset ? wakeCommentId : null },
|
|
);
|
|
// Prove eligibility without retiring the hold. Later gates can still
|
|
// decline this wake; hold retirement and successor creation stay atomic.
|
|
// A bound chat request has already rechecked its current principal
|
|
// above. Treat its new user message like a board comment, but keep
|
|
// failed-run retry actions on their separate exact-request path.
|
|
const ownsReconstructionContinuation = reconciledSourceRunId && executionBlocker?.runId === reconciledSourceRunId
|
|
&& executionBlocker.recoveryActionId === enrichedContextSnapshot.recoveryActionId;
|
|
const continuationHold = () => executionBlocker ?? {
|
|
recoveryActionId: null, cause: "native_continuation_requires_reconciliation" as const, nextAction: continuationWait.message,
|
|
};
|
|
if ((executionBlocker || opts.queuedCommentInterruptId) && !ownsReconstructionContinuation && !(await admitExplicitNativeContinuation({
|
|
db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id,
|
|
agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId,
|
|
reason: taskRetryActorId ? "task_retry" : durableRequest && !failedChatRetry ? "issue_commented" : reason,
|
|
commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId,
|
|
queuedCommentInterruptId: opts.queuedCommentInterruptId,
|
|
queuedCommentRequestId: opts.queuedCommentRequestId,
|
|
taskRetryRequestId: opts.taskRetryRequestId,
|
|
...(durableRequest?.resumeDeferred && !failedChatRetry ? { durableChatRequest: durableRequest } : {}),
|
|
dryRun: true,
|
|
onBlocked: (reason, message) => { continuationBlocked = true; continuationWait = { reason, message }; },
|
|
})) && (executionBlocker || (continuationBlocked && !verifiedConversationReset))) return deferBlockedExecution(continuationHold());
|
|
|
|
|
|
if (
|
|
worktreeExecutionCutoff &&
|
|
issue.createdAt < worktreeExecutionCutoff
|
|
) {
|
|
if (caseRequest) waitForCaseAdmission("heartbeat.worktree_execution_cutoff", "Case handoff is waiting for workspace execution to be enabled");
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "heartbeat.worktree_execution_cutoff",
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: {
|
|
reason: "worktree_execution_cutoff",
|
|
cutoff: worktreeExecutionCutoff.toISOString(),
|
|
issueId: issue.id,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
const cancelStaleScheduledRetry = async (
|
|
scheduledRun: typeof heartbeatRuns.$inferSelect,
|
|
) => {
|
|
const issueCancelled = issue.status === "cancelled";
|
|
if (
|
|
scheduledRun.status !== "scheduled_retry" ||
|
|
(scheduledRun.agentId === issue.assigneeAgentId &&
|
|
!issueCancelled)
|
|
) {
|
|
return false;
|
|
}
|
|
|
|
const now = new Date();
|
|
const reason = issueCancelled
|
|
? "Cancelled because the issue was cancelled before the scheduled retry became due"
|
|
: "Cancelled because the issue was reassigned before the scheduled retry became due";
|
|
await notifyInstructionCollection(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, scheduledRun);
|
|
await notifyScheduledRetryCancellation(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
|
|
await notifyChatRunMilestones(tx, { companyId: scheduledRun.companyId, issueId: issue.id });
|
|
const cancelled = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: now,
|
|
error: reason,
|
|
errorCode: issueCancelled
|
|
? "issue_cancelled"
|
|
: "issue_reassigned",
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, scheduledRun.id),
|
|
eq(heartbeatRuns.status, "scheduled_retry"),
|
|
),
|
|
)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!cancelled) return false;
|
|
|
|
if (scheduledRun.wakeupRequestId) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: now,
|
|
error: reason,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
eq(agentWakeupRequests.id, scheduledRun.wakeupRequestId),
|
|
);
|
|
}
|
|
|
|
if (issue.executionRunId === scheduledRun.id) {
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, issue.id),
|
|
eq(issues.executionRunId, scheduledRun.id),
|
|
),
|
|
);
|
|
}
|
|
|
|
const eventSeq = await allocateHeartbeatRunEventSeq(
|
|
tx as unknown as Db,
|
|
cancelled.id,
|
|
);
|
|
|
|
await tx.insert(heartbeatRunEvents).values({
|
|
companyId: cancelled.companyId,
|
|
runId: cancelled.id,
|
|
agentId: cancelled.agentId,
|
|
seq: eventSeq,
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message: issueCancelled
|
|
? "Scheduled retry cancelled because issue was cancelled before it became due"
|
|
: "Scheduled retry cancelled because issue ownership changed before it became due",
|
|
payload: {
|
|
issueId: issue.id,
|
|
issueStatus: issue.status,
|
|
scheduledRetryAttempt: cancelled.scheduledRetryAttempt,
|
|
scheduledRetryAt: cancelled.scheduledRetryAt
|
|
? new Date(cancelled.scheduledRetryAt).toISOString()
|
|
: null,
|
|
scheduledRetryReason: cancelled.scheduledRetryReason,
|
|
previousRetryAgentId: cancelled.agentId,
|
|
currentAssigneeAgentId: issue.assigneeAgentId,
|
|
},
|
|
});
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({ nextEventSeq: eventSeq + 1, updatedAt: now })
|
|
.where(eq(heartbeatRuns.id, cancelled.id));
|
|
|
|
cancelledRunsToEmit.push(cancelled);
|
|
|
|
return true;
|
|
};
|
|
|
|
let activeExecutionRun = issue.executionRunId
|
|
? await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, issue.executionRunId))
|
|
.then((rows) => rows[0] ?? null)
|
|
: null;
|
|
const manualExecutionHold = activeExecutionRun && await nativeManualAdmissionHoldsExecution(tx as unknown as Db,
|
|
{ companyId: issue.companyId, runId: activeExecutionRun.id, issueId: issue.id });
|
|
|
|
if (
|
|
activeExecutionRun &&
|
|
!manualExecutionHold &&
|
|
!EXECUTION_PATH_HEARTBEAT_RUN_STATUSES.includes(
|
|
activeExecutionRun.status as (typeof EXECUTION_PATH_HEARTBEAT_RUN_STATUSES)[number],
|
|
)
|
|
) {
|
|
activeExecutionRun = null;
|
|
}
|
|
|
|
if (
|
|
activeExecutionRun &&
|
|
(await cancelStaleScheduledRetry(activeExecutionRun))
|
|
) {
|
|
activeExecutionRun = null;
|
|
}
|
|
|
|
// A queued/scheduled run holding the lock for an agent that is
|
|
// no longer the issue's assignee is stale by design — the issue
|
|
// has been re-routed (e.g. blocked → in_review with a different
|
|
// assignee). Cancel it and release the lock; otherwise the new
|
|
// assignee's wake gets parked in `deferred_issue_execution`
|
|
// forever, because the original queued holder will never run
|
|
// (the issue's status / target now belongs to someone else).
|
|
//
|
|
// Race guard: pin the cancel UPDATE to the exact non-running
|
|
// status we read above. A worker could transition the holder
|
|
// from `queued` → `running` between the SELECT and this UPDATE;
|
|
// the status predicate ensures we never clobber a freshly-
|
|
// claimed running run. If zero rows matched, leave
|
|
// `activeExecutionRun` populated so the defer path runs
|
|
// normally against the now-running holder.
|
|
if (
|
|
activeExecutionRun &&
|
|
!manualExecutionHold &&
|
|
activeExecutionRun.status !== "running" &&
|
|
issue.assigneeAgentId &&
|
|
activeExecutionRun.agentId !== issue.assigneeAgentId
|
|
) {
|
|
await notifyInstructionCollection(tx, { companyId: activeExecutionRun.companyId, runId: activeExecutionRun.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: activeExecutionRun.companyId, runId: activeExecutionRun.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: activeExecutionRun.companyId, runId: activeExecutionRun.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, activeExecutionRun);
|
|
await notifyChatRunMilestones(tx, { companyId: activeExecutionRun.companyId, issueId: issue.id });
|
|
const cancelled = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: new Date(),
|
|
error:
|
|
"Execution lock released after issue reassigned to a different agent",
|
|
errorCode: "lock_released_on_reassignment",
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, activeExecutionRun.id),
|
|
eq(heartbeatRuns.status, activeExecutionRun.status),
|
|
),
|
|
)
|
|
.returning();
|
|
if (cancelled.length > 0) {
|
|
cancelledRunsToEmit.push(cancelled[0]);
|
|
if (activeExecutionRun.wakeupRequestId) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: new Date(),
|
|
error:
|
|
"Execution lock released after issue reassigned to a different agent",
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
eq(
|
|
agentWakeupRequests.id,
|
|
activeExecutionRun.wakeupRequestId,
|
|
),
|
|
);
|
|
}
|
|
activeExecutionRun = null;
|
|
}
|
|
}
|
|
|
|
if (!activeExecutionRun && issue.executionRunId) {
|
|
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(issues.id, issue.id));
|
|
}
|
|
|
|
if (!activeExecutionRun) {
|
|
const legacyRun = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, issue.companyId),
|
|
inArray(heartbeatRuns.status, [
|
|
...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES,
|
|
]),
|
|
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`,
|
|
),
|
|
)
|
|
.orderBy(
|
|
sql`case when ${heartbeatRuns.status} = 'running' then 0 else 1 end`,
|
|
asc(heartbeatRuns.createdAt),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (legacyRun) {
|
|
if (await cancelStaleScheduledRetry(legacyRun)) {
|
|
activeExecutionRun = null;
|
|
} else {
|
|
activeExecutionRun = legacyRun;
|
|
const legacyAgent = await tx
|
|
.select({ name: agents.name })
|
|
.from(agents)
|
|
.where(eq(agents.id, legacyRun.agentId))
|
|
.then((rows) => rows[0] ?? null);
|
|
await notifyStatusCardIssueChange(tx, issue.companyId);
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: legacyRun.id,
|
|
executionAgentNameKey: normalizeAgentNameKey(
|
|
legacyAgent?.name,
|
|
),
|
|
executionLockedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(issues.id, issue.id));
|
|
}
|
|
}
|
|
}
|
|
|
|
// Availability is advisory: competing clicks may both read it before
|
|
// admission. The existing issue lock serializes the run and deferred
|
|
// receipt checks with the admission below, without another lock owner.
|
|
if (isManualContextCompactionWake()) {
|
|
if (activeExecutionRun) {
|
|
const compacting = isContextCompactionRun(activeExecutionRun.contextSnapshot);
|
|
throw new ContextCompactionError(compacting ? "already_pending" : "agent_busy",
|
|
compacting ? "The agent is already compacting its context." : "Available when the agent finishes its current run.");
|
|
}
|
|
const [pending] = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, issue.companyId), eq(agentWakeupRequests.agentId, agentId),
|
|
inArray(agentWakeupRequests.status, ["queued", "deferred_issue_execution"]), isNull(agentWakeupRequests.runId),
|
|
executionWaitRequestId ? ne(agentWakeupRequests.id, executionWaitRequestId) : undefined,
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
|
|
sql`${agentWakeupRequests.payload}->>'intent' = ${CONTEXT_COMPACTION_WAKE_REASON}`,
|
|
sql`jsonb_typeof(${agentWakeupRequests.payload}->'requestId') = 'string'`,
|
|
sql`nullif(btrim(${agentWakeupRequests.payload}->>'requestId'), '') is not null`,
|
|
sql`${agentWakeupRequests.payload}->${DEFERRED_WAKE_CONTEXT_KEY}->>'contextCompactionRequestId' = ${agentWakeupRequests.payload}->>'requestId'`,
|
|
)).limit(1);
|
|
if (pending) throw new ContextCompactionError("already_pending", "The agent is already compacting its context.");
|
|
}
|
|
|
|
const dependencyReadiness = await issuesSvc
|
|
.listDependencyReadiness(issue.companyId, [issue.id], tx)
|
|
.then((rows) => rows.get(issue.id) ?? null);
|
|
|
|
// Blocked descendants should stay idle until the final blocker resolves.
|
|
// Human comment/mention wakes are the exception: they may run in a
|
|
// bounded interaction mode so the assignee can answer or triage.
|
|
const blockedInteractionWake =
|
|
dependencyReadiness &&
|
|
!dependencyReadiness.isDependencyReady &&
|
|
allowsIssueInteractionWake(
|
|
enrichedContextSnapshot,
|
|
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
|
|
);
|
|
|
|
if (blockedInteractionWake) {
|
|
enrichedContextSnapshot.dependencyBlockedInteraction = true;
|
|
enrichedContextSnapshot.unresolvedBlockerIssueIds =
|
|
dependencyReadiness.unresolvedBlockerIssueIds;
|
|
enrichedContextSnapshot.unresolvedBlockerCount =
|
|
dependencyReadiness.unresolvedBlockerCount;
|
|
enrichedContextSnapshot.unresolvedBlockerSummaries =
|
|
await listUnresolvedBlockerSummaries(
|
|
tx,
|
|
issue.companyId,
|
|
issue.id,
|
|
dependencyReadiness.unresolvedBlockerIssueIds,
|
|
);
|
|
}
|
|
|
|
if (
|
|
!activeExecutionRun &&
|
|
dependencyReadiness &&
|
|
!dependencyReadiness.isDependencyReady &&
|
|
!blockedInteractionWake
|
|
) {
|
|
if (caseRequest) waitForCaseAdmission("issue_dependencies_blocked", "Case handoff is waiting for task dependencies to resolve");
|
|
await recordExecutionWait(tx as unknown as Db, {
|
|
issueId: issue.id,
|
|
coalesce: coalesceExecutionWait,
|
|
condition: { unresolvedBlockerIssueIds: [...dependencyReadiness.unresolvedBlockerIssueIds].sort() },
|
|
request: {
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "issue_dependencies_blocked",
|
|
payload: {
|
|
...(payload ?? {}),
|
|
issueId,
|
|
unresolvedBlockerIssueIds:
|
|
dependencyReadiness.unresolvedBlockerIssueIds,
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
},
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
if (
|
|
isolatedWorkspacesEnabled &&
|
|
!activeExecutionRun &&
|
|
(issue.status !== "done" || Boolean(opts.queuedCommentInterruptId)) &&
|
|
issue.status !== "cancelled"
|
|
) {
|
|
const issueSettings = parseIssueExecutionWorkspaceSettings(
|
|
issue.executionWorkspaceSettings,
|
|
);
|
|
const resolvedMode = resolveExecutionWorkspaceMode({
|
|
projectPolicy: null,
|
|
issueSettings,
|
|
legacyUseProjectWorkspace: null,
|
|
});
|
|
const workspaceManagedConfig = buildExecutionWorkspaceAdapterConfig(
|
|
{
|
|
agentConfig: parseObject(agent.adapterConfig),
|
|
projectPolicy: null,
|
|
issueSettings,
|
|
mode: resolvedMode,
|
|
legacyUseProjectWorkspace: null,
|
|
},
|
|
);
|
|
const resolvedStrategy = resolveEffectiveWorkspaceStrategyType(
|
|
resolvedMode,
|
|
workspaceManagedConfig,
|
|
);
|
|
const existingExecutionWorkspaceStatus = issue.executionWorkspaceId
|
|
? await tx
|
|
.select({ status: executionWorkspaces.status })
|
|
.from(executionWorkspaces)
|
|
.where(
|
|
and(
|
|
eq(executionWorkspaces.id, issue.executionWorkspaceId),
|
|
eq(executionWorkspaces.companyId, issue.companyId),
|
|
),
|
|
)
|
|
.then((rows) => rows[0]?.status ?? null)
|
|
: null;
|
|
const reuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({
|
|
issueExecutionWorkspaceId: issue.executionWorkspaceId,
|
|
issueExecutionWorkspacePreference:
|
|
issue.executionWorkspacePreference,
|
|
existingExecutionWorkspaceStatus,
|
|
});
|
|
const hasResolvablePriorSessionWorkspace =
|
|
await resolveHasResolvablePriorSessionWorkspace();
|
|
|
|
if (
|
|
isUnrunnableWorktreeCombo({
|
|
issue: {
|
|
projectId: issue.projectId ?? projectId ?? null,
|
|
projectWorkspaceId: issue.projectWorkspaceId,
|
|
executionWorkspaceId: issue.executionWorkspaceId,
|
|
executionWorkspacePreference:
|
|
issue.executionWorkspacePreference,
|
|
},
|
|
resolvedMode,
|
|
resolvedStrategy,
|
|
reusableExecutionWorkspaceAvailable:
|
|
reuseRequest.existingExecutionWorkspaceAvailable,
|
|
hasResolvablePriorSessionWorkspace,
|
|
})
|
|
) {
|
|
if (caseRequest) waitForCaseAdmission(WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE, WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE);
|
|
const now = new Date();
|
|
const issueLabel = formatIssueIdentifierLink(
|
|
issue.identifier,
|
|
issue.id,
|
|
);
|
|
const blockedComment = [
|
|
`Paperclip blocked ${issueLabel} before dispatch because its workspace settings are not runnable.`,
|
|
"",
|
|
`- Code: \`${WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE}\``,
|
|
`- Reason: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE}`,
|
|
`- Next action: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION}`,
|
|
].join("\n");
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
if (issue.status !== "blocked") await notifyChatResponseAvailability(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await notifyRoutineExecutionStatus(tx, issue.id, issue.companyId);
|
|
if (issue.status === "done") {
|
|
const blocked = await issuesSvc.updateForCompany(issue.id, issue.companyId,
|
|
{ status: "blocked", executionState: null }, tx, issueActivityPublications, issuePostCommitActions);
|
|
if (!blocked) throw conflict("The task changed before its workspace settings could be checked");
|
|
} else await tx
|
|
.update(issues)
|
|
.set({
|
|
status: "blocked",
|
|
checkoutRunId: null,
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(issues.id, issue.id));
|
|
await insertIssueComment(tx, {
|
|
companyId: issue.companyId,
|
|
issueId: issue.id,
|
|
body: blockedComment,
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
});
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
|
|
payload: {
|
|
...(payload ?? {}),
|
|
issueId,
|
|
heartbeatSkip: {
|
|
code: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
|
|
reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE,
|
|
remediation:
|
|
WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: now,
|
|
});
|
|
await logActivity(tx as unknown as Db, {
|
|
companyId: issue.companyId,
|
|
actorType: "system",
|
|
actorId: "system",
|
|
agentId,
|
|
runId: null,
|
|
action: "issue.workspace_preflight_blocked",
|
|
entityType: "issue",
|
|
entityId: issue.id,
|
|
details: {
|
|
code: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
|
|
reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE,
|
|
remediation: WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION,
|
|
requestedReason: reason,
|
|
source,
|
|
triggerDetail,
|
|
resolvedMode,
|
|
resolvedStrategy,
|
|
hasResolvablePriorSessionWorkspace,
|
|
},
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
}
|
|
|
|
if (activeExecutionRun) {
|
|
// The case ledger owns retry. Do not place its exact receipt in a
|
|
// generic deferred queue that cannot retain handoff authority.
|
|
if (caseRequest) return { kind: "deferred" as const };
|
|
// The resolved action is already a durable retry outbox. Do not merge
|
|
// its fresh-session contract into unrelated work or create a second
|
|
// deferred wake that could later replay the same reconciliation.
|
|
if (reconciledSourceRunId) return { kind: "deferred" as const };
|
|
|
|
// Offer the input to the active external agent, but keep a durable queued wake
|
|
// until it reads the comment. A finish racing the event tick must
|
|
// leave unread input available for the next assignment.
|
|
const followUpProvider = parseObject(agent.adapterConfig).provider;
|
|
const museFollowUp = followUpProvider === "muse";
|
|
const bindingId = readNonEmptyString(parseObject(agent.adapterConfig)[museFollowUp ? "museBindingId" : followUpProvider === "grok_bot" ? "externalBindingId" : "dotBindingId"]);
|
|
const followUpPayloadKey = museFollowUp ? "museAssignmentFollowUp" : "dotAssignmentFollowUp";
|
|
let followUp: false | { assignmentId: string; consumed: boolean } = false;
|
|
let pendingFollowUps: typeof agentWakeupRequests.$inferSelect[] = [];
|
|
if (agent.adapterType === "paperclip_runner" && ["openai_dot", "grok_bot", "muse"].includes(String(followUpProvider))
|
|
&& bindingId && activeExecutionRun.agentId === agentId && issue.assigneeAgentId === agentId
|
|
&& reason === "issue_commented" && wakeCommentId && opts.allowRunCoalescing !== false
|
|
&& enrichedContextSnapshot.forceFreshSession !== true && !explicitResumeSession && !opts.manualUserWake
|
|
&& !enrichedContextSnapshot.interactionId && !payload?.interactionId && !receiptRequest) {
|
|
// Receipt consumption uses the same issue -> wake -> run -> guard ->
|
|
// binding order. Lock before publishing, then reuse these wake rows.
|
|
pendingFollowUps = await tx.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, agent.companyId), eq(agentWakeupRequests.agentId, agentId),
|
|
isNull(agentWakeupRequests.runId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
|
|
sql`${agentWakeupRequests.payload}->>${followUpPayloadKey} is not null`,
|
|
)).orderBy(asc(agentWakeupRequests.createdAt), asc(agentWakeupRequests.id)).for("update");
|
|
const [currentRun] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, activeExecutionRun.id), eq(heartbeatRuns.companyId, agent.companyId),
|
|
eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.status, "running"),
|
|
eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.nativeIssueId, issue.id),
|
|
)).for("update");
|
|
if (currentRun) followUp = await withExternalAdmissionGuard(tx, agent.companyId, agentId,
|
|
() => (museFollowUp ? publishActiveMuseComment : publishActiveDotComment)(tx as unknown as Db, {
|
|
companyId: agent.companyId, agentId, bindingId, runId: currentRun.id,
|
|
issueId: issue.id, commentId: wakeCommentId,
|
|
}));
|
|
}
|
|
if (followUp) {
|
|
if (!followUp.consumed) await notifyDeferredExecutionWork(tx, { companyId: agent.companyId });
|
|
const assignmentId = followUp.assignmentId;
|
|
const pending = followUp.consumed ? undefined : pendingFollowUps.find(wake =>
|
|
wake.payload?.[followUpPayloadKey] === assignmentId &&
|
|
wake.requestedByActorType === (opts.requestedByActorType ?? null) &&
|
|
wake.requestedByActorId === (opts.requestedByActorId ?? null));
|
|
if (pending) await tx.update(agentWakeupRequests).set({
|
|
payload: withQueuedCommentIdsInWakePayload(pending.payload,
|
|
[...new Set([...queuedCommentIdsFromWakePayload(pending.payload), wakeCommentId!])]),
|
|
coalescedCount: pending.coalescedCount + 1, updatedAt: new Date(),
|
|
}).where(and(eq(agentWakeupRequests.id, pending.id), eq(agentWakeupRequests.companyId, agent.companyId),
|
|
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
else await tx.insert(agentWakeupRequests).values({ companyId: agent.companyId, agentId, source, triggerDetail,
|
|
reason, payload: withQueuedCommentIdsInWakePayload({ ...payload, issueId: issue.id,
|
|
[followUpPayloadKey]: followUp.assignmentId,
|
|
[DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot }, [wakeCommentId!]),
|
|
status: followUp.consumed ? "coalesced" : "deferred_issue_execution",
|
|
runId: followUp.consumed ? activeExecutionRun.id : null,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null, idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: followUp.consumed ? new Date() : null });
|
|
return { kind: "coalesced" as const, run: activeExecutionRun };
|
|
}
|
|
|
|
const admissionScope = wakeQueue.createAdmissionTransactionScope(
|
|
agent.companyId,
|
|
tx as unknown as Db,
|
|
);
|
|
const admission = await wakeQueue.admitWakeBehindIssueExecution(
|
|
admissionScope,
|
|
{
|
|
companyId: agent.companyId,
|
|
issueId: issue.id,
|
|
agentId,
|
|
agentNameKey,
|
|
issueExecutionAgentNameKey: issue.executionAgentNameKey,
|
|
activeExecutionRun: {
|
|
id: activeExecutionRun.id,
|
|
agentId: activeExecutionRun.agentId,
|
|
status: activeExecutionRun.status,
|
|
contextSnapshot: activeExecutionRun.contextSnapshot,
|
|
wakeupRequestId: activeExecutionRun.wakeupRequestId,
|
|
},
|
|
allowRunCoalescing: isConversation(issue) ? false : opts.allowRunCoalescing,
|
|
durableReceipt: receiptRequest
|
|
? {
|
|
id: receiptRequest.id,
|
|
requestedAt: receiptRequest.requestedAt,
|
|
}
|
|
: undefined,
|
|
reason,
|
|
liveRunExecutions,
|
|
wakeCommentId,
|
|
forceFreshSession:
|
|
enrichedContextSnapshot.forceFreshSession === true,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
source,
|
|
triggerDetail,
|
|
payload,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
},
|
|
);
|
|
|
|
if (admission.kind === "coalesced") {
|
|
return {
|
|
kind: "coalesced" as const,
|
|
run: admission.run as typeof heartbeatRuns.$inferSelect,
|
|
};
|
|
}
|
|
if (admission.kind === "deferred") {
|
|
return { kind: "deferred" as const, admitted: true };
|
|
}
|
|
// admission.kind === "proceed": no active run absorbed this wake,
|
|
// so fall through to the ordinary queue path below.
|
|
}
|
|
|
|
// PAP-13775: no live run holds the lock, so this wake would start a
|
|
// fresh adapter session. If this agent's recent runs on this issue
|
|
// keep succeeding without any issue-visible progress and the wake
|
|
// carries no new information, hold it back for an escalating cooldown
|
|
// so external pollers/reconcilers can't storm full-price sessions.
|
|
// Server-side recovery retries insert runs directly and never reach
|
|
// this gate. A verified durable case receipt carries a new phase;
|
|
// its agent-authored instruction comment is not an event-free wake.
|
|
if (
|
|
!caseRequest &&
|
|
isThrottleCandidateIssueRewake({
|
|
reason,
|
|
wakeCommentId: wakeCommentId ?? null,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
forceFreshSession:
|
|
enrichedContextSnapshot.forceFreshSession === true,
|
|
hasExplicitResume: Boolean(explicitResumeSession),
|
|
})
|
|
) {
|
|
const throttleNow = new Date();
|
|
const recentTerminalRuns = await tx
|
|
.select({
|
|
id: heartbeatRuns.id,
|
|
status: heartbeatRuns.status,
|
|
finishedAt: heartbeatRuns.finishedAt,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, agent.companyId),
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
sql`${heartbeatRuns.finishedAt} is not null`,
|
|
gte(
|
|
heartbeatRuns.finishedAt,
|
|
new Date(throttleNow.getTime() - ISSUE_REWAKE_LOOKBACK_MS),
|
|
),
|
|
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`,
|
|
),
|
|
)
|
|
.orderBy(desc(heartbeatRuns.finishedAt))
|
|
.limit(ISSUE_REWAKE_RUN_SAMPLE_LIMIT);
|
|
|
|
if (recentTerminalRuns.length > 0) {
|
|
const sampleRunIds = recentTerminalRuns.map(
|
|
(sampleRun) => sampleRun.id,
|
|
);
|
|
const progressRows = await tx
|
|
.select({ runId: activityLog.runId })
|
|
.from(activityLog)
|
|
.where(
|
|
and(
|
|
eq(activityLog.companyId, agent.companyId),
|
|
eq(activityLog.entityType, "issue"),
|
|
eq(activityLog.entityId, issue.id),
|
|
inArray(activityLog.runId, sampleRunIds),
|
|
inArray(
|
|
activityLog.action,
|
|
ISSUE_PROGRESS_ACTIVITY_ACTIONS,
|
|
),
|
|
),
|
|
);
|
|
const lastRunFinishedAt =
|
|
recentTerminalRuns[0]?.finishedAt ?? null;
|
|
const newInputRows = lastRunFinishedAt
|
|
? await tx
|
|
.select({ id: activityLog.id })
|
|
.from(activityLog)
|
|
.where(
|
|
and(
|
|
eq(activityLog.companyId, agent.companyId),
|
|
eq(activityLog.entityType, "issue"),
|
|
eq(activityLog.entityId, issue.id),
|
|
gt(activityLog.createdAt, lastRunFinishedAt),
|
|
sql`coalesce(${activityLog.details}->>'origin', '') <> 'fast_response'`,
|
|
inArray(
|
|
activityLog.action,
|
|
ISSUE_NEW_INPUT_ACTIVITY_ACTIONS,
|
|
),
|
|
wakeCommentId && opts.requestedByActorType === "agent"
|
|
? ne(activityLog.actorType, "agent")
|
|
: undefined,
|
|
),
|
|
)
|
|
.limit(1)
|
|
: [];
|
|
|
|
const throttleDecision = evaluateIssueRewakeThrottle({
|
|
now: throttleNow,
|
|
recentTerminalRuns,
|
|
runIdsWithIssueProgress: new Set(
|
|
progressRows
|
|
.map((row) => row.runId)
|
|
.filter((runId): runId is string => Boolean(runId)),
|
|
),
|
|
// For an agent comment wake, the query excludes agent-authored
|
|
// activity while preserving genuinely new user/system input.
|
|
// Presentation/author metadata therefore cannot smuggle human
|
|
// wake privilege, nor can it mask an actual human response.
|
|
hasNewIssueInputSinceLastRun: newInputRows.length > 0,
|
|
});
|
|
|
|
if (throttleDecision.blocked) {
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "issue_rewake_throttled",
|
|
payload: {
|
|
...(payload ?? {}),
|
|
issueId,
|
|
heartbeatSkip: {
|
|
reason: "issue_rewake_throttled",
|
|
requestedReason: reason,
|
|
noProgressStreak: throttleDecision.noProgressStreak,
|
|
cooldownMs: throttleDecision.cooldownMs,
|
|
lastRunFinishedAt:
|
|
throttleDecision.lastRunFinishedAt.toISOString(),
|
|
nextAllowedAt:
|
|
throttleDecision.nextAllowedAt.toISOString(),
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: throttleNow,
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
}
|
|
}
|
|
|
|
const dailyCapBlock = await getHeartbeatDailyCapBlock(
|
|
agent,
|
|
policy,
|
|
{},
|
|
tx,
|
|
);
|
|
if (dailyCapBlock) {
|
|
if (caseRequest) waitForCaseAdmission(dailyCapBlock.reason, "Case handoff is waiting for the agent's daily limit to clear");
|
|
if (executionWaitRequestId && executionBlocker) {
|
|
continuationWait = { reason: dailyCapBlock.reason,
|
|
message: "The agent has reached its daily limit. Your message is saved until work can resume." };
|
|
return deferBlockedExecution(executionBlocker);
|
|
}
|
|
const now = new Date();
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: dailyCapBlock.reason,
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: {
|
|
reason:
|
|
"Per-agent heartbeat daily cap reached before adapter invocation.",
|
|
observed: dailyCapBlock.observed,
|
|
limit: dailyCapBlock.limit,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: now,
|
|
});
|
|
if (source === "timer") {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await tx
|
|
.update(agents)
|
|
.set({
|
|
lastHeartbeatAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(agents.id, agentId));
|
|
}
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
const explicitContinuation = await admitExplicitNativeContinuation({
|
|
db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id,
|
|
agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId,
|
|
reason: taskRetryActorId ? "task_retry" : durableRequest && !failedChatRetry ? "issue_commented" : reason,
|
|
commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId,
|
|
queuedCommentInterruptId: opts.queuedCommentInterruptId,
|
|
queuedCommentRequestId: opts.queuedCommentRequestId,
|
|
taskRetryRequestId: opts.taskRetryRequestId,
|
|
...(durableRequest?.resumeDeferred && !failedChatRetry ? { durableChatRequest: durableRequest } : {}),
|
|
onBlocked: (reason, message) => { continuationBlocked = true; continuationWait = { reason, message }; },
|
|
});
|
|
if (!explicitContinuation && (executionBlocker || (continuationBlocked && !verifiedConversationReset)) && !ownsReconstructionContinuation)
|
|
return deferBlockedExecution(continuationHold());
|
|
if (explicitContinuation) {
|
|
enrichedContextSnapshot.forceFreshSession = true;
|
|
enrichedContextSnapshot.previousRunId = explicitContinuation.previousRunId;
|
|
enrichedContextSnapshot.explicitUserContinuation = explicitContinuation;
|
|
}
|
|
|
|
if (opts.queuedCommentInterruptId && issue.status === "done") {
|
|
const reopened = await issuesSvc.updateForCompany(issue.id, issue.companyId,
|
|
{ status: "todo", executionState: null }, tx, issueActivityPublications, issuePostCommitActions);
|
|
if (!reopened) throw conflict("The task changed before its saved message could start");
|
|
await logActivity(tx as unknown as Db, {
|
|
companyId: issue.companyId, actorType: "user", actorId: opts.requestedByActorId!,
|
|
action: "issue.updated", entityType: "issue", entityId: issue.id,
|
|
details: { status: "todo", previousStatus: "done", source: "queued_comment_send", queueId: opts.queuedCommentInterruptId },
|
|
}, issueActivityPublications);
|
|
}
|
|
|
|
const wakeupValues = {
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason,
|
|
payload,
|
|
status: "queued" as const,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
};
|
|
// A deferred durable chat wake already owns this ID and source.
|
|
// Promote that receipt in the same transaction as its successor.
|
|
const wakeupRequest = durableRequest && executionWaitRequestId === durableRequest.id
|
|
? await tx.update(agentWakeupRequests).set({
|
|
payload, status: "queued", error: null, finishedAt: null, updatedAt: new Date(),
|
|
}).where(and(eq(agentWakeupRequests.id, durableRequest.id),
|
|
eq(agentWakeupRequests.companyId, agent.companyId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId)))
|
|
.returning().then(rows => rows[0])
|
|
: await tx.insert(agentWakeupRequests).values(wakeupValues).returning().then(rows => rows[0]);
|
|
if (!wakeupRequest) return { kind: "deferred" as const };
|
|
|
|
// A handoff changes the executor, not the owner of saved user input.
|
|
// Validate its exact stopped source while the issue row is locked;
|
|
// unrelated agents and dedicated continuations keep their own wakes.
|
|
const interruptedRunId = readNonEmptyString(enrichedContextSnapshot.interruptedRunId);
|
|
const handoffSource = source === "assignment" && reason === "issue_assigned" &&
|
|
issue.assigneeAgentId === agentId && interruptedRunId && isUuidLike(interruptedRunId)
|
|
? await tx.select({ agentId: heartbeatRuns.agentId }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, interruptedRunId), eq(heartbeatRuns.companyId, issue.companyId),
|
|
eq(heartbeatRuns.status, "cancelled"), eq(heartbeatRuns.errorCode, "issue_reassigned"),
|
|
ne(heartbeatRuns.agentId, agentId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`,
|
|
or(isNull(heartbeatRuns.nativeIssueId), eq(heartbeatRuns.nativeIssueId, issue.id)),
|
|
)).then(rows => rows[0] ?? null)
|
|
: null;
|
|
// External input owns an exact authorized source batch. Board input
|
|
// must retain its separate receipt, including during normal resume.
|
|
const canCoalesceComments = !durableRequest && !isConversation(issue) && opts.allowRunCoalescing !== false;
|
|
// A resumed receipt already passed admission under this issue lock.
|
|
// Consume it with its successor even when chat keeps other messages
|
|
// in separate turns, or finalization will deliver it a second time.
|
|
const pendingComments =
|
|
(executionWaitRequestId || canCoalesceComments) &&
|
|
(ownsReconstructionContinuation || !(await getExecutionBlocker(tx as unknown as Db, issue.companyId, issue.id)))
|
|
? await tx
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, issue.companyId),
|
|
inArray(agentWakeupRequests.agentId, handoffSource ? [agentId, handoffSource.agentId] : [agentId]),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
|
|
// Chat Retry already pins its complete authorized source
|
|
// batch. Keep separate board input for its own admission.
|
|
canCoalesceComments || (opts.taskRetryRequestId && !failedChatRetry)
|
|
? undefined : eq(agentWakeupRequests.id, executionWaitRequestId!),
|
|
),
|
|
)
|
|
.orderBy(asc(agentWakeupRequests.requestedAt))
|
|
: [];
|
|
const adoptedComments = pendingComments.filter((wake) => {
|
|
if (wake.id === executionWaitRequestId) return true;
|
|
if (wake.id === opts.queuedCommentInterruptId || wake.id === opts.queuedCommentRequestId) return true;
|
|
const deferredPayload = parseObject(wake.payload);
|
|
const deferredContext = parseObject(
|
|
deferredPayload[DEFERRED_WAKE_CONTEXT_KEY],
|
|
);
|
|
// Dedicated interaction wakes carry their own source and session
|
|
// contract. ID-only adoption must not erase that continuation.
|
|
return (
|
|
// Durable chat work must keep its receipt, actor, source, and
|
|
// session contract through normal promotion and authorization.
|
|
!wake.idempotencyKey?.startsWith("chat-inbound:") &&
|
|
!isInteractionResolutionWakePayload(deferredPayload) &&
|
|
!hasInteractionContinuationWakeContext(deferredContext) &&
|
|
["issue_commented", "issue_reopened_via_comment"].includes(String(deferredContext.wakeReason ?? wake.reason)) &&
|
|
queuedCommentIdsFromWakePayload(wake.payload).length > 0
|
|
);
|
|
});
|
|
let adoptedCommentIds = [
|
|
...new Set([
|
|
...adoptedComments.flatMap((wake) =>
|
|
queuedCommentIdsFromWakePayload(wake.payload),
|
|
),
|
|
...queuedCommentIdsFromRunContext(enrichedContextSnapshot),
|
|
]),
|
|
];
|
|
if (opts.queuedCommentRequestId || (opts.taskRetryRequestId && !failedChatRetry) || handoffSource) {
|
|
adoptedCommentIds = await undeliveredLegacyUserCommentIds(tx as unknown as Db,
|
|
agent.companyId, issueId, agentId, adoptedCommentIds);
|
|
}
|
|
await notifyChatRunMilestones(tx, { companyId: agent.companyId, issueId: issueId ?? undefined });
|
|
const newRun = await tx
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
...(explicitContinuation ? { id: explicitContinuationRunId } : {}),
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
scopeKind: readNonEmptyString(enrichedContextSnapshot.issueId) ? "issue" : "company",
|
|
issueId: readNonEmptyString(enrichedContextSnapshot.issueId),
|
|
invocationSource: source,
|
|
triggerDetail,
|
|
status: "queued",
|
|
responsibleUserId: await resolveQueuedResponsibleUserId(),
|
|
wakeupRequestId: wakeupRequest.id,
|
|
retryOfRunId: failedChatRetry
|
|
? durableRequest!.failedRunRetry!.failedRunId
|
|
: opts.failedRunId ?? automaticParentRunId,
|
|
contextSnapshot: adoptedComments.length
|
|
? withQueuedCommentIdsInRunContext(
|
|
enrichedContextSnapshot,
|
|
adoptedCommentIds,
|
|
adoptedComments.map((wake) => wake.id),
|
|
)
|
|
: enrichedContextSnapshot,
|
|
sessionIdBefore: explicitContinuation ? null : sessionBefore,
|
|
continuationAttempt,
|
|
...(reconciledSourceRunId
|
|
? { retryOfRunId: reconciledSourceRunId }
|
|
: {}),
|
|
...(reconciledRestoreRetryCount !== null ? {
|
|
scheduledRetryAttempt: reconciledRestoreRetryCount,
|
|
scheduledRetryReason: "transient_failure",
|
|
} : {}),
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]);
|
|
|
|
if (ownsReconstructionContinuation) {
|
|
await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
|
|
continuationDelivery: "delivered", continuationRunId: newRun.id,
|
|
})}::jsonb` }).where(and(eq(issueRecoveryActions.companyId, issue.companyId),
|
|
eq(issueRecoveryActions.id, String(enrichedContextSnapshot.recoveryActionId)),
|
|
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`));
|
|
}
|
|
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
runId: newRun.id,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
|
|
|
|
if (failedChatRetry && opts.taskRetryRequestId) {
|
|
await tx.update(agentWakeupRequests).set({
|
|
status: "coalesced", runId: newRun.id, finishedAt: new Date(), updatedAt: new Date(),
|
|
}).where(and(eq(agentWakeupRequests.companyId, issue.companyId),
|
|
eq(agentWakeupRequests.id, opts.taskRetryRequestId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
}
|
|
|
|
if (adoptedComments.length) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "coalesced",
|
|
runId: newRun.id,
|
|
finishedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
inArray(
|
|
agentWakeupRequests.id,
|
|
adoptedComments.map((wake) => wake.id),
|
|
),
|
|
);
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
payload: withQueuedCommentIdsInWakePayload(payload, adoptedCommentIds),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
|
|
}
|
|
|
|
// executionRunId is NOT stamped here (enqueueWakeup queues the run but
|
|
// doesn't start it). It will be stamped in claimQueuedRun() once the run
|
|
// transitions to "running" — Fix A (lazy locking).
|
|
|
|
return { kind: "queued" as const, run: newRun };
|
|
}))
|
|
.catch((error) => {
|
|
if (isExternalChatWaitAuthorizationContention(error))
|
|
return { kind: "deferred" as const };
|
|
throw error;
|
|
});
|
|
|
|
for (const publication of issueActivityPublications) publishActivity(publication);
|
|
await executeIssuePostCommitActions(db, issuePostCommitActions);
|
|
|
|
// Telemetry for the cancelled runs is best-effort background work.
|
|
// Fire it here and never await it: none of the lifecycle work below,
|
|
// nor this function's return, depends on it, so a slow telemetry
|
|
// lookup must not delay them.
|
|
for (const cancelledRun of cancelledRunsToEmit) {
|
|
void emitAgentTaskRun(db, cancelledRun);
|
|
}
|
|
|
|
if (outcome.kind === "durable") {
|
|
return outcome.receipt.runId ? getRun(outcome.receipt.runId) : null;
|
|
}
|
|
if (outcome.kind === "deferred" || outcome.kind === "skipped") {
|
|
return null;
|
|
}
|
|
if (outcome.kind === "coalesced") {
|
|
await startNextQueuedRunForAgent(agent.id);
|
|
return outcome.run;
|
|
}
|
|
if (outcome.kind === "replayed") {
|
|
if (outcome.run.status === "queued")
|
|
await startNextQueuedRunForAgent(agent.id);
|
|
return outcome.run;
|
|
}
|
|
|
|
const newRun = outcome.run;
|
|
publishLiveEvent({
|
|
companyId: newRun.companyId,
|
|
type: "heartbeat.run.queued",
|
|
payload: {
|
|
runId: newRun.id,
|
|
agentId: newRun.agentId,
|
|
invocationSource: newRun.invocationSource,
|
|
triggerDetail: newRun.triggerDetail,
|
|
wakeupRequestId: newRun.wakeupRequestId,
|
|
},
|
|
});
|
|
|
|
await startNextQueuedRunForAgent(agent.id);
|
|
return newRun;
|
|
}
|
|
|
|
if (durableRequest) throw new Error("chat_inbound_wakeup_binding_denied");
|
|
|
|
const activeRuns = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
inArray(heartbeatRuns.status, [
|
|
...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES,
|
|
]),
|
|
),
|
|
)
|
|
.orderBy(desc(heartbeatRuns.createdAt));
|
|
|
|
const sameScopeQueuedRun = activeRuns.find(
|
|
(candidate) =>
|
|
candidate.status === "queued" &&
|
|
isSameTaskScope(runTaskKey(candidate), taskKey),
|
|
);
|
|
const sameScopeScheduledRetryRun = activeRuns.find(
|
|
(candidate) =>
|
|
candidate.status === "scheduled_retry" &&
|
|
isSameTaskScope(runTaskKey(candidate), taskKey),
|
|
);
|
|
const sameScopeRunningRun = activeRuns.find(
|
|
(candidate) =>
|
|
candidate.status === "running" &&
|
|
isSameTaskScope(runTaskKey(candidate), taskKey),
|
|
);
|
|
const shouldQueueFollowupForRunningWake =
|
|
Boolean(sameScopeRunningRun) &&
|
|
!sameScopeQueuedRun &&
|
|
shouldQueueFollowupForRunningIssueWake({
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
wakeCommentId,
|
|
});
|
|
// Unscoped manual wakes need their own receipt and execution identity too.
|
|
const rawCoalescedTarget =
|
|
opts.allowRunCoalescing === false || opts.manualUserWake
|
|
? null
|
|
: (sameScopeQueuedRun ??
|
|
sameScopeScheduledRetryRun ??
|
|
(shouldQueueFollowupForRunningWake
|
|
? null
|
|
: (sameScopeRunningRun ?? null)));
|
|
|
|
// A reviewed chat run attests one conversation, so never fold another
|
|
// conversation's reply into it; that reply queues its own run instead.
|
|
const coalescedTargetRun = rawCoalescedTarget &&
|
|
(hasRetainedExternalChatRetry(rawCoalescedTarget.contextSnapshot) ||
|
|
wakeChatConversationsConflict(rawCoalescedTarget.contextSnapshot, enrichedContextSnapshot))
|
|
? null
|
|
: filterZombieCoalesceTarget(rawCoalescedTarget, liveRunExecutions);
|
|
|
|
if (coalescedTargetRun) {
|
|
const mergedContextSnapshot = mergeCoalescedContextSnapshot(
|
|
coalescedTargetRun.contextSnapshot,
|
|
enrichedContextSnapshot,
|
|
{
|
|
preserveExistingInteractionContinuation:
|
|
coalescedTargetRun.status === "queued" ||
|
|
coalescedTargetRun.status === "scheduled_retry",
|
|
},
|
|
);
|
|
const mergedRun = await db
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
contextSnapshot: mergedContextSnapshot,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(heartbeatRuns.id, coalescedTargetRun.id))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? coalescedTargetRun);
|
|
|
|
await db.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason,
|
|
payload,
|
|
status: "coalesced",
|
|
coalescedCount: 1,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
runId: mergedRun.id,
|
|
finishedAt: new Date(),
|
|
});
|
|
return mergedRun;
|
|
}
|
|
|
|
const queueOutcome = await db.transaction(async (tx) => {
|
|
await notifyRunDispatch(tx, { companyId: agent.companyId });
|
|
await tx.execute(
|
|
sql`select id from agents where id = ${agentId} and company_id = ${agent.companyId} for update`,
|
|
);
|
|
|
|
const dailyCapBlock = await getHeartbeatDailyCapBlock(
|
|
agent,
|
|
policy,
|
|
{},
|
|
tx,
|
|
);
|
|
if (dailyCapBlock) {
|
|
const now = new Date();
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: dailyCapBlock.reason,
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: {
|
|
reason:
|
|
"Per-agent heartbeat daily cap reached before adapter invocation.",
|
|
observed: dailyCapBlock.observed,
|
|
limit: dailyCapBlock.limit,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: now,
|
|
});
|
|
if (source === "timer") {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await tx
|
|
.update(agents)
|
|
.set({
|
|
lastHeartbeatAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(agents.id, agentId));
|
|
}
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
const wakeupRequest = await tx
|
|
.insert(agentWakeupRequests)
|
|
.values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason,
|
|
payload,
|
|
status: "queued",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]);
|
|
|
|
const newRun = await tx
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
scopeKind: readNonEmptyString(enrichedContextSnapshot.issueId) ? "issue" : "company",
|
|
issueId: readNonEmptyString(enrichedContextSnapshot.issueId),
|
|
invocationSource: source,
|
|
triggerDetail,
|
|
status: "queued",
|
|
responsibleUserId: await resolveQueuedResponsibleUserId(),
|
|
wakeupRequestId: wakeupRequest.id,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
sessionIdBefore: sessionBefore,
|
|
continuationAttempt,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]);
|
|
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
runId: newRun.id,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
|
|
|
|
return { kind: "queued" as const, run: newRun };
|
|
});
|
|
|
|
if (queueOutcome.kind === "skipped") return null;
|
|
const newRun = queueOutcome.run;
|
|
|
|
publishLiveEvent({
|
|
companyId: newRun.companyId,
|
|
type: "heartbeat.run.queued",
|
|
payload: {
|
|
runId: newRun.id,
|
|
agentId: newRun.agentId,
|
|
invocationSource: newRun.invocationSource,
|
|
triggerDetail: newRun.triggerDetail,
|
|
wakeupRequestId: newRun.wakeupRequestId,
|
|
},
|
|
});
|
|
|
|
await startNextQueuedRunForAgent(agent.id);
|
|
|
|
return newRun;
|
|
}
|
|
|
|
/**
|
|
* Native status commitment deliberately persists dependency/parent wake
|
|
* intents in the same transaction as the authoritative status projection.
|
|
* Those rows are not runnable until the heartbeat scheduler has applied its
|
|
* normal policy, workspace, concurrency, and responsible-user checks. Bridge
|
|
* the durable intent into that scheduler here instead of treating a bare
|
|
* `agent_wakeup_requests` row as if it were already a queued heartbeat run.
|
|
*
|
|
* The intent is claimed before dispatch. A deterministic dispatcher actor id
|
|
* lets a later sweep recover the narrow process-crash window after the real
|
|
* wake was inserted but before the intent was linked to it. Dispatch failure
|
|
* only requeues the intent; it never changes the provider run outcome.
|
|
*/
|
|
type NativeStatusWakeDispatchInput = {
|
|
companyId?: string;
|
|
limit?: number;
|
|
staleClaimMs?: number;
|
|
requestId?: string;
|
|
onDeferredDeadline?: (at: number) => void;
|
|
onError?: (error: unknown) => void;
|
|
};
|
|
type NativeStatusWakeDelivery = { runId: string | null; status: string };
|
|
|
|
function createNativeStatusWakeBatch() {
|
|
const deliveredByIssueScope = new Map<string, NativeStatusWakeDelivery>();
|
|
return (input: NativeStatusWakeDispatchInput = {}) => dispatchNativeStatusWakeBatch(input, deliveredByIssueScope);
|
|
}
|
|
|
|
function dispatchPendingNativeStatusWakeups(input: NativeStatusWakeDispatchInput = {}) {
|
|
return createNativeStatusWakeBatch()(input);
|
|
}
|
|
|
|
async function dispatchNativeStatusWakeBatch(input: NativeStatusWakeDispatchInput,
|
|
deliveredByIssueScope: Map<string, NativeStatusWakeDelivery>,
|
|
) {
|
|
const now = new Date();
|
|
const staleClaimMs = Math.max(1_000, input.staleClaimMs ?? 60_000);
|
|
const candidates = await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
input.companyId
|
|
? eq(agentWakeupRequests.companyId, input.companyId)
|
|
: undefined,
|
|
input.requestId ? eq(agentWakeupRequests.id, input.requestId) : undefined,
|
|
eq(agentWakeupRequests.requestedByActorType, "system"),
|
|
eq(agentWakeupRequests.requestedByActorId, "native-status-committer"),
|
|
inArray(agentWakeupRequests.status, ["queued", "claimed"]),
|
|
isNull(agentWakeupRequests.runId),
|
|
),
|
|
)
|
|
.orderBy(asc(agentWakeupRequests.requestedAt))
|
|
.limit(Math.max(1, Math.min(input.limit ?? 100, 500)));
|
|
|
|
let dispatched = 0;
|
|
let recovered = 0;
|
|
let deferred = 0;
|
|
|
|
for (const candidate of candidates) {
|
|
const payload = parseObject(candidate.payload);
|
|
const wakeContext = parseObject(payload._paperclipWakeContext);
|
|
const issueId =
|
|
readNonEmptyString(payload.issueId) ??
|
|
readNonEmptyString(payload.taskId) ??
|
|
readNonEmptyString(wakeContext.issueId) ??
|
|
null;
|
|
const dispatchActorId = `native-status-wake-dispatch:${candidate.id}`;
|
|
const existingDispatch = await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, candidate.companyId),
|
|
eq(agentWakeupRequests.requestedByActorType, "system"),
|
|
eq(agentWakeupRequests.requestedByActorId, dispatchActorId),
|
|
),
|
|
)
|
|
.orderBy(desc(agentWakeupRequests.requestedAt))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (existingDispatch) {
|
|
const recoveredStatus = existingDispatch.runId
|
|
? "coalesced"
|
|
: existingDispatch.status === "deferred_issue_execution"
|
|
? "coalesced"
|
|
: existingDispatch.status;
|
|
await db.transaction(async tx => {
|
|
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: recoveredStatus,
|
|
runId: existingDispatch.runId,
|
|
finishedAt:
|
|
(existingDispatch.runId ?? existingDispatch.finishedAt)
|
|
? (existingDispatch.finishedAt ?? now)
|
|
: null,
|
|
error: existingDispatch.error,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, candidate.id),
|
|
isNull(agentWakeupRequests.runId),
|
|
),
|
|
);
|
|
});
|
|
recovered += 1;
|
|
continue;
|
|
}
|
|
|
|
if (
|
|
candidate.status === "claimed" &&
|
|
candidate.claimedAt &&
|
|
now.getTime() - candidate.claimedAt.getTime() < staleClaimMs
|
|
) {
|
|
input.onDeferredDeadline?.(candidate.claimedAt.getTime() + staleClaimMs);
|
|
deferred += 1;
|
|
continue;
|
|
}
|
|
|
|
const claimed = await db
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "claimed",
|
|
claimedAt: now,
|
|
error: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, candidate.id),
|
|
isNull(agentWakeupRequests.runId),
|
|
candidate.status === "claimed"
|
|
? eq(agentWakeupRequests.status, "claimed")
|
|
: eq(agentWakeupRequests.status, "queued"),
|
|
),
|
|
)
|
|
.returning({ id: agentWakeupRequests.id });
|
|
if (claimed.length === 0) continue;
|
|
|
|
const scopeKey = issueId
|
|
? `${candidate.companyId}:${candidate.agentId}:${issueId}:${readNativeReviewAssignmentContext(wakeContext)?.nativeReviewInteractionId ?? ""}`
|
|
: null;
|
|
const priorDelivery = scopeKey
|
|
? deliveredByIssueScope.get(scopeKey)
|
|
: null;
|
|
if (priorDelivery) {
|
|
await db.transaction(async tx => {
|
|
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "coalesced",
|
|
runId: priorDelivery.runId,
|
|
finishedAt: new Date(),
|
|
error: null,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, candidate.id));
|
|
});
|
|
recovered += 1;
|
|
continue;
|
|
}
|
|
|
|
let completedOnboardingGuard: WakeupOptions["issueStateGuard"];
|
|
if (issueId) {
|
|
const targetIssue = await db
|
|
.select({
|
|
status: issues.status,
|
|
statusVersion: issues.statusVersion,
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
})
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, candidate.companyId),
|
|
),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
const nativeReview = candidate.reason === "native_completion_review"
|
|
? await getNativeReviewAssignment(db, {
|
|
companyId: candidate.companyId, issueId, agentId: candidate.agentId,
|
|
contextSnapshot: wakeContext,
|
|
})
|
|
: null;
|
|
const onboardingResultReport = targetIssue?.status === "done" && await isCompletedOnboardingHandoffWake(db, {
|
|
companyId: candidate.companyId, issueId, agentId: candidate.agentId,
|
|
reason: candidate.reason, contextSnapshot: wakeContext,
|
|
});
|
|
if (onboardingResultReport) completedOnboardingGuard = { assigneeAgentId: candidate.agentId, statuses: ["done"], statusVersion: targetIssue!.statusVersion };
|
|
if (
|
|
!targetIssue ||
|
|
(["done", "cancelled"].includes(targetIssue.status) && !onboardingResultReport) ||
|
|
(targetIssue.assigneeAgentId !== candidate.agentId && !nativeReview) ||
|
|
(candidate.reason === "native_completion_review" && !nativeReview)
|
|
) {
|
|
await db.transaction(async tx => {
|
|
await notifyChatInboundWakeNotices(tx, { companyId: candidate.companyId, issueId });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "skipped",
|
|
finishedAt: new Date(),
|
|
error: !targetIssue
|
|
? "Native status wake target no longer exists"
|
|
: ["done", "cancelled"].includes(targetIssue.status)
|
|
? `Native status wake target is already ${targetIssue.status}`
|
|
: "Native status wake target has a different assignee",
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, candidate.id));
|
|
});
|
|
recovered += 1;
|
|
continue;
|
|
}
|
|
}
|
|
|
|
try {
|
|
const wakeRun = await enqueueWakeup(candidate.agentId, {
|
|
allowRunCoalescing: false,
|
|
source: candidate.source as WakeupOptions["source"],
|
|
triggerDetail: (candidate.triggerDetail ??
|
|
"system") as WakeupOptions["triggerDetail"],
|
|
reason: candidate.reason,
|
|
payload,
|
|
idempotencyKey: candidate.idempotencyKey,
|
|
requestedByActorType: "system",
|
|
requestedByActorId: dispatchActorId,
|
|
...(completedOnboardingGuard ? { issueStateGuard: completedOnboardingGuard } : {}),
|
|
contextSnapshot: {
|
|
...wakeContext,
|
|
...(issueId ? { issueId, taskId: issueId } : {}),
|
|
wakeReason: candidate.reason,
|
|
source: "native_status_decision",
|
|
statusDecisionSource: "native_status_decision",
|
|
nativeStatusWakeIntentId: candidate.id,
|
|
},
|
|
});
|
|
|
|
const delivered = await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, candidate.companyId),
|
|
eq(agentWakeupRequests.requestedByActorType, "system"),
|
|
eq(agentWakeupRequests.requestedByActorId, dispatchActorId),
|
|
),
|
|
)
|
|
.orderBy(desc(agentWakeupRequests.requestedAt))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
// The committer intent is the durable outbox entry. When admission is
|
|
// blocked, enqueueWakeup creates a separate deferred dispatch receipt;
|
|
// leave the original intent coalesced so the release drain cannot
|
|
// promote both rows (the original has no nativeStatusWakeIntentId
|
|
// provenance and would otherwise run once before the dispatch receipt).
|
|
const deliveredStatus = delivered?.status ?? "queued";
|
|
await db.transaction(async tx => {
|
|
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: wakeRun || deliveredStatus === "deferred_issue_execution"
|
|
? "coalesced"
|
|
: deliveredStatus,
|
|
runId: wakeRun?.id ?? delivered?.runId ?? null,
|
|
finishedAt:
|
|
wakeRun || delivered?.finishedAt
|
|
? (delivered?.finishedAt ?? new Date())
|
|
: null,
|
|
error: delivered?.error ?? null,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, candidate.id));
|
|
});
|
|
|
|
if (scopeKey) {
|
|
deliveredByIssueScope.set(scopeKey, {
|
|
runId: wakeRun?.id ?? delivered?.runId ?? null,
|
|
status: wakeRun ? "coalesced" : (delivered?.status ?? "queued"),
|
|
});
|
|
}
|
|
|
|
if (wakeRun) dispatched += 1;
|
|
else deferred += 1;
|
|
} catch (error) {
|
|
input.onError?.(error);
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "queued",
|
|
claimedAt: null,
|
|
error:
|
|
error instanceof Error
|
|
? error.message.slice(0, 1_000)
|
|
: String(error).slice(0, 1_000),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, candidate.id),
|
|
eq(agentWakeupRequests.status, "claimed"),
|
|
isNull(agentWakeupRequests.runId),
|
|
),
|
|
);
|
|
logger.warn(
|
|
{
|
|
err: error,
|
|
wakeupRequestId: candidate.id,
|
|
agentId: candidate.agentId,
|
|
},
|
|
"failed to dispatch persisted native status wake intent",
|
|
);
|
|
deferred += 1;
|
|
}
|
|
}
|
|
|
|
return { scanned: candidates.length, dispatched, recovered, deferred };
|
|
}
|
|
|
|
return {
|
|
enqueueWakeup,
|
|
dispatchPendingNativeStatusWakeups,
|
|
createNativeStatusWakeBatch,
|
|
startNextQueuedRunForAgent,
|
|
countRunningRunsForAgent,
|
|
releaseRunClaimedJustBeforeSuppression,
|
|
claimQueuedRun,
|
|
withChatControlRecoveryGate,
|
|
trackWakeup,
|
|
resumeQueuedRuns,
|
|
resumeDeferredExecutionReceipts,
|
|
resumeQueuedRunsAfterDrain,
|
|
resumeFreshQueuedRunsAtStartup,
|
|
parseHeartbeatPolicy,
|
|
nextTimerHeartbeatAt,
|
|
claimDueTimerHeartbeat,
|
|
};
|
|
}
|