mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - During a release swap, the outgoing server stops and the incoming server takes over the live runs on remote Runners > - The incoming server reached those runs late: it waited up to 60 s for the outgoing server's controller lease to lapse, re-verified each run's whole log from object storage, and deferred a reattach when the agent directory was busy. Queued starts competed with the reattaches for the same resources. > - On hosted staging, the later a live run was reached, the more likely it was to fail. Runs reached in under 150 s never failed; runs reached after 420 s almost always failed. > - This pull request hands live runs over at the start of shutdown, makes a reattach skip work that the same run already did, and holds queued starts while handed-over runs are being reached, with a 180 s cap. > - The benefit is that live runs are reached within seconds of a swap instead of minutes, so they continue instead of failing. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each run that crossed a swap we measured the gap from the last Runner output to the moment the new server recorded the Runner's process identity: - gap under 150 s: 0 of 9 runs failed - gap over 420 s: 14 of 15 runs failed - median gap: 180 s for runs that resumed, 470 s for runs that failed In one traced run the incoming server waited on `remote_controller_lease_live` three times (09:14:21, 09:15:18, 09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the run's whole log from object storage. A control run spent 23 s in `task.prepare`. Queued starts used the same preparation path, the same 8 object-I/O slots and the same database pool, and nothing gave a reattach priority. The first-attempt failures were `PRP command session.snapshot timed out` and `codex_history_read_failed`. The retry then failed with `runner_harness_state_mismatch: resumed_not_suspended`, `runner_remote_process_identity_unavailable` or `provider_turn_lost_on_restore`. **Expected behavior** Every live run is reached within about 30 s of the swap. Queued work starts after that, or after a bounded wait. **Steps to reproduce** 1. Run a two-server release swap with live remote runs and queued runs. 2. Observe `native.recovery.transition` "waiting" events with `remote_controller_lease_live`, then the claim, then a long `heartbeat.prepare.run_output` span. 3. The new tests in `reattach-phase.test.ts` and `heartbeat-native-early-hand-off.test.ts` model this. **Paperclip version or commit** `master` at426e8a35dc. ## What Changed - Early hand-off at shutdown (`heartbeat/recovery.ts`, `server/src/index.ts`): in deploy-overlap mode, the outgoing server releases the remote controller lease of its live runs at SIGTERM (`recoveryState` `awaiting_runner_reattach`). A run another server already took over is left alone. The incoming server no longer waits for the lease to lapse. - Reattach fast path: - `heartbeat/run-output.ts`, `native-runtime/native-run-log-export.ts`: a reattach starts without waiting for the existing log to be re-verified. - `heartbeat/agent-directory-busy.ts`, `heartbeat.ts`: a reattach that meets a busy agent directory waits in process (at most 30 s) instead of being deferred to a full re-preparation. An ordinary start is still deferred at once, with no added wait. - Reattach phase lifecycle (`heartbeat/reattach-phase.ts`): the watch is a worker on the shared delivery coordinator, queue `reattach-phase`. - Trigger: one startup listing of a previous server's live runs; then the hand-over notification (`notifyReattachHandOver`, sent in the hand-over transaction); the earliest lease deadline it watches (a killed owner); and a 2 s re-check only while handed-over runs are still pending, which ends with the phase (cap 180 s). The bounded re-check exists because a run can also end on another path that sends no notification to this queue. - Source of truth: `native_run_finalizations` and `heartbeat_runs` rows. - Quiet condition: nothing pending and no watched lease, so no deadline and no SQL. It stops with the coordinator at shutdown. - Reached hook for every driver (`native-session-executor.ts`): a reattached run counts as reached when the native session runtime hands the executor its recovered or opened session (`onSession`). Every driver reaches that point only after the Runner answered this server's attach and recovery commands, including an adopted `RunnerdExternalDriver` Runner, which records no new process identity. The identity hook stays as an earlier signal. - Claim wake: when the reattach-phase worker finds runs that just became claimable (hand-over notification or a killed owner's lease deadline), it starts this server's restart-recovery claim for them at once. A validation cell showed the claim otherwise waited about 21.5 s after the hand-over. - `recovery/assigned-work-sweep.ts`: the `deferred_reattach_phase` return now sits before every action path of `act()`, except starting an overdue saved-work recovery (it resumes an existing run). - The in-transaction admission re-check of a queued claim (`confirmRunAdmissionStillOpen`) also honours the reattach phase. A phase that begins during a claim returns the run to the queue through the existing rollback. - A reattach whose log re-verification runs in the background awaits it before the log is finalized (`settleDeferredLogRecovery`, both finalization sites in `run-completion.ts`). - Reattach phase (`heartbeat/reattach-phase.ts`, `heartbeat/run-admission.ts`): `isReattachPhaseActive()` is true while runs that a previous controller handed over (lease released or observed expired, or claimed by this server's restart recovery) are claimable but not yet reached. A run is reached when its process identity is recorded. Authentication is logged as a mark. The cap (180 s, `PAPERCLIP_REATTACH_PHASE_CAP_MS`) counts from the moment the runs became claimable. Later batches join or re-arm the phase. Runs reached after 150 s are counted and logged. - Gates on the phase: - `claimQueuedRun`, start-next, resume-queued and dispatch in `heartbeat/queue.ts`, `run-lifecycle.ts` and `native-runner-execution-driver.ts` leave queued runs queued. - `execution-recovery-resolution.ts`: one line, the first line of `deliverReconciledExecutions`. - `recovery/assigned-work-sweep.ts`: one line, outcome `deferred_reattach_phase`. - `heartbeat/recovery.ts`: a one-line claim hook (`noteReattachRunsClaimed`) after restart recovery claims runs. - The owner of those three files agreed to these one-line gates. - `tests/restart-chaos/daytona-image.ts`: `PAPERCLIP_CHAOS_TEST_ACCEPT_IMAGE_CONTENT_ID` lets a local wave run a controller-only branch against an existing Daytona image. It accepts a content-ID mismatch only for the exact content ID of the named image, and it prints a TEST OVERRIDE warning. This is harness-only. ## Verification Red on master, green on this branch: - `heartbeat/reattach-phase.test.ts` (27 tests). Five of the six two-server scenarios were red on the earlier phase timing: - `does not hold queued starts while the previous owner is alive and not handing over` - `closes the gate when the previous owner hands its runs over, and counts reach latency from the hand-over` - `runs the cap from the hand-over, so a wedged run ends the phase without the overlap eating it` - `begins when a killed owner's lease is observed expired, without any hand-over` - `re-arms for a later batch, including runs the previous owner started after discovery` - `closes the gate at once when this server claims a run between two lease reads, dated from the hand-over` - plus `claimQueuedRun leaves a queued run queued and unstarted, in the admission gate's shape`, `is the same gate the assigned-work sweep and reconciled continuation delivery read`, `ends at the cap, counted from when the runs became claimable, so one wedged Runner cannot hold run starts` - `heartbeat-native-early-hand-off.test.ts`: `releases the remote controller lease of %i live runs at once` (1, 20, 40 runs; 40 runs in 56 ms), `leaves a run another server already took over alone`, `holds the incoming server's queued starts only from the hand-over, not while the outgoing server still drives its runs` - `heartbeat/run-output.test.ts`: `lets a reattach start before its existing log is re-verified` - `heartbeat/agent-directory-busy.test.ts`: `defers an ordinary start at once, with no wait added to the run-start path`, `lets a reattach wait its turn in process instead of a deferred full re-preparation`, `still defers a reattach once its bounded wait runs out` - `tests/restart-chaos/daytona-image.test.ts`: `accepts a mismatch only for the exact image content ID named by the test override` - Review fixes (red before the fix, green after): `reattach-phase.test.ts` `starts the claim of handed-over runs within a second of the hand-over notification`; `assigned-work-invariant.test.ts` `takes no action while handed-over live runs are being reached, and acts once the phase ends`; `heartbeat-native-early-hand-off.test.ts` `never lets the outgoing server claim back a run it handed off, while the incoming server may take it over`; `reattach-phase.test.ts` `retries a failed first listing instead of going quiet, and still finds the previous owner's runs`, `keeps a hand-over notification whose listing failed until a listing succeeds`, `moves the cap earlier when a claimed run turns out to have been handed over before its claim`; `queue.test.ts` `returns the claim to the queue when a reattach phase begins while the claim is being checked`; `run-completion.test.ts` `finalizes the run log in completeRun|failRun only after a reattach's deferred log recovery settles`. `native-session-executor.test.ts` `ends the reattach phase once a reattached %s session is driven by this server again, without the cap or manual settling` over codex_app_server, opencode_server, acpx_runtime, muse_external, grok_bot_mcp and openai_dot_mcp (6 of 6 red without the hook; the test drives the executor with a mocked session runtime, not each real driver). Lifecycle tests for the new worker: `does no database work and keeps no deadline on an instance with nothing to reattach`, `waits for the hand-over notification or the lease deadline instead of polling while the previous owner drives its runs`, `begins at a killed owner's lease deadline, without any hand-over or notification`, `stops with the delivery coordinator at shutdown and ignores a later hand-over`. Local runs on this branch: `tsc --noEmit` is clean for `server`. Passing on this branch: `reattach-phase.test.ts`, `heartbeat-native-early-hand-off.test.ts`, `run-output.test.ts` (53 tests together), `agent-directory-busy.test.ts`, `heartbeat-native-restart-suspension.test.ts` (115 of 115, alone), the new cases in `queue.test.ts` and `run-completion.test.ts`. Under load 30 to 96 on this shared machine, the full `queue.test.ts` and `recovery-wake-herd.test.ts` runs stalled in their embedded-Postgres setup; CI supplies them. Daytona results that exist (local servers with Daytona sandboxes, release-swap cells). They ran on a combined branch with this PR, #21462, #21442 and #21362 (merged), so they cannot be credited to this PR alone: - G5 baseline (masterd1d71a7755): SIGTERM to PRP authentication 21.5 to 38 s at swap 1. - G5c (combined branch 588c6ba267): SIGTERM to authentication 6.2 to 7.7 s at swap 1. All runs that crossed swap 2 were adopted on the same Runner process. The staging chain (`provider recovery timed out after 900000ms`, then `resumed_not_suspended`) did not recur. - Caveat: G5c ran before the phase timing fix in this PR. In G5c the phase opened at process start, its 180 s cap ran out during the overlap, and swap 2 ended `cap_reached` with 0 runs reached, 1 s before the lease claims. The timing fix (the phase begins when the runs become claimable) has not run on Daytona. - G6c (12 live runs and 20 queued, two graceful swaps, no injection, combined branch 588c6ba267): 0 failed runs at either swap; 11 of 11 survivors at both. - Swap 1: hand-off of 11 runs in 291 ms; claims 0.3 to 2.9 s after hand-off; authentication 7.6 to 12.5 s and identity 8.3 to 13.3 s after SIGTERM. The 30 s target was met. - Swap 2: hand-off took 8,562 ms and `hot_restart_prepare` timed out at 19 s, so the outgoing server drained gracefully. The incoming server did not claim on the hand-over; it claimed only when the outgoing server exited, 33 s after SIGTERM. Authentication was 46 to 54 s and identity 47 to 57 s after SIGTERM. The target was missed. Why the claims waited for the exit is not explained. - The host load was 49 to 70 from other work. 7 of 32 tasks timed out in the queue or budget after the swaps; we suspect the overload. ## Risks - Product trade-off: while handed-over live runs are being reached, queued starts wait, up to the 180 s cap. On an instance with no handed-over runs, nothing waits. - Not proven: two-server behavior with the phase timing fix and the notification-driven watch (no cell has run them); warm standby becoming active; runs that the outgoing server starts after discovery. - Early hand-off at SIGTERM orphans a server-side exec tool that is in flight. With #21442 the next controller settles it as interrupted with an unknown outcome; it is never run again. - Swap 2 in G6c: the incoming server claimed only at the outgoing server's exit. That is unexplained. - Cross-session ownership: the gates in `execution-recovery-resolution.ts` and `recovery/assigned-work-sweep.ts`, and the claim hook in `heartbeat/recovery.ts`, are in files that the restart-recovery work owns. They are one-line gates agreed with that owner. That owner said the gate did not need to be inside the claim transaction; that was before the phase could begin during a claim. After review, the in-transaction re-check in `heartbeat/queue.ts` also reads the phase, through the existing rollback. #21427 (open) also edits `execution-recovery-resolution.ts`, `heartbeat/queue.ts` and `recovery/assigned-work-sweep.ts`; whichever merges second needs a rebase. `native-runtime/native-restart-recovery.ts` (claim logic, owned by the restart-recovery work): a 3-line guard in `claimNativeRestartRecoveries` returns a run this process handed off at shutdown as `blocked` / `handed_off_to_next_controller` before takeover is evaluated, so the outgoing server never claims back a run it handed off. #21478 (merged) added a sweep path that starts an overdue saved-work recovery; it is deliberately not gated by the reattach phase, because it resumes an existing run (reattach-like work), which the phase makes room for rather than competes with. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
5799 lines
275 KiB
TypeScript
5799 lines
275 KiB
TypeScript
import { QUEUED_MESSAGES_DELIVERED_REASON } from "../../modules/wake-queue/index.js";
|
|
import { connectionIntentService } from "../connection-intents.js";
|
|
import { verifyBlockerStallWake } from "../recovery/blocker-stall-wakeups.js";
|
|
import { CONTEXT_COMPACTION_WAKE_REASON, ContextCompactionError, isContextCompactionRun, readContextCompactionRequest } from "../context-compaction.js";
|
|
import { isProcessRunAdmissionClosed, isReattachPhaseActive as isProcessReattachPhaseActive } from "./run-admission.js";
|
|
import { continueFreshInTransaction, hasPendingFreshContinuation, isFreshIsolatedContinuationWake, settleCoalescedFreshContinuationHolds } from "../fresh-continuation-fallback.js";
|
|
import { deliverContinuationAsRun, foldWaitingRetriesIntoContinuation, foldWaitingRetriesIntoRecordedContinuation, undeliveredContinuation,
|
|
undeliveredContinuationContext } from "../fresh-continuation-retry-fold.js";
|
|
import { refreshRunStartYieldIfStale } from "../run-start-handover.js";
|
|
import { deliveredQueuedCommentIds } from "../queued-comment-delivery.js";
|
|
import { hasAutomaticInteractionDeliveryHold, INTERACTION_DELIVERY_DENIED, recheckQueuedRunInteractions, withInteractionDeliveryOrigin } from "../agent-interaction-delivery.js";
|
|
import { deferredWorkIntentWait } from "../work-intent-outbox.js";
|
|
import { notifyIssueRecovery } from "../issue-recovery-notifications.js";
|
|
import { createHeartbeatTimerPolicy } from "./timer-policy.js";
|
|
import { QUEUED_RUN_AGING_MS, createQueuedRunDispatcher, hasQueuedRunAged, queuedRunDispatchClass } from "./queued-run-dispatch-order.js";
|
|
import { queuedRunDispatchFacts } from "./queued-run-line.js";
|
|
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
|
|
import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../deferred-execution-wait.js";
|
|
import { assignmentWakePayloadSql, isAssignmentWakePayload, notifyDeferredExecutionWork } from "../deferred-execution-notifications.js";
|
|
import type { DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js";
|
|
import { notifyQueuedRunDispatch, notifyRunDispatch } from "../run-dispatch-notifications.js";
|
|
import { notifyStatusCardIssueChange } from "../status-card-notifications.js";
|
|
import { notifyRoutineExecutionStatus } from "../routine-execution-status-notifications.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
|
|
import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js";
|
|
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
|
|
import { notifyUnsafeWorkspaceExportRecovery, notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
|
|
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
|
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
|
|
import { notifyScheduledRetryCancellation } from "../scheduled-retry-cancellation-notifications.js";
|
|
import { notifyChatResponseAvailability } from "../chat-response-availability.js";
|
|
import { notifyChatInboundWakeNotices } from "../chat-inbound-wake-notice-notifications.js";
|
|
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
|
|
import { assertTaskRetryConversationAllowed, readTaskRetryIntent, readTaskRetryBoardMessageOrigin, retireSupersededTaskRetry, taskRetryMutationAllowed } from "../task-retry.js";
|
|
import { insertIssueComment } from "../issue-comment-writes.js";
|
|
import { withExternalAdmissionGuard, assertNoExternalOverlap } from "../../modules/external-agents/index.js";
|
|
import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js";
|
|
import { normalizeAgentNameKey } from "./retries.js";
|
|
import { publishActiveDotComment } from "../dot-assignment-follow-up.js";
|
|
import { publishActiveMuseComment } from "../muse-assignment-follow-up.js";
|
|
import {
|
|
type WakeupOptions,
|
|
mergeCoalescedContextSnapshot,
|
|
enrichWakeContextSnapshot,
|
|
hasInteractionContinuationWakeContext,
|
|
isInteractionResolutionWakePayload,
|
|
} from "./run-preparation.js";
|
|
import { resolveExecutionWorkspaceReuseRequestForIssue } from "./workspaces.js";
|
|
import {
|
|
CHAT_COMPLETION_WAKE_REASON,
|
|
isCompletedOnboardingHandoffWake,
|
|
} from "../chat-completion-delivery.js";
|
|
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
|
|
import { isVerifiedDependencyInteractionResponse, readQueuedInteractionResponse } from "../queued-interaction-response.js";
|
|
import {
|
|
isConversation,
|
|
isConversationReset,
|
|
isConversationExecutionWake,
|
|
isWaitingConversation,
|
|
} from "../agent-conversations.js";
|
|
import { legacyControllerClaim } from "../legacy-controller-lease.js";
|
|
import {
|
|
admitExplicitNativeContinuation,
|
|
undeliveredLegacyUserCommentIds,
|
|
} from "../explicit-native-continuation.js";
|
|
import { canRetryStoppedRun } from "../cancelled-native-startup.js";
|
|
import { getExecutionBlocker, hasPendingRunCleanup } from "../execution-blocker.js";
|
|
import { nativeManualAdmissionHoldsExecution } from "../native-runtime/native-runner-ownership.js";
|
|
import { taskChatRetrySourceCleanupPending } from "../chat-channels/task-retry-authority.js";
|
|
import { validateNativeWorkspaceReconstruction } from "../native-runtime/native-workspace-reconstruction.js";
|
|
import { isConversationAdapter } from "../conversation-continuation.js";
|
|
import { recordExecutionWait } from "../execution-wait.js";
|
|
import {
|
|
getNativeReviewAssignment,
|
|
readNativeReviewAssignmentContext,
|
|
} from "../native-runtime/native-review-participant.js";
|
|
import { claimQueuedNativeReviewRun } from "../native-runtime/native-review-dispatch.js";
|
|
import { nativeCleanupHoldsAdmission } from "../native-runtime/native-cleanup-admission-hold.js";
|
|
import { adapterExecutionControls } from "../adapter-execution-control.js";
|
|
import { preserveTaskRemoteWorkspaceWriter } from "../native-runtime/native-failed-workspace-recovery.js";
|
|
import { executionFailureRetryCount } from "../execution-recovery-attempt.js";
|
|
import {
|
|
slackMentionAllowsRunStart,
|
|
resumeDeferredChatWake,
|
|
assertDurableChatWakeupReceipt,
|
|
assertDurableChatWakeupRequest,
|
|
withDurableChatAdmission,
|
|
authorizeFailedChatRunRetryWake,
|
|
FailedChatRunRetryAuthorizationError,
|
|
unadmittedChatWakeupCondition,
|
|
} from "../durable-chat-wakeup.js";
|
|
import { agentService } from "../agents.js";
|
|
import { normalizeLegacyRunnerProvider } from "@paperclipai/adapter-utils";
|
|
import { randomUUID } from "node:crypto";
|
|
import {
|
|
and,
|
|
asc,
|
|
desc,
|
|
eq,
|
|
gt,
|
|
gte,
|
|
inArray,
|
|
isNotNull,
|
|
isNull,
|
|
lte,
|
|
ne,
|
|
notInArray,
|
|
or,
|
|
sql,
|
|
} from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
agents,
|
|
caseHandoffs,
|
|
cases,
|
|
agentWakeupRequests,
|
|
activityLog,
|
|
budgetIncidents,
|
|
chatConversations,
|
|
chatEndpoints,
|
|
companies,
|
|
executionWorkspaces,
|
|
heartbeatRunEvents,
|
|
heartbeatRuns,
|
|
issueComments,
|
|
issueRecoveryActions,
|
|
issueRelations,
|
|
issues,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
conflict,
|
|
HttpError,
|
|
notFound,
|
|
} from "../../errors.js";
|
|
import { logger } from "../../middleware/logger.js";
|
|
import { isUniqueViolation } from "../../db-errors.js";
|
|
import { publishLiveEvent } from "../live-events.js";
|
|
import { allocateHeartbeatRunEventSeq } from "../heartbeat-run-events.js";
|
|
import {
|
|
queuedCommentIdsFromWakePayload,
|
|
queuedCommentIdsFromRunContext,
|
|
withQueuedCommentIdsInWakePayload,
|
|
withQueuedCommentIdsInRunContext,
|
|
} from "../issue-queued-comment-queue.js";
|
|
import { parseObject } from "../../adapters/utils.js";
|
|
import { isExternalChatWaitAuthorizationContention } from "../native-runtime/chat-attachment-reuse.js";
|
|
import { emitAgentTaskRun } from "../agent-task-run-telemetry.js";
|
|
import {
|
|
CHAT_CONTROL_RECOVERY_ADMISSION_KEY,
|
|
CHAT_CONTROL_RECOVERY_STOP_CODE,
|
|
CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE,
|
|
chatControlRecoveryAdmission,
|
|
readChatControlRecoveryAdmission,
|
|
readChatControlNativeParent,
|
|
readChatControlRecoveryStop,
|
|
} from "../chat-control-recovery-stop.js";
|
|
import {
|
|
ISSUE_NEW_INPUT_ACTIVITY_ACTIONS,
|
|
ISSUE_PROGRESS_ACTIVITY_ACTIONS,
|
|
ISSUE_REWAKE_LOOKBACK_MS,
|
|
ISSUE_REWAKE_RUN_SAMPLE_LIMIT,
|
|
evaluateIssueRewakeThrottle,
|
|
isThrottleCandidateIssueRewake,
|
|
NATIVE_STATUS_COMMITTER_ACTOR_ID,
|
|
} from "../issue-rewake-throttle.js";
|
|
import { logActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
|
|
import {
|
|
ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
|
|
isDependencyResolvedWakePayload,
|
|
dependencyResolvedWakePayloadSql,
|
|
} from "../issue-dependency-wakeups.js";
|
|
import { resumeIfPauseAlreadyReleased } from "../issue-tree-resume.js";
|
|
import {
|
|
ISSUE_PAUSED_ERROR_CODE,
|
|
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
|
|
isVerifiedIssueTreeControlInteractionWake,
|
|
pauseSuppressionResult,
|
|
} from "../issue-tree-control.js";
|
|
import {
|
|
buildExecutionWorkspaceAdapterConfig,
|
|
isUnrunnableWorktreeCombo,
|
|
parseIssueExecutionWorkspaceSettings,
|
|
resolveEffectiveWorkspaceStrategyType,
|
|
resolveExecutionWorkspaceMode,
|
|
WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
|
|
WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE,
|
|
WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION,
|
|
} from "../execution-workspace-policy.js";
|
|
import { readContinuationAttempt } from "../recovery/index.js";
|
|
import { allowsDependencyBlockedWake, extractWakeCommentIds, wakeChatConversationsConflict } from "../../modules/run-dispatch/index.js";
|
|
import { dispatchForAgent, outsideAgentDispatch } from "../agent-start-lock.js";
|
|
import { AgentRunSlotsFull, assertAgentRunSlotFree } from "./run-slot-admission.js";
|
|
import {
|
|
evaluateAgentInvokability,
|
|
shouldCancelRunsForNonInvokableAgent,
|
|
type AgentOrgRow,
|
|
} from "../agent-invokability.js";
|
|
import { supportsUnscopedTimerHeartbeat } from "../heartbeat-policy.js";
|
|
import { ISSUE_BLOCKER_STALLED_WAKE_REASON, isUuidLike } from "@paperclipai/shared";
|
|
import { retryChatControlAdmission } from "../chat-control-admission-retry.js";
|
|
|
|
import type { HeartbeatServiceOptions } from "../heartbeat.js";
|
|
import type { createHeartbeatRunState } from "./run-state.js";
|
|
import type { createHeartbeatRunPreparation } from "./run-preparation.js";
|
|
import type { HeartbeatRetryDependencies } from "./retries.js";
|
|
import type { HeartbeatRecoveryDependencies, createHeartbeatRecovery } from "./recovery.js";
|
|
import { HEARTBEAT_RUN_TERMINAL_STATUSES } from "./recovery.js";
|
|
import { executeIssuePostCommitActions, type IssuePostCommitAction, type issueService } from "../issues.js";
|
|
import type { issueTreeControlService } from "../issue-tree-control.js";
|
|
import type { budgetService } from "../budgets.js";
|
|
import { isBoardOwnedBudgetBlock } from "../budget-block-ownership.js";
|
|
import type { instanceSettingsService } from "../instance-settings.js";
|
|
import type { createRunDispatch } from "../../modules/run-dispatch/index.js";
|
|
import { hasRetainedExternalChatRetry, type PostCommitEffect as WakeQueuePostCommitEffect, type createWakeQueue } from "../../modules/wake-queue/index.js";
|
|
import { crashPoint } from "@paperclipai/shared/crash-points";
|
|
|
|
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
|
|
type QueuedRunCandidate = Pick<HeartbeatRun, "id" | "companyId" | "agentId" | "status" | "invocationSource" | "createdAt"
|
|
| "startedAt" | "wakeupRequestId" | "retryOfRunId" | "scheduledRetryAt" | "scheduledRetryAttempt" | "scheduledRetryReason"
|
|
| "contextSnapshot">;
|
|
type QueuedDispatchRequest = { attachSetupRequest: boolean; retryRefused: boolean; continuesChain: boolean; triedRunIds: ReadonlySet<string> };
|
|
type HeartbeatRunState = ReturnType<typeof createHeartbeatRunState>;
|
|
type RunPreparation = ReturnType<typeof createHeartbeatRunPreparation>;
|
|
|
|
/** Queue policy uses the service's lifecycle effects and process-wide ownership sets. */
|
|
export interface HeartbeatQueueDependencies extends Pick<HeartbeatRetryDependencies,
|
|
"setWakeupStatus" | "appendRunEvent" | "finalizeAgentStatus" |
|
|
"getAgentInvokability" | "getWorktreeExecutionCutoff" | "applyRunDispatchPostCommitEffects">,
|
|
Pick<RunPreparation, "toAgentOrgRow" | "listCompanyAgentOrgRows" |
|
|
"resolveResponsibleUserIdForRun" | "getIssueExecutionContext" |
|
|
"getRoutineEnvForExecutionIssue" | "resolveResponsibleUserIdForRunSeed">,
|
|
Pick<HeartbeatRunState, "getAgent" | "getRun" | "resolveExplicitResumeSessionOverride" |
|
|
"resolveSessionBeforeForWakeup" | "hasResolvablePriorSessionWorkspaceForWake">,
|
|
Pick<HeartbeatRecoveryDependencies, "setRunStatusFromLive"> {
|
|
sweepPendingCleanupLeases: ReturnType<typeof createHeartbeatRecovery>["sweepPendingCleanupLeases"];
|
|
options: Pick<HeartbeatServiceOptions, "beforeChatControlRecoveryCheck">;
|
|
issuesSvc: Pick<ReturnType<typeof issueService>, "getDependencyReadiness" | "listDependencyReadiness" | "updateForCompany">;
|
|
treeControlSvc: Pick<ReturnType<typeof issueTreeControlService>, "getActivePauseHoldGate">;
|
|
budgets: Pick<ReturnType<typeof budgetService>, "getInvocationBlock">;
|
|
instanceSettings: Pick<ReturnType<typeof instanceSettingsService>, "getExperimental">;
|
|
runDispatch: Pick<ReturnType<typeof createRunDispatch>, "cancelStaleQueuedRun">;
|
|
wakeQueue: Pick<ReturnType<typeof createWakeQueue>, "createAdmissionTransactionScope" | "admitWakeBehindIssueExecution">;
|
|
activeRunExecutions: Set<string>;
|
|
activeRunExecutionPromises: Set<Promise<void>>;
|
|
activeWakeupPromises: Set<Promise<unknown>>;
|
|
liveRunExecutions: { has(id: string): boolean };
|
|
isHeartbeatRunTerminalStatus: (status: string | null | undefined) => boolean;
|
|
isSameTaskScope: (left: string | null, right: string | null) => boolean;
|
|
runTaskKey: (run: HeartbeatRun) => string | null;
|
|
filterZombieCoalesceTarget: <T extends { id: string; status: string }>(run: T | null, live: { has(id: string): boolean }) => T | null;
|
|
getSchedulingSuppression: () => Promise<{ suppressed: boolean; reason: "worktree_instance" | "database_restore_in_progress" | "task_drain" | null }>;
|
|
isRunAdmissionClosed?: () => boolean;
|
|
isReattachPhaseActive?: () => boolean;
|
|
/** Before a dispatch pass, refresh admission inputs that have gone stale; null when current. */
|
|
refreshRunAdmission?: () => Promise<void> | null;
|
|
canStartNewExternalBotWork: (agent: typeof agents.$inferSelect) => Promise<boolean>;
|
|
executeRun: (runId: string) => Promise<void>;
|
|
releaseIssueExecutionAndPromote: (run: Pick<HeartbeatRun, "id" | "companyId">, options?: { suppressImmediateRecovery?: boolean; resumeStrandedQueue?: boolean; deferredPostCommitEffects?: WakeQueuePostCommitEffect[] }) => Promise<unknown>;
|
|
applyWakeQueuePostCommitEffects: (effects: WakeQueuePostCommitEffect[]) => Promise<void>;
|
|
publishRunLifecyclePluginEvent: (run: HeartbeatRun) => void;
|
|
cancelRunInternal: (runId: string, reason?: string,
|
|
options?: { errorCode?: string; resultJson?: Record<string, unknown> }) => Promise<unknown>;
|
|
cancelActiveForAgentInternal: (agentId: string, reason: string) => Promise<unknown>;
|
|
resumeExecutionWaitComments: (scope?: DeferredReceiptRecoveryOptions) => Promise<unknown>;
|
|
resumeQueuedCommentInterrupt: (companyId: string, queueId: string) => Promise<unknown>;
|
|
resumeSavedLegacyComments: (companyId: string, queueId: string) => Promise<unknown>;
|
|
formatIssueIdentifierLink: (identifier: string | null, fallback: string) => string;
|
|
}
|
|
|
|
|
|
|
|
|
|
export const DEFERRED_WAKE_CONTEXT_KEY = "_paperclipWakeContext";
|
|
|
|
export const EXECUTION_PATH_HEARTBEAT_RUN_STATUSES = [
|
|
"queued",
|
|
"running",
|
|
"scheduled_retry",
|
|
] as const;
|
|
|
|
|
|
export class ChatControlRecoveryUnresolvedError extends Error {
|
|
constructor() {
|
|
super(
|
|
"Run admission could not acquire its database locks after bounded retries. No provider work started. Review database contention and send a fresh request; this attempt will not automatically retry.",
|
|
);
|
|
}
|
|
}
|
|
|
|
const RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP = new Set([
|
|
CHAT_COMPLETION_WAKE_REASON,
|
|
"approval_approved",
|
|
ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
|
|
"issue_recovery_action_restored",
|
|
]);
|
|
|
|
|
|
export function shouldRequireIssueCommentForWake(
|
|
contextSnapshot: Record<string, unknown> | null | undefined,
|
|
) {
|
|
if (contextSnapshot?.skipIssueComment === true) return false;
|
|
|
|
const wakeReason = readNonEmptyString(contextSnapshot?.wakeReason);
|
|
return (
|
|
wakeReason === "issue_assigned" ||
|
|
wakeReason === "execution_review_requested" ||
|
|
wakeReason === "execution_approval_requested" ||
|
|
wakeReason === "execution_changes_requested"
|
|
);
|
|
}
|
|
|
|
async function listUnresolvedBlockerSummaries(
|
|
dbOrTx: Pick<Db, "select">,
|
|
companyId: string,
|
|
issueId: string,
|
|
unresolvedBlockerIssueIds: string[],
|
|
) {
|
|
const ids = [...new Set(unresolvedBlockerIssueIds.filter(Boolean))];
|
|
if (ids.length === 0) return [];
|
|
return dbOrTx
|
|
.select({
|
|
id: issues.id,
|
|
identifier: issues.identifier,
|
|
title: issues.title,
|
|
status: issues.status,
|
|
priority: issues.priority,
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
assigneeUserId: issues.assigneeUserId,
|
|
})
|
|
.from(issueRelations)
|
|
.innerJoin(issues, eq(issueRelations.issueId, issues.id))
|
|
.where(
|
|
and(
|
|
eq(issueRelations.companyId, companyId),
|
|
eq(issueRelations.type, "blocks"),
|
|
eq(issueRelations.relatedIssueId, issueId),
|
|
inArray(issues.id, ids),
|
|
),
|
|
)
|
|
.orderBy(asc(issues.title));
|
|
}
|
|
|
|
export function shouldDeferFollowupWakeForSameIssue(input: {
|
|
activeRunStatus: string | null | undefined;
|
|
isSameExecutionAgent: boolean;
|
|
wakeCommentId: string | null | undefined;
|
|
forceFreshSession: boolean;
|
|
}) {
|
|
// A comment follow-up or explicit fresh-session wake needs a new run boundary.
|
|
if (!input.isSameExecutionAgent) return false;
|
|
if (input.activeRunStatus !== "running") return false;
|
|
if (input.wakeCommentId) return true;
|
|
if (input.forceFreshSession) return true;
|
|
return false;
|
|
}
|
|
|
|
export function shouldQueueFollowupForRunningIssueWake(input: {
|
|
contextSnapshot: Record<string, unknown> | null | undefined;
|
|
wakeCommentId: string | null;
|
|
}) {
|
|
if (input.wakeCommentId) return true;
|
|
// A structured interaction response is new input just like a comment. It
|
|
// must run after the turn that created the interaction instead of being
|
|
// merged into that still-running turn.
|
|
if (
|
|
readNonEmptyString(input.contextSnapshot?.interactionId) &&
|
|
readNonEmptyString(input.contextSnapshot?.interactionStatus)
|
|
) {
|
|
return true;
|
|
}
|
|
const wakeReason = readNonEmptyString(input.contextSnapshot?.wakeReason);
|
|
if (wakeReason === "issue_children_completed" && (
|
|
input.contextSnapshot?.onboardingCompletion === true ||
|
|
(input.contextSnapshot?.statusDecisionSource === "native_status_decision" &&
|
|
readNonEmptyString(input.contextSnapshot?.nativeChildCompletionDecisionId))
|
|
)) return true;
|
|
return Boolean(
|
|
wakeReason && RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP.has(wakeReason),
|
|
);
|
|
}
|
|
|
|
/** Agent Chat outbox receipts. The outbox, not a skipped receipt, owns a capped message. */
|
|
const CONVERSATION_COMMENT_KEY_PREFIX = "conversation-comment:";
|
|
const QUEUED_RUN_CLAIM_ATTEMPTS_PER_PASS = 24;
|
|
const QUEUED_RUN_CLAIM_BACKOFF_BASE_MS = 5_000;
|
|
const QUEUED_RUN_CLAIM_BACKOFF_MAX_MS = 60_000;
|
|
const QUEUED_PERSON_RUN_CLAIM_BACKOFF_MS = 2_000;
|
|
function dailyLimitConversationRefusal(cap: { reason: string; observed: number; limit: number }) {
|
|
// No receipt is written, so the person's message stays owed in the outbox
|
|
// until the limit resets (agent-conversations.ts conversationDeliveryHold).
|
|
return conflict("Agent reached its daily limit", { reason: cap.reason, observed: cap.observed, limit: cap.limit });
|
|
}
|
|
|
|
function readNonEmptyString(value: unknown): string | null {
|
|
return typeof value === "string" && value.trim().length > 0 ? value : null;
|
|
}
|
|
|
|
export async function claimDueTimerHeartbeatForAgent(
|
|
db: Db,
|
|
agent: typeof agents.$inferSelect,
|
|
now: Date,
|
|
intervalSec: number,
|
|
) {
|
|
const dueBefore = new Date(now.getTime() - intervalSec * 1000);
|
|
const claimed = await db.transaction(async tx => {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
return tx
|
|
.update(agents)
|
|
.set({ lastHeartbeatAt: now, updatedAt: now })
|
|
.where(and(
|
|
eq(agents.id, agent.id),
|
|
eq(agents.companyId, agent.companyId),
|
|
or(
|
|
lte(agents.lastHeartbeatAt, dueBefore),
|
|
and(isNull(agents.lastHeartbeatAt), lte(agents.createdAt, dueBefore)),
|
|
),
|
|
))
|
|
.returning({ id: agents.id })
|
|
.then((rows) => rows[0] ?? null);
|
|
});
|
|
if (!claimed) return null;
|
|
return { wasFirstHeartbeat: !agent.lastHeartbeatAt };
|
|
}
|
|
|
|
|
|
/** Queue admission and dispatch share the service lifecycle callbacks and execution ownership. */
|
|
// Wake admissions that have not committed their wake and run yet. Process-wide, like
|
|
// the pending-wake set: a server replacement loses these, and only these, pending wakes.
|
|
/** Rolls back a claim whose server closed run admission while it was being taken. */
|
|
class RunAdmissionClosedDuringClaim extends Error {}
|
|
|
|
const unsavedWakeAdmissions = new Set<object>();
|
|
export function unsavedWakeAdmissionCount() { return unsavedWakeAdmissions.size; }
|
|
|
|
export async function notifyOtherAgentsQueuedOnSettledIssue(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect,
|
|
"companyId" | "agentId" | "runtimeMode" | "contextSnapshot">): Promise<boolean> {
|
|
const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
|
|
if (run.runtimeMode !== "native" || !issueId) return false;
|
|
const [waiting] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
ne(heartbeatRuns.agentId, run.agentId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,
|
|
)).limit(1);
|
|
if (!waiting) return false;
|
|
await db.transaction(tx => notifyQueuedRunDispatch(tx, { companyId: run.companyId }));
|
|
return true;
|
|
}
|
|
|
|
export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDependencies) {
|
|
const {
|
|
setRunStatusFromLive,
|
|
setWakeupStatus,
|
|
appendRunEvent,
|
|
releaseIssueExecutionAndPromote,
|
|
applyWakeQueuePostCommitEffects,
|
|
issuesSvc,
|
|
options,
|
|
publishRunLifecyclePluginEvent,
|
|
finalizeAgentStatus,
|
|
getAgent,
|
|
cancelRunInternal,
|
|
toAgentOrgRow,
|
|
getAgentInvokability,
|
|
budgets,
|
|
activeRunExecutions,
|
|
treeControlSvc,
|
|
runDispatch,
|
|
applyRunDispatchPostCommitEffects,
|
|
resolveResponsibleUserIdForRun,
|
|
getIssueExecutionContext,
|
|
isHeartbeatRunTerminalStatus,
|
|
liveRunExecutions,
|
|
getSchedulingSuppression,
|
|
isRunAdmissionClosed: isStartAdmissionClosed = isProcessRunAdmissionClosed,
|
|
isReattachPhaseActive = isProcessReattachPhaseActive,
|
|
refreshRunAdmission = refreshRunStartYieldIfStale,
|
|
resumeExecutionWaitComments,
|
|
getWorktreeExecutionCutoff,
|
|
resumeQueuedCommentInterrupt,
|
|
resumeSavedLegacyComments,
|
|
cancelActiveForAgentInternal,
|
|
listCompanyAgentOrgRows,
|
|
executeRun,
|
|
canStartNewExternalBotWork,
|
|
activeRunExecutionPromises,
|
|
activeWakeupPromises,
|
|
instanceSettings,
|
|
getRun,
|
|
sweepPendingCleanupLeases,
|
|
resolveExplicitResumeSessionOverride,
|
|
resolveSessionBeforeForWakeup,
|
|
hasResolvablePriorSessionWorkspaceForWake,
|
|
getRoutineEnvForExecutionIssue,
|
|
resolveResponsibleUserIdForRunSeed,
|
|
formatIssueIdentifierLink,
|
|
wakeQueue,
|
|
isSameTaskScope,
|
|
runTaskKey,
|
|
filterZombieCoalesceTarget,
|
|
} = dependencies;
|
|
|
|
const isRunAdmissionClosed = () => isStartAdmissionClosed() || isReattachPhaseActive();
|
|
|
|
const { parseHeartbeatPolicy, getHeartbeatDailyCapBlock, hasActionableTimerWork, nextTimerHeartbeatAt } = createHeartbeatTimerPolicy(db);
|
|
|
|
// claimQueuedRun works from the caller's copy of a queued run. Another
|
|
// dispatcher (an overlapping server during a swap) may have started and even
|
|
// finished that run since, so every cancellation it makes is a
|
|
// compare-and-set from "queued": it never overwrites a newer outcome.
|
|
async function cancelRunIfStillQueued(runId: string, patch: Partial<typeof heartbeatRuns.$inferInsert>) {
|
|
const write = await setRunStatusFromLive(runId, "cancelled", ["queued"], patch);
|
|
return write.updated ? write.run : null;
|
|
}
|
|
|
|
async function cancelQueuedRunForHeartbeatDailyCap(
|
|
run: typeof heartbeatRuns.$inferSelect,
|
|
dailyCapBlock: NonNullable<
|
|
Awaited<ReturnType<typeof getHeartbeatDailyCapBlock>>
|
|
>,
|
|
) {
|
|
const now = new Date();
|
|
const reason =
|
|
"Cancelled because the agent reached a per-day heartbeat budget cap before adapter invocation";
|
|
const cancelled = await cancelRunIfStillQueued(run.id, {
|
|
finishedAt: now,
|
|
error: reason,
|
|
errorCode: dailyCapBlock.reason,
|
|
resultJson: {
|
|
...parseObject(run.resultJson),
|
|
stopReason: dailyCapBlock.reason,
|
|
observed: dailyCapBlock.observed,
|
|
limit: dailyCapBlock.limit,
|
|
effectiveTimeoutSec: 0,
|
|
timeoutConfigured: false,
|
|
timeoutSource: "heartbeat_daily_cap_gate",
|
|
timeoutFired: false,
|
|
},
|
|
});
|
|
if (!cancelled) return null;
|
|
|
|
await setWakeupStatus(run.wakeupRequestId, "skipped", {
|
|
finishedAt: now,
|
|
error: reason,
|
|
});
|
|
await releaseConversationCommentsForDailyCap(cancelled);
|
|
|
|
await appendRunEvent(cancelled, {
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message: reason,
|
|
payload: {
|
|
reason: dailyCapBlock.reason,
|
|
observed: dailyCapBlock.observed,
|
|
limit: dailyCapBlock.limit,
|
|
},
|
|
});
|
|
|
|
await releaseIssueExecutionAndPromote(cancelled, {
|
|
suppressImmediateRecovery: true,
|
|
});
|
|
|
|
return cancelled;
|
|
}
|
|
|
|
/**
|
|
* A chat message accepted before the limit was reached must not be consumed
|
|
* when dispatch then cancels its run at the limit. Releasing the outbox key
|
|
* on its receipts (the run's own and any coalesced into it) makes the
|
|
* message owed again; the outbox holds it until the limit resets.
|
|
*/
|
|
async function releaseConversationCommentsForDailyCap(run: typeof heartbeatRuns.$inferSelect) {
|
|
const owned = and(
|
|
eq(agentWakeupRequests.companyId, run.companyId),
|
|
eq(agentWakeupRequests.agentId, run.agentId),
|
|
or(eq(agentWakeupRequests.runId, run.id),
|
|
run.wakeupRequestId ? eq(agentWakeupRequests.id, run.wakeupRequestId) : undefined),
|
|
sql`${agentWakeupRequests.idempotencyKey} like ${CONVERSATION_COMMENT_KEY_PREFIX + "%"}`,
|
|
);
|
|
await db.transaction(async tx => {
|
|
const receipts = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(owned).for("update");
|
|
if (!receipts.length) return;
|
|
// Register the recovery signal before the write it announces.
|
|
await notifyIssueRecovery(tx, { companyId: run.companyId });
|
|
await tx.update(agentWakeupRequests).set({
|
|
idempotencyKey: null,
|
|
payload: sql`coalesce(${agentWakeupRequests.payload}, '{}'::jsonb)
|
|
|| jsonb_build_object('releasedConversationCommentKey', ${agentWakeupRequests.idempotencyKey})`,
|
|
updatedAt: new Date(),
|
|
}).where(and(owned, inArray(agentWakeupRequests.id, receipts.map(row => row.id))));
|
|
});
|
|
}
|
|
|
|
async function markTimerHeartbeatChecked(
|
|
agentId: string,
|
|
source: WakeupOptions["source"],
|
|
) {
|
|
if (source !== "timer") return;
|
|
await db.transaction(async tx => {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
const now = new Date();
|
|
await tx.update(agents).set({ lastHeartbeatAt: now, updatedAt: now })
|
|
.where(eq(agents.id, agentId));
|
|
});
|
|
}
|
|
|
|
const claimDueTimerHeartbeat = (agent: typeof agents.$inferSelect, now: Date, intervalSec: number) =>
|
|
claimDueTimerHeartbeatForAgent(db, agent, now, intervalSec);
|
|
|
|
async function listQueuedRunDependencyReadiness(
|
|
companyId: string,
|
|
queuedRuns: Array<Pick<HeartbeatRun, "contextSnapshot">>,
|
|
) {
|
|
const issueIds = [
|
|
...new Set(
|
|
queuedRuns
|
|
.map((run) =>
|
|
readNonEmptyString(parseObject(run.contextSnapshot).issueId),
|
|
)
|
|
.filter((issueId): issueId is string => Boolean(issueId)),
|
|
),
|
|
];
|
|
if (issueIds.length === 0) {
|
|
return new Map<
|
|
string,
|
|
Awaited<ReturnType<typeof issuesSvc.getDependencyReadiness>>
|
|
>();
|
|
}
|
|
return issuesSvc.listDependencyReadiness(companyId, issueIds);
|
|
}
|
|
|
|
async function countRunningRunsForAgent(agentId: string) {
|
|
const [{ count }] = await db
|
|
.select({ count: sql<number>`count(*)` })
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
eq(heartbeatRuns.status, "running"),
|
|
),
|
|
);
|
|
return Number(count ?? 0);
|
|
}
|
|
|
|
async function withChatControlRecoveryGate(
|
|
run: typeof heartbeatRuns.$inferSelect,
|
|
stage: "claim" | "dispatch",
|
|
onClear: (tx: Db) => Promise<typeof heartbeatRuns.$inferSelect | null>,
|
|
nativeRecovery = false,
|
|
) {
|
|
const issueId = readNonEmptyString(
|
|
parseObject(run.contextSnapshot).issueId,
|
|
);
|
|
if (!issueId || run.invocationSource !== "automation") return onClear(db);
|
|
await options.beforeChatControlRecoveryCheck?.({
|
|
runId: run.id,
|
|
issueId,
|
|
stage,
|
|
});
|
|
if (stage === "dispatch" && nativeRecovery) {
|
|
// Protected historical/already-admitted owners did not previously take
|
|
// this fence's locks. A fresh exact read keeps unrelated Board contention
|
|
// from turning their recovery into a new admission failure.
|
|
const [current] = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, run.id),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.agentId, run.agentId),
|
|
eq(heartbeatRuns.status, "running"),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (
|
|
current &&
|
|
current.wakeupRequestId === run.wakeupRequestId &&
|
|
parseObject(current.contextSnapshot).issueId === issueId
|
|
) {
|
|
const admission = readChatControlRecoveryAdmission(current);
|
|
const expected = readChatControlRecoveryAdmission(run);
|
|
if (
|
|
expected !== "invalid" &&
|
|
(admission === "admitted" ||
|
|
(admission === "historical" && expected === "historical"))
|
|
)
|
|
return onClear(db);
|
|
}
|
|
}
|
|
let terminal: typeof heartbeatRuns.$inferSelect | null = null;
|
|
try {
|
|
const attempt = () => db.transaction(async (tx) => {
|
|
terminal = null;
|
|
// Same queue-edit lock order, then the close committer's conversation
|
|
// row. NOWAIT releases partial locks on contention. Claim defers to the
|
|
// queue; dispatch retries this transaction before considering failure.
|
|
const [issue] = await tx
|
|
.select({ id: issues.id })
|
|
.from(issues)
|
|
.where(
|
|
and(eq(issues.id, issueId), eq(issues.companyId, run.companyId)),
|
|
)
|
|
.for("update", { noWait: true })
|
|
.limit(1);
|
|
if (!issue) return null;
|
|
if (run.wakeupRequestId)
|
|
await tx
|
|
.select({ id: agentWakeupRequests.id })
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, run.wakeupRequestId),
|
|
eq(agentWakeupRequests.companyId, run.companyId),
|
|
eq(agentWakeupRequests.agentId, run.agentId),
|
|
),
|
|
)
|
|
.for("update", { noWait: true })
|
|
.limit(1);
|
|
const [current] = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, run.id),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.agentId, run.agentId),
|
|
),
|
|
)
|
|
.for("update", { noWait: true })
|
|
.limit(1);
|
|
if (
|
|
!current ||
|
|
current.status !== (stage === "claim" ? "queued" : "running")
|
|
)
|
|
return null;
|
|
const admission = readChatControlRecoveryAdmission(current);
|
|
const expectedAdmission = readChatControlRecoveryAdmission(run);
|
|
const admissionLost =
|
|
expectedAdmission !== "historical" && admission === "historical";
|
|
if (
|
|
stage === "dispatch" &&
|
|
nativeRecovery &&
|
|
!admissionLost &&
|
|
(admission === "historical" || admission === "admitted")
|
|
)
|
|
return onClear(tx as unknown as Db);
|
|
// A new warm child can inherit a prior runner's PID during preparation.
|
|
// Those fields do not prove this run's turn was dispatched.
|
|
// Exact admitted/historical recovery retains its existing ownership;
|
|
// a marked, unadmitted bootstrap must earn admission even after crash.
|
|
const proof =
|
|
admission === "invalid" || admissionLost
|
|
? { kind: "unresolved" as const }
|
|
: await readChatControlRecoveryStop(
|
|
tx as unknown as Db,
|
|
{
|
|
companyId: run.companyId,
|
|
issueId,
|
|
agentId: run.agentId,
|
|
sourceRunId: run.id,
|
|
},
|
|
true,
|
|
);
|
|
if (proof.kind === "clear") {
|
|
if (stage === "claim" || admission === "required")
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
runnerProfileJson: {
|
|
...parseObject(current.runnerProfileJson),
|
|
[CHAT_CONTROL_RECOVERY_ADMISSION_KEY]:
|
|
chatControlRecoveryAdmission(
|
|
current,
|
|
stage === "dispatch"
|
|
? "admitted"
|
|
: admission === "admitted"
|
|
? "admitted"
|
|
: "required",
|
|
),
|
|
},
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(heartbeatRuns.id, current.id));
|
|
return onClear(tx as unknown as Db);
|
|
}
|
|
if (proof.kind === "unresolved" && stage === "claim") return null;
|
|
const now = new Date();
|
|
const stopped = proof.kind === "stopped";
|
|
const code = stopped
|
|
? CHAT_CONTROL_RECOVERY_STOP_CODE
|
|
: CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE;
|
|
const error = stopped
|
|
? "Automatic continuation stopped by the committed chat conversation close. Send a new request in chat or on the Board to start fresh work."
|
|
: "Automatic continuation source could not be verified before provider admission. Review the task and send a fresh request; this attempt will not automatically retry.";
|
|
await notifyChatRunMilestones(tx, { companyId: current.companyId, issueId });
|
|
await notifyInstructionCollection(tx, { companyId: current.companyId, runId: current.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: current.companyId, runId: current.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: current.companyId, runId: current.id });
|
|
await notifyExecutionRunWork(tx, { companyId: current.companyId, agentId: current.agentId, runId: current.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, current);
|
|
[terminal] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: stopped ? "cancelled" : "failed",
|
|
errorCode: code,
|
|
error,
|
|
finishedAt: now,
|
|
resultJson: {
|
|
...parseObject(current.resultJson),
|
|
automaticRecovery: {
|
|
code,
|
|
providerDispatched: false,
|
|
...(stopped
|
|
? {
|
|
sourceRunId: proof.sourceRunId,
|
|
conversationId: proof.conversationId,
|
|
publicationId: proof.publicationId,
|
|
}
|
|
: {}),
|
|
},
|
|
},
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, current.id),
|
|
eq(heartbeatRuns.status, current.status),
|
|
),
|
|
)
|
|
.returning();
|
|
if (!terminal) return null;
|
|
if (current.wakeupRequestId)
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: stopped ? "skipped" : "failed",
|
|
error,
|
|
finishedAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, current.wakeupRequestId),
|
|
eq(agentWakeupRequests.companyId, current.companyId),
|
|
eq(agentWakeupRequests.agentId, current.agentId),
|
|
eq(agentWakeupRequests.runId, current.id),
|
|
ne(agentWakeupRequests.status, "cancelled"),
|
|
),
|
|
);
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, current.companyId),
|
|
eq(issues.id, issueId),
|
|
eq(issues.executionRunId, current.id),
|
|
),
|
|
);
|
|
return null;
|
|
});
|
|
const result = stage === "dispatch"
|
|
? await retryChatControlAdmission(attempt)
|
|
: await attempt();
|
|
if (terminal) {
|
|
const settled = terminal as typeof heartbeatRuns.$inferSelect;
|
|
publishLiveEvent({
|
|
companyId: settled.companyId,
|
|
type: "heartbeat.run.status",
|
|
payload: {
|
|
runId: settled.id,
|
|
agentId: settled.agentId,
|
|
status: settled.status,
|
|
errorCode: settled.errorCode,
|
|
error: settled.error,
|
|
},
|
|
});
|
|
publishRunLifecyclePluginEvent(settled);
|
|
if (stage === "dispatch")
|
|
await finalizeAgentStatus(settled.agentId, "cancelled");
|
|
}
|
|
return result;
|
|
} catch (error) {
|
|
if (!isExternalChatWaitAuthorizationContention(error)) throw error;
|
|
if (stage === "dispatch") {
|
|
// No effect was admitted. Let the existing setup-failure path settle
|
|
// this attempt distinctly; it must not become a successful close.
|
|
throw new ChatControlRecoveryUnresolvedError();
|
|
}
|
|
return null;
|
|
}
|
|
}
|
|
|
|
// Run admission can close (SIGTERM drain, or the hand-over yield) while
|
|
// claimQueuedRun awaits its checks, after its first admission check passed.
|
|
// Checked again inside the claim's transaction, right after the write that
|
|
// moves the run to running: throwing rolls the whole claim back, so the run,
|
|
// its wake and the task's execution lock stay exactly as they were, queued
|
|
// for the serving server.
|
|
function confirmRunAdmissionStillOpen(claimedRun: unknown) {
|
|
if (claimedRun && isRunAdmissionClosed()) throw new RunAdmissionClosedDuringClaim();
|
|
}
|
|
|
|
/** System continuations that only make sense for an open task. */
|
|
const SYSTEM_CONTINUATION_WAKE_REASONS = new Set(["issue_recovery_action_restored", "issue_continuation_needed",
|
|
"issue_assignment_recovery", ISSUE_BLOCKER_STALLED_WAKE_REASON, "issue_children_completed"]);
|
|
|
|
/**
|
|
* Cancels a system continuation whose task closed while it was queued,
|
|
* unless it carries a person's message that no run of this agent has
|
|
* received yet (admission can attach deferred comments to a queued system
|
|
* continuation without changing its wake reason). Its wake and run are
|
|
* locked in the order admission's coalescing uses while the message check
|
|
* reads them. The cancel itself goes through the run lifecycle's
|
|
* compare-and-set, so it saves the same never-started proof as every other
|
|
* queued-run cancel (task release needs it to promote messages saved behind
|
|
* this run), and it is fenced on the row version the check saw: a message
|
|
* attached after the check changes the run, the cancel matches nothing, and
|
|
* the check runs again.
|
|
*/
|
|
async function cancelClosedTaskContinuation(run: typeof heartbeatRuns.$inferSelect, issueId: string) {
|
|
for (let attempt = 0; attempt < 3; attempt += 1) {
|
|
const decision = await checkClosedTaskContinuation(run, issueId);
|
|
if (decision.kind !== "cancel") return decision;
|
|
const write = await setRunStatusFromLive(run.id, "cancelled", ["queued"], {
|
|
finishedAt: new Date(), error: "The task was closed before this continuation started.",
|
|
errorCode: "issue_closed_before_start",
|
|
}, undefined, sql`${heartbeatRuns}.xmin::text = ${decision.rowVersion}`);
|
|
if (write.updated && write.run) return { kind: "cancelled" as const, run: write.run };
|
|
if (write.run?.status !== "queued") return { kind: "not_queued" as const };
|
|
}
|
|
// Still changing under us: leave it queued; the next dispatch pass checks again.
|
|
return { kind: "still_queued" as const };
|
|
}
|
|
|
|
async function checkClosedTaskContinuation(run: typeof heartbeatRuns.$inferSelect, issueId: string) {
|
|
return db.transaction(async (tx) => {
|
|
if (run.wakeupRequestId) await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, run.wakeupRequestId), eq(agentWakeupRequests.companyId, run.companyId),
|
|
)).for("update");
|
|
const [locked] = await tx.select({ run: heartbeatRuns, rowVersion: sql<string>`${heartbeatRuns}.xmin::text` })
|
|
.from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued"),
|
|
)).for("update");
|
|
if (!locked) return { kind: "not_queued" as const };
|
|
const current = locked.run;
|
|
const context = parseObject(current.contextSnapshot);
|
|
const commentIds = [...new Set([...extractWakeCommentIds(context), ...queuedCommentIdsFromRunContext(context),
|
|
readNonEmptyString(context.wakeCommentId), readNonEmptyString(context.commentId)]
|
|
.filter((id): id is string => Boolean(id)))];
|
|
if (commentIds.length > 0) {
|
|
const personComments = await tx.select({ id: issueComments.id }).from(issueComments).where(and(
|
|
eq(issueComments.companyId, current.companyId), eq(issueComments.issueId, issueId),
|
|
inArray(issueComments.id, commentIds), eq(issueComments.authorType, "user"),
|
|
isNull(issueComments.createdByRunId), isNull(issueComments.deletedAt)));
|
|
const delivered = await deliveredQueuedCommentIds(tx as unknown as Db, { companyId: current.companyId, issueId,
|
|
agentId: current.agentId, commentIds: personComments.map(comment => comment.id) });
|
|
if (personComments.some(comment => !delivered.has(comment.id))) return { kind: "carries_person_message" as const };
|
|
}
|
|
return { kind: "cancel" as const, rowVersion: locked.rowVersion };
|
|
});
|
|
}
|
|
|
|
async function claimQueuedRun(
|
|
run: typeof heartbeatRuns.$inferSelect,
|
|
companyAgents?: AgentOrgRow[],
|
|
deferredPostCommitEffects?: WakeQueuePostCommitEffect[],
|
|
deferredCancellations?: Array<() => Promise<void>>,
|
|
slots?: { full: boolean },
|
|
) {
|
|
if (run.status !== "queued") return run;
|
|
// Every start path ends here (dispatch passes, executeRun of a queued run,
|
|
// a delivered continuation). A server whose run admission is closed (it
|
|
// received SIGTERM and is draining) or that yields new starts to a
|
|
// replacement already dispatching leaves the run queued for that server:
|
|
// anything it started now would be cut off by its own exit.
|
|
const refreshing = refreshRunAdmission();
|
|
if (refreshing) await refreshing;
|
|
if (isRunAdmissionClosed()) return null;
|
|
// A system continuation queued while its task was open has nothing left to
|
|
// do once another run finished or cancelled the task before this one
|
|
// started. A person's message on a done task is not one of these reasons.
|
|
const runContext = parseObject(run.contextSnapshot);
|
|
const systemContinuationReason = readNonEmptyString(runContext.wakeReason);
|
|
const continuedIssueId = readNonEmptyString(runContext.issueId);
|
|
if (systemContinuationReason && SYSTEM_CONTINUATION_WAKE_REASONS.has(systemContinuationReason) && continuedIssueId
|
|
&& runContext.onboardingCompletion !== true) {
|
|
const [target] = await db.select({ status: issues.status }).from(issues)
|
|
.where(and(eq(issues.companyId, run.companyId), eq(issues.id, continuedIssueId)));
|
|
if (target && ["done", "cancelled"].includes(target.status)) {
|
|
const closed = await cancelClosedTaskContinuation(run, continuedIssueId);
|
|
if (closed.kind === "cancelled") {
|
|
const cancelled = closed.run;
|
|
await setWakeupStatus(run.wakeupRequestId, "skipped", { finishedAt: cancelled.finishedAt ?? new Date(), error: "issue_closed_before_start" });
|
|
await releaseIssueExecutionAndPromote(cancelled, { suppressImmediateRecovery: true, deferredPostCommitEffects });
|
|
}
|
|
// A person's undelivered message rides on this run: it runs as their
|
|
// follow-up, which may reopen the task through the normal path.
|
|
if (closed.kind !== "carries_person_message") return null;
|
|
}
|
|
}
|
|
const interactionDelivery = await recheckQueuedRunInteractions(db, run);
|
|
if (interactionDelivery.run.status !== "queued") return null;
|
|
run = { ...run, contextSnapshot: interactionDelivery.run.contextSnapshot,
|
|
resultJson: interactionDelivery.run.resultJson, updatedAt: interactionDelivery.run.updatedAt };
|
|
if (interactionDelivery.denied) {
|
|
const cancelled = await cancelRunIfStillQueued(run.id, {
|
|
finishedAt: new Date(), error: "Queued agent input is no longer permitted.",
|
|
errorCode: INTERACTION_DELIVERY_DENIED,
|
|
});
|
|
if (cancelled) {
|
|
await setWakeupStatus(run.wakeupRequestId, "skipped", { finishedAt: new Date(), error: INTERACTION_DELIVERY_DENIED });
|
|
await appendRunEvent(cancelled, { eventType: "lifecycle", stream: "system", level: "warn",
|
|
message: "Queued agent input was refused after its interaction permission changed.", payload: { code: INTERACTION_DELIVERY_DENIED, denials: interactionDelivery.denials } });
|
|
await releaseIssueExecutionAndPromote(cancelled, { suppressImmediateRecovery: true, deferredPostCommitEffects });
|
|
}
|
|
return null;
|
|
}
|
|
const [caseHandoffReceipt] = run.wakeupRequestId ? await db.select().from(caseHandoffs).where(and(
|
|
eq(caseHandoffs.wakeupRequestId, run.wakeupRequestId), eq(caseHandoffs.companyId, run.companyId),
|
|
)).limit(1) : [];
|
|
if (caseHandoffReceipt) {
|
|
const [current] = await db.select({ version: cases.handoffVersion, issueId: cases.workIssueId,
|
|
agentId: issues.assigneeAgentId, statusVersion: issues.statusVersion }).from(cases)
|
|
.innerJoin(issues, eq(issues.id, cases.workIssueId)).where(and(eq(cases.id, caseHandoffReceipt.caseId),
|
|
eq(cases.companyId, run.companyId), eq(issues.companyId, run.companyId))).limit(1);
|
|
if (!["pending_dispatch", "delivered"].includes(caseHandoffReceipt.state) || caseHandoffReceipt.toAgentId !== run.agentId ||
|
|
!current || current.version !== caseHandoffReceipt.version || current.issueId !== caseHandoffReceipt.issueId ||
|
|
current.agentId !== run.agentId || (!run.startedAt && current.statusVersion !== caseHandoffReceipt.appliedIssueStatusVersion)) {
|
|
const cancelled = await db.transaction(async tx => {
|
|
const [currentRun] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued")))
|
|
.for("update");
|
|
if (!currentRun) return null;
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, runId: run.id,
|
|
issueId: run.nativeIssueId ?? run.issueId ?? readNonEmptyString(parseObject(run.contextSnapshot).issueId) ?? undefined });
|
|
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExecutionRunWork(tx, { companyId: run.companyId, agentId: run.agentId, runId: run.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, run);
|
|
const [row] = await tx.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date(),
|
|
errorCode: "case_handoff_superseded", error: "This case handoff was cancelled or superseded.", updatedAt: new Date() })
|
|
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued"))).returning();
|
|
return row;
|
|
});
|
|
if (cancelled) await setWakeupStatus(run.wakeupRequestId, "cancelled", { finishedAt: new Date(), error: cancelled.error });
|
|
return null;
|
|
}
|
|
if (!(await instanceSettings.getExperimental()).enableCases) return null;
|
|
}
|
|
const caseIssueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
|
|
if (caseIssueId) {
|
|
const [pending] = await db.select().from(caseHandoffs).where(and(eq(caseHandoffs.companyId, run.companyId), eq(caseHandoffs.issueId, caseIssueId),
|
|
inArray(caseHandoffs.state, ["awaiting_release", "pending_dispatch"]))).limit(1);
|
|
if (pending && (pending.wakeupRequestId !== run.wakeupRequestId || pending.state === "awaiting_release")) return null;
|
|
}
|
|
// Claim is the first execution boundary. startedAt survives same-run
|
|
// retries; restart recovery may also enter executeRun already running.
|
|
// Neither may reinterpret an admitted, already-started message as new work.
|
|
if (!run.startedAt && !(await slackMentionAllowsRunStart(db, run))) {
|
|
const now = new Date();
|
|
const reason = "Message did not @mention the bot";
|
|
// This runs under the agent start lock. Do not use cancelRunInternal,
|
|
// which recursively starts the next run and reacquires that lock.
|
|
const cancelled = await cancelRunIfStillQueued(run.id, {
|
|
finishedAt: now,
|
|
error: reason,
|
|
errorCode: "chat_mention_required",
|
|
});
|
|
if (cancelled) {
|
|
await setWakeupStatus(run.wakeupRequestId, "skipped", { finishedAt: now, error: reason });
|
|
await appendRunEvent(cancelled, {
|
|
eventType: "lifecycle", stream: "system", level: "warn", message: reason,
|
|
});
|
|
await releaseIssueExecutionAndPromote(cancelled, {
|
|
suppressImmediateRecovery: true, deferredPostCommitEffects,
|
|
});
|
|
}
|
|
return null;
|
|
}
|
|
|
|
const agent = await getAgent(run.agentId);
|
|
if (!agent) {
|
|
await cancelRunInternal(
|
|
run.id,
|
|
"Cancelled because the agent no longer exists",
|
|
);
|
|
return null;
|
|
}
|
|
// Keep accepted work in the durable queue until setup completes.
|
|
if (isAgentAwaitingSetup(agent)) return null;
|
|
const invokability = companyAgents
|
|
? evaluateAgentInvokability(toAgentOrgRow(agent), companyAgents)
|
|
: await getAgentInvokability(agent);
|
|
if (!invokability.invokable) {
|
|
if (caseHandoffReceipt) return null;
|
|
await cancelRunInternal(
|
|
run.id,
|
|
`Cancelled because the agent is not invokable: ${invokability.reason}`,
|
|
);
|
|
return null;
|
|
}
|
|
if (!run.startedAt && !await canStartNewExternalBotWork(agent)) return null;
|
|
|
|
const context = parseObject(run.contextSnapshot);
|
|
const budgetBlock = await budgets.getInvocationBlock(
|
|
run.companyId,
|
|
run.agentId,
|
|
{
|
|
issueId: readNonEmptyString(context.issueId),
|
|
projectId: readNonEmptyString(context.projectId),
|
|
},
|
|
);
|
|
if (budgetBlock) {
|
|
if (caseHandoffReceipt) return null;
|
|
// A temporary hold (accounting that is still settling) only delays
|
|
// accepted work. Cancelling the unstarted run would drop the user's
|
|
// message, Retry, or recovery it carries; keep it queued. Settling
|
|
// accounting re-notifies run dispatch. A reached hard stop is the board's.
|
|
if (!run.startedAt && !(await isBoardOwnedBudgetBlock(db, run.companyId, budgetBlock))) {
|
|
logger.info({ runId: run.id, agentId: run.agentId, companyId: run.companyId, reason: "budget_hold_settling",
|
|
scopeType: budgetBlock.scopeType }, "claimQueuedRun: queued run kept queued behind a temporary budget hold");
|
|
return null;
|
|
}
|
|
// Record which budget stops this queued run, so Raise budget and resume
|
|
// continues it (budget-resume-continuation.ts). Saved before the cancel
|
|
// so a crash between them cannot lose it; the resume matches it only
|
|
// against a recorded cancellation with this same reason.
|
|
const budgetAdmissionStop = { scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId,
|
|
reason: budgetBlock.reason.slice(0, 512) };
|
|
const stamped = await db.transaction(async tx => {
|
|
// The same commit touches the scope's open incident. A Raise budget
|
|
// and resume whose repeatable-read snapshot predates this commit then
|
|
// conflicts on that row and retries, so its resume sees the stamp
|
|
// (a still-queued stamped run is claimed and waits for this cancel).
|
|
const open = await tx.update(budgetIncidents).set({ updatedAt: new Date() }).where(and(
|
|
eq(budgetIncidents.companyId, run.companyId), eq(budgetIncidents.scopeType, budgetBlock.scopeType),
|
|
eq(budgetIncidents.scopeId, budgetBlock.scopeId), eq(budgetIncidents.status, "open")))
|
|
.returning({ id: budgetIncidents.id });
|
|
if (!open.length) return false;
|
|
await tx.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb)
|
|
|| jsonb_build_object('budgetAdmissionStop', jsonb_build_object('scopeType', ${budgetAdmissionStop.scopeType}::text,
|
|
'scopeId', ${budgetAdmissionStop.scopeId}::text, 'reason', ${budgetAdmissionStop.reason}::text))` })
|
|
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, run.status)));
|
|
return true;
|
|
});
|
|
if (!stamped) {
|
|
// No open incident: a resume may have just closed it and raised the
|
|
// budget. Read the block again; resumed work stays queued.
|
|
const current = await budgets.getInvocationBlock(run.companyId, run.agentId, {
|
|
issueId: readNonEmptyString(context.issueId), projectId: readNonEmptyString(context.projectId) });
|
|
if (!current) return null;
|
|
if (!run.startedAt && !(await isBoardOwnedBudgetBlock(db, run.companyId, current))) return null;
|
|
await cancelRunInternal(run.id, current.reason);
|
|
return null;
|
|
}
|
|
await cancelRunInternal(run.id, budgetBlock.reason, { resultJson: { budgetAdmissionStop } });
|
|
return null;
|
|
}
|
|
|
|
const dailyCapBlock = await getHeartbeatDailyCapBlock(
|
|
agent,
|
|
parseHeartbeatPolicy(agent),
|
|
{
|
|
excludeRunId: run.id,
|
|
checkRunCap: true,
|
|
checkCostCap: true,
|
|
},
|
|
);
|
|
if (dailyCapBlock) {
|
|
if (caseHandoffReceipt) return null;
|
|
await cancelQueuedRunForHeartbeatDailyCap(run, dailyCapBlock);
|
|
return null;
|
|
}
|
|
|
|
const issueId = readNonEmptyString(context.issueId);
|
|
if (issueId && activeRunExecutions.size > 0) {
|
|
// Native finalization publishes success before workspace synchronization,
|
|
// provider suspension, and lease release finish. A queued comment must
|
|
// not acquire a fresh sandbox while its predecessor still owns that work.
|
|
// The executor's finally block retries this agent after removing its owner.
|
|
const [settlingOwner] = await db.select({ id: heartbeatRuns.id })
|
|
.from(heartbeatRuns)
|
|
.where(and(
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
or(eq(heartbeatRuns.agentId, run.agentId), inArray(heartbeatRuns.status, [...HEARTBEAT_RUN_TERMINAL_STATUSES])),
|
|
eq(heartbeatRuns.runtimeMode, "native"),
|
|
inArray(heartbeatRuns.id, [...activeRunExecutions]),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,
|
|
))
|
|
.limit(1);
|
|
if (settlingOwner) {
|
|
logger.info({ runId: run.id, agentId: run.agentId, companyId: run.companyId, issueId,
|
|
reason: "predecessor_settling_in_process", predecessorRunId: settlingOwner.id },
|
|
"claimQueuedRun: queued run declined at the issue execution claim");
|
|
return null;
|
|
}
|
|
}
|
|
if (issueId && !run.startedAt && agent.adapterType === "paperclip_runner") {
|
|
const cleanupBlock = await nativeCleanupHoldsAdmission(db, { companyId: run.companyId, issueId });
|
|
if (cleanupBlock) {
|
|
logger.info({ runId: run.id, agentId: run.agentId, companyId: run.companyId, issueId,
|
|
reason: "native_session_cleanup_settling", cleanupBlock },
|
|
"claimQueuedRun: queued run kept queued while the task's previous session finishes cleanup");
|
|
return null;
|
|
}
|
|
}
|
|
if (issueId) {
|
|
const activePauseHold = await treeControlSvc.getActivePauseHoldGate(
|
|
run.companyId,
|
|
issueId,
|
|
);
|
|
const treeHoldInteractionWake =
|
|
activePauseHold &&
|
|
(await isVerifiedIssueTreeControlInteractionWake(db, {
|
|
companyId: run.companyId,
|
|
issueId,
|
|
agentId: run.agentId,
|
|
runId: run.id,
|
|
wakeupRequestId: run.wakeupRequestId,
|
|
contextSnapshot: context,
|
|
}));
|
|
if (activePauseHold && !treeHoldInteractionWake) {
|
|
const refuse = async () => {
|
|
// Record the covering hold so its release re-dispatches this work once.
|
|
await cancelRunInternal(
|
|
run.id,
|
|
"Cancelled because issue is held by an active subtree pause hold",
|
|
{ errorCode: ISSUE_PAUSED_ERROR_CODE, resultJson: pauseSuppressionResult(activePauseHold) },
|
|
);
|
|
await logActivity(db, {
|
|
companyId: run.companyId,
|
|
actorType: "system",
|
|
actorId: "system",
|
|
agentId: run.agentId,
|
|
runId: run.id,
|
|
action: "issue.tree_hold_run_interrupted",
|
|
entityType: "heartbeat_run",
|
|
entityId: run.id,
|
|
issueId: issueId,
|
|
details: {
|
|
issueId,
|
|
holdId: activePauseHold.holdId,
|
|
rootIssueId: activePauseHold.rootIssueId,
|
|
source: "heartbeat.claim_queued_run",
|
|
securityPrinciples: [
|
|
"Complete Mediation",
|
|
"Fail Securely",
|
|
"Secure Defaults",
|
|
],
|
|
},
|
|
});
|
|
// A release that committed before this refusal did not see it.
|
|
await resumeIfPauseAlreadyReleased(db, { companyId: run.companyId, issueId,
|
|
holdId: activePauseHold.holdId, wakeup: (agentId, opts) => trackWakeup(agentId, opts) });
|
|
};
|
|
// Cancellation promotes the agent's next queued run under the agent
|
|
// start lock, so a caller holding that lock settles it after release.
|
|
if (deferredCancellations) deferredCancellations.push(refuse);
|
|
else await refuse();
|
|
return null;
|
|
}
|
|
|
|
const dependencyReadiness = await issuesSvc.listDependencyReadiness(
|
|
run.companyId,
|
|
[issueId],
|
|
);
|
|
const readiness = dependencyReadiness.get(issueId);
|
|
const unresolvedBlockerCount = readiness?.unresolvedBlockerCount ?? 0;
|
|
// The run's wake request records who asked for it.
|
|
const blockerStallVerified = unresolvedBlockerCount > 0 && await verifyBlockerStallWake(db, {
|
|
companyId: run.companyId, issueId, contextSnapshot: context,
|
|
requestedByActorType: run.wakeupRequestId
|
|
? (await db.select({ actor: agentWakeupRequests.requestedByActorType }).from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, run.wakeupRequestId)).then(rows => rows[0]?.actor ?? null))
|
|
: null,
|
|
});
|
|
if (
|
|
unresolvedBlockerCount > 0 &&
|
|
!allowsDependencyBlockedWake(
|
|
context,
|
|
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
|
|
blockerStallVerified,
|
|
await isVerifiedDependencyInteractionResponse(db, { companyId: run.companyId,
|
|
issueId, agentId: run.agentId, context }),
|
|
)
|
|
) {
|
|
await cancelQueuedRunForBlockedDependencies(
|
|
run,
|
|
issueId,
|
|
readiness?.unresolvedBlockerIssueIds ?? [],
|
|
);
|
|
logger.info(
|
|
{ runId: run.id, issueId, unresolvedBlockerCount },
|
|
"claimQueuedRun: cancelled blocked queued run",
|
|
);
|
|
return null;
|
|
}
|
|
|
|
const staleness = await runDispatch.cancelStaleQueuedRun({
|
|
runId: run.id,
|
|
companyId: run.companyId,
|
|
expectedStatus: "queued",
|
|
});
|
|
if (staleness.outcome === "cancelled" || staleness.outcome === "deferred") {
|
|
applyRunDispatchPostCommitEffects(staleness.postCommitEffects);
|
|
if (staleness.outcome === "cancelled" && staleness.errorCode === "issue_assignee_changed") {
|
|
await releaseIssueExecutionAndPromote(run, { suppressImmediateRecovery: true, deferredPostCommitEffects }).catch((err) => {
|
|
logger.warn({ err, runId: run.id, issueId }, "claimQueuedRun: new owner's wake will be promoted by recovery");
|
|
});
|
|
}
|
|
logger.info(
|
|
{
|
|
runId: run.id,
|
|
issueId,
|
|
outcome: staleness.outcome,
|
|
errorCode: staleness.outcome === "cancelled" ? staleness.errorCode : undefined,
|
|
},
|
|
"claimQueuedRun: withheld queued run at the execution gate",
|
|
);
|
|
return null;
|
|
}
|
|
}
|
|
|
|
const claimedAt = new Date();
|
|
const responsibleUserId = await resolveResponsibleUserIdForRun({
|
|
run,
|
|
contextSnapshot: context,
|
|
issueContext: issueId
|
|
? await getIssueExecutionContext(run.companyId, issueId)
|
|
: null,
|
|
routineEnvContext: {
|
|
routineId: null,
|
|
env: null,
|
|
responsibleUserId: null,
|
|
},
|
|
});
|
|
// All ordinary and comment claims use the same company-scoped issue
|
|
// lock. A batch may claim several runs before executeRun tracks any owner.
|
|
async function lockIssueExecutionClaim(tx: Db) {
|
|
const [owner] = issueId ? await tx.select({
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
statusVersion: issues.statusVersion,
|
|
executionRunId: issues.executionRunId,
|
|
checkoutRunId: issues.checkoutRunId,
|
|
}).from(issues).where(and(
|
|
eq(issues.id, issueId), eq(issues.companyId, run.companyId),
|
|
)).for("update") : [];
|
|
const ownsIssue = owner?.assigneeAgentId === run.agentId &&
|
|
context.wakeReason !== "source_scoped_recovery_action";
|
|
const [handoff] = issueId ? await tx.select().from(caseHandoffs).where(and(eq(caseHandoffs.companyId, run.companyId), eq(caseHandoffs.issueId, issueId),
|
|
inArray(caseHandoffs.state, ["awaiting_release", "pending_dispatch"]))).limit(1) : [];
|
|
const declined = (reason: string, predecessorRunId?: string) => {
|
|
logger.info({ runId: run.id, agentId: run.agentId, companyId: run.companyId, issueId, reason,
|
|
...(predecessorRunId ? { predecessorRunId } : {}) }, "claimQueuedRun: queued run declined at the issue execution claim");
|
|
return { ownsIssue, blocked: true };
|
|
};
|
|
if (handoff && (handoff.wakeupRequestId !== run.wakeupRequestId || handoff.state === "awaiting_release"))
|
|
return declined("case_handoff_pending");
|
|
if (caseHandoffReceipt) {
|
|
const [current] = await tx.select({ state: caseHandoffs.state, version: caseHandoffs.version,
|
|
currentVersion: cases.handoffVersion, workIssueId: cases.workIssueId,
|
|
appliedIssueStatusVersion: caseHandoffs.appliedIssueStatusVersion,
|
|
toAgentId: caseHandoffs.toAgentId, wakeupRequestId: caseHandoffs.wakeupRequestId }).from(caseHandoffs)
|
|
.innerJoin(cases, eq(cases.id, caseHandoffs.caseId)).where(and(eq(caseHandoffs.id, caseHandoffReceipt.id),
|
|
eq(caseHandoffs.companyId, run.companyId))).limit(1);
|
|
if (!current || !["pending_dispatch", "delivered"].includes(current.state) || current.version !== current.currentVersion ||
|
|
current.workIssueId !== issueId || current.toAgentId !== run.agentId || current.wakeupRequestId !== run.wakeupRequestId ||
|
|
!ownsIssue || (!run.startedAt && current.appliedIssueStatusVersion !== owner?.statusVersion))
|
|
return declined("case_handoff_superseded");
|
|
}
|
|
if (ownsIssue && ["native_safe_replacement", "native_provider_overloaded"].includes(run.scheduledRetryReason ?? "") &&
|
|
owner.checkoutRunId && owner.checkoutRunId !== run.id) {
|
|
return declined("issue_checked_out_by_another_run", owner.checkoutRunId);
|
|
}
|
|
if (run.scheduledRetryReason === "native_provider_overloaded" && owner?.executionRunId &&
|
|
owner.executionRunId !== run.id && owner.executionRunId !== run.retryOfRunId) {
|
|
return declined("issue_execution_owned_by_another_run", owner.executionRunId);
|
|
}
|
|
if (ownsIssue && issueId && await taskChatRetrySourceCleanupPending(tx, run, issueId)) {
|
|
return declined("task_chat_retry_source_cleanup_pending");
|
|
}
|
|
const executionOwnerId = run.scheduledRetryReason === "native_provider_overloaded"
|
|
? run.retryOfRunId : ownsIssue ? owner?.executionRunId : null;
|
|
for (const previousRunId of new Set([executionOwnerId, ownsIssue ? run.retryOfRunId : null])) {
|
|
if (!previousRunId || previousRunId === run.id) continue;
|
|
const [previous] = await tx.select({ status: heartbeatRuns.status, agentId: heartbeatRuns.agentId,
|
|
issueId: heartbeatRuns.issueId, nativeIssueId: heartbeatRuns.nativeIssueId,
|
|
supersededByFreshContinuation: sql<boolean>`${heartbeatRuns.resultJson} ? 'freshContinuationSupersededAt'`,
|
|
contextIssueId: sql<unknown>`coalesce(${heartbeatRuns.contextSnapshot}->>'issueId', ${heartbeatRuns.contextSnapshot}->>'taskId')`,
|
|
})
|
|
.from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, previousRunId),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
));
|
|
if (previousRunId !== executionOwnerId && (previous?.agentId !== run.agentId ||
|
|
(previous.nativeIssueId ?? previous.issueId ?? previous.contextIssueId) !== issueId)) continue;
|
|
// A terminal result can precede workspace/lease cleanup on this or
|
|
// another controller. Local absence alone is not a release receipt.
|
|
if (!isHeartbeatRunTerminalStatus(previous?.status) ||
|
|
liveRunExecutions.has(previousRunId)) {
|
|
return declined("predecessor_still_executing", previousRunId);
|
|
}
|
|
// A fresh continuation replaced the predecessor: its writes are revoked
|
|
// and it shares no workspace with this run (a fenced remote epoch or a
|
|
// new isolated workspace), so its cleanup continues without holding it.
|
|
if (!previous?.supersededByFreshContinuation && await hasPendingRunCleanup(tx, run.companyId, previousRunId)) {
|
|
return declined("predecessor_cleanup_pending", previousRunId);
|
|
}
|
|
}
|
|
return { ownsIssue, blocked: false };
|
|
}
|
|
async function bindClaimedIssueExecution(tx: Db, ownsIssue: boolean, claimedRun: typeof heartbeatRuns.$inferSelect | null | undefined) {
|
|
if (!claimedRun || !issueId || !ownsIssue) return;
|
|
await notifyStatusCardIssueChange(tx, run.companyId);
|
|
await tx.update(issues).set({
|
|
executionRunId: claimedRun.id,
|
|
executionAgentNameKey: normalizeAgentNameKey(agent.name),
|
|
executionLockedAt: claimedAt,
|
|
updatedAt: claimedAt,
|
|
}).where(and(eq(issues.id, issueId), eq(issues.companyId, run.companyId))).catch((error: unknown) => {
|
|
// Another open task of the same routine holds its execution slot, for
|
|
// example a failed run that waits for its scheduled resume. Roll back
|
|
// this claim and keep the run queued until that slot is released.
|
|
if (isUniqueViolation(error, "issues_open_routine_execution_uq")) {
|
|
throw conflict("Another execution for this routine is already in progress");
|
|
}
|
|
throw error;
|
|
});
|
|
}
|
|
const nativeReviewContext = readNativeReviewAssignmentContext(context);
|
|
const queuedCommentIds = queuedCommentIdsFromRunContext(context);
|
|
if (
|
|
issueId &&
|
|
run.invocationSource === "automation" &&
|
|
queuedCommentIds.length > 0
|
|
)
|
|
await options.beforeChatControlRecoveryCheck?.({
|
|
runId: run.id,
|
|
issueId,
|
|
stage: "claim",
|
|
});
|
|
const queuedCommentClaim =
|
|
!nativeReviewContext && issueId && run.wakeupRequestId && queuedCommentIds.length > 0
|
|
? await db
|
|
.transaction(async (tx) => {
|
|
// Match the queue-edit lock order: issue, wake, then run. Once the
|
|
// run becomes running, a concurrent discard must observe the
|
|
// claimed wake and return an explicit conflict; if discard wins,
|
|
// this claim observes the cancelled queue and does no work.
|
|
const issueClaim = await lockIssueExecutionClaim(tx as unknown as Db);
|
|
if (issueClaim.blocked) return { kind: "stale" as const, run: null };
|
|
const wake = await tx
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, run.wakeupRequestId!),
|
|
eq(agentWakeupRequests.companyId, run.companyId),
|
|
eq(agentWakeupRequests.agentId, run.agentId),
|
|
),
|
|
)
|
|
.for("update")
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
const lockedRun = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, run.id),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.agentId, run.agentId),
|
|
),
|
|
)
|
|
.for("update")
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
if (
|
|
!wake ||
|
|
wake.status !== "queued" ||
|
|
wake.runId !== run.id ||
|
|
!lockedRun ||
|
|
lockedRun.status !== "queued" ||
|
|
lockedRun.wakeupRequestId !== wake.id
|
|
) {
|
|
return { kind: "stale" as const, run: null };
|
|
}
|
|
|
|
if (lockedRun.invocationSource === "automation") {
|
|
const admission = readChatControlRecoveryAdmission(lockedRun);
|
|
if (admission === "invalid")
|
|
return { kind: "stale" as const, run: null };
|
|
const proof = await readChatControlRecoveryStop(
|
|
tx as unknown as Db,
|
|
{
|
|
companyId: lockedRun.companyId,
|
|
issueId,
|
|
agentId: lockedRun.agentId,
|
|
sourceRunId: lockedRun.id,
|
|
},
|
|
true,
|
|
);
|
|
if (proof.kind === "unresolved")
|
|
return { kind: "stale" as const, run: null };
|
|
if (proof.kind === "stopped") {
|
|
const error =
|
|
"Automatic continuation stopped by the committed chat conversation close. Send a fresh request in chat or on the Board.";
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
|
|
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExecutionRunWork(tx, { companyId: run.companyId, agentId: run.agentId, runId: run.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, run);
|
|
const [cancelled] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
errorCode: CHAT_CONTROL_RECOVERY_STOP_CODE,
|
|
error,
|
|
finishedAt: claimedAt,
|
|
resultJson: {
|
|
...parseObject(lockedRun.resultJson),
|
|
automaticRecovery: {
|
|
code: CHAT_CONTROL_RECOVERY_STOP_CODE,
|
|
providerDispatched: false,
|
|
sourceRunId: proof.sourceRunId,
|
|
conversationId: proof.conversationId,
|
|
publicationId: proof.publicationId,
|
|
},
|
|
},
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, lockedRun.id),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
),
|
|
)
|
|
.returning();
|
|
if (!cancelled) return { kind: "stale" as const, run: null };
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "skipped",
|
|
error,
|
|
finishedAt: claimedAt,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wake.id));
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, run.companyId),
|
|
eq(issues.executionRunId, run.id),
|
|
),
|
|
);
|
|
return { kind: "cancelled" as const, run: cancelled };
|
|
}
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
runnerProfileJson: {
|
|
...parseObject(lockedRun.runnerProfileJson),
|
|
[CHAT_CONTROL_RECOVERY_ADMISSION_KEY]:
|
|
chatControlRecoveryAdmission(
|
|
lockedRun,
|
|
admission === "admitted" ? "admitted" : "required",
|
|
),
|
|
},
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(eq(heartbeatRuns.id, lockedRun.id));
|
|
}
|
|
const authoritativeIds = queuedCommentIdsFromWakePayload(
|
|
wake.payload,
|
|
);
|
|
if (authoritativeIds.length === 0) {
|
|
// Legacy/direct comment wakes carry comment ids in their ordinary
|
|
// payload and context, not in the authoritative queued-message
|
|
// envelope. Preserve their established claim path; only an
|
|
// explicitly bound queued-message envelope is subject to the
|
|
// live-comment discard gate below.
|
|
await withExternalAdmissionGuard(tx, run.companyId, run.agentId, async () => {
|
|
await assertNoExternalOverlap(tx, run.companyId, run.agentId);
|
|
await assertAgentRunSlotFree(tx, run);
|
|
});
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
|
|
const [claimedRun] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "running",
|
|
runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`,
|
|
...legacyControllerClaim(run.runtimeMode),
|
|
responsibleUserId,
|
|
startedAt: lockedRun.startedAt ?? claimedAt,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, lockedRun.id),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
),
|
|
)
|
|
.returning();
|
|
await bindClaimedIssueExecution(tx as unknown as Db, issueClaim.ownsIssue, claimedRun);
|
|
confirmRunAdmissionStillOpen(claimedRun);
|
|
return claimedRun
|
|
? { kind: "claimed" as const, run: claimedRun }
|
|
: { kind: "stale" as const, run: null };
|
|
}
|
|
const commentRows = await tx
|
|
.select({
|
|
id: issueComments.id,
|
|
deletedAt: issueComments.deletedAt,
|
|
})
|
|
.from(issueComments)
|
|
.where(
|
|
and(
|
|
eq(issueComments.companyId, run.companyId),
|
|
eq(issueComments.issueId, issueId),
|
|
inArray(issueComments.id, authoritativeIds),
|
|
),
|
|
);
|
|
const liveIds = authoritativeIds.filter((commentId) => {
|
|
const comment = commentRows.find((row) => row.id === commentId);
|
|
return Boolean(comment && !comment.deletedAt);
|
|
});
|
|
if (liveIds.length === 0) {
|
|
const reason = "Queued messages were discarded before dispatch";
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
|
|
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExecutionRunWork(tx, { companyId: run.companyId, agentId: run.agentId, runId: run.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, run);
|
|
const [cancelled] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: claimedAt,
|
|
error: reason,
|
|
errorCode: "queued_comment_discarded",
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, lockedRun.id),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
),
|
|
)
|
|
.returning();
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: claimedAt,
|
|
error: reason,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wake.id));
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, run.companyId),
|
|
eq(issues.executionRunId, run.id),
|
|
),
|
|
);
|
|
return {
|
|
kind: "cancelled" as const,
|
|
run: cancelled ?? lockedRun,
|
|
};
|
|
}
|
|
|
|
await withExternalAdmissionGuard(tx, run.companyId, run.agentId, async () => {
|
|
await assertNoExternalOverlap(tx, run.companyId, run.agentId);
|
|
await assertAgentRunSlotFree(tx, run);
|
|
});
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "claimed",
|
|
claimedAt,
|
|
payload: withQueuedCommentIdsInWakePayload(
|
|
wake.payload,
|
|
liveIds,
|
|
),
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wake.id));
|
|
const [claimedRun] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "running",
|
|
runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`,
|
|
...legacyControllerClaim(run.runtimeMode),
|
|
responsibleUserId,
|
|
startedAt: lockedRun.startedAt ?? claimedAt,
|
|
contextSnapshot: withQueuedCommentIdsInRunContext(
|
|
lockedRun.contextSnapshot,
|
|
liveIds,
|
|
),
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, lockedRun.id),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
),
|
|
)
|
|
.returning();
|
|
await bindClaimedIssueExecution(tx as unknown as Db, issueClaim.ownsIssue, claimedRun);
|
|
confirmRunAdmissionStillOpen(claimedRun);
|
|
return claimedRun
|
|
? { kind: "claimed" as const, run: claimedRun }
|
|
: { kind: "stale" as const, run: null };
|
|
})
|
|
.catch((error) => {
|
|
if (isExternalChatWaitAuthorizationContention(error) || error instanceof RunAdmissionClosedDuringClaim)
|
|
return { kind: "stale" as const, run: null };
|
|
if (error instanceof AgentRunSlotsFull) {
|
|
if (slots) slots.full = true;
|
|
return { kind: "stale" as const, run: null };
|
|
}
|
|
throw error;
|
|
})
|
|
: null;
|
|
if (queuedCommentClaim?.kind === "cancelled") {
|
|
await appendRunEvent(queuedCommentClaim.run, {
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message:
|
|
queuedCommentClaim.run.error ??
|
|
"Queued messages were discarded before dispatch",
|
|
});
|
|
publishLiveEvent({
|
|
companyId: queuedCommentClaim.run.companyId,
|
|
type: "heartbeat.run.status",
|
|
payload: {
|
|
runId: queuedCommentClaim.run.id,
|
|
agentId: queuedCommentClaim.run.agentId,
|
|
status: queuedCommentClaim.run.status,
|
|
invocationSource: queuedCommentClaim.run.invocationSource,
|
|
triggerDetail: queuedCommentClaim.run.triggerDetail,
|
|
error: queuedCommentClaim.run.error ?? null,
|
|
errorCode: queuedCommentClaim.run.errorCode ?? null,
|
|
startedAt: queuedCommentClaim.run.startedAt
|
|
? new Date(queuedCommentClaim.run.startedAt).toISOString()
|
|
: null,
|
|
finishedAt: queuedCommentClaim.run.finishedAt
|
|
? new Date(queuedCommentClaim.run.finishedAt).toISOString()
|
|
: null,
|
|
},
|
|
});
|
|
publishRunLifecyclePluginEvent(queuedCommentClaim.run);
|
|
// Fire-and-forget: nothing else in this path depends on the emission,
|
|
// so it must not delay the return.
|
|
void emitAgentTaskRun(db, queuedCommentClaim.run);
|
|
return null;
|
|
}
|
|
const claimed = queuedCommentClaim
|
|
? queuedCommentClaim.run
|
|
: await withChatControlRecoveryGate(run, "claim", async (tx) => {
|
|
const claimValues = {
|
|
status: "running",
|
|
runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`,
|
|
...legacyControllerClaim(run.runtimeMode),
|
|
responsibleUserId,
|
|
startedAt: run.startedAt ?? claimedAt,
|
|
updatedAt: claimedAt,
|
|
};
|
|
if (nativeReviewContext) {
|
|
if (run.scheduledRetryReason === "native_provider_overloaded") {
|
|
const predecessor = await lockIssueExecutionClaim(tx);
|
|
if (predecessor.blocked) return null;
|
|
}
|
|
return tx.transaction(async (reviewTx) => {
|
|
const reviewClaim = await claimQueuedNativeReviewRun(reviewTx as unknown as Db, {
|
|
run, claimedAt, claimValues,
|
|
agentNameKey: normalizeAgentNameKey(agent.name),
|
|
});
|
|
confirmRunAdmissionStillOpen(reviewClaim);
|
|
return reviewClaim;
|
|
});
|
|
}
|
|
return tx.transaction(async (claimTx) => {
|
|
const issueClaim = await lockIssueExecutionClaim(claimTx as unknown as Db);
|
|
if (issueClaim.blocked) return null;
|
|
if (run.wakeupRequestId) await claimTx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, run.wakeupRequestId), eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.agentId, run.agentId),
|
|
)).for("update");
|
|
const [lockedRun] = await claimTx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.agentId, run.agentId),
|
|
)).for("update");
|
|
if (!lockedRun || lockedRun.status !== "queued") return null;
|
|
await withExternalAdmissionGuard(claimTx, run.companyId, run.agentId, async () => {
|
|
await assertNoExternalOverlap(claimTx, run.companyId, run.agentId);
|
|
await assertAgentRunSlotFree(claimTx, run);
|
|
});
|
|
await notifyChatRunMilestones(claimTx, { companyId: run.companyId, runId: run.id, issueId: issueId ?? undefined });
|
|
const claimedRun = await claimTx.update(heartbeatRuns).set(claimValues).where(and(
|
|
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "queued"),
|
|
)).returning().then((rows) => rows[0] ?? null);
|
|
await bindClaimedIssueExecution(claimTx as unknown as Db, issueClaim.ownsIssue, claimedRun);
|
|
confirmRunAdmissionStillOpen(claimedRun);
|
|
return claimedRun;
|
|
});
|
|
}).catch((error: unknown) => {
|
|
if (error instanceof RunAdmissionClosedDuringClaim) return null;
|
|
if (error instanceof AgentRunSlotsFull) {
|
|
if (slots) slots.full = true;
|
|
return null;
|
|
}
|
|
throw error;
|
|
});
|
|
if (!claimed) return null;
|
|
|
|
publishLiveEvent({
|
|
companyId: claimed.companyId,
|
|
type: "heartbeat.run.status",
|
|
payload: {
|
|
runId: claimed.id,
|
|
agentId: claimed.agentId,
|
|
status: claimed.status,
|
|
invocationSource: claimed.invocationSource,
|
|
triggerDetail: claimed.triggerDetail,
|
|
error: claimed.error ?? null,
|
|
errorCode: claimed.errorCode ?? null,
|
|
startedAt: claimed.startedAt
|
|
? new Date(claimed.startedAt).toISOString()
|
|
: null,
|
|
finishedAt: claimed.finishedAt
|
|
? new Date(claimed.finishedAt).toISOString()
|
|
: null,
|
|
},
|
|
});
|
|
publishRunLifecyclePluginEvent(claimed);
|
|
|
|
if (!nativeReviewContext) {
|
|
crashPoint("wake.claim.after_run_running_before_wakeup_claimed");
|
|
await setWakeupStatus(claimed.wakeupRequestId, "claimed", { claimedAt });
|
|
}
|
|
|
|
// Fix A (lazy locking): stamp executionRunId now that the run is actually running,
|
|
// not at queue time. Guard is idempotent — safe if called more than once.
|
|
const claimedContext = parseObject(claimed.contextSnapshot);
|
|
const claimedIssueId = readNonEmptyString(claimedContext.issueId);
|
|
const claimedWakeReason = readNonEmptyString(claimedContext.wakeReason);
|
|
if (
|
|
!nativeReviewContext && claimedIssueId &&
|
|
claimedWakeReason !== "source_scoped_recovery_action"
|
|
) {
|
|
const claimedAgent = await getAgent(claimed.agentId);
|
|
await db
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: claimed.id,
|
|
executionAgentNameKey: normalizeAgentNameKey(claimedAgent?.name),
|
|
executionLockedAt: claimedAt,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, claimedIssueId),
|
|
eq(issues.companyId, claimed.companyId),
|
|
// Mention/context runs can touch an issue, but only the current assignee
|
|
// owns the issue execution lock shown as the active run.
|
|
eq(issues.assigneeAgentId, claimed.agentId),
|
|
["native_safe_replacement", "native_provider_overloaded"].includes(claimed.scheduledRetryReason ?? "")
|
|
? or(
|
|
isNull(issues.checkoutRunId),
|
|
eq(issues.checkoutRunId, claimed.id),
|
|
)
|
|
: undefined,
|
|
or(
|
|
isNull(issues.executionRunId),
|
|
eq(issues.executionRunId, claimed.id),
|
|
),
|
|
),
|
|
);
|
|
}
|
|
|
|
return claimed;
|
|
}
|
|
|
|
// startNextQueuedRunForAgent checks admission suppression once, then claims
|
|
// runs (sets status "running"), then dispatches each to executeRun, which
|
|
// checks suppression again before it does any work. Suppression (task
|
|
// drain, worktree mode, a database restore) can start in the gap between
|
|
// those two checks. When executeRun's check catches that, the run is
|
|
// already claimed — release it back to "queued" so it does not keep a
|
|
// running execution lock that nothing will ever process. This runs inside
|
|
// the same promise startNextQueuedRunForAgent already tracks in
|
|
// activeRunExecutionPromises, so getTaskDrainStatus() keeps reporting the
|
|
// run as active until the release finishes.
|
|
//
|
|
// The run row, the wakeup request, and the issue execution lock all guard
|
|
// the same claim, so one transaction commits all three writes together. A
|
|
// partial write (for example the run flips to "queued" but the wakeup or
|
|
// issue update then fails) would let the execution promise clear from
|
|
// activeRunExecutionPromises while the wakeup stayed "claimed" or the
|
|
// issue stayed locked to a queued run — task-drain status would then read
|
|
// quiescent while the database still held part of the old claim.
|
|
async function releaseRunClaimedJustBeforeSuppression(runId: string) {
|
|
const now = new Date();
|
|
await db.transaction(async (tx) => {
|
|
const [candidate] = await tx.select({ companyId: heartbeatRuns.companyId, contextSnapshot: heartbeatRuns.contextSnapshot })
|
|
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running"))).for("update");
|
|
if (!candidate) return;
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, runId,
|
|
issueId: readNonEmptyString(parseObject(candidate.contextSnapshot).issueId) ?? undefined });
|
|
const released = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "queued",
|
|
startedAt: null,
|
|
responsibleUserId: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running")),
|
|
)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!released) return;
|
|
|
|
if (released.wakeupRequestId) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({ status: "queued", claimedAt: null, updatedAt: now })
|
|
.where(and(eq(agentWakeupRequests.id, released.wakeupRequestId), ne(agentWakeupRequests.status, "cancelled")));
|
|
}
|
|
|
|
const context = parseObject(released.contextSnapshot);
|
|
const issueId = readNonEmptyString(context.issueId);
|
|
if (issueId) {
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, released.companyId),
|
|
eq(issues.executionRunId, released.id),
|
|
),
|
|
);
|
|
}
|
|
});
|
|
}
|
|
|
|
async function cancelQueuedRunForBlockedDependencies(
|
|
run: typeof heartbeatRuns.$inferSelect,
|
|
issueId: string,
|
|
unresolvedBlockerIssueIds: string[],
|
|
) {
|
|
const now = new Date();
|
|
const reason =
|
|
"Cancelled because issue dependencies are still blocked; Paperclip will wake the assignee when blockers resolve";
|
|
const cancelled = await cancelRunIfStillQueued(run.id, {
|
|
finishedAt: now,
|
|
error: reason,
|
|
errorCode: "issue_dependencies_blocked",
|
|
resultJson: {
|
|
...parseObject(run.resultJson),
|
|
stopReason: "issue_dependencies_blocked",
|
|
effectiveTimeoutSec: 0,
|
|
timeoutConfigured: false,
|
|
timeoutSource: "dependency_gate",
|
|
timeoutFired: false,
|
|
},
|
|
});
|
|
if (!cancelled) return null;
|
|
|
|
await setWakeupStatus(run.wakeupRequestId, "skipped", {
|
|
finishedAt: now,
|
|
error: reason,
|
|
});
|
|
|
|
await db.transaction(async tx => {
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId });
|
|
await notifyNativeStatusReassessment(tx, { companyId: run.companyId, issueId });
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, run.companyId),
|
|
eq(issues.id, issueId),
|
|
eq(issues.executionRunId, run.id),
|
|
),
|
|
);
|
|
});
|
|
|
|
await appendRunEvent(cancelled, {
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message: reason,
|
|
payload: {
|
|
issueId,
|
|
unresolvedBlockerIssueIds,
|
|
},
|
|
});
|
|
|
|
return cancelled;
|
|
}
|
|
|
|
async function resumeDeferredExecutionReceipts(scope: DeferredReceiptRecoveryOptions = {}) {
|
|
if ((await getSchedulingSuppression()).suppressed) return;
|
|
await resumeExecutionWaitComments(scope);
|
|
const cutoff = await getWorktreeExecutionCutoff();
|
|
const pendingInterrupts = await db.select({ id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId })
|
|
.from(agentWakeupRequests).innerJoin(companies, eq(companies.id, agentWakeupRequests.companyId))
|
|
.where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
|
|
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
eq(companies.status, "active"),
|
|
sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is not null`,
|
|
lte(agentWakeupRequests.updatedAt, new Date(Date.now() - 30_000)),
|
|
cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined))
|
|
.orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
|
|
for (const wake of pendingInterrupts) {
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date() }).where(and(
|
|
eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
));
|
|
await resumeQueuedCommentInterrupt(wake.companyId, wake.id).catch(err => {
|
|
scope.onError?.(err);
|
|
logger.warn({ err, queueId: wake.id }, "failed to resume interrupted comment queue");
|
|
});
|
|
}
|
|
// Work-intent outbox: a message refused by a pause waits for the pause to
|
|
// lift, not behind the run. Lifting the pause is the operator's explicit
|
|
// authorization to send what the pause refused, as Send is after a Stop:
|
|
// when the pause stopped the task's run, the release drain never promotes
|
|
// queued work behind that acknowledged stop, so the lifted pause issues the
|
|
// task's next explicit wake here. It reconsiders the exact saved receipt
|
|
// through normal admission (a refusal rewrites the receipt in place) and
|
|
// adopts it atomically, so it is delivered once. It runs before the
|
|
// stranded-queue pass, which touches each saved wake.
|
|
const pauseLifted = await db.select({ wake: agentWakeupRequests })
|
|
.from(agentWakeupRequests)
|
|
.innerJoin(issues, and(eq(issues.companyId, agentWakeupRequests.companyId),
|
|
sql`${issues.id}::text = ${agentWakeupRequests.payload}->>'issueId'`,
|
|
eq(issues.assigneeAgentId, agentWakeupRequests.agentId), isNull(issues.executionRunId),
|
|
notInArray(issues.status, ["done", "cancelled"])))
|
|
.innerJoin(companies, and(eq(companies.id, agentWakeupRequests.companyId), eq(companies.status, "active")))
|
|
.where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
|
|
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId),
|
|
// Only messages the pause path saved (never a chat-inbound receipt,
|
|
// which resumes through its own live authorization), and only pauses
|
|
// this query can see lifted. A refusal by any other gate rewrites the
|
|
// receipt's reason, so it never retries in a loop here.
|
|
// A message the reconciler saved after its wake was lost waits on no
|
|
// hold, so the same explicit wake sends it.
|
|
sql`${agentWakeupRequests.payload}->'executionWait'->>'trigger' in ('pause_lifted', 'reconciler_backstop')`,
|
|
sql`${agentWakeupRequests.payload}->'executionWait'->>'reason' in ('company.inactive', 'agent.not_invokable', 'wake_lost')`,
|
|
eq(agentWakeupRequests.requestedByActorType, "user"),
|
|
sql`coalesce(${agentWakeupRequests.idempotencyKey}, '') not like 'chat-inbound:%'`,
|
|
sql`exists (select 1 from agents pause_agent where pause_agent.id = ${agentWakeupRequests.agentId}
|
|
and pause_agent.status not in ('paused', 'terminated', 'pending_approval'))`,
|
|
sql`NOT EXISTS (SELECT 1 FROM heartbeat_runs live_issue_run
|
|
WHERE live_issue_run.company_id = ${issues.companyId}
|
|
AND (live_issue_run.context_snapshot->>'issueId' = ${issues.id}::text
|
|
OR live_issue_run.native_issue_id = ${issues.id})
|
|
AND live_issue_run.status IN ('queued', 'running', 'scheduled_retry'))`,
|
|
cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined))
|
|
.orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
|
|
const pauseLiftedIssues = new Set<string>();
|
|
for (const { wake } of pauseLifted) {
|
|
const issueKey = `${wake.companyId}:${String(wake.payload?.issueId)}`;
|
|
if (pauseLiftedIssues.has(issueKey)) continue;
|
|
pauseLiftedIssues.add(issueKey);
|
|
// Consumption is per comment. The wake adopts every saved message of
|
|
// the task for this agent, so each receipt drops the comments a
|
|
// completed run already received; a receipt left empty is consumed.
|
|
const issueKeyId = String(wake.payload?.issueId);
|
|
const saved = await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, wake.companyId), eq(agentWakeupRequests.agentId, wake.agentId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueKeyId}`));
|
|
for (const receipt of saved) {
|
|
const commentIds = queuedCommentIdsFromWakePayload(receipt.payload);
|
|
const delivered = await deliveredQueuedCommentIds(db, { companyId: wake.companyId, issueId: issueKeyId,
|
|
agentId: wake.agentId, commentIds: commentIds.filter(isUuidLike) });
|
|
if (delivered.size === 0) continue;
|
|
const remaining = commentIds.filter(commentId => !delivered.has(commentId));
|
|
const now = new Date();
|
|
await db.update(agentWakeupRequests).set(remaining.length === 0
|
|
? { status: "cancelled", finishedAt: now, updatedAt: now, error: QUEUED_MESSAGES_DELIVERED_REASON }
|
|
: { payload: withQueuedCommentIdsInWakePayload(parseObject(receipt.payload), remaining), updatedAt: now })
|
|
.where(and(eq(agentWakeupRequests.id, receipt.id), eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
}
|
|
const [current] = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.id, wake.id),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
if (!current) continue;
|
|
wake.payload = current.payload;
|
|
const context = parseObject(parseObject(wake.payload)[DEFERRED_WAKE_CONTEXT_KEY]);
|
|
const { executionWait: _wait, [DEFERRED_WAKE_CONTEXT_KEY]: _context, ...payload } = parseObject(wake.payload);
|
|
await enqueueWakeup(wake.agentId, {
|
|
source: wake.source as WakeupOptions["source"], triggerDetail: wake.triggerDetail as WakeupOptions["triggerDetail"],
|
|
reason: wake.reason ?? undefined, payload, contextSnapshot: context,
|
|
requestedByActorType: wake.requestedByActorType as WakeupOptions["requestedByActorType"],
|
|
requestedByActorId: wake.requestedByActorId ?? undefined,
|
|
}, wake.id, wake.payload?.interactionOrigin).catch(err => {
|
|
if (err instanceof PersistedExecutionWaitRefusal) return;
|
|
scope.onError?.(err);
|
|
logger.warn({ err, requestId: wake.id }, "a message saved during a pause could not resume");
|
|
});
|
|
}
|
|
|
|
// A server restart or a message/cleanup race can leave a deferred wake
|
|
// after its owner has released the issue lock. Revisit it through the same
|
|
// release admission, so recovery holds and operator Stops still apply.
|
|
const strandedQueues = await db.select({ wake: agentWakeupRequests })
|
|
.from(agentWakeupRequests)
|
|
.innerJoin(issues, and(eq(issues.companyId, agentWakeupRequests.companyId),
|
|
sql`${issues.id}::text = ${agentWakeupRequests.payload}->>'issueId'`,
|
|
or(eq(issues.assigneeAgentId, agentWakeupRequests.agentId),
|
|
sql`${agentWakeupRequests.payload} #>> '{_paperclipWakeContext,wakeReason}' = 'issue_comment_mentioned'`)))
|
|
.innerJoin(companies, and(eq(companies.id, issues.companyId), eq(companies.status, "active")))
|
|
.where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
|
|
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
isNull(issues.executionRunId),
|
|
or(and(
|
|
sql`jsonb_typeof(${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}') = 'array'`,
|
|
sql`${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`,
|
|
), sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`,
|
|
dependencyResolvedWakePayloadSql, assignmentWakePayloadSql),
|
|
sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is null`,
|
|
sql`NOT EXISTS (SELECT 1 FROM heartbeat_runs live_issue_run
|
|
WHERE live_issue_run.company_id = ${issues.companyId}
|
|
AND (live_issue_run.context_snapshot->>'issueId' = ${issues.id}::text
|
|
OR live_issue_run.native_issue_id = ${issues.id})
|
|
AND live_issue_run.status IN ('queued', 'running', 'scheduled_retry'))`,
|
|
cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined))
|
|
.orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
|
|
const releasedNativeSources = new Set<string>();
|
|
for (const { wake } of strandedQueues) {
|
|
const ownerWake = isDependencyResolvedWakePayload(wake.payload) || isAssignmentWakePayload(wake.payload);
|
|
if (!ownerWake && !queuedCommentIdsFromWakePayload(wake.payload).length &&
|
|
!await readQueuedInteractionResponse(db, wake.companyId, String(wake.payload?.issueId), wake.payload)) continue;
|
|
const [latest] = await db.select({
|
|
id: heartbeatRuns.id,
|
|
companyId: heartbeatRuns.companyId,
|
|
status: heartbeatRuns.status,
|
|
finishedAt: heartbeatRuns.finishedAt,
|
|
startedAt: heartbeatRuns.startedAt,
|
|
runtimeMode: heartbeatRuns.runtimeMode,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
sourceVersion: sql<string>`${heartbeatRuns}.xmin::text`,
|
|
}).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, wake.companyId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${String(wake.payload?.issueId)}`,
|
|
or(isNull(heartbeatRuns.nativeIssueId), sql`${heartbeatRuns.nativeIssueId}::text = ${String(wake.payload?.issueId)}`),
|
|
)).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date() }).where(and(
|
|
eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
));
|
|
if (!latest || !latest.finishedAt || !isHeartbeatRunTerminalStatus(latest.status)) continue;
|
|
if (latest.runtimeMode !== "legacy" && latest.runtimeMode !== "native") continue;
|
|
if (wake.idempotencyKey?.startsWith("chat-inbound:")) {
|
|
await resumeDeferredChatWake(db, { companyId: wake.companyId, agentId: wake.agentId,
|
|
issueId: String(wake.payload?.issueId), requestId: wake.id }).catch(err => {
|
|
scope.onError?.(err);
|
|
logger.warn({ requestId: wake.id }, "saved chat source could not resume after recovery");
|
|
});
|
|
continue;
|
|
}
|
|
const releaseSource = `${latest.companyId}:${String(wake.payload?.issueId)}:${latest.id}:${latest.sourceVersion}`;
|
|
if (latest.runtimeMode === "native" && releasedNativeSources.has(releaseSource)) continue;
|
|
if (latest.runtimeMode === "legacy") {
|
|
const cancelledAdmission = latest.status === "cancelled" && !latest.startedAt &&
|
|
latest.errorCode === "execution_reconciliation_required";
|
|
if ((latest.status !== "cancelled" || cancelledAdmission) && await getExecutionBlocker(db, wake.companyId, String(wake.payload?.issueId))) {
|
|
await resumeSavedLegacyComments(wake.companyId, wake.id).catch(err => {
|
|
scope.onError?.(err);
|
|
logger.warn({ err, queueId: wake.id }, "failed to deliver saved legacy comment after recovery stopped");
|
|
});
|
|
}
|
|
}
|
|
await releaseIssueExecutionAndPromote(latest, { suppressImmediateRecovery: true, resumeStrandedQueue: true }).then(() => {
|
|
if (latest.runtimeMode === "native") releasedNativeSources.add(releaseSource);
|
|
}).catch(err => {
|
|
scope.onError?.(err);
|
|
logger.warn({ err, queueId: wake.id }, "failed to promote stranded comments");
|
|
});
|
|
}
|
|
|
|
// The cancellation marker is durable intent. Retry while its exact queue
|
|
// is still deferred, including after a failed cleanup promotion or restart.
|
|
// Normal admission still checks process ownership, leases, pauses, and scope.
|
|
const interruptedQueues = await db
|
|
.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId })
|
|
.from(agentWakeupRequests)
|
|
.innerJoin(heartbeatRuns, and(
|
|
sql`${heartbeatRuns.resultJson}->>'queuedCommentInterruptQueueId' = ${agentWakeupRequests.id}::text`,
|
|
eq(heartbeatRuns.companyId, agentWakeupRequests.companyId),
|
|
eq(heartbeatRuns.agentId, agentWakeupRequests.agentId),
|
|
))
|
|
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
|
|
.where(and(
|
|
scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
|
|
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
eq(heartbeatRuns.status, "cancelled"),
|
|
eq(heartbeatRuns.runtimeMode, "legacy"),
|
|
eq(companies.status, "active"),
|
|
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
|
|
));
|
|
for (const run of interruptedQueues) {
|
|
await releaseIssueExecutionAndPromote(run, { suppressImmediateRecovery: true }).catch((err) => {
|
|
scope.onError?.(err);
|
|
logger.error({ err, runId: run.id }, "failed to retry interrupted comment queue");
|
|
});
|
|
}
|
|
|
|
}
|
|
|
|
async function resumeQueuedRuns() {
|
|
if (isRunAdmissionClosed()) return;
|
|
await resumeDeferredExecutionReceipts();
|
|
await resumeQueuedRunsAfterDrain();
|
|
}
|
|
|
|
// Release admission promptly without first scanning unrelated recovery work.
|
|
// Every run still passes the ordinary per-agent claim and dispatch gates.
|
|
async function resumeQueuedRunsAfterDrain(scope?: { companyId: string }) {
|
|
const refreshing = refreshRunAdmission();
|
|
if (refreshing) await refreshing;
|
|
if (isRunAdmissionClosed() || (await getSchedulingSuppression()).suppressed) return;
|
|
const cutoff = await getWorktreeExecutionCutoff();
|
|
const queuedRuns = await db
|
|
.select({ agentId: heartbeatRuns.agentId })
|
|
.from(heartbeatRuns)
|
|
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.status, "queued"),
|
|
scope ? eq(heartbeatRuns.companyId, scope.companyId) : undefined,
|
|
eq(companies.status, "active"),
|
|
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
|
|
),
|
|
);
|
|
|
|
const agentIds = [...new Set(queuedRuns.map((r) => r.agentId))];
|
|
for (const agentId of agentIds) {
|
|
await startNextQueuedRunForAgent(agentId, true, true);
|
|
}
|
|
}
|
|
|
|
// Startup fast path: start queued turns that never executed, on agents with
|
|
// no run left `running` by the previous process and no queued retry or
|
|
// continuation. Those agents' runs depend on restart recovery and wait for
|
|
// the full queued-run pass. Every run still passes the ordinary per-agent
|
|
// claim and dispatch gates, exactly as a new wakeup during startup would.
|
|
async function resumeFreshQueuedRunsAtStartup() {
|
|
if (isRunAdmissionClosed() || (await getSchedulingSuppression()).suppressed) return { agents: 0 };
|
|
const cutoff = await getWorktreeExecutionCutoff();
|
|
const rows = await db
|
|
.selectDistinct({ agentId: heartbeatRuns.agentId })
|
|
.from(heartbeatRuns)
|
|
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.status, "queued"),
|
|
eq(companies.status, "active"),
|
|
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
|
|
sql`not exists (
|
|
select 1 from heartbeat_runs prior
|
|
where prior.agent_id = ${heartbeatRuns.agentId}
|
|
and (prior.status = 'running' or (prior.status = 'queued' and (
|
|
prior.started_at is not null or prior.retry_of_run_id is not null
|
|
or coalesce(prior.process_loss_retry_count, 0) > 0
|
|
or coalesce(prior.continuation_attempt, 0) > 0)))
|
|
)`,
|
|
),
|
|
);
|
|
for (const { agentId } of rows) {
|
|
await startNextQueuedRunForAgent(agentId, true);
|
|
}
|
|
return { agents: rows.length };
|
|
}
|
|
|
|
// A 403 from claimQueuedRun comes from the run's own persisted identity
|
|
// (an unverifiable interrupt receipt, a manual wake with no user). Those rows
|
|
// do not change, so the claim fails the same way on every pass and restart.
|
|
function isPermanentClaimRejection(err: unknown): err is HttpError {
|
|
return err instanceof HttpError && err.status === 403;
|
|
}
|
|
|
|
// Other 4xx rejections can clear later (a responsible user gets assigned, a
|
|
// conflicting claim finishes). Keep the run queued, but do not let it stop
|
|
// the rest of the queue or startup recovery.
|
|
function isDeferrableClaimRejection(err: unknown): err is HttpError {
|
|
return err instanceof HttpError && err.status >= 400 && err.status < 500;
|
|
}
|
|
|
|
// Settle runs that can never be claimed. Letting the error escape stalls the
|
|
// agent's queue and, during startup recovery, stops the server from booting.
|
|
async function cancelRejectedQueuedRuns(
|
|
rejected: Array<{ run: typeof heartbeatRuns.$inferSelect; err: HttpError }>,
|
|
) {
|
|
for (const { run, err } of rejected) {
|
|
logger.warn(
|
|
{ err, runId: run.id, agentId: run.agentId, companyId: run.companyId },
|
|
"cancelling queued heartbeat run whose claim was rejected",
|
|
);
|
|
try {
|
|
await cancelRunInternal(
|
|
run.id,
|
|
`Cancelled because the queued run cannot be claimed: ${err.message}`,
|
|
{ errorCode: "queued_run_claim_rejected" },
|
|
);
|
|
} catch (cancelErr) {
|
|
logger.error(
|
|
{ err: cancelErr, runId: run.id },
|
|
"failed to cancel queued heartbeat run whose claim was rejected; it stays queued for the next recovery pass",
|
|
);
|
|
}
|
|
}
|
|
}
|
|
|
|
async function attachQueuedSetupRequest(agent: typeof agents.$inferSelect) {
|
|
if (isAgentAwaitingSetup(agent)) await connectionIntentService(db)
|
|
.requestForQueuedAgentSetup(agent.companyId, agent.id)
|
|
.catch(() => logger.warn({ agentId: agent.id }, "Could not attach queued agent setup connection request"));
|
|
}
|
|
|
|
const declinedAgedClaimAt = new Map<string, number>();
|
|
|
|
function noteDeclinedAgedClaim(run: QueuedRunCandidate, reason: string) {
|
|
const now = Date.now();
|
|
if (!hasQueuedRunAged(run, new Date(now))) return;
|
|
declinedAgedClaimAt.set(run.id, now);
|
|
logger.warn({ runId: run.id, agentId: run.agentId, companyId: run.companyId, reason,
|
|
queuedForMs: now - run.createdAt.getTime() },
|
|
"long-queued heartbeat run was offered a free slot but could not be claimed; it stays queued and the next oldest run takes its turn");
|
|
}
|
|
|
|
const claimBackoffs = new Map<string, { agentId: string; issueId: string | null; attempts: number; until: number;
|
|
taskStamp: string | null }>();
|
|
|
|
function claimBackedOff(runId: string, now: Date, taskStamp: string | null) {
|
|
const backoff = claimBackoffs.get(runId);
|
|
return Boolean(backoff && backoff.until > now.getTime() && backoff.taskStamp === taskStamp);
|
|
}
|
|
|
|
function backOffClaim(run: QueuedRunCandidate, personRun: boolean, taskStamp: string | null) {
|
|
if (!hasQueuedRunAged(run, new Date())) return;
|
|
const attempts = (claimBackoffs.get(run.id)?.attempts ?? 0) + 1;
|
|
const delayMs = personRun ? QUEUED_PERSON_RUN_CLAIM_BACKOFF_MS
|
|
: Math.min(QUEUED_RUN_CLAIM_BACKOFF_BASE_MS * 2 ** (attempts - 1), QUEUED_RUN_CLAIM_BACKOFF_MAX_MS);
|
|
claimBackoffs.set(run.id, { agentId: run.agentId, issueId: readNonEmptyString(parseObject(run.contextSnapshot).issueId),
|
|
attempts, until: Date.now() + delayMs, taskStamp });
|
|
}
|
|
|
|
function releaseClaimBackoffsForTask(run: Pick<HeartbeatRun, "issueId" | "contextSnapshot">) {
|
|
const issueId = run.issueId ?? readNonEmptyString(parseObject(run.contextSnapshot).issueId);
|
|
if (!issueId) return;
|
|
for (const [runId, backoff] of claimBackoffs) {
|
|
if (backoff.issueId === issueId) claimBackoffs.delete(runId);
|
|
}
|
|
}
|
|
|
|
function queuedRunTaskStamp(run: QueuedRunCandidate,
|
|
issueById: Map<string, { status: string; updatedAt: Date; executionRunId: string | null }>) {
|
|
const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
|
|
const issue = issueId ? issueById.get(issueId) : null;
|
|
return issue ? `${issue.status}:${issue.updatedAt.getTime()}:${issue.executionRunId ?? ""}` : null;
|
|
}
|
|
|
|
function forgetSettledClaimBackoffs(agentId: string, queuedRuns: QueuedRunCandidate[]) {
|
|
const queued = new Set(queuedRuns.map(run => run.id));
|
|
for (const [runId, backoff] of claimBackoffs) {
|
|
if (backoff.agentId === agentId && !queued.has(runId)) claimBackoffs.delete(runId);
|
|
}
|
|
}
|
|
|
|
function pruneDeclinedAgedClaims(now: Date) {
|
|
for (const [id, at] of declinedAgedClaimAt) {
|
|
if (now.getTime() - at >= QUEUED_RUN_AGING_MS) declinedAgedClaimAt.delete(id);
|
|
}
|
|
}
|
|
|
|
async function queuedRunDispatcherForAgent(
|
|
agent: typeof agents.$inferSelect,
|
|
queuedRuns: QueuedRunCandidate[],
|
|
dependencyReadiness: Map<string, { isDependencyReady: boolean }>,
|
|
issueById: Map<string, { status: string; priority: string | null; updatedAt: Date; executionRunId: string | null }>,
|
|
) {
|
|
const now = new Date();
|
|
pruneDeclinedAgedClaims(now);
|
|
const runById = new Map(queuedRuns.map(run => [run.id, run]));
|
|
const { facts, dispatchInput } = await queuedRunDispatchFacts(db, {
|
|
agent, queuedRuns, dependencyReadiness, issueById, now,
|
|
shareDeferred: runId => declinedAgedClaimAt.has(runId) || claimBackedOff(runId, now, queuedRunTaskStamp(runById.get(runId)!, issueById)) });
|
|
const dispatcher = createQueuedRunDispatcher(facts, dispatchInput);
|
|
const byRun = new Map(facts.map(entry => [entry.run, entry]));
|
|
return {
|
|
next: () => dispatcher.next()?.run ?? null,
|
|
claimed: (run: QueuedRunCandidate) => { const entry = byRun.get(run); if (entry) dispatcher.claimed(entry); },
|
|
personRun: (run: QueuedRunCandidate) => {
|
|
const entry = byRun.get(run);
|
|
return Boolean(entry && queuedRunDispatchClass(entry, now) === "person");
|
|
},
|
|
personOrigin: (run: QueuedRunCandidate) => byRun.get(run)?.requestedByUser ?? false,
|
|
};
|
|
}
|
|
|
|
function startNextQueuedRunForAgent(agentId: string, attachSetupRequest = false, retryRefused = false) {
|
|
return requestQueuedDispatch(agentId, { attachSetupRequest, retryRefused, continuesChain: false, triedRunIds: new Set() });
|
|
}
|
|
|
|
function requestQueuedDispatch(agentId: string, request: QueuedDispatchRequest) {
|
|
return dispatchForAgent<QueuedDispatchRequest, Array<typeof heartbeatRuns.$inferSelect>>(agentId, {
|
|
request,
|
|
idle: [],
|
|
merge: (current, next) => ({
|
|
attachSetupRequest: current.attachSetupRequest || next.attachSetupRequest,
|
|
retryRefused: current.retryRefused || next.retryRefused,
|
|
continuesChain: current.continuesChain && next.continuesChain,
|
|
triedRunIds: current.continuesChain && next.continuesChain
|
|
? new Set([...current.triedRunIds, ...next.triedRunIds]) : new Set<string>(),
|
|
}),
|
|
run: (request, signal) => runQueuedDispatchPass(agentId, request, signal),
|
|
track: (pass) => {
|
|
const tracked = pass.then(() => undefined);
|
|
activeRunExecutionPromises.add(tracked);
|
|
void tracked.finally(() => activeRunExecutionPromises.delete(tracked));
|
|
},
|
|
});
|
|
}
|
|
|
|
async function runQueuedDispatchPass(agentId: string, { attachSetupRequest, retryRefused, triedRunIds }: QueuedDispatchRequest,
|
|
signal: AbortSignal) {
|
|
// Cancelled after the pass ends: cancelRunInternal promotes the agent's
|
|
// next queued run, which asks for another pass of this agent.
|
|
const rejectedClaims: Array<{ run: typeof heartbeatRuns.$inferSelect; err: HttpError }> = [];
|
|
const deferredPostCommitEffects: WakeQueuePostCommitEffect[] = [];
|
|
const deferredCancellations: Array<() => Promise<void>> = [];
|
|
let cancellationReason: string | undefined;
|
|
const settle = async () => {
|
|
// Dispatch promoted inputs after the pass has ended.
|
|
try {
|
|
await applyWakeQueuePostCommitEffects(deferredPostCommitEffects);
|
|
} finally {
|
|
if (cancellationReason) await cancelActiveForAgentInternal(agentId, cancellationReason);
|
|
await cancelRejectedQueuedRuns(rejectedClaims);
|
|
for (const cancel of deferredCancellations) {
|
|
await cancel().catch((err) => logger.warn({ err, agentId },
|
|
"queued heartbeat run refused by a pause hold could not be cancelled; the next claim retries it"));
|
|
}
|
|
}
|
|
};
|
|
try {
|
|
return { value: await claimQueuedRunsForAgent(), settle };
|
|
} catch (err) {
|
|
await settle();
|
|
throw err;
|
|
}
|
|
|
|
async function claimQueuedRunsForAgent(): Promise<Array<typeof heartbeatRuns.$inferSelect>> {
|
|
// Null unless this server has not read the run-start hand-over recently.
|
|
const refreshing = refreshRunAdmission();
|
|
if (refreshing) await refreshing;
|
|
if (isRunAdmissionClosed() || (await getSchedulingSuppression()).suppressed) return [];
|
|
const cutoff = await getWorktreeExecutionCutoff();
|
|
const agent = await getAgent(agentId);
|
|
if (!agent) return [];
|
|
const invokability = await getAgentInvokability(agent);
|
|
if (!invokability.invokable) {
|
|
if (attachSetupRequest) await attachQueuedSetupRequest(agent);
|
|
if (shouldCancelRunsForNonInvokableAgent(invokability)) {
|
|
cancellationReason = `Cancelled because the agent is not invokable: ${invokability.reason}`;
|
|
}
|
|
return [];
|
|
}
|
|
const policy = parseHeartbeatPolicy(agent);
|
|
const runningCount = await countRunningRunsForAgent(agentId);
|
|
const availableSlots = Math.max(0, policy.maxConcurrentRuns - runningCount);
|
|
if (availableSlots <= 0) return [];
|
|
|
|
const queuedRuns = await listQueuedRunCandidates(agentId, cutoff);
|
|
forgetSettledClaimBackoffs(agentId, queuedRuns);
|
|
if (queuedRuns.length === 0) return [];
|
|
if (queuedRuns.every(run => !run.startedAt) && !await canStartNewExternalBotWork(agent)) return [];
|
|
|
|
const dependencyReadiness = await listQueuedRunDependencyReadiness(agent.companyId, queuedRuns);
|
|
const queuedIssueIds = [...new Set(queuedRuns.flatMap((run) => {
|
|
const issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
|
|
return issueId ? [issueId] : [];
|
|
}))];
|
|
const issueRows = queuedIssueIds.length > 0
|
|
? await db.select({ id: issues.id, status: issues.status, priority: issues.priority, updatedAt: issues.updatedAt,
|
|
executionRunId: issues.executionRunId }).from(issues)
|
|
.where(and(eq(issues.companyId, agent.companyId), inArray(issues.id, queuedIssueIds)))
|
|
: [];
|
|
const issueById = new Map(issueRows.map((row) => [row.id, row]));
|
|
const companyAgents = await listCompanyAgentOrgRows(agent.companyId);
|
|
const dispatcher = await queuedRunDispatcherForAgent(agent, queuedRuns, dependencyReadiness, issueById);
|
|
|
|
const claimedRuns: Array<typeof heartbeatRuns.$inferSelect> = [];
|
|
const slots = { full: false };
|
|
let claimAttempts = 0;
|
|
const triedThisPass: string[] = [];
|
|
for (let candidate = dispatcher.next(); candidate; candidate = dispatcher.next()) {
|
|
if (claimedRuns.length >= availableSlots || slots.full || isRunAdmissionClosed() || signal.aborted) break;
|
|
const taskStamp = queuedRunTaskStamp(candidate, issueById);
|
|
if (triedRunIds.has(candidate.id) || (!retryRefused && claimBackedOff(candidate.id, new Date(), taskStamp))) continue;
|
|
if (claimAttempts >= QUEUED_RUN_CLAIM_ATTEMPTS_PER_PASS) {
|
|
void requestQueuedDispatch(agentId, { attachSetupRequest: false, retryRefused,
|
|
continuesChain: true, triedRunIds: new Set([...triedRunIds, ...triedThisPass]) });
|
|
break;
|
|
}
|
|
claimAttempts += 1;
|
|
triedThisPass.push(candidate.id);
|
|
const [queuedRun] = await db.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, candidate.id), eq(heartbeatRuns.companyId, candidate.companyId), eq(heartbeatRuns.status, "queued")));
|
|
if (!queuedRun) continue;
|
|
let claimed: typeof heartbeatRuns.$inferSelect | null;
|
|
try {
|
|
claimed = await claimQueuedRun(queuedRun, companyAgents, deferredPostCommitEffects, deferredCancellations, slots);
|
|
} catch (err) {
|
|
if (isPermanentClaimRejection(err)) {
|
|
rejectedClaims.push({ run: queuedRun, err });
|
|
continue;
|
|
}
|
|
if (!isDeferrableClaimRejection(err)) throw err;
|
|
logger.warn(
|
|
{ err, runId: queuedRun.id, agentId: queuedRun.agentId, companyId: queuedRun.companyId },
|
|
"queued heartbeat run claim was rejected; leaving it queued for the next recovery pass",
|
|
);
|
|
noteDeclinedAgedClaim(candidate, err.message);
|
|
backOffClaim(candidate, dispatcher.personRun(candidate), taskStamp);
|
|
continue;
|
|
}
|
|
if (!claimed) {
|
|
if (slots.full) break;
|
|
noteDeclinedAgedClaim(candidate, "claim_declined");
|
|
backOffClaim(candidate, dispatcher.personRun(candidate), taskStamp);
|
|
continue;
|
|
}
|
|
claimedRuns.push(claimed);
|
|
dispatcher.claimed(candidate);
|
|
declinedAgedClaimAt.delete(candidate.id);
|
|
claimBackoffs.delete(candidate.id);
|
|
// Start each run as soon as it is claimed. The claim committed a legacy
|
|
// controller lease that only the run's own execution renews; waiting
|
|
// for the rest of a slow batch to be claimed let that lease lapse.
|
|
startClaimedRun(claimed);
|
|
}
|
|
return claimedRuns;
|
|
}
|
|
|
|
function startClaimedRun(claimedRun: typeof heartbeatRuns.$inferSelect) {
|
|
const execution = outsideAgentDispatch(() => executeRun(claimedRun.id)).catch((err) => {
|
|
logger.error(
|
|
{ err, runId: claimedRun.id },
|
|
"queued heartbeat execution failed",
|
|
);
|
|
});
|
|
// Register the in-flight execution so drainActiveRunExecutions() can await
|
|
// it. executeRun resolves only after its finally block finishes flushing
|
|
// run rows/events, so awaiting this promise guarantees the run's writes
|
|
// have landed before a caller (e.g. a test's afterEach) mutates the DB.
|
|
activeRunExecutionPromises.add(execution);
|
|
void execution.finally(() => {
|
|
// drainActiveRunExecutions loops on activeRunExecutionPromises.size,
|
|
// so an entry that never clears here would hang it forever.
|
|
activeRunExecutionPromises.delete(execution);
|
|
});
|
|
}
|
|
}
|
|
|
|
async function listQueuedRunCandidates(agentId: string, cutoff: Date | null): Promise<QueuedRunCandidate[]> {
|
|
const contextText = (key: string) => sql<string | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}->${key}) = 'string'
|
|
then ${heartbeatRuns.contextSnapshot}->>${key} end`;
|
|
const rows = await db.select({
|
|
id: heartbeatRuns.id,
|
|
companyId: heartbeatRuns.companyId,
|
|
agentId: heartbeatRuns.agentId,
|
|
status: heartbeatRuns.status,
|
|
invocationSource: heartbeatRuns.invocationSource,
|
|
createdAt: heartbeatRuns.createdAt,
|
|
startedAt: heartbeatRuns.startedAt,
|
|
wakeupRequestId: heartbeatRuns.wakeupRequestId,
|
|
retryOfRunId: heartbeatRuns.retryOfRunId,
|
|
scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
|
|
scheduledRetryAttempt: heartbeatRuns.scheduledRetryAttempt,
|
|
scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
|
|
issueId: contextText("issueId"),
|
|
wakeReason: contextText("wakeReason"),
|
|
previousRunId: contextText("previousRunId"),
|
|
}).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
|
|
)).orderBy(asc(heartbeatRuns.createdAt));
|
|
return rows.map(({ issueId, wakeReason, previousRunId, ...run }) => ({
|
|
...run,
|
|
contextSnapshot: Object.fromEntries(Object.entries({ issueId, wakeReason, previousRunId }).filter(([, value]) => value !== null)),
|
|
}));
|
|
}
|
|
|
|
// Public wakeup entry point. Callers dispatch it fire-and-forget, so register
|
|
// the promise in activeWakeupPromises before it starts its asynchronous
|
|
// prologue. drainActiveRunExecutions can then await a wake that is still before
|
|
// run registration. Internal callers reference enqueueWakeup directly and
|
|
// already await it, so they do not need this registration.
|
|
function trackWakeup(
|
|
agentId: string,
|
|
opts: WakeupOptions = {},
|
|
): ReturnType<typeof enqueueWakeup> {
|
|
// Until its wake and run commit, this admission exists only in this process.
|
|
const unsaved = {};
|
|
unsavedWakeAdmissions.add(unsaved);
|
|
const promise = enqueueWakeup(agentId, opts, undefined, undefined, { saved: () => { unsavedWakeAdmissions.delete(unsaved); } });
|
|
activeWakeupPromises.add(promise);
|
|
void promise
|
|
.catch(() => {})
|
|
.finally(() => {
|
|
unsavedWakeAdmissions.delete(unsaved);
|
|
activeWakeupPromises.delete(promise);
|
|
});
|
|
return promise;
|
|
}
|
|
|
|
async function enqueueWakeup(agentId: string, opts: WakeupOptions = {}, executionWaitRequestId?: string, savedInteractionOrigin?: unknown,
|
|
admission?: { saved(): void }) {
|
|
const source = opts.source ?? "on_demand";
|
|
const triggerDetail = opts.triggerDetail ?? null;
|
|
const contextSnapshot: Record<string, unknown> = {
|
|
...(opts.contextSnapshot ?? {}),
|
|
};
|
|
const reason = opts.reason ?? null;
|
|
let payload = opts.payload ? { ...opts.payload } : null;
|
|
// Only the board queue route can record interruption authority on an
|
|
// existing receipt. Never accept this internal marker from a wake caller.
|
|
if (payload) {
|
|
delete payload.queuedCommentInterrupt;
|
|
delete payload.manualUserWake;
|
|
delete payload.taskRetry;
|
|
}
|
|
if (opts.manualUserWake) {
|
|
if (opts.requestedByActorType !== "user" || !opts.requestedByActorId || opts.failedRunId) {
|
|
throw new HttpError(403, "Manual wake requires an authenticated user");
|
|
}
|
|
payload = { ...payload, manualUserWake: true };
|
|
}
|
|
const executionReconciliationWake =
|
|
contextSnapshot.source === "execution.reconciled" ||
|
|
opts.idempotencyKey?.startsWith("execution-reconciliation:") === true;
|
|
const {
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
issueIdFromPayload,
|
|
taskKey,
|
|
wakeCommentId,
|
|
} = enrichWakeContextSnapshot({
|
|
contextSnapshot,
|
|
reason,
|
|
source,
|
|
triggerDetail,
|
|
payload,
|
|
});
|
|
const isManualContextCompactionWake = () => Boolean(
|
|
reason === CONTEXT_COMPACTION_WAKE_REASON && source === "on_demand" && triggerDetail === "manual"
|
|
&& opts.manualUserWake === true && opts.requestedByActorType === "user"
|
|
&& Boolean(opts.requestedByActorId) && opts.allowRunCoalescing === false
|
|
&& payload?.intent === CONTEXT_COMPACTION_WAKE_REASON
|
|
&& readNonEmptyString(payload.requestId) !== null
|
|
&& readContextCompactionRequest(enrichedContextSnapshot)?.requestId === payload.requestId,
|
|
);
|
|
const hasIndependentConversationWakeCause = () => Boolean(
|
|
wakeCommentId || opts.taskRetryRequestId ||
|
|
hasInteractionContinuationWakeContext(enrichedContextSnapshot) ||
|
|
reason === CHAT_COMPLETION_WAKE_REASON || isManualContextCompactionWake(),
|
|
);
|
|
// Keep each request's own server-derived origin, including coalesced wakes.
|
|
// A run's merged context cannot establish which caller authored one receipt.
|
|
// Overwrite caller-supplied nested context rather than trusting it.
|
|
payload = withInteractionDeliveryOrigin(
|
|
{ ...payload, [DEFERRED_WAKE_CONTEXT_KEY]: { ...enrichedContextSnapshot } },
|
|
opts.requestedByActorType, opts.requestedByActorId, savedInteractionOrigin, opts.requestedByActorCompanyId,
|
|
);
|
|
let issueId =
|
|
readNonEmptyString(enrichedContextSnapshot.issueId) ?? issueIdFromPayload;
|
|
if (executionReconciliationWake && !issueId) return null;
|
|
const durableRequest = opts.durableChatRequest;
|
|
let taskRetryActorId: string | null = null;
|
|
delete enrichedContextSnapshot.boardMessageRetry;
|
|
if (opts.taskRetryRequestId) {
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, opts.taskRetryRequestId), eq(agentWakeupRequests.agentId, agentId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
eq(agentWakeupRequests.requestedByActorType, "user"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId ?? ""}`,
|
|
));
|
|
const retry = readTaskRetryIntent(receipt?.payload);
|
|
if (!receipt || retry?.generationId !== receipt.id || retry.canonicalIntentId || retry.queuedChatRequestId ||
|
|
retry.actorId !== receipt.requestedByActorId) return null;
|
|
const chatRetry = retry.chatActionId && durableRequest?.id === retry.chatActionId &&
|
|
durableRequest.failedRunRetry?.failedRunId === retry.sourceRunId;
|
|
if (retry.chatActionId ? !chatRetry :
|
|
opts.requestedByActorType !== "user" || retry.actorId !== opts.requestedByActorId) return null;
|
|
taskRetryActorId = retry.actorId;
|
|
enrichedContextSnapshot.forceFreshSession = true;
|
|
enrichedContextSnapshot.previousRunId = retry.sourceRunId;
|
|
enrichedContextSnapshot.taskRetryRequestId = receipt.id;
|
|
if (retry.boardMessageOrigin) enrichedContextSnapshot.boardMessageRetry = true;
|
|
}
|
|
|
|
|
|
let agent = await getAgent(agentId);
|
|
if (!agent) throw notFound("Agent not found");
|
|
if (issueId && await hasAutomaticInteractionDeliveryHold(db, {
|
|
companyId: agent.companyId, issueId, agentId, payload, context: enrichedContextSnapshot,
|
|
actorType: opts.requestedByActorType, actorId: opts.requestedByActorId,
|
|
})) return null;
|
|
if (issueId) {
|
|
const [pending] = await db.select().from(caseHandoffs).where(and(eq(caseHandoffs.companyId, agent.companyId), eq(caseHandoffs.issueId, issueId),
|
|
inArray(caseHandoffs.state, ["awaiting_release", "pending_dispatch"]))).limit(1);
|
|
if (pending && opts.durableCaseHandoffRequest?.handoffId !== pending.id) return null;
|
|
}
|
|
// Mentions only annotate comments. Ignore legacy callers before creating
|
|
// a run or deferred request; assignment and review have their own wakes.
|
|
if (reason === "issue_comment_mentioned" || enrichedContextSnapshot.wakeReason === "issue_comment_mentioned") return null;
|
|
if (issueId) {
|
|
const conversation = await getIssueExecutionContext(agent.companyId, issueId);
|
|
if (taskRetryActorId && conversation) await assertTaskRetryConversationAllowed(db, conversation, taskRetryActorId);
|
|
if (reason === "issue_children_completed" && conversation?.originKind === "onboarding_first_task") enrichedContextSnapshot.onboardingCompletion = true;
|
|
if (isConversation(conversation)) {
|
|
if (opts.manualUserWake && conversation!.conversationUserId !== opts.requestedByActorId) {
|
|
throw new HttpError(403, "Only the conversation owner can start a chat run");
|
|
}
|
|
if (isConversationExecutionWake(conversation, reason ?? readNonEmptyString(enrichedContextSnapshot.wakeReason))) return null;
|
|
if (agent.id !== conversation!.conversationAgentId) return null;
|
|
if (!(await instanceSettings.getExperimental()).enableAgentChat) return null;
|
|
if (isWaitingConversation(conversation) && !hasIndependentConversationWakeCause()) return null;
|
|
}
|
|
}
|
|
if (agent.adapterType === "paperclip_runner") {
|
|
const oldConfig = parseObject(agent.adapterConfig);
|
|
const nextConfig = normalizeLegacyRunnerProvider(oldConfig);
|
|
if (nextConfig !== oldConfig) {
|
|
await agentService(db).update(
|
|
agent.id,
|
|
{ adapterConfig: nextConfig },
|
|
{
|
|
recordRevision: {
|
|
source: "normalize_runner_provider",
|
|
createdByAgentId: null,
|
|
createdByUserId: null,
|
|
},
|
|
},
|
|
);
|
|
await logActivity(db, {
|
|
companyId: agent.companyId,
|
|
actorType: "system",
|
|
actorId: "heartbeat",
|
|
action: "agent.updated",
|
|
entityType: "agent",
|
|
entityId: agent.id,
|
|
details: { provider: "codex", reason: "native_codex_provider" },
|
|
});
|
|
agent = (await getAgent(agentId))!;
|
|
}
|
|
}
|
|
|
|
if (opts.failedRunId) {
|
|
const failed = await getRun(opts.failedRunId, { includeExecutionEvidence: true });
|
|
if (opts.requestedByActorType !== "user" || !opts.requestedByActorId ||
|
|
reason !== "retry_failed_run" || source !== "on_demand" || triggerDetail !== "manual" ||
|
|
!failed || failed.companyId !== agent.companyId || failed.agentId !== agentId ||
|
|
!(await canRetryStoppedRun(db, failed)) ||
|
|
(failed.nativeIssueId ?? readNonEmptyString(failed.contextSnapshot?.issueId)) !== issueId) {
|
|
throw conflict("The selected failed run cannot be retried for this task.");
|
|
}
|
|
if (!activeRunExecutions.has(failed.id) && !adapterExecutionControls.has(failed.id)) {
|
|
await sweepPendingCleanupLeases({ explicitRetry: {
|
|
companyId: failed.companyId, runId: failed.id, actorId: opts.requestedByActorId,
|
|
} });
|
|
}
|
|
if (isConversationAdapter(agent.adapterType) || agent.adapterType === "paperclip_runner") {
|
|
enrichedContextSnapshot.previousRunId = failed.id;
|
|
enrichedContextSnapshot.forceFreshSession = true;
|
|
}
|
|
}
|
|
|
|
const caseRequest = opts.durableCaseHandoffRequest;
|
|
const waitForCaseAdmission = (reason: string, message: string): never => {
|
|
// The case ledger is the outbox. Roll back this admission attempt rather
|
|
// than consume its one receipt ID in a generic skipped/deferred queue.
|
|
throw conflict(message, { code: "case_handoff_admission_wait", reason });
|
|
};
|
|
let caseReceipt: { id: string; requestedAt: Date } | undefined;
|
|
if (caseRequest) {
|
|
if (opts.taskRetryRequestId || opts.durableChatRequest || opts.durableDotRequest || opts.durableMuseRequest || source !== "assignment" || reason !== "case_handoff")
|
|
throw conflict("Invalid case handoff wake authority");
|
|
const [handoff] = await db.select().from(caseHandoffs).where(and(eq(caseHandoffs.id, caseRequest.handoffId),
|
|
eq(caseHandoffs.companyId, agent.companyId), eq(caseHandoffs.issueId, caseRequest.issueId),
|
|
eq(caseHandoffs.toAgentId, agentId), eq(caseHandoffs.wakeupRequestId, caseRequest.wakeupRequestId),
|
|
inArray(caseHandoffs.state, ["pending_dispatch", "delivered"]))).limit(1);
|
|
if (!handoff || issueId !== handoff.issueId || caseRequest.expectedIssueStatusVersion !== handoff.appliedIssueStatusVersion)
|
|
throw conflict("Case handoff no longer matches its task assignment");
|
|
caseReceipt = { id: handoff.wakeupRequestId, requestedAt: handoff.createdAt };
|
|
opts = { ...opts, allowRunCoalescing: false, idempotencyKey: `case-handoff:${handoff.id}`,
|
|
issueStateGuard: { assigneeAgentId: agentId, statuses: ["todo", "in_progress"], statusVersion: caseRequest.expectedIssueStatusVersion } };
|
|
}
|
|
if (durableRequest) {
|
|
assertDurableChatWakeupRequest(durableRequest, {
|
|
agentId,
|
|
companyId: agent.companyId,
|
|
issueId,
|
|
commentId: wakeCommentId ?? null,
|
|
requestedByActorType: opts.requestedByActorType,
|
|
requestedByActorId: opts.requestedByActorId,
|
|
});
|
|
opts = { ...opts, idempotencyKey: durableRequest.idempotencyKey };
|
|
}
|
|
if (
|
|
Object.hasOwn(enrichedContextSnapshot, "chatFailedRunRetry") &&
|
|
!durableRequest?.failedRunRetry
|
|
) {
|
|
throw new FailedChatRunRetryAuthorizationError();
|
|
}
|
|
if (durableRequest?.failedRunRetry) {
|
|
opts = { ...opts, allowRunCoalescing: false };
|
|
}
|
|
const dotRequest = opts.durableDotRequest;
|
|
const museRequest = opts.durableMuseRequest;
|
|
if (dotRequest && (durableRequest || museRequest || dotRequest.agentId !== agentId ||
|
|
dotRequest.companyId !== agent.companyId || dotRequest.issueId !== issueId ||
|
|
dotRequest.requestId !== payload?.dotRequestId || source !== "assignment" ||
|
|
opts.requestedByActorType !== "agent" || opts.requestedByActorId !== agentId ||
|
|
agent.adapterType !== "paperclip_runner" || !["openai_dot", "grok_bot"].includes(String(parseObject(agent.adapterConfig).provider)) ||
|
|
opts.idempotencyKey !== dotRequest.idempotencyKey)) {
|
|
throw conflict("Dot work request does not match its admission authority.");
|
|
}
|
|
if (museRequest && (durableRequest || dotRequest || museRequest.agentId !== agentId ||
|
|
museRequest.companyId !== agent.companyId || museRequest.issueId !== issueId ||
|
|
museRequest.requestId !== payload?.museRequestId || source !== "assignment" ||
|
|
opts.requestedByActorType !== "agent" || opts.requestedByActorId !== agentId ||
|
|
agent.adapterType !== "paperclip_runner" || parseObject(agent.adapterConfig).provider !== "muse" ||
|
|
parseObject(agent.adapterConfig).museBindingId !== museRequest.bindingId ||
|
|
!Number.isSafeInteger(museRequest.bindingGeneration) || museRequest.bindingGeneration < 1 ||
|
|
payload?.museBindingId !== museRequest.bindingId || payload?.museBindingGeneration !== museRequest.bindingGeneration ||
|
|
payload?.museRequestDigest !== museRequest.requestDigest || !/^sha256:[a-f0-9]{64}$/.test(museRequest.requestDigest) ||
|
|
opts.idempotencyKey !== museRequest.idempotencyKey)) {
|
|
throw conflict("Muse work request does not match its admission authority.");
|
|
}
|
|
const receiptRequest = durableRequest ?? dotRequest ?? museRequest ?? caseReceipt;
|
|
const durableReceiptFields = receiptRequest
|
|
? { id: receiptRequest.id, requestedAt: receiptRequest.requestedAt }
|
|
: {};
|
|
const existingDurableReceipt = async (queryDb: Db) => {
|
|
if (!receiptRequest) return null;
|
|
const receipt = await queryDb
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, receiptRequest.id))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
if (receipt && durableRequest) assertDurableChatWakeupReceipt(durableRequest, receipt);
|
|
if (receipt && caseRequest && (receipt.companyId !== agent.companyId || receipt.agentId !== agentId ||
|
|
receipt.payload?.issueId !== caseRequest.issueId || receipt.idempotencyKey !== `case-handoff:${caseRequest.handoffId}`))
|
|
throw conflict("Case handoff wake receipt identity changed");
|
|
if (receipt && dotRequest && (receipt.companyId !== dotRequest.companyId ||
|
|
receipt.agentId !== dotRequest.agentId || receipt.source !== "assignment" ||
|
|
receipt.requestedByActorType !== "agent" || receipt.requestedByActorId !== dotRequest.agentId ||
|
|
receipt.idempotencyKey !== dotRequest.idempotencyKey || receipt.payload?.issueId !== dotRequest.issueId ||
|
|
receipt.payload?.dotRequestId !== dotRequest.requestId)) {
|
|
throw conflict("requestId was reused for another task.");
|
|
}
|
|
if (receipt && museRequest && (receipt.companyId !== museRequest.companyId ||
|
|
receipt.agentId !== museRequest.agentId || receipt.source !== "assignment" ||
|
|
receipt.requestedByActorType !== "agent" || receipt.requestedByActorId !== museRequest.agentId ||
|
|
receipt.idempotencyKey !== museRequest.idempotencyKey || receipt.payload?.issueId !== museRequest.issueId ||
|
|
receipt.payload?.museRequestId !== museRequest.requestId || receipt.payload?.museBindingId !== museRequest.bindingId ||
|
|
receipt.payload?.museBindingGeneration !== museRequest.bindingGeneration || receipt.payload?.museRequestDigest !== museRequest.requestDigest)) {
|
|
throw conflict("Request ID reused with changed input.");
|
|
}
|
|
if (receipt && durableRequest && receipt.status === "deferred_issue_execution" && !receipt.runId &&
|
|
((durableRequest.failedRunRetry && opts.taskRetryRequestId &&
|
|
receipt.payload?.taskRetryRequestId === opts.taskRetryRequestId) ||
|
|
(durableRequest.resumeDeferred && !durableRequest.failedRunRetry &&
|
|
receipt.payload?.issueId === durableRequest.issueId &&
|
|
queuedCommentIdsFromWakePayload(receipt.payload).includes(durableRequest.commentId)))) {
|
|
// The live inbound owner or a bound Retry may reconsider the same saved receipt.
|
|
// The issue lock, live chat authority and execution holds below still
|
|
// decide whether it can dispatch; this never creates another wake.
|
|
executionWaitRequestId = receipt.id;
|
|
return null;
|
|
}
|
|
return receipt;
|
|
};
|
|
const priorReceipt = await existingDurableReceipt(db);
|
|
if (priorReceipt) {
|
|
// Replaying an admission receipt is not fresh authority to dispatch it.
|
|
// The normal queue owns dispatch and its current execution-policy checks.
|
|
return priorReceipt.runId ? getRun(priorReceipt.runId) : null;
|
|
}
|
|
|
|
const agentDebug = parseObject(parseObject(agent.runtimeConfig).debug);
|
|
const runDebug = parseObject(enrichedContextSnapshot.debug);
|
|
if (
|
|
agentDebug.providerTrace === "raw" &&
|
|
runDebug.providerTrace !== "raw"
|
|
) {
|
|
enrichedContextSnapshot.debug = {
|
|
...runDebug,
|
|
providerTrace: "raw",
|
|
};
|
|
enrichedContextSnapshot.providerTraceRequestedBy = `agent:${agent.id}:debug-setting`;
|
|
enrichedContextSnapshot.providerTraceRequestSource =
|
|
"agent_debug_setting";
|
|
}
|
|
|
|
// Automatic signals are replaceable; user input and interaction delivery
|
|
// retain distinct durable receipts even when the same gate blocks them.
|
|
const coalesceExecutionWait =
|
|
opts.requestedByActorType === "system" &&
|
|
opts.allowRunCoalescing !== false &&
|
|
!receiptRequest &&
|
|
!wakeCommentId &&
|
|
queuedCommentIdsFromRunContext(enrichedContextSnapshot).length === 0 &&
|
|
!isInteractionResolutionWakePayload(payload ?? {}) &&
|
|
!hasInteractionContinuationWakeContext(enrichedContextSnapshot);
|
|
const writeSkippedRequest = async (
|
|
skipReason: string,
|
|
patch: Partial<typeof agentWakeupRequests.$inferInsert> = {},
|
|
waitCondition?: Record<string, unknown>,
|
|
): Promise<{ created: boolean; waitSaved?: boolean }> => {
|
|
// A temporary refusal must leave the handoff retryable, without burning
|
|
// its one durable receipt ID on a skipped wake.
|
|
if (caseRequest) return { created: false };
|
|
if (executionWaitRequestId) {
|
|
const waitSaved = await writeDeferredExecutionWait(db, { companyId: agent.companyId, requestId: executionWaitRequestId }, {
|
|
reason: skipReason, message: patch.error ?? (skipReason === "issue_tree_hold_active"
|
|
? "This task is paused. Resume it to send your saved message."
|
|
: "Waiting for task execution to be enabled. Your message is saved."),
|
|
});
|
|
return { created: false, waitSaved };
|
|
}
|
|
const request = {
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: skipReason,
|
|
payload,
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
...patch,
|
|
};
|
|
if (waitCondition && issueId && isUuidLike(issueId)) {
|
|
const waitIssueId = issueId;
|
|
return db.transaction(async (tx) => {
|
|
await tx.execute(sql`select id from issues where id = ${waitIssueId} and company_id = ${agent.companyId} for update`);
|
|
if (durableRequest) await notifyChatInboundWakeNotices(tx, {
|
|
companyId: durableRequest.companyId, issueId: durableRequest.issueId,
|
|
});
|
|
return recordExecutionWait(tx as unknown as Db, {
|
|
issueId: waitIssueId, request, condition: waitCondition, coalesce: coalesceExecutionWait,
|
|
});
|
|
});
|
|
}
|
|
if (durableRequest) {
|
|
await db.transaction(async tx => {
|
|
await notifyChatInboundWakeNotices(tx, {
|
|
companyId: durableRequest.companyId, issueId: durableRequest.issueId,
|
|
});
|
|
await tx.insert(agentWakeupRequests).values(request);
|
|
});
|
|
} else {
|
|
await db.insert(agentWakeupRequests).values(request);
|
|
}
|
|
return { created: true };
|
|
};
|
|
const writeSkippedHeartbeatRequest = async (
|
|
skipReason: string,
|
|
details: Record<string, unknown>,
|
|
) => {
|
|
await writeSkippedRequest(skipReason, {
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: details,
|
|
},
|
|
});
|
|
};
|
|
|
|
// Work-intent outbox (invariant 4): a user's message refused by a
|
|
// temporary hold is saved as a deferred intent that names the refusal and
|
|
// the hold-release event that promotes it, never written as a terminal
|
|
// skip. Promotion runs normal admission again, once.
|
|
// The release drain promotes a saved message behind the task's latest run,
|
|
// once that run is terminal; a run still stopping after a pause commits is
|
|
// that run. A task that never ran has none: its message stays in the task,
|
|
// and stranded-work recovery starts the task once the hold lifts.
|
|
const canSaveDeferredMessage = async () => {
|
|
if (executionWaitRequestId || caseRequest || durableRequest || !issueId || !isUuidLike(issueId) ||
|
|
opts.requestedByActorType !== "user" || !wakeCommentId) return false;
|
|
const [latest] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.companyId, agent.companyId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`))
|
|
.limit(1);
|
|
return Boolean(latest);
|
|
};
|
|
const saveDeferredMessage = async (refusal: string, message: string, details: Record<string, unknown> = {}) => {
|
|
const savedIssueId = issueId!;
|
|
const condition = deferredWorkIntentWait(refusal, `${message} Your message is saved.`, details);
|
|
await db.transaction(async tx => {
|
|
await tx.execute(sql`select id from issues where id = ${savedIssueId} and company_id = ${agent.companyId} for update`);
|
|
await notifyDeferredExecutionWork(tx, { companyId: agent.companyId });
|
|
await tx.insert(agentWakeupRequests).values({
|
|
companyId: agent.companyId, agentId, source, triggerDetail, reason,
|
|
payload: withQueuedCommentIdsInWakePayload({
|
|
...payload, issueId: savedIssueId, [DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot, executionWait: condition,
|
|
}, [...new Set([...queuedCommentIdsFromRunContext(enrichedContextSnapshot), wakeCommentId!])]),
|
|
status: "deferred_issue_execution",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
});
|
|
});
|
|
return new PersistedExecutionWaitRefusal(message, details);
|
|
};
|
|
|
|
const schedulingSuppression = await getSchedulingSuppression();
|
|
// A task drain holds ADMISSION, not the request. The drain is a
|
|
// process-local pre-restart hold, so a wake that arrives while it is
|
|
// active still names real work that must run once the process comes
|
|
// back: leave it in the durable queue and let the dispatch-side checks
|
|
// (startNextQueuedRunForAgent / executeRun) keep it from starting until
|
|
// the drain lifts or the restart clears it. Writing it as `skipped`
|
|
// here dropped the wake permanently — an accepted plan whose
|
|
// continuation wake landed mid-drain left its issue in `todo` with no
|
|
// run and no path until a person noticed.
|
|
if (
|
|
schedulingSuppression.suppressed &&
|
|
schedulingSuppression.reason !== "task_drain"
|
|
) {
|
|
await writeSkippedHeartbeatRequest("heartbeat.scheduling_suppressed", {
|
|
reason: schedulingSuppression.reason,
|
|
});
|
|
return null;
|
|
}
|
|
|
|
const worktreeExecutionCutoff =
|
|
opts.requestedByActorType === "user"
|
|
? null
|
|
: await getWorktreeExecutionCutoff();
|
|
|
|
const company = await db
|
|
.select({ status: companies.status })
|
|
.from(companies)
|
|
.where(eq(companies.id, agent.companyId))
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!company || company.status !== "active") {
|
|
if (caseRequest) return null;
|
|
const companyStatus = company?.status ?? "missing";
|
|
if (opts.requestedByActorType === "user") {
|
|
// A paused company is a temporary hold; an archived one is not.
|
|
if (companyStatus === "paused" && (await canSaveDeferredMessage()))
|
|
throw await saveDeferredMessage("company.inactive", "Company is not active", { status: companyStatus });
|
|
throw conflict("Company is not active", { status: companyStatus });
|
|
}
|
|
await writeSkippedRequest("company.inactive", {
|
|
error: `Wake suppressed because company status is ${companyStatus}`,
|
|
}, { companyStatus });
|
|
return null;
|
|
}
|
|
|
|
const explicitResumeSession = await resolveExplicitResumeSessionOverride(
|
|
agent,
|
|
payload,
|
|
taskKey,
|
|
);
|
|
if (explicitResumeSession) {
|
|
enrichedContextSnapshot.resumeFromRunId =
|
|
explicitResumeSession.resumeFromRunId;
|
|
enrichedContextSnapshot.resumeSessionDisplayId =
|
|
explicitResumeSession.sessionDisplayId;
|
|
enrichedContextSnapshot.resumeSessionParams =
|
|
explicitResumeSession.sessionParams;
|
|
if (
|
|
!readNonEmptyString(enrichedContextSnapshot.issueId) &&
|
|
explicitResumeSession.issueId
|
|
) {
|
|
enrichedContextSnapshot.issueId = explicitResumeSession.issueId;
|
|
}
|
|
if (
|
|
!readNonEmptyString(enrichedContextSnapshot.taskId) &&
|
|
explicitResumeSession.taskId
|
|
) {
|
|
enrichedContextSnapshot.taskId = explicitResumeSession.taskId;
|
|
}
|
|
if (
|
|
!readNonEmptyString(enrichedContextSnapshot.taskKey) &&
|
|
explicitResumeSession.taskKey
|
|
) {
|
|
enrichedContextSnapshot.taskKey = explicitResumeSession.taskKey;
|
|
}
|
|
issueId = readNonEmptyString(enrichedContextSnapshot.issueId) ?? issueId;
|
|
}
|
|
const effectiveTaskKey =
|
|
readNonEmptyString(enrichedContextSnapshot.taskKey) ?? taskKey;
|
|
const genericTimerWake =
|
|
source === "timer" &&
|
|
!issueId &&
|
|
!wakeCommentId &&
|
|
!readNonEmptyString(enrichedContextSnapshot.taskId) &&
|
|
!effectiveTaskKey;
|
|
// Old or direct timer producers cannot create a doomed taskless Runner
|
|
// run. Explicit session restoration above can supply a real task scope.
|
|
if (genericTimerWake && !supportsUnscopedTimerHeartbeat(agent)) return null;
|
|
const sessionBefore =
|
|
explicitResumeSession?.sessionDisplayId ??
|
|
(await resolveSessionBeforeForWakeup(agent, effectiveTaskKey));
|
|
let hasResolvablePriorSessionWorkspace: boolean | null = null;
|
|
const resolveHasResolvablePriorSessionWorkspace = async () => {
|
|
if (hasResolvablePriorSessionWorkspace !== null)
|
|
return hasResolvablePriorSessionWorkspace;
|
|
hasResolvablePriorSessionWorkspace = issueId
|
|
? await hasResolvablePriorSessionWorkspaceForWake({
|
|
agent,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
taskKey: effectiveTaskKey,
|
|
explicitResumeSession,
|
|
})
|
|
: false;
|
|
return hasResolvablePriorSessionWorkspace;
|
|
};
|
|
const continuationAttempt = readContinuationAttempt(
|
|
enrichedContextSnapshot.livenessContinuationAttempt,
|
|
);
|
|
|
|
let projectId = readNonEmptyString(enrichedContextSnapshot.projectId);
|
|
if (!projectId && issueId) {
|
|
// Look up by either UUID or identifier (e.g. "ENV-13"), but always scope
|
|
// by companyId so a row from another tenant can never be returned even
|
|
// when identifiers collide across companies. Guard the UUID arm because
|
|
// issues.id is a Postgres uuid column — passing "ENV-13" into eq(issues.id, …)
|
|
// would fail with an invalid-input-syntax cast error before the OR is
|
|
// evaluated.
|
|
const lookupIsUuid = isUuidLike(issueId);
|
|
const idMatch = lookupIsUuid
|
|
? or(
|
|
eq(issues.id, issueId),
|
|
eq(issues.identifier, issueId.toUpperCase()),
|
|
)
|
|
: eq(issues.identifier, issueId.toUpperCase());
|
|
const resolvedIssue = await db
|
|
.select({
|
|
id: issues.id,
|
|
projectId: issues.projectId,
|
|
createdAt: issues.createdAt,
|
|
})
|
|
.from(issues)
|
|
.where(and(eq(issues.companyId, agent.companyId), idMatch))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (resolvedIssue) {
|
|
if (
|
|
worktreeExecutionCutoff &&
|
|
resolvedIssue.createdAt < worktreeExecutionCutoff
|
|
) {
|
|
await writeSkippedHeartbeatRequest(
|
|
"heartbeat.worktree_execution_cutoff",
|
|
{
|
|
reason: "worktree_execution_cutoff",
|
|
cutoff: worktreeExecutionCutoff.toISOString(),
|
|
issueId: resolvedIssue.id,
|
|
},
|
|
);
|
|
return null;
|
|
}
|
|
projectId = resolvedIssue.projectId ?? null;
|
|
// Canonicalize context to the UUID so downstream lookups always use UUID
|
|
if (resolvedIssue.id !== issueId) {
|
|
issueId = resolvedIssue.id;
|
|
enrichedContextSnapshot.issueId = issueId;
|
|
if (readNonEmptyString(enrichedContextSnapshot.taskId)) {
|
|
enrichedContextSnapshot.taskId = issueId;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
// Propagate projectId into context so resolveWorkspaceForRun can bind the
|
|
// project workspace even when context.projectId wasn't set by the caller.
|
|
if (projectId && !readNonEmptyString(enrichedContextSnapshot.projectId)) {
|
|
enrichedContextSnapshot.projectId = projectId;
|
|
}
|
|
const isolatedWorkspacesEnabled = issueId
|
|
? (await instanceSettings.getExperimental()).enableIsolatedWorkspaces
|
|
: false;
|
|
let operatorResponsibleUserId: string | null = opts.manualUserWake ? opts.requestedByActorId! : null;
|
|
let queuedResponsibleUserIdPromise: Promise<string> | null = null;
|
|
const resolveQueuedResponsibleUserId = () => {
|
|
if (operatorResponsibleUserId) return Promise.resolve(operatorResponsibleUserId);
|
|
queuedResponsibleUserIdPromise ??= (async () => {
|
|
const queuedIssueContext = issueId
|
|
? await getIssueExecutionContext(agent.companyId, issueId)
|
|
: null;
|
|
const queuedRoutineEnvContext = await getRoutineEnvForExecutionIssue(
|
|
agent.companyId,
|
|
queuedIssueContext,
|
|
);
|
|
const queuedResponsibleUserId =
|
|
await resolveResponsibleUserIdForRunSeed({
|
|
companyId: agent.companyId,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
issueContext: queuedIssueContext,
|
|
routineEnvContext: queuedRoutineEnvContext,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
source,
|
|
triggerDetail,
|
|
});
|
|
if (!queuedResponsibleUserId) {
|
|
throw new HttpError(
|
|
422,
|
|
"Unable to resolve responsible user for heartbeat run dispatch",
|
|
{
|
|
code: "responsible_user_unresolved",
|
|
agentId,
|
|
companyId: agent.companyId,
|
|
issueId: issueId ?? null,
|
|
source,
|
|
triggerDetail,
|
|
wakeReason: readNonEmptyString(
|
|
enrichedContextSnapshot.wakeReason,
|
|
),
|
|
},
|
|
);
|
|
}
|
|
return queuedResponsibleUserId;
|
|
})();
|
|
return queuedResponsibleUserIdPromise;
|
|
};
|
|
|
|
const budgetBlock = await budgets.getInvocationBlock(
|
|
agent.companyId,
|
|
agentId,
|
|
{
|
|
issueId,
|
|
projectId,
|
|
},
|
|
);
|
|
if (budgetBlock) {
|
|
// A budget block that is not a reached hard stop is an explicit pause.
|
|
if ((await canSaveDeferredMessage()) && !(await isBoardOwnedBudgetBlock(db, agent.companyId, budgetBlock)))
|
|
throw await saveDeferredMessage("budget.blocked", budgetBlock.reason,
|
|
{ scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId });
|
|
const wait = await writeSkippedRequest("budget.blocked", { error: budgetBlock.reason }, {
|
|
scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId,
|
|
});
|
|
if (caseRequest) return null;
|
|
const details = { scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId };
|
|
throw wait.waitSaved ? new PersistedExecutionWaitRefusal(budgetBlock.reason, details)
|
|
: conflict(budgetBlock.reason, details);
|
|
}
|
|
|
|
// Setup delays execution, but must not discard work accepted by a caller.
|
|
const invokability = await getAgentInvokability(isAgentAwaitingSetup(agent) ? { ...agent, status: "idle" } : agent);
|
|
if (!invokability.invokable) {
|
|
if (caseRequest) return null;
|
|
// Only a pause defers through the release drain (wake-queue policy); a
|
|
// pending approval still fails a promoted wake, so it is not saved here.
|
|
if (invokability.reason === "paused" && (await canSaveDeferredMessage()))
|
|
throw await saveDeferredMessage("agent.not_invokable", invokability.message,
|
|
{ status: agent.status, reason: invokability.reason });
|
|
let waitSaved = false;
|
|
if (opts.requestedByActorType !== "user" || executionWaitRequestId) {
|
|
const wait = await writeSkippedRequest("agent.not_invokable", {
|
|
error: invokability.message,
|
|
}, { status: agent.status, reason: invokability.reason });
|
|
waitSaved = wait.waitSaved === true;
|
|
}
|
|
const details = { status: agent.status, reason: invokability.reason,
|
|
invalidOrgChain: invokability.invalidOrgChain, ...invokability.details };
|
|
throw waitSaved ? new PersistedExecutionWaitRefusal(invokability.message, details)
|
|
: conflict(invokability.message, details);
|
|
}
|
|
|
|
const policy = parseHeartbeatPolicy(agent);
|
|
|
|
if (source === "timer" && !policy.enabled) {
|
|
await writeSkippedRequest("heartbeat.disabled", {}, { enabled: false });
|
|
return null;
|
|
}
|
|
if (source !== "timer" && !policy.wakeOnDemand) {
|
|
await writeSkippedRequest("heartbeat.wakeOnDemand.disabled", {}, { wakeOnDemand: false });
|
|
return null;
|
|
}
|
|
|
|
if (
|
|
policy.skipTimerWhenNoActionableWork &&
|
|
genericTimerWake &&
|
|
!(await hasActionableTimerWork(agent))
|
|
) {
|
|
await writeSkippedHeartbeatRequest("heartbeat.timer.no_actionable_work", {
|
|
reason:
|
|
"No assigned todo or in_progress issue requires this agent before timer adapter invocation.",
|
|
});
|
|
await markTimerHeartbeatChecked(agentId, source);
|
|
return null;
|
|
}
|
|
|
|
if (issueId) {
|
|
const activePauseHold = await treeControlSvc.getActivePauseHoldGate(
|
|
agent.companyId,
|
|
issueId,
|
|
);
|
|
if (activePauseHold) {
|
|
const treeHoldInteractionWake =
|
|
await isVerifiedIssueTreeControlInteractionWake(db, {
|
|
companyId: agent.companyId,
|
|
issueId,
|
|
agentId,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
requestedByActorType: opts.requestedByActorType,
|
|
requestedByActorId: opts.requestedByActorId,
|
|
});
|
|
|
|
if (!treeHoldInteractionWake) {
|
|
const wait = await writeSkippedRequest("issue_tree_hold_active", {}, {
|
|
holdId: activePauseHold.holdId,
|
|
});
|
|
if (wait.created) await logActivity(db, {
|
|
companyId: agent.companyId,
|
|
actorType: "system",
|
|
actorId: "system",
|
|
agentId,
|
|
runId: null,
|
|
action: "issue.tree_hold_wakeup_deferred",
|
|
entityType: "issue",
|
|
entityId: issueId,
|
|
details: {
|
|
holdId: activePauseHold.holdId,
|
|
rootIssueId: activePauseHold.rootIssueId,
|
|
requestedReason: reason,
|
|
source,
|
|
triggerDetail,
|
|
securityPrinciples: [
|
|
"Complete Mediation",
|
|
"Fail Securely",
|
|
"Secure Defaults",
|
|
],
|
|
},
|
|
});
|
|
return null;
|
|
}
|
|
|
|
enrichedContextSnapshot.treeHoldInteraction = true;
|
|
enrichedContextSnapshot.activeTreeHold = {
|
|
holdId: activePauseHold.holdId,
|
|
rootIssueId: activePauseHold.rootIssueId,
|
|
mode: activePauseHold.mode,
|
|
reason: activePauseHold.reason,
|
|
releasePolicy: activePauseHold.releasePolicy,
|
|
interaction: true,
|
|
};
|
|
}
|
|
}
|
|
|
|
if (issueId) {
|
|
// Mention-triggered wakes can request input from another agent, but they must
|
|
// still respect the issue execution lock so a second agent cannot start on the
|
|
// same issue workspace while the assignee already has a live run.
|
|
const agentNameKey = normalizeAgentNameKey(agent.name);
|
|
|
|
const cancelledRunsToEmit: (typeof heartbeatRuns.$inferSelect)[] = [];
|
|
const issueActivityPublications: ActivityPublication[] = [];
|
|
const issuePostCommitActions: IssuePostCommitAction[] = [];
|
|
|
|
const outcome = await db
|
|
.transaction(withDurableChatAdmission(durableRequest, async (tx) => {
|
|
await tx.execute(
|
|
sql`select id from issues where id = ${issueId} and company_id = ${agent.companyId} for update`,
|
|
);
|
|
|
|
if (executionWaitRequestId) {
|
|
const [pending] = await tx.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, executionWaitRequestId), eq(agentWakeupRequests.companyId, agent.companyId),
|
|
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
// A user message can join a queue originally created by a
|
|
// system wake. Admission validates the saved user comment or board click.
|
|
(opts.queuedCommentInterruptId ?? opts.queuedCommentRequestId) === executionWaitRequestId
|
|
? undefined : eq(agentWakeupRequests.requestedByActorType,
|
|
durableRequest ? opts.requestedByActorType ?? "" : "user"),
|
|
opts.queuedCommentInterruptId === executionWaitRequestId
|
|
? sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt'->>'actorId' = ${opts.requestedByActorId ?? ""}`
|
|
: opts.queuedCommentRequestId === executionWaitRequestId ? undefined
|
|
: eq(agentWakeupRequests.requestedByActorId, opts.requestedByActorId ?? ""),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
|
|
));
|
|
// The issue lock serializes cleanup callbacks and periodic workers.
|
|
// An adopted, discarded, or edited receipt is no longer authority.
|
|
if (!pending || (!opts.taskRetryRequestId && !(wakeCommentId && queuedCommentIdsFromWakePayload(pending.payload).includes(wakeCommentId)) &&
|
|
!(opts.queuedCommentInterruptId && await readQueuedInteractionResponse(tx as unknown as Db,
|
|
agent.companyId, issueId, pending.payload)))) {
|
|
return { kind: "deferred" as const };
|
|
}
|
|
if (opts.taskRetryRequestId && durableRequest?.failedRunRetry) {
|
|
if (pending.id !== durableRequest.id || pending.payload?.taskRetryRequestId !== opts.taskRetryRequestId)
|
|
return { kind: "deferred" as const };
|
|
// The parent board Retry is checked separately below after the
|
|
// connected conversation reauthorizes this exact child receipt.
|
|
} else if (opts.taskRetryRequestId) {
|
|
const retry = readTaskRetryIntent(pending.payload);
|
|
if (pending.id !== opts.taskRetryRequestId || retry?.generationId !== pending.id || retry.chatActionId ||
|
|
retry.actorId !== opts.requestedByActorId || retry.sourceRunId !== enrichedContextSnapshot.previousRunId)
|
|
return { kind: "deferred" as const };
|
|
// A declined Retry stays saved, but never silently: name the gate
|
|
// so the task explains it and a new click can replace this intent.
|
|
const declineRetry = async (reason: string, message: string) => {
|
|
await writeDeferredExecutionWait(tx as unknown as Db, { companyId: agent.companyId,
|
|
requestId: pending.id }, { reason, message }, { replace: true, publications: issueActivityPublications });
|
|
logger.warn({ requestId: pending.id, issueId, reason }, "Task Retry could not start");
|
|
return { kind: "deferred" as const };
|
|
};
|
|
if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: agent.companyId, issueId,
|
|
actorId: retry.actorId, actorSource: retry.actorSource }))
|
|
return declineRetry("retry_authority_denied", "The user who requested Retry can no longer change this task.");
|
|
if (retry.boardMessageOrigin) {
|
|
const origin = await readTaskRetryBoardMessageOrigin(tx as unknown as Db, { companyId: agent.companyId,
|
|
issueId, agentId, sourceRunId: retry.sourceRunId, actorId: retry.actorId });
|
|
if (!origin || origin.commentId !== retry.boardMessageOrigin.commentId ||
|
|
origin.identityContextId !== retry.boardMessageOrigin.identityContextId)
|
|
return declineRetry("retry_source_changed", "The saved message this Retry continues has changed.");
|
|
}
|
|
}
|
|
if (opts.queuedCommentRequestId) {
|
|
const ids = await undeliveredLegacyUserCommentIds(tx as unknown as Db,
|
|
agent.companyId, issueId, agentId, queuedCommentIdsFromWakePayload(pending.payload));
|
|
if (!wakeCommentId || !ids.includes(wakeCommentId)) return { kind: "deferred" as const };
|
|
pending.payload = withQueuedCommentIdsInWakePayload(parseObject(pending.payload), ids);
|
|
await tx.update(agentWakeupRequests).set({ payload: pending.payload }).where(and(
|
|
eq(agentWakeupRequests.id, pending.id), eq(agentWakeupRequests.companyId, agent.companyId),
|
|
));
|
|
}
|
|
if (!opts.queuedCommentInterruptId && !opts.queuedCommentRequestId && pending.payload?.manualUserWake === true) {
|
|
// A persisted manual wake keeps its actor when an execution wait
|
|
// resumes. The locked receipt above has revalidated that actor.
|
|
payload = { ...payload, manualUserWake: true };
|
|
operatorResponsibleUserId = opts.requestedByActorId!;
|
|
}
|
|
if (opts.queuedCommentInterruptId) {
|
|
// The locked board receipt supplies execution authority even when
|
|
// another user authored the messages. Dispatch revalidates the receipt.
|
|
operatorResponsibleUserId = opts.requestedByActorId!;
|
|
}
|
|
if (opts.queuedCommentInterruptId || opts.queuedCommentRequestId) {
|
|
// Edits/discards between the click and dispatch remain authoritative.
|
|
Object.assign(enrichedContextSnapshot, withQueuedCommentIdsInRunContext(
|
|
enrichedContextSnapshot, queuedCommentIdsFromWakePayload(pending.payload),
|
|
));
|
|
}
|
|
}
|
|
let automaticParentRunId: string | null = null;
|
|
if (
|
|
source === "automation" &&
|
|
opts.requestedByActorType === "system"
|
|
) {
|
|
const nativeParent = await readChatControlNativeParent(
|
|
tx as unknown as Db,
|
|
{ companyId: agent.companyId, issueId, agentId },
|
|
opts.requestedByActorId ?? null,
|
|
);
|
|
const genericParent =
|
|
!opts.requestedByActorId && reason === "issue_continuation_needed"
|
|
? readNonEmptyString(enrichedContextSnapshot.retryOfRunId)
|
|
: null;
|
|
automaticParentRunId =
|
|
nativeParent.kind === "parent"
|
|
? nativeParent.runId
|
|
: genericParent;
|
|
if (automaticParentRunId) {
|
|
const [parent] = await tx
|
|
.select({
|
|
agentId: heartbeatRuns.agentId,
|
|
context: heartbeatRuns.contextSnapshot,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, agent.companyId),
|
|
eq(heartbeatRuns.id, automaticParentRunId),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (
|
|
parent &&
|
|
(parseObject(parent.context).issueId ??
|
|
parseObject(parent.context).taskId) === issueId &&
|
|
parent.agentId !== agentId
|
|
)
|
|
automaticParentRunId = null;
|
|
}
|
|
const proof =
|
|
nativeParent.kind === "unresolved"
|
|
? { kind: "unresolved" as const }
|
|
: automaticParentRunId
|
|
? await readChatControlRecoveryStop(
|
|
tx as unknown as Db,
|
|
{
|
|
companyId: agent.companyId,
|
|
issueId,
|
|
agentId,
|
|
sourceRunId: automaticParentRunId,
|
|
},
|
|
true,
|
|
)
|
|
: { kind: "clear" as const };
|
|
if (proof.kind !== "clear") {
|
|
await tx
|
|
.insert(agentWakeupRequests)
|
|
.values({
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason:
|
|
proof.kind === "stopped"
|
|
? CHAT_CONTROL_RECOVERY_STOP_CODE
|
|
: CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE,
|
|
payload: { issueId },
|
|
status: "skipped",
|
|
requestedByActorType: "system",
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
}
|
|
|
|
const durableReceipt = await existingDurableReceipt(
|
|
tx as unknown as Db,
|
|
);
|
|
if (durableReceipt)
|
|
return { kind: "durable" as const, receipt: durableReceipt };
|
|
if (caseRequest) {
|
|
const [handoff] = await tx.select().from(caseHandoffs).where(and(eq(caseHandoffs.id, caseRequest.handoffId),
|
|
eq(caseHandoffs.companyId, agent.companyId),
|
|
eq(caseHandoffs.issueId, caseRequest.issueId), eq(caseHandoffs.state, "pending_dispatch"),
|
|
eq(caseHandoffs.toAgentId, agentId), eq(caseHandoffs.wakeupRequestId, caseRequest.wakeupRequestId))).for("update");
|
|
if (!handoff || handoff.appliedIssueStatusVersion !== caseRequest.expectedIssueStatusVersion)
|
|
throw conflict("Case handoff delivery was superseded");
|
|
}
|
|
const failedChatRetry = await authorizeFailedChatRunRetryWake(
|
|
db,
|
|
tx as unknown as Db,
|
|
{
|
|
phase: "admission",
|
|
wakeupRequestId: durableRequest?.id ?? null,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
issueId,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
},
|
|
);
|
|
if (
|
|
failedChatRetry !== Boolean(durableRequest?.failedRunRetry) ||
|
|
(failedChatRetry &&
|
|
durableRequest?.failedRunRetry?.failedRunId !==
|
|
readNonEmptyString(enrichedContextSnapshot.retryOfRunId))
|
|
) {
|
|
throw new FailedChatRunRetryAuthorizationError();
|
|
}
|
|
if (durableRequest) {
|
|
if (issueId !== durableRequest.issueId)
|
|
throw new Error("chat_inbound_wakeup_binding_denied");
|
|
await durableRequest.authorize(tx as unknown as Db);
|
|
if (opts.taskRetryRequestId) {
|
|
const [retryWake] = await tx.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, agent.companyId), eq(agentWakeupRequests.id, opts.taskRetryRequestId),
|
|
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
)).for("update");
|
|
const retry = readTaskRetryIntent(retryWake?.payload);
|
|
if (!failedChatRetry || retry?.chatActionId !== durableRequest.id ||
|
|
retry.actorId !== taskRetryActorId || retry.sourceRunId !== durableRequest.failedRunRetry?.failedRunId)
|
|
throw new FailedChatRunRetryAuthorizationError();
|
|
if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: agent.companyId, issueId,
|
|
actorId: retry.actorId, actorSource: retry.actorSource })) return { kind: "deferred" as const };
|
|
}
|
|
} else {
|
|
if (
|
|
source === "on_demand" &&
|
|
triggerDetail === "manual" &&
|
|
reason === "retry_failed_run"
|
|
) {
|
|
// Generic Board retry does not carry server-authorized failed-run
|
|
// lineage. Caller-supplied source/comment/retry markers cannot
|
|
// restore it, including for a retired conversation generation.
|
|
const [chatBinding] = await tx
|
|
.select({ id: chatConversations.id })
|
|
.from(chatConversations)
|
|
.innerJoin(chatEndpoints, eq(chatEndpoints.id, chatConversations.endpointId))
|
|
.where(
|
|
and(
|
|
eq(chatEndpoints.externalExecutionPolicy, "restricted"),
|
|
eq(chatConversations.companyId, agent.companyId),
|
|
eq(chatConversations.issueId, issueId),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (chatBinding) {
|
|
throw conflict(
|
|
"Retry needs the exact failed chat request and current access. Send the request again in the current connected conversation.",
|
|
{ code: "chat_failed_run_retry_requires_authorized_context" },
|
|
);
|
|
}
|
|
}
|
|
const [held] = await tx
|
|
.select({ id: issues.id })
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, agent.companyId),
|
|
unadmittedChatWakeupCondition(issues.id, issues.companyId),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (held) {
|
|
throw conflict(
|
|
"This task has external chat input that has not been admitted. Check the connection's Activity; let pending input finish, or restore access and send a new authorized message before starting the task.",
|
|
{ code: "chat_inbound_wakeup_unadmitted", issueId },
|
|
);
|
|
}
|
|
}
|
|
|
|
const issue = await tx
|
|
.select({
|
|
id: issues.id,
|
|
companyId: issues.companyId,
|
|
identifier: issues.identifier,
|
|
conversationAgentId: issues.conversationAgentId,
|
|
conversationUserId: issues.conversationUserId,
|
|
conversationState: issues.conversationState,
|
|
status: issues.status,
|
|
statusVersion: issues.statusVersion,
|
|
projectId: issues.projectId,
|
|
projectWorkspaceId: issues.projectWorkspaceId,
|
|
executionWorkspaceId: issues.executionWorkspaceId,
|
|
executionWorkspacePreference: issues.executionWorkspacePreference,
|
|
executionWorkspaceSettings: issues.executionWorkspaceSettings,
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
assigneeUserId: issues.assigneeUserId,
|
|
monitorNextCheckAt: issues.monitorNextCheckAt,
|
|
monitorWakeRequestedAt: issues.monitorWakeRequestedAt,
|
|
executionRunId: issues.executionRunId,
|
|
executionAgentNameKey: issues.executionAgentNameKey,
|
|
createdAt: issues.createdAt,
|
|
})
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, agent.companyId),
|
|
),
|
|
)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!issue) {
|
|
if (caseRequest) throw conflict("Case handoff task no longer exists");
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "issue_execution_issue_not_found",
|
|
payload,
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
// Completion can settle the conversation while admission is outside
|
|
// the issue lock. Recheck the current state before persisting a wake.
|
|
if (isWaitingConversation(issue) && !hasIndependentConversationWakeCause()) {
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
if (taskRetryActorId) {
|
|
await assertTaskRetryConversationAllowed(tx as unknown as Db, issue, taskRetryActorId);
|
|
if (await retireSupersededTaskRetry(tx as unknown as Db, {
|
|
companyId: issue.companyId, issueId: issue.id, requestId: opts.taskRetryRequestId!,
|
|
sourceRunId: readNonEmptyString(enrichedContextSnapshot.previousRunId)!,
|
|
})) return { kind: "deferred" as const };
|
|
}
|
|
const issueStateGuard = opts.issueStateGuard;
|
|
const activeMonitorRun = issueStateGuard?.monitorNextCheckAt === undefined ? null
|
|
: await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.nativeIssueId, issue.id),
|
|
eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, ["queued", "running", "scheduled_retry"]),
|
|
)).limit(1).then(rows => rows[0] ?? null);
|
|
if (
|
|
issueStateGuard &&
|
|
(!issueStateGuard.statuses.includes(issue.status) ||
|
|
issue.assigneeAgentId !== issueStateGuard.assigneeAgentId ||
|
|
(issueStateGuard.statusVersion !== undefined && issue.statusVersion !== issueStateGuard.statusVersion) ||
|
|
(issueStateGuard.monitorNextCheckAt !== undefined && (
|
|
activeMonitorRun !== null || issue.assigneeUserId !== null ||
|
|
issue.monitorNextCheckAt?.toISOString() !== issueStateGuard.monitorNextCheckAt ||
|
|
issue.monitorWakeRequestedAt?.toISOString() !== issueStateGuard.monitorWakeRequestedAt
|
|
)))
|
|
) {
|
|
if (caseRequest) throw conflict("Case handoff task changed before admission");
|
|
// A deferred monitor retains its schedule; do not create a receipt
|
|
// that could suppress its next admission attempt.
|
|
if (issueStateGuard.monitorNextCheckAt !== undefined) return { kind: "skipped" as const };
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "issue_state_guard_mismatch",
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: {
|
|
reason:
|
|
"Issue status or assignee changed before the wake could be queued.",
|
|
issueId: issue.id,
|
|
expectedStatuses: issueStateGuard.statuses,
|
|
actualStatus: issue.status,
|
|
expectedAssigneeAgentId: issueStateGuard.assigneeAgentId,
|
|
actualAssigneeAgentId: issue.assigneeAgentId,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
if (opts.failedRunId) {
|
|
// The issue lock makes double-clicks and network retries adopt the
|
|
// same successor, including after it has already finished.
|
|
const [previousRetry] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.agentId, agentId),
|
|
eq(heartbeatRuns.retryOfRunId, opts.failedRunId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'retry_failed_run'`,
|
|
)).orderBy(desc(heartbeatRuns.createdAt)).limit(1);
|
|
if (previousRetry) return { kind: "replayed" as const, run: previousRetry };
|
|
}
|
|
|
|
let reconciledSourceRunId: string | null = null;
|
|
let reconciledRestoreRetryCount: number | null = null;
|
|
if (executionReconciliationWake) {
|
|
const actionId = readNonEmptyString(
|
|
enrichedContextSnapshot.recoveryActionId,
|
|
);
|
|
if (
|
|
!actionId ||
|
|
!isUuidLike(actionId) ||
|
|
source !== "automation" ||
|
|
triggerDetail !== "system" ||
|
|
reason !== "issue_recovery_action_restored" ||
|
|
opts.requestedByActorType !== "system" ||
|
|
opts.requestedByActorId !== "execution-recovery" ||
|
|
opts.idempotencyKey !== `execution-reconciliation:${actionId}` ||
|
|
enrichedContextSnapshot.source !== "execution.reconciled" ||
|
|
enrichedContextSnapshot.forceFreshSession !== true ||
|
|
payload?.issueId !== issue.id ||
|
|
payload?.recoveryActionId !== actionId ||
|
|
issue.assigneeAgentId !== agentId ||
|
|
["done", "cancelled"].includes(issue.status)
|
|
)
|
|
return { kind: "skipped" as const };
|
|
|
|
// The issue lock serializes all admissions for this source. Validate
|
|
// the durable operator decision, then reconcile a prior queue commit
|
|
// before considering a new wake (including a now-terminal successor).
|
|
const [action] = await tx
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(
|
|
and(
|
|
eq(issueRecoveryActions.companyId, issue.companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, issue.id),
|
|
eq(issueRecoveryActions.id, actionId),
|
|
),
|
|
)
|
|
.for("update");
|
|
const decision = parseObject(
|
|
action?.evidence.executionReconciliation,
|
|
);
|
|
const sourceRunId = readNonEmptyString(decision.runId);
|
|
if (
|
|
!action ||
|
|
action.status !== "resolved" ||
|
|
action.kind !== "active_run_watchdog" ||
|
|
action.returnOwnerAgentId !== agentId ||
|
|
!sourceRunId ||
|
|
!isUuidLike(sourceRunId) ||
|
|
decision.providerStopped !== true ||
|
|
!["completed", "not_performed", "mixed"].includes(
|
|
String(decision.actionOutcome),
|
|
) ||
|
|
!readNonEmptyString(decision.outcomeEvidence) ||
|
|
enrichedContextSnapshot.previousRunId !== sourceRunId ||
|
|
enrichedContextSnapshot.retryOfRunId !== sourceRunId ||
|
|
!["pending", "delivered"].includes(
|
|
String(action.evidence.continuationDelivery),
|
|
)
|
|
)
|
|
return { kind: "skipped" as const };
|
|
|
|
const [existingWake] = await tx
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, issue.companyId),
|
|
eq(agentWakeupRequests.agentId, agentId),
|
|
eq(agentWakeupRequests.idempotencyKey, opts.idempotencyKey),
|
|
ne(agentWakeupRequests.status, "skipped"),
|
|
),
|
|
)
|
|
.orderBy(asc(agentWakeupRequests.requestedAt))
|
|
.limit(1);
|
|
if (existingWake) {
|
|
if (
|
|
existingWake.payload?.issueId !== issue.id ||
|
|
existingWake.payload?.recoveryActionId !== action.id ||
|
|
existingWake.requestedByActorType !== "system" ||
|
|
existingWake.requestedByActorId !== "execution-recovery" ||
|
|
!existingWake.runId
|
|
)
|
|
return { kind: "deferred" as const };
|
|
const [existingRun] = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, issue.companyId),
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
eq(heartbeatRuns.id, existingWake.runId),
|
|
),
|
|
);
|
|
if (
|
|
!existingRun ||
|
|
existingRun.contextSnapshot?.issueId !== issue.id ||
|
|
existingRun.contextSnapshot?.recoveryActionId !== action.id ||
|
|
existingRun.contextSnapshot?.previousRunId !== sourceRunId
|
|
)
|
|
return { kind: "deferred" as const };
|
|
return { kind: "replayed" as const, run: existingRun };
|
|
}
|
|
if (action.evidence.continuationDelivery !== "pending")
|
|
return { kind: "skipped" as const };
|
|
// Under the task lock: a fresh continuation is the task's next run
|
|
// only while nothing else continued it. A run started since it was
|
|
// recorded (active or already finished, from a person's Retry or
|
|
// saved input) consumes it, and a task that moved to review gets none.
|
|
const freshRecordedAt = readNonEmptyString(parseObject(action.evidence.freshContinuation).recordedAt);
|
|
if (freshRecordedAt) {
|
|
const [intervening] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, issue.companyId), sql`${heartbeatRuns.id}::text <> ${sourceRunId}`,
|
|
or(eq(heartbeatRuns.nativeIssueId, issue.id), eq(heartbeatRuns.issueId, issue.id),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}::text`),
|
|
sql`${heartbeatRuns.createdAt} >= ${freshRecordedAt}::timestamptz`)).limit(1);
|
|
if (intervening || issue.status === "in_review") {
|
|
await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify(
|
|
intervening ? { continuationDelivery: "delivered", continuationRunId: intervening.id, continuationSupersededByRun: true }
|
|
: { continuationDelivery: "invalidated", continuationInvalidatedReason: "in_review" })}::jsonb` })
|
|
.where(and(eq(issueRecoveryActions.companyId, issue.companyId), eq(issueRecoveryActions.id, action.id)));
|
|
await settleCoalescedFreshContinuationHolds(tx as unknown as Db, { companyId: issue.companyId, primaryId: action.id,
|
|
delivery: intervening ? "delivered" : "invalidated", continuationRunId: intervening?.id ?? null });
|
|
return { kind: "skipped" as const };
|
|
}
|
|
}
|
|
const [reconciledRun] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.id, sourceRunId),
|
|
));
|
|
if (decision.workspaceReconstruction) {
|
|
await validateNativeWorkspaceReconstruction(tx as unknown as Db, {
|
|
companyId: issue.companyId, issueId: issue.id, runId: sourceRunId, actionId,
|
|
});
|
|
reconciledRestoreRetryCount = executionFailureRetryCount(reconciledRun!);
|
|
enrichedContextSnapshot.workspaceReconstructionActionId = action.id;
|
|
}
|
|
if (hasWorkspaceRestoreFailure(reconciledRun?.resultJson)) {
|
|
if ((readNonEmptyString(decision.workspaceRepairEvidence)?.length ?? 0) < 20)
|
|
return { kind: "skipped" as const };
|
|
// Repair does not reset the remaining automatic retry budget.
|
|
reconciledRestoreRetryCount = executionFailureRetryCount(reconciledRun!);
|
|
}
|
|
reconciledSourceRunId = sourceRunId;
|
|
}
|
|
|
|
let continuationWait = { reason: "execution_recovery", message: "Waiting for execution recovery. Your message is saved." };
|
|
let continuationBlocked = false;
|
|
const deferBlockedExecution = async (
|
|
executionBlocker: Pick<NonNullable<Awaited<ReturnType<typeof getExecutionBlocker>>>, "recoveryActionId" | "cause" | "nextAction">,
|
|
) => {
|
|
if (caseRequest) waitForCaseAdmission("execution_reconciliation_required", executionBlocker.nextAction);
|
|
opts.issueStateGuard?.onExecutionDeferred?.(executionBlocker.cause);
|
|
const condition = { recoveryActionId: executionBlocker.recoveryActionId, ...continuationWait };
|
|
if (executionWaitRequestId) {
|
|
await writeDeferredExecutionWait(tx as unknown as Db, { companyId: agent.companyId,
|
|
requestId: executionWaitRequestId }, condition, { replace: true, publications: issueActivityPublications });
|
|
return { kind: "deferred" as const, admitted: true };
|
|
}
|
|
if (durableRequest || wakeCommentId ||
|
|
hasInteractionContinuationWakeContext(enrichedContextSnapshot) ||
|
|
readNonEmptyString(enrichedContextSnapshot.nativeStatusWakeIntentId)) {
|
|
await notifyDeferredExecutionWork(tx, { companyId: agent.companyId });
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId, agentId, source, triggerDetail, reason,
|
|
payload: withQueuedCommentIdsInWakePayload({
|
|
...payload,
|
|
issueId: issue.id,
|
|
[DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot,
|
|
executionWait: condition,
|
|
}, [...new Set([
|
|
...queuedCommentIdsFromRunContext(enrichedContextSnapshot),
|
|
...(wakeCommentId ? [wakeCommentId] : []),
|
|
])]),
|
|
status: "deferred_issue_execution",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
});
|
|
} else {
|
|
await recordExecutionWait(tx as unknown as Db, {
|
|
issueId: issue.id, condition, coalesce: coalesceExecutionWait,
|
|
request: {
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId, agentId, source, triggerDetail,
|
|
reason: "execution_reconciliation_required",
|
|
error: executionBlocker.nextAction,
|
|
payload,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
},
|
|
});
|
|
}
|
|
return { kind: "deferred" as const, admitted: true };
|
|
};
|
|
const explicitContinuationRunId = randomUUID();
|
|
const [resetComment] = isConversation(issue) && opts.requestedByActorType === "user" && opts.requestedByActorId && wakeCommentId
|
|
? await tx.select({ body: issueComments.body }).from(issueComments).where(and(
|
|
eq(issueComments.companyId, issue.companyId), eq(issueComments.issueId, issue.id),
|
|
eq(issueComments.id, wakeCommentId), eq(issueComments.authorUserId, opts.requestedByActorId),
|
|
eq(issueComments.authorType, "user"), isNull(issueComments.deletedAt), isNull(issueComments.createdByRunId),
|
|
)).limit(1)
|
|
: [];
|
|
const verifiedConversationReset = Boolean(resetComment && isConversationReset(resetComment.body));
|
|
if (wakeCommentId && opts.requestedByActorType === "user")
|
|
await preserveTaskRemoteWorkspaceWriter(tx as unknown as Db, issue.companyId, issue.id);
|
|
const executionBlocker = await getExecutionBlocker(
|
|
tx as unknown as Db, issue.companyId, issue.id,
|
|
{ conversationResetCommentId: verifiedConversationReset ? wakeCommentId : null },
|
|
);
|
|
// Prove eligibility without retiring the hold. Later gates can still
|
|
// decline this wake; hold retirement and successor creation stay atomic.
|
|
// A bound chat request has already rechecked its current principal
|
|
// above. Treat its new user message like a board comment, but keep
|
|
// failed-run retry actions on their separate exact-request path.
|
|
const ownsReconstructionContinuation = reconciledSourceRunId && executionBlocker?.runId === reconciledSourceRunId
|
|
&& executionBlocker.recoveryActionId === enrichedContextSnapshot.recoveryActionId;
|
|
const continuationHold = () => executionBlocker ?? {
|
|
recoveryActionId: null, cause: "native_continuation_requires_reconciliation" as const, nextAction: continuationWait.message,
|
|
};
|
|
// Only system continuation wakes (sweeps, monitors, timers) reach
|
|
// the fresh-continuation fallback. A person's Retry, message or
|
|
// queued send keeps its own exact admission; the disposition worker
|
|
// settles the hold through the same fallback, and the saved input
|
|
// then continues the task.
|
|
const systemWake = opts.requestedByActorType !== "user" && !taskRetryActorId && !wakeCommentId
|
|
&& !durableRequest && !opts.queuedCommentInterruptId && !opts.queuedCommentRequestId && !opts.failedRunId
|
|
&& !executionReconciliationWake;
|
|
// A recorded fresh continuation that has no run yet is this task's
|
|
// next run: another system wake is coalesced into it, never a second run.
|
|
if (systemWake && !executionBlocker && await hasPendingFreshContinuation(tx as unknown as Db, issue.companyId, issue.id)) {
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields, companyId: agent.companyId, agentId, source, triggerDetail,
|
|
reason: "issue_continuation_pending", payload, status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null, requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null, finishedAt: new Date(),
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
const continuationToDeliver = !systemWake && !executionReconciliationWake && !executionBlocker
|
|
? await undeliveredContinuation(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id }) : null;
|
|
if (continuationToDeliver)
|
|
Object.assign(enrichedContextSnapshot, undeliveredContinuationContext(continuationToDeliver, enrichedContextSnapshot));
|
|
if ((executionBlocker || opts.queuedCommentInterruptId) && !ownsReconstructionContinuation && !(await admitExplicitNativeContinuation({
|
|
db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id,
|
|
agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId,
|
|
reason: taskRetryActorId ? "task_retry" : durableRequest && !failedChatRetry ? "issue_commented" : reason,
|
|
commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId,
|
|
queuedCommentInterruptId: opts.queuedCommentInterruptId,
|
|
queuedCommentRequestId: opts.queuedCommentRequestId,
|
|
taskRetryRequestId: opts.taskRetryRequestId,
|
|
...(durableRequest?.resumeDeferred && !failedChatRetry ? { durableChatRequest: durableRequest } : {}),
|
|
interactionPayload: payload,
|
|
dryRun: true,
|
|
onBlocked: (reason, message) => { continuationBlocked = true; continuationWait = { reason, message }; },
|
|
})) && (executionBlocker || (continuationBlocked && !verifiedConversationReset))) {
|
|
// A hold left by an unfinished run never refuses an ordinary
|
|
// continuation: unless its cause is the person's, the fallback
|
|
// settles it in this transaction and this wake becomes the fresh
|
|
// continuation from the task's saved work.
|
|
// Agent Chat conversations keep their own owner-bound continuation.
|
|
const freshSourceRunId = systemWake && executionBlocker?.recoveryActionId && executionBlocker.runId
|
|
&& !issue.conversationAgentId ? executionBlocker.runId : null;
|
|
// The continuation is recorded as pending and owned by the
|
|
// execution-continuation worker until a run for it exists: a later
|
|
// gate (daily limit, slots, pause, budget), a skip or a crash can
|
|
// never consume it. This system wake is coalesced into it.
|
|
const fresh = freshSourceRunId ? await continueFreshInTransaction(tx as unknown as Db, {
|
|
companyId: issue.companyId, runId: freshSourceRunId, trigger: "continuation_wake", delivery: "record" }) : null;
|
|
if (fresh?.kind !== "continued" && fresh?.kind !== "already_continued") return deferBlockedExecution(continuationHold());
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
|
|
if (
|
|
worktreeExecutionCutoff &&
|
|
issue.createdAt < worktreeExecutionCutoff
|
|
) {
|
|
if (caseRequest) waitForCaseAdmission("heartbeat.worktree_execution_cutoff", "Case handoff is waiting for workspace execution to be enabled");
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "heartbeat.worktree_execution_cutoff",
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: {
|
|
reason: "worktree_execution_cutoff",
|
|
cutoff: worktreeExecutionCutoff.toISOString(),
|
|
issueId: issue.id,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
const cancelStaleScheduledRetry = async (
|
|
scheduledRun: typeof heartbeatRuns.$inferSelect,
|
|
) => {
|
|
const issueCancelled = issue.status === "cancelled";
|
|
if (
|
|
scheduledRun.status !== "scheduled_retry" ||
|
|
(scheduledRun.agentId === issue.assigneeAgentId &&
|
|
!issueCancelled)
|
|
) {
|
|
return false;
|
|
}
|
|
|
|
const now = new Date();
|
|
const reason = issueCancelled
|
|
? "Cancelled because the issue was cancelled before the scheduled retry became due"
|
|
: "Cancelled because the issue was reassigned before the scheduled retry became due";
|
|
await notifyInstructionCollection(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, scheduledRun);
|
|
await notifyScheduledRetryCancellation(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
|
|
await notifyChatRunMilestones(tx, { companyId: scheduledRun.companyId, issueId: issue.id });
|
|
const cancelled = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: now,
|
|
error: reason,
|
|
errorCode: issueCancelled
|
|
? "issue_cancelled"
|
|
: "issue_reassigned",
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, scheduledRun.id),
|
|
eq(heartbeatRuns.status, "scheduled_retry"),
|
|
),
|
|
)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!cancelled) return false;
|
|
|
|
if (scheduledRun.wakeupRequestId) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: now,
|
|
error: reason,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
eq(agentWakeupRequests.id, scheduledRun.wakeupRequestId),
|
|
);
|
|
}
|
|
|
|
if (issue.executionRunId === scheduledRun.id) {
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, issue.id),
|
|
eq(issues.executionRunId, scheduledRun.id),
|
|
),
|
|
);
|
|
}
|
|
|
|
const eventSeq = await allocateHeartbeatRunEventSeq(
|
|
tx as unknown as Db,
|
|
cancelled.id,
|
|
);
|
|
|
|
await tx.insert(heartbeatRunEvents).values({
|
|
companyId: cancelled.companyId,
|
|
runId: cancelled.id,
|
|
agentId: cancelled.agentId,
|
|
seq: eventSeq,
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message: issueCancelled
|
|
? "Scheduled retry cancelled because issue was cancelled before it became due"
|
|
: "Scheduled retry cancelled because issue ownership changed before it became due",
|
|
payload: {
|
|
issueId: issue.id,
|
|
issueStatus: issue.status,
|
|
scheduledRetryAttempt: cancelled.scheduledRetryAttempt,
|
|
scheduledRetryAt: cancelled.scheduledRetryAt
|
|
? new Date(cancelled.scheduledRetryAt).toISOString()
|
|
: null,
|
|
scheduledRetryReason: cancelled.scheduledRetryReason,
|
|
previousRetryAgentId: cancelled.agentId,
|
|
currentAssigneeAgentId: issue.assigneeAgentId,
|
|
},
|
|
});
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({ nextEventSeq: eventSeq + 1, updatedAt: now })
|
|
.where(eq(heartbeatRuns.id, cancelled.id));
|
|
|
|
cancelledRunsToEmit.push(cancelled);
|
|
|
|
return true;
|
|
};
|
|
|
|
let activeExecutionRun = issue.executionRunId
|
|
? await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, issue.executionRunId))
|
|
.then((rows) => rows[0] ?? null)
|
|
: null;
|
|
const manualExecutionHold = activeExecutionRun && await nativeManualAdmissionHoldsExecution(tx as unknown as Db,
|
|
{ companyId: issue.companyId, runId: activeExecutionRun.id, issueId: issue.id });
|
|
|
|
if (
|
|
activeExecutionRun &&
|
|
!manualExecutionHold &&
|
|
!EXECUTION_PATH_HEARTBEAT_RUN_STATUSES.includes(
|
|
activeExecutionRun.status as (typeof EXECUTION_PATH_HEARTBEAT_RUN_STATUSES)[number],
|
|
)
|
|
) {
|
|
activeExecutionRun = null;
|
|
}
|
|
|
|
if (
|
|
activeExecutionRun &&
|
|
(await cancelStaleScheduledRetry(activeExecutionRun))
|
|
) {
|
|
activeExecutionRun = null;
|
|
}
|
|
|
|
// A queued/scheduled run holding the lock for an agent that is
|
|
// no longer the issue's assignee is stale by design — the issue
|
|
// has been re-routed (e.g. blocked → in_review with a different
|
|
// assignee). Cancel it and release the lock; otherwise the new
|
|
// assignee's wake gets parked in `deferred_issue_execution`
|
|
// forever, because the original queued holder will never run
|
|
// (the issue's status / target now belongs to someone else).
|
|
//
|
|
// Race guard: pin the cancel UPDATE to the exact non-running
|
|
// status we read above. A worker could transition the holder
|
|
// from `queued` → `running` between the SELECT and this UPDATE;
|
|
// the status predicate ensures we never clobber a freshly-
|
|
// claimed running run. If zero rows matched, leave
|
|
// `activeExecutionRun` populated so the defer path runs
|
|
// normally against the now-running holder.
|
|
if (
|
|
activeExecutionRun &&
|
|
!manualExecutionHold &&
|
|
activeExecutionRun.status !== "running" &&
|
|
issue.assigneeAgentId &&
|
|
activeExecutionRun.agentId !== issue.assigneeAgentId
|
|
) {
|
|
await notifyInstructionCollection(tx, { companyId: activeExecutionRun.companyId, runId: activeExecutionRun.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: activeExecutionRun.companyId, runId: activeExecutionRun.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: activeExecutionRun.companyId, runId: activeExecutionRun.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, activeExecutionRun);
|
|
await notifyChatRunMilestones(tx, { companyId: activeExecutionRun.companyId, issueId: issue.id });
|
|
const cancelled = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: new Date(),
|
|
error:
|
|
"Execution lock released after issue reassigned to a different agent",
|
|
errorCode: "lock_released_on_reassignment",
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, activeExecutionRun.id),
|
|
eq(heartbeatRuns.status, activeExecutionRun.status),
|
|
),
|
|
)
|
|
.returning();
|
|
if (cancelled.length > 0) {
|
|
cancelledRunsToEmit.push(cancelled[0]);
|
|
if (activeExecutionRun.wakeupRequestId) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: new Date(),
|
|
error:
|
|
"Execution lock released after issue reassigned to a different agent",
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
eq(
|
|
agentWakeupRequests.id,
|
|
activeExecutionRun.wakeupRequestId,
|
|
),
|
|
);
|
|
}
|
|
activeExecutionRun = null;
|
|
}
|
|
}
|
|
|
|
if (!activeExecutionRun && issue.executionRunId) {
|
|
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(issues.id, issue.id));
|
|
}
|
|
|
|
if (!activeExecutionRun) {
|
|
const legacyRun = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, issue.companyId),
|
|
inArray(heartbeatRuns.status, [
|
|
...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES,
|
|
]),
|
|
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`,
|
|
),
|
|
)
|
|
.orderBy(
|
|
sql`case when ${heartbeatRuns.status} = 'running' then 0 else 1 end`,
|
|
asc(heartbeatRuns.createdAt),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (legacyRun) {
|
|
if (await cancelStaleScheduledRetry(legacyRun)) {
|
|
activeExecutionRun = null;
|
|
} else {
|
|
activeExecutionRun = legacyRun;
|
|
const legacyAgent = await tx
|
|
.select({ name: agents.name })
|
|
.from(agents)
|
|
.where(eq(agents.id, legacyRun.agentId))
|
|
.then((rows) => rows[0] ?? null);
|
|
await notifyStatusCardIssueChange(tx, issue.companyId);
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: legacyRun.id,
|
|
executionAgentNameKey: normalizeAgentNameKey(
|
|
legacyAgent?.name,
|
|
),
|
|
executionLockedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(issues.id, issue.id));
|
|
}
|
|
}
|
|
}
|
|
|
|
// Availability is advisory: competing clicks may both read it before
|
|
// admission. The existing issue lock serializes the run and deferred
|
|
// receipt checks with the admission below, without another lock owner.
|
|
if (isManualContextCompactionWake()) {
|
|
if (activeExecutionRun) {
|
|
const compacting = isContextCompactionRun(activeExecutionRun.contextSnapshot);
|
|
throw new ContextCompactionError(compacting ? "already_pending" : "agent_busy",
|
|
compacting ? "The agent is already compacting its context." : "Available when the agent finishes its current run.");
|
|
}
|
|
const [pending] = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, issue.companyId), eq(agentWakeupRequests.agentId, agentId),
|
|
inArray(agentWakeupRequests.status, ["queued", "deferred_issue_execution"]), isNull(agentWakeupRequests.runId),
|
|
executionWaitRequestId ? ne(agentWakeupRequests.id, executionWaitRequestId) : undefined,
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
|
|
sql`${agentWakeupRequests.payload}->>'intent' = ${CONTEXT_COMPACTION_WAKE_REASON}`,
|
|
sql`jsonb_typeof(${agentWakeupRequests.payload}->'requestId') = 'string'`,
|
|
sql`nullif(btrim(${agentWakeupRequests.payload}->>'requestId'), '') is not null`,
|
|
sql`${agentWakeupRequests.payload}->${DEFERRED_WAKE_CONTEXT_KEY}->>'contextCompactionRequestId' = ${agentWakeupRequests.payload}->>'requestId'`,
|
|
)).limit(1);
|
|
if (pending) throw new ContextCompactionError("already_pending", "The agent is already compacting its context.");
|
|
}
|
|
|
|
const dependencyReadiness = await issuesSvc
|
|
.listDependencyReadiness(issue.companyId, [issue.id], tx)
|
|
.then((rows) => rows.get(issue.id) ?? null);
|
|
|
|
// Blocked descendants should stay idle until the final blocker resolves.
|
|
// Human comment/mention wakes are the exception: they may run in a
|
|
// bounded interaction mode so the assignee can answer or triage.
|
|
// A stalled child or blocker (issue_blocker_stalled, once per stall)
|
|
// is the other exception: the assignee must be able to re-delegate,
|
|
// escalate or block with a reason while that dependency is stuck.
|
|
const blockedInteractionWake =
|
|
dependencyReadiness &&
|
|
!dependencyReadiness.isDependencyReady &&
|
|
allowsDependencyBlockedWake(
|
|
enrichedContextSnapshot,
|
|
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
|
|
await verifyBlockerStallWake(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id,
|
|
contextSnapshot: enrichedContextSnapshot, requestedByActorType: opts.requestedByActorType }),
|
|
await isVerifiedDependencyInteractionResponse(tx, { companyId: issue.companyId,
|
|
issueId: issue.id, agentId, context: enrichedContextSnapshot }),
|
|
);
|
|
|
|
if (blockedInteractionWake) {
|
|
enrichedContextSnapshot.dependencyBlockedInteraction = true;
|
|
enrichedContextSnapshot.unresolvedBlockerIssueIds =
|
|
dependencyReadiness.unresolvedBlockerIssueIds;
|
|
enrichedContextSnapshot.unresolvedBlockerCount =
|
|
dependencyReadiness.unresolvedBlockerCount;
|
|
enrichedContextSnapshot.unresolvedBlockerSummaries =
|
|
await listUnresolvedBlockerSummaries(
|
|
tx,
|
|
issue.companyId,
|
|
issue.id,
|
|
dependencyReadiness.unresolvedBlockerIssueIds,
|
|
);
|
|
}
|
|
|
|
if (
|
|
!activeExecutionRun &&
|
|
dependencyReadiness &&
|
|
!dependencyReadiness.isDependencyReady &&
|
|
!blockedInteractionWake
|
|
) {
|
|
if (caseRequest) waitForCaseAdmission("issue_dependencies_blocked", "Case handoff is waiting for task dependencies to resolve");
|
|
await recordExecutionWait(tx as unknown as Db, {
|
|
issueId: issue.id,
|
|
coalesce: coalesceExecutionWait,
|
|
condition: { unresolvedBlockerIssueIds: [...dependencyReadiness.unresolvedBlockerIssueIds].sort() },
|
|
request: {
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "issue_dependencies_blocked",
|
|
payload: {
|
|
...(payload ?? {}),
|
|
issueId,
|
|
unresolvedBlockerIssueIds:
|
|
dependencyReadiness.unresolvedBlockerIssueIds,
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
},
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
if (
|
|
isolatedWorkspacesEnabled &&
|
|
!activeExecutionRun &&
|
|
(issue.status !== "done" || Boolean(opts.queuedCommentInterruptId)) &&
|
|
issue.status !== "cancelled"
|
|
) {
|
|
const issueSettings = parseIssueExecutionWorkspaceSettings(
|
|
issue.executionWorkspaceSettings,
|
|
);
|
|
const resolvedMode = resolveExecutionWorkspaceMode({
|
|
projectPolicy: null,
|
|
issueSettings,
|
|
legacyUseProjectWorkspace: null,
|
|
});
|
|
const workspaceManagedConfig = buildExecutionWorkspaceAdapterConfig(
|
|
{
|
|
agentConfig: parseObject(agent.adapterConfig),
|
|
projectPolicy: null,
|
|
issueSettings,
|
|
mode: resolvedMode,
|
|
legacyUseProjectWorkspace: null,
|
|
},
|
|
);
|
|
const resolvedStrategy = resolveEffectiveWorkspaceStrategyType(
|
|
resolvedMode,
|
|
workspaceManagedConfig,
|
|
);
|
|
const existingExecutionWorkspaceStatus = issue.executionWorkspaceId
|
|
? await tx
|
|
.select({ status: executionWorkspaces.status })
|
|
.from(executionWorkspaces)
|
|
.where(
|
|
and(
|
|
eq(executionWorkspaces.id, issue.executionWorkspaceId),
|
|
eq(executionWorkspaces.companyId, issue.companyId),
|
|
),
|
|
)
|
|
.then((rows) => rows[0]?.status ?? null)
|
|
: null;
|
|
const reuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({
|
|
issueExecutionWorkspaceId: issue.executionWorkspaceId,
|
|
issueExecutionWorkspacePreference:
|
|
issue.executionWorkspacePreference,
|
|
existingExecutionWorkspaceStatus,
|
|
});
|
|
const hasResolvablePriorSessionWorkspace =
|
|
await resolveHasResolvablePriorSessionWorkspace();
|
|
|
|
if (
|
|
isUnrunnableWorktreeCombo({
|
|
issue: {
|
|
projectId: issue.projectId ?? projectId ?? null,
|
|
projectWorkspaceId: issue.projectWorkspaceId,
|
|
executionWorkspaceId: issue.executionWorkspaceId,
|
|
executionWorkspacePreference:
|
|
issue.executionWorkspacePreference,
|
|
},
|
|
resolvedMode,
|
|
resolvedStrategy,
|
|
reusableExecutionWorkspaceAvailable:
|
|
reuseRequest.existingExecutionWorkspaceAvailable,
|
|
hasResolvablePriorSessionWorkspace,
|
|
freshIsolatedContinuation: await isFreshIsolatedContinuationWake(tx as unknown as Db, {
|
|
companyId: issue.companyId, issueId: issue.id, contextSnapshot: enrichedContextSnapshot }),
|
|
})
|
|
) {
|
|
if (caseRequest) waitForCaseAdmission(WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE, WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE);
|
|
const now = new Date();
|
|
const issueLabel = formatIssueIdentifierLink(
|
|
issue.identifier,
|
|
issue.id,
|
|
);
|
|
const blockedComment = [
|
|
`Paperclip blocked ${issueLabel} before dispatch because its workspace settings are not runnable.`,
|
|
"",
|
|
`- Code: \`${WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE}\``,
|
|
`- Reason: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE}`,
|
|
`- Next action: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION}`,
|
|
].join("\n");
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
if (issue.status !== "blocked") await notifyChatResponseAvailability(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await notifyRoutineExecutionStatus(tx, issue.id, issue.companyId);
|
|
if (issue.status === "done") {
|
|
const blocked = await issuesSvc.updateForCompany(issue.id, issue.companyId,
|
|
{ status: "blocked", executionState: null }, tx, issueActivityPublications, issuePostCommitActions);
|
|
if (!blocked) throw conflict("The task changed before its workspace settings could be checked");
|
|
} else await tx
|
|
.update(issues)
|
|
.set({
|
|
status: "blocked",
|
|
checkoutRunId: null,
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(issues.id, issue.id));
|
|
await insertIssueComment(tx, {
|
|
companyId: issue.companyId,
|
|
issueId: issue.id,
|
|
body: blockedComment,
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
});
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
|
|
payload: {
|
|
...(payload ?? {}),
|
|
issueId,
|
|
heartbeatSkip: {
|
|
code: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
|
|
reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE,
|
|
remediation:
|
|
WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: now,
|
|
});
|
|
await logActivity(tx as unknown as Db, {
|
|
companyId: issue.companyId,
|
|
actorType: "system",
|
|
actorId: "system",
|
|
agentId,
|
|
runId: null,
|
|
action: "issue.workspace_preflight_blocked",
|
|
entityType: "issue",
|
|
entityId: issue.id,
|
|
details: {
|
|
code: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
|
|
reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE,
|
|
remediation: WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION,
|
|
requestedReason: reason,
|
|
source,
|
|
triggerDetail,
|
|
resolvedMode,
|
|
resolvedStrategy,
|
|
hasResolvablePriorSessionWorkspace,
|
|
},
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
}
|
|
|
|
if (activeExecutionRun) {
|
|
// The case ledger owns retry. Do not place its exact receipt in a
|
|
// generic deferred queue that cannot retain handoff authority.
|
|
if (caseRequest) return { kind: "deferred" as const };
|
|
// The resolved action is already a durable retry outbox. Do not merge
|
|
// its fresh-session contract into unrelated work or create a second
|
|
// deferred wake that could later replay the same reconciliation.
|
|
if (reconciledSourceRunId) return { kind: "deferred" as const };
|
|
|
|
// Offer the input to the active external agent, but keep a durable queued wake
|
|
// until it reads the comment. A finish racing the event tick must
|
|
// leave unread input available for the next assignment.
|
|
const followUpProvider = parseObject(agent.adapterConfig).provider;
|
|
const museFollowUp = followUpProvider === "muse";
|
|
const bindingId = readNonEmptyString(parseObject(agent.adapterConfig)[museFollowUp ? "museBindingId" : followUpProvider === "grok_bot" ? "externalBindingId" : "dotBindingId"]);
|
|
const followUpPayloadKey = museFollowUp ? "museAssignmentFollowUp" : "dotAssignmentFollowUp";
|
|
let followUp: false | { assignmentId: string; consumed: boolean } = false;
|
|
let pendingFollowUps: typeof agentWakeupRequests.$inferSelect[] = [];
|
|
if (agent.adapterType === "paperclip_runner" && ["openai_dot", "grok_bot", "muse"].includes(String(followUpProvider))
|
|
&& bindingId && activeExecutionRun.agentId === agentId && issue.assigneeAgentId === agentId
|
|
&& reason === "issue_commented" && wakeCommentId && opts.allowRunCoalescing !== false
|
|
&& enrichedContextSnapshot.forceFreshSession !== true && !explicitResumeSession && !opts.manualUserWake
|
|
&& !enrichedContextSnapshot.interactionId && !payload?.interactionId && !receiptRequest) {
|
|
// Receipt consumption uses the same issue -> wake -> run -> guard ->
|
|
// binding order. Lock before publishing, then reuse these wake rows.
|
|
pendingFollowUps = await tx.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, agent.companyId), eq(agentWakeupRequests.agentId, agentId),
|
|
isNull(agentWakeupRequests.runId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
|
|
sql`${agentWakeupRequests.payload}->>${followUpPayloadKey} is not null`,
|
|
)).orderBy(asc(agentWakeupRequests.createdAt), asc(agentWakeupRequests.id)).for("update");
|
|
const [currentRun] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, activeExecutionRun.id), eq(heartbeatRuns.companyId, agent.companyId),
|
|
eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.status, "running"),
|
|
eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.nativeIssueId, issue.id),
|
|
)).for("update");
|
|
if (currentRun) followUp = await withExternalAdmissionGuard(tx, agent.companyId, agentId,
|
|
() => (museFollowUp ? publishActiveMuseComment : publishActiveDotComment)(tx as unknown as Db, {
|
|
companyId: agent.companyId, agentId, bindingId, runId: currentRun.id,
|
|
issueId: issue.id, commentId: wakeCommentId,
|
|
}));
|
|
}
|
|
if (followUp) {
|
|
if (!followUp.consumed) await notifyDeferredExecutionWork(tx, { companyId: agent.companyId });
|
|
const assignmentId = followUp.assignmentId;
|
|
const pending = followUp.consumed ? undefined : pendingFollowUps.find(wake =>
|
|
wake.payload?.[followUpPayloadKey] === assignmentId &&
|
|
wake.requestedByActorType === (opts.requestedByActorType ?? null) &&
|
|
wake.requestedByActorId === (opts.requestedByActorId ?? null));
|
|
if (pending) await tx.update(agentWakeupRequests).set({
|
|
payload: withQueuedCommentIdsInWakePayload(pending.payload,
|
|
[...new Set([...queuedCommentIdsFromWakePayload(pending.payload), wakeCommentId!])]),
|
|
coalescedCount: pending.coalescedCount + 1, updatedAt: new Date(),
|
|
}).where(and(eq(agentWakeupRequests.id, pending.id), eq(agentWakeupRequests.companyId, agent.companyId),
|
|
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
else await tx.insert(agentWakeupRequests).values({ companyId: agent.companyId, agentId, source, triggerDetail,
|
|
reason, payload: withQueuedCommentIdsInWakePayload({ ...payload, issueId: issue.id,
|
|
[followUpPayloadKey]: followUp.assignmentId,
|
|
[DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot }, [wakeCommentId!]),
|
|
status: followUp.consumed ? "coalesced" : "deferred_issue_execution",
|
|
runId: followUp.consumed ? activeExecutionRun.id : null,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null, idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: followUp.consumed ? new Date() : null });
|
|
return { kind: "coalesced" as const, run: activeExecutionRun };
|
|
}
|
|
|
|
const admissionScope = wakeQueue.createAdmissionTransactionScope(
|
|
agent.companyId,
|
|
tx as unknown as Db,
|
|
);
|
|
const admission = await wakeQueue.admitWakeBehindIssueExecution(
|
|
admissionScope,
|
|
{
|
|
companyId: agent.companyId,
|
|
issueId: issue.id,
|
|
agentId,
|
|
agentNameKey,
|
|
issueExecutionAgentNameKey: issue.executionAgentNameKey,
|
|
activeExecutionRun: {
|
|
id: activeExecutionRun.id,
|
|
agentId: activeExecutionRun.agentId,
|
|
status: activeExecutionRun.status,
|
|
contextSnapshot: activeExecutionRun.contextSnapshot,
|
|
wakeupRequestId: activeExecutionRun.wakeupRequestId,
|
|
},
|
|
allowRunCoalescing: isConversation(issue) ? false : opts.allowRunCoalescing,
|
|
durableReceipt: receiptRequest
|
|
? {
|
|
id: receiptRequest.id,
|
|
requestedAt: receiptRequest.requestedAt,
|
|
}
|
|
: undefined,
|
|
reason,
|
|
liveRunExecutions,
|
|
wakeCommentId,
|
|
forceFreshSession:
|
|
enrichedContextSnapshot.forceFreshSession === true,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
source,
|
|
triggerDetail,
|
|
payload,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
},
|
|
);
|
|
|
|
if (admission.kind === "coalesced") {
|
|
return {
|
|
kind: "coalesced" as const,
|
|
run: admission.run as typeof heartbeatRuns.$inferSelect,
|
|
};
|
|
}
|
|
if (admission.kind === "deferred") {
|
|
return { kind: "deferred" as const, admitted: true };
|
|
}
|
|
// admission.kind === "proceed": no active run absorbed this wake,
|
|
// so fall through to the ordinary queue path below.
|
|
}
|
|
|
|
// PAP-13775: no live run holds the lock, so this wake would start a
|
|
// fresh adapter session. If this agent's recent runs on this issue
|
|
// keep succeeding without any issue-visible progress and the wake
|
|
// carries no new information, hold it back for an escalating cooldown
|
|
// so external pollers/reconcilers can't storm full-price sessions.
|
|
// Server-side recovery retries insert runs directly and never reach
|
|
// this gate. A verified durable case receipt carries a new phase;
|
|
// its agent-authored instruction comment is not an event-free wake.
|
|
if (
|
|
!caseRequest &&
|
|
isThrottleCandidateIssueRewake({
|
|
reason,
|
|
wakeCommentId: wakeCommentId ?? null,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
forceFreshSession:
|
|
enrichedContextSnapshot.forceFreshSession === true,
|
|
hasExplicitResume: Boolean(explicitResumeSession),
|
|
declaredContinuationKind: readNonEmptyString(parseObject(payload).continuationKind),
|
|
})
|
|
) {
|
|
const throttleNow = new Date();
|
|
const notNativeStatusCommitterActivity = sql`not (${activityLog.actorType} = 'system' and ${activityLog.actorId} = ${NATIVE_STATUS_COMMITTER_ACTOR_ID})`;
|
|
const recentTerminalRuns = await tx
|
|
.select({
|
|
id: heartbeatRuns.id,
|
|
status: heartbeatRuns.status,
|
|
finishedAt: heartbeatRuns.finishedAt,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, agent.companyId),
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
sql`${heartbeatRuns.finishedAt} is not null`,
|
|
gte(
|
|
heartbeatRuns.finishedAt,
|
|
new Date(throttleNow.getTime() - ISSUE_REWAKE_LOOKBACK_MS),
|
|
),
|
|
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`,
|
|
),
|
|
)
|
|
.orderBy(desc(heartbeatRuns.finishedAt))
|
|
.limit(ISSUE_REWAKE_RUN_SAMPLE_LIMIT);
|
|
|
|
if (recentTerminalRuns.length > 0) {
|
|
const sampleRunIds = recentTerminalRuns.map(
|
|
(sampleRun) => sampleRun.id,
|
|
);
|
|
const progressRows = await tx
|
|
.select({ runId: activityLog.runId })
|
|
.from(activityLog)
|
|
.where(
|
|
and(
|
|
eq(activityLog.companyId, agent.companyId),
|
|
eq(activityLog.entityType, "issue"),
|
|
eq(activityLog.entityId, issue.id),
|
|
inArray(activityLog.runId, sampleRunIds),
|
|
inArray(
|
|
activityLog.action,
|
|
ISSUE_PROGRESS_ACTIVITY_ACTIONS,
|
|
),
|
|
notNativeStatusCommitterActivity,
|
|
),
|
|
);
|
|
const lastRunFinishedAt =
|
|
recentTerminalRuns[0]?.finishedAt ?? null;
|
|
const newInputRows = lastRunFinishedAt
|
|
? await tx
|
|
.select({ id: activityLog.id })
|
|
.from(activityLog)
|
|
.where(
|
|
and(
|
|
eq(activityLog.companyId, agent.companyId),
|
|
eq(activityLog.entityType, "issue"),
|
|
eq(activityLog.entityId, issue.id),
|
|
gt(activityLog.createdAt, lastRunFinishedAt),
|
|
sql`coalesce(${activityLog.details}->>'origin', '') <> 'fast_response'`,
|
|
notNativeStatusCommitterActivity,
|
|
inArray(
|
|
activityLog.action,
|
|
ISSUE_NEW_INPUT_ACTIVITY_ACTIONS,
|
|
),
|
|
wakeCommentId && opts.requestedByActorType === "agent"
|
|
? ne(activityLog.actorType, "agent")
|
|
: undefined,
|
|
),
|
|
)
|
|
.limit(1)
|
|
: [];
|
|
|
|
const throttleDecision = evaluateIssueRewakeThrottle({
|
|
now: throttleNow,
|
|
recentTerminalRuns,
|
|
runIdsWithIssueProgress: new Set(
|
|
progressRows
|
|
.map((row) => row.runId)
|
|
.filter((runId): runId is string => Boolean(runId)),
|
|
),
|
|
// For an agent comment wake, the query excludes agent-authored
|
|
// activity while preserving genuinely new user/system input.
|
|
// Presentation/author metadata therefore cannot smuggle human
|
|
// wake privilege, nor can it mask an actual human response.
|
|
hasNewIssueInputSinceLastRun: newInputRows.length > 0,
|
|
});
|
|
|
|
if (throttleDecision.blocked) {
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "issue_rewake_throttled",
|
|
payload: {
|
|
...(payload ?? {}),
|
|
issueId,
|
|
heartbeatSkip: {
|
|
reason: "issue_rewake_throttled",
|
|
requestedReason: reason,
|
|
noProgressStreak: throttleDecision.noProgressStreak,
|
|
cooldownMs: throttleDecision.cooldownMs,
|
|
lastRunFinishedAt:
|
|
throttleDecision.lastRunFinishedAt.toISOString(),
|
|
nextAllowedAt:
|
|
throttleDecision.nextAllowedAt.toISOString(),
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: throttleNow,
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
}
|
|
}
|
|
|
|
const dailyCapBlock = await getHeartbeatDailyCapBlock(
|
|
agent,
|
|
policy,
|
|
{},
|
|
tx,
|
|
);
|
|
if (dailyCapBlock) {
|
|
if (caseRequest) waitForCaseAdmission(dailyCapBlock.reason, "Case handoff is waiting for the agent's daily limit to clear");
|
|
if (executionWaitRequestId && executionBlocker) {
|
|
continuationWait = { reason: dailyCapBlock.reason,
|
|
message: "The agent has reached its daily limit. Your message is saved until work can resume." };
|
|
return deferBlockedExecution(executionBlocker);
|
|
}
|
|
if (opts.idempotencyKey?.startsWith(CONVERSATION_COMMENT_KEY_PREFIX)) throw dailyLimitConversationRefusal(dailyCapBlock);
|
|
const now = new Date();
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: dailyCapBlock.reason,
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: {
|
|
reason:
|
|
"Per-agent heartbeat daily cap reached before adapter invocation.",
|
|
observed: dailyCapBlock.observed,
|
|
limit: dailyCapBlock.limit,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: now,
|
|
});
|
|
if (source === "timer") {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await tx
|
|
.update(agents)
|
|
.set({
|
|
lastHeartbeatAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(agents.id, agentId));
|
|
}
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
const explicitContinuation = await admitExplicitNativeContinuation({
|
|
db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id,
|
|
agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId,
|
|
reason: taskRetryActorId ? "task_retry" : durableRequest && !failedChatRetry ? "issue_commented" : reason,
|
|
commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId,
|
|
queuedCommentInterruptId: opts.queuedCommentInterruptId,
|
|
queuedCommentRequestId: opts.queuedCommentRequestId,
|
|
taskRetryRequestId: opts.taskRetryRequestId,
|
|
...(durableRequest?.resumeDeferred && !failedChatRetry ? { durableChatRequest: durableRequest } : {}),
|
|
interactionPayload: payload,
|
|
onBlocked: (reason, message) => { continuationBlocked = true; continuationWait = { reason, message }; },
|
|
});
|
|
if (!explicitContinuation && (executionBlocker || (continuationBlocked && !verifiedConversationReset)) && !ownsReconstructionContinuation)
|
|
return deferBlockedExecution(continuationHold());
|
|
if (explicitContinuation) {
|
|
enrichedContextSnapshot.forceFreshSession = true;
|
|
enrichedContextSnapshot.previousRunId = explicitContinuation.previousRunId;
|
|
enrichedContextSnapshot.explicitUserContinuation = explicitContinuation;
|
|
}
|
|
|
|
if (opts.queuedCommentInterruptId && issue.status === "done") {
|
|
const reopened = await issuesSvc.updateForCompany(issue.id, issue.companyId,
|
|
{ status: "todo", executionState: null }, tx, issueActivityPublications, issuePostCommitActions);
|
|
if (!reopened) throw conflict("The task changed before its saved message could start");
|
|
await logActivity(tx as unknown as Db, {
|
|
companyId: issue.companyId, actorType: "user", actorId: opts.requestedByActorId!,
|
|
action: "issue.updated", entityType: "issue", entityId: issue.id,
|
|
details: { status: "todo", previousStatus: "done", source: "queued_comment_send", queueId: opts.queuedCommentInterruptId },
|
|
}, issueActivityPublications);
|
|
}
|
|
|
|
const wakeupValues = {
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason,
|
|
payload,
|
|
status: "queued" as const,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
};
|
|
// A deferred durable chat wake already owns this ID and source.
|
|
// Promote that receipt in the same transaction as its successor.
|
|
const wakeupRequest = durableRequest && executionWaitRequestId === durableRequest.id
|
|
? await tx.update(agentWakeupRequests).set({
|
|
payload, status: "queued", error: null, finishedAt: null, updatedAt: new Date(),
|
|
}).where(and(eq(agentWakeupRequests.id, durableRequest.id),
|
|
eq(agentWakeupRequests.companyId, agent.companyId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId)))
|
|
.returning().then(rows => rows[0])
|
|
: await tx.insert(agentWakeupRequests).values(wakeupValues).returning().then(rows => rows[0]);
|
|
if (!wakeupRequest) return { kind: "deferred" as const };
|
|
|
|
// A handoff changes the executor, not the owner of saved user input.
|
|
// Validate its exact stopped source while the issue row is locked;
|
|
// unrelated agents and dedicated continuations keep their own wakes.
|
|
const interruptedRunId = readNonEmptyString(enrichedContextSnapshot.interruptedRunId);
|
|
const handoffSource = source === "assignment" && reason === "issue_assigned" &&
|
|
issue.assigneeAgentId === agentId && interruptedRunId && isUuidLike(interruptedRunId)
|
|
? await tx.select({ agentId: heartbeatRuns.agentId }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, interruptedRunId), eq(heartbeatRuns.companyId, issue.companyId),
|
|
eq(heartbeatRuns.status, "cancelled"), eq(heartbeatRuns.errorCode, "issue_reassigned"),
|
|
ne(heartbeatRuns.agentId, agentId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`,
|
|
or(isNull(heartbeatRuns.nativeIssueId), eq(heartbeatRuns.nativeIssueId, issue.id)),
|
|
)).then(rows => rows[0] ?? null)
|
|
: null;
|
|
// External input owns an exact authorized source batch. Board input
|
|
// must retain its separate receipt, including during normal resume.
|
|
const canCoalesceComments = !durableRequest && !isConversation(issue) && opts.allowRunCoalescing !== false;
|
|
// A resumed receipt already passed admission under this issue lock.
|
|
// Consume it with its successor even when chat keeps other messages
|
|
// in separate turns, or finalization will deliver it a second time.
|
|
const pendingComments =
|
|
(executionWaitRequestId || canCoalesceComments) &&
|
|
(ownsReconstructionContinuation || !(await getExecutionBlocker(tx as unknown as Db, issue.companyId, issue.id)))
|
|
? await tx
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, issue.companyId),
|
|
inArray(agentWakeupRequests.agentId, handoffSource ? [agentId, handoffSource.agentId] : [agentId]),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
|
|
// Chat Retry already pins its complete authorized source
|
|
// batch. Keep separate board input for its own admission.
|
|
canCoalesceComments || (opts.taskRetryRequestId && !failedChatRetry)
|
|
? undefined : eq(agentWakeupRequests.id, executionWaitRequestId!),
|
|
),
|
|
)
|
|
.orderBy(asc(agentWakeupRequests.requestedAt))
|
|
: [];
|
|
const adoptedComments = pendingComments.filter((wake) => {
|
|
if (wake.id === executionWaitRequestId) return true;
|
|
if (wake.id === opts.queuedCommentInterruptId || wake.id === opts.queuedCommentRequestId) return true;
|
|
const deferredPayload = parseObject(wake.payload);
|
|
const deferredContext = parseObject(
|
|
deferredPayload[DEFERRED_WAKE_CONTEXT_KEY],
|
|
);
|
|
// Dedicated interaction wakes carry their own source and session
|
|
// contract. ID-only adoption must not erase that continuation.
|
|
return (
|
|
// Durable chat work must keep its receipt, actor, source, and
|
|
// session contract through normal promotion and authorization.
|
|
!wake.idempotencyKey?.startsWith("chat-inbound:") &&
|
|
!isInteractionResolutionWakePayload(deferredPayload) &&
|
|
!hasInteractionContinuationWakeContext(deferredContext) &&
|
|
["issue_commented", "issue_reopened_via_comment"].includes(String(deferredContext.wakeReason ?? wake.reason)) &&
|
|
queuedCommentIdsFromWakePayload(wake.payload).length > 0
|
|
);
|
|
});
|
|
let adoptedCommentIds = [
|
|
...new Set([
|
|
...adoptedComments.flatMap((wake) =>
|
|
queuedCommentIdsFromWakePayload(wake.payload),
|
|
),
|
|
...queuedCommentIdsFromRunContext(enrichedContextSnapshot),
|
|
]),
|
|
];
|
|
if (opts.queuedCommentRequestId || (opts.taskRetryRequestId && !failedChatRetry) || handoffSource) {
|
|
adoptedCommentIds = await undeliveredLegacyUserCommentIds(tx as unknown as Db,
|
|
agent.companyId, issueId, agentId, adoptedCommentIds);
|
|
}
|
|
await notifyChatRunMilestones(tx, { companyId: agent.companyId, issueId: issueId ?? undefined });
|
|
const newRun = await tx
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
...(explicitContinuation ? { id: explicitContinuationRunId } : {}),
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
scopeKind: readNonEmptyString(enrichedContextSnapshot.issueId) ? "issue" : "company",
|
|
issueId: readNonEmptyString(enrichedContextSnapshot.issueId),
|
|
invocationSource: source,
|
|
triggerDetail,
|
|
status: "queued",
|
|
responsibleUserId: await resolveQueuedResponsibleUserId(),
|
|
wakeupRequestId: wakeupRequest.id,
|
|
retryOfRunId: failedChatRetry
|
|
? durableRequest!.failedRunRetry!.failedRunId
|
|
: opts.failedRunId ?? automaticParentRunId,
|
|
contextSnapshot: adoptedComments.length
|
|
? withQueuedCommentIdsInRunContext(
|
|
enrichedContextSnapshot,
|
|
adoptedCommentIds,
|
|
adoptedComments.map((wake) => wake.id),
|
|
)
|
|
: enrichedContextSnapshot,
|
|
sessionIdBefore: explicitContinuation ? null : sessionBefore,
|
|
continuationAttempt,
|
|
...(reconciledSourceRunId
|
|
? { retryOfRunId: reconciledSourceRunId }
|
|
: {}),
|
|
...(reconciledRestoreRetryCount !== null ? {
|
|
scheduledRetryAttempt: reconciledRestoreRetryCount,
|
|
scheduledRetryReason: "transient_failure",
|
|
} : {}),
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]);
|
|
|
|
if (ownsReconstructionContinuation) {
|
|
await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
|
|
continuationDelivery: "delivered", continuationRunId: newRun.id,
|
|
})}::jsonb` }).where(and(eq(issueRecoveryActions.companyId, issue.companyId),
|
|
eq(issueRecoveryActions.id, String(enrichedContextSnapshot.recoveryActionId)),
|
|
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`));
|
|
}
|
|
if (continuationToDeliver)
|
|
await deliverContinuationAsRun(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id, runId: newRun.id,
|
|
continuation: continuationToDeliver });
|
|
else if (reconciledSourceRunId)
|
|
await foldWaitingRetriesIntoRecordedContinuation(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id,
|
|
runId: newRun.id, recoveryActionId: readNonEmptyString(enrichedContextSnapshot.recoveryActionId), sourceRunId: reconciledSourceRunId });
|
|
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
runId: newRun.id,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
|
|
|
|
if (failedChatRetry && opts.taskRetryRequestId) {
|
|
await tx.update(agentWakeupRequests).set({
|
|
status: "coalesced", runId: newRun.id, finishedAt: new Date(), updatedAt: new Date(),
|
|
}).where(and(eq(agentWakeupRequests.companyId, issue.companyId),
|
|
eq(agentWakeupRequests.id, opts.taskRetryRequestId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
}
|
|
|
|
if (adoptedComments.length) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "coalesced",
|
|
runId: newRun.id,
|
|
finishedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
inArray(
|
|
agentWakeupRequests.id,
|
|
adoptedComments.map((wake) => wake.id),
|
|
),
|
|
);
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
payload: withQueuedCommentIdsInWakePayload(payload, adoptedCommentIds),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
|
|
}
|
|
|
|
// executionRunId is NOT stamped here (enqueueWakeup queues the run but
|
|
// doesn't start it). It will be stamped in claimQueuedRun() once the run
|
|
// transitions to "running" — Fix A (lazy locking).
|
|
|
|
return { kind: "queued" as const, run: newRun };
|
|
}))
|
|
.catch((error) => {
|
|
if (isExternalChatWaitAuthorizationContention(error))
|
|
return { kind: "deferred" as const };
|
|
throw error;
|
|
});
|
|
|
|
// The admission has committed. What follows is dispatch of saved rows.
|
|
if (outcome.kind !== "deferred" && outcome.kind !== "skipped") admission?.saved();
|
|
for (const publication of issueActivityPublications) publishActivity(publication);
|
|
await executeIssuePostCommitActions(db, issuePostCommitActions);
|
|
|
|
// Telemetry for the cancelled runs is best-effort background work.
|
|
// Fire it here and never await it: none of the lifecycle work below,
|
|
// nor this function's return, depends on it, so a slow telemetry
|
|
// lookup must not delay them.
|
|
for (const cancelledRun of cancelledRunsToEmit) {
|
|
void emitAgentTaskRun(db, cancelledRun);
|
|
}
|
|
|
|
if (outcome.kind === "durable") {
|
|
return outcome.receipt.runId ? getRun(outcome.receipt.runId) : null;
|
|
}
|
|
if (outcome.kind === "deferred" || outcome.kind === "skipped") {
|
|
return null;
|
|
}
|
|
if (outcome.kind === "coalesced") {
|
|
await attachQueuedSetupRequest(agent);
|
|
await startNextQueuedRunForAgent(agent.id);
|
|
return outcome.run;
|
|
}
|
|
if (outcome.kind === "replayed") {
|
|
if (outcome.run.status === "queued") {
|
|
await attachQueuedSetupRequest(agent);
|
|
await startNextQueuedRunForAgent(agent.id);
|
|
}
|
|
return outcome.run;
|
|
}
|
|
|
|
const newRun = outcome.run;
|
|
publishLiveEvent({
|
|
companyId: newRun.companyId,
|
|
type: "heartbeat.run.queued",
|
|
payload: {
|
|
runId: newRun.id,
|
|
agentId: newRun.agentId,
|
|
invocationSource: newRun.invocationSource,
|
|
triggerDetail: newRun.triggerDetail,
|
|
wakeupRequestId: newRun.wakeupRequestId,
|
|
},
|
|
});
|
|
|
|
await attachQueuedSetupRequest(agent);
|
|
await startNextQueuedRunForAgent(agent.id);
|
|
return newRun;
|
|
}
|
|
|
|
if (durableRequest) throw new Error("chat_inbound_wakeup_binding_denied");
|
|
|
|
const activeRuns = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
inArray(heartbeatRuns.status, [
|
|
...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES,
|
|
]),
|
|
),
|
|
)
|
|
.orderBy(desc(heartbeatRuns.createdAt));
|
|
|
|
const sameScopeQueuedRun = activeRuns.find(
|
|
(candidate) =>
|
|
candidate.status === "queued" &&
|
|
isSameTaskScope(runTaskKey(candidate), taskKey),
|
|
);
|
|
const sameScopeScheduledRetryRun = activeRuns.find(
|
|
(candidate) =>
|
|
candidate.status === "scheduled_retry" &&
|
|
isSameTaskScope(runTaskKey(candidate), taskKey),
|
|
);
|
|
const sameScopeRunningRun = activeRuns.find(
|
|
(candidate) =>
|
|
candidate.status === "running" &&
|
|
isSameTaskScope(runTaskKey(candidate), taskKey),
|
|
);
|
|
const shouldQueueFollowupForRunningWake =
|
|
Boolean(sameScopeRunningRun) &&
|
|
!sameScopeQueuedRun &&
|
|
shouldQueueFollowupForRunningIssueWake({
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
wakeCommentId,
|
|
});
|
|
// Unscoped manual wakes need their own receipt and execution identity too.
|
|
const rawCoalescedTarget =
|
|
opts.allowRunCoalescing === false || opts.manualUserWake
|
|
? null
|
|
: (sameScopeQueuedRun ??
|
|
sameScopeScheduledRetryRun ??
|
|
(shouldQueueFollowupForRunningWake
|
|
? null
|
|
: (sameScopeRunningRun ?? null)));
|
|
|
|
// A reviewed chat run attests one conversation, so never fold another
|
|
// conversation's reply into it; that reply queues its own run instead.
|
|
const coalescedTargetRun = rawCoalescedTarget &&
|
|
(hasRetainedExternalChatRetry(rawCoalescedTarget.contextSnapshot) ||
|
|
wakeChatConversationsConflict(rawCoalescedTarget.contextSnapshot, enrichedContextSnapshot))
|
|
? null
|
|
: filterZombieCoalesceTarget(rawCoalescedTarget, liveRunExecutions);
|
|
|
|
if (coalescedTargetRun) {
|
|
const mergedContextSnapshot = mergeCoalescedContextSnapshot(
|
|
coalescedTargetRun.contextSnapshot,
|
|
enrichedContextSnapshot,
|
|
{
|
|
preserveExistingInteractionContinuation:
|
|
coalescedTargetRun.status === "queued" ||
|
|
coalescedTargetRun.status === "scheduled_retry",
|
|
},
|
|
);
|
|
const mergedRun = await db
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
contextSnapshot: mergedContextSnapshot,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(heartbeatRuns.id, coalescedTargetRun.id))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? coalescedTargetRun);
|
|
|
|
await db.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason,
|
|
payload,
|
|
status: "coalesced",
|
|
coalescedCount: 1,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
runId: mergedRun.id,
|
|
finishedAt: new Date(),
|
|
});
|
|
return mergedRun;
|
|
}
|
|
|
|
const queueOutcome = await db.transaction(async (tx) => {
|
|
await notifyRunDispatch(tx, { companyId: agent.companyId });
|
|
await tx.execute(
|
|
sql`select id from agents where id = ${agentId} and company_id = ${agent.companyId} for update`,
|
|
);
|
|
|
|
const dailyCapBlock = await getHeartbeatDailyCapBlock(
|
|
agent,
|
|
policy,
|
|
{},
|
|
tx,
|
|
);
|
|
if (dailyCapBlock) {
|
|
if (opts.idempotencyKey?.startsWith(CONVERSATION_COMMENT_KEY_PREFIX)) throw dailyLimitConversationRefusal(dailyCapBlock);
|
|
const now = new Date();
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: dailyCapBlock.reason,
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: {
|
|
reason:
|
|
"Per-agent heartbeat daily cap reached before adapter invocation.",
|
|
observed: dailyCapBlock.observed,
|
|
limit: dailyCapBlock.limit,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: now,
|
|
});
|
|
if (source === "timer") {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await tx
|
|
.update(agents)
|
|
.set({
|
|
lastHeartbeatAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(agents.id, agentId));
|
|
}
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
const wakeupRequest = await tx
|
|
.insert(agentWakeupRequests)
|
|
.values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason,
|
|
payload,
|
|
status: "queued",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]);
|
|
|
|
const newRun = await tx
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
scopeKind: readNonEmptyString(enrichedContextSnapshot.issueId) ? "issue" : "company",
|
|
issueId: readNonEmptyString(enrichedContextSnapshot.issueId),
|
|
invocationSource: source,
|
|
triggerDetail,
|
|
status: "queued",
|
|
responsibleUserId: await resolveQueuedResponsibleUserId(),
|
|
wakeupRequestId: wakeupRequest.id,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
sessionIdBefore: sessionBefore,
|
|
continuationAttempt,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]);
|
|
|
|
const continuedIssueId = readNonEmptyString(enrichedContextSnapshot.issueId);
|
|
if (continuedIssueId && readNonEmptyString(enrichedContextSnapshot.freshContinuationIsolation)) {
|
|
await tx.execute(sql`select id from issues where company_id = ${agent.companyId} and id = ${continuedIssueId} for update`);
|
|
await foldWaitingRetriesIntoContinuation(tx as unknown as Db, { companyId: agent.companyId, issueId: continuedIssueId, runId: newRun.id,
|
|
sourceRunId: readNonEmptyString(enrichedContextSnapshot.previousRunId) });
|
|
}
|
|
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
runId: newRun.id,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
|
|
|
|
return { kind: "queued" as const, run: newRun };
|
|
});
|
|
|
|
if (queueOutcome.kind === "skipped") return null;
|
|
admission?.saved();
|
|
const newRun = queueOutcome.run;
|
|
|
|
publishLiveEvent({
|
|
companyId: newRun.companyId,
|
|
type: "heartbeat.run.queued",
|
|
payload: {
|
|
runId: newRun.id,
|
|
agentId: newRun.agentId,
|
|
invocationSource: newRun.invocationSource,
|
|
triggerDetail: newRun.triggerDetail,
|
|
wakeupRequestId: newRun.wakeupRequestId,
|
|
},
|
|
});
|
|
|
|
await attachQueuedSetupRequest(agent);
|
|
await startNextQueuedRunForAgent(agent.id);
|
|
|
|
return newRun;
|
|
}
|
|
|
|
/**
|
|
* Native status commitment deliberately persists dependency/parent wake
|
|
* intents in the same transaction as the authoritative status projection.
|
|
* Those rows are not runnable until the heartbeat scheduler has applied its
|
|
* normal policy, workspace, concurrency, and responsible-user checks. Bridge
|
|
* the durable intent into that scheduler here instead of treating a bare
|
|
* `agent_wakeup_requests` row as if it were already a queued heartbeat run.
|
|
*
|
|
* The intent is claimed before dispatch. A deterministic dispatcher actor id
|
|
* lets a later sweep recover the narrow process-crash window after the real
|
|
* wake was inserted but before the intent was linked to it. Dispatch failure
|
|
* only requeues the intent; it never changes the provider run outcome.
|
|
*/
|
|
type NativeStatusWakeDispatchInput = {
|
|
companyId?: string;
|
|
limit?: number;
|
|
staleClaimMs?: number;
|
|
requestId?: string;
|
|
onDeferredDeadline?: (at: number) => void;
|
|
onError?: (error: unknown) => void;
|
|
};
|
|
type NativeStatusWakeDelivery = { runId: string | null; status: string };
|
|
|
|
function createNativeStatusWakeBatch() {
|
|
const deliveredByIssueScope = new Map<string, NativeStatusWakeDelivery>();
|
|
return (input: NativeStatusWakeDispatchInput = {}) => dispatchNativeStatusWakeBatch(input, deliveredByIssueScope);
|
|
}
|
|
|
|
function throttledDispatchUntil(dispatch: typeof agentWakeupRequests.$inferSelect): number | null {
|
|
if (dispatch.status !== "skipped" || dispatch.reason !== "issue_rewake_throttled") return null;
|
|
const at = Date.parse(String(parseObject(parseObject(dispatch.payload).heartbeatSkip).nextAllowedAt ?? ""));
|
|
return Number.isFinite(at) ? at : null;
|
|
}
|
|
|
|
function dispatchPendingNativeStatusWakeups(input: NativeStatusWakeDispatchInput = {}) {
|
|
return createNativeStatusWakeBatch()(input);
|
|
}
|
|
|
|
async function dispatchNativeStatusWakeBatch(input: NativeStatusWakeDispatchInput,
|
|
deliveredByIssueScope: Map<string, NativeStatusWakeDelivery>,
|
|
) {
|
|
const now = new Date();
|
|
const staleClaimMs = Math.max(1_000, input.staleClaimMs ?? 60_000);
|
|
const candidates = await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
input.companyId
|
|
? eq(agentWakeupRequests.companyId, input.companyId)
|
|
: undefined,
|
|
input.requestId ? eq(agentWakeupRequests.id, input.requestId) : undefined,
|
|
eq(agentWakeupRequests.requestedByActorType, "system"),
|
|
eq(agentWakeupRequests.requestedByActorId, "native-status-committer"),
|
|
inArray(agentWakeupRequests.status, ["queued", "claimed"]),
|
|
isNull(agentWakeupRequests.runId),
|
|
),
|
|
)
|
|
.orderBy(asc(agentWakeupRequests.requestedAt))
|
|
.limit(Math.max(1, Math.min(input.limit ?? 100, 500)));
|
|
|
|
let dispatched = 0;
|
|
let recovered = 0;
|
|
let deferred = 0;
|
|
|
|
for (const candidate of candidates) {
|
|
const payload = parseObject(candidate.payload);
|
|
const wakeContext = parseObject(payload._paperclipWakeContext);
|
|
const issueId =
|
|
readNonEmptyString(payload.issueId) ??
|
|
readNonEmptyString(payload.taskId) ??
|
|
readNonEmptyString(wakeContext.issueId) ??
|
|
null;
|
|
const dispatchActorId = `native-status-wake-dispatch:${candidate.id}`;
|
|
const existingDispatch = await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, candidate.companyId),
|
|
eq(agentWakeupRequests.requestedByActorType, "system"),
|
|
eq(agentWakeupRequests.requestedByActorId, dispatchActorId),
|
|
),
|
|
)
|
|
.orderBy(desc(agentWakeupRequests.requestedAt))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
const throttledUntil = existingDispatch ? throttledDispatchUntil(existingDispatch) : null;
|
|
if (throttledUntil !== null && throttledUntil > now.getTime()) {
|
|
input.onDeferredDeadline?.(throttledUntil);
|
|
deferred += 1;
|
|
continue;
|
|
}
|
|
|
|
if (existingDispatch && throttledUntil === null) {
|
|
const recoveredStatus = existingDispatch.runId
|
|
? "coalesced"
|
|
: existingDispatch.status === "deferred_issue_execution"
|
|
? "coalesced"
|
|
: existingDispatch.status;
|
|
await db.transaction(async tx => {
|
|
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: recoveredStatus,
|
|
runId: existingDispatch.runId,
|
|
finishedAt:
|
|
(existingDispatch.runId ?? existingDispatch.finishedAt)
|
|
? (existingDispatch.finishedAt ?? now)
|
|
: null,
|
|
error: existingDispatch.error,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, candidate.id),
|
|
isNull(agentWakeupRequests.runId),
|
|
),
|
|
);
|
|
});
|
|
recovered += 1;
|
|
continue;
|
|
}
|
|
|
|
if (
|
|
candidate.status === "claimed" &&
|
|
candidate.claimedAt &&
|
|
now.getTime() - candidate.claimedAt.getTime() < staleClaimMs
|
|
) {
|
|
input.onDeferredDeadline?.(candidate.claimedAt.getTime() + staleClaimMs);
|
|
deferred += 1;
|
|
continue;
|
|
}
|
|
|
|
const claimed = await db
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "claimed",
|
|
claimedAt: now,
|
|
error: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, candidate.id),
|
|
isNull(agentWakeupRequests.runId),
|
|
candidate.status === "claimed"
|
|
? eq(agentWakeupRequests.status, "claimed")
|
|
: eq(agentWakeupRequests.status, "queued"),
|
|
),
|
|
)
|
|
.returning({ id: agentWakeupRequests.id });
|
|
if (claimed.length === 0) continue;
|
|
|
|
const scopeKey = issueId
|
|
? `${candidate.companyId}:${candidate.agentId}:${issueId}:${readNativeReviewAssignmentContext(wakeContext)?.nativeReviewInteractionId ?? ""}`
|
|
: null;
|
|
const priorDelivery = scopeKey
|
|
? deliveredByIssueScope.get(scopeKey)
|
|
: null;
|
|
if (priorDelivery) {
|
|
await db.transaction(async tx => {
|
|
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "coalesced",
|
|
runId: priorDelivery.runId,
|
|
finishedAt: new Date(),
|
|
error: null,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, candidate.id));
|
|
});
|
|
recovered += 1;
|
|
continue;
|
|
}
|
|
|
|
let completedOnboardingGuard: WakeupOptions["issueStateGuard"];
|
|
if (issueId) {
|
|
const targetIssue = await db
|
|
.select({
|
|
status: issues.status,
|
|
statusVersion: issues.statusVersion,
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
})
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, candidate.companyId),
|
|
),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
const nativeReview = candidate.reason === "native_completion_review"
|
|
? await getNativeReviewAssignment(db, {
|
|
companyId: candidate.companyId, issueId, agentId: candidate.agentId,
|
|
contextSnapshot: wakeContext,
|
|
})
|
|
: null;
|
|
const onboardingResultReport = targetIssue?.status === "done" && await isCompletedOnboardingHandoffWake(db, {
|
|
companyId: candidate.companyId, issueId, agentId: candidate.agentId,
|
|
reason: candidate.reason, contextSnapshot: wakeContext,
|
|
});
|
|
if (onboardingResultReport) completedOnboardingGuard = { assigneeAgentId: candidate.agentId, statuses: ["done"], statusVersion: targetIssue!.statusVersion };
|
|
if (
|
|
!targetIssue ||
|
|
(["done", "cancelled"].includes(targetIssue.status) && !onboardingResultReport) ||
|
|
(targetIssue.assigneeAgentId !== candidate.agentId && !nativeReview) ||
|
|
(candidate.reason === "native_completion_review" && !nativeReview)
|
|
) {
|
|
await db.transaction(async tx => {
|
|
await notifyChatInboundWakeNotices(tx, { companyId: candidate.companyId, issueId });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "skipped",
|
|
finishedAt: new Date(),
|
|
error: !targetIssue
|
|
? "Native status wake target no longer exists"
|
|
: ["done", "cancelled"].includes(targetIssue.status)
|
|
? `Native status wake target is already ${targetIssue.status}`
|
|
: "Native status wake target has a different assignee",
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, candidate.id));
|
|
});
|
|
recovered += 1;
|
|
continue;
|
|
}
|
|
}
|
|
|
|
try {
|
|
const wakeRun = await enqueueWakeup(candidate.agentId, {
|
|
allowRunCoalescing: false,
|
|
source: candidate.source as WakeupOptions["source"],
|
|
triggerDetail: (candidate.triggerDetail ??
|
|
"system") as WakeupOptions["triggerDetail"],
|
|
reason: candidate.reason,
|
|
payload,
|
|
idempotencyKey: candidate.idempotencyKey,
|
|
requestedByActorType: "system",
|
|
requestedByActorId: dispatchActorId,
|
|
...(completedOnboardingGuard ? { issueStateGuard: completedOnboardingGuard } : {}),
|
|
contextSnapshot: {
|
|
...wakeContext,
|
|
...(issueId ? { issueId, taskId: issueId } : {}),
|
|
wakeReason: candidate.reason,
|
|
source: "native_status_decision",
|
|
statusDecisionSource: "native_status_decision",
|
|
nativeStatusWakeIntentId: candidate.id,
|
|
},
|
|
}, undefined, candidate.payload?.interactionOrigin);
|
|
|
|
const delivered = await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, candidate.companyId),
|
|
eq(agentWakeupRequests.requestedByActorType, "system"),
|
|
eq(agentWakeupRequests.requestedByActorId, dispatchActorId),
|
|
),
|
|
)
|
|
.orderBy(desc(agentWakeupRequests.requestedAt))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
const throttledDeliveryUntil = !wakeRun && delivered ? throttledDispatchUntil(delivered) : null;
|
|
if (throttledDeliveryUntil !== null) {
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ status: "queued", claimedAt: null, finishedAt: null, error: null, updatedAt: new Date() })
|
|
.where(and(eq(agentWakeupRequests.id, candidate.id), isNull(agentWakeupRequests.runId)));
|
|
input.onDeferredDeadline?.(throttledDeliveryUntil);
|
|
deferred += 1;
|
|
continue;
|
|
}
|
|
|
|
// The committer intent is the durable outbox entry. When admission is
|
|
// blocked, enqueueWakeup creates a separate deferred dispatch receipt;
|
|
// leave the original intent coalesced so the release drain cannot
|
|
// promote both rows (the original has no nativeStatusWakeIntentId
|
|
// provenance and would otherwise run once before the dispatch receipt).
|
|
const deliveredStatus = delivered?.status ?? "queued";
|
|
await db.transaction(async tx => {
|
|
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: wakeRun || deliveredStatus === "deferred_issue_execution"
|
|
? "coalesced"
|
|
: deliveredStatus,
|
|
runId: wakeRun?.id ?? delivered?.runId ?? null,
|
|
finishedAt:
|
|
wakeRun || delivered?.finishedAt
|
|
? (delivered?.finishedAt ?? new Date())
|
|
: null,
|
|
error: delivered?.error ?? null,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, candidate.id));
|
|
});
|
|
|
|
if (scopeKey) {
|
|
deliveredByIssueScope.set(scopeKey, {
|
|
runId: wakeRun?.id ?? delivered?.runId ?? null,
|
|
status: wakeRun ? "coalesced" : (delivered?.status ?? "queued"),
|
|
});
|
|
}
|
|
|
|
if (wakeRun) dispatched += 1;
|
|
else deferred += 1;
|
|
} catch (error) {
|
|
input.onError?.(error);
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "queued",
|
|
claimedAt: null,
|
|
error:
|
|
error instanceof Error
|
|
? error.message.slice(0, 1_000)
|
|
: String(error).slice(0, 1_000),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, candidate.id),
|
|
eq(agentWakeupRequests.status, "claimed"),
|
|
isNull(agentWakeupRequests.runId),
|
|
),
|
|
);
|
|
logger.warn(
|
|
{
|
|
err: error,
|
|
wakeupRequestId: candidate.id,
|
|
agentId: candidate.agentId,
|
|
},
|
|
"failed to dispatch persisted native status wake intent",
|
|
);
|
|
deferred += 1;
|
|
}
|
|
}
|
|
|
|
return { scanned: candidates.length, dispatched, recovered, deferred };
|
|
}
|
|
|
|
return {
|
|
enqueueWakeup,
|
|
dispatchPendingNativeStatusWakeups,
|
|
createNativeStatusWakeBatch,
|
|
startNextQueuedRunForAgent,
|
|
countRunningRunsForAgent,
|
|
releaseClaimBackoffsForTask,
|
|
releaseRunClaimedJustBeforeSuppression,
|
|
claimQueuedRun,
|
|
withChatControlRecoveryGate,
|
|
trackWakeup,
|
|
resumeQueuedRuns,
|
|
resumeDeferredExecutionReceipts,
|
|
resumeQueuedRunsAfterDrain,
|
|
resumeFreshQueuedRunsAtStartup,
|
|
parseHeartbeatPolicy,
|
|
nextTimerHeartbeatAt,
|
|
claimDueTimerHeartbeat,
|
|
};
|
|
}
|