mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs are interrupted when a server restarts, a deploy swaps
servers, a controller lease is lost, or a sandbox Runner cannot be
reached again
> - Exact recovery (resume the same session, replace, retry) refuses in
many ways, and each refusal leaves the task blocked or idle for a person
> - On staging, 577 of 1,000 runs in 11 hours failed or were
interrupted, across 25 error codes. About 900 earlier PRs each fixed one
refusal code, and the rate did not converge
> - This pull request adds one universal fallback: when an unfinished
run cannot be resumed exactly, for any reason that is not the person's,
the task continues in a fresh run from its last saved workspace and its
conversation
> - The benefit is that a new failure code next week falls back without
code changes, and tasks stop waiting on a person for infrastructure
faults
## Linked Issues or Issue Description
**What happened?**
After a server restart or deploy swap, many runs end with codes such as
`native_continuation_requires_reconciliation`
(`runner_remote_process_identity_unavailable`), `controller_lease_lost`,
`process_lost`, `server_shutdown_interrupted` and
`native_session_interrupted` (including `runner_harness_state_mismatch:
resumed_not_suspended` / `resumed_unavailable`). The disposition worker
then settles the recovery hold as "blocked, no replay". The task stays
Blocked until a person presses Retry or reconciles the run.
**Expected behavior**
A task waits on a person only for reasons that are the person's: an
approval, a question, a credential to share, a budget stop, a provider
usage limit, or a Stop they asked for. For every other failure, the
system starts a new run for the same task from the last verified saved
workspace, without a person.
**Steps to reproduce**
1. Start a task on a sandbox agent.
2. Restart or swap the server while the run is in progress, so the new
server cannot re-reach the Runner.
3. See that the run fails with
`native_continuation_requires_reconciliation` and the task stays Blocked
with no successor run.
**Paperclip version or commit**
`master` at 9e910ff9ae.
## What Changed
- New module `server/src/services/fresh-continuation-fallback.ts`. One
entry point, `ensureFreshContinuation` / `continueFreshInTransaction`,
takes a run that ended without finishing and guarantees one of:
something already follows it (a newer run, a linked replacement, an
operator decision), exact recovery is still under way (checked again
later), the cause is the person's, or one fresh continuation is
recorded.
- Classification, not an allowlist: `PERSON_OWNED_FAILURE_CODES` plus
task-level waits (pending approval, pending question, paused or
unavailable agent, paused company, review in progress, a Stop that a
person asked for, a Blocked status that another decision owns). Every
other code, including unknown future codes, falls back.
- Idempotent per source run. The task row lock serializes callers on
every server; the recorded `freshContinuation.sourceRunId` makes later
calls a no-op.
- Isolation, never waiting on proof. A local process still running on
this host is settled by its own owner first. If the old local writer is
proven gone, the task workspace is kept. Otherwise the continuation gets
a new isolated workspace (`fresh_isolated_workspace`), so two runs never
write one directory. The decision records the stop proof honestly
(`providerStopProof: isolated`).
- Claim admission no longer waits on the cleanup of a predecessor that a
fresh continuation replaced (`heartbeat/queue.ts`, predecessor loop).
Without this, the successor stayed queued behind the unreachable lease,
which is the staging failure itself.
- Only reconciliation holds that this decision owns are settled.
Workspace restore holds that workspace recovery still owns, and resolved
history, are untouched.
- `continuation_user_authorization_missing` is the system's failure (a
Retry's authorization was not carried forward), not the person's.
- Safety: the old run keeps its fencing. When it still owns its remote
workspace epoch, the epoch is abandoned
(`abandonUnverifiedRemoteWorkspaceEpoch`), so the old writer can never
publish and the next run restores the last committed save into a new
root on a new allocation. The unverified files stay in the abandoned
root, unapplied. The native coordinator records `replacementDenied:
fresh_continuation`, so no exact replay also starts.
- The replaced run loses its write authority. The fallback records
`freshContinuationSupersededAt` on the old run. The agent-JWT middleware
(`loadAgentRunAuthentication` reads one more field in its existing row
read) and `assertAgentRunWriteAllowed` refuse every write of the old
token with `agent_run_superseded`. Before this change, a failed or
interrupted run's token could still write. So an agent that is still
alive in an unreachable sandbox cannot act on the task next to its
continuation.
- The opening context (`recoveryOutcomes` in the execution continuation)
says that the previous attempt was cut off and lists the actions with
unverified outcomes, so the agent checks before it repeats a side
effect. No copy asks a person to inspect or repair a sandbox, workspace
or run. The hold text written at run end no longer says "Inspect the
original failure and reconcile".
- Delivery reuses the existing path: the settled hold gets an
`executionReconciliation` decision and `continuationDelivery: pending`,
and `deliverReconciledExecutions` starts the run (fresh session,
`retryOfRunId` = source).
- Budget per task, not per restart: at most 6 fresh continuations per
task per rolling hour (the server-restart lane's cap). At the cap the
hold stays with the system and is checked again when the window has
room. The failure allowance is never charged.
- Call sites:
- `settleUnrecoverableExecutions` (execution-recovery-resolution.ts):
every hold whose exact recovery is ruled out now goes through the
fallback before it would block the task. Interrupted preparations keep
their own replay lane first.
- `continueHeldExecutions` (same worker): picks up holds saved by older
releases (settled-but-blocked no-replay holds, finished or exhausted
saved-work recoveries) for the task's latest unfinished run.
- Wake admission (heartbeat/queue.ts): an ordinary system continuation
wake (sweep, monitor, timer) that a hold would defer now calls the
fallback in the same transaction and becomes the fresh continuation
itself. This is the path the idle-task sweep uses. A person's Retry,
message or queued send keeps its exact admission; the disposition worker
settles the hold, and the saved input then continues.
- `executionControlSchedule.disposition` wakes the worker for held tasks
and for the next due recheck.
### Human gate versus system uncertainty
One documented rule (`isHumanGate` plus `personOwnedTaskWait`) decides
what the fallback never touches. The rule goes by cause, never by owner
type:
- Human gates: a pending approval, a pending question, a user-owned
hold, an owner-gone attestation, a recorded explicit continuation, an
operator reconciliation, a person's Stop.
- System uncertainty: a no-replay record ("outcome unknown after an
interruption"), a hold that is `board`-owned only by default, an
unverifiable controller process space or host.
For uncertainty the fallback starts a fresh run, never a replay. The
interrupted turn is not re-sent. No tool call or write is re-executed.
Governed actions go through their approval gate again. The earlier
`automaticRecovery` decision is kept on the hold as
`automaticRecoveryHistory`, beside the attributed
`freshContinuationSettlement` and an activity entry.
### Restart recovery refused for an unverifiable controller
The case is `controller_process_space_unverified`, its remote variant,
or `controller_host_unrecorded` / `controller_host_unprovable`. Its
lease has lapsed; a live, renewing lease is still waited for (server
overlap). After `NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS` (2 minutes) the
run ends (failed, `native_restart_recovery_blocked`). That frees the
task's execution lock and the agent's slot, and a system-owned hold lets
the fallback continue the task. Recording the stops is optional. The "a
board operator must record both stops" text is replaced.
Board/operator attention sites in native restart recovery:
- `surfaceNativeControllerNamespaceAttention`: copy replaced; the run
now ends after the bounded wait.
- `requestNativeControllerStopConfirmation`
(native-controller-host-recovery.ts): copy says confirmation is
optional; the same bounded end applies.
- The blocked restart-recovery hold (`nextAction` "Inspect the preserved
checkpoint and reconcile…"): copy replaced; the fallback settles the
hold by cause.
- `owner-gone-attestation.ts` (generic resource records from older
releases): unchanged. It is not a run-recovery path, and it is listed as
a follow-up.
### Every wait and its bound
| Wait | Owner that rechecks | Worst-case total before the fresh
continuation |
| --- | --- | --- |
| `exact_recovery_pending` (execution settling, native finalizer,
controller lease, interrupted-preparation lane, saved-work recovery,
restore hold, retained-source fallback, live local process) |
Disposition worker, woken at each deadline; rechecks at 1, 2, 3 and 4
minutes, then every 4 | About 10 minutes from the first pending check.
Up to 15 while the owner shows progress (a new event, output, lease
renewal or coordinator update since the last check). |
| Saved-work (workspace) recovery that is `scheduled`,
`awaiting_evidence`, or `running` without progress | Held-run scan, now
selected and scheduled like the other holds | About 10 minutes; 15 at
most |
| Saved-work recovery that is `running` and advancing (revision,
attempt, state or hold update since the last check) | Held-run scan |
Its own 30-minute budget |
| Unverifiable lapsed controller in restart recovery | Restart recovery
(`retryAt`) | 2 minutes, then the run ends and the fallback continues it
within one disposition pass |
| Hourly cap (`backoff`) | Disposition worker, woken at the deadline |
Until the rolling hour frees a slot, at most 1 hour. This is the
deliberate restart-lane bound, not a stuck state. |
| Person-owned or dependency wait | The person's own action wakes the
task; rechecked hourly | Not bounded: it is the person's |
Restart-recovery classifications (`claimNativeRestartRecoveries`, which
`recoverNativeRunsAfterRestart` calls):
| Classification | Who rechecks, when | Bound |
| --- | --- | --- |
| Claims: `reattach_existing_runner`, `reattach_remote_runner`,
`resume_dead_runner`, `bootstrap_incomplete` | The claiming server
resumes exactly | Exact resume; no fallback |
| `already_finalized` | Nobody; it is terminal | None needed |
| `blocked` | The run is terminal and its hold goes to the fallback
(disposition worker) | Fallback bounds above |
| `awaiting_evidence` / `current_controller_preparing` | This process's
own executor | Ends with that executor |
| `awaiting_evidence` / `recovery_rows_busy` | Restart dispatch after 1
second | Self-resolving: the row lock clears |
| `awaiting_evidence` / a live lease (`controller_lease_held`,
`controller_lease_grace`, `controller_still_alive`) | Restart dispatch
at the lease-renew interval | Never ended while the old controller
renews its lease |
| `awaiting_evidence` / unverifiable lapsed controller (process space or
host) | Restart dispatch every 2 minutes | Ends 2 minutes after the
lease lapsed |
| `awaiting_evidence` / any other reason (startup cleanup, finalizer or
cleanup owner, recovery failure, routine busy, future reasons) | Restart
dispatch at the owner's `retryAt`, or within 2 minutes when it has none
| Ends 2 minutes after the lease lapsed, or after 10 minutes, whichever
comes first. It never ends while a lease is live. |
Ending a run sets it failed (`native_restart_recovery_blocked`),
releases the task lock and the agent's slot, and records a system-owned
hold. The fallback then continues the task once, isolated.
No wait needs longer. Saved-work recovery and retained-source checks
keep their own attempt budgets. After 10 to 15 minutes the fallback
continues anyway in a new root, without the unverified files, which stay
recorded and unapplied.
### When the fallback preempts saved-work recovery
The unverified files are not destroyed. They stay where recovery found
them: in the abandoned remote epoch's root, or in the old local
workspace that the task is no longer bound to. They are recorded and
never applied. Settling the hold records the decision on it, so the
retained-workspace worker stops scheduling it (its query requires no
`executionReconciliation` unless recovery already succeeded). If an
attempt already in flight finishes later, it cannot publish into the
fresh run's workspace. For a remote workspace, the epoch it targets is
abandoned and the writer is cleared. For a local one, the task now
points at a new workspace. Whenever a wait is preempted, the
continuation is isolated: a fenced remote epoch or a new local
workspace.
### What people see
The task's interruption presentation shows "working" while the primary
fresh-continuation record is pending and has not started (coalesced
holds never count, and they follow the primary's final state), and a
timed wait with no Retry while a check is scheduled. The Retry banner
appears only when the system has given up or the cause is the person's.
Holds settled together with the primary record are `coalesced`; they no
longer show "closed or reassigned". A pending continuation is never a
second run: if a person's Retry or saved input started a run after it
was recorded, delivery links to that run instead. A task that finished
before delivery gets no run.
### Readers of what the fallback writes
- Recheck deadline (`freshContinuationCheck`):
- `unresolvedExecutionDispositions` and
`heldExecutionsAwaitingFreshContinuation` read it to skip work that is
not due.
- `nextFreshContinuationCheckAt` and
`executionControlSchedule.disposition` read it to schedule the worker.
- Tests: the cap deadline is scheduled, and a cancelled task goes quiet.
- Isolation decision:
- The task's workspace binding is cleared. Every path that starts the
next run (system continuation, saved input, Retry, sweep wake) resolves
the workspace from that binding at run start (heartbeat.ts
`issueContext.executionWorkspaceId`).
- `freshContinuationIsolation` in the run context is read by
`readFreshWorkspaceReadiness` for the shared-directory case.
- `freshContinuationSupersededAt` on the old run is read by the auth
middleware, `assertAgentRunWriteAllowed` and the claim gate.
## Verification
- `pnpm vitest run src/services/fresh-continuation-fallback.test.ts`
(server): every test passes (it covers the cases below).
- The continuation starts (leaves the queue) while the unreachable old
lease is `active` or `pending_cleanup`. This was red before the
claim-admission fix.
- Isolation: a writer on another host gets a new isolated workspace; a
process alive on this host waits for its owner; a proven-gone writer
keeps the workspace.
- Owned holds only: a restore hold and resolved history are untouched.
- The recheck schedule goes quiet when the task is cancelled.
- `continuation_user_authorization_missing` continues under the task's
responsible user.
- One test per top-five staging code (native and legacy hold shapes):
the task gets exactly one fresh successor run from saved state.
- `runner_harness_state_mismatch` (`resumed_not_suspended`,
`resumed_unavailable`) falls back.
- An unknown code falls back.
- Two concurrent callers plus the disposition worker create one
successor.
- Old-saved/new-resumed: a settled-but-blocked hold from the previous
release, with its coordinator in `awaiting_runner_reattach`, continues
on the first pass.
- An ordinary system continuation wake is admitted as the fresh
continuation.
- Fencing: after the continuation is recorded, the old run's token is
refused for writes (`agent_run_superseded`) and the continuation's token
is not. The old run's remote workspace epoch is abandoned: the writer is
cleared, the location and provider root are cleared, and the epoch
advances. So the next preparation restores the last save into a new root
(`prepareRemoteWorkspace`: `sameAllocation` is false when the location
is null, server/src/services/remote-workspaces.ts:437-470).
- The opening context names the cut-off run and an unverified tool
action.
- Person-owned causes (pending approval, credential not shared, budget
stop) are not continued.
- The hourly cap backs off without blocking, then continues.
- Red/green: with the wiring reverted to `master`, 9 of these tests fail
(the task stays Blocked with no successor).
- Affected suites pass locally: `execution-recovery-dead-controller`,
`completed-turn-cleanup-hold-settle`, `execution-control-work`,
`heartbeat-process-recovery`, `issue-recovery-actions`,
`native-safe-replacement`, `native-failed-workspace-recovery`,
`task-retry`, `runner-lost-mid-turn-continuation`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
wake-queue `postgres`, `agent-conversations`, `native-startup-retry`,
`governed-wait-runner-loss`, `native-preclaim-continuation`.
- `pnpm verify --explain` selects the full suite (the wake queue is
shared), so CI supplies full coverage.
Existing expectations changed because they asserted the "wait for a
person" behavior this PR replaces:
- `issue-monitor-scheduler.test.ts`: "explains held monitor dispatch
without replaying uncertain execution" is now "continues a held task in
a fresh run when its monitor is due". A due monitor on a task held by a
`process_lost` hold now starts one fresh run instead of deferring.
- `agent-run-authentication.test.ts`: the auth row also reads
`freshContinuationSupersededAt` (still under 512 bytes), and a
superseded run is refused.
- `assigned-work-invariant.test.ts` (#21139): "with the real wake queue,
a hold that defers the continuation leaves a violation…" is now "…a hold
reached by the sweep's wake becomes one pending fresh continuation". The
sweep's wake now yields one pending fresh continuation, and the next
sweep adds no attempt.
- `heartbeat-native-restart-suspension.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: awaiting-evidence
dispositions now carry a `retryAt`, because every wait is rechecked. A
lapsed unverifiable controller no longer waits forever.
- `native-remote-retry-admission.test.ts`: the blocked-hold text is now
"This run stopped before it finished. Your saved work is kept." The old
text asked people to reconcile.
Holdless legacy runs: in the 1,000 staging runs, 0 tasks end on
`controller_lease_lost` or `process_lost` (every one has a later run).
12 end on `server_shutdown_interrupted` with no retry scheduled (attempt
0). On staging now, 9 of those 12 sit `in_progress` with no run and no
hold, 1 is done and 2 have a live run. Those 9 belong to the idle-task
sweep (#21139).
## Risks
- Speed: nothing is added to the normal path before `turn.start`. The
fallback runs only on the disposition worker after exact recovery is
ruled out, or at wake admission when a hold already defers the wake. The
auth middleware reads one more JSON field in the row read it already
does.
- Behavior shift: tasks that today stay Blocked for a person after an
infrastructure failure now continue automatically. A side effect with an
unknown outcome is not replayed; the new run is told about it and must
check first.
- After deploy, the disposition worker continues tasks already blocked
by these codes, at most 25 per pass and at most 6 per task per hour. No
migration. No manual Retry is needed.
- The old executor is not proven stopped before the successor starts.
Fencing is: its API writes are refused, its workspace epoch is
abandoned, the native `replacementDenied` marker is set, and the
successor has a separate run identity.
Known gaps (follow-ups):
- No quarantine attachment: unverified files stay unapplied in the
abandoned workspace root (recorded on the workspace in
`remoteWorkspaceAbandonedEpochs`). They are not read into a task
attachment, because the sandbox is usually unreachable in these
failures.
- Agent Chat conversations are excluded at wake admission. They keep
their owner-bound continuation.
- Under a `reuse_by_environment` lease policy the new allocation can be
on the same provider sandbox. It always uses a new root and a new epoch,
and the old run cannot publish or write.
- Tasks left `in_progress` with no run and no hold (for example after
`server_shutdown_interrupted` with no retry scheduled) belong to the
idle-task sweep. That sweep's wake reaches this fallback when a hold
exists.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
5507 lines
255 KiB
TypeScript
5507 lines
255 KiB
TypeScript
import { QUEUED_MESSAGES_DELIVERED_REASON } from "../../modules/wake-queue/index.js";
|
|
import { connectionIntentService } from "../connection-intents.js";
|
|
import { verifyBlockerStallWake } from "../recovery/blocker-stall-wakeups.js";
|
|
import { CONTEXT_COMPACTION_WAKE_REASON, ContextCompactionError, isContextCompactionRun, readContextCompactionRequest } from "../context-compaction.js";
|
|
import { isProcessRunAdmissionClosed } from "./run-admission.js";
|
|
import { continueFreshInTransaction, hasPendingFreshContinuation, settleCoalescedFreshContinuationHolds } from "../fresh-continuation-fallback.js";
|
|
import { refreshRunStartYieldIfStale } from "../run-start-handover.js";
|
|
import { deliveredQueuedCommentIds } from "../queued-comment-delivery.js";
|
|
import { hasAutomaticInteractionDeliveryHold, INTERACTION_DELIVERY_DENIED, recheckQueuedRunInteractions, withInteractionDeliveryOrigin } from "../agent-interaction-delivery.js";
|
|
import { deferredWorkIntentWait } from "../work-intent-outbox.js";
|
|
import { notifyIssueRecovery } from "../issue-recovery-notifications.js";
|
|
import { createHeartbeatTimerPolicy } from "./timer-policy.js";
|
|
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
|
|
import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../deferred-execution-wait.js";
|
|
import { notifyDeferredExecutionWork } from "../deferred-execution-notifications.js";
|
|
import type { DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js";
|
|
import { notifyRunDispatch } from "../run-dispatch-notifications.js";
|
|
import { notifyStatusCardIssueChange } from "../status-card-notifications.js";
|
|
import { notifyRoutineExecutionStatus } from "../routine-execution-status-notifications.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
|
|
import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js";
|
|
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
|
|
import { notifyUnsafeWorkspaceExportRecovery, notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
|
|
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
|
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
|
|
import { notifyScheduledRetryCancellation } from "../scheduled-retry-cancellation-notifications.js";
|
|
import { notifyChatResponseAvailability } from "../chat-response-availability.js";
|
|
import { notifyChatInboundWakeNotices } from "../chat-inbound-wake-notice-notifications.js";
|
|
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
|
|
import { assertTaskRetryConversationAllowed, readTaskRetryIntent, readTaskRetryBoardMessageOrigin, retireSupersededTaskRetry, taskRetryMutationAllowed } from "../task-retry.js";
|
|
import { insertIssueComment } from "../issue-comment-writes.js";
|
|
import { withExternalAdmissionGuard, assertNoExternalOverlap } from "../../modules/external-agents/index.js";
|
|
import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js";
|
|
import { normalizeAgentNameKey } from "./retries.js";
|
|
import { publishActiveDotComment } from "../dot-assignment-follow-up.js";
|
|
import { publishActiveMuseComment } from "../muse-assignment-follow-up.js";
|
|
import {
|
|
type WakeupOptions,
|
|
mergeCoalescedContextSnapshot,
|
|
enrichWakeContextSnapshot,
|
|
hasInteractionContinuationWakeContext,
|
|
isInteractionResolutionWakePayload,
|
|
} from "./run-preparation.js";
|
|
import { resolveExecutionWorkspaceReuseRequestForIssue } from "./workspaces.js";
|
|
import {
|
|
CHAT_COMPLETION_WAKE_REASON,
|
|
isCompletedOnboardingHandoffWake,
|
|
} from "../chat-completion-delivery.js";
|
|
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
|
|
import { isVerifiedDependencyInteractionResponse, readQueuedInteractionResponse } from "../queued-interaction-response.js";
|
|
import {
|
|
isConversation,
|
|
isConversationReset,
|
|
isConversationExecutionWake,
|
|
isWaitingConversation,
|
|
} from "../agent-conversations.js";
|
|
import { legacyControllerClaim } from "../legacy-controller-lease.js";
|
|
import {
|
|
admitExplicitNativeContinuation,
|
|
undeliveredLegacyUserCommentIds,
|
|
} from "../explicit-native-continuation.js";
|
|
import { canRetryStoppedRun } from "../cancelled-native-startup.js";
|
|
import { getExecutionBlocker, hasPendingRunCleanup } from "../execution-blocker.js";
|
|
import { nativeManualAdmissionHoldsExecution } from "../native-runtime/native-runner-ownership.js";
|
|
import { taskChatRetrySourceCleanupPending } from "../chat-channels/task-retry-authority.js";
|
|
import { validateNativeWorkspaceReconstruction } from "../native-runtime/native-workspace-reconstruction.js";
|
|
import { isConversationAdapter } from "../conversation-continuation.js";
|
|
import { recordExecutionWait } from "../execution-wait.js";
|
|
import {
|
|
getNativeReviewAssignment,
|
|
readNativeReviewAssignmentContext,
|
|
} from "../native-runtime/native-review-participant.js";
|
|
import { claimQueuedNativeReviewRun } from "../native-runtime/native-review-dispatch.js";
|
|
import { nativeCleanupHoldsAdmission } from "../native-runtime/native-cleanup-admission-hold.js";
|
|
import { adapterExecutionControls } from "../adapter-execution-control.js";
|
|
import { preserveTaskRemoteWorkspaceWriter } from "../native-runtime/native-failed-workspace-recovery.js";
|
|
import { executionFailureRetryCount } from "../execution-recovery-attempt.js";
|
|
import {
|
|
slackMentionAllowsRunStart,
|
|
resumeDeferredChatWake,
|
|
assertDurableChatWakeupReceipt,
|
|
assertDurableChatWakeupRequest,
|
|
withDurableChatAdmission,
|
|
authorizeFailedChatRunRetryWake,
|
|
FailedChatRunRetryAuthorizationError,
|
|
unadmittedChatWakeupCondition,
|
|
} from "../durable-chat-wakeup.js";
|
|
import { agentService } from "../agents.js";
|
|
import { normalizeLegacyRunnerProvider } from "@paperclipai/adapter-utils";
|
|
import { randomUUID } from "node:crypto";
|
|
import {
|
|
and,
|
|
asc,
|
|
desc,
|
|
eq,
|
|
gt,
|
|
gte,
|
|
inArray,
|
|
isNull,
|
|
lte,
|
|
ne,
|
|
notInArray,
|
|
or,
|
|
sql,
|
|
} from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
agents,
|
|
caseHandoffs,
|
|
cases,
|
|
agentWakeupRequests,
|
|
activityLog,
|
|
budgetIncidents,
|
|
chatConversations,
|
|
chatEndpoints,
|
|
companies,
|
|
executionWorkspaces,
|
|
heartbeatRunEvents,
|
|
heartbeatRuns,
|
|
issueComments,
|
|
issueRecoveryActions,
|
|
issueRelations,
|
|
issues,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
conflict,
|
|
HttpError,
|
|
notFound,
|
|
} from "../../errors.js";
|
|
import { logger } from "../../middleware/logger.js";
|
|
import { isUniqueViolation } from "../../db-errors.js";
|
|
import { publishLiveEvent } from "../live-events.js";
|
|
import { allocateHeartbeatRunEventSeq } from "../heartbeat-run-events.js";
|
|
import {
|
|
queuedCommentIdsFromWakePayload,
|
|
queuedCommentIdsFromRunContext,
|
|
withQueuedCommentIdsInWakePayload,
|
|
withQueuedCommentIdsInRunContext,
|
|
} from "../issue-queued-comment-queue.js";
|
|
import { parseObject } from "../../adapters/utils.js";
|
|
import { isExternalChatWaitAuthorizationContention } from "../native-runtime/chat-attachment-reuse.js";
|
|
import { emitAgentTaskRun } from "../agent-task-run-telemetry.js";
|
|
import {
|
|
CHAT_CONTROL_RECOVERY_ADMISSION_KEY,
|
|
CHAT_CONTROL_RECOVERY_STOP_CODE,
|
|
CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE,
|
|
chatControlRecoveryAdmission,
|
|
readChatControlRecoveryAdmission,
|
|
readChatControlNativeParent,
|
|
readChatControlRecoveryStop,
|
|
} from "../chat-control-recovery-stop.js";
|
|
import {
|
|
ISSUE_NEW_INPUT_ACTIVITY_ACTIONS,
|
|
ISSUE_PROGRESS_ACTIVITY_ACTIONS,
|
|
ISSUE_REWAKE_LOOKBACK_MS,
|
|
ISSUE_REWAKE_RUN_SAMPLE_LIMIT,
|
|
evaluateIssueRewakeThrottle,
|
|
isThrottleCandidateIssueRewake,
|
|
NATIVE_STATUS_COMMITTER_ACTOR_ID,
|
|
} from "../issue-rewake-throttle.js";
|
|
import { logActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
|
|
import {
|
|
ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
|
|
isDependencyResolvedWakePayload,
|
|
dependencyResolvedWakePayloadSql,
|
|
} from "../issue-dependency-wakeups.js";
|
|
import { resumeIfPauseAlreadyReleased } from "../issue-tree-resume.js";
|
|
import {
|
|
ISSUE_PAUSED_ERROR_CODE,
|
|
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
|
|
isVerifiedIssueTreeControlInteractionWake,
|
|
pauseSuppressionResult,
|
|
} from "../issue-tree-control.js";
|
|
import {
|
|
buildExecutionWorkspaceAdapterConfig,
|
|
isUnrunnableWorktreeCombo,
|
|
parseIssueExecutionWorkspaceSettings,
|
|
resolveEffectiveWorkspaceStrategyType,
|
|
resolveExecutionWorkspaceMode,
|
|
WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
|
|
WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE,
|
|
WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION,
|
|
} from "../execution-workspace-policy.js";
|
|
import { readContinuationAttempt } from "../recovery/index.js";
|
|
import { allowsDependencyBlockedWake, wakeChatConversationsConflict } from "../../modules/run-dispatch/index.js";
|
|
import { withAgentStartLock } from "../agent-start-lock.js";
|
|
import {
|
|
evaluateAgentInvokability,
|
|
shouldCancelRunsForNonInvokableAgent,
|
|
type AgentOrgRow,
|
|
} from "../agent-invokability.js";
|
|
import { supportsUnscopedTimerHeartbeat } from "../heartbeat-policy.js";
|
|
import { isUuidLike } from "@paperclipai/shared";
|
|
import { retryChatControlAdmission } from "../chat-control-admission-retry.js";
|
|
|
|
import type { HeartbeatServiceOptions } from "../heartbeat.js";
|
|
import type { createHeartbeatRunState } from "./run-state.js";
|
|
import type { createHeartbeatRunPreparation } from "./run-preparation.js";
|
|
import type { HeartbeatRetryDependencies } from "./retries.js";
|
|
import type { HeartbeatRecoveryDependencies, createHeartbeatRecovery } from "./recovery.js";
|
|
import { executeIssuePostCommitActions, type IssuePostCommitAction, type issueService } from "../issues.js";
|
|
import type { issueTreeControlService } from "../issue-tree-control.js";
|
|
import type { budgetService } from "../budgets.js";
|
|
import { isBoardOwnedBudgetBlock } from "../budget-block-ownership.js";
|
|
import type { instanceSettingsService } from "../instance-settings.js";
|
|
import type { createRunDispatch } from "../../modules/run-dispatch/index.js";
|
|
import { hasRetainedExternalChatRetry, type PostCommitEffect as WakeQueuePostCommitEffect, type createWakeQueue } from "../../modules/wake-queue/index.js";
|
|
import { crashPoint } from "@paperclipai/shared/crash-points";
|
|
|
|
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
|
|
type HeartbeatRunState = ReturnType<typeof createHeartbeatRunState>;
|
|
type RunPreparation = ReturnType<typeof createHeartbeatRunPreparation>;
|
|
|
|
/** Queue policy uses the service's lifecycle effects and process-wide ownership sets. */
|
|
export interface HeartbeatQueueDependencies extends Pick<HeartbeatRetryDependencies,
|
|
"setWakeupStatus" | "appendRunEvent" | "finalizeAgentStatus" |
|
|
"getAgentInvokability" | "getWorktreeExecutionCutoff" | "applyRunDispatchPostCommitEffects">,
|
|
Pick<RunPreparation, "toAgentOrgRow" | "listCompanyAgentOrgRows" |
|
|
"resolveResponsibleUserIdForRun" | "getIssueExecutionContext" |
|
|
"getRoutineEnvForExecutionIssue" | "resolveResponsibleUserIdForRunSeed">,
|
|
Pick<HeartbeatRunState, "getAgent" | "getRun" | "resolveExplicitResumeSessionOverride" |
|
|
"resolveSessionBeforeForWakeup" | "hasResolvablePriorSessionWorkspaceForWake">,
|
|
Pick<HeartbeatRecoveryDependencies, "setRunStatus"> {
|
|
sweepPendingCleanupLeases: ReturnType<typeof createHeartbeatRecovery>["sweepPendingCleanupLeases"];
|
|
options: Pick<HeartbeatServiceOptions, "beforeChatControlRecoveryCheck">;
|
|
issuesSvc: Pick<ReturnType<typeof issueService>, "getDependencyReadiness" | "listDependencyReadiness" | "updateForCompany">;
|
|
treeControlSvc: Pick<ReturnType<typeof issueTreeControlService>, "getActivePauseHoldGate">;
|
|
budgets: Pick<ReturnType<typeof budgetService>, "getInvocationBlock">;
|
|
instanceSettings: Pick<ReturnType<typeof instanceSettingsService>, "getExperimental">;
|
|
runDispatch: Pick<ReturnType<typeof createRunDispatch>, "cancelStaleQueuedRun">;
|
|
wakeQueue: Pick<ReturnType<typeof createWakeQueue>, "createAdmissionTransactionScope" | "admitWakeBehindIssueExecution">;
|
|
activeRunExecutions: Set<string>;
|
|
activeRunExecutionPromises: Set<Promise<void>>;
|
|
activeWakeupPromises: Set<Promise<unknown>>;
|
|
liveRunExecutions: { has(id: string): boolean };
|
|
isHeartbeatRunTerminalStatus: (status: string | null | undefined) => boolean;
|
|
isSameTaskScope: (left: string | null, right: string | null) => boolean;
|
|
runTaskKey: (run: HeartbeatRun) => string | null;
|
|
filterZombieCoalesceTarget: <T extends { id: string; status: string }>(run: T | null, live: { has(id: string): boolean }) => T | null;
|
|
getSchedulingSuppression: () => Promise<{ suppressed: boolean; reason: "worktree_instance" | "database_restore_in_progress" | "task_drain" | null }>;
|
|
isRunAdmissionClosed?: () => boolean;
|
|
/** Before a dispatch pass, refresh admission inputs that have gone stale; null when current. */
|
|
refreshRunAdmission?: () => Promise<void> | null;
|
|
canStartNewExternalBotWork: (agent: typeof agents.$inferSelect) => Promise<boolean>;
|
|
executeRun: (runId: string) => Promise<void>;
|
|
releaseIssueExecutionAndPromote: (run: Pick<HeartbeatRun, "id" | "companyId">, options?: { suppressImmediateRecovery?: boolean; resumeStrandedQueue?: boolean; deferredPostCommitEffects?: WakeQueuePostCommitEffect[] }) => Promise<unknown>;
|
|
applyWakeQueuePostCommitEffects: (effects: WakeQueuePostCommitEffect[]) => Promise<void>;
|
|
publishRunLifecyclePluginEvent: (run: HeartbeatRun) => void;
|
|
cancelRunInternal: (runId: string, reason?: string,
|
|
options?: { errorCode?: string; resultJson?: Record<string, unknown> }) => Promise<unknown>;
|
|
cancelActiveForAgentInternal: (agentId: string, reason: string) => Promise<unknown>;
|
|
resumeExecutionWaitComments: (scope?: DeferredReceiptRecoveryOptions) => Promise<unknown>;
|
|
resumeQueuedCommentInterrupt: (companyId: string, queueId: string) => Promise<unknown>;
|
|
resumeSavedLegacyComments: (companyId: string, queueId: string) => Promise<unknown>;
|
|
formatIssueIdentifierLink: (identifier: string | null, fallback: string) => string;
|
|
}
|
|
|
|
|
|
|
|
|
|
export const DEFERRED_WAKE_CONTEXT_KEY = "_paperclipWakeContext";
|
|
|
|
export const EXECUTION_PATH_HEARTBEAT_RUN_STATUSES = [
|
|
"queued",
|
|
"running",
|
|
"scheduled_retry",
|
|
] as const;
|
|
|
|
|
|
export class ChatControlRecoveryUnresolvedError extends Error {
|
|
constructor() {
|
|
super(
|
|
"Run admission could not acquire its database locks after bounded retries. No provider work started. Review database contention and send a fresh request; this attempt will not automatically retry.",
|
|
);
|
|
}
|
|
}
|
|
|
|
const RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP = new Set([
|
|
CHAT_COMPLETION_WAKE_REASON,
|
|
"approval_approved",
|
|
ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
|
|
"issue_recovery_action_restored",
|
|
]);
|
|
|
|
|
|
export function shouldRequireIssueCommentForWake(
|
|
contextSnapshot: Record<string, unknown> | null | undefined,
|
|
) {
|
|
if (contextSnapshot?.skipIssueComment === true) return false;
|
|
|
|
const wakeReason = readNonEmptyString(contextSnapshot?.wakeReason);
|
|
return (
|
|
wakeReason === "issue_assigned" ||
|
|
wakeReason === "execution_review_requested" ||
|
|
wakeReason === "execution_approval_requested" ||
|
|
wakeReason === "execution_changes_requested"
|
|
);
|
|
}
|
|
|
|
async function listUnresolvedBlockerSummaries(
|
|
dbOrTx: Pick<Db, "select">,
|
|
companyId: string,
|
|
issueId: string,
|
|
unresolvedBlockerIssueIds: string[],
|
|
) {
|
|
const ids = [...new Set(unresolvedBlockerIssueIds.filter(Boolean))];
|
|
if (ids.length === 0) return [];
|
|
return dbOrTx
|
|
.select({
|
|
id: issues.id,
|
|
identifier: issues.identifier,
|
|
title: issues.title,
|
|
status: issues.status,
|
|
priority: issues.priority,
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
assigneeUserId: issues.assigneeUserId,
|
|
})
|
|
.from(issueRelations)
|
|
.innerJoin(issues, eq(issueRelations.issueId, issues.id))
|
|
.where(
|
|
and(
|
|
eq(issueRelations.companyId, companyId),
|
|
eq(issueRelations.type, "blocks"),
|
|
eq(issueRelations.relatedIssueId, issueId),
|
|
inArray(issues.id, ids),
|
|
),
|
|
)
|
|
.orderBy(asc(issues.title));
|
|
}
|
|
|
|
export function shouldDeferFollowupWakeForSameIssue(input: {
|
|
activeRunStatus: string | null | undefined;
|
|
isSameExecutionAgent: boolean;
|
|
wakeCommentId: string | null | undefined;
|
|
forceFreshSession: boolean;
|
|
}) {
|
|
// A comment follow-up or explicit fresh-session wake needs a new run boundary.
|
|
if (!input.isSameExecutionAgent) return false;
|
|
if (input.activeRunStatus !== "running") return false;
|
|
if (input.wakeCommentId) return true;
|
|
if (input.forceFreshSession) return true;
|
|
return false;
|
|
}
|
|
|
|
export function shouldQueueFollowupForRunningIssueWake(input: {
|
|
contextSnapshot: Record<string, unknown> | null | undefined;
|
|
wakeCommentId: string | null;
|
|
}) {
|
|
if (input.wakeCommentId) return true;
|
|
// A structured interaction response is new input just like a comment. It
|
|
// must run after the turn that created the interaction instead of being
|
|
// merged into that still-running turn.
|
|
if (
|
|
readNonEmptyString(input.contextSnapshot?.interactionId) &&
|
|
readNonEmptyString(input.contextSnapshot?.interactionStatus)
|
|
) {
|
|
return true;
|
|
}
|
|
const wakeReason = readNonEmptyString(input.contextSnapshot?.wakeReason);
|
|
if (wakeReason === "issue_children_completed" && (
|
|
input.contextSnapshot?.onboardingCompletion === true ||
|
|
(input.contextSnapshot?.statusDecisionSource === "native_status_decision" &&
|
|
readNonEmptyString(input.contextSnapshot?.nativeChildCompletionDecisionId))
|
|
)) return true;
|
|
return Boolean(
|
|
wakeReason && RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP.has(wakeReason),
|
|
);
|
|
}
|
|
|
|
/** Agent Chat outbox receipts. The outbox, not a skipped receipt, owns a capped message. */
|
|
const CONVERSATION_COMMENT_KEY_PREFIX = "conversation-comment:";
|
|
function dailyLimitConversationRefusal(cap: { reason: string; observed: number; limit: number }) {
|
|
// No receipt is written, so the person's message stays owed in the outbox
|
|
// until the limit resets (agent-conversations.ts conversationDeliveryHold).
|
|
return conflict("Agent reached its daily limit", { reason: cap.reason, observed: cap.observed, limit: cap.limit });
|
|
}
|
|
|
|
function readNonEmptyString(value: unknown): string | null {
|
|
return typeof value === "string" && value.trim().length > 0 ? value : null;
|
|
}
|
|
|
|
export async function claimDueTimerHeartbeatForAgent(
|
|
db: Db,
|
|
agent: typeof agents.$inferSelect,
|
|
now: Date,
|
|
intervalSec: number,
|
|
) {
|
|
const dueBefore = new Date(now.getTime() - intervalSec * 1000);
|
|
const claimed = await db.transaction(async tx => {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
return tx
|
|
.update(agents)
|
|
.set({ lastHeartbeatAt: now, updatedAt: now })
|
|
.where(and(
|
|
eq(agents.id, agent.id),
|
|
eq(agents.companyId, agent.companyId),
|
|
or(
|
|
lte(agents.lastHeartbeatAt, dueBefore),
|
|
and(isNull(agents.lastHeartbeatAt), lte(agents.createdAt, dueBefore)),
|
|
),
|
|
))
|
|
.returning({ id: agents.id })
|
|
.then((rows) => rows[0] ?? null);
|
|
});
|
|
if (!claimed) return null;
|
|
return { wasFirstHeartbeat: !agent.lastHeartbeatAt };
|
|
}
|
|
|
|
|
|
/** Queue admission and dispatch share the service lifecycle callbacks and execution ownership. */
|
|
// Wake admissions that have not committed their wake and run yet. Process-wide, like
|
|
// the pending-wake set: a server replacement loses these, and only these, pending wakes.
|
|
const unsavedWakeAdmissions = new Set<object>();
|
|
export function unsavedWakeAdmissionCount() { return unsavedWakeAdmissions.size; }
|
|
|
|
export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDependencies) {
|
|
const {
|
|
setRunStatus,
|
|
setWakeupStatus,
|
|
appendRunEvent,
|
|
releaseIssueExecutionAndPromote,
|
|
applyWakeQueuePostCommitEffects,
|
|
issuesSvc,
|
|
options,
|
|
publishRunLifecyclePluginEvent,
|
|
finalizeAgentStatus,
|
|
getAgent,
|
|
cancelRunInternal,
|
|
toAgentOrgRow,
|
|
getAgentInvokability,
|
|
budgets,
|
|
activeRunExecutions,
|
|
treeControlSvc,
|
|
runDispatch,
|
|
applyRunDispatchPostCommitEffects,
|
|
resolveResponsibleUserIdForRun,
|
|
getIssueExecutionContext,
|
|
isHeartbeatRunTerminalStatus,
|
|
liveRunExecutions,
|
|
getSchedulingSuppression,
|
|
isRunAdmissionClosed = isProcessRunAdmissionClosed,
|
|
refreshRunAdmission = refreshRunStartYieldIfStale,
|
|
resumeExecutionWaitComments,
|
|
getWorktreeExecutionCutoff,
|
|
resumeQueuedCommentInterrupt,
|
|
resumeSavedLegacyComments,
|
|
cancelActiveForAgentInternal,
|
|
listCompanyAgentOrgRows,
|
|
executeRun,
|
|
canStartNewExternalBotWork,
|
|
activeRunExecutionPromises,
|
|
activeWakeupPromises,
|
|
instanceSettings,
|
|
getRun,
|
|
sweepPendingCleanupLeases,
|
|
resolveExplicitResumeSessionOverride,
|
|
resolveSessionBeforeForWakeup,
|
|
hasResolvablePriorSessionWorkspaceForWake,
|
|
getRoutineEnvForExecutionIssue,
|
|
resolveResponsibleUserIdForRunSeed,
|
|
formatIssueIdentifierLink,
|
|
wakeQueue,
|
|
isSameTaskScope,
|
|
runTaskKey,
|
|
filterZombieCoalesceTarget,
|
|
} = dependencies;
|
|
|
|
const { parseHeartbeatPolicy, getHeartbeatDailyCapBlock, hasActionableTimerWork, nextTimerHeartbeatAt } = createHeartbeatTimerPolicy(db);
|
|
|
|
async function cancelQueuedRunForHeartbeatDailyCap(
|
|
run: typeof heartbeatRuns.$inferSelect,
|
|
dailyCapBlock: NonNullable<
|
|
Awaited<ReturnType<typeof getHeartbeatDailyCapBlock>>
|
|
>,
|
|
) {
|
|
const now = new Date();
|
|
const reason =
|
|
"Cancelled because the agent reached a per-day heartbeat budget cap before adapter invocation";
|
|
const cancelled = await setRunStatus(run.id, "cancelled", {
|
|
finishedAt: now,
|
|
error: reason,
|
|
errorCode: dailyCapBlock.reason,
|
|
resultJson: {
|
|
...parseObject(run.resultJson),
|
|
stopReason: dailyCapBlock.reason,
|
|
observed: dailyCapBlock.observed,
|
|
limit: dailyCapBlock.limit,
|
|
effectiveTimeoutSec: 0,
|
|
timeoutConfigured: false,
|
|
timeoutSource: "heartbeat_daily_cap_gate",
|
|
timeoutFired: false,
|
|
},
|
|
});
|
|
if (!cancelled) return null;
|
|
|
|
await setWakeupStatus(run.wakeupRequestId, "skipped", {
|
|
finishedAt: now,
|
|
error: reason,
|
|
});
|
|
await releaseConversationCommentsForDailyCap(cancelled);
|
|
|
|
await appendRunEvent(cancelled, {
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message: reason,
|
|
payload: {
|
|
reason: dailyCapBlock.reason,
|
|
observed: dailyCapBlock.observed,
|
|
limit: dailyCapBlock.limit,
|
|
},
|
|
});
|
|
|
|
await releaseIssueExecutionAndPromote(cancelled, {
|
|
suppressImmediateRecovery: true,
|
|
});
|
|
|
|
return cancelled;
|
|
}
|
|
|
|
/**
|
|
* A chat message accepted before the limit was reached must not be consumed
|
|
* when dispatch then cancels its run at the limit. Releasing the outbox key
|
|
* on its receipts (the run's own and any coalesced into it) makes the
|
|
* message owed again; the outbox holds it until the limit resets.
|
|
*/
|
|
async function releaseConversationCommentsForDailyCap(run: typeof heartbeatRuns.$inferSelect) {
|
|
const owned = and(
|
|
eq(agentWakeupRequests.companyId, run.companyId),
|
|
eq(agentWakeupRequests.agentId, run.agentId),
|
|
or(eq(agentWakeupRequests.runId, run.id),
|
|
run.wakeupRequestId ? eq(agentWakeupRequests.id, run.wakeupRequestId) : undefined),
|
|
sql`${agentWakeupRequests.idempotencyKey} like ${CONVERSATION_COMMENT_KEY_PREFIX + "%"}`,
|
|
);
|
|
await db.transaction(async tx => {
|
|
const receipts = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(owned).for("update");
|
|
if (!receipts.length) return;
|
|
// Register the recovery signal before the write it announces.
|
|
await notifyIssueRecovery(tx, { companyId: run.companyId });
|
|
await tx.update(agentWakeupRequests).set({
|
|
idempotencyKey: null,
|
|
payload: sql`coalesce(${agentWakeupRequests.payload}, '{}'::jsonb)
|
|
|| jsonb_build_object('releasedConversationCommentKey', ${agentWakeupRequests.idempotencyKey})`,
|
|
updatedAt: new Date(),
|
|
}).where(and(owned, inArray(agentWakeupRequests.id, receipts.map(row => row.id))));
|
|
});
|
|
}
|
|
|
|
async function markTimerHeartbeatChecked(
|
|
agentId: string,
|
|
source: WakeupOptions["source"],
|
|
) {
|
|
if (source !== "timer") return;
|
|
await db.transaction(async tx => {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
const now = new Date();
|
|
await tx.update(agents).set({ lastHeartbeatAt: now, updatedAt: now })
|
|
.where(eq(agents.id, agentId));
|
|
});
|
|
}
|
|
|
|
const claimDueTimerHeartbeat = (agent: typeof agents.$inferSelect, now: Date, intervalSec: number) =>
|
|
claimDueTimerHeartbeatForAgent(db, agent, now, intervalSec);
|
|
|
|
function issueRunPriorityRank(priority: string | null | undefined) {
|
|
switch (priority) {
|
|
case "critical":
|
|
return 0;
|
|
case "high":
|
|
return 1;
|
|
case "medium":
|
|
return 2;
|
|
case "low":
|
|
return 3;
|
|
default:
|
|
return 4;
|
|
}
|
|
}
|
|
|
|
async function listQueuedRunDependencyReadiness(
|
|
companyId: string,
|
|
queuedRuns: Array<typeof heartbeatRuns.$inferSelect>,
|
|
) {
|
|
const issueIds = [
|
|
...new Set(
|
|
queuedRuns
|
|
.map((run) =>
|
|
readNonEmptyString(parseObject(run.contextSnapshot).issueId),
|
|
)
|
|
.filter((issueId): issueId is string => Boolean(issueId)),
|
|
),
|
|
];
|
|
if (issueIds.length === 0) {
|
|
return new Map<
|
|
string,
|
|
Awaited<ReturnType<typeof issuesSvc.getDependencyReadiness>>
|
|
>();
|
|
}
|
|
return issuesSvc.listDependencyReadiness(companyId, issueIds);
|
|
}
|
|
|
|
async function countRunningRunsForAgent(agentId: string) {
|
|
const [{ count }] = await db
|
|
.select({ count: sql<number>`count(*)` })
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
eq(heartbeatRuns.status, "running"),
|
|
),
|
|
);
|
|
return Number(count ?? 0);
|
|
}
|
|
|
|
async function withChatControlRecoveryGate(
|
|
run: typeof heartbeatRuns.$inferSelect,
|
|
stage: "claim" | "dispatch",
|
|
onClear: (tx: Db) => Promise<typeof heartbeatRuns.$inferSelect | null>,
|
|
nativeRecovery = false,
|
|
) {
|
|
const issueId = readNonEmptyString(
|
|
parseObject(run.contextSnapshot).issueId,
|
|
);
|
|
if (!issueId || run.invocationSource !== "automation") return onClear(db);
|
|
await options.beforeChatControlRecoveryCheck?.({
|
|
runId: run.id,
|
|
issueId,
|
|
stage,
|
|
});
|
|
if (stage === "dispatch" && nativeRecovery) {
|
|
// Protected historical/already-admitted owners did not previously take
|
|
// this fence's locks. A fresh exact read keeps unrelated Board contention
|
|
// from turning their recovery into a new admission failure.
|
|
const [current] = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, run.id),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.agentId, run.agentId),
|
|
eq(heartbeatRuns.status, "running"),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (
|
|
current &&
|
|
current.wakeupRequestId === run.wakeupRequestId &&
|
|
parseObject(current.contextSnapshot).issueId === issueId
|
|
) {
|
|
const admission = readChatControlRecoveryAdmission(current);
|
|
const expected = readChatControlRecoveryAdmission(run);
|
|
if (
|
|
expected !== "invalid" &&
|
|
(admission === "admitted" ||
|
|
(admission === "historical" && expected === "historical"))
|
|
)
|
|
return onClear(db);
|
|
}
|
|
}
|
|
let terminal: typeof heartbeatRuns.$inferSelect | null = null;
|
|
try {
|
|
const attempt = () => db.transaction(async (tx) => {
|
|
terminal = null;
|
|
// Same queue-edit lock order, then the close committer's conversation
|
|
// row. NOWAIT releases partial locks on contention. Claim defers to the
|
|
// queue; dispatch retries this transaction before considering failure.
|
|
const [issue] = await tx
|
|
.select({ id: issues.id })
|
|
.from(issues)
|
|
.where(
|
|
and(eq(issues.id, issueId), eq(issues.companyId, run.companyId)),
|
|
)
|
|
.for("update", { noWait: true })
|
|
.limit(1);
|
|
if (!issue) return null;
|
|
if (run.wakeupRequestId)
|
|
await tx
|
|
.select({ id: agentWakeupRequests.id })
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, run.wakeupRequestId),
|
|
eq(agentWakeupRequests.companyId, run.companyId),
|
|
eq(agentWakeupRequests.agentId, run.agentId),
|
|
),
|
|
)
|
|
.for("update", { noWait: true })
|
|
.limit(1);
|
|
const [current] = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, run.id),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.agentId, run.agentId),
|
|
),
|
|
)
|
|
.for("update", { noWait: true })
|
|
.limit(1);
|
|
if (
|
|
!current ||
|
|
current.status !== (stage === "claim" ? "queued" : "running")
|
|
)
|
|
return null;
|
|
const admission = readChatControlRecoveryAdmission(current);
|
|
const expectedAdmission = readChatControlRecoveryAdmission(run);
|
|
const admissionLost =
|
|
expectedAdmission !== "historical" && admission === "historical";
|
|
if (
|
|
stage === "dispatch" &&
|
|
nativeRecovery &&
|
|
!admissionLost &&
|
|
(admission === "historical" || admission === "admitted")
|
|
)
|
|
return onClear(tx as unknown as Db);
|
|
// A new warm child can inherit a prior runner's PID during preparation.
|
|
// Those fields do not prove this run's turn was dispatched.
|
|
// Exact admitted/historical recovery retains its existing ownership;
|
|
// a marked, unadmitted bootstrap must earn admission even after crash.
|
|
const proof =
|
|
admission === "invalid" || admissionLost
|
|
? { kind: "unresolved" as const }
|
|
: await readChatControlRecoveryStop(
|
|
tx as unknown as Db,
|
|
{
|
|
companyId: run.companyId,
|
|
issueId,
|
|
agentId: run.agentId,
|
|
sourceRunId: run.id,
|
|
},
|
|
true,
|
|
);
|
|
if (proof.kind === "clear") {
|
|
if (stage === "claim" || admission === "required")
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
runnerProfileJson: {
|
|
...parseObject(current.runnerProfileJson),
|
|
[CHAT_CONTROL_RECOVERY_ADMISSION_KEY]:
|
|
chatControlRecoveryAdmission(
|
|
current,
|
|
stage === "dispatch"
|
|
? "admitted"
|
|
: admission === "admitted"
|
|
? "admitted"
|
|
: "required",
|
|
),
|
|
},
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(heartbeatRuns.id, current.id));
|
|
return onClear(tx as unknown as Db);
|
|
}
|
|
if (proof.kind === "unresolved" && stage === "claim") return null;
|
|
const now = new Date();
|
|
const stopped = proof.kind === "stopped";
|
|
const code = stopped
|
|
? CHAT_CONTROL_RECOVERY_STOP_CODE
|
|
: CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE;
|
|
const error = stopped
|
|
? "Automatic continuation stopped by the committed chat conversation close. Send a new request in chat or on the Board to start fresh work."
|
|
: "Automatic continuation source could not be verified before provider admission. Review the task and send a fresh request; this attempt will not automatically retry.";
|
|
await notifyChatRunMilestones(tx, { companyId: current.companyId, issueId });
|
|
await notifyInstructionCollection(tx, { companyId: current.companyId, runId: current.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: current.companyId, runId: current.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: current.companyId, runId: current.id });
|
|
await notifyExecutionRunWork(tx, { companyId: current.companyId, agentId: current.agentId, runId: current.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, current);
|
|
[terminal] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: stopped ? "cancelled" : "failed",
|
|
errorCode: code,
|
|
error,
|
|
finishedAt: now,
|
|
resultJson: {
|
|
...parseObject(current.resultJson),
|
|
automaticRecovery: {
|
|
code,
|
|
providerDispatched: false,
|
|
...(stopped
|
|
? {
|
|
sourceRunId: proof.sourceRunId,
|
|
conversationId: proof.conversationId,
|
|
publicationId: proof.publicationId,
|
|
}
|
|
: {}),
|
|
},
|
|
},
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, current.id),
|
|
eq(heartbeatRuns.status, current.status),
|
|
),
|
|
)
|
|
.returning();
|
|
if (!terminal) return null;
|
|
if (current.wakeupRequestId)
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: stopped ? "skipped" : "failed",
|
|
error,
|
|
finishedAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, current.wakeupRequestId),
|
|
eq(agentWakeupRequests.companyId, current.companyId),
|
|
eq(agentWakeupRequests.agentId, current.agentId),
|
|
eq(agentWakeupRequests.runId, current.id),
|
|
ne(agentWakeupRequests.status, "cancelled"),
|
|
),
|
|
);
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, current.companyId),
|
|
eq(issues.id, issueId),
|
|
eq(issues.executionRunId, current.id),
|
|
),
|
|
);
|
|
return null;
|
|
});
|
|
const result = stage === "dispatch"
|
|
? await retryChatControlAdmission(attempt)
|
|
: await attempt();
|
|
if (terminal) {
|
|
const settled = terminal as typeof heartbeatRuns.$inferSelect;
|
|
publishLiveEvent({
|
|
companyId: settled.companyId,
|
|
type: "heartbeat.run.status",
|
|
payload: {
|
|
runId: settled.id,
|
|
agentId: settled.agentId,
|
|
status: settled.status,
|
|
errorCode: settled.errorCode,
|
|
error: settled.error,
|
|
},
|
|
});
|
|
publishRunLifecyclePluginEvent(settled);
|
|
if (stage === "dispatch")
|
|
await finalizeAgentStatus(settled.agentId, "cancelled");
|
|
}
|
|
return result;
|
|
} catch (error) {
|
|
if (!isExternalChatWaitAuthorizationContention(error)) throw error;
|
|
if (stage === "dispatch") {
|
|
// No effect was admitted. Let the existing setup-failure path settle
|
|
// this attempt distinctly; it must not become a successful close.
|
|
throw new ChatControlRecoveryUnresolvedError();
|
|
}
|
|
return null;
|
|
}
|
|
}
|
|
|
|
async function claimQueuedRun(
|
|
run: typeof heartbeatRuns.$inferSelect,
|
|
companyAgents?: AgentOrgRow[],
|
|
deferredPostCommitEffects?: WakeQueuePostCommitEffect[],
|
|
deferredCancellations?: Array<() => Promise<void>>,
|
|
) {
|
|
if (run.status !== "queued") return run;
|
|
const interactionDelivery = await recheckQueuedRunInteractions(db, run);
|
|
if (interactionDelivery.run.status !== "queued") return null;
|
|
run = { ...run, contextSnapshot: interactionDelivery.run.contextSnapshot,
|
|
resultJson: interactionDelivery.run.resultJson, updatedAt: interactionDelivery.run.updatedAt };
|
|
if (interactionDelivery.denied) {
|
|
const cancelled = await setRunStatus(run.id, "cancelled", {
|
|
finishedAt: new Date(), error: "Queued agent input is no longer permitted.",
|
|
errorCode: INTERACTION_DELIVERY_DENIED,
|
|
});
|
|
if (cancelled) {
|
|
await setWakeupStatus(run.wakeupRequestId, "skipped", { finishedAt: new Date(), error: INTERACTION_DELIVERY_DENIED });
|
|
await appendRunEvent(cancelled, { eventType: "lifecycle", stream: "system", level: "warn",
|
|
message: "Queued agent input was refused after its interaction permission changed.", payload: { code: INTERACTION_DELIVERY_DENIED, denials: interactionDelivery.denials } });
|
|
await releaseIssueExecutionAndPromote(cancelled, { suppressImmediateRecovery: true, deferredPostCommitEffects });
|
|
}
|
|
return null;
|
|
}
|
|
const [caseHandoffReceipt] = run.wakeupRequestId ? await db.select().from(caseHandoffs).where(and(
|
|
eq(caseHandoffs.wakeupRequestId, run.wakeupRequestId), eq(caseHandoffs.companyId, run.companyId),
|
|
)).limit(1) : [];
|
|
if (caseHandoffReceipt) {
|
|
const [current] = await db.select({ version: cases.handoffVersion, issueId: cases.workIssueId,
|
|
agentId: issues.assigneeAgentId, statusVersion: issues.statusVersion }).from(cases)
|
|
.innerJoin(issues, eq(issues.id, cases.workIssueId)).where(and(eq(cases.id, caseHandoffReceipt.caseId),
|
|
eq(cases.companyId, run.companyId), eq(issues.companyId, run.companyId))).limit(1);
|
|
if (!["pending_dispatch", "delivered"].includes(caseHandoffReceipt.state) || caseHandoffReceipt.toAgentId !== run.agentId ||
|
|
!current || current.version !== caseHandoffReceipt.version || current.issueId !== caseHandoffReceipt.issueId ||
|
|
current.agentId !== run.agentId || (!run.startedAt && current.statusVersion !== caseHandoffReceipt.appliedIssueStatusVersion)) {
|
|
const cancelled = await db.transaction(async tx => {
|
|
const [currentRun] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued")))
|
|
.for("update");
|
|
if (!currentRun) return null;
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, runId: run.id,
|
|
issueId: run.nativeIssueId ?? run.issueId ?? readNonEmptyString(parseObject(run.contextSnapshot).issueId) ?? undefined });
|
|
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExecutionRunWork(tx, { companyId: run.companyId, agentId: run.agentId, runId: run.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, run);
|
|
const [row] = await tx.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date(),
|
|
errorCode: "case_handoff_superseded", error: "This case handoff was cancelled or superseded.", updatedAt: new Date() })
|
|
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued"))).returning();
|
|
return row;
|
|
});
|
|
if (cancelled) await setWakeupStatus(run.wakeupRequestId, "cancelled", { finishedAt: new Date(), error: cancelled.error });
|
|
return null;
|
|
}
|
|
if (!(await instanceSettings.getExperimental()).enableCases) return null;
|
|
}
|
|
const caseIssueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
|
|
if (caseIssueId) {
|
|
const [pending] = await db.select().from(caseHandoffs).where(and(eq(caseHandoffs.companyId, run.companyId), eq(caseHandoffs.issueId, caseIssueId),
|
|
inArray(caseHandoffs.state, ["awaiting_release", "pending_dispatch"]))).limit(1);
|
|
if (pending && (pending.wakeupRequestId !== run.wakeupRequestId || pending.state === "awaiting_release")) return null;
|
|
}
|
|
// Claim is the first execution boundary. startedAt survives same-run
|
|
// retries; restart recovery may also enter executeRun already running.
|
|
// Neither may reinterpret an admitted, already-started message as new work.
|
|
if (!run.startedAt && !(await slackMentionAllowsRunStart(db, run))) {
|
|
const now = new Date();
|
|
const reason = "Message did not @mention the bot";
|
|
// This runs under the agent start lock. Do not use cancelRunInternal,
|
|
// which recursively starts the next run and reacquires that lock.
|
|
const cancelled = await setRunStatus(run.id, "cancelled", {
|
|
finishedAt: now,
|
|
error: reason,
|
|
errorCode: "chat_mention_required",
|
|
});
|
|
if (cancelled) {
|
|
await setWakeupStatus(run.wakeupRequestId, "skipped", { finishedAt: now, error: reason });
|
|
await appendRunEvent(cancelled, {
|
|
eventType: "lifecycle", stream: "system", level: "warn", message: reason,
|
|
});
|
|
await releaseIssueExecutionAndPromote(cancelled, {
|
|
suppressImmediateRecovery: true, deferredPostCommitEffects,
|
|
});
|
|
}
|
|
return null;
|
|
}
|
|
|
|
const agent = await getAgent(run.agentId);
|
|
if (!agent) {
|
|
await cancelRunInternal(
|
|
run.id,
|
|
"Cancelled because the agent no longer exists",
|
|
);
|
|
return null;
|
|
}
|
|
// Keep accepted work in the durable queue until setup completes.
|
|
if (isAgentAwaitingSetup(agent)) return null;
|
|
const invokability = companyAgents
|
|
? evaluateAgentInvokability(toAgentOrgRow(agent), companyAgents)
|
|
: await getAgentInvokability(agent);
|
|
if (!invokability.invokable) {
|
|
if (caseHandoffReceipt) return null;
|
|
await cancelRunInternal(
|
|
run.id,
|
|
`Cancelled because the agent is not invokable: ${invokability.reason}`,
|
|
);
|
|
return null;
|
|
}
|
|
if (!run.startedAt && !await canStartNewExternalBotWork(agent)) return null;
|
|
|
|
const context = parseObject(run.contextSnapshot);
|
|
const budgetBlock = await budgets.getInvocationBlock(
|
|
run.companyId,
|
|
run.agentId,
|
|
{
|
|
issueId: readNonEmptyString(context.issueId),
|
|
projectId: readNonEmptyString(context.projectId),
|
|
},
|
|
);
|
|
if (budgetBlock) {
|
|
if (caseHandoffReceipt) return null;
|
|
// A temporary hold (accounting that is still settling) only delays
|
|
// accepted work. Cancelling the unstarted run would drop the user's
|
|
// message, Retry, or recovery it carries; keep it queued. Settling
|
|
// accounting re-notifies run dispatch. A reached hard stop is the board's.
|
|
if (!run.startedAt && !(await isBoardOwnedBudgetBlock(db, run.companyId, budgetBlock))) {
|
|
logger.info({ runId: run.id, agentId: run.agentId, companyId: run.companyId, reason: "budget_hold_settling",
|
|
scopeType: budgetBlock.scopeType }, "claimQueuedRun: queued run kept queued behind a temporary budget hold");
|
|
return null;
|
|
}
|
|
// Record which budget stops this queued run, so Raise budget and resume
|
|
// continues it (budget-resume-continuation.ts). Saved before the cancel
|
|
// so a crash between them cannot lose it; the resume matches it only
|
|
// against a recorded cancellation with this same reason.
|
|
const budgetAdmissionStop = { scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId,
|
|
reason: budgetBlock.reason.slice(0, 512) };
|
|
const stamped = await db.transaction(async tx => {
|
|
// The same commit touches the scope's open incident. A Raise budget
|
|
// and resume whose repeatable-read snapshot predates this commit then
|
|
// conflicts on that row and retries, so its resume sees the stamp
|
|
// (a still-queued stamped run is claimed and waits for this cancel).
|
|
const open = await tx.update(budgetIncidents).set({ updatedAt: new Date() }).where(and(
|
|
eq(budgetIncidents.companyId, run.companyId), eq(budgetIncidents.scopeType, budgetBlock.scopeType),
|
|
eq(budgetIncidents.scopeId, budgetBlock.scopeId), eq(budgetIncidents.status, "open")))
|
|
.returning({ id: budgetIncidents.id });
|
|
if (!open.length) return false;
|
|
await tx.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb)
|
|
|| jsonb_build_object('budgetAdmissionStop', jsonb_build_object('scopeType', ${budgetAdmissionStop.scopeType}::text,
|
|
'scopeId', ${budgetAdmissionStop.scopeId}::text, 'reason', ${budgetAdmissionStop.reason}::text))` })
|
|
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, run.status)));
|
|
return true;
|
|
});
|
|
if (!stamped) {
|
|
// No open incident: a resume may have just closed it and raised the
|
|
// budget. Read the block again; resumed work stays queued.
|
|
const current = await budgets.getInvocationBlock(run.companyId, run.agentId, {
|
|
issueId: readNonEmptyString(context.issueId), projectId: readNonEmptyString(context.projectId) });
|
|
if (!current) return null;
|
|
if (!run.startedAt && !(await isBoardOwnedBudgetBlock(db, run.companyId, current))) return null;
|
|
await cancelRunInternal(run.id, current.reason);
|
|
return null;
|
|
}
|
|
await cancelRunInternal(run.id, budgetBlock.reason, { resultJson: { budgetAdmissionStop } });
|
|
return null;
|
|
}
|
|
|
|
const dailyCapBlock = await getHeartbeatDailyCapBlock(
|
|
agent,
|
|
parseHeartbeatPolicy(agent),
|
|
{
|
|
excludeRunId: run.id,
|
|
checkRunCap: true,
|
|
checkCostCap: true,
|
|
},
|
|
);
|
|
if (dailyCapBlock) {
|
|
if (caseHandoffReceipt) return null;
|
|
await cancelQueuedRunForHeartbeatDailyCap(run, dailyCapBlock);
|
|
return null;
|
|
}
|
|
|
|
const issueId = readNonEmptyString(context.issueId);
|
|
if (issueId && activeRunExecutions.size > 0) {
|
|
// Native finalization publishes success before workspace synchronization,
|
|
// provider suspension, and lease release finish. A queued comment must
|
|
// not acquire a fresh sandbox while its predecessor still owns that work.
|
|
// The executor's finally block retries this agent after removing its owner.
|
|
const [settlingOwner] = await db.select({ id: heartbeatRuns.id })
|
|
.from(heartbeatRuns)
|
|
.where(and(
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.agentId, run.agentId),
|
|
eq(heartbeatRuns.runtimeMode, "native"),
|
|
inArray(heartbeatRuns.id, [...activeRunExecutions]),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,
|
|
))
|
|
.limit(1);
|
|
if (settlingOwner) {
|
|
logger.info({ runId: run.id, agentId: run.agentId, companyId: run.companyId, issueId,
|
|
reason: "predecessor_settling_in_process", predecessorRunId: settlingOwner.id },
|
|
"claimQueuedRun: queued run declined at the issue execution claim");
|
|
return null;
|
|
}
|
|
}
|
|
if (issueId && !run.startedAt && agent.adapterType === "paperclip_runner") {
|
|
const cleanupBlock = await nativeCleanupHoldsAdmission(db, { companyId: run.companyId, issueId });
|
|
if (cleanupBlock) {
|
|
logger.info({ runId: run.id, agentId: run.agentId, companyId: run.companyId, issueId,
|
|
reason: "native_session_cleanup_settling", cleanupBlock },
|
|
"claimQueuedRun: queued run kept queued while the task's previous session finishes cleanup");
|
|
return null;
|
|
}
|
|
}
|
|
if (issueId) {
|
|
const activePauseHold = await treeControlSvc.getActivePauseHoldGate(
|
|
run.companyId,
|
|
issueId,
|
|
);
|
|
const treeHoldInteractionWake =
|
|
activePauseHold &&
|
|
(await isVerifiedIssueTreeControlInteractionWake(db, {
|
|
companyId: run.companyId,
|
|
issueId,
|
|
agentId: run.agentId,
|
|
runId: run.id,
|
|
wakeupRequestId: run.wakeupRequestId,
|
|
contextSnapshot: context,
|
|
}));
|
|
if (activePauseHold && !treeHoldInteractionWake) {
|
|
const refuse = async () => {
|
|
// Record the covering hold so its release re-dispatches this work once.
|
|
await cancelRunInternal(
|
|
run.id,
|
|
"Cancelled because issue is held by an active subtree pause hold",
|
|
{ errorCode: ISSUE_PAUSED_ERROR_CODE, resultJson: pauseSuppressionResult(activePauseHold) },
|
|
);
|
|
await logActivity(db, {
|
|
companyId: run.companyId,
|
|
actorType: "system",
|
|
actorId: "system",
|
|
agentId: run.agentId,
|
|
runId: run.id,
|
|
action: "issue.tree_hold_run_interrupted",
|
|
entityType: "heartbeat_run",
|
|
entityId: run.id,
|
|
issueId: issueId,
|
|
details: {
|
|
issueId,
|
|
holdId: activePauseHold.holdId,
|
|
rootIssueId: activePauseHold.rootIssueId,
|
|
source: "heartbeat.claim_queued_run",
|
|
securityPrinciples: [
|
|
"Complete Mediation",
|
|
"Fail Securely",
|
|
"Secure Defaults",
|
|
],
|
|
},
|
|
});
|
|
// A release that committed before this refusal did not see it.
|
|
await resumeIfPauseAlreadyReleased(db, { companyId: run.companyId, issueId,
|
|
holdId: activePauseHold.holdId, wakeup: (agentId, opts) => trackWakeup(agentId, opts) });
|
|
};
|
|
// Cancellation promotes the agent's next queued run under the agent
|
|
// start lock, so a caller holding that lock settles it after release.
|
|
if (deferredCancellations) deferredCancellations.push(refuse);
|
|
else await refuse();
|
|
return null;
|
|
}
|
|
|
|
const dependencyReadiness = await issuesSvc.listDependencyReadiness(
|
|
run.companyId,
|
|
[issueId],
|
|
);
|
|
const readiness = dependencyReadiness.get(issueId);
|
|
const unresolvedBlockerCount = readiness?.unresolvedBlockerCount ?? 0;
|
|
// The run's wake request records who asked for it.
|
|
const blockerStallVerified = unresolvedBlockerCount > 0 && await verifyBlockerStallWake(db, {
|
|
companyId: run.companyId, issueId, contextSnapshot: context,
|
|
requestedByActorType: run.wakeupRequestId
|
|
? (await db.select({ actor: agentWakeupRequests.requestedByActorType }).from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, run.wakeupRequestId)).then(rows => rows[0]?.actor ?? null))
|
|
: null,
|
|
});
|
|
if (
|
|
unresolvedBlockerCount > 0 &&
|
|
!allowsDependencyBlockedWake(
|
|
context,
|
|
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
|
|
blockerStallVerified,
|
|
await isVerifiedDependencyInteractionResponse(db, { companyId: run.companyId,
|
|
issueId, agentId: run.agentId, context }),
|
|
)
|
|
) {
|
|
await cancelQueuedRunForBlockedDependencies(
|
|
run,
|
|
issueId,
|
|
readiness?.unresolvedBlockerIssueIds ?? [],
|
|
);
|
|
logger.info(
|
|
{ runId: run.id, issueId, unresolvedBlockerCount },
|
|
"claimQueuedRun: cancelled blocked queued run",
|
|
);
|
|
return null;
|
|
}
|
|
|
|
const staleness = await runDispatch.cancelStaleQueuedRun({
|
|
runId: run.id,
|
|
companyId: run.companyId,
|
|
expectedStatus: "queued",
|
|
});
|
|
if (staleness.outcome === "cancelled" || staleness.outcome === "deferred") {
|
|
applyRunDispatchPostCommitEffects(staleness.postCommitEffects);
|
|
logger.info(
|
|
{
|
|
runId: run.id,
|
|
issueId,
|
|
outcome: staleness.outcome,
|
|
errorCode: staleness.outcome === "cancelled" ? staleness.errorCode : undefined,
|
|
},
|
|
"claimQueuedRun: withheld queued run at the execution gate",
|
|
);
|
|
return null;
|
|
}
|
|
}
|
|
|
|
const claimedAt = new Date();
|
|
const responsibleUserId = await resolveResponsibleUserIdForRun({
|
|
run,
|
|
contextSnapshot: context,
|
|
issueContext: issueId
|
|
? await getIssueExecutionContext(run.companyId, issueId)
|
|
: null,
|
|
routineEnvContext: {
|
|
routineId: null,
|
|
env: null,
|
|
responsibleUserId: null,
|
|
},
|
|
});
|
|
// All ordinary and comment claims use the same company-scoped issue
|
|
// lock. A batch may claim several runs before executeRun tracks any owner.
|
|
async function lockIssueExecutionClaim(tx: Db) {
|
|
const [owner] = issueId ? await tx.select({
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
statusVersion: issues.statusVersion,
|
|
executionRunId: issues.executionRunId,
|
|
checkoutRunId: issues.checkoutRunId,
|
|
}).from(issues).where(and(
|
|
eq(issues.id, issueId), eq(issues.companyId, run.companyId),
|
|
)).for("update") : [];
|
|
const ownsIssue = owner?.assigneeAgentId === run.agentId &&
|
|
context.wakeReason !== "source_scoped_recovery_action";
|
|
const [handoff] = issueId ? await tx.select().from(caseHandoffs).where(and(eq(caseHandoffs.companyId, run.companyId), eq(caseHandoffs.issueId, issueId),
|
|
inArray(caseHandoffs.state, ["awaiting_release", "pending_dispatch"]))).limit(1) : [];
|
|
const declined = (reason: string, predecessorRunId?: string) => {
|
|
logger.info({ runId: run.id, agentId: run.agentId, companyId: run.companyId, issueId, reason,
|
|
...(predecessorRunId ? { predecessorRunId } : {}) }, "claimQueuedRun: queued run declined at the issue execution claim");
|
|
return { ownsIssue, blocked: true };
|
|
};
|
|
if (handoff && (handoff.wakeupRequestId !== run.wakeupRequestId || handoff.state === "awaiting_release"))
|
|
return declined("case_handoff_pending");
|
|
if (caseHandoffReceipt) {
|
|
const [current] = await tx.select({ state: caseHandoffs.state, version: caseHandoffs.version,
|
|
currentVersion: cases.handoffVersion, workIssueId: cases.workIssueId,
|
|
appliedIssueStatusVersion: caseHandoffs.appliedIssueStatusVersion,
|
|
toAgentId: caseHandoffs.toAgentId, wakeupRequestId: caseHandoffs.wakeupRequestId }).from(caseHandoffs)
|
|
.innerJoin(cases, eq(cases.id, caseHandoffs.caseId)).where(and(eq(caseHandoffs.id, caseHandoffReceipt.id),
|
|
eq(caseHandoffs.companyId, run.companyId))).limit(1);
|
|
if (!current || !["pending_dispatch", "delivered"].includes(current.state) || current.version !== current.currentVersion ||
|
|
current.workIssueId !== issueId || current.toAgentId !== run.agentId || current.wakeupRequestId !== run.wakeupRequestId ||
|
|
!ownsIssue || (!run.startedAt && current.appliedIssueStatusVersion !== owner?.statusVersion))
|
|
return declined("case_handoff_superseded");
|
|
}
|
|
if (ownsIssue && ["native_safe_replacement", "native_provider_overloaded"].includes(run.scheduledRetryReason ?? "") &&
|
|
owner.checkoutRunId && owner.checkoutRunId !== run.id) {
|
|
return declined("issue_checked_out_by_another_run", owner.checkoutRunId);
|
|
}
|
|
if (run.scheduledRetryReason === "native_provider_overloaded" && owner?.executionRunId &&
|
|
owner.executionRunId !== run.id && owner.executionRunId !== run.retryOfRunId) {
|
|
return declined("issue_execution_owned_by_another_run", owner.executionRunId);
|
|
}
|
|
if (ownsIssue && issueId && await taskChatRetrySourceCleanupPending(tx, run, issueId)) {
|
|
return declined("task_chat_retry_source_cleanup_pending");
|
|
}
|
|
const executionOwnerId = run.scheduledRetryReason === "native_provider_overloaded"
|
|
? run.retryOfRunId : ownsIssue ? owner?.executionRunId : null;
|
|
for (const previousRunId of new Set([executionOwnerId, ownsIssue ? run.retryOfRunId : null])) {
|
|
if (!previousRunId || previousRunId === run.id) continue;
|
|
const [previous] = await tx.select({ status: heartbeatRuns.status, agentId: heartbeatRuns.agentId,
|
|
issueId: heartbeatRuns.issueId, nativeIssueId: heartbeatRuns.nativeIssueId,
|
|
supersededByFreshContinuation: sql<boolean>`${heartbeatRuns.resultJson} ? 'freshContinuationSupersededAt'`,
|
|
contextIssueId: sql<unknown>`coalesce(${heartbeatRuns.contextSnapshot}->>'issueId', ${heartbeatRuns.contextSnapshot}->>'taskId')`,
|
|
})
|
|
.from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, previousRunId),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
));
|
|
if (previousRunId !== executionOwnerId && (previous?.agentId !== run.agentId ||
|
|
(previous.nativeIssueId ?? previous.issueId ?? previous.contextIssueId) !== issueId)) continue;
|
|
// A terminal result can precede workspace/lease cleanup on this or
|
|
// another controller. Local absence alone is not a release receipt.
|
|
if (!isHeartbeatRunTerminalStatus(previous?.status) ||
|
|
liveRunExecutions.has(previousRunId)) {
|
|
return declined("predecessor_still_executing", previousRunId);
|
|
}
|
|
// A fresh continuation replaced the predecessor: its writes are revoked
|
|
// and it shares no workspace with this run (a fenced remote epoch or a
|
|
// new isolated workspace), so its cleanup continues without holding it.
|
|
if (!previous?.supersededByFreshContinuation && await hasPendingRunCleanup(tx, run.companyId, previousRunId)) {
|
|
return declined("predecessor_cleanup_pending", previousRunId);
|
|
}
|
|
}
|
|
return { ownsIssue, blocked: false };
|
|
}
|
|
async function bindClaimedIssueExecution(tx: Db, ownsIssue: boolean, claimedRun: typeof heartbeatRuns.$inferSelect | null | undefined) {
|
|
if (!claimedRun || !issueId || !ownsIssue) return;
|
|
await notifyStatusCardIssueChange(tx, run.companyId);
|
|
await tx.update(issues).set({
|
|
executionRunId: claimedRun.id,
|
|
executionAgentNameKey: normalizeAgentNameKey(agent.name),
|
|
executionLockedAt: claimedAt,
|
|
updatedAt: claimedAt,
|
|
}).where(and(eq(issues.id, issueId), eq(issues.companyId, run.companyId))).catch((error: unknown) => {
|
|
// Another open task of the same routine holds its execution slot, for
|
|
// example a failed run that waits for its scheduled resume. Roll back
|
|
// this claim and keep the run queued until that slot is released.
|
|
if (isUniqueViolation(error, "issues_open_routine_execution_uq")) {
|
|
throw conflict("Another execution for this routine is already in progress");
|
|
}
|
|
throw error;
|
|
});
|
|
}
|
|
const nativeReviewContext = readNativeReviewAssignmentContext(context);
|
|
const queuedCommentIds = queuedCommentIdsFromRunContext(context);
|
|
if (
|
|
issueId &&
|
|
run.invocationSource === "automation" &&
|
|
queuedCommentIds.length > 0
|
|
)
|
|
await options.beforeChatControlRecoveryCheck?.({
|
|
runId: run.id,
|
|
issueId,
|
|
stage: "claim",
|
|
});
|
|
const queuedCommentClaim =
|
|
!nativeReviewContext && issueId && run.wakeupRequestId && queuedCommentIds.length > 0
|
|
? await db
|
|
.transaction(async (tx) => {
|
|
// Match the queue-edit lock order: issue, wake, then run. Once the
|
|
// run becomes running, a concurrent discard must observe the
|
|
// claimed wake and return an explicit conflict; if discard wins,
|
|
// this claim observes the cancelled queue and does no work.
|
|
const issueClaim = await lockIssueExecutionClaim(tx as unknown as Db);
|
|
if (issueClaim.blocked) return { kind: "stale" as const, run: null };
|
|
const wake = await tx
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, run.wakeupRequestId!),
|
|
eq(agentWakeupRequests.companyId, run.companyId),
|
|
eq(agentWakeupRequests.agentId, run.agentId),
|
|
),
|
|
)
|
|
.for("update")
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
const lockedRun = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, run.id),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.agentId, run.agentId),
|
|
),
|
|
)
|
|
.for("update")
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
if (
|
|
!wake ||
|
|
wake.status !== "queued" ||
|
|
wake.runId !== run.id ||
|
|
!lockedRun ||
|
|
lockedRun.status !== "queued" ||
|
|
lockedRun.wakeupRequestId !== wake.id
|
|
) {
|
|
return { kind: "stale" as const, run: null };
|
|
}
|
|
|
|
if (lockedRun.invocationSource === "automation") {
|
|
const admission = readChatControlRecoveryAdmission(lockedRun);
|
|
if (admission === "invalid")
|
|
return { kind: "stale" as const, run: null };
|
|
const proof = await readChatControlRecoveryStop(
|
|
tx as unknown as Db,
|
|
{
|
|
companyId: lockedRun.companyId,
|
|
issueId,
|
|
agentId: lockedRun.agentId,
|
|
sourceRunId: lockedRun.id,
|
|
},
|
|
true,
|
|
);
|
|
if (proof.kind === "unresolved")
|
|
return { kind: "stale" as const, run: null };
|
|
if (proof.kind === "stopped") {
|
|
const error =
|
|
"Automatic continuation stopped by the committed chat conversation close. Send a fresh request in chat or on the Board.";
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
|
|
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExecutionRunWork(tx, { companyId: run.companyId, agentId: run.agentId, runId: run.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, run);
|
|
const [cancelled] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
errorCode: CHAT_CONTROL_RECOVERY_STOP_CODE,
|
|
error,
|
|
finishedAt: claimedAt,
|
|
resultJson: {
|
|
...parseObject(lockedRun.resultJson),
|
|
automaticRecovery: {
|
|
code: CHAT_CONTROL_RECOVERY_STOP_CODE,
|
|
providerDispatched: false,
|
|
sourceRunId: proof.sourceRunId,
|
|
conversationId: proof.conversationId,
|
|
publicationId: proof.publicationId,
|
|
},
|
|
},
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, lockedRun.id),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
),
|
|
)
|
|
.returning();
|
|
if (!cancelled) return { kind: "stale" as const, run: null };
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "skipped",
|
|
error,
|
|
finishedAt: claimedAt,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wake.id));
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, run.companyId),
|
|
eq(issues.executionRunId, run.id),
|
|
),
|
|
);
|
|
return { kind: "cancelled" as const, run: cancelled };
|
|
}
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
runnerProfileJson: {
|
|
...parseObject(lockedRun.runnerProfileJson),
|
|
[CHAT_CONTROL_RECOVERY_ADMISSION_KEY]:
|
|
chatControlRecoveryAdmission(
|
|
lockedRun,
|
|
admission === "admitted" ? "admitted" : "required",
|
|
),
|
|
},
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(eq(heartbeatRuns.id, lockedRun.id));
|
|
}
|
|
const authoritativeIds = queuedCommentIdsFromWakePayload(
|
|
wake.payload,
|
|
);
|
|
if (authoritativeIds.length === 0) {
|
|
// Legacy/direct comment wakes carry comment ids in their ordinary
|
|
// payload and context, not in the authoritative queued-message
|
|
// envelope. Preserve their established claim path; only an
|
|
// explicitly bound queued-message envelope is subject to the
|
|
// live-comment discard gate below.
|
|
await withExternalAdmissionGuard(tx, run.companyId, run.agentId, () => assertNoExternalOverlap(tx, run.companyId, run.agentId));
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
|
|
const [claimedRun] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "running",
|
|
runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`,
|
|
...legacyControllerClaim(run.runtimeMode),
|
|
responsibleUserId,
|
|
startedAt: lockedRun.startedAt ?? claimedAt,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, lockedRun.id),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
),
|
|
)
|
|
.returning();
|
|
await bindClaimedIssueExecution(tx as unknown as Db, issueClaim.ownsIssue, claimedRun);
|
|
return claimedRun
|
|
? { kind: "claimed" as const, run: claimedRun }
|
|
: { kind: "stale" as const, run: null };
|
|
}
|
|
const commentRows = await tx
|
|
.select({
|
|
id: issueComments.id,
|
|
deletedAt: issueComments.deletedAt,
|
|
})
|
|
.from(issueComments)
|
|
.where(
|
|
and(
|
|
eq(issueComments.companyId, run.companyId),
|
|
eq(issueComments.issueId, issueId),
|
|
inArray(issueComments.id, authoritativeIds),
|
|
),
|
|
);
|
|
const liveIds = authoritativeIds.filter((commentId) => {
|
|
const comment = commentRows.find((row) => row.id === commentId);
|
|
return Boolean(comment && !comment.deletedAt);
|
|
});
|
|
if (liveIds.length === 0) {
|
|
const reason = "Queued messages were discarded before dispatch";
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
|
|
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyExecutionRunWork(tx, { companyId: run.companyId, agentId: run.agentId, runId: run.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, run);
|
|
const [cancelled] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: claimedAt,
|
|
error: reason,
|
|
errorCode: "queued_comment_discarded",
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, lockedRun.id),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
),
|
|
)
|
|
.returning();
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: claimedAt,
|
|
error: reason,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wake.id));
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, run.companyId),
|
|
eq(issues.executionRunId, run.id),
|
|
),
|
|
);
|
|
return {
|
|
kind: "cancelled" as const,
|
|
run: cancelled ?? lockedRun,
|
|
};
|
|
}
|
|
|
|
await withExternalAdmissionGuard(tx, run.companyId, run.agentId, () => assertNoExternalOverlap(tx, run.companyId, run.agentId));
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "claimed",
|
|
claimedAt,
|
|
payload: withQueuedCommentIdsInWakePayload(
|
|
wake.payload,
|
|
liveIds,
|
|
),
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wake.id));
|
|
const [claimedRun] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "running",
|
|
runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`,
|
|
...legacyControllerClaim(run.runtimeMode),
|
|
responsibleUserId,
|
|
startedAt: lockedRun.startedAt ?? claimedAt,
|
|
contextSnapshot: withQueuedCommentIdsInRunContext(
|
|
lockedRun.contextSnapshot,
|
|
liveIds,
|
|
),
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, lockedRun.id),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
),
|
|
)
|
|
.returning();
|
|
await bindClaimedIssueExecution(tx as unknown as Db, issueClaim.ownsIssue, claimedRun);
|
|
return claimedRun
|
|
? { kind: "claimed" as const, run: claimedRun }
|
|
: { kind: "stale" as const, run: null };
|
|
})
|
|
.catch((error) => {
|
|
if (isExternalChatWaitAuthorizationContention(error))
|
|
return { kind: "stale" as const, run: null };
|
|
throw error;
|
|
})
|
|
: null;
|
|
if (queuedCommentClaim?.kind === "cancelled") {
|
|
await appendRunEvent(queuedCommentClaim.run, {
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message:
|
|
queuedCommentClaim.run.error ??
|
|
"Queued messages were discarded before dispatch",
|
|
});
|
|
publishLiveEvent({
|
|
companyId: queuedCommentClaim.run.companyId,
|
|
type: "heartbeat.run.status",
|
|
payload: {
|
|
runId: queuedCommentClaim.run.id,
|
|
agentId: queuedCommentClaim.run.agentId,
|
|
status: queuedCommentClaim.run.status,
|
|
invocationSource: queuedCommentClaim.run.invocationSource,
|
|
triggerDetail: queuedCommentClaim.run.triggerDetail,
|
|
error: queuedCommentClaim.run.error ?? null,
|
|
errorCode: queuedCommentClaim.run.errorCode ?? null,
|
|
startedAt: queuedCommentClaim.run.startedAt
|
|
? new Date(queuedCommentClaim.run.startedAt).toISOString()
|
|
: null,
|
|
finishedAt: queuedCommentClaim.run.finishedAt
|
|
? new Date(queuedCommentClaim.run.finishedAt).toISOString()
|
|
: null,
|
|
},
|
|
});
|
|
publishRunLifecyclePluginEvent(queuedCommentClaim.run);
|
|
// Fire-and-forget: nothing else in this path depends on the emission,
|
|
// so it must not delay the return.
|
|
void emitAgentTaskRun(db, queuedCommentClaim.run);
|
|
return null;
|
|
}
|
|
const claimed = queuedCommentClaim
|
|
? queuedCommentClaim.run
|
|
: await withChatControlRecoveryGate(run, "claim", async (tx) => {
|
|
const claimValues = {
|
|
status: "running",
|
|
runnerProfileJson: sql`(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) || ${JSON.stringify({ adapterDispatch: { adapterType: agent.adapterType } })}::jsonb`,
|
|
...legacyControllerClaim(run.runtimeMode),
|
|
responsibleUserId,
|
|
startedAt: run.startedAt ?? claimedAt,
|
|
updatedAt: claimedAt,
|
|
};
|
|
if (nativeReviewContext) {
|
|
if (run.scheduledRetryReason === "native_provider_overloaded") {
|
|
const predecessor = await lockIssueExecutionClaim(tx);
|
|
if (predecessor.blocked) return null;
|
|
}
|
|
return claimQueuedNativeReviewRun(tx, {
|
|
run, claimedAt, claimValues,
|
|
agentNameKey: normalizeAgentNameKey(agent.name),
|
|
});
|
|
}
|
|
return tx.transaction(async (claimTx) => {
|
|
const issueClaim = await lockIssueExecutionClaim(claimTx as unknown as Db);
|
|
if (issueClaim.blocked) return null;
|
|
if (run.wakeupRequestId) await claimTx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, run.wakeupRequestId), eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.agentId, run.agentId),
|
|
)).for("update");
|
|
const [lockedRun] = await claimTx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.agentId, run.agentId),
|
|
)).for("update");
|
|
if (!lockedRun || lockedRun.status !== "queued") return null;
|
|
await withExternalAdmissionGuard(claimTx, run.companyId, run.agentId, () => assertNoExternalOverlap(claimTx, run.companyId, run.agentId));
|
|
await notifyChatRunMilestones(claimTx, { companyId: run.companyId, runId: run.id, issueId: issueId ?? undefined });
|
|
const claimedRun = await claimTx.update(heartbeatRuns).set(claimValues).where(and(
|
|
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "queued"),
|
|
)).returning().then((rows) => rows[0] ?? null);
|
|
await bindClaimedIssueExecution(claimTx as unknown as Db, issueClaim.ownsIssue, claimedRun);
|
|
return claimedRun;
|
|
});
|
|
});
|
|
if (!claimed) return null;
|
|
|
|
publishLiveEvent({
|
|
companyId: claimed.companyId,
|
|
type: "heartbeat.run.status",
|
|
payload: {
|
|
runId: claimed.id,
|
|
agentId: claimed.agentId,
|
|
status: claimed.status,
|
|
invocationSource: claimed.invocationSource,
|
|
triggerDetail: claimed.triggerDetail,
|
|
error: claimed.error ?? null,
|
|
errorCode: claimed.errorCode ?? null,
|
|
startedAt: claimed.startedAt
|
|
? new Date(claimed.startedAt).toISOString()
|
|
: null,
|
|
finishedAt: claimed.finishedAt
|
|
? new Date(claimed.finishedAt).toISOString()
|
|
: null,
|
|
},
|
|
});
|
|
publishRunLifecyclePluginEvent(claimed);
|
|
|
|
if (!nativeReviewContext) {
|
|
crashPoint("wake.claim.after_run_running_before_wakeup_claimed");
|
|
await setWakeupStatus(claimed.wakeupRequestId, "claimed", { claimedAt });
|
|
}
|
|
|
|
// Fix A (lazy locking): stamp executionRunId now that the run is actually running,
|
|
// not at queue time. Guard is idempotent — safe if called more than once.
|
|
const claimedContext = parseObject(claimed.contextSnapshot);
|
|
const claimedIssueId = readNonEmptyString(claimedContext.issueId);
|
|
const claimedWakeReason = readNonEmptyString(claimedContext.wakeReason);
|
|
if (
|
|
!nativeReviewContext && claimedIssueId &&
|
|
claimedWakeReason !== "source_scoped_recovery_action"
|
|
) {
|
|
const claimedAgent = await getAgent(claimed.agentId);
|
|
await db
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: claimed.id,
|
|
executionAgentNameKey: normalizeAgentNameKey(claimedAgent?.name),
|
|
executionLockedAt: claimedAt,
|
|
updatedAt: claimedAt,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, claimedIssueId),
|
|
eq(issues.companyId, claimed.companyId),
|
|
// Mention/context runs can touch an issue, but only the current assignee
|
|
// owns the issue execution lock shown as the active run.
|
|
eq(issues.assigneeAgentId, claimed.agentId),
|
|
["native_safe_replacement", "native_provider_overloaded"].includes(claimed.scheduledRetryReason ?? "")
|
|
? or(
|
|
isNull(issues.checkoutRunId),
|
|
eq(issues.checkoutRunId, claimed.id),
|
|
)
|
|
: undefined,
|
|
or(
|
|
isNull(issues.executionRunId),
|
|
eq(issues.executionRunId, claimed.id),
|
|
),
|
|
),
|
|
);
|
|
}
|
|
|
|
return claimed;
|
|
}
|
|
|
|
// startNextQueuedRunForAgent checks admission suppression once, then claims
|
|
// runs (sets status "running"), then dispatches each to executeRun, which
|
|
// checks suppression again before it does any work. Suppression (task
|
|
// drain, worktree mode, a database restore) can start in the gap between
|
|
// those two checks. When executeRun's check catches that, the run is
|
|
// already claimed — release it back to "queued" so it does not keep a
|
|
// running execution lock that nothing will ever process. This runs inside
|
|
// the same promise startNextQueuedRunForAgent already tracks in
|
|
// activeRunExecutionPromises, so getTaskDrainStatus() keeps reporting the
|
|
// run as active until the release finishes.
|
|
//
|
|
// The run row, the wakeup request, and the issue execution lock all guard
|
|
// the same claim, so one transaction commits all three writes together. A
|
|
// partial write (for example the run flips to "queued" but the wakeup or
|
|
// issue update then fails) would let the execution promise clear from
|
|
// activeRunExecutionPromises while the wakeup stayed "claimed" or the
|
|
// issue stayed locked to a queued run — task-drain status would then read
|
|
// quiescent while the database still held part of the old claim.
|
|
async function releaseRunClaimedJustBeforeSuppression(runId: string) {
|
|
const now = new Date();
|
|
await db.transaction(async (tx) => {
|
|
const [candidate] = await tx.select({ companyId: heartbeatRuns.companyId, contextSnapshot: heartbeatRuns.contextSnapshot })
|
|
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running"))).for("update");
|
|
if (!candidate) return;
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, runId,
|
|
issueId: readNonEmptyString(parseObject(candidate.contextSnapshot).issueId) ?? undefined });
|
|
const released = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "queued",
|
|
startedAt: null,
|
|
responsibleUserId: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running")),
|
|
)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!released) return;
|
|
|
|
if (released.wakeupRequestId) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({ status: "queued", claimedAt: null, updatedAt: now })
|
|
.where(and(eq(agentWakeupRequests.id, released.wakeupRequestId), ne(agentWakeupRequests.status, "cancelled")));
|
|
}
|
|
|
|
const context = parseObject(released.contextSnapshot);
|
|
const issueId = readNonEmptyString(context.issueId);
|
|
if (issueId) {
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, released.companyId),
|
|
eq(issues.executionRunId, released.id),
|
|
),
|
|
);
|
|
}
|
|
});
|
|
}
|
|
|
|
async function cancelQueuedRunForBlockedDependencies(
|
|
run: typeof heartbeatRuns.$inferSelect,
|
|
issueId: string,
|
|
unresolvedBlockerIssueIds: string[],
|
|
) {
|
|
const now = new Date();
|
|
const reason =
|
|
"Cancelled because issue dependencies are still blocked; Paperclip will wake the assignee when blockers resolve";
|
|
const cancelled = await setRunStatus(run.id, "cancelled", {
|
|
finishedAt: now,
|
|
error: reason,
|
|
errorCode: "issue_dependencies_blocked",
|
|
resultJson: {
|
|
...parseObject(run.resultJson),
|
|
stopReason: "issue_dependencies_blocked",
|
|
effectiveTimeoutSec: 0,
|
|
timeoutConfigured: false,
|
|
timeoutSource: "dependency_gate",
|
|
timeoutFired: false,
|
|
},
|
|
});
|
|
if (!cancelled) return null;
|
|
|
|
await setWakeupStatus(run.wakeupRequestId, "skipped", {
|
|
finishedAt: now,
|
|
error: reason,
|
|
});
|
|
|
|
await db.transaction(async tx => {
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId });
|
|
await notifyNativeStatusReassessment(tx, { companyId: run.companyId, issueId });
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, run.companyId),
|
|
eq(issues.id, issueId),
|
|
eq(issues.executionRunId, run.id),
|
|
),
|
|
);
|
|
});
|
|
|
|
await appendRunEvent(cancelled, {
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message: reason,
|
|
payload: {
|
|
issueId,
|
|
unresolvedBlockerIssueIds,
|
|
},
|
|
});
|
|
|
|
return cancelled;
|
|
}
|
|
|
|
async function resumeDeferredExecutionReceipts(scope: DeferredReceiptRecoveryOptions = {}) {
|
|
if ((await getSchedulingSuppression()).suppressed) return;
|
|
await resumeExecutionWaitComments(scope);
|
|
const cutoff = await getWorktreeExecutionCutoff();
|
|
const pendingInterrupts = await db.select({ id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId })
|
|
.from(agentWakeupRequests).innerJoin(companies, eq(companies.id, agentWakeupRequests.companyId))
|
|
.where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
|
|
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
eq(companies.status, "active"),
|
|
sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is not null`,
|
|
lte(agentWakeupRequests.updatedAt, new Date(Date.now() - 30_000)),
|
|
cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined))
|
|
.orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
|
|
for (const wake of pendingInterrupts) {
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date() }).where(and(
|
|
eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
));
|
|
await resumeQueuedCommentInterrupt(wake.companyId, wake.id).catch(err => {
|
|
scope.onError?.(err);
|
|
logger.warn({ err, queueId: wake.id }, "failed to resume interrupted comment queue");
|
|
});
|
|
}
|
|
// Work-intent outbox: a message refused by a pause waits for the pause to
|
|
// lift, not behind the run. Lifting the pause is the operator's explicit
|
|
// authorization to send what the pause refused, as Send is after a Stop:
|
|
// when the pause stopped the task's run, the release drain never promotes
|
|
// queued work behind that acknowledged stop, so the lifted pause issues the
|
|
// task's next explicit wake here. It reconsiders the exact saved receipt
|
|
// through normal admission (a refusal rewrites the receipt in place) and
|
|
// adopts it atomically, so it is delivered once. It runs before the
|
|
// stranded-queue pass, which touches each saved wake.
|
|
const pauseLifted = await db.select({ wake: agentWakeupRequests })
|
|
.from(agentWakeupRequests)
|
|
.innerJoin(issues, and(eq(issues.companyId, agentWakeupRequests.companyId),
|
|
sql`${issues.id}::text = ${agentWakeupRequests.payload}->>'issueId'`,
|
|
eq(issues.assigneeAgentId, agentWakeupRequests.agentId), isNull(issues.executionRunId),
|
|
notInArray(issues.status, ["done", "cancelled"])))
|
|
.innerJoin(companies, and(eq(companies.id, agentWakeupRequests.companyId), eq(companies.status, "active")))
|
|
.where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
|
|
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId),
|
|
// Only messages the pause path saved (never a chat-inbound receipt,
|
|
// which resumes through its own live authorization), and only pauses
|
|
// this query can see lifted. A refusal by any other gate rewrites the
|
|
// receipt's reason, so it never retries in a loop here.
|
|
// A message the reconciler saved after its wake was lost waits on no
|
|
// hold, so the same explicit wake sends it.
|
|
sql`${agentWakeupRequests.payload}->'executionWait'->>'trigger' in ('pause_lifted', 'reconciler_backstop')`,
|
|
sql`${agentWakeupRequests.payload}->'executionWait'->>'reason' in ('company.inactive', 'agent.not_invokable', 'wake_lost')`,
|
|
eq(agentWakeupRequests.requestedByActorType, "user"),
|
|
sql`coalesce(${agentWakeupRequests.idempotencyKey}, '') not like 'chat-inbound:%'`,
|
|
sql`exists (select 1 from agents pause_agent where pause_agent.id = ${agentWakeupRequests.agentId}
|
|
and pause_agent.status not in ('paused', 'terminated', 'pending_approval'))`,
|
|
sql`NOT EXISTS (SELECT 1 FROM heartbeat_runs live_issue_run
|
|
WHERE live_issue_run.company_id = ${issues.companyId}
|
|
AND (live_issue_run.context_snapshot->>'issueId' = ${issues.id}::text
|
|
OR live_issue_run.native_issue_id = ${issues.id})
|
|
AND live_issue_run.status IN ('queued', 'running', 'scheduled_retry'))`,
|
|
cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined))
|
|
.orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
|
|
const pauseLiftedIssues = new Set<string>();
|
|
for (const { wake } of pauseLifted) {
|
|
const issueKey = `${wake.companyId}:${String(wake.payload?.issueId)}`;
|
|
if (pauseLiftedIssues.has(issueKey)) continue;
|
|
pauseLiftedIssues.add(issueKey);
|
|
// Consumption is per comment. The wake adopts every saved message of
|
|
// the task for this agent, so each receipt drops the comments a
|
|
// completed run already received; a receipt left empty is consumed.
|
|
const issueKeyId = String(wake.payload?.issueId);
|
|
const saved = await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, wake.companyId), eq(agentWakeupRequests.agentId, wake.agentId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueKeyId}`));
|
|
for (const receipt of saved) {
|
|
const commentIds = queuedCommentIdsFromWakePayload(receipt.payload);
|
|
const delivered = await deliveredQueuedCommentIds(db, { companyId: wake.companyId, issueId: issueKeyId,
|
|
agentId: wake.agentId, commentIds: commentIds.filter(isUuidLike) });
|
|
if (delivered.size === 0) continue;
|
|
const remaining = commentIds.filter(commentId => !delivered.has(commentId));
|
|
const now = new Date();
|
|
await db.update(agentWakeupRequests).set(remaining.length === 0
|
|
? { status: "cancelled", finishedAt: now, updatedAt: now, error: QUEUED_MESSAGES_DELIVERED_REASON }
|
|
: { payload: withQueuedCommentIdsInWakePayload(parseObject(receipt.payload), remaining), updatedAt: now })
|
|
.where(and(eq(agentWakeupRequests.id, receipt.id), eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
}
|
|
const [current] = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.id, wake.id),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
if (!current) continue;
|
|
wake.payload = current.payload;
|
|
const context = parseObject(parseObject(wake.payload)[DEFERRED_WAKE_CONTEXT_KEY]);
|
|
const { executionWait: _wait, [DEFERRED_WAKE_CONTEXT_KEY]: _context, ...payload } = parseObject(wake.payload);
|
|
await enqueueWakeup(wake.agentId, {
|
|
source: wake.source as WakeupOptions["source"], triggerDetail: wake.triggerDetail as WakeupOptions["triggerDetail"],
|
|
reason: wake.reason ?? undefined, payload, contextSnapshot: context,
|
|
requestedByActorType: wake.requestedByActorType as WakeupOptions["requestedByActorType"],
|
|
requestedByActorId: wake.requestedByActorId ?? undefined,
|
|
}, wake.id, wake.payload?.interactionOrigin).catch(err => {
|
|
if (err instanceof PersistedExecutionWaitRefusal) return;
|
|
scope.onError?.(err);
|
|
logger.warn({ err, requestId: wake.id }, "a message saved during a pause could not resume");
|
|
});
|
|
}
|
|
|
|
// A server restart or a message/cleanup race can leave a deferred wake
|
|
// after its owner has released the issue lock. Revisit it through the same
|
|
// release admission, so recovery holds and operator Stops still apply.
|
|
const strandedQueues = await db.select({ wake: agentWakeupRequests })
|
|
.from(agentWakeupRequests)
|
|
.innerJoin(issues, and(eq(issues.companyId, agentWakeupRequests.companyId),
|
|
sql`${issues.id}::text = ${agentWakeupRequests.payload}->>'issueId'`,
|
|
or(eq(issues.assigneeAgentId, agentWakeupRequests.agentId),
|
|
sql`${agentWakeupRequests.payload} #>> '{_paperclipWakeContext,wakeReason}' = 'issue_comment_mentioned'`)))
|
|
.innerJoin(companies, and(eq(companies.id, issues.companyId), eq(companies.status, "active")))
|
|
.where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
|
|
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
isNull(issues.executionRunId),
|
|
or(and(
|
|
sql`jsonb_typeof(${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}') = 'array'`,
|
|
sql`${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`,
|
|
), sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`,
|
|
dependencyResolvedWakePayloadSql),
|
|
sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is null`,
|
|
sql`NOT EXISTS (SELECT 1 FROM heartbeat_runs live_issue_run
|
|
WHERE live_issue_run.company_id = ${issues.companyId}
|
|
AND (live_issue_run.context_snapshot->>'issueId' = ${issues.id}::text
|
|
OR live_issue_run.native_issue_id = ${issues.id})
|
|
AND live_issue_run.status IN ('queued', 'running', 'scheduled_retry'))`,
|
|
cutoff ? gte(agentWakeupRequests.requestedAt, cutoff) : undefined))
|
|
.orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
|
|
const releasedNativeSources = new Set<string>();
|
|
for (const { wake } of strandedQueues) {
|
|
const dependencyWake = isDependencyResolvedWakePayload(wake.payload);
|
|
if (!dependencyWake && !queuedCommentIdsFromWakePayload(wake.payload).length &&
|
|
!await readQueuedInteractionResponse(db, wake.companyId, String(wake.payload?.issueId), wake.payload)) continue;
|
|
const [latest] = await db.select({
|
|
id: heartbeatRuns.id,
|
|
companyId: heartbeatRuns.companyId,
|
|
status: heartbeatRuns.status,
|
|
finishedAt: heartbeatRuns.finishedAt,
|
|
startedAt: heartbeatRuns.startedAt,
|
|
runtimeMode: heartbeatRuns.runtimeMode,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
sourceVersion: sql<string>`${heartbeatRuns}.xmin::text`,
|
|
}).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, wake.companyId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${String(wake.payload?.issueId)}`,
|
|
or(isNull(heartbeatRuns.nativeIssueId), sql`${heartbeatRuns.nativeIssueId}::text = ${String(wake.payload?.issueId)}`),
|
|
)).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date() }).where(and(
|
|
eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
));
|
|
if (!latest || !latest.finishedAt || !isHeartbeatRunTerminalStatus(latest.status)) continue;
|
|
if (latest.runtimeMode !== "legacy" && latest.runtimeMode !== "native") continue;
|
|
if (wake.idempotencyKey?.startsWith("chat-inbound:")) {
|
|
await resumeDeferredChatWake(db, { companyId: wake.companyId, agentId: wake.agentId,
|
|
issueId: String(wake.payload?.issueId), requestId: wake.id }).catch(err => {
|
|
scope.onError?.(err);
|
|
logger.warn({ requestId: wake.id }, "saved chat source could not resume after recovery");
|
|
});
|
|
continue;
|
|
}
|
|
const releaseSource = `${latest.companyId}:${String(wake.payload?.issueId)}:${latest.id}:${latest.sourceVersion}`;
|
|
if (latest.runtimeMode === "native" && releasedNativeSources.has(releaseSource)) continue;
|
|
if (latest.runtimeMode === "legacy") {
|
|
const cancelledAdmission = latest.status === "cancelled" && !latest.startedAt &&
|
|
latest.errorCode === "execution_reconciliation_required";
|
|
if ((latest.status !== "cancelled" || cancelledAdmission) && await getExecutionBlocker(db, wake.companyId, String(wake.payload?.issueId))) {
|
|
await resumeSavedLegacyComments(wake.companyId, wake.id).catch(err => {
|
|
scope.onError?.(err);
|
|
logger.warn({ err, queueId: wake.id }, "failed to deliver saved legacy comment after recovery stopped");
|
|
});
|
|
}
|
|
}
|
|
await releaseIssueExecutionAndPromote(latest, { suppressImmediateRecovery: true, resumeStrandedQueue: true }).then(() => {
|
|
if (latest.runtimeMode === "native") releasedNativeSources.add(releaseSource);
|
|
}).catch(err => {
|
|
scope.onError?.(err);
|
|
logger.warn({ err, queueId: wake.id }, "failed to promote stranded comments");
|
|
});
|
|
}
|
|
|
|
// The cancellation marker is durable intent. Retry while its exact queue
|
|
// is still deferred, including after a failed cleanup promotion or restart.
|
|
// Normal admission still checks process ownership, leases, pauses, and scope.
|
|
const interruptedQueues = await db
|
|
.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId })
|
|
.from(agentWakeupRequests)
|
|
.innerJoin(heartbeatRuns, and(
|
|
sql`${heartbeatRuns.resultJson}->>'queuedCommentInterruptQueueId' = ${agentWakeupRequests.id}::text`,
|
|
eq(heartbeatRuns.companyId, agentWakeupRequests.companyId),
|
|
eq(heartbeatRuns.agentId, agentWakeupRequests.agentId),
|
|
))
|
|
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
|
|
.where(and(
|
|
scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
|
|
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
eq(heartbeatRuns.status, "cancelled"),
|
|
eq(heartbeatRuns.runtimeMode, "legacy"),
|
|
eq(companies.status, "active"),
|
|
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
|
|
));
|
|
for (const run of interruptedQueues) {
|
|
await releaseIssueExecutionAndPromote(run, { suppressImmediateRecovery: true }).catch((err) => {
|
|
scope.onError?.(err);
|
|
logger.error({ err, runId: run.id }, "failed to retry interrupted comment queue");
|
|
});
|
|
}
|
|
|
|
}
|
|
|
|
async function resumeQueuedRuns() {
|
|
if (isRunAdmissionClosed()) return;
|
|
await resumeDeferredExecutionReceipts();
|
|
await resumeQueuedRunsAfterDrain();
|
|
}
|
|
|
|
// Release admission promptly without first scanning unrelated recovery work.
|
|
// Every run still passes the ordinary per-agent claim and dispatch gates.
|
|
async function resumeQueuedRunsAfterDrain(scope?: { companyId: string }) {
|
|
const refreshing = refreshRunAdmission();
|
|
if (refreshing) await refreshing;
|
|
if (isRunAdmissionClosed() || (await getSchedulingSuppression()).suppressed) return;
|
|
const cutoff = await getWorktreeExecutionCutoff();
|
|
const queuedRuns = await db
|
|
.select({ agentId: heartbeatRuns.agentId })
|
|
.from(heartbeatRuns)
|
|
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.status, "queued"),
|
|
scope ? eq(heartbeatRuns.companyId, scope.companyId) : undefined,
|
|
eq(companies.status, "active"),
|
|
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
|
|
),
|
|
);
|
|
|
|
const agentIds = [...new Set(queuedRuns.map((r) => r.agentId))];
|
|
for (const agentId of agentIds) {
|
|
await startNextQueuedRunForAgent(agentId, true);
|
|
}
|
|
}
|
|
|
|
// Startup fast path: start queued turns that never executed, on agents with
|
|
// no run left `running` by the previous process and no queued retry or
|
|
// continuation. Those agents' runs depend on restart recovery and wait for
|
|
// the full queued-run pass. Every run still passes the ordinary per-agent
|
|
// claim and dispatch gates, exactly as a new wakeup during startup would.
|
|
async function resumeFreshQueuedRunsAtStartup() {
|
|
if ((await getSchedulingSuppression()).suppressed) return { agents: 0 };
|
|
const cutoff = await getWorktreeExecutionCutoff();
|
|
const rows = await db
|
|
.selectDistinct({ agentId: heartbeatRuns.agentId })
|
|
.from(heartbeatRuns)
|
|
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.status, "queued"),
|
|
eq(companies.status, "active"),
|
|
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
|
|
sql`not exists (
|
|
select 1 from heartbeat_runs prior
|
|
where prior.agent_id = ${heartbeatRuns.agentId}
|
|
and (prior.status = 'running' or (prior.status = 'queued' and (
|
|
prior.started_at is not null or prior.retry_of_run_id is not null
|
|
or coalesce(prior.process_loss_retry_count, 0) > 0
|
|
or coalesce(prior.continuation_attempt, 0) > 0)))
|
|
)`,
|
|
),
|
|
);
|
|
for (const { agentId } of rows) {
|
|
await startNextQueuedRunForAgent(agentId, true);
|
|
}
|
|
return { agents: rows.length };
|
|
}
|
|
|
|
// A 403 from claimQueuedRun comes from the run's own persisted identity
|
|
// (an unverifiable interrupt receipt, a manual wake with no user). Those rows
|
|
// do not change, so the claim fails the same way on every pass and restart.
|
|
function isPermanentClaimRejection(err: unknown): err is HttpError {
|
|
return err instanceof HttpError && err.status === 403;
|
|
}
|
|
|
|
// Other 4xx rejections can clear later (a responsible user gets assigned, a
|
|
// conflicting claim finishes). Keep the run queued, but do not let it stop
|
|
// the rest of the queue or startup recovery.
|
|
function isDeferrableClaimRejection(err: unknown): err is HttpError {
|
|
return err instanceof HttpError && err.status >= 400 && err.status < 500;
|
|
}
|
|
|
|
// Settle runs that can never be claimed. Letting the error escape stalls the
|
|
// agent's queue and, during startup recovery, stops the server from booting.
|
|
async function cancelRejectedQueuedRuns(
|
|
rejected: Array<{ run: typeof heartbeatRuns.$inferSelect; err: HttpError }>,
|
|
) {
|
|
for (const { run, err } of rejected) {
|
|
logger.warn(
|
|
{ err, runId: run.id, agentId: run.agentId, companyId: run.companyId },
|
|
"cancelling queued heartbeat run whose claim was rejected",
|
|
);
|
|
try {
|
|
await cancelRunInternal(
|
|
run.id,
|
|
`Cancelled because the queued run cannot be claimed: ${err.message}`,
|
|
{ errorCode: "queued_run_claim_rejected" },
|
|
);
|
|
} catch (cancelErr) {
|
|
logger.error(
|
|
{ err: cancelErr, runId: run.id },
|
|
"failed to cancel queued heartbeat run whose claim was rejected; it stays queued for the next recovery pass",
|
|
);
|
|
}
|
|
}
|
|
}
|
|
|
|
async function attachQueuedSetupRequest(agent: typeof agents.$inferSelect) {
|
|
if (isAgentAwaitingSetup(agent)) await connectionIntentService(db)
|
|
.requestForQueuedAgentSetup(agent.companyId, agent.id)
|
|
.catch(() => logger.warn({ agentId: agent.id }, "Could not attach queued agent setup connection request"));
|
|
}
|
|
|
|
async function startNextQueuedRunForAgent(agentId: string, attachSetupRequest = false) {
|
|
// Null unless this server has not read the run-start hand-over recently.
|
|
const refreshing = refreshRunAdmission();
|
|
if (refreshing) await refreshing;
|
|
if (isRunAdmissionClosed() || (await getSchedulingSuppression()).suppressed) return [];
|
|
const cutoff = await getWorktreeExecutionCutoff();
|
|
// Cancelled after the start lock is released: cancelRunInternal promotes the
|
|
// agent's next queued run, which takes this same lock.
|
|
const rejectedClaims: Array<{ run: typeof heartbeatRuns.$inferSelect; err: HttpError }> = [];
|
|
const deferredPostCommitEffects: WakeQueuePostCommitEffect[] = [];
|
|
const deferredCancellations: Array<() => Promise<void>> = [];
|
|
let cancellationReason: string | undefined;
|
|
|
|
return withAgentStartLock(agentId, async () => {
|
|
const agent = await getAgent(agentId);
|
|
if (!agent) return [];
|
|
const invokability = await getAgentInvokability(agent);
|
|
if (!invokability.invokable) {
|
|
if (attachSetupRequest) await attachQueuedSetupRequest(agent);
|
|
if (shouldCancelRunsForNonInvokableAgent(invokability)) {
|
|
cancellationReason = `Cancelled because the agent is not invokable: ${invokability.reason}`;
|
|
}
|
|
return [];
|
|
}
|
|
const policy = parseHeartbeatPolicy(agent);
|
|
const runningCount = await countRunningRunsForAgent(agentId);
|
|
const availableSlots = Math.max(
|
|
0,
|
|
policy.maxConcurrentRuns - runningCount,
|
|
);
|
|
if (availableSlots <= 0) return [];
|
|
|
|
const queuedRuns = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
eq(heartbeatRuns.status, "queued"),
|
|
cutoff ? gte(heartbeatRuns.createdAt, cutoff) : undefined,
|
|
),
|
|
)
|
|
.orderBy(asc(heartbeatRuns.createdAt));
|
|
if (queuedRuns.length === 0) return [];
|
|
if (queuedRuns.every(run => !run.startedAt) && !await canStartNewExternalBotWork(agent)) return [];
|
|
|
|
const dependencyReadiness = await listQueuedRunDependencyReadiness(
|
|
agent.companyId,
|
|
queuedRuns,
|
|
);
|
|
const queuedIssueIds = [
|
|
...new Set(
|
|
queuedRuns
|
|
.map((run) =>
|
|
readNonEmptyString(parseObject(run.contextSnapshot).issueId),
|
|
)
|
|
.filter((issueId): issueId is string => Boolean(issueId)),
|
|
),
|
|
];
|
|
const issueRows = await db
|
|
.select({
|
|
id: issues.id,
|
|
status: issues.status,
|
|
priority: issues.priority,
|
|
})
|
|
.from(issues)
|
|
.where(
|
|
queuedIssueIds.length > 0
|
|
? and(
|
|
eq(issues.companyId, agent.companyId),
|
|
inArray(issues.id, queuedIssueIds),
|
|
)
|
|
: sql`false`,
|
|
);
|
|
const issueById = new Map(issueRows.map((row) => [row.id, row]));
|
|
const companyAgents = await listCompanyAgentOrgRows(agent.companyId);
|
|
const prioritizedRuns = [...queuedRuns].sort((left, right) => {
|
|
const leftIssueId = readNonEmptyString(
|
|
parseObject(left.contextSnapshot).issueId,
|
|
);
|
|
const rightIssueId = readNonEmptyString(
|
|
parseObject(right.contextSnapshot).issueId,
|
|
);
|
|
const leftReadiness = leftIssueId
|
|
? dependencyReadiness.get(leftIssueId)
|
|
: null;
|
|
const rightReadiness = rightIssueId
|
|
? dependencyReadiness.get(rightIssueId)
|
|
: null;
|
|
const leftReady = leftIssueId
|
|
? (leftReadiness?.isDependencyReady ?? true)
|
|
: true;
|
|
const rightReady = rightIssueId
|
|
? (rightReadiness?.isDependencyReady ?? true)
|
|
: true;
|
|
const leftIssue = leftIssueId ? issueById.get(leftIssueId) : null;
|
|
const rightIssue = rightIssueId ? issueById.get(rightIssueId) : null;
|
|
const leftRank = leftIssueId
|
|
? leftReady
|
|
? leftIssue?.status === "in_progress"
|
|
? 0
|
|
: 1
|
|
: 3
|
|
: 2;
|
|
const rightRank = rightIssueId
|
|
? rightReady
|
|
? rightIssue?.status === "in_progress"
|
|
? 0
|
|
: 1
|
|
: 3
|
|
: 2;
|
|
if (leftRank !== rightRank) return leftRank - rightRank;
|
|
const leftPriorityRank = issueRunPriorityRank(leftIssue?.priority);
|
|
const rightPriorityRank = issueRunPriorityRank(rightIssue?.priority);
|
|
if (leftPriorityRank !== rightPriorityRank)
|
|
return leftPriorityRank - rightPriorityRank;
|
|
return left.createdAt.getTime() - right.createdAt.getTime();
|
|
});
|
|
|
|
const claimedRuns: Array<typeof heartbeatRuns.$inferSelect> = [];
|
|
for (const queuedRun of prioritizedRuns) {
|
|
if (claimedRuns.length >= availableSlots || isRunAdmissionClosed()) break;
|
|
let claimed: typeof heartbeatRuns.$inferSelect | null;
|
|
try {
|
|
claimed = await claimQueuedRun(queuedRun, companyAgents, deferredPostCommitEffects, deferredCancellations);
|
|
} catch (err) {
|
|
if (isPermanentClaimRejection(err)) {
|
|
rejectedClaims.push({ run: queuedRun, err });
|
|
continue;
|
|
}
|
|
if (!isDeferrableClaimRejection(err)) throw err;
|
|
logger.warn(
|
|
{ err, runId: queuedRun.id, agentId: queuedRun.agentId, companyId: queuedRun.companyId },
|
|
"queued heartbeat run claim was rejected; leaving it queued for the next recovery pass",
|
|
);
|
|
continue;
|
|
}
|
|
if (claimed) claimedRuns.push(claimed);
|
|
}
|
|
if (claimedRuns.length === 0) return [];
|
|
|
|
for (const claimedRun of claimedRuns) {
|
|
const execution = executeRun(claimedRun.id).catch((err) => {
|
|
logger.error(
|
|
{ err, runId: claimedRun.id },
|
|
"queued heartbeat execution failed",
|
|
);
|
|
});
|
|
// Register the in-flight execution so drainActiveRunExecutions() can await
|
|
// it. executeRun resolves only after its finally block finishes flushing
|
|
// run rows/events, so awaiting this promise guarantees the run's writes
|
|
// have landed before a caller (e.g. a test's afterEach) mutates the DB.
|
|
activeRunExecutionPromises.add(execution);
|
|
void execution.finally(() => {
|
|
// drainActiveRunExecutions loops on activeRunExecutionPromises.size,
|
|
// so an entry that never clears here would hang it forever.
|
|
activeRunExecutionPromises.delete(execution);
|
|
});
|
|
}
|
|
return claimedRuns;
|
|
}).finally(async () => {
|
|
// Dispatch promoted inputs after the agent start lock is released.
|
|
try {
|
|
await applyWakeQueuePostCommitEffects(deferredPostCommitEffects);
|
|
} finally {
|
|
if (cancellationReason) await cancelActiveForAgentInternal(agentId, cancellationReason);
|
|
await cancelRejectedQueuedRuns(rejectedClaims);
|
|
for (const cancel of deferredCancellations) {
|
|
await cancel().catch((err) => logger.warn({ err, agentId },
|
|
"queued heartbeat run refused by a pause hold could not be cancelled; the next claim retries it"));
|
|
}
|
|
}
|
|
});
|
|
}
|
|
|
|
// Public wakeup entry point. Callers dispatch it fire-and-forget, so register
|
|
// the promise in activeWakeupPromises before it starts its asynchronous
|
|
// prologue. drainActiveRunExecutions can then await a wake that is still before
|
|
// run registration. Internal callers reference enqueueWakeup directly and
|
|
// already await it, so they do not need this registration.
|
|
function trackWakeup(
|
|
agentId: string,
|
|
opts: WakeupOptions = {},
|
|
): ReturnType<typeof enqueueWakeup> {
|
|
// Until its wake and run commit, this admission exists only in this process.
|
|
const unsaved = {};
|
|
unsavedWakeAdmissions.add(unsaved);
|
|
const promise = enqueueWakeup(agentId, opts, undefined, undefined, { saved: () => { unsavedWakeAdmissions.delete(unsaved); } });
|
|
activeWakeupPromises.add(promise);
|
|
void promise
|
|
.catch(() => {})
|
|
.finally(() => {
|
|
unsavedWakeAdmissions.delete(unsaved);
|
|
activeWakeupPromises.delete(promise);
|
|
});
|
|
return promise;
|
|
}
|
|
|
|
async function enqueueWakeup(agentId: string, opts: WakeupOptions = {}, executionWaitRequestId?: string, savedInteractionOrigin?: unknown,
|
|
admission?: { saved(): void }) {
|
|
const source = opts.source ?? "on_demand";
|
|
const triggerDetail = opts.triggerDetail ?? null;
|
|
const contextSnapshot: Record<string, unknown> = {
|
|
...(opts.contextSnapshot ?? {}),
|
|
};
|
|
const reason = opts.reason ?? null;
|
|
let payload = opts.payload ? { ...opts.payload } : null;
|
|
// Only the board queue route can record interruption authority on an
|
|
// existing receipt. Never accept this internal marker from a wake caller.
|
|
if (payload) {
|
|
delete payload.queuedCommentInterrupt;
|
|
delete payload.manualUserWake;
|
|
delete payload.taskRetry;
|
|
}
|
|
if (opts.manualUserWake) {
|
|
if (opts.requestedByActorType !== "user" || !opts.requestedByActorId || opts.failedRunId) {
|
|
throw new HttpError(403, "Manual wake requires an authenticated user");
|
|
}
|
|
payload = { ...payload, manualUserWake: true };
|
|
}
|
|
const executionReconciliationWake =
|
|
contextSnapshot.source === "execution.reconciled" ||
|
|
opts.idempotencyKey?.startsWith("execution-reconciliation:") === true;
|
|
const {
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
issueIdFromPayload,
|
|
taskKey,
|
|
wakeCommentId,
|
|
} = enrichWakeContextSnapshot({
|
|
contextSnapshot,
|
|
reason,
|
|
source,
|
|
triggerDetail,
|
|
payload,
|
|
});
|
|
const isManualContextCompactionWake = () => Boolean(
|
|
reason === CONTEXT_COMPACTION_WAKE_REASON && source === "on_demand" && triggerDetail === "manual"
|
|
&& opts.manualUserWake === true && opts.requestedByActorType === "user"
|
|
&& Boolean(opts.requestedByActorId) && opts.allowRunCoalescing === false
|
|
&& payload?.intent === CONTEXT_COMPACTION_WAKE_REASON
|
|
&& readNonEmptyString(payload.requestId) !== null
|
|
&& readContextCompactionRequest(enrichedContextSnapshot)?.requestId === payload.requestId,
|
|
);
|
|
const hasIndependentConversationWakeCause = () => Boolean(
|
|
wakeCommentId || opts.taskRetryRequestId ||
|
|
hasInteractionContinuationWakeContext(enrichedContextSnapshot) ||
|
|
reason === CHAT_COMPLETION_WAKE_REASON || isManualContextCompactionWake(),
|
|
);
|
|
// Keep each request's own server-derived origin, including coalesced wakes.
|
|
// A run's merged context cannot establish which caller authored one receipt.
|
|
// Overwrite caller-supplied nested context rather than trusting it.
|
|
payload = withInteractionDeliveryOrigin(
|
|
{ ...payload, [DEFERRED_WAKE_CONTEXT_KEY]: { ...enrichedContextSnapshot } },
|
|
opts.requestedByActorType, opts.requestedByActorId, savedInteractionOrigin, opts.requestedByActorCompanyId,
|
|
);
|
|
let issueId =
|
|
readNonEmptyString(enrichedContextSnapshot.issueId) ?? issueIdFromPayload;
|
|
if (executionReconciliationWake && !issueId) return null;
|
|
const durableRequest = opts.durableChatRequest;
|
|
let taskRetryActorId: string | null = null;
|
|
delete enrichedContextSnapshot.boardMessageRetry;
|
|
if (opts.taskRetryRequestId) {
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, opts.taskRetryRequestId), eq(agentWakeupRequests.agentId, agentId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
eq(agentWakeupRequests.requestedByActorType, "user"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId ?? ""}`,
|
|
));
|
|
const retry = readTaskRetryIntent(receipt?.payload);
|
|
if (!receipt || retry?.generationId !== receipt.id || retry.canonicalIntentId || retry.queuedChatRequestId ||
|
|
retry.actorId !== receipt.requestedByActorId) return null;
|
|
const chatRetry = retry.chatActionId && durableRequest?.id === retry.chatActionId &&
|
|
durableRequest.failedRunRetry?.failedRunId === retry.sourceRunId;
|
|
if (retry.chatActionId ? !chatRetry :
|
|
opts.requestedByActorType !== "user" || retry.actorId !== opts.requestedByActorId) return null;
|
|
taskRetryActorId = retry.actorId;
|
|
enrichedContextSnapshot.forceFreshSession = true;
|
|
enrichedContextSnapshot.previousRunId = retry.sourceRunId;
|
|
enrichedContextSnapshot.taskRetryRequestId = receipt.id;
|
|
if (retry.boardMessageOrigin) enrichedContextSnapshot.boardMessageRetry = true;
|
|
}
|
|
|
|
|
|
let agent = await getAgent(agentId);
|
|
if (!agent) throw notFound("Agent not found");
|
|
if (issueId && await hasAutomaticInteractionDeliveryHold(db, {
|
|
companyId: agent.companyId, issueId, agentId, payload, context: enrichedContextSnapshot,
|
|
actorType: opts.requestedByActorType, actorId: opts.requestedByActorId,
|
|
})) return null;
|
|
if (issueId) {
|
|
const [pending] = await db.select().from(caseHandoffs).where(and(eq(caseHandoffs.companyId, agent.companyId), eq(caseHandoffs.issueId, issueId),
|
|
inArray(caseHandoffs.state, ["awaiting_release", "pending_dispatch"]))).limit(1);
|
|
if (pending && opts.durableCaseHandoffRequest?.handoffId !== pending.id) return null;
|
|
}
|
|
// Mentions only annotate comments. Ignore legacy callers before creating
|
|
// a run or deferred request; assignment and review have their own wakes.
|
|
if (reason === "issue_comment_mentioned" || enrichedContextSnapshot.wakeReason === "issue_comment_mentioned") return null;
|
|
if (issueId) {
|
|
const conversation = await getIssueExecutionContext(agent.companyId, issueId);
|
|
if (taskRetryActorId && conversation) await assertTaskRetryConversationAllowed(db, conversation, taskRetryActorId);
|
|
if (reason === "issue_children_completed" && conversation?.originKind === "onboarding_first_task") enrichedContextSnapshot.onboardingCompletion = true;
|
|
if (isConversation(conversation)) {
|
|
if (opts.manualUserWake && conversation!.conversationUserId !== opts.requestedByActorId) {
|
|
throw new HttpError(403, "Only the conversation owner can start a chat run");
|
|
}
|
|
if (isConversationExecutionWake(conversation, reason ?? readNonEmptyString(enrichedContextSnapshot.wakeReason))) return null;
|
|
if (agent.id !== conversation!.conversationAgentId) return null;
|
|
if (!(await instanceSettings.getExperimental()).enableAgentChat) return null;
|
|
if (isWaitingConversation(conversation) && !hasIndependentConversationWakeCause()) return null;
|
|
}
|
|
}
|
|
if (agent.adapterType === "paperclip_runner") {
|
|
const oldConfig = parseObject(agent.adapterConfig);
|
|
const nextConfig = normalizeLegacyRunnerProvider(oldConfig);
|
|
if (nextConfig !== oldConfig) {
|
|
await agentService(db).update(
|
|
agent.id,
|
|
{ adapterConfig: nextConfig },
|
|
{
|
|
recordRevision: {
|
|
source: "normalize_runner_provider",
|
|
createdByAgentId: null,
|
|
createdByUserId: null,
|
|
},
|
|
},
|
|
);
|
|
await logActivity(db, {
|
|
companyId: agent.companyId,
|
|
actorType: "system",
|
|
actorId: "heartbeat",
|
|
action: "agent.updated",
|
|
entityType: "agent",
|
|
entityId: agent.id,
|
|
details: { provider: "codex", reason: "native_codex_provider" },
|
|
});
|
|
agent = (await getAgent(agentId))!;
|
|
}
|
|
}
|
|
|
|
if (opts.failedRunId) {
|
|
const failed = await getRun(opts.failedRunId, { includeExecutionEvidence: true });
|
|
if (opts.requestedByActorType !== "user" || !opts.requestedByActorId ||
|
|
reason !== "retry_failed_run" || source !== "on_demand" || triggerDetail !== "manual" ||
|
|
!failed || failed.companyId !== agent.companyId || failed.agentId !== agentId ||
|
|
!(await canRetryStoppedRun(db, failed)) ||
|
|
(failed.nativeIssueId ?? readNonEmptyString(failed.contextSnapshot?.issueId)) !== issueId) {
|
|
throw conflict("The selected failed run cannot be retried for this task.");
|
|
}
|
|
if (!activeRunExecutions.has(failed.id) && !adapterExecutionControls.has(failed.id)) {
|
|
await sweepPendingCleanupLeases({ explicitRetry: {
|
|
companyId: failed.companyId, runId: failed.id, actorId: opts.requestedByActorId,
|
|
} });
|
|
}
|
|
if (isConversationAdapter(agent.adapterType) || agent.adapterType === "paperclip_runner") {
|
|
enrichedContextSnapshot.previousRunId = failed.id;
|
|
enrichedContextSnapshot.forceFreshSession = true;
|
|
}
|
|
}
|
|
|
|
const caseRequest = opts.durableCaseHandoffRequest;
|
|
const waitForCaseAdmission = (reason: string, message: string): never => {
|
|
// The case ledger is the outbox. Roll back this admission attempt rather
|
|
// than consume its one receipt ID in a generic skipped/deferred queue.
|
|
throw conflict(message, { code: "case_handoff_admission_wait", reason });
|
|
};
|
|
let caseReceipt: { id: string; requestedAt: Date } | undefined;
|
|
if (caseRequest) {
|
|
if (opts.taskRetryRequestId || opts.durableChatRequest || opts.durableDotRequest || opts.durableMuseRequest || source !== "assignment" || reason !== "case_handoff")
|
|
throw conflict("Invalid case handoff wake authority");
|
|
const [handoff] = await db.select().from(caseHandoffs).where(and(eq(caseHandoffs.id, caseRequest.handoffId),
|
|
eq(caseHandoffs.companyId, agent.companyId), eq(caseHandoffs.issueId, caseRequest.issueId),
|
|
eq(caseHandoffs.toAgentId, agentId), eq(caseHandoffs.wakeupRequestId, caseRequest.wakeupRequestId),
|
|
inArray(caseHandoffs.state, ["pending_dispatch", "delivered"]))).limit(1);
|
|
if (!handoff || issueId !== handoff.issueId || caseRequest.expectedIssueStatusVersion !== handoff.appliedIssueStatusVersion)
|
|
throw conflict("Case handoff no longer matches its task assignment");
|
|
caseReceipt = { id: handoff.wakeupRequestId, requestedAt: handoff.createdAt };
|
|
opts = { ...opts, allowRunCoalescing: false, idempotencyKey: `case-handoff:${handoff.id}`,
|
|
issueStateGuard: { assigneeAgentId: agentId, statuses: ["todo", "in_progress"], statusVersion: caseRequest.expectedIssueStatusVersion } };
|
|
}
|
|
if (durableRequest) {
|
|
assertDurableChatWakeupRequest(durableRequest, {
|
|
agentId,
|
|
companyId: agent.companyId,
|
|
issueId,
|
|
commentId: wakeCommentId ?? null,
|
|
requestedByActorType: opts.requestedByActorType,
|
|
requestedByActorId: opts.requestedByActorId,
|
|
});
|
|
opts = { ...opts, idempotencyKey: durableRequest.idempotencyKey };
|
|
}
|
|
if (
|
|
Object.hasOwn(enrichedContextSnapshot, "chatFailedRunRetry") &&
|
|
!durableRequest?.failedRunRetry
|
|
) {
|
|
throw new FailedChatRunRetryAuthorizationError();
|
|
}
|
|
if (durableRequest?.failedRunRetry) {
|
|
opts = { ...opts, allowRunCoalescing: false };
|
|
}
|
|
const dotRequest = opts.durableDotRequest;
|
|
const museRequest = opts.durableMuseRequest;
|
|
if (dotRequest && (durableRequest || museRequest || dotRequest.agentId !== agentId ||
|
|
dotRequest.companyId !== agent.companyId || dotRequest.issueId !== issueId ||
|
|
dotRequest.requestId !== payload?.dotRequestId || source !== "assignment" ||
|
|
opts.requestedByActorType !== "agent" || opts.requestedByActorId !== agentId ||
|
|
agent.adapterType !== "paperclip_runner" || !["openai_dot", "grok_bot"].includes(String(parseObject(agent.adapterConfig).provider)) ||
|
|
opts.idempotencyKey !== dotRequest.idempotencyKey)) {
|
|
throw conflict("Dot work request does not match its admission authority.");
|
|
}
|
|
if (museRequest && (durableRequest || dotRequest || museRequest.agentId !== agentId ||
|
|
museRequest.companyId !== agent.companyId || museRequest.issueId !== issueId ||
|
|
museRequest.requestId !== payload?.museRequestId || source !== "assignment" ||
|
|
opts.requestedByActorType !== "agent" || opts.requestedByActorId !== agentId ||
|
|
agent.adapterType !== "paperclip_runner" || parseObject(agent.adapterConfig).provider !== "muse" ||
|
|
parseObject(agent.adapterConfig).museBindingId !== museRequest.bindingId ||
|
|
!Number.isSafeInteger(museRequest.bindingGeneration) || museRequest.bindingGeneration < 1 ||
|
|
payload?.museBindingId !== museRequest.bindingId || payload?.museBindingGeneration !== museRequest.bindingGeneration ||
|
|
payload?.museRequestDigest !== museRequest.requestDigest || !/^sha256:[a-f0-9]{64}$/.test(museRequest.requestDigest) ||
|
|
opts.idempotencyKey !== museRequest.idempotencyKey)) {
|
|
throw conflict("Muse work request does not match its admission authority.");
|
|
}
|
|
const receiptRequest = durableRequest ?? dotRequest ?? museRequest ?? caseReceipt;
|
|
const durableReceiptFields = receiptRequest
|
|
? { id: receiptRequest.id, requestedAt: receiptRequest.requestedAt }
|
|
: {};
|
|
const existingDurableReceipt = async (queryDb: Db) => {
|
|
if (!receiptRequest) return null;
|
|
const receipt = await queryDb
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, receiptRequest.id))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
if (receipt && durableRequest) assertDurableChatWakeupReceipt(durableRequest, receipt);
|
|
if (receipt && caseRequest && (receipt.companyId !== agent.companyId || receipt.agentId !== agentId ||
|
|
receipt.payload?.issueId !== caseRequest.issueId || receipt.idempotencyKey !== `case-handoff:${caseRequest.handoffId}`))
|
|
throw conflict("Case handoff wake receipt identity changed");
|
|
if (receipt && dotRequest && (receipt.companyId !== dotRequest.companyId ||
|
|
receipt.agentId !== dotRequest.agentId || receipt.source !== "assignment" ||
|
|
receipt.requestedByActorType !== "agent" || receipt.requestedByActorId !== dotRequest.agentId ||
|
|
receipt.idempotencyKey !== dotRequest.idempotencyKey || receipt.payload?.issueId !== dotRequest.issueId ||
|
|
receipt.payload?.dotRequestId !== dotRequest.requestId)) {
|
|
throw conflict("requestId was reused for another task.");
|
|
}
|
|
if (receipt && museRequest && (receipt.companyId !== museRequest.companyId ||
|
|
receipt.agentId !== museRequest.agentId || receipt.source !== "assignment" ||
|
|
receipt.requestedByActorType !== "agent" || receipt.requestedByActorId !== museRequest.agentId ||
|
|
receipt.idempotencyKey !== museRequest.idempotencyKey || receipt.payload?.issueId !== museRequest.issueId ||
|
|
receipt.payload?.museRequestId !== museRequest.requestId || receipt.payload?.museBindingId !== museRequest.bindingId ||
|
|
receipt.payload?.museBindingGeneration !== museRequest.bindingGeneration || receipt.payload?.museRequestDigest !== museRequest.requestDigest)) {
|
|
throw conflict("Request ID reused with changed input.");
|
|
}
|
|
if (receipt && durableRequest && receipt.status === "deferred_issue_execution" && !receipt.runId &&
|
|
((durableRequest.failedRunRetry && opts.taskRetryRequestId &&
|
|
receipt.payload?.taskRetryRequestId === opts.taskRetryRequestId) ||
|
|
(durableRequest.resumeDeferred && !durableRequest.failedRunRetry &&
|
|
receipt.payload?.issueId === durableRequest.issueId &&
|
|
queuedCommentIdsFromWakePayload(receipt.payload).includes(durableRequest.commentId)))) {
|
|
// The live inbound owner or a bound Retry may reconsider the same saved receipt.
|
|
// The issue lock, live chat authority and execution holds below still
|
|
// decide whether it can dispatch; this never creates another wake.
|
|
executionWaitRequestId = receipt.id;
|
|
return null;
|
|
}
|
|
return receipt;
|
|
};
|
|
const priorReceipt = await existingDurableReceipt(db);
|
|
if (priorReceipt) {
|
|
// Replaying an admission receipt is not fresh authority to dispatch it.
|
|
// The normal queue owns dispatch and its current execution-policy checks.
|
|
return priorReceipt.runId ? getRun(priorReceipt.runId) : null;
|
|
}
|
|
|
|
const agentDebug = parseObject(parseObject(agent.runtimeConfig).debug);
|
|
const runDebug = parseObject(enrichedContextSnapshot.debug);
|
|
if (
|
|
agentDebug.providerTrace === "raw" &&
|
|
runDebug.providerTrace !== "raw"
|
|
) {
|
|
enrichedContextSnapshot.debug = {
|
|
...runDebug,
|
|
providerTrace: "raw",
|
|
};
|
|
enrichedContextSnapshot.providerTraceRequestedBy = `agent:${agent.id}:debug-setting`;
|
|
enrichedContextSnapshot.providerTraceRequestSource =
|
|
"agent_debug_setting";
|
|
}
|
|
|
|
// Automatic signals are replaceable; user input and interaction delivery
|
|
// retain distinct durable receipts even when the same gate blocks them.
|
|
const coalesceExecutionWait =
|
|
opts.requestedByActorType === "system" &&
|
|
opts.allowRunCoalescing !== false &&
|
|
!receiptRequest &&
|
|
!wakeCommentId &&
|
|
queuedCommentIdsFromRunContext(enrichedContextSnapshot).length === 0 &&
|
|
!isInteractionResolutionWakePayload(payload ?? {}) &&
|
|
!hasInteractionContinuationWakeContext(enrichedContextSnapshot);
|
|
const writeSkippedRequest = async (
|
|
skipReason: string,
|
|
patch: Partial<typeof agentWakeupRequests.$inferInsert> = {},
|
|
waitCondition?: Record<string, unknown>,
|
|
): Promise<{ created: boolean; waitSaved?: boolean }> => {
|
|
// A temporary refusal must leave the handoff retryable, without burning
|
|
// its one durable receipt ID on a skipped wake.
|
|
if (caseRequest) return { created: false };
|
|
if (executionWaitRequestId) {
|
|
const waitSaved = await writeDeferredExecutionWait(db, { companyId: agent.companyId, requestId: executionWaitRequestId }, {
|
|
reason: skipReason, message: patch.error ?? (skipReason === "issue_tree_hold_active"
|
|
? "This task is paused. Resume it to send your saved message."
|
|
: "Waiting for task execution to be enabled. Your message is saved."),
|
|
});
|
|
return { created: false, waitSaved };
|
|
}
|
|
const request = {
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: skipReason,
|
|
payload,
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
...patch,
|
|
};
|
|
if (waitCondition && issueId && isUuidLike(issueId)) {
|
|
const waitIssueId = issueId;
|
|
return db.transaction(async (tx) => {
|
|
await tx.execute(sql`select id from issues where id = ${waitIssueId} and company_id = ${agent.companyId} for update`);
|
|
if (durableRequest) await notifyChatInboundWakeNotices(tx, {
|
|
companyId: durableRequest.companyId, issueId: durableRequest.issueId,
|
|
});
|
|
return recordExecutionWait(tx as unknown as Db, {
|
|
issueId: waitIssueId, request, condition: waitCondition, coalesce: coalesceExecutionWait,
|
|
});
|
|
});
|
|
}
|
|
if (durableRequest) {
|
|
await db.transaction(async tx => {
|
|
await notifyChatInboundWakeNotices(tx, {
|
|
companyId: durableRequest.companyId, issueId: durableRequest.issueId,
|
|
});
|
|
await tx.insert(agentWakeupRequests).values(request);
|
|
});
|
|
} else {
|
|
await db.insert(agentWakeupRequests).values(request);
|
|
}
|
|
return { created: true };
|
|
};
|
|
const writeSkippedHeartbeatRequest = async (
|
|
skipReason: string,
|
|
details: Record<string, unknown>,
|
|
) => {
|
|
await writeSkippedRequest(skipReason, {
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: details,
|
|
},
|
|
});
|
|
};
|
|
|
|
// Work-intent outbox (invariant 4): a user's message refused by a
|
|
// temporary hold is saved as a deferred intent that names the refusal and
|
|
// the hold-release event that promotes it, never written as a terminal
|
|
// skip. Promotion runs normal admission again, once.
|
|
// The release drain promotes a saved message behind the task's latest run,
|
|
// once that run is terminal; a run still stopping after a pause commits is
|
|
// that run. A task that never ran has none: its message stays in the task,
|
|
// and stranded-work recovery starts the task once the hold lifts.
|
|
const canSaveDeferredMessage = async () => {
|
|
if (executionWaitRequestId || caseRequest || durableRequest || !issueId || !isUuidLike(issueId) ||
|
|
opts.requestedByActorType !== "user" || !wakeCommentId) return false;
|
|
const [latest] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.companyId, agent.companyId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`))
|
|
.limit(1);
|
|
return Boolean(latest);
|
|
};
|
|
const saveDeferredMessage = async (refusal: string, message: string, details: Record<string, unknown> = {}) => {
|
|
const savedIssueId = issueId!;
|
|
const condition = deferredWorkIntentWait(refusal, `${message} Your message is saved.`, details);
|
|
await db.transaction(async tx => {
|
|
await tx.execute(sql`select id from issues where id = ${savedIssueId} and company_id = ${agent.companyId} for update`);
|
|
await notifyDeferredExecutionWork(tx, { companyId: agent.companyId });
|
|
await tx.insert(agentWakeupRequests).values({
|
|
companyId: agent.companyId, agentId, source, triggerDetail, reason,
|
|
payload: withQueuedCommentIdsInWakePayload({
|
|
...payload, issueId: savedIssueId, [DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot, executionWait: condition,
|
|
}, [...new Set([...queuedCommentIdsFromRunContext(enrichedContextSnapshot), wakeCommentId!])]),
|
|
status: "deferred_issue_execution",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
});
|
|
});
|
|
return new PersistedExecutionWaitRefusal(message, details);
|
|
};
|
|
|
|
const schedulingSuppression = await getSchedulingSuppression();
|
|
// A task drain holds ADMISSION, not the request. The drain is a
|
|
// process-local pre-restart hold, so a wake that arrives while it is
|
|
// active still names real work that must run once the process comes
|
|
// back: leave it in the durable queue and let the dispatch-side checks
|
|
// (startNextQueuedRunForAgent / executeRun) keep it from starting until
|
|
// the drain lifts or the restart clears it. Writing it as `skipped`
|
|
// here dropped the wake permanently — an accepted plan whose
|
|
// continuation wake landed mid-drain left its issue in `todo` with no
|
|
// run and no path until a person noticed.
|
|
if (
|
|
schedulingSuppression.suppressed &&
|
|
schedulingSuppression.reason !== "task_drain"
|
|
) {
|
|
await writeSkippedHeartbeatRequest("heartbeat.scheduling_suppressed", {
|
|
reason: schedulingSuppression.reason,
|
|
});
|
|
return null;
|
|
}
|
|
|
|
const worktreeExecutionCutoff =
|
|
opts.requestedByActorType === "user"
|
|
? null
|
|
: await getWorktreeExecutionCutoff();
|
|
|
|
const company = await db
|
|
.select({ status: companies.status })
|
|
.from(companies)
|
|
.where(eq(companies.id, agent.companyId))
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!company || company.status !== "active") {
|
|
if (caseRequest) return null;
|
|
const companyStatus = company?.status ?? "missing";
|
|
if (opts.requestedByActorType === "user") {
|
|
// A paused company is a temporary hold; an archived one is not.
|
|
if (companyStatus === "paused" && (await canSaveDeferredMessage()))
|
|
throw await saveDeferredMessage("company.inactive", "Company is not active", { status: companyStatus });
|
|
throw conflict("Company is not active", { status: companyStatus });
|
|
}
|
|
await writeSkippedRequest("company.inactive", {
|
|
error: `Wake suppressed because company status is ${companyStatus}`,
|
|
}, { companyStatus });
|
|
return null;
|
|
}
|
|
|
|
const explicitResumeSession = await resolveExplicitResumeSessionOverride(
|
|
agent,
|
|
payload,
|
|
taskKey,
|
|
);
|
|
if (explicitResumeSession) {
|
|
enrichedContextSnapshot.resumeFromRunId =
|
|
explicitResumeSession.resumeFromRunId;
|
|
enrichedContextSnapshot.resumeSessionDisplayId =
|
|
explicitResumeSession.sessionDisplayId;
|
|
enrichedContextSnapshot.resumeSessionParams =
|
|
explicitResumeSession.sessionParams;
|
|
if (
|
|
!readNonEmptyString(enrichedContextSnapshot.issueId) &&
|
|
explicitResumeSession.issueId
|
|
) {
|
|
enrichedContextSnapshot.issueId = explicitResumeSession.issueId;
|
|
}
|
|
if (
|
|
!readNonEmptyString(enrichedContextSnapshot.taskId) &&
|
|
explicitResumeSession.taskId
|
|
) {
|
|
enrichedContextSnapshot.taskId = explicitResumeSession.taskId;
|
|
}
|
|
if (
|
|
!readNonEmptyString(enrichedContextSnapshot.taskKey) &&
|
|
explicitResumeSession.taskKey
|
|
) {
|
|
enrichedContextSnapshot.taskKey = explicitResumeSession.taskKey;
|
|
}
|
|
issueId = readNonEmptyString(enrichedContextSnapshot.issueId) ?? issueId;
|
|
}
|
|
const effectiveTaskKey =
|
|
readNonEmptyString(enrichedContextSnapshot.taskKey) ?? taskKey;
|
|
const genericTimerWake =
|
|
source === "timer" &&
|
|
!issueId &&
|
|
!wakeCommentId &&
|
|
!readNonEmptyString(enrichedContextSnapshot.taskId) &&
|
|
!effectiveTaskKey;
|
|
// Old or direct timer producers cannot create a doomed taskless Runner
|
|
// run. Explicit session restoration above can supply a real task scope.
|
|
if (genericTimerWake && !supportsUnscopedTimerHeartbeat(agent)) return null;
|
|
const sessionBefore =
|
|
explicitResumeSession?.sessionDisplayId ??
|
|
(await resolveSessionBeforeForWakeup(agent, effectiveTaskKey));
|
|
let hasResolvablePriorSessionWorkspace: boolean | null = null;
|
|
const resolveHasResolvablePriorSessionWorkspace = async () => {
|
|
if (hasResolvablePriorSessionWorkspace !== null)
|
|
return hasResolvablePriorSessionWorkspace;
|
|
hasResolvablePriorSessionWorkspace = issueId
|
|
? await hasResolvablePriorSessionWorkspaceForWake({
|
|
agent,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
taskKey: effectiveTaskKey,
|
|
explicitResumeSession,
|
|
})
|
|
: false;
|
|
return hasResolvablePriorSessionWorkspace;
|
|
};
|
|
const continuationAttempt = readContinuationAttempt(
|
|
enrichedContextSnapshot.livenessContinuationAttempt,
|
|
);
|
|
|
|
let projectId = readNonEmptyString(enrichedContextSnapshot.projectId);
|
|
if (!projectId && issueId) {
|
|
// Look up by either UUID or identifier (e.g. "ENV-13"), but always scope
|
|
// by companyId so a row from another tenant can never be returned even
|
|
// when identifiers collide across companies. Guard the UUID arm because
|
|
// issues.id is a Postgres uuid column — passing "ENV-13" into eq(issues.id, …)
|
|
// would fail with an invalid-input-syntax cast error before the OR is
|
|
// evaluated.
|
|
const lookupIsUuid = isUuidLike(issueId);
|
|
const idMatch = lookupIsUuid
|
|
? or(
|
|
eq(issues.id, issueId),
|
|
eq(issues.identifier, issueId.toUpperCase()),
|
|
)
|
|
: eq(issues.identifier, issueId.toUpperCase());
|
|
const resolvedIssue = await db
|
|
.select({
|
|
id: issues.id,
|
|
projectId: issues.projectId,
|
|
createdAt: issues.createdAt,
|
|
})
|
|
.from(issues)
|
|
.where(and(eq(issues.companyId, agent.companyId), idMatch))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (resolvedIssue) {
|
|
if (
|
|
worktreeExecutionCutoff &&
|
|
resolvedIssue.createdAt < worktreeExecutionCutoff
|
|
) {
|
|
await writeSkippedHeartbeatRequest(
|
|
"heartbeat.worktree_execution_cutoff",
|
|
{
|
|
reason: "worktree_execution_cutoff",
|
|
cutoff: worktreeExecutionCutoff.toISOString(),
|
|
issueId: resolvedIssue.id,
|
|
},
|
|
);
|
|
return null;
|
|
}
|
|
projectId = resolvedIssue.projectId ?? null;
|
|
// Canonicalize context to the UUID so downstream lookups always use UUID
|
|
if (resolvedIssue.id !== issueId) {
|
|
issueId = resolvedIssue.id;
|
|
enrichedContextSnapshot.issueId = issueId;
|
|
if (readNonEmptyString(enrichedContextSnapshot.taskId)) {
|
|
enrichedContextSnapshot.taskId = issueId;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
// Propagate projectId into context so resolveWorkspaceForRun can bind the
|
|
// project workspace even when context.projectId wasn't set by the caller.
|
|
if (projectId && !readNonEmptyString(enrichedContextSnapshot.projectId)) {
|
|
enrichedContextSnapshot.projectId = projectId;
|
|
}
|
|
const isolatedWorkspacesEnabled = issueId
|
|
? (await instanceSettings.getExperimental()).enableIsolatedWorkspaces
|
|
: false;
|
|
let operatorResponsibleUserId: string | null = opts.manualUserWake ? opts.requestedByActorId! : null;
|
|
let queuedResponsibleUserIdPromise: Promise<string> | null = null;
|
|
const resolveQueuedResponsibleUserId = () => {
|
|
if (operatorResponsibleUserId) return Promise.resolve(operatorResponsibleUserId);
|
|
queuedResponsibleUserIdPromise ??= (async () => {
|
|
const queuedIssueContext = issueId
|
|
? await getIssueExecutionContext(agent.companyId, issueId)
|
|
: null;
|
|
const queuedRoutineEnvContext = await getRoutineEnvForExecutionIssue(
|
|
agent.companyId,
|
|
queuedIssueContext,
|
|
);
|
|
const queuedResponsibleUserId =
|
|
await resolveResponsibleUserIdForRunSeed({
|
|
companyId: agent.companyId,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
issueContext: queuedIssueContext,
|
|
routineEnvContext: queuedRoutineEnvContext,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
source,
|
|
triggerDetail,
|
|
});
|
|
if (!queuedResponsibleUserId) {
|
|
throw new HttpError(
|
|
422,
|
|
"Unable to resolve responsible user for heartbeat run dispatch",
|
|
{
|
|
code: "responsible_user_unresolved",
|
|
agentId,
|
|
companyId: agent.companyId,
|
|
issueId: issueId ?? null,
|
|
source,
|
|
triggerDetail,
|
|
wakeReason: readNonEmptyString(
|
|
enrichedContextSnapshot.wakeReason,
|
|
),
|
|
},
|
|
);
|
|
}
|
|
return queuedResponsibleUserId;
|
|
})();
|
|
return queuedResponsibleUserIdPromise;
|
|
};
|
|
|
|
const budgetBlock = await budgets.getInvocationBlock(
|
|
agent.companyId,
|
|
agentId,
|
|
{
|
|
issueId,
|
|
projectId,
|
|
},
|
|
);
|
|
if (budgetBlock) {
|
|
// A budget block that is not a reached hard stop is an explicit pause.
|
|
if ((await canSaveDeferredMessage()) && !(await isBoardOwnedBudgetBlock(db, agent.companyId, budgetBlock)))
|
|
throw await saveDeferredMessage("budget.blocked", budgetBlock.reason,
|
|
{ scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId });
|
|
const wait = await writeSkippedRequest("budget.blocked", { error: budgetBlock.reason }, {
|
|
scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId,
|
|
});
|
|
if (caseRequest) return null;
|
|
const details = { scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId };
|
|
throw wait.waitSaved ? new PersistedExecutionWaitRefusal(budgetBlock.reason, details)
|
|
: conflict(budgetBlock.reason, details);
|
|
}
|
|
|
|
// Setup delays execution, but must not discard work accepted by a caller.
|
|
const invokability = await getAgentInvokability(isAgentAwaitingSetup(agent) ? { ...agent, status: "idle" } : agent);
|
|
if (!invokability.invokable) {
|
|
if (caseRequest) return null;
|
|
// Only a pause defers through the release drain (wake-queue policy); a
|
|
// pending approval still fails a promoted wake, so it is not saved here.
|
|
if (invokability.reason === "paused" && (await canSaveDeferredMessage()))
|
|
throw await saveDeferredMessage("agent.not_invokable", invokability.message,
|
|
{ status: agent.status, reason: invokability.reason });
|
|
let waitSaved = false;
|
|
if (opts.requestedByActorType !== "user" || executionWaitRequestId) {
|
|
const wait = await writeSkippedRequest("agent.not_invokable", {
|
|
error: invokability.message,
|
|
}, { status: agent.status, reason: invokability.reason });
|
|
waitSaved = wait.waitSaved === true;
|
|
}
|
|
const details = { status: agent.status, reason: invokability.reason,
|
|
invalidOrgChain: invokability.invalidOrgChain, ...invokability.details };
|
|
throw waitSaved ? new PersistedExecutionWaitRefusal(invokability.message, details)
|
|
: conflict(invokability.message, details);
|
|
}
|
|
|
|
const policy = parseHeartbeatPolicy(agent);
|
|
|
|
if (source === "timer" && !policy.enabled) {
|
|
await writeSkippedRequest("heartbeat.disabled", {}, { enabled: false });
|
|
return null;
|
|
}
|
|
if (source !== "timer" && !policy.wakeOnDemand) {
|
|
await writeSkippedRequest("heartbeat.wakeOnDemand.disabled", {}, { wakeOnDemand: false });
|
|
return null;
|
|
}
|
|
|
|
if (
|
|
policy.skipTimerWhenNoActionableWork &&
|
|
genericTimerWake &&
|
|
!(await hasActionableTimerWork(agent))
|
|
) {
|
|
await writeSkippedHeartbeatRequest("heartbeat.timer.no_actionable_work", {
|
|
reason:
|
|
"No assigned todo or in_progress issue requires this agent before timer adapter invocation.",
|
|
});
|
|
await markTimerHeartbeatChecked(agentId, source);
|
|
return null;
|
|
}
|
|
|
|
if (issueId) {
|
|
const activePauseHold = await treeControlSvc.getActivePauseHoldGate(
|
|
agent.companyId,
|
|
issueId,
|
|
);
|
|
if (activePauseHold) {
|
|
const treeHoldInteractionWake =
|
|
await isVerifiedIssueTreeControlInteractionWake(db, {
|
|
companyId: agent.companyId,
|
|
issueId,
|
|
agentId,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
requestedByActorType: opts.requestedByActorType,
|
|
requestedByActorId: opts.requestedByActorId,
|
|
});
|
|
|
|
if (!treeHoldInteractionWake) {
|
|
const wait = await writeSkippedRequest("issue_tree_hold_active", {}, {
|
|
holdId: activePauseHold.holdId,
|
|
});
|
|
if (wait.created) await logActivity(db, {
|
|
companyId: agent.companyId,
|
|
actorType: "system",
|
|
actorId: "system",
|
|
agentId,
|
|
runId: null,
|
|
action: "issue.tree_hold_wakeup_deferred",
|
|
entityType: "issue",
|
|
entityId: issueId,
|
|
details: {
|
|
holdId: activePauseHold.holdId,
|
|
rootIssueId: activePauseHold.rootIssueId,
|
|
requestedReason: reason,
|
|
source,
|
|
triggerDetail,
|
|
securityPrinciples: [
|
|
"Complete Mediation",
|
|
"Fail Securely",
|
|
"Secure Defaults",
|
|
],
|
|
},
|
|
});
|
|
return null;
|
|
}
|
|
|
|
enrichedContextSnapshot.treeHoldInteraction = true;
|
|
enrichedContextSnapshot.activeTreeHold = {
|
|
holdId: activePauseHold.holdId,
|
|
rootIssueId: activePauseHold.rootIssueId,
|
|
mode: activePauseHold.mode,
|
|
reason: activePauseHold.reason,
|
|
releasePolicy: activePauseHold.releasePolicy,
|
|
interaction: true,
|
|
};
|
|
}
|
|
}
|
|
|
|
if (issueId) {
|
|
// Mention-triggered wakes can request input from another agent, but they must
|
|
// still respect the issue execution lock so a second agent cannot start on the
|
|
// same issue workspace while the assignee already has a live run.
|
|
const agentNameKey = normalizeAgentNameKey(agent.name);
|
|
|
|
const cancelledRunsToEmit: (typeof heartbeatRuns.$inferSelect)[] = [];
|
|
const issueActivityPublications: ActivityPublication[] = [];
|
|
const issuePostCommitActions: IssuePostCommitAction[] = [];
|
|
|
|
const outcome = await db
|
|
.transaction(withDurableChatAdmission(durableRequest, async (tx) => {
|
|
await tx.execute(
|
|
sql`select id from issues where id = ${issueId} and company_id = ${agent.companyId} for update`,
|
|
);
|
|
|
|
if (executionWaitRequestId) {
|
|
const [pending] = await tx.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, executionWaitRequestId), eq(agentWakeupRequests.companyId, agent.companyId),
|
|
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
// A user message can join a queue originally created by a
|
|
// system wake. Admission validates the saved user comment or board click.
|
|
(opts.queuedCommentInterruptId ?? opts.queuedCommentRequestId) === executionWaitRequestId
|
|
? undefined : eq(agentWakeupRequests.requestedByActorType,
|
|
durableRequest ? opts.requestedByActorType ?? "" : "user"),
|
|
opts.queuedCommentInterruptId === executionWaitRequestId
|
|
? sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt'->>'actorId' = ${opts.requestedByActorId ?? ""}`
|
|
: opts.queuedCommentRequestId === executionWaitRequestId ? undefined
|
|
: eq(agentWakeupRequests.requestedByActorId, opts.requestedByActorId ?? ""),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
|
|
));
|
|
// The issue lock serializes cleanup callbacks and periodic workers.
|
|
// An adopted, discarded, or edited receipt is no longer authority.
|
|
if (!pending || (!opts.taskRetryRequestId && !(wakeCommentId && queuedCommentIdsFromWakePayload(pending.payload).includes(wakeCommentId)) &&
|
|
!(opts.queuedCommentInterruptId && await readQueuedInteractionResponse(tx as unknown as Db,
|
|
agent.companyId, issueId, pending.payload)))) {
|
|
return { kind: "deferred" as const };
|
|
}
|
|
if (opts.taskRetryRequestId && durableRequest?.failedRunRetry) {
|
|
if (pending.id !== durableRequest.id || pending.payload?.taskRetryRequestId !== opts.taskRetryRequestId)
|
|
return { kind: "deferred" as const };
|
|
// The parent board Retry is checked separately below after the
|
|
// connected conversation reauthorizes this exact child receipt.
|
|
} else if (opts.taskRetryRequestId) {
|
|
const retry = readTaskRetryIntent(pending.payload);
|
|
if (pending.id !== opts.taskRetryRequestId || retry?.generationId !== pending.id || retry.chatActionId ||
|
|
retry.actorId !== opts.requestedByActorId || retry.sourceRunId !== enrichedContextSnapshot.previousRunId)
|
|
return { kind: "deferred" as const };
|
|
// A declined Retry stays saved, but never silently: name the gate
|
|
// so the task explains it and a new click can replace this intent.
|
|
const declineRetry = async (reason: string, message: string) => {
|
|
await writeDeferredExecutionWait(tx as unknown as Db, { companyId: agent.companyId,
|
|
requestId: pending.id }, { reason, message }, { replace: true, publications: issueActivityPublications });
|
|
logger.warn({ requestId: pending.id, issueId, reason }, "Task Retry could not start");
|
|
return { kind: "deferred" as const };
|
|
};
|
|
if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: agent.companyId, issueId,
|
|
actorId: retry.actorId, actorSource: retry.actorSource }))
|
|
return declineRetry("retry_authority_denied", "The user who requested Retry can no longer change this task.");
|
|
if (retry.boardMessageOrigin) {
|
|
const origin = await readTaskRetryBoardMessageOrigin(tx as unknown as Db, { companyId: agent.companyId,
|
|
issueId, agentId, sourceRunId: retry.sourceRunId, actorId: retry.actorId });
|
|
if (!origin || origin.commentId !== retry.boardMessageOrigin.commentId ||
|
|
origin.identityContextId !== retry.boardMessageOrigin.identityContextId)
|
|
return declineRetry("retry_source_changed", "The saved message this Retry continues has changed.");
|
|
}
|
|
}
|
|
if (opts.queuedCommentRequestId) {
|
|
const ids = await undeliveredLegacyUserCommentIds(tx as unknown as Db,
|
|
agent.companyId, issueId, agentId, queuedCommentIdsFromWakePayload(pending.payload));
|
|
if (!wakeCommentId || !ids.includes(wakeCommentId)) return { kind: "deferred" as const };
|
|
pending.payload = withQueuedCommentIdsInWakePayload(parseObject(pending.payload), ids);
|
|
await tx.update(agentWakeupRequests).set({ payload: pending.payload }).where(and(
|
|
eq(agentWakeupRequests.id, pending.id), eq(agentWakeupRequests.companyId, agent.companyId),
|
|
));
|
|
}
|
|
if (!opts.queuedCommentInterruptId && !opts.queuedCommentRequestId && pending.payload?.manualUserWake === true) {
|
|
// A persisted manual wake keeps its actor when an execution wait
|
|
// resumes. The locked receipt above has revalidated that actor.
|
|
payload = { ...payload, manualUserWake: true };
|
|
operatorResponsibleUserId = opts.requestedByActorId!;
|
|
}
|
|
if (opts.queuedCommentInterruptId) {
|
|
// The locked board receipt supplies execution authority even when
|
|
// another user authored the messages. Dispatch revalidates the receipt.
|
|
operatorResponsibleUserId = opts.requestedByActorId!;
|
|
}
|
|
if (opts.queuedCommentInterruptId || opts.queuedCommentRequestId) {
|
|
// Edits/discards between the click and dispatch remain authoritative.
|
|
Object.assign(enrichedContextSnapshot, withQueuedCommentIdsInRunContext(
|
|
enrichedContextSnapshot, queuedCommentIdsFromWakePayload(pending.payload),
|
|
));
|
|
}
|
|
}
|
|
let automaticParentRunId: string | null = null;
|
|
if (
|
|
source === "automation" &&
|
|
opts.requestedByActorType === "system"
|
|
) {
|
|
const nativeParent = await readChatControlNativeParent(
|
|
tx as unknown as Db,
|
|
{ companyId: agent.companyId, issueId, agentId },
|
|
opts.requestedByActorId ?? null,
|
|
);
|
|
const genericParent =
|
|
!opts.requestedByActorId && reason === "issue_continuation_needed"
|
|
? readNonEmptyString(enrichedContextSnapshot.retryOfRunId)
|
|
: null;
|
|
automaticParentRunId =
|
|
nativeParent.kind === "parent"
|
|
? nativeParent.runId
|
|
: genericParent;
|
|
if (automaticParentRunId) {
|
|
const [parent] = await tx
|
|
.select({
|
|
agentId: heartbeatRuns.agentId,
|
|
context: heartbeatRuns.contextSnapshot,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, agent.companyId),
|
|
eq(heartbeatRuns.id, automaticParentRunId),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (
|
|
parent &&
|
|
(parseObject(parent.context).issueId ??
|
|
parseObject(parent.context).taskId) === issueId &&
|
|
parent.agentId !== agentId
|
|
)
|
|
automaticParentRunId = null;
|
|
}
|
|
const proof =
|
|
nativeParent.kind === "unresolved"
|
|
? { kind: "unresolved" as const }
|
|
: automaticParentRunId
|
|
? await readChatControlRecoveryStop(
|
|
tx as unknown as Db,
|
|
{
|
|
companyId: agent.companyId,
|
|
issueId,
|
|
agentId,
|
|
sourceRunId: automaticParentRunId,
|
|
},
|
|
true,
|
|
)
|
|
: { kind: "clear" as const };
|
|
if (proof.kind !== "clear") {
|
|
await tx
|
|
.insert(agentWakeupRequests)
|
|
.values({
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason:
|
|
proof.kind === "stopped"
|
|
? CHAT_CONTROL_RECOVERY_STOP_CODE
|
|
: CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE,
|
|
payload: { issueId },
|
|
status: "skipped",
|
|
requestedByActorType: "system",
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
}
|
|
|
|
const durableReceipt = await existingDurableReceipt(
|
|
tx as unknown as Db,
|
|
);
|
|
if (durableReceipt)
|
|
return { kind: "durable" as const, receipt: durableReceipt };
|
|
if (caseRequest) {
|
|
const [handoff] = await tx.select().from(caseHandoffs).where(and(eq(caseHandoffs.id, caseRequest.handoffId),
|
|
eq(caseHandoffs.companyId, agent.companyId),
|
|
eq(caseHandoffs.issueId, caseRequest.issueId), eq(caseHandoffs.state, "pending_dispatch"),
|
|
eq(caseHandoffs.toAgentId, agentId), eq(caseHandoffs.wakeupRequestId, caseRequest.wakeupRequestId))).for("update");
|
|
if (!handoff || handoff.appliedIssueStatusVersion !== caseRequest.expectedIssueStatusVersion)
|
|
throw conflict("Case handoff delivery was superseded");
|
|
}
|
|
const failedChatRetry = await authorizeFailedChatRunRetryWake(
|
|
db,
|
|
tx as unknown as Db,
|
|
{
|
|
phase: "admission",
|
|
wakeupRequestId: durableRequest?.id ?? null,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
issueId,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
},
|
|
);
|
|
if (
|
|
failedChatRetry !== Boolean(durableRequest?.failedRunRetry) ||
|
|
(failedChatRetry &&
|
|
durableRequest?.failedRunRetry?.failedRunId !==
|
|
readNonEmptyString(enrichedContextSnapshot.retryOfRunId))
|
|
) {
|
|
throw new FailedChatRunRetryAuthorizationError();
|
|
}
|
|
if (durableRequest) {
|
|
if (issueId !== durableRequest.issueId)
|
|
throw new Error("chat_inbound_wakeup_binding_denied");
|
|
await durableRequest.authorize(tx as unknown as Db);
|
|
if (opts.taskRetryRequestId) {
|
|
const [retryWake] = await tx.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, agent.companyId), eq(agentWakeupRequests.id, opts.taskRetryRequestId),
|
|
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
)).for("update");
|
|
const retry = readTaskRetryIntent(retryWake?.payload);
|
|
if (!failedChatRetry || retry?.chatActionId !== durableRequest.id ||
|
|
retry.actorId !== taskRetryActorId || retry.sourceRunId !== durableRequest.failedRunRetry?.failedRunId)
|
|
throw new FailedChatRunRetryAuthorizationError();
|
|
if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: agent.companyId, issueId,
|
|
actorId: retry.actorId, actorSource: retry.actorSource })) return { kind: "deferred" as const };
|
|
}
|
|
} else {
|
|
if (
|
|
source === "on_demand" &&
|
|
triggerDetail === "manual" &&
|
|
reason === "retry_failed_run"
|
|
) {
|
|
// Generic Board retry does not carry server-authorized failed-run
|
|
// lineage. Caller-supplied source/comment/retry markers cannot
|
|
// restore it, including for a retired conversation generation.
|
|
const [chatBinding] = await tx
|
|
.select({ id: chatConversations.id })
|
|
.from(chatConversations)
|
|
.innerJoin(chatEndpoints, eq(chatEndpoints.id, chatConversations.endpointId))
|
|
.where(
|
|
and(
|
|
eq(chatEndpoints.externalExecutionPolicy, "restricted"),
|
|
eq(chatConversations.companyId, agent.companyId),
|
|
eq(chatConversations.issueId, issueId),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (chatBinding) {
|
|
throw conflict(
|
|
"Retry needs the exact failed chat request and current access. Send the request again in the current connected conversation.",
|
|
{ code: "chat_failed_run_retry_requires_authorized_context" },
|
|
);
|
|
}
|
|
}
|
|
const [held] = await tx
|
|
.select({ id: issues.id })
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, agent.companyId),
|
|
unadmittedChatWakeupCondition(issues.id, issues.companyId),
|
|
),
|
|
)
|
|
.limit(1);
|
|
if (held) {
|
|
throw conflict(
|
|
"This task has external chat input that has not been admitted. Check the connection's Activity; let pending input finish, or restore access and send a new authorized message before starting the task.",
|
|
{ code: "chat_inbound_wakeup_unadmitted", issueId },
|
|
);
|
|
}
|
|
}
|
|
|
|
const issue = await tx
|
|
.select({
|
|
id: issues.id,
|
|
companyId: issues.companyId,
|
|
identifier: issues.identifier,
|
|
conversationAgentId: issues.conversationAgentId,
|
|
conversationUserId: issues.conversationUserId,
|
|
conversationState: issues.conversationState,
|
|
status: issues.status,
|
|
statusVersion: issues.statusVersion,
|
|
projectId: issues.projectId,
|
|
projectWorkspaceId: issues.projectWorkspaceId,
|
|
executionWorkspaceId: issues.executionWorkspaceId,
|
|
executionWorkspacePreference: issues.executionWorkspacePreference,
|
|
executionWorkspaceSettings: issues.executionWorkspaceSettings,
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
assigneeUserId: issues.assigneeUserId,
|
|
monitorNextCheckAt: issues.monitorNextCheckAt,
|
|
monitorWakeRequestedAt: issues.monitorWakeRequestedAt,
|
|
executionRunId: issues.executionRunId,
|
|
executionAgentNameKey: issues.executionAgentNameKey,
|
|
createdAt: issues.createdAt,
|
|
})
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, agent.companyId),
|
|
),
|
|
)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!issue) {
|
|
if (caseRequest) throw conflict("Case handoff task no longer exists");
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "issue_execution_issue_not_found",
|
|
payload,
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
// Completion can settle the conversation while admission is outside
|
|
// the issue lock. Recheck the current state before persisting a wake.
|
|
if (isWaitingConversation(issue) && !hasIndependentConversationWakeCause()) {
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
if (taskRetryActorId) {
|
|
await assertTaskRetryConversationAllowed(tx as unknown as Db, issue, taskRetryActorId);
|
|
if (await retireSupersededTaskRetry(tx as unknown as Db, {
|
|
companyId: issue.companyId, issueId: issue.id, requestId: opts.taskRetryRequestId!,
|
|
sourceRunId: readNonEmptyString(enrichedContextSnapshot.previousRunId)!,
|
|
})) return { kind: "deferred" as const };
|
|
}
|
|
const issueStateGuard = opts.issueStateGuard;
|
|
const activeMonitorRun = issueStateGuard?.monitorNextCheckAt === undefined ? null
|
|
: await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.nativeIssueId, issue.id),
|
|
eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, ["queued", "running", "scheduled_retry"]),
|
|
)).limit(1).then(rows => rows[0] ?? null);
|
|
if (
|
|
issueStateGuard &&
|
|
(!issueStateGuard.statuses.includes(issue.status) ||
|
|
issue.assigneeAgentId !== issueStateGuard.assigneeAgentId ||
|
|
(issueStateGuard.statusVersion !== undefined && issue.statusVersion !== issueStateGuard.statusVersion) ||
|
|
(issueStateGuard.monitorNextCheckAt !== undefined && (
|
|
activeMonitorRun !== null || issue.assigneeUserId !== null ||
|
|
issue.monitorNextCheckAt?.toISOString() !== issueStateGuard.monitorNextCheckAt ||
|
|
issue.monitorWakeRequestedAt?.toISOString() !== issueStateGuard.monitorWakeRequestedAt
|
|
)))
|
|
) {
|
|
if (caseRequest) throw conflict("Case handoff task changed before admission");
|
|
// A deferred monitor retains its schedule; do not create a receipt
|
|
// that could suppress its next admission attempt.
|
|
if (issueStateGuard.monitorNextCheckAt !== undefined) return { kind: "skipped" as const };
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "issue_state_guard_mismatch",
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: {
|
|
reason:
|
|
"Issue status or assignee changed before the wake could be queued.",
|
|
issueId: issue.id,
|
|
expectedStatuses: issueStateGuard.statuses,
|
|
actualStatus: issue.status,
|
|
expectedAssigneeAgentId: issueStateGuard.assigneeAgentId,
|
|
actualAssigneeAgentId: issue.assigneeAgentId,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
if (opts.failedRunId) {
|
|
// The issue lock makes double-clicks and network retries adopt the
|
|
// same successor, including after it has already finished.
|
|
const [previousRetry] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.agentId, agentId),
|
|
eq(heartbeatRuns.retryOfRunId, opts.failedRunId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'retry_failed_run'`,
|
|
)).orderBy(desc(heartbeatRuns.createdAt)).limit(1);
|
|
if (previousRetry) return { kind: "replayed" as const, run: previousRetry };
|
|
}
|
|
|
|
let reconciledSourceRunId: string | null = null;
|
|
let reconciledRestoreRetryCount: number | null = null;
|
|
if (executionReconciliationWake) {
|
|
const actionId = readNonEmptyString(
|
|
enrichedContextSnapshot.recoveryActionId,
|
|
);
|
|
if (
|
|
!actionId ||
|
|
!isUuidLike(actionId) ||
|
|
source !== "automation" ||
|
|
triggerDetail !== "system" ||
|
|
reason !== "issue_recovery_action_restored" ||
|
|
opts.requestedByActorType !== "system" ||
|
|
opts.requestedByActorId !== "execution-recovery" ||
|
|
opts.idempotencyKey !== `execution-reconciliation:${actionId}` ||
|
|
enrichedContextSnapshot.source !== "execution.reconciled" ||
|
|
enrichedContextSnapshot.forceFreshSession !== true ||
|
|
payload?.issueId !== issue.id ||
|
|
payload?.recoveryActionId !== actionId ||
|
|
issue.assigneeAgentId !== agentId ||
|
|
["done", "cancelled"].includes(issue.status)
|
|
)
|
|
return { kind: "skipped" as const };
|
|
|
|
// The issue lock serializes all admissions for this source. Validate
|
|
// the durable operator decision, then reconcile a prior queue commit
|
|
// before considering a new wake (including a now-terminal successor).
|
|
const [action] = await tx
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(
|
|
and(
|
|
eq(issueRecoveryActions.companyId, issue.companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, issue.id),
|
|
eq(issueRecoveryActions.id, actionId),
|
|
),
|
|
)
|
|
.for("update");
|
|
const decision = parseObject(
|
|
action?.evidence.executionReconciliation,
|
|
);
|
|
const sourceRunId = readNonEmptyString(decision.runId);
|
|
if (
|
|
!action ||
|
|
action.status !== "resolved" ||
|
|
action.kind !== "active_run_watchdog" ||
|
|
action.returnOwnerAgentId !== agentId ||
|
|
!sourceRunId ||
|
|
!isUuidLike(sourceRunId) ||
|
|
decision.providerStopped !== true ||
|
|
!["completed", "not_performed", "mixed"].includes(
|
|
String(decision.actionOutcome),
|
|
) ||
|
|
!readNonEmptyString(decision.outcomeEvidence) ||
|
|
enrichedContextSnapshot.previousRunId !== sourceRunId ||
|
|
enrichedContextSnapshot.retryOfRunId !== sourceRunId ||
|
|
!["pending", "delivered"].includes(
|
|
String(action.evidence.continuationDelivery),
|
|
)
|
|
)
|
|
return { kind: "skipped" as const };
|
|
|
|
const [existingWake] = await tx
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, issue.companyId),
|
|
eq(agentWakeupRequests.agentId, agentId),
|
|
eq(agentWakeupRequests.idempotencyKey, opts.idempotencyKey),
|
|
ne(agentWakeupRequests.status, "skipped"),
|
|
),
|
|
)
|
|
.orderBy(asc(agentWakeupRequests.requestedAt))
|
|
.limit(1);
|
|
if (existingWake) {
|
|
if (
|
|
existingWake.payload?.issueId !== issue.id ||
|
|
existingWake.payload?.recoveryActionId !== action.id ||
|
|
existingWake.requestedByActorType !== "system" ||
|
|
existingWake.requestedByActorId !== "execution-recovery" ||
|
|
!existingWake.runId
|
|
)
|
|
return { kind: "deferred" as const };
|
|
const [existingRun] = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, issue.companyId),
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
eq(heartbeatRuns.id, existingWake.runId),
|
|
),
|
|
);
|
|
if (
|
|
!existingRun ||
|
|
existingRun.contextSnapshot?.issueId !== issue.id ||
|
|
existingRun.contextSnapshot?.recoveryActionId !== action.id ||
|
|
existingRun.contextSnapshot?.previousRunId !== sourceRunId
|
|
)
|
|
return { kind: "deferred" as const };
|
|
return { kind: "replayed" as const, run: existingRun };
|
|
}
|
|
if (action.evidence.continuationDelivery !== "pending")
|
|
return { kind: "skipped" as const };
|
|
// Under the task lock: a fresh continuation is the task's next run
|
|
// only while nothing else continued it. A run started since it was
|
|
// recorded (active or already finished, from a person's Retry or
|
|
// saved input) consumes it, and a task that moved to review gets none.
|
|
const freshRecordedAt = readNonEmptyString(parseObject(action.evidence.freshContinuation).recordedAt);
|
|
if (freshRecordedAt) {
|
|
const [intervening] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, issue.companyId), sql`${heartbeatRuns.id}::text <> ${sourceRunId}`,
|
|
or(eq(heartbeatRuns.nativeIssueId, issue.id), eq(heartbeatRuns.issueId, issue.id),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}::text`),
|
|
sql`${heartbeatRuns.createdAt} >= ${freshRecordedAt}::timestamptz`)).limit(1);
|
|
if (intervening || issue.status === "in_review") {
|
|
await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify(
|
|
intervening ? { continuationDelivery: "delivered", continuationRunId: intervening.id, continuationSupersededByRun: true }
|
|
: { continuationDelivery: "invalidated", continuationInvalidatedReason: "in_review" })}::jsonb` })
|
|
.where(and(eq(issueRecoveryActions.companyId, issue.companyId), eq(issueRecoveryActions.id, action.id)));
|
|
await settleCoalescedFreshContinuationHolds(tx as unknown as Db, { companyId: issue.companyId, primaryId: action.id,
|
|
delivery: intervening ? "delivered" : "invalidated", continuationRunId: intervening?.id ?? null });
|
|
return { kind: "skipped" as const };
|
|
}
|
|
}
|
|
const [reconciledRun] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.id, sourceRunId),
|
|
));
|
|
if (decision.workspaceReconstruction) {
|
|
await validateNativeWorkspaceReconstruction(tx as unknown as Db, {
|
|
companyId: issue.companyId, issueId: issue.id, runId: sourceRunId, actionId,
|
|
});
|
|
reconciledRestoreRetryCount = executionFailureRetryCount(reconciledRun!);
|
|
enrichedContextSnapshot.workspaceReconstructionActionId = action.id;
|
|
}
|
|
if (hasWorkspaceRestoreFailure(reconciledRun?.resultJson)) {
|
|
if ((readNonEmptyString(decision.workspaceRepairEvidence)?.length ?? 0) < 20)
|
|
return { kind: "skipped" as const };
|
|
// Repair does not reset the remaining automatic retry budget.
|
|
reconciledRestoreRetryCount = executionFailureRetryCount(reconciledRun!);
|
|
}
|
|
reconciledSourceRunId = sourceRunId;
|
|
}
|
|
|
|
let continuationWait = { reason: "execution_recovery", message: "Waiting for execution recovery. Your message is saved." };
|
|
let continuationBlocked = false;
|
|
const deferBlockedExecution = async (
|
|
executionBlocker: Pick<NonNullable<Awaited<ReturnType<typeof getExecutionBlocker>>>, "recoveryActionId" | "cause" | "nextAction">,
|
|
) => {
|
|
if (caseRequest) waitForCaseAdmission("execution_reconciliation_required", executionBlocker.nextAction);
|
|
opts.issueStateGuard?.onExecutionDeferred?.(executionBlocker.cause);
|
|
const condition = { recoveryActionId: executionBlocker.recoveryActionId, ...continuationWait };
|
|
if (executionWaitRequestId) {
|
|
await writeDeferredExecutionWait(tx as unknown as Db, { companyId: agent.companyId,
|
|
requestId: executionWaitRequestId }, condition, { replace: true, publications: issueActivityPublications });
|
|
return { kind: "deferred" as const, admitted: true };
|
|
}
|
|
if (durableRequest || wakeCommentId ||
|
|
hasInteractionContinuationWakeContext(enrichedContextSnapshot) ||
|
|
readNonEmptyString(enrichedContextSnapshot.nativeStatusWakeIntentId)) {
|
|
await notifyDeferredExecutionWork(tx, { companyId: agent.companyId });
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId, agentId, source, triggerDetail, reason,
|
|
payload: withQueuedCommentIdsInWakePayload({
|
|
...payload,
|
|
issueId: issue.id,
|
|
[DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot,
|
|
executionWait: condition,
|
|
}, [...new Set([
|
|
...queuedCommentIdsFromRunContext(enrichedContextSnapshot),
|
|
...(wakeCommentId ? [wakeCommentId] : []),
|
|
])]),
|
|
status: "deferred_issue_execution",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
});
|
|
} else {
|
|
await recordExecutionWait(tx as unknown as Db, {
|
|
issueId: issue.id, condition, coalesce: coalesceExecutionWait,
|
|
request: {
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId, agentId, source, triggerDetail,
|
|
reason: "execution_reconciliation_required",
|
|
error: executionBlocker.nextAction,
|
|
payload,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
},
|
|
});
|
|
}
|
|
return { kind: "deferred" as const, admitted: true };
|
|
};
|
|
const explicitContinuationRunId = randomUUID();
|
|
const [resetComment] = isConversation(issue) && opts.requestedByActorType === "user" && opts.requestedByActorId && wakeCommentId
|
|
? await tx.select({ body: issueComments.body }).from(issueComments).where(and(
|
|
eq(issueComments.companyId, issue.companyId), eq(issueComments.issueId, issue.id),
|
|
eq(issueComments.id, wakeCommentId), eq(issueComments.authorUserId, opts.requestedByActorId),
|
|
eq(issueComments.authorType, "user"), isNull(issueComments.deletedAt), isNull(issueComments.createdByRunId),
|
|
)).limit(1)
|
|
: [];
|
|
const verifiedConversationReset = Boolean(resetComment && isConversationReset(resetComment.body));
|
|
if (wakeCommentId && opts.requestedByActorType === "user")
|
|
await preserveTaskRemoteWorkspaceWriter(tx as unknown as Db, issue.companyId, issue.id);
|
|
const executionBlocker = await getExecutionBlocker(
|
|
tx as unknown as Db, issue.companyId, issue.id,
|
|
{ conversationResetCommentId: verifiedConversationReset ? wakeCommentId : null },
|
|
);
|
|
// Prove eligibility without retiring the hold. Later gates can still
|
|
// decline this wake; hold retirement and successor creation stay atomic.
|
|
// A bound chat request has already rechecked its current principal
|
|
// above. Treat its new user message like a board comment, but keep
|
|
// failed-run retry actions on their separate exact-request path.
|
|
const ownsReconstructionContinuation = reconciledSourceRunId && executionBlocker?.runId === reconciledSourceRunId
|
|
&& executionBlocker.recoveryActionId === enrichedContextSnapshot.recoveryActionId;
|
|
const continuationHold = () => executionBlocker ?? {
|
|
recoveryActionId: null, cause: "native_continuation_requires_reconciliation" as const, nextAction: continuationWait.message,
|
|
};
|
|
// Only system continuation wakes (sweeps, monitors, timers) reach
|
|
// the fresh-continuation fallback. A person's Retry, message or
|
|
// queued send keeps its own exact admission; the disposition worker
|
|
// settles the hold through the same fallback, and the saved input
|
|
// then continues the task.
|
|
const systemWake = opts.requestedByActorType !== "user" && !taskRetryActorId && !wakeCommentId
|
|
&& !durableRequest && !opts.queuedCommentInterruptId && !opts.queuedCommentRequestId && !opts.failedRunId
|
|
&& !executionReconciliationWake;
|
|
// A recorded fresh continuation that has no run yet is this task's
|
|
// next run: another system wake is coalesced into it, never a second run.
|
|
if (systemWake && !executionBlocker && await hasPendingFreshContinuation(tx as unknown as Db, issue.companyId, issue.id))
|
|
return { kind: "skipped" as const };
|
|
if ((executionBlocker || opts.queuedCommentInterruptId) && !ownsReconstructionContinuation && !(await admitExplicitNativeContinuation({
|
|
db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id,
|
|
agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId,
|
|
reason: taskRetryActorId ? "task_retry" : durableRequest && !failedChatRetry ? "issue_commented" : reason,
|
|
commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId,
|
|
queuedCommentInterruptId: opts.queuedCommentInterruptId,
|
|
queuedCommentRequestId: opts.queuedCommentRequestId,
|
|
taskRetryRequestId: opts.taskRetryRequestId,
|
|
...(durableRequest?.resumeDeferred && !failedChatRetry ? { durableChatRequest: durableRequest } : {}),
|
|
interactionPayload: payload,
|
|
dryRun: true,
|
|
onBlocked: (reason, message) => { continuationBlocked = true; continuationWait = { reason, message }; },
|
|
})) && (executionBlocker || (continuationBlocked && !verifiedConversationReset))) {
|
|
// A hold left by an unfinished run never refuses an ordinary
|
|
// continuation: unless its cause is the person's, the fallback
|
|
// settles it in this transaction and this wake becomes the fresh
|
|
// continuation from the task's saved work.
|
|
// Agent Chat conversations keep their own owner-bound continuation.
|
|
const freshSourceRunId = systemWake && executionBlocker?.recoveryActionId && executionBlocker.runId
|
|
&& !issue.conversationAgentId ? executionBlocker.runId : null;
|
|
// The continuation is recorded as pending and owned by the
|
|
// execution-continuation worker until a run for it exists: a later
|
|
// gate (daily limit, slots, pause, budget), a skip or a crash can
|
|
// never consume it. This system wake is coalesced into it.
|
|
const fresh = freshSourceRunId ? await continueFreshInTransaction(tx as unknown as Db, {
|
|
companyId: issue.companyId, runId: freshSourceRunId, trigger: "continuation_wake", delivery: "record" }) : null;
|
|
if (fresh?.kind !== "continued" && fresh?.kind !== "already_continued") return deferBlockedExecution(continuationHold());
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
|
|
if (
|
|
worktreeExecutionCutoff &&
|
|
issue.createdAt < worktreeExecutionCutoff
|
|
) {
|
|
if (caseRequest) waitForCaseAdmission("heartbeat.worktree_execution_cutoff", "Case handoff is waiting for workspace execution to be enabled");
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "heartbeat.worktree_execution_cutoff",
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: {
|
|
reason: "worktree_execution_cutoff",
|
|
cutoff: worktreeExecutionCutoff.toISOString(),
|
|
issueId: issue.id,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
const cancelStaleScheduledRetry = async (
|
|
scheduledRun: typeof heartbeatRuns.$inferSelect,
|
|
) => {
|
|
const issueCancelled = issue.status === "cancelled";
|
|
if (
|
|
scheduledRun.status !== "scheduled_retry" ||
|
|
(scheduledRun.agentId === issue.assigneeAgentId &&
|
|
!issueCancelled)
|
|
) {
|
|
return false;
|
|
}
|
|
|
|
const now = new Date();
|
|
const reason = issueCancelled
|
|
? "Cancelled because the issue was cancelled before the scheduled retry became due"
|
|
: "Cancelled because the issue was reassigned before the scheduled retry became due";
|
|
await notifyInstructionCollection(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, scheduledRun);
|
|
await notifyScheduledRetryCancellation(tx, { companyId: scheduledRun.companyId, runId: scheduledRun.id });
|
|
await notifyChatRunMilestones(tx, { companyId: scheduledRun.companyId, issueId: issue.id });
|
|
const cancelled = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: now,
|
|
error: reason,
|
|
errorCode: issueCancelled
|
|
? "issue_cancelled"
|
|
: "issue_reassigned",
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, scheduledRun.id),
|
|
eq(heartbeatRuns.status, "scheduled_retry"),
|
|
),
|
|
)
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!cancelled) return false;
|
|
|
|
if (scheduledRun.wakeupRequestId) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: now,
|
|
error: reason,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
eq(agentWakeupRequests.id, scheduledRun.wakeupRequestId),
|
|
);
|
|
}
|
|
|
|
if (issue.executionRunId === scheduledRun.id) {
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, issue.id),
|
|
eq(issues.executionRunId, scheduledRun.id),
|
|
),
|
|
);
|
|
}
|
|
|
|
const eventSeq = await allocateHeartbeatRunEventSeq(
|
|
tx as unknown as Db,
|
|
cancelled.id,
|
|
);
|
|
|
|
await tx.insert(heartbeatRunEvents).values({
|
|
companyId: cancelled.companyId,
|
|
runId: cancelled.id,
|
|
agentId: cancelled.agentId,
|
|
seq: eventSeq,
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message: issueCancelled
|
|
? "Scheduled retry cancelled because issue was cancelled before it became due"
|
|
: "Scheduled retry cancelled because issue ownership changed before it became due",
|
|
payload: {
|
|
issueId: issue.id,
|
|
issueStatus: issue.status,
|
|
scheduledRetryAttempt: cancelled.scheduledRetryAttempt,
|
|
scheduledRetryAt: cancelled.scheduledRetryAt
|
|
? new Date(cancelled.scheduledRetryAt).toISOString()
|
|
: null,
|
|
scheduledRetryReason: cancelled.scheduledRetryReason,
|
|
previousRetryAgentId: cancelled.agentId,
|
|
currentAssigneeAgentId: issue.assigneeAgentId,
|
|
},
|
|
});
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({ nextEventSeq: eventSeq + 1, updatedAt: now })
|
|
.where(eq(heartbeatRuns.id, cancelled.id));
|
|
|
|
cancelledRunsToEmit.push(cancelled);
|
|
|
|
return true;
|
|
};
|
|
|
|
let activeExecutionRun = issue.executionRunId
|
|
? await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, issue.executionRunId))
|
|
.then((rows) => rows[0] ?? null)
|
|
: null;
|
|
const manualExecutionHold = activeExecutionRun && await nativeManualAdmissionHoldsExecution(tx as unknown as Db,
|
|
{ companyId: issue.companyId, runId: activeExecutionRun.id, issueId: issue.id });
|
|
|
|
if (
|
|
activeExecutionRun &&
|
|
!manualExecutionHold &&
|
|
!EXECUTION_PATH_HEARTBEAT_RUN_STATUSES.includes(
|
|
activeExecutionRun.status as (typeof EXECUTION_PATH_HEARTBEAT_RUN_STATUSES)[number],
|
|
)
|
|
) {
|
|
activeExecutionRun = null;
|
|
}
|
|
|
|
if (
|
|
activeExecutionRun &&
|
|
(await cancelStaleScheduledRetry(activeExecutionRun))
|
|
) {
|
|
activeExecutionRun = null;
|
|
}
|
|
|
|
// A queued/scheduled run holding the lock for an agent that is
|
|
// no longer the issue's assignee is stale by design — the issue
|
|
// has been re-routed (e.g. blocked → in_review with a different
|
|
// assignee). Cancel it and release the lock; otherwise the new
|
|
// assignee's wake gets parked in `deferred_issue_execution`
|
|
// forever, because the original queued holder will never run
|
|
// (the issue's status / target now belongs to someone else).
|
|
//
|
|
// Race guard: pin the cancel UPDATE to the exact non-running
|
|
// status we read above. A worker could transition the holder
|
|
// from `queued` → `running` between the SELECT and this UPDATE;
|
|
// the status predicate ensures we never clobber a freshly-
|
|
// claimed running run. If zero rows matched, leave
|
|
// `activeExecutionRun` populated so the defer path runs
|
|
// normally against the now-running holder.
|
|
if (
|
|
activeExecutionRun &&
|
|
!manualExecutionHold &&
|
|
activeExecutionRun.status !== "running" &&
|
|
issue.assigneeAgentId &&
|
|
activeExecutionRun.agentId !== issue.assigneeAgentId
|
|
) {
|
|
await notifyInstructionCollection(tx, { companyId: activeExecutionRun.companyId, runId: activeExecutionRun.id });
|
|
await notifyExternalAgentSettlement(tx, { companyId: activeExecutionRun.companyId, runId: activeExecutionRun.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: activeExecutionRun.companyId, runId: activeExecutionRun.id });
|
|
await notifyUnsafeWorkspaceExportRunChange(tx, activeExecutionRun);
|
|
await notifyChatRunMilestones(tx, { companyId: activeExecutionRun.companyId, issueId: issue.id });
|
|
const cancelled = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: new Date(),
|
|
error:
|
|
"Execution lock released after issue reassigned to a different agent",
|
|
errorCode: "lock_released_on_reassignment",
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, activeExecutionRun.id),
|
|
eq(heartbeatRuns.status, activeExecutionRun.status),
|
|
),
|
|
)
|
|
.returning();
|
|
if (cancelled.length > 0) {
|
|
cancelledRunsToEmit.push(cancelled[0]);
|
|
if (activeExecutionRun.wakeupRequestId) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: new Date(),
|
|
error:
|
|
"Execution lock released after issue reassigned to a different agent",
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
eq(
|
|
agentWakeupRequests.id,
|
|
activeExecutionRun.wakeupRequestId,
|
|
),
|
|
);
|
|
}
|
|
activeExecutionRun = null;
|
|
}
|
|
}
|
|
|
|
if (!activeExecutionRun && issue.executionRunId) {
|
|
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(issues.id, issue.id));
|
|
}
|
|
|
|
if (!activeExecutionRun) {
|
|
const legacyRun = await tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, issue.companyId),
|
|
inArray(heartbeatRuns.status, [
|
|
...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES,
|
|
]),
|
|
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`,
|
|
),
|
|
)
|
|
.orderBy(
|
|
sql`case when ${heartbeatRuns.status} = 'running' then 0 else 1 end`,
|
|
asc(heartbeatRuns.createdAt),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (legacyRun) {
|
|
if (await cancelStaleScheduledRetry(legacyRun)) {
|
|
activeExecutionRun = null;
|
|
} else {
|
|
activeExecutionRun = legacyRun;
|
|
const legacyAgent = await tx
|
|
.select({ name: agents.name })
|
|
.from(agents)
|
|
.where(eq(agents.id, legacyRun.agentId))
|
|
.then((rows) => rows[0] ?? null);
|
|
await notifyStatusCardIssueChange(tx, issue.companyId);
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: legacyRun.id,
|
|
executionAgentNameKey: normalizeAgentNameKey(
|
|
legacyAgent?.name,
|
|
),
|
|
executionLockedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(issues.id, issue.id));
|
|
}
|
|
}
|
|
}
|
|
|
|
// Availability is advisory: competing clicks may both read it before
|
|
// admission. The existing issue lock serializes the run and deferred
|
|
// receipt checks with the admission below, without another lock owner.
|
|
if (isManualContextCompactionWake()) {
|
|
if (activeExecutionRun) {
|
|
const compacting = isContextCompactionRun(activeExecutionRun.contextSnapshot);
|
|
throw new ContextCompactionError(compacting ? "already_pending" : "agent_busy",
|
|
compacting ? "The agent is already compacting its context." : "Available when the agent finishes its current run.");
|
|
}
|
|
const [pending] = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, issue.companyId), eq(agentWakeupRequests.agentId, agentId),
|
|
inArray(agentWakeupRequests.status, ["queued", "deferred_issue_execution"]), isNull(agentWakeupRequests.runId),
|
|
executionWaitRequestId ? ne(agentWakeupRequests.id, executionWaitRequestId) : undefined,
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
|
|
sql`${agentWakeupRequests.payload}->>'intent' = ${CONTEXT_COMPACTION_WAKE_REASON}`,
|
|
sql`jsonb_typeof(${agentWakeupRequests.payload}->'requestId') = 'string'`,
|
|
sql`nullif(btrim(${agentWakeupRequests.payload}->>'requestId'), '') is not null`,
|
|
sql`${agentWakeupRequests.payload}->${DEFERRED_WAKE_CONTEXT_KEY}->>'contextCompactionRequestId' = ${agentWakeupRequests.payload}->>'requestId'`,
|
|
)).limit(1);
|
|
if (pending) throw new ContextCompactionError("already_pending", "The agent is already compacting its context.");
|
|
}
|
|
|
|
const dependencyReadiness = await issuesSvc
|
|
.listDependencyReadiness(issue.companyId, [issue.id], tx)
|
|
.then((rows) => rows.get(issue.id) ?? null);
|
|
|
|
// Blocked descendants should stay idle until the final blocker resolves.
|
|
// Human comment/mention wakes are the exception: they may run in a
|
|
// bounded interaction mode so the assignee can answer or triage.
|
|
// A stalled child or blocker (issue_blocker_stalled, once per stall)
|
|
// is the other exception: the assignee must be able to re-delegate,
|
|
// escalate or block with a reason while that dependency is stuck.
|
|
const blockedInteractionWake =
|
|
dependencyReadiness &&
|
|
!dependencyReadiness.isDependencyReady &&
|
|
allowsDependencyBlockedWake(
|
|
enrichedContextSnapshot,
|
|
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
|
|
await verifyBlockerStallWake(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id,
|
|
contextSnapshot: enrichedContextSnapshot, requestedByActorType: opts.requestedByActorType }),
|
|
await isVerifiedDependencyInteractionResponse(tx, { companyId: issue.companyId,
|
|
issueId: issue.id, agentId, context: enrichedContextSnapshot }),
|
|
);
|
|
|
|
if (blockedInteractionWake) {
|
|
enrichedContextSnapshot.dependencyBlockedInteraction = true;
|
|
enrichedContextSnapshot.unresolvedBlockerIssueIds =
|
|
dependencyReadiness.unresolvedBlockerIssueIds;
|
|
enrichedContextSnapshot.unresolvedBlockerCount =
|
|
dependencyReadiness.unresolvedBlockerCount;
|
|
enrichedContextSnapshot.unresolvedBlockerSummaries =
|
|
await listUnresolvedBlockerSummaries(
|
|
tx,
|
|
issue.companyId,
|
|
issue.id,
|
|
dependencyReadiness.unresolvedBlockerIssueIds,
|
|
);
|
|
}
|
|
|
|
if (
|
|
!activeExecutionRun &&
|
|
dependencyReadiness &&
|
|
!dependencyReadiness.isDependencyReady &&
|
|
!blockedInteractionWake
|
|
) {
|
|
if (caseRequest) waitForCaseAdmission("issue_dependencies_blocked", "Case handoff is waiting for task dependencies to resolve");
|
|
await recordExecutionWait(tx as unknown as Db, {
|
|
issueId: issue.id,
|
|
coalesce: coalesceExecutionWait,
|
|
condition: { unresolvedBlockerIssueIds: [...dependencyReadiness.unresolvedBlockerIssueIds].sort() },
|
|
request: {
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "issue_dependencies_blocked",
|
|
payload: {
|
|
...(payload ?? {}),
|
|
issueId,
|
|
unresolvedBlockerIssueIds:
|
|
dependencyReadiness.unresolvedBlockerIssueIds,
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: new Date(),
|
|
},
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
if (
|
|
isolatedWorkspacesEnabled &&
|
|
!activeExecutionRun &&
|
|
(issue.status !== "done" || Boolean(opts.queuedCommentInterruptId)) &&
|
|
issue.status !== "cancelled"
|
|
) {
|
|
const issueSettings = parseIssueExecutionWorkspaceSettings(
|
|
issue.executionWorkspaceSettings,
|
|
);
|
|
const resolvedMode = resolveExecutionWorkspaceMode({
|
|
projectPolicy: null,
|
|
issueSettings,
|
|
legacyUseProjectWorkspace: null,
|
|
});
|
|
const workspaceManagedConfig = buildExecutionWorkspaceAdapterConfig(
|
|
{
|
|
agentConfig: parseObject(agent.adapterConfig),
|
|
projectPolicy: null,
|
|
issueSettings,
|
|
mode: resolvedMode,
|
|
legacyUseProjectWorkspace: null,
|
|
},
|
|
);
|
|
const resolvedStrategy = resolveEffectiveWorkspaceStrategyType(
|
|
resolvedMode,
|
|
workspaceManagedConfig,
|
|
);
|
|
const existingExecutionWorkspaceStatus = issue.executionWorkspaceId
|
|
? await tx
|
|
.select({ status: executionWorkspaces.status })
|
|
.from(executionWorkspaces)
|
|
.where(
|
|
and(
|
|
eq(executionWorkspaces.id, issue.executionWorkspaceId),
|
|
eq(executionWorkspaces.companyId, issue.companyId),
|
|
),
|
|
)
|
|
.then((rows) => rows[0]?.status ?? null)
|
|
: null;
|
|
const reuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({
|
|
issueExecutionWorkspaceId: issue.executionWorkspaceId,
|
|
issueExecutionWorkspacePreference:
|
|
issue.executionWorkspacePreference,
|
|
existingExecutionWorkspaceStatus,
|
|
});
|
|
const hasResolvablePriorSessionWorkspace =
|
|
await resolveHasResolvablePriorSessionWorkspace();
|
|
|
|
if (
|
|
isUnrunnableWorktreeCombo({
|
|
issue: {
|
|
projectId: issue.projectId ?? projectId ?? null,
|
|
projectWorkspaceId: issue.projectWorkspaceId,
|
|
executionWorkspaceId: issue.executionWorkspaceId,
|
|
executionWorkspacePreference:
|
|
issue.executionWorkspacePreference,
|
|
},
|
|
resolvedMode,
|
|
resolvedStrategy,
|
|
reusableExecutionWorkspaceAvailable:
|
|
reuseRequest.existingExecutionWorkspaceAvailable,
|
|
hasResolvablePriorSessionWorkspace,
|
|
})
|
|
) {
|
|
if (caseRequest) waitForCaseAdmission(WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE, WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE);
|
|
const now = new Date();
|
|
const issueLabel = formatIssueIdentifierLink(
|
|
issue.identifier,
|
|
issue.id,
|
|
);
|
|
const blockedComment = [
|
|
`Paperclip blocked ${issueLabel} before dispatch because its workspace settings are not runnable.`,
|
|
"",
|
|
`- Code: \`${WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE}\``,
|
|
`- Reason: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE}`,
|
|
`- Next action: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION}`,
|
|
].join("\n");
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
if (issue.status !== "blocked") await notifyChatResponseAvailability(tx, { companyId: issue.companyId, issueId: issue.id });
|
|
await notifyRoutineExecutionStatus(tx, issue.id, issue.companyId);
|
|
if (issue.status === "done") {
|
|
const blocked = await issuesSvc.updateForCompany(issue.id, issue.companyId,
|
|
{ status: "blocked", executionState: null }, tx, issueActivityPublications, issuePostCommitActions);
|
|
if (!blocked) throw conflict("The task changed before its workspace settings could be checked");
|
|
} else await tx
|
|
.update(issues)
|
|
.set({
|
|
status: "blocked",
|
|
checkoutRunId: null,
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(issues.id, issue.id));
|
|
await insertIssueComment(tx, {
|
|
companyId: issue.companyId,
|
|
issueId: issue.id,
|
|
body: blockedComment,
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
});
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
|
|
payload: {
|
|
...(payload ?? {}),
|
|
issueId,
|
|
heartbeatSkip: {
|
|
code: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
|
|
reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE,
|
|
remediation:
|
|
WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: now,
|
|
});
|
|
await logActivity(tx as unknown as Db, {
|
|
companyId: issue.companyId,
|
|
actorType: "system",
|
|
actorId: "system",
|
|
agentId,
|
|
runId: null,
|
|
action: "issue.workspace_preflight_blocked",
|
|
entityType: "issue",
|
|
entityId: issue.id,
|
|
details: {
|
|
code: WORKSPACE_WORKTREE_REQUIRES_PROJECT_CODE,
|
|
reason: WORKSPACE_WORKTREE_REQUIRES_PROJECT_MESSAGE,
|
|
remediation: WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION,
|
|
requestedReason: reason,
|
|
source,
|
|
triggerDetail,
|
|
resolvedMode,
|
|
resolvedStrategy,
|
|
hasResolvablePriorSessionWorkspace,
|
|
},
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
}
|
|
|
|
if (activeExecutionRun) {
|
|
// The case ledger owns retry. Do not place its exact receipt in a
|
|
// generic deferred queue that cannot retain handoff authority.
|
|
if (caseRequest) return { kind: "deferred" as const };
|
|
// The resolved action is already a durable retry outbox. Do not merge
|
|
// its fresh-session contract into unrelated work or create a second
|
|
// deferred wake that could later replay the same reconciliation.
|
|
if (reconciledSourceRunId) return { kind: "deferred" as const };
|
|
|
|
// Offer the input to the active external agent, but keep a durable queued wake
|
|
// until it reads the comment. A finish racing the event tick must
|
|
// leave unread input available for the next assignment.
|
|
const followUpProvider = parseObject(agent.adapterConfig).provider;
|
|
const museFollowUp = followUpProvider === "muse";
|
|
const bindingId = readNonEmptyString(parseObject(agent.adapterConfig)[museFollowUp ? "museBindingId" : followUpProvider === "grok_bot" ? "externalBindingId" : "dotBindingId"]);
|
|
const followUpPayloadKey = museFollowUp ? "museAssignmentFollowUp" : "dotAssignmentFollowUp";
|
|
let followUp: false | { assignmentId: string; consumed: boolean } = false;
|
|
let pendingFollowUps: typeof agentWakeupRequests.$inferSelect[] = [];
|
|
if (agent.adapterType === "paperclip_runner" && ["openai_dot", "grok_bot", "muse"].includes(String(followUpProvider))
|
|
&& bindingId && activeExecutionRun.agentId === agentId && issue.assigneeAgentId === agentId
|
|
&& reason === "issue_commented" && wakeCommentId && opts.allowRunCoalescing !== false
|
|
&& enrichedContextSnapshot.forceFreshSession !== true && !explicitResumeSession && !opts.manualUserWake
|
|
&& !enrichedContextSnapshot.interactionId && !payload?.interactionId && !receiptRequest) {
|
|
// Receipt consumption uses the same issue -> wake -> run -> guard ->
|
|
// binding order. Lock before publishing, then reuse these wake rows.
|
|
pendingFollowUps = await tx.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, agent.companyId), eq(agentWakeupRequests.agentId, agentId),
|
|
isNull(agentWakeupRequests.runId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
|
|
sql`${agentWakeupRequests.payload}->>${followUpPayloadKey} is not null`,
|
|
)).orderBy(asc(agentWakeupRequests.createdAt), asc(agentWakeupRequests.id)).for("update");
|
|
const [currentRun] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, activeExecutionRun.id), eq(heartbeatRuns.companyId, agent.companyId),
|
|
eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.status, "running"),
|
|
eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.nativeIssueId, issue.id),
|
|
)).for("update");
|
|
if (currentRun) followUp = await withExternalAdmissionGuard(tx, agent.companyId, agentId,
|
|
() => (museFollowUp ? publishActiveMuseComment : publishActiveDotComment)(tx as unknown as Db, {
|
|
companyId: agent.companyId, agentId, bindingId, runId: currentRun.id,
|
|
issueId: issue.id, commentId: wakeCommentId,
|
|
}));
|
|
}
|
|
if (followUp) {
|
|
if (!followUp.consumed) await notifyDeferredExecutionWork(tx, { companyId: agent.companyId });
|
|
const assignmentId = followUp.assignmentId;
|
|
const pending = followUp.consumed ? undefined : pendingFollowUps.find(wake =>
|
|
wake.payload?.[followUpPayloadKey] === assignmentId &&
|
|
wake.requestedByActorType === (opts.requestedByActorType ?? null) &&
|
|
wake.requestedByActorId === (opts.requestedByActorId ?? null));
|
|
if (pending) await tx.update(agentWakeupRequests).set({
|
|
payload: withQueuedCommentIdsInWakePayload(pending.payload,
|
|
[...new Set([...queuedCommentIdsFromWakePayload(pending.payload), wakeCommentId!])]),
|
|
coalescedCount: pending.coalescedCount + 1, updatedAt: new Date(),
|
|
}).where(and(eq(agentWakeupRequests.id, pending.id), eq(agentWakeupRequests.companyId, agent.companyId),
|
|
eq(agentWakeupRequests.agentId, agentId), eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
else await tx.insert(agentWakeupRequests).values({ companyId: agent.companyId, agentId, source, triggerDetail,
|
|
reason, payload: withQueuedCommentIdsInWakePayload({ ...payload, issueId: issue.id,
|
|
[followUpPayloadKey]: followUp.assignmentId,
|
|
[DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot }, [wakeCommentId!]),
|
|
status: followUp.consumed ? "coalesced" : "deferred_issue_execution",
|
|
runId: followUp.consumed ? activeExecutionRun.id : null,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null, idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: followUp.consumed ? new Date() : null });
|
|
return { kind: "coalesced" as const, run: activeExecutionRun };
|
|
}
|
|
|
|
const admissionScope = wakeQueue.createAdmissionTransactionScope(
|
|
agent.companyId,
|
|
tx as unknown as Db,
|
|
);
|
|
const admission = await wakeQueue.admitWakeBehindIssueExecution(
|
|
admissionScope,
|
|
{
|
|
companyId: agent.companyId,
|
|
issueId: issue.id,
|
|
agentId,
|
|
agentNameKey,
|
|
issueExecutionAgentNameKey: issue.executionAgentNameKey,
|
|
activeExecutionRun: {
|
|
id: activeExecutionRun.id,
|
|
agentId: activeExecutionRun.agentId,
|
|
status: activeExecutionRun.status,
|
|
contextSnapshot: activeExecutionRun.contextSnapshot,
|
|
wakeupRequestId: activeExecutionRun.wakeupRequestId,
|
|
},
|
|
allowRunCoalescing: isConversation(issue) ? false : opts.allowRunCoalescing,
|
|
durableReceipt: receiptRequest
|
|
? {
|
|
id: receiptRequest.id,
|
|
requestedAt: receiptRequest.requestedAt,
|
|
}
|
|
: undefined,
|
|
reason,
|
|
liveRunExecutions,
|
|
wakeCommentId,
|
|
forceFreshSession:
|
|
enrichedContextSnapshot.forceFreshSession === true,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
source,
|
|
triggerDetail,
|
|
payload,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
},
|
|
);
|
|
|
|
if (admission.kind === "coalesced") {
|
|
return {
|
|
kind: "coalesced" as const,
|
|
run: admission.run as typeof heartbeatRuns.$inferSelect,
|
|
};
|
|
}
|
|
if (admission.kind === "deferred") {
|
|
return { kind: "deferred" as const, admitted: true };
|
|
}
|
|
// admission.kind === "proceed": no active run absorbed this wake,
|
|
// so fall through to the ordinary queue path below.
|
|
}
|
|
|
|
// PAP-13775: no live run holds the lock, so this wake would start a
|
|
// fresh adapter session. If this agent's recent runs on this issue
|
|
// keep succeeding without any issue-visible progress and the wake
|
|
// carries no new information, hold it back for an escalating cooldown
|
|
// so external pollers/reconcilers can't storm full-price sessions.
|
|
// Server-side recovery retries insert runs directly and never reach
|
|
// this gate. A verified durable case receipt carries a new phase;
|
|
// its agent-authored instruction comment is not an event-free wake.
|
|
if (
|
|
!caseRequest &&
|
|
isThrottleCandidateIssueRewake({
|
|
reason,
|
|
wakeCommentId: wakeCommentId ?? null,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
forceFreshSession:
|
|
enrichedContextSnapshot.forceFreshSession === true,
|
|
hasExplicitResume: Boolean(explicitResumeSession),
|
|
declaredContinuationKind: readNonEmptyString(parseObject(payload).continuationKind),
|
|
})
|
|
) {
|
|
const throttleNow = new Date();
|
|
const notNativeStatusCommitterActivity = sql`not (${activityLog.actorType} = 'system' and ${activityLog.actorId} = ${NATIVE_STATUS_COMMITTER_ACTOR_ID})`;
|
|
const recentTerminalRuns = await tx
|
|
.select({
|
|
id: heartbeatRuns.id,
|
|
status: heartbeatRuns.status,
|
|
finishedAt: heartbeatRuns.finishedAt,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, agent.companyId),
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
sql`${heartbeatRuns.finishedAt} is not null`,
|
|
gte(
|
|
heartbeatRuns.finishedAt,
|
|
new Date(throttleNow.getTime() - ISSUE_REWAKE_LOOKBACK_MS),
|
|
),
|
|
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`,
|
|
),
|
|
)
|
|
.orderBy(desc(heartbeatRuns.finishedAt))
|
|
.limit(ISSUE_REWAKE_RUN_SAMPLE_LIMIT);
|
|
|
|
if (recentTerminalRuns.length > 0) {
|
|
const sampleRunIds = recentTerminalRuns.map(
|
|
(sampleRun) => sampleRun.id,
|
|
);
|
|
const progressRows = await tx
|
|
.select({ runId: activityLog.runId })
|
|
.from(activityLog)
|
|
.where(
|
|
and(
|
|
eq(activityLog.companyId, agent.companyId),
|
|
eq(activityLog.entityType, "issue"),
|
|
eq(activityLog.entityId, issue.id),
|
|
inArray(activityLog.runId, sampleRunIds),
|
|
inArray(
|
|
activityLog.action,
|
|
ISSUE_PROGRESS_ACTIVITY_ACTIONS,
|
|
),
|
|
notNativeStatusCommitterActivity,
|
|
),
|
|
);
|
|
const lastRunFinishedAt =
|
|
recentTerminalRuns[0]?.finishedAt ?? null;
|
|
const newInputRows = lastRunFinishedAt
|
|
? await tx
|
|
.select({ id: activityLog.id })
|
|
.from(activityLog)
|
|
.where(
|
|
and(
|
|
eq(activityLog.companyId, agent.companyId),
|
|
eq(activityLog.entityType, "issue"),
|
|
eq(activityLog.entityId, issue.id),
|
|
gt(activityLog.createdAt, lastRunFinishedAt),
|
|
sql`coalesce(${activityLog.details}->>'origin', '') <> 'fast_response'`,
|
|
notNativeStatusCommitterActivity,
|
|
inArray(
|
|
activityLog.action,
|
|
ISSUE_NEW_INPUT_ACTIVITY_ACTIONS,
|
|
),
|
|
wakeCommentId && opts.requestedByActorType === "agent"
|
|
? ne(activityLog.actorType, "agent")
|
|
: undefined,
|
|
),
|
|
)
|
|
.limit(1)
|
|
: [];
|
|
|
|
const throttleDecision = evaluateIssueRewakeThrottle({
|
|
now: throttleNow,
|
|
recentTerminalRuns,
|
|
runIdsWithIssueProgress: new Set(
|
|
progressRows
|
|
.map((row) => row.runId)
|
|
.filter((runId): runId is string => Boolean(runId)),
|
|
),
|
|
// For an agent comment wake, the query excludes agent-authored
|
|
// activity while preserving genuinely new user/system input.
|
|
// Presentation/author metadata therefore cannot smuggle human
|
|
// wake privilege, nor can it mask an actual human response.
|
|
hasNewIssueInputSinceLastRun: newInputRows.length > 0,
|
|
});
|
|
|
|
if (throttleDecision.blocked) {
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: "issue_rewake_throttled",
|
|
payload: {
|
|
...(payload ?? {}),
|
|
issueId,
|
|
heartbeatSkip: {
|
|
reason: "issue_rewake_throttled",
|
|
requestedReason: reason,
|
|
noProgressStreak: throttleDecision.noProgressStreak,
|
|
cooldownMs: throttleDecision.cooldownMs,
|
|
lastRunFinishedAt:
|
|
throttleDecision.lastRunFinishedAt.toISOString(),
|
|
nextAllowedAt:
|
|
throttleDecision.nextAllowedAt.toISOString(),
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: throttleNow,
|
|
});
|
|
return { kind: "skipped" as const };
|
|
}
|
|
}
|
|
}
|
|
|
|
const dailyCapBlock = await getHeartbeatDailyCapBlock(
|
|
agent,
|
|
policy,
|
|
{},
|
|
tx,
|
|
);
|
|
if (dailyCapBlock) {
|
|
if (caseRequest) waitForCaseAdmission(dailyCapBlock.reason, "Case handoff is waiting for the agent's daily limit to clear");
|
|
if (executionWaitRequestId && executionBlocker) {
|
|
continuationWait = { reason: dailyCapBlock.reason,
|
|
message: "The agent has reached its daily limit. Your message is saved until work can resume." };
|
|
return deferBlockedExecution(executionBlocker);
|
|
}
|
|
if (opts.idempotencyKey?.startsWith(CONVERSATION_COMMENT_KEY_PREFIX)) throw dailyLimitConversationRefusal(dailyCapBlock);
|
|
const now = new Date();
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: dailyCapBlock.reason,
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: {
|
|
reason:
|
|
"Per-agent heartbeat daily cap reached before adapter invocation.",
|
|
observed: dailyCapBlock.observed,
|
|
limit: dailyCapBlock.limit,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: now,
|
|
});
|
|
if (source === "timer") {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await tx
|
|
.update(agents)
|
|
.set({
|
|
lastHeartbeatAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(agents.id, agentId));
|
|
}
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
const explicitContinuation = await admitExplicitNativeContinuation({
|
|
db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id,
|
|
agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId,
|
|
reason: taskRetryActorId ? "task_retry" : durableRequest && !failedChatRetry ? "issue_commented" : reason,
|
|
commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId,
|
|
queuedCommentInterruptId: opts.queuedCommentInterruptId,
|
|
queuedCommentRequestId: opts.queuedCommentRequestId,
|
|
taskRetryRequestId: opts.taskRetryRequestId,
|
|
...(durableRequest?.resumeDeferred && !failedChatRetry ? { durableChatRequest: durableRequest } : {}),
|
|
interactionPayload: payload,
|
|
onBlocked: (reason, message) => { continuationBlocked = true; continuationWait = { reason, message }; },
|
|
});
|
|
if (!explicitContinuation && (executionBlocker || (continuationBlocked && !verifiedConversationReset)) && !ownsReconstructionContinuation)
|
|
return deferBlockedExecution(continuationHold());
|
|
if (explicitContinuation) {
|
|
enrichedContextSnapshot.forceFreshSession = true;
|
|
enrichedContextSnapshot.previousRunId = explicitContinuation.previousRunId;
|
|
enrichedContextSnapshot.explicitUserContinuation = explicitContinuation;
|
|
}
|
|
|
|
if (opts.queuedCommentInterruptId && issue.status === "done") {
|
|
const reopened = await issuesSvc.updateForCompany(issue.id, issue.companyId,
|
|
{ status: "todo", executionState: null }, tx, issueActivityPublications, issuePostCommitActions);
|
|
if (!reopened) throw conflict("The task changed before its saved message could start");
|
|
await logActivity(tx as unknown as Db, {
|
|
companyId: issue.companyId, actorType: "user", actorId: opts.requestedByActorId!,
|
|
action: "issue.updated", entityType: "issue", entityId: issue.id,
|
|
details: { status: "todo", previousStatus: "done", source: "queued_comment_send", queueId: opts.queuedCommentInterruptId },
|
|
}, issueActivityPublications);
|
|
}
|
|
|
|
const wakeupValues = {
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason,
|
|
payload,
|
|
status: "queued" as const,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
};
|
|
// A deferred durable chat wake already owns this ID and source.
|
|
// Promote that receipt in the same transaction as its successor.
|
|
const wakeupRequest = durableRequest && executionWaitRequestId === durableRequest.id
|
|
? await tx.update(agentWakeupRequests).set({
|
|
payload, status: "queued", error: null, finishedAt: null, updatedAt: new Date(),
|
|
}).where(and(eq(agentWakeupRequests.id, durableRequest.id),
|
|
eq(agentWakeupRequests.companyId, agent.companyId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId)))
|
|
.returning().then(rows => rows[0])
|
|
: await tx.insert(agentWakeupRequests).values(wakeupValues).returning().then(rows => rows[0]);
|
|
if (!wakeupRequest) return { kind: "deferred" as const };
|
|
|
|
// A handoff changes the executor, not the owner of saved user input.
|
|
// Validate its exact stopped source while the issue row is locked;
|
|
// unrelated agents and dedicated continuations keep their own wakes.
|
|
const interruptedRunId = readNonEmptyString(enrichedContextSnapshot.interruptedRunId);
|
|
const handoffSource = source === "assignment" && reason === "issue_assigned" &&
|
|
issue.assigneeAgentId === agentId && interruptedRunId && isUuidLike(interruptedRunId)
|
|
? await tx.select({ agentId: heartbeatRuns.agentId }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, interruptedRunId), eq(heartbeatRuns.companyId, issue.companyId),
|
|
eq(heartbeatRuns.status, "cancelled"), eq(heartbeatRuns.errorCode, "issue_reassigned"),
|
|
ne(heartbeatRuns.agentId, agentId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`,
|
|
or(isNull(heartbeatRuns.nativeIssueId), eq(heartbeatRuns.nativeIssueId, issue.id)),
|
|
)).then(rows => rows[0] ?? null)
|
|
: null;
|
|
// External input owns an exact authorized source batch. Board input
|
|
// must retain its separate receipt, including during normal resume.
|
|
const canCoalesceComments = !durableRequest && !isConversation(issue) && opts.allowRunCoalescing !== false;
|
|
// A resumed receipt already passed admission under this issue lock.
|
|
// Consume it with its successor even when chat keeps other messages
|
|
// in separate turns, or finalization will deliver it a second time.
|
|
const pendingComments =
|
|
(executionWaitRequestId || canCoalesceComments) &&
|
|
(ownsReconstructionContinuation || !(await getExecutionBlocker(tx as unknown as Db, issue.companyId, issue.id)))
|
|
? await tx
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, issue.companyId),
|
|
inArray(agentWakeupRequests.agentId, handoffSource ? [agentId, handoffSource.agentId] : [agentId]),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
|
|
// Chat Retry already pins its complete authorized source
|
|
// batch. Keep separate board input for its own admission.
|
|
canCoalesceComments || (opts.taskRetryRequestId && !failedChatRetry)
|
|
? undefined : eq(agentWakeupRequests.id, executionWaitRequestId!),
|
|
),
|
|
)
|
|
.orderBy(asc(agentWakeupRequests.requestedAt))
|
|
: [];
|
|
const adoptedComments = pendingComments.filter((wake) => {
|
|
if (wake.id === executionWaitRequestId) return true;
|
|
if (wake.id === opts.queuedCommentInterruptId || wake.id === opts.queuedCommentRequestId) return true;
|
|
const deferredPayload = parseObject(wake.payload);
|
|
const deferredContext = parseObject(
|
|
deferredPayload[DEFERRED_WAKE_CONTEXT_KEY],
|
|
);
|
|
// Dedicated interaction wakes carry their own source and session
|
|
// contract. ID-only adoption must not erase that continuation.
|
|
return (
|
|
// Durable chat work must keep its receipt, actor, source, and
|
|
// session contract through normal promotion and authorization.
|
|
!wake.idempotencyKey?.startsWith("chat-inbound:") &&
|
|
!isInteractionResolutionWakePayload(deferredPayload) &&
|
|
!hasInteractionContinuationWakeContext(deferredContext) &&
|
|
["issue_commented", "issue_reopened_via_comment"].includes(String(deferredContext.wakeReason ?? wake.reason)) &&
|
|
queuedCommentIdsFromWakePayload(wake.payload).length > 0
|
|
);
|
|
});
|
|
let adoptedCommentIds = [
|
|
...new Set([
|
|
...adoptedComments.flatMap((wake) =>
|
|
queuedCommentIdsFromWakePayload(wake.payload),
|
|
),
|
|
...queuedCommentIdsFromRunContext(enrichedContextSnapshot),
|
|
]),
|
|
];
|
|
if (opts.queuedCommentRequestId || (opts.taskRetryRequestId && !failedChatRetry) || handoffSource) {
|
|
adoptedCommentIds = await undeliveredLegacyUserCommentIds(tx as unknown as Db,
|
|
agent.companyId, issueId, agentId, adoptedCommentIds);
|
|
}
|
|
await notifyChatRunMilestones(tx, { companyId: agent.companyId, issueId: issueId ?? undefined });
|
|
const newRun = await tx
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
...(explicitContinuation ? { id: explicitContinuationRunId } : {}),
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
scopeKind: readNonEmptyString(enrichedContextSnapshot.issueId) ? "issue" : "company",
|
|
issueId: readNonEmptyString(enrichedContextSnapshot.issueId),
|
|
invocationSource: source,
|
|
triggerDetail,
|
|
status: "queued",
|
|
responsibleUserId: await resolveQueuedResponsibleUserId(),
|
|
wakeupRequestId: wakeupRequest.id,
|
|
retryOfRunId: failedChatRetry
|
|
? durableRequest!.failedRunRetry!.failedRunId
|
|
: opts.failedRunId ?? automaticParentRunId,
|
|
contextSnapshot: adoptedComments.length
|
|
? withQueuedCommentIdsInRunContext(
|
|
enrichedContextSnapshot,
|
|
adoptedCommentIds,
|
|
adoptedComments.map((wake) => wake.id),
|
|
)
|
|
: enrichedContextSnapshot,
|
|
sessionIdBefore: explicitContinuation ? null : sessionBefore,
|
|
continuationAttempt,
|
|
...(reconciledSourceRunId
|
|
? { retryOfRunId: reconciledSourceRunId }
|
|
: {}),
|
|
...(reconciledRestoreRetryCount !== null ? {
|
|
scheduledRetryAttempt: reconciledRestoreRetryCount,
|
|
scheduledRetryReason: "transient_failure",
|
|
} : {}),
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]);
|
|
|
|
if (ownsReconstructionContinuation) {
|
|
await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
|
|
continuationDelivery: "delivered", continuationRunId: newRun.id,
|
|
})}::jsonb` }).where(and(eq(issueRecoveryActions.companyId, issue.companyId),
|
|
eq(issueRecoveryActions.id, String(enrichedContextSnapshot.recoveryActionId)),
|
|
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`));
|
|
}
|
|
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
runId: newRun.id,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
|
|
|
|
if (failedChatRetry && opts.taskRetryRequestId) {
|
|
await tx.update(agentWakeupRequests).set({
|
|
status: "coalesced", runId: newRun.id, finishedAt: new Date(), updatedAt: new Date(),
|
|
}).where(and(eq(agentWakeupRequests.companyId, issue.companyId),
|
|
eq(agentWakeupRequests.id, opts.taskRetryRequestId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
}
|
|
|
|
if (adoptedComments.length) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "coalesced",
|
|
runId: newRun.id,
|
|
finishedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
inArray(
|
|
agentWakeupRequests.id,
|
|
adoptedComments.map((wake) => wake.id),
|
|
),
|
|
);
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
payload: withQueuedCommentIdsInWakePayload(payload, adoptedCommentIds),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
|
|
}
|
|
|
|
// executionRunId is NOT stamped here (enqueueWakeup queues the run but
|
|
// doesn't start it). It will be stamped in claimQueuedRun() once the run
|
|
// transitions to "running" — Fix A (lazy locking).
|
|
|
|
return { kind: "queued" as const, run: newRun };
|
|
}))
|
|
.catch((error) => {
|
|
if (isExternalChatWaitAuthorizationContention(error))
|
|
return { kind: "deferred" as const };
|
|
throw error;
|
|
});
|
|
|
|
// The admission has committed. What follows is dispatch of saved rows.
|
|
if (outcome.kind !== "deferred" && outcome.kind !== "skipped") admission?.saved();
|
|
for (const publication of issueActivityPublications) publishActivity(publication);
|
|
await executeIssuePostCommitActions(db, issuePostCommitActions);
|
|
|
|
// Telemetry for the cancelled runs is best-effort background work.
|
|
// Fire it here and never await it: none of the lifecycle work below,
|
|
// nor this function's return, depends on it, so a slow telemetry
|
|
// lookup must not delay them.
|
|
for (const cancelledRun of cancelledRunsToEmit) {
|
|
void emitAgentTaskRun(db, cancelledRun);
|
|
}
|
|
|
|
if (outcome.kind === "durable") {
|
|
return outcome.receipt.runId ? getRun(outcome.receipt.runId) : null;
|
|
}
|
|
if (outcome.kind === "deferred" || outcome.kind === "skipped") {
|
|
return null;
|
|
}
|
|
if (outcome.kind === "coalesced") {
|
|
await attachQueuedSetupRequest(agent);
|
|
await startNextQueuedRunForAgent(agent.id);
|
|
return outcome.run;
|
|
}
|
|
if (outcome.kind === "replayed") {
|
|
if (outcome.run.status === "queued") {
|
|
await attachQueuedSetupRequest(agent);
|
|
await startNextQueuedRunForAgent(agent.id);
|
|
}
|
|
return outcome.run;
|
|
}
|
|
|
|
const newRun = outcome.run;
|
|
publishLiveEvent({
|
|
companyId: newRun.companyId,
|
|
type: "heartbeat.run.queued",
|
|
payload: {
|
|
runId: newRun.id,
|
|
agentId: newRun.agentId,
|
|
invocationSource: newRun.invocationSource,
|
|
triggerDetail: newRun.triggerDetail,
|
|
wakeupRequestId: newRun.wakeupRequestId,
|
|
},
|
|
});
|
|
|
|
await attachQueuedSetupRequest(agent);
|
|
await startNextQueuedRunForAgent(agent.id);
|
|
return newRun;
|
|
}
|
|
|
|
if (durableRequest) throw new Error("chat_inbound_wakeup_binding_denied");
|
|
|
|
const activeRuns = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.agentId, agentId),
|
|
inArray(heartbeatRuns.status, [
|
|
...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES,
|
|
]),
|
|
),
|
|
)
|
|
.orderBy(desc(heartbeatRuns.createdAt));
|
|
|
|
const sameScopeQueuedRun = activeRuns.find(
|
|
(candidate) =>
|
|
candidate.status === "queued" &&
|
|
isSameTaskScope(runTaskKey(candidate), taskKey),
|
|
);
|
|
const sameScopeScheduledRetryRun = activeRuns.find(
|
|
(candidate) =>
|
|
candidate.status === "scheduled_retry" &&
|
|
isSameTaskScope(runTaskKey(candidate), taskKey),
|
|
);
|
|
const sameScopeRunningRun = activeRuns.find(
|
|
(candidate) =>
|
|
candidate.status === "running" &&
|
|
isSameTaskScope(runTaskKey(candidate), taskKey),
|
|
);
|
|
const shouldQueueFollowupForRunningWake =
|
|
Boolean(sameScopeRunningRun) &&
|
|
!sameScopeQueuedRun &&
|
|
shouldQueueFollowupForRunningIssueWake({
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
wakeCommentId,
|
|
});
|
|
// Unscoped manual wakes need their own receipt and execution identity too.
|
|
const rawCoalescedTarget =
|
|
opts.allowRunCoalescing === false || opts.manualUserWake
|
|
? null
|
|
: (sameScopeQueuedRun ??
|
|
sameScopeScheduledRetryRun ??
|
|
(shouldQueueFollowupForRunningWake
|
|
? null
|
|
: (sameScopeRunningRun ?? null)));
|
|
|
|
// A reviewed chat run attests one conversation, so never fold another
|
|
// conversation's reply into it; that reply queues its own run instead.
|
|
const coalescedTargetRun = rawCoalescedTarget &&
|
|
(hasRetainedExternalChatRetry(rawCoalescedTarget.contextSnapshot) ||
|
|
wakeChatConversationsConflict(rawCoalescedTarget.contextSnapshot, enrichedContextSnapshot))
|
|
? null
|
|
: filterZombieCoalesceTarget(rawCoalescedTarget, liveRunExecutions);
|
|
|
|
if (coalescedTargetRun) {
|
|
const mergedContextSnapshot = mergeCoalescedContextSnapshot(
|
|
coalescedTargetRun.contextSnapshot,
|
|
enrichedContextSnapshot,
|
|
{
|
|
preserveExistingInteractionContinuation:
|
|
coalescedTargetRun.status === "queued" ||
|
|
coalescedTargetRun.status === "scheduled_retry",
|
|
},
|
|
);
|
|
const mergedRun = await db
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
contextSnapshot: mergedContextSnapshot,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(heartbeatRuns.id, coalescedTargetRun.id))
|
|
.returning()
|
|
.then((rows) => rows[0] ?? coalescedTargetRun);
|
|
|
|
await db.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason,
|
|
payload,
|
|
status: "coalesced",
|
|
coalescedCount: 1,
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
runId: mergedRun.id,
|
|
finishedAt: new Date(),
|
|
});
|
|
return mergedRun;
|
|
}
|
|
|
|
const queueOutcome = await db.transaction(async (tx) => {
|
|
await notifyRunDispatch(tx, { companyId: agent.companyId });
|
|
await tx.execute(
|
|
sql`select id from agents where id = ${agentId} and company_id = ${agent.companyId} for update`,
|
|
);
|
|
|
|
const dailyCapBlock = await getHeartbeatDailyCapBlock(
|
|
agent,
|
|
policy,
|
|
{},
|
|
tx,
|
|
);
|
|
if (dailyCapBlock) {
|
|
if (opts.idempotencyKey?.startsWith(CONVERSATION_COMMENT_KEY_PREFIX)) throw dailyLimitConversationRefusal(dailyCapBlock);
|
|
const now = new Date();
|
|
await tx.insert(agentWakeupRequests).values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason: dailyCapBlock.reason,
|
|
payload: {
|
|
...(payload ?? {}),
|
|
heartbeatSkip: {
|
|
reason:
|
|
"Per-agent heartbeat daily cap reached before adapter invocation.",
|
|
observed: dailyCapBlock.observed,
|
|
limit: dailyCapBlock.limit,
|
|
},
|
|
},
|
|
status: "skipped",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
finishedAt: now,
|
|
});
|
|
if (source === "timer") {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
await tx
|
|
.update(agents)
|
|
.set({
|
|
lastHeartbeatAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(agents.id, agentId));
|
|
}
|
|
return { kind: "skipped" as const };
|
|
}
|
|
|
|
const wakeupRequest = await tx
|
|
.insert(agentWakeupRequests)
|
|
.values({
|
|
...durableReceiptFields,
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
source,
|
|
triggerDetail,
|
|
reason,
|
|
payload,
|
|
status: "queued",
|
|
requestedByActorType: opts.requestedByActorType ?? null,
|
|
requestedByActorId: opts.requestedByActorId ?? null,
|
|
idempotencyKey: opts.idempotencyKey ?? null,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]);
|
|
|
|
const newRun = await tx
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
companyId: agent.companyId,
|
|
agentId,
|
|
scopeKind: readNonEmptyString(enrichedContextSnapshot.issueId) ? "issue" : "company",
|
|
issueId: readNonEmptyString(enrichedContextSnapshot.issueId),
|
|
invocationSource: source,
|
|
triggerDetail,
|
|
status: "queued",
|
|
responsibleUserId: await resolveQueuedResponsibleUserId(),
|
|
wakeupRequestId: wakeupRequest.id,
|
|
contextSnapshot: enrichedContextSnapshot,
|
|
sessionIdBefore: sessionBefore,
|
|
continuationAttempt,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]);
|
|
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
runId: newRun.id,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
|
|
|
|
return { kind: "queued" as const, run: newRun };
|
|
});
|
|
|
|
if (queueOutcome.kind === "skipped") return null;
|
|
admission?.saved();
|
|
const newRun = queueOutcome.run;
|
|
|
|
publishLiveEvent({
|
|
companyId: newRun.companyId,
|
|
type: "heartbeat.run.queued",
|
|
payload: {
|
|
runId: newRun.id,
|
|
agentId: newRun.agentId,
|
|
invocationSource: newRun.invocationSource,
|
|
triggerDetail: newRun.triggerDetail,
|
|
wakeupRequestId: newRun.wakeupRequestId,
|
|
},
|
|
});
|
|
|
|
await attachQueuedSetupRequest(agent);
|
|
await startNextQueuedRunForAgent(agent.id);
|
|
|
|
return newRun;
|
|
}
|
|
|
|
/**
|
|
* Native status commitment deliberately persists dependency/parent wake
|
|
* intents in the same transaction as the authoritative status projection.
|
|
* Those rows are not runnable until the heartbeat scheduler has applied its
|
|
* normal policy, workspace, concurrency, and responsible-user checks. Bridge
|
|
* the durable intent into that scheduler here instead of treating a bare
|
|
* `agent_wakeup_requests` row as if it were already a queued heartbeat run.
|
|
*
|
|
* The intent is claimed before dispatch. A deterministic dispatcher actor id
|
|
* lets a later sweep recover the narrow process-crash window after the real
|
|
* wake was inserted but before the intent was linked to it. Dispatch failure
|
|
* only requeues the intent; it never changes the provider run outcome.
|
|
*/
|
|
type NativeStatusWakeDispatchInput = {
|
|
companyId?: string;
|
|
limit?: number;
|
|
staleClaimMs?: number;
|
|
requestId?: string;
|
|
onDeferredDeadline?: (at: number) => void;
|
|
onError?: (error: unknown) => void;
|
|
};
|
|
type NativeStatusWakeDelivery = { runId: string | null; status: string };
|
|
|
|
function createNativeStatusWakeBatch() {
|
|
const deliveredByIssueScope = new Map<string, NativeStatusWakeDelivery>();
|
|
return (input: NativeStatusWakeDispatchInput = {}) => dispatchNativeStatusWakeBatch(input, deliveredByIssueScope);
|
|
}
|
|
|
|
function throttledDispatchUntil(dispatch: typeof agentWakeupRequests.$inferSelect): number | null {
|
|
if (dispatch.status !== "skipped" || dispatch.reason !== "issue_rewake_throttled") return null;
|
|
const at = Date.parse(String(parseObject(parseObject(dispatch.payload).heartbeatSkip).nextAllowedAt ?? ""));
|
|
return Number.isFinite(at) ? at : null;
|
|
}
|
|
|
|
function dispatchPendingNativeStatusWakeups(input: NativeStatusWakeDispatchInput = {}) {
|
|
return createNativeStatusWakeBatch()(input);
|
|
}
|
|
|
|
async function dispatchNativeStatusWakeBatch(input: NativeStatusWakeDispatchInput,
|
|
deliveredByIssueScope: Map<string, NativeStatusWakeDelivery>,
|
|
) {
|
|
const now = new Date();
|
|
const staleClaimMs = Math.max(1_000, input.staleClaimMs ?? 60_000);
|
|
const candidates = await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
input.companyId
|
|
? eq(agentWakeupRequests.companyId, input.companyId)
|
|
: undefined,
|
|
input.requestId ? eq(agentWakeupRequests.id, input.requestId) : undefined,
|
|
eq(agentWakeupRequests.requestedByActorType, "system"),
|
|
eq(agentWakeupRequests.requestedByActorId, "native-status-committer"),
|
|
inArray(agentWakeupRequests.status, ["queued", "claimed"]),
|
|
isNull(agentWakeupRequests.runId),
|
|
),
|
|
)
|
|
.orderBy(asc(agentWakeupRequests.requestedAt))
|
|
.limit(Math.max(1, Math.min(input.limit ?? 100, 500)));
|
|
|
|
let dispatched = 0;
|
|
let recovered = 0;
|
|
let deferred = 0;
|
|
|
|
for (const candidate of candidates) {
|
|
const payload = parseObject(candidate.payload);
|
|
const wakeContext = parseObject(payload._paperclipWakeContext);
|
|
const issueId =
|
|
readNonEmptyString(payload.issueId) ??
|
|
readNonEmptyString(payload.taskId) ??
|
|
readNonEmptyString(wakeContext.issueId) ??
|
|
null;
|
|
const dispatchActorId = `native-status-wake-dispatch:${candidate.id}`;
|
|
const existingDispatch = await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, candidate.companyId),
|
|
eq(agentWakeupRequests.requestedByActorType, "system"),
|
|
eq(agentWakeupRequests.requestedByActorId, dispatchActorId),
|
|
),
|
|
)
|
|
.orderBy(desc(agentWakeupRequests.requestedAt))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
const throttledUntil = existingDispatch ? throttledDispatchUntil(existingDispatch) : null;
|
|
if (throttledUntil !== null && throttledUntil > now.getTime()) {
|
|
input.onDeferredDeadline?.(throttledUntil);
|
|
deferred += 1;
|
|
continue;
|
|
}
|
|
|
|
if (existingDispatch && throttledUntil === null) {
|
|
const recoveredStatus = existingDispatch.runId
|
|
? "coalesced"
|
|
: existingDispatch.status === "deferred_issue_execution"
|
|
? "coalesced"
|
|
: existingDispatch.status;
|
|
await db.transaction(async tx => {
|
|
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: recoveredStatus,
|
|
runId: existingDispatch.runId,
|
|
finishedAt:
|
|
(existingDispatch.runId ?? existingDispatch.finishedAt)
|
|
? (existingDispatch.finishedAt ?? now)
|
|
: null,
|
|
error: existingDispatch.error,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, candidate.id),
|
|
isNull(agentWakeupRequests.runId),
|
|
),
|
|
);
|
|
});
|
|
recovered += 1;
|
|
continue;
|
|
}
|
|
|
|
if (
|
|
candidate.status === "claimed" &&
|
|
candidate.claimedAt &&
|
|
now.getTime() - candidate.claimedAt.getTime() < staleClaimMs
|
|
) {
|
|
input.onDeferredDeadline?.(candidate.claimedAt.getTime() + staleClaimMs);
|
|
deferred += 1;
|
|
continue;
|
|
}
|
|
|
|
const claimed = await db
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "claimed",
|
|
claimedAt: now,
|
|
error: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, candidate.id),
|
|
isNull(agentWakeupRequests.runId),
|
|
candidate.status === "claimed"
|
|
? eq(agentWakeupRequests.status, "claimed")
|
|
: eq(agentWakeupRequests.status, "queued"),
|
|
),
|
|
)
|
|
.returning({ id: agentWakeupRequests.id });
|
|
if (claimed.length === 0) continue;
|
|
|
|
const scopeKey = issueId
|
|
? `${candidate.companyId}:${candidate.agentId}:${issueId}:${readNativeReviewAssignmentContext(wakeContext)?.nativeReviewInteractionId ?? ""}`
|
|
: null;
|
|
const priorDelivery = scopeKey
|
|
? deliveredByIssueScope.get(scopeKey)
|
|
: null;
|
|
if (priorDelivery) {
|
|
await db.transaction(async tx => {
|
|
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "coalesced",
|
|
runId: priorDelivery.runId,
|
|
finishedAt: new Date(),
|
|
error: null,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, candidate.id));
|
|
});
|
|
recovered += 1;
|
|
continue;
|
|
}
|
|
|
|
let completedOnboardingGuard: WakeupOptions["issueStateGuard"];
|
|
if (issueId) {
|
|
const targetIssue = await db
|
|
.select({
|
|
status: issues.status,
|
|
statusVersion: issues.statusVersion,
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
})
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
eq(issues.companyId, candidate.companyId),
|
|
),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
const nativeReview = candidate.reason === "native_completion_review"
|
|
? await getNativeReviewAssignment(db, {
|
|
companyId: candidate.companyId, issueId, agentId: candidate.agentId,
|
|
contextSnapshot: wakeContext,
|
|
})
|
|
: null;
|
|
const onboardingResultReport = targetIssue?.status === "done" && await isCompletedOnboardingHandoffWake(db, {
|
|
companyId: candidate.companyId, issueId, agentId: candidate.agentId,
|
|
reason: candidate.reason, contextSnapshot: wakeContext,
|
|
});
|
|
if (onboardingResultReport) completedOnboardingGuard = { assigneeAgentId: candidate.agentId, statuses: ["done"], statusVersion: targetIssue!.statusVersion };
|
|
if (
|
|
!targetIssue ||
|
|
(["done", "cancelled"].includes(targetIssue.status) && !onboardingResultReport) ||
|
|
(targetIssue.assigneeAgentId !== candidate.agentId && !nativeReview) ||
|
|
(candidate.reason === "native_completion_review" && !nativeReview)
|
|
) {
|
|
await db.transaction(async tx => {
|
|
await notifyChatInboundWakeNotices(tx, { companyId: candidate.companyId, issueId });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "skipped",
|
|
finishedAt: new Date(),
|
|
error: !targetIssue
|
|
? "Native status wake target no longer exists"
|
|
: ["done", "cancelled"].includes(targetIssue.status)
|
|
? `Native status wake target is already ${targetIssue.status}`
|
|
: "Native status wake target has a different assignee",
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, candidate.id));
|
|
});
|
|
recovered += 1;
|
|
continue;
|
|
}
|
|
}
|
|
|
|
try {
|
|
const wakeRun = await enqueueWakeup(candidate.agentId, {
|
|
allowRunCoalescing: false,
|
|
source: candidate.source as WakeupOptions["source"],
|
|
triggerDetail: (candidate.triggerDetail ??
|
|
"system") as WakeupOptions["triggerDetail"],
|
|
reason: candidate.reason,
|
|
payload,
|
|
idempotencyKey: candidate.idempotencyKey,
|
|
requestedByActorType: "system",
|
|
requestedByActorId: dispatchActorId,
|
|
...(completedOnboardingGuard ? { issueStateGuard: completedOnboardingGuard } : {}),
|
|
contextSnapshot: {
|
|
...wakeContext,
|
|
...(issueId ? { issueId, taskId: issueId } : {}),
|
|
wakeReason: candidate.reason,
|
|
source: "native_status_decision",
|
|
statusDecisionSource: "native_status_decision",
|
|
nativeStatusWakeIntentId: candidate.id,
|
|
},
|
|
}, undefined, candidate.payload?.interactionOrigin);
|
|
|
|
const delivered = await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, candidate.companyId),
|
|
eq(agentWakeupRequests.requestedByActorType, "system"),
|
|
eq(agentWakeupRequests.requestedByActorId, dispatchActorId),
|
|
),
|
|
)
|
|
.orderBy(desc(agentWakeupRequests.requestedAt))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
const throttledDeliveryUntil = !wakeRun && delivered ? throttledDispatchUntil(delivered) : null;
|
|
if (throttledDeliveryUntil !== null) {
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ status: "queued", claimedAt: null, finishedAt: null, error: null, updatedAt: new Date() })
|
|
.where(and(eq(agentWakeupRequests.id, candidate.id), isNull(agentWakeupRequests.runId)));
|
|
input.onDeferredDeadline?.(throttledDeliveryUntil);
|
|
deferred += 1;
|
|
continue;
|
|
}
|
|
|
|
// The committer intent is the durable outbox entry. When admission is
|
|
// blocked, enqueueWakeup creates a separate deferred dispatch receipt;
|
|
// leave the original intent coalesced so the release drain cannot
|
|
// promote both rows (the original has no nativeStatusWakeIntentId
|
|
// provenance and would otherwise run once before the dispatch receipt).
|
|
const deliveredStatus = delivered?.status ?? "queued";
|
|
await db.transaction(async tx => {
|
|
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: wakeRun || deliveredStatus === "deferred_issue_execution"
|
|
? "coalesced"
|
|
: deliveredStatus,
|
|
runId: wakeRun?.id ?? delivered?.runId ?? null,
|
|
finishedAt:
|
|
wakeRun || delivered?.finishedAt
|
|
? (delivered?.finishedAt ?? new Date())
|
|
: null,
|
|
error: delivered?.error ?? null,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(agentWakeupRequests.id, candidate.id));
|
|
});
|
|
|
|
if (scopeKey) {
|
|
deliveredByIssueScope.set(scopeKey, {
|
|
runId: wakeRun?.id ?? delivered?.runId ?? null,
|
|
status: wakeRun ? "coalesced" : (delivered?.status ?? "queued"),
|
|
});
|
|
}
|
|
|
|
if (wakeRun) dispatched += 1;
|
|
else deferred += 1;
|
|
} catch (error) {
|
|
input.onError?.(error);
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "queued",
|
|
claimedAt: null,
|
|
error:
|
|
error instanceof Error
|
|
? error.message.slice(0, 1_000)
|
|
: String(error).slice(0, 1_000),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, candidate.id),
|
|
eq(agentWakeupRequests.status, "claimed"),
|
|
isNull(agentWakeupRequests.runId),
|
|
),
|
|
);
|
|
logger.warn(
|
|
{
|
|
err: error,
|
|
wakeupRequestId: candidate.id,
|
|
agentId: candidate.agentId,
|
|
},
|
|
"failed to dispatch persisted native status wake intent",
|
|
);
|
|
deferred += 1;
|
|
}
|
|
}
|
|
|
|
return { scanned: candidates.length, dispatched, recovered, deferred };
|
|
}
|
|
|
|
return {
|
|
enqueueWakeup,
|
|
dispatchPendingNativeStatusWakeups,
|
|
createNativeStatusWakeBatch,
|
|
startNextQueuedRunForAgent,
|
|
countRunningRunsForAgent,
|
|
releaseRunClaimedJustBeforeSuppression,
|
|
claimQueuedRun,
|
|
withChatControlRecoveryGate,
|
|
trackWakeup,
|
|
resumeQueuedRuns,
|
|
resumeDeferredExecutionReceipts,
|
|
resumeQueuedRunsAfterDrain,
|
|
resumeFreshQueuedRunsAtStartup,
|
|
parseHeartbeatPolicy,
|
|
nextTimerHeartbeatAt,
|
|
claimDueTimerHeartbeat,
|
|
};
|
|
}
|