mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 13:20:54 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs can retain accepted results after an unsafe workspace export fails. > - Historical recovery can retire an old notice or resume the accepted result without another provider turn. > - The periodic reaper repeatedly scans failures that cannot currently recover. > - Existing rows and committed changes can identify the actual recovery obligation. > - This pull request wakes recovery from those changes and retries only owed work. > - Dormant history becomes quiet while ownership and explicit operator decisions remain protected. ## Linked Issues or Issue Description **What happened?** The native finalization reaper repeatedly visits historical unsafe workspace-export failures. Cancelled tasks, newer runs, changed contracts, and explicit operator decisions can make those rows permanently ineligible. They still produce periodic database reads. **What did you expect?** Retire stale unsafe-export notices and recover eligible accepted results. Stop reading dormant history until a relevant committed change occurs. Keep a retry while a real obligation is blocked by ownership or a transient failure. **Steps to reproduce** Retain an unsafe-export terminal failure whose task is cancelled or superseded. Leave the instance idle after startup. The periodic reaper continues reading that history. Prerequisite #17680 is merged into master. Related: #14402 established safe export finalization; #14329 was an earlier historical recovery proposal. #17679 owns separate policy-review cleanup. This PR changes only historical unsafe-export scheduling and its producers. ## What Changed - Register a scoped worker through the existing coordinator and scheduler. Remove this duty from periodic reaping. Preserve the manual/startup recovery entry point. - Distinguish stale notices, eligible accepted results, dormant history, and concrete ownership contention. Reuse the existing mutation owner and lock order. Preserve explicit settlement vetoes and exact result/contract identity. - Signal relevant task, run, action, and deletion writes before they commit. Existing transaction boundaries handle nested commits and uncertain outcomes. Notifications contain only scope IDs. - Retain a retry only for owed work. Reconstruct work at startup, preserve drain/resume, and stop through the shared lifecycle. - Add concurrent, idempotent partial index migration 0410 for unsafe terminal failures. Keep the five newest generated schema snapshots. - Document producer coverage, query plans, supported writer boundaries, and test failures in the unsafe-export audit. ## Verification - Source `2a13dd08a6cc52f0adab09e3df9ce31e97b746b9` passes all 2188 cases across 24 complete files, without skips. The main selection passes 2129 cases in 192.58 seconds; safe-replacement passes 59 in 26.04 seconds. Coverage includes all 968 native-session cases, real export/action producers, explicit continuation, workspace reconstruction, interaction routing, Stop, queue and Slack consumers. The audit gives per-file counts. - Twenty separate fresh PostgreSQL 17 databases migrate through 0410 before setup. Four embedded-only fixture imports temporarily use their own external database on this IPC-exhausted host, then are restored byte for byte. Assertions and timing limits stay unchanged. This is qualified external-database acceptance, not an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-head migration order pass. Normal integration preserves the complete original runtime/schema patch and unrelated upstream blobs. Original concurrent index SQL, all prior executable history, journal entries and retained snapshot bytes are unchanged; five snapshots 0406 through 0410 remain. Added comment removal preserves the full non-comment TypeScript token streams. - A populated 0409-to0410 upgrade preserves all 408 prior migration receipts and 38 companies. It appends exactly one receipt; replay adds none. The index is valid and ready on fresh and upgraded databases. - The owning audit retains earlier fixture failures, skipped checks and notice-retirement regression history. Older combined stack checks retain their original scope in their owning audits. Actual parent `73d24aa5acfcf17b3f676b4da7cd05eb70b60b99` is reconciled through a normal merge. Its tree equals the reviewed ancestor and the complete accepted child tree is unchanged. Fresh published-head CI and Greptile 5/5 remain required. No deployed sleep, fleet-cost or live provider qualification claim is made. - Inherited prefix correction: current integration `8f43c9f81fac23b50997e7f0e1d0d9cf383a8109` changes only two values in a previously unselected corpus fixture. All production code, the complete package tree and all 24 accepted test files remain byte-identical to the 2188-case source above. The additional complete 72-case corpus passes in 14.00 seconds and again on the same database in 13.08 seconds. Retained PSC/OUT and sentinel rows remain unchanged. The repeat is not double-counted. The parent audit retains the original collision and rejected cleanup attempt. - Latest parent context: integration `4d51231abe8b68c21ce896d0d2ac16865978c8ab` carries only the parent's one-argument GitHub observation correction and an independent master cleanup assertion. All production code, packages, 24 accepted suite files and the additional corpus are unchanged. The two focused native-presentation cases pass in 15.14 seconds and the complete cleanup-removal suite passes all six cases in 8.31 seconds on two fresh 0410 PostgreSQL fixtures. This does not claim the full parent workflow was rerun here. Current head `b28399babac72ba9db1c4a4ef882c5e87aded38c` has passing CI and Greptile 5/5 with no actionable findings. The first CI attempt lost runners: fourteen jobs explicitly received simultaneous shutdown signals, two reported lost runner communication, and two aggregate gates failed downstream. One failed-job retry on unchanged source passes. The underlying host shutdown cause remains unconfirmed; no code fix or extra retry is claimed. [CI run and attempts](https://github.com/paperclipai/paperclip/actions/runs/40819408528). No Ponytail verdict is present. ## Risks - A missing future producer can delay dormant-history discovery until restart. The audit names current producers. Independent SQL writers and other server processes remain outside the process-local signal contract. - Physical ownership and uncertain operations retain retries. Time alone does not authorize taking workspace ownership. No provider turn or unsafe archive retry is added. - The recovery owner remains authoritative if state changes after discovery. Terminal UI status does not authorize resource destruction or accepted-result replay. - The partial index adds matching-row write/storage cost. It builds concurrently through the existing restart-safe migration runner. - This proves the changed worker becomes quiet. It does not prove host suspension, external wake coverage, or fleet cost savings. Other sleep migrations remain. This slice still requires fresh current-head CI and reviews. ## Model Used OpenAI Codex, based on GPT-6, with reasoning and code-execution tools. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1336 lines
59 KiB
TypeScript
1336 lines
59 KiB
TypeScript
import { notifyUnsafeWorkspaceExportRecovery } from "../../../services/unsafe-workspace-export-notifications.js";
|
|
import { notifyCancelledWorkspacePreservation } from "../../../services/cancelled-workspace-preservation-notifications.js";
|
|
import { notifyInstructionCollection } from "../../../services/instruction-collection-notifications.js";
|
|
import { notifyScheduledRetryCancellation } from "../../../services/scheduled-retry-cancellation-notifications.js";
|
|
import { hasConversationContinuationPolicy } from "../../../services/conversation-continuation.js";
|
|
import { getExecutionBlocker } from "../../../services/execution-blocker.js";
|
|
import { validateReconstructedWorkspaceDispatch } from "../../../services/native-runtime/native-workspace-reconstruction.js";
|
|
import { getNativeReviewAssignment } from "../../../services/native-runtime/native-review-participant.js";
|
|
import { and, asc, eq, gte, inArray, lte, or, sql } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
agentWakeupRequests,
|
|
agents,
|
|
budgetReservations,
|
|
companies,
|
|
connectionIntentDeliveries,
|
|
heartbeatRuns,
|
|
issueRecoveryActions,
|
|
issueComments,
|
|
issues,
|
|
nativeRunFinalizations,
|
|
projects,
|
|
} from "@paperclipai/db";
|
|
import { ISSUE_DISPOSITION_REPAIR_RETRY_REASON } from "@paperclipai/shared";
|
|
import { withAccountingTransaction } from "../../../services/accounting-transaction.js";
|
|
import type { ActivityPublication } from "../../../services/activity-log.js";
|
|
import { parseObject } from "../../../adapters/utils.js";
|
|
import { evaluateAgentInvokabilityFromDb } from "../../../services/agent-invokability.js";
|
|
import { budgetService, budgetServiceInTransaction, recheckAdmittedRunBudgetReadOnly } from "../../../services/budgets.js";
|
|
import { lockRunBudgetRecoveryOwnerInTransaction, recheckRunBudgetReservationInTransaction } from "../../../services/budget-reservations.js";
|
|
import { isCompletedOnboardingHandoffWake } from "../../../services/chat-completion-delivery.js";
|
|
import { isHeartbeatWakeOnDemandEnabled } from "../../../services/heartbeat-policy.js";
|
|
import { notifyChatRunMilestones } from "../../../services/chat-run-milestone-notifications.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../../services/delivery-work-notifications.js";
|
|
import { connectionContinuationPendingResponse } from "../../../services/satisfied-connection-intents.js";
|
|
import { collectDispositionRepairSourceState } from "../../../services/recovery/disposition-repair.js";
|
|
import { legacyDispositionEpisode, legacyDispositionFingerprint } from "../../../services/recovery/legacy-continuation.js";
|
|
import { appendHeartbeatRunEvent } from "../../../services/heartbeat-run-events.js";
|
|
import { emitAgentTaskRun } from "../../../services/agent-task-run-telemetry.js";
|
|
import { issueService } from "../../../services/issues.js";
|
|
import {
|
|
issueTreeControlService,
|
|
isVerifiedIssueTreeControlInteractionWake,
|
|
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
|
|
} from "../../../services/issue-tree-control.js";
|
|
import {
|
|
continuationSummaryParksExecutor,
|
|
getIssueContinuationSummaryDocument,
|
|
} from "../../../services/issue-continuation-summary.js";
|
|
import { parseIssueExecutionState } from "../../../services/issue-execution-policy.js";
|
|
import { decideQueuedRunStaleness, decideScheduledRetryGate } from "../domain/policy.js";
|
|
import type {
|
|
QueuedRunFacts,
|
|
ReviewParticipantFacts,
|
|
RetryReasonKind,
|
|
ScheduledRetryFacts,
|
|
} from "../domain/policy.js";
|
|
import {
|
|
MAX_TURN_CONTINUATION_RETRY_REASON,
|
|
allowsIssueInteractionWake,
|
|
deriveCommentId,
|
|
isNonAssigneeWorkspaceBusyRetry,
|
|
isResolvedInteractionContinuationWakeContext,
|
|
} from "../domain/wake-context.js";
|
|
import type {
|
|
CancelStaleQueuedRunInput,
|
|
DispatchResolvedInteractionInput,
|
|
DispatchResolvedInteractionOutcome,
|
|
DueRetryRun,
|
|
EvaluateScheduledRetryGateInput,
|
|
ListDueRetriesInput,
|
|
PromoteOrCancelDueRetryInput,
|
|
RunDispatchWriter,
|
|
ScheduledRetryReader,
|
|
} from "../application/ports.js";
|
|
import type {
|
|
CancelStaleQueuedRunOutcome,
|
|
PostCommitEffect,
|
|
PromoteScheduledRetryOutcome,
|
|
} from "../application/types.js";
|
|
import { RunDispatchApplicationError } from "../application/types.js";
|
|
|
|
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
|
|
type LoadGateFactsInput = {
|
|
conversationContinuation: boolean;
|
|
responsibleUserId: string | null;
|
|
runId: string;
|
|
companyId: string;
|
|
agentId: string;
|
|
contextSnapshot: Record<string, unknown>;
|
|
scheduledRetryReason: string | null;
|
|
retryReasonOverride?: string | null;
|
|
wakeupRequestId: string | null;
|
|
};
|
|
type LoadGateFactsResult =
|
|
| { agentFound: true; facts: ScheduledRetryFacts }
|
|
| { agentFound: false; issueId: string | null };
|
|
type LoadStalenessFactsInput = {
|
|
responsibleUserId: string | null;
|
|
conversationContinuation: boolean;
|
|
runId: string;
|
|
companyId: string;
|
|
agentId: string;
|
|
issueId: string;
|
|
contextSnapshot: Record<string, unknown>;
|
|
scheduledRetryReason: string | null;
|
|
};
|
|
type CancelSuppressedRetryInput = {
|
|
runId: string;
|
|
companyId: string;
|
|
now: Date;
|
|
reason: string;
|
|
errorCode: string;
|
|
issueId: string | null;
|
|
details: Record<string, unknown>;
|
|
};
|
|
type CancelSuppressedRetryResult = { applied: true; run: HeartbeatRun } | { applied: false };
|
|
|
|
const NO_REVIEW_PARTICIPANT: ReviewParticipantFacts = {
|
|
isInReview: false,
|
|
hasParticipant: false,
|
|
participantIsAgent: false,
|
|
participantAgentId: null,
|
|
currentStageType: null,
|
|
currentParticipant: null,
|
|
};
|
|
|
|
async function readNativeReviewParticipantFacts(db: Db, input: {
|
|
companyId: string; issueId: string; agentId: string; contextSnapshot: unknown;
|
|
}): Promise<ReviewParticipantFacts | null> {
|
|
const review = await getNativeReviewAssignment(db, input);
|
|
return review ? {
|
|
isInReview: true, hasParticipant: true, participantIsAgent: true,
|
|
participantAgentId: input.agentId, currentStageType: "native_completion_review",
|
|
currentParticipant: { type: "agent", agentId: input.agentId, interactionId: review.interaction.id },
|
|
} : null;
|
|
}
|
|
|
|
function readNonEmptyString(value: unknown): string | null {
|
|
return typeof value === "string" && value.trim().length > 0 ? value : null;
|
|
}
|
|
|
|
function classifyRetryReasonKind(retryReason: string | null): RetryReasonKind {
|
|
if (retryReason === MAX_TURN_CONTINUATION_RETRY_REASON) return "max_turn_continuation";
|
|
if (retryReason === ISSUE_DISPOSITION_REPAIR_RETRY_REASON) return "disposition_repair";
|
|
if ((retryReason === "ai_connection_busy" || retryReason === "ai_connection_pool_wait")) return "ai_connection_wait";
|
|
if (retryReason === "native_safe_replacement" || retryReason === "native_provider_overloaded") return "native_safe_replacement";
|
|
return "other";
|
|
}
|
|
|
|
function buildReviewParticipantFacts(input: {
|
|
isInReview: boolean;
|
|
executionState: ReturnType<typeof parseIssueExecutionState>;
|
|
}): ReviewParticipantFacts {
|
|
const currentParticipant = input.executionState?.currentParticipant ?? null;
|
|
return {
|
|
isInReview: input.isInReview,
|
|
hasParticipant: currentParticipant !== null,
|
|
participantIsAgent: currentParticipant?.type === "agent",
|
|
participantAgentId:
|
|
currentParticipant?.type === "agent" ? (currentParticipant.agentId ?? null) : null,
|
|
currentStageType: input.executionState?.currentStageType ?? null,
|
|
currentParticipant: currentParticipant as Record<string, unknown> | null,
|
|
};
|
|
}
|
|
|
|
function queuedEffect(run: HeartbeatRun): PostCommitEffect {
|
|
return {
|
|
kind: "run_queued",
|
|
companyId: run.companyId,
|
|
runId: run.id,
|
|
agentId: run.agentId,
|
|
invocationSource: run.invocationSource,
|
|
triggerDetail: run.triggerDetail,
|
|
wakeupRequestId: run.wakeupRequestId,
|
|
};
|
|
}
|
|
|
|
function statusEffect(run: HeartbeatRun, previousStatus: string | null): PostCommitEffect {
|
|
return {
|
|
kind: "run_status_published",
|
|
companyId: run.companyId,
|
|
runId: run.id,
|
|
agentId: run.agentId,
|
|
status: run.status,
|
|
invocationSource: run.invocationSource,
|
|
triggerDetail: run.triggerDetail,
|
|
error: run.error,
|
|
errorCode: run.errorCode,
|
|
contextSource: readNonEmptyString(parseObject(run.contextSnapshot).source),
|
|
startedAt: run.startedAt,
|
|
finishedAt: run.finishedAt,
|
|
result: parseObject(run.resultJson),
|
|
issueId: readNonEmptyString(parseObject(run.contextSnapshot).issueId),
|
|
previousStatus,
|
|
};
|
|
}
|
|
|
|
export function createPostgresRunDispatchAdapter(
|
|
db: Db,
|
|
): ScheduledRetryReader & RunDispatchWriter {
|
|
const budgets = budgetService(db);
|
|
const treeControlSvc = issueTreeControlService(db);
|
|
const issuesSvc = issueService(db);
|
|
|
|
async function withRunOwnershipLocks<T>(
|
|
input: { runId: string; companyId: string; recoveryLeaseOwner?: string; enforceContinuationOwnership?: boolean },
|
|
onMissing: () => T,
|
|
operation: (tx: Db, run: HeartbeatRun, publications: ActivityPublication[]) => Promise<T>,
|
|
mode: "accounting" | "ownership",
|
|
): Promise<T> {
|
|
// Queue editing and claiming already use issue -> wake -> run. Read the
|
|
// immutable issue reference without a lock first, then acquire issue ->
|
|
// run here as well. Locking the run first can deadlock with a claimant
|
|
// that owns the issue lock and is waiting for the run.
|
|
for (let attempt = 0; attempt < 3; attempt += 1) {
|
|
const hint = await db
|
|
.select({ contextSnapshot: heartbeatRuns.contextSnapshot })
|
|
.from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!hint) return onMissing();
|
|
// Already-admitted ordinary work owes the budget/run gate, not another
|
|
// continuation or chat-control issue lock. Its authority is captured on
|
|
// the run and reservation; protected recovery retains its admission.
|
|
const hintedIssueId = input.enforceContinuationOwnership === false ? null
|
|
: readNonEmptyString(parseObject(hint.contextSnapshot).issueId);
|
|
|
|
const locked = async (typedTx: Db, publications: ActivityPublication[]) => {
|
|
// Recovery always takes coordinator before issue/run. Only the
|
|
// accounting phase acquires the company lock and evaluates budgets.
|
|
await lockRunBudgetRecoveryOwnerInTransaction(typedTx, input.companyId, input.runId, input.recoveryLeaseOwner);
|
|
if (hintedIssueId) {
|
|
await typedTx
|
|
.select({ id: issues.id })
|
|
.from(issues)
|
|
.where(and(eq(issues.id, hintedIssueId), eq(issues.companyId, input.companyId)))
|
|
.for("update");
|
|
}
|
|
|
|
const run = await typedTx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)),
|
|
)
|
|
// Keep the run status stable through the semantic decision and any
|
|
// resulting mutation and synchronous dispatch handoff. Never await
|
|
// adapter-owned work while this transaction holds the row locks.
|
|
.for("update")
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!run) return { kind: "missing" as const };
|
|
|
|
const lockedIssueId = input.enforceContinuationOwnership === false ? null
|
|
: readNonEmptyString(parseObject(run.contextSnapshot).issueId);
|
|
if (lockedIssueId !== hintedIssueId) return { kind: "retry" as const };
|
|
return { kind: "value" as const, value: await operation(typedTx, run, publications) };
|
|
};
|
|
const result: { kind: "missing" | "retry" } | { kind: "value"; value: T } = mode === "accounting"
|
|
? await withAccountingTransaction(db, input.companyId, locked)
|
|
: await db.transaction(tx => locked(tx as unknown as Db, []));
|
|
|
|
if (result.kind === "missing") return onMissing();
|
|
if (result.kind === "value") return result.value;
|
|
}
|
|
|
|
throw new Error(
|
|
`run-dispatch: run ${input.runId} changed issue context repeatedly while acquiring locks`,
|
|
);
|
|
}
|
|
|
|
function withIssueThenRunLocks<T>(
|
|
input: { runId: string; companyId: string; recoveryLeaseOwner?: string; enforceContinuationOwnership?: boolean },
|
|
onMissing: () => T,
|
|
operation: (tx: Db, run: HeartbeatRun, publications: ActivityPublication[]) => Promise<T>,
|
|
) {
|
|
return withRunOwnershipLocks(input, onMissing, operation, "accounting");
|
|
}
|
|
|
|
async function loadGateFacts(
|
|
input: LoadGateFactsInput,
|
|
now: Date,
|
|
tx?: unknown,
|
|
publications?: ActivityPublication[],
|
|
): Promise<LoadGateFactsResult> {
|
|
// Semantic adapter operations pass their transaction here so the fact
|
|
// read and the state transition share one unit of work. This helper is
|
|
// deliberately not exposed through the module's public API.
|
|
const dbOrTx = (tx as Db | undefined) ?? db;
|
|
const budgetsForRead = tx ? budgetServiceInTransaction(dbOrTx, publications) : budgets;
|
|
const treeControlForRead = tx ? issueTreeControlService(dbOrTx) : treeControlSvc;
|
|
const issuesSvcForRead = tx ? issueService(dbOrTx) : issuesSvc;
|
|
|
|
const agent = await dbOrTx
|
|
.select()
|
|
.from(agents)
|
|
.where(and(eq(agents.id, input.agentId), eq(agents.companyId, input.companyId)))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!agent) {
|
|
return { agentFound: false, issueId: readNonEmptyString(input.contextSnapshot.issueId) };
|
|
}
|
|
|
|
const retryReason =
|
|
input.retryReasonOverride ??
|
|
readNonEmptyString(input.contextSnapshot.retryReason) ??
|
|
input.scheduledRetryReason ??
|
|
null;
|
|
const issueId = readNonEmptyString(input.contextSnapshot.issueId);
|
|
const retryReasonKind = classifyRetryReasonKind(retryReason);
|
|
|
|
// Facts fill in through the same rule order `decideScheduledRetryGate`
|
|
// evaluates. A field still awaiting its own read keeps a value that
|
|
// rule never blocks on, EXCEPT `issueFound`, which starts unresolved
|
|
// and gets an explicit early check once the issue read lands — a
|
|
// false default there would look exactly like "the issue is gone" the
|
|
// moment `issueId` is set, before the read that would prove it. Once a
|
|
// group of fields is real, this calls the gate on the facts gathered
|
|
// so far, so a rule an earlier read already decided stops the run
|
|
// before it pays for a later, unrelated read — a sweep of many due
|
|
// retries must not query pause holds or dependencies for a retry a
|
|
// budget block or a stale assignment already suppresses.
|
|
const facts: ScheduledRetryFacts = {
|
|
runId: input.runId,
|
|
runAgentId: input.agentId,
|
|
issueId,
|
|
retryReasonKind,
|
|
enforceIssueExecutionLock: retryReasonKind === "max_turn_continuation" || retryReasonKind === "ai_connection_wait",
|
|
isNonAssigneeWorkspaceBusyRetry: isNonAssigneeWorkspaceBusyRetry(retryReason, input.contextSnapshot),
|
|
budgetBlock: null,
|
|
agentInvokable: true,
|
|
agentInvokabilityDetails: {},
|
|
agentInvokabilityInvalidOrgChain: false,
|
|
heartbeatWakeOnDemandEnabled: true,
|
|
issueFound: false,
|
|
issueStatus: null,
|
|
issueAssigneeAgentId: null,
|
|
issueExecutionRunId: null,
|
|
reviewParticipant: NO_REVIEW_PARTICIPANT,
|
|
activePauseHold: null,
|
|
dependenciesBlocked: null,
|
|
dispositionRepair: null,
|
|
};
|
|
const isBlocked = () => !decideScheduledRetryGate(facts, now).allowed;
|
|
|
|
const budgetBlock = await budgetsForRead.getInvocationBlock(input.companyId, input.agentId, {
|
|
issueId,
|
|
projectId: readNonEmptyString(input.contextSnapshot.projectId),
|
|
});
|
|
facts.budgetBlock = budgetBlock
|
|
? { reason: budgetBlock.reason, scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId }
|
|
: null;
|
|
if (facts.budgetBlock) return { agentFound: true, facts };
|
|
|
|
const agentInvokability = await evaluateAgentInvokabilityFromDb(dbOrTx, agent);
|
|
facts.agentInvokable = agentInvokability.invokable;
|
|
facts.agentInvokabilityDetails = agentInvokability.invokable ? {} : agentInvokability.details;
|
|
facts.agentInvokabilityInvalidOrgChain = agentInvokability.invokable
|
|
? false
|
|
: agentInvokability.invalidOrgChain;
|
|
if (!facts.agentInvokable) return { agentFound: true, facts };
|
|
|
|
facts.heartbeatWakeOnDemandEnabled = isHeartbeatWakeOnDemandEnabled(agent);
|
|
if (!facts.heartbeatWakeOnDemandEnabled) return { agentFound: true, facts };
|
|
|
|
if (!issueId) return { agentFound: true, facts };
|
|
|
|
const issueQuery = dbOrTx
|
|
.select({
|
|
id: issues.id,
|
|
companyId: issues.companyId,
|
|
projectId: issues.projectId,
|
|
status: issues.status,
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
assigneeUserId: issues.assigneeUserId,
|
|
executionRunId: issues.executionRunId,
|
|
checkoutRunId: issues.checkoutRunId,
|
|
executionPolicy: issues.executionPolicy,
|
|
executionState: issues.executionState,
|
|
monitorNextCheckAt: issues.monitorNextCheckAt,
|
|
})
|
|
.from(issues)
|
|
.where(and(eq(issues.id, issueId), eq(issues.companyId, input.companyId)));
|
|
// Locking the row here, only when a caller opened a transaction for the
|
|
// promote-or-cancel write, is what makes that write's decision hold: a
|
|
// concurrent reassignment, pause, or status change blocks on this lock
|
|
// instead of landing between this read and that write.
|
|
const issue = await (tx ? issueQuery.for("update") : issueQuery).then(
|
|
(rows) => rows[0] ?? null,
|
|
);
|
|
|
|
facts.issueFound = issue !== null;
|
|
if (!facts.issueFound) return { agentFound: true, facts };
|
|
|
|
facts.issueStatus = issue.status;
|
|
facts.issueAssigneeAgentId = issue.assigneeAgentId;
|
|
facts.issueExecutionRunId = issue.executionRunId;
|
|
facts.issueCheckoutRunId = issue.checkoutRunId;
|
|
if (input.conversationContinuation) {
|
|
facts.pendingResponse = await connectionContinuationPendingResponse(dbOrTx as unknown as Db,
|
|
issue, input.responsibleUserId, input.runId);
|
|
}
|
|
facts.reviewParticipant = await readNativeReviewParticipantFacts(dbOrTx, {
|
|
companyId: input.companyId, issueId, agentId: input.agentId,
|
|
contextSnapshot: input.contextSnapshot,
|
|
}) ?? buildReviewParticipantFacts({
|
|
isInReview: issue.status === "in_review",
|
|
executionState: parseIssueExecutionState(issue.executionState),
|
|
});
|
|
|
|
// The gate checks a disposition repair's own supersession before it
|
|
// checks ownership or status, so this read must land before the next
|
|
// isBlocked() call — checking isBlocked() first would report the wrong
|
|
// suppression reason for a superseded repair.
|
|
if (retryReasonKind === "disposition_repair") {
|
|
const expectedFingerprint = readNonEmptyString(input.contextSnapshot.dispositionRepairFingerprint);
|
|
const sourceState = await collectDispositionRepairSourceState(dbOrTx, {
|
|
issue,
|
|
excludeRunId: input.runId,
|
|
excludeWakeupRequestId: input.wakeupRequestId,
|
|
});
|
|
let currentFingerprint = sourceState.fingerprint;
|
|
let validSource = true;
|
|
if (readNonEmptyString(parseObject(input.contextSnapshot.legacyDispositionEpisode).id)) {
|
|
// Legacy repair reserves a slot in a persisted episode. Its fingerprint
|
|
// identifies that episode, not the older parked-summary state snapshot.
|
|
// Still recheck every active/wait/ownership gate before promotion.
|
|
const episode = legacyDispositionEpisode({ id: input.runId, contextSnapshot: input.contextSnapshot });
|
|
const sourceId = readNonEmptyString(input.contextSnapshot.dispositionRepairSourceRunId)
|
|
?? readNonEmptyString(input.contextSnapshot.retryOfRunId);
|
|
const source = sourceId ? await dbOrTx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, sourceId), eq(heartbeatRuns.companyId, input.companyId),
|
|
)).limit(1).then(rows => rows[0]) : null;
|
|
validSource = Boolean(source && source.status === "succeeded" && source.agentId === input.agentId
|
|
&& (source.contextSnapshot?.issueId ?? source.contextSnapshot?.taskId) === issueId
|
|
&& legacyDispositionEpisode(source).id === episode.id
|
|
&& episode.attempt >= 1 && episode.attempt <= episode.maxAttempts);
|
|
currentFingerprint = legacyDispositionFingerprint(input.companyId, issueId, input.agentId, episode.id);
|
|
}
|
|
facts.dispositionRepair = {
|
|
expectedFingerprintPresent: expectedFingerprint !== null,
|
|
fingerprintMatches: validSource && currentFingerprint === expectedFingerprint,
|
|
hasActiveExecutionPath: sourceState.hasActiveExecutionPath,
|
|
hasDurableWaitingPath: sourceState.hasDurableWaitingPath,
|
|
expectedFingerprint,
|
|
currentFingerprint,
|
|
durablePathReason: sourceState.durablePathReason,
|
|
};
|
|
}
|
|
if (isBlocked()) return { agentFound: true, facts };
|
|
|
|
const activePauseHold = await treeControlForRead.getActivePauseHoldGate(input.companyId, issueId);
|
|
facts.activePauseHold = activePauseHold
|
|
? { holdId: activePauseHold.holdId, rootIssueId: activePauseHold.rootIssueId }
|
|
: null;
|
|
if (isBlocked()) return { agentFound: true, facts };
|
|
|
|
const dependencyReadiness = await issuesSvcForRead.listDependencyReadiness(input.companyId, [issueId]);
|
|
const readiness = dependencyReadiness.get(issueId);
|
|
facts.dependenciesBlocked =
|
|
readiness && !readiness.isDependencyReady
|
|
? {
|
|
unresolvedBlockerIssueIds: readiness.unresolvedBlockerIssueIds,
|
|
unresolvedBlockerCount: readiness.unresolvedBlockerCount,
|
|
}
|
|
: null;
|
|
|
|
return { agentFound: true, facts };
|
|
}
|
|
|
|
async function evaluateScheduledRetryGate(input: EvaluateScheduledRetryGateInput) {
|
|
const run = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!run) {
|
|
throw new RunDispatchApplicationError(
|
|
"run_not_found",
|
|
`run-dispatch: run ${input.runId} was not found while evaluating retry eligibility`,
|
|
);
|
|
}
|
|
const factsResult = await loadGateFacts(
|
|
{
|
|
runId: run.id,
|
|
companyId: run.companyId,
|
|
agentId: run.agentId,
|
|
conversationContinuation: run.runtimeMode === "legacy" && hasConversationContinuationPolicy(run.resultJson),
|
|
responsibleUserId: run.responsibleUserId,
|
|
contextSnapshot: parseObject(run.contextSnapshot),
|
|
scheduledRetryReason: run.scheduledRetryReason,
|
|
retryReasonOverride: input.retryReasonOverride,
|
|
wakeupRequestId: run.wakeupRequestId,
|
|
},
|
|
input.now,
|
|
);
|
|
if (!factsResult.agentFound) {
|
|
return {
|
|
allowed: false as const,
|
|
reason: "Scheduled retry suppressed because the agent no longer exists",
|
|
errorCode: "agent_not_invokable" as const,
|
|
issueId: factsResult.issueId,
|
|
details: { agentId: run.agentId },
|
|
};
|
|
}
|
|
return decideScheduledRetryGate(factsResult.facts, input.now);
|
|
}
|
|
|
|
async function listDueRetries(input: ListDueRetriesInput): Promise<DueRetryRun[]> {
|
|
const rows = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.status, "scheduled_retry"),
|
|
lte(heartbeatRuns.scheduledRetryAt, input.now),
|
|
input.cutoff ? gte(heartbeatRuns.createdAt, input.cutoff) : undefined,
|
|
),
|
|
)
|
|
.orderBy(
|
|
asc(heartbeatRuns.scheduledRetryAt),
|
|
asc(heartbeatRuns.createdAt),
|
|
asc(heartbeatRuns.id),
|
|
)
|
|
.limit(input.limit);
|
|
|
|
return rows.map((row) => ({
|
|
runId: row.id,
|
|
companyId: row.companyId,
|
|
}));
|
|
}
|
|
|
|
async function loadStalenessFacts(
|
|
input: LoadStalenessFactsInput,
|
|
_now: Date,
|
|
tx?: unknown,
|
|
): Promise<QueuedRunFacts> {
|
|
const dbOrTx = (tx as Db | undefined) ?? db;
|
|
const issueId = input.issueId;
|
|
const context = input.contextSnapshot;
|
|
|
|
const issueQuery = dbOrTx
|
|
.select({
|
|
id: issues.id,
|
|
companyId: issues.companyId,
|
|
projectId: issues.projectId,
|
|
assigneeUserId: issues.assigneeUserId,
|
|
status: issues.status,
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
executionRunId: issues.executionRunId,
|
|
checkoutRunId: issues.checkoutRunId,
|
|
executionState: issues.executionState,
|
|
})
|
|
.from(issues)
|
|
.where(and(eq(issues.id, issueId), eq(issues.companyId, input.companyId)));
|
|
const issue = await (tx ? issueQuery.for("update") : issueQuery).then(
|
|
(rows) => rows[0] ?? null,
|
|
);
|
|
|
|
const wakeCommentId = deriveCommentId(context);
|
|
const isInteractionWake = allowsIssueInteractionWake(
|
|
context,
|
|
ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS,
|
|
);
|
|
const resumeIntent = context.resumeIntent === true || context.followUpRequested === true;
|
|
const wakeReason = readNonEmptyString(context.wakeReason);
|
|
const retryReason =
|
|
readNonEmptyString(context.retryReason) ?? input.scheduledRetryReason ?? null;
|
|
const interactionResolvedAt = readNonEmptyString(context.interactionResolvedAt);
|
|
const hasResolvedInteractionEvidence =
|
|
interactionResolvedAt !== null && !Number.isNaN(Date.parse(interactionResolvedAt));
|
|
const isResolvedInteractionContinuation = isResolvedInteractionContinuationWakeContext(context);
|
|
|
|
const continuationParkApplies = Boolean(
|
|
issue &&
|
|
issue.status === "in_progress" &&
|
|
!wakeCommentId &&
|
|
!hasResolvedInteractionEvidence &&
|
|
(wakeReason === "issue_continuation_needed" || retryReason === "issue_continuation_needed"),
|
|
);
|
|
let continuationSummaryBody: string | null = null;
|
|
let continuationParksExecutor = false;
|
|
if (continuationParkApplies) {
|
|
const queuedWake = parseObject(context.paperclipWake);
|
|
const queuedContinuationSummary =
|
|
readNonEmptyString(parseObject(context.paperclipContinuationSummary).body) ??
|
|
readNonEmptyString(parseObject(queuedWake.continuationSummary).body);
|
|
const currentContinuationSummary = queuedContinuationSummary
|
|
? null
|
|
: await getIssueContinuationSummaryDocument(dbOrTx, issueId);
|
|
continuationSummaryBody = queuedContinuationSummary ?? currentContinuationSummary?.body ?? null;
|
|
continuationParksExecutor = continuationSummaryParksExecutor(continuationSummaryBody);
|
|
}
|
|
|
|
const recoveryActionId = readNonEmptyString(context.recoveryActionId);
|
|
const isAuthorizedSourceScopedRecovery =
|
|
issue && wakeReason === "source_scoped_recovery_action" && recoveryActionId
|
|
? await dbOrTx
|
|
.select({ id: issueRecoveryActions.id })
|
|
.from(issueRecoveryActions)
|
|
.where(
|
|
and(
|
|
eq(issueRecoveryActions.id, recoveryActionId),
|
|
eq(issueRecoveryActions.companyId, input.companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, issue.id),
|
|
eq(issueRecoveryActions.ownerAgentId, input.agentId),
|
|
inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
|
),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => Boolean(rows[0]))
|
|
: false;
|
|
|
|
const retryReasonKind = classifyRetryReasonKind(retryReason);
|
|
// Dependency edges can change after scheduled promotion without changing
|
|
// the displayed status. Read them again under the queued/final issue lock.
|
|
const readiness = issue && retryReasonKind === "native_safe_replacement"
|
|
? (await issueService(dbOrTx).listDependencyReadiness(input.companyId, [issueId])).get(issueId)
|
|
: null;
|
|
return {
|
|
runId: input.runId,
|
|
runAgentId: input.agentId,
|
|
issueId,
|
|
retryReasonKind,
|
|
dependenciesBlocked: readiness && !readiness.isDependencyReady ? {
|
|
unresolvedBlockerIssueIds: readiness.unresolvedBlockerIssueIds,
|
|
unresolvedBlockerCount: readiness.unresolvedBlockerCount,
|
|
} : null,
|
|
pendingResponse: issue && (input.conversationContinuation || context.connectionIntentResolution === "existing_connection")
|
|
? await connectionContinuationPendingResponse(dbOrTx as unknown as Db, issue, input.responsibleUserId, input.runId)
|
|
: null,
|
|
issueFound: issue !== null,
|
|
issueStatus: issue?.status ?? null,
|
|
issueAssigneeAgentId: issue?.assigneeAgentId ?? null,
|
|
issueExecutionRunId: issue?.executionRunId ?? null,
|
|
issueCheckoutRunId: issue?.checkoutRunId ?? null,
|
|
isResolvedInteractionContinuation,
|
|
isConnectionContinuation: (isResolvedInteractionContinuation && context.interactionKind === "connection_intent")
|
|
|| context.source === "connection_tools.refreshed",
|
|
isInteractionWake,
|
|
isAuthorizedSourceScopedRecovery,
|
|
isNonAssigneeWorkspaceBusyRetry: isNonAssigneeWorkspaceBusyRetry(retryReason, context),
|
|
resumeIntent,
|
|
wakeCommentIdPresent: Boolean(wakeCommentId),
|
|
isCompletedOnboardingHandoffWake: await isCompletedOnboardingHandoffWake(dbOrTx, {
|
|
companyId: input.companyId, issueId, agentId: input.agentId,
|
|
reason: wakeReason, contextSnapshot: context,
|
|
}),
|
|
continuationParkApplies,
|
|
continuationParksExecutor,
|
|
continuationSummaryBody,
|
|
wakeReason,
|
|
retryReason,
|
|
reviewParticipant: issue
|
|
? await readNativeReviewParticipantFacts(dbOrTx, {
|
|
companyId: input.companyId, issueId, agentId: input.agentId,
|
|
contextSnapshot: context,
|
|
}) ?? buildReviewParticipantFacts({
|
|
isInReview: issue.status === "in_review",
|
|
executionState: issue.status === "in_review" ? parseIssueExecutionState(issue.executionState) : null,
|
|
})
|
|
: NO_REVIEW_PARTICIPANT,
|
|
};
|
|
}
|
|
|
|
async function promoteDueRetryInTx(
|
|
tx: Db,
|
|
input: PromoteOrCancelDueRetryInput,
|
|
issueId: string | null,
|
|
): Promise<
|
|
| { applied: true; run: HeartbeatRun; postCommitEffects: PostCommitEffect[] }
|
|
| { applied: false }
|
|
> {
|
|
await notifyInstructionCollection(tx, { companyId: input.companyId, runId: input.runId });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: input.companyId, runId: input.runId });
|
|
await notifyScheduledRetryCancellation(tx, { companyId: input.companyId, runId: input.runId });
|
|
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: issueId ?? undefined });
|
|
const [row] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "queued",
|
|
updatedAt: input.now,
|
|
resultJson: sql`case when ${heartbeatRuns.resultJson}->'executionWait'->>'cause' = 'execution_owner_active'
|
|
then ${heartbeatRuns.resultJson} - 'executionWait' else ${heartbeatRuns.resultJson} end`,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, input.runId),
|
|
eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.status, "scheduled_retry"),
|
|
lte(heartbeatRuns.scheduledRetryAt, input.now),
|
|
),
|
|
)
|
|
.returning();
|
|
if (!row) return { applied: false };
|
|
|
|
await appendHeartbeatRunEvent(tx as unknown as Db, {
|
|
companyId: row.companyId,
|
|
runId: row.id,
|
|
agentId: row.agentId,
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "info",
|
|
message: "Scheduled retry became due and was promoted to the queued run pool",
|
|
payload: {
|
|
scheduledRetryAttempt: row.scheduledRetryAttempt,
|
|
scheduledRetryAt: row.scheduledRetryAt ? new Date(row.scheduledRetryAt).toISOString() : null,
|
|
scheduledRetryReason: row.scheduledRetryReason,
|
|
},
|
|
});
|
|
|
|
return { applied: true as const, run: row, postCommitEffects: [queuedEffect(row)] };
|
|
}
|
|
|
|
async function cancelSuppressedRetryInTx(
|
|
tx: Db,
|
|
input: CancelSuppressedRetryInput,
|
|
): Promise<CancelSuppressedRetryResult> {
|
|
await notifyInstructionCollection(tx, { companyId: input.companyId, runId: input.runId });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: input.companyId, runId: input.runId });
|
|
await notifyScheduledRetryCancellation(tx, { companyId: input.companyId, runId: input.runId });
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined });
|
|
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined });
|
|
const [row] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: input.now,
|
|
error: input.reason,
|
|
errorCode: input.errorCode,
|
|
updatedAt: input.now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, input.runId),
|
|
eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.status, "scheduled_retry"),
|
|
lte(heartbeatRuns.scheduledRetryAt, input.now),
|
|
),
|
|
)
|
|
.returning();
|
|
if (!row) return { applied: false };
|
|
|
|
if (row.wakeupRequestId) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({ status: "cancelled", finishedAt: input.now, error: input.reason, updatedAt: input.now })
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, row.wakeupRequestId),
|
|
eq(agentWakeupRequests.companyId, row.companyId),
|
|
),
|
|
);
|
|
}
|
|
|
|
if (input.issueId) {
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: input.now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, row.companyId),
|
|
eq(issues.id, input.issueId),
|
|
eq(issues.executionRunId, row.id),
|
|
),
|
|
);
|
|
}
|
|
|
|
await appendHeartbeatRunEvent(tx as unknown as Db, {
|
|
companyId: row.companyId,
|
|
runId: row.id,
|
|
agentId: row.agentId,
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message: input.reason,
|
|
payload: {
|
|
...input.details,
|
|
scheduledRetryAttempt: row.scheduledRetryAttempt,
|
|
scheduledRetryAt: row.scheduledRetryAt ? new Date(row.scheduledRetryAt).toISOString() : null,
|
|
scheduledRetryReason: row.scheduledRetryReason,
|
|
},
|
|
});
|
|
|
|
return { applied: true, run: row };
|
|
}
|
|
|
|
async function promoteOrCancelDueRetry(
|
|
input: PromoteOrCancelDueRetryInput,
|
|
): Promise<PromoteScheduledRetryOutcome> {
|
|
const now = input.now;
|
|
|
|
const promoteLockedRun = async (tx: Db, run: HeartbeatRun, publications: ActivityPublication[]) => {
|
|
if (
|
|
run.status !== "scheduled_retry" ||
|
|
// The locked row is also the cancellation claim. If Stop won first,
|
|
// leave settlement to that owner rather than promote its parked run.
|
|
run.resultJson?.startupCancellation ||
|
|
!run.scheduledRetryAt ||
|
|
new Date(run.scheduledRetryAt).getTime() > now.getTime()
|
|
) {
|
|
return { outcome: { outcome: "not_promoted" as const }, telemetryRun: null };
|
|
}
|
|
const factsResult = await loadGateFacts(
|
|
{
|
|
runId: run.id,
|
|
companyId: run.companyId,
|
|
agentId: run.agentId,
|
|
conversationContinuation: run.runtimeMode === "legacy" && hasConversationContinuationPolicy(run.resultJson),
|
|
responsibleUserId: run.responsibleUserId,
|
|
contextSnapshot: parseObject(run.contextSnapshot),
|
|
scheduledRetryReason: run.scheduledRetryReason,
|
|
retryReasonOverride: run.scheduledRetryReason,
|
|
wakeupRequestId: run.wakeupRequestId,
|
|
},
|
|
now,
|
|
tx,
|
|
publications,
|
|
);
|
|
|
|
if (!factsResult.agentFound) {
|
|
const cancelled = await cancelSuppressedRetryInTx(tx as unknown as Db, {
|
|
runId: input.runId,
|
|
companyId: input.companyId,
|
|
now,
|
|
reason: "Scheduled retry suppressed because the agent no longer exists",
|
|
errorCode: "agent_not_invokable",
|
|
issueId: factsResult.issueId,
|
|
details: { agentId: run.agentId },
|
|
});
|
|
return cancelled.applied
|
|
? {
|
|
outcome: {
|
|
outcome: "gate_suppressed" as const,
|
|
reason: "Scheduled retry suppressed because the agent no longer exists",
|
|
errorCode: "agent_not_invokable" as const,
|
|
},
|
|
telemetryRun: cancelled.run,
|
|
}
|
|
: { outcome: { outcome: "not_promoted" as const }, telemetryRun: null };
|
|
}
|
|
|
|
const gate = decideScheduledRetryGate(factsResult.facts, now);
|
|
|
|
// Preserve legacy transient retry behavior for runs that only carry a
|
|
// loose task context rather than a persisted issue row: a missing
|
|
// issue suppresses a max-turn continuation, but every other retry
|
|
// reason proceeds to promotion anyway.
|
|
const isLegacyMissingIssueException =
|
|
!hasConversationContinuationPolicy(run.resultJson) &&
|
|
!gate.allowed &&
|
|
gate.errorCode === "issue_not_found" &&
|
|
factsResult.facts.retryReasonKind !== "max_turn_continuation" &&
|
|
factsResult.facts.retryReasonKind !== "native_safe_replacement";
|
|
|
|
if (!gate.allowed && !isLegacyMissingIssueException) {
|
|
const cancelled = await cancelSuppressedRetryInTx(tx as unknown as Db, {
|
|
runId: input.runId,
|
|
companyId: input.companyId,
|
|
now,
|
|
reason: gate.reason,
|
|
errorCode: gate.errorCode,
|
|
issueId: gate.issueId,
|
|
details: gate.details,
|
|
});
|
|
return cancelled.applied
|
|
? {
|
|
outcome: {
|
|
outcome: "gate_suppressed" as const,
|
|
reason: gate.reason,
|
|
errorCode: gate.errorCode,
|
|
},
|
|
telemetryRun: cancelled.run,
|
|
}
|
|
: { outcome: { outcome: "not_promoted" as const }, telemetryRun: null };
|
|
}
|
|
|
|
const issueId = factsResult.facts.issueId;
|
|
if (issueId && (await deferRetryForCleanupInTx(tx, run, issueId, now))) {
|
|
return { outcome: { outcome: "not_promoted" as const }, telemetryRun: null };
|
|
}
|
|
|
|
const promoted = await promoteDueRetryInTx(tx as unknown as Db, {
|
|
runId: input.runId,
|
|
companyId: input.companyId,
|
|
now,
|
|
}, issueId);
|
|
return promoted.applied
|
|
? {
|
|
outcome: {
|
|
outcome: "promoted" as const,
|
|
postCommitEffects: promoted.postCommitEffects,
|
|
},
|
|
telemetryRun: null,
|
|
}
|
|
: { outcome: { outcome: "not_promoted" as const }, telemetryRun: null };
|
|
};
|
|
const transactionResult = await withIssueThenRunLocks(
|
|
input,
|
|
() => ({ outcome: { outcome: "not_promoted" as const }, telemetryRun: null }),
|
|
promoteLockedRun,
|
|
);
|
|
|
|
// Telemetry is best-effort background work; fire it only after the
|
|
// transaction above has committed, so a suppressed retry is never
|
|
// published before its cancellation is durable, and never published at
|
|
// all if the transaction rolled back.
|
|
if (transactionResult.telemetryRun) {
|
|
void emitAgentTaskRun(db, transactionResult.telemetryRun);
|
|
}
|
|
|
|
return transactionResult.outcome;
|
|
}
|
|
|
|
async function cancelStaleRunInTx(
|
|
tx: Db,
|
|
run: HeartbeatRun,
|
|
issueId: string,
|
|
decision: Extract<ReturnType<typeof decideQueuedRunStaleness>, { stale: true }>,
|
|
expectedStatus: "queued" | "running",
|
|
now: Date,
|
|
providerDispatchNotStarted = false,
|
|
): Promise<CancelStaleQueuedRunOutcome> {
|
|
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, run.companyId);
|
|
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId });
|
|
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
|
|
const [row] = await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: now,
|
|
error: decision.reason,
|
|
errorCode: decision.errorCode,
|
|
resultJson: {
|
|
...parseObject(run.resultJson),
|
|
stopReason: decision.errorCode,
|
|
...(providerDispatchNotStarted && decision.errorCode === "issue_waiting_for_response"
|
|
? { preDispatchResponseWait: { version: 1, providerWorkStarted: false } }
|
|
: {}),
|
|
...(decision.errorCode === "execution_reconciliation_required"
|
|
? { executionWait: decision.details }
|
|
: {}),
|
|
effectiveTimeoutSec: 0,
|
|
timeoutConfigured: false,
|
|
timeoutSource: "stale_queued_run_gate",
|
|
timeoutFired: false,
|
|
},
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, run.id),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.status, expectedStatus),
|
|
),
|
|
)
|
|
.returning();
|
|
// A concurrent claimant or canceller already moved the run off
|
|
// `expectedStatus`: the caller's staleness decision lost the race, so
|
|
// this write must not overwrite whatever status won it.
|
|
if (!row) return { outcome: "lost_race" };
|
|
|
|
if (row.wakeupRequestId) {
|
|
await tx
|
|
.update(agentWakeupRequests)
|
|
.set({ status: "skipped", finishedAt: now, error: decision.reason, updatedAt: now })
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.id, row.wakeupRequestId),
|
|
eq(agentWakeupRequests.companyId, row.companyId),
|
|
),
|
|
);
|
|
}
|
|
|
|
const context = parseObject(row.contextSnapshot);
|
|
if (decision.errorCode === "issue_waiting_for_response" && context.connectionIntentResolution === "existing_connection"
|
|
&& context.source === "connection_intent.resolved" && context.interactionKind === "connection_intent"
|
|
&& typeof context.interactionId === "string") {
|
|
// The partial wake-key uniqueness fence permits another delivery after
|
|
// this skipped receipt. Rearm atomically so restored access or a later
|
|
// answer can resume the untouched request after a restart.
|
|
await tx.update(connectionIntentDeliveries).set({ deliveredAt: null, nextAttemptAt: now }).where(and(
|
|
eq(connectionIntentDeliveries.companyId, row.companyId),
|
|
eq(connectionIntentDeliveries.interactionId, context.interactionId),
|
|
));
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.connection);
|
|
}
|
|
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, row.companyId),
|
|
eq(issues.id, issueId),
|
|
eq(issues.executionRunId, row.id),
|
|
),
|
|
);
|
|
|
|
await appendHeartbeatRunEvent(tx as unknown as Db, {
|
|
companyId: row.companyId,
|
|
runId: row.id,
|
|
agentId: row.agentId,
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "warn",
|
|
message: decision.reason,
|
|
payload: decision.details,
|
|
});
|
|
|
|
return {
|
|
outcome: "cancelled",
|
|
reason: decision.reason,
|
|
errorCode: decision.errorCode,
|
|
postCommitEffects: [statusEffect(row, expectedStatus)],
|
|
};
|
|
}
|
|
|
|
async function decideCurrentRunStaleness(tx: Db, run: HeartbeatRun, now: Date, publications?: ActivityPublication[]) {
|
|
const contextSnapshot = parseObject(run.contextSnapshot);
|
|
const issueId = readNonEmptyString(contextSnapshot.issueId);
|
|
if (!issueId) return { issueId: null, facts: null, executionBlocker: null, decision: { stale: false as const } };
|
|
if (typeof contextSnapshot.workspaceReconstructionActionId === "string") {
|
|
try {
|
|
await validateReconstructedWorkspaceDispatch(tx, run);
|
|
const current = await loadGateFacts({ runId: run.id, companyId: run.companyId, agentId: run.agentId,
|
|
responsibleUserId: run.responsibleUserId, conversationContinuation: true, contextSnapshot,
|
|
scheduledRetryReason: run.scheduledRetryReason, retryReasonOverride: "native_safe_replacement", wakeupRequestId: run.wakeupRequestId,
|
|
}, now, tx, publications);
|
|
if (!current.agentFound || !decideScheduledRetryGate(current.facts, now).allowed) throw new Error("reconstruction_execution_not_eligible");
|
|
}
|
|
catch {
|
|
return { issueId, facts: null, executionBlocker: null, decision: { stale: true as const,
|
|
errorCode: "execution_reconciliation_required" as const,
|
|
reason: "Workspace reconstruction changed before execution. Inspect the saved sources and current task before continuing.",
|
|
details: { issueId, recoveryActionId: contextSnapshot.workspaceReconstructionActionId },
|
|
} };
|
|
}
|
|
}
|
|
const recovery = await getExecutionBlocker(tx, run.companyId, issueId, { conversationResetCommentId: deriveCommentId(contextSnapshot) });
|
|
if (recovery) return { issueId, facts: null, executionBlocker: recovery, decision: { stale: true as const,
|
|
errorCode: "execution_reconciliation_required" as const, reason: recovery.nextAction,
|
|
details: { issueId, recoveryActionId: recovery.recoveryActionId },
|
|
} };
|
|
const facts = await loadStalenessFacts(
|
|
{
|
|
runId: run.id,
|
|
companyId: run.companyId,
|
|
agentId: run.agentId,
|
|
responsibleUserId: run.responsibleUserId,
|
|
conversationContinuation: run.runtimeMode === "legacy" && hasConversationContinuationPolicy(run.resultJson),
|
|
issueId,
|
|
contextSnapshot,
|
|
scheduledRetryReason: run.scheduledRetryReason,
|
|
},
|
|
now,
|
|
tx,
|
|
);
|
|
return { issueId, facts, executionBlocker: null, decision: decideQueuedRunStaleness(facts, now) };
|
|
}
|
|
|
|
/**
|
|
* Keep the existing attempt durable while the previous execution releases
|
|
* ownership. Only an unstarted legacy conversation retry can wait here;
|
|
* reconciliation of uncertain actions and native execution retain their gates.
|
|
* Both callers hold the issue and run locks, in that order.
|
|
*/
|
|
async function deferRetryForCleanupInTx(
|
|
tx: Db,
|
|
run: HeartbeatRun,
|
|
issueId: string,
|
|
now: Date,
|
|
observedBlocker?: Awaited<ReturnType<typeof getExecutionBlocker>>,
|
|
) {
|
|
if (
|
|
run.runtimeMode !== "legacy" || !run.retryOfRunId || run.startedAt ||
|
|
run.processPid || run.processGroupId ||
|
|
!hasConversationContinuationPolicy(run.resultJson) ||
|
|
!["scheduled_retry", "queued"].includes(run.status)
|
|
) return null;
|
|
// Use the dispatch gate's observation when supplied. A second read could
|
|
// see cleanup finish and accidentally cancel the retry for the old hold.
|
|
const blocker = observedBlocker === undefined
|
|
? await getExecutionBlocker(tx, run.companyId, issueId)
|
|
: observedBlocker;
|
|
if (blocker?.cause !== "execution_owner_active") return null;
|
|
|
|
const previousWait = parseObject(run.resultJson?.executionWait);
|
|
const scheduledRetryAt = new Date(now.getTime() + 30_000);
|
|
const executionWait = { issueId, runId: blocker.runId, cause: blocker.cause };
|
|
await notifyScheduledRetryCancellation(tx, { companyId: run.companyId, runId: run.id });
|
|
if (run.status === "queued") await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId });
|
|
const [waiting] = await tx.update(heartbeatRuns)
|
|
.set({
|
|
status: "scheduled_retry",
|
|
scheduledRetryAt,
|
|
updatedAt: now,
|
|
contextSnapshot: { ...parseObject(run.contextSnapshot), scheduledRetryAt: scheduledRetryAt.toISOString() },
|
|
resultJson: { ...parseObject(run.resultJson), executionWait },
|
|
})
|
|
.where(and(
|
|
eq(heartbeatRuns.id, run.id),
|
|
eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.status, run.status),
|
|
))
|
|
.returning();
|
|
if (!waiting) return null;
|
|
// Repeated sweeps retain one wait diagnostic, not a new attempt or event.
|
|
if (previousWait.cause !== blocker.cause || previousWait.runId !== blocker.runId) {
|
|
await appendHeartbeatRunEvent(tx, {
|
|
companyId: run.companyId,
|
|
runId: run.id,
|
|
agentId: run.agentId,
|
|
eventType: "lifecycle",
|
|
stream: "system",
|
|
level: "info",
|
|
message: "Scheduled retry is waiting for execution cleanup; the same attempt will continue after ownership is released",
|
|
payload: { ...executionWait, scheduledRetryAttempt: run.scheduledRetryAttempt },
|
|
});
|
|
}
|
|
return { outcome: "deferred" as const, postCommitEffects: [statusEffect(waiting, run.status)] };
|
|
}
|
|
|
|
async function cancelStaleQueuedRun(
|
|
input: CancelStaleQueuedRunInput,
|
|
): Promise<CancelStaleQueuedRunOutcome> {
|
|
const cancelLockedRun = async (tx: Db, run: HeartbeatRun, publications: ActivityPublication[]) => {
|
|
if (run.status !== input.expectedStatus) return { outcome: "lost_race" as const };
|
|
const { issueId, facts, decision, executionBlocker } = await decideCurrentRunStaleness(tx, run, input.now, publications);
|
|
if (!decision.stale || !issueId) {
|
|
if (input.expectedStatus === "queued" && facts?.isInteractionWake) {
|
|
// Preserve the authority accepted under the issue/run locks. Later
|
|
// preflight reads can observe a reassignment; they must not turn an
|
|
// assignee comment into a non-assignee subscription-wait exception.
|
|
await tx.update(heartbeatRuns).set({
|
|
runnerProfileJson: {
|
|
...parseObject(run.runnerProfileJson),
|
|
aiConnectionNonAssigneeCommentWake: facts.issueAssigneeAgentId !== run.agentId,
|
|
},
|
|
}).where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId)));
|
|
}
|
|
return { outcome: "not_stale" as const };
|
|
}
|
|
if (input.expectedStatus === "queued" && decision.errorCode === "execution_reconciliation_required") {
|
|
const deferred = await deferRetryForCleanupInTx(tx, run, issueId, input.now, executionBlocker);
|
|
if (deferred) return deferred;
|
|
}
|
|
return cancelStaleRunInTx(tx, run, issueId, decision, input.expectedStatus, input.now);
|
|
};
|
|
|
|
return withIssueThenRunLocks(
|
|
input,
|
|
() => {
|
|
throw new RunDispatchApplicationError(
|
|
"run_not_found",
|
|
`run-dispatch: run ${input.runId} disappeared during stale-run validation`,
|
|
);
|
|
},
|
|
cancelLockedRun,
|
|
);
|
|
}
|
|
|
|
async function dispatchResolvedInteractionIfCurrent<T>(
|
|
input: DispatchResolvedInteractionInput<T>,
|
|
): Promise<DispatchResolvedInteractionOutcome<T>> {
|
|
const dispatchLockedRun = async (tx: Db, run: HeartbeatRun, publications: ActivityPublication[]) => {
|
|
if (run.status !== input.expectedStatus || run.resultJson?.cancellation || run.resultJson?.startupCancellation) {
|
|
return { dispatched: false as const, cancellation: { outcome: "lost_race" as const } };
|
|
}
|
|
if (input.enforceContinuationOwnership !== false) {
|
|
const { issueId, decision: initialDecision } = await decideCurrentRunStaleness(
|
|
tx,
|
|
run,
|
|
input.now,
|
|
publications,
|
|
);
|
|
const decision =
|
|
!initialDecision.stale && issueId
|
|
? await tx
|
|
.select({ executionRunId: issues.executionRunId })
|
|
.from(issues)
|
|
.where(and(eq(issues.id, issueId), eq(issues.companyId, run.companyId)))
|
|
.then((rows) => rows[0] ?? null)
|
|
.then((issue) =>
|
|
issue?.executionRunId === run.id
|
|
? initialDecision
|
|
: {
|
|
stale: true as const,
|
|
errorCode: "issue_execution_lock_changed" as const,
|
|
reason:
|
|
"Cancelled because continuation no longer owns the issue execution lock before adapter dispatch",
|
|
details: {
|
|
issueId,
|
|
expectedExecutionRunId: run.id,
|
|
currentExecutionRunId: issue?.executionRunId ?? null,
|
|
},
|
|
},
|
|
)
|
|
: initialDecision;
|
|
|
|
if (decision.stale && issueId) {
|
|
const cancellation = await cancelStaleRunInTx(
|
|
tx,
|
|
run,
|
|
issueId,
|
|
decision,
|
|
input.expectedStatus,
|
|
input.now,
|
|
true, // This locked gate has not handed off to the provider callback.
|
|
);
|
|
return { dispatched: false as const, cancellation };
|
|
}
|
|
}
|
|
|
|
const admissionBlockReason = await recheckRunBudgetReservationInTransaction(
|
|
tx, publications, run, input.reservationId, input.recoveryLeaseOwner,
|
|
);
|
|
// Commit budget-owned hold/enforcement changes before the caller handles
|
|
// denial. Throwing here would roll back the very stop intent we found.
|
|
if (admissionBlockReason) return { dispatched: false as const, admissionBlockReason };
|
|
|
|
const priorDispatch = parseObject(parseObject(run.usageJson).budgetDispatch);
|
|
const recoveryLeaseOwner = input.recoveryLeaseOwner ?? null;
|
|
if (priorDispatch.reservationId === input.reservationId && (priorDispatch.recoveryLeaseOwner ?? null) === recoveryLeaseOwner)
|
|
return { dispatched: false as const, admissionBlockReason: "Provider dispatch already admitted this reservation generation" };
|
|
// The committed claim is the dispatch linearization point. A concurrent
|
|
// charge wins before it, or targets already-admitted work afterwards.
|
|
// Preserve usage/cancellation evidence and fence duplicate handoffs even
|
|
// for zero-valued holds. A newly proven native recovery owner has its own
|
|
// generation; its ordinary stopped-execution checks remain upstream.
|
|
const claim = { reservationId: input.reservationId, recoveryLeaseOwner, admittedAt: new Date().toISOString() };
|
|
await tx.update(heartbeatRuns).set({
|
|
usageJson: sql`coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || ${JSON.stringify({ budgetDispatch: claim })}::jsonb`,
|
|
}).where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId)));
|
|
return { dispatched: true as const, claim };
|
|
};
|
|
|
|
const admission = await withIssueThenRunLocks(
|
|
input,
|
|
() => {
|
|
throw new RunDispatchApplicationError(
|
|
"run_not_found",
|
|
`run-dispatch: run ${input.runId} disappeared before resolved-interaction dispatch`,
|
|
);
|
|
},
|
|
dispatchLockedRun,
|
|
);
|
|
if (!admission.dispatched) return admission;
|
|
// Accounting has committed. Close the separate issue/run ownership race
|
|
// under short row locks, without reacquiring company accounting or awaiting
|
|
// provider work. Mutating budget evaluation never runs in this second phase.
|
|
return withRunOwnershipLocks<DispatchResolvedInteractionOutcome<T>>(input, () => ({ dispatched: false as const,
|
|
cancellation: { outcome: "lost_race" as const } }), async (tx, run) => {
|
|
const claim = parseObject(parseObject(run.usageJson).budgetDispatch);
|
|
if (run.status !== input.expectedStatus || run.costAccountedAt || run.resultJson?.cancellation || run.resultJson?.startupCancellation
|
|
|| claim.reservationId !== admission.claim.reservationId || claim.recoveryLeaseOwner !== admission.claim.recoveryLeaseOwner
|
|
|| claim.admittedAt !== admission.claim.admittedAt || claim.handedOffAt)
|
|
return { dispatched: false as const, cancellation: { outcome: "lost_race" as const } };
|
|
|
|
const [scope] = await tx.select({ companyStatus: companies.status, agentStatus: agents.status,
|
|
lifecycleState: agents.lifecycleState, reservationId: budgetReservations.id,
|
|
projectId: budgetReservations.projectId, currentProjectId: projects.id, projectPausedAt: projects.pausedAt })
|
|
.from(companies).innerJoin(agents, and(eq(agents.companyId, companies.id), eq(agents.id, run.agentId)))
|
|
.leftJoin(budgetReservations, and(eq(budgetReservations.companyId, companies.id), eq(budgetReservations.runId, run.id),
|
|
eq(budgetReservations.id, input.reservationId), eq(budgetReservations.agentId, run.agentId), eq(budgetReservations.state, "held")))
|
|
.leftJoin(projects, and(eq(projects.companyId, companies.id), eq(projects.id, budgetReservations.projectId)))
|
|
.where(eq(companies.id, run.companyId));
|
|
if (!scope || scope.reservationId !== input.reservationId)
|
|
return { dispatched: false as const, admissionBlockReason: "Provider dispatch no longer owns a held budget reservation" };
|
|
if (scope.projectId && !scope.currentProjectId)
|
|
return { dispatched: false as const, admissionBlockReason: "Reserved project no longer exists" };
|
|
if (scope.companyStatus !== "active" || scope.projectPausedAt || scope.lifecycleState !== "ready"
|
|
|| ["paused", "terminated", "pending_approval"].includes(scope.agentStatus))
|
|
return { dispatched: false as const, admissionBlockReason: "Execution scope is paused or no longer ready for work" };
|
|
|
|
if (input.enforceContinuationOwnership !== false) {
|
|
const contextSnapshot = parseObject(run.contextSnapshot);
|
|
const issueId = readNonEmptyString(contextSnapshot.issueId);
|
|
if (issueId) {
|
|
// These facts read current issue authority. The reconstruction/source
|
|
// coordinator validation remains in the accounting phase, avoiding a
|
|
// source coordinator lock after issue/run in this final phase.
|
|
const facts = await loadStalenessFacts({ runId: run.id, companyId: run.companyId, agentId: run.agentId,
|
|
responsibleUserId: run.responsibleUserId,
|
|
conversationContinuation: run.runtimeMode === "legacy" && hasConversationContinuationPolicy(run.resultJson),
|
|
issueId, contextSnapshot, scheduledRetryReason: run.scheduledRetryReason }, input.now, tx);
|
|
const semantic = decideQueuedRunStaleness(facts, input.now);
|
|
const decision = semantic.stale || facts.issueExecutionRunId === run.id ? semantic : {
|
|
stale: true as const, errorCode: "issue_execution_lock_changed" as const,
|
|
reason: "Cancelled because continuation no longer owns the issue execution lock before adapter dispatch",
|
|
details: { issueId, expectedExecutionRunId: run.id, currentExecutionRunId: facts.issueExecutionRunId },
|
|
};
|
|
if (decision.stale) return { dispatched: false as const,
|
|
cancellation: await cancelStaleRunInTx(tx, run, issueId, decision, input.expectedStatus, input.now, true) };
|
|
}
|
|
}
|
|
const budgetBlock = await recheckAdmittedRunBudgetReadOnly(tx, { companyId: run.companyId, agentId: run.agentId,
|
|
runId: run.id, projectId: scope.projectId, reservationId: input.reservationId });
|
|
if (budgetBlock) return { dispatched: false as const, admissionBlockReason: budgetBlock };
|
|
if (input.recoveryLeaseOwner) {
|
|
const [owner] = run.runtimeMode === "native" ? await tx.select({ id: nativeRunFinalizations.runId }).from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id),
|
|
eq(nativeRunFinalizations.leaseOwner, input.recoveryLeaseOwner), sql`${nativeRunFinalizations.leaseExpiresAt} > clock_timestamp()`)) : [];
|
|
if (!owner) return { dispatched: false as const, admissionBlockReason: "Native recovery no longer owns provider dispatch" };
|
|
}
|
|
await tx.update(heartbeatRuns).set({ usageJson: sql`coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb)
|
|
|| ${JSON.stringify({ budgetDispatch: { ...claim, handedOffAt: new Date().toISOString() } })}::jsonb` })
|
|
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId)));
|
|
let resultPromise: Promise<T>;
|
|
try { resultPromise = Promise.resolve(input.dispatch(() => {})); }
|
|
catch (error) { resultPromise = Promise.reject(error); }
|
|
void resultPromise.catch(() => {});
|
|
return { dispatched: true as const, resultPromise };
|
|
}, "ownership");
|
|
}
|
|
|
|
return {
|
|
evaluateScheduledRetryGate,
|
|
listDueRetries,
|
|
cancelStaleQueuedRun,
|
|
dispatchResolvedInteractionIfCurrent,
|
|
promoteOrCancelDueRetry,
|
|
};
|
|
}
|