mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 22:56:58 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents split work into tasks. A task can have a parent task. > - Today agents can nest tasks to any depth: parent, child, grandchild, and deeper. > - Deep agent-made trees are hard to follow, and an agent that works on a child task makes grandchildren by default. > - This pull request limits agent-created trees to two levels: a top-level task and its children. > - A request for a grandchild does not fail. The server puts the new task beside the requester and tells the agent. > - Board users keep full control. They can still nest to any depth. Existing deep trees stay valid. > - The pull request also records which task created each new task, so the audit trail stays complete when the parent changes. ## Linked Issues or Issue Description No public issue exists. This is the feature description. Related: #16853 (budgets that include descendant work) also reasons about task trees; this change does not alter its traversal. **Subsystem affected** Issues (tasks): create, child create and move in the issue service and routes. Agent tools, chat suggestions and creation provenance use these paths. **Problem or motivation** Agents create subtasks as deep as they want. An agent that runs on a child task gets that child as the default parent, so its follow-ups become grandchildren. The trees become deep and hard to read. The parent link is also the only record of which task created a follow-up, so that record is lost when the parent changes. **Proposed solution** Limit agent-created trees to a parent and its children. When an agent asks for a grandchild, put the new task under the requester's parent and return `parentAdjusted`. Do not reject the request. Keep board users unrestricted. Record the originating task in a new server-set `origin_issue_id` column. Keep `origin_run_id` as the originating run. **Alternatives considered** Reject grandchild requests with an error. We did not do this, because agents then fail mid-task. Add a database constraint. We did not do this, because existing deep trees must stay valid and nesting can return later. ## What Changed - Add `MAX_AGENT_TASK_HIERARCHY_DEPTH` (2) and the `IssueParentAdjustment` type in `packages/shared`. - Add `resolveNewTaskParent` and `planIssueParentChange` in `server/src/services/issues.ts`. `create`, `createChild` and `update` use the agent limit by default (`parentDepthPolicy: "flatten"`). - The issue create, `/children` and PATCH routes pass `allow_nested` for board users. For agents, the routes check authorization against the parent that the server actually writes. - The requester keeps its behavior when the server moves a task beside it: `blockParentUntilDone` blocks the requester, and the new task inherits the requester's workspace and privacy. - The server never moves a task above a low-trust boundary root that contains the requester. - These subsystems keep any depth, because they find their work by parent: watchdog reviews, monitor recovery, email-channel placement, and routines that a person configured. - An agent move of a task that has children skips only the parent change and returns `reason: "has_children"`. - Add `issues.origin_issue_id` (nullable, `on delete set null`, indexed) in migration `0434`. Only the server sets it. The `createdFrom` detail field, the `createdFromIssueId` filter and task-creation provenance read it first. - Chat prompts now let a chat agent create one task, a task with children, or separate tasks. The conversation is never a parent. - The runner `create_task` tool accepts an optional `parentTaskId`. A chat agent can create a top-level task and then put subtasks under it. The run must be able to read and change the parent. A conversation is never a parent. The depth limit still applies. Project tools map the field to `parentId` through the issue route. The protocol manifest, semantic contracts and eval fixture are regenerated. - Watchdog runs that have an active watchdog keep their requested parent on all create paths, because their access is limited to the watched subtree. - Fix a bug that the test drive found: the chat (MCP) `create_task` ignored the requested `status`, so a `backlog` task became `todo` and started to run. It now uses the same rule as the native tool. - Update `skills/paperclip`, `doc/SPEC-implementation.md`, and add the plan `doc/plans/2026-11-05-two-level-task-hierarchy.md`. ## Verification - `cd server && npx vitest run src/__tests__/agent-task-hierarchy-depth.test.ts src/__tests__/agent-task-hierarchy-depth-routes.test.ts`: 19 new tests for service and route behavior, all pass. - Related suites (64 files: issues service, thread interactions, privacy, watchdogs, provenance, runner tool authority, decisions, routines, project tools): 1208 tests pass. Two tests changed because of the intended behavior and pass now. Two `email-channels.integration` tests fail locally because the test cannot find a bundled skill file from this worktree path. This change does not touch that code. - `packages/db`: migration safety, snapshot drift and journal tests pass. - Typecheck passes for `server`, `packages/shared`, `packages/db`, `cli` and `ui` (local `ui` errors come only from generated Storybook catalog files that are absent). - Local test drive on an isolated instance with real Claude agents: 29 of 29 use cases pass, including an agent on a child task (follow-ups placed beside it, with the notice) and a chat that creates a top-level task with three subtasks. - I did not run the full `pnpm test:run` or `pnpm build` locally. CI runs them. ## Risks - Behavior change for agents: some follow-ups that were grandchildren are now siblings. The parent of the requester is woken when all its children finish, which now includes these tasks. - The runner `create_task` input contract adds one optional field, so the protocol manifest and the pack contract change. The tool output does not change: it shows the actual `parentId`, and the activity log records `parentAdjusted`. - Migration `0434` adds a nullable column, a foreign key and an index. It does not backfill rows. - `IssueCreationSource.run` is now nullable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1394 lines
51 KiB
TypeScript
1394 lines
51 KiB
TypeScript
import { notifyStatusCardIssueChange } from "../status-card-notifications.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
|
|
import { notifyIssueReviewState } from "../issue-review-state-notifications.js";
|
|
import { insertIssueComment } from "../issue-comment-writes.js";
|
|
import {
|
|
issues,
|
|
issueComments,
|
|
heartbeatRuns,
|
|
nativeRunFinalizations,
|
|
companies,
|
|
agentTaskSessions,
|
|
agentWakeupRequests,
|
|
agentSessionGoalActions,
|
|
agents,
|
|
activityLog,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
type IssueExecutionMonitorPolicy,
|
|
type IssueExecutionMonitorClearReason,
|
|
type IssueExecutionMonitorRecoveryPolicy,
|
|
PROVIDER_QUOTA_MONITOR_SERVICE_NAME,
|
|
EXECUTION_RECONCILIATION_CAUSES,
|
|
isUuidLike,
|
|
} from "@paperclipai/shared";
|
|
import {
|
|
and,
|
|
eq,
|
|
notInArray,
|
|
desc,
|
|
sql,
|
|
isNull,
|
|
inArray,
|
|
or,
|
|
lt,
|
|
lte,
|
|
asc,
|
|
getTableColumns,
|
|
gte,
|
|
} from "drizzle-orm";
|
|
import { RECOVERY_ORIGIN_KINDS } from "../recovery/index.js";
|
|
import { visibleIssueCondition } from "../issue-visibility.js";
|
|
import { REVIEW_PATH_RECOVERY_INSTRUCTION } from "../recovery/review-path-recovery.js";
|
|
import { withRecoveryContext } from "../recovery/status-only-context.js";
|
|
import { logActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
|
|
import {
|
|
normalizeIssueExecutionPolicy,
|
|
buildIssueMonitorClearedPatch,
|
|
parseIssueExecutionState,
|
|
buildIssueMonitorTriggeredPatch,
|
|
} from "../issue-execution-policy.js";
|
|
import { conflict, HttpError, notFound } from "../../errors.js";
|
|
import { EXECUTION_REVIEW_PARTICIPANT_RECOVERY_WAKE_REASON } from "./run-state.js";
|
|
import { parseObject } from "../../adapters/utils.js";
|
|
import {
|
|
legacyExecutionNeedsReconciliationWithEvidence,
|
|
terminalizeLegacyExecution,
|
|
} from "../legacy-execution-recovery.js";
|
|
import { logger } from "../../middleware/logger.js";
|
|
import { failRunnerGoalAction } from "../runner-goals.js";
|
|
import { evaluateAgentInvokability } from "../agent-invokability.js";
|
|
import { supportsUnscopedTimerHeartbeat } from "../heartbeat-policy.js";
|
|
import type { Db } from "@paperclipai/db";
|
|
import type { issueService } from "../issues.js";
|
|
import type { createHeartbeatQueue } from "./queue.js";
|
|
import type { createHeartbeatRetries, HeartbeatRetryDependencies } from "./retries.js";
|
|
import type { createHeartbeatRunPreparation } from "./run-preparation.js";
|
|
import type { NativeSessionGoalControl } from "../../vendor/paperclip-runner/index.js";
|
|
|
|
type HeartbeatQueue = ReturnType<typeof createHeartbeatQueue>;
|
|
type RunPreparation = ReturnType<typeof createHeartbeatRunPreparation>;
|
|
|
|
export interface HeartbeatSchedulingDependencies
|
|
extends Pick<HeartbeatRetryDependencies, "getRun" | "getAgent" | "getWorktreeExecutionCutoff">,
|
|
Pick<HeartbeatQueue, "enqueueWakeup" | "parseHeartbeatPolicy" | "nextTimerHeartbeatAt" | "claimDueTimerHeartbeat">,
|
|
Pick<RunPreparation, "toAgentOrgRow" | "groupAgentOrgRowsByCompany"> {
|
|
issuesSvc: Pick<ReturnType<typeof issueService>, "listReviewAttention" | "create">;
|
|
scheduleBoundedRetryForRun: ReturnType<typeof createHeartbeatRetries>["scheduleBoundedRetryForRun"];
|
|
getSchedulingSuppression: (fresh?: boolean) => Promise<{ suppressed: boolean }>;
|
|
}
|
|
|
|
function readNonEmptyString(value: unknown): string | null {
|
|
return typeof value === "string" && value.trim().length > 0 ? value : null;
|
|
}
|
|
|
|
export function parseNativeSessionGoalControl(
|
|
value: unknown,
|
|
): NativeSessionGoalControl | null {
|
|
const candidate = parseObject(value);
|
|
const requestId = readNonEmptyString(candidate.requestId);
|
|
const action = readNonEmptyString(candidate.action);
|
|
if (
|
|
!requestId ||
|
|
!action ||
|
|
!["create", "edit", "replace", "pause", "resume", "clear"].includes(action)
|
|
)
|
|
return null;
|
|
const objective = readNonEmptyString(candidate.objective);
|
|
const tokenBudget =
|
|
candidate.tokenBudget === null
|
|
? null
|
|
: typeof candidate.tokenBudget === "number" &&
|
|
Number.isSafeInteger(candidate.tokenBudget) &&
|
|
candidate.tokenBudget > 0
|
|
? candidate.tokenBudget
|
|
: undefined;
|
|
return {
|
|
requestId,
|
|
action: action as NativeSessionGoalControl["action"],
|
|
...(objective ? { objective } : {}),
|
|
...(tokenBudget !== undefined ? { tokenBudget } : {}),
|
|
};
|
|
}
|
|
|
|
export function formatIssueIdentifierLink(
|
|
identifier: string | null,
|
|
fallback: string,
|
|
) {
|
|
if (!identifier) return fallback;
|
|
const prefix = identifier.split("-")[0];
|
|
if (!prefix || !/^[A-Z][A-Z0-9]*-\d+$/.test(identifier)) return identifier;
|
|
return `[${identifier}](/${prefix}/issues/${identifier})`;
|
|
}
|
|
|
|
const EXECUTION_REVIEW_PARTICIPANT_RECOVERY_RETRY_REASON =
|
|
"execution_review_participant_recovery";
|
|
|
|
// Construction only binds dependencies. Scheduler calls retain the existing queue admission gates.
|
|
export function createHeartbeatScheduling(
|
|
db: Db,
|
|
dependencies: HeartbeatSchedulingDependencies,
|
|
) {
|
|
const {
|
|
issuesSvc,
|
|
enqueueWakeup,
|
|
getRun,
|
|
getAgent,
|
|
scheduleBoundedRetryForRun,
|
|
getSchedulingSuppression,
|
|
getWorktreeExecutionCutoff,
|
|
toAgentOrgRow,
|
|
groupAgentOrgRowsByCompany,
|
|
parseHeartbeatPolicy,
|
|
nextTimerHeartbeatAt,
|
|
claimDueTimerHeartbeat,
|
|
} = dependencies;
|
|
|
|
const issueMonitorDispatchColumns = {
|
|
id: issues.id,
|
|
companyId: issues.companyId,
|
|
projectId: issues.projectId,
|
|
goalId: issues.goalId,
|
|
identifier: issues.identifier,
|
|
title: issues.title,
|
|
status: issues.status,
|
|
priority: issues.priority,
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
assigneeUserId: issues.assigneeUserId,
|
|
billingCode: issues.billingCode,
|
|
executionPolicy: issues.executionPolicy,
|
|
executionState: issues.executionState,
|
|
monitorNextCheckAt: issues.monitorNextCheckAt,
|
|
monitorWakeRequestedAt: issues.monitorWakeRequestedAt,
|
|
monitorLastTriggeredAt: issues.monitorLastTriggeredAt,
|
|
monitorAttemptCount: issues.monitorAttemptCount,
|
|
monitorNotes: issues.monitorNotes,
|
|
monitorScheduledBy: issues.monitorScheduledBy,
|
|
};
|
|
|
|
interface IssueMonitorDispatchRow {
|
|
id: string;
|
|
companyId: string;
|
|
projectId: string | null;
|
|
goalId: string | null;
|
|
identifier: string | null;
|
|
title: string;
|
|
status: string;
|
|
priority: string;
|
|
assigneeAgentId: string | null;
|
|
assigneeUserId: string | null;
|
|
billingCode: string | null;
|
|
executionPolicy: Record<string, unknown> | null;
|
|
executionState: Record<string, unknown> | null;
|
|
monitorNextCheckAt: Date | null;
|
|
monitorWakeRequestedAt: Date | null;
|
|
monitorLastTriggeredAt: Date | null;
|
|
monitorAttemptCount: number | null;
|
|
monitorNotes: string | null;
|
|
monitorScheduledBy: string | null;
|
|
}
|
|
|
|
function parseMonitorDate(value: string | null | undefined) {
|
|
if (!value) return null;
|
|
const date = new Date(value);
|
|
return Number.isNaN(date.getTime()) ? null : date;
|
|
}
|
|
|
|
function issueMonitorLimitClearReason(input: {
|
|
monitor: IssueExecutionMonitorPolicy | null;
|
|
nextAttemptCount: number;
|
|
now: Date;
|
|
}): IssueExecutionMonitorClearReason | null {
|
|
const timeoutAt = parseMonitorDate(input.monitor?.timeoutAt ?? null);
|
|
if (timeoutAt && input.now.getTime() >= timeoutAt.getTime()) {
|
|
return "timeout_exceeded";
|
|
}
|
|
const maxAttempts = input.monitor?.maxAttempts ?? null;
|
|
if (maxAttempts !== null && input.nextAttemptCount > maxAttempts) {
|
|
return "max_attempts_exhausted";
|
|
}
|
|
return null;
|
|
}
|
|
|
|
function monitorRecoveryPolicy(
|
|
monitor: IssueExecutionMonitorPolicy | null,
|
|
): IssueExecutionMonitorRecoveryPolicy {
|
|
return monitor?.recoveryPolicy ?? "wake_owner";
|
|
}
|
|
|
|
function monitorRecoveryDetails(input: {
|
|
claimed: IssueMonitorDispatchRow;
|
|
scheduledAtIso: string;
|
|
nextAttemptCount: number;
|
|
clearReason: IssueExecutionMonitorClearReason;
|
|
recoveryPolicy: IssueExecutionMonitorRecoveryPolicy;
|
|
monitor: IssueExecutionMonitorPolicy | null;
|
|
source: "manual" | "scheduled";
|
|
}) {
|
|
return {
|
|
identifier: input.claimed.identifier,
|
|
nextCheckAt: input.scheduledAtIso,
|
|
attemptedAttemptCount: input.nextAttemptCount,
|
|
notes: input.claimed.monitorNotes ?? null,
|
|
serviceName: input.monitor?.serviceName ?? null,
|
|
timeoutAt: input.monitor?.timeoutAt ?? null,
|
|
maxAttempts: input.monitor?.maxAttempts ?? null,
|
|
clearReason: input.clearReason,
|
|
recoveryPolicy: input.recoveryPolicy,
|
|
source: input.source,
|
|
};
|
|
}
|
|
|
|
|
|
function monitorRecoveryComment(input: {
|
|
issue: IssueMonitorDispatchRow;
|
|
clearReason: IssueExecutionMonitorClearReason;
|
|
recoveryPolicy: IssueExecutionMonitorRecoveryPolicy;
|
|
nextAttemptCount: number;
|
|
}) {
|
|
const label = formatIssueIdentifierLink(
|
|
input.issue.identifier,
|
|
input.issue.id,
|
|
);
|
|
const reason =
|
|
input.clearReason === "timeout_exceeded"
|
|
? "its timeout was reached"
|
|
: "its maximum attempt count was reached";
|
|
return [
|
|
`Paperclip cleared the scheduled external-service monitor for ${label} because ${reason}.`,
|
|
"",
|
|
`- Attempt count: ${input.nextAttemptCount}`,
|
|
`- Recovery policy: ${input.recoveryPolicy}`,
|
|
"",
|
|
"Next action: inspect the external service state, record the result on this issue, and restore an explicit execution or waiting path if more work remains.",
|
|
].join("\n");
|
|
}
|
|
|
|
async function findOpenIssueMonitorRecoveryIssue(
|
|
claimed: IssueMonitorDispatchRow,
|
|
) {
|
|
return db
|
|
.select()
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, claimed.companyId),
|
|
eq(issues.originKind, RECOVERY_ORIGIN_KINDS.strandedIssueRecovery),
|
|
eq(issues.originId, claimed.id),
|
|
visibleIssueCondition(),
|
|
notInArray(issues.status, ["done", "cancelled"]),
|
|
),
|
|
)
|
|
.orderBy(desc(issues.createdAt))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
}
|
|
|
|
async function performIssueMonitorRecovery(input: {
|
|
claimed: IssueMonitorDispatchRow;
|
|
scheduledAtIso: string;
|
|
nextAttemptCount: number;
|
|
clearReason: IssueExecutionMonitorClearReason;
|
|
recoveryPolicy: IssueExecutionMonitorRecoveryPolicy;
|
|
monitor: IssueExecutionMonitorPolicy | null;
|
|
actorType: "user" | "agent" | "system";
|
|
actorId: string;
|
|
agentId: string | null;
|
|
runId: string | null;
|
|
activitySource: "manual" | "scheduled";
|
|
}) {
|
|
const reviewPathLost =
|
|
input.claimed.status === "in_review" &&
|
|
(await issuesSvc
|
|
.listReviewAttention(input.claimed.companyId, [input.claimed])
|
|
.then(
|
|
(attention) => attention.get(input.claimed.id)?.state === "stalled",
|
|
));
|
|
const reviewPathContext = reviewPathLost
|
|
? {
|
|
reviewPathLost: true,
|
|
reviewPathConsumedRef: `monitor:${input.claimed.id}:${input.clearReason}:${input.scheduledAtIso}`,
|
|
reviewPathInstruction: REVIEW_PATH_RECOVERY_INSTRUCTION,
|
|
}
|
|
: null;
|
|
const details = monitorRecoveryDetails({
|
|
claimed: input.claimed,
|
|
scheduledAtIso: input.scheduledAtIso,
|
|
nextAttemptCount: input.nextAttemptCount,
|
|
clearReason: input.clearReason,
|
|
recoveryPolicy: input.recoveryPolicy,
|
|
monitor: input.monitor,
|
|
source: input.activitySource,
|
|
});
|
|
|
|
if (input.recoveryPolicy === "create_recovery_issue") {
|
|
let recoveryIssue = await findOpenIssueMonitorRecoveryIssue(
|
|
input.claimed,
|
|
);
|
|
if (!recoveryIssue) {
|
|
recoveryIssue = await issuesSvc.create(input.claimed.companyId, {
|
|
title: `Recover external-service monitor for ${input.claimed.identifier ?? input.claimed.title}`,
|
|
description: monitorRecoveryComment({
|
|
issue: input.claimed,
|
|
clearReason: input.clearReason,
|
|
recoveryPolicy: input.recoveryPolicy,
|
|
nextAttemptCount: input.nextAttemptCount,
|
|
}),
|
|
status: "todo",
|
|
priority: "high",
|
|
parentId: input.claimed.id,
|
|
// Recovery work is structural system machinery tied to the stranded task.
|
|
parentDepthPolicy: "allow_nested",
|
|
projectId: input.claimed.projectId,
|
|
goalId: input.claimed.goalId,
|
|
assigneeAgentId: input.claimed.assigneeAgentId,
|
|
originKind: RECOVERY_ORIGIN_KINDS.strandedIssueRecovery,
|
|
originId: input.claimed.id,
|
|
originFingerprint: `issue_monitor:${input.clearReason}`,
|
|
billingCode: input.claimed.billingCode,
|
|
});
|
|
}
|
|
|
|
if (recoveryIssue.assigneeAgentId) {
|
|
await enqueueWakeup(recoveryIssue.assigneeAgentId, {
|
|
source: "automation",
|
|
triggerDetail: "system",
|
|
reason: "issue_monitor_recovery_issue",
|
|
idempotencyKey: `issue-monitor-recovery-issue:${input.claimed.id}:${input.clearReason}:${input.scheduledAtIso}`,
|
|
payload: withRecoveryContext(
|
|
{ issueId: recoveryIssue.id, sourceIssueId: input.claimed.id },
|
|
"status_only",
|
|
),
|
|
requestedByActorType: input.actorType,
|
|
requestedByActorId: input.actorId,
|
|
contextSnapshot: withRecoveryContext(
|
|
{
|
|
issueId: recoveryIssue.id,
|
|
sourceIssueId: input.claimed.id,
|
|
source: "issue.monitor.recovery_issue",
|
|
wakeReason: "issue_monitor_recovery_issue",
|
|
},
|
|
"status_only",
|
|
),
|
|
});
|
|
}
|
|
|
|
await logActivity(db, {
|
|
companyId: input.claimed.companyId,
|
|
actorType: input.actorType,
|
|
actorId: input.actorId,
|
|
agentId: input.agentId,
|
|
runId: input.runId,
|
|
action: "issue.monitor_recovery_issue_created",
|
|
entityType: "issue",
|
|
entityId: input.claimed.id,
|
|
details: {
|
|
...details,
|
|
recoveryIssueId: recoveryIssue.id,
|
|
recoveryIdentifier: recoveryIssue.identifier,
|
|
},
|
|
});
|
|
return;
|
|
}
|
|
|
|
const recoveryKey = `issue-monitor-recovery:${input.claimed.id}:${input.scheduledAtIso}`;
|
|
|
|
|
|
const acceptedRecoveryKeys = [
|
|
recoveryKey,
|
|
`issue-monitor-recovery:${input.claimed.id}:max_attempts_exhausted:${input.scheduledAtIso}`,
|
|
`issue-monitor-recovery:${input.claimed.id}:timeout_exceeded:${input.scheduledAtIso}`,
|
|
];
|
|
|
|
|
|
|
|
const [accepted] = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests)
|
|
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, agentWakeupRequests.runId),
|
|
eq(heartbeatRuns.companyId, agentWakeupRequests.companyId),
|
|
eq(heartbeatRuns.agentId, agentWakeupRequests.agentId),
|
|
sql`coalesce(${heartbeatRuns.contextSnapshot}->>'issueId', ${heartbeatRuns.contextSnapshot}->>'taskId',
|
|
${heartbeatRuns.nativeIssueId}::text) = ${input.claimed.id}`))
|
|
.where(and(eq(agentWakeupRequests.companyId, input.claimed.companyId),
|
|
eq(agentWakeupRequests.agentId, input.claimed.assigneeAgentId!),
|
|
inArray(agentWakeupRequests.idempotencyKey, acceptedRecoveryKeys),
|
|
inArray(agentWakeupRequests.reason, ["issue_monitor_recovery", "issue_execution_same_name"]),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${input.claimed.id}`,
|
|
sql`${agentWakeupRequests.status} <> 'skipped'`)).limit(1);
|
|
const wake = accepted ?? await enqueueWakeup(input.claimed.assigneeAgentId!, {
|
|
issueStateGuard: { statuses: [input.claimed.status], assigneeAgentId: input.claimed.assigneeAgentId!,
|
|
monitorNextCheckAt: input.scheduledAtIso,
|
|
monitorWakeRequestedAt: input.claimed.monitorWakeRequestedAt!.toISOString() },
|
|
source: "automation",
|
|
triggerDetail: "system",
|
|
reason: "issue_monitor_recovery",
|
|
idempotencyKey: recoveryKey,
|
|
payload: withRecoveryContext(
|
|
{
|
|
issueId: input.claimed.id,
|
|
monitorAttemptCount: input.nextAttemptCount,
|
|
monitorNotes: input.claimed.monitorNotes ?? null,
|
|
clearReason: input.clearReason,
|
|
serviceName: input.monitor?.serviceName ?? null,
|
|
timeoutAt: input.monitor?.timeoutAt ?? null,
|
|
maxAttempts: input.monitor?.maxAttempts ?? null,
|
|
...(reviewPathContext ?? {}),
|
|
},
|
|
"status_only",
|
|
),
|
|
requestedByActorType: input.actorType,
|
|
requestedByActorId: input.actorId,
|
|
contextSnapshot: withRecoveryContext(
|
|
{
|
|
issueId: input.claimed.id,
|
|
source: "issue.monitor.recovery",
|
|
wakeReason: "issue_monitor_recovery",
|
|
monitorAttemptCount: input.nextAttemptCount,
|
|
monitorNotes: input.claimed.monitorNotes ?? null,
|
|
clearReason: input.clearReason,
|
|
serviceName: input.monitor?.serviceName ?? null,
|
|
timeoutAt: input.monitor?.timeoutAt ?? null,
|
|
maxAttempts: input.monitor?.maxAttempts ?? null,
|
|
...(reviewPathContext ?? {}),
|
|
},
|
|
"status_only",
|
|
),
|
|
});
|
|
|
|
if (!wake) return false;
|
|
|
|
await logActivity(db, {
|
|
companyId: input.claimed.companyId,
|
|
actorType: input.actorType,
|
|
actorId: input.actorId,
|
|
agentId: input.agentId,
|
|
runId: input.runId,
|
|
action: "issue.monitor_recovery_wake_queued",
|
|
entityType: "issue",
|
|
entityId: input.claimed.id,
|
|
details,
|
|
});
|
|
return true;
|
|
}
|
|
|
|
async function clearIssueMonitorAndRecover(input: {
|
|
claimed: IssueMonitorDispatchRow;
|
|
policy: ReturnType<typeof normalizeIssueExecutionPolicy>;
|
|
scheduledAtIso: string;
|
|
nextAttemptCount: number;
|
|
clearReason: IssueExecutionMonitorClearReason;
|
|
recoveryPolicy: IssueExecutionMonitorRecoveryPolicy;
|
|
monitor: IssueExecutionMonitorPolicy | null;
|
|
now: Date;
|
|
actorType: "user" | "agent" | "system";
|
|
actorId: string;
|
|
agentId: string | null;
|
|
runId: string | null;
|
|
activitySource: "manual" | "scheduled";
|
|
}) {
|
|
const recover = () => performIssueMonitorRecovery(input);
|
|
if (input.recoveryPolicy === "wake_owner") {
|
|
try {
|
|
if (!await recover()) {
|
|
await releaseIssueMonitorWithDiagnostic(input.claimed, input, {
|
|
action: "issue.monitor_deferred", reason: "monitor_recovery_deferred",
|
|
});
|
|
return { outcome: "skipped" as const, reason: "monitor_recovery_deferred" };
|
|
}
|
|
} catch (error) {
|
|
await releaseIssueMonitorWithDiagnostic(input.claimed, input, {
|
|
action: "issue.monitor_dispatch_failed", reason: "monitor_recovery_failed",
|
|
});
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
|
|
|
|
|
|
const publications: ActivityPublication[] = [];
|
|
const cleared = await db.transaction(async tx => {
|
|
await notifyIssueReviewState(tx, { companyId: input.claimed.companyId, issueId: input.claimed.id });
|
|
const rows = await tx.update(issues).set({
|
|
...monitorOnlyDispatchPatch(buildIssueMonitorClearedPatch({
|
|
issue: input.claimed, policy: input.policy, clearReason: input.clearReason, clearedAt: input.now,
|
|
})),
|
|
updatedAt: input.now,
|
|
}).where(issueMonitorClaimCondition(input.claimed)).returning({ id: issues.id });
|
|
if (rows.length && input.recoveryPolicy === "escalate_to_board") {
|
|
await insertIssueComment(tx, {
|
|
companyId: input.claimed.companyId,
|
|
issueId: input.claimed.id,
|
|
body: monitorRecoveryComment({
|
|
issue: input.claimed,
|
|
clearReason: input.clearReason,
|
|
recoveryPolicy: input.recoveryPolicy,
|
|
nextAttemptCount: input.nextAttemptCount,
|
|
}),
|
|
});
|
|
|
|
await logActivity(tx as unknown as Db, {
|
|
companyId: input.claimed.companyId,
|
|
actorType: input.actorType,
|
|
actorId: input.actorId,
|
|
agentId: input.agentId,
|
|
runId: input.runId,
|
|
action: "issue.monitor_escalated_to_board",
|
|
entityType: "issue",
|
|
entityId: input.claimed.id,
|
|
details: monitorRecoveryDetails({ ...input, source: input.activitySource }),
|
|
}, publications);
|
|
}
|
|
|
|
return rows;
|
|
});
|
|
for (const publication of publications) publishActivity(publication);
|
|
if (cleared.length === 0) return { outcome: "skipped" as const, reason: "monitor_replaced" };
|
|
|
|
await logActivity(db, {
|
|
companyId: input.claimed.companyId,
|
|
actorType: input.actorType,
|
|
actorId: input.actorId,
|
|
agentId: input.agentId,
|
|
runId: input.runId,
|
|
action: "issue.monitor_exhausted",
|
|
entityType: "issue",
|
|
entityId: input.claimed.id,
|
|
details: monitorRecoveryDetails({
|
|
claimed: input.claimed,
|
|
scheduledAtIso: input.scheduledAtIso,
|
|
nextAttemptCount: input.nextAttemptCount,
|
|
clearReason: input.clearReason,
|
|
recoveryPolicy: input.recoveryPolicy,
|
|
monitor: input.monitor,
|
|
source: input.activitySource,
|
|
}),
|
|
});
|
|
|
|
if (input.recoveryPolicy === "create_recovery_issue") await recover();
|
|
|
|
return { outcome: "skipped" as const, reason: input.clearReason };
|
|
}
|
|
|
|
function monitorOnlyDispatchPatch<T extends ReturnType<typeof buildIssueMonitorClearedPatch>>(patch: T) {
|
|
// Admission and consumption are separate transactions. Preserve any review
|
|
// policy/state changes made between them; only the monitor belongs to us.
|
|
return {
|
|
...patch,
|
|
executionPolicy: sql`nullif(${issues.executionPolicy} - 'monitor', '{}'::jsonb)`,
|
|
executionState: sql`jsonb_set(coalesce(${issues.executionState}, ${JSON.stringify(patch.executionState)}::jsonb),
|
|
'{monitor}', ${JSON.stringify(patch.executionState?.monitor ?? null)}::jsonb)`,
|
|
};
|
|
}
|
|
|
|
function issueMonitorClaimCondition(claimed: IssueMonitorDispatchRow) {
|
|
return and(eq(issues.id, claimed.id), eq(issues.companyId, claimed.companyId),
|
|
eq(issues.assigneeAgentId, claimed.assigneeAgentId!), isNull(issues.assigneeUserId),
|
|
eq(issues.status, claimed.status), eq(issues.monitorNextCheckAt, claimed.monitorNextCheckAt!),
|
|
eq(issues.monitorWakeRequestedAt, claimed.monitorWakeRequestedAt!));
|
|
}
|
|
|
|
async function releaseIssueMonitorWithDiagnostic(
|
|
claimed: IssueMonitorDispatchRow,
|
|
input: Pick<Parameters<typeof dispatchClaimedIssueMonitor>[1],
|
|
"now" | "actorType" | "actorId" | "agentId" | "runId" | "activitySource">,
|
|
diagnostic: { action: "issue.monitor_deferred" | "issue.monitor_dispatch_failed";
|
|
reason: string; blockerCause?: string; httpStatus?: number },
|
|
) {
|
|
const publications: ActivityPublication[] = [];
|
|
try {
|
|
await db.transaction(async (tx) => {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
|
await notifyStatusCardIssueChange(tx, claimed.companyId);
|
|
const released = await tx.update(issues)
|
|
.set({ monitorWakeRequestedAt: null, updatedAt: input.now })
|
|
.where(issueMonitorClaimCondition(claimed)).returning({ id: issues.id });
|
|
if (released.length === 0) return;
|
|
// The issue lock serializes diagnostics from competing scheduler instances.
|
|
// Bound recurring reads to this issue's recent indexed activity history.
|
|
const [recent] = await tx.select({ id: activityLog.id }).from(activityLog).where(and(
|
|
eq(activityLog.companyId, claimed.companyId), eq(activityLog.entityType, "issue"),
|
|
eq(activityLog.entityId, claimed.id), eq(activityLog.action, diagnostic.action),
|
|
gte(activityLog.createdAt, new Date(Date.now() - 5 * 60_000)),
|
|
sql`${activityLog.details}->>'nextCheckAt' = ${claimed.monitorNextCheckAt!.toISOString()}`,
|
|
sql`${activityLog.details}->>'reason' = ${diagnostic.reason}`,
|
|
sql`coalesce(${activityLog.details}->>'blockerCause', '') = ${diagnostic.blockerCause ?? ""}`,
|
|
)).limit(1);
|
|
if (recent) return;
|
|
await logActivity(tx as unknown as Db, {
|
|
companyId: claimed.companyId, actorType: input.actorType, actorId: input.actorId,
|
|
agentId: input.agentId, runId: input.runId, action: diagnostic.action,
|
|
entityType: "issue", entityId: claimed.id,
|
|
details: { nextCheckAt: claimed.monitorNextCheckAt!.toISOString(),
|
|
checkedAt: input.now.toISOString(), attemptCount: claimed.monitorAttemptCount ?? 0,
|
|
reason: diagnostic.reason, source: input.activitySource,
|
|
...(diagnostic.blockerCause ? { blockerCause: diagnostic.blockerCause } : {}),
|
|
...(diagnostic.httpStatus ? { httpStatus: diagnostic.httpStatus } : {}),
|
|
},
|
|
}, publications);
|
|
});
|
|
for (const publication of publications) publishActivity(publication);
|
|
} catch (err) {
|
|
// Diagnostics are optional. Release the original claim even if audit storage
|
|
// failed, and never clobber a monitor that was replaced in the meantime.
|
|
await db.transaction(async tx => {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
|
await notifyStatusCardIssueChange(tx, claimed.companyId);
|
|
await tx.update(issues).set({ monitorWakeRequestedAt: null, updatedAt: input.now })
|
|
.where(issueMonitorClaimCondition(claimed));
|
|
});
|
|
logger.warn({ err, issueId: claimed.id }, "issue monitor diagnostic could not be recorded");
|
|
}
|
|
}
|
|
|
|
async function dispatchClaimedIssueMonitor(
|
|
claimed: IssueMonitorDispatchRow,
|
|
input: {
|
|
now: Date;
|
|
source: "automation" | "on_demand";
|
|
triggerDetail: "manual" | "system";
|
|
wakeReason: string;
|
|
actorType: "user" | "agent" | "system";
|
|
actorId: string;
|
|
agentId: string | null;
|
|
runId: string | null;
|
|
clearOnClientError: boolean;
|
|
activitySource: "manual" | "scheduled";
|
|
},
|
|
) {
|
|
if (!claimed.assigneeAgentId || !claimed.monitorNextCheckAt) {
|
|
throw conflict("Issue monitor is not ready to dispatch");
|
|
}
|
|
|
|
const scheduledAtIso = claimed.monitorNextCheckAt.toISOString();
|
|
const nextAttemptCount = (claimed.monitorAttemptCount ?? 0) + 1;
|
|
const policy = normalizeIssueExecutionPolicy(
|
|
claimed.executionPolicy ?? null,
|
|
);
|
|
const monitor = policy?.monitor ?? null;
|
|
const clearReason = issueMonitorLimitClearReason({
|
|
monitor,
|
|
nextAttemptCount,
|
|
now: input.now,
|
|
});
|
|
const recoveryPolicy = monitorRecoveryPolicy(monitor);
|
|
const monitorMetadata = {
|
|
serviceName: monitor?.serviceName ?? null,
|
|
timeoutAt: monitor?.timeoutAt ?? null,
|
|
maxAttempts: monitor?.maxAttempts ?? null,
|
|
recoveryPolicy: monitor?.recoveryPolicy ?? null,
|
|
};
|
|
const executionState =
|
|
claimed.status === "in_review"
|
|
? parseIssueExecutionState(claimed.executionState)
|
|
: null;
|
|
const currentParticipant =
|
|
executionState?.status === "pending"
|
|
? executionState.currentParticipant
|
|
: null;
|
|
const reviewParticipantAgentId =
|
|
currentParticipant?.type === "agent" ? currentParticipant.agentId : null;
|
|
const isProviderQuotaReviewMonitor =
|
|
monitor?.serviceName === PROVIDER_QUOTA_MONITOR_SERVICE_NAME &&
|
|
Boolean(reviewParticipantAgentId);
|
|
const targetAgentId = isProviderQuotaReviewMonitor
|
|
? reviewParticipantAgentId
|
|
: claimed.assigneeAgentId;
|
|
if (!targetAgentId) {
|
|
throw conflict("Issue monitor has no agent target");
|
|
}
|
|
const wakeReason = isProviderQuotaReviewMonitor
|
|
? EXECUTION_REVIEW_PARTICIPANT_RECOVERY_WAKE_REASON
|
|
: input.wakeReason;
|
|
const reviewRecoveryContext = isProviderQuotaReviewMonitor
|
|
? {
|
|
retryReason: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_RETRY_REASON,
|
|
currentStageId: executionState?.currentStageId ?? null,
|
|
currentStageType: executionState?.currentStageType ?? null,
|
|
reviewRecoveryInstruction:
|
|
"The previous reviewer run reached provider quota. Resume this execution-review stage now that the quota wait has elapsed.",
|
|
}
|
|
: {};
|
|
|
|
if (clearReason) {
|
|
return clearIssueMonitorAndRecover({
|
|
claimed,
|
|
policy,
|
|
scheduledAtIso,
|
|
nextAttemptCount,
|
|
clearReason,
|
|
recoveryPolicy,
|
|
monitor,
|
|
now: input.now,
|
|
actorType: input.actorType,
|
|
actorId: input.actorId,
|
|
agentId: input.agentId,
|
|
runId: input.runId,
|
|
activitySource: input.activitySource,
|
|
});
|
|
}
|
|
|
|
try {
|
|
if (monitor?.serviceName === PROVIDER_QUOTA_MONITOR_SERVICE_NAME) {
|
|
// Normalized monitor projections redact externalRef. Read the claimed
|
|
// persisted policy only on this server-owned quota recovery path.
|
|
const sourceRunId = readNonEmptyString(
|
|
parseObject(parseObject(claimed.executionPolicy).monitor).externalRef,
|
|
);
|
|
const sourceRun =
|
|
sourceRunId && isUuidLike(sourceRunId)
|
|
? await getRun(sourceRunId, { unsafeFullResultJson: true })
|
|
: null;
|
|
if (
|
|
!sourceRun ||
|
|
sourceRun.companyId !== claimed.companyId ||
|
|
sourceRun.agentId !== targetAgentId ||
|
|
sourceRun.contextSnapshot?.issueId !== claimed.id ||
|
|
!["failed", "timed_out", "interrupted", "cancelled"].includes(
|
|
sourceRun.status,
|
|
)
|
|
) {
|
|
throw conflict(
|
|
"The quota recovery source changed; inspect the current task execution.",
|
|
);
|
|
}
|
|
if (sourceRun.runtimeMode === "native") {
|
|
throw conflict(
|
|
"Native execution recovery owns this provider failure; a quota monitor cannot start a replacement.",
|
|
);
|
|
}
|
|
if (await legacyExecutionNeedsReconciliationWithEvidence(db, sourceRun)) {
|
|
await terminalizeLegacyExecution({
|
|
db,
|
|
run: sourceRun,
|
|
status: sourceRun.status,
|
|
});
|
|
} else {
|
|
const targetAgent = await getAgent(targetAgentId);
|
|
if (!targetAgent)
|
|
throw conflict("The quota recovery agent is unavailable.");
|
|
const scheduled = await scheduleBoundedRetryForRun(
|
|
sourceRun,
|
|
targetAgent,
|
|
{
|
|
now: input.now,
|
|
...(isProviderQuotaReviewMonitor
|
|
? {
|
|
retryReason:
|
|
EXECUTION_REVIEW_PARTICIPANT_RECOVERY_RETRY_REASON,
|
|
wakeReason:
|
|
EXECUTION_REVIEW_PARTICIPANT_RECOVERY_WAKE_REASON,
|
|
}
|
|
: {}),
|
|
},
|
|
);
|
|
if (scheduled.outcome === "not_scheduled")
|
|
throw conflict(scheduled.reason);
|
|
}
|
|
} else {
|
|
let executionDeferredCause: string | null = null;
|
|
const wake = await enqueueWakeup(targetAgentId, {
|
|
issueStateGuard: { statuses: [claimed.status], assigneeAgentId: claimed.assigneeAgentId,
|
|
monitorNextCheckAt: scheduledAtIso, monitorWakeRequestedAt: claimed.monitorWakeRequestedAt!.toISOString(),
|
|
onExecutionDeferred: cause => { executionDeferredCause = cause; } },
|
|
source: input.source,
|
|
triggerDetail: input.triggerDetail,
|
|
reason: wakeReason,
|
|
idempotencyKey: `issue-monitor:${claimed.id}:${scheduledAtIso}`,
|
|
payload: {
|
|
issueId: claimed.id,
|
|
nextCheckAt: scheduledAtIso,
|
|
monitorAttemptCount: nextAttemptCount,
|
|
monitorNotes: claimed.monitorNotes ?? null,
|
|
...monitorMetadata,
|
|
...reviewRecoveryContext,
|
|
source: input.activitySource,
|
|
},
|
|
requestedByActorType: input.actorType,
|
|
requestedByActorId: input.actorId,
|
|
contextSnapshot: {
|
|
issueId: claimed.id,
|
|
source: isProviderQuotaReviewMonitor
|
|
? "issue.execution_review_recovery"
|
|
: "issue.monitor",
|
|
wakeReason,
|
|
nextCheckAt: scheduledAtIso,
|
|
monitorAttemptCount: nextAttemptCount,
|
|
monitorNotes: claimed.monitorNotes ?? null,
|
|
...monitorMetadata,
|
|
...reviewRecoveryContext,
|
|
manualTrigger: input.activitySource === "manual",
|
|
},
|
|
});
|
|
if (!wake) {
|
|
const blockerCause = executionDeferredCause &&
|
|
([...EXECUTION_RECONCILIATION_CAUSES, "execution_owner_active"] as readonly string[]).includes(executionDeferredCause)
|
|
? executionDeferredCause : undefined;
|
|
const reason = executionDeferredCause ? "execution_reconciliation_required" : "monitor_dispatch_deferred";
|
|
await releaseIssueMonitorWithDiagnostic(claimed, input, {
|
|
action: "issue.monitor_deferred", reason, ...(blockerCause ? { blockerCause } : {}),
|
|
});
|
|
return { outcome: "skipped" as const, reason };
|
|
}
|
|
}
|
|
|
|
const consumed = await db.transaction(async tx => {
|
|
await notifyStatusCardIssueChange(tx, claimed.companyId);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
|
return tx.update(issues)
|
|
.set({
|
|
...monitorOnlyDispatchPatch(buildIssueMonitorTriggeredPatch({
|
|
issue: claimed,
|
|
policy,
|
|
triggeredAt: input.now,
|
|
})),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(issueMonitorClaimCondition(claimed))
|
|
.returning({ id: issues.id });
|
|
});
|
|
if (consumed.length === 0) return { outcome: "triggered" as const };
|
|
|
|
await logActivity(db, {
|
|
companyId: claimed.companyId,
|
|
actorType: input.actorType,
|
|
actorId: input.actorId,
|
|
agentId: input.agentId,
|
|
runId: input.runId,
|
|
action: "issue.monitor_triggered",
|
|
entityType: "issue",
|
|
entityId: claimed.id,
|
|
details: {
|
|
identifier: claimed.identifier,
|
|
nextCheckAt: scheduledAtIso,
|
|
lastTriggeredAt: input.now.toISOString(),
|
|
attemptCount: nextAttemptCount,
|
|
notes: claimed.monitorNotes ?? null,
|
|
...monitorMetadata,
|
|
source: input.activitySource,
|
|
},
|
|
});
|
|
|
|
return { outcome: "triggered" as const };
|
|
} catch (err) {
|
|
if (err instanceof HttpError && err.status >= 400 && err.status < 500) {
|
|
if (input.clearOnClientError) {
|
|
await db.transaction(async tx => {
|
|
await notifyIssueReviewState(tx, { companyId: claimed.companyId, issueId: claimed.id });
|
|
await tx
|
|
.update(issues)
|
|
.set({
|
|
...monitorOnlyDispatchPatch(buildIssueMonitorClearedPatch({
|
|
issue: claimed,
|
|
policy,
|
|
clearReason: "dispatch_skipped",
|
|
clearedAt: input.now,
|
|
})),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(issueMonitorClaimCondition(claimed));
|
|
});
|
|
|
|
await logActivity(db, {
|
|
companyId: claimed.companyId,
|
|
actorType: input.actorType,
|
|
actorId: input.actorId,
|
|
agentId: input.agentId,
|
|
runId: input.runId,
|
|
action: "issue.monitor_skipped",
|
|
entityType: "issue",
|
|
entityId: claimed.id,
|
|
details: {
|
|
identifier: claimed.identifier,
|
|
nextCheckAt: scheduledAtIso,
|
|
attemptCount: nextAttemptCount,
|
|
notes: claimed.monitorNotes ?? null,
|
|
reason: err.message,
|
|
source: input.activitySource,
|
|
},
|
|
});
|
|
|
|
return { outcome: "skipped" as const, reason: err.message };
|
|
}
|
|
}
|
|
await releaseIssueMonitorWithDiagnostic(claimed, input, {
|
|
action: "issue.monitor_dispatch_failed", reason: "monitor_dispatch_failed",
|
|
...(err instanceof HttpError ? { httpStatus: err.status } : {}),
|
|
});
|
|
throw err;
|
|
}
|
|
}
|
|
|
|
function noActiveNativeMonitorRun() {
|
|
return sql`not exists (select 1 from ${heartbeatRuns} monitor_run
|
|
where monitor_run.company_id = ${issues.companyId}
|
|
and monitor_run.native_issue_id = ${issues.id}
|
|
and monitor_run.runtime_mode = 'native'
|
|
and (monitor_run.status in ('queued', 'running', 'scheduled_retry')
|
|
or exists (select 1 from ${nativeRunFinalizations} monitor_finalization
|
|
where monitor_finalization.company_id = monitor_run.company_id
|
|
and monitor_finalization.run_id = monitor_run.id
|
|
and monitor_finalization.phase not in ('committed', 'terminal_failure'))))`;
|
|
}
|
|
|
|
async function triggerIssueMonitor(
|
|
issueId: string,
|
|
input?: {
|
|
now?: Date;
|
|
actorType?: "user" | "agent" | "system";
|
|
actorId?: string | null;
|
|
agentId?: string | null;
|
|
runId?: string | null;
|
|
wakeReason?: string;
|
|
},
|
|
) {
|
|
const now = input?.now ?? new Date();
|
|
const actorType = input?.actorType ?? "system";
|
|
const actorId =
|
|
input?.actorId ?? (actorType === "system" ? "heartbeat_scheduler" : null);
|
|
if (!actorId) {
|
|
throw conflict("Issue monitor trigger requires an actor");
|
|
}
|
|
|
|
const issue = await db
|
|
.select(issueMonitorDispatchColumns)
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!issue) {
|
|
throw notFound("Issue not found");
|
|
}
|
|
if (!issue.monitorNextCheckAt) {
|
|
throw conflict("Issue has no scheduled monitor");
|
|
}
|
|
if (!issue.assigneeAgentId || issue.assigneeUserId) {
|
|
throw conflict("Issue monitor requires an agent assignee");
|
|
}
|
|
if (issue.status === "blocked" && actorType === "user" &&
|
|
normalizeIssueExecutionPolicy(issue.executionPolicy ?? null)?.monitor?.serviceName === PROVIDER_QUOTA_MONITOR_SERVICE_NAME) {
|
|
// Quota recovery schedules a bounded retry directly, rather than using
|
|
// ordinary wake admission. It must not consume a blocked task's monitor.
|
|
throw conflict("Resume this task before checking provider usage.", { code: "blocked_quota_monitor" });
|
|
}
|
|
// A human can request one check of a blocked task. Admission still owns
|
|
// dependencies, recovery, pause and budget gates; automatic ticks stay idle.
|
|
const allowedStatuses = actorType === "user"
|
|
? ["in_progress", "in_review", "blocked"]
|
|
: ["in_progress", "in_review"];
|
|
if (!allowedStatuses.includes(issue.status)) {
|
|
throw conflict(
|
|
"Issue monitor can only run while the issue is in progress or in review",
|
|
);
|
|
}
|
|
|
|
const staleClaimThreshold = new Date(now.getTime() - 5 * 60 * 1000);
|
|
const claimed = await db.transaction(async (tx) => {
|
|
await notifyStatusCardIssueChange(tx, issue.companyId);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
|
const [updated] = await tx
|
|
.update(issues)
|
|
.set({
|
|
monitorWakeRequestedAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, issueId),
|
|
noActiveNativeMonitorRun(),
|
|
sql`${issues.monitorNextCheckAt} is not null`,
|
|
isNull(issues.assigneeUserId),
|
|
sql`${issues.assigneeAgentId} is not null`,
|
|
inArray(issues.status, allowedStatuses),
|
|
or(
|
|
isNull(issues.monitorWakeRequestedAt),
|
|
lt(issues.monitorWakeRequestedAt, staleClaimThreshold),
|
|
),
|
|
),
|
|
)
|
|
.returning();
|
|
return (updated ?? null) as IssueMonitorDispatchRow | null;
|
|
});
|
|
|
|
if (!claimed) {
|
|
throw conflict("Issue monitor check is already in progress or waiting for the previous run to settle");
|
|
}
|
|
|
|
return dispatchClaimedIssueMonitor(claimed, {
|
|
now,
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
wakeReason: input?.wakeReason ?? "issue_monitor_due",
|
|
actorType,
|
|
actorId,
|
|
agentId: input?.agentId ?? null,
|
|
runId: input?.runId ?? null,
|
|
clearOnClientError: false,
|
|
activitySource: "manual",
|
|
});
|
|
}
|
|
|
|
async function tickDueIssueMonitors(now: Date, canContinue: () => boolean) {
|
|
const staleClaimThreshold = new Date(now.getTime() - 5 * 60 * 1000);
|
|
const dueMonitors = await db
|
|
.select(issueMonitorDispatchColumns)
|
|
.from(issues)
|
|
.innerJoin(companies, eq(companies.id, issues.companyId))
|
|
.where(
|
|
and(
|
|
eq(companies.status, "active"),
|
|
noActiveNativeMonitorRun(),
|
|
sql`${issues.monitorNextCheckAt} is not null`,
|
|
lte(issues.monitorNextCheckAt, now),
|
|
isNull(issues.assigneeUserId),
|
|
sql`${issues.assigneeAgentId} is not null`,
|
|
inArray(issues.status, ["in_progress", "in_review"]),
|
|
or(
|
|
isNull(issues.monitorWakeRequestedAt),
|
|
lt(issues.monitorWakeRequestedAt, staleClaimThreshold),
|
|
),
|
|
),
|
|
)
|
|
.orderBy(asc(issues.monitorNextCheckAt), asc(issues.updatedAt))
|
|
.limit(50);
|
|
|
|
let triggered = 0;
|
|
let skipped = 0;
|
|
|
|
for (const due of dueMonitors) {
|
|
if (!canContinue()) break;
|
|
const claimed = await db.transaction(async (tx) => {
|
|
await notifyStatusCardIssueChange(tx, due.companyId);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
|
const [updated] = await tx
|
|
.update(issues)
|
|
.set({
|
|
monitorWakeRequestedAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.id, due.id),
|
|
noActiveNativeMonitorRun(),
|
|
sql`${issues.monitorNextCheckAt} is not null`,
|
|
lte(issues.monitorNextCheckAt, now),
|
|
isNull(issues.assigneeUserId),
|
|
sql`${issues.assigneeAgentId} is not null`,
|
|
inArray(issues.status, ["in_progress", "in_review"]),
|
|
or(
|
|
isNull(issues.monitorWakeRequestedAt),
|
|
lt(issues.monitorWakeRequestedAt, staleClaimThreshold),
|
|
),
|
|
),
|
|
)
|
|
.returning();
|
|
return (updated ?? null) as IssueMonitorDispatchRow | null;
|
|
});
|
|
|
|
if (!claimed) continue;
|
|
if (!canContinue()) break;
|
|
|
|
try {
|
|
const result = await dispatchClaimedIssueMonitor(claimed, {
|
|
now,
|
|
source: "automation",
|
|
triggerDetail: "system",
|
|
wakeReason: "issue_monitor_due",
|
|
actorType: "system",
|
|
actorId: "heartbeat_scheduler",
|
|
agentId: null,
|
|
runId: null,
|
|
clearOnClientError: true,
|
|
activitySource: "scheduled",
|
|
});
|
|
if (result.outcome === "triggered") triggered += 1;
|
|
if (result.outcome === "skipped") skipped += 1;
|
|
} catch (err) {
|
|
logger.error({ err, issueId: claimed.id }, "issue monitor tick failed");
|
|
}
|
|
}
|
|
|
|
return {
|
|
checked: dueMonitors.length,
|
|
triggered,
|
|
skipped,
|
|
};
|
|
}
|
|
|
|
async function tickIssueMonitors(now = new Date(), canContinue: () => boolean = () => true) {
|
|
if (!canContinue() || (await getSchedulingSuppression(true)).suppressed) {
|
|
return { checked: 0, triggered: 0, skipped: 0 };
|
|
}
|
|
return tickDueIssueMonitors(now, canContinue);
|
|
}
|
|
|
|
async function nextIssueMonitorAt(): Promise<number | null> {
|
|
if ((await getSchedulingSuppression(true)).suppressed) return null;
|
|
const [row] = await db.select({ at: sql<string | null>`min(greatest(${issues.monitorNextCheckAt},
|
|
${issues.monitorWakeRequestedAt} + interval '5 minutes' + interval '1 millisecond'))` })
|
|
.from(issues).innerJoin(companies, eq(companies.id, issues.companyId))
|
|
.where(and(eq(companies.status, "active"), noActiveNativeMonitorRun(),
|
|
sql`${issues.monitorNextCheckAt} is not null`, isNull(issues.assigneeUserId),
|
|
sql`${issues.assigneeAgentId} is not null`, inArray(issues.status, ["in_progress", "in_review"])));
|
|
|
|
|
|
return row?.at == null ? null : new Date(row.at).getTime();
|
|
}
|
|
|
|
async function recoverActiveSessionGoals() {
|
|
if ((await getSchedulingSuppression()).suppressed) {
|
|
return { scanned: 0, enqueued: 0 };
|
|
}
|
|
const sessions = await db
|
|
.select({
|
|
id: agentTaskSessions.id,
|
|
companyId: agentTaskSessions.companyId,
|
|
agentId: agentTaskSessions.agentId,
|
|
issueId: agentTaskSessions.taskKey,
|
|
revision: agentTaskSessions.goalRevision,
|
|
})
|
|
.from(agentTaskSessions)
|
|
.innerJoin(
|
|
issues,
|
|
and(
|
|
sql`${issues.id}::text = ${agentTaskSessions.taskKey}`,
|
|
eq(issues.companyId, agentTaskSessions.companyId),
|
|
eq(issues.assigneeAgentId, agentTaskSessions.agentId),
|
|
),
|
|
)
|
|
.where(
|
|
and(
|
|
eq(agentTaskSessions.goalDesiredState, "active"),
|
|
eq(agentTaskSessions.goalStatus, "active"),
|
|
notInArray(issues.status, ["done", "cancelled"]),
|
|
),
|
|
);
|
|
let enqueued = 0;
|
|
for (const session of sessions) {
|
|
const run = await enqueueWakeup(session.agentId, {
|
|
source: "automation",
|
|
triggerDetail: "system",
|
|
reason: "goal_control",
|
|
payload: { issueId: session.issueId, intent: "goal_recovery" },
|
|
idempotencyKey: `goal_recovery:${session.id}:${session.revision}`,
|
|
requestedByActorType: "system",
|
|
contextSnapshot: {
|
|
issueId: session.issueId,
|
|
taskKey: session.issueId,
|
|
resumeSessionGoalHeartbeat: true,
|
|
skipIssueComment: true,
|
|
},
|
|
});
|
|
if (run) enqueued += 1;
|
|
}
|
|
return { scanned: sessions.length, enqueued };
|
|
}
|
|
|
|
async function recoverPendingSessionGoalActions() {
|
|
if ((await getSchedulingSuppression()).suppressed) {
|
|
return { scanned: 0, enqueued: 0, alreadyQueued: 0, invalid: 0 };
|
|
}
|
|
const pending = await db
|
|
.select({
|
|
id: agentSessionGoalActions.id,
|
|
requestId: agentSessionGoalActions.requestId,
|
|
payload: agentSessionGoalActions.payloadJson,
|
|
companyId: agentTaskSessions.companyId,
|
|
agentId: agentTaskSessions.agentId,
|
|
adapterType: agentTaskSessions.adapterType,
|
|
issueId: agentTaskSessions.taskKey,
|
|
})
|
|
.from(agentSessionGoalActions)
|
|
.innerJoin(
|
|
agentTaskSessions,
|
|
eq(agentTaskSessions.id, agentSessionGoalActions.sessionId),
|
|
)
|
|
.innerJoin(
|
|
issues,
|
|
and(
|
|
sql`${issues.id}::text = ${agentTaskSessions.taskKey}`,
|
|
eq(issues.companyId, agentTaskSessions.companyId),
|
|
eq(issues.assigneeAgentId, agentTaskSessions.agentId),
|
|
),
|
|
)
|
|
.where(
|
|
inArray(agentSessionGoalActions.status, [
|
|
"pending",
|
|
"delivering",
|
|
"delivered",
|
|
]),
|
|
)
|
|
.orderBy(asc(agentSessionGoalActions.createdAt));
|
|
|
|
let enqueued = 0;
|
|
let alreadyQueued = 0;
|
|
let invalid = 0;
|
|
for (const action of pending) {
|
|
const control = parseNativeSessionGoalControl(action.payload);
|
|
if (!control || control.requestId !== action.requestId) {
|
|
invalid += 1;
|
|
await failRunnerGoalAction(
|
|
db,
|
|
{
|
|
companyId: action.companyId,
|
|
issueId: action.issueId,
|
|
agentId: action.agentId,
|
|
adapterType: action.adapterType,
|
|
},
|
|
action.requestId,
|
|
"session_goal_control_payload_invalid",
|
|
).catch(() => undefined);
|
|
continue;
|
|
}
|
|
|
|
const inFlight = await db
|
|
.select({ contextSnapshot: heartbeatRuns.contextSnapshot })
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, action.companyId),
|
|
eq(heartbeatRuns.agentId, action.agentId),
|
|
inArray(heartbeatRuns.status, [
|
|
"queued",
|
|
"scheduled_retry",
|
|
"running",
|
|
]),
|
|
),
|
|
)
|
|
.then((runs) =>
|
|
runs.some(
|
|
(run) =>
|
|
readNonEmptyString(
|
|
parseObject(run.contextSnapshot).goalControlRequestId,
|
|
) === action.requestId,
|
|
),
|
|
);
|
|
if (inFlight) {
|
|
alreadyQueued += 1;
|
|
continue;
|
|
}
|
|
|
|
const run = await enqueueWakeup(action.agentId, {
|
|
source: "automation",
|
|
triggerDetail: "system",
|
|
reason: "goal_control",
|
|
payload: {
|
|
issueId: action.issueId,
|
|
requestId: action.requestId,
|
|
intent: "goal_control_recovery",
|
|
},
|
|
idempotencyKey: `goal_control_recovery:${action.id}`,
|
|
requestedByActorType: "system",
|
|
contextSnapshot: {
|
|
issueId: action.issueId,
|
|
taskKey: action.issueId,
|
|
// Goal controls reconcile the provider session itself and remain
|
|
// valid after issue terminalization (for example, clearing a
|
|
// completed goal from its retained widget).
|
|
resumeIntent: true,
|
|
goalControlRequestId: action.requestId,
|
|
runnerGoalControl: control,
|
|
skipIssueComment: true,
|
|
},
|
|
});
|
|
if (run) enqueued += 1;
|
|
}
|
|
return { scanned: pending.length, enqueued, alreadyQueued, invalid };
|
|
}
|
|
|
|
async function eligibleTimerAgents() {
|
|
const cutoff = await getWorktreeExecutionCutoff();
|
|
|
|
const allAgents = await db
|
|
.select({ ...getTableColumns(agents) })
|
|
.from(agents)
|
|
.innerJoin(companies, eq(companies.id, agents.companyId))
|
|
.where(eq(companies.status, "active"));
|
|
const agentsByCompany = groupAgentOrgRowsByCompany(
|
|
allAgents.map(toAgentOrgRow),
|
|
);
|
|
const eligible: Array<{ agent: typeof agents.$inferSelect; intervalSec: number }> = [];
|
|
|
|
for (const agent of allAgents) {
|
|
const invokability = evaluateAgentInvokability(
|
|
toAgentOrgRow(agent),
|
|
agentsByCompany.get(agent.companyId) ?? [],
|
|
);
|
|
if (!invokability.invokable) continue;
|
|
if (!supportsUnscopedTimerHeartbeat(agent)) continue;
|
|
const policy = parseHeartbeatPolicy(agent);
|
|
if (!policy.enabled || policy.intervalSec <= 0) continue;
|
|
|
|
if (cutoff) {
|
|
const eligibleIssue = await db
|
|
.select({ id: issues.id })
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, agent.companyId),
|
|
eq(issues.assigneeAgentId, agent.id),
|
|
inArray(issues.status, ["todo", "in_progress"]),
|
|
gte(issues.createdAt, cutoff),
|
|
),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!eligibleIssue) continue;
|
|
}
|
|
|
|
eligible.push({ agent, intervalSec: policy.intervalSec });
|
|
}
|
|
return eligible;
|
|
}
|
|
|
|
async function nextAgentTimerAt(now = new Date()): Promise<number | null> {
|
|
if ((await getSchedulingSuppression(true)).suppressed) return null;
|
|
let next: number | null = null;
|
|
for (const { agent } of await eligibleTimerAgents()) {
|
|
const deadline = await nextTimerHeartbeatAt(agent, now);
|
|
if (deadline !== null) next = next === null ? deadline : Math.min(next, deadline);
|
|
}
|
|
return next;
|
|
}
|
|
|
|
async function tickAgentTimers(now = new Date(), canContinue: () => boolean = () => true) {
|
|
if (!canContinue() || (await getSchedulingSuppression(true)).suppressed) return { checked: 0, enqueued: 0, skipped: 0 };
|
|
let checked = 0, enqueued = 0, skipped = 0;
|
|
for (const { agent, intervalSec } of await eligibleTimerAgents()) {
|
|
if (!canContinue()) break;
|
|
checked++;
|
|
const deadline = await nextTimerHeartbeatAt(agent, now);
|
|
if (deadline === null || deadline > now.getTime()) continue;
|
|
if (!canContinue()) break;
|
|
const timerClaim = await claimDueTimerHeartbeat(agent, now, intervalSec);
|
|
if (!timerClaim) continue;
|
|
const run = await enqueueWakeup(agent.id, {
|
|
source: "timer", triggerDetail: "system", reason: "heartbeat_timer",
|
|
requestedByActorType: "system", requestedByActorId: "heartbeat_scheduler",
|
|
contextSnapshot: { source: "scheduler", reason: "interval_elapsed", now: now.toISOString(),
|
|
timerClaimWasFirstHeartbeat: timerClaim.wasFirstHeartbeat },
|
|
});
|
|
if (run) enqueued++; else skipped++;
|
|
}
|
|
return { checked, enqueued, skipped };
|
|
}
|
|
|
|
|
|
async function tickTimers(now = new Date()) {
|
|
const timers = await tickAgentTimers(now);
|
|
const monitors = await tickIssueMonitors(now);
|
|
return { checked: timers.checked + monitors.checked,
|
|
enqueued: timers.enqueued + monitors.triggered, skipped: timers.skipped + monitors.skipped };
|
|
}
|
|
|
|
return {
|
|
triggerIssueMonitor,
|
|
tickTimers,
|
|
tickAgentTimers,
|
|
nextAgentTimerAt,
|
|
tickIssueMonitors,
|
|
nextIssueMonitorAt,
|
|
recoverActiveSessionGoals,
|
|
recoverPendingSessionGoalActions,
|
|
};
|
|
}
|