mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 04:25:52 +01:00
## Thinking Path > - Paperclip manages work and human decisions for AI-agent companies. > - Automatic decision queues store membership for pending attention sources. > - The old attention maintenance scan created that membership as a side effect. > - Moving approval retention out of the scan also removed approval queue discovery. > - Queue API consumers do not build the attention feed first, so they can miss pending PR approvals. > - This pull request gives approval queue discovery its own worker and commit signals. > - The worker becomes quiet after it saves every missing membership. ## Linked Issues or Issue Description Refs #17497. Stacked on #17508, following #17505. This is a correctness dependency for the approval retention migration in #17497. Do not treat that migration as complete without this queue-population path. A real PostgreSQL regression found no PR queue when a pending approval had a linked PR work product and the maintenance feed excluded approvals. Related: #13364 proposes pruning stale seeded memberships. This PR preserves existing membership retention and does not implement pruning. #15149 documents the decision-queue APIs. ## What Changed - Add an approval queue worker that selects missing PR-queue memberships from existing durable records. Bound each pass to 100 rows. Continue past failed rows and revisit them on retry. - Reuse the queue service's seed writer through a normalized source contract. Keep queue rules, uniqueness, and audit writes in that service. Keep feed readers compatible. - Wake from committed approval and link changes, issue/company visibility, PR work-product changes and imports, and queue-rule changes. Reuse the existing transaction boundaries. - Recheck approval status under a row lock. Serialize queue-rule changes with the seed writer. Preserve required recovery during drain and shutdown. - Register startup reconstruction under the existing maintenance switch. Leave other attention-source seeding with its current owner until its own migration. - Document the complete producer audit and stack dependency. Add a validated external PostgreSQL option to the existing queue route suite so it runs on hosts without embedded database support. ## Verification Current tested revision: `7948136ef8`. This PR targets master after #17508 merged as `8406047b4e`. Normal ancestry reconciliation preserved the complete tested integration tree, before the review repairs below. Only the original 25 seeding paths differ from master, and database schema/migration metadata is identical. - **56 cases pass in the three complete affected suites:** 28 approval seeding, 6 queue routes and 22 task-watchdog cases. Server typecheck and production build pass. - Three real PostgreSQL retry regressions failed then passed: new matching arrivals cannot postpone a recovered failure indefinitely; successful partial progress resets old failure penalties; an empty cursor rollover does not reset outage backoff. Exact one- and two-second retry assertions, eventual complete recovery and existing capped-backoff/company isolation tests pass. - Three genuinely empty external databases failed before setup repair because tables were missing. Each suite now applies migrations itself. All 55 cases passed on a second set of fresh databases; adding the final rollover guard then passed all 56 cases using those now-migrated fixtures. No production migration or table is added. - The initial expanded fresh run passed 54 cases but failed a cleanup expectation that omitted a required cursor rollover pass. The fixture now permits that explicit continuation while preserving exact deadline and complete-membership assertions. All failure records remain in `doc/idle-approval-queue-seeding.md`. - The prior integrated revision `4e725e2055` passed **522 cases in fifteen complete producer/consumer suites**, plus types/build. Its unchanged consumer code and historical deadlock, cross-company amplification, backoff, gateway integration and setup evidence remain documented; those counts are not relabeled as current acceptance. Commands: server Vitest for approval queue seeding, queue routes and task watchdogs; server typecheck/build. Fresh exact-head Greptile/Ponytail reviews and all CI checks remain required. No full local repository-suite, deployed suspension or cost claim is made. ## Risks - Signals cover the supported in-process service writers. Independent SQL writers and multiple server replicas need a separate notification contract. - The source predicate must stay aligned with unscoped attention maintenance. The first linked issue, company scope, pending status, and execution visibility are tested. A pending approval itself keeps its linked external conversation active. - Existing enabled seed rules can re-add manually removed matching memberships. This preserves the old scan's behavior. Completed or archived membership is not pruned. - Queue seeding and retention are separate owners. Other attention sources still need their own event-driven discovery before the aggregate scan can be retired. - This PR does not extend Cloud wake coverage, relax sleep admission, or claim measured host/database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1145 lines
51 KiB
TypeScript
1145 lines
51 KiB
TypeScript
import { notifyApprovalWork } from "./approval-work-notifications.js";
|
|
import { notifyCompanyRetentionWork } from "./retention-work-notifications.js";
|
|
import { addCents, compareCents, normalizeCents } from "@paperclipai/shared";
|
|
import { and, desc, eq, inArray, sql, or, isNull } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
|
|
import {
|
|
agents,
|
|
approvals,
|
|
budgetIncidents,
|
|
budgetPolicies,
|
|
budgetReservations,
|
|
companies,
|
|
costEvents,
|
|
heartbeatRuns,
|
|
projects,
|
|
} from "@paperclipai/db";
|
|
import type {
|
|
BudgetIncident,
|
|
BudgetIncidentResolutionInput,
|
|
BudgetMetric,
|
|
BudgetOverview,
|
|
PauseReason,
|
|
BudgetPolicy,
|
|
BudgetPolicySummary,
|
|
BudgetPolicyUpsertInput,
|
|
BudgetScopeType,
|
|
BudgetThresholdType,
|
|
BudgetWindowKind,
|
|
} from "@paperclipai/shared";
|
|
import { HttpError, notFound, unprocessable } from "../errors.js";
|
|
import { logger } from "../middleware/logger.js";
|
|
import { withAccountingTransaction } from "./accounting-transaction.js";
|
|
import { logActivity, type LogActivityInput, type ActivityPublication } from "./activity-log.js";
|
|
import { computeObservedSpend, computeObservedSpends, type ObservedSpend } from "./budget-exposure.js";
|
|
import { notifyDeliveryWork, DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
|
import { listPendingBudgetCancellationScopes } from "./heartbeat/budget-cancellation.js";
|
|
import { withBudgetSpan } from "./budget-tracing.js";
|
|
export { computeObservedSpend, computeObservedSpends } from "./budget-exposure.js";
|
|
export type { BudgetExposure } from "./budget-exposure.js";
|
|
|
|
type ScopeRecord = {
|
|
companyId: string;
|
|
name: string;
|
|
paused: boolean;
|
|
pauseReason: PauseReason | null;
|
|
budgetHeld: boolean;
|
|
};
|
|
|
|
type PolicyRow = typeof budgetPolicies.$inferSelect;
|
|
type IncidentRow = typeof budgetIncidents.$inferSelect;
|
|
|
|
export type BudgetEnforcementScope = {
|
|
companyId: string;
|
|
scopeType: BudgetScopeType;
|
|
scopeId: string;
|
|
/** Snapshot taken under the company lock, before a later budget grant can admit new work. */
|
|
createdBefore?: Date;
|
|
/** Revalidated at the durable cancellation write, under the admission lock. */
|
|
enforcement?: { policyId: string; version: number };
|
|
};
|
|
|
|
export type BudgetServiceHooks = {
|
|
cancelWorkForScope?: (scope: BudgetEnforcementScope) => Promise<void>;
|
|
};
|
|
|
|
function currentUtcMonthWindow(now = new Date()) {
|
|
const year = now.getUTCFullYear();
|
|
const month = now.getUTCMonth();
|
|
const start = new Date(Date.UTC(year, month, 1, 0, 0, 0, 0));
|
|
const end = new Date(Date.UTC(year, month + 1, 1, 0, 0, 0, 0));
|
|
return { start, end };
|
|
}
|
|
|
|
function resolveWindow(windowKind: BudgetWindowKind, now = new Date()) {
|
|
if (windowKind === "lifetime") {
|
|
return {
|
|
start: new Date(Date.UTC(1970, 0, 1, 0, 0, 0, 0)),
|
|
end: new Date(Date.UTC(9999, 0, 1, 0, 0, 0, 0)),
|
|
};
|
|
}
|
|
return currentUtcMonthWindow(now);
|
|
}
|
|
|
|
function budgetStatusFromObserved(
|
|
observedAmount: string,
|
|
amount: number,
|
|
warnPercent: number,
|
|
): BudgetPolicySummary["status"] {
|
|
if (amount <= 0) return "ok";
|
|
if (compareCents(observedAmount, amount) >= 0) return "hard_stop";
|
|
if (compareCents(observedAmount, Math.ceil((amount * warnPercent) / 100)) >= 0) return "warning";
|
|
return "ok";
|
|
}
|
|
|
|
function normalizeScopeName(scopeType: BudgetScopeType, name: string) {
|
|
if (scopeType === "company") return name;
|
|
return name.trim().length > 0 ? name : scopeType;
|
|
}
|
|
|
|
async function resolveScopeRecord(db: Db, scopeType: BudgetScopeType, scopeId: string): Promise<ScopeRecord> {
|
|
if (scopeType === "company") {
|
|
const row = await db
|
|
.select({
|
|
companyId: companies.id,
|
|
name: companies.name,
|
|
status: companies.status,
|
|
pauseReason: companies.pauseReason,
|
|
pausedAt: companies.pausedAt,
|
|
})
|
|
.from(companies)
|
|
.where(eq(companies.id, scopeId))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!row) throw notFound("Company not found");
|
|
return {
|
|
companyId: row.companyId,
|
|
name: row.name,
|
|
paused: row.status === "paused" || Boolean(row.pausedAt),
|
|
pauseReason: (row.pauseReason as ScopeRecord["pauseReason"]) ?? null,
|
|
budgetHeld: row.pauseReason === "budget",
|
|
};
|
|
}
|
|
|
|
if (scopeType === "agent") {
|
|
const row = await db
|
|
.select({
|
|
companyId: agents.companyId,
|
|
name: agents.name,
|
|
status: agents.status,
|
|
pauseReason: agents.pauseReason,
|
|
lifecycleHolds: agents.lifecycleHolds,
|
|
})
|
|
.from(agents)
|
|
.where(eq(agents.id, scopeId))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!row) throw notFound("Agent not found");
|
|
return {
|
|
companyId: row.companyId,
|
|
name: row.name,
|
|
paused: row.status === "paused",
|
|
pauseReason: (row.pauseReason as ScopeRecord["pauseReason"]) ?? null,
|
|
budgetHeld: row.lifecycleHolds.includes("budget"),
|
|
};
|
|
}
|
|
|
|
const row = await db
|
|
.select({
|
|
companyId: projects.companyId,
|
|
name: projects.name,
|
|
pauseReason: projects.pauseReason,
|
|
pausedAt: projects.pausedAt,
|
|
})
|
|
.from(projects)
|
|
.where(eq(projects.id, scopeId))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!row) throw notFound("Project not found");
|
|
return {
|
|
companyId: row.companyId,
|
|
name: row.name,
|
|
paused: Boolean(row.pausedAt),
|
|
pauseReason: (row.pauseReason as ScopeRecord["pauseReason"]) ?? null,
|
|
budgetHeld: row.pauseReason === "budget",
|
|
};
|
|
}
|
|
|
|
// Keep a large dashboard from filling the database pool with queued reads.
|
|
async function mapBudgetReads<T, R>(rows: T[], read: (row: T) => Promise<R>): Promise<R[]> {
|
|
const result: R[] = [];
|
|
for (let start = 0; start < rows.length; start += 4) {
|
|
result.push(...await Promise.all(rows.slice(start, start + 4).map(read)));
|
|
}
|
|
return result;
|
|
}
|
|
|
|
/** Filter in PostgreSQL so one run never loads unrelated agents' policies. */
|
|
export function budgetPoliciesForRun(companyId: string, agentId: string | null, projectId: string | null) {
|
|
return or(
|
|
and(eq(budgetPolicies.scopeType, "company"), eq(budgetPolicies.scopeId, companyId)),
|
|
agentId ? and(eq(budgetPolicies.scopeType, "agent"), eq(budgetPolicies.scopeId, agentId)) : undefined,
|
|
projectId ? and(eq(budgetPolicies.scopeType, "project"), eq(budgetPolicies.scopeId, projectId)) : undefined,
|
|
);
|
|
}
|
|
|
|
function unpricedUsageBlocks(policy: PolicyRow, observed: ObservedSpend) {
|
|
return policy.unpricedUsagePolicy !== "allow"
|
|
&& (observed.unpricedEventCount > 0 || observed.unknownBrowserRunCount > 0);
|
|
}
|
|
|
|
function observedBlocks(policy: PolicyRow, observed: Awaited<ReturnType<typeof computeObservedSpend>>) {
|
|
if (!policy.isActive || !policy.hardStopEnabled || policy.amount <= 0) return false;
|
|
return compareCents(observed.totalExact, policy.amount) >= 0 || (observed.pendingRunCount - observed.recoveringRunCount) > 0 || unpricedUsageBlocks(policy, observed);
|
|
}
|
|
export async function policyBlocks(db: Db, policy: PolicyRow) {
|
|
if (!policy.isActive || !policy.hardStopEnabled || policy.amount <= 0) return false;
|
|
return observedBlocks(policy, await computeObservedSpend(db, policy));
|
|
}
|
|
|
|
/** Supplemental handoff check after authoritative accounting admission commits.
|
|
* Read only: the caller can hold issue/run ownership, so this must never take
|
|
* the company lock, reconcile holds, or deliver effects. A later concurrent
|
|
* charge targets already-admitted work through the normal durable Stop path. */
|
|
export async function recheckAdmittedRunBudgetReadOnly(db: Db, input: {
|
|
companyId: string; agentId: string; runId: string; projectId: string | null; reservationId: string;
|
|
}): Promise<string | null> {
|
|
return withBudgetSpan("budget.admission.evaluate", async () => {
|
|
const [reservation] = await db.select({ amountCents: budgetReservations.amountCents }).from(budgetReservations).where(and(
|
|
eq(budgetReservations.id, input.reservationId), eq(budgetReservations.companyId, input.companyId),
|
|
eq(budgetReservations.runId, input.runId), eq(budgetReservations.agentId, input.agentId),
|
|
sql`${budgetReservations.projectId} is not distinct from ${input.projectId}::uuid`,
|
|
eq(budgetReservations.state, "held"),
|
|
));
|
|
if (!reservation) return "Provider dispatch no longer owns a held budget reservation";
|
|
const policies = await db.select().from(budgetPolicies).where(and(eq(budgetPolicies.companyId, input.companyId),
|
|
eq(budgetPolicies.isActive, true), eq(budgetPolicies.hardStopEnabled, true), eq(budgetPolicies.metric, "billed_cents"),
|
|
sql`${budgetPolicies.amount} > 0`, budgetPoliciesForRun(input.companyId, input.agentId, input.projectId)));
|
|
const observations = await computeObservedSpends(db, policies);
|
|
for (const [index, policy] of policies.entries()) {
|
|
const observed = observations[index];
|
|
if (observedBlocks(policy, observed)) return "Applicable budget no longer permits provider work";
|
|
const capacity = compareCents(addCents(observed.ledgerCents, observed.effectiveHeldCents), policy.amount);
|
|
if (capacity > 0 || (compareCents(reservation.amountCents, 0) === 0 && capacity >= 0))
|
|
return "Available budget is reserved by unfinished runs";
|
|
}
|
|
return null;
|
|
});
|
|
}
|
|
|
|
function buildApprovalPayload(input: {
|
|
policy: PolicyRow;
|
|
scopeName: string;
|
|
thresholdType: BudgetThresholdType;
|
|
amountObserved: number;
|
|
amountObservedExact: string;
|
|
windowStart: Date;
|
|
windowEnd: Date;
|
|
}) {
|
|
return {
|
|
scopeType: input.policy.scopeType,
|
|
scopeId: input.policy.scopeId,
|
|
scopeName: input.scopeName,
|
|
metric: input.policy.metric,
|
|
windowKind: input.policy.windowKind,
|
|
thresholdType: input.thresholdType,
|
|
budgetAmount: input.policy.amount,
|
|
observedAmount: input.amountObserved,
|
|
observedAmountExact: input.amountObservedExact,
|
|
warnPercent: input.policy.warnPercent,
|
|
windowStart: input.windowStart.toISOString(),
|
|
windowEnd: input.windowEnd.toISOString(),
|
|
policyId: input.policy.id,
|
|
guidance: "Resolve pending or unpriced accounting, raise the budget, or keep the scope paused.",
|
|
};
|
|
}
|
|
|
|
async function markApprovalStatus(
|
|
db: Db,
|
|
approvalId: string | null,
|
|
status: "approved" | "rejected",
|
|
decisionNote: string | null | undefined,
|
|
decidedByUserId: string,
|
|
) {
|
|
if (!approvalId) return;
|
|
const [approval] = await db.select({ companyId: approvals.companyId }).from(approvals).where(eq(approvals.id, approvalId));
|
|
if (!approval) return;
|
|
await notifyApprovalWork(db, approval.companyId);
|
|
await db
|
|
.update(approvals)
|
|
.set({
|
|
status,
|
|
decisionNote: decisionNote ?? null,
|
|
decidedByUserId,
|
|
decidedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(approvals.id, approvalId));
|
|
}
|
|
|
|
/** Internal: caller must hold the company accounting transaction lock. */
|
|
export function budgetServiceInTransaction(db: Db, publications: ActivityPublication[] = []) {
|
|
const recordActivity = (input: LogActivityInput) => logActivity(db, input, publications);
|
|
async function pauseScopeForBudget(policy: PolicyRow) {
|
|
const now = new Date();
|
|
if (policy.scopeType === "agent") {
|
|
const { applyAgentBudgetHoldInTransaction } = await import("./agent-lifecycle.js");
|
|
await applyAgentBudgetHoldInTransaction(db, policy.companyId, policy.scopeId, true);
|
|
return;
|
|
}
|
|
|
|
if (policy.scopeType === "project") {
|
|
await db
|
|
.update(projects)
|
|
.set({
|
|
pauseReason: "budget",
|
|
pausedAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(and(eq(projects.id, policy.scopeId), isNull(projects.pausedAt)));
|
|
return;
|
|
}
|
|
|
|
const scope = await resolveScopeRecord(db, "company", policy.scopeId);
|
|
if (scope.paused) return;
|
|
await notifyCompanyRetentionWork(db, policy.companyId);
|
|
await notifyDeliveryWork(db, DELIVERY_QUEUES.agentLifecycle, policy.companyId);
|
|
await notifyDeliveryWork(db, DELIVERY_QUEUES.skillSourceSync);
|
|
await db
|
|
.update(companies)
|
|
.set({
|
|
status: "paused",
|
|
pauseReason: "budget",
|
|
pausedAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(and(eq(companies.id, policy.scopeId), eq(companies.status, "active")));
|
|
}
|
|
|
|
async function pauseAndCancelScopeForBudget(policy: PolicyRow, requestCancellation = true) {
|
|
await pauseScopeForBudget(policy);
|
|
if (!requestCancellation) return;
|
|
// A pending generation already covers all admitted work in this scope.
|
|
// Rechecks must not invalidate a worker that is delivering that intent.
|
|
await notifyDeliveryWork(db, DELIVERY_QUEUES.budgetEnforcement, policy.companyId);
|
|
await db.update(budgetPolicies).set({
|
|
enforcementVersion: sql`case when ${budgetPolicies.enforcementVersion} > ${budgetPolicies.enforcementDeliveredVersion}
|
|
then ${budgetPolicies.enforcementVersion} else ${budgetPolicies.enforcementVersion} + 1 end`,
|
|
updatedAt: sql`case when ${budgetPolicies.enforcementVersion} > ${budgetPolicies.enforcementDeliveredVersion}
|
|
then ${budgetPolicies.updatedAt} else now() end`,
|
|
}).where(eq(budgetPolicies.id, policy.id));
|
|
}
|
|
|
|
async function resumeScopeFromBudget(policy: PolicyRow) {
|
|
const policies = await db.select().from(budgetPolicies).where(and(
|
|
eq(budgetPolicies.companyId, policy.companyId), eq(budgetPolicies.scopeType, policy.scopeType),
|
|
eq(budgetPolicies.scopeId, policy.scopeId), eq(budgetPolicies.isActive, true), eq(budgetPolicies.hardStopEnabled, true),
|
|
));
|
|
const observations = await computeObservedSpends(db, policies);
|
|
if (policies.some((candidate, index) => observedBlocks(candidate, observations[index]))) return;
|
|
const now = new Date();
|
|
if (policy.scopeType === "agent") {
|
|
const { applyAgentBudgetHoldInTransaction } = await import("./agent-lifecycle.js");
|
|
await applyAgentBudgetHoldInTransaction(db, policy.companyId, policy.scopeId, false);
|
|
return;
|
|
}
|
|
|
|
if (policy.scopeType === "project") {
|
|
await db
|
|
.update(projects)
|
|
.set({
|
|
pauseReason: null,
|
|
pausedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(and(eq(projects.id, policy.scopeId), eq(projects.pauseReason, "budget")));
|
|
return;
|
|
}
|
|
|
|
await notifyDeliveryWork(db, DELIVERY_QUEUES.skillSourceSync);
|
|
const scope = await resolveScopeRecord(db, "company", policy.scopeId);
|
|
if (scope.pauseReason !== "budget") return;
|
|
await notifyCompanyRetentionWork(db, policy.companyId);
|
|
await notifyDeliveryWork(db, DELIVERY_QUEUES.agentLifecycle, policy.companyId);
|
|
await db
|
|
.update(companies)
|
|
.set({
|
|
status: "active",
|
|
pauseReason: null,
|
|
pausedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(and(eq(companies.id, policy.scopeId), eq(companies.status, "paused"), eq(companies.pauseReason, "budget")));
|
|
}
|
|
|
|
async function getPolicyRow(policyId: string) {
|
|
const policy = await db
|
|
.select()
|
|
.from(budgetPolicies)
|
|
.where(eq(budgetPolicies.id, policyId))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!policy) throw notFound("Budget policy not found");
|
|
return policy;
|
|
}
|
|
|
|
async function listPolicyRows(companyId: string) {
|
|
return db
|
|
.select()
|
|
.from(budgetPolicies)
|
|
.where(eq(budgetPolicies.companyId, companyId))
|
|
.orderBy(desc(budgetPolicies.updatedAt));
|
|
}
|
|
|
|
async function buildPolicySummary(policy: PolicyRow): Promise<BudgetPolicySummary> {
|
|
const scope = await resolveScopeRecord(db, policy.scopeType as BudgetScopeType, policy.scopeId);
|
|
const observed = await computeObservedSpend(db, policy);
|
|
const { total: observedAmount, totalExact: observedAmountExact, unpricedEventCount, pendingRunCount } = observed;
|
|
const { windowStart: start, windowEnd: end } = observed;
|
|
const amount = policy.isActive ? policy.amount : 0;
|
|
const utilizationPercent =
|
|
amount > 0 ? Number(((observedAmount / amount) * 100).toFixed(2)) : 0;
|
|
return {
|
|
policyId: policy.id,
|
|
companyId: policy.companyId,
|
|
scopeType: policy.scopeType as BudgetScopeType,
|
|
scopeId: policy.scopeId,
|
|
scopeName: normalizeScopeName(policy.scopeType as BudgetScopeType, scope.name),
|
|
metric: policy.metric as BudgetMetric,
|
|
windowKind: policy.windowKind as BudgetWindowKind,
|
|
amount,
|
|
observedAmount,
|
|
observedAmountExact,
|
|
reservationCents: policy.reservationCents,
|
|
unpricedEventCount,
|
|
pendingRunCount,
|
|
unpricedUsagePolicy: policy.unpricedUsagePolicy as "block" | "allow",
|
|
remainingAmount: amount > 0 ? Math.max(0, amount - observedAmount) : 0,
|
|
utilizationPercent,
|
|
warnPercent: policy.warnPercent,
|
|
hardStopEnabled: policy.hardStopEnabled,
|
|
notifyEnabled: policy.notifyEnabled,
|
|
isActive: policy.isActive,
|
|
status: observedBlocks(policy, observed) ? "hard_stop" : policy.isActive
|
|
? budgetStatusFromObserved(observedAmountExact, amount, policy.warnPercent)
|
|
: "ok",
|
|
paused: scope.paused,
|
|
pauseReason: scope.pauseReason,
|
|
windowStart: start,
|
|
windowEnd: end,
|
|
};
|
|
}
|
|
|
|
async function createIncidentIfNeeded(
|
|
policy: PolicyRow,
|
|
thresholdType: BudgetThresholdType,
|
|
observed: Awaited<ReturnType<typeof computeObservedSpend>>,
|
|
) {
|
|
const { windowStart: start, windowEnd: end } = observed;
|
|
const existing = await db
|
|
.select()
|
|
.from(budgetIncidents)
|
|
.where(
|
|
and(
|
|
eq(budgetIncidents.policyId, policy.id),
|
|
eq(budgetIncidents.windowStart, start),
|
|
eq(budgetIncidents.thresholdType, thresholdType),
|
|
or(eq(budgetIncidents.status, "open"), and(eq(budgetIncidents.status, "dismissed"), eq(budgetIncidents.amountLimit, policy.amount))),
|
|
),
|
|
)
|
|
.orderBy(desc(budgetIncidents.createdAt))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (existing) return { incident: existing, created: false };
|
|
|
|
const { total: amountObserved, totalExact: amountObservedExact } = observed;
|
|
const scope = await resolveScopeRecord(db, policy.scopeType as BudgetScopeType, policy.scopeId);
|
|
const payload = buildApprovalPayload({
|
|
policy,
|
|
scopeName: normalizeScopeName(policy.scopeType as BudgetScopeType, scope.name),
|
|
thresholdType,
|
|
amountObserved,
|
|
amountObservedExact,
|
|
windowStart: start,
|
|
windowEnd: end,
|
|
});
|
|
|
|
if (thresholdType === "hard") await notifyApprovalWork(db, policy.companyId);
|
|
const approval = thresholdType === "hard"
|
|
? await db
|
|
.insert(approvals)
|
|
.values({
|
|
companyId: policy.companyId,
|
|
type: "budget_override_required",
|
|
requestedByUserId: null,
|
|
requestedByAgentId: null,
|
|
status: "pending",
|
|
payload,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null)
|
|
: null;
|
|
|
|
const incident = await db
|
|
.insert(budgetIncidents)
|
|
.values({
|
|
companyId: policy.companyId,
|
|
policyId: policy.id,
|
|
scopeType: policy.scopeType,
|
|
scopeId: policy.scopeId,
|
|
metric: policy.metric,
|
|
windowKind: policy.windowKind,
|
|
windowStart: start,
|
|
windowEnd: end,
|
|
thresholdType,
|
|
amountLimit: policy.amount,
|
|
amountObserved: sql`${amountObservedExact}::numeric`,
|
|
status: "open",
|
|
approvalId: approval?.id ?? null,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0] ?? null);
|
|
return incident ? { incident, created: true } : null;
|
|
}
|
|
|
|
async function resolveOpenSoftIncidents(policyId: string) {
|
|
await db
|
|
.update(budgetIncidents)
|
|
.set({
|
|
status: "resolved",
|
|
resolvedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
and(
|
|
eq(budgetIncidents.policyId, policyId),
|
|
eq(budgetIncidents.thresholdType, "soft"),
|
|
eq(budgetIncidents.status, "open"),
|
|
),
|
|
);
|
|
}
|
|
|
|
async function resolveOpenIncidentsForPolicy(
|
|
policyId: string,
|
|
approvalStatus: "approved" | "rejected" | null,
|
|
decidedByUserId: string | null,
|
|
thresholdType?: BudgetThresholdType,
|
|
) {
|
|
const condition = and(eq(budgetIncidents.policyId, policyId), eq(budgetIncidents.status, "open"),
|
|
thresholdType ? eq(budgetIncidents.thresholdType, thresholdType) : undefined);
|
|
const openRows = await db
|
|
.select()
|
|
.from(budgetIncidents)
|
|
.where(condition);
|
|
|
|
await db
|
|
.update(budgetIncidents)
|
|
.set({
|
|
status: "resolved",
|
|
resolvedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(condition);
|
|
|
|
if (!approvalStatus || !decidedByUserId) return;
|
|
for (const row of openRows) {
|
|
await markApprovalStatus(db, row.approvalId ?? null, approvalStatus, "Resolved via budget update", decidedByUserId);
|
|
}
|
|
}
|
|
|
|
async function hydrateIncidentRows(rows: IncidentRow[]): Promise<BudgetIncident[]> {
|
|
const approvalIds = rows.map((row) => row.approvalId).filter((value): value is string => Boolean(value));
|
|
const approvalRows = approvalIds.length > 0
|
|
? await db
|
|
.select({ id: approvals.id, status: approvals.status })
|
|
.from(approvals)
|
|
.where(inArray(approvals.id, approvalIds))
|
|
: [];
|
|
const approvalStatusById = new Map(approvalRows.map((row) => [row.id, row.status]));
|
|
|
|
return mapBudgetReads(rows, async (row) => {
|
|
const scope = await resolveScopeRecord(db, row.scopeType as BudgetScopeType, row.scopeId);
|
|
return {
|
|
id: row.id,
|
|
companyId: row.companyId,
|
|
policyId: row.policyId,
|
|
scopeType: row.scopeType as BudgetScopeType,
|
|
scopeId: row.scopeId,
|
|
scopeName: normalizeScopeName(row.scopeType as BudgetScopeType, scope.name),
|
|
metric: row.metric as BudgetMetric,
|
|
windowKind: row.windowKind as BudgetWindowKind,
|
|
windowStart: row.windowStart,
|
|
windowEnd: row.windowEnd,
|
|
thresholdType: row.thresholdType as BudgetThresholdType,
|
|
amountLimit: row.amountLimit,
|
|
amountObserved: row.amountObserved,
|
|
status: row.status as BudgetIncident["status"],
|
|
approvalId: row.approvalId ?? null,
|
|
approvalStatus: row.approvalId ? approvalStatusById.get(row.approvalId) ?? null : null,
|
|
resolvedAt: row.resolvedAt ?? null,
|
|
createdAt: row.createdAt,
|
|
updatedAt: row.updatedAt,
|
|
};
|
|
});
|
|
}
|
|
|
|
async function reconcileScope(companyId: string, scopeType: BudgetScopeType, scopeId: string,
|
|
prepared?: { scope: ScopeRecord; policies: PolicyRow[]; observations: ObservedSpend[] }) {
|
|
const scope = prepared?.scope ?? await resolveScopeRecord(db, scopeType, scopeId);
|
|
if (scope.companyId !== companyId) throw notFound("Budget scope not found");
|
|
const policies = prepared?.policies ?? await db.select().from(budgetPolicies).where(and(
|
|
eq(budgetPolicies.companyId, companyId), eq(budgetPolicies.scopeType, scopeType), eq(budgetPolicies.scopeId, scopeId),
|
|
));
|
|
let recoveringPolicy: PolicyRow | undefined;
|
|
const observations = prepared?.observations ?? await computeObservedSpends(db, policies);
|
|
for (const [index, policy] of policies.entries()) {
|
|
const observed = observations[index];
|
|
if (observedBlocks(policy, observed)) {
|
|
await createIncidentIfNeeded(policy, "hard", observed);
|
|
await pauseAndCancelScopeForBudget(policy, !scope.budgetHeld);
|
|
const title = scopeType === "company" ? "Company" : scopeType === "agent" ? "Agent" : "Project";
|
|
if (observed.pendingRunCount > 0) {
|
|
return { scopeType, scopeId, scopeName: scope.name, reason: `${title} cannot start work while completed runs await accounting.` };
|
|
}
|
|
if (unpricedUsageBlocks(policy, observed)) {
|
|
return { scopeType, scopeId, scopeName: scope.name, reason: `${title} cannot start work because recorded usage has no reliable price.` };
|
|
}
|
|
return { scopeType, scopeId, scopeName: scope.name, reason: scopeType === "project"
|
|
? "Project cannot start work because its budget hard-stop is still exceeded."
|
|
: `${title} is paused because its budget hard-stop was reached.` };
|
|
}
|
|
// An open native coordinator must finish the original run's accounting.
|
|
// Deny fresh work without pausing/cancelling that recovery or its hold.
|
|
if (policy.isActive && policy.hardStopEnabled && policy.amount > 0 && observed.recoveringRunCount > 0) recoveringPolicy ??= policy;
|
|
// A closed UTC window, inactive policy, or budget raise releases only
|
|
// budget-owned pauses. Manual pauses remain operator decisions.
|
|
await resolveOpenIncidentsForPolicy(policy.id, "approved", "budget_service", "hard");
|
|
if (!policy.isActive || !policy.notifyEnabled || compareCents(observed.totalExact, Math.ceil(policy.amount * policy.warnPercent / 100)) < 0) {
|
|
await resolveOpenIncidentsForPolicy(policy.id, "approved", "budget_service", "soft");
|
|
}
|
|
}
|
|
// Agent status has its own admission gate and error code. Preserve a
|
|
// manual pause without misclassifying it as a budget failure.
|
|
if (scopeType !== "agent" && scope.paused && scope.pauseReason !== "budget") {
|
|
return { scopeType, scopeId, scopeName: scope.name, reason: `${scopeType === "company" ? "Company" : "Project"} is paused and cannot start new work.` };
|
|
}
|
|
if (scope.paused && scope.pauseReason === "budget" && !policies.length) {
|
|
return { scopeType, scopeId, scopeName: scope.name, reason: "Budget pause requires a policy or an explicit operator resume." };
|
|
}
|
|
if (scope.budgetHeld && policies[0]) await resumeScopeFromBudget(policies[0]);
|
|
if (recoveringPolicy) return { scopeType, scopeId, scopeName: scope.name,
|
|
reason: "New work must wait while a native run recovers its unfinished accounting." };
|
|
return null;
|
|
}
|
|
|
|
async function evaluateAffectedScopes(companyId: string, agentId: string | null, projectId: string | null) {
|
|
const candidatePolicies = await db
|
|
.select()
|
|
.from(budgetPolicies)
|
|
.where(
|
|
and(
|
|
eq(budgetPolicies.companyId, companyId),
|
|
eq(budgetPolicies.isActive, true),
|
|
budgetPoliciesForRun(companyId, agentId, projectId),
|
|
),
|
|
);
|
|
|
|
const policies = candidatePolicies.filter(policy => policy.metric === "billed_cents" && policy.amount > 0);
|
|
const observations = await computeObservedSpends(db, policies);
|
|
for (const [index, policy] of policies.entries()) {
|
|
const observed = observations[index];
|
|
const observedAmount = observed.total;
|
|
const softThreshold = Math.ceil((policy.amount * policy.warnPercent) / 100);
|
|
|
|
if (policy.notifyEnabled && compareCents(observed.totalExact, softThreshold) >= 0 && (!policy.hardStopEnabled || compareCents(observed.totalExact, policy.amount) < 0)) {
|
|
const softIncident = await createIncidentIfNeeded(policy, "soft", observed);
|
|
if (softIncident?.created) {
|
|
await recordActivity({
|
|
companyId: policy.companyId,
|
|
actorType: "system",
|
|
actorId: "budget_service",
|
|
action: "budget.soft_threshold_crossed",
|
|
entityType: "budget_incident",
|
|
entityId: softIncident.incident.id,
|
|
details: {
|
|
scopeType: policy.scopeType,
|
|
scopeId: policy.scopeId,
|
|
amountObserved: observedAmount,
|
|
amountLimit: policy.amount,
|
|
},
|
|
});
|
|
}
|
|
}
|
|
|
|
if (observedBlocks(policy, observed)) {
|
|
await resolveOpenSoftIncidents(policy.id);
|
|
const hardIncident = await createIncidentIfNeeded(policy, "hard", observed);
|
|
await pauseAndCancelScopeForBudget(policy);
|
|
if (hardIncident?.created) {
|
|
await recordActivity({
|
|
companyId: policy.companyId,
|
|
actorType: "system",
|
|
actorId: "budget_service",
|
|
action: "budget.hard_threshold_crossed",
|
|
entityType: "budget_incident",
|
|
entityId: hardIncident.incident.id,
|
|
details: {
|
|
scopeType: policy.scopeType,
|
|
scopeId: policy.scopeId,
|
|
amountObserved: observedAmount,
|
|
amountLimit: policy.amount,
|
|
approvalId: hardIncident.incident.approvalId ?? null,
|
|
},
|
|
});
|
|
}
|
|
}
|
|
}
|
|
// A final receipt or price correction can reduce a partial observation.
|
|
// Release only this scope's budget-owned hold after all its policies agree.
|
|
const healthyScopes = new Map<string, PolicyRow>();
|
|
for (const [index, policy] of policies.entries()) {
|
|
if (!observedBlocks(policy, observations[index])) healthyScopes.set(`${policy.scopeType}:${policy.scopeId}`, policy);
|
|
}
|
|
for (const policy of healthyScopes.values()) {
|
|
const scope = await resolveScopeRecord(db, policy.scopeType as BudgetScopeType, policy.scopeId);
|
|
if (scope.budgetHeld) await resumeScopeFromBudget(policy);
|
|
}
|
|
}
|
|
|
|
async function evaluateAdmissionInTransaction(companyId: string, agentId: string | null,
|
|
context?: { issueId?: string | null; projectId?: string | null }) {
|
|
const scopes: Array<{ scopeType: BudgetScopeType; scopeId: string }> = [
|
|
{ scopeType: "company", scopeId: companyId }, ...(agentId ? [{ scopeType: "agent" as const, scopeId: agentId }] : []),
|
|
...(context?.projectId ? [{ scopeType: "project" as const, scopeId: context.projectId }] : []),
|
|
];
|
|
const records = await Promise.all(scopes.map(scope => resolveScopeRecord(db, scope.scopeType, scope.scopeId)));
|
|
if (records.some(scope => scope.companyId !== companyId)) throw notFound("Budget scope not found");
|
|
const policies = await db.select().from(budgetPolicies).where(and(eq(budgetPolicies.companyId, companyId),
|
|
budgetPoliciesForRun(companyId, agentId, context?.projectId ?? null)));
|
|
const observations = await computeObservedSpends(db, policies);
|
|
const reservationIndexes = policies.map((policy, i) => policy.isActive && policy.hardStopEnabled && policy.metric === "billed_cents" && policy.amount > 0 ? i : -1).filter(i => i >= 0);
|
|
const admission = (block: Awaited<ReturnType<typeof reconcileScope>>) => ({ block,
|
|
policies: reservationIndexes.map(i => policies[i]), observations: reservationIndexes.map(i => observations[i]) });
|
|
for (const [index, { scopeType, scopeId }] of scopes.entries()) {
|
|
const indexes = policies.map((policy, i) => policy.scopeType === scopeType && policy.scopeId === scopeId ? i : -1).filter(i => i >= 0);
|
|
const block = await reconcileScope(companyId, scopeType, scopeId, { scope: records[index],
|
|
policies: indexes.map(i => policies[i]), observations: indexes.map(i => observations[i]) });
|
|
if (block) return admission(block);
|
|
}
|
|
return admission(null);
|
|
}
|
|
|
|
const evaluateAdmission = (...args: Parameters<typeof evaluateAdmissionInTransaction>) =>
|
|
withBudgetSpan("budget.admission.evaluate", () => evaluateAdmissionInTransaction(...args));
|
|
|
|
return {
|
|
reconcileScope,
|
|
listPolicies: async (companyId: string): Promise<BudgetPolicy[]> => {
|
|
const rows = await listPolicyRows(companyId);
|
|
return rows.map((row) => ({
|
|
...row,
|
|
scopeType: row.scopeType as BudgetScopeType,
|
|
unpricedUsagePolicy: row.unpricedUsagePolicy as "block" | "allow",
|
|
metric: row.metric as BudgetMetric,
|
|
windowKind: row.windowKind as BudgetWindowKind,
|
|
}));
|
|
},
|
|
|
|
upsertPolicy: async (
|
|
companyId: string,
|
|
input: BudgetPolicyUpsertInput,
|
|
actorUserId: string | null,
|
|
): Promise<BudgetPolicySummary> => {
|
|
const scope = await resolveScopeRecord(db, input.scopeType, input.scopeId);
|
|
if (scope.companyId !== companyId) {
|
|
throw unprocessable("Budget scope does not belong to company");
|
|
}
|
|
|
|
if (input.reservationCents !== undefined && compareCents(input.reservationCents, 0) < 0) throw unprocessable("Reservation cannot be negative");
|
|
const metric = input.metric ?? "billed_cents";
|
|
const windowKind = input.windowKind ?? (input.scopeType === "project" ? "lifetime" : "calendar_month_utc");
|
|
const existing = await db
|
|
.select()
|
|
.from(budgetPolicies)
|
|
.where(
|
|
and(
|
|
eq(budgetPolicies.companyId, companyId),
|
|
eq(budgetPolicies.scopeType, input.scopeType),
|
|
eq(budgetPolicies.scopeId, input.scopeId),
|
|
eq(budgetPolicies.metric, metric),
|
|
eq(budgetPolicies.windowKind, windowKind),
|
|
),
|
|
)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!existing && input.amount === undefined) throw unprocessable("Amount is required for a new budget policy");
|
|
const amount = Math.max(0, Math.floor(input.amount ?? existing!.amount));
|
|
// A saved zero disables enforcement. Raising it is an explicit limit
|
|
// setting; editing a disabled positive policy must preserve that choice.
|
|
const nextIsActive = amount > 0 && (input.isActive ?? (existing?.amount === 0 ? true : existing?.isActive) ?? true);
|
|
const now = new Date();
|
|
await notifyDeliveryWork(db, DELIVERY_QUEUES.budgetEnforcement, companyId);
|
|
const row = existing
|
|
? await db
|
|
.update(budgetPolicies)
|
|
.set({
|
|
amount,
|
|
reservationCents: normalizeCents(input.reservationCents ?? existing.reservationCents ?? 0),
|
|
warnPercent: input.warnPercent ?? existing.warnPercent,
|
|
hardStopEnabled: input.hardStopEnabled ?? existing.hardStopEnabled,
|
|
notifyEnabled: input.notifyEnabled ?? existing.notifyEnabled,
|
|
unpricedUsagePolicy: input.unpricedUsagePolicy ?? existing.unpricedUsagePolicy,
|
|
isActive: nextIsActive,
|
|
updatedByUserId: actorUserId,
|
|
updatedAt: now,
|
|
enforcementVersion: sql`${budgetPolicies.enforcementVersion} + 1`,
|
|
})
|
|
.where(eq(budgetPolicies.id, existing.id))
|
|
.returning()
|
|
.then((rows) => rows[0])
|
|
: await db
|
|
.insert(budgetPolicies)
|
|
.values({
|
|
companyId,
|
|
scopeType: input.scopeType,
|
|
scopeId: input.scopeId,
|
|
metric,
|
|
windowKind,
|
|
amount,
|
|
reservationCents: normalizeCents(input.reservationCents ?? 0),
|
|
warnPercent: input.warnPercent ?? 80,
|
|
hardStopEnabled: input.hardStopEnabled ?? true,
|
|
notifyEnabled: input.notifyEnabled ?? true,
|
|
unpricedUsagePolicy: input.unpricedUsagePolicy ?? "block",
|
|
isActive: nextIsActive,
|
|
createdByUserId: actorUserId,
|
|
updatedByUserId: actorUserId,
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]);
|
|
|
|
if (input.scopeType === "company" && windowKind === "calendar_month_utc") {
|
|
await db
|
|
.update(companies)
|
|
.set({
|
|
budgetMonthlyCents: amount,
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(companies.id, input.scopeId));
|
|
}
|
|
|
|
if (input.scopeType === "agent" && windowKind === "calendar_month_utc") {
|
|
await db
|
|
.update(agents)
|
|
.set({
|
|
budgetMonthlyCents: amount,
|
|
updatedAt: now,
|
|
})
|
|
.where(eq(agents.id, input.scopeId));
|
|
}
|
|
|
|
if (row.isActive && amount > 0) {
|
|
const observed = await computeObservedSpend(db, row);
|
|
if (!observedBlocks(row, observed)) {
|
|
await resumeScopeFromBudget(row);
|
|
await resolveOpenIncidentsForPolicy(row.id, "approved", actorUserId ?? "budget_service", "hard");
|
|
if (row.notifyEnabled && compareCents(observed.totalExact, Math.ceil(row.amount * row.warnPercent / 100)) >= 0) {
|
|
await createIncidentIfNeeded(row, "soft", observed);
|
|
} else {
|
|
await resolveOpenIncidentsForPolicy(row.id, null, null, "soft");
|
|
}
|
|
} else {
|
|
const softThreshold = Math.ceil((row.amount * row.warnPercent) / 100);
|
|
if (row.notifyEnabled && compareCents(observed.totalExact, softThreshold) >= 0 && (!row.hardStopEnabled || compareCents(observed.totalExact, row.amount) < 0)) {
|
|
await createIncidentIfNeeded(row, "soft", observed);
|
|
}
|
|
if (observedBlocks(row, observed)) {
|
|
await resolveOpenSoftIncidents(row.id);
|
|
await createIncidentIfNeeded(row, "hard", observed);
|
|
await pauseAndCancelScopeForBudget(row);
|
|
}
|
|
}
|
|
} else {
|
|
await resumeScopeFromBudget(row);
|
|
await resolveOpenIncidentsForPolicy(row.id, "approved", actorUserId ?? "budget_service");
|
|
}
|
|
|
|
await recordActivity({
|
|
companyId,
|
|
actorType: "user",
|
|
actorId: actorUserId ?? "board",
|
|
action: "budget.policy_upserted",
|
|
entityType: "budget_policy",
|
|
entityId: row.id,
|
|
details: {
|
|
scopeType: row.scopeType,
|
|
scopeId: row.scopeId,
|
|
amount: row.amount,
|
|
windowKind: row.windowKind,
|
|
},
|
|
});
|
|
|
|
return buildPolicySummary(row);
|
|
},
|
|
|
|
overview: async (companyId: string): Promise<BudgetOverview> => {
|
|
const rows = await listPolicyRows(companyId);
|
|
const policies = await mapBudgetReads(rows, buildPolicySummary);
|
|
const activeIncidentRows = await db
|
|
.select()
|
|
.from(budgetIncidents)
|
|
.where(and(eq(budgetIncidents.companyId, companyId), eq(budgetIncidents.status, "open")))
|
|
.orderBy(desc(budgetIncidents.createdAt));
|
|
const activeIncidents = await hydrateIncidentRows(activeIncidentRows);
|
|
return {
|
|
companyId,
|
|
policies,
|
|
activeIncidents,
|
|
pausedAgentCount: new Set(policies.filter((policy) => policy.scopeType === "agent" && policy.paused).map((policy) => policy.scopeId)).size,
|
|
pausedProjectCount: new Set(policies.filter((policy) => policy.scopeType === "project" && policy.paused).map((policy) => policy.scopeId)).size,
|
|
pendingApprovalCount: activeIncidents.filter((incident) => incident.approvalStatus === "pending").length,
|
|
};
|
|
},
|
|
|
|
evaluateAffectedScopes,
|
|
evaluateCostEvent: (event: typeof costEvents.$inferSelect) => evaluateAffectedScopes(event.companyId, event.agentId, event.projectId),
|
|
|
|
evaluateAdmission,
|
|
getInvocationBlock: async (companyId: string, agentId: string | null, context?: { issueId?: string | null; projectId?: string | null }) =>
|
|
(await evaluateAdmission(companyId, agentId, context)).block,
|
|
|
|
resolveIncident: async (
|
|
companyId: string,
|
|
incidentId: string,
|
|
input: BudgetIncidentResolutionInput,
|
|
actorUserId: string,
|
|
): Promise<BudgetIncident> => {
|
|
const incident = await db
|
|
.select()
|
|
.from(budgetIncidents)
|
|
.where(eq(budgetIncidents.id, incidentId))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!incident) throw notFound("Budget incident not found");
|
|
if (incident.companyId !== companyId) throw notFound("Budget incident not found");
|
|
|
|
const policy = await getPolicyRow(incident.policyId);
|
|
if (input.action === "raise_budget_and_resume") {
|
|
const nextAmount = Math.max(0, Math.floor(input.amount ?? 0));
|
|
const observed = await computeObservedSpend(db, policy);
|
|
if (unpricedUsageBlocks(policy, observed)) {
|
|
throw unprocessable("Resolve unpriced usage or explicitly allow it in the budget policy before resuming");
|
|
}
|
|
if (observed.pendingRunCount > 0) throw unprocessable("Completed runs must finish accounting before resuming");
|
|
if (compareCents(observed.totalExact, nextAmount) >= 0) {
|
|
throw unprocessable("New budget must exceed current observed spend");
|
|
}
|
|
|
|
const now = new Date();
|
|
await notifyDeliveryWork(db, DELIVERY_QUEUES.budgetEnforcement, companyId);
|
|
await db
|
|
.update(budgetPolicies)
|
|
.set({
|
|
amount: nextAmount,
|
|
isActive: true,
|
|
updatedByUserId: actorUserId,
|
|
updatedAt: now,
|
|
enforcementVersion: sql`${budgetPolicies.enforcementVersion} + 1`,
|
|
})
|
|
.where(eq(budgetPolicies.id, policy.id));
|
|
|
|
if (policy.scopeType === "company" && policy.windowKind === "calendar_month_utc") {
|
|
await db
|
|
.update(companies)
|
|
.set({ budgetMonthlyCents: nextAmount, updatedAt: now })
|
|
.where(eq(companies.id, policy.scopeId));
|
|
}
|
|
|
|
if (policy.scopeType === "agent" && policy.windowKind === "calendar_month_utc") {
|
|
await db
|
|
.update(agents)
|
|
.set({ budgetMonthlyCents: nextAmount, updatedAt: now })
|
|
.where(eq(agents.id, policy.scopeId));
|
|
}
|
|
|
|
await resumeScopeFromBudget(policy);
|
|
await db
|
|
.update(budgetIncidents)
|
|
.set({
|
|
status: "resolved",
|
|
resolvedAt: now,
|
|
updatedAt: now,
|
|
})
|
|
.where(and(eq(budgetIncidents.policyId, policy.id), eq(budgetIncidents.status, "open")));
|
|
|
|
await markApprovalStatus(db, incident.approvalId ?? null, "approved", input.decisionNote, actorUserId);
|
|
} else {
|
|
await db
|
|
.update(budgetIncidents)
|
|
.set({
|
|
status: "dismissed",
|
|
resolvedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(budgetIncidents.id, incident.id));
|
|
await markApprovalStatus(db, incident.approvalId ?? null, "rejected", input.decisionNote, actorUserId);
|
|
}
|
|
|
|
await recordActivity({
|
|
companyId: incident.companyId,
|
|
actorType: "user",
|
|
actorId: actorUserId,
|
|
action: "budget.incident_resolved",
|
|
entityType: "budget_incident",
|
|
entityId: incident.id,
|
|
details: {
|
|
action: input.action,
|
|
amount: input.amount ?? null,
|
|
scopeType: incident.scopeType,
|
|
scopeId: incident.scopeId,
|
|
},
|
|
});
|
|
|
|
const [updated] = await hydrateIncidentRows([{
|
|
...incident,
|
|
status: input.action === "raise_budget_and_resume" ? "resolved" : "dismissed",
|
|
resolvedAt: new Date(),
|
|
updatedAt: new Date(),
|
|
}]);
|
|
return updated!;
|
|
},
|
|
};
|
|
}
|
|
|
|
/** Fence the durable stop intent against policy edits and admission. External
|
|
* process shutdown must happen after this transaction, without holding locks
|
|
* that the executor needs to persist its final usage receipt. */
|
|
export async function withCurrentBudgetEnforcement<T>(db: Db, scope: BudgetEnforcementScope, work: (tx: Db) => Promise<T>): Promise<T | null> {
|
|
return withAccountingTransaction(db, scope.companyId, async (tx) => {
|
|
if (scope.enforcement) {
|
|
const [policy] = await tx.select().from(budgetPolicies).where(and(
|
|
eq(budgetPolicies.id, scope.enforcement.policyId), eq(budgetPolicies.companyId, scope.companyId),
|
|
));
|
|
if (!policy || policy.scopeType !== scope.scopeType || policy.scopeId !== scope.scopeId
|
|
|| policy.enforcementVersion !== scope.enforcement.version || !(await policyBlocks(tx, policy))) return null;
|
|
}
|
|
return work(tx);
|
|
});
|
|
}
|
|
|
|
async function deliverScopes<T>(items: T[], deliver: (item: T) => Promise<void>) {
|
|
let next = 0;
|
|
await Promise.all(Array.from({ length: Math.min(4, items.length) }, async () => {
|
|
while (next < items.length) await deliver(items[next++]);
|
|
}));
|
|
}
|
|
|
|
/** Cancellation is an at-least-once external effect. Only the delivery worker
|
|
* invokes this: request paths commit intent and return without awaiting it. */
|
|
export async function deliverBudgetEnforcement(db: Db, hooks: BudgetServiceHooks, companyId?: string, _agentId?: string | null) {
|
|
if (!hooks.cancelWorkForScope) return;
|
|
const attempted = new Set<string>();
|
|
const scopeKey = (scope: BudgetEnforcementScope) => `${scope.companyId}:${scope.scopeType}:${scope.scopeId}`;
|
|
const pending = await db.select().from(budgetPolicies).where(and(
|
|
sql`${budgetPolicies.enforcementVersion} > ${budgetPolicies.enforcementDeliveredVersion}`,
|
|
companyId ? eq(budgetPolicies.companyId, companyId) : undefined,
|
|
)).orderBy(budgetPolicies.updatedAt, budgetPolicies.id).limit(100);
|
|
await deliverScopes(pending, policy => withBudgetSpan("budget.enforcement.scope", async () => {
|
|
try {
|
|
const cancellation = await withAccountingTransaction(db, policy.companyId, async tx => {
|
|
const [current] = await tx.select().from(budgetPolicies).where(eq(budgetPolicies.id, policy.id));
|
|
if (!current || current.enforcementVersion !== policy.enforcementVersion || !(await policyBlocks(tx, current))) return null;
|
|
return { companyId: policy.companyId, scopeType: policy.scopeType as BudgetScopeType, scopeId: policy.scopeId,
|
|
createdBefore: new Date(), enforcement: { policyId: policy.id, version: policy.enforcementVersion } };
|
|
});
|
|
if (cancellation) {
|
|
attempted.add(scopeKey(cancellation));
|
|
await hooks.cancelWorkForScope!(cancellation);
|
|
}
|
|
await db.update(budgetPolicies).set({ enforcementDeliveredVersion: policy.enforcementVersion })
|
|
.where(and(eq(budgetPolicies.id, policy.id), eq(budgetPolicies.enforcementVersion, policy.enforcementVersion)));
|
|
} catch (error) {
|
|
logger.warn({ err: error, policyId: policy.id }, "Budget cancellation delivery pending; accounting is committed");
|
|
// Rotate a failed scope behind other pending work. A poisoned delivery
|
|
// must not monopolize the bounded batch on every retry.
|
|
await db.update(budgetPolicies).set({ updatedAt: new Date() }).where(and(eq(budgetPolicies.id, policy.id),
|
|
eq(budgetPolicies.enforcementVersion, policy.enforcementVersion))).catch(() => undefined);
|
|
}
|
|
}, { "budget.enforcement.queue_delay_ms": Math.max(0, Date.now() - policy.updatedAt.getTime()) }));
|
|
// A committed run Stop outlives a policy edit or deletion. It remains owed
|
|
// even if no policy generation now authorizes a fresh cancellation.
|
|
const committed = (await listPendingBudgetCancellationScopes(db))
|
|
.filter(scope => (!companyId || scope.companyId === companyId) && !attempted.has(scopeKey(scope)));
|
|
await deliverScopes(committed, async scope => {
|
|
try { await hooks.cancelWorkForScope!(scope); }
|
|
catch (error) { logger.warn({ err: error, companyId: scope.companyId, scopeType: scope.scopeType, scopeId: scope.scopeId }, "Committed budget Stop remains pending"); }
|
|
});
|
|
}
|
|
|
|
export async function hasPendingBudgetEnforcement(db: Db): Promise<boolean> {
|
|
const [pending] = await db.select({ id: budgetPolicies.id }).from(budgetPolicies)
|
|
.where(sql`${budgetPolicies.enforcementVersion} > ${budgetPolicies.enforcementDeliveredVersion}`).limit(1);
|
|
return Boolean(pending) || (await listPendingBudgetCancellationScopes(db, 1)).length > 0;
|
|
}
|
|
|
|
const budgetOwnedScope = or(
|
|
and(eq(budgetPolicies.scopeType, "company"), sql`exists (select 1 from ${companies} where ${companies.id} = ${budgetPolicies.scopeId} and ${companies.pauseReason} = 'budget')`),
|
|
and(eq(budgetPolicies.scopeType, "agent"), sql`exists (select 1 from ${agents} where ${agents.id} = ${budgetPolicies.scopeId} and ${agents.lifecycleHolds} @> '["budget"]'::jsonb)`),
|
|
and(eq(budgetPolicies.scopeType, "project"), sql`exists (select 1 from ${projects} where ${projects.id} = ${budgetPolicies.scopeId} and ${projects.pauseReason} = 'budget')`),
|
|
)!;
|
|
|
|
const openIncidentScope = sql`exists (select 1 from ${budgetIncidents}
|
|
where ${budgetIncidents.policyId} = ${budgetPolicies.id} and ${budgetIncidents.status} = 'open')`;
|
|
const unfinishedRunScope = sql`exists (select 1 from ${heartbeatRuns}
|
|
where ${heartbeatRuns.companyId} = ${budgetPolicies.companyId} and ${heartbeatRuns.costAccountingPending} = true
|
|
and ${heartbeatRuns.costAccountedAt} is null and (${budgetPolicies.scopeType} = 'company'
|
|
or (${budgetPolicies.scopeType} = 'agent' and ${heartbeatRuns.agentId} = ${budgetPolicies.scopeId})
|
|
or (${budgetPolicies.scopeType} = 'project' and ${heartbeatRuns.usageJson}->'ledgerScope'->>'projectId' = ${budgetPolicies.scopeId}::text)))`;
|
|
|
|
export async function nextBudgetReconciliationAt(db: Db): Promise<Date | null> {
|
|
const [held] = await db.select({ id: budgetPolicies.id }).from(budgetPolicies)
|
|
.where(and(eq(budgetPolicies.windowKind, "calendar_month_utc"), or(budgetOwnedScope, openIncidentScope))).limit(1);
|
|
return held ? currentUtcMonthWindow().end : null;
|
|
}
|
|
|
|
/** Startup repairs configured policy scopes, not every agent. Afterwards only
|
|
* held monthly scopes owe a timed reevaluation; writes reconcile their own scopes. */
|
|
export async function reconcileDueBudgetPolicies(db: Db, options: { startup?: boolean; cursor?: string | null } = {}): Promise<{ nextCursor: string | null }> {
|
|
const actionable = options.startup ? or(budgetOwnedScope, openIncidentScope,
|
|
sql`${budgetPolicies.enforcementVersion} > ${budgetPolicies.enforcementDeliveredVersion}`,
|
|
and(eq(budgetPolicies.isActive, true), eq(budgetPolicies.hardStopEnabled, true), sql`${budgetPolicies.amount} > 0`, unfinishedRunScope))
|
|
: and(eq(budgetPolicies.windowKind, "calendar_month_utc"), or(budgetOwnedScope, openIncidentScope));
|
|
const rows = await db.select({ id: budgetPolicies.id, companyId: budgetPolicies.companyId, scopeType: budgetPolicies.scopeType, scopeId: budgetPolicies.scopeId })
|
|
.from(budgetPolicies).where(and(actionable, options.cursor ? sql`${budgetPolicies.id} > ${options.cursor}::uuid` : undefined))
|
|
.orderBy(budgetPolicies.id).limit(100);
|
|
const scopes = new Map(rows.map(scope => [`${scope.companyId}:${scope.scopeType}:${scope.scopeId}`, scope]));
|
|
for (const scope of scopes.values()) {
|
|
try {
|
|
await withAccountingTransaction(db, scope.companyId, (tx, publications) => budgetServiceInTransaction(tx, publications)
|
|
.reconcileScope(scope.companyId, scope.scopeType as BudgetScopeType, scope.scopeId));
|
|
} catch (error) {
|
|
if (error instanceof HttpError && error.status === 404) continue;
|
|
throw error;
|
|
}
|
|
}
|
|
return { nextCursor: rows.length === 100 ? rows[rows.length - 1].id : null };
|
|
}
|
|
|
|
export function budgetService(db: Db, _hooks: BudgetServiceHooks = {}) {
|
|
const reads = budgetServiceInTransaction(db);
|
|
const mutate = <T>(companyId: string, work: (service: ReturnType<typeof budgetServiceInTransaction>) => Promise<T>) =>
|
|
withAccountingTransaction(db, companyId, (tx, publications) => work(budgetServiceInTransaction(tx, publications)));
|
|
return {
|
|
...reads,
|
|
evaluateAdmission: (companyId: string, agentId: string | null, context?: { issueId?: string | null; projectId?: string | null }) =>
|
|
mutate(companyId, service => service.evaluateAdmission(companyId, agentId, context)),
|
|
evaluateAffectedScopes: (companyId: string, agentId: string | null, projectId: string | null) =>
|
|
mutate(companyId, service => service.evaluateAffectedScopes(companyId, agentId, projectId)),
|
|
reconcileScope: (companyId: string, scopeType: BudgetScopeType, scopeId: string) =>
|
|
mutate(companyId, service => service.reconcileScope(companyId, scopeType, scopeId)),
|
|
reconcilePolicies: () => reconcileDueBudgetPolicies(db, { startup: true }),
|
|
upsertPolicy: (companyId: string, input: BudgetPolicyUpsertInput, actorUserId: string | null) =>
|
|
mutate(companyId, service => service.upsertPolicy(companyId, input, actorUserId)),
|
|
evaluateCostEvent: (event: typeof costEvents.$inferSelect) => mutate(event.companyId, service => service.evaluateCostEvent(event)),
|
|
getInvocationBlock: (companyId: string, agentId: string | null, context?: { issueId?: string | null; projectId?: string | null }) =>
|
|
mutate(companyId, service => service.getInvocationBlock(companyId, agentId, context)),
|
|
resolveIncident: (companyId: string, incidentId: string, input: BudgetIncidentResolutionInput, actorUserId: string) =>
|
|
mutate(companyId, service => service.resolveIncident(companyId, incidentId, input, actorUserId)),
|
|
};
|
|
}
|