Files
PaperClipAI/server/src/services/budgets.ts
T
DottaandPaperclip f836fec59e perf(budgets): avoid company lock for routine eligibility checks (#18830)
## Thinking Path

> - Paperclip helps people manage AI agents at work.
> - The budget service checks whether an agent can start work.
> - Preliminary checks acquired the company accounting lock even with no
budget policy.
> - Independent checks then waited behind accounting writes for the same
company.
> - This pull request reads one consistent snapshot before it decides
whether reconciliation is needed.
> - Healthy checks avoid the writer lock. State changes and final paid
admission retain their existing authority.
> - The benefit is less company-wide contention during routine
eligibility checks.

## Linked Issues or Issue Description

Refs #17697 (the preliminary-check portion). Refs #17817 (related
serialization in agent configuration).

Related approach: #17966. That PR adds a precheck for policies that
cannot block. This PR also avoids the lock for healthy active hard-stop
policies and manual pause reads. It uses a repeatable-read snapshot and
leaves public transactional admission locked. It does not add the
inactive-policy spend filtering proposed in #17966.

**Current behavior**

Every public `getInvocationBlock` call takes the company writer lock.
Healthy checks can wait behind an unrelated receipt or settlement
operation.

**Proposed behavior**

Read current scope state, policies, exposure and relevant incidents in a
read-only snapshot. Return directly when no transition is needed.
Otherwise discard the snapshot and reconcile current state under the
company lock.

**Reason and benefit**

Independent preliminary checks can finish while another transaction owns
the company row. Final reservations and paid-work admission still
serialize against current accounting state.

## What Changed

- Separate preliminary eligibility inspection from mutating
reconciliation in the budget service.
- Share scope loading, pause decisions and soft-incident resolution
conditions between the two paths.
- Keep no-policy checks free of exposure scans and incident reads.
- Add inspection and reconciliation spans to the existing operator-gated
Observability path.
- Add PostgreSQL contention tests and regressions for pauses, holds,
incidents, recovering native accounting and company boundaries.
- Use the shared fixture startup timeout. Per-test deadlines and the 500
ms lock timeout stay intact.
- Update the implementation contract and tracing documentation. Add
dated investigation and future-design notes. The future accounting
architecture remains a proposal.

## Verification

- 73 targeted tests passed across eight files on implementation commit
`244db565`. No tests were skipped. The files cover budget service,
enforcement scoping and delivery, live receipt attempts, budget history,
run settlement, heartbeat accounting and budget tracing.
- Real PostgreSQL tests hold the company accounting lock while routine
checks finish. The same reader's authoritative admission instead fails
with PostgreSQL lock timeout `55P03`.
- Full workspace typecheck passed before the rebase: `node
scripts/typecheck-low-memory.mjs`.
- Full workspace build passed before the rebase:
`PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm -r --workspace-concurrency=1
build`. The Runner release binary was rebuilt from source.
- `node scripts/check-module-boundaries.mjs` and `git diff --check`
passed.
- Initial local setup failed before assertions: cold fixture setup
exceeded 20 seconds, then a missing SDK build was exposed. The fixture
now uses the shared 90-second startup allowance. The SDK was built
before the successful test run. An incomplete local Copilot artifact
also blocked the first build; the pinned installer repaired it before
the successful canonical retry.
- The branch rebased cleanly onto master `83b5f7d486`. All 73 targeted
tests then passed again on clean head
`244db565ae206502b100c3cdf5cd86a0ae91b372` (eight files, no skips). The
full local Vitest suite was not run; remote CI must verify the wider
suite.
- Initial remote CI failed three monitor-scheduler cases at the suite's
three-second heartbeat-idle wait. The unchanged suite then passed all 21
tests locally (`pnpm exec vitest run
src/__tests__/issue-monitor-scheduler.test.ts`, 70.13 seconds). The one
failed-job rerun was cancelled after the review follow-up was pushed; it
produced no pass/fail result. No scheduler assertions or timeouts were
changed. On updated head `ba2ab2a0`, the unchanged monitor suite passed
all 21 tests in 38.381 seconds in [server shard
10/12](https://github.com/paperclipai/paperclip/actions/runs/41702687492/job/126151059714).
That shard passed 115 files and 2,677 tests, with no failures (one file
and 20 tests skipped by existing suite configuration).
- The independent review identified missing recovering-run coverage. The
added contention case requires the locked fallback while another
accounting writer holds the company row, then verifies the recovery
block without pausing or cancelling recovery. The regression is in
follow-up commit `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`. All 19
budget-service tests passed locally (no skips, 48.89 seconds), and `pnpm
--filter @paperclipai/server typecheck` passed after the follow-up.
- The updated CI run then hit a separate plugin-refresh fixture timeout
at its final re-enable settle. The unchanged
`external-object-refresh-work.test.ts` suite passed all 24 tests locally
(52.02 seconds). Inspection identified a possible fake-clock race: its
quiet helper can observe no scheduled deadline while reconstruction is
still active, then encounter the existing 30-second reconstruction
cooldown. The one same-head failed-job retry
[passed](https://github.com/paperclipai/paperclip/actions/runs/41702687492/job/126158227876):
refresh suite 24/24, shard 115 files and 4,125 tests passed (one file
and four tests skipped by existing configuration). No assertions,
timeouts, or product code were changed.
- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/41702687492)
passed on exact head `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`: build,
typecheck, Runner checks, all server/workspace suites and E2E. Both
required aggregates are successful. All 59 checks are terminal: 54
successful, two advisory neutral and three expected skips. The PR is
mergeable with a clean merge state.
- Deployment latency and CPU effects have not been measured.
- Greptile reviewed updated head
`ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`: 5/5, no actionable findings
or unresolved Greptile threads.
- Independent Paperclip review on the updated head: advisory 4/5, no
blocker, recovery gap confirmed resolved. One canonical maintainability
advisory remains visible and is documented below.

## Risks

- Preliminary inspection can precede a concurrent policy or pause
change. It grants no paid dispatch authority. The final locked admission
and reservation checks remain required.
- Healthy configured policies add an incident lookup. An unheld blocking
scope can compute exposure during both inspection and locked
revalidation. Already-held scopes skip the first exposure computation.
- Holds and incidents must retain their existing transition effects. The
fallback reads them again under the company lock, rather than mutating
from the earlier snapshot.
- Eligibility inspection and reconciliation still maintain separate
transition lists. Future reconciliation changes must update the
inspection predicate and its contention coverage together; consolidating
the effects model remains a nonblocking review follow-up.
- No schema or API response change is required. This change does not
relax the current budget enforcement contract.

## Model Used

OpenAI Codex with model `gpt-6.1-sol`. Used reasoning, repository
inspection, code execution and delegated review. The session did not
expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 13:48:56 -06:00

1244 lines
57 KiB
TypeScript

import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js";
import { notifyRunDispatch } from "./run-dispatch-notifications.js";
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
import { notifyExternalObjectRefresh } from "./external-object-refresh-notifications.js";
import { notifyApprovalWork } from "./approval-work-notifications.js";
import { notifyAttentionRetentionWork, notifyCompanyRetentionWork } from "./retention-work-notifications.js";
import { addCents, compareCents, normalizeCents } from "@paperclipai/shared";
import { and, desc, eq, inArray, sql, or, isNull } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
agents,
approvals,
budgetIncidents,
budgetPolicies,
budgetReservations,
companies,
costEvents,
heartbeatRuns,
projects,
} from "@paperclipai/db";
import type {
BudgetIncident,
BudgetIncidentResolutionInput,
BudgetMetric,
BudgetOverview,
PauseReason,
BudgetPolicy,
BudgetPolicySummary,
BudgetPolicyUpsertInput,
BudgetScopeType,
BudgetThresholdType,
BudgetWindowKind,
} from "@paperclipai/shared";
import { HttpError, notFound, unprocessable } from "../errors.js";
import { logger } from "../middleware/logger.js";
import { withAccountingReadSnapshot, withAccountingTransaction } from "./accounting-transaction.js";
import { logActivity, type LogActivityInput, type ActivityPublication } from "./activity-log.js";
import { computeObservedSpend, computeObservedSpends, type ObservedSpend } from "./budget-exposure.js";
import { notifyDeliveryWork, DELIVERY_QUEUES } from "./delivery-work-notifications.js";
import { listPendingBudgetCancellationScopes } from "./heartbeat/budget-cancellation.js";
import { withBudgetSpan } from "./budget-tracing.js";
export { computeObservedSpend, computeObservedSpends } from "./budget-exposure.js";
export type { BudgetExposure } from "./budget-exposure.js";
type ScopeRecord = {
companyId: string;
name: string;
paused: boolean;
pauseReason: PauseReason | null;
budgetHeld: boolean;
};
type PolicyRow = typeof budgetPolicies.$inferSelect;
type IncidentRow = typeof budgetIncidents.$inferSelect;
export type BudgetEnforcementScope = {
companyId: string;
scopeType: BudgetScopeType;
scopeId: string;
/** Snapshot taken under the company lock, before a later budget grant can admit new work. */
createdBefore?: Date;
/** Revalidated at the durable cancellation write, under the admission lock. */
enforcement?: { policyId: string; version: number };
};
export type BudgetServiceHooks = {
cancelWorkForScope?: (scope: BudgetEnforcementScope) => Promise<void>;
};
function currentUtcMonthWindow(now = new Date()) {
const year = now.getUTCFullYear();
const month = now.getUTCMonth();
const start = new Date(Date.UTC(year, month, 1, 0, 0, 0, 0));
const end = new Date(Date.UTC(year, month + 1, 1, 0, 0, 0, 0));
return { start, end };
}
function resolveWindow(windowKind: BudgetWindowKind, now = new Date()) {
if (windowKind === "lifetime") {
return {
start: new Date(Date.UTC(1970, 0, 1, 0, 0, 0, 0)),
end: new Date(Date.UTC(9999, 0, 1, 0, 0, 0, 0)),
};
}
return currentUtcMonthWindow(now);
}
function budgetStatusFromObserved(
observedAmount: string,
amount: number,
warnPercent: number,
): BudgetPolicySummary["status"] {
if (amount <= 0) return "ok";
if (compareCents(observedAmount, amount) >= 0) return "hard_stop";
if (compareCents(observedAmount, Math.ceil((amount * warnPercent) / 100)) >= 0) return "warning";
return "ok";
}
function normalizeScopeName(scopeType: BudgetScopeType, name: string) {
if (scopeType === "company") return name;
return name.trim().length > 0 ? name : scopeType;
}
async function resolveScopeRecord(db: Db, scopeType: BudgetScopeType, scopeId: string): Promise<ScopeRecord> {
if (scopeType === "company") {
const row = await db
.select({
companyId: companies.id,
name: companies.name,
status: companies.status,
pauseReason: companies.pauseReason,
pausedAt: companies.pausedAt,
})
.from(companies)
.where(eq(companies.id, scopeId))
.then((rows) => rows[0] ?? null);
if (!row) throw notFound("Company not found");
return {
companyId: row.companyId,
name: row.name,
paused: row.status === "paused" || Boolean(row.pausedAt),
pauseReason: (row.pauseReason as ScopeRecord["pauseReason"]) ?? null,
budgetHeld: row.pauseReason === "budget",
};
}
if (scopeType === "agent") {
const row = await db
.select({
companyId: agents.companyId,
name: agents.name,
status: agents.status,
pauseReason: agents.pauseReason,
lifecycleHolds: agents.lifecycleHolds,
})
.from(agents)
.where(eq(agents.id, scopeId))
.then((rows) => rows[0] ?? null);
if (!row) throw notFound("Agent not found");
return {
companyId: row.companyId,
name: row.name,
paused: row.status === "paused",
pauseReason: (row.pauseReason as ScopeRecord["pauseReason"]) ?? null,
budgetHeld: row.lifecycleHolds.includes("budget"),
};
}
const row = await db
.select({
companyId: projects.companyId,
name: projects.name,
pauseReason: projects.pauseReason,
pausedAt: projects.pausedAt,
})
.from(projects)
.where(eq(projects.id, scopeId))
.then((rows) => rows[0] ?? null);
if (!row) throw notFound("Project not found");
return {
companyId: row.companyId,
name: row.name,
paused: Boolean(row.pausedAt),
pauseReason: (row.pauseReason as ScopeRecord["pauseReason"]) ?? null,
budgetHeld: row.pauseReason === "budget",
};
}
async function resolveSummaryScopeRecord(db: Db, companyId: string, scopeType: BudgetScopeType, scopeId: string): Promise<ScopeRecord> {
let scope: ScopeRecord;
try {
scope = await resolveScopeRecord(db, scopeType, scopeId);
} catch (error) {
if (!(error instanceof HttpError) || error.status !== 404 || (scopeType !== "agent" && scopeType !== "project")) throw error;
return { companyId, name: scopeType === "agent" ? "Deleted agent" : "Deleted project",
paused: false, pauseReason: null, budgetHeld: false };
}
if (scope.companyId !== companyId) throw notFound("Budget scope not found");
return scope;
}
// Keep a large dashboard from filling the database pool with queued reads.
async function mapBudgetReads<T, R>(rows: T[], read: (row: T) => Promise<R>): Promise<R[]> {
const result: R[] = [];
for (let start = 0; start < rows.length; start += 4) {
result.push(...await Promise.all(rows.slice(start, start + 4).map(read)));
}
return result;
}
/** Filter in PostgreSQL so one run never loads unrelated agents' policies. */
export function budgetPoliciesForRun(companyId: string, agentId: string | null, projectId: string | null) {
return or(
and(eq(budgetPolicies.scopeType, "company"), eq(budgetPolicies.scopeId, companyId)),
agentId ? and(eq(budgetPolicies.scopeType, "agent"), eq(budgetPolicies.scopeId, agentId)) : undefined,
projectId ? and(eq(budgetPolicies.scopeType, "project"), eq(budgetPolicies.scopeId, projectId)) : undefined,
);
}
type AdmissionContext = { issueId?: string | null; projectId?: string | null };
async function readAdmissionScopes(db: Db, companyId: string, agentId: string | null, context?: AdmissionContext) {
const scopes: Array<{ scopeType: BudgetScopeType; scopeId: string }> = [
{ scopeType: "company", scopeId: companyId }, ...(agentId ? [{ scopeType: "agent" as const, scopeId: agentId }] : []),
...(context?.projectId ? [{ scopeType: "project" as const, scopeId: context.projectId }] : []),
];
const records = await Promise.all(scopes.map(scope => resolveScopeRecord(db, scope.scopeType, scope.scopeId)));
if (records.some(scope => scope.companyId !== companyId)) throw notFound("Budget scope not found");
const policies = await db.select().from(budgetPolicies).where(and(eq(budgetPolicies.companyId, companyId),
budgetPoliciesForRun(companyId, agentId, context?.projectId ?? null)));
return { scopes, records, policies };
}
function scopePauseBlock(scopeType: BudgetScopeType, scopeId: string, scope: ScopeRecord, hasPolicies: boolean) {
if (scopeType !== "agent" && scope.paused && scope.pauseReason !== "budget") {
return { scopeType, scopeId, scopeName: scope.name, reason: `${scopeType === "company" ? "Company" : "Project"} is paused and cannot start new work.` };
}
if (scope.paused && scope.pauseReason === "budget" && !hasPolicies) {
return { scopeType, scopeId, scopeName: scope.name, reason: "Budget pause requires a policy or an explicit operator resume." };
}
return null;
}
async function inspectInvocationEligibility(db: Db, companyId: string, agentId: string | null, context?: AdmissionContext) {
const { scopes, records, policies } = await readAdmissionScopes(db, companyId, agentId, context);
if (policies.length > 0) {
if (records.some(scope => scope.budgetHeld)) return { needsReconciliation: true } as const;
const observations = await computeObservedSpends(db, policies);
if (policies.some((policy, index) => observedBlocks(policy, observations[index])
|| (policy.isActive && policy.hardStopEnabled && policy.amount > 0 && observations[index].recoveringRunCount > 0))) {
return { needsReconciliation: true } as const;
}
const incidents = await db.select({ policyId: budgetIncidents.policyId, thresholdType: budgetIncidents.thresholdType })
.from(budgetIncidents).where(and(eq(budgetIncidents.companyId, companyId),
inArray(budgetIncidents.policyId, policies.map(policy => policy.id)), eq(budgetIncidents.status, "open")));
const byId = new Map(policies.map((policy, index) => [policy.id, { policy, observed: observations[index] }]));
if (incidents.some(incident => {
const { policy, observed } = byId.get(incident.policyId)!;
return incident.thresholdType === "hard" || shouldResolveSoftIncident(policy, observed);
})) return { needsReconciliation: true } as const;
}
for (const [index, { scopeType, scopeId }] of scopes.entries()) {
const block = scopePauseBlock(scopeType, scopeId, records[index],
policies.some(policy => policy.scopeType === scopeType && policy.scopeId === scopeId));
if (block) return { needsReconciliation: false, block } as const;
}
return { needsReconciliation: false, block: null } as const;
}
function unpricedUsageBlocks(policy: PolicyRow, observed: ObservedSpend) {
return policy.unpricedUsagePolicy !== "allow"
&& (observed.unpricedEventCount > 0 || observed.unknownBrowserRunCount > 0);
}
function shouldResolveSoftIncident(policy: PolicyRow, observed: ObservedSpend) {
return !policy.isActive || !policy.notifyEnabled
|| compareCents(observed.totalExact, Math.ceil(policy.amount * policy.warnPercent / 100)) < 0;
}
function observedBlocks(policy: PolicyRow, observed: Awaited<ReturnType<typeof computeObservedSpend>>) {
if (!policy.isActive || !policy.hardStopEnabled || policy.amount <= 0) return false;
return compareCents(observed.totalExact, policy.amount) >= 0 || (observed.pendingRunCount - observed.recoveringRunCount) > 0 || unpricedUsageBlocks(policy, observed);
}
export async function policyBlocks(db: Db, policy: PolicyRow) {
if (!policy.isActive || !policy.hardStopEnabled || policy.amount <= 0) return false;
return observedBlocks(policy, await computeObservedSpend(db, policy));
}
/** Supplemental handoff check after authoritative accounting admission commits.
* Read only: the caller can hold issue/run ownership, so this must never take
* the company lock, reconcile holds, or deliver effects. A later concurrent
* charge targets already-admitted work through the normal durable Stop path. */
export async function recheckAdmittedRunBudgetReadOnly(db: Db, input: {
companyId: string; agentId: string; runId: string; projectId: string | null; reservationId: string;
}): Promise<string | null> {
return withBudgetSpan("budget.admission.evaluate", async () => {
const [reservation] = await db.select({ amountCents: budgetReservations.amountCents }).from(budgetReservations).where(and(
eq(budgetReservations.id, input.reservationId), eq(budgetReservations.companyId, input.companyId),
eq(budgetReservations.runId, input.runId), eq(budgetReservations.agentId, input.agentId),
sql`${budgetReservations.projectId} is not distinct from ${input.projectId}::uuid`,
eq(budgetReservations.state, "held"),
));
if (!reservation) return "Provider dispatch no longer owns a held budget reservation";
const policies = await db.select().from(budgetPolicies).where(and(eq(budgetPolicies.companyId, input.companyId),
eq(budgetPolicies.isActive, true), eq(budgetPolicies.hardStopEnabled, true), eq(budgetPolicies.metric, "billed_cents"),
sql`${budgetPolicies.amount} > 0`, budgetPoliciesForRun(input.companyId, input.agentId, input.projectId)));
const observations = await computeObservedSpends(db, policies);
for (const [index, policy] of policies.entries()) {
const observed = observations[index];
if (observedBlocks(policy, observed)) return "Applicable budget no longer permits provider work";
const capacity = compareCents(addCents(observed.ledgerCents, observed.effectiveHeldCents), policy.amount);
if (capacity > 0 || (compareCents(reservation.amountCents, 0) === 0 && capacity >= 0))
return "Available budget is reserved by unfinished runs";
}
return null;
});
}
function buildApprovalPayload(input: {
policy: PolicyRow;
scopeName: string;
thresholdType: BudgetThresholdType;
amountObserved: number;
amountObservedExact: string;
windowStart: Date;
windowEnd: Date;
}) {
return {
scopeType: input.policy.scopeType,
scopeId: input.policy.scopeId,
scopeName: input.scopeName,
metric: input.policy.metric,
windowKind: input.policy.windowKind,
thresholdType: input.thresholdType,
budgetAmount: input.policy.amount,
observedAmount: input.amountObserved,
observedAmountExact: input.amountObservedExact,
warnPercent: input.policy.warnPercent,
windowStart: input.windowStart.toISOString(),
windowEnd: input.windowEnd.toISOString(),
policyId: input.policy.id,
guidance: "Resolve pending or unpriced accounting, raise the budget, or keep the scope paused.",
};
}
async function markApprovalStatus(
db: Db,
approvalId: string | null,
status: "approved" | "rejected",
decisionNote: string | null | undefined,
decidedByUserId: string,
) {
if (!approvalId) return;
const [approval] = await db.select({ companyId: approvals.companyId }).from(approvals).where(eq(approvals.id, approvalId));
if (!approval) return;
await notifyApprovalWork(db, approval.companyId, { approvalId });
await db
.update(approvals)
.set({
status,
decisionNote: decisionNote ?? null,
decidedByUserId,
decidedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(approvals.id, approvalId));
}
/** Internal: caller must hold the company accounting transaction lock. */
export function budgetServiceInTransaction(db: Db, publications: ActivityPublication[] = []) {
const recordActivity = (input: LogActivityInput) => logActivity(db, input, publications);
async function pauseScopeForBudget(policy: PolicyRow) {
const now = new Date();
if (policy.scopeType === "agent") {
const { applyAgentBudgetHoldInTransaction } = await import("./agent-lifecycle.js");
await applyAgentBudgetHoldInTransaction(db, policy.companyId, policy.scopeId, true);
return;
}
if (policy.scopeType === "project") {
// Admission rechecks the same hard stop. Lock current eligibility before
// publishing intent so a no-op pause cannot reactivate its own receipts.
const [project] = await db.select({ pausedAt: projects.pausedAt }).from(projects)
.where(and(eq(projects.id, policy.scopeId), eq(projects.companyId, policy.companyId))).for("update");
if (!project || project.pausedAt !== null) return;
await notifyRunDispatch(db, { companyId: policy.companyId });
await notifyDeferredExecutionEligibility(db, { companyId: policy.companyId });
await db
.update(projects)
.set({
pauseReason: "budget",
pausedAt: now,
updatedAt: now,
})
.where(and(eq(projects.id, policy.scopeId), eq(projects.companyId, policy.companyId), isNull(projects.pausedAt)));
return;
}
const scope = await resolveScopeRecord(db, "company", policy.scopeId);
if (scope.paused) return;
await notifyRunDispatch(db, { companyId: policy.companyId });
await notifyDeferredExecutionEligibility(db, { companyId: policy.companyId });
await notifyCompanyRetentionWork(db, policy.companyId);
await notifyStatusCardIssueChange(db, policy.companyId);
await notifyExternalObjectRefresh(db, policy.companyId);
await notifyDeliveryWork(db, DELIVERY_QUEUES.agentTimers);
await notifyDeliveryWork(db, DELIVERY_QUEUES.routineSchedules);
await notifyDeliveryWork(db, DELIVERY_QUEUES.issueMonitors);
await notifyDeliveryWork(db, DELIVERY_QUEUES.agentLifecycle, policy.companyId);
await notifyDeliveryWork(db, DELIVERY_QUEUES.skillSourceSync);
await db
.update(companies)
.set({
status: "paused",
pauseReason: "budget",
pausedAt: now,
updatedAt: now,
})
.where(and(eq(companies.id, policy.scopeId), eq(companies.status, "active")));
}
async function pauseAndCancelScopeForBudget(policy: PolicyRow, requestCancellation = true) {
await pauseScopeForBudget(policy);
if (!requestCancellation) return;
// A pending generation already covers all admitted work in this scope.
// Rechecks must not invalidate a worker that is delivering that intent.
await notifyDeliveryWork(db, DELIVERY_QUEUES.budgetEnforcement, policy.companyId);
await db.update(budgetPolicies).set({
enforcementVersion: sql`case when ${budgetPolicies.enforcementVersion} > ${budgetPolicies.enforcementDeliveredVersion}
then ${budgetPolicies.enforcementVersion} else ${budgetPolicies.enforcementVersion} + 1 end`,
updatedAt: sql`case when ${budgetPolicies.enforcementVersion} > ${budgetPolicies.enforcementDeliveredVersion}
then ${budgetPolicies.updatedAt} else now() end`,
}).where(eq(budgetPolicies.id, policy.id));
}
async function resumeScopeFromBudget(policy: PolicyRow) {
const policies = await db.select().from(budgetPolicies).where(and(
eq(budgetPolicies.companyId, policy.companyId), eq(budgetPolicies.scopeType, policy.scopeType),
eq(budgetPolicies.scopeId, policy.scopeId), eq(budgetPolicies.isActive, true), eq(budgetPolicies.hardStopEnabled, true),
));
const observations = await computeObservedSpends(db, policies);
if (policies.some((candidate, index) => observedBlocks(candidate, observations[index]))) return;
const now = new Date();
if (policy.scopeType === "agent") {
const { applyAgentBudgetHoldInTransaction } = await import("./agent-lifecycle.js");
await applyAgentBudgetHoldInTransaction(db, policy.companyId, policy.scopeId, false);
return;
}
if (policy.scopeType === "project") {
await notifyRunDispatch(db, { companyId: policy.companyId });
await notifyDeferredExecutionEligibility(db, { companyId: policy.companyId });
await db
.update(projects)
.set({
pauseReason: null,
pausedAt: null,
updatedAt: now,
})
.where(and(eq(projects.id, policy.scopeId), eq(projects.pauseReason, "budget")));
return;
}
await notifyDeliveryWork(db, DELIVERY_QUEUES.skillSourceSync);
const scope = await resolveScopeRecord(db, "company", policy.scopeId);
if (scope.pauseReason !== "budget") return;
await notifyRunDispatch(db, { companyId: policy.companyId });
await notifyDeferredExecutionEligibility(db, { companyId: policy.companyId });
await notifyCompanyRetentionWork(db, policy.companyId);
await notifyStatusCardIssueChange(db, policy.companyId);
await notifyExternalObjectRefresh(db, policy.companyId);
await notifyDeliveryWork(db, DELIVERY_QUEUES.agentTimers);
await notifyDeliveryWork(db, DELIVERY_QUEUES.routineSchedules);
await notifyDeliveryWork(db, DELIVERY_QUEUES.issueMonitors);
await notifyDeliveryWork(db, DELIVERY_QUEUES.agentLifecycle, policy.companyId);
await db
.update(companies)
.set({
status: "active",
pauseReason: null,
pausedAt: null,
updatedAt: now,
})
.where(and(eq(companies.id, policy.scopeId), eq(companies.status, "paused"), eq(companies.pauseReason, "budget")));
}
async function getPolicyRow(policyId: string) {
const policy = await db
.select()
.from(budgetPolicies)
.where(eq(budgetPolicies.id, policyId))
.then((rows) => rows[0] ?? null);
if (!policy) throw notFound("Budget policy not found");
return policy;
}
async function listPolicyRows(companyId: string) {
return db
.select()
.from(budgetPolicies)
.where(eq(budgetPolicies.companyId, companyId))
.orderBy(desc(budgetPolicies.updatedAt));
}
async function buildPolicySummary(policy: PolicyRow): Promise<BudgetPolicySummary> {
const scope = await resolveSummaryScopeRecord(db, policy.companyId, policy.scopeType as BudgetScopeType, policy.scopeId);
const observed = await computeObservedSpend(db, policy);
const { total: observedAmount, totalExact: observedAmountExact, unpricedEventCount, pendingRunCount } = observed;
const { windowStart: start, windowEnd: end } = observed;
const amount = policy.isActive ? policy.amount : 0;
const utilizationPercent =
amount > 0 ? Number(((observedAmount / amount) * 100).toFixed(2)) : 0;
return {
policyId: policy.id,
companyId: policy.companyId,
scopeType: policy.scopeType as BudgetScopeType,
scopeId: policy.scopeId,
scopeName: normalizeScopeName(policy.scopeType as BudgetScopeType, scope.name),
metric: policy.metric as BudgetMetric,
windowKind: policy.windowKind as BudgetWindowKind,
amount,
observedAmount,
observedAmountExact,
reservationCents: policy.reservationCents,
unpricedEventCount,
pendingRunCount,
unpricedUsagePolicy: policy.unpricedUsagePolicy as "block" | "allow",
remainingAmount: amount > 0 ? Math.max(0, amount - observedAmount) : 0,
utilizationPercent,
warnPercent: policy.warnPercent,
hardStopEnabled: policy.hardStopEnabled,
notifyEnabled: policy.notifyEnabled,
isActive: policy.isActive,
status: observedBlocks(policy, observed) ? "hard_stop" : policy.isActive
? budgetStatusFromObserved(observedAmountExact, amount, policy.warnPercent)
: "ok",
paused: scope.paused,
pauseReason: scope.pauseReason,
windowStart: start,
windowEnd: end,
};
}
async function createIncidentIfNeeded(
policy: PolicyRow,
thresholdType: BudgetThresholdType,
observed: Awaited<ReturnType<typeof computeObservedSpend>>,
) {
const { windowStart: start, windowEnd: end } = observed;
const existing = await db
.select()
.from(budgetIncidents)
.where(
and(
eq(budgetIncidents.policyId, policy.id),
eq(budgetIncidents.windowStart, start),
eq(budgetIncidents.thresholdType, thresholdType),
or(eq(budgetIncidents.status, "open"), and(eq(budgetIncidents.status, "dismissed"), eq(budgetIncidents.amountLimit, policy.amount))),
),
)
.orderBy(desc(budgetIncidents.createdAt))
.then((rows) => rows[0] ?? null);
if (existing) return { incident: existing, created: false };
const { total: amountObserved, totalExact: amountObservedExact } = observed;
const scope = await resolveScopeRecord(db, policy.scopeType as BudgetScopeType, policy.scopeId);
const payload = buildApprovalPayload({
policy,
scopeName: normalizeScopeName(policy.scopeType as BudgetScopeType, scope.name),
thresholdType,
amountObserved,
amountObservedExact,
windowStart: start,
windowEnd: end,
});
await notifyAttentionRetentionWork(db, { companyId: policy.companyId, sourceKind: "budget_alert" });
if (thresholdType === "hard") await notifyApprovalWork(db, policy.companyId);
const approval = thresholdType === "hard"
? await db
.insert(approvals)
.values({
companyId: policy.companyId,
type: "budget_override_required",
requestedByUserId: null,
requestedByAgentId: null,
status: "pending",
payload,
})
.returning()
.then((rows) => rows[0] ?? null)
: null;
const incident = await db
.insert(budgetIncidents)
.values({
companyId: policy.companyId,
policyId: policy.id,
scopeType: policy.scopeType,
scopeId: policy.scopeId,
metric: policy.metric,
windowKind: policy.windowKind,
windowStart: start,
windowEnd: end,
thresholdType,
amountLimit: policy.amount,
amountObserved: sql`${amountObservedExact}::numeric`,
status: "open",
approvalId: approval?.id ?? null,
})
.returning()
.then((rows) => rows[0] ?? null);
return incident ? { incident, created: true } : null;
}
async function resolveOpenSoftIncidents(policyId: string, companyId: string) {
await notifyAttentionRetentionWork(db, { companyId, sourceKind: "budget_alert" });
await db
.update(budgetIncidents)
.set({
status: "resolved",
resolvedAt: new Date(),
updatedAt: new Date(),
})
.where(
and(
eq(budgetIncidents.policyId, policyId),
eq(budgetIncidents.thresholdType, "soft"),
eq(budgetIncidents.status, "open"),
),
);
}
async function resolveOpenIncidentsForPolicy(
policyId: string,
companyId: string,
approvalStatus: "approved" | "rejected" | null,
decidedByUserId: string | null,
thresholdType?: BudgetThresholdType,
) {
await notifyAttentionRetentionWork(db, { companyId, sourceKind: "budget_alert" });
const condition = and(eq(budgetIncidents.policyId, policyId), eq(budgetIncidents.status, "open"),
thresholdType ? eq(budgetIncidents.thresholdType, thresholdType) : undefined);
const openRows = await db
.select()
.from(budgetIncidents)
.where(condition);
await db
.update(budgetIncidents)
.set({
status: "resolved",
resolvedAt: new Date(),
updatedAt: new Date(),
})
.where(condition);
if (!approvalStatus || !decidedByUserId) return;
for (const row of openRows) {
await markApprovalStatus(db, row.approvalId ?? null, approvalStatus, "Resolved via budget update", decidedByUserId);
}
}
async function hydrateIncidentRows(rows: IncidentRow[]): Promise<BudgetIncident[]> {
const approvalIds = rows.map((row) => row.approvalId).filter((value): value is string => Boolean(value));
const approvalRows = approvalIds.length > 0
? await db
.select({ id: approvals.id, status: approvals.status })
.from(approvals)
.where(inArray(approvals.id, approvalIds))
: [];
const approvalStatusById = new Map(approvalRows.map((row) => [row.id, row.status]));
return mapBudgetReads(rows, async (row) => {
const scope = await resolveSummaryScopeRecord(db, row.companyId, row.scopeType as BudgetScopeType, row.scopeId);
return {
id: row.id,
companyId: row.companyId,
policyId: row.policyId,
scopeType: row.scopeType as BudgetScopeType,
scopeId: row.scopeId,
scopeName: normalizeScopeName(row.scopeType as BudgetScopeType, scope.name),
metric: row.metric as BudgetMetric,
windowKind: row.windowKind as BudgetWindowKind,
windowStart: row.windowStart,
windowEnd: row.windowEnd,
thresholdType: row.thresholdType as BudgetThresholdType,
amountLimit: row.amountLimit,
amountObserved: row.amountObserved,
status: row.status as BudgetIncident["status"],
approvalId: row.approvalId ?? null,
approvalStatus: row.approvalId ? approvalStatusById.get(row.approvalId) ?? null : null,
resolvedAt: row.resolvedAt ?? null,
createdAt: row.createdAt,
updatedAt: row.updatedAt,
};
});
}
async function reconcileScope(companyId: string, scopeType: BudgetScopeType, scopeId: string,
prepared?: { scope: ScopeRecord; policies: PolicyRow[]; observations: ObservedSpend[] }) {
const scope = prepared?.scope ?? await resolveScopeRecord(db, scopeType, scopeId);
if (scope.companyId !== companyId) throw notFound("Budget scope not found");
const policies = prepared?.policies ?? await db.select().from(budgetPolicies).where(and(
eq(budgetPolicies.companyId, companyId), eq(budgetPolicies.scopeType, scopeType), eq(budgetPolicies.scopeId, scopeId),
));
let recoveringPolicy: PolicyRow | undefined;
const observations = prepared?.observations ?? await computeObservedSpends(db, policies);
for (const [index, policy] of policies.entries()) {
const observed = observations[index];
if (observedBlocks(policy, observed)) {
await createIncidentIfNeeded(policy, "hard", observed);
await pauseAndCancelScopeForBudget(policy, !scope.budgetHeld);
const title = scopeType === "company" ? "Company" : scopeType === "agent" ? "Agent" : "Project";
if (observed.pendingRunCount > 0) {
return { scopeType, scopeId, scopeName: scope.name, reason: `${title} cannot start work while completed runs await accounting.` };
}
if (unpricedUsageBlocks(policy, observed)) {
return { scopeType, scopeId, scopeName: scope.name, reason: `${title} cannot start work because recorded usage has no reliable price.` };
}
return { scopeType, scopeId, scopeName: scope.name, reason: scopeType === "project"
? "Project cannot start work because its budget hard-stop is still exceeded."
: `${title} is paused because its budget hard-stop was reached.` };
}
// An open native coordinator must finish the original run's accounting.
// Deny fresh work without pausing/cancelling that recovery or its hold.
if (policy.isActive && policy.hardStopEnabled && policy.amount > 0 && observed.recoveringRunCount > 0) recoveringPolicy ??= policy;
// A closed UTC window, inactive policy, or budget raise releases only
// budget-owned pauses. Manual pauses remain operator decisions.
await resolveOpenIncidentsForPolicy(policy.id, policy.companyId, "approved", "budget_service", "hard");
if (shouldResolveSoftIncident(policy, observed)) {
await resolveOpenIncidentsForPolicy(policy.id, policy.companyId, "approved", "budget_service", "soft");
}
}
const pauseBlock = scopePauseBlock(scopeType, scopeId, scope, policies.length > 0);
if (pauseBlock) return pauseBlock;
if (scope.budgetHeld && policies[0]) await resumeScopeFromBudget(policies[0]);
if (recoveringPolicy) return { scopeType, scopeId, scopeName: scope.name,
reason: "New work must wait while a native run recovers its unfinished accounting." };
return null;
}
async function evaluateAffectedScopes(companyId: string, agentId: string | null, projectId: string | null) {
const candidatePolicies = await db
.select()
.from(budgetPolicies)
.where(
and(
eq(budgetPolicies.companyId, companyId),
eq(budgetPolicies.isActive, true),
budgetPoliciesForRun(companyId, agentId, projectId),
),
);
const policies = candidatePolicies.filter(policy => policy.metric === "billed_cents" && policy.amount > 0);
const observations = await computeObservedSpends(db, policies);
for (const [index, policy] of policies.entries()) {
const observed = observations[index];
const observedAmount = observed.total;
const softThreshold = Math.ceil((policy.amount * policy.warnPercent) / 100);
if (policy.notifyEnabled && compareCents(observed.totalExact, softThreshold) >= 0 && (!policy.hardStopEnabled || compareCents(observed.totalExact, policy.amount) < 0)) {
const softIncident = await createIncidentIfNeeded(policy, "soft", observed);
if (softIncident?.created) {
await recordActivity({
companyId: policy.companyId,
actorType: "system",
actorId: "budget_service",
action: "budget.soft_threshold_crossed",
entityType: "budget_incident",
entityId: softIncident.incident.id,
details: {
scopeType: policy.scopeType,
scopeId: policy.scopeId,
amountObserved: observedAmount,
amountLimit: policy.amount,
},
});
}
}
if (observedBlocks(policy, observed)) {
await resolveOpenSoftIncidents(policy.id, policy.companyId);
const hardIncident = await createIncidentIfNeeded(policy, "hard", observed);
await pauseAndCancelScopeForBudget(policy);
if (hardIncident?.created) {
await recordActivity({
companyId: policy.companyId,
actorType: "system",
actorId: "budget_service",
action: "budget.hard_threshold_crossed",
entityType: "budget_incident",
entityId: hardIncident.incident.id,
details: {
scopeType: policy.scopeType,
scopeId: policy.scopeId,
amountObserved: observedAmount,
amountLimit: policy.amount,
approvalId: hardIncident.incident.approvalId ?? null,
},
});
}
}
}
// A final receipt or price correction can reduce a partial observation.
// Release only this scope's budget-owned hold after all its policies agree.
const healthyScopes = new Map<string, PolicyRow>();
for (const [index, policy] of policies.entries()) {
if (!observedBlocks(policy, observations[index])) healthyScopes.set(`${policy.scopeType}:${policy.scopeId}`, policy);
}
for (const policy of healthyScopes.values()) {
const scope = await resolveScopeRecord(db, policy.scopeType as BudgetScopeType, policy.scopeId);
if (scope.budgetHeld) await resumeScopeFromBudget(policy);
}
}
async function evaluateAdmissionInTransaction(companyId: string, agentId: string | null,
context?: { issueId?: string | null; projectId?: string | null }) {
const { scopes, records, policies } = await readAdmissionScopes(db, companyId, agentId, context);
const observations = await computeObservedSpends(db, policies);
const reservationIndexes = policies.map((policy, i) => policy.isActive && policy.hardStopEnabled && policy.metric === "billed_cents" && policy.amount > 0 ? i : -1).filter(i => i >= 0);
const admission = (block: Awaited<ReturnType<typeof reconcileScope>>) => ({ block,
policies: reservationIndexes.map(i => policies[i]), observations: reservationIndexes.map(i => observations[i]) });
for (const [index, { scopeType, scopeId }] of scopes.entries()) {
const indexes = policies.map((policy, i) => policy.scopeType === scopeType && policy.scopeId === scopeId ? i : -1).filter(i => i >= 0);
const block = await reconcileScope(companyId, scopeType, scopeId, { scope: records[index],
policies: indexes.map(i => policies[i]), observations: indexes.map(i => observations[i]) });
if (block) return admission(block);
}
return admission(null);
}
const evaluateAdmission = (...args: Parameters<typeof evaluateAdmissionInTransaction>) =>
withBudgetSpan("budget.admission.evaluate", () => evaluateAdmissionInTransaction(...args));
return {
reconcileScope,
listPolicies: async (companyId: string): Promise<BudgetPolicy[]> => {
const rows = await listPolicyRows(companyId);
return rows.map((row) => ({
...row,
scopeType: row.scopeType as BudgetScopeType,
unpricedUsagePolicy: row.unpricedUsagePolicy as "block" | "allow",
metric: row.metric as BudgetMetric,
windowKind: row.windowKind as BudgetWindowKind,
}));
},
upsertPolicy: async (
companyId: string,
input: BudgetPolicyUpsertInput,
actorUserId: string | null,
): Promise<BudgetPolicySummary> => {
const scope = await resolveScopeRecord(db, input.scopeType, input.scopeId);
if (scope.companyId !== companyId) {
throw unprocessable("Budget scope does not belong to company");
}
if (input.reservationCents !== undefined && compareCents(input.reservationCents, 0) < 0) throw unprocessable("Reservation cannot be negative");
const metric = input.metric ?? "billed_cents";
const windowKind = input.windowKind ?? (input.scopeType === "project" ? "lifetime" : "calendar_month_utc");
const existing = await db
.select()
.from(budgetPolicies)
.where(
and(
eq(budgetPolicies.companyId, companyId),
eq(budgetPolicies.scopeType, input.scopeType),
eq(budgetPolicies.scopeId, input.scopeId),
eq(budgetPolicies.metric, metric),
eq(budgetPolicies.windowKind, windowKind),
),
)
.then((rows) => rows[0] ?? null);
if (!existing && input.amount === undefined) throw unprocessable("Amount is required for a new budget policy");
const amount = Math.max(0, Math.floor(input.amount ?? existing!.amount));
// A saved zero disables enforcement. Raising it is an explicit limit
// setting; editing a disabled positive policy must preserve that choice.
const nextIsActive = amount > 0 && (input.isActive ?? (existing?.amount === 0 ? true : existing?.isActive) ?? true);
const now = new Date();
await notifyDeliveryWork(db, DELIVERY_QUEUES.budgetEnforcement, companyId);
const row = existing
? await db
.update(budgetPolicies)
.set({
amount,
reservationCents: normalizeCents(input.reservationCents ?? existing.reservationCents ?? 0),
warnPercent: input.warnPercent ?? existing.warnPercent,
hardStopEnabled: input.hardStopEnabled ?? existing.hardStopEnabled,
notifyEnabled: input.notifyEnabled ?? existing.notifyEnabled,
unpricedUsagePolicy: input.unpricedUsagePolicy ?? existing.unpricedUsagePolicy,
isActive: nextIsActive,
updatedByUserId: actorUserId,
updatedAt: now,
enforcementVersion: sql`${budgetPolicies.enforcementVersion} + 1`,
})
.where(eq(budgetPolicies.id, existing.id))
.returning()
.then((rows) => rows[0])
: await db
.insert(budgetPolicies)
.values({
companyId,
scopeType: input.scopeType,
scopeId: input.scopeId,
metric,
windowKind,
amount,
reservationCents: normalizeCents(input.reservationCents ?? 0),
warnPercent: input.warnPercent ?? 80,
hardStopEnabled: input.hardStopEnabled ?? true,
notifyEnabled: input.notifyEnabled ?? true,
unpricedUsagePolicy: input.unpricedUsagePolicy ?? "block",
isActive: nextIsActive,
createdByUserId: actorUserId,
updatedByUserId: actorUserId,
})
.returning()
.then((rows) => rows[0]);
if (input.scopeType === "company" && windowKind === "calendar_month_utc") {
await db
.update(companies)
.set({
budgetMonthlyCents: amount,
updatedAt: now,
})
.where(eq(companies.id, input.scopeId));
}
if (input.scopeType === "agent" && windowKind === "calendar_month_utc") {
await db
.update(agents)
.set({
budgetMonthlyCents: amount,
updatedAt: now,
})
.where(eq(agents.id, input.scopeId));
}
if (row.isActive && amount > 0) {
const observed = await computeObservedSpend(db, row);
if (!observedBlocks(row, observed)) {
await resumeScopeFromBudget(row);
await resolveOpenIncidentsForPolicy(row.id, row.companyId, "approved", actorUserId ?? "budget_service", "hard");
if (row.notifyEnabled && compareCents(observed.totalExact, Math.ceil(row.amount * row.warnPercent / 100)) >= 0) {
await createIncidentIfNeeded(row, "soft", observed);
} else {
await resolveOpenIncidentsForPolicy(row.id, row.companyId, null, null, "soft");
}
} else {
const softThreshold = Math.ceil((row.amount * row.warnPercent) / 100);
if (row.notifyEnabled && compareCents(observed.totalExact, softThreshold) >= 0 && (!row.hardStopEnabled || compareCents(observed.totalExact, row.amount) < 0)) {
await createIncidentIfNeeded(row, "soft", observed);
}
if (observedBlocks(row, observed)) {
await resolveOpenSoftIncidents(row.id, row.companyId);
await createIncidentIfNeeded(row, "hard", observed);
await pauseAndCancelScopeForBudget(row);
}
}
} else {
await resumeScopeFromBudget(row);
await resolveOpenIncidentsForPolicy(row.id, row.companyId, "approved", actorUserId ?? "budget_service");
}
await recordActivity({
companyId,
actorType: "user",
actorId: actorUserId ?? "board",
action: "budget.policy_upserted",
entityType: "budget_policy",
entityId: row.id,
details: {
scopeType: row.scopeType,
scopeId: row.scopeId,
amount: row.amount,
windowKind: row.windowKind,
},
});
return buildPolicySummary(row);
},
overview: async (companyId: string): Promise<BudgetOverview> => {
const rows = await listPolicyRows(companyId);
const policies = await mapBudgetReads(rows, buildPolicySummary);
const activeIncidentRows = await db
.select()
.from(budgetIncidents)
.where(and(eq(budgetIncidents.companyId, companyId), eq(budgetIncidents.status, "open")))
.orderBy(desc(budgetIncidents.createdAt));
const activeIncidents = await hydrateIncidentRows(activeIncidentRows);
return {
companyId,
policies,
activeIncidents,
pausedAgentCount: new Set(policies.filter((policy) => policy.scopeType === "agent" && policy.paused).map((policy) => policy.scopeId)).size,
pausedProjectCount: new Set(policies.filter((policy) => policy.scopeType === "project" && policy.paused).map((policy) => policy.scopeId)).size,
pendingApprovalCount: activeIncidents.filter((incident) => incident.approvalStatus === "pending").length,
};
},
evaluateAffectedScopes,
evaluateCostEvent: (event: typeof costEvents.$inferSelect) => evaluateAffectedScopes(event.companyId, event.agentId, event.projectId),
evaluateAdmission,
getInvocationBlock: async (companyId: string, agentId: string | null, context?: { issueId?: string | null; projectId?: string | null }) =>
(await evaluateAdmission(companyId, agentId, context)).block,
resolveIncident: async (
companyId: string,
incidentId: string,
input: BudgetIncidentResolutionInput,
actorUserId: string,
): Promise<BudgetIncident> => {
const incident = await db
.select()
.from(budgetIncidents)
.where(eq(budgetIncidents.id, incidentId))
.then((rows) => rows[0] ?? null);
if (!incident) throw notFound("Budget incident not found");
if (incident.companyId !== companyId) throw notFound("Budget incident not found");
const policy = await getPolicyRow(incident.policyId);
if (policy.companyId !== companyId || policy.scopeType !== incident.scopeType || policy.scopeId !== incident.scopeId) {
throw notFound("Budget scope not found");
}
const scope = input.action === "keep_paused"
? await resolveSummaryScopeRecord(db, companyId, policy.scopeType as BudgetScopeType, policy.scopeId)
: await resolveScopeRecord(db, policy.scopeType as BudgetScopeType, policy.scopeId);
if (scope.companyId !== companyId) throw notFound("Budget scope not found");
if (input.action === "raise_budget_and_resume") {
const nextAmount = Math.max(0, Math.floor(input.amount ?? 0));
const observed = await computeObservedSpend(db, policy);
if (unpricedUsageBlocks(policy, observed)) {
throw unprocessable("Resolve unpriced usage or explicitly allow it in the budget policy before resuming");
}
if (observed.pendingRunCount > 0) throw unprocessable("Completed runs must finish accounting before resuming");
if (compareCents(observed.totalExact, nextAmount) >= 0) {
throw unprocessable("New budget must exceed current observed spend");
}
const now = new Date();
await notifyAttentionRetentionWork(db, { companyId, sourceKind: "budget_alert" });
await notifyDeliveryWork(db, DELIVERY_QUEUES.budgetEnforcement, companyId);
await db
.update(budgetPolicies)
.set({
amount: nextAmount,
isActive: true,
updatedByUserId: actorUserId,
updatedAt: now,
enforcementVersion: sql`${budgetPolicies.enforcementVersion} + 1`,
})
.where(eq(budgetPolicies.id, policy.id));
if (policy.scopeType === "company" && policy.windowKind === "calendar_month_utc") {
await db
.update(companies)
.set({ budgetMonthlyCents: nextAmount, updatedAt: now })
.where(eq(companies.id, policy.scopeId));
}
if (policy.scopeType === "agent" && policy.windowKind === "calendar_month_utc") {
await db
.update(agents)
.set({ budgetMonthlyCents: nextAmount, updatedAt: now })
.where(eq(agents.id, policy.scopeId));
}
await resumeScopeFromBudget(policy);
await db
.update(budgetIncidents)
.set({
status: "resolved",
resolvedAt: now,
updatedAt: now,
})
.where(and(eq(budgetIncidents.policyId, policy.id), eq(budgetIncidents.status, "open")));
await markApprovalStatus(db, incident.approvalId ?? null, "approved", input.decisionNote, actorUserId);
} else {
await db
.update(budgetIncidents)
.set({
status: "dismissed",
resolvedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(budgetIncidents.id, incident.id));
await markApprovalStatus(db, incident.approvalId ?? null, "rejected", input.decisionNote, actorUserId);
}
await recordActivity({
companyId: incident.companyId,
actorType: "user",
actorId: actorUserId,
action: "budget.incident_resolved",
entityType: "budget_incident",
entityId: incident.id,
details: {
action: input.action,
amount: input.amount ?? null,
scopeType: incident.scopeType,
scopeId: incident.scopeId,
},
});
const [updated] = await hydrateIncidentRows([{
...incident,
status: input.action === "raise_budget_and_resume" ? "resolved" : "dismissed",
resolvedAt: new Date(),
updatedAt: new Date(),
}]);
return updated!;
},
};
}
/** Fence the durable stop intent against policy edits and admission. External
* process shutdown must happen after this transaction, without holding locks
* that the executor needs to persist its final usage receipt. */
export async function withCurrentBudgetEnforcement<T>(db: Db, scope: BudgetEnforcementScope, work: (tx: Db) => Promise<T>): Promise<T | null> {
return withAccountingTransaction(db, scope.companyId, async (tx) => {
if (scope.enforcement) {
const [policy] = await tx.select().from(budgetPolicies).where(and(
eq(budgetPolicies.id, scope.enforcement.policyId), eq(budgetPolicies.companyId, scope.companyId),
));
if (!policy || policy.scopeType !== scope.scopeType || policy.scopeId !== scope.scopeId
|| policy.enforcementVersion !== scope.enforcement.version || !(await policyBlocks(tx, policy))) return null;
}
return work(tx);
});
}
async function deliverScopes<T>(items: T[], deliver: (item: T) => Promise<void>) {
let next = 0;
await Promise.all(Array.from({ length: Math.min(4, items.length) }, async () => {
while (next < items.length) await deliver(items[next++]);
}));
}
/** Cancellation is an at-least-once external effect. Only the delivery worker
* invokes this: request paths commit intent and return without awaiting it. */
export async function deliverBudgetEnforcement(db: Db, hooks: BudgetServiceHooks, companyId?: string, _agentId?: string | null) {
if (!hooks.cancelWorkForScope) return;
const attempted = new Set<string>();
const scopeKey = (scope: BudgetEnforcementScope) => `${scope.companyId}:${scope.scopeType}:${scope.scopeId}`;
const pending = await db.select().from(budgetPolicies).where(and(
sql`${budgetPolicies.enforcementVersion} > ${budgetPolicies.enforcementDeliveredVersion}`,
companyId ? eq(budgetPolicies.companyId, companyId) : undefined,
)).orderBy(budgetPolicies.updatedAt, budgetPolicies.id).limit(100);
await deliverScopes(pending, policy => withBudgetSpan("budget.enforcement.scope", async () => {
try {
const cancellation = await withAccountingTransaction(db, policy.companyId, async tx => {
const [current] = await tx.select().from(budgetPolicies).where(eq(budgetPolicies.id, policy.id));
if (!current || current.enforcementVersion !== policy.enforcementVersion || !(await policyBlocks(tx, current))) return null;
return { companyId: policy.companyId, scopeType: policy.scopeType as BudgetScopeType, scopeId: policy.scopeId,
createdBefore: new Date(), enforcement: { policyId: policy.id, version: policy.enforcementVersion } };
});
if (cancellation) {
attempted.add(scopeKey(cancellation));
await hooks.cancelWorkForScope!(cancellation);
}
await db.update(budgetPolicies).set({ enforcementDeliveredVersion: policy.enforcementVersion })
.where(and(eq(budgetPolicies.id, policy.id), eq(budgetPolicies.enforcementVersion, policy.enforcementVersion)));
} catch (error) {
logger.warn({ err: error, policyId: policy.id }, "Budget cancellation delivery pending; accounting is committed");
// Rotate a failed scope behind other pending work. A poisoned delivery
// must not monopolize the bounded batch on every retry.
await db.update(budgetPolicies).set({ updatedAt: new Date() }).where(and(eq(budgetPolicies.id, policy.id),
eq(budgetPolicies.enforcementVersion, policy.enforcementVersion))).catch(() => undefined);
}
}, { "budget.enforcement.queue_delay_ms": Math.max(0, Date.now() - policy.updatedAt.getTime()) }));
// A committed run Stop outlives a policy edit or deletion. It remains owed
// even if no policy generation now authorizes a fresh cancellation.
const committed = (await listPendingBudgetCancellationScopes(db))
.filter(scope => (!companyId || scope.companyId === companyId) && !attempted.has(scopeKey(scope)));
await deliverScopes(committed, async scope => {
try { await hooks.cancelWorkForScope!(scope); }
catch (error) { logger.warn({ err: error, companyId: scope.companyId, scopeType: scope.scopeType, scopeId: scope.scopeId }, "Committed budget Stop remains pending"); }
});
}
export async function hasPendingBudgetEnforcement(db: Db): Promise<boolean> {
const [pending] = await db.select({ id: budgetPolicies.id }).from(budgetPolicies)
.where(sql`${budgetPolicies.enforcementVersion} > ${budgetPolicies.enforcementDeliveredVersion}`).limit(1);
return Boolean(pending) || (await listPendingBudgetCancellationScopes(db, 1)).length > 0;
}
const budgetOwnedScope = or(
and(eq(budgetPolicies.scopeType, "company"), sql`exists (select 1 from ${companies} where ${companies.id} = ${budgetPolicies.scopeId} and ${companies.pauseReason} = 'budget')`),
and(eq(budgetPolicies.scopeType, "agent"), sql`exists (select 1 from ${agents} where ${agents.id} = ${budgetPolicies.scopeId} and ${agents.lifecycleHolds} @> '["budget"]'::jsonb)`),
and(eq(budgetPolicies.scopeType, "project"), sql`exists (select 1 from ${projects} where ${projects.id} = ${budgetPolicies.scopeId} and ${projects.pauseReason} = 'budget')`),
)!;
const openIncidentScope = sql`exists (select 1 from ${budgetIncidents}
where ${budgetIncidents.policyId} = ${budgetPolicies.id} and ${budgetIncidents.status} = 'open')`;
const unfinishedRunScope = sql`exists (select 1 from ${heartbeatRuns}
where ${heartbeatRuns.companyId} = ${budgetPolicies.companyId} and ${heartbeatRuns.costAccountingPending} = true
and ${heartbeatRuns.costAccountedAt} is null and (${budgetPolicies.scopeType} = 'company'
or (${budgetPolicies.scopeType} = 'agent' and ${heartbeatRuns.agentId} = ${budgetPolicies.scopeId})
or (${budgetPolicies.scopeType} = 'project' and ${heartbeatRuns.usageJson}->'ledgerScope'->>'projectId' = ${budgetPolicies.scopeId}::text)))`;
export async function nextBudgetReconciliationAt(db: Db): Promise<Date | null> {
const [held] = await db.select({ id: budgetPolicies.id }).from(budgetPolicies)
.where(and(eq(budgetPolicies.windowKind, "calendar_month_utc"), or(budgetOwnedScope, openIncidentScope))).limit(1);
return held ? currentUtcMonthWindow().end : null;
}
/** Startup repairs configured policy scopes, not every agent. Afterwards only
* held monthly scopes owe a timed reevaluation; writes reconcile their own scopes. */
export async function reconcileDueBudgetPolicies(db: Db, options: { startup?: boolean; cursor?: string | null } = {}): Promise<{ nextCursor: string | null }> {
const actionable = options.startup ? or(budgetOwnedScope, openIncidentScope,
sql`${budgetPolicies.enforcementVersion} > ${budgetPolicies.enforcementDeliveredVersion}`,
and(eq(budgetPolicies.isActive, true), eq(budgetPolicies.hardStopEnabled, true), sql`${budgetPolicies.amount} > 0`, unfinishedRunScope))
: and(eq(budgetPolicies.windowKind, "calendar_month_utc"), or(budgetOwnedScope, openIncidentScope));
const rows = await db.select({ id: budgetPolicies.id, companyId: budgetPolicies.companyId, scopeType: budgetPolicies.scopeType, scopeId: budgetPolicies.scopeId })
.from(budgetPolicies).where(and(actionable, options.cursor ? sql`${budgetPolicies.id} > ${options.cursor}::uuid` : undefined))
.orderBy(budgetPolicies.id).limit(100);
const scopes = new Map(rows.map(scope => [`${scope.companyId}:${scope.scopeType}:${scope.scopeId}`, scope]));
for (const scope of scopes.values()) {
try {
await withAccountingTransaction(db, scope.companyId, (tx, publications) => budgetServiceInTransaction(tx, publications)
.reconcileScope(scope.companyId, scope.scopeType as BudgetScopeType, scope.scopeId));
} catch (error) {
if (error instanceof HttpError && error.status === 404) continue;
throw error;
}
}
return { nextCursor: rows.length === 100 ? rows[rows.length - 1].id : null };
}
export function budgetService(db: Db, _hooks: BudgetServiceHooks = {}) {
const reads = budgetServiceInTransaction(db);
const mutate = <T>(companyId: string, work: (service: ReturnType<typeof budgetServiceInTransaction>) => Promise<T>) =>
withAccountingTransaction(db, companyId, (tx, publications) => work(budgetServiceInTransaction(tx, publications)));
return {
...reads,
evaluateAdmission: (companyId: string, agentId: string | null, context?: { issueId?: string | null; projectId?: string | null }) =>
mutate(companyId, service => service.evaluateAdmission(companyId, agentId, context)),
evaluateAffectedScopes: (companyId: string, agentId: string | null, projectId: string | null) =>
mutate(companyId, service => service.evaluateAffectedScopes(companyId, agentId, projectId)),
reconcileScope: (companyId: string, scopeType: BudgetScopeType, scopeId: string) =>
mutate(companyId, service => service.reconcileScope(companyId, scopeType, scopeId)),
reconcilePolicies: () => reconcileDueBudgetPolicies(db, { startup: true }),
upsertPolicy: (companyId: string, input: BudgetPolicyUpsertInput, actorUserId: string | null) =>
mutate(companyId, service => service.upsertPolicy(companyId, input, actorUserId)),
evaluateCostEvent: (event: typeof costEvents.$inferSelect) => mutate(event.companyId, service => service.evaluateCostEvent(event)),
getInvocationBlock: async (companyId: string, agentId: string | null, context?: AdmissionContext) => {
const eligibility = await withBudgetSpan("budget.admission.inspect", () =>
withAccountingReadSnapshot(db, companyId, tx => inspectInvocationEligibility(tx, companyId, agentId, context)));
if (!eligibility.needsReconciliation) return eligibility.block;
return withBudgetSpan("budget.admission.reconcile", () =>
mutate(companyId, service => service.getInvocationBlock(companyId, agentId, context)));
},
resolveIncident: (companyId: string, incidentId: string, input: BudgetIncidentResolutionInput, actorUserId: string) =>
mutate(companyId, service => service.resolveIncident(companyId, incidentId, input, actorUserId)),
};
}