mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
fix(budgets): bound the accounting hold so one missing receipt cannot wedge a company (#19700)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Pending run accounting holds new company work until it settles (#19497, #19606, #19673) > - A run settles only with closure evidence: a verified local stop, or a terminated remote sandbox. Without that evidence it held the whole company forever. In restart-chaos storm 4, a single run held 35 tasks for hours > - Two more shapes kept the hold alive. A run whose result had already been accepted never counted as closed. A recovery that already owned its held reservation was refused by other tasks' recovering accounting, so concurrent recoveries waited on each other > - This pull request bounds the hold, closes accepted-result runs on a verified stop, and lets an owned recovery finish > - The benefit is that no single missing receipt can wedge a company, while genuinely unknown spend still surfaces under the blocking policy ## Linked Issues or Issue Description Follow-up to #19497, #19606 and #19673. **What happened?** - Storm 4: run `f879895f` timed out in `session.open`. Its local stop was never verified (its process identity is still recorded), recovery ended in `terminal_failure`, and its receipt never became ready. From then on, every new-work wake got 409 "New work must wait while a native run recovers its unfinished accounting". That was 93 refusals, 35 blocked tasks, and no Retry. - Storm 4: run `6d3c76bf` was resuming as the recovery owner with its own held reservation. `reserveRunBudget` refused it because other tasks' runs were recovering. - Wave s7401: run `0a3a849d` failed after its result was accepted, with a verified local stop. It had 10 receipts across 2 sources, all incomplete with 0 tokens. It stayed "completed runs await accounting" because closure only covered coordinators in terminal failure with no result. **What did you expect?** A run without closure evidence holds new work only for a bounded time, then settles visibly. An accepted-result run with a verified stop closes like a terminal failure. A recovery that already owns admitted work can finish. **Steps to reproduce** 1. Restart the control plane during several native turns so that one run's local stop is never verified. 2. Observe the run stay "recovering" and every new wake refused indefinitely. ## What Changed - `run-cost-accounting.ts`: - A coordinator counts as settled when it ended in terminal failure without a result, or when it has an accepted result. The verified-stop and remote-termination closures use that rule, which covers the s7401 shape. - A native run whose provider never started settles at once, like a pre-provider failure, with no charge and its reservation released. Its recovery has settled, no usage receipt was ever journaled (a provider turn journals one as soon as it starts), and its local process never launched or was verified stopped (wave s7403, run 3c1d7b70, `session.open` timeout). - New bound `ACCOUNTING_CLOSURE_TIMEOUT_MS`: the remote runner reconnect grace (7 minutes) plus 3 minutes. A terminal native run with a settled coordinator and no closure evidence settles at the bound: - its known usage is recorded as unpriced (or estimated from the rate card), with provenance evidence "closure evidence never arrived…"; - it is marked `accountingClosureTimedOut`; - a `cost.accounting_closure_timed_out` activity entry names the run. - The reconciler selects these runs, and `readCostAccountingWork` schedules a wake at the bound. - `budget-exposure.ts`: a timed-out closure's final charge is exempt from none of the unpriced-block exemptions. Under the `block` policy it raises the normal budget notice ("recorded usage has no reliable price") instead of a silent hold. - `budgets.ts`, `budget-reservations.ts`, `run-dispatch/adapters/postgres.ts`: when a validated recovery owner continues admitted work, admission and both dispatch rechecks apply only hard stops and pauses. Pending and recovering accounting holds still gate new work. - `doc/SPEC-implementation.md`: documents the bound, the accepted-result closure and the provenance. ## Verification - `closed-native-receipt-accounting.test.ts` (embedded Postgres). All of these fail on master and pass here: - s7401 shape (accepted result, verified stop, all receipts incomplete with 0 tokens): settles and clears admission. - A run without closure evidence: held before the bound with a scheduled wake, then settled with the "closure evidence never arrived" provenance, the `accountingClosureTimedOut` flag and an activity entry. Admission clears under `allow`. - Storm 4 `f879895f` shape (terminal failure, recorded process identity, no stop event): held, then released after the bound. - The blocking policy still blocks on a timed-out closure. - s7403 shape (no receipts, model unknown): a run whose process never launched, or was verified stopped, settles at once with no charge and its reservation released. With an unverified process it waits for the bound. - Storm 4 `6d3c76bf` shape: a recovery owner with its held reservation reserves past another task's recovering accounting, while new work for another task still waits. - Existing suites pass locally: budget-enforcement-scoping, budget-enforcement-worker, native-stop-accounting, cost-data-quality, heartbeat-cost-accounting, run-cost-accounting, budgets-service. Server typecheck passes. - The other storm 4 terminal-failure runs (`2e477ae9`, `e4d2e14d`, `6d3c76bf`, `7d21a44c`, `4bc49df0`, `1d85fbd9`) were already settled by #19673 in that database (`cost_accounting_pending=false`) and were not holding admission. `f879895f` alone held it. ## Risks - After the bound, a run whose provider might still be running settles as unpriced. Under the `block` policy that raises a hard-stop budget notice, by design. Under `allow`, work continues and a late receipt still reconciles. - A recovery owner skips accounting holds, so concurrent recoveries can spend while other tasks' accounting is unknown. Hard stops still apply, consistent with best-effort budget enforcement. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
4612fa9fb4
commit
8a34c1b62c
8 files changed
+344
-31
No files matched your search
@@ -1884,7 +1884,8 @@ to block budget admission.
|
||||
After native recovery reaches terminal failure, an incomplete receipt holds new
|
||||
work as recovering accounting; it does not open a hard-stop incident. When the
|
||||
run no longer records a provider process (recovery clears it only after a
|
||||
verified stop), no producer can deliver the receipt. A remote run reaches the
|
||||
verified stop), no producer can deliver the receipt. The same holds after the
|
||||
run's result was accepted. A remote run reaches the
|
||||
same point when every sandbox allocation it used carries a provider
|
||||
termination receipt (stopped or destroyed); a sandbox that may still run keeps
|
||||
the debt pending. Accounting then records
|
||||
@@ -1947,8 +1948,12 @@ prices require no cleanup and must not be presented as zero-cost receipts.
|
||||
board cost reader with a non-viewer membership, and uses the existing audited,
|
||||
idempotent accounting service. It never executes an agent. The UI offers retry
|
||||
only when the saved run is eligible to record costs. Missing final usage changes
|
||||
its display label after fifteen minutes; elapsed time never settles the debt or
|
||||
discards a late receipt. Raw provider errors and credentials are excluded from
|
||||
its display label after fifteen minutes. Elapsed time never discards a late
|
||||
receipt. It settles the debt only for a native run whose recovery has ended
|
||||
(terminal failure or an accepted result) and whose closure evidence has not
|
||||
arrived after ten minutes (the remote runner reconnect grace plus a margin): the known usage is recorded as unpriced with
|
||||
provenance saying closure evidence never arrived, an activity entry names the
|
||||
run, and the blocking unpriced-usage policy keeps counting that charge. Raw provider errors and credentials are excluded from
|
||||
cost details. ACP usage retains the runtime-selected model, including defaults;
|
||||
conflicting before/after model IDs do not authorize a single-model estimate.
|
||||
Remote OpenCode billing uses only credentials forwarded to the execution target.
|
||||
|
||||
@@ -4,11 +4,11 @@ import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
agents, budgetIncidents, budgetPolicies, budgetReservations, companies, costEvents, createDb, environmentLeases, environments,
|
||||
heartbeatRunEvents, heartbeatRuns, issues,
|
||||
nativeRunFinalizations, type Db,
|
||||
activityLog, completionContracts, nativeRunFinalizations, nativeRunResults, type Db,
|
||||
} from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { budgetService, flushBudgetEvaluationWork, recheckAdmittedRunBudgetReadOnly } from "../services/budgets.js";
|
||||
import { accountRunCost, reconcileRunCosts } from "../services/run-cost-accounting.js";
|
||||
import { accountRunCost, readCostAccountingWork, reconcileRunCosts } from "../services/run-cost-accounting.js";
|
||||
import { persistUsageReceipt } from "../services/usage-receipts.js";
|
||||
import { recheckRunBudgetReservationInTransaction, reserveRunBudget } from "../services/budget-reservations.js";
|
||||
|
||||
@@ -197,6 +197,57 @@ const support = await getEmbeddedPostgresTestSupport();
|
||||
await expect(reserveRunBudget(db, f.companyId, other, null)).rejects.toThrow(/recovers its unfinished accounting/);
|
||||
});
|
||||
|
||||
// Storm 4 (run 6d3c76bf): after a restart storm several tasks recover at once.
|
||||
// A recovery that already owns its run's held reservation was refused by
|
||||
// another task's recovering accounting, so no recovery could finish.
|
||||
it("lets a recovery that owns its held reservation continue past other tasks' accounting holds", async () => {
|
||||
const f = await fixture({ phase: "observed" });
|
||||
const [otherIssue] = await db.insert(issues).values({ companyId: f.companyId, title: "Recovering task", status: "in_progress",
|
||||
assigneeAgentId: f.otherAgentId }).returning();
|
||||
const recovering = randomUUID(), owner = `recovery:${randomUUID()}`;
|
||||
await db.insert(heartbeatRuns).values({ id: recovering, companyId: f.companyId, agentId: f.otherAgentId, status: "running",
|
||||
runtimeMode: "native", scopeKind: "issue", issueId: otherIssue.id, nativeIssueId: otherIssue.id });
|
||||
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: otherIssue.id, runId: recovering,
|
||||
phase: "observed", attempt: 2, leaseOwner: owner, leaseExpiresAt: new Date(Date.now() + 60_000) });
|
||||
await db.insert(budgetReservations).values({ companyId: f.companyId, runId: recovering, agentId: f.otherAgentId,
|
||||
amountCents: "0", state: "held", providerStartedAt: new Date() });
|
||||
expect(await f.block()).toMatchObject({ reason: "New work must wait while a native run recovers its unfinished accounting." });
|
||||
await expect(reserveRunBudget(db, f.companyId, recovering, null, {}, owner)).resolves.toMatchObject({ reused: true });
|
||||
// New work for another task still waits.
|
||||
const fresh = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: fresh, companyId: f.companyId, agentId: f.otherAgentId, status: "queued",
|
||||
runtimeMode: "native", scopeKind: "issue", issueId: otherIssue.id, nativeIssueId: null });
|
||||
await expect(reserveRunBudget(db, f.companyId, fresh, null)).rejects.toThrow(/recovers its unfinished accounting/);
|
||||
});
|
||||
|
||||
it("lets an owned recovery dispatch when other runs' held reservations fill the budget", async () => {
|
||||
const f = await fixture({ phase: "observed" });
|
||||
const [otherIssue] = await db.insert(issues).values({ companyId: f.companyId, title: "Recovering task", status: "in_progress",
|
||||
assigneeAgentId: f.otherAgentId }).returning();
|
||||
const recovering = randomUUID(), busy = randomUUID(), owner = `recovery:${randomUUID()}`;
|
||||
await db.insert(heartbeatRuns).values([
|
||||
{ id: recovering, companyId: f.companyId, agentId: f.otherAgentId, status: "running",
|
||||
runtimeMode: "native", scopeKind: "issue", issueId: otherIssue.id, nativeIssueId: otherIssue.id },
|
||||
{ id: busy, companyId: f.companyId, agentId: f.otherAgentId, status: "running" },
|
||||
]);
|
||||
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: otherIssue.id, runId: recovering,
|
||||
phase: "observed", attempt: 2, leaseOwner: owner, leaseExpiresAt: new Date(Date.now() + 60_000) });
|
||||
// Held reservations total 600 of the 500-cent budget.
|
||||
const [own] = await db.insert(budgetReservations).values([
|
||||
{ companyId: f.companyId, runId: recovering, agentId: f.otherAgentId, amountCents: "300", state: "held", providerStartedAt: new Date() },
|
||||
{ companyId: f.companyId, runId: busy, agentId: f.otherAgentId, amountCents: "300", state: "held", providerStartedAt: new Date() },
|
||||
]).returning();
|
||||
await expect(reserveRunBudget(db, f.companyId, recovering, null, {}, owner)).resolves.toMatchObject({ reused: true });
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, recovering));
|
||||
expect(await db.transaction(tx => recheckRunBudgetReservationInTransaction(tx as unknown as typeof db, [],
|
||||
run, own.id, owner))).toBeNull();
|
||||
expect(await recheckAdmittedRunBudgetReadOnly(db, { companyId: f.companyId, agentId: f.otherAgentId, runId: recovering,
|
||||
projectId: null, reservationId: own.id, admittedRecovery: true })).toBeNull();
|
||||
// Without the recovery ownership, the same dispatch is refused.
|
||||
expect(await recheckAdmittedRunBudgetReadOnly(db, { companyId: f.companyId, agentId: f.otherAgentId, runId: recovering,
|
||||
projectId: null, reservationId: own.id })).not.toBeNull();
|
||||
});
|
||||
|
||||
// RC-06e: unpriced charges with no observed usage, or with a known partial
|
||||
// cost, hard-stopped a company at 120 of 2000 cents.
|
||||
it.each([
|
||||
@@ -375,6 +426,156 @@ const support = await getEmbeddedPostgresTestSupport();
|
||||
expect(await f.charges()).toHaveLength(0);
|
||||
});
|
||||
|
||||
// Wave s7401: a local ACPX Claude run was killed after its result was
|
||||
// accepted. Ten receipts across two sources were all incomplete with zero
|
||||
// tokens, its local stop was verified, and the company refused all new work.
|
||||
async function acceptedResultFixture(options: { stopEvidence?: boolean; finishedMinutesAgo?: number } = {}) {
|
||||
const f = await fixture({ stopEvidence: options.stopEvidence });
|
||||
const contractId = randomUUID();
|
||||
await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issue.id, revision: 1,
|
||||
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "result-worker-v1", risk: "standard",
|
||||
completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
|
||||
contractJson: { revision: "result-worker-v1", objective: "Finish", criteria: [{ id: "objective", requirement: "Complete the task" }] },
|
||||
canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" });
|
||||
await db.update(heartbeatRuns).set({ completionContractId: contractId }).where(eq(heartbeatRuns.id, f.run.id));
|
||||
const [accepted] = await db.insert(nativeRunResults).values({ companyId: f.companyId, issueId: f.issue.id, runId: f.run.id,
|
||||
completionContractId: contractId, serverFingerprint: "s7401", schemaStatus: "accepted", resultJson: {},
|
||||
canonicalSha256: "c".repeat(64) }).returning();
|
||||
await db.update(nativeRunFinalizations).set({ phase: "committed", resultId: accepted.id, failureCode: null })
|
||||
.where(eq(nativeRunFinalizations.runId, f.run.id));
|
||||
await db.update(heartbeatRuns).set({ errorCode: "adapter_failed",
|
||||
finishedAt: new Date(Date.now() - (options.finishedMinutesAgo ?? 0) * 60_000),
|
||||
usageJson: sql`${heartbeatRuns.usageJson} || ${JSON.stringify({ accountingReceiptSourceFences: { [randomUUID()]: 5 },
|
||||
accountingReceiptSequence: 5 })}::jsonb` }).where(eq(heartbeatRuns.id, f.run.id));
|
||||
return f;
|
||||
}
|
||||
|
||||
it("closes a result-accepted run's lost receipt once its local stop is verified", async () => {
|
||||
const f = await acceptedResultFixture();
|
||||
expect(await f.block()).toMatchObject({ reason: "Company cannot start work while completed runs await accounting." });
|
||||
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
|
||||
await flushBudgetEvaluationWork(db);
|
||||
expect(await f.charges()).toMatchObject([{ costStatus: "unpriced" }]);
|
||||
expect(await f.block()).toBeNull();
|
||||
});
|
||||
|
||||
// RC-06g: with no closure evidence at all, one run must not hold a company's
|
||||
// new work forever.
|
||||
it("holds a run without closure evidence only until the bound, then settles it visibly", async () => {
|
||||
const recent = await acceptedResultFixture({ stopEvidence: false, finishedMinutesAgo: 5 });
|
||||
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 0 });
|
||||
expect(await recent.block()).not.toBeNull();
|
||||
// The worker schedules a wake for the bound instead of polling.
|
||||
const work = await readCostAccountingWork(db, Date.now());
|
||||
expect(work.nextRunAt).not.toBeNull();
|
||||
expect(work.nextRunAt!).toBeGreaterThan(Date.now());
|
||||
// One cutoff decides both: past the bound the run is actionable now.
|
||||
const later = await readCostAccountingWork(db, work.nextRunAt! + 1);
|
||||
expect(later.pending).toBe(true);
|
||||
|
||||
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() - 11 * 60_000) }).where(eq(heartbeatRuns.id, recent.run.id));
|
||||
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
|
||||
await flushBudgetEvaluationWork(db);
|
||||
const [charge] = await recent.charges();
|
||||
expect(charge).toMatchObject({ costStatus: "unpriced" });
|
||||
expect(charge.pricingProvenance).toMatchObject({ evidence: expect.stringContaining("closure evidence never arrived") });
|
||||
expect((await recent.readRun()).usageJson).toMatchObject({ accountingClosureTimedOut: true });
|
||||
// A board-visible notice names the run.
|
||||
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, recent.companyId),
|
||||
eq(activityLog.action, "cost.accounting_closure_timed_out"), eq(activityLog.entityId, recent.run.id)))).toHaveLength(1);
|
||||
expect(await recent.block()).toBeNull();
|
||||
});
|
||||
|
||||
// Storm 4 (run f879895f): session.open timed out, the local stop was never
|
||||
// verified (the identity is still recorded), and recovery ended in terminal
|
||||
// failure. That one run held all new company work for hours.
|
||||
it("bounds the hold of a terminal failure whose stop was never verified", async () => {
|
||||
const f = await fixture({ processPid: 15099, stopEvidence: false });
|
||||
await db.update(heartbeatRuns).set({ errorCode: "native_session_interrupted", finishedAt: new Date(Date.now() - 2 * 60_000) })
|
||||
.where(eq(heartbeatRuns.id, f.run.id));
|
||||
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 0 });
|
||||
expect(await f.block()).toMatchObject({ reason: "New work must wait while a native run recovers its unfinished accounting." });
|
||||
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() - 11 * 60_000) }).where(eq(heartbeatRuns.id, f.run.id));
|
||||
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
|
||||
await flushBudgetEvaluationWork(db);
|
||||
expect((await f.readRun()).usageJson).toMatchObject({ accountingClosureTimedOut: true });
|
||||
expect(await f.block()).toBeNull();
|
||||
});
|
||||
|
||||
// Wave s7403 (run 3c1d7b70): the final bootstrap died in session.open. No
|
||||
// usage receipt was ever journaled and the model was never known, yet the
|
||||
// run held company work like a lost receipt.
|
||||
async function neverStartedFixture(processState: "never_launched" | "verified_stop" | "unverified") {
|
||||
const f = await fixture({ processPid: processState === "unverified" ? 15099 : null,
|
||||
stopEvidence: processState === "verified_stop" });
|
||||
await db.update(heartbeatRuns).set({ errorCode: "native_session_interrupted",
|
||||
usageJson: sql`(${heartbeatRuns.usageJson} - 'accountingReceiptOwner' - 'budgetExposure') || '{"accountingReceiptSequence":0,"model":null,"costStatus":null,"provider":null,"biller":null}'::jsonb` })
|
||||
.where(eq(heartbeatRuns.id, f.run.id));
|
||||
if (processState === "never_launched") await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.run.id));
|
||||
return f;
|
||||
}
|
||||
|
||||
it.each(["never_launched", "verified_stop"] as const)("settles a run whose provider never started at once (%s)", async processState => {
|
||||
const f = await neverStartedFixture(processState);
|
||||
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
|
||||
await flushBudgetEvaluationWork(db);
|
||||
expect((await f.readRun()).costAccountingPending).toBe(false);
|
||||
expect(await f.charges()).toHaveLength(0);
|
||||
expect((await f.reservation()).state).toBe("released");
|
||||
expect(await f.block()).toBeNull();
|
||||
});
|
||||
|
||||
it("keeps waiting for the bound when the provider process was never verified stopped", async () => {
|
||||
const f = await neverStartedFixture("unverified");
|
||||
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 0 });
|
||||
expect((await f.readRun()).costAccountingPending).toBe(true);
|
||||
});
|
||||
|
||||
it("bounds a succeeded run whose accepted result left its receipt incomplete", async () => {
|
||||
const f = await acceptedResultFixture({ stopEvidence: false, finishedMinutesAgo: 11 });
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", errorCode: null }).where(eq(heartbeatRuns.id, f.run.id));
|
||||
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
|
||||
await flushBudgetEvaluationWork(db);
|
||||
expect((await f.readRun()).usageJson).toMatchObject({ accountingClosureTimedOut: true });
|
||||
expect(await f.block()).toBeNull();
|
||||
});
|
||||
|
||||
it("never schedules a wake for a bound that already passed or for a capture failure", async () => {
|
||||
const past = await acceptedResultFixture({ stopEvidence: false, finishedMinutesAgo: 11 });
|
||||
await db.update(heartbeatRuns).set({ usageJson: sql`${heartbeatRuns.usageJson} || '{"accountingCaptureFailed":true}'::jsonb` })
|
||||
.where(eq(heartbeatRuns.id, past.run.id));
|
||||
expect((await readCostAccountingWork(db, Date.now())).nextRunAt).toBeNull();
|
||||
});
|
||||
|
||||
it("stops blocking once a complete late receipt accounts for a timed-out closure", async () => {
|
||||
const f = await acceptedResultFixture({ stopEvidence: false, finishedMinutesAgo: 11 });
|
||||
await db.update(budgetPolicies).set({ unpricedUsagePolicy: "block" }).where(eq(budgetPolicies.id, f.policy.id));
|
||||
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
|
||||
await flushBudgetEvaluationWork(db);
|
||||
expect((await f.block())?.reason).toMatch(/no reliable price|budget hard-stop/);
|
||||
await persistUsageReceipt(db, { schema: "paperclip/accounting-receipt/v1", id: randomUUID(),
|
||||
companyId: f.companyId, runId: f.run.id, adapterType: "paperclip_runner", sourceId: f.sourceId, sequence: 6,
|
||||
receivedAt: new Date().toISOString(),
|
||||
producerEvent: { sourceEventId: randomUUID(), eventType: "turn.completed" },
|
||||
receipt: { complete: true, usageBasis: "per_run", provider: "anthropic", biller: "anthropic",
|
||||
model: "claude-sonnet-5", billingType: "metered_api", costUsd: 0.25, costStatus: "reported",
|
||||
pricingProvenance: { source: "provider_reported", version: "test" },
|
||||
usage: { inputTokens: 1200, cachedInputTokens: 0, outputTokens: 80 } } });
|
||||
await accountRunCost(db, f.run.id);
|
||||
await budgetService(db).evaluateAffectedScopes(f.companyId, f.agentId, null);
|
||||
await flushBudgetEvaluationWork(db);
|
||||
expect((await f.readRun()).usageJson).toMatchObject({ accountingClosureTimedOut: false });
|
||||
expect(await f.block()).toBeNull();
|
||||
});
|
||||
|
||||
it("keeps the blocking policy for a timed-out closure, since the run may still spend", async () => {
|
||||
const f = await acceptedResultFixture({ stopEvidence: false, finishedMinutesAgo: 11 });
|
||||
await db.update(budgetPolicies).set({ unpricedUsagePolicy: "block" }).where(eq(budgetPolicies.id, f.policy.id));
|
||||
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
|
||||
await flushBudgetEvaluationWork(db);
|
||||
expect((await f.block())?.reason).toMatch(/no reliable price|budget hard-stop/);
|
||||
});
|
||||
|
||||
it("keeps the debt pending while native recovery can still produce a result", async () => {
|
||||
const f = await fixture({ phase: "observed" });
|
||||
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 0 });
|
||||
|
||||
@@ -1501,7 +1501,8 @@ export function createPostgresRunDispatchAdapter(
|
||||
}
|
||||
}
|
||||
const budgetBlock = stopRecovery ? null : await recheckAdmittedRunBudgetReadOnly(tx, { companyId: run.companyId, agentId: run.agentId,
|
||||
runId: run.id, projectId: scope.projectId, reservationId: input.reservationId });
|
||||
runId: run.id, projectId: scope.projectId, reservationId: input.reservationId,
|
||||
admittedRecovery: Boolean(input.recoveryLeaseOwner) });
|
||||
if (budgetBlock) return { dispatched: false as const, admissionBlockReason: budgetBlock };
|
||||
if (input.recoveryLeaseOwner) {
|
||||
const [owner] = run.runtimeMode === "native" ? await tx.select({ id: nativeRunFinalizations.runId }).from(nativeRunFinalizations).where(and(
|
||||
|
||||
@@ -152,13 +152,20 @@ async function readObservedSpends(db: Db, policies: SpendPolicy[], now: Date, ex
|
||||
// Unknown billing with no tokens (for example Cursor Cloud) still counts.
|
||||
// The lost-receipt exemption covers only that run's settled final charge;
|
||||
// later growth from a complete receipt is ordinary usage.
|
||||
const exemptFromUnpricedBlock = sql`((${costEvents.idempotencyKey} = 'heartbeat:' || ${costEvents.heartbeatRunId}::text || ':final'
|
||||
// A final charge settled only because closure evidence never arrived is
|
||||
// genuinely unknown spend (its provider may still have run), so none of
|
||||
// these exemptions apply to it.
|
||||
const closureTimedOutCharge = sql`(${costEvents.idempotencyKey} = 'heartbeat:' || ${costEvents.heartbeatRunId}::text || ':final'
|
||||
and exists (select 1 from ${heartbeatRuns} timed_out_run where timed_out_run.id = ${costEvents.heartbeatRunId}
|
||||
and timed_out_run.company_id = ${costEvents.companyId}
|
||||
and timed_out_run.usage_json->>'accountingClosureTimedOut' = 'true'))`;
|
||||
const exemptFromUnpricedBlock = sql`(not ${closureTimedOutCharge} and ((${costEvents.idempotencyKey} = 'heartbeat:' || ${costEvents.heartbeatRunId}::text || ':final'
|
||||
and exists (select 1 from ${heartbeatRuns} lost_receipt_run where lost_receipt_run.id = ${costEvents.heartbeatRunId}
|
||||
and lost_receipt_run.company_id = ${costEvents.companyId}
|
||||
and lost_receipt_run.usage_json->>'accountingReceiptUnavailable' = 'true'))
|
||||
or ${costEvents.costCents} > 0
|
||||
or (${costEvents.billingType} in ('metered_api', 'api', 'subscription') and ${costEvents.inputTokens} = 0 and ${costEvents.cachedInputTokens} = 0
|
||||
and ${costEvents.outputTokens} = 0))`;
|
||||
and ${costEvents.outputTokens} = 0)))`;
|
||||
ledgerColumns[`unpriced${index}`] = sql`count(*) filter (where ${ledger} and ${costEvents.costStatus} = 'unpriced' and ${costEvents.billingType} <> 'subscription_included'
|
||||
and not ${exemptFromUnpricedBlock})::int`;
|
||||
runColumns[`live${index}`] = sql`coalesce(sum(${liveCost}) filter (where ${run} and ${inWindow}), 0)::text`;
|
||||
|
||||
@@ -54,12 +54,16 @@ export async function recheckRunBudgetReservationInTransaction(
|
||||
}
|
||||
if (stopRecovery) return null;
|
||||
const { block, policies, observations } = await budgetServiceInTransaction(tx, publications)
|
||||
.evaluateAdmission(run.companyId, run.agentId, { projectId: reservation.projectId, runId: run.id });
|
||||
.evaluateAdmission(run.companyId, run.agentId, { projectId: reservation.projectId, runId: run.id,
|
||||
admittedRecovery: Boolean(recoveryLeaseOwner) });
|
||||
if (block) return block.reason;
|
||||
const [agent] = await tx.select({ status: agents.status, lifecycleState: agents.lifecycleState }).from(agents)
|
||||
.where(and(eq(agents.companyId, run.companyId), eq(agents.id, run.agentId)));
|
||||
if (!agent || agent.lifecycleState !== "ready" || ["paused", "terminated", "pending_approval"].includes(agent.status))
|
||||
return "Agent is paused or its lifecycle is not ready for new work";
|
||||
// An owned recovery reuses the hold reserveRunBudget already admitted;
|
||||
// other runs' holds filling the budget must not refuse it.
|
||||
if (recoveryLeaseOwner) return null;
|
||||
for (const [index, policy] of policies.entries()) {
|
||||
const observed = observations[index];
|
||||
// This operation's hold is already included in exposure. Do not propose it
|
||||
@@ -102,7 +106,7 @@ export async function reserveRunBudget(db: Db, companyId: string, runId: string,
|
||||
// Recovery has already made this run queued/running, so its own pending
|
||||
// accounting is excluded. Current overruns, other unfinished runs and
|
||||
// scope pauses still apply before any more paid work starts.
|
||||
const block = await budgetServiceInTransaction(tx, publications).getInvocationBlock(companyId, run.agentId, { projectId, runId });
|
||||
const block = await budgetServiceInTransaction(tx, publications).getInvocationBlock(companyId, run.agentId, { projectId, runId, admittedRecovery: true });
|
||||
if (block) return { admissionBlockReason: block.reason };
|
||||
const [agent] = await tx.select({ status: agents.status, lifecycleState: agents.lifecycleState }).from(agents)
|
||||
.where(and(eq(agents.companyId, companyId), eq(agents.id, run.agentId)));
|
||||
|
||||
@@ -201,7 +201,12 @@ export function budgetPoliciesForRun(companyId: string, agentId: string | null,
|
||||
}
|
||||
|
||||
/** `runId` names the run being admitted, so its own pending accounting never holds it. */
|
||||
type AdmissionContext = { issueId?: string | null; projectId?: string | null; runId?: string | null };
|
||||
type AdmissionContext = { issueId?: string | null; projectId?: string | null; runId?: string | null;
|
||||
/** A validated recovery owner is continuing work that was already admitted.
|
||||
* Accounting holds gate new work only; recoveries must be able to finish,
|
||||
* or concurrent recoveries after a restart storm wait on each other. Hard
|
||||
* stops and pauses still apply. */
|
||||
admittedRecovery?: boolean };
|
||||
|
||||
async function readAdmissionScopes(db: Db, companyId: string, agentId: string | null, context?: AdmissionContext) {
|
||||
const scopes: Array<{ scopeType: BudgetScopeType; scopeId: string }> = [
|
||||
@@ -287,6 +292,8 @@ export async function policyBlocks(db: Db, policy: PolicyRow) {
|
||||
* charge targets already-admitted work through the normal durable Stop path. */
|
||||
export async function recheckAdmittedRunBudgetReadOnly(db: Db, input: {
|
||||
companyId: string; agentId: string; runId: string; projectId: string | null; reservationId: string;
|
||||
/** See AdmissionContext.admittedRecovery. */
|
||||
admittedRecovery?: boolean;
|
||||
}): Promise<string | null> {
|
||||
return withBudgetSpan("budget.admission.evaluate", async () => {
|
||||
const [reservation] = await db.select({ amountCents: budgetReservations.amountCents }).from(budgetReservations).where(and(
|
||||
@@ -302,7 +309,11 @@ export async function recheckAdmittedRunBudgetReadOnly(db: Db, input: {
|
||||
const observations = await computeObservedSpends(db, policies, undefined, input.runId);
|
||||
for (const [index, policy] of policies.entries()) {
|
||||
const observed = observations[index];
|
||||
if (observedBlocks(policy, observed)) return "Applicable budget no longer permits provider work";
|
||||
if (input.admittedRecovery ? hardStopReached(policy, observed) : observedBlocks(policy, observed))
|
||||
return "Applicable budget no longer permits provider work";
|
||||
// An owned recovery reuses the hold reserveRunBudget already admitted;
|
||||
// other runs' holds filling the budget must not refuse it.
|
||||
if (input.admittedRecovery) continue;
|
||||
const capacity = compareCents(addCents(observed.ledgerCents, observed.effectiveHeldCents), policy.amount);
|
||||
if (capacity > 0 || (compareCents(reservation.amountCents, 0) === 0 && capacity >= 0))
|
||||
return "Available budget is reserved by unfinished runs";
|
||||
@@ -834,12 +845,12 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica
|
||||
if (pause) return admission(pause);
|
||||
for (const { policy, observed } of scoped) {
|
||||
const title = scopeType === "company" ? "Company" : scopeType === "agent" ? "Agent" : "Project";
|
||||
if (observedBlocks(policy, observed)) return admission({ scopeType, scopeId, scopeName: scope.name,
|
||||
if (context?.admittedRecovery ? hardStopReached(policy, observed) : observedBlocks(policy, observed)) return admission({ scopeType, scopeId, scopeName: scope.name,
|
||||
reason: observed.pendingRunCount > 0 ? `${title} cannot start work while completed runs await accounting.`
|
||||
: unpricedUsageBlocks(policy, observed) ? `${title} cannot start work because recorded usage has no reliable price.`
|
||||
: scopeType === "project" ? "Project cannot start work because its budget hard-stop is still exceeded."
|
||||
: `${title} is paused because its budget hard-stop was reached.` });
|
||||
if (policy.isActive && policy.hardStopEnabled && policy.amount > 0 && observed.recoveringRunCount > 0)
|
||||
if (!context?.admittedRecovery && policy.isActive && policy.hardStopEnabled && policy.amount > 0 && observed.recoveringRunCount > 0)
|
||||
return admission({ scopeType, scopeId, scopeName: scope.name,
|
||||
reason: "New work must wait while a native run recovers its unfinished accounting." });
|
||||
}
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
import { notifyStrandedAgentStatus } from "./live-run-recovery-notifications.js";
|
||||
import { usdToUnits, unitsToCents } from "@paperclipai/shared";
|
||||
import { and, asc, eq, inArray, or, sql } from "drizzle-orm";
|
||||
import { budgetReservations, agents, environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, issues, projects, decisionInvocations, promptInjectionInvocations, type Db } from "@paperclipai/db";
|
||||
import { and, asc, eq, inArray, isNull, or, sql } from "drizzle-orm";
|
||||
import { budgetReservations, agents, environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, runUsageReceipts, issues, projects, decisionInvocations, promptInjectionInvocations, type Db } from "@paperclipai/db";
|
||||
import { withOperationAccountingTransaction } from "./accounting-transaction.js";
|
||||
import { withBudgetSpan } from "./budget-tracing.js";
|
||||
import { conflict } from "../errors.js";
|
||||
import { logActivity } from "./activity-log.js";
|
||||
import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "./native-runtime/native-runner-recovery-contract.js";
|
||||
import { createCostEventInTransaction } from "./costs.js";
|
||||
import type { BudgetServiceHooks } from "./budgets.js";
|
||||
import { queueBudgetEvaluation } from "./budget-evaluation-work.js";
|
||||
@@ -35,6 +37,7 @@ async function readAccountingClosure(db: Db, runId: string) {
|
||||
preProviderFailure: sql<boolean>`(${heartbeatRuns.resultJson}->'executionRecovery'->>'providerWorkStarted' = 'false'
|
||||
or ${heartbeatRuns.usageJson}->>'accountingProviderWorkStarted' = 'false') is true`,
|
||||
coordinatorId: nativeRunFinalizations.runId, resultId: nativeRunFinalizations.resultId, phase: nativeRunFinalizations.phase,
|
||||
providerWorkNeverStarted: sql<boolean>`(${nativeProviderWorkNeverStarted}) is true`,
|
||||
processPid: heartbeatRuns.processPid, processGroupId: heartbeatRuns.processGroupId,
|
||||
}).from(heartbeatRuns).leftJoin(nativeRunFinalizations, and(
|
||||
eq(nativeRunFinalizations.runId, heartbeatRuns.id), eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
|
||||
@@ -73,10 +76,15 @@ const remoteExecutionTerminated = sql`exists (select 1 from ${environmentLeases}
|
||||
* does not clear the recorded remote process identity.
|
||||
* Missing identity alone is never proof. One predicate serves the reconciler
|
||||
* scan and settlement so they cannot disagree. */
|
||||
const closedNativeReceiptSource = sql`${heartbeatRuns.runtimeMode} = 'native'
|
||||
and exists (select 1 from ${nativeRunFinalizations} where ${nativeRunFinalizations.runId} = ${heartbeatRuns.id}
|
||||
/** Native recovery will not run this provider again: it ended in terminal
|
||||
* failure, or the run's result was already accepted. */
|
||||
const settledNativeCoordinator = sql`exists (select 1 from ${nativeRunFinalizations} where ${nativeRunFinalizations.runId} = ${heartbeatRuns.id}
|
||||
and ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId}
|
||||
and ${nativeRunFinalizations.resultId} is null and ${nativeRunFinalizations.phase} = 'terminal_failure')
|
||||
and ((${nativeRunFinalizations.resultId} is null and ${nativeRunFinalizations.phase} = 'terminal_failure')
|
||||
or ${nativeRunFinalizations.resultId} is not null))`;
|
||||
|
||||
const closedNativeReceiptSource = sql`${heartbeatRuns.runtimeMode} = 'native'
|
||||
and ${settledNativeCoordinator}
|
||||
and ((${heartbeatRuns.status} = 'failed'
|
||||
and ${heartbeatRuns.processPid} is null and ${heartbeatRuns.processGroupId} is null
|
||||
and (select process_event.event_type from ${heartbeatRunEvents} process_event
|
||||
@@ -86,10 +94,54 @@ const closedNativeReceiptSource = sql`${heartbeatRuns.runtimeMode} = 'native'
|
||||
order by process_event.seq desc limit 1) = 'native.local_process_stopped')
|
||||
or (${heartbeatRuns.status} in ('failed', 'cancelled', 'timed_out', 'interrupted') and ${remoteExecutionTerminated}))`;
|
||||
|
||||
async function nativeReceiptSourceClosed(db: Db, runId: string) {
|
||||
const [row] = await db.select({ closed: sql<boolean>`${closedNativeReceiptSource}` }).from(heartbeatRuns)
|
||||
/** A native run whose provider never started has nothing to account: recovery
|
||||
* settled, not one usage receipt was ever journaled for the run (a provider
|
||||
* turn journals one as soon as it starts), and its local process either never
|
||||
* launched or was verified stopped. Settle it at once, like a pre-provider
|
||||
* failure, instead of holding company work. */
|
||||
const nativeProviderWorkNeverStarted = sql`${heartbeatRuns.runtimeMode} = 'native'
|
||||
and ${heartbeatRuns.status} in ('failed', 'cancelled', 'timed_out', 'interrupted')
|
||||
and ${settledNativeCoordinator}
|
||||
and coalesce((${heartbeatRuns.usageJson}->>'accountingReceiptSequence')::int, 0) = 0
|
||||
and ${heartbeatRuns.usageJson}->'accountingReceiptSourceFences' is null
|
||||
and not exists (select 1 from ${runUsageReceipts} journaled_receipt
|
||||
where journaled_receipt.company_id = ${heartbeatRuns.companyId} and journaled_receipt.run_id = ${heartbeatRuns.id})
|
||||
and ${heartbeatRuns.processPid} is null and ${heartbeatRuns.processGroupId} is null
|
||||
and coalesce((select process_event.event_type from ${heartbeatRunEvents} process_event
|
||||
where process_event.company_id = ${heartbeatRuns.companyId} and process_event.run_id = ${heartbeatRuns.id}
|
||||
and process_event.source_event_id is null
|
||||
and process_event.event_type in ('native.process_start_requested', 'native.process_identity_recorded', 'native.local_process_stopped')
|
||||
order by process_event.seq desc limit 1), 'native.local_process_stopped') = 'native.local_process_stopped'`;
|
||||
|
||||
/** Waiting for closure evidence is bounded. Past the remote runner reconnect
|
||||
* grace plus a margin, a terminal run whose recovery has settled no longer
|
||||
* holds new company work: its debt settles as unpriced, flagged so the
|
||||
* blocking policy still counts it and the board sees which run it was. */
|
||||
export const ACCOUNTING_CLOSURE_TIMEOUT_MS = REMOTE_RUNNER_RECONNECT_GRACE_MS + 3 * 60_000;
|
||||
/** `at` is one shared cutoff, so the actionable scan and the wake query
|
||||
* cannot both miss a run whose bound falls between them. */
|
||||
function closureEvidenceTimedOut(at: Date = new Date()) {
|
||||
return sql`${heartbeatRuns.runtimeMode} = 'native'
|
||||
and ${heartbeatRuns.status} in ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted')
|
||||
and ${settledNativeCoordinator}
|
||||
and coalesce(${heartbeatRuns.finishedAt}, ${heartbeatRuns.updatedAt})
|
||||
<= ${at.toISOString()}::timestamptz - make_interval(secs => ${ACCOUNTING_CLOSURE_TIMEOUT_MS / 1000})`;
|
||||
}
|
||||
|
||||
async function readNativeReceiptClosure(db: Db, runId: string) {
|
||||
const [row] = await db.select({ closed: sql<boolean>`${closedNativeReceiptSource}`,
|
||||
timedOut: sql<boolean>`${closureEvidenceTimedOut()}`, neverStarted: sql<boolean>`${nativeProviderWorkNeverStarted}` }).from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
return row?.closed === true;
|
||||
if (row?.neverStarted === true) return "no_provider_work" as const;
|
||||
if (row?.closed === true) return "closed" as const;
|
||||
return row?.timedOut === true ? "timed_out" as const : null;
|
||||
}
|
||||
|
||||
function withClosureTimeoutEvidence(provenance: Record<string, unknown>, timedOut: boolean) {
|
||||
if (!timedOut) return provenance;
|
||||
const evidence = text(provenance.evidence);
|
||||
return { source: "unknown", version: "accounting-receipt/v1", ...provenance,
|
||||
evidence: [evidence, "closure evidence never arrived; settled after the accounting closure timeout, and the run may have spent more"].filter(Boolean).join("; ") };
|
||||
}
|
||||
|
||||
/** A receipt lost with its stopped provider still carries the usage observed
|
||||
@@ -135,7 +187,7 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
|
||||
? receiptIndex ?? await indexPendingUsageReceipts() : undefined;
|
||||
const closure = await readAccountingClosure(db, runId);
|
||||
if (!closure) return false;
|
||||
if (!identity.accountedAt && closure.runtimeMode === "native" && !closure.preProviderFailure
|
||||
if (!identity.accountedAt && closure.runtimeMode === "native" && !closure.preProviderFailure && !closure.providerWorkNeverStarted
|
||||
&& closure.coordinatorId && !closure.resultId && closure.phase !== "terminal_failure") return false;
|
||||
const pendingReceipts = !identity.accountedAt
|
||||
? await readPendingRunUsageReceipts({ companyId: identity.companyId, runId }, undefined, index) : [];
|
||||
@@ -172,8 +224,10 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
|
||||
// A failed capture may follow an older complete attempt. Neither replay
|
||||
// nor terminal status can certify that the missing work was free.
|
||||
if (object(run.usageJson).accountingCaptureFailed === true) return false;
|
||||
const providerWorkNeverStarted = run.runtimeMode === "native" && object(run.usageJson).accountingReceiptReady === false
|
||||
&& await readNativeReceiptClosure(tx, run.id) === "no_provider_work";
|
||||
const preProviderFailure = object(object(run.resultJson).executionRecovery).providerWorkStarted === false
|
||||
|| object(run.usageJson).accountingProviderWorkStarted === false;
|
||||
|| object(run.usageJson).accountingProviderWorkStarted === false || providerWorkNeverStarted;
|
||||
// A failed native turn is not necessarily the end of this heartbeat: the
|
||||
// coordinator can resume the same run. Only its closed provider boundary
|
||||
// authorizes acknowledgement; finalization may still run after a result.
|
||||
@@ -188,7 +242,9 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
|
||||
// deliver it: settle the known part as unpriced usage. A later receipt
|
||||
// still posts through late-receipt reconciliation.
|
||||
const receiptUnavailable = !preProviderFailure && object(run.usageJson).accountingReceiptReady === false;
|
||||
if (receiptUnavailable && !(await nativeReceiptSourceClosed(tx, run.id))) return false;
|
||||
const receiptClosure = receiptUnavailable ? await readNativeReceiptClosure(tx, run.id) : null;
|
||||
if (receiptUnavailable && !receiptClosure) return false;
|
||||
const closureTimedOut = receiptClosure === "timed_out";
|
||||
if (!preProviderFailure) await ensureRuntimeAccountingBaseline(tx, run.companyId, run.agentId);
|
||||
// Acknowledgement is part of the same transaction. Budget checks inside
|
||||
// this transaction must not mistake this receipt for missing accounting.
|
||||
@@ -228,7 +284,7 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
|
||||
issueId: text(scope.issueId), projectId: text(scope.projectId), billingCode: text(scope.billingCode),
|
||||
provider: text(usage.provider) ?? "unknown", biller: text(usage.biller) ?? text(usage.provider) ?? "unknown",
|
||||
providerRequestId: text(usage.providerRequestId),
|
||||
pricingProvenance: lostReceiptEstimate?.pricingProvenance ?? object(usage.pricingProvenance),
|
||||
pricingProvenance: withClosureTimeoutEvidence(lostReceiptEstimate?.pricingProvenance ?? object(usage.pricingProvenance), closureTimedOut),
|
||||
model: text(usage.model) ?? "unknown", billingType,
|
||||
costStatus: lostReceiptEstimate ? "estimated" : (costUsd === null || usage.costStatus === "unpriced" || receiptUnavailable) && billingType !== "subscription_included" && hasBillableEvidence ? "unpriced" : usage.costStatus === "estimated" ? "estimated" : "reported",
|
||||
inputTokens: amount(usage.inputTokens) ?? 0, cachedInputTokens: amount(usage.cachedInputTokens) ?? 0, outputTokens: amount(usage.outputTokens) ?? 0,
|
||||
@@ -267,11 +323,18 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
|
||||
// A complete receipt arriving later may price this charge (see usage-receipts).
|
||||
...(receiptUnavailable ? { usageJson: sql`coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || ${JSON.stringify({
|
||||
accountingReceiptUnavailable: true,
|
||||
// Unlike a verified stop, the run may still have spent: the blocking
|
||||
// unpriced-usage policy keeps counting this charge.
|
||||
...(closureTimedOut ? { accountingClosureTimedOut: true } : {}),
|
||||
// Late reconciliation measures a later receipt from the accepted money.
|
||||
...(lostReceiptEstimate ? { costUsdExact: lostReceiptEstimate.costUsdExact, costStatus: "estimated",
|
||||
pricingProvenance: lostReceiptEstimate.pricingProvenance } : {}),
|
||||
})}::jsonb` } : {}),
|
||||
}).where(eq(heartbeatRuns.id, run.id));
|
||||
if (closureTimedOut) await logActivity(tx, { companyId: run.companyId, actorType: "system", actorId: "cost_accounting",
|
||||
action: "cost.accounting_closure_timed_out", entityType: "heartbeat_run", entityId: run.id, runId: run.id, agentId: run.agentId,
|
||||
details: { reason: "Final usage never arrived and no stop of its provider was verified; the run's known usage was recorded as unpriced.",
|
||||
timeoutMinutes: ACCOUNTING_CLOSURE_TIMEOUT_MS / 60_000 } }, publications);
|
||||
return true;
|
||||
}), { "budget.accounting.owner": "run" });
|
||||
for (const file of recoveredFiles) {
|
||||
@@ -281,7 +344,7 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
|
||||
return accounted;
|
||||
}
|
||||
|
||||
function actionableRunAccounting() {
|
||||
function actionableRunAccounting(at: Date = new Date()) {
|
||||
return and(
|
||||
eq(heartbeatRuns.costAccountingPending, true), inArray(heartbeatRuns.status, terminalStatuses),
|
||||
or(and(sql`${heartbeatRuns.costAccountedAt} is not null`,
|
||||
@@ -295,7 +358,7 @@ function actionableRunAccounting() {
|
||||
and ${heartbeatRuns.runtimeMode} = 'native' and ${nativeRunFinalizations.resultId} is null
|
||||
and ${nativeRunFinalizations.phase} <> 'terminal_failure'))`,
|
||||
sql`(${heartbeatRuns.usageJson}->>'accountingReceiptReady' is distinct from 'false' or ${heartbeatRuns.resultJson}->'executionRecovery'->>'providerWorkStarted' = 'false' or ${heartbeatRuns.usageJson}->>'accountingProviderWorkStarted' = 'false'
|
||||
or (${closedNativeReceiptSource}))`,
|
||||
or (${closedNativeReceiptSource}) or (${closureEvidenceTimedOut(at)}) or (${nativeProviderWorkNeverStarted}))`,
|
||||
)),
|
||||
);
|
||||
}
|
||||
@@ -303,7 +366,8 @@ function actionableRunAccounting() {
|
||||
/** Only unfinished, actionable debt warrants retries. Incomplete provider
|
||||
* receipts retain their accounting/idle hold and wake when evidence arrives. */
|
||||
export async function readCostAccountingWork(db: Db, now = Date.now(), receiptDirectory?: string) {
|
||||
const [pending] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(actionableRunAccounting()).limit(1);
|
||||
const cutoff = new Date(now);
|
||||
const [pending] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(actionableRunAccounting(cutoff)).limit(1);
|
||||
const [deadline] = await db.execute<{ at: Date | string | null }>(sql`
|
||||
select min(started_at) + interval '120 seconds 1 millisecond' as at from (
|
||||
select ${decisionInvocations.startedAt} as started_at from ${decisionInvocations}
|
||||
@@ -312,7 +376,20 @@ export async function readCostAccountingWork(db: Db, now = Date.now(), receiptDi
|
||||
select ${promptInjectionInvocations.startedAt} from ${promptInjectionInvocations}
|
||||
where ${promptInjectionInvocations.status} = 'running'
|
||||
) running_service_invocations`);
|
||||
const nextRunAt = deadline?.at == null ? null : new Date(deadline.at).getTime();
|
||||
// Wake when a run waiting for closure evidence reaches its bound.
|
||||
const [closure] = await db.select({ at: sql<Date | string | null>`min(coalesce(${heartbeatRuns.finishedAt}, ${heartbeatRuns.updatedAt}))` })
|
||||
.from(heartbeatRuns).where(and(eq(heartbeatRuns.costAccountingPending, true), isNull(heartbeatRuns.costAccountedAt),
|
||||
sql`${heartbeatRuns.usageJson}->>'accountingReceiptReady' = 'false'`,
|
||||
sql`${heartbeatRuns.usageJson}->>'accountingCaptureFailed' is distinct from 'true'`,
|
||||
sql`${heartbeatRuns.runtimeMode} = 'native' and ${heartbeatRuns.status} in ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted')`,
|
||||
settledNativeCoordinator,
|
||||
// Only future bounds: a run already past it is in the actionable scan,
|
||||
// and a past deadline must never schedule an immediate pass again.
|
||||
sql`coalesce(${heartbeatRuns.finishedAt}, ${heartbeatRuns.updatedAt})
|
||||
> ${cutoff.toISOString()}::timestamptz - make_interval(secs => ${ACCOUNTING_CLOSURE_TIMEOUT_MS / 1000})`));
|
||||
const closureAt = closure?.at == null ? null : new Date(closure.at).getTime() + ACCOUNTING_CLOSURE_TIMEOUT_MS + 1;
|
||||
const serviceAt = deadline?.at == null ? null : new Date(deadline.at).getTime();
|
||||
const nextRunAt = serviceAt === null ? closureAt : closureAt === null ? serviceAt : Math.min(serviceAt, closureAt);
|
||||
const receipts = await indexPendingUsageReceipts(receiptDirectory);
|
||||
return {
|
||||
pending: !!pending || receipts.size > 0 || (nextRunAt !== null && nextRunAt <= now),
|
||||
|
||||
@@ -340,7 +340,14 @@ export async function reconcileLateRunUsageReceiptInTransaction(tx: Db, run: typ
|
||||
accountingLateReceiptDebts: debts, accountingLateReceiptId: nextId,
|
||||
accountingLateReceiptAttemptedId: nextId && debts[nextId] ? nextId : null,
|
||||
...(reconciled && advanceReceiptIdentity ? { accountingReconciledReceiptId: currentId,
|
||||
accountingReconciledReceiptSourceId: late!.sourceId, accountingReconciledReceiptSequence: late!.sequence } : {}),
|
||||
accountingReconciledReceiptSourceId: late!.sourceId,
|
||||
accountingReconciledReceiptSequence: late!.sequence } : {}),
|
||||
// A complete receipt now accounts for the usage a closure timeout
|
||||
// could not see, so its unknown-spend charge stops blocking.
|
||||
// Only a complete whole-run receipt from the run's own source does;
|
||||
// an increment reported by a later run does not.
|
||||
...(reconciled && late?.receipt.complete && !late.lateDelta && late.sourceId === previous.accountingReceiptSourceId
|
||||
&& previous.accountingClosureTimedOut === true ? { accountingClosureTimedOut: false } : {}),
|
||||
},
|
||||
}).where(eq(heartbeatRuns.id, run.id));
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user