fix(accounting): a run's unproven accounting never holds the company (#20254)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, each lost provider receipt leaves an accounting
debt that settles later
> - In restart-chaos G1 (seed 9802, task t02), six debts each waited the
full 10-minute closure bound, and the board's accepted Retry waited with
them
> - The last debt settled as an unpriced charge. Under the blocking
unpriced-usage policy, that charge hard-stopped the whole company, which
cancelled the Retry, and the task showed "budget"
> - Dotta's plan decision 1 says accounting is never a company-wide
gate: admission reads per-run exposure and real hard stops only
> - This pull request makes those debts settle at once when the stop is
proven, records accounting-settled charges without a company hold, and
labels the remaining accounting wait correctly
> - The benefit is that a restart no longer stalls a company's work for
10 minutes or stops it outright

## Linked Issues or Issue Description

Refs #20250 (Phase 2: a run that only inherited its predecessor's runner
identity settles at once; this PR builds on it), #19944, #19826, #19857

**What happened?**
Restart-chaos wave w-20261109T111954-s9802 (G1), task t02, master
c5da0b88ad. After a K-fault restart at 11:27, six `run_accounting_debt`
records were "unprovable". Each settled exactly 10 minutes after its run
finished: bf9ea6b4 and c2e222a2 at 11:30, 097d0036 at 11:35, f64f9f46
and aa2c9f03 at 11:36, and 0b4b404f at 11:37:23.
- Four runs were `cancelled` (issue_terminalized), with
`native.local_process_stopped` as their last process event and no
recorded pid. The receipt-closed rule accepted that stop proof only for
`failed` runs.
- 0b4b404f failed with `native_session_cleanup_quarantined` before it
started any process. It journaled no receipt, but it carried pid 11356
of the previous run's warm runner, so the never-started rule did not
apply.
- At 11:37:23, 0b4b404f settled as an unpriced unknown-billing charge
with zero tokens. With `unpricedUsagePolicy: block`, the company hit a
hard stop at 11:37:24 (24 of 3000 cents). The pause cancelled the
board's Retry (751751ea, accepted at 11:28:37), and the task showed
`userAction: "budget"`.

**Expected behavior**
Accounting debts whose provider stop is proven settle at once.
Accounting never holds or pauses the whole company. A wait caused by
unpriced usage is labeled as accounting, not budget.

**Steps to reproduce**
Restart the control plane while native runs are being cancelled, with
the company's unpriced-usage policy set to block. Before this change,
the debts wait 10 minutes and then hard-stop the company.

## What Changed

- `run-cost-accounting.ts`: a verified local process stop closes the
receipt source for `cancelled`, `timed_out` and `interrupted` runs as
well as `failed` ones. The inherited-pid run (0b4b404f) is covered by
#20250, so this PR does not change the never-started rule.
- `budget-exposure.ts`: a final charge that accounting settled without
its receipt (receipt unavailable, closure timed out, capture failed, or
lost on restore) never trips the blocking unpriced-usage policy.
Unpriced usage from a complete receipt still does. A capture failure
stays exempt even if a later report adds usage, because its receipt is
never certified complete. This removes the restart-derived special case,
because the rule now covers every accounting-settled charge.
- `task-interruption-presentation.ts`: a `budget.blocked` wait caused by
the unpriced-usage refusal is shown as `accounting.unpriced_usage`, with
accounting wording, for both a held Retry and a never-started task. Only
the fixed refusal phrase is matched; the stored message is never shown.
- `doc/SPEC-implementation.md`: the admission and accounting rules are
updated to match.

## Verification

- `closed-native-receipt-accounting.test.ts`: new tests for cancelled,
timed-out and interrupted runs with a verified stop; they were red
before the fix. When a timed-out closure settles under the blocking
policy, the board's saved Retry is promoted (due at once) and never
cancelled. That assertion was red on master, where the hard stop
cancelled it. Seven tests that asserted the old rule (an
accounting-settled charge keeps the blocking policy) now assert that the
charge is recorded and the company stays active and admits work. All 61
pass.
- `task-interruption-query.test.ts` (Postgres): a Retry held by the
unpriced refusal, and a skipped wake, are labeled
`accounting.unpriced_usage`. A budget hard stop keeps `budget.blocked`.
The test was red before the fix.
- Also passing: `run-cost-accounting`, `cost-data-quality`,
`idle-accounting-work`, `native-usage-delivery.integration`,
`task-interruption-presentation`, and the accounting and budget tests in
`heartbeat-process-recovery`. Server typecheck has no errors in the
changed files. The only local errors are in
`native-session-executor.ts`, from a stale local runner build.
- Not run: the G1 chaos rerun. Phase 3 runs it after merge.

## Risks

- Behavior change: a charge that accounting settled without a receipt no
longer hard-stops a company under the blocking policy. That includes a
remote run whose closure evidence never arrived, which may still have
spent. The charge stays in Costs with provenance and an activity entry,
and the spend still counts toward the budget limit. This follows plan
decision 1.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-09 07:02:47 -06:00
1 parent 7044ff96e3
commit 4c15eea770
6 files changed
+137 -60

No files matched your search

+10 -7
View File
@@ -1769,7 +1769,10 @@ hard stop refuses new work even while evaluation lags.
waits on another run's cleanup or accounting completeness. Only these refuse it:
- spend at the limit (ledger plus observed running spend), or unpriced usage
under a blocking policy (the hard stop);
a provider reported, under a blocking policy (the hard stop). A charge that
accounting settled without its receipt (lost, closure evidence that never
arrived, or a failed capture) is recorded and shown in Costs, but it is never
a company gate;
- an explicit pause of the company, project or agent;
- an inactive company.
@@ -1986,12 +1989,12 @@ its display label after fifteen minutes. Elapsed time never discards a late
receipt. It settles the debt only for a native run whose recovery has ended
(terminal failure or an accepted result) and whose closure evidence has not
arrived after ten minutes (the remote runner reconnect grace plus a margin): the known usage is recorded as unpriced with
provenance saying closure evidence never arrived, an activity entry names the
run, and the blocking unpriced-usage policy keeps counting that charge unless
a control-plane restart caused it. A restart-derived receipt (the run replaced
its receipt source, recovered from a restart, or lost a provider turn on
restore) never hard-stops or pauses a company under the blocking policy,
whatever its token count; its activity entry and provenance remain the notice. A lost
provenance saying closure evidence never arrived, and an activity entry names
the run. Like any charge accounting settled without its receipt, it never
hard-stops or pauses a company under the blocking policy, whatever its token
count; its activity entry and provenance remain the notice. A verified local
process stop closes the receipt source for a failed, cancelled, timed-out or
interrupted run alike, so a restart-cancelled run settles at once. A lost
receipt that already carries a provider-reported cost settles at that cost as
an estimate marked as a lower bound, not as unpriced. A failed native run that
never started provider work (no usage receipt, no replaced receipt source, and a
@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import {
agents, budgetIncidents, budgetPolicies, budgetReservations, companies, costEvents, createDb, environmentLeases, environments,
agentWakeupRequests, agents, budgetIncidents, budgetPolicies, budgetReservations, companies, costEvents, createDb, environmentLeases, environments,
heartbeatRunEvents, heartbeatRuns, issues,
activityLog, completionContracts, nativeRunFinalizations, nativeRunResults, type Db,
} from "@paperclipai/db";
@@ -595,14 +595,27 @@ const support = await getEmbeddedPostgresTestSupport();
expect(await f.block()).toBeNull();
});
it("does not count an empty fence map as restart evidence for a timed-out closure", async () => {
// G1 wave s9802 (runs bf9ea6b4, c2e222a2, aa2c9f03, f64f9f46): a restart
// cancelled these runs, and recovery verified each local process stopped.
// The stop proof counted only for failed runs, so each cancelled run held its
// task's Retry for the full 10-minute closure bound.
it.each(["cancelled", "timed_out", "interrupted"] as const)("settles a %s run once its local stop is verified", async status => {
const f = await fixture();
await db.update(heartbeatRuns).set({ status, errorCode: "issue_terminalized" }).where(eq(heartbeatRuns.id, f.run.id));
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
const run = await f.readRun();
expect(run.costAccountingPending).toBe(false);
expect(run.usageJson).not.toHaveProperty("accountingClosureTimedOut");
});
it("records a timed-out closure with an empty fence map without blocking", async () => {
const f = await acceptedResultFixture({ stopEvidence: false, finishedMinutesAgo: 11, restart: false });
await db.update(budgetPolicies).set({ unpricedUsagePolicy: "block" }).where(eq(budgetPolicies.id, f.policy.id));
await db.update(heartbeatRuns).set({ usageJson: sql`${heartbeatRuns.usageJson} || '{"accountingReceiptSourceFences":{}}'::jsonb` })
.where(eq(heartbeatRuns.id, f.run.id));
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
await flushBudgetEvaluationWork(db);
expect((await f.block())?.reason).toMatch(/no reliable price|budget hard-stop/);
expect(await f.block()).toBeNull();
});
it("keeps waiting for the bound when the provider process was never verified stopped", async () => {
@@ -655,21 +668,27 @@ const support = await getEmbeddedPostgresTestSupport();
expect(next!).toBeGreaterThan(Date.now());
});
it("keeps a settled capture failure under the blocking unpriced-usage policy", async () => {
// Accounting is never a company gate (plan decision 1): a charge accounting
// settled without its receipt is recorded and shown in Costs, but it never
// trips the blocking unpriced-usage policy.
it("records a settled capture failure without tripping the blocking unpriced-usage policy", async () => {
const f = await fixture({ unpricedUsagePolicy: "block" });
await captureFailed(f, 11, { inputTokens: 1200, outputTokens: 80, model: "unknown-model" });
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
await flushBudgetEvaluationWork(db);
expect((await f.block())?.reason).toMatch(/no reliable price|budget hard-stop/);
expect(await f.charges()).toMatchObject([{ costStatus: "unpriced" }]);
expect(await f.block()).toBeNull();
const [company] = await db.select().from(companies).where(eq(companies.id, f.companyId));
expect(company).toMatchObject({ status: "active", pauseReason: null });
});
it("keeps a capture failure after a receipt-source handoff under the blocking policy", async () => {
it("records a capture failure after a receipt-source handoff without blocking", async () => {
const f = await fixture({ unpricedUsagePolicy: "block" });
await captureFailed(f, 11, { inputTokens: 1200, outputTokens: 80, model: "unknown-model",
accountingReceiptSourceFences: { [f.sourceId]: 2 } });
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
await flushBudgetEvaluationWork(db);
expect((await f.block())?.reason).toMatch(/no reliable price|budget hard-stop/);
expect(await f.block()).toBeNull();
});
it("settles a capture failure at no less than its observed live price", async () => {
@@ -716,12 +735,12 @@ const support = await getEmbeddedPostgresTestSupport();
expect((await readCostAccountingWork(db, Date.now())).nextRunAt).toBeNull();
});
it("stops blocking once a complete late receipt accounts for a timed-out closure", async () => {
it("prices a timed-out closure from a complete late receipt, never blocking", async () => {
const f = await acceptedResultFixture({ stopEvidence: false, finishedMinutesAgo: 11, restart: false });
await db.update(budgetPolicies).set({ unpricedUsagePolicy: "block" }).where(eq(budgetPolicies.id, f.policy.id));
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
await flushBudgetEvaluationWork(db);
expect((await f.block())?.reason).toMatch(/no reliable price|budget hard-stop/);
expect(await f.block()).toBeNull();
await persistUsageReceipt(db, { schema: "paperclip/accounting-receipt/v1", id: randomUUID(),
companyId: f.companyId, runId: f.run.id, adapterType: "paperclip_runner", sourceId: f.sourceId, sequence: 6,
receivedAt: new Date().toISOString(),
@@ -795,7 +814,7 @@ const support = await getEmbeddedPostgresTestSupport();
expect(await f.block()).toBeNull();
});
it("keeps the blocking policy for a timed-out closure whose only recovery was an ordinary same-run retry", async () => {
it("records a timed-out closure whose only recovery was an ordinary same-run retry without blocking", async () => {
const f = await acceptedResultFixture({ stopEvidence: false, finishedMinutesAgo: 11, restart: false });
await db.update(budgetPolicies).set({ unpricedUsagePolicy: "block" }).where(eq(budgetPolicies.id, f.policy.id));
await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.run.id, agentId: f.agentId, seq: 10,
@@ -803,15 +822,31 @@ const support = await getEmbeddedPostgresTestSupport();
payload: { restartKind: "hot", trigger: "same_run_retry", runnerDisposition: "reattach_remote_runner" } });
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
await flushBudgetEvaluationWork(db);
expect((await f.block())?.reason).toMatch(/no reliable price|budget hard-stop/);
expect(await f.block()).toBeNull();
});
it("keeps the blocking policy for a timed-out closure with no restart evidence, since the run may still spend", async () => {
// G1 wave s9802: the timed-out closure's unpriced charge hard-stopped the
// company and cancelled an accepted board Retry. Its exposure is recorded;
// it never holds the company.
it("records a timed-out closure with no restart evidence without holding the company", async () => {
const f = await acceptedResultFixture({ stopEvidence: false, finishedMinutesAgo: 11, restart: false });
await db.update(budgetPolicies).set({ unpricedUsagePolicy: "block" }).where(eq(budgetPolicies.id, f.policy.id));
// The board's accepted Retry, saved while the run's accounting was open.
const [retry] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId,
source: "on_demand", reason: "task_retry", status: "deferred_issue_execution", requestedByActorType: "user",
updatedAt: new Date(), payload: { issueId: f.issue.id, taskRetry: { sourceRunId: f.run.id },
executionWait: { reason: "execution_settling", message: "Retry is waiting for the previous execution to stop." } } }).returning();
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 1 });
await flushBudgetEvaluationWork(db);
expect((await f.block())?.reason).toMatch(/no reliable price|budget hard-stop/);
expect((await f.readRun()).usageJson).toMatchObject({ accountingClosureTimedOut: true });
expect(await f.block()).toBeNull();
// The cleared hold re-dispatches it: still saved, never cancelled, and due now.
const [promoted] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.id));
expect(promoted.status).toBe("deferred_issue_execution");
expect(promoted.updatedAt.getTime()).toBeLessThanOrEqual(Date.now() - 29_000);
expect(await budgetService(db).getInvocationBlock(f.companyId, f.agentId, { issueId: null, projectId: null })).toBeNull();
const [company] = await db.select().from(companies).where(eq(companies.id, f.companyId));
expect(company).toMatchObject({ status: "active", pauseReason: null });
});
it("keeps the debt pending while native recovery can still produce a result", async () => {
@@ -124,6 +124,36 @@ describe.skipIf(!support.supported)("task interruption latest run query", () =>
expect((await taskInterruptionPresentation(db, parent))?.retryWait?.reason).not.toBe("dependency_blocked");
});
// G1 wave s9802 t02: an accepted Retry held by the unpriced-usage refusal
// read as a budget wait. It is an accounting matter; name it so.
it("labels a Retry held by unpriced usage as accounting, not budget", async () => {
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Unpriced", issuePrefix: `U${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Agent", adapterType: "paperclip_runner" });
const [task] = await db.insert(issues).values({ companyId, title: "Task", identifier: "CHA-2",
status: "blocked", assigneeAgentId: agentId }).returning();
const [todo] = await db.insert(issues).values({ companyId, title: "Not started", identifier: "CHA-3",
status: "todo", assigneeAgentId: agentId }).returning();
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: task.id, scopeKind: "issue",
status: "failed", runtimeMode: "native", errorCode: "native_session_cleanup_quarantined", finishedAt: new Date(Date.now() - 60_000) });
const refusal = "Company cannot start work because recorded usage has no reliable price.";
await db.insert(agentWakeupRequests).values([{ companyId, agentId, source: "on_demand", reason: "task_retry",
status: "deferred_issue_execution", requestedByActorType: "user", requestedAt: new Date(Date.now() - 30_000),
payload: { issueId: task.id, taskRetry: { sourceRunId: runId }, executionWait: { reason: "budget.blocked", message: refusal } } },
{ companyId, agentId, source: "assignment", reason: "budget.blocked", status: "skipped", error: refusal,
requestedAt: new Date(), payload: { issueId: todo.id } }]);
const held = await taskInterruptionPresentation(db, task);
expect(held?.retryWait).toMatchObject({ reason: "accounting.unpriced_usage", message: expect.stringContaining("has no price") });
expect(held?.retryWait?.message).not.toContain("budget to allow");
const waiting = (await taskInterruptionPresentations(db, companyId, [todo])).get(todo.id);
expect(waiting?.retryWait).toMatchObject({ reason: "accounting.unpriced_usage", message: expect.stringContaining("has no price") });
// A budget hard stop keeps its budget label.
await db.update(agentWakeupRequests).set({ payload: { issueId: task.id, taskRetry: { sourceRunId: runId },
executionWait: { reason: "budget.blocked", message: "Company is paused because its budget hard-stop was reached." } } })
.where(eq(agentWakeupRequests.reason, "task_retry"));
expect((await taskInterruptionPresentation(db, task))?.retryWait?.reason).toBe("budget.blocked");
});
it("shows a queued-message Interrupt as handing off, not as a failed attempt", async () => {
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), userId = "board-user";
await db.insert(companies).values({ id: companyId, name: "Interrupted task", issuePrefix: `Q${companyId.slice(0, 7)}` });
+14 -28
View File
@@ -127,44 +127,30 @@ async function readObservedSpends(db: Db, policies: SpendPolicy[], now: Date): P
and ${budgetReservations.runId} = ${heartbeatRuns.id} and ${budgetReservations.state} = 'held')`;
ledgerPredicates.push(ledger); heldPredicates.push(held); runPredicates.push(run);
ledgerColumns[`total${index}`] = sql`coalesce(sum(${costEvents.costCents}) filter (where ${ledger}), 0)::text`;
// "Usage with no reliable price" means observed usage that could not be
// priced. These unpriced charges are not that, so they never trip the
// blocking policy (they stay visible in Costs):
// - a receipt lost with its stopped provider (restart-derived, any usage);
// "Usage with no reliable price" means usage from a complete receipt that
// could not be priced. These unpriced charges are not that, so they never
// trip the blocking policy (they stay visible in Costs):
// - a final charge accounting settled without its receipt: lost with a
// stopped provider, closure evidence that never arrived, or a failed
// capture (its receipt is never certified complete, even if a later
// report adds usage). Accounting is never a company gate (plan decision 1); its
// exposure is recorded, and the activity entry is the board's notice;
// - a charge that carries a known partial cost;
// - metered API or subscription billing with no observed tokens.
// Unknown billing with no tokens (for example Cursor Cloud) still counts.
// The lost-receipt exemption covers only that run's settled final charge;
// later growth from a complete receipt is ordinary usage.
// A final charge settled only because closure evidence never arrived is
// genuinely unknown spend (its provider may still have run), so none of
// these exemptions apply to it, unless a control-plane restart caused it.
// A restart-derived receipt never trips the blocking policy, whatever its
// token count: the run replaced its receipt source, recovered from the
// restart, or lost a provider turn on restore. Its activity entry and
// provenance remain the board's notice.
// The receipt-unavailable exemption covers only that run's settled final
// charge; later growth from a complete receipt is ordinary usage.
const finalCharge = sql`${costEvents.idempotencyKey} = 'heartbeat:' || ${costEvents.heartbeatRunId}::text || ':final'`;
// Every usage recorder binds an empty fence map; only a replaced receipt
// source (a non-empty map) is restart evidence.
const restartDerived = (alias: string) => sql.raw(`(coalesce(${alias}.usage_json->'accountingReceiptSourceFences', '{}'::jsonb) <> '{}'::jsonb
or coalesce(${alias}.usage_json->'pricingProvenance'->>'evidence', '') like '%lost on restore%'
or exists (select 1 from heartbeat_run_events restart_event where restart_event.run_id = ${alias}.id
and restart_event.company_id = ${alias}.company_id and restart_event.event_type = 'native.recovery.transition'
and restart_event.payload->>'trigger' is distinct from 'same_run_retry'))`);
const closureTimedOutCharge = sql`(${finalCharge}
and exists (select 1 from ${heartbeatRuns} timed_out_run where timed_out_run.id = ${costEvents.heartbeatRunId}
and timed_out_run.company_id = ${costEvents.companyId}
and timed_out_run.usage_json->>'accountingClosureTimedOut' = 'true'
-- A failed capture is unknown spend whatever its source history.
and (timed_out_run.usage_json->>'accountingCaptureFailed' = 'true' or not ${restartDerived("timed_out_run")})))`;
const exemptFromUnpricedBlock = sql`(not ${closureTimedOutCharge} and ((${finalCharge}
const exemptFromUnpricedBlock = sql`((${finalCharge}
and exists (select 1 from ${heartbeatRuns} lost_receipt_run where lost_receipt_run.id = ${costEvents.heartbeatRunId}
and lost_receipt_run.company_id = ${costEvents.companyId}
and (lost_receipt_run.usage_json->>'accountingReceiptUnavailable' = 'true'
or lost_receipt_run.usage_json->>'accountingClosureTimedOut' = 'true'
or lost_receipt_run.usage_json->>'accountingCaptureFailed' = 'true'
or coalesce(lost_receipt_run.usage_json->'pricingProvenance'->>'evidence', '') like '%lost on restore%')))
or ${costEvents.costCents} > 0
or (${costEvents.billingType} in ('metered_api', 'api', 'subscription') and ${costEvents.inputTokens} = 0 and ${costEvents.cachedInputTokens} = 0
and ${costEvents.outputTokens} = 0)))`;
and ${costEvents.outputTokens} = 0))`;
ledgerColumns[`unpriced${index}`] = sql`count(*) filter (where ${ledger} and ${costEvents.costStatus} = 'unpriced' and ${costEvents.billingType} <> 'subscription_included'
and not ${exemptFromUnpricedBlock})::int`;
runColumns[`live${index}`] = sql`coalesce(sum(${liveCost}) filter (where ${run} and ${inWindow}), 0)::text`;
+19 -10
View File
@@ -71,9 +71,11 @@ const remoteExecutionTerminated = sql`exists (select 1 from ${environmentLeases}
/** A native run's receipt source is closed only with positive stop evidence,
* after recovery ended in terminal failure without a result:
* - Local: the run no longer records a process identity and its latest
* server-authored process event is the verified local stop. Only a failed
* run closes this way; a cancelled local run keeps its accounting drain,
* since Stop recovery can still reattach a recorder.
* server-authored process event is the verified local stop. This holds for
* any terminal status (failed, cancelled, timed out, interrupted): a stopped
* process reports no more usage. Settlement still imports spooled receipts,
* and a receipt a reattached recorder delivers later posts through
* late-receipt reconciliation.
* - Remote: every allocation has a termination receipt for its sandbox, so
* neither the original nor a reattached runner can report usage. That proof
* does not clear the recorded remote process identity.
@@ -86,9 +88,11 @@ const settledNativeCoordinator = sql`exists (select 1 from ${nativeRunFinalizati
and ((${nativeRunFinalizations.resultId} is null and ${nativeRunFinalizations.phase} = 'terminal_failure')
or ${nativeRunFinalizations.resultId} is not null))`;
// A verified local stop closes the receipt source for any terminal status: a
// restart cancels, times out or interrupts a run as often as it fails one.
const closedNativeReceiptSource = sql`${heartbeatRuns.runtimeMode} = 'native'
and ${settledNativeCoordinator}
and ((${heartbeatRuns.status} = 'failed'
and ((${heartbeatRuns.status} in ('failed', 'cancelled', 'timed_out', 'interrupted')
and ${heartbeatRuns.processPid} is null and ${heartbeatRuns.processGroupId} is null
and (select process_event.event_type from ${heartbeatRunEvents} process_event
where process_event.company_id = ${heartbeatRuns.companyId} and process_event.run_id = ${heartbeatRuns.id}
@@ -120,8 +124,10 @@ const nativeProviderWorkNeverStarted = sql`${heartbeatRuns.runtimeMode} = 'nativ
/** Waiting for closure evidence is bounded. Past the remote runner reconnect
* grace plus a margin, a terminal run whose recovery has settled no longer
* holds new company work: its debt settles as unpriced, flagged so the
* blocking policy still counts it and the board sees which run it was. */
* holds new company work: its debt settles as unpriced and is flagged so the
* board sees which run it was. The flag also exempts the charge from the
* blocking unpriced-usage policy (budget-exposure.ts): accounting is never a
* company gate. */
/** A run's accounting settled: an admission hold may have cleared. Held work
* (queued runs, saved messages and Retries, and tasks whose start or
* continuation was refused) waits passively, so wake its owners. */
@@ -144,8 +150,9 @@ function closureEvidenceTimedOut(at: Date = new Date()) {
* certify what it missed. It is bounded like missing closure evidence: past
* the same bound, once native recovery will not run the provider again, its
* known usage settles as unpriced (or as a lower-bound estimate) and is
* flagged so the blocking policy still counts it. Without the bound such a
* run held company admission until its UTC budget window rolled over. */
* flagged, which exempts it from the blocking unpriced-usage policy
* (budget-exposure.ts). Without the bound such a run held company admission
* until its UTC budget window rolled over. */
function captureFailureBoundPassed(at: Date = new Date()) {
return sql`${heartbeatRuns.usageJson}->>'accountingCaptureFailed' = 'true'
and ${heartbeatRuns.status} in ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted')
@@ -396,8 +403,10 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
// A complete receipt arriving later may price this charge (see usage-receipts).
...(receiptUnavailable ? { usageJson: sql`coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || ${JSON.stringify({
accountingReceiptUnavailable: true,
// Unlike a verified stop, the run may still have spent: the blocking
// unpriced-usage policy keeps counting this charge.
// Unlike a verified stop, the run may still have spent. The flag names
// the run in Costs; like any charge accounting settled without its
// receipt, it is exempt from the blocking unpriced-usage policy
// (budget-exposure.ts), because accounting is never a company gate.
...(closureTimedOut ? { accountingClosureTimedOut: true } : {}),
// Late reconciliation measures a later receipt from the accepted money.
...(lostReceiptEstimate ? { costUsdExact: lostReceiptEstimate.costUsdExact, costStatus: "estimated",
@@ -95,6 +95,14 @@ function boundedJsonString(value: SQL, limit: number) {
return sql<string | null>`case when jsonb_typeof(${value}) = 'string' then left(${value} #>> '{}', ${limit}) else null end`;
}
/** Projected reason for a budget.blocked refusal caused by unpriced usage. */
export const UNPRICED_USAGE_WAIT = "accounting.unpriced_usage";
/** Admission's unpriced-usage refusal text (budgets.ts). Only this fixed
* phrase is matched; the stored message itself is never displayed. */
const unpricedUsageWait = (reason: SQL, message: SQL) =>
sql<string | null>`case when ${reason} = 'budget.blocked' and ${message} like '%recorded usage has no reliable price%'
then ${UNPRICED_USAGE_WAIT} else ${reason} end`;
// Persisted wait messages can contain diagnostics. Ordinary task views expose
// only these stable categories, never arbitrary payloads or source identifiers.
const retryWaitMessages: Record<string, string> = {
@@ -109,6 +117,9 @@ const retryWaitMessages: Record<string, string> = {
execution_active: "Retry is waiting for the current run to finish. Your message is saved.",
decision_pending: "Retry is waiting for a pending approval or question to be resolved. Your message is saved.",
"budget.blocked": "Retry is waiting for the budget to allow new work. Check Costs for unrecorded or unpriced usage.",
// Admission records an unpriced-usage refusal as budget.blocked; the
// projection shows it as the accounting matter it is.
[UNPRICED_USAGE_WAIT]: "Retry is waiting because recorded usage has no price. Check Costs, or allow unpriced usage in the budget policy. Your message is saved.",
finalizing_saved_result: "Paperclip is finishing the saved result before Retry can continue.",
// Another click cannot release a copyback owner that stopped without reporting.
// Retry starts a new recovery generation; the agent does not repeat its work.
@@ -136,6 +147,7 @@ function dependencyRetryWaitMessage(dependencies: RetryDependencies): string {
// assigned-task liveness sweep admits it once the hold clears; no Retry needed.
const admissionWaitMessages: Record<string, string> = {
"budget.blocked": "Waiting for the budget to allow new work. The task starts automatically when it does.",
[UNPRICED_USAGE_WAIT]: "Waiting because recorded usage has no price. Check Costs, or allow unpriced usage in the budget policy.",
};
/** Recognize old persisted failures without displaying arbitrary run error text. */
@@ -349,7 +361,8 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
// A new message must not hide an existing Retry's admission explanation.
db.selectDistinctOn([wakeIssue, isRetryWait], { issueId: wakeIssue, id: agentWakeupRequests.id, requestedAt: agentWakeupRequests.requestedAt,
sourceRunId: boundedJsonString(sql`${agentWakeupRequests.payload}#>'{taskRetry,sourceRunId}'`, 160),
executionWaitReason: boundedJsonString(sql`${agentWakeupRequests.payload}#>'{executionWait,reason}'`, 80),
executionWaitReason: boundedJsonString(sql`to_jsonb(${unpricedUsageWait(sql`${agentWakeupRequests.payload}#>>'{executionWait,reason}'`,
sql`${agentWakeupRequests.payload}#>>'{executionWait,message}'`)})`, 80),
preparationStatus: chatActions.status, preparationRetryAt: chatActionDeadline("failed_run_retry"),
preparationUpdatedAt: chatActions.updatedAt })
.from(agentWakeupRequests).leftJoin(chatActions, and(eq(chatActions.companyId, agentWakeupRequests.companyId),
@@ -377,7 +390,8 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
db.select({ status: companies.status, pauseReason: companies.pauseReason }).from(companies).where(eq(companies.id, companyId)),
// Only consulted for tasks without a run: the latest wake says whether admission refused it.
startingIssueIds.length ? db.selectDistinctOn([wakeIssue], { issueId: wakeIssue, id: agentWakeupRequests.id,
status: agentWakeupRequests.status, reason: agentWakeupRequests.reason, agentId: agentWakeupRequests.agentId })
status: agentWakeupRequests.status, reason: unpricedUsageWait(sql`${agentWakeupRequests.reason}`, sql`${agentWakeupRequests.error}`),
agentId: agentWakeupRequests.agentId })
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId), inArray(wakeIssue, startingIssueIds)))
.orderBy(wakeIssue, desc(agentWakeupRequests.requestedAt), desc(agentWakeupRequests.id)) : [],
]);