mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-08 21:03:51 +02:00
fix(adapters): record unpriced CLI usage (#9505)
## Thinking Path > - Paperclip is the open source control plane people use to manage AI-agent companies > - Budgets and spend telemetry are control-plane safety features, not just reporting > - Local Codex and Claude adapters can execute through either ACP or their native CLI engines > - The ACP lane records usage and reported cost, but CLI JSON output often reports tokens without a price > - The CLI lane was either losing per-run usage semantics or coercing missing cost to zero, making real usage indistinguishable from a genuinely free run > - This pull request preserves CLI usage as per-run totals and records token-bearing runs without a reported price as explicitly unpriced ledger events > - The benefit is accurate usage accounting and a visible pricing gap instead of silently misleading zero-cost telemetry ## Linked Issues or Issue Description Refs #9471 Refs #9230 **Bug description** A `codex_local` run using the CLI engine can emit a final `turn.completed` event with millions of input tokens and tens of thousands of output tokens while the agent's spend ledger remains indistinguishable from a true zero-usage, zero-cost run. Claude CLI output has the same missing-price edge case. **Expected behavior** Token-bearing CLI runs should persist their usage. If the adapter reports a price, the ledger should record it as reported; if the CLI reports usage but no price, the ledger should explicitly mark the event as unpriced rather than silently treating missing price data as a reported `$0` cost. **Reproduction shape** 1. Configure `codex_local` with `engine: cli`. 2. Run a task that produces a `turn.completed` usage payload. 3. Observe token usage in the run stream. 4. Before this change, missing price data is represented as ordinary zero-cost spend and the CLI usage basis is not consistently propagated. ## What Changed - Mark Codex and Claude native CLI usage totals as `per_run` and propagate that basis through success and failure results. - Stop coercing missing Claude CLI cost to `0`. - Add `cost_status` to cost events with `reported` and `unpriced` values, including an idempotent migration and shared validation/types. - Persist token-bearing runs without a reported price as `unpriced` ledger events while retaining zero cents until an authoritative price exists. - Add parser, execute-path, heartbeat-accounting, and cost-service regression coverage for both local CLI adapters. - Document the cost-status invariant and CLI accounting behavior. ## Verification - `pnpm exec vitest run packages/adapters/codex-local/src/server/parse.test.ts packages/adapters/claude-local/src/server/parse.test.ts server/src/__tests__/codex-local-execute.test.ts server/src/__tests__/claude-local-execute.test.ts server/src/__tests__/heartbeat-cost-accounting.test.ts server/src/__tests__/costs-service.test.ts` — 6 files / 102 tests passed. - `pnpm --filter @paperclipai/shared typecheck` - `pnpm --filter @paperclipai/db typecheck` — includes migration numbering and safety checks. - `pnpm --filter @paperclipai/adapter-codex-local typecheck` - `pnpm --filter @paperclipai/adapter-claude-local typecheck` - `pnpm --filter @paperclipai/server typecheck` ## Risks - Existing cost rows default to `reported`, preserving current interpretation; only new token-bearing events with absent cost are marked `unpriced`. - This change does not invent model pricing. Budget hard stops still cannot charge an unknown amount, but operators and evals can now distinguish missing pricing from a genuinely reported zero cost. - Consumers that enumerate cost-event fields should tolerate the additive `costStatus` field. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, exact model `gpt-5.3-codex`, with repository tool use and code execution; default reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
ce7dedf33d
commit
efcce9cc8e
20 files changed
+122
-6
No files matched your search
@@ -720,6 +720,9 @@ describe("claude execute", () => {
|
||||
|
||||
expect(result.exitCode).toBe(0);
|
||||
expect(result.errorMessage).toBeNull();
|
||||
expect(result.usage).toEqual({ inputTokens: 1, cachedInputTokens: 0, outputTokens: 1 });
|
||||
expect(result.usageBasis).toBe("per_run");
|
||||
expect(result.costUsd).toBeNull();
|
||||
expect(loggedCommand).toBe(commandPath);
|
||||
expect(loggedEnv.HOME).toBe(root);
|
||||
expect(loggedEnv.CLAUDE_CONFIG_DIR).toBe(claudeConfigDir);
|
||||
|
||||
@@ -178,6 +178,9 @@ describe("codex execute", () => {
|
||||
|
||||
expect(result.exitCode).toBe(0);
|
||||
expect(result.errorMessage).toBeNull();
|
||||
expect(result.usage).toEqual({ inputTokens: 1, cachedInputTokens: 0, outputTokens: 1 });
|
||||
expect(result.usageBasis).toBe("per_run");
|
||||
expect(result.costUsd).toBeNull();
|
||||
|
||||
const capture = JSON.parse(await fs.readFile(capturePath, "utf8")) as CapturePayload;
|
||||
expect(capture.codexHome).toBe(managedCodexHome);
|
||||
|
||||
@@ -3,6 +3,7 @@ import request from "supertest";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterAll, afterEach, beforeAll } from "vitest";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import {
|
||||
createDb,
|
||||
companies,
|
||||
@@ -432,6 +433,48 @@ describeEmbeddedPostgres("cost and finance aggregate overflow handling", () => {
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
it("persists unpriced token usage without inflating monthly spend", async () => {
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Paperclip",
|
||||
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
await db.insert(agents).values({
|
||||
id: agentId,
|
||||
companyId,
|
||||
name: "CLI Agent",
|
||||
role: "engineer",
|
||||
status: "active",
|
||||
adapterType: "codex_local",
|
||||
adapterConfig: {},
|
||||
runtimeConfig: {},
|
||||
permissions: {},
|
||||
});
|
||||
|
||||
const event = await costs.createEvent(companyId, {
|
||||
agentId,
|
||||
provider: "openai",
|
||||
biller: "chatgpt",
|
||||
billingType: "subscription_included",
|
||||
costStatus: "unpriced",
|
||||
model: "gpt-5.6-terra",
|
||||
inputTokens: 2_732_577,
|
||||
cachedInputTokens: 2_632_998,
|
||||
outputTokens: 32_644,
|
||||
costCents: 0,
|
||||
occurredAt: new Date("2026-07-13T14:22:54.000Z"),
|
||||
});
|
||||
|
||||
expect(event.costStatus).toBe("unpriced");
|
||||
expect(event.inputTokens).toBe(2_732_577);
|
||||
const [agent] = await db.select().from(agents).where(eq(agents.id, agentId));
|
||||
expect(agent?.spentMonthlyCents).toBe(0);
|
||||
});
|
||||
|
||||
it("aggregates cost event sums above int32 without raising Postgres integer overflow", async () => {
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { resolveLedgerCostStatus } from "../services/heartbeat.js";
|
||||
|
||||
describe("heartbeat cost accounting", () => {
|
||||
it("marks token-bearing CLI usage without a reported cost as unpriced", () => {
|
||||
expect(resolveLedgerCostStatus({
|
||||
costUsd: null,
|
||||
inputTokens: 2_732_577,
|
||||
cachedInputTokens: 2_632_998,
|
||||
outputTokens: 32_644,
|
||||
})).toBe("unpriced");
|
||||
});
|
||||
|
||||
it("marks reported CLI cost as priced", () => {
|
||||
expect(resolveLedgerCostStatus({
|
||||
costUsd: 1.25,
|
||||
inputTokens: 2_090,
|
||||
cachedInputTokens: 300_000,
|
||||
outputTokens: 77_000,
|
||||
})).toBe("reported");
|
||||
});
|
||||
});
|
||||
@@ -12,6 +12,7 @@ import {
|
||||
envBindingSchema,
|
||||
isEnvironmentDriverSupportedForAdapter,
|
||||
type BillingType,
|
||||
type CostStatus,
|
||||
type EnvironmentLeaseStatus,
|
||||
type ExecutionWorkspace,
|
||||
type ExecutionWorkspaceConfig,
|
||||
@@ -2367,6 +2368,16 @@ function normalizeBilledCostCents(costUsd: number | null | undefined, billingTyp
|
||||
return Math.max(0, Math.round(costUsd * 100));
|
||||
}
|
||||
|
||||
export function resolveLedgerCostStatus(input: {
|
||||
costUsd: number | null | undefined;
|
||||
inputTokens: number;
|
||||
cachedInputTokens: number;
|
||||
outputTokens: number;
|
||||
}): CostStatus {
|
||||
const hasTokenUsage = input.inputTokens > 0 || input.cachedInputTokens > 0 || input.outputTokens > 0;
|
||||
return input.costUsd == null && hasTokenUsage ? "unpriced" : "reported";
|
||||
}
|
||||
|
||||
async function resolveLedgerScopeForRun(
|
||||
db: Db,
|
||||
companyId: string,
|
||||
@@ -10867,6 +10878,12 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
const billingType = normalizeLedgerBillingType(result.billingType);
|
||||
const additionalCostCents = normalizeBilledCostCents(result.costUsd, billingType);
|
||||
const hasTokenUsage = inputTokens > 0 || outputTokens > 0 || cachedInputTokens > 0;
|
||||
const costStatus = resolveLedgerCostStatus({
|
||||
costUsd: result.costUsd,
|
||||
inputTokens,
|
||||
cachedInputTokens,
|
||||
outputTokens,
|
||||
});
|
||||
const provider = result.provider ?? "unknown";
|
||||
const biller = resolveLedgerBiller(result);
|
||||
const ledgerScope = await resolveLedgerScopeForRun(db, agent.companyId, run);
|
||||
@@ -10897,6 +10914,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
provider,
|
||||
biller,
|
||||
billingType,
|
||||
costStatus,
|
||||
model: result.model ?? "unknown",
|
||||
inputTokens,
|
||||
cachedInputTokens,
|
||||
@@ -12967,6 +12985,12 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
biller: resolveLedgerBiller(adapterResult),
|
||||
model: readNonEmptyString(adapterResult.model) ?? "unknown",
|
||||
...(adapterResult.costUsd != null ? { costUsd: adapterResult.costUsd } : {}),
|
||||
costStatus: resolveLedgerCostStatus({
|
||||
costUsd: adapterResult.costUsd,
|
||||
inputTokens: normalizedUsage?.inputTokens ?? 0,
|
||||
cachedInputTokens: normalizedUsage?.cachedInputTokens ?? 0,
|
||||
outputTokens: normalizedUsage?.outputTokens ?? 0,
|
||||
}),
|
||||
billingType: normalizeLedgerBillingType(adapterResult.billingType),
|
||||
} as Record<string, unknown>)
|
||||
: null;
|
||||
|
||||
Reference in new issue
Block a user