From 1a007c4ef1df02e3968aa2a2bcfac5e2ee847567 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 9 Nov 2026 07:12:01 -0600 Subject: [PATCH] fix(accounting): settle a native run that only inherited its predecessor's runner identity as no provider work (#20250) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each finished run settles its accounting. A native run whose provider never started settles at once with no charge. Other runs wait for a usage receipt, bounded by a closure timeout. > - A company budget policy can block unpriced usage. Unpriced cost rows then trip a hard stop that pauses the company. > - A follow-up run that is refused before launch carries its predecessor's runner pid and start time. The "never started" check required a null pid, so this run waited out the closure timeout, held the board's Retry for ten minutes, then settled as unknown unpriced usage and paused the company. The task said "budget" although almost nothing was spent. > - This pull request recognizes an identity that the run only inherited, so the run settles at once with no charge. > - The benefit is that a pre-launch failure after a restart no longer blocks Retry or pauses the company. ## Linked Issues or Issue Description **What happened?** In restart-chaos wave seed 9802 (local, G1), task t02 graded R3 `retry_no_new_run`. Follow-up run 0b4b404f failed 0.3 s after start with `native_session_cleanup_quarantined`. It recorded no `native.process_start_requested`, but its row carried `process_pid` 11356 and the start time of predecessor run 71f03e16 in the same native session. Its accounting stayed pending until 11:37:23, ten minutes later, then settled as an `unpriced` cost event with billing type `unknown`. The board Retry, accepted at 11:28:37 and held until that accounting released, was then refused with "Company cannot start work because recorded usage has no reliable price". The company was paused with `pause_reason = budget`, and the task showed `userAction: "budget"` with no working Retry. **Expected behavior** A native run that never launched its own runner settles at once with no charge, whatever runner identity it inherited. The held Retry dispatches, and the company keeps running. **Steps to reproduce** 1. Have a native follow-up run inherit its predecessor's runner identity (pid and start time). 2. Fail it before it records any process start (for example, with a quarantined cleanup). 3. Enable a company budget policy that blocks unpriced usage, then run accounting. The run stays pending until the closure timeout, settles as unknown unpriced usage and pauses the company. **Paperclip version or commit** master at 0d80b74c21 ## What Changed - `server/src/services/heartbeat/runtime-selection.ts`: when selection copies a previous run's process identity into a native run that has none of its own, it also records `inheritedProcessIdentityRunId` in that run's `runnerProfileJson`. - `server/src/services/run-cost-accounting.ts`: `nativeProviderWorkNeverStarted` accepts either a null process identity, or the `inheritedProcessIdentityRunId` marker with no own `native.process_start_requested` / `native.process_identity_recorded` event. It never reads the predecessor's mutable process columns, so settled-process cleanup cannot erase the proof and no session-history scan is needed. - Tests: - `runtime-selection.test.ts`: selection writes the marker only when a prior identity is copied. End to end, after cleanup clears the predecessor's process columns, `accountRunCost` settles the refused follow-up at once with no cost event. - `closed-native-receipt-accounting.test.ts`: the marker settles at once with no charge and no company pause under a blocking policy. An own recorded start, or an identity without the marker, stays pending. ## Verification - Red before the fix: the inherited case stays pending (`accountRunCost` returns `false`). - `runtime-selection` (except six dynamic-skill tests that need a local `.claude/skills/slack/SKILL.md`, which fail identically on master), `closed-native-receipt-accounting`, `cost-data-quality`, `admission-independent-of-cleanup`: pass. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server`: passes. ## Risks - Low. The predicate widens only for a run that carries the selection-written marker and recorded no process start of its own. Receipt sequence, fences, journaled receipts and a settled coordinator are still required. - Previous-release story: forward-only. Runs selected by an older build have no marker. Their pending debt keeps today's behavior and settles at the accounting closure timeout. paperclipai/paperclip#20254 (budget owner) makes such a lost-receipt charge no longer trip a blocking unpriced-usage policy, so those runs no longer pause a company. No schema change. - Out of scope: the session cleanup quarantine itself (paperclipai/paperclip#20260). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../closed-native-receipt-accounting.test.ts | 34 +++++++++++++ .../heartbeat/runtime-selection.test.ts | 50 ++++++++++++++++++- .../services/heartbeat/runtime-selection.ts | 7 +++ server/src/services/run-cost-accounting.ts | 8 ++- 4 files changed, 97 insertions(+), 2 deletions(-) diff --git a/server/src/__tests__/closed-native-receipt-accounting.test.ts b/server/src/__tests__/closed-native-receipt-accounting.test.ts index c03b8aa17b5..0e2a0388150 100644 --- a/server/src/__tests__/closed-native-receipt-accounting.test.ts +++ b/server/src/__tests__/closed-native-receipt-accounting.test.ts @@ -288,6 +288,40 @@ const support = await getEmbeddedPostgresTestSupport(); expect((await f.block())?.reason).toMatch(/no reliable price|budget hard-stop/); }); + // Chaos wave s9802 t02: a follow-up run refused by a quarantined cleanup + // before any launch carried its predecessor's runner identity. Its debt + // waited ten minutes, then settled as unknown unpriced usage, and the + // blocking policy paused the company and held the board's Retry. + it.each([ + { label: "settles at once without a charge", inherited: true, ownStart: false }, + { label: "keeps the debt pending for its own recorded runner start", inherited: true, ownStart: true }, + { label: "keeps the debt pending for an identity it did not record as inherited", inherited: false, ownStart: false }, + ])("$label: a run that only carries its predecessor's runner identity", async ({ inherited, ownStart }) => { + const f = await fixture({ unpricedUsagePolicy: "block", processPid: 11356, failureCode: "native_session_cleanup_quarantined" }); + await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.run.id)); + if (ownStart) await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.run.id, agentId: f.agentId, + seq: 1, eventType: "native.process_start_requested", stream: "system" }); + await db.update(heartbeatRuns).set({ errorCode: "native_session_cleanup_quarantined", + runnerProfileJson: inherited ? { inheritedProcessIdentityRunId: randomUUID() } : {}, + usageJson: { accountingReceiptSourceId: f.sourceId, accountingReceiptSequence: 0, accountingReceiptReady: false, + accountingReceiptUnavailable: true, accountingReceiptSourceFences: {}, + ledgerScope: { issueId: f.issue.id, projectId: null, billingCode: null } } }) + .where(eq(heartbeatRuns.id, f.run.id)); + const settled = await accountRunCost(db, f.run.id); + if (!inherited || ownStart) { + expect(settled).toBe(false); + expect((await f.readRun()).costAccountingPending).toBe(true); + return; + } + expect(settled).toBe(true); + await budgetService(db).evaluateAffectedScopes(f.companyId, f.agentId, null); + await flushBudgetEvaluationWork(db); + expect(await f.charges()).toEqual([]); + const [company] = await db.select().from(companies).where(eq(companies.id, f.companyId)); + expect(company).toMatchObject({ status: "active", pauseReason: null }); + expect(await f.block()).toBeNull(); + }); + it("keeps the debt pending while the provider process identity is still recorded", async () => { const f = await fixture({ processPid: 424242 }); expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 0 }); diff --git a/server/src/services/heartbeat/runtime-selection.test.ts b/server/src/services/heartbeat/runtime-selection.test.ts index 51415004914..381c4c329ab 100644 --- a/server/src/services/heartbeat/runtime-selection.test.ts +++ b/server/src/services/heartbeat/runtime-selection.test.ts @@ -4,7 +4,8 @@ import { execFile as execFileCallback } from "node:child_process"; import { promisify } from "node:util"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { activityLog, agents, authUsers, companyMemberships, companySkillRepositorySnapshots, principalPermissionGrants, environments, environmentLeases, companies, completionContracts, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, workspaceOperations, type Db } from "@paperclipai/db"; +import { activityLog, agents, authUsers, companyMemberships, companySkillRepositorySnapshots, principalPermissionGrants, environments, environmentLeases, companies, completionContracts, costEvents, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, workspaceOperations, type Db } from "@paperclipai/db"; +import { accountRunCost } from "../run-cost-accounting.js"; import { eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; @@ -330,6 +331,53 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = if (changed) expect(await readFile(join(canonical, "memory.txt"), "utf8")).toBe("Synthetic changed memory.\n"); }, 45_000); + it.each([true, false])("records which earlier run a follow-up's process identity came from (prior identity: %s)", async priorIdentity => { + const { input, run, agent, issue } = await fixture(); + const sessionId = randomUUID(); + const prior = await selectHeartbeatRuntime(db, input); + if (!prior.selected || !prior.nativeExecution) throw new Error("Expected prior native selection"); + const priorInput = { ...prior.nativeExecution, session: { ...prior.nativeExecution.session, normalizedSessionId: sessionId } }; + const priorProfile = (await read(run.id)).runnerProfileJson; + const processStartedAt = new Date(Date.now() - 60_000); + await db.update(heartbeatRuns).set({ status: "succeeded", nativeSessionId: sessionId, + ...(priorIdentity ? { processPid: 11_356, processGroupId: 11_356, processStartedAt } : {}), + runnerProfileJson: { ...priorProfile, nativeExecutionInput: priorInput, sessionCheckpoint: { + backendKind: "runner", driverKind: "codex_app_server", sessionId: "retained-provider-thread", + providerSessionId: "retained-provider-thread", identity: { ...priorInput.binding, sessionId }, + } }, + }).where(eq(heartbeatRuns.id, run.id)); + const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: agent.companyId, agentId: agent.id, + issueId: issue.id, scopeKind: "issue", status: "running", nativeSessionId: sessionId, + createdAt: new Date(run.createdAt.getTime() + 1000), + }).returning(); + input.run = nextRun!; + const selected = await selectHeartbeatRuntime(db, input); + if (!selected.selected) throw new Error("Expected follow-up native selection"); + const saved = await read(nextRun!.id); + if (priorIdentity) { + expect(saved).toMatchObject({ processPid: 11_356, processGroupId: 11_356 }); + expect(saved.runnerProfileJson?.inheritedProcessIdentityRunId).toBe(run.id); + // Settled-process cleanup clears the predecessor before the follow-up's + // accounting runs; the follow-up was refused before it launched. + await db.update(heartbeatRuns).set({ processPid: null, processGroupId: null, processStartedAt: null }) + .where(eq(heartbeatRuns.id, run.id)); + const sourceId = randomUUID(); + await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), costAccountingPending: true, + errorCode: "native_session_cleanup_quarantined", + usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0, accountingReceiptReady: false, + accountingReceiptUnavailable: true, accountingReceiptSourceFences: {}, + ledgerScope: { issueId: issue.id, projectId: null, billingCode: null } } }) + .where(eq(heartbeatRuns.id, nextRun!.id)); + await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", failureCode: "native_session_cleanup_quarantined" }) + .where(eq(nativeRunFinalizations.runId, nextRun!.id)); + expect(await accountRunCost(db, nextRun!.id)).toBe(true); + expect(await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, nextRun!.id))).toEqual([]); + } else { + expect(saved.processPid).toBeNull(); + expect(saved.runnerProfileJson?.inheritedProcessIdentityRunId).toBeUndefined(); + } + }); + it.each(["matching", "different-session", "absent"])("persists the migrated logical cwd only for the receipt's exact session: %s", async receipt => { const sameSession = receipt === "matching"; const { input, run, agent, issue } = await fixture(); diff --git a/server/src/services/heartbeat/runtime-selection.ts b/server/src/services/heartbeat/runtime-selection.ts index 9d0e54afe62..6af0b2490f6 100644 --- a/server/src/services/heartbeat/runtime-selection.ts +++ b/server/src/services/heartbeat/runtime-selection.ts @@ -1019,6 +1019,13 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele : {}), nativeExecutionInput: lockedProfile.nativeExecutionInput ?? nativeExecution, + ...(typeof lockedProfile.inheritedProcessIdentityRunId === "string" + ? { inheritedProcessIdentityRunId: lockedProfile.inheritedProcessIdentityRunId } + : lockedRun.processPid == null && lockedRun.processGroupId == null && lockedRun.processStartedAt == null + && previousNativeRun?.nativeSessionId === nativeSessionId + && (previousNativeRun.processPid != null || previousNativeRun.processGroupId != null) + ? { inheritedProcessIdentityRunId: previousNativeRun.id } + : {}), nativeResourceDisposition: providerResourceDispositionForRun, nativeToolContractFingerprint: nativeToolContractFingerprintForTarget( diff --git a/server/src/services/run-cost-accounting.ts b/server/src/services/run-cost-accounting.ts index 81c7f524a33..3965daeb6a8 100644 --- a/server/src/services/run-cost-accounting.ts +++ b/server/src/services/run-cost-accounting.ts @@ -108,6 +108,12 @@ const closedNativeReceiptSource = sql`${heartbeatRuns.runtimeMode} = 'native' * failure, instead of holding company work. A usage recorder binds every run * with an empty fence map; only a replaced receipt source (a non-empty map) * shows that provider work began. */ +const inheritedProcessIdentityOnly = sql`${heartbeatRuns.runnerProfileJson}->>'inheritedProcessIdentityRunId' is not null + and not exists (select 1 from ${heartbeatRunEvents} own_process + where own_process.company_id = ${heartbeatRuns.companyId} and own_process.run_id = ${heartbeatRuns.id} + and own_process.source_event_id is null + and own_process.event_type in ('native.process_start_requested', 'native.process_identity_recorded'))`; + const nativeProviderWorkNeverStarted = sql`${heartbeatRuns.runtimeMode} = 'native' and ${heartbeatRuns.status} in ('failed', 'cancelled', 'timed_out', 'interrupted') and ${settledNativeCoordinator} @@ -115,7 +121,7 @@ const nativeProviderWorkNeverStarted = sql`${heartbeatRuns.runtimeMode} = 'nativ and coalesce(${heartbeatRuns.usageJson}->'accountingReceiptSourceFences', '{}'::jsonb) = '{}'::jsonb and not exists (select 1 from ${runUsageReceipts} journaled_receipt where journaled_receipt.company_id = ${heartbeatRuns.companyId} and journaled_receipt.run_id = ${heartbeatRuns.id}) - and ${heartbeatRuns.processPid} is null and ${heartbeatRuns.processGroupId} is null + and ((${heartbeatRuns.processPid} is null and ${heartbeatRuns.processGroupId} is null) or (${inheritedProcessIdentityOnly})) and coalesce((select process_event.event_type from ${heartbeatRunEvents} process_event where process_event.company_id = ${heartbeatRuns.companyId} and process_event.run_id = ${heartbeatRuns.id} and process_event.source_event_id is null