mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
fix(accounting): settle a native run that only inherited its predecessor's runner identity as no provider work (#20250)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each finished run settles its accounting. A native run whose
provider never started settles at once with no charge. Other runs wait
for a usage receipt, bounded by a closure timeout.
> - A company budget policy can block unpriced usage. Unpriced cost rows
then trip a hard stop that pauses the company.
> - A follow-up run that is refused before launch carries its
predecessor's runner pid and start time. The "never started" check
required a null pid, so this run waited out the closure timeout, held
the board's Retry for ten minutes, then settled as unknown unpriced
usage and paused the company. The task said "budget" although almost
nothing was spent.
> - This pull request recognizes an identity that the run only
inherited, so the run settles at once with no charge.
> - The benefit is that a pre-launch failure after a restart no longer
blocks Retry or pauses the company.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos wave seed 9802 (local, G1), task t02 graded R3
`retry_no_new_run`. Follow-up run 0b4b404f failed 0.3 s after start with
`native_session_cleanup_quarantined`. It recorded no
`native.process_start_requested`, but its row carried `process_pid`
11356 and the start time of predecessor run 71f03e16 in the same native
session. Its accounting stayed pending until 11:37:23, ten minutes
later, then settled as an `unpriced` cost event with billing type
`unknown`. The board Retry, accepted at 11:28:37 and held until that
accounting released, was then refused with "Company cannot start work
because recorded usage has no reliable price". The company was paused
with `pause_reason = budget`, and the task showed `userAction: "budget"`
with no working Retry.
**Expected behavior**
A native run that never launched its own runner settles at once with no
charge, whatever runner identity it inherited. The held Retry
dispatches, and the company keeps running.
**Steps to reproduce**
1. Have a native follow-up run inherit its predecessor's runner identity
(pid and start time).
2. Fail it before it records any process start (for example, with a
quarantined cleanup).
3. Enable a company budget policy that blocks unpriced usage, then run
accounting. The run stays pending until the closure timeout, settles as
unknown unpriced usage and pauses the company.
**Paperclip version or commit**
master at 0d80b74c21
## What Changed
- `server/src/services/heartbeat/runtime-selection.ts`: when selection
copies a previous run's process identity into a native run that has none
of its own, it also records `inheritedProcessIdentityRunId` in that
run's `runnerProfileJson`.
- `server/src/services/run-cost-accounting.ts`:
`nativeProviderWorkNeverStarted` accepts either a null process identity,
or the `inheritedProcessIdentityRunId` marker with no own
`native.process_start_requested` / `native.process_identity_recorded`
event. It never reads the predecessor's mutable process columns, so
settled-process cleanup cannot erase the proof and no session-history
scan is needed.
- Tests:
- `runtime-selection.test.ts`: selection writes the marker only when a
prior identity is copied. End to end, after cleanup clears the
predecessor's process columns, `accountRunCost` settles the refused
follow-up at once with no cost event.
- `closed-native-receipt-accounting.test.ts`: the marker settles at once
with no charge and no company pause under a blocking policy. An own
recorded start, or an identity without the marker, stays pending.
## Verification
- Red before the fix: the inherited case stays pending (`accountRunCost`
returns `false`).
- `runtime-selection` (except six dynamic-skill tests that need a local
`.claude/skills/slack/SKILL.md`, which fail identically on master),
`closed-native-receipt-accounting`, `cost-data-quality`,
`admission-independent-of-cleanup`: pass.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server`: passes.
## Risks
- Low. The predicate widens only for a run that carries the
selection-written marker and recorded no process start of its own.
Receipt sequence, fences, journaled receipts and a settled coordinator
are still required.
- Previous-release story: forward-only. Runs selected by an older build
have no marker. Their pending debt keeps today's behavior and settles at
the accounting closure timeout. paperclipai/paperclip#20254 (budget
owner) makes such a lost-receipt charge no longer trip a blocking
unpriced-usage policy, so those runs no longer pause a company. No
schema change.
- Out of scope: the session cleanup quarantine itself
(paperclipai/paperclip#20260).
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
4c15eea770
commit
1a007c4ef1
4 files changed
+97
-2
No files matched your search
@@ -288,6 +288,40 @@ const support = await getEmbeddedPostgresTestSupport();
|
||||
expect((await f.block())?.reason).toMatch(/no reliable price|budget hard-stop/);
|
||||
});
|
||||
|
||||
// Chaos wave s9802 t02: a follow-up run refused by a quarantined cleanup
|
||||
// before any launch carried its predecessor's runner identity. Its debt
|
||||
// waited ten minutes, then settled as unknown unpriced usage, and the
|
||||
// blocking policy paused the company and held the board's Retry.
|
||||
it.each([
|
||||
{ label: "settles at once without a charge", inherited: true, ownStart: false },
|
||||
{ label: "keeps the debt pending for its own recorded runner start", inherited: true, ownStart: true },
|
||||
{ label: "keeps the debt pending for an identity it did not record as inherited", inherited: false, ownStart: false },
|
||||
])("$label: a run that only carries its predecessor's runner identity", async ({ inherited, ownStart }) => {
|
||||
const f = await fixture({ unpricedUsagePolicy: "block", processPid: 11356, failureCode: "native_session_cleanup_quarantined" });
|
||||
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.run.id));
|
||||
if (ownStart) await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.run.id, agentId: f.agentId,
|
||||
seq: 1, eventType: "native.process_start_requested", stream: "system" });
|
||||
await db.update(heartbeatRuns).set({ errorCode: "native_session_cleanup_quarantined",
|
||||
runnerProfileJson: inherited ? { inheritedProcessIdentityRunId: randomUUID() } : {},
|
||||
usageJson: { accountingReceiptSourceId: f.sourceId, accountingReceiptSequence: 0, accountingReceiptReady: false,
|
||||
accountingReceiptUnavailable: true, accountingReceiptSourceFences: {},
|
||||
ledgerScope: { issueId: f.issue.id, projectId: null, billingCode: null } } })
|
||||
.where(eq(heartbeatRuns.id, f.run.id));
|
||||
const settled = await accountRunCost(db, f.run.id);
|
||||
if (!inherited || ownStart) {
|
||||
expect(settled).toBe(false);
|
||||
expect((await f.readRun()).costAccountingPending).toBe(true);
|
||||
return;
|
||||
}
|
||||
expect(settled).toBe(true);
|
||||
await budgetService(db).evaluateAffectedScopes(f.companyId, f.agentId, null);
|
||||
await flushBudgetEvaluationWork(db);
|
||||
expect(await f.charges()).toEqual([]);
|
||||
const [company] = await db.select().from(companies).where(eq(companies.id, f.companyId));
|
||||
expect(company).toMatchObject({ status: "active", pauseReason: null });
|
||||
expect(await f.block()).toBeNull();
|
||||
});
|
||||
|
||||
it("keeps the debt pending while the provider process identity is still recorded", async () => {
|
||||
const f = await fixture({ processPid: 424242 });
|
||||
expect(await reconcileRunCosts(db)).toMatchObject({ accounted: 0 });
|
||||
|
||||
@@ -4,7 +4,8 @@ import { execFile as execFileCallback } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { activityLog, agents, authUsers, companyMemberships, companySkillRepositorySnapshots, principalPermissionGrants, environments, environmentLeases, companies, completionContracts, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, workspaceOperations, type Db } from "@paperclipai/db";
|
||||
import { activityLog, agents, authUsers, companyMemberships, companySkillRepositorySnapshots, principalPermissionGrants, environments, environmentLeases, companies, completionContracts, costEvents, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, workspaceOperations, type Db } from "@paperclipai/db";
|
||||
import { accountRunCost } from "../run-cost-accounting.js";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
@@ -330,6 +331,53 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
|
||||
if (changed) expect(await readFile(join(canonical, "memory.txt"), "utf8")).toBe("Synthetic changed memory.\n");
|
||||
}, 45_000);
|
||||
|
||||
it.each([true, false])("records which earlier run a follow-up's process identity came from (prior identity: %s)", async priorIdentity => {
|
||||
const { input, run, agent, issue } = await fixture();
|
||||
const sessionId = randomUUID();
|
||||
const prior = await selectHeartbeatRuntime(db, input);
|
||||
if (!prior.selected || !prior.nativeExecution) throw new Error("Expected prior native selection");
|
||||
const priorInput = { ...prior.nativeExecution, session: { ...prior.nativeExecution.session, normalizedSessionId: sessionId } };
|
||||
const priorProfile = (await read(run.id)).runnerProfileJson;
|
||||
const processStartedAt = new Date(Date.now() - 60_000);
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", nativeSessionId: sessionId,
|
||||
...(priorIdentity ? { processPid: 11_356, processGroupId: 11_356, processStartedAt } : {}),
|
||||
runnerProfileJson: { ...priorProfile, nativeExecutionInput: priorInput, sessionCheckpoint: {
|
||||
backendKind: "runner", driverKind: "codex_app_server", sessionId: "retained-provider-thread",
|
||||
providerSessionId: "retained-provider-thread", identity: { ...priorInput.binding, sessionId },
|
||||
} },
|
||||
}).where(eq(heartbeatRuns.id, run.id));
|
||||
const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: agent.companyId, agentId: agent.id,
|
||||
issueId: issue.id, scopeKind: "issue", status: "running", nativeSessionId: sessionId,
|
||||
createdAt: new Date(run.createdAt.getTime() + 1000),
|
||||
}).returning();
|
||||
input.run = nextRun!;
|
||||
const selected = await selectHeartbeatRuntime(db, input);
|
||||
if (!selected.selected) throw new Error("Expected follow-up native selection");
|
||||
const saved = await read(nextRun!.id);
|
||||
if (priorIdentity) {
|
||||
expect(saved).toMatchObject({ processPid: 11_356, processGroupId: 11_356 });
|
||||
expect(saved.runnerProfileJson?.inheritedProcessIdentityRunId).toBe(run.id);
|
||||
// Settled-process cleanup clears the predecessor before the follow-up's
|
||||
// accounting runs; the follow-up was refused before it launched.
|
||||
await db.update(heartbeatRuns).set({ processPid: null, processGroupId: null, processStartedAt: null })
|
||||
.where(eq(heartbeatRuns.id, run.id));
|
||||
const sourceId = randomUUID();
|
||||
await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), costAccountingPending: true,
|
||||
errorCode: "native_session_cleanup_quarantined",
|
||||
usageJson: { accountingReceiptSourceId: sourceId, accountingReceiptSequence: 0, accountingReceiptReady: false,
|
||||
accountingReceiptUnavailable: true, accountingReceiptSourceFences: {},
|
||||
ledgerScope: { issueId: issue.id, projectId: null, billingCode: null } } })
|
||||
.where(eq(heartbeatRuns.id, nextRun!.id));
|
||||
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", failureCode: "native_session_cleanup_quarantined" })
|
||||
.where(eq(nativeRunFinalizations.runId, nextRun!.id));
|
||||
expect(await accountRunCost(db, nextRun!.id)).toBe(true);
|
||||
expect(await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, nextRun!.id))).toEqual([]);
|
||||
} else {
|
||||
expect(saved.processPid).toBeNull();
|
||||
expect(saved.runnerProfileJson?.inheritedProcessIdentityRunId).toBeUndefined();
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["matching", "different-session", "absent"])("persists the migrated logical cwd only for the receipt's exact session: %s", async receipt => {
|
||||
const sameSession = receipt === "matching";
|
||||
const { input, run, agent, issue } = await fixture();
|
||||
|
||||
@@ -1019,6 +1019,13 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
|
||||
: {}),
|
||||
nativeExecutionInput:
|
||||
lockedProfile.nativeExecutionInput ?? nativeExecution,
|
||||
...(typeof lockedProfile.inheritedProcessIdentityRunId === "string"
|
||||
? { inheritedProcessIdentityRunId: lockedProfile.inheritedProcessIdentityRunId }
|
||||
: lockedRun.processPid == null && lockedRun.processGroupId == null && lockedRun.processStartedAt == null
|
||||
&& previousNativeRun?.nativeSessionId === nativeSessionId
|
||||
&& (previousNativeRun.processPid != null || previousNativeRun.processGroupId != null)
|
||||
? { inheritedProcessIdentityRunId: previousNativeRun.id }
|
||||
: {}),
|
||||
nativeResourceDisposition: providerResourceDispositionForRun,
|
||||
nativeToolContractFingerprint:
|
||||
nativeToolContractFingerprintForTarget(
|
||||
|
||||
@@ -108,6 +108,12 @@ const closedNativeReceiptSource = sql`${heartbeatRuns.runtimeMode} = 'native'
|
||||
* failure, instead of holding company work. A usage recorder binds every run
|
||||
* with an empty fence map; only a replaced receipt source (a non-empty map)
|
||||
* shows that provider work began. */
|
||||
const inheritedProcessIdentityOnly = sql`${heartbeatRuns.runnerProfileJson}->>'inheritedProcessIdentityRunId' is not null
|
||||
and not exists (select 1 from ${heartbeatRunEvents} own_process
|
||||
where own_process.company_id = ${heartbeatRuns.companyId} and own_process.run_id = ${heartbeatRuns.id}
|
||||
and own_process.source_event_id is null
|
||||
and own_process.event_type in ('native.process_start_requested', 'native.process_identity_recorded'))`;
|
||||
|
||||
const nativeProviderWorkNeverStarted = sql`${heartbeatRuns.runtimeMode} = 'native'
|
||||
and ${heartbeatRuns.status} in ('failed', 'cancelled', 'timed_out', 'interrupted')
|
||||
and ${settledNativeCoordinator}
|
||||
@@ -115,7 +121,7 @@ const nativeProviderWorkNeverStarted = sql`${heartbeatRuns.runtimeMode} = 'nativ
|
||||
and coalesce(${heartbeatRuns.usageJson}->'accountingReceiptSourceFences', '{}'::jsonb) = '{}'::jsonb
|
||||
and not exists (select 1 from ${runUsageReceipts} journaled_receipt
|
||||
where journaled_receipt.company_id = ${heartbeatRuns.companyId} and journaled_receipt.run_id = ${heartbeatRuns.id})
|
||||
and ${heartbeatRuns.processPid} is null and ${heartbeatRuns.processGroupId} is null
|
||||
and ((${heartbeatRuns.processPid} is null and ${heartbeatRuns.processGroupId} is null) or (${inheritedProcessIdentityOnly}))
|
||||
and coalesce((select process_event.event_type from ${heartbeatRunEvents} process_event
|
||||
where process_event.company_id = ${heartbeatRuns.companyId} and process_event.run_id = ${heartbeatRuns.id}
|
||||
and process_event.source_event_id is null
|
||||
|
||||
Reference in new issue
Block a user