fix(accounting): close a remote-Runner turn completed across a session restore as a lower bound (#20524)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs settle their charge once the final usage receipt is
complete.
> - #20517 fixed one case for the local Runner. If a restart stops the
provider before a turn's final usage report, and the restored Runner
then reports that turn completed, the turn now closes as a lower bound
instead of waiting 10 minutes.
> - Remote Runners deliver usage through a separate producer-owned
receiver. That receiver dropped `session.resumed`, so the same turn
there still waited for the 10-minute closure timeout.
> - This pull request applies the #20517 rule to that receiver.
> - The benefit is that remote ACPX and OpenCode runs settle promptly
with an honest lower-bound label after a mid-turn restore.

## Linked Issues or Issue Description

Refs #20517.

**What happened?**

After #20517, `session.resumed` reached turn accounting only through the
local Runner's fallback recorder. The remote producer receiver
(`native-usage-delivery.ts`) and the session-authority filter
(`isNativeUsageAccountingEvent`) both dropped it. A remote ACPX or
OpenCode turn that was open across a session restore, and then reported
terminal without a complete report, left the receipt incomplete until
the closure timeout.

**Expected behavior**

Same rule as the local Runner. Such a turn closes as a lower bound right
away. A complete report that arrives after the restore is still used,
and a turn that starts after the restore stays pending until its own
report is complete.

**Steps to reproduce**

`native-usage-delivery.integration.test.ts > a turn restored mid-turn on
a remote Runner > closes the turn as a lower bound, also after the
controller restarts between restore and terminal` fails on master: the
receipt stays `accountingReceiptReady: false`.

## What Changed

- **Receiver (`createNativeUsageReceiver`).** It now accepts
`session.resumed` for turn-reporting providers (ACPX, OpenCode) while a
turn is open. It journals the restore as a receipt like any other
producer event, so turn accounting marks the open turn restored. A
replacement controller rebuilds the same state from that journal.
- **Restores never fail delivery.** A restore from another run or
source, or one with no open turn, is ignored, not rejected. An
unacknowledged lifecycle event would otherwise replay without end.
- **Session-authority filter.** `acceptAccountingEvent` now uses a new
`isNativeUsageDeliveryEvent`: the usage events plus `session.resumed`. A
restore for another run is ignored there too.
- **Recovery routing is unchanged.** `acceptRecoveryAccountingEvent` and
`onRecoveryAccountingEvent` keep the usage-only
`isNativeUsageAccountingEvent`, so a source run's restore never requires
a proven recovery source.
- **Turn accounting** gets a `hasOpenTurn()` helper.
- **Docs:** `doc/SPEC-implementation.md` now covers both Runners.

## Verification

- `native-usage-delivery.integration.test.ts`, describe `a turn restored
mid-turn on a remote Runner` (real PostgreSQL):

  | Test | On master | Here |
  |---|---|---|
| `closes the turn as a lower bound, also after the controller restarts
between restore and terminal`: the restore is journaled by one receiver
and the terminal arrives at a rebuilt one; replaying everything again is
idempotent | fails | passes |
| `keeps the turn's own final report when it arrives after the restore`
(guard) | passes | passes |
| `keeps a turn started after the restore pending, and journals nothing
for a restore with no open turn` (guard) | passes | passes |

- `native-session-executor.test.ts > binds retained warm transport usage
to the successor run authority` covers routing through the
session-authority filter:
  - a current-run restore reaches the successor's capture;
  - a draining-run restore reaches the drained capture;
  - an unrelated-run restore resolves with no capture;
  - recovery routing ignores a source restore.

It fails when the filter is reverted to `isNativeUsageAccountingEvent`.
- Related suites pass: `native-usage-delivery.integration` and
`native-usage-accounting` (46 tests), and the usage, accounting, receipt
and authority cases in `native-session-executor.test.ts` (224 tests).
Server typecheck passes.
- **The live remote path is unproven.** No Daytona wave ran because the
Daytona Runner image has not been rebuilt yet. The local Runner
counterpart (#20517) passed restart-chaos L1, seed 20386, 4/4
(`w-20261110T104332-s20386`).

## Risks

- **Upgrade safety.** Usage state saved by current master has no
journaled restore and rebuilds as before. A receiver that predates this
change replays a journaled restore as an ignored event, which is the
behaviour on master. A restore that an older controller already
acknowledged is not replayed, so that run waits for the closure timeout
as it does today.
- **One extra receipt per mid-turn restore**, only for ACPX and OpenCode
and only while a turn is open. Nothing is added before `turn.start`: a
restore with no open turn returns immediately.
- **Codex and other providers are unaffected.** They do not use turn
accounting.
- No schema or persisted-format change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with shell, file-edit and code-execution tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-10 05:42:52 -06:00
1 parent 16c30ac32b
commit c0593b69f1
6 files changed
+92 -6

No files matched your search

+3 -1
View File
@@ -2033,7 +2033,9 @@ run's other turns do, and its provenance says the observed usage is a lower
bound. The same applies to a turn that was open when the provider session was
restored after process loss and that the restored Runner then reports
terminal without a complete usage report: the provider process that would have
sent the final report was stopped. Raw provider errors and credentials are excluded from
sent the final report was stopped. Local and remote Runners both apply it; the
remote producer journals the restore as a receipt so a replacement controller
rebuilds the same turn state. Raw provider errors and credentials are excluded from
cost details. ACP usage retains the runtime-selected model, including defaults;
conflicting before/after model IDs do not authorize a single-model estimate.
Remote OpenCode billing uses only credentials forwarded to the execution target.
@@ -8698,12 +8698,25 @@ describe("native terminal-turn accounting", () => {
expect(firstCapture).toHaveBeenCalledTimes(2);
expect(secondCapture).toHaveBeenCalledTimes(1);
await expect(retainedHook({ ...firstReport, runId: "unrelated-run" })).rejects.toThrow("native_usage_run_binding_mismatch");
// A session restore reaches the producer that owns its run, so an open
// turn can close as a lower bound; another run's restore is ignored.
const currentResume = { ...secondReport, eventType: "session.resumed" } as PrpEvent;
const drainingResume = { ...firstReport, eventType: "session.resumed" } as PrpEvent;
await retainedHook(currentResume);
expect(secondCapture).toHaveBeenLastCalledWith(currentResume);
await retainedHook(drainingResume);
expect(firstCapture).toHaveBeenLastCalledWith(drainingResume);
const [firstCalls, secondCalls] = [firstCapture.mock.calls.length, secondCapture.mock.calls.length];
await expect(retainedHook({ ...currentResume, runId: "unrelated-run" })).resolves.toBeUndefined();
expect([firstCapture.mock.calls.length, secondCapture.mock.calls.length]).toEqual([firstCalls, secondCalls]);
const recoveryHook = state.createTransport.mock.calls.at(-1)![0].onRecoveryAccountingEvent!;
const source = { runnerInstanceId: "warm-producer", environmentLeaseId: "warm-usage-workspace",
runId: first.binding.runId, normalizedSessionId: first.session.normalizedSessionId!, turnId: "source-turn", itemId: "source-item" };
await recoveryHook(firstReport, source);
expect(firstCapture).toHaveBeenCalledTimes(3);
expect(secondCapture).toHaveBeenCalledTimes(1);
expect(firstCapture).toHaveBeenCalledTimes(firstCalls + 1);
expect(secondCapture).toHaveBeenCalledTimes(secondCalls);
// Recovery routing stays usage-only: a source restore needs no proven source.
await expect(recoveryHook({ ...drainingResume, runId: "foreign-source" }, source)).resolves.toBeUndefined();
await expect(recoveryHook({ ...firstReport, runId: "foreign-source" }, source)).rejects.toThrow("native_recovery_usage_source_unproven");
} finally {
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
@@ -26,7 +26,7 @@ import { prepareRemoteOpenCodeArtifacts, type OpenCodeLaunchArtifacts } from "./
import { rehydratePinnedSkillAssets } from "./runtime-context.js";
import { createNativeUsageDelivery } from "./native-usage-delivery.js";
import { createNativeRecoveryUsageDelivery } from "./native-recovery-usage-delivery.js";
import { createNativeTurnAccounting, NATIVE_LOST_TURN_USAGE_EVIDENCE, nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage, usesNativeUsageDelivery, isNativeUsageAccountingEvent } from "./native-usage-accounting.js";
import { createNativeTurnAccounting, NATIVE_LOST_TURN_USAGE_EVIDENCE, nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage, usesNativeUsageDelivery, isNativeUsageAccountingEvent, isNativeUsageDeliveryEvent } from "./native-usage-accounting.js";
import { crashPoint } from "@paperclipai/shared/crash-points";
export { nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage } from "./native-usage-accounting.js";
import { measureRunnerOperation, recordRunnerTimingReceipt } from "./runner-operation-timing.js";
@@ -1968,7 +1968,9 @@ class SessionAuthorityEpoch {
}
async acceptAccountingEvent(event: PrpEvent) {
if (!isNativeUsageAccountingEvent(event)) return;
if (!isNativeUsageDeliveryEvent(event)) return;
// A restore carries no usage; it never fails delivery for another run.
if (!isNativeUsageAccountingEvent(event) && event.runId !== this.runId && event.runId !== this.#drainingRun?.runId) return;
this.#assertCurrent();
if (event.runId !== this.runId && event.runId !== this.#drainingRun?.runId) throw new Error("native_usage_run_binding_mismatch");
await (event.runId === this.runId ? this.onAccountingEvent : this.#drainingRun?.capture)?.(event);
@@ -143,6 +143,11 @@ export function createNativeTurnAccounting(provider: NativeExecutionInput["provi
return true;
},
snapshot,
/** A restore matters only while a turn is open. */
hasOpenTurn() {
for (const turn of turns.values()) if (!turn.terminal) return true;
return false;
},
finish(turnId: string | null) {
const turn = turnId ? turns.get(turnId) : undefined;
if (turn) turn.terminal = true;
@@ -215,3 +220,11 @@ export function usesNativeUsageDelivery(input: {
export function isNativeUsageAccountingEvent(event: Pick<PrpEvent, "eventType">) {
return ["usage.reported", "turn.started", "turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"].includes(event.eventType);
}
/** Producer delivery for the run itself also needs the session restore: a
* turn open across it can close as a lower bound (see createNativeTurnAccounting).
* Recovery routing keeps isNativeUsageAccountingEvent: a source run's restore
* carries no usage and must not require a proven recovery source. */
export function isNativeUsageDeliveryEvent(event: Pick<PrpEvent, "eventType">) {
return isNativeUsageAccountingEvent(event) || event.eventType === "session.resumed";
}
@@ -559,6 +559,54 @@ describe("producer-owned native usage", () => {
expect((await read()).usageJson).toMatchObject({ accountingReceiptReady: true });
});
describe("a turn restored mid-turn on a remote Runner", () => {
const restoredSeed = async () => ({ ...await seed(), provider: { kind: "opencode" as const, model: "test-model" } });
const partial = (input: Awaited<ReturnType<typeof restoredSeed>>, sequence: number) => {
const value = event(input, sequence, "usage.reported");
(value.payload as Record<string, unknown>).runDeltaAvailable = false;
return value;
};
const read = async (runId: string) => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0];
// The local Runner rule from #20517: K1 stopped the provider before the
// turn's final report; the restored Runner reported the turn completed.
it("closes the turn as a lower bound, also after the controller restarts between restore and terminal", async () => {
const input = await restoredSeed();
const history = [event(input, 1, "turn.started"), partial(input, 2), event(input, 3, "session.resumed", undefined as never)];
const done = event(input, 4, "turn.completed");
let delivery = await createNativeUsageDelivery(db, input);
for (const value of history) await delivery.capture(value);
// A replacement controller rebuilds the receiver from the saved journal.
delivery = await createNativeUsageDelivery(db, input);
await delivery.capture(done);
expect((await read(input.runId)).usageJson).toMatchObject({ accountingReceiptReady: true,
pricingProvenance: { evidence: expect.stringContaining("lower bound") } });
// Replaying every event into another receiver is idempotent.
delivery = await createNativeUsageDelivery(db, input);
for (const value of [...history, done]) await delivery.capture(value);
expect((await read(input.runId)).usageJson).toMatchObject({ accountingReceiptReady: true });
});
it("keeps the turn's own final report when it arrives after the restore", async () => {
const input = await restoredSeed();
const delivery = await createNativeUsageDelivery(db, input);
for (const value of [event(input, 1, "turn.started"), partial(input, 2), event(input, 3, "session.resumed", undefined as never),
event(input, 4, "usage.reported"), event(input, 5, "turn.completed")]) await delivery.capture(value);
const usage = (await read(input.runId)).usageJson;
expect(usage).toMatchObject({ accountingReceiptReady: true, outputTokens: 10 });
expect(JSON.stringify(usage?.pricingProvenance ?? {})).not.toContain("lower bound");
});
it("keeps a turn started after the restore pending, and journals nothing for a restore with no open turn", async () => {
const input = await restoredSeed();
const delivery = await createNativeUsageDelivery(db, input);
for (const value of [event(input, 1, "session.resumed", undefined as never), event(input, 2, "turn.started"), partial(input, 3),
event(input, 4, "turn.completed")]) await delivery.capture(value);
expect((await read(input.runId)).usageJson).toMatchObject({ accountingReceiptReady: false });
expect(await db.select().from(runUsageReceipts).where(eq(runUsageReceipts.runId, input.runId))).toHaveLength(3);
});
});
it("replays partial and final usage once after a lost commit response and restores turn aggregation", async () => {
const input = { ...await seed(), provider: { kind: "opencode" as const, model: "test-model" } };
let loseResponse = false;
@@ -155,7 +155,15 @@ function createNativeUsageReceiver(db: Db, input: NativeUsageDeliveryInput, sour
let pending: UsageReceiptEnvelope | undefined;
let chain = Promise.resolve();
const capture = async (event: PrpEvent) => {
if (event.eventType !== "usage.reported" && event.eventType !== "turn.started" && !terminal.has(event.eventType)) return;
// A restore is journaled only for turn-reporting providers with an open
// turn, so a rebuilt receiver can close that turn as a lower bound.
// Receivers that predate it replay the receipt as an ignored event.
if (event.eventType === "session.resumed") {
// Never fail delivery for a lifecycle event: an unacknowledged event
// would replay without end. Another source's restore is not ours.
if (!turnAccounting || event.runId !== input.runId || event.sourceInstanceId !== input.runnerInstanceId
|| (!turnAccounting.hasOpenTurn() && !accepted.has(event.sourceEventId))) return;
} else if (event.eventType !== "usage.reported" && event.eventType !== "turn.started" && !terminal.has(event.eventType)) return;
if (event.runId !== input.runId || event.sourceInstanceId !== input.runnerInstanceId) throw new Error("native_usage_source_mismatch");
const payload = event.payload as Record<string, unknown>;
if (event.eventType === "usage.reported" && await captureLateUsageForOwner(db, input, event, payload)) return;