Commit Graph
2 Commits
Author SHA1 Message Date
DottaandPaperclip 199670ba99 fix(accounting): prevent usage receipt stalls during streamed run updates (#20385)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner commits usage receipts before it acknowledges retained
events.
> - Usage receipts and streamed run logs update the same run row.
> - A repeatable-read snapshot can become stale before the receipt
writer locks that row.
> - Continuous log writes can exhaust all accounting retries and stall
the retained event.
> - This pull request uses the existing run lock with READ COMMITTED for
standalone receipt writes.
> - Other accounting operations keep their consistent snapshot. Stall
diagnostics retain safe nested error codes.

## Linked Issues or Issue Description

Related: #20235.

**What happened?**

A hosted native run stopped on a retained usage event. Its diagnostic
reported `Error at source seq N`. The saved outbox and accounting
records locate the failing delivery boundary. Traces show repeated
accounting transactions while streamed events still updated the run row.
The historical inner exception is unavailable.

A controlled PostgreSQL reproduction proves a matching defect: a run-log
append after company validation makes the receipt writer's
repeatable-read snapshot stale. Repeated appends exhaust all four
retries with SQLSTATE `40001`. This is a demonstrated failure mechanism;
attribution of the historical incident remains an inference.

**Expected behavior**

Commit the receipt under its existing run-row lock despite concurrent
run-log writes. Observe concurrent finalization and preserve
late-receipt debt. Keep duplicate receipts idempotent and reject
conflicting or cross-company receipts.

**Steps to reproduce**

Run `server/src/services/usage-receipts-read.test.ts`. The concurrency
regression uses real PostgreSQL and the production run-log append
function. It commits a log append after company validation and before
the receipt locks the run. Before the repair, the receipt rejects with a
wrapped `40001` after four attempts. After the repair, it commits once.
The finalization variant verifies that a newly settled run receives
late-receipt debt without replacing settled state.

## What Changed

- Allow aggregate-lock-protected accounting writers to select READ
COMMITTED. Keep repeatable read as the default.
- Use READ COMMITTED only for standalone usage receipt writes. Keep the
existing run lock, immutable receipt journal, replay checks and
transactional budget-work publication.
- Add PostgreSQL regressions for concurrent log appends, concurrent
finalization, duplicate delivery and the unchanged default snapshot.
- Read the outer event-commit error and at most three nested causes for
an existing safe error code. Exclude exception messages and SQL.
- Document receipt isolation and bounded stall diagnostics.

## Verification

- Red: the production receipt writer exhausted four retries with
PostgreSQL `40001` under controlled concurrent run-log appends.
- Red: wrapped database and transport stall diagnostics returned `Error`
instead of `40001` and `ECONNRESET`.
- Green: Runner control-plane tests passed all 207 cases. All 68 server
assertions passed across receipt reads, native usage delivery and cost
accounting. Cost-accounting teardown exceeded its 30-second limit twice
on this loaded host, after all 28 assertions passed. The workspace
project retained its configured 30-second hook limit despite the CLI
override. This is a local suite cleanup failure, not a passing suite;
the same revision passed the complete server suites in CI.
- Server layered TypeScript checking passed, including source and test
layers. Runner TypeScript checking passed. Server checks cover
`ccfe3bdeb1ab88537a03fd6af9044f66863939a5`; the unchanged Runner checks
cover its parent.
- Local environment: macOS arm64, Node 26.4.0, disposable embedded
PostgreSQL. Native usage-delivery tests use the real Rust receiver.
- CI at `ccfe3bdeb1ab88537a03fd6af9044f66863939a5`:
[build](https://github.com/paperclipai/paperclip/actions/runs/42714760702/job/129493568428),
[typecheck](https://github.com/paperclipai/paperclip/actions/runs/42714760702/job/129493568367),
native Runner integration, browser shards and all nine serialized server
groups passed. All 12 general server shards passed in the [full
run](https://github.com/paperclipai/paperclip/actions/runs/42714760702).
Final rollup: 59 successful checks/statuses, three intentionally skipped
checks, no pending or failed checks.
- [Greptile reviewed this revision at 5/5 with no actionable
findings](https://github.com/paperclipai/paperclip/pull/20385#issuecomment-6753163397).
- Hosted evidence came from the task, run, saved sandbox outbox,
accounting receipts and traces. The run started fresh and did not cross
a deployment. No original agent turn was replayed.
- The hosted repair has not been deployed. The original task has not
been resumed. A fixed-revision remote-provider user journey remains
unverified.

## Risks

- READ COMMITTED changes snapshot timing for standalone receipt writes.
The run-row lock protects all mutable receipt/accounting reads and
serializes competing receipts and finalization. Other accounting
operations retain repeatable read.
- Receipt identity, replay conflict checks, company scoping, budget
publication, retry bounds and acknowledgement rules remain intact.
- No persisted format, provider identity or database schema changes.
Existing saved state needs no migration.
- Diagnostics inspect at most four objects and retain only the existing
identifier format, with an 80-character code limit.
- The reproduced race matches the incident boundary and timing, but the
original inner exception was not recorded. Rollout and live acceptance
remain necessary before incident resolution.

## Model Used

- OpenAI Codex, GPT-6 family, with code and shell tools. The exact model
ID and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 17:35:13 -06:00
Devin FoleyandPaperclip 08dafde1d6 perf(accounting): narrow locked usage receipt run reads (#17559)
Select only the five run fields used by usage receipt persistence under the existing company and row locks. Keep receipt history, replay checks, budget updates, and stored execution evidence intact.

Validation: 37 focused tests, full build and typecheck, exact-head Greptile 5/5 (medium), and full green CI with no unresolved threads. The first CI attempt failed an unchanged Runner shutdown test; its qualified local case and same-head failed-lane rerun passed.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-24 23:58:43 -07:00