DottaandClaude Opus 5.5
8a34c1b62c
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Pending run accounting holds new company work until it settles (#19497, #19606, #19673) > - A run settles only with closure evidence: a verified local stop, or a terminated remote sandbox. Without that evidence it held the whole company forever. In restart-chaos storm 4, a single run held 35 tasks for hours > - Two more shapes kept the hold alive. A run whose result had already been accepted never counted as closed. A recovery that already owned its held reservation was refused by other tasks' recovering accounting, so concurrent recoveries waited on each other > - This pull request bounds the hold, closes accepted-result runs on a verified stop, and lets an owned recovery finish > - The benefit is that no single missing receipt can wedge a company, while genuinely unknown spend still surfaces under the blocking policy ## Linked Issues or Issue Description Follow-up to #19497, #19606 and #19673. **What happened?** - Storm 4: run `f879895f` timed out in `session.open`. Its local stop was never verified (its process identity is still recorded), recovery ended in `terminal_failure`, and its receipt never became ready. From then on, every new-work wake got 409 "New work must wait while a native run recovers its unfinished accounting". That was 93 refusals, 35 blocked tasks, and no Retry. - Storm 4: run `6d3c76bf` was resuming as the recovery owner with its own held reservation. `reserveRunBudget` refused it because other tasks' runs were recovering. - Wave s7401: run `0a3a849d` failed after its result was accepted, with a verified local stop. It had 10 receipts across 2 sources, all incomplete with 0 tokens. It stayed "completed runs await accounting" because closure only covered coordinators in terminal failure with no result. **What did you expect?** A run without closure evidence holds new work only for a bounded time, then settles visibly. An accepted-result run with a verified stop closes like a terminal failure. A recovery that already owns admitted work can finish. **Steps to reproduce** 1. Restart the control plane during several native turns so that one run's local stop is never verified. 2. Observe the run stay "recovering" and every new wake refused indefinitely. ## What Changed - `run-cost-accounting.ts`: - A coordinator counts as settled when it ended in terminal failure without a result, or when it has an accepted result. The verified-stop and remote-termination closures use that rule, which covers the s7401 shape. - A native run whose provider never started settles at once, like a pre-provider failure, with no charge and its reservation released. Its recovery has settled, no usage receipt was ever journaled (a provider turn journals one as soon as it starts), and its local process never launched or was verified stopped (wave s7403, run 3c1d7b70, `session.open` timeout). - New bound `ACCOUNTING_CLOSURE_TIMEOUT_MS`: the remote runner reconnect grace (7 minutes) plus 3 minutes. A terminal native run with a settled coordinator and no closure evidence settles at the bound: - its known usage is recorded as unpriced (or estimated from the rate card), with provenance evidence "closure evidence never arrived…"; - it is marked `accountingClosureTimedOut`; - a `cost.accounting_closure_timed_out` activity entry names the run. - The reconciler selects these runs, and `readCostAccountingWork` schedules a wake at the bound. - `budget-exposure.ts`: a timed-out closure's final charge is exempt from none of the unpriced-block exemptions. Under the `block` policy it raises the normal budget notice ("recorded usage has no reliable price") instead of a silent hold. - `budgets.ts`, `budget-reservations.ts`, `run-dispatch/adapters/postgres.ts`: when a validated recovery owner continues admitted work, admission and both dispatch rechecks apply only hard stops and pauses. Pending and recovering accounting holds still gate new work. - `doc/SPEC-implementation.md`: documents the bound, the accepted-result closure and the provenance. ## Verification - `closed-native-receipt-accounting.test.ts` (embedded Postgres). All of these fail on master and pass here: - s7401 shape (accepted result, verified stop, all receipts incomplete with 0 tokens): settles and clears admission. - A run without closure evidence: held before the bound with a scheduled wake, then settled with the "closure evidence never arrived" provenance, the `accountingClosureTimedOut` flag and an activity entry. Admission clears under `allow`. - Storm 4 `f879895f` shape (terminal failure, recorded process identity, no stop event): held, then released after the bound. - The blocking policy still blocks on a timed-out closure. - s7403 shape (no receipts, model unknown): a run whose process never launched, or was verified stopped, settles at once with no charge and its reservation released. With an unverified process it waits for the bound. - Storm 4 `6d3c76bf` shape: a recovery owner with its held reservation reserves past another task's recovering accounting, while new work for another task still waits. - Existing suites pass locally: budget-enforcement-scoping, budget-enforcement-worker, native-stop-accounting, cost-data-quality, heartbeat-cost-accounting, run-cost-accounting, budgets-service. Server typecheck passes. - The other storm 4 terminal-failure runs (`2e477ae9`, `e4d2e14d`, `6d3c76bf`, `7d21a44c`, `4bc49df0`, `1d85fbd9`) were already settled by #19673 in that database (`cost_accounting_pending=false`) and were not holding admission. `f879895f` alone held it. ## Risks - After the bound, a run whose provider might still be running settles as unpriced. Under the `block` policy that raises a hard-stop budget notice, by design. Under `allow`, work continues and a late receipt still reconciles. - A recovery owner skips accounting holds, so concurrent recoveries can spend while other tasks' accounting is unknown. Hard stops still apply, consistent with best-effort budget enforcement. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Feature modules
A feature module under server/src/modules/<name>/ uses three layers. Each
layer has one rule: which layer below it, it may import from.
adapters → application → domain
domain/holds pure business rules. A domain file takes plain data in and returns plain data out. A domain file must not importdrizzle-orm,@paperclipai/db, a service underserver/src/services/, a route underserver/src/routes/, or a Node.js I/O module (node:child_process,node:fs,node:net). A domain function must not read the system clock; the caller passesnowas an explicitDatevalue.application/holds use cases and the ports they need. A use case takes its ports as constructor arguments and calls domain functions for policy decisions. An application file must not importdrizzle-orm, a SQL client, a concrete adapter, or the server's HTTP error helpers. The outer service or route translates application errors into transport responses.adapters/holds the concrete implementations of the ports: Postgres queries, transactions, and process control. An adapter file may importdrizzle-orm,@paperclipai/db, and Node.js I/O modules.
A module exposes its commands through index.ts, which composes the adapters
and the use cases behind a factory function. Code outside the module imports
that entry point, never a file inside domain/, application/, or
adapters/ directly.
pnpm check:module-boundaries enforces these rules for production source
files. It also checks access to the company deletion entry point below.
Company deletion
The existing company deletion service owns the database transaction and deletion
order. A module can expose a separate company-deletion.ts entry point.
Only services/company-deletion.ts can import this entry point.
The entry point exports an implementation of CompanyDeletionParticipant from
lib/company-deletion.ts. Do not export it through the normal module index.
The service must complete external cleanup before it starts the transaction. It must lock the company before it deletes dependent records or calls a module. Creation commands must use the same company lock. The service removes dependent records before it calls the module. The module checks its deletion conditions and deletes the records it owns. Use the supplied transaction. Do not start another transaction or call external systems. Throw an error if deletion cannot proceed. An error must roll back all database changes in the deletion transaction.
Call each module explicitly in dependency order. No module registry is required.
The agent lifecycle module also has a restricted configuration.ts entry point.
It saves configuration and invalidates verification in the caller's transaction.
The lifecycle guide lists its permitted callers.
Ordinary configuration updates use the root lifecycle command.
The agent lifecycle module owns agent creation, pause, resume, and termination. Use its commands for these changes.
The execution admission module owns project task-execution policy and the
durable admission records that charge project capacity. Today it exposes only
internal policy commands and the deletion guards that project, agent, and
company deletion call. Settlement releases active charges but keeps each
admission's complete project set for later retained-session resolution.
Execution claims still belong to services/heartbeat/queue.ts and native
review dispatch. See execution admission accounting.