PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-28 12:04:41 +01:00

Commit Graph

Author	SHA1	Message	Date
Dotta	051b425fdf	fix(evals): price explicit response model identities (#17808 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E operations reserve model cost before each provider request. > - Providers can return an exact snapshot name after a request selects a model alias. > - The estimator required the response name to equal the request alias. It left valid snapshot usage unpriced and invalidated its reviewed bound. > - This pull request lets a versioned pricecard declare a small list of exact response names that share its reviewed rates. > - The benefit is correct cost coverage while request selection and unknown-model handling stay strict. ## Linked Issues or Issue Description Refs #17765 for the shared eval gateway request policy. What happened? An eval request can select `gpt-5.4-mini` and receive `gpt-5.4-mini-2026-03-17`. Even with complete usage, the estimator returns `null`. The gateway also rejects that response identity as evidence for its reviewed reservation bound. Expected behavior A pricecard can name exact response identities with the same verified rates. Unlisted snapshots stay unknown. A declared response identity must not become an allowed request identity. Steps to reproduce 1. Configure a route with an alias pricecard and an exact response snapshot. 2. Return complete Responses usage for that snapshot through the gateway. 3. Observe that the old estimator leaves the charge unknown and invalidates the bound. Paperclip version or commit Reproduced in the shared estimator and gateway at `507dc70be26c191e372f09e58dc0965e73bf8f57`. Deployment mode Other: the credential-free Product E2E operations harness. The same library serves attached deployment targets. ## What Changed - Add optional `responseModels` to versioned eval pricecards. Validate one to sixteen unique exact additional identities. - Reuse one response matcher in cost estimation, reviewed bound checks and supported request-policy cache-write attribution. Cache attribution also requires the reviewed provider and request identity to match. Keep exact request-model admission unchanged. - Admit the field in the strict operations manifest. Include it in the existing whole-price bound digest. - Cover fragmented JSON and SSE forwarding, persisted usage, durable ledger replay, partial responses, unknown snapshots, invalid mappings, stale bound evidence and unchanged request admission. - Document mapping provenance, version changes and preservation of historical ledgers. ## Verification - Focused provider usage, gateway and operations tests: 185 passed, zero skipped on the cache attribution follow-up. The exact snapshot assertions failed before the initial repair. Three positive cache-write snapshot tests also failed before the follow-up and now pass. - `pnpm test:e2e:runner:typecheck`: passed after building required workspace declarations. - `pnpm test:e2e:runner:unit`: 4,311 passed, two skipped and four failures. The same four failures reproduce on unchanged base `507dc70` (79 passed, four failed in the two affected files): two Daytona Copilot image identity assertions and two Pi catalog/profile expectation assertions. This PR does not change those files or their source inputs. - `pnpm -r typecheck` and `pnpm build`: passed again on the exact cache attribution follow-up commit. - The private Cloud consumer passed 57 offline compatibility tests again on this exact follow-up source. These cover the declared snapshot estimate, exact request selection, unknown snapshot and incomplete usage handling, ledger holds, and source export admission. No provider calls were made. - `git diff --check`: passed. - `pnpm test:run`: interrupted after repeated embedded PostgreSQL startup failures in existing native-session and other database suites. The failed attempt is preserved and does not count as a full local suite pass. Only this checkout's verified test process group was stopped; it left no processes running. The first sandboxed workspace typecheck could not open a local tsx IPC socket; the later authorized check passes. An initial runner typecheck also lacked fresh-checkout workspace build declarations; its later run passes. - All forwarding tests use local synthetic response bytes. No provider calls, releases, deployments or paid eval campaigns were run for this repair. ## Risks - An incorrect mapping can apply incorrect rates. The pricecard author must verify provider provenance and change the price version. Validation bounds the list and uses exact matching only. - Existing pricecards remain exact-name only when the new field is absent. Adding or changing a mapping invalidates old reviewed bound evidence. Historical ledgers and receipts are not rewritten. ## Model Used OpenAI Codex, GPT-6 family. The exact runtime deployment ID and context window are not exposed in this session. The agent used reasoning, source editing, terminal commands and credential-free tests. No model inference was used by the eval harness for this repair. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-10-26 21:34:30 +00:00
DottaandPaperclip	b6e690044a	feat(evals): allow reviewed bounded provisional accounting (#17365 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E campaigns include model, setup, storage and observability costs. > - Some billing windows close after the tests and reports finish. > - Requiring settled receipts for retained storage can block a good campaign for weeks. > - A reservation alone does not prove that a pending charge has an upper bound. > - This change permits provisional completion only with reviewed evidence for the exact cost configuration. > - Reports preserve unknown amounts and holds until later reconciliation. ## Linked Issues or Issue Description Refs: #17360, #17354 What existing behavior does this improve? The Product E2E scheduled accounting gate and its retained cost reports. Current behavior Every pending receipt fails finalization. A completed test can wait through the retained-storage billing window. The journal has dollar reservations but no explicit qualification of their bounds. Proposed behavior Permit provisional completion when all pending charges have reviewed full-lifecycle bounds, no requests remain in flight, no receipts conflict, and known costs plus held amounts fit the campaign limit. Keep all existing holds and receipt-finality rules. Missing evidence still fails. Reason and benefit An operator can distinguish delayed but bounded billing from unbounded or missing costs. The campaign can finish without claiming that its invoice is settled. Breaking changes The journal accepts optional bound metadata and an invalidation record. New reports add an optional bounded-pending count. Current readers retain old journals and reports without granting them bounds. Older code must not read new bound-bearing journals. The existing daily/weekly enable gates and budget amounts stay unchanged. Additional context This implements the operator-approved provisional-cost policy for the existing eval harness. It does not change product budgets or the agent-eval roadmap. A targeted GitHub search found no duplicate open PR. ## What Changed - Accounting owns a narrow evidence contract: a reviewed audit hash, exact configuration hash and full-lifecycle scope. - Gateway reservations retain that evidence before dispatch. Changed limits or rates fail before dispatch. Unexpected routing, model identity or excessive usage invalidates a bound durably. - Operating allowances can carry separately reviewed bounds for each cost component, including its full storage lifetime. No existing allowance becomes qualified by default. - Reports and Braintrust experiment metadata distinguish provisional accounting, complete receipts, blocked accounting and unavailable evidence. Public output omits private audit hashes. - Scheduled finalization allows qualified provisional costs and explicitly rejects spending overruns, including overruns with complete receipts. - Add regressions for legacy data, restarts, late reconciliation, conflicts, malformed evidence, routing uncertainty and exceeded limits. Document the evidence and migration boundaries. ## Verification - Full offline eval-support suite: 4,062 Vitest tests passed, two existing tests skipped; 137 Node tests passed. - `pnpm test:e2e:runner:typecheck`: passed. - `git diff --check`: passed. - Read the retained hosted qualification journal with the new policy. All six legacy pending charges remain unqualified. Known cost `$0.421245200`, held `$19.327680000`, and committed `$19.748925200` remain unchanged. The journal's SHA-256 is unchanged. The original failed workflow remains failed. - No paid models, network provider calls, receipt imports or reservation releases were used for this verification. - Rebased onto master `4d90634f74`; 142 targeted tests and typecheck passed. - Review reproduced a lost OpenAI input-total limit check when cache details were missing (1 failed, 47 passed). Fix `5fef9ba7c149951a51eae468614a4b89f0b3d6db` preserves unpriced provider usage bounds. Then 178 focused tests, full offline support and typecheck passed. - Current-head hosted CI passed: https://github.com/paperclipai/paperclip/actions/runs/40020746585 (attempt 2). Typecheck, build, all general/Runner/serialized tests and browser gates are green. - CI attempt 1 hit a PostgreSQL deadlock in unchanged chat-test cleanup; its skipped flag restoration caused a second assertion failure. One failed-jobs-only retry passed. Original failure evidence is preserved in attempt 1. No paid eval campaign was rerun. - Greptile reviewed exact head `5fef9ba7c149951a51eae468614a4b89f0b3d6db` at 5/5; its one finding is fixed and resolved. The optional Paperclip review is paused and supplied no assessment. ## Risks - Evidence hashes bind an audit to a configuration; they do not prove the audit is true. Operators must review real provider limits, all price tiers, full resource lifetimes and enforceable infrastructure limits before configuring bounds. - Provisional completion is not invoice settlement. Later billing can reveal overruns, which must be reconciled and inspected. The ledger cannot stop external billing. - Existing pilot journals, configured allowances and price cards have no new bound approval. This PR alone does not qualify or enable recurring paid work. - The gateway cannot qualify a canceled Braintrust request without route confirmation. It preserves that unknown hold. - No new dependency, secret, repository variable, workflow, schedule or cloud resource changes. ## Model Used OpenAI Codex, GPT-6 family. The exact deployment ID and context-window size are not exposed in this session. Used reasoning, repository tools and code execution. No paid evaluation model calls were made for this change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-23 13:57:57 -07:00
DottaandPaperclip	3bd709df8b	Operationalize Product E2E accounting, tracing, and scheduling (#17009 ) ## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E tests exercise real agents, tools, setup and cleanup. > - Repeated runs need retained costs and traces, including failed attempts. > - A passing retry must not hide earlier spending or feedback. > - Recurring work also needs admission limits, private reports and failure detection. > - This change supplies those controls while keeping paid activation behind existing gates. ## Linked Issues or Issue Description Refs #16996 and #17158. This is the consolidated landing of the reviewed eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033, #17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096, #17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127, #17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository permits squash merges only. A local rehearsal showed that squashing each slice into master causes repeated conflicts in dependent slices. This PR preserves their commits and lands their combined implementation in one reviewed integration. Superseded slices will close after this merge. ## What Changed - Retain every attempt's cost receipt, including setup, failures, interruptions and retries. Separate reported amounts, estimates, missing costs and reserved capacity in reports and dashboards. - Add bounded request admission, exact model pricecards, gateway error accounting, cache policy and reconciliation evidence. Refuse unsupported routes and requests that cannot fit the allowance. - Integrate Braintrust gateway traces and OTLP correlation, with secret filtering and experiment imports. Record phase timing, including setup, feedback capture and cleanup. - Collect complaint and suggestion evidence after each attempt, including recovery after failures. Retain explicit review dispositions independently from the product grade. - Add gated daily and weekly selections, measured forecasts, setup/readiness checks, wall-time limits and retained monitor notifications. Include declared non-model overhead in admission while reporting the existing subscription separately. - Keep recurring history private, bound uploads and retention, and use isolated archive environments. Keep public publication an explicit operation. Report and publication jobs install only the committed lockfile, with scripts disabled and isolated stores; stale pins fail closed. - Integrate current master, preserve its Product E2E journey billing and privacy checks, and test that shared setup receipts do not double-count estimated agent usage. Preserve upstream fixture repairs. ## Verification - Complete integration baseline `0649a5ce7544ab20825c2183d2f421157db49f24` includes master `a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head as ancestors. - On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012 Vitest tests passed, two skipped, across 193 files; all 137 Node source-contract tests passed. - Focused billing and Product E2E journey contract checks: 24 tests passed. They cover estimates, unknown receipts, single-owner setup costs, browser-only execution and private publication guards. - Monitor and workflow wiring: all 96 Node tests passed. `node scripts/check-token-gates.mjs`: all gates clean. - `pnpm test:e2e:runner:typecheck` passes after the standard Runner TypeScript build refreshed local declarations. The first attempt rejected the new master `runner_prepared_launch` union member because this checkout retained an older generated declaration; the source union already contained that member. - Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes dependency re-resolution from archive-authorized reporting jobs after the security scan flagged it. All 15 workflow-security tests pass after updating the old two-install assertion. - Hosted [CI run 39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180) passes on the current repaired head, including typecheck, build, general tests, Runner checks, browser shards and canary dry run. Greptile is 5/5 on this exact head with no unresolved threads. The security scan passes after the fixed lockfile-resolution thread was resolved. - Paperclip Review returned a terminal neutral check but an incomplete 0-file assessment on the repaired head: its platform blocked both diff responses as prompt-injection content. This is not a successful code review. No bypass or fabricated pass was attempted; current-head Greptile, security and hosted CI are independent gates. - No paid test, reservation release or recurring-run activation accompanies this landing. Live route qualification and invoice reconciliation remain separate evidence. <details> <summary>Retained earlier validation failures</summary> Earlier broad local runs had two 15-second test timeouts (CLI poll cadence and Retry validation), one embedded-database setup timeout, and later PostgreSQL semaphore exhaustion. The exact timed-out tests passed separately without larger timeouts; the broad-run causes were not established. Those local failures remain distinct from subsequent hosted CI results. The independent #17158 fix records its Muse fixture collision, its UUID correction and hosted verification. Local Muse execution was also blocked by macOS embedded-PostgreSQL startup. No historical failure is treated as a successful paid qualification. </details> ## Risks - Admission is a local control, not a provider invoice cap. Missing provider receipts or resource charges retain unknown amounts and holds; estimates do not become billed dollars. - The bounded path admits only documented local suites and provider routes. Ordinary legacy or separate Runner protocol campaigns have different controls. Merging this code does not qualify additional routes. - Recurrence still needs the existing enable gates and current qualification/allowance evidence. Archive permissions and resources must match the documented private-environment contract. - Legacy reports may lack complete attempt or phase evidence. Reports preserve that limitation and do not infer zero usage from missing data. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact deployment variant and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks; full-suite host limitation disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-23 07:18:37 -07:00

Author

SHA1

Message

Date

Dotta

051b425fdf

fix(evals): price explicit response model identities (#17808 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Product E2E operations reserve model cost before each provider
request.
> - Providers can return an exact snapshot name after a request selects
a model alias.
> - The estimator required the response name to equal the request alias.
It left valid snapshot usage unpriced and invalidated its reviewed
bound.
> - This pull request lets a versioned pricecard declare a small list of
exact response names that share its reviewed rates.
> - The benefit is correct cost coverage while request selection and
unknown-model handling stay strict.

## Linked Issues or Issue Description

Refs #17765 for the shared eval gateway request policy.

**What happened?**

An eval request can select `gpt-5.4-mini` and receive
`gpt-5.4-mini-2026-03-17`. Even with complete usage, the estimator
returns `null`. The gateway also rejects that response identity as
evidence for its reviewed reservation bound.

**Expected behavior**

A pricecard can name exact response identities with the same verified
rates. Unlisted snapshots stay unknown. A declared response identity
must not become an allowed request identity.

**Steps to reproduce**

1. Configure a route with an alias pricecard and an exact response
snapshot.
2. Return complete Responses usage for that snapshot through the
gateway.
3. Observe that the old estimator leaves the charge unknown and
invalidates the bound.

**Paperclip version or commit**

Reproduced in the shared estimator and gateway at
`507dc70be26c191e372f09e58dc0965e73bf8f57`.

**Deployment mode**

Other: the credential-free Product E2E operations harness. The same
library serves attached deployment targets.

## What Changed

- Add optional `responseModels` to versioned eval pricecards. Validate
one to sixteen unique exact additional identities.
- Reuse one response matcher in cost estimation, reviewed bound checks
and supported request-policy cache-write attribution. Cache attribution
also requires the reviewed provider and request identity to match. Keep
exact request-model admission unchanged.
- Admit the field in the strict operations manifest. Include it in the
existing whole-price bound digest.
- Cover fragmented JSON and SSE forwarding, persisted usage, durable
ledger replay, partial responses, unknown snapshots, invalid mappings,
stale bound evidence and unchanged request admission.
- Document mapping provenance, version changes and preservation of
historical ledgers.

## Verification

- Focused provider usage, gateway and operations tests: **185 passed,
zero skipped** on the cache attribution follow-up. The exact snapshot
assertions failed before the initial repair. Three positive cache-write
snapshot tests also failed before the follow-up and now pass.
- `pnpm test:e2e:runner:typecheck`: passed after building required
workspace declarations.
- `pnpm test:e2e:runner:unit`: 4,311 passed, two skipped and four
failures. The same four failures reproduce on unchanged base `507dc70`
(79 passed, four failed in the two affected files): two Daytona Copilot
image identity assertions and two Pi catalog/profile expectation
assertions. This PR does not change those files or their source inputs.
- `pnpm -r typecheck` and `pnpm build`: passed again on the exact cache
attribution follow-up commit.
- The private Cloud consumer passed 57 offline compatibility tests again
on this exact follow-up source. These cover the declared snapshot
estimate, exact request selection, unknown snapshot and incomplete usage
handling, ledger holds, and source export admission. No provider calls
were made.
- `git diff --check`: passed.
- `pnpm test:run`: interrupted after repeated embedded PostgreSQL
startup failures in existing native-session and other database suites.
The failed attempt is preserved and does not count as a full local suite
pass. Only this checkout's verified test process group was stopped; it
left no processes running. The first sandboxed workspace typecheck could
not open a local tsx IPC socket; the later authorized check passes. An
initial runner typecheck also lacked fresh-checkout workspace build
declarations; its later run passes.
- All forwarding tests use local synthetic response bytes. No provider
calls, releases, deployments or paid eval campaigns were run for this
repair.

## Risks

- An incorrect mapping can apply incorrect rates. The pricecard author
must verify provider provenance and change the price version. Validation
bounds the list and uses exact matching only.
- Existing pricecards remain exact-name only when the new field is
absent. Adding or changing a mapping invalidates old reviewed bound
evidence. Historical ledgers and receipts are not rewritten.

## Model Used

OpenAI Codex, GPT-6 family. The exact runtime deployment ID and context
window are not exposed in this session. The agent used reasoning, source
editing, terminal commands and credential-free tests. No model inference
was used by the eval harness for this repair.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

2026-10-26 21:34:30 +00:00

DottaandPaperclip

b6e690044a

feat(evals): allow reviewed bounded provisional accounting (#17365 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Product E2E campaigns include model, setup, storage and
observability costs.
> - Some billing windows close after the tests and reports finish.
> - Requiring settled receipts for retained storage can block a good
campaign for weeks.
> - A reservation alone does not prove that a pending charge has an
upper bound.
> - This change permits provisional completion only with reviewed
evidence for the exact cost configuration.
> - Reports preserve unknown amounts and holds until later
reconciliation.

## Linked Issues or Issue Description

Refs: #17360, #17354

**What existing behavior does this improve?**

The Product E2E scheduled accounting gate and its retained cost reports.

**Current behavior**

Every pending receipt fails finalization. A completed test can wait
through the retained-storage billing window. The journal has dollar
reservations but no explicit qualification of their bounds.

**Proposed behavior**

Permit provisional completion when all pending charges have reviewed
full-lifecycle bounds, no requests remain in flight, no receipts
conflict, and known costs plus held amounts fit the campaign limit. Keep
all existing holds and receipt-finality rules. Missing evidence still
fails.

**Reason and benefit**

An operator can distinguish delayed but bounded billing from unbounded
or missing costs. The campaign can finish without claiming that its
invoice is settled.

**Breaking changes**

The journal accepts optional bound metadata and an invalidation record.
New reports add an optional bounded-pending count. Current readers
retain old journals and reports without granting them bounds. Older code
must not read new bound-bearing journals. The existing daily/weekly
enable gates and budget amounts stay unchanged.

**Additional context**

This implements the operator-approved provisional-cost policy for the
existing eval harness. It does not change product budgets or the
agent-eval roadmap. A targeted GitHub search found no duplicate open PR.

## What Changed

- Accounting owns a narrow evidence contract: a reviewed audit hash,
exact configuration hash and full-lifecycle scope.
- Gateway reservations retain that evidence before dispatch. Changed
limits or rates fail before dispatch. Unexpected routing, model identity
or excessive usage invalidates a bound durably.
- Operating allowances can carry separately reviewed bounds for each
cost component, including its full storage lifetime. No existing
allowance becomes qualified by default.
- Reports and Braintrust experiment metadata distinguish provisional
accounting, complete receipts, blocked accounting and unavailable
evidence. Public output omits private audit hashes.
- Scheduled finalization allows qualified provisional costs and
explicitly rejects spending overruns, including overruns with complete
receipts.
- Add regressions for legacy data, restarts, late reconciliation,
conflicts, malformed evidence, routing uncertainty and exceeded limits.
Document the evidence and migration boundaries.

## Verification

- Full offline eval-support suite: 4,062 Vitest tests passed, two
existing tests skipped; 137 Node tests passed.
- `pnpm test:e2e:runner:typecheck`: passed.
- `git diff --check`: passed.
- Read the retained hosted qualification journal with the new policy.
All six legacy pending charges remain unqualified. Known cost
`$0.421245200`, held `$19.327680000`, and committed `$19.748925200`
remain unchanged. The journal's SHA-256 is unchanged. The original
failed workflow remains failed.
- No paid models, network provider calls, receipt imports or reservation
releases were used for this verification.
- Rebased onto master `4d90634f74`; 142 targeted tests and typecheck
passed.
- Review reproduced a lost OpenAI input-total limit check when cache
details were missing (1 failed, 47 passed). Fix
`5fef9ba7c149951a51eae468614a4b89f0b3d6db` preserves unpriced provider
usage bounds. Then 178 focused tests, full offline support and typecheck
passed.
- Current-head hosted CI passed:
https://github.com/paperclipai/paperclip/actions/runs/40020746585
(attempt 2). Typecheck, build, all general/Runner/serialized tests and
browser gates are green.
- CI attempt 1 hit a PostgreSQL deadlock in unchanged chat-test cleanup;
its skipped flag restoration caused a second assertion failure. One
failed-jobs-only retry passed. Original failure evidence is preserved in
attempt 1. No paid eval campaign was rerun.
- Greptile reviewed exact head
`5fef9ba7c149951a51eae468614a4b89f0b3d6db` at 5/5; its one finding is
fixed and resolved. The optional Paperclip review is paused and supplied
no assessment.

## Risks

- Evidence hashes bind an audit to a configuration; they do not prove
the audit is true. Operators must review real provider limits, all price
tiers, full resource lifetimes and enforceable infrastructure limits
before configuring bounds.
- Provisional completion is not invoice settlement. Later billing can
reveal overruns, which must be reconciled and inspected. The ledger
cannot stop external billing.
- Existing pilot journals, configured allowances and price cards have no
new bound approval. This PR alone does not qualify or enable recurring
paid work.
- The gateway cannot qualify a canceled Braintrust request without route
confirmation. It preserves that unknown hold.
- No new dependency, secret, repository variable, workflow, schedule or
cloud resource changes.

## Model Used

OpenAI Codex, GPT-6 family. The exact deployment ID and context-window
size are not exposed in this session. Used reasoning, repository tools
and code execution. No paid evaluation model calls were made for this
change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-23 13:57:57 -07:00

DottaandPaperclip

3bd709df8b

Operationalize Product E2E accounting, tracing, and scheduling (#17009 )

## Thinking Path

> - Paperclip manages agents and the work they perform.
> - Product E2E tests exercise real agents, tools, setup and cleanup.
> - Repeated runs need retained costs and traces, including failed
attempts.
> - A passing retry must not hide earlier spending or feedback.
> - Recurring work also needs admission limits, private reports and
failure detection.
> - This change supplies those controls while keeping paid activation
behind existing gates.

## Linked Issues or Issue Description

Refs #16996 and #17158. This is the consolidated landing of the reviewed
eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033,
#17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096,
#17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127,
#17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository
permits squash merges only. A local rehearsal showed that squashing each
slice into master causes repeated conflicts in dependent slices. This PR
preserves their commits and lands their combined implementation in one
reviewed integration. Superseded slices will close after this merge.

## What Changed

- Retain every attempt's cost receipt, including setup, failures,
interruptions and retries. Separate reported amounts, estimates, missing
costs and reserved capacity in reports and dashboards.
- Add bounded request admission, exact model pricecards, gateway error
accounting, cache policy and reconciliation evidence. Refuse unsupported
routes and requests that cannot fit the allowance.
- Integrate Braintrust gateway traces and OTLP correlation, with secret
filtering and experiment imports. Record phase timing, including setup,
feedback capture and cleanup.
- Collect complaint and suggestion evidence after each attempt,
including recovery after failures. Retain explicit review dispositions
independently from the product grade.
- Add gated daily and weekly selections, measured forecasts,
setup/readiness checks, wall-time limits and retained monitor
notifications. Include declared non-model overhead in admission while
reporting the existing subscription separately.
- Keep recurring history private, bound uploads and retention, and use
isolated archive environments. Keep public publication an explicit
operation. Report and publication jobs install only the committed
lockfile, with scripts disabled and isolated stores; stale pins fail
closed.
- Integrate current master, preserve its Product E2E journey billing and
privacy checks, and test that shared setup receipts do not double-count
estimated agent usage. Preserve upstream fixture repairs.

## Verification

- Complete integration baseline
`0649a5ce7544ab20825c2183d2f421157db49f24` includes master
`a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head
as ancestors.
- On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012
Vitest tests passed, two skipped, across 193 files; all 137 Node
source-contract tests passed.
- Focused billing and Product E2E journey contract checks: 24 tests
passed. They cover estimates, unknown receipts, single-owner setup
costs, browser-only execution and private publication guards.
- Monitor and workflow wiring: all 96 Node tests passed. `node
scripts/check-token-gates.mjs`: all gates clean.
- `pnpm test:e2e:runner:typecheck` passes after the standard Runner
TypeScript build refreshed local declarations. The first attempt
rejected the new master `runner_prepared_launch` union member because
this checkout retained an older generated declaration; the source union
already contained that member.
- Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes
dependency re-resolution from archive-authorized reporting jobs after
the security scan flagged it. All 15 workflow-security tests pass after
updating the old two-install assertion.
- Hosted [CI run
39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180)
passes on the current repaired head, including typecheck, build, general
tests, Runner checks, browser shards and canary dry run. Greptile is 5/5
on this exact head with no unresolved threads. The security scan passes
after the fixed lockfile-resolution thread was resolved.
- Paperclip Review returned a terminal neutral check but an incomplete
0-file assessment on the repaired head: its platform blocked both diff
responses as prompt-injection content. This is not a successful code
review. No bypass or fabricated pass was attempted; current-head
Greptile, security and hosted CI are independent gates.
- No paid test, reservation release or recurring-run activation
accompanies this landing. Live route qualification and invoice
reconciliation remain separate evidence.

<details>
<summary>Retained earlier validation failures</summary>

Earlier broad local runs had two 15-second test timeouts (CLI poll
cadence and Retry validation), one embedded-database setup timeout, and
later PostgreSQL semaphore exhaustion. The exact timed-out tests passed
separately without larger timeouts; the broad-run causes were not
established. Those local failures remain distinct from subsequent hosted
CI results. The independent #17158 fix records its Muse fixture
collision, its UUID correction and hosted verification. Local Muse
execution was also blocked by macOS embedded-PostgreSQL startup. No
historical failure is treated as a successful paid qualification.

</details>

## Risks

- Admission is a local control, not a provider invoice cap. Missing
provider receipts or resource charges retain unknown amounts and holds;
estimates do not become billed dollars.
- The bounded path admits only documented local suites and provider
routes. Ordinary legacy or separate Runner protocol campaigns have
different controls. Merging this code does not qualify additional
routes.
- Recurrence still needs the existing enable gates and current
qualification/allowance evidence. Archive permissions and resources must
match the documented private-environment contract.
- Legacy reports may lack complete attempt or phase evidence. Reports
preserve that limitation and do not infer zero usage from missing data.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The exact deployment variant and context-window size are not
exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted checks;
full-suite host limitation disclosed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-23 07:18:37 -07:00