2 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	661f33f994	fix(evals): verify stored gateway request coverage (#17432) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E campaigns retain a request ledger and send native traces and gateway logs to Braintrust. > - Native trace verification did not prove that each charged gateway request had a stored log row. > - A hosted probe showed that `x-bt-span-id` identifies the row `id`, while the stored `span_id` differs. Propagated campaign metadata was absent. > - This pull request joins retained request receipts to the correct Braintrust rows during scheduled finalization. > - Operators can see missing request coverage without changing costs, releasing holds, or rerunning agents. ## Linked Issues or Issue Description Refs #17354 and #17371. **Describe the bug** A campaign could pass native trace and experiment indexing checks while gateway request logs remained unverified. The runbook also claimed that propagated metadata made a request searchable after cancellation before response headers. The hosted gateway did not preserve that metadata in retained evidence. **Steps to reproduce** Inspect a retained gateway response header and query project logs by its `x-bt-span-id`. The matching record uses that value as `id`, not `span_id`. Compare every model reservation with its saved response receipt, including setup and canceled work. **Expected behavior** Verify each request against the stored row and report absent or conflicting evidence. Keep trace coverage separate from final usage and invoice reconciliation. ## What Changed - Added a bounded, read-only gateway index verifier. It matches row IDs, provider and model, and retains the actual span and trace IDs. - Included setup, failed and canceled requests in coverage. Missing receipts or row IDs, duplicate attribution, conflicting records and evidence changes prevent verification. - Added a separate scheduled gateway coverage gate and private receipt. Native trace and experiment checks remain independent. - Shared the existing bounded Braintrust query reader. Queries select only identity fields, with at most 100 IDs per batch, three rounds and a 45-second deadline. - Corrected the runbook's metadata guarantee and documented the historical `gatewaySpanId` receipt field. ## Verification - `pnpm test:e2e:runner:unit`: 4,120 passed, two existing skips; all 137 Node contracts passed. - Final repair: 54 focused tests passed, including scheduled-finalization coverage and all five optional identity fields. Each conflicting-field regression failed before the fix; absent or matching partial metadata remains supported. - `pnpm test:e2e:runner:typecheck`: passed after rebuilding stale local plugin SDK declarations. The initial stale-declaration failure remains recorded in the validation logs. - `git diff --check`: passed. - Read-only live verification of retained evidence: all ten local Codex requests and the one canceled drain-probe request matched stored rows. The hosted Claude campaign matched eight of ten; its two pre-header cancellations correctly remained incomplete. - The first hosted lookup exceeded the 1 MiB read limit when selecting full metadata. The final identity-only projection passed the same lookup. - Original journals stayed byte-identical. These checks made zero model calls and did not change provider data or release holds. - Full repository CI passed on `957ad0b684b80267d230fc4fb56d4f002ef089ad`: [run 40105676630](https://github.com/paperclipai/paperclip/actions/runs/40105676630), including typecheck, tests and build. Both current-head automated reviews scored 5/5 with no open findings. - The superseded initial CI run was canceled after the repair push to release the queue; its completed results remain available. No failed job was retried on the final commit. ## Risks Missing gateway IDs now fail scheduled trace coverage. A failed or canceled call may have no stored row, so this can require investigation even if product assertions pass. Indexing can lag; bounded query retries do not retry model work. Stored identity does not prove final billing or gateway span nesting under native attempt traces. Private identifiers remain in private evidence only. Recurrence and billing settings are unchanged. ## Model Used OpenAI Codex, GPT-6 family. The runtime does not expose a more specific model ID or context-window size in this session. Used reasoning, repository tools, local code execution and read-only API verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-24 08:23:57 -07:00
DottaandPaperclip	3bd709df8b	Operationalize Product E2E accounting, tracing, and scheduling (#17009) ... ## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E tests exercise real agents, tools, setup and cleanup. > - Repeated runs need retained costs and traces, including failed attempts. > - A passing retry must not hide earlier spending or feedback. > - Recurring work also needs admission limits, private reports and failure detection. > - This change supplies those controls while keeping paid activation behind existing gates. ## Linked Issues or Issue Description Refs #16996 and #17158. This is the consolidated landing of the reviewed eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033, #17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096, #17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127, #17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository permits squash merges only. A local rehearsal showed that squashing each slice into master causes repeated conflicts in dependent slices. This PR preserves their commits and lands their combined implementation in one reviewed integration. Superseded slices will close after this merge. ## What Changed - Retain every attempt's cost receipt, including setup, failures, interruptions and retries. Separate reported amounts, estimates, missing costs and reserved capacity in reports and dashboards. - Add bounded request admission, exact model pricecards, gateway error accounting, cache policy and reconciliation evidence. Refuse unsupported routes and requests that cannot fit the allowance. - Integrate Braintrust gateway traces and OTLP correlation, with secret filtering and experiment imports. Record phase timing, including setup, feedback capture and cleanup. - Collect complaint and suggestion evidence after each attempt, including recovery after failures. Retain explicit review dispositions independently from the product grade. - Add gated daily and weekly selections, measured forecasts, setup/readiness checks, wall-time limits and retained monitor notifications. Include declared non-model overhead in admission while reporting the existing subscription separately. - Keep recurring history private, bound uploads and retention, and use isolated archive environments. Keep public publication an explicit operation. Report and publication jobs install only the committed lockfile, with scripts disabled and isolated stores; stale pins fail closed. - Integrate current master, preserve its Product E2E journey billing and privacy checks, and test that shared setup receipts do not double-count estimated agent usage. Preserve upstream fixture repairs. ## Verification - Complete integration baseline `0649a5ce7544ab20825c2183d2f421157db49f24` includes master `a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head as ancestors. - On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012 Vitest tests passed, two skipped, across 193 files; all 137 Node source-contract tests passed. - Focused billing and Product E2E journey contract checks: 24 tests passed. They cover estimates, unknown receipts, single-owner setup costs, browser-only execution and private publication guards. - Monitor and workflow wiring: all 96 Node tests passed. `node scripts/check-token-gates.mjs`: all gates clean. - `pnpm test:e2e:runner:typecheck` passes after the standard Runner TypeScript build refreshed local declarations. The first attempt rejected the new master `runner_prepared_launch` union member because this checkout retained an older generated declaration; the source union already contained that member. - Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes dependency re-resolution from archive-authorized reporting jobs after the security scan flagged it. All 15 workflow-security tests pass after updating the old two-install assertion. - Hosted [CI run 39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180) passes on the current repaired head, including typecheck, build, general tests, Runner checks, browser shards and canary dry run. Greptile is 5/5 on this exact head with no unresolved threads. The security scan passes after the fixed lockfile-resolution thread was resolved. - Paperclip Review returned a terminal neutral check but an incomplete 0-file assessment on the repaired head: its platform blocked both diff responses as prompt-injection content. This is not a successful code review. No bypass or fabricated pass was attempted; current-head Greptile, security and hosted CI are independent gates. - No paid test, reservation release or recurring-run activation accompanies this landing. Live route qualification and invoice reconciliation remain separate evidence. <details> <summary>Retained earlier validation failures</summary> Earlier broad local runs had two 15-second test timeouts (CLI poll cadence and Retry validation), one embedded-database setup timeout, and later PostgreSQL semaphore exhaustion. The exact timed-out tests passed separately without larger timeouts; the broad-run causes were not established. Those local failures remain distinct from subsequent hosted CI results. The independent #17158 fix records its Muse fixture collision, its UUID correction and hosted verification. Local Muse execution was also blocked by macOS embedded-PostgreSQL startup. No historical failure is treated as a successful paid qualification. </details> ## Risks - Admission is a local control, not a provider invoice cap. Missing provider receipts or resource charges retain unknown amounts and holds; estimates do not become billed dollars. - The bounded path admits only documented local suites and provider routes. Ordinary legacy or separate Runner protocol campaigns have different controls. Merging this code does not qualify additional routes. - Recurrence still needs the existing enable gates and current qualification/allowance evidence. Archive permissions and resources must match the documented private-environment contract. - Legacy reports may lack complete attempt or phase evidence. Reports preserve that limitation and do not infer zero usage from missing data. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact deployment variant and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks; full-suite host limitation disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-23 07:18:37 -07:00