mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 12:59:13 +01:00
3bd0d81c7bdf9ebee55bbfb48383dca060cf7f71
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
cff331f6ca |
fix(evals): retain local controller cost origin (#17990)
## Thinking Path > - Paperclip manages AI agents and their work. > - Its eval journals retain model costs and operating reservations under one cap. > - The operating policy records the controller's execution origin. > - The schema currently accepts only GitHub-hosted and AWS Fleet origins. > - A local controller must therefore use an inaccurate label or fail admission. > - This PR adds the local origin and keeps the hosted-only schedule gate. ## Linked Issues or Issue Description Refs #17986. No duplicate open PR or issue was found in the operating-overhead/local search. **What existing behavior does this improve?** The Product E2E operating allowance and its retained journal policy. **Current behavior** At master `40c1ab36cd`, an unscheduled local controller cannot set `runner: "local"`. The generic allowance validator rejects it. Local qualification can use the same accounting controls but cannot record its origin correctly. **Proposed behavior** Accept the local origin in the generic allowance. Keep all component, price, cap, expiry, and receipt rules. The scheduled validator still requires `github-hosted` and rejects a local allowance. **Reason and benefit** Local qualification can retain an accurate execution origin without claiming that its reservations are billed GitHub charges. ## What Changed - Add `local` to the operating allowance's runner type and validator. - Exercise local journal creation, model admission, retained overhead, restart, and reconciliation in the existing boundary test. - Check that a local allowance fails Product scheduled admission. - Pin new hosted Runner controller admission to `github-hosted`. Preserve original labels when reading historical manifests; those reads cannot authorize a new worker. - Exercise the hosted-origin rejection and historical/local reads in both source and copied portable runtimes. - Document the local label and its accounting limits. ## Verification - Before the fix: the new local persistence case and schedule-boundary assertion failed; eight tests passed. - Focused ledger, overhead, operations, and scheduled finalization tests: 71 passed across five files. - Controlled Runner source and portable tests: 86 passed. The new hosted-origin regression failed before the fix. - Full Product support suite at the first commit: 4,524 passed, three skipped, and one retained embedded PostgreSQL bootstrap failure. The same database failure was previously reproduced on a clean baseline. - Separate Node test phase on the host: 137 passed. - Actual Linux controller: database lifecycle tests pass. After installing its missing `zip` utility, the full Product suite has 4,527 passes and one Copilot parent-version coverage failure. That same failure reproduces on clean master `40c1ab36cd` with the same Linux filesystem and dependencies (14 passes, one failure). Copilot files are unchanged. The initial seven ZIP-related failures and the watcher failure remain retained. - The final fresh Linux semantic check passes: `GOMEMLIMIT=2GiB GOGC=50 GOMAXPROCS=1 node node_modules/typescript/bin/tsc --singleThreaded -p tests/runner-e2e/tsconfig.json`. The parallel compiler OOM is retained. No semantic checks were omitted. - Both fresh reviews are 5/5, with the addressed thread resolved. Hosted Build first failed during unchanged Pi setup prefetch with an operation timeout. One job rerun recovered; the original failure remains retained. All 58 latest-head checks are now acceptable. - No paid model requests, sandbox creation, schedule changes, or historical journal edits. ## Risks - The caller supplies this origin. It must describe the actual controller. - `local` does not prove zero compute cost, provide a default allowance, or release a hold. - Both maintained hosted admission paths require the GitHub-hosted controller origin. A previously accepted hosted policy with an AWS/local label now fails new admission. Historical reads preserve the original evidence. ## Model Used - OpenAI GPT-6 through Codex, with tool use and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
bf3ef12f3a |
fix(evals): qualify model spend separately from operating costs (#17928)
## Thinking Path > - Paperclip manages agent work and its budgets. > - Product E2E campaigns must explain each model request and attempt. > - The current completion gate also requires complete operating-cost evidence. > - Long-lived observability and storage charges can block a campaign after its model receipts are complete. > - An operator can now choose model-request readiness in a retained allowance. > - The report keeps full accounting, operating holds and model readiness separate. ## Linked Issues or Issue Description Refs #17889, #17523, #17452. **What happened?** A hosted campaign could pass both product cases and retain complete model receipts, yet fail readiness because operating charges were unresolved. **Expected behavior** An explicit model-readiness policy permits complete model receipts to qualify a run. Unknown operating charges stay visible and reserved. Missing model receipts and budget violations still block qualification. **Steps to reproduce** Use a campaign with four unresolved operating reservations and one completed model request. The regression checks readiness before and after the final model receipt, then checks an operating overrun. The default policy retains the original full-accounting gate. ## What Changed - Add the explicit `accountingReadiness: "model_requests"` allowance policy. Bind it into operating-bound configuration when present. - Derive a model subtotal and receipt-coverage projection from the retained journal. Show it beside full accounting and retained holds. - Use the selected readiness policy in scheduled finalization. Keep product, cleanup, feedback, trace and budget checks. - Refresh two existing eval prerequisites found by the full unit suite: the active Copilot image identity and the model-account catalog mapping. - Document the policy and its limit: it does not certify a full-lifecycle operating-cost ceiling. ## Verification - Regression failed on the old implementation with `Invalid operating overhead fields` and passed after the change. - Focused accounting/operations/finalization checks: 54 tests passed before the final policy assertions. - Existing image/catalog failures reproduced, then their 12 checks passed after the metadata repairs. - Product E2E unit suite: 4,360 passed and 2 skipped, plus 137 Node checks. Product E2E and workspace typechecks and workspace build passed. - Review regressions: 35 focused accounting/finalization checks passed on |
||
|
DottaandPaperclip
|
3bd709df8b |
Operationalize Product E2E accounting, tracing, and scheduling (#17009)
## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E tests exercise real agents, tools, setup and cleanup. > - Repeated runs need retained costs and traces, including failed attempts. > - A passing retry must not hide earlier spending or feedback. > - Recurring work also needs admission limits, private reports and failure detection. > - This change supplies those controls while keeping paid activation behind existing gates. ## Linked Issues or Issue Description Refs #16996 and #17158. This is the consolidated landing of the reviewed eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033, #17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096, #17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127, #17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository permits squash merges only. A local rehearsal showed that squashing each slice into master causes repeated conflicts in dependent slices. This PR preserves their commits and lands their combined implementation in one reviewed integration. Superseded slices will close after this merge. ## What Changed - Retain every attempt's cost receipt, including setup, failures, interruptions and retries. Separate reported amounts, estimates, missing costs and reserved capacity in reports and dashboards. - Add bounded request admission, exact model pricecards, gateway error accounting, cache policy and reconciliation evidence. Refuse unsupported routes and requests that cannot fit the allowance. - Integrate Braintrust gateway traces and OTLP correlation, with secret filtering and experiment imports. Record phase timing, including setup, feedback capture and cleanup. - Collect complaint and suggestion evidence after each attempt, including recovery after failures. Retain explicit review dispositions independently from the product grade. - Add gated daily and weekly selections, measured forecasts, setup/readiness checks, wall-time limits and retained monitor notifications. Include declared non-model overhead in admission while reporting the existing subscription separately. - Keep recurring history private, bound uploads and retention, and use isolated archive environments. Keep public publication an explicit operation. Report and publication jobs install only the committed lockfile, with scripts disabled and isolated stores; stale pins fail closed. - Integrate current master, preserve its Product E2E journey billing and privacy checks, and test that shared setup receipts do not double-count estimated agent usage. Preserve upstream fixture repairs. ## Verification - Complete integration baseline `0649a5ce7544ab20825c2183d2f421157db49f24` includes master `a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head as ancestors. - On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012 Vitest tests passed, two skipped, across 193 files; all 137 Node source-contract tests passed. - Focused billing and Product E2E journey contract checks: 24 tests passed. They cover estimates, unknown receipts, single-owner setup costs, browser-only execution and private publication guards. - Monitor and workflow wiring: all 96 Node tests passed. `node scripts/check-token-gates.mjs`: all gates clean. - `pnpm test:e2e:runner:typecheck` passes after the standard Runner TypeScript build refreshed local declarations. The first attempt rejected the new master `runner_prepared_launch` union member because this checkout retained an older generated declaration; the source union already contained that member. - Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes dependency re-resolution from archive-authorized reporting jobs after the security scan flagged it. All 15 workflow-security tests pass after updating the old two-install assertion. - Hosted [CI run 39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180) passes on the current repaired head, including typecheck, build, general tests, Runner checks, browser shards and canary dry run. Greptile is 5/5 on this exact head with no unresolved threads. The security scan passes after the fixed lockfile-resolution thread was resolved. - Paperclip Review returned a terminal neutral check but an incomplete 0-file assessment on the repaired head: its platform blocked both diff responses as prompt-injection content. This is not a successful code review. No bypass or fabricated pass was attempted; current-head Greptile, security and hosted CI are independent gates. - No paid test, reservation release or recurring-run activation accompanies this landing. Live route qualification and invoice reconciliation remain separate evidence. <details> <summary>Retained earlier validation failures</summary> Earlier broad local runs had two 15-second test timeouts (CLI poll cadence and Retry validation), one embedded-database setup timeout, and later PostgreSQL semaphore exhaustion. The exact timed-out tests passed separately without larger timeouts; the broad-run causes were not established. Those local failures remain distinct from subsequent hosted CI results. The independent #17158 fix records its Muse fixture collision, its UUID correction and hosted verification. Local Muse execution was also blocked by macOS embedded-PostgreSQL startup. No historical failure is treated as a successful paid qualification. </details> ## Risks - Admission is a local control, not a provider invoice cap. Missing provider receipts or resource charges retain unknown amounts and holds; estimates do not become billed dollars. - The bounded path admits only documented local suites and provider routes. Ordinary legacy or separate Runner protocol campaigns have different controls. Merging this code does not qualify additional routes. - Recurrence still needs the existing enable gates and current qualification/allowance evidence. Archive permissions and resources must match the documented private-environment contract. - Legacy reports may lack complete attempt or phase evidence. Reports preserve that limitation and do not infer zero usage from missing data. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact deployment variant and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks; full-suite host limitation disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |