2 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	41c42895fa	feat(evals): enforce fixed Runner request allocations (#17452) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner evals execute real provider harnesses against maintained fixtures. > - Turn-level usage arrives after requests have already spent money. > - Parallel cells need distinct allowances before requests start. > - Missing output must preserve the remaining allowance and any known charges. > - This pull request adds a controller that reuses the existing request ledger and gateway. > - It makes a bounded local Runner attempt reviewable before hosted scheduling is enabled. ## Linked Issues or Issue Description Refs #17440. That PR added the explicit Runner gateway route. This PR owns request admission and retained campaign accounting for that route. The existing hosted matrix is not enabled or changed here. Its global allocation, rerun rejection, and private evidence archive remain separate integration work. ## What Changed - Validate a trusted manifest with exact request, CLI, package, daemon, model, and source identities. Assign each selected cell a distinct allowance. Count operating overhead once. Reject allocations above the campaign limit before launch. - Run native Codex or ACPX Claude through the shared request proxy and append-only ledger. Pass only the proxy credential to the child. Refuse local attempt reuse and preserve existing output files. - Close gateway admission on cancellation. Retain the ledger, receipts, route descriptor, child result, and cleanup evidence outside the disposable workspace. - Recompute campaign accounting from retained ledgers. Missing or invalid final evidence holds the full cell allowance while preserving known charges. Reports distinguish provisional and incomplete costs. - Accept namespaces containing only local function/custom tools. Disable web search in the controlled Codex profile. Continue to reject hosted tools, including tools hidden inside a namespace. - Add controller tests to the existing Runner static test path and document this source-checkout entry point. ## Verification - 20 controller tests pass. They use real child processes, local HTTP, and the actual ledger. Cases cover reservations before dispatch, concurrent cell isolation, caps, cancellation, replay rejection, lost artifacts, changed receipts, and Braintrust Claude routing. Upstream responses are synthetic. - 52 shared gateway tests and 55 Runner CLI tests pass. The namespace acceptance and web-search configuration regressions failed before the repair and pass afterward. - Runner TypeScript/Rust checks, Runner build, shared E2E typecheck, and `git diff --check` pass. - The first actual-harness probe found that Codex advertised both a local tool namespace and web search. It reached no upstream transport. That failure is retained; the compatibility repair does not allow paid hosted tools. - An earlier test failure showed that separate dynamic imports created different admission-error classes. The shared runtime bridge now loads one module graph. Another regression showed that a missing final report hid known ledger charges; the collector now preserves those charges. - The repaired actual-harness probe passed its control checks for native Codex and ACPX Claude: each reached exactly one synthetic upstream request after the ledger reservation; both retained the unresolved hold and route evidence; both recorded clean process exit. A scan of 41 retained files found no synthetic upstream credential. Deliberate provider rejection preserved the failed Codex turn and Claude timeout; these are not semantic passes. Prices, operating bounds, and the package archive identity in this probe were synthetic. - Review regressions reproduced lost request/receipt evidence, lost admission metadata, executable replacement after verification, and a SIGTERM-resistant descendant surviving the CLI. The repairs now pass. Executable bytes are staged before launch; a trusted Node resolution hook preserves relative imports. A symlink-path regression also failed before its repair and now passes. The final staged native probe repeated both one-request checks successfully. - Full current-head [CI](https://github.com/paperclipai/paperclip/actions/runs/40115932485) passed on `4781487215116afd10a415d4db53b1ec8e004dd5`: all 58 check runs and the legacy status are successful, neutral, or skipped. Greptile and Paperclip both reviewed this commit at 5/5. All five finding threads are fixed and resolved. No paid provider request ran during this change. ## Risks - This is a trusted source-checkout controller, not a published package command. It needs the repository's installed shared cost dependencies. - The manifest declares reviewed prices and full-lifecycle bounds. Synthetic tests do not qualify real prices, invoices, storage, or Braintrust metering. - The attempt-directory guard is local. Hosted fanout still needs unique global allocations, rerun rejection, and durable private retention. Do not enable the existing matrix on this change alone. - Forced child termination can leave cleanup unverified. That attempt remains incomplete and holds its full allocation. These checks do not claim a host security boundary against an adversarial target. - The current private publisher does not archive the separate controller directory. Native trace relay, private report integration, and live qualification remain open. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. This runtime does not expose the exact model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-26 07:53:57 -05:00
DottaandPaperclip	d1fdfced9d	feat(evals): route Runner sessions through a controller gateway (#17440) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Runner Evals test provider behavior against a mock control plane. > - A campaign budget needs to observe and admit each provider request. > - The existing eval CLI does not select the provider-home mode needed for a controller-assigned route. > - A proxy URL alone therefore cannot establish controlled Codex routing. > - This change adds an explicit route with an isolated home and retained identity. > - The controller can use that route as the basis for request accounting and admission. ## Linked Issues or Issue Description **What happened?** The direct Runner eval CLI passes the ambient environment to the native transport. The transport deliberately removes an environment-only preconfigured-home marker unless its explicit option is also set. This prevents a controller from assigning a Codex gateway through that marker alone. **Expected behavior** A controller can explicitly route a supported eval attempt through its local proxy. The provider receives the proxy token and isolated configuration. The artifact identifies the assigned route on success and failure. Ordinary direct runs keep their current path. **Steps to reproduce** Run the new CLI tests for `--gateway-config`. They validate the descriptor, inspect the actual transport inputs and provider sanitizers, and check retained artifacts and owned-home cleanup. The local probe also ran built native Codex and ACPX Claude against a fake loopback API using only synthetic credentials. Related request-accounting work: https://github.com/paperclipai/paperclip/pull/17432. ## What Changed - Add an explicit gateway descriptor bound to the requested provider, exact model, campaign, execution, and attempt. - Accept only the controller's IPv4 loopback URL and a separate proxy token. - Create a fresh provider home for native Codex and ACPX Claude. Exclude ambient provider credentials, subscription logins, proxies, and startup hooks. - Retain route identity and descriptor hash in successful and failed artifacts. Remove the owned home in finalization. - Document the CLI contract and its limits. The Runner eval CLI owns this behavior. It uses the existing native preconfigured-home contract. Production routing, permissions, and provider protocols are unchanged. ## Verification - Initial verification passed 56 focused gateway/CLI tests and 246 CLI and adjacent provider-isolation tests. After the admission repair, all 247 CLI and adjacent provider-isolation tests passed, with TypeScript checks and build passing again. - Runner TypeScript checks and build passed. - The native daemon rebuilt from the candidate base with locked, offline dependencies. - Local fake-API probe: six authenticated Responses requests from native Codex and six authenticated Messages requests from ACPX Claude reached the assigned endpoint. Claude also made an unauthenticated hello check. The final commit retained both route artifacts. All 27 retained files were scanned for the synthetic proxy token and contained none. Both temporary homes were removed, and all four recorded provider/daemon PIDs were absent after exit. - The fake API intentionally rejected inference. Codex retained a failed turn; Claude retained a timeout. These are route observations, not passing semantic evals or live provider qualification. No paid provider calls ran. - The first focused command used the wrong directory and discovered no tests. It was corrected before the successful runs above. - Full Runner validation: 3,967 tests passed, 13 skipped, and five failed in three unchanged installation and native-settlement files. The failures were two installation timeouts and three retained-cleanup settlement failures. One isolated run passed all 109 applicable tests with six skips. This shows recovery, not a proven cause for the original five failures; those remain retained. Hosted [CI 40111220754](https://github.com/paperclipai/paperclip/actions/runs/40111220754) passed on final commit `a5c47b64e56c1228fca718b72a81ddc3b1d185a4`, including repository typecheck, build, tests and Runner checks. All 57 checks passed or were neutral/skipped, and the legacy status passed. Greptile and Paperclip reviews both scored 5/5 on that commit; no findings remain unresolved. - Rebased onto current master `71c04c0bfb205cc9e7f7bc20c29fd76051b44cc4`. Its added change concerns server cleanup; Runner source is unchanged. - An actual ACPX Codex probe failed before reaching the fake API. Review and source inspection confirmed that Runner deliberately rejects its preconfigured home to protect Runner state. This route is now rejected before service creation. Both explicit and default ACPX Codex admission regressions failed before the repair. The unsupported lane is not presented as qualified. ## Risks This is an explicit new CLI route. Unsupported providers fail before service creation. Upstream route is a controller declaration, not independent billing evidence. The local probe saw six Codex requests where legacy usage counted one, so the controller's request ledger must remain the budget source. Campaign allocation, proxy lifecycle, hosted workflow wiring, dedicated OTLP relay routing, and paid qualification are separate remaining integration steps. This PR enables none of those schedules. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, including the information available in this session - [x] I checked ROADMAP.md; this is eval infrastructure, not a competing core feature - [x] I searched for duplicate PRs for this exact route change - [x] I described the issue in-PR using the bug template labels - [x] I have not referenced internal or instance-local issues - [x] My branch describes the change and contains no internal issue ID - [x] Relevant focused tests pass - [x] I added tests for the new contract and lifecycle - [x] I updated the relevant documentation - [x] I described the risks and unqualified behavior - [x] All current-head Paperclip CI gates are green - [x] Current-head Greptile is 5/5 with no unresolved findings - [x] I will address review findings before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-24 09:34:10 -07:00