mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-05 12:29:37 +01:00
865a191bcef151eac5ebee5b1b89ebc44aa6271f
179
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dotta
|
865a191bce | Evaluate native delivered work with an outstanding human review gate | ||
|
Dotta
|
b53c596bc2 | Preserve paused inventory owners with explicit readiness proofs | ||
|
Dotta
|
1c84384d41 | feat(evals): bind one production chat attachment to its human receipt | ||
|
Dotta
|
1be8e44467 | Evaluate actual chat brief delegation and worker attachment access | ||
|
Dotta
|
0d22c3cc1d | test(e2e): cover secure keys, held resume and draft upload recovery | ||
|
Dotta
|
f4ef7e81b9 | Implement authorized GitHub repository project eval journeys | ||
|
Dotta
|
ad031f1c59 | Reject negated goal rationale in final purpose calibration | ||
|
Dotta
|
5f45ed9fd6 | test(e2e): prove native pause and controller continuity boundaries | ||
|
Dotta
|
131fd66001 | Isolate goal purpose conditional context and grade business meaning | ||
|
Dotta
|
af18a43ca2 | Bind inbox summary grading to actual responsible-user access | ||
|
Dotta
|
8b27fe681d | Correct goal alignment grading to match catalog semantics | ||
|
Dotta
|
875371a262 | test(e2e): verify runtime surfaces and actual task workspace binding | ||
|
Dotta
|
f1762c5609 | Evaluate multi-page open task inventory and honest partial results | ||
|
Dotta
|
45b248d2f0 | test(e2e): cover runtime history and human primary controls | ||
|
Dotta
|
b5f0e6399f | Add authenticated inbox scope and denied archive evals | ||
|
Dotta
|
2574cbc18e | Evaluate private task owner sharing boundary with real observer | ||
|
Dotta
|
3a713fabbe | Add owner-driven project sharing eval and strengthen evidence controls | ||
|
Dotta
|
e652b0de6e | feat(evals): retain genuine pre-query chat history boundaries | ||
|
Dotta
|
5591141864 | test(e2e): add runtime agent and dependency cycle oracles | ||
|
Dotta
|
4c9de65edd | fix(evals): load source chat domains and retain canonical report IDs | ||
|
Dotta
|
ba144e8503 | test(evals): add external chat catalog checkpoint and provider prerequisites | ||
|
Dotta
|
2200f1c2e8 | test(evals): verify private projects with independent human and agent observers | ||
|
Dotta
|
10d0ce6361 | test(evals): add identity scenarios and separate browser boundary dispatch | ||
|
Dotta
|
0c67cb39be | test(evals): exercise delegated task creation and lost send recovery | ||
|
Dotta
|
6ddec3a40d | test(evals): bind goal outcomes to their actual chat phases | ||
|
Dotta
|
caba17c3f0 | test(evals): freeze complete goal chat journeys and calibration | ||
|
DottaandPaperclip
|
de73104b34 |
test: isolate chat case modules and fingerprint executable dependencies
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
bc7b66912b |
test: establish frozen chat catalog campaign and executable case modules
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2ee36a4c0a |
test: add chat feature scenarios and real task workflow evals (#17794)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent Chat lets a person assign work through a conversation. > - Feature and API maps do not prove that a natural request produces the right saved result. > - Product E2E already owns isolated fixtures, browser runs, evidence and cleanup. > - This pull request adds a chat scenario catalog and two real task workflows in that framework. > - Independent checks verify the task, its author, the saved output and the reply in chat. > - The benefit is a small executable foundation for broader chat coverage. ## Linked Issues or Issue Description **What existing behavior does this improve?** It adds real task creation and delegation coverage for Agent Chat. **Current behavior** Existing maps describe features and agent access. Existing chat evals cover specific protocols and handoffs. They do not connect a broad human-request catalog to independent saved-state checks. **Proposed behavior** Keep 244 planned scenarios across 35 feature areas. Promote two requests into the explicit-only `chat-features` suite. One saves a backlog task without starting work. The other delegates an announcement and returns the saved result to the original chat. **Reason and benefit** A successful tool call or confident reply does not prove the requested result exists. These evals check public API state, attribution and browser access after a real model run. **Breaking changes** No API or data schema changes. Chat instructions now require actual relative task links. The local E2E private-state scanner recognizes Codex account ownership metadata while continuing to reject credentials and model-visible identity leaks. **Additional context** Related prior work: #13969 and #13762. This improves verification of the existing Agent Chat roadmap item. It does not add a new core product feature. ## What Changed - Add YAML scenarios with capability IDs, starting context, user requests and expected outcomes. Validate all area files against their schema, global ID uniqueness and exact feature-map capability coverage. - Add two local native Codex subscription cells. Send the exact catalog request once through Chromium. - Seed only prerequisites through public APIs. Use shipped agent instructions and explicit disposable budget policies for unpriced subscription usage. - Check task ownership, creation audit, no premature execution, current worker-authored output, automatic completion delivery and usable links after reload. - Keep private Codex account ownership metadata distinct from bearer credentials. Scan rollout message content, all tokens and published evidence as before. - Teach normal chat instructions to use relative task links instead of invented hosts. Update tests, maps and the runbook. - Repair existing E2E contract drift: include the active Copilot identity in the image digest and update Pi profile expectations. Keep global catalog cardinality in its owning catalog test. Product E2E owns the new flow and grader. Its existing launch and report paths remain in use. The shared scanner change is limited to owner-private temporary Codex runtime metadata. Other callers retain the full secret scan. No UI, endpoint, schema or permission behavior changes. ## Verification - `node scripts/check-maps.mjs` passed: 35 areas and 191 capability IDs. - Product E2E typecheck and focused tests passed before rebase: 225 Vitest tests and 137 Node tests. Conversation service tests passed 40/40. Five existing runner turn-binding regressions passed. - Historical live measurement used real Chromium, server, database and native Codex with no automatic retries. The initial same-definition batch passed 5/6 cells. Backlog passed 3/3; delegation passed 2/3. One CEO startup failed because the runner did not receive a provider turn identity. The cause is unresolved. - An unchanged follow-up passed delegation three times and backlog twice. Combined evidence is backlog 5/5 and delegation 5/6. Each attempt passed cleanup. Later passes do not erase the earlier failure or prove a runner repair. - Earlier failed revisions exposed an invented task hostname, private account metadata false positives and missing funding for unpriced subscription usage. Those led to the specific changes above; the failed records remain retained locally. - The runbook records campaign identifiers and definition provenance. Generated result artifacts are not committed. Historical live results predate this branch rebase; fresh PR-source live evidence and final validation are recorded below. - Full workspace typecheck and production build passed on the rebased branch. Product E2E typecheck passed. - The full Product E2E support suite first reproduced four baseline failures: two missing active Copilot image-identity inputs and two stale Pi/catalog expectations. The narrow contract fixes above resolved them. The full rerun passed 4,282 Vitest tests plus 137 Node tests, with two skipped. Image checks verify mutation changes the hash, unchanged input stays stable, and an input closure without Copilot is unaffected. Original failed logs remain retained. - The repo-wide local Vitest run has reported two failures in unchanged service files. The Workspace connection test passed an isolated recheck and its full file passed 469/469 unchanged. The recovery test hit a 15-second timeout; its full file passed 89/89 unchanged. The run later hit explicit PostgreSQL `semget` resource exhaustion and was interrupted after those observed failures; no final aggregate counts or full local pass are claimed. Its complete available log is retained. No assertion or timeout was weakened. Matching remote server CI passed on the same source. - On clean PR head `39ad80b523ce194c90aafafd077d48dd9a36b4c5`, backlog passed in `local-2026-10-26T19-24-13-014Z`. The parallel delegation fixture failed during PostgreSQL initdb, before browser or model execution; its failed cleanup grade is retained. A disposable standalone initdb probe then passed. No semaphore sets or kernel settings were changed before that probe/recheck. After confirming no remaining fixture process, its private temporary files were removed. - One separate delegation recheck passed in `local-2026-10-26T19-29-24-800Z`, with one attempt, zero automatic retries and successful cleanup. Both passing current-source cells use definition `3f2c9ba2d03e3dcddde47ddab3b43bc52de977f25e256fceae0a24c9128e204b`. The earlier bootstrap failure and the historical provider startup failure remain distinct. These local results omit automatic source-SHA metadata; the clean checkout SHA is recorded here from the execution session. - The later broad local run reported `could not create semaphores: No space left on device`. A bounded cleanup removed 32 current-user PostgreSQL sets with 17 members, marker 537, a dead recorded creator and zero waiters. No live process or kernel limit was changed by that cleanup. The test process group was subsequently stopped; no broad rerun is planned. This direct later resource failure does not prove the precise cause of the earlier live initdb failure. - Review follow-up adds only static catalog validation and documentation of the existing five-minute journey bound. The executable definition is unchanged. Focused catalog/oracle tests passed 28/28; no redundant live run is claimed. - Final head `30fcb36ac9ae1f0270a589dbfafcec91ca6a03f3` passed all required and underlying CI checks in [run 40391402347](https://github.com/paperclipai/paperclip/actions/runs/40391402347), including `ci / verify` and `ci / e2e`. Greptile and Paperclip Review both scored this head 5/5, with all review threads resolved. The static follow-up leaves every executable definition input byte-identical to the live-tested source at `39ad80b`. Normal squash auto-merge is enabled. Actual merge is pending eligible code-owner approval for the owned dependency manifest; no protection is bypassed. ## Risks - Live model behavior can vary. One missing turn-start event remains unexplained; this PR does not claim provider reliability or broad catalog qualification. - Subscription costs are unpriced. Explicit fixture funding admits that usage. Run counts and deadlines remain bounded, but the dollar limit cannot cap unknown cost. - Account identity exceptions apply only to known private Codex runtime files. Token reuse, unknown files and model-visible rollout content remain blocking findings. - Deterministic checks establish saved state, output structure, required brief facts and access. They do not score all prose quality. Initial samples also require human inspection. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, browser and repository tools. The precise model revision and context window are not exposed to this session. Live eval agents used `gpt-5.6-sol`; that is the evaluated model, not a claim about the authoring model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run local tests; focused suites pass and the interrupted full-run failures are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
313c1db2c3 |
fix(evals): share bounded request policy with qualification (#17765)
## Thinking Path > - Paperclip manages agents and their work. > - Product E2E evaluations send model work through a gateway with cost limits. > - The gateway checks the request body before it reserves money or contacts a provider. > - A qualification probe needs that same check before it records a spending-cap denial. > - Namespace support already exists. Qualification must share that policy without sending a paid request. > - This pull request exposes the pure request validator and bounds the supported local tool definitions. > - The benefit is one request policy for qualification and forwarding. ## Linked Issues or Issue Description Refs: #17452, #17469. **What happened?** The request validator was private to `forwardEvalRequest`. A qualification caller could not reuse the request policy before admission. A separate check could accept a request body that forwarding rejects before a reservation. **Expected behavior** Qualification and forwarding use the same pure validator. One-level OpenAI namespaces with local function or custom tools remain supported. Invalid tool definitions fail before accounting or transport work. **Steps to reproduce** Use the synthetic native-daemon request fixture in `eval-gateway.test.ts`. It has seven local function tools and one namespace with five local function tools. Call the shared validator, then forward through the fake provider. Run the malformed namespace and tool-count cases to check rejection before side effects. The namespace behavior follows the [OpenAI function-calling guide](https://developers.openai.com/api/docs/guides/function-calling). This change preserves the one-level contract introduced in #17452 and hardened in #17469. ## What Changed - Export `boundedEvalPayload` with credential-free route fields. `forwardEvalRequest` calls that function. - Keep one namespace level with function or custom leaves. Require exact leaf types, nonblank string leaf names, and a closed namespace wrapper. - Limit OpenAI tool definitions to 1,024 per request. Count namespace wrappers and their children. - Add native-daemon-shaped forwarding coverage, malformed and hosted tool rejection, count and depth boundaries, and Anthropic compatibility checks. - Document the shared qualification policy in the Product E2E security guide. - Compile the real observability validator in the test parent. The owned CLI fixture imports JavaScript, so its process group does not include a test-only TypeScript compiler service. The parent resolves the compiler from the root package, which declares that dependency. Production cleanup checks and attempt deadlines stay intact. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts eval-gateway.test.ts cost-ledger.test.ts provider-usage.test.ts operating-overhead.test.ts operations.test.ts`: 173 passed. - `pnpm test:e2e:runner:typecheck`: passed. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - Separate Node support checks: 137 passed. - `pnpm test:e2e:runner:unit`: 4,271 passed, two skipped, four failed in unchanged Daytona image and Pi catalog contracts. The same four failures reproduce on untouched base commit `a67a18c171816b097f46eae78fc58c7e55c9c565` (79 passed, four failed in those files). - `pnpm test:run`: interrupted and incomplete after required CI completed. Initial database startup failed because this checkout omitted the embedded PostgreSQL package postinstall aliases. The package hydrator restored them, and 14 real database checks passed. Earlier failures and the full output remain preserved; this local full run is not counted as passing verification. Its owned process group exited after SIGINT, with no owned survivors. - An offline full native authentication probe used the real Rust daemon and pinned Codex 0.160.0 Darwin SDK. The same captured request failed the older operations policy before reservation and reached a mock reservation with this policy. The OS sandbox allowed loopback only. It used synthetic credentials, retained no raw model payload, and made no provider calls. - Controller fixture suite: 70 passed. Three cold comparisons each confirmed that the old child started an owned compiler service and the corrected child did not. On Darwin the old compiler also exited cleanly, so this does not prove the exact historical Linux process race. Fresh Linux CI is required. - Independent review passed. The private qualification integration passed 14 exact-source compatibility checks and 67 shared-source tests without provider calls. - Required GitHub CI checks passed on the current head. The separate Entire approval gate still awaits an eligible reviewer; the delegated author account's explicitly AI-assisted approval does not satisfy it. ## Risks Requests with more than 1,024 OpenAI tool definitions or extra namespace wrapper fields now fail closed. New provider schema fields will need review. Anthropic tool handling, model prices, billing restrictions, output caps, and reservation ordering remain unchanged. This offline request-policy proof does not establish a completed model turn or Linux or Cloud onboarding acceptance. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model identifier and context window are not exposed in this session. The model used reasoning, tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
bbe1016d81 |
feat(evals): retain bounded native tracing in the Runner controller (#17493)
Retain bounded native tracing in the Runner controller with attempt-bound relays, private export receipts, offline cost replay, conservative holds and bounded failure diagnostics.
Verified exact head
|
||
|
DottaandPaperclip
|
3f91d58dd5 |
fix(ci): provide Copilot auth only to protected paid eval cells (#17743)
## Thinking Path > - Paperclip manages AI agents and their work. > - Paid runner evals test real provider integrations. > - Copilot cells use the saved-token connection path. > - The two trusted workflows did not pass the Copilot credential to those cells. > - This PR supplies it only when the selected cell needs it. > - Existing environment and actor gates protect access in this public repository. ## Linked Issues or Issue Description Related: #15705 and #15706. A search found no open PR that adds this workflow credential binding. **Describe the bug** Copilot Product and direct-protocol eval cells require `COPILOT_GITHUB_TOKEN`. The protected paid workflows do not currently pass that environment secret to the selected execution step. **Expected behavior** Only a selected Copilot cell receives this credential. Ordinary PR, build, catalog, image and reporting jobs receive none. **Steps to reproduce** Dispatch the trusted full-stack workflow for one Copilot cell. Credential validation fails before the task starts, although the protected environment can hold a valid token. **Environment** GitHub Actions in the public `paperclipai/paperclip` repository, with the reviewed workflow on `master` and the `runner-e2e-paid` environment. ## What Changed - Pass `COPILOT_GITHUB_TOKEN` in the paid Product and direct-protocol execution steps, only when `matrix.credentialName` selects it. - Add the missing Copilot protocol-catalog credential mapping and verify it reaches the selected execution matrix. - Add security checks for the selected-cell binding and bounded protected-job scope. Extend the scan that rejects provider credentials in unrelated workflows. - Document the Copilot credential scope. No secret values, account data or private test evidence are in this PR. ## Verification - Product workflow security: 16 tests passed on this head. - Protocol catalog and workflow security: 21 tests passed, including the Copilot matrix regression reproduced red before its fix. - Current-head required `ci / verify` and `ci / e2e` are green. Greptile is 5/5 with zero unresolved review threads. - The first CI attempt retained runner-shutdown failures; one failed-job retry passed on the unchanged head. Both attempts remain in [the CI run](https://github.com/paperclipai/paperclip/actions/runs/40342429106). - `git diff --check` passed. - A selected-credential scan found zero matching secret values in the seven changed files. - The environment setting was separately checked: `master` only, required approval from existing allowlisted code owners, and administrator bypass disabled. The new secret exists only at environment scope; no repository-wide duplicate was created. - The packaged Copilot runtime verified the selected token and the explicit model without a model prompt. A fresh local UI journey then created an agent, completed its first task and verified the downloaded artifact and cleanup. ## Risks The selected paid cells gain access to one additional environment credential. Missing credentials still fail closed. Default-branch checks, initiating/rerunning actor checks, protected environment access and per-cell credential selection remain in place. This PR adds no trigger and gives no credential to builds or ordinary PR checks. ## Model Used OpenAI Codex, based on GPT-6, with repository tools and browser regression tests. The exact runtime model identifier is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
49bd7a4ba6 |
feat(evals): retain portable Runner controller evidence (#17469)
Retain private Runner request evidence for offline cost reconciliation. Bound post-child reads, reject hidden paid tools, exclude configured credentials, bind incomplete results, and replace redacted proof atomically.
Verified at
|
||
|
DottaandPaperclip
|
41c42895fa |
feat(evals): enforce fixed Runner request allocations (#17452)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner evals execute real provider harnesses against maintained fixtures. > - Turn-level usage arrives after requests have already spent money. > - Parallel cells need distinct allowances before requests start. > - Missing output must preserve the remaining allowance and any known charges. > - This pull request adds a controller that reuses the existing request ledger and gateway. > - It makes a bounded local Runner attempt reviewable before hosted scheduling is enabled. ## Linked Issues or Issue Description Refs #17440. That PR added the explicit Runner gateway route. This PR owns request admission and retained campaign accounting for that route. The existing hosted matrix is not enabled or changed here. Its global allocation, rerun rejection, and private evidence archive remain separate integration work. ## What Changed - Validate a trusted manifest with exact request, CLI, package, daemon, model, and source identities. Assign each selected cell a distinct allowance. Count operating overhead once. Reject allocations above the campaign limit before launch. - Run native Codex or ACPX Claude through the shared request proxy and append-only ledger. Pass only the proxy credential to the child. Refuse local attempt reuse and preserve existing output files. - Close gateway admission on cancellation. Retain the ledger, receipts, route descriptor, child result, and cleanup evidence outside the disposable workspace. - Recompute campaign accounting from retained ledgers. Missing or invalid final evidence holds the full cell allowance while preserving known charges. Reports distinguish provisional and incomplete costs. - Accept namespaces containing only local function/custom tools. Disable web search in the controlled Codex profile. Continue to reject hosted tools, including tools hidden inside a namespace. - Add controller tests to the existing Runner static test path and document this source-checkout entry point. ## Verification - 20 controller tests pass. They use real child processes, local HTTP, and the actual ledger. Cases cover reservations before dispatch, concurrent cell isolation, caps, cancellation, replay rejection, lost artifacts, changed receipts, and Braintrust Claude routing. Upstream responses are synthetic. - 52 shared gateway tests and 55 Runner CLI tests pass. The namespace acceptance and web-search configuration regressions failed before the repair and pass afterward. - Runner TypeScript/Rust checks, Runner build, shared E2E typecheck, and `git diff --check` pass. - The first actual-harness probe found that Codex advertised both a local tool namespace and web search. It reached no upstream transport. That failure is retained; the compatibility repair does not allow paid hosted tools. - An earlier test failure showed that separate dynamic imports created different admission-error classes. The shared runtime bridge now loads one module graph. Another regression showed that a missing final report hid known ledger charges; the collector now preserves those charges. - The repaired actual-harness probe passed its control checks for native Codex and ACPX Claude: each reached exactly one synthetic upstream request after the ledger reservation; both retained the unresolved hold and route evidence; both recorded clean process exit. A scan of 41 retained files found no synthetic upstream credential. Deliberate provider rejection preserved the failed Codex turn and Claude timeout; these are not semantic passes. Prices, operating bounds, and the package archive identity in this probe were synthetic. - Review regressions reproduced lost request/receipt evidence, lost admission metadata, executable replacement after verification, and a SIGTERM-resistant descendant surviving the CLI. The repairs now pass. Executable bytes are staged before launch; a trusted Node resolution hook preserves relative imports. A symlink-path regression also failed before its repair and now passes. The final staged native probe repeated both one-request checks successfully. - Full current-head [CI](https://github.com/paperclipai/paperclip/actions/runs/40115932485) passed on `4781487215116afd10a415d4db53b1ec8e004dd5`: all 58 check runs and the legacy status are successful, neutral, or skipped. Greptile and Paperclip both reviewed this commit at 5/5. All five finding threads are fixed and resolved. No paid provider request ran during this change. ## Risks - This is a trusted source-checkout controller, not a published package command. It needs the repository's installed shared cost dependencies. - The manifest declares reviewed prices and full-lifecycle bounds. Synthetic tests do not qualify real prices, invoices, storage, or Braintrust metering. - The attempt-directory guard is local. Hosted fanout still needs unique global allocations, rerun rejection, and durable private retention. Do not enable the existing matrix on this change alone. - Forced child termination can leave cleanup unverified. That attempt remains incomplete and holds its full allocation. These checks do not claim a host security boundary against an adversarial target. - The current private publisher does not archive the separate controller directory. Native trace relay, private report integration, and live qualification remain open. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. This runtime does not expose the exact model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
aec86c794b |
fix(runner): preserve native work through controller outages (#17594)
Keep remote native work under its authenticated connection lease through controller outages. Hold durable ACPX tools through reconnection, align the native Claude HTTP MCP client budget, and wait for the recovered handler before dispatch. Preserve Stop, cancellation, exact identity and artifact checks. Verified with real-process regressions and a real Claude Opus 5.5 Daytona workflow: 125 seconds offline during useful work, original process and lease retained, no failed native tool calls, queued follow-up completed, independently verified ZIP and committed workspace receipts. All current-head CI gates pass and Greptile is 5/5 with all findings resolved. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
e6ec781a1a |
fix(e2e): verify review outcomes and preserve report measurements (#17556)
## Thinking Path > - Paperclip's Product E2E journeys verify user outcomes in the real app and GitHub. > - The review journey must show the score and failed check for its exact reviewed commit, then the visible task status before and after reload. > - GitHub includes a hidden “Re-run failed checks” menu item before the visible check result. > - The journey selected that hidden item and failed even after native work and publication completed. A subsequent attempt exposed the same first-text-match problem with Done in the task UI. > - This change reads the selected check's status header and binds its URL, score, and commit. > - Browser fixtures retain this failure and reject plausible wrong projections. ## Linked Issues or Issue Description Refs: #17457, #17513, #17551. **What happened?** The bot completed its native run and task, then published its summary, inline precision finding, and failed 2/5 check. The browser assertion selected a hidden “Re-run failed checks” menu item before the visible result, so the journey still failed. After that was corrected, the next live attempt passed the check projection but selected hidden Done text before the visible task status. **Expected behavior** Verify the actual selected GitHub check. A hidden menu item or a word in the model's narrative cannot prove its conclusion. **Steps to reproduce** 1. Render a hidden “Re-run failed checks” item before a visible failed check. 2. Use the old first text match for “failed”. 3. Observe the false visibility failure. **Paperclip version or commit** The base includes the journey changes from #17457 and #17513. ## What Changed - Validate the check URL against the configured repository and select its exact check-run ID. - Require the expected score and full reviewed commit inside that check. - Read failure status only from that check's header. - Give each projection step its own phase so failures identify the correct boundary. - Stage the verified persisted screenshot as the standard report final-state artifact, rejecting a passing report if it is absent. - Preserve measured billing summaries during regeneration when raw run usage is absent; newer raw usage takes precedence. - Extend the catalog-wide history fixture to include the new journey evidence contract. - Read task completion from the visible Status property picker before and after reload, with separate phase labels. - Add eighteen isolated browser fixtures to the existing browser-support suite and document both boundaries. ## Verification - Reproduced the old selector failure in the live GitHub page and a credential-free browser fixture before fixing it. - All 60 browser-support cases pass, including eighteen focused projection cases. They reject the wrong check status, score, commit, page, and repository reference, plus hidden or unrelated Done text. - All 163 focused billing, history, accounting, report, and journey tests pass, as does runner E2E typechecking. - A regression reproduced discarded saved token totals. Retained totals now survive per-cell and campaign aggregation while missing dollar prices stay unknown; fresh raw usage still overrides stale summaries. - Diff and credential-value checks pass. - The earlier failed attempts remain failed. A fresh complete QA journey passed all 18 checkpoints and cleanup with a successful native run, a below-threshold review and inline precision finding, the correctly failed check, visible Done status, and reload persistence. The later reference-validation commit changes only browser fixtures; the live driver bytes are unchanged. - A report regression reproduced the missing standard final-state artifact. The packager now aliases only the hash-verified persisted screenshot for passing results; missing final-state proof is rejected. Presentation regeneration preserves the original live grade and dispatches no provider work. ## Risks This depends on GitHub's check-run header structure. A markup change fails the journey instead of accepting narrative text as a result. The product runtime and behavioral oracle stay unchanged. Usage-price coverage remains a separate evidence dimension. A saved billing summary is reused only when raw usage is absent, preserving the recorded coverage and unknown prices. The separate advisory Paperclip Review remains queued after its prior assigned attempt timed out. Its normal Retry control rejected the obsolete failed chat request as unsafe to retry. Required repository gates `ci / verify` and `ci / e2e` pass, all 55 completed CI checks are green, and final-head Greptile is 5/5 with no unresolved findings. The advisory queue is not represented as a completed assessment. ## Model Used OpenAI Codex, based on GPT-6. The exact runtime model ID and context-window size are not exposed in this session. Used for code, browser fixtures, and live evidence analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <paperclip@paperclip.ing> Co-authored-by: Paperclip <bot@paperclip.ing> |
||
|
|
a0e5166e97 |
Fix GitHub journey registration diagnostics and native retry cleanup (#17513)
## Thinking Path > - Paperclip manages work performed by AI agents. > - The Product E2E harness verifies a GitHub review-bot journey. > - GitHub can return a registration page without the expected creation control. > - The journey could then wait for the full login allowance without useful evidence. > - A failed native run can also retain a retry after its terminal status is visible. > - This change records a safe registration boundary and uses the normal Stop route for owned retries. > - Operators can diagnose the setup failure and confirm cleanup without changing the behavioral oracle. ## Linked Issues or Issue Description Refs: #17457. **What happened?** The GitHub review journey waited at App registration with no useful handoff evidence. Its cleanup also skipped failed native runs that still owned a durable retry. **Expected behavior** The journey should report a bounded registration failure or reauthentication boundary. Cleanup should Stop an owned native retry and require its audited acknowledgement. **Steps to reproduce** 1. Run the configured GitHub review journey. 2. Return a GitHub registration page without the expected creation button, or fail a native review run while its coordinator can retry. 3. Inspect the result and cleanup evidence. **Paperclip version or commit** Source built from current master. This extends the driver merged in #17457. ## What Changed - Bound the registration render wait to 30 seconds. Keep the separate attended confirmation allowance. - Retain GitHub origin, path, and control-presence flags. Retain headings, button labels, and alerts only on known registration error pages without authentication fields. - Exclude query parameters, entered values, page bodies, grant payloads, and authentication content from that diagnostic. - Add an optional private authentication hook for ordinary GitHub access reconfirmation. Detect passkey-first sudo at the manifest URL without retaining its authentication content. - Stop owned failed native retries through the existing run-only cancellation route. Verify company, agent, task, and audit acknowledgement. - Use native Stop correlation only for native runs. Queued legacy or unresolved runs use the ordinary cancel request. Stop scheduled retries and wait for them to leave the live state set. - Keep pending accounting debt intact. Add regression coverage and journey documentation. ## Verification - Focused journey and report suite: 58 tests passed. - Runner E2E TypeScript check passed. - `git diff --check` passed. Credential-value scan passed. - Live fixture reproduced the missing creation control at the manifest handoff. The new diagnostic retained its path and control-presence flags. No model work started. Draft cleanup passed. - All 58 current-head checks are terminal and green. Greptile scored the exact head 5/5, with no unresolved review threads. ## Risks GitHub pages that take more than 30 seconds to render the creation control now fail early. Native cleanup fails closed if the server does not provide the expected Stop acknowledgement. The live setup-to-Done behavioral journey is still unqualified. These changes do not repair a missing provider usage receipt or weaken its accounting gate. ## Model Used OpenAI Codex, based on GPT-6. Exact runtime model identifier and context-window size are not exposed in this session. Used for code execution, browser tooling, and regression analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Paperclip <paperclip@paperclip.ing> |
||
|
DottaandPaperclip
|
970127a022 |
test: retain onboarding Connect failure evidence (#17529)
## Thinking Path > - Paperclip manages agents and their work through normal company controls. > - Product E2E journeys check these controls through the browser and public API. > - Cloud onboarding uses one Connect action to validate the environment and hire the first agent. > - A failed Connect action previously left only a general journey error in the report. > - This pull request retains a closed diagnostic receipt for that action. > - The report can show the observed failure boundary without retaining private response text. ## Linked Issues or Issue Description **What existing behavior does this improve?** Cloud onboarding Product E2E failure evidence and report detail. **Current behavior** A validation or hire failure can reach the journey as a general operation failure. The report does not retain which Connect response failed. **Proposed behavior** Observe only the same-origin, same-company validation and hire POST requests made during one Connect action. Retain a closed receipt with the checkpoint, response state, bounded HTTP status, and allowlisted adapter, outcome, check code, level, and server code. The Get started control and saved-agent readback remain required. **Reason and benefit** The next failure can be attributed to the exact observed browser/API boundary. This change does not claim the cause of a historical failure. **Breaking changes** The new receipt is optional. Historical results remain valid. Explicit validation and hire rejections now stop the journey with a precise closed error code. This changes only E2E source; it does not change product behavior or qualification gates. ## What Changed - Add a bounded Connect observer with exact request matching and listener cleanup. - Retain known server codes from `details.code` or `code` without response messages, hints, headers, request bodies, provider text, or credentials. - Validate the receipt as a closed schema and render its allowlisted fields in existing report detail. - Include the new helper in the journey definition hash. - Add route, privacy, malformed-body, deadline, late-response, delayed-failure, success, and historical compatibility tests. Document the evidence boundary. ## Verification - Focused Product E2E tests passed: 3 files, 58 tests on the review-fix head. - E2E typecheck, workspace typecheck, and full build passed on the preceding implementation head. The review fix changes only report labels, checkpoint timing, and the matching tests. - The local full `pnpm test:run` command stopped progressing at the general-server start header. Its owned session was interrupted. This is not recorded as a pass. - All 58 remote checks completed on the final review-fix head: 55 passed, one neutral, and two expected skips. Both Greptile and Paperclip reviews are 5/5 with no unresolved review threads. - No live campaign was run with this patch. The observed historical failure is preserved. ## Risks - Unknown or missing responses remain explicit diagnostic states. They do not establish a product root cause. - Response body reads stop after five seconds or the action deadline and accept at most 64 KiB of JSON. Checks are capped at 32 retained entries. - New server or check codes appear as `unrecognized` until the closed allowlist is updated. - The production launch classifier, native gateway qualification, and Get started requirement are unchanged. ## Model Used OpenAI Codex (GPT-6 family), with code execution and tool use. The host does not expose the exact model ID or context window to this run; these values are unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
044628f59e |
Add explicit native tracing to the Runner eval CLI (#17483)
## Thinking Path > - Paperclip manages AI agents and their work. > - Runner Evals exercise real providers against a mock control plane. > - The controlled gateway route creates a disposable provider home. > - That route had no explicit native trace configuration. > - Codex skips normal home preparation for a preconfigured home. > - This change validates and applies the existing observability launch contract before provider creation. > - Operators can collect native evidence without passing vendor keys to the child. ## Linked Issues or Issue Description Refs #17440. Related: #17452 and #17469. This PR is based directly on master and does not include those pending controller changes. **Current behavior** The eval-session CLI cannot pass an explicit native observability launch. Passing environment alone cannot configure Codex in the disposable gateway home. **Proposed behavior** Accept an opt-in descriptor bound to the gateway attempt. Validate the existing provider launch contract, add a scoped relay token in memory, and apply it only to the owned provider home and environment. **Reason and benefit** A controller can route native evidence through its bounded local relay while the CLI keeps configuration distinct from verified ingestion. This is the CLI contract; hosted controller integration remains separate work. ## What Changed - Add a strict, bounded descriptor reader for native Codex and ACPX Claude. Bind campaign, execution, attempt, project, trace, span, and mock authority identifiers. - Reuse Runner observability materialization and redaction. Preserve the model gateway route. Reject upstream destinations, extra environment fields, credentials in headers, and unknown exporter settings. - Add `--observability-config`, requiring an explicit gateway. Keep direct and unconfigured paths unchanged. - Retain secret-free configured/unverified metadata in success and failure artifacts. - Document the contract and controller responsibilities. No workflow, runtime dependency, paid schedule, or public report changes. ## Verification - Eval CLI regression selection: 81 passed before the test-only relocation. The 17 new cross-package observability tests now pass from `tests/runner-e2e/runner-eval-observability.test.ts`; their integration-suite typecheck also passes. They use the actual shared launch generator, real disposable homes, CLI success/failure paths, and credential redaction. The first run exposed two test setup references introduced by a broad edit; both were corrected and its failure log remains retained. - Runner TypeScript build and release daemon build passed. Full `pnpm -r typecheck` passed. - Actual native Codex and ACPX Claude reached a local receiver with authenticated OTLP protobuf and the expected Braintrust parent. Codex sent four log and two trace batches; Claude sent three log and one trace batch. All recorded processes exited, disposable homes were removed, and 29 retained files contained neither synthetic credential. - The local model endpoint deliberately returned 401. Codex retained a failed terminal turn; Claude retained a provider timeout. These are route/export proofs, not semantic eval passes. Native retries reached only the fake endpoint. No paid provider or Braintrust request was made. - The first local probe failed before provider launch because its TypeScript loader path was wrong. It was corrected to the installed loader; both attempts remain retained. - Full repository build passed at initial head |
||
|
DottaandPaperclip
|
6d46851920 |
test(e2e): add attended GitHub review bot journey (#17457)
## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E checks verify real user paths through those agents. > - A recorded GitHub review-bot walkthrough had no repeatable registered case. > - App setup, automatic triggering, publication, and task completion need separate evidence. > - This pull request adds an explicit attended journey to the existing catalog and report tools. > - Reusable setup profiles and independent assertions preserve the same outcome across compatible targets. ## Linked Issues or Issue Description Refs: #17262. This case remains an explicit attended attached-target flow. It does not replace the shared deployment-target or paid-operations framework. **What existing behavior does this improve?** The Product E2E catalog can now select and retain evidence for GitHub review-bot setup and its first automatic review. **Current behavior** Recorded walkthroughs cannot be rerun as a fixed registered case. A published review alone can hide a failed run or unfinished task. **Proposed behavior** Create and install an owned App, save automatic new-PR settings, create a defective PR, and require correlated review delivery, an inline finding, a consistent check, a successful run, and a Done task. Retain failures and confirm owned-resource cleanup. **Reason and benefit** The case makes a useful external integration reproducible without treating setup, unit checks, or report rendering as proof of a completed live journey. ## What Changed - Add two explicit attended cells for an existing configured agent in local or Daytona execution. - Declare hosting, account, connection, and credential prerequisites through private setup bindings. - Add fixed browser controls, ownership checks, independent grading, and cleanup. - Package declared screenshots through the existing sanitizer, manifest, validator, and dashboard. - Reject operations-managed execution until budgeted routing is qualified. - Document hosted QA credential setup and evidence limits. ## Verification - Focused catalog, oracle, report, connection, and ownership checks: 127 tests passed. - Inherited-environment preflight regression failed against the previous commit and passed after the fix. Actual-run environment verification remains required. - Harness TypeScript check and full workspace typecheck passed. Full build passed. The local full-test attempt was interrupted after repeated embedded PostgreSQL startup failures; its exit code of zero does not establish a pass; Exact-head Linux CI passed the full test, typecheck, build, Runner, serialized, browser, and canary gates on `abbddeade848edefc8de75f11cc9008b2cd52764`. - A real hosted attempt published a rated review and inline finding. It failed when its native session timed out before the task reached Done. Cleanup passed. Full live qualification remains open. - Credential scan found no known credential values in the authored files or retained report corpus. ## Risks - The full case creates external test resources and uses the configured provider. It requires an explicit selection and current task authority. It is excluded from default and CI matrices. - Time and run limits are not a hard dollar cap. Missing dollar cost stays unknown. Fixture setup does not weaken budget policy. - Fresh personal OAuth, unattended authentication, runtime skill-read proof, and cross-host qualification are separate claims. - The selected live fixture needs environment/lifecycle recovery before it can adopt a longer provider deadline. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, code execution, and browser tools. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, including the available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the enhancement template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all Greptile and reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
661f33f994 |
fix(evals): verify stored gateway request coverage (#17432)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E campaigns retain a request ledger and send native traces and gateway logs to Braintrust. > - Native trace verification did not prove that each charged gateway request had a stored log row. > - A hosted probe showed that `x-bt-span-id` identifies the row `id`, while the stored `span_id` differs. Propagated campaign metadata was absent. > - This pull request joins retained request receipts to the correct Braintrust rows during scheduled finalization. > - Operators can see missing request coverage without changing costs, releasing holds, or rerunning agents. ## Linked Issues or Issue Description Refs #17354 and #17371. **Describe the bug** A campaign could pass native trace and experiment indexing checks while gateway request logs remained unverified. The runbook also claimed that propagated metadata made a request searchable after cancellation before response headers. The hosted gateway did not preserve that metadata in retained evidence. **Steps to reproduce** Inspect a retained gateway response header and query project logs by its `x-bt-span-id`. The matching record uses that value as `id`, not `span_id`. Compare every model reservation with its saved response receipt, including setup and canceled work. **Expected behavior** Verify each request against the stored row and report absent or conflicting evidence. Keep trace coverage separate from final usage and invoice reconciliation. ## What Changed - Added a bounded, read-only gateway index verifier. It matches row IDs, provider and model, and retains the actual span and trace IDs. - Included setup, failed and canceled requests in coverage. Missing receipts or row IDs, duplicate attribution, conflicting records and evidence changes prevent verification. - Added a separate scheduled gateway coverage gate and private receipt. Native trace and experiment checks remain independent. - Shared the existing bounded Braintrust query reader. Queries select only identity fields, with at most 100 IDs per batch, three rounds and a 45-second deadline. - Corrected the runbook's metadata guarantee and documented the historical `gatewaySpanId` receipt field. ## Verification - `pnpm test:e2e:runner:unit`: 4,120 passed, two existing skips; all 137 Node contracts passed. - Final repair: 54 focused tests passed, including scheduled-finalization coverage and all five optional identity fields. Each conflicting-field regression failed before the fix; absent or matching partial metadata remains supported. - `pnpm test:e2e:runner:typecheck`: passed after rebuilding stale local plugin SDK declarations. The initial stale-declaration failure remains recorded in the validation logs. - `git diff --check`: passed. - Read-only live verification of retained evidence: all ten local Codex requests and the one canceled drain-probe request matched stored rows. The hosted Claude campaign matched eight of ten; its two pre-header cancellations correctly remained incomplete. - The first hosted lookup exceeded the 1 MiB read limit when selecting full metadata. The final identity-only projection passed the same lookup. - Original journals stayed byte-identical. These checks made zero model calls and did not change provider data or release holds. - Full repository CI passed on `957ad0b684b80267d230fc4fb56d4f002ef089ad`: [run 40105676630](https://github.com/paperclipai/paperclip/actions/runs/40105676630), including typecheck, tests and build. Both current-head automated reviews scored 5/5 with no open findings. - The superseded initial CI run was canceled after the repair push to release the queue; its completed results remain available. No failed job was retried on the final commit. ## Risks Missing gateway IDs now fail scheduled trace coverage. A failed or canceled call may have no stored row, so this can require investigation even if product assertions pass. Indexing can lag; bounded query retries do not retry model work. Stored identity does not prove final billing or gateway span nesting under native attempt traces. Private identifiers remain in private evidence only. Recurrence and billing settings are unchanged. ## Model Used OpenAI Codex, GPT-6 family. The runtime does not expose a more specific model ID or context-window size in this session. Used reasoning, repository tools, local code execution and read-only API verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
87bf89756e |
fix: support Daytona runner upgrades and ignore obsolete Claude options (#17361)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Existing Claude and Codex agents can upgrade to Paperclip Runner. > - A Daytona computer needs platform qualification before Runner becomes its default. > - The upgrade notice required that default, so it hid the action that could start qualification. > - Saved Claude turn limits also caused conversion to fail even though Runner can run without them. > - This pull request verifies the computer on click and ignores obsolete Claude options with a warning. > - Agents retain their model, account, environment, instructions, and permission policy. ## Linked Issues or Issue Description Related: #17122. **What happened?** A legacy Claude agent on a Daytona snapshot had no upgrade button until its environment had platform qualification. A saved `maxTurnsPerRun: 1000` then caused conversion to return HTTP 422. **Expected behavior** The upgrade action verifies a supported Daytona computer and converts the agent. Unsupported tuning options produce a visible warning and do not block execution. **Steps to reproduce** 1. Create a legacy Claude agent with `maxTurnsPerRun: 1000` on an active Daytona snapshot. 2. Open Runtime before the environment has platform qualification. 3. Try to upgrade to Paperclip Runner. **Paperclip version or commit** After #17122, reproduced against `9c442cf48e`. ## What Changed - Offer upgrade on probe-capable Daytona snapshots and immutable sandbox images. Perform the public environment probe only after a click. Save after fresh adapter discovery confirms support. - Resolve the actual computer for explicit, instance-default, and hidden managed environments. Preserve pending edits and reject stale probe results. - Ignore Claude `maxTurnsPerRun`, `chrome`, `warmHandleIdleMs`, and `acpWarmHandleIdleMs`. Return structured warnings from create, hire, and update. Show warnings in Runtime and after a save. - Keep validation for commands, credentials, filesystem, network, and permission settings. - Retain the legacy turn limit in the existing live conversion fixture. Update regression tests, upgrade stories, and capability maps. ## Verification - Red: the old route returns HTTP 422 for the saved turn limit; the old form hides the Daytona upgrade notice. - Green: 417 focused server and form tests pass. A further 188 tests pass after the review fixes. Qualification coverage includes failed probes, unsupported targets, changed settings, managed defaults, unsaved environment variables, and forced Kubernetes targets. - `pnpm -r typecheck` and `pnpm build` pass. Token gates, map checks, map tests, and Storybook catalog checks pass. - Real local Claude Opus 5.5 conversion and task pass, with the downloaded artifact checked against seeded source bytes, SHA-256, and Git commit. - Real browser upgrade and full Claude Opus 5.5 task on a disposable named Daytona snapshot pass at `b81cec7d8c`. The saved agent retains its model, high effort, credentials, timeout, and grace settings while dropping the turn limit with a visible warning. Exactly one successful run completed the task. The downloaded artifact matched independently seeded source bytes, SHA-256, and Git commit; the source was unchanged, no interactions remained pending, and the delivery link survived a browser reload. The snapshot used current runtime assets built from public source. The supplied Daytona account could not access the original staging snapshot, so this verifies an equivalent local fixture rather than that exact snapshot. Disposable test resources were cleaned up. - Earlier live attempts remain recorded: an image-based sandbox creation failed, and the first named-snapshot probe returned failure with unresolved provisional cleanup identity. Later named-snapshot probing and the task succeeded; the initial creation failures are not attributed to this PR or declared resolved. A separate attempt using the Oct 18 cached runtime failed the current provider-pack compatibility check; rebuilding the runtime from matching public source removed that mismatch. Owned resources were identified and cleaned up, with diagnostic databases retained. - 54 checks passed at `b81cec7d8c`, with no failed checks; Greptile and Paperclip Review both scored 5/5, with both review threads resolved. Master advanced during the live proof, so the branch was rebased with both sides of additive import/test conflicts retained. On the rebased head `2b74222b94`, 472 focused tests, UI typecheck, and UI build pass. All required CI gates pass; Greptile and Paperclip Review both score 5/5 on this head, with no unresolved review threads. The PR is mergeable. The serial local `pnpm test:run` was stopped after the CI test lanes passed; that local run is not claimed as a pass. ## Risks - An explicit upgrade can allocate a temporary probe sandbox and take several minutes. Probe failures leave the agent on its existing runner. - Ignored options no longer enforce a turn cap, Chrome integration, or legacy idle-time tuning. The UI warns about each active ignored option. - Unsafe or ambiguous execution settings still fail validation. This change does not weaken environment qualification or runtime artifact checks. ## Model Used OpenAI Codex, GPT-6, with reasoning, code execution, and browser tools. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
52a66af29d |
perf(evals): tighten measured campaign resource ceilings (#17381)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E checks real provider behavior under a campaign budget. > - Its request and trace limits also determine the operating-cost forecast. > - The initial limits greatly exceed the measured marker and feedback cases. > - This pull request lowers those defaults while keeping the same selected cases and full model context limits. > - The benefit is a smaller bounded resource envelope with explicit quota failures and readable historical evidence. ## Linked Issues or Issue Description Refs: #17365 Refs: #17371 **What existing behavior does this improve?** The resource ceilings for bounded Product E2E campaigns and their OTLP relay. **Current behavior** Daily and weekly campaigns allow 60 and 120 model requests. The native relay defaults to 256 MiB of expanded source and 16,384 records per attempt. Both October 23 two-case qualification campaigns used ten model requests. The largest native attempt used 6,531,801 expanded source bytes and 2,455 records. **Proposed behavior** Use 30 daily and 60 weekly model requests. Default native exports to 16 MiB of expanded source and 4,096 records. Preserve the older explicit relay maxima so prior receipts remain valid and deliberate controller overrides still work. **Reason and benefit** The smaller limits leave headroom for the measured cases and reduce the maximum source traffic in the operating forecast. They do not change the daily or weekly dollar cap, model context ceiling, per-request reservation, selected cases, or attempt policy. **Breaking changes** A new bounded attempt that exceeds the smaller default trace allowance receives a quota error. It cannot silently omit required evidence. Existing receipts remain readable. No paid recurrence is enabled by this change. ## What Changed - Set scheduled model request caps to 30 daily and 60 weekly. - Add smaller native relay defaults while preserving the existing explicit maximum and historical receipt validation. - Exercise default count and expanded-source exhaustion through the HTTP relay. Keep a regression for larger historical receipt policies. - Document the measured basis, default capacities, and failure behavior. ## Verification - `pnpm test:e2e:runner:unit`: 4,095 Vitest tests passed, two existing skips; 137 Node contract tests passed. - `pnpm test:e2e:runner:typecheck`: passed. - `git diff --check`: passed. - The relay test accepts two 3-MiB log payloads, rejects the third before upstream dispatch, and retains its quota receipt. - Read-only replay of the prior real hosted campaign preserved all five ingestion receipts and 123,044 payload bytes exactly. Its accounting journal remained byte-for-byte unchanged; zero provider calls. - Full repository typecheck, tests, build and reviews are pending hosted CI. No paid model call was made for this change. Live qualification of the smaller defaults remains open. ## Risks - More verbose future cases may exhaust the new defaults. Such a failure remains visible and requires evidence before increasing a limit. - Source-byte ceilings are not Braintrust invoice units. Operating-cost qualification and reconciliation remain separate requirements. - Manual controller overrides can still use the earlier larger maxima; their cost audit must account for the limits actually recorded in the receipt. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution and tool use. The host does not expose an exact model build identifier or context window size for this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
eadc560712 |
fix(evals): retain usage during bounded gateway shutdown (#17371)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E tests use a local gateway to record model usage and reserve costs. > - Test cleanup can disconnect a background request before its provider sends final usage. > - Immediate abort then leaves the request cost unknown and its full reservation held. > - This pull request lets an admitted response finish within a short drain period. > - The benefit is better accounting without a new request or a retry. ## Linked Issues or Issue Description Refs #17365 and #17354. **What happened?** Two hosted cases passed, but cleanup canceled background requests before the gateway received their usage. Both requests kept their full holds. A local HTTP regression reproduces this loss with OpenAI and Anthropic responses. **Expected behavior** Retain final usage when an already-dispatched response finishes within a bounded cleanup period. Keep the client outcome canceled. Preserve the hold when final usage is still unknown. **Steps to reproduce** 1. Send a request through the eval gateway. 2. Disconnect the client before the provider returns final usage. 3. Close the gateway and let the provider response arrive. 4. Before this fix, the gateway aborts immediately and loses the final receipt. **Paperclip version or commit** Reproduced against master `b6e690044a821faab52d5466db36c4330c754a2d`. ## What Changed - Drain an already-dispatched response for at most five seconds after a disconnect. Keep the original provider timeout and campaign deadline. - Stop forwarding response data to a disconnected client. Stop admission as soon as shutdown begins. - Retain `clientDisconnected` on the private receipt and keep the outcome canceled, even when final usage arrives. - Make repeated shutdown calls share completion. - Add HTTP and SSE regression tests. Document the drain and its accounting limits. ## Verification - Red: five regression cases failed against the previous gateway implementation. Four further cases reproduced late cancellation during review. - `pnpm test:e2e:runner:unit`: 4,072 Vitest tests and 137 Node contract tests passed. Two existing Vitest tests were skipped. - `pnpm exec tsc --noEmit -p tests/runner-e2e/tsconfig.json`: passed. - `git diff --check`: passed. - Regressions cover both providers, an interrupted SSE stream, grace expiry, a shorter provider deadline, no dispatch during interrupted identity resolution, no new admission, repeated shutdown, client-only disconnects, and disconnects during each final receipt/ledger write. - Full repository typecheck, tests, and build passed in [CI run 40026846417](https://github.com/paperclipai/paperclip/actions/runs/40026846417) on `40ef13bd4d5df6b2ca0a9666fb485adf2a565c28`. Greptile and Paperclip reviews are both 5/5. No paid model test was run for this change. Live qualification remains open. ## Risks - Cleanup can take up to five more seconds. The admitted request can continue generating during that period, inside its existing reservation and deadlines. - An abort does not prove that the provider stopped billing. Missing usage keeps the unknown hold. - This change does not establish invoice settlement or rewrite historical receipts. It changes only the eval gateway. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, and tool use. The host does not expose an exact model build identifier or context window size for this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a967d479bd |
fix(e2e): retain verified onboarding artifact evidence (#17370)
Retain the bounded verified onboarding download and independent oracle in private normalized results before cleanup. Preserve the receipt when later readback or trace checks fail without marking the journey as passed. Refs #17262. Refs #17253. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b6e690044a |
feat(evals): allow reviewed bounded provisional accounting (#17365)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E campaigns include model, setup, storage and observability costs. > - Some billing windows close after the tests and reports finish. > - Requiring settled receipts for retained storage can block a good campaign for weeks. > - A reservation alone does not prove that a pending charge has an upper bound. > - This change permits provisional completion only with reviewed evidence for the exact cost configuration. > - Reports preserve unknown amounts and holds until later reconciliation. ## Linked Issues or Issue Description Refs: #17360, #17354 **What existing behavior does this improve?** The Product E2E scheduled accounting gate and its retained cost reports. **Current behavior** Every pending receipt fails finalization. A completed test can wait through the retained-storage billing window. The journal has dollar reservations but no explicit qualification of their bounds. **Proposed behavior** Permit provisional completion when all pending charges have reviewed full-lifecycle bounds, no requests remain in flight, no receipts conflict, and known costs plus held amounts fit the campaign limit. Keep all existing holds and receipt-finality rules. Missing evidence still fails. **Reason and benefit** An operator can distinguish delayed but bounded billing from unbounded or missing costs. The campaign can finish without claiming that its invoice is settled. **Breaking changes** The journal accepts optional bound metadata and an invalidation record. New reports add an optional bounded-pending count. Current readers retain old journals and reports without granting them bounds. Older code must not read new bound-bearing journals. The existing daily/weekly enable gates and budget amounts stay unchanged. **Additional context** This implements the operator-approved provisional-cost policy for the existing eval harness. It does not change product budgets or the agent-eval roadmap. A targeted GitHub search found no duplicate open PR. ## What Changed - Accounting owns a narrow evidence contract: a reviewed audit hash, exact configuration hash and full-lifecycle scope. - Gateway reservations retain that evidence before dispatch. Changed limits or rates fail before dispatch. Unexpected routing, model identity or excessive usage invalidates a bound durably. - Operating allowances can carry separately reviewed bounds for each cost component, including its full storage lifetime. No existing allowance becomes qualified by default. - Reports and Braintrust experiment metadata distinguish provisional accounting, complete receipts, blocked accounting and unavailable evidence. Public output omits private audit hashes. - Scheduled finalization allows qualified provisional costs and explicitly rejects spending overruns, including overruns with complete receipts. - Add regressions for legacy data, restarts, late reconciliation, conflicts, malformed evidence, routing uncertainty and exceeded limits. Document the evidence and migration boundaries. ## Verification - Full offline eval-support suite: 4,062 Vitest tests passed, two existing tests skipped; 137 Node tests passed. - `pnpm test:e2e:runner:typecheck`: passed. - `git diff --check`: passed. - Read the retained hosted qualification journal with the new policy. All six legacy pending charges remain unqualified. Known cost `$0.421245200`, held `$19.327680000`, and committed `$19.748925200` remain unchanged. The journal's SHA-256 is unchanged. The original failed workflow remains failed. - No paid models, network provider calls, receipt imports or reservation releases were used for this verification. - Rebased onto master `4d90634f74`; 142 targeted tests and typecheck passed. - Review reproduced a lost OpenAI input-total limit check when cache details were missing (1 failed, 47 passed). Fix `5fef9ba7c149951a51eae468614a4b89f0b3d6db` preserves unpriced provider usage bounds. Then 178 focused tests, full offline support and typecheck passed. - Current-head hosted CI passed: https://github.com/paperclipai/paperclip/actions/runs/40020746585 (attempt 2). Typecheck, build, all general/Runner/serialized tests and browser gates are green. - CI attempt 1 hit a PostgreSQL deadlock in unchanged chat-test cleanup; its skipped flag restoration caused a second assertion failure. One failed-jobs-only retry passed. Original failure evidence is preserved in attempt 1. No paid eval campaign was rerun. - Greptile reviewed exact head `5fef9ba7c149951a51eae468614a4b89f0b3d6db` at 5/5; its one finding is fixed and resolved. The optional Paperclip review is paused and supplied no assessment. ## Risks - Evidence hashes bind an audit to a configuration; they do not prove the audit is true. Operators must review real provider limits, all price tiers, full resource lifetimes and enforceable infrastructure limits before configuring bounds. - Provisional completion is not invoice settlement. Later billing can reveal overruns, which must be reconciled and inspected. The ledger cannot stop external billing. - Existing pilot journals, configured allowances and price cards have no new bound approval. This PR alone does not qualify or enable recurring paid work. - The gateway cannot qualify a canceled Braintrust request without route confirmation. It preserves that unknown hold. - No new dependency, secret, repository variable, workflow, schedule or cloud resource changes. ## Model Used OpenAI Codex, GPT-6 family. The exact deployment ID and context-window size are not exposed in this session. Used reasoning, repository tools and code execution. No paid evaluation model calls were made for this change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
17f1e386e8 |
feat(evals): compare retained Anthropic billing exports (#17360)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E evaluations retain request costs and failed attempts in a journal. > - Provider exports can arrive later and contain costs at a wider time scope. > - Comparing those exports by hand makes repeated accounting work hard to audit. > - This change adds a read-only command for the observed Anthropic export format. > - It binds the comparison to the retained manifest, journal and exact export bytes. > - Operators can see the unmatched amount without assigning it to canceled requests or releasing their holds. ## Linked Issues or Issue Description Refs: #17009, #17354 **Problem or motivation** A hosted campaign retained eight complete model receipts and two canceled requests. A later provider export covered the campaign, but comparing its tokens, rounded costs and the journal required a one-off script. Repeated observations must not be counted twice or treated as request-level billing proof. **Proposed solution** Add a private comparison command for one standard Claude Sonnet 5 campaign and one UTC date. Validate the selected workspace, key label, model and pricing dimensions. Compare provider totals with one authoritative observation per request. Retain source hashes and signed differences. Keep provisional exports, partial receipts and missing model identities explicit. **Alternatives considered** Manual comparisons remain possible but are hard to repeat. Importing a day total as a request cost would invent attribution. This command leaves journal and budget mutations to their existing owners. **Roadmap alignment** This is a focused operational tool for the existing Product E2E accounting subsystem. It does not change the product budgeting or agent-eval roadmap features. A targeted search found no duplicate billing-comparison PR or issue. ## What Changed - Add the provider-specific CSV parser, scope checks, token pricing comparison and private CLI in the Product E2E accounting owner. - Report complete and partial receipt coverage, source hashes, rounded provider charges, separate non-model charges, and signed unmatched amounts. - Reject ambiguous scopes, unsupported price dimensions, duplicate rows, contradictory token observations, changed journals and incomplete input. - Write a new mode-0600 output without replacing existing files or changing the journal. - Document the supported export shape and the limits of comparison evidence. ## Verification - After rebasing onto master `e48f1867b3b45f74edca4645d640846bcafeee4c`, 104 focused and adjacent tests passed across comparison, ledger, provider pricing, operating receipts and gateway behavior. - `pnpm test:e2e:runner:typecheck`: passed after rebase. - `git diff --check origin/master...HEAD`: passed. - Ran the actual CLI on retained hosted-campaign CSV exports and journal. It reproduced the prior manual calculation: provider-token estimate `$0.428027200`, with `$0.006782000` unmatched to retained request cost receipts. No provider or network call was made. - The first retained-data run exposed an empty provider key-status field and canceled receipts without a returned model identity. Those cases are now covered by the fixtures; the rerun passed. No request identity or final charge was invented. - Regression tests first reproduced loss of token fields on unknown-cost receipts (2 failed, 13 passed). The repair retains each known field independently of cost and rejects conflicting equal-authority token values. - Full PR CI passed on `f74ef1c31391ee62b686fcf300601e9cf8e73a7e`, including typecheck, tests, build and browser suites. Greptile and Paperclip each reviewed that exact head at 5/5 with no remaining actionable findings. ## Risks - The initial supported schema is standard global Claude Sonnet 5 with context buckets at or below 200k, one campaign and one UTC date. Other provider shapes fail explicitly. - Matching workspace/key labels does not prove exclusive key use or a complete provider export. The output remains a comparison and cannot release a reservation. - Provider costs are rounded and can lag. The token estimate uses retained prices and is kept separate from provider charges and non-model costs. - Config and exports are private inputs. Output includes hashes instead of raw key labels. This file is not a public report projection. - No runtime dependency, schedule, credential, paid execution, journal format or admission rule changes. ## Model Used OpenAI Codex, GPT-6 family. The exact deployment ID and context-window size are not exposed in this session. Used reasoning, repository tools and code execution. No paid evaluation model calls were made for this change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e48f1867b3 |
fix(evals): correlate canceled gateway requests with their ledger (#17354)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E evaluations use a budget ledger and a local model gateway. > - The gateway writes each request reservation before it calls a provider. > - A client can cancel before the gateway returns response headers. > - Project-only logging context cannot then bind a late gateway log to its reservation. > - This change sends the reserved request identity in Braintrust log metadata before dispatch. > - Operators can use that identity to inspect late records without guessing from time or session order. ## Linked Issues or Issue Description Refs: #17009 **What happened?** A hosted evaluation passed its product checks but retained two canceled requests with no response headers. Later gateway logs had partial usage. Their time and session matched, but they had no ledger request identity. **Expected behavior** A gateway log must carry the ledger request ID even when the client receives no headers. A log match must not imply final billing. **Steps to reproduce** Run the gateway regression with a transport that receives a request and then cancels before response headers. Before this change, both requests have the same project-only parent header. ## What Changed - Add a small Braintrust context encoder owned by the eval gateway integration. It uses the existing SpanComponentsV4 project-log format and an allowlist of request, campaign, execution, attempt and phase metadata. - Save the exact context in the private request receipt before dispatch. Send it as the gateway parent header. - Test both provider routes, repeated identical requests, cancellation holds and direct-provider isolation. - Document the lookup field and the limits of historical and partial evidence. ## Verification - The new cancellation regressions failed before the change: 2 failed, 37 passed. - Gateway, operations and tracing checks: 82 passed. - `pnpm test:e2e:runner:typecheck`: passed after rebuilding stale generated plugin-SDK types from current master. The initial stale-type failure remains recorded locally. - Official Braintrust SDK 3.38.0 decoded the actual generated context. Its `startSpan` implementation emitted the reserved identity in the initial LLM log. The check captured logs in memory and forbade all network calls. - `git diff --check`: passed. - Full repository CI passed on `529446a9295f2dc6fa6499a2f976480d74626097`, including typecheck, tests, build and both required gates. Greptile is 5/5 on that head; its sole finding is fixed and resolved. - The optional Paperclip review turn ended without a final response. Its check remains queued and provides no review assessment. No passing review is inferred from it. - Live hosted acceptance of this header change remains unqualified; no paid test was dispatched for this fix. ## Risks - The encoder follows Braintrust's versioned span format. A future vendor change can require an update. It adds no package dependency. - A late log can have partial usage. Existing reservations remain held until valid cost evidence resolves them. - Historical project-only requests cannot gain the new identity retroactively. - Only synthetic eval identifiers enter the new metadata. Credentials, prompts and responses are excluded. Direct provider routes keep their current headers. ## Model Used OpenAI Codex, GPT-6 family. The exact deployment ID and context-window size are not exposed in this session. Used reasoning, repository tools and code execution. No paid evaluation model calls were made for this change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
59d1b7b43d |
fix(evals): collect usage when GitHub skips jobs without a runner (#17352)
## Thinking Path > - Paperclip manages AI agents and their operating budgets. > - Product E2E runs retain accounting for each campaign. > - The usage collector reads completed GitHub jobs without rerunning paid work. > - GitHub returns a null runner ID for jobs that it skips before assigning a runner. > - The collector rejected those jobs and could not retain the completed campaign inventory. > - This change accepts that skipped-job representation while preserving started-job checks. ## Linked Issues or Issue Description Refs #17009. **What happened?** The usage collector rejected completed qualification run 39971835384. GitHub returned skipped jobs with null runner IDs. The collector accepted only zero for jobs without a runner. **Expected behavior** Retain completed run usage when skipped jobs have a null or zero runner ID. Do not count VM time for those jobs. Continue to reject missing runner identity for started jobs. **Steps to reproduce** Read the completed first-attempt Product E2E run with operating-receipts.ts github. Before this fix, a skipped job with runner_id set to null throws Invalid workflow job identity. **Paperclip version or commit** |
||
|
DottaandPaperclip
|
ed3f067dbd |
fix(evals): select operating allowance by cadence (#17326)
## Thinking Path > - Paperclip manages AI agents and their operating budgets. > - Product E2E runs exercise real agents through a bounded gateway. > - The daily and weekly selections share one repository configuration variable. > - Their different request limits need different operating allowances. > - The existing parser accepts only one allowance, which cannot represent both reviewed limits. > - This change selects the allowance for the actual cadence and retains all existing validation. ## Linked Issues or Issue Description Refs #17009. **What happened?** The daily selection needs an $11 operating reservation; the weekly selection needs $21. A shared $21 allowance cannot fit the $20 daily cap. A shared $11 allowance does not cover the reviewed weekly resource bound. A configuration containing both allowances was rejected before paid admission. **Expected behavior** Each cadence uses its own reviewed allowance. Missing or malformed selected allowances stop admission before paid work. **Steps to reproduce** Set the operating allowance JSON to an object with daily and weekly allowances. Prepare either campaign through prepareScheduledCampaign. Before this change, both calls reject the outer object. **Paperclip version or commit** |