7 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	0fd82e386e	feat(evals): stream completed chat catalog results to TestOps (#18647) ... ## Thinking Path > - Paperclip lets people manage AI agents and inspect their work. > - The chat catalog uses real Product E2E cases and reports their original grades to TestOps. > - The current report matrix waits for every selected worker. > - A large selection can hide completed results for hours. > - Each case can use the existing private reader and safe exporter after its own worker ends. > - This PR gives each case a separate upload and readback job, then verifies final coverage without another upload. > - People can inspect completed cases while the rest of the selection runs. ## Linked Issues or Issue Description Refs #18542, #18625. **What existing behavior does this improve?** The existing chat catalog reports results to TestOps only after all selected workers finish. **Current behavior** One slow case holds back the report and upload jobs for all other cases. **Proposed behavior** Each finished case gets its own protected reader, safe projection, upload and backend verification. The final job checks the whole selection through read-only requests. It does not upload results again. **Reason and benefit** A person can inspect a completed grade and its safe evidence receipts while other cases run. This changes future runs. It does not change an in-flight run. ## What Changed - Add one same-commit reusable workflow with separate worker, private-reader and TestOps-upload jobs. Each job uses only its own protected environment. Callers pass empty named secret placeholders and do not inherit secrets. - Reuse the original exporter and grade checks. Freeze each case's three Allure JSON files once. Allow known sibling results during per-case readback. Keep final exact selection coverage. - Submit each bundle at most once. Reconcile uncertain responses by readback. Retain only closed publication, failure and upload-intent receipts. A rerun attempt still fails the existing attempt-one admission guard. - Preserve failed original grades, failed cleanup and missing evidence. No missing original can become a passing result. - Keep external-input cases in the serial lane. Small native-only selections allow two cases at once. Small mixed selections allow one native and one external case. Larger selections retain eight native cases and one external case. - Add source, privacy and failure controls to the existing unit roster. Document the reporting flow and runner-group admission requirement. - Fix an existing corruption-test fixture that could write the same byte back to a random IV. Read and flip one bit instead. Keep the storage rejection and unpublished-plaintext assertions unchanged. ## Verification - 152 focused Node controls pass across the shared client, API, selection, CLI and streaming tests. The retained parent controls show the missing multipart and partial-readback behavior, plus the small external-selection error before its fix. - The focused workflow security control passes. Actionlint passes for both workflows. Changed JavaScript files pass syntax checks. Git diff checks pass. - Local controls use synthetic data and mocked HTTP. They make no provider call and do not start an app, database or native build. - Previous head `b51406df` passed PR CI (48 jobs pass and one is skipped). The review-driven test-only successor `0c65cc21` then failed one unchanged snapshot-storage corruption test, plus the final CI summary. The latest head includes its narrow fixture repair. Its first CI attempt separately hit runner shutdown/lost-communication failures, two setup/timeouts and a heartbeat-cancel cleanup HTTP 500/lock timeout; those observations are retained without claiming their underlying cause. One same-head failed-and-cancelled-only recovery now passes at `bf9c3e8f`: 48 jobs pass and one is skipped, including required `ci / verify` and `ci / e2e`. All 31 already-passing jobs retained their original execution records; all 15 recovered leaves and both dependent summaries passed. The first attempt remains recorded; the recovery does not establish the underlying causes of those failures. A small AWS run after delivery must prove that one case and both attachments are visible before another original case ends. That live proof is pending; local controls do not prove vendor ingestion or environment-secret resolution. - Retained earlier fixture-calibration failures remain recorded. The final integrated Node control run is green. A deterministic Node crypto/filesystem control also reproduces the no-op corruption mutation with a valid IV byte equal to `X`, then passes after flipping that byte. This control uses no app or database. The failing CI byte was not captured, so the exact historical byte remains unknown. ## Risks - The called workflow needs its exact master ref in the existing selected-workflow runner group. No IAM role change is needed. - Environment-only secrets must resolve in the called jobs. The caller passes no actual model or reporting key. The small live proof must confirm this behavior. - TestOps does not document an idempotency key for this upload route. An uncertain write is never resent automatically. If readback cannot verify it, the workflow fails with retained closed receipts. - The existing job run can already be marked finished by the selection watch. The live proof must record its actual stage and prove subsequent uploads are accepted. This PR does not reopen, stop or close that job run. - Backend ingestion does not prove that an already-open TestOps page refreshes automatically. - The corruption-test repair changes no production storage or cryptography. New-head AWS CI must qualify the existing database-backed owner test. - Each case adds short protected reader and upload jobs. The app source, original case clocks, graders, model, budgets, raw-evidence privacy boundary and zero eval retries stay unchanged. The versioned stream concurrency group serializes new streaming runs; an old immutable batch run can continue in its original group. ## Model Used OpenAI `gpt-6.1-sol`, with tool use and code execution. The session does not expose a context-window size. Independent source peers used the same model family. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 14:18:00 -06:00
DottaandPaperclip	774ddc7271	test(evals): add the full chat catalog to TestOps (#18542) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat feature evaluations check real user requests through the product. > - The YAML catalog has 244 scenarios, but the shipped TestOps completion route selects only three. > - The preserved catalog already supplies the execution mappings and independent oracles. > - A larger TestOps plan needs exact identities, bounded selection and source-bound private evidence. > - This pull request adds that plan route without merging the catalog app into master or replacing its grades. > - Users can run the full catalog or a selected subset and see the actual outcome and prerequisites. ## Linked Issues or Issue Description Refs: #18516 and #18515. This extends their completion pilot and reuses its trusted reporting path. **What existing behavior does this improve?** The existing three-case TestOps completion workflow. **Current behavior** The remaining catalog scenarios cannot be selected through that TestOps job. The one-case backlog plan has a separate strict singleton contract. **Proposed behavior** A separate candidate-scoped plan maps all 244 scenarios. Existing case aliases, jobs, plans and result history stay intact. Registration is distinct from a measured pass. ## What Changed - Add a source-derived 244-row manifest. It binds the 242 domain descriptors and two legacy tasks to the unchanged YAML queries, contexts, outcomes and original bounds. The JSON snapshot avoids importing the unmerged domain tree into master. - Reconcile exact case identities and a dedicated full-catalog plan. Reuse legacy aliases, including the existing backlog case, without renaming them. Preserve history and stop on conflicting aliases or uncertain writes. - Extend the shared completion adapter with a separate trusted full-catalog workflow context. Return a compact observed binding that fits one 48 KB protected variable. Keep the three-case pilot contract intact. - Add a fixed-source manual workflow. Small selections run serially. Larger selections allow eight independent workers and one serial external-input worker. Each worker and report reader keeps a singleton approval and exact-key AWS session. The eight-cell private authority limit is unchanged. - Validate and merge metadata-only singleton projections before one protected Allure upload and complete paginated readback. Preserve failed originals and cleanup failures beside passed results. Keep screenshots and raw evidence private. - Discard a changing full-catalog page-count snapshot and retry within the same observer bounds. Malformed page fields still fail at once. A pass still needs one consistent exact selection and all final evidence checks. - Keep workflow source and app source separate. The full lane builds candidate `3b66c3c144360ddf32714a0742fd774b8ceafb16`. Its only changes from the existing `de0941835cc8f0d01c982574ee258e110babdbeb` pin are finite TestOps host admission and its offline test. This is not current-master app qualification. - Give full selections above three cases a bounded metadata-read allowance above the existing 120-second deadline ceiling. Readback has two upper bounds: at most twelve polls with five-second pauses, and a size-derived deadline of at most 374 seconds, whichever exhausts first. There is no guaranteed minimum wait or processing SLA. Small selections and pilot deadlines stay unchanged. Final identity, grade and evidence predicates stay intact. - Document the real readiness limits and setup path. Five coverage gaps, six evidence exceptions, eighteen additional-credential declarations and thirty-seven external-input declarations overlap. They are not new pass claims. ## Verification - Related Node controls passed: pilot API and dispatch, 244-row manifest, idempotent reconciliation, full selection, three-page result readback, source guards and partition/output-size bounds. The latest ten adapter controls include the actual registration receipt's variable-size limit and the real bootstrap facade's closed failure codes and privacy checks. - A deterministic 244-result replay with one second per HTTP response failed at the old deadline and now verifies after 127 virtual seconds. A three-second response replay still rejects at 375 virtual seconds. The full-lane allowance accounts for paginated reads and four-way result batches; it is not a guaranteed network SLA. All 55 related API/adapter controls pass. They also reproduce the changing-page-count failure before the fix, then verify the consistent snapshot. Permanently changing counts fail after twelve polls. A malformed numeric field still fails before a retry. - Owning export controls: 26 passed. They include all 244 synthetic carriers, mixed passed/failed originals, missing and foreign projections, privacy filtering and exact source binding. - Candidate host owner: 31 offline controls passed. The new finite-catalog case failed before admission changed. - Preserve two actual local regressions: canonical scenario provenance failed before its one-line fix; the full registration binding exceeded the variable limit before compact output. Both now pass. - Initial local harness setup attempts failed before test execution. The final narrow owner configuration avoids provider-bundle setup. - Full repository tests, build and typecheck were not run locally. Normal required PR CI provides the remote checks. - Owning TypeScript no-emit, actionlint, paid-workflow policy with unsafe controls, and diff checks passed. - Local tests used Node 26.4.0. These are synthetic source controls. No local app, database, native build or provider qualification ran. - CI [41307571776](https://github.com/paperclipai/paperclip/actions/runs/41307571776) on `23e518be` recorded one 15-second timeout in `attention-retention-work.test.ts:153`. All other ordinary leaves passed. The test used a fixed JavaScript date but its retention deadline comes from PostgreSQL `updatedAt`. The exact failing await and original SQL timestamp were not captured. The corrected pre-sync head’s server shard (2/12) passed on AWS in CI 41309030928. Current master independently fixed the same clock assumption in #18543; conflict resolution adopts that whole upstream file. No local PostgreSQL run or timeout increase was used. - Current-head normal CI [41463567187](https://github.com/paperclipai/paperclip/actions/runs/41463567187) passes all required checks on `fc6c2309cfc7d1e5a83f0a597c7fa1d2beb987e1`, with Greptile 5/5 and all review threads resolved. The final diff has 23 catalog files. Current master’s whole retention test is adopted; the redundant earlier test change is absent. The valid changing-page snapshot regression passes with all 55 related API/adapter controls. No new TestOps import or live evaluation has run for this change. After delivery, use metadata-only bootstrap and a three-case normal TestOps selection to verify the new route. Do not claim a new 244-case measurement. ## Risks - The app target remains the preserved catalog candidate. Each result records its target and workflow revisions. No old campaign ordinal, acceptance gate or core fix is changed. - Some mapped cases need private account configuration or have known source/evidence gaps. Their original checks and evidence rules can still fail. A registration does not establish readiness or success. - Whole-plan execution can take hours and spend provider and AWS money. Original per-case limits and zero retries remain. Unknown setup usage is not free. This PR does not run all 244 cases. - The raw private `SOURCE_REF` field retains the prior integration-lineage label. It is not a verified checked-out branch. The actual candidate checkout and all authority checks use immutable `3b66c3c144360ddf32714a0742fd774b8ceafb16`; public projections set `source.ref` to null. - TestOps has no atomic bulk upsert. Reconciliation retains partial progress and performs no automatic resend after an uncertain response. - The trusted inventory keeps a 12 MiB limit per public carrier file. The larger parser ceiling does not promise admission above that limit. The existing 244-case fixture produces a 1,359,465-byte normalized carrier; this is not a guarantee for future payload sizes. - Raw evidence remains under the existing private retention owner. Public GitHub artifacts and TestOps attachments contain only allowlisted metadata and availability receipts. - The new workflow needs the exact master entry in the existing runner group's selected-workflow list before a live run. No repository or runner-group visibility is broadened by this source change. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) assisted with implementation and independent source review, using code and tools. The host does not report the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run the relevant local controls and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 08:58:44 -06:00
Dotta	e122478368	test(evals): run and record the chat completion pilot through TestOps (#18516) ... ## Thinking Path > - Paperclip evals must verify real user outcomes. > - A passing manual pilot does not make its cases runnable from TestOps. > - The existing TestOps adapter owns one separate backlog case. > - The three-case completion pilot needs its own exact selection and reporting path. > - This workflow binds a real TestOps job run to reviewed catalog source and AWS execution. > - Results retain their original verdict, checks, source and private evidence. ## Linked Issues or Issue Description Refs #18364, #18377, #18515 and #18509. This is the main-side companion to merged #18515. The immutable evaluated source is its actual integration-branch squash merge, `de0941835cc8f0d01c982574ee258e110babdbeb`; its tree matches the reviewed catalog head. The host admission remains inert until this main-only workflow exists. The broader catalog stays off master. The existing TestOps connection can run only the separate backlog cell. The project-work, project-defaults and instance-restart cases currently require manual campaign dispatch and private retrieval. This change creates a dedicated reusable TestOps plan and records fresh execution outcomes there. Source checks and registration do not establish a new live pass. ## What Changed - Add a default-branch-only workflow with exact TestOps selection, immutable source/build outputs, serial AWS execution and zero eval retries. No source/model/budget selector is accepted. - Register the three exact reusable cases and a separate plan with the existing reviewed GitHub integration. Preserve opaque job definitions. Validate the observed single Branch/defaultValue carrier, PATCH only enable/integration, and use the job’s default branch without rewriting its definition or adding plan overrides. Fence the initial operator launch by actual GitHub run number and nonce; normal TestOps Run remains repeatable. - Reuse the existing private-evidence authority, scanners, bounds and exact-key S3 policies. Authenticated selection supplies only this original attempt's exact source and case identity. The three trusted protocol files are unchanged. - Skip report retrieval after an unsuccessful build or skipped execution; retain evidence and failed grades when evaluation actually ran. - Check out trusted main after the target in both build and execution jobs, and reject any target tree containing the reserved trusted-helper path. A real temporary-Git-tree regression exercises the actual YAML guard. - Keep reporting credentials in separate protected jobs. Export only closed verdict/check/accounting metadata and private-evidence receipts to TestOps. Verify final PNGs privately and record their checksums; keep all screenshot bytes in protected AWS storage and out of public GitHub artifacts. Preserve failed grades and failed cleanup. - Verify actual same-run case results and complete result-owned attachment metadata within a fixed processing window. Missing or inherited attachments never qualify. Keep source/result links and stable case identities. ## Verification - 68 dispatch, selection, registration and readback controls pass. A checkout-boundary regression fails on the old ordering, accepts an ordinary target and rejects a candidate-owned helper tree before trusted checkout. They cover real workflow environment wiring, actual GitHub metadata guards, exact subset selection, wrong sources/cases/jobs, replays and attachment readiness. - 38 exporter controls pass, including the existing backlog exporter and the unchanged private-evidence authority consumer. Red/green controls prove project work accepts the catalog’s 1–3 run range in both export gates, while 0/4 runs are rejected; the actual pinned-source control verifies all three cases’ run bounds. Cancelled, failed and scheduled TestOps job stages are rejected even with complete results. The three new cases retain all 35 independent checks; private diagnostics and all image bytes are excluded from the public artifact. Extra media files and directories are rejected. - The actual checked-in workflow environment is exercised against the actual pinned source matrix and host function for all three cases before provider access. Local fixture controls make zero provider calls and do not claim live AWS qualification. - All 71 workflow-policy controls pass. Actionlint and owning typecheck pass. Relevant TypeScript build dependencies pass. - Full support suite: 5,827 passed, 13 failed, four skipped. A clean unchanged-master baseline reproduces all 13 failures in the same five files. They include fixture expectation drift, operating-receipt fixtures, a profile pin, pnpm bootstrap counts and local database startup. This change updates the exact TestOps workflow registries; the two affected boundary controls now pass. Baseline failures remain retained and are not presented as a green full support suite. - The catalog dependency passed CI and current-head 5/5 review, has no unresolved threads, and is merged. All current-head CI checks pass; both Greptile and Paperclip reviews are 5/5, with no unresolved threads. No new TestOps mutation, provider call or passing launch is claimed yet. A fresh three-case TestOps launch and verified server-side results remain the delivery gate. ## Risks This is a paid execution and reporting boundary. The new workflow accepts only trusted default-branch actor identities and a verified project/plan/job/case selection. Its evaluated source is immutable and separate from main reporting code. The full catalog is not promoted. Screenshots, raw evidence and hidden provider data stay in protected storage. Public GitHub artifacts carry only the closed metadata projection. TestOps receives checks and private-evidence availability/checksum receipts. Unknown setup or AWS accounting stays unknown. One passing launch will not establish full-catalog reliability. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code execution and tool use. Exact hosted revision and context-window size are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-01 06:09:47 -06:00
DottaandPaperclip	987e0932bf	test(testops): inspect a fixed retained chat result (#18477) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - The chat feature evaluation checks a real task request on AWS. > - The case passed, but its final TestOps reader rejected attachment metadata. > - The later browser view passes the same check. It uses a different identity and time. > - Another model run cannot explain that difference. > - This pull request adds a fixed read-only check of the retained measurement with the reporting API identity. > - The check separates its own executor from the original measurement and preserves the original grade. ## Linked Issues or Issue Description Refs #18470 and #18462. **What happened?** The real chat case passed all ten checks and cleanup in [run 41217990335](https://github.com/paperclipai/paperclip/actions/runs/41217990335). Final verification failed at `attachment_page / attachment_rows`. The reader had already accepted the source, run, case, status, page counts, and unique attachment IDs. Later interactive metadata had valid IDs and false attachment flags. That later view does not explain the original API caller failure. **Expected behavior** A read-only operation can inspect the same retained result with the existing protected reporting identity. It must report the exact current attachment flags without running a model, uploading results, or changing the original grade. **Steps to reproduce** 1. Read the closed failure from the linked evaluation run. 2. After this workflow is reviewed and deployed, dispatch its fixed no-input operation on master. 3. Inspect its closed receipt. It records both the new executor and the original measurement. **Paperclip version/commit** The retained measurement used `e40de0c9e88210dc476dcf651461c0a8d1b98a59`. **Deployment mode** GitHub Actions metadata job with the existing protected reporting environment. No app, database, provider, or AWS worker starts. ## What Changed - Add one manual no-input workflow for the fixed retained measurement. - Admit the live executor and the original run separately. Verify exact selection and sanitized export artifact hashes before TestOps authentication. - Read only the five existing TestOps metadata routes with the existing client. Its normal API-token session exchange is the only authentication POST. - Record closed per-row attachment types and flags. Replay the unchanged verifier against the captured responses using historical context as data. Keep the actual process environment unchanged. - Add nine mock and archive-safety tests to the existing Node unit roster. ## Verification - `node --test tests/runner-e2e/chat-feature-testops-retained-verification.node-test.mjs`: 9 passed on Node 26.4.0. - Three Node syntax checks, actionlint, the relevant workflow policy check, and `git diff --check` passed. - The real pinned sanitized export ZIP passes the new safe-export extractor and existing bound reader. The first local attempt used a noncanonical `/var` temporary path and hit the existing canonical-path guard; the canonical `/private/tmp` attempt passed without a source change. - Corrected a review finding in failure attribution. Only actual authentication or protected GET errors now receive `apiFailure`. Historical replay failures retain their original phase and predicate diagnostics. - Initial test run: eight passed and one test expected the generic upload error instead of the existing specific job-run error. Corrected that test expectation. No verifier behavior changed. - Tests exercise separate identities, foreign source and actor rejection, origin and artifact integrity, exact five GETs, all three attachment-row refusals, secret redaction, wrong job/run/source, and real archive traversal/symlink rejection. - The unchanged API and export readers are source-pinned. Existing evaluation and acceptance code is unchanged. - [Normal CI 41220476243](https://github.com/paperclipai/paperclip/actions/runs/41220476243) passed on exact head `b79e85f71567f662d6f3d680ec66ec49a132589a`: 55 successes, three skipped checks, and two neutral reviews. Both required checks passed. Greptile is 5/5 and all review threads are resolved. - The nine new controls are registered in the family unit command. The separate normal PR-CI runner suites do not execute that roster, so their green status is not proof these nine ran on Node 24. The manual workflow explicitly runs them on Node 24 before any protected API read. - The real protected API operation has not run. A separately approved merge and metadata dispatch remain required. No model, app, database, or native build ran locally. ## Risks - The fixed artifacts can expire. Any absent or changed artifact fails before TestOps authentication. - The current protected response snapshot may differ from the original failure-time state. The receipt states that limit and makes no historical cause claim. - This is a fixed diagnostic operation. It does not repair the original workflow or grant authority to run another evaluation. - The workflow uses the existing protected reporting token. It logs no response body, unknown field, credential, or unknown URL. Only closed receipts are retained. ## Model Used OpenAI `gpt-6.1-sol`, through Codex, with reasoning, code editing, shell tools, and source review. The host does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 22:16:20 -05:00
DottaandPaperclip	9e1735ec17	test(evals): add a fixed TestOps AWS chat feature route (#18322) ... ## Thinking Path > - Paperclip lets people manage AI agents and inspect their saved work. > - Chat feature evaluations test a real request against saved tasks, audit records, links and reload behavior. > - The current suite has Codex and OpenCode cells but no dedicated runnable DeepSeek 1017 backlog plan in TestOps. > - TestOps must select the real case before the workflow can access a model key. > - This PR adds one fixed, default-branch AWS route for native OpenCode and DeepSeek 1017. > - The route keeps original evidence private and sends only a verified result and approved screenshots to TestOps. ## Linked Issues or Issue Description **Subsystem affected** Product E2E evaluation definitions and GitHub Actions delivery. **Problem or motivation** A dedicated TestOps plan cannot run the existing backlog chat journey with native OpenCode. A result must prove saved work and cleanup rather than report that a model call completed. **Proposed solution** Add one OpenCode backlog cell and one fixed public workflow. TestOps selection validation runs in a token-isolated job before builds or inference. The workflow retains private originals and reports the bound outcome to the selected TestOps case. **Alternatives considered** A private-repository wrapper would need a new cross-repository GitHub write credential. The new route uses the existing TestOps GitHub integration instead. Existing manual candidate approval paths remain separate. **Roadmap alignment** This extends the existing feature evaluation infrastructure. It changes no production feature, global runtime default or model default. Current main #18319 supplies the v2 grader, delegation queue accounting and final screenshot capture; these improvements are preserved. Related product work: #16814 and #17126 concern backlog handling. This PR changes neither product implementation and does not claim those changes are required. ## What Changed - Add exactly one native OpenCode backlog cell with DeepSeek 1017 under the separate `runner-opencode-deepseek-1017` profile. Preserve all four existing Codex and production OpenCode cells and their models. - Add a default-branch-only workflow with fixed evaluation, zero-result registration and one plan-launch modes. Require GitHub actor checks and authenticated singleton selection before inference. - Add a separate shipped-main evidence authority. Reuse the existing strict private storage, scanners, inventories and scoped AWS sessions. - Bind sanitized results, generated JUnit and producer-approved screenshots to the exact source, run, attempt and case. Keep failed grades and cleanup failures failed. - Add TestOps API, CLI and result adapters with pure controls and normal unit-test wiring. Keep the reporting token out of the evaluated process. - Verify the current server case result and attachment metadata after upload, without masking an earlier failed upload or evaluation. - Apply only the existing reviewed lock correction, verify unchanged tracked dependency inputs and bind the lock receipt across the compiled-output handoff before frozen installation. - Bind the initial administrator launch to one approved workflow run number and live GitHub source/actor metadata before TestOps authentication. Use bounded REST case reconciliation and only the reviewed existing GitHub integration. - Make two existing CI tests observe their intended state deterministically: a fake clock for copy-feedback lifetime and a completed-pass barrier for retry-cancellation batches. Production behavior and the original assertions remain unchanged. The two test-only fixes are intentionally included because their observation boundaries blocked this delivery’s required normal CI. They preserve every original assertion and change no production behavior. The original failed runs remain recorded, and the exact causes of those observations remain unknown. ## Verification - 157 focused profile, catalog, oracle, filesystem admission, export, private-path and workflow controls passed on the current-main composition. The existing stock OpenCode models and the new exact DeepSeek 1017 selection are checked separately. - 44 API, CLI, filesystem and post-upload readback controls passed, including foreign selection, changed hashes, unsafe files, failed outcomes, expired readback and no-resend behavior. - The new exact private session and trace layout controls produced four genuine parent failures and pass with the selected-path extension. The frozen 244-case candidate approval registry remains unchanged. - The lock guard has seven filesystem controls plus seven existing canonical-owner controls. The unrestricted old workflow failed its actual policy calibration; the corrected frozen-install route passed. - Eval-owning TypeScript checks, workflow lint and Git whitespace checks passed. The separate UI test correction was checked with the focused jsdom unit suite. The server test barrier has pure wrapper and real scheduler controls; its DB-backed route was not run locally. No additional local typecheck/build was run for these test-only changes. - Full native builds, provider execution, app/database behavior and TestOps registration are reserved for GitHub Actions/AWS. They have not run locally. - The initial normal CI run failed the new-workflow policy allowlist and a heartbeat scheduling assertion. The policy fix has an actual parent red/current green control and rejects five unsafe route variants. Heartbeat source is unchanged and its failure cause remains unknown. The original run is retained. - The launch-fence actual caller fails against the published parent and passes the five own-run contradiction controls before authentication. - The prior `a5aa359d` normal CI completed successfully. The complete `f9e1f289` run passed all leaf jobs except an existing copy-feedback UI assertion, plus its dependent verification gate. The four owning UI files were unchanged from the earlier passing head and actual tested merge; the exact scheduling cause remains unknown. - The correction changes only that test to actual React `act` and a scoped fake clock. All eight component tests pass; early-hide and never-clear source variants each fail the intended lifetime assertion. Production clipboard/card behavior is unchanged. - The `2f27f3fc` run passed the corrected UI shard and every leaf job except the existing retry-cancellation intermediate-count assertion, plus its dependent verification gate. The seven relevant owners match the earlier passing run and historical source comparison; the exact CI scheduling cause remains unknown. - The test-only barrier observes a persisted completed worker pass before asserting the unchanged 20-item cap and two-company fairness, then releases it for the original 29-item drain and 180-second idle check. Two extracted-wrapper controls and three actual scheduler controls pass. The real DB-backed test awaits AWS qualification. - On `836e80ac`, both repaired test shards passed in AWS. The original attempt retained a separate E2E cleanup HTTP 500 from a PostgreSQL lock timeout and one runner shutdown. No production repair or precise contention/shutdown cause is claimed. - Exactly one targeted retry of E2E shard 3 and its aggregate gate passed in attempt 2; exactly one recovery of the interrupted server shard and its verification gate passed in attempt 3. Other passed jobs were carried forward. The final normal run is successful. No whole-workflow retry or evaluation/model retry was launched. - Both literal required checks (`ci / verify` and `ci / e2e`) are successful on the exact final source `836e80ac`. Greptile is 5/5 and no review threads remain open. Actual TestOps registration and the OpenCode/DeepSeek feature result remain unqualified until their separate executions. ## Risks - Actual zero-result registration, tenant API shapes and public-repository dispatch access must be checked before the first evaluation. Missing or ambiguous metadata stops the route. - The real OpenCode/DeepSeek backlog result and Allure outcome remain unqualified until that run. No passing result is seeded. - One human query, one accounted work run, 120-second turn, 420-second attempt, 300-second settled-state bound and zero retries remain unchanged. Setup usage is outside work accounting and stays unknown. - The reporting token was configured through the separately reviewed protected credential bootstrap. It is unavailable to AWS evaluation jobs. - Metadata reconciliation is capped at ten pages, 1,000 cases and four parallel detail reads. Missing or inconsistent metadata stops before inference. The initial administrator launch counter must be created once after registration; it does not restrict normal TestOps Run actions. - Existing manual workflows, historical evaluation grades and candidate approval records remain unchanged. The original CI failures are preserved; the test corrections do not establish retrospective causes. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning and tool use. The runtime did not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-30 15:08:16 -05:00
DottaandPaperclip	3bd709df8b	Operationalize Product E2E accounting, tracing, and scheduling (#17009) ... ## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E tests exercise real agents, tools, setup and cleanup. > - Repeated runs need retained costs and traces, including failed attempts. > - A passing retry must not hide earlier spending or feedback. > - Recurring work also needs admission limits, private reports and failure detection. > - This change supplies those controls while keeping paid activation behind existing gates. ## Linked Issues or Issue Description Refs #16996 and #17158. This is the consolidated landing of the reviewed eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033, #17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096, #17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127, #17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository permits squash merges only. A local rehearsal showed that squashing each slice into master causes repeated conflicts in dependent slices. This PR preserves their commits and lands their combined implementation in one reviewed integration. Superseded slices will close after this merge. ## What Changed - Retain every attempt's cost receipt, including setup, failures, interruptions and retries. Separate reported amounts, estimates, missing costs and reserved capacity in reports and dashboards. - Add bounded request admission, exact model pricecards, gateway error accounting, cache policy and reconciliation evidence. Refuse unsupported routes and requests that cannot fit the allowance. - Integrate Braintrust gateway traces and OTLP correlation, with secret filtering and experiment imports. Record phase timing, including setup, feedback capture and cleanup. - Collect complaint and suggestion evidence after each attempt, including recovery after failures. Retain explicit review dispositions independently from the product grade. - Add gated daily and weekly selections, measured forecasts, setup/readiness checks, wall-time limits and retained monitor notifications. Include declared non-model overhead in admission while reporting the existing subscription separately. - Keep recurring history private, bound uploads and retention, and use isolated archive environments. Keep public publication an explicit operation. Report and publication jobs install only the committed lockfile, with scripts disabled and isolated stores; stale pins fail closed. - Integrate current master, preserve its Product E2E journey billing and privacy checks, and test that shared setup receipts do not double-count estimated agent usage. Preserve upstream fixture repairs. ## Verification - Complete integration baseline `0649a5ce7544ab20825c2183d2f421157db49f24` includes master `a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head as ancestors. - On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012 Vitest tests passed, two skipped, across 193 files; all 137 Node source-contract tests passed. - Focused billing and Product E2E journey contract checks: 24 tests passed. They cover estimates, unknown receipts, single-owner setup costs, browser-only execution and private publication guards. - Monitor and workflow wiring: all 96 Node tests passed. `node scripts/check-token-gates.mjs`: all gates clean. - `pnpm test:e2e:runner:typecheck` passes after the standard Runner TypeScript build refreshed local declarations. The first attempt rejected the new master `runner_prepared_launch` union member because this checkout retained an older generated declaration; the source union already contained that member. - Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes dependency re-resolution from archive-authorized reporting jobs after the security scan flagged it. All 15 workflow-security tests pass after updating the old two-install assertion. - Hosted [CI run 39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180) passes on the current repaired head, including typecheck, build, general tests, Runner checks, browser shards and canary dry run. Greptile is 5/5 on this exact head with no unresolved threads. The security scan passes after the fixed lockfile-resolution thread was resolved. - Paperclip Review returned a terminal neutral check but an incomplete 0-file assessment on the repaired head: its platform blocked both diff responses as prompt-injection content. This is not a successful code review. No bypass or fabricated pass was attempted; current-head Greptile, security and hosted CI are independent gates. - No paid test, reservation release or recurring-run activation accompanies this landing. Live route qualification and invoice reconciliation remain separate evidence. <details> <summary>Retained earlier validation failures</summary> Earlier broad local runs had two 15-second test timeouts (CLI poll cadence and Retry validation), one embedded-database setup timeout, and later PostgreSQL semaphore exhaustion. The exact timed-out tests passed separately without larger timeouts; the broad-run causes were not established. Those local failures remain distinct from subsequent hosted CI results. The independent #17158 fix records its Muse fixture collision, its UUID correction and hosted verification. Local Muse execution was also blocked by macOS embedded-PostgreSQL startup. No historical failure is treated as a successful paid qualification. </details> ## Risks - Admission is a local control, not a provider invoice cap. Missing provider receipts or resource charges retain unknown amounts and holds; estimates do not become billed dollars. - The bounded path admits only documented local suites and provider routes. Ordinary legacy or separate Runner protocol campaigns have different controls. Merging this code does not qualify additional routes. - Recurrence still needs the existing enable gates and current qualification/allowance evidence. Archive permissions and resources must match the documented private-environment contract. - Legacy reports may lack complete attempt or phase evidence. Reports preserve that limitation and do not infer zero usage from missing data. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact deployment variant and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks; full-suite host limitation disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-23 07:18:37 -07:00
DottaandPaperclip	5739a53f5f	fix(tests): sanitize installed-consumer validation processes (#16797) ... Start installed-consumer validation through Node and remove ambient Node startup settings before test processes start. Keep pinned historical source fixtures available in shallow checkouts and set filesystem fixture conditions explicitly. Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>	2026-10-19 14:12:45 -05:00