mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 17:51:58 +01:00
a9b18e1cbecabd8cd929e0a47f49a67d52dce3c1
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
5d40e75253 |
fix(evals): distinguish scheduled TestOps results from uploads (#18676)
## Thinking Path > - Paperclip lets people manage AI agents and inspect their work. > - The chat catalog reports each original Product E2E case to TestOps. > - TestOps creates scheduled result rows before a case runs. > - Those rows have identity fields but no status property. > - The publisher currently mistakes any case row for an uploaded outcome. > - This PR distinguishes scheduled membership from an outcome after strict identity checks. > - Each case can upload once and wait for processing without changing its grade. ## Linked Issues or Issue Description Refs #18647, #18666. **What happened?** A real TestOps launch had all three scheduled result rows before any model case started. Each row had the expected case, project, launch and job-run identity. Its status property was absent. The publisher treated the row as already uploaded, skipped its only POST and then rejected readback. **Expected behavior** A fully bound scheduled row permits the first immutable case upload. It does not count as an outcome. Readback waits within the existing bound until the original status and exact attachments are ready. **Steps to reproduce** Run the new scheduled-row control against the parent API. It rejects before sending the initial result. Fixing only that initial check still rejects the first scheduled readback after one POST. ## What Changed - Share the strict result-detail identity check between initial inspection and readback. - Recognize an absent status property as scheduled only after that check. Present valid statuses, including unknown, stay readback-only. Malformed statuses and foreign identities fail closed. - Keep scheduled full-catalog readback unready within the same poll and deadline bounds. Keep the final status, evidence and attachment checks unchanged. - Calibrate three planned rows, delayed and permanent processing, unknown outcomes, wrong job/source and explicit-null status. Retain the once-only upload and uncertain-response controls. Use the registered legacy full-name alias in the fixtures. ## Verification - The real-shape parent control fails with `result_mismatch` after the first upload is suppressed. - A controlled initial-only fix sends one POST and still fails on scheduled readback. The final two-seam fix passes. - All 77 focused controls pass: `node --test tests/runner-e2e/chat-catalog-testops-streaming.test.mjs tests/runner-e2e/chat-completion-testops-api.test.mjs tests/runner-e2e/chat-full-catalog-testops-adapter.test.mjs`. - Both changed files pass Node syntax checks. `git diff --check` passes. - All local controls use synthetic data and mocked HTTP. No local app, database, native build or provider runs. - The affected live run was cancelled during build after metadata proved no original case had started. Its build instance is terminated. No original grade was changed or inferred. - Latest-head GitHub CI is green: 48 successful jobs and one skipped. Typecheck, build, browser and required aggregate checks passed. - The first CI attempt failed a heartbeat scheduling test’s five-second idle wait after its 51-admission assertion passed. Its six direct owners were byte-identical to the prior passing CI. One same-head failed-job recovery passed the server shard and required aggregate; all 46 earlier successful jobs were carried unchanged. The original failure is preserved, and its underlying cause remains unresolved. No heartbeat source, fixture or timeout was changed. - Live streaming qualification remains pending. No user command or documentation contract changes. ## Risks - The observed JSON distinction is status-property absence, not a result-stage or UUID field. Those fields are not used. - A scheduled row must match the current project, launch, case, registered full name, job run and GitHub source URL. It never qualifies as passed or complete. - The same 12-poll and time limits remain. A permanent scheduled state fails. A present outcome never causes another POST. An uncertain write still uses GET-only reconciliation. - Backend handling of uploaded scheduled rows still needs live qualification. Duplicate, foreign and final-coverage guards remain; this PR does not assume server deduplication. - Earlier dispatch and cancelled setup records remain separate. The first integration HTTP 500 cause is still unknown. No in-flight legacy workflow is changed or rerun. ## Model Used OpenAI `gpt-6.1-sol`, with tool use and code execution. The session does not expose a context-window size. The independent source peer uses the same model family. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
410b0e2701 |
fix(evals): accept refreshed catalog workflow URL (#18666)
## Thinking Path > - Paperclip lets people manage AI agents and inspect their work. > - The chat catalog reports original Product E2E grades to TestOps. > - TestOps can refresh its job metadata from GitHub. > - That operation returns a canonical master workflow-file URL. > - The catalog guard currently rejects that exact URL before case selection completes. > - This PR accepts that fixed URL and keeps the existing source and identity checks. > - The catalog can use the refreshed job without changing its cases or grades. ## Linked Issues or Issue Description Refs #18542, #18647. **What happened?** A supported TestOps job refresh changed the catalog job URL to the GitHub `blob/master/.github/workflows/chat-catalog-testops.yml` form. The selection guard rejects it as `job_mismatch`. **Expected behavior** The guard accepts that exact parent workflow URL. It still rejects another owner, branch or workflow. **Steps to reproduce** Run the new adapter regression against the parent commit. It fails in `validatePilotSelection` before the job-run request. ## What Changed - Add one exact canonical master URL to the full-catalog workflow URL guard. - Add a regression through the real selection validator with refreshed job metadata. Keep negative checks for another owner, branch and the called case workflow. ## Verification - The new selection regression fails on the parent with `job_mismatch` and passes with this change. - All 73 focused controls pass: `node --test tests/runner-e2e/chat-full-catalog-testops-adapter.test.mjs tests/runner-e2e/chat-completion-testops-api.test.mjs tests/runner-e2e/chat-catalog-testops-streaming.test.mjs`. - Both changed files pass Node syntax checks. `git diff --check` passes. - These controls use synthetic metadata and mocked HTTP. No local app, database, native build or provider runs. - Removing the one added predicate restores the parent API file exactly. The branch, job, case, source, evidence and grade checks remain unchanged. The pilot URL policy also stays unchanged. - Current-head GitHub CI passes: 48 jobs pass and one is skipped, including the required typecheck, test, build and final summaries. Live streaming qualification remains pending. No documentation change is needed for this internal URL compatibility fix. ## Risks - The alias accepts only this repository, master branch and parent workflow in the full-catalog lane. It grants no alternate source or execution authority. - An earlier TestOps dispatch failed with an integration HTTP 500 before GitHub created a run. Its cause remains unknown. This patch addresses the separate, reproduced metadata compatibility failure; it does not claim to cause or cure that HTTP 500. - Existing in-flight workflows keep their original source. This PR does not patch, rerun or upload into them. ## Model Used OpenAI `gpt-6.1-sol`, with tool use and code execution. The session does not expose a context-window size. The independent source peer uses the same model family. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0fd82e386e |
feat(evals): stream completed chat catalog results to TestOps (#18647)
## Thinking Path > - Paperclip lets people manage AI agents and inspect their work. > - The chat catalog uses real Product E2E cases and reports their original grades to TestOps. > - The current report matrix waits for every selected worker. > - A large selection can hide completed results for hours. > - Each case can use the existing private reader and safe exporter after its own worker ends. > - This PR gives each case a separate upload and readback job, then verifies final coverage without another upload. > - People can inspect completed cases while the rest of the selection runs. ## Linked Issues or Issue Description Refs #18542, #18625. **What existing behavior does this improve?** The existing chat catalog reports results to TestOps only after all selected workers finish. **Current behavior** One slow case holds back the report and upload jobs for all other cases. **Proposed behavior** Each finished case gets its own protected reader, safe projection, upload and backend verification. The final job checks the whole selection through read-only requests. It does not upload results again. **Reason and benefit** A person can inspect a completed grade and its safe evidence receipts while other cases run. This changes future runs. It does not change an in-flight run. ## What Changed - Add one same-commit reusable workflow with separate worker, private-reader and TestOps-upload jobs. Each job uses only its own protected environment. Callers pass empty named secret placeholders and do not inherit secrets. - Reuse the original exporter and grade checks. Freeze each case's three Allure JSON files once. Allow known sibling results during per-case readback. Keep final exact selection coverage. - Submit each bundle at most once. Reconcile uncertain responses by readback. Retain only closed publication, failure and upload-intent receipts. A rerun attempt still fails the existing attempt-one admission guard. - Preserve failed original grades, failed cleanup and missing evidence. No missing original can become a passing result. - Keep external-input cases in the serial lane. Small native-only selections allow two cases at once. Small mixed selections allow one native and one external case. Larger selections retain eight native cases and one external case. - Add source, privacy and failure controls to the existing unit roster. Document the reporting flow and runner-group admission requirement. - Fix an existing corruption-test fixture that could write the same byte back to a random IV. Read and flip one bit instead. Keep the storage rejection and unpublished-plaintext assertions unchanged. ## Verification - 152 focused Node controls pass across the shared client, API, selection, CLI and streaming tests. The retained parent controls show the missing multipart and partial-readback behavior, plus the small external-selection error before its fix. - The focused workflow security control passes. Actionlint passes for both workflows. Changed JavaScript files pass syntax checks. Git diff checks pass. - Local controls use synthetic data and mocked HTTP. They make no provider call and do not start an app, database or native build. - Previous head `b51406df` passed PR CI (48 jobs pass and one is skipped). The review-driven test-only successor `0c65cc21` then failed one unchanged snapshot-storage corruption test, plus the final CI summary. The latest head includes its narrow fixture repair. Its first CI attempt separately hit runner shutdown/lost-communication failures, two setup/timeouts and a heartbeat-cancel cleanup HTTP 500/lock timeout; those observations are retained without claiming their underlying cause. One same-head failed-and-cancelled-only recovery now passes at `bf9c3e8f`: 48 jobs pass and one is skipped, including required `ci / verify` and `ci / e2e`. All 31 already-passing jobs retained their original execution records; all 15 recovered leaves and both dependent summaries passed. The first attempt remains recorded; the recovery does not establish the underlying causes of those failures. A small AWS run after delivery must prove that one case and both attachments are visible before another original case ends. That live proof is pending; local controls do not prove vendor ingestion or environment-secret resolution. - Retained earlier fixture-calibration failures remain recorded. The final integrated Node control run is green. A deterministic Node crypto/filesystem control also reproduces the no-op corruption mutation with a valid IV byte equal to `X`, then passes after flipping that byte. This control uses no app or database. The failing CI byte was not captured, so the exact historical byte remains unknown. ## Risks - The called workflow needs its exact master ref in the existing selected-workflow runner group. No IAM role change is needed. - Environment-only secrets must resolve in the called jobs. The caller passes no actual model or reporting key. The small live proof must confirm this behavior. - TestOps does not document an idempotency key for this upload route. An uncertain write is never resent automatically. If readback cannot verify it, the workflow fails with retained closed receipts. - The existing job run can already be marked finished by the selection watch. The live proof must record its actual stage and prove subsequent uploads are accepted. This PR does not reopen, stop or close that job run. - Backend ingestion does not prove that an already-open TestOps page refreshes automatically. - The corruption-test repair changes no production storage or cryptography. New-head AWS CI must qualify the existing database-backed owner test. - Each case adds short protected reader and upload jobs. The app source, original case clocks, graders, model, budgets, raw-evidence privacy boundary and zero eval retries stay unchanged. The versioned stream concurrency group serializes new streaming runs; an old immutable batch run can continue in its original group. ## Model Used OpenAI `gpt-6.1-sol`, with tool use and code execution. The session does not expose a context-window size. Independent source peers used the same model family. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
97c4dc969e |
test(evals): retire duplicate chat TestOps plans (#18625)
## Thinking Path > - Paperclip manages AI agents and preserves their work history. > - Product E2E cases use TestOps plans to select real AWS evaluations. > - The full chat catalog now includes the former smoke and completion-pilot cases. > - The two old bootstrap workflows can recreate their smaller plans after deletion. > - This pull request removes those workflows and stops their plan-creation functions before API access. > - The full-catalog workflow keeps its shared clients, evidence rules, case identities, and measured results. ## Linked Issues or Issue Description Refs #18542, #18516. The user requested removal of the duplicate smoke and completion-pilot plans and investigation of their recreation paths. Both old workflows accepted manual dispatch only. Their bootstrap operations created plans by name when the plans were absent. They had no schedules in their workflow definitions. The old workflow records were disabled before plan removal. An existing pilot execution was allowed to finish successfully. This change does not cancel executions or delete cases, results, launches, or private evidence. ## What Changed - Remove `chat-feature-testops.yml` and `chat-completion-testops.yml`. - Make the former singleton plan creator reject every call before API reads or mutation markers. - Reject pilot bootstrap before API access. Keep the full-catalog branch and its integration validation. - Remove tests that require the retired workflow files. Keep shared helper and full-catalog controls. Run the trusted-checkout boundary test against the active catalog workflow. - Replace old setup instructions with retirement guidance and the active full-catalog route. The master sync preserves the incoming typed launch-search fix from #18620, including both regression tests. The only conflict was the retired pilot documentation. Its search-format note remains as a retained-helper contract. ## Verification - Two new retirement controls fail against the parent code because it reaches an API or mutation marker. Both pass after the change. - Six owning Node suites after the master sync: 146 passed (144 before the sync). Shared contract: 22 passed. Catalog and retained export controls: 12 passed. Focused workflow and release controls: 7 passed. - Two modified API modules pass Node syntax checks. `git diff --check` passes. - The pure workflow-security file has the same four failures and 13 passes on both the untouched parent and this branch. Two assertions expect eight trusted bootstrap steps where master now has nine. Two lists omit the already-shipped catalog workflow. This change does not repair or relax those unrelated assertions. - A broader release-test invocation had 73 passes and 20 sandbox failures from blocked mock HTTP listeners or npm permissions. Its focused affected workflow control passes. A standard Vitest setup attempt stopped on a missing `acpx/runtime` bundle dependency before tests. No local app, database, provider, or native runtime was qualified. - Current-head PR CI passes: 48 successful jobs and one skipped job, including both required checks. Greptile is 5/5. Independent source review passes. No new model run is needed for this retirement. ## Risks - Existing integrations that point at either retired workflow will stop dispatching. This is the requested behavior; use the full catalog instead. - Historical evidence readers still use shared helpers. Those helpers remain, including the transport and official-context parser used by the active catalog. - Workflow removal is a repository change. TestOps plan deletion and disabling obsolete executors are separate service actions. Neither action should delete shared cases or measurements. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with source editing, shell tools, and independent source review. The context-window size is not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
a6afe3e230 |
Fix TestOps launch preflight search encoding (#18620)
## Thinking Path > - Paperclip manages AI agents that do real work. > - Product evals verify the saved outcome of that work. > - TestOps plans select and dispatch these evals. > - Operator launchers check a unique launch name before they request work. > - TestOps expects an encoded filter, but the launchers send plain name text. > - This change uses the vendor filter format in the shared TestOps API owner. > - The existing checks can then prevent duplicate launch requests. ## Linked Issues or Issue Description Refs #18516, #18608, and #18542. The completion pilot registered successfully after #18608. Its optional API launcher then stopped with HTTP 400 in [run 41477551241](https://github.com/paperclipai/paperclip/actions/runs/41477551241). It stopped before the launch mutation marker. No evaluation was dispatched by that request. The launch search API uses Base64 of UTF-8 JSON filters. The launchers sent plain name text. I confirmed the format in the deployed public [vendor filter codec](https://paperclip.testops.cloud/es-testops-CICoDnkU.js) and its launch UI. The encoded name filter displayed the existing pilot launch. The TestOps UI separately dispatched [the genuine three-case pilot](https://github.com/paperclipai/paperclip/actions/runs/41478906153); this PR does not create another run or claim that pilot passed. ## What Changed - Add the typed name-filter encoder to the existing TestOps API module. - Use it in the backlog and completion launch preflight requests. The full-catalog route uses the same completion adapter. - Preserve the exact launch name, nonce, selection, authorization, once-only markers, and no-resend behavior. - Add protocol controls through both real launch functions. Document the wire format. ## Verification - Red: both new consumer controls reproduce the HTTP 400 boundary with plain name text. The other 109 API controls pass. - Green: all 111 API controls pass after the fix. All 166 related API, pilot contract, and full-catalog controls pass with Node 24.22.2. - The first broad local invocation lacked the `yaml` dependency in the new worktree. Its failure log is retained. Reusing the existing installed dependencies allowed all 166 controls to run and pass without further source changes. - The vendor-format query displayed the existing uniquely named pilot launch in the authenticated TestOps UI. No launch POST or model call was used for this read-only verification. - Diff check passes. All required CI checks pass on exact head `b55bdc6753fa042e4c2465e7b9b366b8126b4730` ([run 41480643021](https://github.com/paperclipai/paperclip/actions/runs/41480643021)). Both current-head reviews are 5/5, with zero unresolved findings. No TypeScript, compiled product, workflow, grader, model, clock, retry, or evidence-boundary changes. ## Risks The vendor can change its search format. The format is kept in the existing provider-specific API module, and both launch consumers have protocol controls. Existing duplicate detection and uncertain-response refusal remain in place. The already-running UI pilot is independent of this fix. ## Model Used OpenAI GPT-6 through Codex assisted with reasoning, repository inspection, code execution, and native accessibility tool use. The host does not expose the exact model revision or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Dotta
|
9203ab52f0 |
fix(evals): bind complete unassigned TestOps plan buckets (#18608)
## Thinking Path > - Paperclip manages agents that do real work. > - Product evals check the saved outcome of that work. > - TestOps plans select the exact automated cases to run. > - A new plan can report its unassigned cases as a job-statistics bucket. > - The pilot registration guard rejects that bucket as a foreign job. > - This change binds only a complete, known unassigned bucket and records assignment diagnostics before writes. ## Linked Issues or Issue Description Refs #18516 and #18542. The metadata-only bootstrap in [41464514278](https://github.com/paperclipai/paperclip/actions/runs/41464514278) created the dedicated plan and verified its GitHub job definition, then stopped at `plan_job_mismatch` before job or case-assignment mutations. No model work ran. The retained receipt did not include the assignment rows. The live failure's exact shape remains unconfirmed. The API's job-statistics carrier permits an absent job. The old guard rejects a complete unassigned bucket. This PR covers that case and records the actual rows in the next closed inspection. ## What Changed - Accept one closed unassigned bucket with the exact selected-case count. Support an explicit null job or an omitted optional job field. - Assign the exact selected cases once. Keep the existing owned assignment without assigning it twice. - Reject foreign jobs, mixed buckets, wrong counts and unknown unassigned fields before mutations. - Retain only bounded assignment kinds, numeric job IDs and counts in the existing observation receipt. - Merge current master, including the full-catalog adapter from #18542. Retain its context-aware plan and job checks alongside the new pre-write assignment inspection. The helper consumes that context’s expected case count. - Document the inspection and binding rules. The helper takes the expected count from its owner. This does not import or run the full-catalog route from #18542. ## Verification - Red: both complete unassigned carriers fail on the old guard; the other 70 dispatch/API controls pass. - Green: all 72 dispatch/API controls passed before the master sync. After the sync, all 104 related pilot and full-catalog Node controls pass. They include ordinary registration, no duplicate assignment, foreign-job rejection, incomplete and unknown bucket rejection, launch fences, uncertain-response protection and result readback. - Diff check passes. No TypeScript or compiled product code changes. Full repository checks were not run locally. All required CI checks pass on current head `a8bec6418065a7faae28c8daab3efdfc9b184797` ([run 41473510972](https://github.com/paperclipai/paperclip/actions/runs/41473510972)). Both current-head reviews are 5/5 with no unresolved threads. Before the master sync, one unchanged server timing test failed on attempt one and passed on the only targeted rerun; that original failure remains in CI attempt one. No rerun was needed on the current head. - Live acceptance still requires another metadata-only inspection, followed by a genuine three-case TestOps launch. This PR does not claim a new passing measurement. ## Risks An overly broad unassigned rule could replace an existing job. The new rule permits only one known bucket with the exact selected-case count and no foreign job. All final binding checks still apply. Unknown assignment shapes stop before writes. Original case queries, model, clocks, graders, evidence rules and retries are unchanged. ## Model Used OpenAI GPT-6 through Codex assisted with repository inspection, code execution and tool use. The host does not expose the exact model revision or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run the relevant local controls and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
774ddc7271 |
test(evals): add the full chat catalog to TestOps (#18542)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat feature evaluations check real user requests through the product. > - The YAML catalog has 244 scenarios, but the shipped TestOps completion route selects only three. > - The preserved catalog already supplies the execution mappings and independent oracles. > - A larger TestOps plan needs exact identities, bounded selection and source-bound private evidence. > - This pull request adds that plan route without merging the catalog app into master or replacing its grades. > - Users can run the full catalog or a selected subset and see the actual outcome and prerequisites. ## Linked Issues or Issue Description Refs: #18516 and #18515. This extends their completion pilot and reuses its trusted reporting path. **What existing behavior does this improve?** The existing three-case TestOps completion workflow. **Current behavior** The remaining catalog scenarios cannot be selected through that TestOps job. The one-case backlog plan has a separate strict singleton contract. **Proposed behavior** A separate candidate-scoped plan maps all 244 scenarios. Existing case aliases, jobs, plans and result history stay intact. Registration is distinct from a measured pass. ## What Changed - Add a source-derived 244-row manifest. It binds the 242 domain descriptors and two legacy tasks to the unchanged YAML queries, contexts, outcomes and original bounds. The JSON snapshot avoids importing the unmerged domain tree into master. - Reconcile exact case identities and a dedicated full-catalog plan. Reuse legacy aliases, including the existing backlog case, without renaming them. Preserve history and stop on conflicting aliases or uncertain writes. - Extend the shared completion adapter with a separate trusted full-catalog workflow context. Return a compact observed binding that fits one 48 KB protected variable. Keep the three-case pilot contract intact. - Add a fixed-source manual workflow. Small selections run serially. Larger selections allow eight independent workers and one serial external-input worker. Each worker and report reader keeps a singleton approval and exact-key AWS session. The eight-cell private authority limit is unchanged. - Validate and merge metadata-only singleton projections before one protected Allure upload and complete paginated readback. Preserve failed originals and cleanup failures beside passed results. Keep screenshots and raw evidence private. - Discard a changing full-catalog page-count snapshot and retry within the same observer bounds. Malformed page fields still fail at once. A pass still needs one consistent exact selection and all final evidence checks. - Keep workflow source and app source separate. The full lane builds candidate `3b66c3c144360ddf32714a0742fd774b8ceafb16`. Its only changes from the existing `de0941835cc8f0d01c982574ee258e110babdbeb` pin are finite TestOps host admission and its offline test. This is not current-master app qualification. - Give full selections above three cases a bounded metadata-read allowance above the existing 120-second deadline ceiling. Readback has two upper bounds: at most twelve polls with five-second pauses, and a size-derived deadline of at most 374 seconds, whichever exhausts first. There is no guaranteed minimum wait or processing SLA. Small selections and pilot deadlines stay unchanged. Final identity, grade and evidence predicates stay intact. - Document the real readiness limits and setup path. Five coverage gaps, six evidence exceptions, eighteen additional-credential declarations and thirty-seven external-input declarations overlap. They are not new pass claims. ## Verification - Related Node controls passed: pilot API and dispatch, 244-row manifest, idempotent reconciliation, full selection, three-page result readback, source guards and partition/output-size bounds. The latest ten adapter controls include the actual registration receipt's variable-size limit and the real bootstrap facade's closed failure codes and privacy checks. - A deterministic 244-result replay with one second per HTTP response failed at the old deadline and now verifies after 127 virtual seconds. A three-second response replay still rejects at 375 virtual seconds. The full-lane allowance accounts for paginated reads and four-way result batches; it is not a guaranteed network SLA. All 55 related API/adapter controls pass. They also reproduce the changing-page-count failure before the fix, then verify the consistent snapshot. Permanently changing counts fail after twelve polls. A malformed numeric field still fails before a retry. - Owning export controls: 26 passed. They include all 244 synthetic carriers, mixed passed/failed originals, missing and foreign projections, privacy filtering and exact source binding. - Candidate host owner: 31 offline controls passed. The new finite-catalog case failed before admission changed. - Preserve two actual local regressions: canonical scenario provenance failed before its one-line fix; the full registration binding exceeded the variable limit before compact output. Both now pass. - Initial local harness setup attempts failed before test execution. The final narrow owner configuration avoids provider-bundle setup. - Full repository tests, build and typecheck were not run locally. Normal required PR CI provides the remote checks. - Owning TypeScript no-emit, actionlint, paid-workflow policy with unsafe controls, and diff checks passed. - Local tests used Node 26.4.0. These are synthetic source controls. No local app, database, native build or provider qualification ran. - CI [41307571776](https://github.com/paperclipai/paperclip/actions/runs/41307571776) on `23e518be` recorded one 15-second timeout in `attention-retention-work.test.ts:153`. All other ordinary leaves passed. The test used a fixed JavaScript date but its retention deadline comes from PostgreSQL `updatedAt`. The exact failing await and original SQL timestamp were not captured. The corrected pre-sync head’s server shard (2/12) passed on AWS in CI 41309030928. Current master independently fixed the same clock assumption in #18543; conflict resolution adopts that whole upstream file. No local PostgreSQL run or timeout increase was used. - Current-head normal CI [41463567187](https://github.com/paperclipai/paperclip/actions/runs/41463567187) passes all required checks on `fc6c2309cfc7d1e5a83f0a597c7fa1d2beb987e1`, with Greptile 5/5 and all review threads resolved. The final diff has 23 catalog files. Current master’s whole retention test is adopted; the redundant earlier test change is absent. The valid changing-page snapshot regression passes with all 55 related API/adapter controls. No new TestOps import or live evaluation has run for this change. After delivery, use metadata-only bootstrap and a three-case normal TestOps selection to verify the new route. Do not claim a new 244-case measurement. ## Risks - The app target remains the preserved catalog candidate. Each result records its target and workflow revisions. No old campaign ordinal, acceptance gate or core fix is changed. - Some mapped cases need private account configuration or have known source/evidence gaps. Their original checks and evidence rules can still fail. A registration does not establish readiness or success. - Whole-plan execution can take hours and spend provider and AWS money. Original per-case limits and zero retries remain. Unknown setup usage is not free. This PR does not run all 244 cases. - The raw private `SOURCE_REF` field retains the prior integration-lineage label. It is not a verified checked-out branch. The actual candidate checkout and all authority checks use immutable `3b66c3c144360ddf32714a0742fd774b8ceafb16`; public projections set `source.ref` to null. - TestOps has no atomic bulk upsert. Reconciliation retains partial progress and performs no automatic resend after an uncertain response. - The trusted inventory keeps a 12 MiB limit per public carrier file. The larger parser ceiling does not promise admission above that limit. The existing 244-case fixture produces a 1,359,465-byte normalized carrier; this is not a guarantee for future payload sizes. - Raw evidence remains under the existing private retention owner. Public GitHub artifacts and TestOps attachments contain only allowlisted metadata and availability receipts. - The new workflow needs the exact master entry in the existing runner group's selected-workflow list before a live run. No repository or runner-group visibility is broadened by this source change. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) assisted with implementation and independent source review, using code and tools. The host does not report the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run the relevant local controls and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
e122478368 |
test(evals): run and record the chat completion pilot through TestOps (#18516)
## Thinking Path > - Paperclip evals must verify real user outcomes. > - A passing manual pilot does not make its cases runnable from TestOps. > - The existing TestOps adapter owns one separate backlog case. > - The three-case completion pilot needs its own exact selection and reporting path. > - This workflow binds a real TestOps job run to reviewed catalog source and AWS execution. > - Results retain their original verdict, checks, source and private evidence. ## Linked Issues or Issue Description Refs #18364, #18377, #18515 and #18509. This is the main-side companion to merged #18515. The immutable evaluated source is its actual integration-branch squash merge, `de0941835cc8f0d01c982574ee258e110babdbeb`; its tree matches the reviewed catalog head. The host admission remains inert until this main-only workflow exists. The broader catalog stays off master. The existing TestOps connection can run only the separate backlog cell. The project-work, project-defaults and instance-restart cases currently require manual campaign dispatch and private retrieval. This change creates a dedicated reusable TestOps plan and records fresh execution outcomes there. Source checks and registration do not establish a new live pass. ## What Changed - Add a default-branch-only workflow with exact TestOps selection, immutable source/build outputs, serial AWS execution and zero eval retries. No source/model/budget selector is accepted. - Register the three exact reusable cases and a separate plan with the existing reviewed GitHub integration. Preserve opaque job definitions. Validate the observed single Branch/defaultValue carrier, PATCH only enable/integration, and use the job’s default branch without rewriting its definition or adding plan overrides. Fence the initial operator launch by actual GitHub run number and nonce; normal TestOps Run remains repeatable. - Reuse the existing private-evidence authority, scanners, bounds and exact-key S3 policies. Authenticated selection supplies only this original attempt's exact source and case identity. The three trusted protocol files are unchanged. - Skip report retrieval after an unsuccessful build or skipped execution; retain evidence and failed grades when evaluation actually ran. - Check out trusted main after the target in both build and execution jobs, and reject any target tree containing the reserved trusted-helper path. A real temporary-Git-tree regression exercises the actual YAML guard. - Keep reporting credentials in separate protected jobs. Export only closed verdict/check/accounting metadata and private-evidence receipts to TestOps. Verify final PNGs privately and record their checksums; keep all screenshot bytes in protected AWS storage and out of public GitHub artifacts. Preserve failed grades and failed cleanup. - Verify actual same-run case results and complete result-owned attachment metadata within a fixed processing window. Missing or inherited attachments never qualify. Keep source/result links and stable case identities. ## Verification - 68 dispatch, selection, registration and readback controls pass. A checkout-boundary regression fails on the old ordering, accepts an ordinary target and rejects a candidate-owned helper tree before trusted checkout. They cover real workflow environment wiring, actual GitHub metadata guards, exact subset selection, wrong sources/cases/jobs, replays and attachment readiness. - 38 exporter controls pass, including the existing backlog exporter and the unchanged private-evidence authority consumer. Red/green controls prove project work accepts the catalog’s 1–3 run range in both export gates, while 0/4 runs are rejected; the actual pinned-source control verifies all three cases’ run bounds. Cancelled, failed and scheduled TestOps job stages are rejected even with complete results. The three new cases retain all 35 independent checks; private diagnostics and all image bytes are excluded from the public artifact. Extra media files and directories are rejected. - The actual checked-in workflow environment is exercised against the actual pinned source matrix and host function for all three cases before provider access. Local fixture controls make zero provider calls and do not claim live AWS qualification. - All 71 workflow-policy controls pass. Actionlint and owning typecheck pass. Relevant TypeScript build dependencies pass. - Full support suite: 5,827 passed, 13 failed, four skipped. A clean unchanged-master baseline reproduces all 13 failures in the same five files. They include fixture expectation drift, operating-receipt fixtures, a profile pin, pnpm bootstrap counts and local database startup. This change updates the exact TestOps workflow registries; the two affected boundary controls now pass. Baseline failures remain retained and are not presented as a green full support suite. - The catalog dependency passed CI and current-head 5/5 review, has no unresolved threads, and is merged. All current-head CI checks pass; both Greptile and Paperclip reviews are 5/5, with no unresolved threads. No new TestOps mutation, provider call or passing launch is claimed yet. A fresh three-case TestOps launch and verified server-side results remain the delivery gate. ## Risks This is a paid execution and reporting boundary. The new workflow accepts only trusted default-branch actor identities and a verified project/plan/job/case selection. Its evaluated source is immutable and separate from main reporting code. The full catalog is not promoted. Screenshots, raw evidence and hidden provider data stay in protected storage. Public GitHub artifacts carry only the closed metadata projection. TestOps receives checks and private-evidence availability/checksum receipts. Unknown setup or AWS accounting stays unknown. One passing launch will not establish full-catalog reliability. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code execution and tool use. Exact hosted revision and context-window size are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |