PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-04 02:17:33 +01:00

Commit Graph

Author	SHA1	Message	Date
DottaandPaperclip	0fd82e386e	feat(evals): stream completed chat catalog results to TestOps (#18647 ) ## Thinking Path > - Paperclip lets people manage AI agents and inspect their work. > - The chat catalog uses real Product E2E cases and reports their original grades to TestOps. > - The current report matrix waits for every selected worker. > - A large selection can hide completed results for hours. > - Each case can use the existing private reader and safe exporter after its own worker ends. > - This PR gives each case a separate upload and readback job, then verifies final coverage without another upload. > - People can inspect completed cases while the rest of the selection runs. ## Linked Issues or Issue Description Refs #18542, #18625. What existing behavior does this improve? The existing chat catalog reports results to TestOps only after all selected workers finish. Current behavior One slow case holds back the report and upload jobs for all other cases. Proposed behavior Each finished case gets its own protected reader, safe projection, upload and backend verification. The final job checks the whole selection through read-only requests. It does not upload results again. Reason and benefit A person can inspect a completed grade and its safe evidence receipts while other cases run. This changes future runs. It does not change an in-flight run. ## What Changed - Add one same-commit reusable workflow with separate worker, private-reader and TestOps-upload jobs. Each job uses only its own protected environment. Callers pass empty named secret placeholders and do not inherit secrets. - Reuse the original exporter and grade checks. Freeze each case's three Allure JSON files once. Allow known sibling results during per-case readback. Keep final exact selection coverage. - Submit each bundle at most once. Reconcile uncertain responses by readback. Retain only closed publication, failure and upload-intent receipts. A rerun attempt still fails the existing attempt-one admission guard. - Preserve failed original grades, failed cleanup and missing evidence. No missing original can become a passing result. - Keep external-input cases in the serial lane. Small native-only selections allow two cases at once. Small mixed selections allow one native and one external case. Larger selections retain eight native cases and one external case. - Add source, privacy and failure controls to the existing unit roster. Document the reporting flow and runner-group admission requirement. - Fix an existing corruption-test fixture that could write the same byte back to a random IV. Read and flip one bit instead. Keep the storage rejection and unpublished-plaintext assertions unchanged. ## Verification - 152 focused Node controls pass across the shared client, API, selection, CLI and streaming tests. The retained parent controls show the missing multipart and partial-readback behavior, plus the small external-selection error before its fix. - The focused workflow security control passes. Actionlint passes for both workflows. Changed JavaScript files pass syntax checks. Git diff checks pass. - Local controls use synthetic data and mocked HTTP. They make no provider call and do not start an app, database or native build. - Previous head `b51406df` passed PR CI (48 jobs pass and one is skipped). The review-driven test-only successor `0c65cc21` then failed one unchanged snapshot-storage corruption test, plus the final CI summary. The latest head includes its narrow fixture repair. Its first CI attempt separately hit runner shutdown/lost-communication failures, two setup/timeouts and a heartbeat-cancel cleanup HTTP 500/lock timeout; those observations are retained without claiming their underlying cause. One same-head failed-and-cancelled-only recovery now passes at `bf9c3e8f`: 48 jobs pass and one is skipped, including required `ci / verify` and `ci / e2e`. All 31 already-passing jobs retained their original execution records; all 15 recovered leaves and both dependent summaries passed. The first attempt remains recorded; the recovery does not establish the underlying causes of those failures. A small AWS run after delivery must prove that one case and both attachments are visible before another original case ends. That live proof is pending; local controls do not prove vendor ingestion or environment-secret resolution. - Retained earlier fixture-calibration failures remain recorded. The final integrated Node control run is green. A deterministic Node crypto/filesystem control also reproduces the no-op corruption mutation with a valid IV byte equal to `X`, then passes after flipping that byte. This control uses no app or database. The failing CI byte was not captured, so the exact historical byte remains unknown. ## Risks - The called workflow needs its exact master ref in the existing selected-workflow runner group. No IAM role change is needed. - Environment-only secrets must resolve in the called jobs. The caller passes no actual model or reporting key. The small live proof must confirm this behavior. - TestOps does not document an idempotency key for this upload route. An uncertain write is never resent automatically. If readback cannot verify it, the workflow fails with retained closed receipts. - The existing job run can already be marked finished by the selection watch. The live proof must record its actual stage and prove subsequent uploads are accepted. This PR does not reopen, stop or close that job run. - Backend ingestion does not prove that an already-open TestOps page refreshes automatically. - The corruption-test repair changes no production storage or cryptography. New-head AWS CI must qualify the existing database-backed owner test. - Each case adds short protected reader and upload jobs. The app source, original case clocks, graders, model, budgets, raw-evidence privacy boundary and zero eval retries stay unchanged. The versioned stream concurrency group serializes new streaming runs; an old immutable batch run can continue in its original group. ## Model Used OpenAI `gpt-6.1-sol`, with tool use and code execution. The session does not expose a context-window size. Independent source peers used the same model family. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 14:18:00 -06:00
DottaandPaperclip	97c4dc969e	test(evals): retire duplicate chat TestOps plans (#18625 ) ## Thinking Path > - Paperclip manages AI agents and preserves their work history. > - Product E2E cases use TestOps plans to select real AWS evaluations. > - The full chat catalog now includes the former smoke and completion-pilot cases. > - The two old bootstrap workflows can recreate their smaller plans after deletion. > - This pull request removes those workflows and stops their plan-creation functions before API access. > - The full-catalog workflow keeps its shared clients, evidence rules, case identities, and measured results. ## Linked Issues or Issue Description Refs #18542, #18516. The user requested removal of the duplicate smoke and completion-pilot plans and investigation of their recreation paths. Both old workflows accepted manual dispatch only. Their bootstrap operations created plans by name when the plans were absent. They had no schedules in their workflow definitions. The old workflow records were disabled before plan removal. An existing pilot execution was allowed to finish successfully. This change does not cancel executions or delete cases, results, launches, or private evidence. ## What Changed - Remove `chat-feature-testops.yml` and `chat-completion-testops.yml`. - Make the former singleton plan creator reject every call before API reads or mutation markers. - Reject pilot bootstrap before API access. Keep the full-catalog branch and its integration validation. - Remove tests that require the retired workflow files. Keep shared helper and full-catalog controls. Run the trusted-checkout boundary test against the active catalog workflow. - Replace old setup instructions with retirement guidance and the active full-catalog route. The master sync preserves the incoming typed launch-search fix from #18620, including both regression tests. The only conflict was the retired pilot documentation. Its search-format note remains as a retained-helper contract. ## Verification - Two new retirement controls fail against the parent code because it reaches an API or mutation marker. Both pass after the change. - Six owning Node suites after the master sync: 146 passed (144 before the sync). Shared contract: 22 passed. Catalog and retained export controls: 12 passed. Focused workflow and release controls: 7 passed. - Two modified API modules pass Node syntax checks. `git diff --check` passes. - The pure workflow-security file has the same four failures and 13 passes on both the untouched parent and this branch. Two assertions expect eight trusted bootstrap steps where master now has nine. Two lists omit the already-shipped catalog workflow. This change does not repair or relax those unrelated assertions. - A broader release-test invocation had 73 passes and 20 sandbox failures from blocked mock HTTP listeners or npm permissions. Its focused affected workflow control passes. A standard Vitest setup attempt stopped on a missing `acpx/runtime` bundle dependency before tests. No local app, database, provider, or native runtime was qualified. - Current-head PR CI passes: 48 successful jobs and one skipped job, including both required checks. Greptile is 5/5. Independent source review passes. No new model run is needed for this retirement. ## Risks - Existing integrations that point at either retired workflow will stop dispatching. This is the requested behavior; use the full catalog instead. - Historical evidence readers still use shared helpers. Those helpers remain, including the transport and official-context parser used by the active catalog. - Workflow removal is a repository change. TestOps plan deletion and disabling obsolete executors are separate service actions. Neither action should delete shared cases or measurements. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with source editing, shell tools, and independent source review. The context-window size is not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 11:07:32 -06:00
Dotta	e122478368	test(evals): run and record the chat completion pilot through TestOps (#18516 ) ## Thinking Path > - Paperclip evals must verify real user outcomes. > - A passing manual pilot does not make its cases runnable from TestOps. > - The existing TestOps adapter owns one separate backlog case. > - The three-case completion pilot needs its own exact selection and reporting path. > - This workflow binds a real TestOps job run to reviewed catalog source and AWS execution. > - Results retain their original verdict, checks, source and private evidence. ## Linked Issues or Issue Description Refs #18364, #18377, #18515 and #18509. This is the main-side companion to merged #18515. The immutable evaluated source is its actual integration-branch squash merge, `de0941835cc8f0d01c982574ee258e110babdbeb`; its tree matches the reviewed catalog head. The host admission remains inert until this main-only workflow exists. The broader catalog stays off master. The existing TestOps connection can run only the separate backlog cell. The project-work, project-defaults and instance-restart cases currently require manual campaign dispatch and private retrieval. This change creates a dedicated reusable TestOps plan and records fresh execution outcomes there. Source checks and registration do not establish a new live pass. ## What Changed - Add a default-branch-only workflow with exact TestOps selection, immutable source/build outputs, serial AWS execution and zero eval retries. No source/model/budget selector is accepted. - Register the three exact reusable cases and a separate plan with the existing reviewed GitHub integration. Preserve opaque job definitions. Validate the observed single Branch/defaultValue carrier, PATCH only enable/integration, and use the job’s default branch without rewriting its definition or adding plan overrides. Fence the initial operator launch by actual GitHub run number and nonce; normal TestOps Run remains repeatable. - Reuse the existing private-evidence authority, scanners, bounds and exact-key S3 policies. Authenticated selection supplies only this original attempt's exact source and case identity. The three trusted protocol files are unchanged. - Skip report retrieval after an unsuccessful build or skipped execution; retain evidence and failed grades when evaluation actually ran. - Check out trusted main after the target in both build and execution jobs, and reject any target tree containing the reserved trusted-helper path. A real temporary-Git-tree regression exercises the actual YAML guard. - Keep reporting credentials in separate protected jobs. Export only closed verdict/check/accounting metadata and private-evidence receipts to TestOps. Verify final PNGs privately and record their checksums; keep all screenshot bytes in protected AWS storage and out of public GitHub artifacts. Preserve failed grades and failed cleanup. - Verify actual same-run case results and complete result-owned attachment metadata within a fixed processing window. Missing or inherited attachments never qualify. Keep source/result links and stable case identities. ## Verification - 68 dispatch, selection, registration and readback controls pass. A checkout-boundary regression fails on the old ordering, accepts an ordinary target and rejects a candidate-owned helper tree before trusted checkout. They cover real workflow environment wiring, actual GitHub metadata guards, exact subset selection, wrong sources/cases/jobs, replays and attachment readiness. - 38 exporter controls pass, including the existing backlog exporter and the unchanged private-evidence authority consumer. Red/green controls prove project work accepts the catalog’s 1–3 run range in both export gates, while 0/4 runs are rejected; the actual pinned-source control verifies all three cases’ run bounds. Cancelled, failed and scheduled TestOps job stages are rejected even with complete results. The three new cases retain all 35 independent checks; private diagnostics and all image bytes are excluded from the public artifact. Extra media files and directories are rejected. - The actual checked-in workflow environment is exercised against the actual pinned source matrix and host function for all three cases before provider access. Local fixture controls make zero provider calls and do not claim live AWS qualification. - All 71 workflow-policy controls pass. Actionlint and owning typecheck pass. Relevant TypeScript build dependencies pass. - Full support suite: 5,827 passed, 13 failed, four skipped. A clean unchanged-master baseline reproduces all 13 failures in the same five files. They include fixture expectation drift, operating-receipt fixtures, a profile pin, pnpm bootstrap counts and local database startup. This change updates the exact TestOps workflow registries; the two affected boundary controls now pass. Baseline failures remain retained and are not presented as a green full support suite. - The catalog dependency passed CI and current-head 5/5 review, has no unresolved threads, and is merged. All current-head CI checks pass; both Greptile and Paperclip reviews are 5/5, with no unresolved threads. No new TestOps mutation, provider call or passing launch is claimed yet. A fresh three-case TestOps launch and verified server-side results remain the delivery gate. ## Risks This is a paid execution and reporting boundary. The new workflow accepts only trusted default-branch actor identities and a verified project/plan/job/case selection. Its evaluated source is immutable and separate from main reporting code. The full catalog is not promoted. Screenshots, raw evidence and hidden provider data stay in protected storage. Public GitHub artifacts carry only the closed metadata projection. TestOps receives checks and private-evidence availability/checksum receipts. Unknown setup or AWS accounting stays unknown. One passing launch will not establish full-catalog reliability. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code execution and tool use. Exact hosted revision and context-window size are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-01 06:09:47 -06:00

Author

SHA1

Message

Date

DottaandPaperclip

0fd82e386e

feat(evals): stream completed chat catalog results to TestOps (#18647 )

## Thinking Path

> - Paperclip lets people manage AI agents and inspect their work.
> - The chat catalog uses real Product E2E cases and reports their
original grades to TestOps.
> - The current report matrix waits for every selected worker.
> - A large selection can hide completed results for hours.
> - Each case can use the existing private reader and safe exporter
after its own worker ends.
> - This PR gives each case a separate upload and readback job, then
verifies final coverage without another upload.
> - People can inspect completed cases while the rest of the selection
runs.

## Linked Issues or Issue Description

Refs #18542, #18625.

**What existing behavior does this improve?**

The existing chat catalog reports results to TestOps only after all
selected workers finish.

**Current behavior**

One slow case holds back the report and upload jobs for all other cases.

**Proposed behavior**

Each finished case gets its own protected reader, safe projection,
upload and backend verification. The final job checks the whole
selection through read-only requests. It does not upload results again.

**Reason and benefit**

A person can inspect a completed grade and its safe evidence receipts
while other cases run. This changes future runs. It does not change an
in-flight run.

## What Changed

- Add one same-commit reusable workflow with separate worker,
private-reader and TestOps-upload jobs. Each job uses only its own
protected environment. Callers pass empty named secret placeholders and
do not inherit secrets.
- Reuse the original exporter and grade checks. Freeze each case's three
Allure JSON files once. Allow known sibling results during per-case
readback. Keep final exact selection coverage.
- Submit each bundle at most once. Reconcile uncertain responses by
readback. Retain only closed publication, failure and upload-intent
receipts. A rerun attempt still fails the existing attempt-one admission
guard.
- Preserve failed original grades, failed cleanup and missing evidence.
No missing original can become a passing result.
- Keep external-input cases in the serial lane. Small native-only
selections allow two cases at once. Small mixed selections allow one
native and one external case. Larger selections retain eight native
cases and one external case.
- Add source, privacy and failure controls to the existing unit roster.
Document the reporting flow and runner-group admission requirement.
- Fix an existing corruption-test fixture that could write the same byte
back to a random IV. Read and flip one bit instead. Keep the storage
rejection and unpublished-plaintext assertions unchanged.

## Verification

- 152 focused Node controls pass across the shared client, API,
selection, CLI and streaming tests. The retained parent controls show
the missing multipart and partial-readback behavior, plus the small
external-selection error before its fix.
- The focused workflow security control passes. Actionlint passes for
both workflows. Changed JavaScript files pass syntax checks. Git diff
checks pass.
- Local controls use synthetic data and mocked HTTP. They make no
provider call and do not start an app, database or native build.
- Previous head `b51406df` passed PR CI (48 jobs pass and one is
skipped). The review-driven test-only successor `0c65cc21` then failed
one unchanged snapshot-storage corruption test, plus the final CI
summary. The latest head includes its narrow fixture repair. Its first
CI attempt separately hit runner shutdown/lost-communication failures,
two setup/timeouts and a heartbeat-cancel cleanup HTTP 500/lock timeout;
those observations are retained without claiming their underlying cause.
One same-head failed-and-cancelled-only recovery now passes at
`bf9c3e8f`: 48 jobs pass and one is skipped, including required `ci /
verify` and `ci / e2e`. All 31 already-passing jobs retained their
original execution records; all 15 recovered leaves and both dependent
summaries passed. The first attempt remains recorded; the recovery does
not establish the underlying causes of those failures. A small AWS run
after delivery must prove that one case and both attachments are visible
before another original case ends. That live proof is pending; local
controls do not prove vendor ingestion or environment-secret resolution.
- Retained earlier fixture-calibration failures remain recorded. The
final integrated Node control run is green. A deterministic Node
crypto/filesystem control also reproduces the no-op corruption mutation
with a valid IV byte equal to `X`, then passes after flipping that byte.
This control uses no app or database. The failing CI byte was not
captured, so the exact historical byte remains unknown.

## Risks

- The called workflow needs its exact master ref in the existing
selected-workflow runner group. No IAM role change is needed.
- Environment-only secrets must resolve in the called jobs. The caller
passes no actual model or reporting key. The small live proof must
confirm this behavior.
- TestOps does not document an idempotency key for this upload route. An
uncertain write is never resent automatically. If readback cannot verify
it, the workflow fails with retained closed receipts.
- The existing job run can already be marked finished by the selection
watch. The live proof must record its actual stage and prove subsequent
uploads are accepted. This PR does not reopen, stop or close that job
run.
- Backend ingestion does not prove that an already-open TestOps page
refreshes automatically.
- The corruption-test repair changes no production storage or
cryptography. New-head AWS CI must qualify the existing database-backed
owner test.
- Each case adds short protected reader and upload jobs. The app source,
original case clocks, graders, model, budgets, raw-evidence privacy
boundary and zero eval retries stay unchanged. The versioned stream
concurrency group serializes new streaming runs; an old immutable batch
run can continue in its original group.

## Model Used

OpenAI `gpt-6.1-sol`, with tool use and code execution. The session does
not expose a context-window size. Independent source peers used the same
model family.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-11-02 14:18:00 -06:00

DottaandPaperclip

97c4dc969e

test(evals): retire duplicate chat TestOps plans (#18625 )

## Thinking Path

> - Paperclip manages AI agents and preserves their work history.
> - Product E2E cases use TestOps plans to select real AWS evaluations.
> - The full chat catalog now includes the former smoke and
completion-pilot cases.
> - The two old bootstrap workflows can recreate their smaller plans
after deletion.
> - This pull request removes those workflows and stops their
plan-creation functions before API access.
> - The full-catalog workflow keeps its shared clients, evidence rules,
case identities, and measured results.

## Linked Issues or Issue Description

Refs #18542, #18516.

The user requested removal of the duplicate smoke and completion-pilot
plans and investigation of their recreation paths. Both old workflows
accepted manual dispatch only. Their bootstrap operations created plans
by name when the plans were absent. They had no schedules in their
workflow definitions.

The old workflow records were disabled before plan removal. An existing
pilot execution was allowed to finish successfully. This change does not
cancel executions or delete cases, results, launches, or private
evidence.

## What Changed

- Remove `chat-feature-testops.yml` and `chat-completion-testops.yml`.
- Make the former singleton plan creator reject every call before API
reads or mutation markers.
- Reject pilot bootstrap before API access. Keep the full-catalog branch
and its integration validation.
- Remove tests that require the retired workflow files. Keep shared
helper and full-catalog controls. Run the trusted-checkout boundary test
against the active catalog workflow.
- Replace old setup instructions with retirement guidance and the active
full-catalog route.

The master sync preserves the incoming typed launch-search fix from
#18620, including both regression tests. The only conflict was the
retired pilot documentation. Its search-format note remains as a
retained-helper contract.

## Verification

- Two new retirement controls fail against the parent code because it
reaches an API or mutation marker. Both pass after the change.
- Six owning Node suites after the master sync: 146 passed (144 before
the sync). Shared contract: 22 passed. Catalog and retained export
controls: 12 passed. Focused workflow and release controls: 7 passed.
- Two modified API modules pass Node syntax checks. `git diff --check`
passes.
- The pure workflow-security file has the same four failures and 13
passes on both the untouched parent and this branch. Two assertions
expect eight trusted bootstrap steps where master now has nine. Two
lists omit the already-shipped catalog workflow. This change does not
repair or relax those unrelated assertions.
- A broader release-test invocation had 73 passes and 20 sandbox
failures from blocked mock HTTP listeners or npm permissions. Its
focused affected workflow control passes. A standard Vitest setup
attempt stopped on a missing `acpx/runtime` bundle dependency before
tests. No local app, database, provider, or native runtime was
qualified.
- Current-head PR CI passes: 48 successful jobs and one skipped job,
including both required checks. Greptile is 5/5. Independent source
review passes. No new model run is needed for this retirement.

## Risks

- Existing integrations that point at either retired workflow will stop
dispatching. This is the requested behavior; use the full catalog
instead.
- Historical evidence readers still use shared helpers. Those helpers
remain, including the transport and official-context parser used by the
active catalog.
- Workflow removal is a repository change. TestOps plan deletion and
disabling obsolete executors are separate service actions. Neither
action should delete shared cases or measurements.

## Model Used

OpenAI `gpt-6.1-sol` through Codex, with source editing, shell tools,
and independent source review. The context-window size is not asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-11-02 11:07:32 -06:00

Dotta

e122478368

test(evals): run and record the chat completion pilot through TestOps (#18516 )

## Thinking Path

> - Paperclip evals must verify real user outcomes.
> - A passing manual pilot does not make its cases runnable from
TestOps.
> - The existing TestOps adapter owns one separate backlog case.
> - The three-case completion pilot needs its own exact selection and
reporting path.
> - This workflow binds a real TestOps job run to reviewed catalog
source and AWS execution.
> - Results retain their original verdict, checks, source and private
evidence.

## Linked Issues or Issue Description

Refs #18364, #18377, #18515 and #18509. This is the main-side companion
to merged #18515. The immutable evaluated source is its actual
integration-branch squash merge,
`de0941835cc8f0d01c982574ee258e110babdbeb`; its tree matches the
reviewed catalog head. The host admission remains inert until this
main-only workflow exists. The broader catalog stays off master.

The existing TestOps connection can run only the separate backlog cell.
The project-work, project-defaults and instance-restart cases currently
require manual campaign dispatch and private retrieval. This change
creates a dedicated reusable TestOps plan and records fresh execution
outcomes there. Source checks and registration do not establish a new
live pass.

## What Changed

- Add a default-branch-only workflow with exact TestOps selection,
immutable source/build outputs, serial AWS execution and zero eval
retries. No source/model/budget selector is accepted.
- Register the three exact reusable cases and a separate plan with the
existing reviewed GitHub integration. Preserve opaque job definitions.
Validate the observed single Branch/defaultValue carrier, PATCH only
enable/integration, and use the job’s default branch without rewriting
its definition or adding plan overrides. Fence the initial operator
launch by actual GitHub run number and nonce; normal TestOps Run remains
repeatable.
- Reuse the existing private-evidence authority, scanners, bounds and
exact-key S3 policies. Authenticated selection supplies only this
original attempt's exact source and case identity. The three trusted
protocol files are unchanged.
- Skip report retrieval after an unsuccessful build or skipped
execution; retain evidence and failed grades when evaluation actually
ran.
- Check out trusted main after the target in both build and execution
jobs, and reject any target tree containing the reserved trusted-helper
path. A real temporary-Git-tree regression exercises the actual YAML
guard.
- Keep reporting credentials in separate protected jobs. Export only
closed verdict/check/accounting metadata and private-evidence receipts
to TestOps. Verify final PNGs privately and record their checksums; keep
all screenshot bytes in protected AWS storage and out of public GitHub
artifacts. Preserve failed grades and failed cleanup.
- Verify actual same-run case results and complete result-owned
attachment metadata within a fixed processing window. Missing or
inherited attachments never qualify. Keep source/result links and stable
case identities.

## Verification

- 68 dispatch, selection, registration and readback controls pass. A
checkout-boundary regression fails on the old ordering, accepts an
ordinary target and rejects a candidate-owned helper tree before trusted
checkout. They cover real workflow environment wiring, actual GitHub
metadata guards, exact subset selection, wrong sources/cases/jobs,
replays and attachment readiness.
- 38 exporter controls pass, including the existing backlog exporter and
the unchanged private-evidence authority consumer. Red/green controls
prove project work accepts the catalog’s 1–3 run range in both export
gates, while 0/4 runs are rejected; the actual pinned-source control
verifies all three cases’ run bounds. Cancelled, failed and scheduled
TestOps job stages are rejected even with complete results. The three
new cases retain all 35 independent checks; private diagnostics and all
image bytes are excluded from the public artifact. Extra media files and
directories are rejected.
- The actual checked-in workflow environment is exercised against the
actual pinned source matrix and host function for all three cases before
provider access. Local fixture controls make zero provider calls and do
not claim live AWS qualification.
- All 71 workflow-policy controls pass. Actionlint and owning typecheck
pass. Relevant TypeScript build dependencies pass.
- Full support suite: 5,827 passed, 13 failed, four skipped. A clean
unchanged-master baseline reproduces all 13 failures in the same five
files. They include fixture expectation drift, operating-receipt
fixtures, a profile pin, pnpm bootstrap counts and local database
startup. This change updates the exact TestOps workflow registries; the
two affected boundary controls now pass. Baseline failures remain
retained and are not presented as a green full support suite.
- The catalog dependency passed CI and current-head 5/5 review, has no
unresolved threads, and is merged. All current-head CI checks pass; both
Greptile and Paperclip reviews are 5/5, with no unresolved threads. No
new TestOps mutation, provider call or passing launch is claimed yet. A
fresh three-case TestOps launch and verified server-side results remain
the delivery gate.

## Risks

This is a paid execution and reporting boundary. The new workflow
accepts only trusted default-branch actor identities and a verified
project/plan/job/case selection. Its evaluated source is immutable and
separate from main reporting code. The full catalog is not promoted.
Screenshots, raw evidence and hidden provider data stay in protected
storage. Public GitHub artifacts carry only the closed metadata
projection. TestOps receives checks and private-evidence
availability/checksum receipts. Unknown setup or AWS accounting stays
unknown. One passing launch will not establish full-catalog reliability.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code execution
and tool use. Exact hosted revision and context-window size are not
exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

2026-11-01 06:09:47 -06:00