test(evals): run Product regression from TestOps (#18983)

## Thinking Path

> - Paperclip manages agent work through governed product workflows.
> - Product E2E tests check those workflows through the real app and
runner.
> - The saved Product regression plan has 20 imported case identities.
> - The plan has no live executor, so historical baseline rows do not
qualify its current behavior.
> - The existing paid full-stack workflow already owns all 20 required
journeys.
> - This change binds TestOps selection and reporting to that workflow
without changing the product oracle or paid admission.
> - Maintainers can run the plan and inspect current case verdicts and
retained evidence.

## Linked Issues or Issue Description

**What happened?**

The standing Product regression plan has no executor. Its dashboard
points at a historical 355-result snapshot across several suites and
revisions.

**Expected behavior**

A TestOps plan run must select the intended existing full-stack cells,
execute them on GitHub, and report their original case IDs, assertions
and source revision. Missing evidence must remain a visible failure.

**Steps to reproduce**

Open the Product regression plan and inspect Executors. There is no live
job. The existing full-stack workflow accepts exact cell IDs but has no
TestOps selection or reporting path.

**Paperclip version or commit**

The branch includes reviewed master correction 04a24f4e06. This change
does not alter scenario prompts, models, deadlines or assertions.

Related public work: #18542 establishes the separate chat-catalog lane.
#18666 repairs its workflow URL binding. These changes do not implement
this Product plan.

## What Changed

- Add optional TestOps service inputs and a manual bootstrap selector to
the existing full-stack workflow.
- Map the original 20 case IDs and full names to existing native
Codex/local cells, with the existing native OpenCode/local
task-workspaces case.
- Keep reporting tokens in trusted default-branch selection and
publication jobs. Keep the existing paid provider authorization and
protected environments.
- Bind retained results to the workflow, job run, launch, source
revision and selection hash. Preserve missing-result failures and
earlier attempt verdict receipts.
- Verify uploaded identities, coverage and verdicts. Document full-plan
and subset reporting limits.
- Keep the outstanding-decision reservation live while its existing
deletion-fence test asserts the original 409 guard. Release the test
provider in finally. Production deletion and accounting code stay
unchanged.

## Verification

- `node --test tests/runner-e2e/product-regression-testops.test.mjs`: 13
positive and negative controls pass, including the separated CLI
handoff.
- Existing paid-workflow security regression passes.
- `actionlint -shellcheck= .github/workflows/runner-full-stack-e2e.yml`:
passes.
- The Product E2E catalog compatibility test confirms all 20 exact cells
exist: passes.
- `pnpm test:e2e:runner:typecheck`: passes.
- Fresh Greptile review of final head 61ef7a6e is 5/5 (11 files, no new
findings); both initial reporting findings are repaired and resolved.
- `server/src/__tests__/decision-models.test.ts`: all 37 controls pass
on a disposable official PostgreSQL 18 container in 22.02 seconds. The
unchanged focused case passed once and all five local samples; the
original Linux CI failure remains retained. This is not a claimed
deterministic local red reproduction. The new gate explicitly preserves
the outstanding-reservation precondition.
- The existing master readiness-fixture correction passes all three
retained-workspace controls on the same database in 25.79 seconds. It is
incorporated from master, not duplicated in this PR's diff.
- Final-head Linux CI also passed all 37 decision-model controls in
7.047 seconds; the server shard passed 120 files and 2,004 tests. See
[the retained successful
shard](https://github.com/paperclipai/paperclip/actions/runs/41870214517/job/126715358759).
- Previous CI failures are retained: [three stale readiness-fixture
cases](https://github.com/paperclipai/paperclip/actions/runs/41864061890/job/126694812885)
and [the reservation fixture
race](https://github.com/paperclipai/paperclip/actions/runs/41864061890/job/126694812894).
The separate [Canary Dry
Run](https://github.com/paperclipai/paperclip/actions/runs/41864061890/job/126694811605)
recorded a lost self-hosted runner connection before any step. The
fixture repairs are verified on changed source; the separate
Spot-interrupted CI attempt and its bounded retry are recorded below.
- Actual one-cell GitHub canary on the unchanged product source:
https://github.com/paperclipai/paperclip/actions/runs/41860189801 . The
native Codex/local message-marker case passed all six assertions and
cleanup in 35.24 seconds. The persisted issue is Done and its one run
succeeded. This is supporting evidence; it is not TestOps acceptance.
- Final-head CI attempt 1 was interrupted: three failed jobs have EC2
`Server.SpotInstanceTermination` receipts; six cancelled jobs show the
same simultaneous shutdown. Original logs are retained. One [failed-job
retry](https://github.com/paperclipai/paperclip/actions/runs/41870214517/attempts/2)
passed all nine interrupted checks while preserving prior successful
jobs.
- Final-head full repository CI passes, including both required `ci /
verify` and `ci / e2e` gates. GitHub still requires a non-author
code-owner approval for the workflow change; normal auto-merge is
queued. A fresh TestOps-triggered subset/full-plan acceptance remains
pending. Local projection checks do not establish a live Product pass.

## Risks

- The first bootstrap upload must establish the official TestOps job
before the saved plan can use it.
- This lane tests one declared local native configuration per case. It
does not qualify the historical 355-row matrix.
- Upload failures remain visible. A missing retained report produces
broken case results and cannot pass.
- The existing non-TestOps campaign path must keep its selection and
scheduling behavior; paid-workflow regression coverage remains in place.

## Model Used

OpenAI Codex with reasoning, code execution and tool use. The host does
not expose the exact model ID or context window to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

.github/workflows/runner-full-stack-e2e.yml

@@ -48,6 +48,18 @@ on:
         description: "Comma-separated full suite.profile.environment.case IDs; exclusive with other selectors"
         type: string
         required: false
+      testops_case:
+        description: "Product regression TestOps bootstrap: all, case name or case ID; leave empty for normal campaigns"
+        type: string
+        required: false
+      ALLURE_JOB_RUN_ID:
+        description: "Allure TestOps service input; its requested selection takes precedence"
+        type: string
+        required: false
+      ALLURE_USERNAME:
+        description: "Allure TestOps service input"
+        type: string
+        required: false
 
 permissions:
   contents: read
@@ -284,10 +296,67 @@ jobs:
           retention-days: 30
           if-no-files-found: error
 
+  testops_selection:
+    name: Bind Product regression TestOps selection
+    needs: authorize
+    if: github.event_name == 'workflow_dispatch' && (inputs.ALLURE_JOB_RUN_ID != '' || inputs.testops_case != '')
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    environment: runner-e2e-testops-reporting
+    permissions:
+      contents: read
+      actions: read
+    outputs:
+      ids: ${{ steps.selection.outputs.ids }}
+      job_run_started: ${{ steps.selection.outputs.job_run_started }}
+      job_run_id: ${{ steps.selection.outputs.job_run_id }}
+      selection_sha256: ${{ steps.selection.outputs.selection_sha256 }}
+      launch_url: ${{ steps.selection.outputs.launch_url }}
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+        with:
+          ref: ${{ github.sha }}
+          persist-credentials: false
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
+        with:
+          node-version: 24
+      - uses: allure-framework/setup-allurectl@2d89ee0b7ba2f59dbe9849b6df5763cab78e14db # v1
+        with:
+          allurectl-version: '2.23.1'
+          allure-endpoint: https://paperclip.testops.cloud
+          allure-project-id: '1'
+          allure-token: ${{ secrets.ALLURE_TOKEN }}
+      - name: Resolve only admitted plan cases before provider jobs
+        id: selection
+        env:
+          SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
+          EVAL_CADENCE: ${{ inputs.cadence || 'manual' }}
+          SELECT_ID: ${{ inputs.id }}
+          SELECT_SUITE: ${{ inputs.suite }}
+          SELECT_GROUP: ${{ inputs.group }}
+          SELECT_PROFILE: ${{ inputs.profile }}
+          SELECT_ENVIRONMENT: ${{ inputs.environment }}
+          SELECT_CASE: ${{ inputs.case }}
+          TESTOPS_CASE: ${{ inputs.testops_case }}
+          ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
+          ALLURE_USERNAME: ${{ inputs.ALLURE_USERNAME }}
+          ALLURE_LAUNCH_NAME: Product regression · ${{ github.run_number }}
+          ALLURE_LAUNCH_TAGS: product-regression,github-actions
+        run: node tests/runner-e2e/product-regression-testops-cli.mjs select product-testops/selection.json
+      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+        with:
+          name: product-testops-selection-${{ github.run_id }}-${{ github.run_attempt }}
+          path: product-testops/selection.json
+          if-no-files-found: error
+          retention-days: 30
+
   catalog:
     name: Validate catalog and select cells
-    needs: [authorize, target_lock]
-    if: github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true')
+    needs: [authorize, target_lock, testops_selection]
+    if: >-
+      always() && needs.authorize.result == 'success' && needs.target_lock.result == 'success' &&
+      (needs.testops_selection.result == 'success' || needs.testops_selection.result == 'skipped') &&
+      (github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true'))
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
@@ -371,7 +440,7 @@ jobs:
           SELECT_PROFILE: ${{ inputs.profile }}
           SELECT_ENVIRONMENT: ${{ inputs.environment }}
           SELECT_CASE: ${{ inputs.case }}
-          SELECT_ID: ${{ inputs.id }}
+          SELECT_ID: ${{ needs.testops_selection.outputs.ids || inputs.id }}
           MAX_PARALLEL: ${{ vars.RUNNER_E2E_MAX_PARALLEL || needs.authorize.outputs.max_parallel_default }}
           MAX_PARALLEL_LIMIT: ${{ needs.authorize.outputs.max_parallel_limit }}
           REQUESTED_MAX_PARALLEL: ${{ inputs.max_parallel }}
@@ -1946,6 +2015,69 @@ jobs:
             echo "ready=false" >> "$GITHUB_OUTPUT"
           fi
 
+  publish_testops:
+    name: Publish bound Product regression results to TestOps
+    needs: [authorize, testops_selection, catalog, report]
+    if: always() && needs.testops_selection.outputs.job_run_started == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    environment: runner-e2e-testops-reporting
+    permissions:
+      contents: read
+      actions: read
+    env:
+      SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
+      SELECTION_SHA256: ${{ needs.testops_selection.outputs.selection_sha256 }}
+      EVAL_CADENCE: manual
+      ALLURE_JOB_RUN_ID: ${{ needs.testops_selection.outputs.job_run_id || inputs.ALLURE_JOB_RUN_ID }}
+      ALLURE_USERNAME: ${{ inputs.ALLURE_USERNAME }}
+      ALLURE_RESULTS: ${{ github.workspace }}/product-regression-allure
+      REPORT_DIR: ${{ github.workspace }}/product-report
+      EXPORT_SUMMARY: ${{ github.workspace }}/product-regression-export.json
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+        with:
+          # Only trusted workflow code sees the reporting token. Candidate
+          # reports are parsed as data and never loaded as executable modules.
+          ref: ${{ github.sha }}
+          persist-credentials: false
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
+        with:
+          node-version: 24
+      - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
+        if: needs.testops_selection.result == 'success' && !cancelled()
+        id: download_selection
+        with:
+          name: product-testops-selection-${{ github.run_id }}-${{ github.run_attempt }}
+          path: product-testops
+      - name: Download retained full-stack report
+        if: needs.testops_selection.result == 'success' && !cancelled()
+        continue-on-error: true
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
+        with:
+          name: runner-e2e-report-${{ github.run_id }}-${{ github.run_attempt }}
+          path: product-report
+      - uses: allure-framework/setup-allurectl@2d89ee0b7ba2f59dbe9849b6df5763cab78e14db # v1
+        if: always()
+        with:
+          allurectl-version: '2.23.1'
+          allure-endpoint: https://paperclip.testops.cloud
+          allure-project-id: '1'
+          allure-token: ${{ secrets.ALLURE_TOKEN }}
+      - name: Project original verdicts and explicitly report missing results
+        if: steps.download_selection.outcome == 'success' && !cancelled()
+        id: project
+        run: node tests/runner-e2e/product-regression-testops-cli.mjs project product-testops/selection.json
+      - name: Upload only the bound result projection
+        if: steps.project.outcome == 'success'
+        run: allurectl upload --validate-json "$ALLURE_RESULTS"
+      - name: Verify server-side case identities, completeness and verdicts
+        if: steps.project.outcome == 'success'
+        run: node tests/runner-e2e/product-regression-testops-cli.mjs verify product-testops/selection.json
+      - name: Finish the original TestOps job run
+        if: always()
+        run: allurectl job-run stop
+
   publish_history:
     name: Publish S3 history and Pages bundle with declared screenshots
     needs: [authorize, catalog, report]