Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-03 17:51:58 +01:00
Code Issues Packages Projects Releases Wiki Activity

test(evals): add the full chat catalog to TestOps (#18542)

Browse Source 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat feature evaluations check real user requests through the
product.
> - The YAML catalog has 244 scenarios, but the shipped TestOps
completion route selects only three.
> - The preserved catalog already supplies the execution mappings and
independent oracles.
> - A larger TestOps plan needs exact identities, bounded selection and
source-bound private evidence.
> - This pull request adds that plan route without merging the catalog
app into master or replacing its grades.
> - Users can run the full catalog or a selected subset and see the
actual outcome and prerequisites.

## Linked Issues or Issue Description

Refs: #18516 and #18515. This extends their completion pilot and reuses
its trusted reporting path.

**What existing behavior does this improve?**
The existing three-case TestOps completion workflow.

**Current behavior**
The remaining catalog scenarios cannot be selected through that TestOps
job. The one-case backlog plan has a separate strict singleton contract.

**Proposed behavior**
A separate candidate-scoped plan maps all 244 scenarios. Existing case
aliases, jobs, plans and result history stay intact. Registration is
distinct from a measured pass.

## What Changed

- Add a source-derived 244-row manifest. It binds the 242 domain
descriptors and two legacy tasks to the unchanged YAML queries,
contexts, outcomes and original bounds. The JSON snapshot avoids
importing the unmerged domain tree into master.
- Reconcile exact case identities and a dedicated full-catalog plan.
Reuse legacy aliases, including the existing backlog case, without
renaming them. Preserve history and stop on conflicting aliases or
uncertain writes.
- Extend the shared completion adapter with a separate trusted
full-catalog workflow context. Return a compact observed binding that
fits one 48 KB protected variable. Keep the three-case pilot contract
intact.
- Add a fixed-source manual workflow. Small selections run serially.
Larger selections allow eight independent workers and one serial
external-input worker. Each worker and report reader keeps a singleton
approval and exact-key AWS session. The eight-cell private authority
limit is unchanged.
- Validate and merge metadata-only singleton projections before one
protected Allure upload and complete paginated readback. Preserve failed
originals and cleanup failures beside passed results. Keep screenshots
and raw evidence private.
- Discard a changing full-catalog page-count snapshot and retry within
the same observer bounds. Malformed page fields still fail at once. A
pass still needs one consistent exact selection and all final evidence
checks.
- Keep workflow source and app source separate. The full lane builds
candidate `3b66c3c144360ddf32714a0742fd774b8ceafb16`. Its only changes
from the existing `de0941835cc8f0d01c982574ee258e110babdbeb` pin are
finite TestOps host admission and its offline test. This is not
current-master app qualification.
- Give full selections above three cases a bounded metadata-read
allowance above the existing 120-second deadline ceiling. Readback has
two upper bounds: at most twelve polls with five-second pauses, and a
size-derived deadline of at most 374 seconds, whichever exhausts first.
There is no guaranteed minimum wait or processing SLA. Small selections
and pilot deadlines stay unchanged. Final identity, grade and evidence
predicates stay intact.
- Document the real readiness limits and setup path. Five coverage gaps,
six evidence exceptions, eighteen additional-credential declarations and
thirty-seven external-input declarations overlap. They are not new pass
claims.

## Verification

- Related Node controls passed: pilot API and dispatch, 244-row
manifest, idempotent reconciliation, full selection, three-page result
readback, source guards and partition/output-size bounds. The latest ten
adapter controls include the actual registration receipt's variable-size
limit and the real bootstrap facade's closed failure codes and privacy
checks.
- A deterministic 244-result replay with one second per HTTP response
failed at the old deadline and now verifies after 127 virtual seconds. A
three-second response replay still rejects at 375 virtual seconds. The
full-lane allowance accounts for paginated reads and four-way result
batches; it is not a guaranteed network SLA. All 55 related API/adapter
controls pass. They also reproduce the changing-page-count failure
before the fix, then verify the consistent snapshot. Permanently
changing counts fail after twelve polls. A malformed numeric field still
fails before a retry.
- Owning export controls: 26 passed. They include all 244 synthetic
carriers, mixed passed/failed originals, missing and foreign
projections, privacy filtering and exact source binding.
- Candidate host owner: 31 offline controls passed. The new
finite-catalog case failed before admission changed.
- Preserve two actual local regressions: canonical scenario provenance
failed before its one-line fix; the full registration binding exceeded
the variable limit before compact output. Both now pass.
- Initial local harness setup attempts failed before test execution. The
final narrow owner configuration avoids provider-bundle setup.
- Full repository tests, build and typecheck were not run locally.
Normal required PR CI provides the remote checks.
- Owning TypeScript no-emit, actionlint, paid-workflow policy with
unsafe controls, and diff checks passed.
- Local tests used Node 26.4.0. These are synthetic source controls. No
local app, database, native build or provider qualification ran.
- CI
[41307571776](https://github.com/paperclipai/paperclip/actions/runs/41307571776)
on `23e518be` recorded one 15-second timeout in
`attention-retention-work.test.ts:153`. All other ordinary leaves
passed. The test used a fixed JavaScript date but its retention deadline
comes from PostgreSQL `updatedAt`. The exact failing await and original
SQL timestamp were not captured. The corrected pre-sync head’s server
shard (2/12) passed on AWS in CI 41309030928. Current master
independently fixed the same clock assumption in #18543; conflict
resolution adopts that whole upstream file. No local PostgreSQL run or
timeout increase was used.
- Current-head normal CI
[41463567187](https://github.com/paperclipai/paperclip/actions/runs/41463567187)
passes all required checks on
`fc6c2309cfc7d1e5a83f0a597c7fa1d2beb987e1`, with Greptile 5/5 and all
review threads resolved. The final diff has 23 catalog files. Current
master’s whole retention test is adopted; the redundant earlier test
change is absent. The valid changing-page snapshot regression passes
with all 55 related API/adapter controls. No new TestOps import or live
evaluation has run for this change. After delivery, use metadata-only
bootstrap and a three-case normal TestOps selection to verify the new
route. Do not claim a new 244-case measurement.

## Risks

- The app target remains the preserved catalog candidate. Each result
records its target and workflow revisions. No old campaign ordinal,
acceptance gate or core fix is changed.
- Some mapped cases need private account configuration or have known
source/evidence gaps. Their original checks and evidence rules can still
fail. A registration does not establish readiness or success.
- Whole-plan execution can take hours and spend provider and AWS money.
Original per-case limits and zero retries remain. Unknown setup usage is
not free. This PR does not run all 244 cases.
- The raw private `SOURCE_REF` field retains the prior
integration-lineage label. It is not a verified checked-out branch. The
actual candidate checkout and all authority checks use immutable
`3b66c3c144360ddf32714a0742fd774b8ceafb16`; public projections set
`source.ref` to null.
- TestOps has no atomic bulk upsert. Reconciliation retains partial
progress and performs no automatic resend after an uncertain response.
- The trusted inventory keeps a 12 MiB limit per public carrier file.
The larger parser ceiling does not promise admission above that limit.
The existing 244-case fixture produces a 1,359,465-byte normalized
carrier; this is not a guarantee for future payload sizes.
- Raw evidence remains under the existing private retention owner.
Public GitHub artifacts and TestOps attachments contain only allowlisted
metadata and availability receipts.
- The new workflow needs the exact master entry in the existing runner
group's selected-workflow list before a live run. No repository or
runner-group visibility is broadened by this source change.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) assisted with implementation and
independent source review, using code and tools. The host does not
report the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run the relevant local controls and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-02 08:58:44 -06:00
1 parent 39e5ccd1a8
commit 774ddc7271
23 files changed
+13642 -86
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/chat-catalog-testops.yml
+498
View File
@@ -0,0 +1,498 @@
name: Chat catalog TestOps
run-name: Chat catalog TestOps / ${{ inputs.operation || 'evaluate' }} / ${{ inputs.ALLURE_JOB_RUN_ID || github.run_id }}
on:
workflow_dispatch:
inputs:
operation:
type: choice
options: [evaluate, bootstrap]
default: evaluate
description: Run the exact TestOps selection, or register its catalog plan
ALLURE_JOB_RUN_ID:
type: string
required: false
description: TestOps job-run identity; source, model and clocks are fixed
ALLURE_USERNAME:
type: string
required: false
description: Reserved service metadata; never grants authority
permissions:
contents: read
actions: read
concurrency:
group: chat-catalog-testops
cancel-in-progress: false
env:
NODE_OPTIONS: ''
NODE_PATH: ''
AWS_EC2_METADATA_DISABLED: 'true'
jobs:
authorize:
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
operation: ${{ steps.guard.outputs.operation }}
target_sha: ${{ steps.guard.outputs.target_sha }}
runner: runs-on/fleet=paperclip-public-pr-x64/env=public-ci
steps:
- &checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ github.sha }}
persist-credentials: false
- &node
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- &guard
name: Verify actual main-only dispatch and both actors
id: guard
env:
GH_TOKEN: ${{ github.token }}
TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
run: node tests/runner-e2e/chat-completion-testops-cli.mjs guard
select:
name: Register the dedicated plan or validate its real TestOps selection
needs: authorize
runs-on: ubuntu-latest
timeout-minutes: 15
environment: runner-e2e-testops-reporting
outputs:
artifact_id: ${{ steps.selection_upload.outputs.artifact-id }}
selection_sha256: ${{ steps.selection.outputs.selection_sha256 }}
execution_ids: ${{ steps.selection.outputs.execution_ids }}
matrix: ${{ steps.selection.outputs.matrix }}
native_matrix: ${{ steps.selection.outputs.native_matrix }}
external_matrix: ${{ steps.selection.outputs.external_matrix }}
native_count: ${{ steps.selection.outputs.native_count }}
external_count: ${{ steps.selection.outputs.external_count }}
parallelism: ${{ steps.selection.outputs.parallelism }}
policy: ${{ steps.selection.outputs.policy }}
steps:
- *checkout
- *node
- *guard
- &allure
if: needs.authorize.outputs.operation != 'launch-plan'
uses: allure-framework/setup-allurectl@2d89ee0b7ba2f59dbe9849b6df5763cab78e14db # v1
with:
allurectl-version: 2.23.1
allure-endpoint: https://paperclip.testops.cloud
allure-project-id: '1'
allure-token: ${{ secrets.ALLURE_TOKEN }}
- name: Register all 244 catalog cases without synthetic results
if: needs.authorize.outputs.operation == 'bootstrap'
env:
GH_TOKEN: ${{ github.token }}
TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
ALLURE_ENDPOINT: https://paperclip.testops.cloud
ALLURE_PROJECT_ID: '1'
ALLURE_RESULTS: ${{ runner.temp }}/chat-completion-bootstrap-results
CHAT_TESTOPS_INTEGRATION_ID: ${{ vars.CHAT_TESTOPS_INTEGRATION_ID }}
run: |
mkdir -m 700 "$ALLURE_RESULTS"
allurectl watch -- node tests/runner-e2e/chat-completion-testops-cli.mjs bootstrap
node tests/runner-e2e/chat-completion-testops-cli.mjs bootstrap-context
- name: Validate case, plan, job, launch and same-run source before provider access
id: selection
if: needs.authorize.outputs.operation == 'evaluate'
env:
GH_TOKEN: ${{ github.token }}
TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
ALLURE_ENDPOINT: https://paperclip.testops.cloud
ALLURE_PROJECT_ID: '1'
ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
ALLURE_RESULTS: ${{ runner.temp }}/chat-completion-selection-results
CHAT_FULL_CATALOG_TESTOPS_BINDING: ${{ vars.CHAT_FULL_CATALOG_TESTOPS_BINDING }}
run: |
mkdir -m 700 "$ALLURE_RESULTS"
allurectl watch -- node tests/runner-e2e/chat-completion-testops-cli.mjs selection
- name: Retain token-free registration receipts
if: always() && needs.authorize.outputs.operation != 'evaluate'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: chat-completion-testops-setup-${{ github.run_id }}-1
path: |
${{ runner.temp }}/chat-completion-testops-bootstrap/
retention-days: 30
if-no-files-found: error
- name: Retain the exact authenticated selection
id: selection_upload
if: steps.selection.outcome == 'success'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: chat-completion-testops-selection-${{ github.run_id }}-1
path: testops-pilot-selection.json
retention-days: 30
if-no-files-found: error
build:
name: Build the immutable catalog source without service or model credentials
needs: [authorize, select]
if: needs.authorize.outputs.operation == 'evaluate'
runs-on: ${{ needs.authorize.outputs.runner }}
timeout-minutes: 45
outputs:
artifact_id: ${{ steps.bundle.outputs.artifact-id }}
steps:
- *checkout
- *node
- *guard
- &pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: 'false'
NPM_CONFIG_FUND: 'false'
NPM_CONFIG_UPDATE_NOTIFIER: 'false'
with:
version: 9.15.4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ needs.authorize.outputs.target_sha }}
persist-credentials: false
- name: Reject target-owned trusted helper paths
id: target_boundary
run: test -z "$(git ls-tree --name-only HEAD -- .testops-trusted)"
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ github.sha }}
path: .testops-trusted
persist-credentials: false
- name: Resolve the fixed candidate graph with lifecycle scripts disabled
run: |
pnpm install --lockfile-only --ignore-scripts
pnpm install --frozen-lockfile --ignore-scripts
- name: Build existing TypeScript and native Runner outputs
run: |
pnpm --filter @paperclipai/paperclip-runner build:typescript
- name: Exercise the real workflow and pinned source host contract before model access
run: node --import ./cli/node_modules/tsx/dist/loader.mjs .testops-trusted/tests/runner-e2e/chat-completion-testops-host-control.mjs
- name: Build native Runner outputs
run: pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
- name: Package the resolved lock and exact source outputs
env:
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
run: |
set -euo pipefail
test "$(git rev-parse HEAD)" = "$TARGET_SHA"
tar -czf chat-completion-build.tar.gz pnpm-lock.yaml packages/paperclip-eval-kernel/dist packages/paperclip-runner/dist packages/paperclip-runner/runner/target/debug/paperclip-runnerd packages/paperclip-runner/runner/target/debug/conformance-tracer packages/paperclip-runner/runner/target/debug/fake-harness packages/paperclip-runner/runner/target/debug/fake-codex-app-server packages/paperclip-runner/runner/target/debug/fake-acpx-sidecar
sha256sum chat-completion-build.tar.gz > chat-completion-build.tar.gz.sha256
- name: Retain credential-free immutable outputs
id: bundle
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: chat-completion-build-${{ github.run_id }}-1
path: |
chat-completion-build.tar.gz
chat-completion-build.tar.gz.sha256
retention-days: 1
if-no-files-found: error
chat:
name: ${{ matrix.executionId }}
needs: [authorize, select, build]
if: needs.authorize.outputs.operation == 'evaluate' && needs.select.outputs.native_count != '0'
strategy:
fail-fast: false
max-parallel: ${{ fromJSON(needs.select.outputs.parallelism) }}
matrix: ${{ fromJSON(needs.select.outputs.native_matrix) }}
runs-on: ${{ needs.authorize.outputs.runner }}
timeout-minutes: ${{ matrix.timeout_minutes }}
environment: runner-e2e-private-paid
permissions:
contents: read
actions: read
id-token: write
env: &retention
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
MANUAL_PRIVATE_IDS: ${{ matrix.private_ids }}
MANUAL_PRIVATE_APPROVALS_JSON: ${{ matrix.private_policy }}
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
steps: &catalog_worker_steps
- *checkout
- *node
- *guard
- *pnpm
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ needs.authorize.outputs.target_sha }}
persist-credentials: false
- name: Reject target-owned trusted helper paths
id: target_boundary
run: test -z "$(git ls-tree --name-only HEAD -- .testops-trusted)"
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ github.sha }}
path: .testops-trusted
persist-credentials: false
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
artifact-ids: ${{ needs.build.outputs.artifact_id }}
path: chat-completion-build
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
artifact-ids: ${{ needs.select.outputs.artifact_id }}
path: chat-completion-testops-selection
- name: Verify source-bound build and authenticated selection before credentials
env:
GH_TOKEN: ${{ github.token }}
TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
EXPECTED_SELECTION_SHA256: ${{ needs.select.outputs.selection_sha256 }}
ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
run: |
set -euo pipefail
test "$(git -C .testops-trusted rev-parse HEAD)" = "$GITHUB_SHA"
node .testops-trusted/tests/runner-e2e/chat-completion-testops-cli.mjs verify-selection chat-completion-testops-selection/testops-pilot-selection.json
(cd chat-completion-build && sha256sum -c chat-completion-build.tar.gz.sha256)
tar -xzf chat-completion-build/chat-completion-build.tar.gz -C "$GITHUB_WORKSPACE"
pnpm install --frozen-lockfile --ignore-scripts
(cd .testops-trusted && node tests/runner-e2e/prepare-report-lock.mjs && pnpm install --frozen-lockfile --ignore-scripts --store-dir "$RUNNER_TEMP/chat-completion-retention-store")
- name: Qualify the pinned provider runtime and preinstalled browser
run: |
node packages/paperclip-runner/scripts/materialize-opencode-binary.mjs
node -e 'const fs=require("node:fs");fs.chmodSync(process.execPath,fs.statSync(process.execPath).mode&~0o022);if(fs.statSync(process.execPath).mode&0o022)throw Error("Writable provider Node")'
google-chrome --version
pnpm exec playwright install ffmpeg
- name: Bind exact private evidence under the authenticated TestOps authority
id: private
env:
TARGET_ROOT: ${{ github.workspace }}
SELECTED_EXECUTION_ID: ${{ matrix.executionId }}
run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts prepare
- name: Obtain only this cell's exact-key storage session
id: cell_credentials
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
with:
role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }}
aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
inline-session-policy: ${{ steps.private.outputs.session_policy }}
output-env-credentials: false
output-credentials: true
disable-retry: true
action-timeout-s: 60
- name: Verify private custody before delivering the model key
env: &cell_session
SELECTED_EXECUTION_ID: ${{ matrix.executionId }}
PRIVATE_EVIDENCE_ROOT: ${{ steps.private.outputs.root }}
AWS_ACCESS_KEY_ID: ${{ steps.cell_credentials.outputs.aws-access-key-id }}
AWS_SECRET_ACCESS_KEY: ${{ steps.cell_credentials.outputs.aws-secret-access-key }}
AWS_SESSION_TOKEN: ${{ steps.cell_credentials.outputs.aws-session-token }}
AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts verify
- name: Execute only the selected original case with zero retries
env:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
ANTHROPIC_API_KEY: ${{ contains(matrix.required_credentials, 'ANTHROPIC_API_KEY') && secrets.ANTHROPIC_API_KEY || '' }}
PAPERCLIP_CHAT_COMPLETION_TESTOPS: v1
ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
PAPERCLIP_RUNNER_E2E_PRIVATE_PREFLIGHT: ${{ steps.private.outputs.root }}/preflight.json
PRIVATE_EVIDENCE_ROOT: ${{ steps.private.outputs.root }}
PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-1-${{ matrix.executionId }}
PAPERCLIP_E2E_ACCOUNTING_CAMPAIGN_ID: gha-${{ github.run_id }}-1
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: refs/heads/codex/chat-catalog-master-integration
PAPERCLIP_PLAYWRIGHT_CHANNEL: chrome
run: pnpm test:e2e:runner -- --id "${{ matrix.executionId }}" --max-parallel 1 --max-automatic-retries 0 > "$PRIVATE_EVIDENCE_ROOT/launcher-output.log" 2>&1
- name: Retain original evidence after success or failure
if: always() && steps.cell_credentials.outcome == 'success'
env: *cell_session
run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts upload-cell
chat_external:
name: ${{ matrix.executionId }}
needs: [authorize, select, build]
if: needs.authorize.outputs.operation == 'evaluate' && needs.select.outputs.external_count != '0'
strategy:
fail-fast: false
max-parallel: 1
matrix: ${{ fromJSON(needs.select.outputs.external_matrix) }}
runs-on: ${{ needs.authorize.outputs.runner }}
timeout-minutes: ${{ matrix.timeout_minutes }}
environment: runner-e2e-private-paid
permissions:
contents: read
actions: read
id-token: write
env: *retention
steps: *catalog_worker_steps
report:
name: Retain private originals and build the exact safe TestOps projection
needs: [authorize, select, build, chat, chat_external]
if: always() && !cancelled() && needs.authorize.outputs.operation == 'evaluate' && needs.select.result == 'success' && needs.build.result == 'success' && (needs.chat.result == 'success' || needs.chat.result == 'failure' || needs.chat_external.result == 'success' || needs.chat_external.result == 'failure')
runs-on: ubuntu-latest
timeout-minutes: 20
environment: runner-e2e-private-history
permissions:
contents: read
actions: read
id-token: write
env: *retention
strategy:
fail-fast: false
max-parallel: 4
matrix: ${{ fromJSON(needs.select.outputs.matrix) }}
steps:
- *checkout
- *node
- *guard
- *pnpm
- name: Install trusted reporting dependencies with lifecycle scripts disabled
run: |
node tests/runner-e2e/prepare-report-lock.mjs
pnpm install --frozen-lockfile --ignore-scripts --store-dir "$RUNNER_TEMP/chat-completion-report-store"
- name: Prepare unchanged exact-key report and cell policies
id: session
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts report-session
- name: Obtain exact first-cell reader
id: reader_0
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
with:
role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }}
aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
inline-session-policy: ${{ steps.session.outputs.read_policy_0 }}
output-env-credentials: false
output-credentials: true
disable-retry: true
action-timeout-s: 60
- name: Download the first exact original
env:
PRIVATE_REPORT_ROOT: ${{ github.workspace }}
SELECTED_CELL_INDEX: '0'
AWS_ACCESS_KEY_ID: ${{ steps.reader_0.outputs.aws-access-key-id }}
AWS_SECRET_ACCESS_KEY: ${{ steps.reader_0.outputs.aws-secret-access-key }}
AWS_SESSION_TOKEN: ${{ steps.reader_0.outputs.aws-session-token }}
AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts download-cell
- name: Verify every selected original and its completeness receipt
env:
PRIVATE_REPORT_ROOT: ${{ github.workspace }}
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts verify-downloads
- name: Build the existing report while preserving failed source grades
id: normalize
continue-on-error: true
env:
PAPERCLIP_RUNNER_E2E_REPORT_ROOT: ${{ github.workspace }}/selected-runner-e2e
PAPERCLIP_RUNNER_E2E_REPORT_OUT: ${{ github.workspace }}/runner-e2e-merged-report/normalized
PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: ${{ matrix.private_ids }}
PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-1
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: refs/heads/codex/chat-catalog-master-integration
run: |
mkdir -p runner-e2e-merged-report
pnpm test:e2e:runner:report > runner-e2e-merged-report/report.log 2>&1
- name: Export only exact reviewed synthetic fixture results
id: export
if: always() && steps.reader_0.outcome == 'success'
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/chat-completion-testops-export.ts
- name: Require a complete safe projection even after a failed normalization
if: always()
env:
EXPORT_OUTCOME: ${{ steps.export.outcome }}
run: test "$EXPORT_OUTCOME" = success
- name: Retain the metadata-only TestOps projection
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: chat-catalog-testops-cell-${{ github.run_id }}-${{ matrix.key }}
path: runner-e2e-testops/
retention-days: 30
if-no-files-found: error
aggregate:
name: Validate and merge every selected metadata-only projection
needs: [authorize, select, report]
if: always() && !cancelled() && needs.authorize.outputs.operation == 'evaluate' && (needs.report.result == 'success' || needs.report.result == 'failure')
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
artifact_id: ${{ steps.projection.outputs.artifact-id }}
steps:
- *checkout
- *node
- *guard
- *pnpm
- name: Install trusted reporting dependencies without lifecycle scripts
run: |
node tests/runner-e2e/prepare-report-lock.mjs
pnpm install --frozen-lockfile --ignore-scripts
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
artifact-ids: ${{ needs.select.outputs.artifact_id }}
path: chat-completion-testops-selection
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: chat-catalog-testops-cell-${{ github.run_id }}-*
path: catalog-projections
- name: Authenticate selection and validate all singleton exports before aggregation
env:
GH_TOKEN: ${{ github.token }}
TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
EXPECTED_SELECTION_SHA256: ${{ needs.select.outputs.selection_sha256 }}
TESTOPS_CATALOG_SELECTION: chat-completion-testops-selection/testops-pilot-selection.json
run: |
node tests/runner-e2e/chat-completion-testops-cli.mjs verify-selection "$TESTOPS_CATALOG_SELECTION"
node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/chat-full-catalog-testops-merge.ts
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
id: projection
with:
name: chat-catalog-testops-projection-${{ github.run_id }}-1
path: runner-e2e-testops/
retention-days: 30
if-no-files-found: error
upload:
name: Record and independently verify each TestOps result
needs: [authorize, select, chat, chat_external, report, aggregate]
if: always() && !cancelled() && needs.authorize.outputs.operation == 'evaluate' && needs.aggregate.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 10
environment: runner-e2e-testops-reporting
steps:
- *checkout
- *node
- *guard
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
artifact-ids: ${{ needs.aggregate.outputs.artifact_id }}
path: runner-e2e-testops
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
artifact-ids: ${{ needs.select.outputs.artifact_id }}
path: chat-completion-testops-selection
- *allure
- name: Upload original outcomes, checks and private-evidence receipts
id: allure_upload
env: &upload_env
GH_TOKEN: ${{ github.token }}
TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
ALLURE_ENDPOINT: https://paperclip.testops.cloud
ALLURE_PROJECT_ID: '1'
ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
EXPECTED_SELECTION_SHA256: ${{ needs.select.outputs.selection_sha256 }}
ALLURE_RESULTS: ${{ runner.temp }}/chat-completion-allure-results
run: allurectl watch -- node tests/runner-e2e/chat-completion-testops-cli.mjs upload runner-e2e-testops
- name: Verify actual selected results and complete result-owned attachments
id: verification
if: always() && !cancelled() && (steps.allure_upload.outcome == 'success' || steps.allure_upload.outcome == 'failure')
env: *upload_env
run: node tests/runner-e2e/chat-completion-testops-cli.mjs verify-upload runner-e2e-testops
- name: Retain closed TestOps verification receipts
if: always() && !cancelled() && (steps.verification.outcome == 'success' || steps.verification.outcome == 'failure')
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: chat-completion-testops-verification-${{ github.run_id }}-1
path: |
${{ runner.temp }}/chat-completion-testops-verify-upload/
${{ runner.temp }}/chat-completion-testops-upload/exported.json
retention-days: 30
if-no-files-found: error
doc/evals.md
+6 -1
View File
@@ -59,7 +59,12 @@ using ordinary production instructions, real browser messages and independent
state/provenance checks. A separately selected native OpenCode/DeepSeek1017
backlog cell measures TestOps integration acceptance with the same oracle and
original bounds. Its source registration is not an AWS or TestOps pass. The
broader YAML catalog remains planning inventory.
[full-catalog TestOps lane](../tests/runner-e2e/CHAT-FEATURES.md#full-catalog-in-testops)
maps all 244 YAML scenarios into a separate candidate-scoped plan. Its app and
fixtures use a preserved immutable catalog revision; trusted workflow helpers run
from master. Registration, known gaps and private prerequisites remain distinct
from measured results. This lane does not qualify the current master app or reset
earlier campaign accounting.
The explicit-only [Daytona restart continuity journey](../tests/runner-e2e/README.md)
extends the existing native restart oracle to a 300-second controller outage
scripts/__tests__/release-verify-workflow.test.mjs
+26 -5
View File
@@ -581,6 +581,7 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
readWorkflow("runner-protocol-live-evals.yml"),
readWorkflow("chat-feature-testops.yml"),
readWorkflow("chat-completion-testops.yml"),
readWorkflow("chat-catalog-testops.yml"),
];
for (const workflow of actionPinWorkflows) {
@@ -622,6 +623,7 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
"runner-protocol-live-evals.yml",
"chat-feature-testops.yml",
"chat-completion-testops.yml",
"chat-catalog-testops.yml",
];
const paidWorkflowNameSet = new Set(paidWorkflowNames);
const providerSecretReference =
@@ -654,26 +656,30 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
(block) => [...block.matchAll(providerSecretReference)].length > 0,
);
assert.ok(providerJobs.length > 0, `${name} needs a provider-secret job`);
if (name === "chat-completion-testops.yml") {
if (name === "chat-completion-testops.yml" || name === "chat-catalog-testops.yml") {
assert.equal(providerJobs.length, 1);
const expectedKeys = name === "chat-catalog-testops.yml" ? 2 : 1;
const assertBoundary = (block) => {
assert.match(block, /^ chat:\n/);
assert.match(block, /\n needs: \[authorize, select, build\]\n/);
assert.match(block, /\n if: needs\.authorize\.outputs\.operation == 'evaluate'\n/);
if (expectedKeys === 2) assert.ok(block.includes("if: needs.authorize.outputs.operation == 'evaluate' && needs.select.outputs.native_count != '0'"));
else assert.match(block, /\n if: needs\.authorize\.outputs\.operation == 'evaluate'\n/);
assert.match(block, /\n environment: runner-e2e-private-paid\n/);
assert.match(block, /\n - \*checkout\n - \*node\n - \*guard\n/);
assert.match(block, /chat-completion-testops-cli\.mjs verify-selection/);
assert.match(block, /OPENROUTER_API_KEY: \$\{\{ secrets\.OPENROUTER_API_KEY \}\}/);
assert.match(block, /--id "\$\{\{ matrix\.executionId \}\}" --max-parallel 1 --max-automatic-retries 0/);
assert.match(block, /PAPERCLIP_CHAT_COMPLETION_TESTOPS: v1/);
assert.match(block, /max-parallel: 1/);
if (expectedKeys === 2) assert.ok(block.includes('max-parallel: ${{ fromJSON(needs.select.outputs.parallelism) }}'));
else assert.match(block, /max-parallel: 1/);
assert.doesNotMatch(block, /ALLURE_TOKEN/);
assert.equal([...block.matchAll(providerSecretReference)].length, 1);
assert.equal([...block.matchAll(providerSecretReference)].length, expectedKeys);
if (expectedKeys === 2) assert.ok(block.includes("ANTHROPIC_API_KEY: ${{ contains(matrix.required_credentials, 'ANTHROPIC_API_KEY') && secrets.ANTHROPIC_API_KEY || '' }}"));
assert.ok(block.indexOf(" - *guard") < block.search(providerSecretReference));
assert.ok(block.indexOf("chat-completion-testops-cli.mjs verify-selection") < block.search(providerSecretReference));
};
const block = providerJobs[0]; assertBoundary(block);
assert.equal([...workflow.matchAll(providerSecretReference)].length, 1);
assert.equal([...workflow.matchAll(providerSecretReference)].length, expectedKeys);
for (const mutant of [
block.replace("needs: [authorize, select, build]", "needs: [authorize, build]"),
block.replace("environment: runner-e2e-private-paid", "environment: unprotected"),
@@ -682,6 +688,21 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
block.replace('--id "${{ matrix.executionId }}"', "--all"),
block.replace("PAPERCLIP_CHAT_COMPLETION_TESTOPS: v1", "PAPERCLIP_CHAT_COMPLETION_TESTOPS: unknown"),
]) assert.throws(() => assertBoundary(mutant));
if (expectedKeys === 2) assert.throws(() => assertBoundary(block.replace("contains(matrix.required_credentials, 'ANTHROPIC_API_KEY')", "true")));
if (expectedKeys === 2) {
const external = jobBlocks.find(value => value.startsWith(' chat_external:\n'));
const assertExternal = value => {
assert.match(value, /needs: \[authorize, select, build\]/);
assert.ok(value.includes("if: needs.authorize.outputs.operation == 'evaluate' && needs.select.outputs.external_count != '0'"));
assert.match(value, /environment: runner-e2e-private-paid/);
assert.match(value, /max-parallel: 1/);
assert.ok(value.includes('matrix: ${{ fromJSON(needs.select.outputs.external_matrix) }}'));
assert.match(value, /env: \*retention/);
assert.match(value, /steps: \*catalog_worker_steps/);
};
assertExternal(external);
for (const mutant of [external.replace('*catalog_worker_steps', '*unguarded'), external.replace('max-parallel: 1', 'max-parallel: 8'), external.replace('environment: runner-e2e-private-paid', 'environment: unprotected')]) assert.throws(() => assertExternal(mutant));
}
continue;
}
if (name === "chat-feature-testops.yml") {
tests/runner-e2e/CHAT-FEATURES.md
+73
View File
@@ -241,6 +241,79 @@ metadata. It rejects arbitrary source/model/budget selectors, branch workflows,
replays and foreign cases. All secret-bearing jobs recheck actual GitHub run and
actor identities. The existing one-case backlog plan remains independent.
## Full catalog in TestOps
The [Chat catalog TestOps workflow](../../.github/workflows/chat-catalog-testops.yml)
registers all 244 scenarios from the 35 YAML areas in a separate
**Chat feature catalog — OpenCode 1017 (candidate)** plan. Its source-derived
manifest is `chat-full-catalog-testops-manifest.json`. The existing backlog plan
and three-case completion pilot keep their own jobs, selections and history.
Legacy aliases reuse existing TestOps case identities without renaming them.
The catalog contains 236 Agent Chat routes, five browser boundaries, one external
assistant route and two Slack routes. An execution mapping is not a passing
measurement. Five implementations declare incomplete coverage. Six routes do not
produce the generic final PNG required by the unchanged evidence owner. Eighteen
declare additional credentials and 37 declare external inputs. These groups
overlap. Exact requirements and limitations appear in each case's preconditions.
A declared credential does not establish whether its runtime reference exists.
This lane runs the app and fixtures from candidate
`3b66c3c144360ddf32714a0742fd774b8ceafb16`. Its app and domain definitions are
unchanged from `de0941835cc8f0d01c982574ee258e110babdbeb`; the successor adds only
finite TestOps host admission. Trusted selection, evidence transfer and publishing
use the workflow's immutable master revision. Each result records both revisions.
A pass qualifies that candidate measurement, not the current master app. Other
lanes sharing a case retain their own sources. This integration does not merge
the broad catalog branch or its core fixes, reset earlier campaign ordinals, or
change the previous 196-of-244 acceptance gate.
Native execution uses OpenCode and
`openrouter/deepseek/deepseek-v4.1-flash-1017` on the existing AWS fleet. Each case
keeps its original query, oracle, run counts, clocks, budgets and zero automatic
retries. Browser boundaries and the external assistant retain their original
surfaces and may have zero native work runs. The external companion keeps its own
declared provider requirement. Setup inference and unpriced usage are not free.
Initial setup uses the existing reporting environment:
1. Dispatch reviewed master with `operation=bootstrap`. It reconciles the exact
244 cases and the dedicated plan, then registers and binds its own GitHub job.
Missing cases use one marked POST each. Existing cases retain their identities
and history. Alias conflicts, foreign selections and uncertain writes stop
setup. It makes no model call or synthetic test result.
2. Read the registration receipt. Store its compact `binding` JSON as
`CHAT_FULL_CATALOG_TESTOPS_BINDING`. It contains observed plan, job, integration
and case IDs plus registered full names, and fits one GitHub environment
variable. The job must retain its observed single `Branch/defaultValue`
definition for `refs/heads/master`.
3. Use normal TestOps **Run** for the full plan or a selected subset. The service's
actual expected case list controls execution. Do not manually dispatch another
evaluation after an uncertain request. Initial route verification uses a small
selection and does not qualify all 244 cases.
Each worker executes one case. Selections of up to three cases run serially.
Larger selections allow eight independent cases and one external-input case at
once. External-input cases remain serial because they can share QA resources.
A full run can take hours. Its elapsed time depends on setup and fleet capacity;
the declared case clocks are not a completion-time guarantee. Each worker and
report reader has a singleton private-evidence approval and exact-key AWS
session; the existing eight-cell
authority limit is unchanged. Reports wait for all selected workers. A
credential-free job then validates and merges their metadata projections. The
protected reporter uploads original outcomes and verifies every selected result,
including paginated selections over 100 cases. Failed originals and failed
cleanup remain failed beside passed results. Missing originals cannot become
fabricated measurements or a full-selection pass.
Raw evidence and final PNGs remain in protected storage. Public GitHub artifacts
and TestOps attachments contain only allowlisted provenance, checks, accounting
and private-evidence receipts. Raw conversations, screenshots, provider sessions
and credentials do not enter the public artifact path. Additional credentials
reach only cases that declare them. External inputs use the existing selected-case
private-input owner, disposable-resource checks and oracle commitments.
Unconfigured inputs remain setup blockers.
## TestOps-backed AWS backlog cell
The dedicated `Chat feature TestOps` workflow accepts only the shipped default
tests/runner-e2e/chat-completion-testops-api.mjs
+74 -32
View File
@@ -1,5 +1,7 @@
import { testopsClient, endpoint } from './chat-feature-testops-api.mjs';
import { cases, planName, targetSha, workflow, positive, reject, validateSelection } from './chat-completion-testops-contract.mjs';
import { cases, planName, positive, reject, validateSelection, catalogDefinition } from './chat-completion-testops-contract.mjs';
import { reconcileFullCatalog } from './chat-full-catalog-testops-reconcile.mjs';
export { testopsClient, endpoint };
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
@@ -8,37 +10,61 @@ const pageRows = value => {
|| value.totalPages < 0 || value.totalPages > 1) reject('incomplete_page');
return value.content;
};
export function bindingFromEnv(env) {
async function resultRows(get, launchId, maximum, full) {
const rows = []; let pages, total;
for (let index = 0; index < (full ? 3 : 1); index++) {
const page = await get(`/api/testresult?${new URLSearchParams({ launchId: String(launchId), page: String(index), size: '100' })}`);
if (!full) {
const content = pageRows(page);
if (!Number.isSafeInteger(page.totalElements) || page.totalElements !== content.length) reject('result_page');
return content;
}
if (!Array.isArray(page?.content) || page.content.length > 100 || page.number !== index
|| !Number.isSafeInteger(page.totalPages) || page.totalPages < 0 || page.totalPages > 3
|| !Number.isSafeInteger(page.totalElements) || page.totalElements < 0 || page.totalElements > maximum) reject('result_page');
if (pages !== undefined && (pages !== page.totalPages || total !== page.totalElements)) return null;
pages = page.totalPages; total = page.totalElements; rows.push(...page.content);
if (index + 1 >= pages) break;
}
if (rows.length !== total) reject('result_page');
return rows;
}
export function bindingFromEnv(env, context = {}) {
const { cases, bindingKey } = catalogDefinition(context);
let value;
try { value = JSON.parse(env.CHAT_COMPLETION_TESTOPS_BINDING); } catch { reject('binding_missing'); }
try { value = JSON.parse(env[bindingKey]); } catch { reject('binding_missing'); }
if (!positive(value.planId) || !positive(value.jobId) || !positive(value.integrationId)
|| !Array.isArray(value.cases) || value.cases.length !== cases.length) reject('binding');
const selected = cases.map(row => {
const matched = value.cases.filter(v => v.key === row.key);
if (matched.length !== 1 || !positive(matched[0].caseId)) reject('binding');
return { ...row, caseId: matched[0].caseId };
const registeredFullName = matched[0].registeredFullName ?? row.fullName;
if (![row.fullName, ...(row.aliases ?? [])].includes(registeredFullName)) reject('binding');
return { ...row, caseId: matched[0].caseId, ...(matched[0].registeredFullName === undefined ? {} : { registeredFullName }) };
});
if (new Set(selected.map(row => row.caseId)).size !== cases.length) reject('binding');
return { planId: value.planId, jobId: value.jobId, integrationId: value.integrationId, cases: selected };
}
export function isWorkflowUrl(url, workflowId) {
export function isWorkflowUrl(url, workflowId, context = {}) {
const { workflow } = catalogDefinition(context);
return url === `https://github.com/paperclipai/paperclip/actions/workflows/${workflow.split('/').at(-1)}`
|| positive(workflowId) && url === `https://github.com/paperclipai/paperclip/actions/workflows/${workflowId}`
|| url === 'https://github.com/paperclipai/paperclip/actions?query=workflow%3A%22Chat+completion+TestOps%22';
|| url === (context.catalog === 'full' ? 'https://github.com/paperclipai/paperclip/actions?query=workflow%3A%22Chat+catalog+TestOps%22' : 'https://github.com/paperclipai/paperclip/actions?query=workflow%3A%22Chat+completion+TestOps%22');
}
function assertCase(actual, expected) {
if (actual.id !== expected.caseId || actual.projectId !== 1 || actual.fullName !== expected.fullName
if (actual.id !== expected.caseId || actual.projectId !== 1 || actual.fullName !== (expected.registeredFullName ?? expected.fullName)
|| actual.automated !== true || actual.deleted === true) reject('case_mismatch');
}
function assertPlan(actual, binding) {
function assertPlan(actual, binding, context = {}) {
const { planName } = catalogDefinition(context);
const selected = binding.cases.map(row => row.caseId).sort((a, b) => a - b);
if (actual.id !== binding.planId || actual.projectId !== 1 || actual.name !== planName || actual.testCasesCount !== selected.length
|| !Array.isArray(actual.treeSelection?.leafsInclude) || !same([...actual.treeSelection.leafsInclude].sort((a, b) => a - b), selected)
|| actual.treeSelection.inverted === true || (actual.treeSelection.groupsInclude?.length ?? 0) !== 0
|| (actual.treeSelection.leafsExclude?.length ?? 0) !== 0) reject('plan_mismatch');
}
function assertJob(job, binding, workflowId) {
if (job.id !== binding.jobId || job.type !== 'github' || !isWorkflowUrl(job.url, workflowId)) reject('job_mismatch');
function assertJob(job, binding, workflowId, context = {}) {
if (job.id !== binding.jobId || job.type !== 'github' || !isWorkflowUrl(job.url, workflowId, context)) reject('job_mismatch');
}
function assertBranchDefinition(job) {
// Observed job-definition DTO, not the id/value plan-override DTO.
@@ -48,12 +74,13 @@ function assertBranchDefinition(job) {
}
export async function validatePilotSelection(api, env, context, workflowId) {
const binding = bindingFromEnv(env);
const { cases, targetSha, planName } = catalogDefinition(context);
const binding = bindingFromEnv(env, context);
for (const row of binding.cases) assertCase(await api(`/api/testcase/${row.caseId}`), row);
assertPlan(await api(`/api/testplan/${binding.planId}`), binding);
assertPlan(await api(`/api/testplan/${binding.planId}`), binding, context);
const jobs = await api(`/api/testplan/${binding.planId}/job`);
if (!Array.isArray(jobs) || jobs.length !== 1 || jobs[0].job?.id !== binding.jobId || jobs[0].testCasesCount !== cases.length) reject('plan_job_mismatch');
assertJob(await api(`/api/job/${binding.jobId}`), binding, workflowId);
assertJob(await api(`/api/job/${binding.jobId}`), binding, workflowId, context);
const run = await api(`/api/jobrun/${env.ALLURE_JOB_RUN_ID}`);
if (run.id !== Number(env.ALLURE_JOB_RUN_ID) || run.job?.id !== binding.jobId || run.job?.type !== 'github'
|| !['scheduled', 'in_progress'].includes(run.stage) || !positive(run.launchId)
@@ -63,7 +90,7 @@ export async function validatePilotSelection(api, env, context, workflowId) {
const planned = await api(`/api/jobrun/${run.id}/plan?expected=true`);
if (!Array.isArray(planned) || !planned.length || planned.length > cases.length || new Set(planned.map(row => row.id)).size !== planned.length) reject('selection');
const selected = planned.map(row => {
const found = binding.cases.find(value => value.caseId === row.id && value.fullName === row.selector);
const found = binding.cases.find(value => value.caseId === row.id && (value.registeredFullName ?? value.fullName) === row.selector);
if (!found) reject('foreign_selection');
return found;
});
@@ -102,10 +129,11 @@ export async function inspectPilotCatalog(api) {
return { cases: [...found.values()], plans: plans.map(row => ({ id: row.id, name: row.name })) };
}
export async function bootstrapPilotPlan(api, integrationId, markOnce) {
export async function bootstrapPilotPlan(api, integrationId, markOnce, context = {}) {
if (!positive(integrationId)) reject('integration');
const integration = await api(`/api/integration/${integrationId}/project/1`);
if (integration.id !== integrationId || integration.projectId !== 1 || integration.info?.type !== 'github' || integration.disabled !== false) reject('integration');
if (context.catalog === 'full') return { ...await reconcileFullCatalog(api, markOnce), integrationId };
const inspected = await inspectPilotCatalog(api);
if (inspected.plans.length && inspected.cases.length !== cases.length) reject('plan_conflict');
const selected = [...inspected.cases];
@@ -131,14 +159,15 @@ export async function bootstrapPilotPlan(api, integrationId, markOnce) {
}
export async function completePilotRegistration(api, prepared, official, context, workflowId, markOnce, observe = async () => {}) {
const { cases, targetSha, planName } = catalogDefinition(context);
const run = await api(`/api/jobrun/${official.jobRunId}`);
if (run.id !== Number(official.jobRunId) || run.job?.type !== 'github' || !positive(run.job?.id)
|| run.launchId !== official.launchId || run.url !== `https://github.com/paperclipai/paperclip/actions/runs/${context.runId}`) reject('registration');
const launch = await api(`/api/launch/${official.launchId}`);
if (launch.id !== official.launchId || launch.projectId !== 1) reject('registration');
const binding = { ...prepared, jobId: run.job.id };
assertPlan(await api(`/api/testplan/${binding.planId}`), binding);
const job = await api(`/api/job/${binding.jobId}`); assertJob(job, binding, workflowId);
assertPlan(await api(`/api/testplan/${binding.planId}`), binding, context);
const job = await api(`/api/job/${binding.jobId}`); assertJob(job, binding, workflowId, context);
await observe({ schema: 'paperclip.chat-completion-testops-registration-observed/v1', ...context,
projectId: 1, jobId: job.id, launchId: official.launchId, allureJobRunId: official.jobRunId,
parameterCount: Array.isArray(job.parameters) ? job.parameters.length : null,
@@ -154,7 +183,7 @@ export async function completePilotRegistration(api, prepared, official, context
// PATCH only the documented enable/integration fields. Preserve the opaque
// definition exactly; no guessed parameter-definition write codec is used.
await api(`/api/job/${binding.jobId}`, 'PATCH', { integrationId: binding.integrationId, canRun: true });
const patched = await api(`/api/job/${binding.jobId}`); assertJob(patched, binding, workflowId); assertBranchDefinition(patched);
const patched = await api(`/api/job/${binding.jobId}`); assertJob(patched, binding, workflowId, context); assertBranchDefinition(patched);
if (!same(patched.parameters, job.parameters) || (Object.hasOwn(patched, 'canRun') && patched.canRun !== true)) reject('job_parameters');
if (!assignments.length) {
await markOnce('assign_plan');
@@ -163,19 +192,23 @@ export async function completePilotRegistration(api, prepared, official, context
}
const bound = await api(`/api/testplan/${binding.planId}/job`);
if (!Array.isArray(bound) || bound.length !== 1 || bound[0].job?.id !== binding.jobId || bound[0].testCasesCount !== cases.length) reject('plan_job_mismatch');
return { schema: 'paperclip.chat-completion-testops-registration/v1', ...context, binding,
const persistedBinding = context.catalog === 'full' ? { planId: binding.planId, jobId: binding.jobId,
integrationId: binding.integrationId, cases: binding.cases.map(row => ({ key: row.key, caseId: row.caseId,
registeredFullName: row.registeredFullName ?? row.fullName })) } : binding;
return { schema: 'paperclip.chat-completion-testops-registration/v1', ...context, binding: persistedBinding,
parametersMode: 'job-defaults', branchDefinition: { id: job.parameters[0].id, name: 'Branch', defaultValue: 'refs/heads/master' },
planUrl: `${endpoint}/testplan/${binding.planId}`, jobUrl: `${endpoint}/project/1/jobs`, qualification: 'registration_only', actualResult: null };
}
export async function launchPilotPlan(api, env, context, workflowId, markOnce) {
const binding = bindingFromEnv(env);
const { cases, targetSha, planName } = catalogDefinition(context);
const binding = bindingFromEnv(env, context);
if (!/^[1-9][0-9]{0,15}$/.test(env.CHAT_COMPLETION_TESTOPS_LAUNCH_RUN_NUMBER ?? '')
|| env.GITHUB_RUN_NUMBER !== env.CHAT_COMPLETION_TESTOPS_LAUNCH_RUN_NUMBER
|| !/^[a-f0-9]{32}$/.test(env.CHAT_COMPLETION_TESTOPS_LAUNCH_NONCE ?? '')) reject('launch_fence');
for (const row of binding.cases) assertCase(await api(`/api/testcase/${row.caseId}`), row);
assertPlan(await api(`/api/testplan/${binding.planId}`), binding);
const job = await api(`/api/job/${binding.jobId}`); assertJob(job, binding, workflowId);
assertPlan(await api(`/api/testplan/${binding.planId}`), binding, context);
const job = await api(`/api/job/${binding.jobId}`); assertJob(job, binding, workflowId, context);
assertBranchDefinition(job);
const assignments = await api(`/api/testplan/${binding.planId}/job`);
if (!Array.isArray(assignments) || assignments.length !== 1 || assignments[0].job?.id !== binding.jobId
@@ -192,6 +225,7 @@ export async function launchPilotPlan(api, env, context, workflowId, markOnce) {
}
export async function verifyPilotUpload(api, selection, context, exported, { now = Date.now, wait = ms => new Promise(resolve => setTimeout(resolve, ms)) } = {}) {
const { cases, targetSha, planName } = catalogDefinition(context);
const selected = validateSelection(selection, context, selection.allureJobRunId);
if (!Array.isArray(exported) || exported.length !== selected.length || new Set(exported.map(row => row.caseId)).size !== selected.length) reject('export_mismatch');
for (const row of exported) {
@@ -205,7 +239,9 @@ export async function verifyPilotUpload(api, selection, context, exported, { now
|| !Number.isSafeInteger(value.contentLength) || value.contentLength <= 0 || value.contentLength > 12 * 1024 * 1024
|| value.contentType !== 'application/json')) reject('export_mismatch');
}
const deadline = now() + 120_000;
const metadataReadAllowance = context.catalog === 'full' && selected.length > 3
? (2 + Math.ceil(selected.length / 100) + 2 * Math.ceil(selected.length / 4)) * 2000 : 0;
const deadline = now() + 120_000 + metadataReadAllowance;
const get = route => {
const remaining = deadline - now(); if (remaining < 1) reject('upload_unverified');
return api(route, 'GET', undefined, Math.min(30_000, remaining));
@@ -219,22 +255,25 @@ export async function verifyPilotUpload(api, selection, context, exported, { now
|| run.launchId !== selection.launchId || run.url !== runUrl) reject('jobrun_mismatch');
const launch = await get(`/api/launch/${selection.launchId}`);
if (launch.id !== selection.launchId || launch.projectId !== 1) reject('launch_mismatch');
const page = await get(`/api/testresult?${new URLSearchParams({ launchId: String(selection.launchId), page: '0', size: '100' })}`);
const listed = pageRows(page);
if (!Number.isSafeInteger(page.totalElements) || page.totalElements !== listed.length || listed.length > selected.length
const listed = await resultRows(get, selection.launchId, selected.length, context.catalog === 'full');
if (listed === null) {
if (poll < 11 && now() + 5000 < deadline) await wait(5000);
continue;
}
if (listed.length > selected.length
|| new Set(listed.map(row => row.id)).size !== listed.length || new Set(listed.map(row => row.testCaseId)).size !== listed.length) reject('result_page');
let ready = listed.length === selected.length; const verified = [];
for (const listedRow of listed) {
const verifyResult = async listedRow => {
const expected = selected.find(value => value.caseId === listedRow.testCaseId);
if (!expected || !positive(listedRow.id)) reject('foreign_result');
const wanted = exported.find(value => value.caseId === expected.caseId);
const result = await get(`/api/testresult/${listedRow.id}`);
if (result.id !== listedRow.id || result.projectId !== 1 || result.launchId !== selection.launchId || result.testCaseId !== expected.caseId
|| result.fullName !== expected.fullName || result.jobRun?.id !== Number(selection.allureJobRunId)
|| result.fullName !== (expected.registeredFullName ?? expected.fullName) || result.jobRun?.id !== Number(selection.allureJobRunId)
|| result.jobRun?.launchId !== selection.launchId || result.jobRun?.url !== runUrl
|| result.jobRun.job != null && (result.jobRun.job.id !== selection.jobId
|| result.jobRun.job.type != null && result.jobRun.job.type !== 'github'
|| result.jobRun.job.url != null && !isWorkflowUrl(result.jobRun.job.url, context.workflowId))
|| result.jobRun.job.url != null && !isWorkflowUrl(result.jobRun.job.url, context.workflowId, context))
|| !Array.isArray(result.links) || result.links.some(link => typeof link.url === 'string'
&& (link.url.startsWith('https://github.com/paperclipai/paperclip/actions/runs/') && link.url !== runUrl
|| link.url.startsWith('https://github.com/paperclipai/paperclip/commit/') && link.url !== `https://github.com/paperclipai/paperclip/commit/${targetSha}`))
@@ -247,9 +286,12 @@ export async function verifyPilotUpload(api, selection, context, exported, { now
// ready. Keep every final requirement; wait within the existing bound.
const complete = result.status === wanted.status && attachments.every(row => row.missed === false && row.fromTestCase === false)
&& same(signature(attachments), signature(wanted.attachments));
ready &&= complete;
verified.push({ executionId: expected.executionId, caseId: expected.caseId, resultId: result.id,
status: result.status, cleanup: wanted.cleanup, attachmentsVerified: complete, resultUrl: `${endpoint}/testresult/${result.id}` });
return { executionId: expected.executionId, caseId: expected.caseId, resultId: result.id,
status: result.status, cleanup: wanted.cleanup, attachmentsVerified: complete, resultUrl: `${endpoint}/testresult/${result.id}` };
};
for (let offset = 0; offset < listed.length; offset += context.catalog === 'full' ? 4 : 1) {
const batch = await Promise.all(listed.slice(offset, offset + (context.catalog === 'full' ? 4 : 1)).map(verifyResult));
ready &&= batch.every(row => row.attachmentsVerified); verified.push(...batch);
}
if (ready && now() < deadline) return { schema: 'paperclip.chat-completion-testops-verification/v1', ...context,
projectId: 1, planId: selection.planId, jobId: selection.jobId, launchId: selection.launchId,
tests/runner-e2e/chat-completion-testops-cli.mjs
+51 -11
View File
@@ -3,12 +3,41 @@ import { promisify } from 'node:util';
import { appendFile, lstat, mkdir, open, readFile, realpath, writeFile } from 'node:fs/promises';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
import { admitDispatch, targetSha, workflow, hash, reject, validateSelection, evidencePolicy } from './chat-completion-testops-contract.mjs';
import { admitDispatch, hash, reject, validateSelection, evidencePolicy, catalogDefinition } from './chat-completion-testops-contract.mjs';
import { testopsClient, bootstrapPilotPlan, completePilotRegistration, validatePilotSelection, launchPilotPlan, verifyPilotUpload } from './chat-completion-testops-api.mjs';
import { parseOfficialBootstrapContext } from './testops-chat-reporter.mjs';
import { exportPilotAllure, readPilotExport } from './chat-completion-testops-results.mjs';
const execute = promisify(execFile);
const fullCatalogFailureCodes = new Set([
'case_alias_conflict', 'case_identity', 'case_metadata', 'catalog_case', 'catalog_identity',
'catalog_incomplete', 'catalog_page', 'created_case', 'created_plan', 'manifest', 'manifest_alias',
'mutation_marker', 'plan_conflict', 'plan_identity', 'plan_page', 'plan_scope', 'reconciliation_changed',
].map(code => `chat_full_catalog_testops_${code}`));
function compactCase(row) {
return Object.fromEntries(['key', 'scenario', 'executionId', 'fullName', 'registeredFullName', 'caseId'].filter(key => row[key] !== undefined).map(key => [key, row[key]]));
}
export function executionMatrix(selection, context, jobRunId) {
const selected = validateSelection(selection, context, jobRunId);
const matrix = { include: selected.map((row, index) => ({ executionId: row.executionId, key: row.key,
...(context.catalog === 'full' ? { index, required_credentials: row.requiredCredentials,
external: row.externalInput != null, timeout_minutes: Math.ceil(row.attemptTimeoutMs / 60_000) + 15,
private_ids: JSON.stringify([row.executionId]),
private_policy: JSON.stringify(evidencePolicy({ ...selection, cases: [row] }, context, jobRunId)) } : {}) })) };
if (JSON.stringify(matrix).length * 2 > 768_000) reject('matrix_size');
return matrix;
}
export function executionPartition(selection, context, jobRunId) {
const matrix = executionMatrix(selection, context, jobRunId);
if (context.catalog !== 'full') reject('catalog');
const small = matrix.include.length <= 3;
const native = { include: matrix.include.filter(row => small || !row.external) };
const external = { include: matrix.include.filter(row => !small && row.external) };
const size = [matrix, native, external, selection.cases.map(row => row.executionId)]
.reduce((sum, value) => sum + JSON.stringify(value).length * 2, 0);
if (size > 1_000_000) reject('matrix_size');
return { matrix, native, external, parallelism: small ? '1' : '8' };
}
async function readOwned(file, root, limit) {
const absolute = path.resolve(file), base = await realpath(root);
if (!absolute.startsWith(`${base}${path.sep}`)) reject('file_path');
@@ -29,14 +58,14 @@ export async function actualContext(env, exec = execute) {
{ env, shell: false, timeout: 30_000, maxBuffer: 512_000, encoding: 'utf8' });
const run = JSON.parse(stdout), context = admitDispatch(env, event, run.triggering_actor?.id);
if (run.id !== Number(context.runId) || run.run_attempt !== 1 || run.head_sha !== context.workflowSha || run.head_branch !== 'master'
|| run.event !== 'workflow_dispatch' || run.path !== workflow || run.repository?.full_name !== 'paperclipai/paperclip' || run.repository.private !== false
|| run.event !== 'workflow_dispatch' || run.path !== catalogDefinition(context).workflow || run.repository?.full_name !== 'paperclipai/paperclip' || run.repository.private !== false
|| run.actor?.id !== context.actorId || run.run_number !== Number(env.GITHUB_RUN_NUMBER)
|| !Number.isSafeInteger(run.workflow_id) || run.workflow_id <= 0) reject('actual_run');
return { ...context, workflowId: run.workflow_id };
}
async function readSelection(file, env, context) {
if (!/^[a-f0-9]{64}$/.test(env.EXPECTED_SELECTION_SHA256 ?? '')) reject('selection_digest');
const bytes = await readOwned(file, env.GITHUB_WORKSPACE, 32_768);
const bytes = await readOwned(file, env.GITHUB_WORKSPACE, context.catalog === 'full' ? 262_144 : 32_768);
if (hash(bytes) !== env.EXPECTED_SELECTION_SHA256) reject('selection_digest');
const selected = JSON.parse(bytes); validateSelection(selected, context, env.ALLURE_JOB_RUN_ID); return selected;
}
@@ -66,7 +95,7 @@ export async function run(command, argument, env = process.env, { fetchImpl = fe
if (!['guard', 'bootstrap', 'bootstrap-context', 'selection', 'launch-plan', 'verify-selection', 'upload', 'verify-upload'].includes(command)) reject('operation');
const context = await actualContext(env, exec);
if (command === 'guard') {
await output(env, 'operation', context.operation); await output(env, 'target_sha', targetSha);
await output(env, 'operation', context.operation); await output(env, 'target_sha', context.targetSha);
await output(env, 'workflow_id', String(context.workflowId)); return 0;
}
if (command === 'verify-selection') {
@@ -79,13 +108,13 @@ export async function run(command, argument, env = process.env, { fetchImpl = fe
state = await stateDirectory(env, command === 'bootstrap-context' ? 'bootstrap' : command, command === 'bootstrap-context');
const api = await testopsClient(env.ALLURE_TOKEN, fetchImpl);
if (command === 'bootstrap') {
const prepared = await bootstrapPilotPlan(api, Number(env.CHAT_TESTOPS_INTEGRATION_ID), mutationMarker(state));
await writeReceipt(state, 'prepared.json', prepared); return 0;
const prepared = await bootstrapPilotPlan(api, Number(env.CHAT_TESTOPS_INTEGRATION_ID), mutationMarker(state), context);
await writeReceipt(state, 'prepared.json', context.catalog === 'full' ? { ...prepared, cases: prepared.cases.map(compactCase) } : prepared); return 0;
}
if (command === 'bootstrap-context') {
const official = parseOfficialBootstrapContext((await exec('allurectl', ['job-run', 'env'],
{ env, shell: false, timeout: 30_000, maxBuffer: 16_384, encoding: 'utf8' })).stdout);
const prepared = JSON.parse(await readOwned(path.join(state, 'prepared.json'), env.RUNNER_TEMP, 32_768));
const prepared = JSON.parse(await readOwned(path.join(state, 'prepared.json'), env.RUNNER_TEMP, context.catalog === 'full' ? 262_144 : 32_768));
const receipt = await completePilotRegistration(api, prepared, official, context, context.workflowId, mutationMarker(state),
observation => writeReceipt(state, 'registration-observed.json', observation));
await writeReceipt(state, 'registration.json', receipt); return 0;
@@ -96,13 +125,23 @@ export async function run(command, argument, env = process.env, { fetchImpl = fe
}
if (command === 'selection') {
const selection = await validatePilotSelection(api, env, context, context.workflowId);
if (context.catalog === 'full') selection.cases = selection.cases.map(compactCase);
await writeReceipt(state, 'selection.json', selection);
const bytes = Buffer.from(JSON.stringify(selection, null, 2));
await writeFile(path.join(env.GITHUB_WORKSPACE, 'testops-pilot-selection.json'), bytes, { flag: 'wx', mode: 0o600 });
await output(env, 'selection_sha256', hash(bytes));
await output(env, 'execution_ids', JSON.stringify(selection.cases.map(row => row.executionId)));
await output(env, 'matrix', JSON.stringify({ include: selection.cases.map(row => ({ executionId: row.executionId, key: row.key })) }));
await output(env, 'policy', JSON.stringify(evidencePolicy(selection, context, env.ALLURE_JOB_RUN_ID))); return 0;
const matrix = executionMatrix(selection, context, env.ALLURE_JOB_RUN_ID);
await output(env, 'matrix', JSON.stringify(matrix));
if (context.catalog === 'full') {
const partition = executionPartition(selection, context, env.ALLURE_JOB_RUN_ID);
for (const name of ['native', 'external']) {
const { include } = partition[name];
await output(env, `${name}_matrix`, JSON.stringify({ include })); await output(env, `${name}_count`, String(include.length));
}
await output(env, 'parallelism', partition.parallelism);
}
await output(env, 'policy', JSON.stringify(context.catalog === 'full' ? null : evidencePolicy(selection, context, env.ALLURE_JOB_RUN_ID))); return 0;
}
const selected = await readSelection(path.join(env.GITHUB_WORKSPACE, 'chat-completion-testops-selection', 'testops-pilot-selection.json'), env, context);
if (command === 'upload') {
@@ -110,7 +149,7 @@ export async function run(command, argument, env = process.env, { fetchImpl = fe
await writeReceipt(state, 'exported.json', exported); return exported.some(row => row.status !== 'passed') ? 1 : 0;
}
const prior = path.join(env.RUNNER_TEMP, 'chat-completion-testops-upload', 'exported.json');
const exported = JSON.parse(await readOwned(prior, env.RUNNER_TEMP, 32_768));
const exported = JSON.parse(await readOwned(prior, env.RUNNER_TEMP, context.catalog === 'full' ? 262_144 : 32_768));
const { campaign } = await readPilotExport(path.resolve(argument), context, selected);
if (exported.some(row => {
const original = campaign.results.find(value => value.executionId === row.executionId);
@@ -119,7 +158,8 @@ export async function run(command, argument, env = process.env, { fetchImpl = fe
const verification = await verifyPilotUpload(api, selected, context, exported);
await writeReceipt(state, 'verification.json', verification); return 0;
} catch (error) {
const code = /^(chat_completion_testops_[a-z_]{1,80}|testops_(?:api|auth)_http(?:_[1-5][0-9]{2})?)$/.test(error?.message ?? '') ? error.message : 'chat_completion_testops_unknown';
const code = fullCatalogFailureCodes.has(error?.message)
|| /^(chat_completion_testops_[a-z_]{1,80}|testops_(?:api|auth)_http(?:_[1-5][0-9]{2})?)$/.test(error?.message ?? '') ? error.message : 'chat_completion_testops_unknown';
const receipt = { command, code, httpStatus: Number.isInteger(error?.status) ? error.status : null };
if (state) await writeReceipt(state, 'failure.json', receipt).catch(() => {});
process.stderr.write(`Chat completion TestOps stopped: ${JSON.stringify(receipt)}\n`); return 1;
tests/runner-e2e/chat-completion-testops-contract.d.mts
+11 -5
View File
@@ -1,13 +1,16 @@
export interface PilotCase {
key: "control" | "defaults" | "work";
key: string;
scenario: string;
name: string;
checks: number;
aliases?: readonly string[];
registeredFullName?: string;
minimumRuns: number;
maximumRuns: number;
executionId: string;
fullName: string;
}
export type CatalogCase = Omit<PilotCase, "checks"> & { checks: number | null };
export const cases: readonly Readonly<PilotCase>[];
export const targetSha: string;
export const launchSha256: string;
@@ -15,14 +18,17 @@ export const workflow: string;
export const planName: string;
export const model: string;
export function hash(value: string | Uint8Array): string;
export function validRunCount(ids: readonly string[] | undefined, descriptor: PilotCase): boolean;
export interface PilotContext { runId: string; attempt: string; workflowSha: string; targetSha: string }
export function validRunCount(ids: readonly string[] | undefined, descriptor: CatalogCase): boolean;
export interface PilotContext { catalog?: "full"; runId: string; attempt: string; workflowSha: string; targetSha: string }
export interface PilotSelection extends PilotContext {
schema: string; projectId: number; planId: number; jobId: number; launchId: number;
allureJobRunId: string; cases: (PilotCase & { caseId: number })[];
allureJobRunId: string; cases: (CatalogCase & { caseId: number })[];
}
export function validateSelection(selection: PilotSelection, context: PilotContext, jobRunId: string): (PilotCase & { caseId: number })[];
export function validateSelection(selection: PilotSelection, context: PilotContext, jobRunId: string): (CatalogCase & { caseId: number })[];
export function evidencePolicy(selection: PilotSelection, context: PilotContext, jobRunId: string): {
version: number; approvals: { workflowSha: string; targetSha: string; runId: string; attempt: string;
executionIds: string[]; protocol: string; launchSha256: string }[];
};
export const catalogWorkflow: string;
export function catalogDefinition(context?: { catalog?: "full" }): { cases: readonly Readonly<CatalogCase>[]; planName: string; targetSha: string; workflow: string; bindingKey: string };
tests/runner-e2e/chat-completion-testops-contract.mjs
+26 -8
View File
@@ -1,4 +1,5 @@
import { createHash } from 'node:crypto';
import { fullCatalogCases, fullCatalogPlanName, catalogTargetSha } from './chat-full-catalog-testops-manifest.mjs';
// The catalog remains on its preservation/integration branch. Updating this
// immutable source pin requires a reviewed source change.
@@ -7,6 +8,7 @@ export const launchSha256 = 'f2e69957c350b220b06edca49194a4bd07d70bc882f80838eeb
export const workflow = '.github/workflows/chat-completion-testops.yml';
export const planName = 'Chat completion pilot — OpenCode 1017';
export const model = 'openrouter/deepseek/deepseek-v4.1-flash-1017';
export const catalogWorkflow = '.github/workflows/chat-catalog-testops.yml';
export const cases = Object.freeze([
{ key: 'control', scenario: 'instance-operations.restart-service', name: 'Restart an instance and retain the conversation', checks: 9, minimumRuns: 1, maximumRuns: 1 },
{ key: 'defaults', scenario: 'projects.project-defaults', name: 'Save project defaults for future work', checks: 11, minimumRuns: 3, maximumRuns: 3 },
@@ -14,6 +16,12 @@ export const cases = Object.freeze([
].map(row => Object.freeze({ ...row, executionId: `chat-features.runner-opencode.local.${row.scenario}`,
fullName: `paperclip.Product E2E.chat-features.${row.scenario}` })));
export const positive = value => Number.isSafeInteger(value) && value > 0;
export function catalogDefinition(context = {}) {
if (context.catalog === 'full') return { cases: fullCatalogCases, planName: fullCatalogPlanName,
targetSha: catalogTargetSha, workflow: catalogWorkflow, bindingKey: 'CHAT_FULL_CATALOG_TESTOPS_BINDING' };
if (context.catalog !== undefined) reject('catalog');
return { cases, planName, targetSha, workflow, bindingKey: 'CHAT_COMPLETION_TESTOPS_BINDING' };
}
export const hash = value => createHash('sha256').update(value).digest('hex');
// These bounds are copied from the pinned source, not inferred from one
// measurement. The real-source control checks them before provider access.
@@ -27,39 +35,49 @@ export function admitDispatch(env, event, triggeringActorId) {
try { allowed = JSON.parse(env.TESTOPS_ALLOWED_ACTOR_IDS); } catch { reject('actor'); }
const branch = event.repository?.default_branch;
const operation = event.inputs?.operation ?? 'evaluate';
const full = env.GITHUB_WORKFLOW_REF === `paperclipai/paperclip/${catalogWorkflow}@refs/heads/master`;
const definition = catalogDefinition(full ? { catalog: 'full' } : {});
if (env.GITHUB_ACTIONS !== 'true' || env.GITHUB_EVENT_NAME !== 'workflow_dispatch'
|| env.GITHUB_REPOSITORY !== 'paperclipai/paperclip' || event.repository?.full_name !== 'paperclipai/paperclip'
|| event.repository?.private !== false || branch !== 'master' || env.GITHUB_REF !== 'refs/heads/master'
|| event.ref !== env.GITHUB_REF || env.GITHUB_WORKFLOW_REF !== `paperclipai/paperclip/${workflow}@refs/heads/master`
|| event.ref !== env.GITHUB_REF || env.GITHUB_WORKFLOW_REF !== `paperclipai/paperclip/${definition.workflow}@refs/heads/master`
|| !/^[a-f0-9]{40}$/.test(env.GITHUB_SHA ?? '') || env.GITHUB_RUN_ATTEMPT !== '1'
|| !numeric(env.GITHUB_RUN_ID) || !numeric(env.GITHUB_ACTOR_ID) || event.sender?.id !== Number(env.GITHUB_ACTOR_ID)
|| !Array.isArray(allowed) || !allowed.length || allowed.some(v => !positive(v))
|| !allowed.includes(Number(env.GITHUB_ACTOR_ID)) || !allowed.includes(triggeringActorId)
|| !event.inputs || Object.keys(event.inputs).some(k => !['operation', 'ALLURE_JOB_RUN_ID', 'ALLURE_USERNAME'].includes(k))
|| !['bootstrap', 'evaluate', 'launch-plan'].includes(operation)
|| full && operation === 'launch-plan'
|| (operation === 'evaluate' ? !numeric(event.inputs.ALLURE_JOB_RUN_ID) : (event.inputs.ALLURE_JOB_RUN_ID ?? '') !== '')) reject('dispatch');
return Object.freeze({ runId: env.GITHUB_RUN_ID, attempt: '1', workflowSha: env.GITHUB_SHA, targetSha, operation,
return Object.freeze({ runId: env.GITHUB_RUN_ID, attempt: '1', workflowSha: env.GITHUB_SHA, targetSha: definition.targetSha, operation,
...(full ? { catalog: 'full' } : {}),
actorId: Number(env.GITHUB_ACTOR_ID), triggeringActorId });
}
export function selectedCases(rows) {
export function selectedCases(rows, context = {}) {
const { cases } = catalogDefinition(context);
if (!Array.isArray(rows) || !rows.length || rows.length > cases.length || new Set(rows.map(row => row.executionId)).size !== rows.length) reject('selection');
const selected = cases.filter(row => rows.some(value => value.executionId === row.executionId));
if (selected.length !== rows.length) reject('selection');
for (const row of rows) {
const expected = selected.find(value => value.executionId === row.executionId);
if (!positive(row.caseId) || row.fullName !== expected.fullName || row.key !== expected.key || row.scenario !== expected.scenario) reject('selection');
if (!positive(row.caseId) || row.fullName !== expected.fullName || row.key !== expected.key || row.scenario !== expected.scenario
|| row.registeredFullName !== undefined && ![expected.fullName, ...(expected.aliases ?? [])].includes(row.registeredFullName)) reject('selection');
}
if (new Set(rows.map(row => row.caseId)).size !== rows.length) reject('selection');
return selected.map(expected => ({ ...expected, caseId: rows.find(row => row.executionId === expected.executionId).caseId }));
return selected.map(expected => {
const actual = rows.find(row => row.executionId === expected.executionId);
return { ...expected, caseId: actual.caseId, ...(actual.registeredFullName === undefined ? {} : { registeredFullName: actual.registeredFullName }) };
});
}
export function validateSelection(selection, context, jobRunId) {
const definition = catalogDefinition(context);
if (selection.schema !== 'paperclip.chat-completion-testops-selection/v1' || selection.projectId !== 1
|| selection.runId !== context.runId || selection.attempt !== '1' || selection.workflowSha !== context.workflowSha
|| selection.targetSha !== targetSha || selection.allureJobRunId !== jobRunId || !numeric(jobRunId)
|| selection.targetSha !== definition.targetSha || selection.catalog !== context.catalog || selection.allureJobRunId !== jobRunId || !numeric(jobRunId)
|| ['planId', 'jobId', 'launchId'].some(key => !positive(selection[key]))) reject('selection_binding');
return selectedCases(selection.cases);
return selectedCases(selection.cases, context);
}
// This authority is emitted only by the trusted main-only selector after an
@@ -67,7 +85,7 @@ export function validateSelection(selection, context, jobRunId) {
// scanners, byte limits and exact-key S3 policies remain the enforcement owner.
export function evidencePolicy(selection, context, jobRunId) {
const selected = validateSelection(selection, context, jobRunId);
return { version: 1, approvals: [{ workflowSha: context.workflowSha, targetSha,
return { version: 1, approvals: [{ workflowSha: context.workflowSha, targetSha: catalogDefinition(context).targetSha,
runId: context.runId, attempt: '1', executionIds: selected.map(row => row.executionId),
protocol: 'paperclip.manual-private-evidence/v1', launchSha256 }] };
}
tests/runner-e2e/chat-completion-testops-export.test.ts
+47 -2
View File
@@ -6,12 +6,16 @@ import { exportCompletionPilot } from "./chat-completion-testops-export.js";
import { exportPilotAllure } from "./chat-completion-testops-results.mjs";
import { cases, targetSha, evidencePolicy, hash } from "./chat-completion-testops-contract.mjs";
import { authorizeManualPrivateEvidence } from "./manual-private-retention.js";
import { fullCatalogCases, catalogTargetSha } from "./chat-full-catalog-testops-manifest.mjs";
import { mergeCatalogProjections } from "./chat-full-catalog-testops-merge.js";
import type { CatalogCase, PilotContext } from "./chat-completion-testops-contract.mjs";
const roots: string[] = [];
afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
const png = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+/l9sAAAAASUVORK5CYII=", "base64");
const binding = { runId: "123456", attempt: "1", workflowSha: "a".repeat(40), targetSha };
async function fixture() {
async function fixture(descriptors: readonly CatalogCase[] = cases, context: PilotContext = binding) {
const cases = descriptors, binding = context, targetSha = context.targetSha;
const root = await realpath(await mkdtemp(path.join(os.tmpdir(), "chat-completion-export-"))); roots.push(root);
const report = path.join(root, "report"), output = path.join(root, "projection"), allure = path.join(root, "allure");
const results = cases.map(row => ({
@@ -22,7 +26,7 @@ async function fixture() {
profileId: "runner-opencode", environmentId: "local", caseId: row.scenario, provider: "opencode",
model: "openrouter/deepseek/deepseek-v4.1-flash-1017", runtimeMode: "native",
startedAt: "2026-11-01T00:00:00Z", finishedAt: "2026-11-01T00:00:01Z", durationMs: 1000,
matcherResults: Array.from({ length: row.checks }, (_, i) => ({ matcher: { kind: "json_path", path: `fixture.check-${i}`, expected: true }, passed: true, detail: "OPAQUE_PROVIDER_BODY" })),
matcherResults: Array.from({ length: row.checks ?? 2 }, (_, i) => ({ matcher: { kind: "json_path", path: `fixture.check-${i}`, expected: true }, passed: true, detail: "OPAQUE_PROVIDER_BODY" })),
screenshots: [], usage: { hiddenReasoning: "PRIVATE_REASONING_MUST_NOT_EXPORT" }, unknownField: "PRIVATE_PROVIDER_EXTRA",
}));
for (const row of cases) {
@@ -37,6 +41,47 @@ async function fixture() {
await save(); return { root, report, output, allure, campaign, results, selected, save };
}
it("exports the complete244 fixture selection without treating registration as a live qualification", async () => {
const context = { ...binding, catalog: "full" as const, targetSha: catalogTargetSha };
const f = await fixture(fullCatalogCases, context);
for (const row of f.selected.cases) {
const identity = { ...context, executionIds: [row.executionId] };
expect(authorizeManualPrivateEvidence(identity, JSON.stringify(evidencePolicy({ ...f.selected, cases: [row] }, context, "77"))).executionIds).toEqual([row.executionId]);
}
await exportCompletionPilot(f.report, f.output, context, f.campaign.expected);
const exported = await exportPilotAllure(f.output, f.allure, context, f.selected);
expect(exported).toHaveLength(244);
const serialized = (await readdir(f.allure)).filter(name => name.endsWith("-result.json"));
expect(serialized).toHaveLength(244);
const rendered = await Promise.all(serialized.map(async name => JSON.parse(await readFile(path.join(f.allure, name), "utf8"))));
expect(rendered.find(row => row.fullName === "paperclip.Product E2E.chat-features.tasks.create-backlog").parameters)
.toContainEqual({ name: "canonicalScenario", value: "tasks.create-backlog" });
expect(await readFile(path.join(f.output, "normalized-results.json"), "utf8")).not.toMatch(/PRIVATE_|OPAQUE_PROVIDER_BODY/);
});
it("merges distinct source-bound singleton projections and retains a failed original beside passed originals", async () => {
const context = { ...binding, catalog: "full" as const, targetSha: catalogTargetSha };
const descriptors = fullCatalogCases.filter(row => ["tasks.create-backlog", "projects.project-work", "tasks.edit-properties"].includes(row.key));
expect(descriptors).toHaveLength(3);
const f = await fixture(descriptors, context);
const inputs = path.join(f.root, "projections"); await mkdir(inputs);
const results = [...f.results];
Object.assign(results[1]!, { status: "failed", cleanup: "failed", evidenceValid: false, runIds: [] });
for (const [index, result] of results.entries()) {
f.campaign.expected = [result.executionId]; f.campaign.results = [result]; await f.save();
await exportCompletionPilot(f.report, path.join(inputs, `cell-${index}`), context, [result.executionId]);
}
const merged = path.join(f.root, "merged");
await mergeCatalogProjections(inputs, merged, f.selected, context);
const exported = await exportPilotAllure(merged, f.allure, context, f.selected);
expect(exported.map(row => row.status).sort()).toEqual(["failed", "passed", "passed"]);
const missing = path.join(f.root, "missing"); await mkdir(missing);
await expect(mergeCatalogProjections(missing, path.join(f.root, "invalid"), f.selected, context)).rejects.toThrow("Incomplete projection set");
const bindingFile = path.join(inputs, "cell-0", "binding.json");
const record = JSON.parse(await readFile(bindingFile, "utf8")); record.targetSha = "f".repeat(40); await writeFile(bindingFile, JSON.stringify(record));
await expect(mergeCatalogProjections(inputs, path.join(f.root, "foreign"), f.selected, context)).rejects.toThrow();
});
it("converts three original cases into three exact TestOps outcomes with all 35 checks and private-evidence receipts", async () => {
const f = await fixture(); await writeFile(path.join(f.report, "private-provider.log"), "PRIVATE_LOG_MUST_NOT_EXPORT");
await exportCompletionPilot(f.report, f.output, binding, f.campaign.expected);
tests/runner-e2e/chat-completion-testops-export.ts
+8 -5
View File
@@ -5,11 +5,11 @@ import { buildRunnerCampaign } from "./history.js";
import { assertSecretFree, findSecretLeakInJsonValues } from "./redaction.js";
import { validateRetainedRunnerResult } from "./result-validation.js";
import { publicBilling } from "./testops-chat-export.js";
import { cases, targetSha, model, hash, validRunCount } from "./chat-completion-testops-contract.mjs";
import { model, hash, validRunCount, catalogDefinition } from "./chat-completion-testops-contract.mjs";
import type { RunnerE2EResult } from "./types.js";
const pngMagic = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]);
interface Binding { runId: string; attempt: string; workflowSha: string; targetSha: string }
interface Binding { runId: string; attempt: string; workflowSha: string; targetSha: string; catalog?: "full" }
async function ownedRead(root: string, relative: string, limit: number) {
if (!path.isAbsolute(root) || await realpath(root) !== root || relative.split("/").some(part => !/^[A-Za-z0-9._-]+$/.test(part) || [".", ".."].includes(part))) throw new Error("Invalid pilot evidence path");
@@ -32,8 +32,9 @@ async function ownedRead(root: string, relative: string, limit: number) {
* verified here, but only its checksum and availability receipt cross jobs.
*/
export async function exportCompletionPilot(root: string, output: string, binding: Binding, expectedIds: string[]) {
const { cases, targetSha } = catalogDefinition(binding);
if (!/^[1-9][0-9]{0,15}$/.test(binding.runId) || binding.attempt !== "1" || !/^[a-f0-9]{40}$/.test(binding.workflowSha)
|| binding.targetSha !== targetSha || !Array.isArray(expectedIds) || !expectedIds.length || expectedIds.length > 3
|| binding.targetSha !== targetSha || !Array.isArray(expectedIds) || !expectedIds.length || expectedIds.length > cases.length
|| new Set(expectedIds).size !== expectedIds.length || expectedIds.some(id => !cases.some(row => row.executionId === id))) throw new Error("Invalid pilot binding");
const campaignId = `gha-${binding.runId}-1`, runUrl = `https://github.com/paperclipai/paperclip/actions/runs/${binding.runId}`;
const raw = JSON.parse((await ownedRead(root, "normalized-results.json", 8 * 1024 * 1024)).toString("utf8"));
@@ -55,7 +56,7 @@ export async function exportCompletionPilot(root: string, output: string, bindin
if (checks.length > 200 || checks.some(check => check.matcher.kind !== "json_path" || !/^[A-Za-z0-9._-]{1,150}$/.test(check.matcher.path)
|| check.matcher.expected !== true || typeof check.passed !== "boolean")) throw new Error("Invalid pilot checks");
if (original.status === "passed" && (original.cleanup !== "passed" || original.evidenceValid !== true
|| checks.length !== descriptor.checks || checks.some(check => !check.passed) || !validRunCount(original.runIds, descriptor))) throw new Error("Incomplete pilot pass");
|| (descriptor.checks == null ? checks.length < 1 : checks.length !== descriptor.checks) || checks.some(check => !check.passed) || !validRunCount(original.runIds, descriptor))) throw new Error("Incomplete pilot pass");
let image: Buffer | undefined;
const relative = `evidence/${id}/attempt-1/final-state.png`;
try { image = await ownedRead(root, relative, 12 * 1024 * 1024); }
@@ -101,7 +102,9 @@ export async function exportCompletionPilot(root: string, output: string, bindin
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
exportCompletionPilot(path.resolve(process.env.TESTOPS_PILOT_REPORT_ROOT ?? "runner-e2e-merged-report/normalized"),
path.resolve(process.env.TESTOPS_PILOT_EXPORT_ROOT ?? "runner-e2e-testops"), {
runId: process.env.GITHUB_RUN_ID ?? "", attempt: process.env.GITHUB_RUN_ATTEMPT ?? "", workflowSha: process.env.GITHUB_SHA ?? "", targetSha,
runId: process.env.GITHUB_RUN_ID ?? "", attempt: process.env.GITHUB_RUN_ATTEMPT ?? "", workflowSha: process.env.GITHUB_SHA ?? "",
...(process.env.GITHUB_WORKFLOW_REF === "paperclipai/paperclip/.github/workflows/chat-catalog-testops.yml@refs/heads/master" ? { catalog: "full" as const } : {}),
targetSha: catalogDefinition(process.env.GITHUB_WORKFLOW_REF === "paperclipai/paperclip/.github/workflows/chat-catalog-testops.yml@refs/heads/master" ? { catalog: "full" } : {}).targetSha,
}, JSON.parse(process.env.MANUAL_PRIVATE_IDS ?? "null"))
.catch(() => { process.stderr.write("TestOps pilot projection rejected; originals remain protected\n"); process.exitCode = 1; });
}
tests/runner-e2e/chat-completion-testops-host-control.mjs
+9 -3
View File
@@ -3,19 +3,21 @@ import { mkdtemp, readFile, realpath, rm } from 'node:fs/promises';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
import { parse } from 'yaml';
import { cases, model, targetSha, workflow } from './chat-completion-testops-contract.mjs';
import { model, catalogDefinition, catalogWorkflow } from './chat-completion-testops-contract.mjs';
/** A credential-free caller/source control with synthetic IMDS responses.
* It does not qualify an actual AWS host or make a provider call.
*/
export async function verifyHostContract(workspace, temporaryRoot, env = process.env) {
if (env.ALLURE_TOKEN || env.OPENROUTER_API_KEY || !path.isAbsolute(workspace)) throw new Error('Credential-free source control required');
const full = env.GITHUB_WORKFLOW_REF === `paperclipai/paperclip/${catalogWorkflow}@refs/heads/master`;
const { cases, targetSha, workflow } = catalogDefinition(full ? { catalog: 'full' } : {});
const source = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: workspace, encoding: 'utf8' }).trim();
if (source !== targetSha) throw new Error('Source control pin mismatch');
const definition = parse(await readFile(new URL(`../../${workflow}`, import.meta.url), 'utf8'));
const paid = definition.jobs.chat.steps.find(step => step.env?.OPENROUTER_API_KEY);
if (paid.env.PAPERCLIP_CHAT_COMPLETION_TESTOPS !== 'v1' || paid.env.ALLURE_JOB_RUN_ID !== '${{ inputs.ALLURE_JOB_RUN_ID }}'
|| !paid.run.includes('--max-automatic-retries 0') || definition.jobs.chat.strategy['max-parallel'] !== 1) throw new Error('Workflow host contract mismatch');
|| !paid.run.includes('--max-automatic-retries 0') || definition.jobs.chat.strategy['max-parallel'] !== (full ? '${{ fromJSON(needs.select.outputs.parallelism) }}' : 1)) throw new Error('Workflow host contract mismatch');
const { runnerMatrix } = await import(pathToFileURL(path.join(workspace, 'tests/runner-e2e/catalog.ts')).href);
const { prepareChatRemoteHost } = await import(pathToFileURL(path.join(workspace, 'tests/runner-e2e/chat-remote-host.ts')).href);
const root = await realpath(await mkdtemp(path.join(temporaryRoot, 'chat-completion-host-control-')));
@@ -25,7 +27,11 @@ export async function verifyHostContract(workspace, temporaryRoot, env = process
if (!execution || execution.task.id !== descriptor.scenario || execution.profile.model !== model
|| execution.profile.id !== 'runner-opencode' || execution.environment.id !== 'local'
|| execution.task.expectedRunCount !== descriptor.maximumRuns
|| (execution.task.minimumExpectedRunCount ?? execution.task.expectedRunCount) !== descriptor.minimumRuns) throw new Error('Pinned source selection mismatch');
|| (execution.task.minimumExpectedRunCount ?? execution.task.expectedRunCount) !== descriptor.minimumRuns
|| full && (execution.task.turnTimeoutMs !== descriptor.turnTimeoutMs
|| execution.task.attemptTimeoutMs.local !== descriptor.attemptTimeoutMs
|| execution.suite.definitionMetadata?.companyAndAgentBudgetCents !== 500
|| execution.suite.definitionMetadata?.maximumAttemptsPerCell !== 1)) throw new Error('Pinned source selection mismatch');
const controlEnv = { GITHUB_ACTIONS: 'true', GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/heads/master',
GITHUB_WORKFLOW_REF: `paperclipai/paperclip/${workflow}@refs/heads/master`, GITHUB_RUN_ID: '1', GITHUB_RUN_ATTEMPT: '1',
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: targetSha, PAPERCLIP_CHAT_COMPLETION_TESTOPS: paid.env.PAPERCLIP_CHAT_COMPLETION_TESTOPS,
tests/runner-e2e/chat-completion-testops-results.d.mts
+4 -4
View File
@@ -1,9 +1,9 @@
import type { RunnerE2EResult } from "./types.js";
import type { PilotCase } from "./chat-completion-testops-contract.mjs";
interface Context { runId: string; attempt: string; workflowSha: string; targetSha: string }
import type { CatalogCase } from "./chat-completion-testops-contract.mjs";
interface Context { catalog?: "full"; runId: string; attempt: string; workflowSha: string; targetSha: string }
interface Selection extends Context {
schema: string; projectId: number; planId: number; jobId: number; launchId: number;
allureJobRunId: string; cases: (PilotCase & { caseId: number })[];
allureJobRunId: string; cases: (CatalogCase & { caseId: number })[];
}
export interface PilotOutcome {
caseId: number; executionId: string; status: "passed" | "failed"; cleanup: string;
@@ -11,7 +11,7 @@ export interface PilotOutcome {
}
export function readPilotExport(root: string, context: Context, selection: Selection): Promise<{
campaign: { campaignId: string; results: RunnerE2EResult[] };
selected: (PilotCase & { caseId: number })[]; retained: Map<string, Buffer>;
selected: (CatalogCase & { caseId: number })[]; retained: Map<string, Buffer>;
privateEvidence: { executionId: string; requiredEvidenceComplete: boolean | null; finalScreenshot: { sha256: string; bytes: number } | null }[];
}>;
export function exportPilotAllure(root: string, output: string, context: Context, selection: Selection): Promise<PilotOutcome[]>;
tests/runner-e2e/chat-completion-testops-results.mjs
+10 -8
View File
@@ -1,7 +1,7 @@
import { randomUUID } from 'node:crypto';
import { lstat, mkdir, readFile, readdir, realpath, writeFile } from 'node:fs/promises';
import path from 'node:path';
import { targetSha, model, hash, validateSelection, validRunCount, reject } from './chat-completion-testops-contract.mjs';
import { model, hash, validateSelection, validRunCount, reject, catalogDefinition } from './chat-completion-testops-contract.mjs';
async function inventory(root, relative = '') {
const found = []; const entries = await readdir(path.join(root, relative));
@@ -21,12 +21,13 @@ async function inventory(root, relative = '') {
export async function readPilotExport(root, context, selection) {
const selected = validateSelection(selection, context, selection.allureJobRunId);
const { targetSha } = catalogDefinition(context);
const stat = await lstat(root);
if (!stat.isDirectory() || stat.isSymbolicLink() || await realpath(root) !== path.resolve(root)) reject('export_path');
const files = await inventory(root);
if (!files.includes('binding.json')) reject('export_binding');
const bindingBytes = await readFile(path.join(root, 'binding.json'));
if (bindingBytes.length > 32_768) reject('export_binding');
if (bindingBytes.length > (context.catalog === 'full' ? 262_144 : 32_768)) reject('export_binding');
const binding = JSON.parse(bindingBytes);
if (binding.schema !== 'paperclip.chat-completion-testops-export/v1' || binding.runId !== context.runId || binding.attempt !== '1'
|| binding.workflowSha !== context.workflowSha || binding.targetSha !== targetSha || binding.campaignId !== `gha-${context.runId}-1`
@@ -47,7 +48,7 @@ export async function readPilotExport(root, context, selection) {
retained.set(row.path, bytes);
}
const normalized = retained.get('normalized-results.json');
if (normalized.length > 2_000_000) reject('export_inventory');
if (normalized.length > (context.catalog === 'full' ? 16_000_000 : 2_000_000)) reject('export_inventory');
const campaign = JSON.parse(normalized);
if (campaign.schema !== 'paperclip.runner-e2e.campaign/v2' || campaign.campaignId !== binding.campaignId || campaign.retries !== 0
|| !Array.isArray(campaign.results) || campaign.results.length !== selected.length
@@ -65,7 +66,7 @@ export async function readPilotExport(root, context, selection) {
if (!evidence || ![true, false, null].includes(evidence.requiredEvidenceComplete)
|| evidence.finalScreenshot !== null && (!/^[a-f0-9]{64}$/.test(evidence.finalScreenshot?.sha256 ?? '')
|| !Number.isSafeInteger(evidence.finalScreenshot.bytes) || evidence.finalScreenshot.bytes < 24 || evidence.finalScreenshot.bytes > 12 * 1024 * 1024)) reject('private_evidence_receipt');
if (row.status === 'passed' && (row.cleanup !== 'passed' || row.matcherResults.length !== expected.checks
if (row.status === 'passed' && (row.cleanup !== 'passed' || (expected.checks == null ? row.matcherResults.length < 1 : row.matcherResults.length !== expected.checks)
|| row.matcherResults.some(check => !check.passed) || !validRunCount(row.runIds, expected)
|| evidence.requiredEvidenceComplete !== true || evidence.finalScreenshot === null)) reject('export_incomplete_pass');
}
@@ -74,6 +75,7 @@ export async function readPilotExport(root, context, selection) {
export async function exportPilotAllure(root, output, context, selection) {
const { campaign, selected, privateEvidence } = await readPilotExport(root, context, selection);
const { targetSha } = catalogDefinition(context);
await mkdir(output, { mode: 0o700 }).catch(error => { if (error.code !== 'EEXIST') throw error; });
const stat = await lstat(output);
if (!stat.isDirectory() || stat.isSymbolicLink() || process.getuid && stat.uid !== process.getuid() || (await readdir(output)).length) reject('allure_output');
@@ -105,14 +107,14 @@ export async function exportPilotAllure(root, output, context, selection) {
archiveKind: `manual-cell-${hash(result.executionId)}`,
note: 'Only this availability receipt and checksum are attached. Original screenshots and diagnostics require authenticated private-history retrieval.' };
await attach('Private evidence receipt', 'application/json', Buffer.from(JSON.stringify(evidenceReceipt, null, 2)), 'json');
const value = { uuid: randomUUID(), name: selectedCase.name, fullName: selectedCase.fullName,
testCaseId: hash(`${JSON.stringify(selectedCase.fullName)}\n`),
historyId: hash(JSON.stringify([selectedCase.fullName, 'runner-opencode', model, 'local'])), status, stage: 'finished',
const value = { uuid: randomUUID(), name: selectedCase.name, fullName: selectedCase.registeredFullName ?? selectedCase.fullName,
testCaseId: hash(`${JSON.stringify(selectedCase.registeredFullName ?? selectedCase.fullName)}\n`),
historyId: hash(JSON.stringify([selectedCase.registeredFullName ?? selectedCase.fullName, 'runner-opencode', model, 'local'])), status, stage: 'finished',
statusDetails: { message: status === 'passed' ? 'Real Product E2E assertions and cleanup passed' : 'Original evaluation or cleanup failed' },
labels: [{ name: 'AS_ID', value: String(selectedCase.caseId) }, { name: 'parentSuite', value: 'Product E2E' },
{ name: 'suite', value: 'chat-features' }, { name: 'area', value: 'Chat and collaboration' },
{ name: 'framework', value: 'Paperclip Evals' }, { name: 'tag', value: 'chat-completion-testops' }],
parameters: [{ name: 'canonicalScenario', value: selectedCase.scenario }, { name: 'model', value: model },
parameters: [{ name: 'canonicalScenario', value: selectedCase.canonicalScenario ?? selectedCase.scenario }, { name: 'model', value: model },
{ name: 'environment', value: 'GitHub Actions / AWS' }],
links: [{ name: 'GitHub Actions run', type: 'custom', url: `https://github.com/paperclipai/paperclip/actions/runs/${context.runId}` },
{ name: 'Evaluated source', type: 'custom', url: `https://github.com/paperclipai/paperclip/commit/${targetSha}` }],
tests/runner-e2e/chat-full-catalog-testops-adapter.test.mjs
+226
View File
@@ -0,0 +1,226 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { fullCatalogCases, fullCatalogPlanName, catalogTargetSha } from './chat-full-catalog-testops-manifest.mjs';
import { admitDispatch, validateSelection, evidencePolicy, catalogWorkflow, catalogDefinition } from './chat-completion-testops-contract.mjs';
import { bindingFromEnv, validatePilotSelection, verifyPilotUpload, completePilotRegistration } from './chat-completion-testops-api.mjs';
import { executionMatrix, executionPartition, run } from './chat-completion-testops-cli.mjs';
import { mkdtemp, writeFile, readFile, readdir, rm, realpath } from 'node:fs/promises';
import os from 'node:os';
import path from 'node:path';
const context = { catalog: 'full', runId: '123', attempt: '1', workflowSha: 'a'.repeat(40), targetSha: catalogTargetSha };
const runUrl = 'https://github.com/paperclipai/paperclip/actions/runs/123';
const selected = fullCatalogCases.map((row, index) => ({ ...row, caseId: row.key === 'tasks.create-backlog' ? 388 : 1000 + index,
registeredFullName: row.aliases[0] ?? row.fullName }));
const compact = row => ({ key: row.key, caseId: row.caseId, registeredFullName: row.registeredFullName });
const binding = { planId: 10, jobId: 11, integrationId: 1, cases: selected.map(compact) };
const selection = { schema: 'paperclip.chat-completion-testops-selection/v1', ...context,
projectId: 1, planId: 10, jobId: 11, launchId: 12, allureJobRunId: '77', cases: selected };
const env = { CHAT_FULL_CATALOG_TESTOPS_BINDING: JSON.stringify(binding), ALLURE_JOB_RUN_ID: '77' };
const attachments = [
{ name: 'Execution provenance and checks', contentType: 'application/json', contentLength: 100 },
{ name: 'Private evidence receipt', contentType: 'application/json', contentLength: 200 },
];
const exported = selected.map(row => ({ executionId: row.executionId, caseId: row.caseId, status: 'passed', cleanup: 'passed', attachments }));
function api(replace = {}) {
const calls = [];
const get = async route => {
calls.push(route);
if (Object.hasOwn(replace, route)) return structuredClone(replace[route]);
let match;
if ((match = route.match(/^\/api\/testcase\/(\d+)$/))) {
const row = selected.find(value => value.caseId === Number(match[1]));
return { id: row.caseId, projectId: 1, fullName: row.registeredFullName, automated: true };
}
if (route === '/api/testplan/10') return { id: 10, projectId: 1, name: fullCatalogPlanName,
testCasesCount: 244, treeSelection: { inverted: false, leafsInclude: selected.map(row => row.caseId) } };
if (route === '/api/testplan/10/job') return [{ job: { id: 11 }, testCasesCount: 244 }];
if (route === '/api/job/11') return { id: 11, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-catalog-testops.yml',
parameters: [{ id: 34, name: 'Branch', defaultValue: 'refs/heads/master' }] };
if (route === '/api/jobrun/77') return { id: 77, job: { id: 11, type: 'github' }, stage: 'in_progress', launchId: 12, url: runUrl };
if (route === '/api/launch/12') return { id: 12, projectId: 1, closed: false };
if (route === '/api/jobrun/77/plan?expected=true') return selected.map(row => ({ id: row.caseId, selector: row.registeredFullName }));
if ((match = route.match(/^\/api\/testresult\?launchId=12&page=(\d)&size=100$/))) {
const index = Number(match[1]);
return { content: selected.slice(index * 100, (index + 1) * 100).map((row, offset) => ({ id: 2000 + index * 100 + offset, testCaseId: row.caseId })),
totalElements: 244, totalPages: 3, number: index };
}
if ((match = route.match(/^\/api\/testresult\/(\d+)$/))) {
const id = Number(match[1]), row = selected[id - 2000];
return { id, projectId: 1, launchId: 12, testCaseId: row.caseId, fullName: row.registeredFullName,
jobRun: { id: 77, launchId: 12, url: runUrl }, links: [], status: 'passed' };
}
if (route.startsWith('/api/testresult/attachment?')) return { content: attachments.map((row, index) => ({ ...row, id: 3000 + index,
missed: false, fromTestCase: false })), totalElements: 2, totalPages: 1 };
throw new Error(`Unexpected route ${route}`);
};
return { get, calls };
}
test('the complete 244 mapping uses compact actual IDs and preserves the legacy case388 alias', async () => {
assert.ok(Buffer.byteLength(env.CHAT_FULL_CATALOG_TESTOPS_BINDING) < 48_000);
assert.equal(bindingFromEnv(env, context).cases.length, 244);
const f = api(), value = await validatePilotSelection(f.get, env, context, 99);
assert.equal(validateSelection(value, context, '77').length, 244);
assert.equal(value.cases.find(row => row.caseId === 388).registeredFullName, 'paperclip.Product E2E.chat-features.create-backlog');
assert.equal(value.targetSha, catalogTargetSha);
});
test('the actual full registration receipt fits one protected variable without a later source change', async () => {
const prepared = { planId: 10, integrationId: 1, cases: selected.map(row => Object.fromEntries(
['key', 'scenario', 'executionId', 'fullName', 'registeredFullName', 'caseId'].map(key => [key, row[key]]))) };
const f = api();
const receipt = await completePilotRegistration(f.get, prepared, { jobRunId: '77', launchId: 12 }, context, 99, async () => {});
assert.ok(Buffer.byteLength(JSON.stringify(receipt.binding)) < 48_000);
assert.equal(bindingFromEnv({ CHAT_FULL_CATALOG_TESTOPS_BINDING: JSON.stringify(receipt.binding) }, context).cases.length, 244);
assert.equal(receipt.binding.cases.find(row => row.caseId === 388).registeredFullName, 'paperclip.Product E2E.chat-features.create-backlog');
});
test('every case has a singleton private approval instead of broadening the original eight-cell authority', () => {
const matrix = executionMatrix(selection, context, '77');
assert.equal(matrix.include.length, 244);
for (const row of selected) {
const policy = evidencePolicy({ ...selection, cases: [row] }, context, '77');
assert.deepEqual(policy.approvals[0].executionIds, [row.executionId]);
assert.equal(policy.approvals[0].targetSha, catalogTargetSha);
const cell = matrix.include.find(value => value.executionId === row.executionId);
assert.deepEqual(JSON.parse(cell.private_ids), [row.executionId]);
assert.deepEqual(JSON.parse(cell.private_policy), policy);
assert.deepEqual(cell.required_credentials, row.requiredCredentials);
}
});
test('full catalog source admission is independent from the old three-case workflow', () => {
const event = { repository: { full_name: 'paperclipai/paperclip', private: false, default_branch: 'master' }, ref: 'refs/heads/master',
sender: { id: 42 }, inputs: { operation: 'evaluate', ALLURE_JOB_RUN_ID: '77' } };
const dispatchEnv = { GITHUB_ACTIONS: 'true', GITHUB_EVENT_NAME: 'workflow_dispatch', GITHUB_REPOSITORY: event.repository.full_name,
GITHUB_REF: event.ref, GITHUB_WORKFLOW_REF: `paperclipai/paperclip/${catalogWorkflow}@refs/heads/master`, GITHUB_SHA: context.workflowSha,
GITHUB_RUN_ATTEMPT: '1', GITHUB_RUN_ID: '123', GITHUB_ACTOR_ID: '42', TESTOPS_ALLOWED_ACTOR_IDS: '[42]' };
assert.equal(admitDispatch(dispatchEnv, event, 42).targetSha, catalogTargetSha);
assert.equal(catalogDefinition({}).cases.length, 3);
assert.throws(() => admitDispatch({ ...dispatchEnv, GITHUB_RUN_ATTEMPT: '2' }, event, 42), /dispatch/);
assert.throws(() => admitDispatch(dispatchEnv, { ...event, inputs: { operation: 'launch-plan' } }, 42), /dispatch/);
assert.throws(() => validateSelection({ ...selection, targetSha: 'b'.repeat(40) }, context, '77'), /selection_binding/);
});
test('native and external partitions preserve exact selections, serial small runs and the GitHub output bound', () => {
for (const rows of [selected, selected.slice(0, 3), selected.filter(row => row.externalInput), [selected[0], selected.find(row => row.externalInput)]]) {
const input = { ...selection, cases: rows }, partition = executionPartition(input, context, '77');
const merged = [...partition.native.include, ...partition.external.include].map(row => row.executionId);
assert.equal(new Set(merged).size, rows.length);
assert.deepEqual(merged.sort(), rows.map(row => row.executionId).sort());
if (rows.length <= 3) {
assert.equal(partition.parallelism, '1'); assert.equal(partition.external.include.length, 0);
} else {
assert.equal(partition.parallelism, '8'); assert.ok(partition.external.include.every(row => row.external));
}
assert.ok([partition.matrix, partition.native, partition.external, rows.map(row => row.executionId)]
.reduce((sum, value) => sum + JSON.stringify(value).length * 2, 0) < 1_000_000);
for (const cell of merged) assert.ok(partition.matrix.include.find(row => row.executionId === cell).timeout_minutes <= 45);
}
});
test('full server readback verifies every result across three complete pages', async () => {
const f = api();
const value = await verifyPilotUpload(f.get, selection, context, exported);
assert.equal(value.verification, 'VERIFIED'); assert.equal(value.results.length, 244);
assert.ok(f.calls.includes('/api/testresult?launchId=12&page=2&size=100'));
});
test('a full ready snapshot fits its read budget while a late snapshot still fails closed', async t => {
async function replay(latency) {
let clock = 0, outcome; const pending = [], f = api();
const verification = verifyPilotUpload(route => new Promise((resolve, reject) => {
pending.push({ at: clock + latency, complete: () => { f.get(route).then(resolve, reject); } });
}), selection, context, exported, { now: () => clock });
verification.then(value => { outcome = { value }; }, error => { outcome = { error }; });
for (let tick = 0; tick < 600 && !outcome; tick++) {
await new Promise(resolve => setImmediate(resolve));
if (outcome) break;
assert.ok(pending.length > 0, 'the controlled response chain must make progress');
clock = Math.min(...pending.map(row => row.at));
const ready = pending.filter(row => row.at <= clock);
for (const row of ready) { pending.splice(pending.indexOf(row), 1); row.complete(); }
}
assert.ok(outcome, 'bounded controlled snapshot replay must settle');
return { ...outcome, clock };
}
const ready = await replay(1000);
assert.ifError(ready.error);
assert.equal(ready.value.verification, 'VERIFIED'); assert.equal(ready.value.results.length, 244);
assert.ok(ready.clock > 120_000);
const late = await replay(3000);
assert.match(late.error?.message ?? '', /upload_unverified/);
t.diagnostic(JSON.stringify({ fixtureOnly: true, selectedCases: 244, readyElapsedMs: ready.clock, lateElapsedMs: late.clock }));
});
test('changing valid page counts retry only within the existing snapshot bounds', async () => {
async function replay(alwaysChanging, malformed = false) {
const f = api(); let polls = 0, waits = 0, clock = 0;
const get = async route => {
if (route === '/api/jobrun/77') polls++;
const page = await f.get(route);
if (route === '/api/testresult?launchId=12&page=0&size=100' && (alwaysChanging || polls === 1)) {
return { ...page, totalElements: 150, totalPages: 2 };
}
if (malformed && route === '/api/testresult?launchId=12&page=1&size=100') return { ...page, totalElements: '244' };
return page;
};
try {
const value = await verifyPilotUpload(get, selection, context, exported, { now: () => clock,
wait: async ms => { clock += ms; waits++; } });
return { value, polls, waits };
} catch (error) { return { error, polls, waits }; }
}
const converged = await replay(false);
assert.ifError(converged.error);
assert.equal(converged.value.verification, 'VERIFIED'); assert.equal(converged.value.results.length, 244);
assert.equal(converged.polls, 2); assert.equal(converged.waits, 1);
const moving = await replay(true);
assert.match(moving.error?.message ?? '', /upload_unverified/);
assert.equal(moving.polls, 12); assert.equal(moving.waits, 11);
const malformed = await replay(false, true);
assert.match(malformed.error?.message ?? '', /result_page/);
assert.equal(malformed.polls, 1); assert.equal(malformed.waits, 0);
});
test('an incomplete last page and a wrong registered selector cannot qualify a full catalog run', async () => {
const missing = api({ '/api/testresult?launchId=12&page=2&size=100': { content: [], totalElements: 244, totalPages: 3, number: 2 } });
await assert.rejects(verifyPilotUpload(missing.get, selection, context, exported), /result_page/);
const wrong = api({ '/api/jobrun/77/plan?expected=true': [{ id: 388, selector: fullCatalogCases[0].fullName }] });
await assert.rejects(validatePilotSelection(wrong.get, env, context, 99), /foreign_selection/);
});
test('the real bootstrap facade preserves fixed reconciliation failures and discards unknown lookalike messages', async t => {
for (const unknown of [false, true]) {
const root = await realpath(await mkdtemp(path.join(os.tmpdir(), 'catalog-bootstrap-failure-')));
let stderr = ''; const capture = t.mock.method(process.stderr, 'write', value => { stderr += value; return true; });
try {
const event = { repository: { full_name: 'paperclipai/paperclip', private: false, default_branch: 'master' },
ref: 'refs/heads/master', sender: { id: 42 }, inputs: { operation: 'bootstrap' } };
const eventPath = path.join(root, 'event.json'); await writeFile(eventPath, JSON.stringify(event));
const runtime = { GITHUB_ACTIONS: 'true', GITHUB_EVENT_NAME: 'workflow_dispatch', GITHUB_REPOSITORY: event.repository.full_name,
GITHUB_REF: event.ref, GITHUB_WORKFLOW_REF: `paperclipai/paperclip/${catalogWorkflow}@refs/heads/master`, GITHUB_SHA: context.workflowSha,
GITHUB_RUN_ATTEMPT: '1', GITHUB_RUN_ID: '123', GITHUB_ACTOR_ID: '42', TESTOPS_ALLOWED_ACTOR_IDS: '[42]', GITHUB_RUN_NUMBER: '1',
GITHUB_EVENT_PATH: eventPath, GH_TOKEN: 'SYNTHETIC_GITHUB_SECRET', ALLURE_TOKEN: 'SYNTHETIC_REPORTING_SECRET',
RUNNER_TEMP: root, GITHUB_WORKSPACE: root, CHAT_TESTOPS_INTEGRATION_ID: '1' };
const exec = async () => ({ stdout: JSON.stringify({ id: 123, run_attempt: 1, run_number: 1, workflow_id: 99,
head_sha: context.workflowSha, head_branch: 'master', event: 'workflow_dispatch', path: catalogWorkflow,
repository: event.repository, actor: { id: 42 }, triggering_actor: { id: 42 } }) });
const fetchImpl = async (url, options) => {
if (unknown) throw new Error('chat_full_catalog_testops_catalog_page_private_body');
if (url.endsWith('/api/uaa/oauth/token')) return Response.json({ access_token: 'SYNTHETIC_BEARER_SECRET' });
assert.equal(options.method, 'GET');
if (url.endsWith('/api/integration/1/project/1')) return Response.json({ id: 1, projectId: 1, info: { type: 'github' }, disabled: false });
assert.ok(url.includes('/api/testcase?'));
return Response.json({ content: [], number: 0, totalPages: 2, totalElements: 0, private: 'SYNTHETIC_PRIVATE_BODY' });
};
assert.equal(await run('bootstrap', undefined, runtime, { exec, fetchImpl }), 1);
const state = path.join(root, 'chat-completion-testops-bootstrap');
const failure = JSON.parse(await readFile(path.join(state, 'failure.json'), 'utf8'));
assert.equal(failure.code, unknown ? 'chat_completion_testops_unknown' : 'chat_full_catalog_testops_catalog_page');
assert.deepEqual(await readdir(state), ['failure.json']);
assert.doesNotMatch(stderr + JSON.stringify(failure), /SYNTHETIC_|PRIVATE_BODY|catalog_page_private_body/);
} finally { capture.mock.restore(); await rm(root, { recursive: true, force: true }); }
}
});
tests/runner-e2e/chat-full-catalog-testops-manifest.d.mts
+5
View File
@@ -0,0 +1,5 @@
import type { CatalogCase } from './chat-completion-testops-contract.mjs';
export const catalogTargetSha: string;
export const catalogManifestSha256: string;
export const fullCatalogPlanName: string;
export const fullCatalogCases: readonly Readonly<CatalogCase>[];
tests/runner-e2e/chat-full-catalog-testops-manifest.json
+11971
View File
File diff suppressed because it is too large. Load diff
tests/runner-e2e/chat-full-catalog-testops-manifest.mjs
+57
View File
@@ -0,0 +1,57 @@
import { createHash } from 'node:crypto';
import { readFileSync } from 'node:fs';
const bytes = readFileSync(new URL('./chat-full-catalog-testops-manifest.json', import.meta.url));
const manifest = JSON.parse(bytes);
const legacy = new Set(['tasks.create-backlog', 'tasks.delegate-announcement']);
const surfaces = new Set(['agent-chat', 'browser-boundary', 'scoped-external-assistant', 'native-slack-thread']);
const nonnegative = value => Number.isSafeInteger(value) && value >= 0;
const text = value => typeof value === 'string' && value.trim().length > 0;
export function validateFullCatalogManifest(value) {
if (value?.schema !== 'paperclip.chat-full-catalog-testops-manifest/v1'
|| !/^[a-f0-9]{40}$/.test(value.targetSha ?? '') || !/^[a-f0-9]{40}$/.test(value.metadataSourceSha ?? '')
|| value.profile !== 'runner-opencode' || value.model !== 'openrouter/deepseek/deepseek-v4.1-flash-1017'
|| !Array.isArray(value.cases) || value.cases.length !== 244) throw new Error('chat_full_catalog_manifest_invalid');
for (const row of value.cases) {
const task = legacy.has(row.key) ? row.key.slice(6) : row.key;
if (!/^[a-z0-9-]+\.[a-z0-9-]+$/.test(row.key ?? '') || row.canonicalScenario !== row.key || row.scenario !== task
|| row.area !== row.key.split('.')[0] || !text(row.name) || !text(row.title)
|| row.executionId !== `chat-features.runner-opencode.local.${task}`
|| row.fullName !== `paperclip.Product E2E.chat-features.${row.key}`
|| JSON.stringify(row.aliases) !== JSON.stringify(legacy.has(row.key) ? [`paperclip.Product E2E.chat-features.${task}`] : [])
|| !nonnegative(row.minimumRuns) || !nonnegative(row.maximumRuns) || row.minimumRuns > row.maximumRuns
|| !nonnegative(row.turnTimeoutMs) || !nonnegative(row.declaredTurnTimeoutMs)
|| !Number.isSafeInteger(row.attemptTimeoutMs) || row.attemptTimeoutMs <= 30_000
|| !surfaces.has(row.journeyKind) || JSON.stringify(row.surface) !== JSON.stringify([row.journeyKind])
|| !Array.isArray(row.coverageGaps) || !row.coverageGaps.every(text)
|| !Array.isArray(row.requiredCredentials) || !row.requiredCredentials.every(text)
|| row.externalInput !== null && (typeof row.externalInput !== 'object' || Array.isArray(row.externalInput))
|| row.installedSource !== null && !text(row.installedSource)
|| ![row.originalQuery, row.originalContext, row.expectedOutcome, row.oracleVersion, row.evidenceKind, row.evidenceReadiness].every(text)
|| !Array.isArray(row.capabilities) || row.capabilities.length === 0 || !row.capabilities.every(text)
|| !Array.isArray(row.sourceFiles) || row.sourceFiles.length === 0 || !row.sourceFiles.every(text)
|| row.checks !== null && (!Number.isSafeInteger(row.checks) || row.checks <= 0)
|| row.screenshot !== null && !/^[a-z0-9-]+\.png$/.test(row.screenshot)) throw new Error('chat_full_catalog_row_invalid');
}
for (const key of ['key', 'executionId', 'fullName']) {
if (new Set(value.cases.map(row => row[key])).size !== 244) throw new Error('chat_full_catalog_identity_conflict');
}
if (!Array.isArray(value.source?.scenarioFiles) || value.source.scenarioFiles.length !== 35
|| !/^[a-f0-9]{64}$/.test(value.source.descriptorSha256 ?? '')) throw new Error('chat_full_catalog_provenance_invalid');
return value;
}
function freeze(value) {
if (value && typeof value === 'object') {
for (const item of Object.values(value)) freeze(item);
Object.freeze(value);
}
return value;
}
validateFullCatalogManifest(manifest);
export const catalogTargetSha = manifest.targetSha;
export const catalogManifestSha256 = createHash('sha256').update(bytes).digest('hex');
export const fullCatalogPlanName = 'Chat feature catalog — OpenCode 1017 (candidate)';
export const fullCatalogCases = freeze(manifest.cases);
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs
+96
View File
@@ -0,0 +1,96 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
import { readFileSync, readdirSync } from 'node:fs';
import { createRequire } from 'node:module';
import { catalogTargetSha, catalogManifestSha256, fullCatalogPlanName, fullCatalogCases, validateFullCatalogManifest } from './chat-full-catalog-testops-manifest.mjs';
const { parseDocument } = createRequire(import.meta.url)('yaml');
const bytes = readFileSync(new URL('./chat-full-catalog-testops-manifest.json', import.meta.url));
const manifest = JSON.parse(bytes);
const row = id => fullCatalogCases.find(value => value.key === id);
test('the manifest joins all244 source YAML scenarios without importing a domain or fabricating a result', () => {
const directory = new URL('./chat-scenarios/', import.meta.url);
const scenarios = readdirSync(directory).filter(file => file.endsWith('.yaml') && file !== 'defaults.yaml').flatMap(file => {
const parsed = parseDocument(readFileSync(new URL(file, directory), 'utf8'), { uniqueKeys: true });
assert.deepEqual(parsed.errors, []);
return parsed.toJS().scenarios;
});
assert.equal(scenarios.length, 244);
assert.equal(fullCatalogCases.length, 244);
assert.deepEqual(fullCatalogCases.map(value => value.key).sort(), scenarios.map(value => value.id).sort());
for (const scenario of scenarios) {
const actual = row(scenario.id);
assert.equal(actual.originalQuery, scenario.query);
assert.equal(actual.originalContext, scenario.context);
assert.equal(actual.expectedOutcome, scenario.expected_outcome);
assert.ok(!Object.hasOwn(actual, 'status') && !Object.hasOwn(actual, 'passed'));
}
});
test('legacy aliases preserve existing TestOps identities while execution uses actual legacy task IDs', () => {
for (const task of ['create-backlog', 'delegate-announcement']) {
const actual = row(`tasks.${task}`);
assert.equal(actual.scenario, task);
assert.equal(actual.executionId, `chat-features.runner-opencode.local.${task}`);
assert.equal(actual.fullName, `paperclip.Product E2E.chat-features.tasks.${task}`);
assert.deepEqual(actual.aliases, [`paperclip.Product E2E.chat-features.${task}`]);
}
});
test('source bounds retain legacy and pilot runs, native zero turn policy and original finite attempts', () => {
for (const [id, minimum, maximum, attempt, checks] of [
['tasks.create-backlog', 1, 1, 420000, 10], ['tasks.delegate-announcement', 3, 3, 420000, 15],
['instance-operations.restart-service', 1, 1, 420000, 9], ['projects.project-defaults', 3, 3, 660000, 11],
['projects.project-work', 1, 3, 300000, 15],
]) {
const actual = row(id);
assert.deepEqual([actual.minimumRuns, actual.maximumRuns, actual.attemptTimeoutMs, actual.checks], [minimum, maximum, attempt, checks]);
assert.equal(actual.turnTimeoutMs, id.startsWith('tasks.') ? 120000 : 0);
}
assert.equal(row('access.sign-in').maximumRuns, 0);
assert.equal(row('assistant-connections.use-scoped-client').maximumRuns, 0);
assert.equal(fullCatalogCases.filter(value => value.turnTimeoutMs === 0).length, 242);
});
test('definition gaps and private prerequisites remain explicit instead of inferred runnable passes', () => {
assert.deepEqual(fullCatalogCases.filter(value => value.coverageGaps.length).map(value => value.key), [
'access.invite-teammate', 'access.manage-membership', 'activity.routine-audit', 'agents.restore-instructions', 'budgets-costs.subscription-fees',
]);
assert.equal(fullCatalogCases.filter(value => value.requiredCredentials.length).length, 18);
assert.equal(fullCatalogCases.filter(value => value.externalInput !== null).length, 37);
assert.equal(row('skills.enable-for-agent').coverageGaps.length, 0);
});
test('surface evidence declarations never manufacture a generic final screenshot for boundary cases', () => {
assert.equal(row('agent-chat.chat-disabled').screenshot, 'runtime-disabled-agent-chat.png');
for (const id of ['access.sign-in', 'onboarding.choose-first-runtime', 'onboarding.new-instance', 'onboarding.resume-wizard', 'assistant-connections.use-scoped-client']) {
assert.equal(row(id).screenshot, null);
assert.match(row(id).evidenceReadiness, /requires-final-state\.png/);
}
assert.equal(row('projects.project-work').screenshot, 'final-state.png');
assert.ok(manifest.source.requiredOriginalPassEvidence.includes('final-state.png'));
});
test('source identities and descriptor provenance are stable, deeply frozen and deterministic', () => {
assert.equal(catalogTargetSha, manifest.targetSha);
assert.equal(catalogManifestSha256, createHash('sha256').update(bytes).digest('hex'));
assert.equal(fullCatalogPlanName, 'Chat feature catalog — OpenCode 1017 (candidate)');
assert.equal(new Set(fullCatalogCases.map(value => value.fullName)).size, 244);
assert.equal(manifest.source.scenarioFiles.length, 35);
assert.equal(manifest.metadataSourceSha, 'de0941835cc8f0d01c982574ee258e110babdbeb');
assert.throws(() => { fullCatalogCases[0].coverageGaps.push('changed'); }, TypeError);
assert.throws(() => { fullCatalogCases[0].maximumRuns = 999; }, TypeError);
assert.ok(!bytes.toString().match(/sk-or-v1-[a-f0-9]{32,}/));
});
test('foreign identity, duplicated case and changed run-bound metadata are rejected by the pure loader', () => {
const copy = () => structuredClone(manifest);
const foreign = copy(); foreign.cases[0].executionId = 'chat-features.other.local.case';
assert.throws(() => validateFullCatalogManifest(foreign), /row_invalid/);
const duplicate = copy(); duplicate.cases[1] = duplicate.cases[0];
assert.throws(() => validateFullCatalogManifest(duplicate), /identity_conflict/);
const bounds = copy(); bounds.cases[0].minimumRuns = bounds.cases[0].maximumRuns + 1;
assert.throws(() => validateFullCatalogManifest(bounds), /row_invalid/);
});
tests/runner-e2e/chat-full-catalog-testops-merge.ts
+59
View File
@@ -0,0 +1,59 @@
import { lstat, mkdir, readFile, readdir, realpath, writeFile } from "node:fs/promises";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { buildRunnerCampaign } from "./history.js";
import { readPilotExport } from "./chat-completion-testops-results.mjs";
import { hash, validateSelection } from "./chat-completion-testops-contract.mjs";
import type { PilotContext, PilotSelection } from "./chat-completion-testops-contract.mjs";
import type { RunnerE2EResult } from "./types.js";
import { findSecretLeakInJsonValues } from "./redaction.js";
export async function mergeCatalogProjections(root: string, output: string, selection: PilotSelection, context: PilotContext) {
if (context.catalog !== "full") throw new Error("Full catalog context required");
const rootStat = await lstat(root);
if (!rootStat.isDirectory() || rootStat.isSymbolicLink() || await realpath(root) !== path.resolve(root)) throw new Error("Invalid projection root");
const selected = validateSelection(selection, context, selection.allureJobRunId);
const directories = await readdir(root);
if (directories.length !== selected.length || directories.some(name => !/^[A-Za-z0-9._-]+$/.test(name))) throw new Error("Incomplete projection set");
const results: RunnerE2EResult[] = [], privateEvidence: unknown[] = [], seen = new Set<string>();
for (const name of directories) {
const directory = path.join(root, name), stat = await lstat(directory);
if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Invalid projection path");
const bindingPath = path.join(directory, "binding.json"), bindingStat = await lstat(bindingPath);
if (!bindingStat.isFile() || bindingStat.isSymbolicLink() || bindingStat.nlink !== 1 || bindingStat.size > 262_144) throw new Error("Invalid projection binding");
const bytes = await readFile(bindingPath);
if (bytes.length > 262_144) throw new Error("Invalid projection binding");
const binding = JSON.parse(bytes.toString("utf8"));
if (!Array.isArray(binding.executionIds) || binding.executionIds.length !== 1 || seen.has(binding.executionIds[0])) throw new Error("Duplicate projection");
const row = selected.find(value => value.executionId === binding.executionIds[0]);
if (!row) throw new Error("Foreign projection");
const validated = await readPilotExport(directory, context, { ...selection, cases: [row] });
seen.add(row.executionId); results.push(...validated.campaign.results); privateEvidence.push(...validated.privateEvidence);
}
const ordered = selected.map(row => results.find(result => result.executionId === row.executionId)!);
const campaign = buildRunnerCampaign({ campaignId: `gha-${context.runId}-1`, generatedAt: new Date().toISOString(),
expected: selected.map(row => row.executionId), results: ordered });
if (findSecretLeakInJsonValues(campaign, [], { includeShapes: true })) throw new Error("Unsafe merged projection");
const files = new Map([
["normalized-results.json", Buffer.from(`${JSON.stringify(campaign, null, 2)}\n`)],
["junit.xml", Buffer.from(`<?xml version="1.0"?><testsuite name="Chat feature catalog" tests="${ordered.length}" failures="${ordered.filter(row => row.status !== "passed" || row.cleanup !== "passed").length}"/>\n`)],
]);
if ([...files.values()].reduce((sum, bytes) => sum + bytes.length, 0) > 32 * 1024 * 1024) throw new Error("Projection exceeds allowance");
await mkdir(output, { mode: 0o700 });
for (const [name, data] of files) await writeFile(path.join(output, name), data, { flag: "wx", mode: 0o600 });
const binding = { schema: "paperclip.chat-completion-testops-export/v1", ...context, campaignId: campaign.campaignId,
executionIds: selected.map(row => row.executionId), privateEvidence,
files: [...files].map(([name, bytes]) => ({ path: name, bytes: bytes.length, sha256: hash(bytes) })) };
await writeFile(path.join(output, "binding.json"), JSON.stringify(binding, null, 2), { flag: "wx", mode: 0o600 });
await readPilotExport(output, context, selection);
return binding;
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
const bytes = await readFile(process.env.TESTOPS_CATALOG_SELECTION!);
if (hash(bytes) !== process.env.EXPECTED_SELECTION_SHA256) throw new Error("Selection digest mismatch");
const selection = JSON.parse(bytes.toString("utf8"));
const context = { catalog: "full" as const, runId: process.env.GITHUB_RUN_ID!, attempt: process.env.GITHUB_RUN_ATTEMPT!,
workflowSha: process.env.GITHUB_SHA!, targetSha: selection.targetSha };
await mergeCatalogProjections(path.resolve("catalog-projections"), path.resolve("runner-e2e-testops"), selection, context);
}
tests/runner-e2e/chat-full-catalog-testops-reconcile.mjs
+175
View File
@@ -0,0 +1,175 @@
import { createHash } from 'node:crypto';
import { catalogTargetSha, fullCatalogCases, fullCatalogPlanName } from './chat-full-catalog-testops-manifest.mjs';
const positive = value => Number.isSafeInteger(value) && value > 0;
const fail = code => { throw new Error(`chat_full_catalog_testops_${code}`); };
const equal = (left, right) => JSON.stringify(left) === JSON.stringify(right);
const metadataFields = ['name', 'automated', 'description', 'precondition', 'expectedResult'];
const marker = (operation, key) => `${operation}_${createHash('sha256').update(key).digest('hex').replace(/[0-9a-f]/g, value => String.fromCharCode(97 + parseInt(value, 16)))}`;
function manifestIndex() {
if (!Array.isArray(fullCatalogCases) || fullCatalogCases.length !== 244
|| !/^[a-f0-9]{40}$/.test(catalogTargetSha) || !fullCatalogPlanName) fail('manifest');
const names = new Map(), keys = new Set();
for (const row of fullCatalogCases) {
if (!row.key || keys.has(row.key) || row.key !== row.canonicalScenario || !row.fullName
|| !Array.isArray(row.aliases) || !row.name || !row.originalQuery || !row.originalContext || !row.expectedOutcome
|| !Array.isArray(row.capabilities) || !Array.isArray(row.surface)
|| !Array.isArray(row.coverageGaps) || !Array.isArray(row.requiredCredentials)) fail('manifest');
keys.add(row.key);
for (const name of [row.fullName, ...row.aliases]) {
if (typeof name !== 'string' || !name || names.has(name)) fail('manifest_alias');
names.set(name, row);
}
}
return names;
}
function desiredMetadata(row) {
const gaps = row.coverageGaps.length ? row.coverageGaps.join('\n') : 'No coverage gap is declared in the source metadata; live qualification is separate.';
const requirements = [
`Full-catalog lane targets candidate ${catalogTargetSha}; other lanes keep their own actual source; target is recorded per result. This registration does not qualify current master or record a PASS.`,
`Assumed context:\n${row.originalContext}`,
`Required credential names: ${row.requiredCredentials.join(', ') || 'None declared for this case; the controller profile credential is separate.'}`,
`External input contract: ${row.externalInput == null ? 'None declared.' : JSON.stringify(row.externalInput)}`,
`Installed-source requirement: ${row.installedSource ?? 'None declared.'}`,
`Source evidence kind: ${row.evidenceKind}`,
`Source screenshot carrier: ${row.screenshot ?? 'None declared.'}`,
`Source evidence readiness: ${row.evidenceReadiness}`,
`Coverage gaps (the original oracle remains unchanged):\n${gaps}`,
];
return {
name: row.name,
automated: true,
description: [
`Canonical chat scenario: ${row.canonicalScenario}`,
`Full-catalog lane targets candidate ${catalogTargetSha}; other lanes keep their own actual source; target is recorded per result. Executable registration is not a result or a qualification claim.`,
`Human request:\n${row.originalQuery}`,
`Capabilities: ${row.capabilities.join(', ') || 'None declared.'}`,
`Surfaces: ${row.surface.join(', ') || 'None declared.'}`,
`Native OpenCode / OpenRouter DeepSeek1017. Original source bounds and evidence gates apply.`,
`Accounted work runs: minimum ${row.minimumRuns ?? 'unknown'}, maximum ${row.maximumRuns ?? 'unknown'}. Turn limit: ${row.turnTimeoutMs} ms. Attempt limit: ${row.attemptTimeoutMs} ms.`,
`Coverage gaps:\n${gaps}`,
].join('\n\n'),
precondition: requirements.join('\n\n'),
expectedResult: row.expectedOutcome,
};
}
function caseMetadata(actual) {
return Object.fromEntries(metadataFields.map(field => [field, actual[field]]));
}
function assertCase(actual, row, caseId, registeredFullName) {
if (actual?.id !== caseId || actual.projectId !== 1 || actual.deleted === true
|| typeof actual.automated !== 'boolean' || actual.fullName !== registeredFullName
|| ![row.fullName, ...row.aliases].includes(actual.fullName)) fail('case_identity');
}
function assertSelection(plan, allowedIds, exact = false) {
const selection = plan.treeSelection;
if (!positive(plan.id) || plan.projectId !== 1 || plan.name !== fullCatalogPlanName
|| !selection || selection.inverted !== false || !Array.isArray(selection.leafsInclude)
|| selection.leafsInclude.some(value => !positive(value) || !allowedIds.has(value))
|| new Set(selection.leafsInclude).size !== selection.leafsInclude.length
|| !Number.isSafeInteger(plan.testCasesCount) || plan.testCasesCount !== selection.leafsInclude.length
|| ['groupsInclude', 'groupsExclude', 'leafsExclude', 'path'].some(key => selection[key] != null
&& (!Array.isArray(selection[key]) || selection[key].length !== 0))
|| plan.baseRql != null && plan.baseRql !== ''
|| exact && selection.leafsInclude.length !== allowedIds.size) fail('plan_scope');
}
export async function inspectFullCatalog(api) {
const names = manifestIndex(), seen = new Set(), found = new Map();
let totalPages, totalElements;
for (let page = 0; page < 10; page++) {
const listed = await api(`/api/testcase?${new URLSearchParams({ projectId: '1', page: String(page), size: '100', sort: 'id,asc' })}`);
if (!Array.isArray(listed?.content) || listed.content.length > 100 || listed.number !== page
|| !Number.isSafeInteger(listed.totalPages) || listed.totalPages < 0 || listed.totalPages > 10
|| !Number.isSafeInteger(listed.totalElements) || listed.totalElements < 0 || listed.totalElements > 1000
|| listed.totalPages !== Math.ceil(listed.totalElements / 100)
|| listed.content.length !== Math.min(100, Math.max(0, listed.totalElements - page * 100))
|| totalPages !== undefined && (totalPages !== listed.totalPages || totalElements !== listed.totalElements)) fail('catalog_page');
totalPages = listed.totalPages; totalElements = listed.totalElements;
const ids = listed.content.map(value => {
if (!positive(value.id) || seen.has(value.id)) fail('catalog_identity');
seen.add(value.id); return value.id;
});
for (let offset = 0; offset < ids.length; offset += 4) {
const batch = ids.slice(offset, offset + 4);
const actuals = await Promise.all(batch.map(caseId => api(`/api/testcase/${caseId}`)));
for (const [index, actual] of actuals.entries()) {
if (actual?.id !== batch[index] || actual.projectId !== 1
|| actual.fullName != null && typeof actual.fullName !== 'string') fail('catalog_case');
const row = names.get(actual.fullName);
if (!row) continue;
assertCase(actual, row, actual.id, actual.fullName);
if (found.has(row.key)) fail('case_alias_conflict');
found.set(row.key, { ...row, caseId: actual.id, registeredFullName: actual.fullName, currentMetadata: caseMetadata(actual) });
}
}
if (page + 1 >= totalPages) break;
}
if (seen.size !== totalElements) fail('catalog_incomplete');
const listedPlans = await api(`/api/testplan?${new URLSearchParams({ projectId: '1', name: fullCatalogPlanName, page: '0', size: '100' })}`);
if (!Array.isArray(listedPlans?.content) || listedPlans.content.length > 100
|| !Number.isSafeInteger(listedPlans.totalPages) || listedPlans.totalPages < 0 || listedPlans.totalPages > 1
|| !Number.isSafeInteger(listedPlans.totalElements) || listedPlans.totalElements !== listedPlans.content.length) fail('plan_page');
const plans = listedPlans.content.filter(plan => plan.name === fullCatalogPlanName);
if (plans.length > 1) fail('plan_conflict');
const cases = fullCatalogCases.flatMap(row => found.has(row.key) ? [found.get(row.key)] : []);
if (plans.length) {
if (!positive(plans[0].id) || plans[0].projectId !== 1) fail('plan_identity');
const actual = await api(`/api/testplan/${plans[0].id}`);
if (actual?.id !== plans[0].id) fail('plan_identity');
assertSelection(actual, new Set(cases.map(row => row.caseId)));
plans[0] = actual;
}
return { projectId: 1, catalogCount: totalElements, cases, plans };
}
export async function reconcileFullCatalog(api, markOnce) {
if (typeof markOnce !== 'function') fail('mutation_marker');
const inspected = await inspectFullCatalog(api), existing = new Map(inspected.cases.map(row => [row.key, row]));
const reconciled = [];
for (const row of fullCatalogCases) {
const wanted = desiredMetadata(row);
let bound = existing.get(row.key);
if (!bound) {
await markOnce(marker('create_case', row.key));
const created = await api('/api/testcase', 'POST', { projectId: 1, fullName: row.fullName, ...wanted });
if (!positive(created?.id) || reconciled.some(value => value.caseId === created.id)) fail('created_case');
const actual = await api(`/api/testcase/${created.id}`);
assertCase(actual, row, created.id, row.fullName);
if (!equal(caseMetadata(actual), wanted)) fail('case_metadata');
bound = { ...row, caseId: actual.id, registeredFullName: actual.fullName };
} else if (!equal(bound.currentMetadata, wanted)) {
await markOnce(marker('patch_case', row.key));
await api(`/api/testcase/${bound.caseId}`, 'PATCH', wanted);
const actual = await api(`/api/testcase/${bound.caseId}`);
assertCase(actual, row, bound.caseId, bound.registeredFullName);
if (!equal(caseMetadata(actual), wanted)) fail('case_metadata');
}
reconciled.push({ ...row, caseId: bound.caseId, registeredFullName: bound.registeredFullName });
}
const checked = await inspectFullCatalog(api);
if (checked.cases.length !== fullCatalogCases.length
|| checked.cases.some((row, index) => row.caseId !== reconciled[index].caseId
|| row.registeredFullName !== reconciled[index].registeredFullName
|| !equal(row.currentMetadata, desiredMetadata(fullCatalogCases[index])))) fail('reconciliation_changed');
const selection = { inverted: false, leafsInclude: reconciled.map(row => row.caseId), groupsInclude: [], groupsExclude: [], leafsExclude: [] };
let plan = checked.plans[0];
if (!plan) {
await markOnce('create_plan');
const created = await api('/api/testplan', 'POST', { projectId: 1, name: fullCatalogPlanName, treeSelection: selection });
if (!positive(created?.id)) fail('created_plan');
plan = await api(`/api/testplan/${created.id}`);
if (plan?.id !== created.id) fail('created_plan');
} else if (!equal([...plan.treeSelection.leafsInclude].sort((a, b) => a - b), [...selection.leafsInclude].sort((a, b) => a - b))) {
await markOnce('patch_plan');
await api(`/api/testplan/${plan.id}`, 'PATCH', { treeSelection: selection });
plan = await api(`/api/testplan/${plan.id}`);
}
assertSelection(plan, new Set(selection.leafsInclude), true);
return { planId: plan.id, cases: reconciled };
}
tests/runner-e2e/chat-full-catalog-testops-reconcile.test.mjs
+207
View File
@@ -0,0 +1,207 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { fullCatalogCases, fullCatalogPlanName, catalogTargetSha } from './chat-full-catalog-testops-manifest.mjs';
import { inspectFullCatalog, reconcileFullCatalog } from './chat-full-catalog-testops-reconcile.mjs';
const clone = value => structuredClone(value);
const backlog = fullCatalogCases.find(row => row.canonicalScenario === 'tasks.create-backlog');
const pilotKeys = ['instance-operations.restart-service', 'projects.project-defaults', 'projects.project-work'];
function storedCase(row, caseId, fullName = row.fullName) {
return { id: caseId, projectId: 1, fullName, automated: true, deleted: false, name: row.name,
description: 'Retained prior description', precondition: '', expectedResult: '', retainedHistory: ['original measurement'] };
}
function fixture({ existing = true, partialPlan = false, transform = value => value, uncertain = null } = {}) {
const cases = new Map(), plans = new Map(), calls = [], claimed = new Set();
const unrelated = { id: 2, projectId: 1, fullName: 'unrelated.original.case', automated: true, name: 'Unrelated', retainedHistory: ['unchanged'] };
cases.set(unrelated.id, clone(unrelated));
if (existing) {
cases.set(388, storedCase(backlog, 388, backlog.aliases[0]));
pilotKeys.forEach((key, index) => cases.set(500 + index, storedCase(fullCatalogCases.find(row => row.key === key), 500 + index)));
}
plans.set(36, { id: 36, projectId: 1, name: 'Chat feature evals', testCasesCount: 1, treeSelection: { inverted: false, leafsInclude: [388] } });
plans.set(40, { id: 40, projectId: 1, name: 'Chat completion pilot — OpenCode 1017', testCasesCount: 3,
treeSelection: { inverted: false, leafsInclude: [500, 501, 502] } });
if (partialPlan) plans.set(90, { id: 90, projectId: 1, name: fullCatalogPlanName, testCasesCount: 2,
treeSelection: { inverted: false, leafsInclude: [388, 500] } });
let nextId = 10000, uncertainUsed = false;
const api = async (route, method = 'GET', body) => {
calls.push({ route, method, ...(body === undefined ? {} : { body: clone(body) }) });
const url = new URL(route, 'https://mock.invalid');
let value;
if (route.startsWith('/api/testcase?')) {
const rows = [...cases.values()].sort((a, b) => a.id - b.id), page = Number(url.searchParams.get('page'));
value = { content: rows.slice(page * 100, (page + 1) * 100).map(row => ({ id: row.id })),
number: page, totalPages: Math.ceil(rows.length / 100), totalElements: rows.length };
} else if (/^\/api\/testcase\/\d+$/.test(route)) {
const caseId = Number(route.split('/').at(-1));
if (method === 'PATCH') Object.assign(cases.get(caseId), clone(body));
else assert.equal(method, 'GET');
value = cases.get(caseId);
} else if (route === '/api/testcase' && method === 'POST') {
if (uncertain === 'before' && !uncertainUsed) { uncertainUsed = true; throw new Error('uncertain transport'); }
value = { ...clone(body), id: nextId++, deleted: false };
cases.set(value.id, clone(value));
if (uncertain === 'after' && !uncertainUsed) { uncertainUsed = true; throw new Error('uncertain transport'); }
} else if (route.startsWith('/api/testplan?')) {
const rows = [...plans.values()].filter(row => row.name.includes(url.searchParams.get('name')));
value = { content: rows, totalElements: rows.length, totalPages: rows.length ? 1 : 0 };
} else if (/^\/api\/testplan\/\d+$/.test(route)) {
const planId = Number(route.split('/').at(-1));
if (method === 'PATCH') {
Object.assign(plans.get(planId), clone(body));
plans.get(planId).testCasesCount = body.treeSelection.leafsInclude.length;
} else assert.equal(method, 'GET');
value = plans.get(planId);
} else if (route === '/api/testplan' && method === 'POST') {
value = { ...clone(body), id: 90, testCasesCount: body.treeSelection.leafsInclude.length };
plans.set(value.id, clone(value));
} else assert.fail(`Unexpected ${method} ${route}`);
return clone(transform(clone(value), { route, method, body, cases, plans, calls }));
};
const markOnce = async stage => {
assert.match(stage, /^[a-z_]{1,80}$/);
if (claimed.has(stage)) throw new Error('already attempted');
claimed.add(stage); calls.push({ marker: stage });
};
return { api, markOnce, cases, plans, calls, claimed, unrelated };
}
const writes = f => f.calls.filter(row => row.method && row.method !== 'GET');
test('reconciles the actual 244-row manifest while retaining legacy388, pilot identities, history and unrelated plans', async () => {
const f = fixture(), oldPlans = clone([...f.plans.values()]);
const receipt = await reconcileFullCatalog(f.api, f.markOnce);
assert.equal(receipt.cases.length, 244); assert.equal(receipt.planId, 90);
assert.equal(new Set(receipt.cases.map(row => row.caseId)).size, 244);
const boundBacklog = receipt.cases.find(row => row.key === backlog.key);
assert.equal(boundBacklog.caseId, 388); assert.equal(boundBacklog.fullName, backlog.fullName);
assert.equal(boundBacklog.registeredFullName, backlog.aliases[0]);
assert.equal(f.cases.get(388).fullName, backlog.aliases[0]);
assert.deepEqual(f.cases.get(388).retainedHistory, ['original measurement']);
assert.deepEqual(f.cases.get(2), f.unrelated);
oldPlans.forEach(plan => assert.deepEqual(f.plans.get(plan.id), plan));
assert.equal(writes(f).filter(row => row.route === '/api/testcase').length, 240);
for (const bound of receipt.cases) {
const actual = f.cases.get(bound.caseId);
assert.equal(actual.automated, true); assert.equal(actual.expectedResult, bound.expectedOutcome);
assert.ok(actual.description.includes(bound.originalQuery));
assert.ok(actual.precondition.includes(bound.originalContext));
assert.ok(actual.description.includes(catalogTargetSha));
for (const field of ['description', 'precondition']) {
assert.ok(actual[field].includes(`Full-catalog lane targets candidate ${catalogTargetSha}; other lanes keep their own actual source; target is recorded per result.`));
}
assert.ok(actual.precondition.includes(`Source evidence kind: ${bound.evidenceKind}`));
assert.ok(actual.precondition.includes(`Source screenshot carrier: ${bound.screenshot ?? 'None declared.'}`));
assert.ok(actual.precondition.includes(`Source evidence readiness: ${bound.evidenceReadiness}`));
bound.coverageGaps.forEach(gap => assert.ok(actual.description.includes(gap)));
bound.requiredCredentials.forEach(name => assert.ok(actual.precondition.includes(name)));
assert.equal(Object.hasOwn(actual, 'statusId'), false);
}
assert.deepEqual(f.plans.get(90).treeSelection.leafsInclude, receipt.cases.map(row => row.caseId));
for (const [index, call] of f.calls.entries()) if (call.method && call.method !== 'GET') {
assert.ok(f.calls[index - 1].marker);
if (call.method === 'PATCH' && call.route.startsWith('/api/testcase/')) {
assert.deepEqual(Object.keys(call.body).sort(), ['automated', 'description', 'expectedResult', 'name', 'precondition']);
}
}
});
test('a second full reconciliation performs no writes or new mutation claims', async () => {
const f = fixture();
const first = await reconcileFullCatalog(f.api, f.markOnce), count = writes(f).length, claims = f.claimed.size;
const second = await reconcileFullCatalog(f.api, f.markOnce);
assert.deepEqual(second, first); assert.equal(writes(f).length, count); assert.equal(f.claimed.size, claims);
});
test('resumes an owned partial plan by extending only its explicit known-case selection', async () => {
const f = fixture({ partialPlan: true });
const value = await reconcileFullCatalog(f.api, f.markOnce);
assert.equal(value.planId, 90);
assert.equal(writes(f).filter(row => row.route === '/api/testplan').length, 0);
assert.deepEqual(writes(f).filter(row => row.route === '/api/testplan/90').map(row => Object.keys(row.body)), [['treeSelection']]);
assert.equal(f.plans.get(90).testCasesCount, 244);
});
for (const kind of ['alias', 'canonical']) {
test(`rejects duplicate ${kind} case identities before any mutation`, async () => {
const f = fixture();
const row = kind === 'alias' ? backlog : fullCatalogCases.find(value => !value.aliases.length);
if (kind === 'canonical') f.cases.set(600, storedCase(row, 600));
f.cases.set(601, storedCase(row, 601));
await assert.rejects(reconcileFullCatalog(f.api, f.markOnce), /case_alias_conflict/);
assert.equal(writes(f).length, 0); assert.equal(f.claimed.size, 0);
});
}
test('a matching-name plan containing a foreign case is not overwritten', async () => {
const f = fixture({ partialPlan: true });
f.plans.get(90).treeSelection.leafsInclude = [2]; f.plans.get(90).testCasesCount = 1;
await assert.rejects(reconcileFullCatalog(f.api, f.markOnce), /plan_scope/);
assert.equal(writes(f).length, 0);
});
test('duplicate dedicated plan names stop before case metadata patches', async () => {
const f = fixture({ partialPlan: true }); f.plans.set(91, { ...clone(f.plans.get(90)), id: 91 });
await assert.rejects(reconcileFullCatalog(f.api, f.markOnce), /plan_conflict/);
assert.equal(writes(f).length, 0);
});
test('per-ID project mismatch stops before writes even when list IDs look valid', async () => {
const f = fixture({ transform: (value, { route }) => route === '/api/testcase/388' ? { ...value, projectId: 2 } : value });
await assert.rejects(reconcileFullCatalog(f.api, f.markOnce), /catalog_case/);
assert.equal(writes(f).length, 0);
});
for (const mode of ['changed_total', 'duplicate_id', 'short_page']) {
test(`complete pagination rejects ${mode} without writing`, async () => {
const f = fixture({ transform: (value, { route }) => {
if (!route.startsWith('/api/testcase?')) return value;
if (mode === 'changed_total' && value.number === 1) { value.totalElements++; }
if (mode === 'duplicate_id' && value.number === 1) value.content[0].id = 2;
if (mode === 'short_page' && value.number === 0) value.content.pop();
return value;
} });
for (let i = 0; i < 110; i++) f.cases.set(2000 + i, { ...clone(f.unrelated), id: 2000 + i, fullName: `foreign.${i}` });
await assert.rejects(reconcileFullCatalog(f.api, f.markOnce), /catalog_page|catalog_identity/);
assert.equal(writes(f).length, 0);
});
}
test('an uncertain unapplied case POST is not automatically resent with the same marker', async () => {
const f = fixture({ existing: false, uncertain: 'before' });
await assert.rejects(reconcileFullCatalog(f.api, f.markOnce), /uncertain transport/);
await assert.rejects(reconcileFullCatalog(f.api, f.markOnce), /already attempted/);
assert.equal(writes(f).length, 1); assert.equal(f.plans.has(90), false);
});
test('resumption reuses a case durably created before an uncertain response rather than creating it twice', async () => {
const f = fixture({ existing: false, uncertain: 'after' });
await assert.rejects(reconcileFullCatalog(f.api, f.markOnce), /uncertain transport/);
const createdId = [...f.cases.keys()].find(id => id !== 2);
const value = await reconcileFullCatalog(f.api, f.markOnce);
assert.equal(value.cases[0].caseId, createdId);
assert.equal(writes(f).filter(row => row.route === '/api/testcase').length, 244);
assert.equal(new Set(value.cases.map(row => row.caseId)).size, 244);
});
test('a duplicate appearing after case updates prevents final plan admission', async () => {
let injected = false;
const f = fixture({ transform: (value, { route, cases, calls }) => {
if (!injected && route.startsWith('/api/testcase?') && calls.some(row => row.method === 'POST')) {
injected = true; cases.set(9000, storedCase(backlog, 9000));
const rows = [...cases.values()].sort((a, b) => a.id - b.id);
return { ...value, content: rows.slice(0, 100).map(row => ({ id: row.id })),
totalElements: rows.length, totalPages: Math.ceil(rows.length / 100) };
}
return value;
} });
await assert.rejects(reconcileFullCatalog(f.api, f.markOnce), /case_alias_conflict/);
assert.equal(writes(f).filter(row => row.route.startsWith('/api/testplan')).length, 0);
});
test('inspection alone reports known bindings without mutation markers or writes', async () => {
const f = fixture();
const value = await inspectFullCatalog(f.api);
assert.equal(value.cases.length, 4); assert.equal(value.plans.length, 0);
assert.equal(value.cases.find(row => row.caseId === 388).registeredFullName, backlog.aliases[0]);
assert.equal(writes(f).length, 0);
});
tests/runner-e2e/run-unit.mjs
+2 -1
View File
@@ -23,7 +23,8 @@ for (const [name, command] of [
["Node tests", ["--test", "tests/runner-e2e/artifact-budget.test.mjs", "tests/runner-e2e/native-completion-checks.test.mjs",
"tests/runner-e2e/native-completion-git-source.test.mjs", "tests/runner-e2e/native-completion-source-contract.test.mjs",
"tests/runner-e2e/chat-feature-testops-api.test.mjs", "tests/runner-e2e/chat-feature-testops-cli.test.mjs", "tests/runner-e2e/chat-feature-testops-results.test.mjs", "tests/runner-e2e/testops-chat-reporter.test.mjs", "tests/runner-e2e/chat-feature-testops-retained-verification.node-test.mjs",
"tests/runner-e2e/chat-completion-testops-api.test.mjs", "tests/runner-e2e/chat-completion-testops-contract.test.mjs"]],
"tests/runner-e2e/chat-completion-testops-api.test.mjs", "tests/runner-e2e/chat-completion-testops-contract.test.mjs",
"tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs", "tests/runner-e2e/chat-full-catalog-testops-reconcile.test.mjs", "tests/runner-e2e/chat-full-catalog-testops-adapter.test.mjs"]],
]) {
const result = spawnSync(process.execPath, command, { cwd: root, env, stdio: "inherit" });
if (result.error || result.signal) {
tests/runner-e2e/vitest.config.ts
+1 -1
View File
@@ -7,7 +7,7 @@ export default defineConfig({
globalSetup: ["./verified-provider-bundles.setup.mjs"],
include: ["**/*.test.ts", "**/*.test.mjs"],
// These use node:test and run separately in test:e2e:runner:unit.
exclude: [...configDefaults.exclude, "native-completion-{checks,git-source,source-contract}.test.mjs", "artifact-budget.test.mjs", "chat-feature-testops-api.test.mjs", "chat-feature-testops-cli.test.mjs", "chat-feature-testops-results.test.mjs", "testops-chat-reporter.test.mjs", "chat-completion-testops-{api,contract}.test.mjs"],
exclude: [...configDefaults.exclude, "native-completion-{checks,git-source,source-contract}.test.mjs", "artifact-budget.test.mjs", "chat-feature-testops-api.test.mjs", "chat-feature-testops-cli.test.mjs", "chat-feature-testops-results.test.mjs", "testops-chat-reporter.test.mjs", "chat-completion-testops-{api,contract}.test.mjs", "chat-full-catalog-testops-*.test.mjs"],
testTimeout: 30_000,
},
});