mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 06:04:11 +01:00
test(evals): add a fixed TestOps AWS chat feature route (#18322)
## Thinking Path > - Paperclip lets people manage AI agents and inspect their saved work. > - Chat feature evaluations test a real request against saved tasks, audit records, links and reload behavior. > - The current suite has Codex and OpenCode cells but no dedicated runnable DeepSeek 1017 backlog plan in TestOps. > - TestOps must select the real case before the workflow can access a model key. > - This PR adds one fixed, default-branch AWS route for native OpenCode and DeepSeek 1017. > - The route keeps original evidence private and sends only a verified result and approved screenshots to TestOps. ## Linked Issues or Issue Description **Subsystem affected** Product E2E evaluation definitions and GitHub Actions delivery. **Problem or motivation** A dedicated TestOps plan cannot run the existing backlog chat journey with native OpenCode. A result must prove saved work and cleanup rather than report that a model call completed. **Proposed solution** Add one OpenCode backlog cell and one fixed public workflow. TestOps selection validation runs in a token-isolated job before builds or inference. The workflow retains private originals and reports the bound outcome to the selected TestOps case. **Alternatives considered** A private-repository wrapper would need a new cross-repository GitHub write credential. The new route uses the existing TestOps GitHub integration instead. Existing manual candidate approval paths remain separate. **Roadmap alignment** This extends the existing feature evaluation infrastructure. It changes no production feature, global runtime default or model default. Current main #18319 supplies the v2 grader, delegation queue accounting and final screenshot capture; these improvements are preserved. Related product work: #16814 and #17126 concern backlog handling. This PR changes neither product implementation and does not claim those changes are required. ## What Changed - Add exactly one native OpenCode backlog cell with DeepSeek 1017 under the separate `runner-opencode-deepseek-1017` profile. Preserve all four existing Codex and production OpenCode cells and their models. - Add a default-branch-only workflow with fixed evaluation, zero-result registration and one plan-launch modes. Require GitHub actor checks and authenticated singleton selection before inference. - Add a separate shipped-main evidence authority. Reuse the existing strict private storage, scanners, inventories and scoped AWS sessions. - Bind sanitized results, generated JUnit and producer-approved screenshots to the exact source, run, attempt and case. Keep failed grades and cleanup failures failed. - Add TestOps API, CLI and result adapters with pure controls and normal unit-test wiring. Keep the reporting token out of the evaluated process. - Verify the current server case result and attachment metadata after upload, without masking an earlier failed upload or evaluation. - Apply only the existing reviewed lock correction, verify unchanged tracked dependency inputs and bind the lock receipt across the compiled-output handoff before frozen installation. - Bind the initial administrator launch to one approved workflow run number and live GitHub source/actor metadata before TestOps authentication. Use bounded REST case reconciliation and only the reviewed existing GitHub integration. - Make two existing CI tests observe their intended state deterministically: a fake clock for copy-feedback lifetime and a completed-pass barrier for retry-cancellation batches. Production behavior and the original assertions remain unchanged. The two test-only fixes are intentionally included because their observation boundaries blocked this delivery’s required normal CI. They preserve every original assertion and change no production behavior. The original failed runs remain recorded, and the exact causes of those observations remain unknown. ## Verification - 157 focused profile, catalog, oracle, filesystem admission, export, private-path and workflow controls passed on the current-main composition. The existing stock OpenCode models and the new exact DeepSeek 1017 selection are checked separately. - 44 API, CLI, filesystem and post-upload readback controls passed, including foreign selection, changed hashes, unsafe files, failed outcomes, expired readback and no-resend behavior. - The new exact private session and trace layout controls produced four genuine parent failures and pass with the selected-path extension. The frozen 244-case candidate approval registry remains unchanged. - The lock guard has seven filesystem controls plus seven existing canonical-owner controls. The unrestricted old workflow failed its actual policy calibration; the corrected frozen-install route passed. - Eval-owning TypeScript checks, workflow lint and Git whitespace checks passed. The separate UI test correction was checked with the focused jsdom unit suite. The server test barrier has pure wrapper and real scheduler controls; its DB-backed route was not run locally. No additional local typecheck/build was run for these test-only changes. - Full native builds, provider execution, app/database behavior and TestOps registration are reserved for GitHub Actions/AWS. They have not run locally. - The initial normal CI run failed the new-workflow policy allowlist and a heartbeat scheduling assertion. The policy fix has an actual parent red/current green control and rejects five unsafe route variants. Heartbeat source is unchanged and its failure cause remains unknown. The original run is retained. - The launch-fence actual caller fails against the published parent and passes the five own-run contradiction controls before authentication. - The prior `a5aa359d` normal CI completed successfully. The complete `f9e1f289` run passed all leaf jobs except an existing copy-feedback UI assertion, plus its dependent verification gate. The four owning UI files were unchanged from the earlier passing head and actual tested merge; the exact scheduling cause remains unknown. - The correction changes only that test to actual React `act` and a scoped fake clock. All eight component tests pass; early-hide and never-clear source variants each fail the intended lifetime assertion. Production clipboard/card behavior is unchanged. - The `2f27f3fc` run passed the corrected UI shard and every leaf job except the existing retry-cancellation intermediate-count assertion, plus its dependent verification gate. The seven relevant owners match the earlier passing run and historical source comparison; the exact CI scheduling cause remains unknown. - The test-only barrier observes a persisted completed worker pass before asserting the unchanged 20-item cap and two-company fairness, then releases it for the original 29-item drain and 180-second idle check. Two extracted-wrapper controls and three actual scheduler controls pass. The real DB-backed test awaits AWS qualification. - On `836e80ac`, both repaired test shards passed in AWS. The original attempt retained a separate E2E cleanup HTTP 500 from a PostgreSQL lock timeout and one runner shutdown. No production repair or precise contention/shutdown cause is claimed. - Exactly one targeted retry of E2E shard 3 and its aggregate gate passed in attempt 2; exactly one recovery of the interrupted server shard and its verification gate passed in attempt 3. Other passed jobs were carried forward. The final normal run is successful. No whole-workflow retry or evaluation/model retry was launched. - Both literal required checks (`ci / verify` and `ci / e2e`) are successful on the exact final source `836e80ac`. Greptile is 5/5 and no review threads remain open. Actual TestOps registration and the OpenCode/DeepSeek feature result remain unqualified until their separate executions. ## Risks - Actual zero-result registration, tenant API shapes and public-repository dispatch access must be checked before the first evaluation. Missing or ambiguous metadata stops the route. - The real OpenCode/DeepSeek backlog result and Allure outcome remain unqualified until that run. No passing result is seeded. - One human query, one accounted work run, 120-second turn, 420-second attempt, 300-second settled-state bound and zero retries remain unchanged. Setup usage is outside work accounting and stays unknown. - The reporting token was configured through the separately reviewed protected credential bootstrap. It is unavailable to AWS evaluation jobs. - Metadata reconciliation is capped at ten pages, 1,000 cases and four parallel detail reads. Missing or inconsistent metadata stops before inference. The initial administrator launch counter must be created once after registration; it does not restrict normal TestOps Run actions. - Existing manual workflows, historical evaluation grades and candidate approval records remain unchanged. The original CI failures are preserved; the test corrections do not establish retrospective causes. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning and tool use. The runtime did not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
a4308b9791
commit
9e1735ec17
34 files changed
+2802
-74
No files matched your search
@@ -0,0 +1,448 @@
|
||||
name: Chat feature TestOps
|
||||
run-name: Chat feature TestOps / ${{ inputs.operation || 'evaluate' }} / ${{ inputs.ALLURE_JOB_RUN_ID || github.run_id }}
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
operation:
|
||||
type: choice
|
||||
options: [evaluate, bootstrap, launch-plan]
|
||||
default: evaluate
|
||||
description: Fixed evaluation, zero-result registration, or one TestOps plan launch
|
||||
ALLURE_JOB_RUN_ID:
|
||||
type: string
|
||||
required: false
|
||||
description: TestOps supplies its job-run identity; no test/source selector is accepted
|
||||
ALLURE_USERNAME:
|
||||
type: string
|
||||
required: false
|
||||
description: Reserved TestOps service metadata; never used as authorization
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: chat-feature-testops
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
authorize:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
outputs:
|
||||
sha: ${{ steps.guard.outputs.sha }}
|
||||
operation: ${{ steps.guard.outputs.operation }}
|
||||
triggering_actor_id: ${{ steps.guard.outputs.triggering_actor_id }}
|
||||
runner: runs-on/fleet=paperclip-public-pr-x64/env=public-ci
|
||||
steps:
|
||||
- &checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
||||
with:
|
||||
ref: ${{ github.sha }}
|
||||
persist-credentials: false
|
||||
- &node
|
||||
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
||||
with:
|
||||
node-version: 24
|
||||
- &guard
|
||||
name: Verify fixed default-source dispatch and both actor identities
|
||||
id: guard
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
NODE_OPTIONS: ''
|
||||
NODE_PATH: ''
|
||||
run: node tests/runner-e2e/testops-chat-dispatch.ts
|
||||
|
||||
select:
|
||||
name: Validate TestOps selection or register zero-result setup
|
||||
needs: authorize
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
environment: runner-e2e-testops-reporting
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
outputs:
|
||||
artifact_id: ${{ steps.selection_upload.outputs.artifact-id }}
|
||||
sha256: ${{ steps.selection.outputs.sha256 }}
|
||||
steps:
|
||||
- *checkout
|
||||
- *node
|
||||
- *guard
|
||||
- &allure
|
||||
if: needs.authorize.outputs.operation != 'launch-plan'
|
||||
uses: allure-framework/setup-allurectl@2d89ee0b7ba2f59dbe9849b6df5763cab78e14db # v1
|
||||
with:
|
||||
allurectl-version: 2.23.1
|
||||
allure-endpoint: https://paperclip.testops.cloud
|
||||
allure-project-id: '1'
|
||||
allure-token: ${{ secrets.ALLURE_TOKEN }}
|
||||
- name: Register the public job without an evaluation or a synthetic result
|
||||
if: needs.authorize.outputs.operation == 'bootstrap'
|
||||
env:
|
||||
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
|
||||
ALLURE_ENDPOINT: https://paperclip.testops.cloud
|
||||
ALLURE_PROJECT_ID: '1'
|
||||
CHAT_TESTOPS_GITHUB_WORKFLOW_ID: ${{ vars.CHAT_TESTOPS_GITHUB_WORKFLOW_ID }}
|
||||
ALLURE_RESULTS: ${{ runner.temp }}/chat-testops-bootstrap-results
|
||||
run: |
|
||||
mkdir -m 700 "$ALLURE_RESULTS"
|
||||
allurectl watch -- node tests/runner-e2e/testops-chat-reporter.mjs bootstrap
|
||||
- name: Validate the one real TestOps selection before any build or provider access
|
||||
id: selection
|
||||
if: needs.authorize.outputs.operation == 'evaluate'
|
||||
env:
|
||||
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
|
||||
ALLURE_ENDPOINT: https://paperclip.testops.cloud
|
||||
ALLURE_PROJECT_ID: '1'
|
||||
ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
|
||||
CHAT_TESTOPS_CASE_ID: ${{ vars.CHAT_TESTOPS_CASE_ID }}
|
||||
CHAT_TESTOPS_PLAN_ID: ${{ vars.CHAT_TESTOPS_PLAN_ID }}
|
||||
CHAT_TESTOPS_JOB_ID: ${{ vars.CHAT_TESTOPS_JOB_ID }}
|
||||
CHAT_TESTOPS_GITHUB_WORKFLOW_ID: ${{ vars.CHAT_TESTOPS_GITHUB_WORKFLOW_ID }}
|
||||
ALLURE_RESULTS: ${{ runner.temp }}/chat-testops-selection-results
|
||||
run: |
|
||||
mkdir -m 700 "$ALLURE_RESULTS"
|
||||
allurectl watch -- node tests/runner-e2e/testops-chat-reporter.mjs selection
|
||||
- name: Start exactly the validated TestOps plan and exit without waiting for its child
|
||||
if: needs.authorize.outputs.operation == 'launch-plan'
|
||||
env:
|
||||
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
|
||||
ALLURE_ENDPOINT: https://paperclip.testops.cloud
|
||||
ALLURE_PROJECT_ID: '1'
|
||||
CHAT_TESTOPS_CASE_ID: ${{ vars.CHAT_TESTOPS_CASE_ID }}
|
||||
CHAT_TESTOPS_PLAN_ID: ${{ vars.CHAT_TESTOPS_PLAN_ID }}
|
||||
CHAT_TESTOPS_JOB_ID: ${{ vars.CHAT_TESTOPS_JOB_ID }}
|
||||
CHAT_TESTOPS_GITHUB_WORKFLOW_ID: ${{ vars.CHAT_TESTOPS_GITHUB_WORKFLOW_ID }}
|
||||
CHAT_TESTOPS_OPERATION_PARAMETER_ID: ${{ vars.CHAT_TESTOPS_OPERATION_PARAMETER_ID }}
|
||||
CHAT_TESTOPS_INTEGRATION_ID: ${{ vars.CHAT_TESTOPS_INTEGRATION_ID }}
|
||||
CHAT_TESTOPS_LAUNCH_NONCE: ${{ vars.CHAT_TESTOPS_LAUNCH_NONCE }}
|
||||
CHAT_TESTOPS_LAUNCH_RUN_NUMBER: ${{ vars.CHAT_TESTOPS_LAUNCH_RUN_NUMBER }}
|
||||
TESTOPS_TRIGGERING_ACTOR_ID: ${{ needs.authorize.outputs.triggering_actor_id }}
|
||||
TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: node tests/runner-e2e/testops-chat-reporter.mjs launch-plan
|
||||
- name: Retain only closed metadata from zero-result setup
|
||||
if: always() && needs.authorize.outputs.operation == 'bootstrap'
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
||||
with:
|
||||
name: chat-testops-setup-${{ github.run_id }}-1
|
||||
path: |
|
||||
${{ runner.temp }}/chat-testops-bootstrap-state/metadata.json
|
||||
${{ runner.temp }}/chat-testops-bootstrap-state/once.json
|
||||
${{ runner.temp }}/chat-testops-bootstrap-state/plan.json
|
||||
${{ runner.temp }}/chat-testops-bootstrap-state/job.json
|
||||
${{ runner.temp }}/chat-testops-bootstrap-state/failure.json
|
||||
retention-days: 30
|
||||
if-no-files-found: error
|
||||
- name: Retain the one plan-launch receipt and stop state
|
||||
if: always() && needs.authorize.outputs.operation == 'launch-plan'
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
||||
with:
|
||||
name: chat-testops-plan-launch-${{ github.run_id }}-1
|
||||
path: |
|
||||
${{ runner.temp }}/chat-testops-launch-plan-state/once.json
|
||||
${{ runner.temp }}/chat-testops-launch-plan-state/launch.json
|
||||
${{ runner.temp }}/chat-testops-launch-plan-state/failure.json
|
||||
retention-days: 30
|
||||
if-no-files-found: error
|
||||
- name: Retain only the token-free exact selection receipt
|
||||
id: selection_upload
|
||||
if: steps.selection.outcome == 'success'
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
||||
with:
|
||||
name: chat-testops-selection-${{ github.run_id }}-1
|
||||
path: testops-selection.json
|
||||
retention-days: 30
|
||||
if-no-files-found: error
|
||||
|
||||
build:
|
||||
name: Build the immutable shipped native candidate without model credentials
|
||||
needs: [authorize, select]
|
||||
if: needs.authorize.outputs.operation == 'evaluate'
|
||||
runs-on: ${{ needs.authorize.outputs.runner }}
|
||||
timeout-minutes: 45
|
||||
outputs:
|
||||
artifact_id: ${{ steps.bundle.outputs.artifact-id }}
|
||||
steps:
|
||||
- *checkout
|
||||
- *node
|
||||
- *guard
|
||||
- &pnpm
|
||||
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
||||
env:
|
||||
NPM_CONFIG_AUDIT: 'false'
|
||||
NPM_CONFIG_FUND: 'false'
|
||||
NPM_CONFIG_UPDATE_NOTIFIER: 'false'
|
||||
with:
|
||||
version: 9.15.4
|
||||
- name: Install only the source-reviewed graph without lifecycle scripts
|
||||
run: |
|
||||
node tests/runner-e2e/testops-chat-lock.ts > chat-testops-lock.json
|
||||
pnpm install --frozen-lockfile --ignore-scripts
|
||||
- name: Build the existing TypeScript and native runner owners
|
||||
run: |
|
||||
pnpm --filter @paperclipai/paperclip-runner build:typescript
|
||||
pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
|
||||
- name: Package source-bound outputs and resolved lock
|
||||
run: |
|
||||
set -euo pipefail
|
||||
test "$(git rev-parse HEAD)" = "$GITHUB_SHA"
|
||||
node tests/runner-e2e/testops-chat-lock.ts > chat-testops-lock.json
|
||||
tar -czf chat-testops-build.tar.gz chat-testops-lock.json pnpm-lock.yaml packages/paperclip-eval-kernel/dist packages/paperclip-runner/dist packages/paperclip-runner/runner/target/debug/paperclip-runnerd packages/paperclip-runner/runner/target/debug/conformance-tracer packages/paperclip-runner/runner/target/debug/fake-harness packages/paperclip-runner/runner/target/debug/fake-codex-app-server packages/paperclip-runner/runner/target/debug/fake-acpx-sidecar
|
||||
sha256sum chat-testops-build.tar.gz > chat-testops-build.tar.gz.sha256
|
||||
- name: Retain credential-free immutable outputs
|
||||
id: bundle
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
||||
with:
|
||||
name: chat-testops-build-${{ github.sha }}-${{ github.run_id }}-1
|
||||
path: |
|
||||
chat-testops-build.tar.gz
|
||||
chat-testops-build.tar.gz.sha256
|
||||
retention-days: 1
|
||||
if-no-files-found: error
|
||||
|
||||
chat:
|
||||
name: One real native OpenCode backlog request on AWS
|
||||
needs: [authorize, select, build]
|
||||
if: needs.authorize.outputs.operation == 'evaluate'
|
||||
runs-on: ${{ needs.authorize.outputs.runner }}
|
||||
timeout-minutes: 32
|
||||
environment: runner-e2e-private-paid
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
env: &retention
|
||||
PAPERCLIP_TESTOPS_CHAT_MAIN_ADMISSION: v1
|
||||
TESTOPS_ADMITTED_SHA: ${{ needs.authorize.outputs.sha }}
|
||||
TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TESTOPS_TRIGGERING_ACTOR_ID: ${{ needs.authorize.outputs.triggering_actor_id }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.sha }}
|
||||
MANUAL_PRIVATE_IDS: '["chat-features.runner-opencode-deepseek-1017.local.create-backlog"]'
|
||||
SELECTED_EXECUTION_ID: chat-features.runner-opencode-deepseek-1017.local.create-backlog
|
||||
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
|
||||
NODE_OPTIONS: ''
|
||||
NODE_PATH: ''
|
||||
AWS_EC2_METADATA_DISABLED: 'true'
|
||||
steps:
|
||||
- *checkout
|
||||
- *node
|
||||
- *guard
|
||||
- *pnpm
|
||||
- name: Check out the same admitted source for trusted retention
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
||||
with:
|
||||
ref: ${{ needs.authorize.outputs.sha }}
|
||||
path: .testops-trusted
|
||||
persist-credentials: false
|
||||
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
||||
with:
|
||||
artifact-ids: ${{ needs.build.outputs.artifact_id }}
|
||||
path: chat-testops-build
|
||||
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
||||
with:
|
||||
artifact-ids: ${{ needs.select.outputs.artifact_id }}
|
||||
path: chat-testops-selection
|
||||
- name: Verify immutable outputs and token-free TestOps selection
|
||||
env:
|
||||
EXPECTED_SELECTION_SHA256: ${{ needs.select.outputs.sha256 }}
|
||||
ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
(cd chat-testops-build && sha256sum -c chat-testops-build.tar.gz.sha256)
|
||||
tar -xzf chat-testops-build/chat-testops-build.tar.gz -C "$GITHUB_WORKSPACE"
|
||||
node tests/runner-e2e/testops-chat-reporter.mjs verify-selection chat-testops-selection/testops-selection.json
|
||||
node tests/runner-e2e/testops-chat-lock.ts > chat-testops-worker-lock.json
|
||||
cmp chat-testops-lock.json chat-testops-worker-lock.json
|
||||
cp pnpm-lock.yaml .testops-trusted/pnpm-lock.yaml
|
||||
pnpm install --frozen-lockfile --ignore-scripts
|
||||
(cd .testops-trusted && pnpm install --frozen-lockfile --ignore-scripts --store-dir "$RUNNER_TEMP/chat-retention-store")
|
||||
- name: Materialize and qualify pinned OpenCode runtime and browser
|
||||
run: |
|
||||
node packages/paperclip-runner/scripts/materialize-opencode-binary.mjs
|
||||
node -e 'const fs=require("node:fs");fs.chmodSync(process.execPath,fs.statSync(process.execPath).mode&~0o022);if(fs.statSync(process.execPath).mode&0o022)throw Error("Writable provider Node")'
|
||||
google-chrome --version
|
||||
pnpm exec playwright install ffmpeg
|
||||
- name: Prepare private evidence with shipped-main authority
|
||||
id: private
|
||||
env:
|
||||
TARGET_ROOT: ${{ github.workspace }}
|
||||
run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts prepare
|
||||
- name: Obtain only the exact private cell storage session
|
||||
id: cell_credentials
|
||||
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
|
||||
with:
|
||||
role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }}
|
||||
aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
|
||||
inline-session-policy: ${{ steps.private.outputs.session_policy }}
|
||||
output-env-credentials: false
|
||||
output-credentials: true
|
||||
disable-retry: true
|
||||
action-timeout-s: 60
|
||||
- name: Verify strict private custody before the model key
|
||||
env: &cell_session
|
||||
PRIVATE_EVIDENCE_ROOT: ${{ steps.private.outputs.root }}
|
||||
AWS_ACCESS_KEY_ID: ${{ steps.cell_credentials.outputs.aws-access-key-id }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ steps.cell_credentials.outputs.aws-secret-access-key }}
|
||||
AWS_SESSION_TOKEN: ${{ steps.cell_credentials.outputs.aws-session-token }}
|
||||
AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
|
||||
run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts verify
|
||||
- name: Execute only the fixed original request with zero retries
|
||||
env:
|
||||
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
|
||||
PAPERCLIP_RUNNER_E2E_PRIVATE_PREFLIGHT: ${{ steps.private.outputs.root }}/preflight.json
|
||||
PRIVATE_EVIDENCE_ROOT: ${{ steps.private.outputs.root }}
|
||||
PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-1-chat-features.runner-opencode-deepseek-1017.local.create-backlog
|
||||
PAPERCLIP_E2E_ACCOUNTING_CAMPAIGN_ID: gha-${{ github.run_id }}-1
|
||||
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.sha }}
|
||||
PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ github.ref }}
|
||||
PAPERCLIP_PLAYWRIGHT_CHANNEL: chrome
|
||||
run: pnpm test:e2e:runner -- --id chat-features.runner-opencode-deepseek-1017.local.create-backlog --max-parallel 1 --max-automatic-retries 0 > "$PRIVATE_EVIDENCE_ROOT/launcher-output.log" 2>&1
|
||||
- name: Retain exact original grade and private evidence, including failures
|
||||
if: always() && steps.cell_credentials.outcome == 'success'
|
||||
env: *cell_session
|
||||
run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts upload-cell
|
||||
|
||||
report:
|
||||
name: Retain private originals and export only reviewed fixture results
|
||||
needs: [authorize, select, chat]
|
||||
if: always() && !cancelled() && needs.authorize.outputs.operation == 'evaluate' && needs.select.result == 'success'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
environment: runner-e2e-private-history
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
id-token: write
|
||||
env: *retention
|
||||
steps:
|
||||
- *checkout
|
||||
- *node
|
||||
- *guard
|
||||
- *pnpm
|
||||
- name: Install trusted reporting dependencies only
|
||||
run: |
|
||||
node tests/runner-e2e/prepare-report-lock.mjs
|
||||
pnpm install --frozen-lockfile --ignore-scripts --store-dir "$RUNNER_TEMP/chat-report-store"
|
||||
- name: Prepare exact private report and cell-read policies
|
||||
id: session
|
||||
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts report-session
|
||||
- name: Obtain private report writer
|
||||
id: report_credentials
|
||||
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
|
||||
with:
|
||||
role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }}
|
||||
aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
|
||||
inline-session-policy: ${{ steps.session.outputs.session_policy }}
|
||||
output-env-credentials: false
|
||||
output-credentials: true
|
||||
disable-retry: true
|
||||
action-timeout-s: 60
|
||||
- name: Obtain exact private cell reader
|
||||
id: read_credentials
|
||||
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
|
||||
with:
|
||||
role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }}
|
||||
aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
|
||||
inline-session-policy: ${{ steps.session.outputs.read_policy_0 }}
|
||||
output-env-credentials: false
|
||||
output-credentials: true
|
||||
disable-retry: true
|
||||
action-timeout-s: 60
|
||||
- name: Download the exact private original and verify all inventories
|
||||
env:
|
||||
PRIVATE_REPORT_ROOT: ${{ github.workspace }}
|
||||
SELECTED_CELL_INDEX: '0'
|
||||
AWS_ACCESS_KEY_ID: ${{ steps.read_credentials.outputs.aws-access-key-id }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ steps.read_credentials.outputs.aws-secret-access-key }}
|
||||
AWS_SESSION_TOKEN: ${{ steps.read_credentials.outputs.aws-session-token }}
|
||||
AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
|
||||
run: |
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts download-cell
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts verify-downloads
|
||||
- name: Build the existing normalized campaign without exposing original logs
|
||||
env:
|
||||
PAPERCLIP_RUNNER_E2E_REPORT_ROOT: ${{ github.workspace }}/selected-runner-e2e
|
||||
PAPERCLIP_RUNNER_E2E_REPORT_OUT: ${{ github.workspace }}/runner-e2e-merged-report/normalized
|
||||
PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: '["chat-features.runner-opencode-deepseek-1017.local.create-backlog"]'
|
||||
PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-1
|
||||
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.sha }}
|
||||
PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ github.ref }}
|
||||
run: |
|
||||
mkdir -p runner-e2e-merged-report
|
||||
pnpm test:e2e:runner:report > runner-e2e-merged-report/report.log 2>&1
|
||||
- name: Retain the complete merged report privately even when its grade failed
|
||||
id: private_report
|
||||
if: always() && steps.report_credentials.outcome == 'success'
|
||||
env:
|
||||
PRIVATE_REPORT_ROOT: ${{ github.workspace }}
|
||||
PRIVATE_REPORT_SOURCE: ${{ github.workspace }}/runner-e2e-merged-report
|
||||
AWS_ACCESS_KEY_ID: ${{ steps.report_credentials.outputs.aws-access-key-id }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ steps.report_credentials.outputs.aws-secret-access-key }}
|
||||
AWS_SESSION_TOKEN: ${{ steps.report_credentials.outputs.aws-session-token }}
|
||||
AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
|
||||
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts upload-report
|
||||
- name: Export only the bound normalized result, JUnit and marked PNGs
|
||||
id: export
|
||||
if: always() && steps.private_report.outcome == 'success'
|
||||
env:
|
||||
TESTOPS_CHAT_TARGET_SHA: ${{ needs.authorize.outputs.sha }}
|
||||
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/testops-chat-export.ts
|
||||
- name: Upload only the sanitized TestOps projection
|
||||
if: always() && steps.export.outcome == 'success'
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
||||
with:
|
||||
name: runner-e2e-testops-${{ github.run_id }}-1
|
||||
path: runner-e2e-testops/
|
||||
retention-days: 30
|
||||
if-no-files-found: error
|
||||
|
||||
upload:
|
||||
name: Report the original outcome to its exact TestOps case
|
||||
needs: [authorize, select, chat, report]
|
||||
if: always() && !cancelled() && needs.authorize.outputs.operation == 'evaluate' && needs.select.result == 'success'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
environment: runner-e2e-testops-reporting
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
steps:
|
||||
- *checkout
|
||||
- *node
|
||||
- *guard
|
||||
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
||||
with:
|
||||
name: runner-e2e-testops-${{ github.run_id }}-1
|
||||
path: runner-e2e-testops
|
||||
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
||||
with:
|
||||
artifact-ids: ${{ needs.select.outputs.artifact_id }}
|
||||
path: chat-testops-selection
|
||||
- *allure
|
||||
- name: Verify and upload only the bound safe case outcome
|
||||
id: allure_upload
|
||||
env: &testops_reporter
|
||||
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
|
||||
ALLURE_ENDPOINT: https://paperclip.testops.cloud
|
||||
ALLURE_PROJECT_ID: '1'
|
||||
ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
|
||||
EXPECTED_SELECTION_SHA256: ${{ needs.select.outputs.sha256 }}
|
||||
ALLURE_RESULTS: ${{ runner.temp }}/chat-testops-allure-results
|
||||
run: allurectl watch -- node tests/runner-e2e/testops-chat-reporter.mjs upload runner-e2e-testops
|
||||
- name: Verify the actual TestOps result after upload processing
|
||||
id: verify_upload
|
||||
if: always() && !cancelled() && (steps.allure_upload.outcome == 'success' || steps.allure_upload.outcome == 'failure')
|
||||
env: *testops_reporter
|
||||
run: node tests/runner-e2e/testops-chat-reporter.mjs verify-upload runner-e2e-testops
|
||||
- name: Retain only closed actual TestOps verification metadata
|
||||
if: always() && !cancelled() && (steps.verify_upload.outcome == 'success' || steps.verify_upload.outcome == 'failure')
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
||||
with:
|
||||
name: chat-testops-verification-${{ github.run_id }}-1
|
||||
path: |
|
||||
${{ runner.temp }}/chat-testops-verify-upload-state/verification.json
|
||||
${{ runner.temp }}/chat-testops-verify-upload-state/failure.json
|
||||
retention-days: 30
|
||||
if-no-files-found: error
|
||||
+4
-1
@@ -43,7 +43,10 @@ two natural-language task scenarios into local native Codex subscription and
|
||||
OpenCode Product E2E cells. They
|
||||
verify saving an unstarted task and delegating a saved announcement back to chat,
|
||||
using ordinary production instructions, real browser messages and independent
|
||||
state/provenance checks. The broader YAML catalog remains planning inventory.
|
||||
state/provenance checks. A separately selected native OpenCode/DeepSeek1017
|
||||
backlog cell measures TestOps integration acceptance with the same oracle and
|
||||
original bounds. Its source registration is not an AWS or TestOps pass. The
|
||||
broader YAML catalog remains planning inventory.
|
||||
|
||||
The explicit-only [Daytona restart continuity journey](../tests/runner-e2e/README.md)
|
||||
extends the existing native restart oracle to a 125-second controller outage,
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# Chat feature TestOps smoke — 2026-10-30
|
||||
|
||||
## Finish line
|
||||
|
||||
Create one dedicated runnable TestOps plan and obtain one genuine passing
|
||||
`tasks.create-backlog` result through TestOps → GitHub Actions/AWS → native
|
||||
OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`. The result must prove
|
||||
the saved task, assignment, backlog-at-creation state, no worker execution,
|
||||
agent attribution, rendered link and reload persistence. Confirm cleanup,
|
||||
safe screenshot/report access and the final TestOps result link.
|
||||
|
||||
This is a new integration-acceptance measurement. It preserves the original
|
||||
244-scenario campaign, attempts, grades and existing Codex measurements.
|
||||
Delegation and the other catalog scenarios are deferred.
|
||||
|
||||
## Source and execution contract
|
||||
|
||||
- Public selector: `chat-features.runner-opencode-deepseek-1017.local.create-backlog`.
|
||||
- Existing YAML query, fixture and independent backlog oracle stay unchanged.
|
||||
- One human query and one accounted controller work run; 120-second turns,
|
||||
420-second attempt, 300-second settled-state observation, zero retries.
|
||||
- Existing 500-cent company/agent policies and unknown setup usage remain
|
||||
explicit. Morgan is fixture context and must not execute this backlog task.
|
||||
- Only the existing protected `OPENROUTER_API_KEY` is required. Agent
|
||||
configuration uses the normal encrypted secret-reference path.
|
||||
- OpenCode 1017 is selected only for this cell and remains a qualification
|
||||
candidate until the real run. Other suites retain their existing models.
|
||||
|
||||
## Ownership and integration
|
||||
|
||||
Campaign owns public catalog/profile calibration and these progress notes.
|
||||
Core owns the public AWS workflow/execution adapter, one coherent public PR,
|
||||
and the sole actual operation. Coordinator supplies the public TestOps
|
||||
selection/reporting helper and the separate private credential sealer.
|
||||
The public workflow owns the dedicated plan. Source peers precede publication.
|
||||
|
||||
The current Cloud TestOps adapter is scoped to `qa-basics` Product journeys;
|
||||
the new chat plan must have its own exact plan/case mapping. Reuse normalized
|
||||
result/check/provenance and existing packaged evidence. Do not upload private
|
||||
traces, provider sessions, raw logs or unreviewed media as TestOps attachments.
|
||||
|
||||
## Current progress
|
||||
|
||||
The public patch adds only the OpenCode backlog selection while preserving
|
||||
the two Codex cells. A genuine parent selector control failed; all 68 focused
|
||||
oracle/catalog controls and the owning type check now pass. The expected
|
||||
suite-size assertion was updated for the single added cell. Independent source
|
||||
review passed and the nine-file profile slice is committed as `0a996033`.
|
||||
No local app, database, native SDK,
|
||||
provider, model, install or build qualification was run.
|
||||
|
||||
The separate protected credential bootstrap seals the existing reporting token
|
||||
for the source-pinned public reporting environment. Delivery is complete without
|
||||
exporting plaintext. The public adapter has 53 focused authority/export/workflow
|
||||
controls and 39 API/CLI/result controls passing, plus owning typechecking and
|
||||
workflow lint. Unchanged controls are reused after narrow source corrections.
|
||||
The independent composition source peer passed.
|
||||
The public infrastructure [PR 18322](https://github.com/paperclipai/paperclip/pull/18322)
|
||||
is open. Its first normal CI run rejected the new workflow in the existing paid
|
||||
workflow allowlist and reported a separate heartbeat scheduling assertion; both
|
||||
failures remain recorded. The policy correction preserves the older workflow
|
||||
guards and rejects unsafe new-route variants. The reporting token is configured
|
||||
through the reviewed encrypted bootstrap.
|
||||
|
||||
Review also added a one-time initial administrator launch fence. A protected
|
||||
expected GitHub run number and actual own-run metadata must match before any
|
||||
TestOps mutation. REST catalog reconciliation preserves exact case aliases;
|
||||
job wiring uses only the reviewed existing project GitHub integration. These
|
||||
provider-free helper controls pass. Actual registration IDs, tenant API behavior,
|
||||
provider execution and the TestOps result remain unqualified. Dedicated plan,
|
||||
AWS evaluation and final result links remain pending. Source checks alone do not
|
||||
satisfy the finish line.
|
||||
|
||||
The final source consolidation adds actual admission-wrapper filesystem coverage,
|
||||
the existing reviewed lock normalization with frozen installs and a matched
|
||||
build/worker receipt, plus same-job server-result readback after Allure upload.
|
||||
Focused checks cover 41 admission controls, 31 lock/owner/workflow controls and
|
||||
44 API/reporter controls; unchanged profile/oracle evidence is reused. This is
|
||||
source verification. The plan and actual model result are still not created.
|
||||
|
||||
Current-main composition preserves the four existing Codex and production-qualified
|
||||
OpenCode cells introduced by #18319. The TestOps cell has its own
|
||||
`runner-opencode-deepseek-1017` profile, so the existing OpenCode model remains
|
||||
unchanged. It uses the shipped v2 grader, queue accounting and `final-state.png`
|
||||
capture. The new profile keeps the original query, one work run, one human message,
|
||||
120-second turn, 420-second attempt, 300-second observation bound and zero retries.
|
||||
The existing private session and trace-asset owners admit only this additional
|
||||
exact selector; the frozen 244-case candidate-approval registry remains unchanged.
|
||||
Source checks do not establish a TestOps registration or model outcome.
|
||||
@@ -579,6 +579,7 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
|
||||
readWorkflow("runner-full-stack-e2e.yml"),
|
||||
readWorkflow("e2e.yml"),
|
||||
readWorkflow("runner-protocol-live-evals.yml"),
|
||||
readWorkflow("chat-feature-testops.yml"),
|
||||
];
|
||||
|
||||
for (const workflow of actionPinWorkflows) {
|
||||
@@ -618,6 +619,7 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
|
||||
"runner-full-stack-e2e.yml",
|
||||
"runner-live-evals.yml",
|
||||
"runner-protocol-live-evals.yml",
|
||||
"chat-feature-testops.yml",
|
||||
];
|
||||
const paidWorkflowNameSet = new Set(paidWorkflowNames);
|
||||
const providerSecretReference =
|
||||
@@ -650,6 +652,38 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
|
||||
(block) => [...block.matchAll(providerSecretReference)].length > 0,
|
||||
);
|
||||
assert.ok(providerJobs.length > 0, `${name} needs a provider-secret job`);
|
||||
if (name === "chat-feature-testops.yml") {
|
||||
assert.equal(providerJobs.length, 1);
|
||||
const assertBoundary = (block) => {
|
||||
assert.match(block, /^ chat:\n/);
|
||||
assert.match(block, /\n needs: \[authorize, select, build\]\n/);
|
||||
assert.match(block, /\n if: needs\.authorize\.outputs\.operation == 'evaluate'\n/);
|
||||
assert.match(block, /\n environment: runner-e2e-private-paid\n/);
|
||||
assert.match(block, /\n - \*checkout\n - \*node\n - \*guard\n/);
|
||||
assert.match(block, /testops-chat-reporter\.mjs verify-selection chat-testops-selection\/testops-selection\.json/);
|
||||
assert.match(block, /OPENROUTER_API_KEY: \$\{\{ secrets\.OPENROUTER_API_KEY \}\}/);
|
||||
assert.match(block, /--id chat-features\.runner-opencode-deepseek-1017\.local\.create-backlog --max-parallel 1 --max-automatic-retries 0/);
|
||||
assert.doesNotMatch(block, /ALLURE_TOKEN/);
|
||||
assert.equal([...block.matchAll(providerSecretReference)].length, 1);
|
||||
assert.ok(block.indexOf(" - *guard") < block.search(providerSecretReference));
|
||||
assert.ok(block.indexOf("testops-chat-reporter.mjs verify-selection") < block.search(providerSecretReference));
|
||||
};
|
||||
const block = providerJobs[0];
|
||||
assertBoundary(block);
|
||||
assert.equal([...workflow.matchAll(providerSecretReference)].length, 1);
|
||||
assert.match(workflow, /ref: \$\{\{ github\.sha \}\}/);
|
||||
assert.match(workflow, /run: node tests\/runner-e2e\/testops-chat-dispatch\.ts/);
|
||||
assert.match(workflow, /TESTOPS_ALLOWED_ACTOR_IDS: \$\{\{ vars\.RUNNER_E2E_ALLOWED_ACTOR_IDS \}\}/);
|
||||
for (const mutant of [
|
||||
block.replace("needs: [authorize, select, build]", "needs: [authorize, build]"),
|
||||
block.replace("environment: runner-e2e-private-paid", "environment: unprotected"),
|
||||
block.replace(" - *guard\n", ""),
|
||||
block.replace("testops-chat-reporter.mjs verify-selection", "testops-chat-reporter.mjs unverified-selection"),
|
||||
block.replace("OPENROUTER_API_KEY:", "ALLURE_TOKEN:"),
|
||||
block.replace("--id chat-features.runner-opencode-deepseek-1017.local.create-backlog", "--id chat-features.runner-opencode.local.create-backlog"),
|
||||
]) assert.throws(() => assertBoundary(mutant));
|
||||
continue;
|
||||
}
|
||||
for (const block of providerJobs) {
|
||||
if (name === "runner-protocol-live-evals.yml") {
|
||||
assert.ok(block.includes("name: ${{ needs.authorize.outputs.accounting_mode == 'controlled' && 'runner-e2e-private-paid' || 'runner-e2e-paid' }}"));
|
||||
|
||||
@@ -580,16 +580,33 @@ describeEmbeddedPostgres("issue scheduled retry routes", () => {
|
||||
.where(inArray(heartbeatRuns.id, work.map(row => row.id)));
|
||||
clock();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => { throw error; } });
|
||||
const register = coordinator.register.bind(coordinator);
|
||||
let initial: Awaited<ReturnType<typeof states>> | undefined;
|
||||
let releaseFirstPass!: () => void;
|
||||
const firstPassHeld = new Promise<void>(resolve => { releaseFirstPass = resolve; });
|
||||
vi.spyOn(coordinator, "register").mockImplementation((queue, task) => register(queue, {
|
||||
...task,
|
||||
run: async signal => {
|
||||
const result = await task.run(signal);
|
||||
if (initial) return result;
|
||||
const cancelled = (await states()).filter(row => row.status === "cancelled");
|
||||
if (cancelled.length === 0) return result;
|
||||
initial = cancelled;
|
||||
await firstPassHeld;
|
||||
return result;
|
||||
},
|
||||
}));
|
||||
try {
|
||||
await heartbeatService(db, { runtimeEnv: {} }).registerScheduledRetryCancellationWork(coordinator, () => true).ready;
|
||||
expect((await states()).filter(row => row.status === "cancelled")).toHaveLength(0);
|
||||
const earlierId = "00000000-0000-4000-8000-000000000000";
|
||||
await writeIntent({ companyId: a.companyId, retryRunId: earlierId });
|
||||
work.push({ id: earlierId });
|
||||
await writeIntent({ companyId: a.companyId, retryRunId: earlierId });
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await expect.poll(async () => (await states()).filter(row => row.status === "cancelled").length).toBe(20);
|
||||
const initial = (await states()).filter(row => row.status === "cancelled");
|
||||
expect(new Set(initial.map(row => row.companyId)).size).toBe(2);
|
||||
await expect.poll(() => initial).toBeDefined();
|
||||
expect(initial).toHaveLength(20);
|
||||
expect(new Set(initial!.map(row => row.companyId)).size).toBe(2);
|
||||
releaseFirstPass();
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect((await states()).filter(row => row.status === "cancelled")).toHaveLength(29);
|
||||
@@ -600,7 +617,7 @@ describeEmbeddedPostgres("issue scheduled retry routes", () => {
|
||||
const reads = vi.spyOn(db, "select");
|
||||
await vi.advanceTimersByTimeAsync(180_000);
|
||||
expect(reads).not.toHaveBeenCalled(); reads.mockRestore();
|
||||
} finally { await coordinator.stop(); }
|
||||
} finally { releaseFirstPass(); await coordinator.stop(); }
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -29,7 +29,8 @@ and removes it with the isolated instance.
|
||||
|
||||
The local native OpenCode profile uses the current production-qualified model
|
||||
and `OPENROUTER_API_KEY` through the existing encrypted company-secret fixture.
|
||||
Each of the two journeys has a cell for each profile. All four cells are manual.
|
||||
Each of the two journeys has a cell for each original profile. These four cells
|
||||
remain unchanged; the TestOps profile below adds one separately named backlog cell.
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner:typecheck
|
||||
@@ -40,13 +41,41 @@ pnpm test:e2e:runner -- --id chat-features.runner-codex-subscription.local.deleg
|
||||
pnpm test:e2e:runner -- --id chat-features.runner-opencode.local.delegate-announcement --max-automatic-retries 0
|
||||
```
|
||||
|
||||
Run creation first and inspect its packaged report before delegation. For initial
|
||||
Run creation first and inspect its packaged report before delegation. For the original Codex
|
||||
acceptance, run each cell three times against fresh fixtures on the same definition
|
||||
revision. Keep every attempt, including failures; a retry does not erase one. A
|
||||
fixture or product correction starts a new three-sample measurement. All cells
|
||||
are explicit-only and excluded from `--all`; this does not implement or qualify
|
||||
the rest of the YAML inventory.
|
||||
|
||||
## OpenCode TestOps integration acceptance
|
||||
|
||||
The additional explicit cell
|
||||
`chat-features.runner-opencode-deepseek-1017.local.create-backlog` runs the same backlog
|
||||
query, public fixture and saved-state oracle on native OpenCode with
|
||||
`openrouter/deepseek/deepseek-v4.1-flash-1017`. It requires only the selected
|
||||
`OPENROUTER_API_KEY` profile credential. The launcher stores an encrypted secret
|
||||
reference for the disposable agents; no key belongs in catalog data or chat.
|
||||
This selected model is a qualification candidate. Other OpenCode suites keep
|
||||
their existing model, and delegation is not registered for this profile.
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner -- --id chat-features.runner-opencode-deepseek-1017.local.create-backlog --max-automatic-retries 0
|
||||
```
|
||||
|
||||
Use a separately named TestOps integration-acceptance plan and launch for this
|
||||
single cell. It retains one human request, one accounted controller work run,
|
||||
120-second turns, a seven-minute attempt, the five-minute stable-state bound
|
||||
and zero automatic retries. Morgan is created as fixture context but must not
|
||||
execute the saved backlog task. Existing setup probes and unknown usage remain
|
||||
reported separately. The original 244-scenario campaign, failed attempts and
|
||||
Codex measurements are not reset or regraded by this measurement.
|
||||
|
||||
The existing normalized result, JUnit and packaged report carry the outcome to
|
||||
the execution integration. Catalog selection and provider-free oracle checks
|
||||
do not establish an AWS model pass or a successful TestOps upload. Accept this
|
||||
integration only after the real AWS execution and its TestOps result agree.
|
||||
|
||||
## Bounds and grading
|
||||
|
||||
Creation expects one accounted native work run; delegation expects three (CEO
|
||||
@@ -151,3 +180,71 @@ in the history. Across the original final-definition batch and this recheck,
|
||||
backlog is 5/5 and delegation is 5/6. The local
|
||||
`results/chat-feature-recheck/verification.md` records the follow-up evidence;
|
||||
its standard dashboard is explicitly scoped to the latest campaign.
|
||||
|
||||
## TestOps-backed AWS backlog cell
|
||||
|
||||
The dedicated `Chat feature TestOps` workflow accepts only the shipped default
|
||||
branch and the fixed OpenCode backlog cell. `evaluate` is the default operation
|
||||
and requires TestOps' `ALLURE_JOB_RUN_ID`. A protected reporting job verifies the
|
||||
project, singleton plan, assigned public workflow, selected case and current
|
||||
GitHub run before any build or provider access. The evaluated process receives
|
||||
no TestOps token. It retains the original one-query, one-work-run, 120-second
|
||||
turn, 420-second attempt, 300-second settled-state and zero-retry limits.
|
||||
|
||||
Two metadata operations prepare that route. `bootstrap` registers the actual
|
||||
public CI job through pinned `allurectl` with an empty results directory and
|
||||
reconciles only the dedicated case and singleton plan. It produces no passing
|
||||
test result. After the registered job and protected case/plan/job identifiers
|
||||
are verified, `launch-plan` sends one TestOps plan-run request and exits.
|
||||
The TestOps job then dispatches `evaluate`; an operator must not dispatch a
|
||||
second evaluation manually. Missing or ambiguous registration and selection
|
||||
metadata stops before inference.
|
||||
|
||||
The workflow reuses the existing AWS fleet and strict private evidence storage.
|
||||
Its shipped-main authority is separate from manual candidate approval. Original
|
||||
logs, provider records and failure evidence remain private. The reporting adapter
|
||||
exports only a source/run-bound normalized result, JUnit and screenshots already
|
||||
marked by the owned fixture screenshot producer with matching file hashes.
|
||||
A separate protected reporting job converts that projection to the exact TestOps
|
||||
case. Failed grades and cleanup failures remain failed.
|
||||
|
||||
Reporting setup uses the master-only `runner-e2e-testops-reporting` environment,
|
||||
with the existing reporting token delivered through the reviewed sealed-box
|
||||
bootstrap. `CHAT_TESTOPS_CASE_ID`, `CHAT_TESTOPS_PLAN_ID` and
|
||||
`CHAT_TESTOPS_JOB_ID` must name the observed dedicated mapping; the numeric
|
||||
`CHAT_TESTOPS_GITHUB_WORKFLOW_ID` supports TestOps' registered workflow URL.
|
||||
No arbitrary source, model, case, retry or budget input is accepted.
|
||||
|
||||
The initial `launch-plan` administrator operation also requires a protected
|
||||
`CHAT_TESTOPS_LAUNCH_RUN_NUMBER`. After registration, the trusted operator creates
|
||||
this variable once with the reviewed next workflow run number and reads it back.
|
||||
The launcher compares GitHub's actual run number, attempt, workflow, source and
|
||||
actors before any TestOps mutation. A later dispatch or replay cannot reuse that
|
||||
approved number. Missing or uncertain metadata stops; an uncertain request needs
|
||||
reconciliation rather than a new send. The nonce and launch search provide
|
||||
additional evidence. Normal TestOps **Run** uses `evaluate` and is independent of
|
||||
this one-time administrator fence. The operation parameter identifier and launch
|
||||
nonce are also protected configuration, not dispatch inputs.
|
||||
|
||||
`CHAT_TESTOPS_INTEGRATION_ID` names the reviewed existing project GitHub
|
||||
integration. The launcher verifies its project, type and enabled state, then binds
|
||||
only the newly registered public job to it. It never changes integration settings
|
||||
or credentials. Since the job response does not expose that association, only the
|
||||
real TestOps-to-GitHub dispatch can qualify it.
|
||||
|
||||
After `allurectl watch` finishes, the same protected reporter checks the actual
|
||||
TestOps job-run, launch, selected case, result source/status and attachment
|
||||
metadata. Processing reads are bounded to 12 polls, five-second gaps and a
|
||||
120-second processing deadline. Unknown or missing results remain unverified.
|
||||
The check downloads no attachment content and claims no server-side attachment
|
||||
byte hash. Its closed receipt is retained separately. An upload success alone
|
||||
is not a passing feature result, and verifying an original failure cannot turn
|
||||
that failure into a pass.
|
||||
|
||||
Build and worker installs use the existing reviewed report-lock policy. Only its
|
||||
committed input or output lock is accepted; its authenticated correction is the
|
||||
only allowed normalization. Dependency manifests, patch inputs and policy must
|
||||
remain unchanged. Installs are frozen with lifecycle scripts disabled. The
|
||||
credential-free build rechecks that boundary before packaging, and the worker
|
||||
matches the source/lock receipt before installing. Arbitrary resolver drift is
|
||||
rejected before provider access.
|
||||
@@ -5,7 +5,10 @@ only company, agents and project context through public APIs. Both agents use
|
||||
production role instructions; evaluator outcomes are never injected. Real browser
|
||||
submission, task audits, worker document provenance, completion delivery and
|
||||
reload readback establish the outcome. The existing fixture registry, subscription
|
||||
credential staging and whole-company cleanup own setup and teardown.
|
||||
credential staging and whole-company cleanup own setup and teardown. The explicit
|
||||
OpenCode backlog cell instead uses an encrypted OpenRouter profile secret
|
||||
reference. It preserves the same fixture and oracle without registering
|
||||
OpenCode delegation or changing the other suites' model.
|
||||
|
||||
[Deployment-targeted journeys](PRODUCT-JOURNEYS.md) resolve only their requested
|
||||
fixture graph through `FixtureRegistry.setupAll(requested)`. Omitting the
|
||||
|
||||
@@ -219,10 +219,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 4, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2,
|
||||
2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(774);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(774);
|
||||
expect(validateRunnerCatalog()).toHaveLength(775);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(775);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(48);
|
||||
|
||||
@@ -26,7 +26,7 @@ import { copilotProviderDeathTasks } from "./copilot-active-stop-tasks.js";
|
||||
import { instructionPersistenceTask } from "./instruction-persistence.js";
|
||||
import { apiResponseReadingTask } from "./api-response-reading.js";
|
||||
import { taskTitleTasks, taskTitleDefinitionDigest, TASK_TITLE_BUDGET_CENTS } from "./task-titles.js";
|
||||
import { CHAT_FEATURE_SUITE, CHAT_FEATURE_BUDGET_CENTS, chatFeatureTasks, chatFeatureProfile, chatFeatureScenarios, chatFeatureDefinitionDigest } from "./chat-feature-cases.js";
|
||||
import { CHAT_FEATURE_SUITE, CHAT_FEATURE_BUDGET_CENTS, CHAT_FEATURE_OPENCODE_MODEL, chatFeatureTasks, chatFeatureProfile, chatFeatureScenarios, chatFeatureDefinitionDigest } from "./chat-feature-cases.js";
|
||||
import { blockerTasks, blockerProfile } from "./blocker-cases.js";
|
||||
import { accountingTasks } from "./accounting-cases.js";
|
||||
import { continuationTasks, questionResumeTask } from "./continuation-cases.js";
|
||||
@@ -1609,10 +1609,14 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
id: CHAT_FEATURE_SUITE, label: "Chat feature journeys", manualOnly: true,
|
||||
description: "Natural catalog requests create backlog work or delegate an announcement and return its saved result.",
|
||||
groups: ["chat", "native"],
|
||||
profiles: [nativeProfile({ id: "runner-codex-subscription", label: "Native Codex subscription",
|
||||
provider: "codex", model: DEFAULT_CODEX_LOCAL_MODEL, credential: "CODEX_AUTH_JSON", supportedEnvironments: ["local"] }),
|
||||
runnerProfiles.find(profile => profile.id === "runner-opencode")!].map(chatFeatureProfile),
|
||||
environments: [localEnvironment], tasks: chatFeatureTasks, expectedMatrixSize: 4,
|
||||
profiles: [chatFeatureProfile(nativeProfile({ id: "runner-codex-subscription", label: "Native Codex subscription",
|
||||
provider: "codex", model: DEFAULT_CODEX_LOCAL_MODEL, credential: "CODEX_AUTH_JSON", supportedEnvironments: ["local"] })),
|
||||
chatFeatureProfile(runnerProfiles.find(profile => profile.id === "runner-opencode")!),
|
||||
chatFeatureProfile(nativeProfile({ id: "runner-opencode-deepseek-1017", label: "Native OpenCode DeepSeek 1017",
|
||||
provider: "opencode", model: CHAT_FEATURE_OPENCODE_MODEL, credential: "OPENROUTER_API_KEY", supportedEnvironments: ["local"],
|
||||
modelQualification: { source: "candidate_runner_profile", qualificationId: "chat-feature-testops:deepseek-v4.1-flash-1017" } }))],
|
||||
environments: [localEnvironment], tasks: chatFeatureTasks, expectedMatrixSize: 5,
|
||||
excludedExecutionIds: ["chat-features.runner-opencode-deepseek-1017.local.delegate-announcement"],
|
||||
definitionMetadata: { version: 2, definitionDigest: chatFeatureDefinitionDigest, scheduling: "explicit-only",
|
||||
instructions: "production-role-defaults", companyAndAgentBudgetCents: CHAT_FEATURE_BUDGET_CENTS,
|
||||
unpricedUsagePolicy: "allow-through-public-budget-policy", taskRunLimits: { ceo: 3, worker: 1 },
|
||||
|
||||
@@ -6,6 +6,7 @@ import type { RunnerProfileFixture, RunnerTaskFixture } from "./types.js";
|
||||
export const CHAT_FEATURE_SUITE = "chat-features";
|
||||
export const CHAT_FEATURE_BUDGET_CENTS = 500;
|
||||
export const CHAT_FEATURE_GRADER = "paperclip.chat-features.v2";
|
||||
export const CHAT_FEATURE_OPENCODE_MODEL = "openrouter/deepseek/deepseek-v4.1-flash-1017";
|
||||
export const CHAT_FEATURE_CASES = ["create-backlog", "delegate-announcement"] as const;
|
||||
export type ChatFeatureCase = typeof CHAT_FEATURE_CASES[number];
|
||||
export interface ChatScenario {
|
||||
@@ -48,7 +49,7 @@ export function chatFeatureProfile(profile: RunnerProfileFixture): RunnerProfile
|
||||
return { ...payload, name: "CEO", title: "CEO", role: "ceo",
|
||||
capabilities: "Coordinates company work and delegates tasks to the team.",
|
||||
budgetMonthlyCents: CHAT_FEATURE_BUDGET_CENTS,
|
||||
adapterConfig: { ...adapterConfig, timeoutSec: 120, modelReasoningEffort: "low" },
|
||||
adapterConfig: { ...adapterConfig, timeoutSec: 120, ...(profile.id === "runner-opencode-deepseek-1017" ? {} : { modelReasoningEffort: "low" }) },
|
||||
runtimeConfig: { ...(payload.runtimeConfig as Record<string, unknown>),
|
||||
heartbeat: { enabled: false, maxConcurrentRuns: 1, maxDailyRuns: 3, maxDailyCostCents: CHAT_FEATURE_BUDGET_CENTS } },
|
||||
};
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { chatFeatureExecutionRuns, gradeChatFeature, type ChatFeatureObservation } from "./chat-feature-scoring.js";
|
||||
import { chatFeatureScenarios, readChatFeatureScenarios } from "./chat-feature-cases.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { runnerMatrix, runnerProfiles } from "./catalog.js";
|
||||
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
|
||||
|
||||
const announcement = "Welcome to Acorn Notes, launching November 12! Join us with invitation code ACORN123.";
|
||||
@@ -107,9 +107,15 @@ describe("chat feature saved-state oracle", () => {
|
||||
});
|
||||
|
||||
describe("chat feature selection", () => {
|
||||
it("registers the two authored queries as explicit local subscription and OpenCode cells", () => {
|
||||
const cells = runnerMatrix.filter(cell => cell.suite.id === "chat-features");
|
||||
it("preserves the four authored subscription and production OpenCode cells", () => {
|
||||
const cells = runnerMatrix.filter(cell => cell.suite.id === "chat-features" && cell.profile.id !== "runner-opencode-deepseek-1017");
|
||||
expect(cells).toHaveLength(4);
|
||||
const stock = runnerProfiles.find(profile => profile.id === "runner-opencode")!;
|
||||
expect(cells.filter(cell => cell.profile.id === stock.id).map(cell => cell.profile.model)).toEqual([stock.model, stock.model]);
|
||||
for (const cell of cells.filter(cell => cell.profile.id === stock.id)) {
|
||||
const payload = cell.profile.buildAgent({ environmentId: "env", environmentFixtureId: "local", workspacePath: "/tmp/fixture", executionId: cell.id, secretRefs: { OPENROUTER_API_KEY: { type: "secret_ref", secretId: "fixture-openrouter", version: "latest" } } });
|
||||
expect(payload.adapterConfig).toMatchObject({ model: stock.model, modelReasoningEffort: "low", timeoutSec: 120 });
|
||||
}
|
||||
for (const cell of cells) {
|
||||
expect(cell.environment.id).toBe("local");
|
||||
expect(cell.requiredCredentials).toEqual([cell.profile.id === "runner-opencode" ? "OPENROUTER_API_KEY" : "CODEX_AUTH_JSON"]);
|
||||
@@ -119,6 +125,30 @@ describe("chat feature selection", () => {
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(row => row.id === cell.id)).toBe(false);
|
||||
}
|
||||
});
|
||||
it("selects one bounded OpenCode backlog cell with the requested model and secret reference", () => {
|
||||
const cells = runnerMatrix.filter(cell => cell.suite.id === "chat-features" && cell.profile.id === "runner-opencode-deepseek-1017");
|
||||
expect(cells.map(cell => cell.id)).toEqual(["chat-features.runner-opencode-deepseek-1017.local.create-backlog"]);
|
||||
const cell = cells[0]!;
|
||||
expect(cell.profile).toMatchObject({ provider: "opencode", model: "openrouter/deepseek/deepseek-v4.1-flash-1017", expectedRuntimeMode: "native" });
|
||||
expect(cell.profile.modelQualification.source).toBe("candidate_runner_profile");
|
||||
expect(cell.requiredCredentials).toEqual(["OPENROUTER_API_KEY"]);
|
||||
expect(cell.task).toMatchObject({ expectedRunCount: 1, turnTimeoutMs: 120_000,
|
||||
attemptTimeoutMs: { local: 420_000 }, automaticRetryPolicy: "single_attempt" });
|
||||
expect(cell.task.buildPrompt("nonce")).toBe(chatFeatureScenarios[0]!.query);
|
||||
const secret = { type: "secret_ref", secretId: "fixture-openrouter", version: "latest" } as const;
|
||||
const input = { environmentId: "env", environmentFixtureId: "local", workspacePath: "/tmp/fixture", executionId: "backlog" } as const;
|
||||
const agent = cell.profile.buildAgent({ ...input, secretRefs: { OPENROUTER_API_KEY: secret } });
|
||||
expect(agent).toMatchObject({ role: "ceo", adapterType: "paperclip_runner", budgetMonthlyCents: 500,
|
||||
adapterConfig: { provider: "opencode", model: "openrouter/deepseek/deepseek-v4.1-flash-1017", timeoutSec: 120,
|
||||
env: { OPENROUTER_API_KEY: secret } },
|
||||
runtimeConfig: { heartbeat: { enabled: false, maxConcurrentRuns: 1, maxDailyRuns: 3, maxDailyCostCents: 500 } } });
|
||||
expect(agent).not.toHaveProperty("instructionsBundle");
|
||||
expect(agent.adapterConfig).not.toHaveProperty("modelReasoningEffort");
|
||||
expect(agent.adapterConfig).not.toHaveProperty("codexPermissionMode");
|
||||
expect(() => cell.profile.buildAgent({ ...input, secretRefs: {} })).toThrow("Missing fixture secret reference OPENROUTER_API_KEY");
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--id", cell.id])).map(row => row.id)).toEqual([cell.id]);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(row => row.id === cell.id)).toBe(false);
|
||||
});
|
||||
it("refuses missing, malformed or duplicate catalog entries", () => {
|
||||
for (const yaml of ["{", "schema_version: 2", "schema_version: 1\nfeature: tasks\nstatus: planned\nscenarios: []"])
|
||||
expect(() => readChatFeatureScenarios(yaml)).toThrow();
|
||||
|
||||
@@ -0,0 +1,342 @@
|
||||
export const endpoint = 'https://paperclip.testops.cloud';
|
||||
export const planName = 'Chat feature evals';
|
||||
export const fullName = 'paperclip.Product E2E.chat-features.create-backlog';
|
||||
export const aliases = [fullName, 'paperclip.Product E2E.chat-features.tasks.create-backlog'];
|
||||
class SetupFailure extends Error {
|
||||
constructor(code, status = null) { super(code); this.code = code; this.status = status; }
|
||||
}
|
||||
const fail = code => {
|
||||
const match = /^(testops_(?:api|auth)|chat_testops_github)_http_([1-5][0-9]{2})$/.exec(code);
|
||||
throw new SetupFailure(match ? `${match[1]}_http` : code, match ? Number(match[2]) : null);
|
||||
};
|
||||
const diagnosticCodes = new Set(['untrusted_chat_testops_context', 'testops_token_missing', 'testops_auth_http',
|
||||
'testops_api_http', 'testops_auth_invalid', 'testops_route_invalid', 'testops_response_too_large',
|
||||
'testops_response_invalid', 'testops_metadata_page_incomplete', 'testops_case_identity_invalid',
|
||||
'testops_case_alias_conflict', 'testops_plan_name_conflict', 'testops_plan_identity_invalid',
|
||||
'testops_integration_identity_invalid', 'testops_bootstrap_identity_invalid', 'testops_plan_without_matching_case',
|
||||
'testops_created_case_identity_invalid', 'testops_created_plan_identity_invalid', 'testops_plan_scope_mismatch',
|
||||
'testops_operation_invalid', 'testops_output_invalid', 'untrusted_public_testops_context', 'chat_testops_identity_missing',
|
||||
'chat_testops_case_mismatch', 'chat_testops_plan_mismatch', 'chat_testops_plan_job_mismatch', 'chat_testops_job_mismatch',
|
||||
'chat_testops_jobrun_mismatch', 'chat_testops_launch_mismatch', 'chat_testops_selection_mismatch', 'chat_testops_parameter_mismatch', 'chat_testops_nonce_invalid', 'chat_testops_nonce_conflict', 'chat_testops_catalog_incomplete', 'chat_testops_launch_admission', 'chat_testops_github_http', 'chat_testops_integration_mismatch', 'chat_testops_upload_mismatch', 'chat_testops_upload_unverified']);
|
||||
export function failureProjection(error, stage) {
|
||||
const stages = ['context', 'output_admission', 'authentication', 'metadata_inspection', 'plan_bootstrap', 'selection_validation', 'launch_admission', 'launch_preflight', 'plan_launch', 'bootstrap_job_metadata', 'result_export', 'upload_verification', 'receipt'];
|
||||
return { stage: stages.includes(stage) ? stage : 'unknown',
|
||||
code: error instanceof SetupFailure && diagnosticCodes.has(error.code) ? error.code : 'unknown_failure',
|
||||
httpStatus: error instanceof SetupFailure && Number.isInteger(error.status) && error.status >= 100 && error.status < 600 ? error.status : null };
|
||||
}
|
||||
const id = value => Number.isSafeInteger(value) && value > 0;
|
||||
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
|
||||
|
||||
export async function testopsClient(token, fetchImpl = fetch) {
|
||||
if (!token) fail('testops_token_missing');
|
||||
const auth = await fetchImpl(`${endpoint}/api/uaa/oauth/token`, { method: 'POST', redirect: 'error',
|
||||
headers: { Accept: 'application/json' }, body: new URLSearchParams({ grant_type: 'apitoken', scope: 'openid', token }),
|
||||
signal: AbortSignal.timeout(30_000) });
|
||||
if (!auth.ok) fail(`testops_auth_http_${auth.status}`);
|
||||
const value = await auth.json();
|
||||
if (typeof value.access_token !== 'string' || !value.access_token) fail('testops_auth_invalid');
|
||||
return async (route, method = 'GET', body, timeoutMs = 30_000) => {
|
||||
if (!Number.isSafeInteger(timeoutMs) || timeoutMs < 1 || timeoutMs > 30_000) fail('chat_testops_upload_unverified');
|
||||
if (!route.startsWith('/api/') || route.includes('://')) fail('testops_route_invalid');
|
||||
const response = await fetchImpl(`${endpoint}${route}`, { method, redirect: 'error',
|
||||
headers: { Accept: 'application/json', Authorization: `Bearer ${value.access_token}`, ...(body ? { 'Content-Type': 'application/json' } : {}) },
|
||||
...(body ? { body: JSON.stringify(body) } : {}), signal: AbortSignal.timeout(timeoutMs) });
|
||||
if (!response.ok) fail(`testops_api_http_${response.status}`);
|
||||
if (response.status === 204) return null;
|
||||
const text = await response.text();
|
||||
if (Buffer.byteLength(text) > 2_000_000) fail('testops_response_too_large');
|
||||
try { return JSON.parse(text); } catch { fail('testops_response_invalid'); }
|
||||
};
|
||||
}
|
||||
|
||||
const rows = value => {
|
||||
if (!Array.isArray(value?.content) || value.content.length > 100 || value.totalPages > 1) fail('testops_metadata_page_incomplete');
|
||||
return value.content;
|
||||
};
|
||||
export async function inspectChatMetadata(api) {
|
||||
const cases = new Map(), catalogIds = new Set();
|
||||
let totalPages, totalElements;
|
||||
for (let page = 0; page < 10; page++) {
|
||||
const catalog = await api(`/api/testcase?${new URLSearchParams({ projectId: '1', page: String(page), size: '100', sort: 'id,asc' })}`);
|
||||
if (!Array.isArray(catalog?.content) || catalog.content.length > 100 || catalog.number !== page
|
||||
|| !Number.isSafeInteger(catalog.totalPages) || catalog.totalPages < 0 || catalog.totalPages > 10
|
||||
|| !Number.isSafeInteger(catalog.totalElements) || catalog.totalElements < 0 || catalog.totalElements > 1000
|
||||
|| totalPages !== undefined && (totalPages !== catalog.totalPages || totalElements !== catalog.totalElements)) fail('chat_testops_catalog_incomplete');
|
||||
totalPages = catalog.totalPages; totalElements = catalog.totalElements;
|
||||
const ids = catalog.content.map(row => {
|
||||
if (!id(row.id) || catalogIds.has(row.id)) fail('chat_testops_catalog_incomplete');
|
||||
catalogIds.add(row.id); return row.id;
|
||||
});
|
||||
for (let offset = 0; offset < ids.length; offset += 4) {
|
||||
const batch = ids.slice(offset, offset + 4);
|
||||
const details = await Promise.all(batch.map(value => api(`/api/testcase/${value}`)));
|
||||
for (const [index, value] of details.entries()) {
|
||||
if (value?.id !== batch[index] || value.projectId !== 1 || value.fullName != null && typeof value.fullName !== 'string') fail('testops_case_identity_invalid');
|
||||
if (!aliases.includes(value.fullName)) continue;
|
||||
if (value.deleted === true || typeof value.automated !== 'boolean') fail('testops_case_identity_invalid');
|
||||
cases.set(value.id, { id: value.id, fullName: value.fullName, automated: value.automated });
|
||||
}
|
||||
}
|
||||
if (page + 1 >= totalPages) break;
|
||||
if (page === 9) fail('chat_testops_catalog_incomplete');
|
||||
}
|
||||
if (catalogIds.size !== totalElements) fail('chat_testops_catalog_incomplete');
|
||||
const plans = rows(await api(`/api/testplan?${new URLSearchParams({ projectId: '1', name: planName, page: '0', size: '100' })}`))
|
||||
.filter(value => value.name === planName);
|
||||
if (plans.length > 1) fail('testops_plan_name_conflict');
|
||||
if (plans.some(value => !id(value.id) || value.projectId !== 1)) fail('testops_plan_identity_invalid');
|
||||
const integrations = rows(await api('/api/integration/project/1?page=0&size=100'))
|
||||
.filter(value => value.info?.type === 'github')
|
||||
.map(value => ({ id: value.id, type: 'github', disabled: typeof value.disabled === 'boolean' ? value.disabled : null }));
|
||||
if (integrations.some(value => !id(value.id))) fail('testops_integration_identity_invalid');
|
||||
return { projectId: 1, caseConflict: cases.size > 1, cases: [...cases.values()], plans: plans.map(value => ({ id: value.id, name: planName, count: value.testCasesCount })), integrations };
|
||||
}
|
||||
|
||||
export async function ensureChatPlan(api, inspected, markOnce) {
|
||||
if (inspected.projectId !== 1 || inspected.plans.length > 1) fail('testops_bootstrap_identity_invalid');
|
||||
if (inspected.caseConflict === true || inspected.cases.length > 1) fail('testops_case_alias_conflict');
|
||||
let selected = inspected.cases[0], plan = inspected.plans[0];
|
||||
if (plan && !selected) fail('testops_plan_without_matching_case');
|
||||
if (!selected || !plan) await markOnce();
|
||||
if (!selected) {
|
||||
const created = await api('/api/testcase', 'POST', { projectId: 1, name: 'Create a task backlog through chat', fullName,
|
||||
automated: true, description: 'Canonical scenario: tasks.create-backlog. Readiness and recorded outcomes remain separate.',
|
||||
expectedResult: 'The real agent creates the requested backlog; public Product E2E assertions and cleanup must pass.' });
|
||||
if (!id(created?.id) || created.projectId !== 1 || created.fullName !== fullName) fail('testops_created_case_identity_invalid');
|
||||
selected = { id: created.id, fullName, automated: created.automated === true };
|
||||
}
|
||||
if (!plan) {
|
||||
const created = await api('/api/testplan', 'POST', { projectId: 1, name: planName,
|
||||
treeSelection: { inverted: false, leafsInclude: [selected.id] } });
|
||||
if (!id(created?.id) || created.projectId !== 1 || created.name !== planName) fail('testops_created_plan_identity_invalid');
|
||||
plan = { id: created.id, name: planName };
|
||||
}
|
||||
const actual = await api(`/api/testplan/${plan.id}`);
|
||||
if (actual.projectId !== 1 || actual.name !== planName || actual.testCasesCount !== 1
|
||||
|| actual.treeSelection?.inverted === true || !same(actual.treeSelection?.leafsInclude, [selected.id])
|
||||
|| (actual.treeSelection?.groupsInclude?.length ?? 0) > 0 || (actual.treeSelection?.leafsExclude?.length ?? 0) > 0) fail('testops_plan_scope_mismatch');
|
||||
return { projectId: 1, case: selected, plan: { id: plan.id, name: planName, count: 1 },
|
||||
caseUrl: `${endpoint}/project/1/test-cases/${selected.id}`, planUrl: `${endpoint}/testplan/${plan.id}`,
|
||||
actualResult: null, note: 'Plan preparation is not an executed measurement or a passing result.' };
|
||||
}
|
||||
|
||||
|
||||
export const executionId = 'chat-features.runner-opencode-deepseek-1017.local.create-backlog';
|
||||
export const scenario = 'tasks.create-backlog';
|
||||
const positive = value => typeof value === 'string' && /^[1-9][0-9]{0,15}$/.test(value) && Number.isSafeInteger(Number(value));
|
||||
export function publicContext(env) {
|
||||
if (env.GITHUB_REPOSITORY !== 'paperclipai/paperclip' || env.GITHUB_REF !== 'refs/heads/master'
|
||||
|| env.GITHUB_EVENT_NAME !== 'workflow_dispatch' || env.GITHUB_RUN_ATTEMPT !== '1'
|
||||
|| env.GITHUB_WORKFLOW_REF !== 'paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master'
|
||||
|| !/^[a-f0-9]{40}$/.test(env.GITHUB_SHA ?? '') || !positive(env.GITHUB_RUN_ID)) fail('untrusted_public_testops_context');
|
||||
return { runId: env.GITHUB_RUN_ID, attempt: '1', workflowSha: env.GITHUB_SHA, targetSha: env.GITHUB_SHA };
|
||||
}
|
||||
|
||||
export async function validateChatSelection(api, env) {
|
||||
const context = publicContext(env);
|
||||
const names = ['ALLURE_JOB_RUN_ID', 'CHAT_TESTOPS_CASE_ID', 'CHAT_TESTOPS_PLAN_ID', 'CHAT_TESTOPS_JOB_ID'];
|
||||
if (names.some(name => !positive(env[name]))) fail('chat_testops_identity_missing');
|
||||
const [caseId, planId, jobId, jobRunId] = ['CHAT_TESTOPS_CASE_ID', 'CHAT_TESTOPS_PLAN_ID', 'CHAT_TESTOPS_JOB_ID', 'ALLURE_JOB_RUN_ID'].map(name => Number(env[name]));
|
||||
const selected = await api(`/api/testcase/${caseId}`);
|
||||
if (selected.id !== caseId || selected.projectId !== 1 || !aliases.includes(selected.fullName) || selected.deleted === true || selected.automated !== true) fail('chat_testops_case_mismatch');
|
||||
const plan = await api(`/api/testplan/${planId}`);
|
||||
if (plan.id !== planId || plan.projectId !== 1 || plan.name !== planName || plan.testCasesCount !== 1
|
||||
|| plan.treeSelection?.inverted === true || !same(plan.treeSelection?.leafsInclude, [caseId])
|
||||
|| (plan.treeSelection?.groupsInclude?.length ?? 0) > 0 || (plan.treeSelection?.leafsExclude?.length ?? 0) > 0) fail('chat_testops_plan_mismatch');
|
||||
const assignments = await api(`/api/testplan/${planId}/job`);
|
||||
if (!Array.isArray(assignments) || assignments.length !== 1 || assignments[0].job?.id !== jobId || assignments[0].testCasesCount !== 1) fail('chat_testops_plan_job_mismatch');
|
||||
const job = await api(`/api/job/${jobId}`);
|
||||
const jobUrls = ['https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml'];
|
||||
if (positive(env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID)) jobUrls.push(`https://github.com/paperclipai/paperclip/actions/workflows/${env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID}`);
|
||||
if (job.id !== jobId || job.type !== 'github' || !jobUrls.includes(job.url)) fail('chat_testops_job_mismatch');
|
||||
const run = await api(`/api/jobrun/${jobRunId}`);
|
||||
if (run.id !== jobRunId || run.job?.id !== jobId || run.job?.type !== 'github'
|
||||
|| !['scheduled', 'in_progress'].includes(run.stage) || !id(run.launchId)
|
||||
|| run.url !== `https://github.com/paperclipai/paperclip/actions/runs/${context.runId}`) fail('chat_testops_jobrun_mismatch');
|
||||
const launch = await api(`/api/launch/${run.launchId}`);
|
||||
if (launch.id !== run.launchId || launch.projectId !== 1 || launch.closed === true) fail('chat_testops_launch_mismatch');
|
||||
const selection = await api(`/api/jobrun/${jobRunId}/plan?expected=true`);
|
||||
if (!Array.isArray(selection) || selection.length !== 1 || selection[0].id !== caseId || selection[0].selector !== selected.fullName) fail('chat_testops_selection_mismatch');
|
||||
return { schema: 'paperclip.chat-feature-testops-selection/v1', ...context, allureJobRunId: String(jobRunId),
|
||||
executionId, canonicalScenario: scenario, projectId: 1, planId, jobId, caseId, fullName: selected.fullName, launchId: run.launchId };
|
||||
}
|
||||
|
||||
function configuredIds(env) {
|
||||
const names = ['CHAT_TESTOPS_CASE_ID', 'CHAT_TESTOPS_PLAN_ID', 'CHAT_TESTOPS_JOB_ID', 'CHAT_TESTOPS_OPERATION_PARAMETER_ID', 'CHAT_TESTOPS_INTEGRATION_ID'];
|
||||
if (names.some(name => !positive(env[name]))) fail('chat_testops_identity_missing');
|
||||
return Object.fromEntries(names.map((name, index) => [['caseId', 'planId', 'jobId', 'parameterId', 'integrationId'][index], Number(env[name])]));
|
||||
}
|
||||
function publicJobUrl(url, env) {
|
||||
return url === 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml'
|
||||
|| positive(env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID) && url === `https://github.com/paperclipai/paperclip/actions/workflows/${env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID}`;
|
||||
}
|
||||
async function registeredPlan(api, env) {
|
||||
const ids = configuredIds(env);
|
||||
const selected = await api(`/api/testcase/${ids.caseId}`);
|
||||
if (selected.id !== ids.caseId || selected.projectId !== 1 || !aliases.includes(selected.fullName) || selected.deleted === true || selected.automated !== true) fail('chat_testops_case_mismatch');
|
||||
const plan = await api(`/api/testplan/${ids.planId}`);
|
||||
if (plan.id !== ids.planId || plan.projectId !== 1 || plan.name !== planName || plan.testCasesCount !== 1
|
||||
|| plan.treeSelection?.inverted === true || !same(plan.treeSelection?.leafsInclude, [ids.caseId])
|
||||
|| (plan.treeSelection?.groupsInclude?.length ?? 0) > 0 || (plan.treeSelection?.leafsExclude?.length ?? 0) > 0) fail('chat_testops_plan_mismatch');
|
||||
const job = await api(`/api/job/${ids.jobId}`);
|
||||
if (job.id !== ids.jobId || job.type !== 'github' || !publicJobUrl(job.url, env)) fail('chat_testops_job_mismatch');
|
||||
const integration = await api(`/api/integration/${ids.integrationId}/project/1`);
|
||||
if (integration.id !== ids.integrationId || integration.projectId !== 1 || integration.info?.type !== 'github' || integration.disabled !== false) fail('chat_testops_integration_mismatch');
|
||||
const definition = await api(`/api/environment/${ids.parameterId}`);
|
||||
if (definition.id !== ids.parameterId || definition.name !== 'operation' || !Array.isArray(job.parameters) || job.parameters.length !== 1
|
||||
|| job.parameters[0].id !== ids.parameterId || !['bootstrap', 'evaluate'].includes(job.parameters[0].value)) fail('chat_testops_parameter_mismatch');
|
||||
return { ...ids, selected, plan, job };
|
||||
}
|
||||
|
||||
export async function bootstrapJobMetadata(api, env) {
|
||||
if (!positive(env.ALLURE_JOB_RUN_ID)) return { status: 'official_job_run_id_missing', actualResult: null };
|
||||
const run = await api(`/api/jobrun/${env.ALLURE_JOB_RUN_ID}`);
|
||||
if (!id(run?.job?.id)) fail('chat_testops_jobrun_mismatch');
|
||||
const job = await api(`/api/job/${run.job.id}`);
|
||||
if (job.id !== run.job.id || job.type !== 'github' || !Array.isArray(job.parameters) || job.parameters.length > 4) fail('chat_testops_job_mismatch');
|
||||
const parameters = [];
|
||||
for (const parameter of job.parameters) {
|
||||
if (!id(parameter.id)) fail('chat_testops_parameter_mismatch');
|
||||
const definition = await api(`/api/environment/${parameter.id}`);
|
||||
const known = ['operation', 'ALLURE_JOB_RUN_ID', 'ALLURE_USERNAME'].includes(definition.name);
|
||||
parameters.push({ id: parameter.id, name: known ? definition.name : 'UNKNOWN',
|
||||
value: definition.name === 'operation' && ['bootstrap', 'evaluate', 'launch-plan'].includes(parameter.value) ? parameter.value : 'NOT_PROJECTED' });
|
||||
}
|
||||
return { status: 'observed', projectId: 1, jobId: job.id, allureJobRunId: env.ALLURE_JOB_RUN_ID,
|
||||
publicWorkflowMatches: publicJobUrl(job.url, env), currentRunUrlMatches: run.url === `https://github.com/paperclipai/paperclip/actions/runs/${env.GITHUB_RUN_ID}`,
|
||||
launchId: id(run.launchId) ? run.launchId : null, parameters, actualResult: null };
|
||||
}
|
||||
|
||||
const launchAdmissions = new WeakSet();
|
||||
export async function admitLaunchPlan(env, fetchImpl = fetch) {
|
||||
const source = publicContext(env);
|
||||
const expected = env.CHAT_TESTOPS_LAUNCH_RUN_NUMBER, actual = env.GITHUB_RUN_NUMBER;
|
||||
if (!positive(expected) || !positive(actual) || actual !== expected || !positive(env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID)
|
||||
|| !positive(env.GITHUB_ACTOR_ID) || !positive(env.TESTOPS_TRIGGERING_ACTOR_ID) || !env.GH_TOKEN) fail('chat_testops_launch_admission');
|
||||
let allowed;
|
||||
try { allowed = JSON.parse(env.TESTOPS_ALLOWED_ACTOR_IDS ?? 'null'); } catch { fail('chat_testops_launch_admission'); }
|
||||
if (!Array.isArray(allowed) || !allowed.length || allowed.some(value => !id(value)) || !allowed.includes(Number(env.GITHUB_ACTOR_ID)) || !allowed.includes(Number(env.TESTOPS_TRIGGERING_ACTOR_ID))) fail('chat_testops_launch_admission');
|
||||
const read = async route => {
|
||||
const response = await fetchImpl(`https://api.github.com/repos/paperclipai/paperclip${route ? `/${route}` : ''}`, {
|
||||
method: 'GET', redirect: 'error', headers: { Accept: 'application/vnd.github+json', Authorization: `Bearer ${env.GH_TOKEN}` },
|
||||
signal: AbortSignal.timeout(30_000) });
|
||||
if (!response.ok) fail(`chat_testops_github_http_${response.status}`);
|
||||
const text = await response.text();
|
||||
if (Buffer.byteLength(text) > 512_000) fail('chat_testops_launch_admission');
|
||||
try { return JSON.parse(text); } catch { fail('chat_testops_launch_admission'); }
|
||||
};
|
||||
const repository = await read('');
|
||||
const workflow = await read('actions/workflows/chat-feature-testops.yml');
|
||||
const run = await read(`actions/runs/${source.runId}`);
|
||||
if (repository.full_name !== 'paperclipai/paperclip' || repository.private !== false || repository.default_branch !== 'master'
|
||||
|| workflow.id !== Number(env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID) || workflow.path !== '.github/workflows/chat-feature-testops.yml' || workflow.state !== 'active'
|
||||
|| run.id !== Number(source.runId) || run.workflow_id !== workflow.id || run.run_attempt !== 1 || run.run_number !== Number(expected)
|
||||
|| run.head_sha !== source.workflowSha || run.head_branch !== 'master' || run.event !== 'workflow_dispatch'
|
||||
|| run.repository?.full_name !== 'paperclipai/paperclip' || run.repository?.private !== false
|
||||
|| run.actor?.id !== Number(env.GITHUB_ACTOR_ID) || run.triggering_actor?.id !== Number(env.TESTOPS_TRIGGERING_ACTOR_ID) || !allowed.includes(run.triggering_actor.id)) fail('chat_testops_launch_admission');
|
||||
const receipt = Object.freeze({ schema: 'paperclip.chat-feature-testops-launch-admission/v1', ...source,
|
||||
workflowId: workflow.id, runNumber: run.run_number, expectedRunNumber: Number(expected), actorId: run.actor.id, triggeringActorId: run.triggering_actor.id });
|
||||
launchAdmissions.add(receipt); return receipt;
|
||||
}
|
||||
|
||||
export async function launchChatPlan(api, env, markOnce, admission) {
|
||||
const source = publicContext(env);
|
||||
if (!admission || !launchAdmissions.has(admission) || admission.runId !== source.runId || admission.workflowSha !== source.workflowSha
|
||||
|| admission.targetSha !== source.targetSha || String(admission.expectedRunNumber) !== env.CHAT_TESTOPS_LAUNCH_RUN_NUMBER
|
||||
|| String(admission.runNumber) !== env.GITHUB_RUN_NUMBER) fail('chat_testops_launch_admission');
|
||||
const nonce = env.CHAT_TESTOPS_LAUNCH_NONCE;
|
||||
if (!/^[a-f0-9]{32}$/.test(nonce ?? '')) fail('chat_testops_nonce_invalid');
|
||||
const known = await registeredPlan(api, env);
|
||||
const assignments = await api(`/api/testplan/${known.planId}/job`);
|
||||
if (!Array.isArray(assignments) || assignments.length > 1 || assignments.some(row => row.job?.id !== known.jobId || row.testCasesCount !== 1)) fail('chat_testops_plan_job_mismatch');
|
||||
const launchName = `${planName} · ${nonce}`;
|
||||
const existing = rows(await api(`/api/launch?${new URLSearchParams({ projectId: '1', search: launchName, page: '0', size: '100' })}`));
|
||||
if (existing.length) {
|
||||
if (existing.length !== 1 || existing[0].name !== launchName || !id(existing[0].id)) fail('chat_testops_nonce_conflict');
|
||||
return { schema: 'paperclip.chat-feature-testops-launch/v1', ...source, nonce, planId: known.planId,
|
||||
jobId: known.jobId, caseId: known.caseId, launchId: existing[0].id, status: 'EXISTING_NO_RESEND', actualResult: null };
|
||||
}
|
||||
await markOnce({ ...source, launchAdmission: admission, nonce, planId: known.planId, jobId: known.jobId, caseId: known.caseId });
|
||||
const parameters = [{ id: known.parameterId, value: 'evaluate' }];
|
||||
await api(`/api/job/${known.jobId}`, 'PATCH', { integrationId: known.integrationId, canRun: true, parameters });
|
||||
const patched = await api(`/api/job/${known.jobId}`);
|
||||
if (patched.id !== known.jobId || patched.type !== 'github' || !publicJobUrl(patched.url, env) || !same(patched.parameters, parameters)) fail('chat_testops_parameter_mismatch');
|
||||
if (!assignments.length) await api(`/api/testplan/${known.planId}/assign`, 'POST', { jobId: known.jobId,
|
||||
selection: { inverted: false, manual: false, leafsInclude: [known.caseId] } });
|
||||
await api(`/api/testplan/${known.planId}/jobparameter`, 'POST', { jobId: known.jobId, parameters });
|
||||
const bound = await api(`/api/testplan/${known.planId}/job`);
|
||||
if (!Array.isArray(bound) || bound.length !== 1 || bound[0].job?.id !== known.jobId || bound[0].testCasesCount !== 1
|
||||
|| !same(bound[0].job.parameters, parameters)) fail('chat_testops_plan_job_mismatch');
|
||||
const launched = await api(`/api/testplan/${known.planId}/run`, 'POST', { launchName });
|
||||
if (!id(launched?.id) || launched.projectId !== 1 || launched.name !== launchName) fail('chat_testops_launch_mismatch');
|
||||
return { schema: 'paperclip.chat-feature-testops-launch/v1', ...source, nonce, projectId: 1,
|
||||
planId: known.planId, jobId: known.jobId, caseId: known.caseId, launchId: launched.id,
|
||||
status: 'REQUESTED', integrationId: known.integrationId, integrationBindingRequested: true, integrationAssociationReadback: 'not_exposed_by_job_dto', executionId, canonicalScenario: scenario, actualRunId: null, actualResult: null };
|
||||
}
|
||||
|
||||
|
||||
export async function verifyChatUpload(api, env, selected, exported, { wait = ms => new Promise(resolve => setTimeout(resolve, ms)), now = Date.now } = {}) {
|
||||
const context = publicContext(env);
|
||||
if (selected.runId !== context.runId || selected.attempt !== '1' || selected.workflowSha !== context.workflowSha
|
||||
|| selected.targetSha !== context.targetSha || selected.allureJobRunId !== env.ALLURE_JOB_RUN_ID
|
||||
|| selected.executionId !== executionId || selected.canonicalScenario !== scenario || selected.projectId !== 1
|
||||
|| !aliases.includes(selected.fullName) || ['planId', 'jobId', 'caseId', 'launchId'].some(key => !id(selected[key]))
|
||||
|| exported.caseId !== selected.caseId || exported.campaignId !== `gha-${context.runId}-1` || exported.count !== 1
|
||||
|| !['passed', 'failed'].includes(exported.status) || !['passed', 'failed', 'not_started'].includes(exported.cleanup)
|
||||
|| exported.status === 'passed' && exported.cleanup !== 'passed' || !Array.isArray(exported.attachmentMetadata)
|
||||
|| exported.attachmentMetadata.length < 2 || exported.attachmentMetadata.length > 10) fail('chat_testops_upload_mismatch');
|
||||
const allowedAttachments = new Map([['Synthetic task and source conversation', 'image/png'],
|
||||
['Execution provenance and checks', 'application/json'], ['Sanitized Product E2E record and accounting', 'application/json']]);
|
||||
if (exported.attachmentMetadata.some(row => allowedAttachments.get(row.name) !== row.contentType
|
||||
|| !Number.isSafeInteger(row.contentLength) || row.contentLength <= 0 || row.contentLength > 12 * 1024 * 1024)
|
||||
|| exported.attachmentMetadata.filter(row => row.name === 'Execution provenance and checks').length !== 1
|
||||
|| exported.attachmentMetadata.filter(row => row.name === 'Sanitized Product E2E record and accounting').length !== 1
|
||||
|| exported.status === 'passed' && !exported.attachmentMetadata.some(row => row.contentType === 'image/png')) fail('chat_testops_upload_mismatch');
|
||||
const signature = rows => rows.map(row => JSON.stringify([row.name, row.contentType, row.contentLength])).sort();
|
||||
const runUrl = `https://github.com/paperclipai/paperclip/actions/runs/${context.runId}`;
|
||||
const sourceUrl = `https://github.com/paperclipai/paperclip/commit/${context.targetSha}`;
|
||||
const deadline = now() + 120_000;
|
||||
const get = async route => {
|
||||
const remaining = deadline - now();
|
||||
if (remaining < 1) fail('chat_testops_upload_unverified');
|
||||
const value = await api(route, 'GET', undefined, Math.min(30_000, Math.floor(remaining)));
|
||||
if (now() >= deadline) fail('chat_testops_upload_unverified');
|
||||
return value;
|
||||
};
|
||||
for (let poll = 0; poll < 12 && now() < deadline; poll++) {
|
||||
const run = await get(`/api/jobrun/${selected.allureJobRunId}`);
|
||||
if (run.id !== Number(selected.allureJobRunId) || run.job?.id !== selected.jobId || run.job?.type !== 'github'
|
||||
|| run.launchId !== selected.launchId || run.url !== runUrl || !['in_progress', 'finished'].includes(run.stage)) fail('chat_testops_jobrun_mismatch');
|
||||
const launch = await get(`/api/launch/${selected.launchId}`);
|
||||
if (launch.id !== selected.launchId || launch.projectId !== 1) fail('chat_testops_launch_mismatch');
|
||||
const listed = await get(`/api/testresult?${new URLSearchParams({ launchId: String(selected.launchId), page: '0', size: '100' })}`);
|
||||
if (!Array.isArray(listed?.content) || listed.content.length > 1 || !Number.isSafeInteger(listed.totalElements)
|
||||
|| listed.totalElements !== listed.content.length || !Number.isSafeInteger(listed.totalPages) || listed.totalPages < 0 || listed.totalPages > 1) fail('chat_testops_upload_mismatch');
|
||||
if (listed.content.length) {
|
||||
const row = listed.content[0];
|
||||
if (!id(row.id) || row.testCaseId !== selected.caseId) fail('chat_testops_upload_mismatch');
|
||||
const result = await get(`/api/testresult/${row.id}`);
|
||||
if (result.id !== row.id || result.projectId !== 1 || result.launchId !== selected.launchId || result.testCaseId !== selected.caseId
|
||||
|| result.fullName !== selected.fullName || result.jobRun?.id !== Number(selected.allureJobRunId)
|
||||
|| result.jobRun?.job?.id !== selected.jobId || result.jobRun?.launchId !== selected.launchId || result.jobRun?.url !== runUrl
|
||||
|| !Array.isArray(result.links) || !result.links.some(link => link.url === runUrl)
|
||||
|| !result.links.some(link => link.url === sourceUrl)) fail('chat_testops_upload_mismatch');
|
||||
if (!['passed', 'failed', 'broken', 'skipped', 'unknown'].includes(result.status)) fail('chat_testops_upload_mismatch');
|
||||
if (result.status !== 'unknown' && result.status !== exported.status) fail('chat_testops_upload_mismatch');
|
||||
const attached = await get(`/api/testresult/attachment?${new URLSearchParams({ testResultId: String(row.id), page: '0', size: '100' })}`);
|
||||
if (!Array.isArray(attached?.content) || attached.content.length > 10 || !Number.isSafeInteger(attached.totalElements)
|
||||
|| attached.totalElements !== attached.content.length || !Number.isSafeInteger(attached.totalPages) || attached.totalPages < 0 || attached.totalPages > 1
|
||||
|| new Set(attached.content.map(value => value.id)).size !== attached.content.length
|
||||
|| attached.content.some(value => !id(value.id) || value.missed === true || value.fromTestCase === true)) fail('chat_testops_upload_mismatch');
|
||||
if (now() < deadline && result.status === exported.status && same(signature(attached.content), signature(exported.attachmentMetadata))) {
|
||||
return { schema: 'paperclip.chat-feature-testops-upload-verification/v1', ...context, projectId: 1, planId: selected.planId,
|
||||
jobId: selected.jobId, allureJobRunId: selected.allureJobRunId, launchId: selected.launchId, caseId: selected.caseId,
|
||||
resultId: result.id, executionId, status: result.status, cleanup: exported.cleanup, verification: 'VERIFIED',
|
||||
attachments: exported.attachmentMetadata.map(value => ({ ...value })),
|
||||
launchUrl: `${endpoint}/launch/${selected.launchId}`, resultUrl: `${endpoint}/testresult/${result.id}` };
|
||||
}
|
||||
}
|
||||
if (poll < 11 && now() + 5000 < deadline) await wait(5000);
|
||||
}
|
||||
fail('chat_testops_upload_unverified');
|
||||
}
|
||||
@@ -0,0 +1,392 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { mkdtemp, readFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { endpoint, planName, fullName, inspectChatMetadata, ensureChatPlan, testopsClient, failureProjection } from './chat-feature-testops-api.mjs';
|
||||
|
||||
const caseRow = { id: 91, projectId: 1, fullName, automated: true };
|
||||
const plan = { id: 8, projectId: 1, name: planName, testCasesCount: 1, treeSelection: { inverted: false, leafsInclude: [91] } };
|
||||
const page = content => ({ content, totalPages: 1 });
|
||||
const inspectApi = (cases = [caseRow], plans = [plan]) => async route => {
|
||||
if (route.startsWith('/api/testcase?')) return { content: cases.map(row => ({ id: row.id, name: 'Not exported', automated: row.automated })), totalElements: cases.length, totalPages: cases.length ? 1 : 0, number: 0 };
|
||||
if (/^\/api\/testcase\/[0-9]+$/.test(route)) return cases.find(row => row.id === Number(route.split('/').at(-1)));
|
||||
if (route.startsWith('/api/testplan?')) return page(plans);
|
||||
if (route.startsWith('/api/integration/project/1')) return page([{ id: 2, info: { type: 'github' }, disabled: false, settings: { secret: 'DO_NOT_EXPORT' } }]);
|
||||
assert.fail(route);
|
||||
};
|
||||
|
||||
test('inspect reconciles exact historical case identity and discards integration settings', async () => {
|
||||
const result = await inspectChatMetadata(inspectApi());
|
||||
assert.deepEqual(result.cases, [{ id: 91, fullName, automated: true }]);
|
||||
assert.deepEqual(result.integrations, [{ id: 2, type: 'github', disabled: false }]);
|
||||
assert.ok(!JSON.stringify(result).includes('DO_NOT_EXPORT'));
|
||||
});
|
||||
|
||||
test('multiple alias identities, foreign project and incomplete metadata stop before mutation', async () => {
|
||||
const conflicts = await inspectChatMetadata(inspectApi([caseRow, { ...caseRow, id: 92, fullName: 'paperclip.Product E2E.chat-features.tasks.create-backlog' }]));
|
||||
assert.equal(conflicts.caseConflict, true); assert.deepEqual(conflicts.cases.map(row => row.id), [91, 92]);
|
||||
await assert.rejects(ensureChatPlan(() => assert.fail('no ambiguous write'), conflicts, () => assert.fail('no ambiguous marker')));
|
||||
await assert.rejects(inspectChatMetadata(inspectApi([{ ...caseRow, projectId: 2 }])), /identity_invalid/);
|
||||
await assert.rejects(inspectChatMetadata(async () => ({ content: [], totalPages: 11, totalElements: 1100, number: 0 })), /catalog_incomplete/);
|
||||
});
|
||||
|
||||
test('existing exact plan reuses case and plan without a write', async () => {
|
||||
const result = await ensureChatPlan(async route => { assert.equal(route, '/api/testplan/8'); return plan; },
|
||||
await inspectChatMetadata(inspectApi()), () => assert.fail('must not mark or write'));
|
||||
assert.equal(result.case.id, 91);
|
||||
assert.equal(result.plan.id, 8);
|
||||
assert.equal(result.actualResult, null);
|
||||
});
|
||||
|
||||
test('bootstrap creates only the missing exact plan after exclusive marker and reads back scope', async () => {
|
||||
const calls = [];
|
||||
const result = await ensureChatPlan(async (route, method, body) => {
|
||||
calls.push({ route, method, body });
|
||||
if (method === 'POST') { assert.equal(calls[0].route, 'MARK'); assert.deepEqual(body, { projectId: 1, name: planName, treeSelection: { inverted: false, leafsInclude: [91] } }); }
|
||||
return plan;
|
||||
}, { projectId: 1, cases: [caseRow], plans: [] }, async () => calls.push({ route: 'MARK' }));
|
||||
assert.equal(calls.filter(row => row.method === 'POST').length, 1);
|
||||
assert.equal(result.plan.count, 1);
|
||||
});
|
||||
|
||||
test('empty project bootstrap links the genuinely created case into only the new plan', async () => {
|
||||
const calls = [];
|
||||
const createdCase = { ...caseRow, id: 92 };
|
||||
const createdPlan = { ...plan, treeSelection: { inverted: false, leafsInclude: [92] } };
|
||||
const result = await ensureChatPlan(async (route, method, body) => {
|
||||
calls.push({ route, method, body });
|
||||
assert.equal(calls[0].route, 'MARK');
|
||||
if (route === '/api/testcase') {
|
||||
assert.equal(body.projectId, 1); assert.equal(body.fullName, fullName); assert.equal(body.automated, true);
|
||||
return createdCase;
|
||||
}
|
||||
if (route === '/api/testplan') {
|
||||
assert.deepEqual(body, { projectId: 1, name: planName, treeSelection: { inverted: false, leafsInclude: [92] } });
|
||||
return createdPlan;
|
||||
}
|
||||
assert.equal(route, '/api/testplan/8'); assert.equal(method, undefined); return createdPlan;
|
||||
}, { projectId: 1, cases: [], plans: [] }, async () => calls.push({ route: 'MARK' }));
|
||||
assert.equal(result.case.id, 92); assert.equal(result.plan.id, 8);
|
||||
assert.deepEqual(calls.map(call => call.route), ['MARK', '/api/testcase', '/api/testplan', '/api/testplan/8']);
|
||||
});
|
||||
|
||||
test('known conflicting plan stops before creating an otherwise missing case or marker', async () => {
|
||||
await assert.rejects(ensureChatPlan(() => assert.fail('no API write or read needed'),
|
||||
{ projectId: 1, cases: [], plans: [plan] }, () => assert.fail('must not mark')), /plan_without_matching_case/);
|
||||
});
|
||||
|
||||
test('new case is created only after alias reconciliation; an uncertain mutation is never resent', async () => {
|
||||
let writes = 0, markers = 0;
|
||||
await assert.rejects(ensureChatPlan(async (_route, method) => {
|
||||
assert.equal(method, 'POST'); writes++; throw new Error('uncertain transport');
|
||||
}, { projectId: 1, cases: [], plans: [] }, async () => { markers++; }), /uncertain/);
|
||||
assert.equal(writes, 1); assert.equal(markers, 1);
|
||||
});
|
||||
|
||||
test('existing broad or conflicting plan never expands or changes its scope', async () => {
|
||||
for (const actual of [{ ...plan, testCasesCount: 2 }, { ...plan, treeSelection: { leafsInclude: [92] } },
|
||||
{ ...plan, treeSelection: { leafsInclude: [91], inverted: true } }, { ...plan, projectId: 2 }]) {
|
||||
await assert.rejects(ensureChatPlan(async (_route, method) => { assert.equal(method, undefined); return actual; },
|
||||
{ projectId: 1, cases: [caseRow], plans: [plan] }, () => assert.fail('no writes')), /scope_mismatch/);
|
||||
}
|
||||
});
|
||||
|
||||
test('API credential stays in memory, redirects fail closed and response text is not an error', async () => {
|
||||
const calls = [];
|
||||
const api = await testopsClient('PRIVATE_API_TOKEN', async (url, options) => {
|
||||
calls.push({ url, options });
|
||||
return calls.length === 1 ? { ok: true, json: async () => ({ access_token: 'PRIVATE_BEARER' }) }
|
||||
: { ok: false, status: 403, text: () => assert.fail('must not inspect arbitrary error body') };
|
||||
});
|
||||
await assert.rejects(api('/api/testplan?projectId=1'), error => { assert.deepEqual(failureProjection(error, 'metadata_inspection'), { stage: 'metadata_inspection', code: 'testops_api_http', httpStatus: 403 }); return true; });
|
||||
assert.equal(calls[0].url, `${endpoint}/api/uaa/oauth/token`);
|
||||
assert.equal(calls[1].options.redirect, 'error');
|
||||
await assert.rejects(api('https://foreign.invalid'), /route_invalid/);
|
||||
assert.equal(calls.length, 2);
|
||||
});
|
||||
|
||||
test('unknown exception text is never copied into closed diagnostics', () => {
|
||||
const value = failureProjection(new Error('PRIVATE_RESPONSE_TOKEN_BODY'), 'untrusted raw stage');
|
||||
assert.deepEqual(value, { stage: 'unknown', code: 'unknown_failure', httpStatus: null });
|
||||
assert.ok(!JSON.stringify(value).includes('PRIVATE'));
|
||||
});
|
||||
|
||||
|
||||
import { validateChatSelection } from './chat-feature-testops-api.mjs';
|
||||
const publicEnv = { GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/heads/master',
|
||||
GITHUB_EVENT_NAME: 'workflow_dispatch', GITHUB_RUN_ATTEMPT: '1', GITHUB_SHA: 'a'.repeat(40), GITHUB_RUN_ID: '123',
|
||||
GITHUB_WORKFLOW_REF: 'paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master',
|
||||
ALLURE_JOB_RUN_ID: '9', CHAT_TESTOPS_CASE_ID: '91', CHAT_TESTOPS_PLAN_ID: '8', CHAT_TESTOPS_JOB_ID: '7' };
|
||||
const selectedApi = replacements => async route => {
|
||||
const values = { '/api/testcase/91': caseRow, '/api/testplan/8': plan,
|
||||
'/api/testplan/8/job': [{ job: { id: 7 }, testCasesCount: 1 }],
|
||||
'/api/job/7': { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml' },
|
||||
'/api/jobrun/9': { id: 9, job: { id: 7, type: 'github' }, stage: 'scheduled', launchId: 10, url: 'https://github.com/paperclipai/paperclip/actions/runs/123' },
|
||||
'/api/launch/10': { id: 10, projectId: 1, closed: false },
|
||||
'/api/jobrun/9/plan?expected=true': [{ id: 91, selector: fullName }] };
|
||||
assert.ok(Object.hasOwn(values, route));
|
||||
return replacements?.[route] ?? values[route];
|
||||
};
|
||||
|
||||
test('real API selection is exact singleton/project/job/launch and current source bound before model access', async () => {
|
||||
const selected = await validateChatSelection(selectedApi(), publicEnv);
|
||||
assert.equal(selected.runId, '123'); assert.equal(selected.workflowSha, 'a'.repeat(40));
|
||||
assert.equal(selected.executionId, 'chat-features.runner-opencode-deepseek-1017.local.create-backlog');
|
||||
assert.equal(selected.canonicalScenario, 'tasks.create-backlog'); assert.equal(selected.allureJobRunId, '9');
|
||||
assert.equal(selected.launchId, 10);
|
||||
});
|
||||
|
||||
test('selection refuses missing correlation, a different case, broad plan, foreign job or closed launch', async () => {
|
||||
await assert.rejects(validateChatSelection(selectedApi(), { ...publicEnv, ALLURE_JOB_RUN_ID: '' }), /identity_missing/);
|
||||
for (const [route, value] of [
|
||||
['/api/testcase/91', { ...caseRow, projectId: 2 }],
|
||||
['/api/testplan/8', { ...plan, testCasesCount: 2 }],
|
||||
['/api/job/7', { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip-cloud/actions/workflows/qa-staging-smoke.yml' }],
|
||||
['/api/testplan/8/job', [{ job: { id: 70 }, testCasesCount: 1 }]],
|
||||
['/api/jobrun/9', { id: 9, job: { id: 70, type: 'github' }, stage: 'scheduled', launchId: 10 }],
|
||||
['/api/jobrun/9', { id: 9, job: { id: 7, type: 'github' }, stage: 'scheduled', launchId: 10, url: 'https://github.com/paperclipai/paperclip/actions/runs/456' }],
|
||||
['/api/launch/10', { id: 10, projectId: 1, closed: true }],
|
||||
['/api/jobrun/9/plan?expected=true', [{ id: 92, selector: fullName }]],
|
||||
]) await assert.rejects(validateChatSelection(selectedApi({ [route]: value }), publicEnv), /mismatch/);
|
||||
});
|
||||
|
||||
test('selection rejects every non-master, foreign and second-attempt execution before API access', async () => {
|
||||
for (const change of [{ GITHUB_REF: 'refs/heads/feature' }, { GITHUB_REPOSITORY: 'foreign' },
|
||||
{ GITHUB_EVENT_NAME: 'pull_request' }, { GITHUB_RUN_ATTEMPT: '2' }, { GITHUB_WORKFLOW_REF: 'other' }]) {
|
||||
await assert.rejects(validateChatSelection(() => assert.fail('must stop before API'), { ...publicEnv, ...change }), /untrusted/);
|
||||
}
|
||||
});
|
||||
|
||||
import { launchChatPlan, bootstrapJobMetadata, admitLaunchPlan } from './chat-feature-testops-api.mjs';
|
||||
const launchEnv = { ...publicEnv, ALLURE_JOB_RUN_ID: '', CHAT_TESTOPS_OPERATION_PARAMETER_ID: '17', CHAT_TESTOPS_INTEGRATION_ID: '2', CHAT_TESTOPS_LAUNCH_NONCE: '1'.repeat(32), CHAT_TESTOPS_LAUNCH_RUN_NUMBER: '2', GITHUB_RUN_NUMBER: '2',
|
||||
CHAT_TESTOPS_GITHUB_WORKFLOW_ID: '99', GITHUB_ACTOR_ID: '34892728', TESTOPS_TRIGGERING_ACTOR_ID: '34892728', TESTOPS_ALLOWED_ACTOR_IDS: '[34892728]', GH_TOKEN: 'SYNTHETIC_GH_TOKEN' };
|
||||
const ownRun = { id: 123, workflow_id: 99, run_attempt: 1, run_number: 2, head_sha: 'a'.repeat(40), head_branch: 'master', event: 'workflow_dispatch',
|
||||
repository: { full_name: 'paperclipai/paperclip', private: false }, actor: { id: 34892728 }, triggering_actor: { id: 34892728 } };
|
||||
async function fence(env = launchEnv, changes = {}, workflowChanges = {}, repoChanges = {}) {
|
||||
return admitLaunchPlan(env, async url => ({ ok: true, text: async () => JSON.stringify(url.endsWith('/repos/paperclipai/paperclip') ? { full_name: 'paperclipai/paperclip', private: false, default_branch: 'master', ...repoChanges }
|
||||
: url.includes('/actions/runs/') ? { ...ownRun, ...changes }
|
||||
: { id: 99, path: '.github/workflows/chat-feature-testops.yml', state: 'active', ...workflowChanges }) }));
|
||||
}
|
||||
function launchServer({ wrongParameter = false, wrongAssignment = false, uncertainRun = false, integrationChange = {} } = {}) {
|
||||
const calls = [], parameters = [{ id: 17, value: 'bootstrap' }]; let assigned = false, launched = false, requests = 0;
|
||||
const api = async (route, method, body) => {
|
||||
calls.push({ route, method, body });
|
||||
if (route.startsWith('/api/launch?')) return { totalPages: 1, content: launched ? [{ id: 10, name: `Chat feature evals · ${'1'.repeat(32)}` }] : [] };
|
||||
if (route === '/api/integration/2/project/1') return { id: 2, projectId: 1, info: { type: 'github' }, disabled: false, ...integrationChange };
|
||||
if (route === '/api/environment/17') return { id: 17, name: wrongParameter ? 'OTHER_PRIVATE_PARAMETER' : 'operation' };
|
||||
if (route === '/api/job/7') {
|
||||
if (method === 'PATCH') { assert.deepEqual(body, { integrationId: 2, canRun: true, parameters: [{ id: 17, value: 'evaluate' }] }); parameters[0].value = 'evaluate'; }
|
||||
return { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml', parameters };
|
||||
}
|
||||
if (route === '/api/testplan/8/job') return wrongAssignment ? [{ job: { id: 70 }, testCasesCount: 1 }]
|
||||
: assigned ? [{ job: { id: 7, parameters }, testCasesCount: 1 }] : [];
|
||||
if (route === '/api/testplan/8/assign') { assert.equal(method, 'POST'); assert.deepEqual(body, { jobId: 7, selection: { inverted: false, manual: false, leafsInclude: [91] } }); assigned = true; return null; }
|
||||
if (route === '/api/testplan/8/jobparameter') { assert.deepEqual(body, { jobId: 7, parameters: [{ id: 17, value: 'evaluate' }] }); return null; }
|
||||
if (route === '/api/testplan/8/run') { requests++; assert.equal(method, 'POST'); if (uncertainRun) throw new Error('synthetic uncertain transport'); launched = true; return { id: 10, projectId: 1, name: body.launchName }; }
|
||||
return selectedApi()(route);
|
||||
};
|
||||
return { api, calls, requests: () => requests };
|
||||
}
|
||||
|
||||
test('launch configures only the exact registered public job/plan then issues one run request and exits', async () => {
|
||||
const server = launchServer(); let markers = 0;
|
||||
const result = await launchChatPlan(server.api, launchEnv, async () => { markers++; assert.equal(server.calls.some(row => row.method), false); }, await fence());
|
||||
assert.equal(markers, 1); assert.equal(server.requests(), 1); assert.equal(result.status, 'REQUESTED');
|
||||
assert.equal(result.launchId, 10); assert.equal(result.actualRunId, null); assert.equal(result.actualResult, null);
|
||||
assert.ok(!server.calls.some(row => /jobrun.*rerun|dispatches/.test(row.route)));
|
||||
});
|
||||
|
||||
test('observed launch nonce prevents a second run request across separate local entry attempts', async () => {
|
||||
const server = launchServer(); await launchChatPlan(server.api, launchEnv, async () => {}, await fence());
|
||||
const result = await launchChatPlan(server.api, launchEnv, () => assert.fail('existing launch must not mark or write'), await fence());
|
||||
assert.equal(result.status, 'EXISTING_NO_RESEND'); assert.equal(server.requests(), 1);
|
||||
});
|
||||
|
||||
test('foreign assignment and unobserved operation parameter stop before any write', async () => {
|
||||
for (const options of [{ wrongAssignment: true }, { wrongParameter: true }]) {
|
||||
const server = launchServer(options);
|
||||
await assert.rejects(launchChatPlan(server.api, launchEnv, () => assert.fail('no marker'), await fence()));
|
||||
assert.equal(server.calls.some(row => row.method), false); assert.equal(server.requests(), 0);
|
||||
}
|
||||
});
|
||||
|
||||
test('uncertain plan-run transport is sent once and never followed by another mutation', async () => {
|
||||
const server = launchServer({ uncertainRun: true });
|
||||
await assert.rejects(launchChatPlan(server.api, launchEnv, async () => {}, await fence()), /uncertain/);
|
||||
assert.equal(server.requests(), 1);
|
||||
assert.equal(server.calls.at(-1).route, '/api/testplan/8/run');
|
||||
});
|
||||
|
||||
test('official bootstrap job metadata projects only fixed parameter names and operation enums', async () => {
|
||||
const job = await bootstrapJobMetadata(async route => {
|
||||
if (route === '/api/jobrun/9') return { id: 9, job: { id: 7 }, launchId: 10, url: 'https://github.com/paperclipai/paperclip/actions/runs/123' };
|
||||
if (route === '/api/job/7') return { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml', parameters: [{ id: 17, value: 'bootstrap' }, { id: 18, value: 'SYNTHETIC_PRIVATE_VALUE' }] };
|
||||
if (route === '/api/environment/17') return { id: 17, name: 'operation' };
|
||||
return { id: 18, name: 'UNRELATED_NAME' };
|
||||
}, publicEnv);
|
||||
assert.equal(job.publicWorkflowMatches, true); assert.equal(job.parameters[0].value, 'bootstrap');
|
||||
assert.deepEqual(job.parameters[1], { id: 18, name: 'UNKNOWN', value: 'NOT_PROJECTED' });
|
||||
assert.ok(!JSON.stringify(job).includes('SYNTHETIC_PRIVATE_VALUE'));
|
||||
});
|
||||
|
||||
test('documented stable-ID list pagination is complete before locally matching exact alias detail metadata', async () => {
|
||||
const calls = [];
|
||||
const api = async route => {
|
||||
calls.push(route);
|
||||
if (route.startsWith('/api/testcase?')) {
|
||||
const query = new URLSearchParams(route.split('?')[1]);
|
||||
assert.equal(query.get('projectId'), '1'); assert.equal(query.get('sort'), 'id,asc');
|
||||
const number = Number(query.get('page'));
|
||||
return { number, totalPages: 2, totalElements: 2, content: [{ id: number ? 92 : 91 }] };
|
||||
}
|
||||
if (route === '/api/testcase/91') return { ...caseRow, description: 'SYNTHETIC_PRIVATE_DESCRIPTION' };
|
||||
if (route === '/api/testcase/92') return { id: 92, projectId: 1, fullName: 'paperclip.Product E2E.unrelated.case', automated: true, expectedResult: 'DO_NOT_EXPORT' };
|
||||
return inspectApi([], [])(route);
|
||||
};
|
||||
const result = await inspectChatMetadata(api);
|
||||
assert.deepEqual(result.cases, [{ id: 91, fullName, automated: true }]);
|
||||
assert.ok(calls.some(route => route.includes('page=1')));
|
||||
assert.ok(!calls.some(route => route.includes('__search')));
|
||||
assert.ok(!JSON.stringify(result).includes('SYNTHETIC_PRIVATE_DESCRIPTION'));
|
||||
assert.ok(!JSON.stringify(result).includes('DO_NOT_EXPORT'));
|
||||
});
|
||||
|
||||
test('changing catalog totals or repeated page IDs cannot become a not-found result', async () => {
|
||||
for (const duplicate of [false, true]) {
|
||||
await assert.rejects(inspectChatMetadata(async route => {
|
||||
if (route.startsWith('/api/testcase?')) {
|
||||
const number = Number(new URLSearchParams(route.split('?')[1]).get('page'));
|
||||
return { number, totalPages: 2, totalElements: number && !duplicate ? 3 : 2, content: [{ id: number && !duplicate ? 92 : 91 }] };
|
||||
}
|
||||
return caseRow;
|
||||
}), /catalog_incomplete/);
|
||||
}
|
||||
});
|
||||
|
||||
test('catalog detail reads have a finite four-request concurrency bound', async () => {
|
||||
let active = 0, maximum = 0;
|
||||
const api = async route => {
|
||||
if (route.startsWith('/api/testcase?')) return { number: 0, totalPages: 1, totalElements: 7, content: Array.from({ length: 7 }, (_, index) => ({ id: index + 1 })) };
|
||||
if (/^\/api\/testcase\/[0-9]+$/.test(route)) {
|
||||
active++; maximum = Math.max(maximum, active); await new Promise(resolve => queueMicrotask(resolve)); active--;
|
||||
return { id: Number(route.split('/').at(-1)), projectId: 1, fullName: null, automated: false };
|
||||
}
|
||||
return inspectApi([], [])(route);
|
||||
};
|
||||
assert.deepEqual((await inspectChatMetadata(api)).cases, []);
|
||||
assert.equal(maximum, 4);
|
||||
});
|
||||
|
||||
test('fresh dispatch or replay cannot acquire the fixed initial launch counter even with stale TestOps search', async () => {
|
||||
for (const env of [{ ...launchEnv, GITHUB_RUN_NUMBER: '3' }, { ...launchEnv, GITHUB_RUN_ATTEMPT: '2' },
|
||||
{ ...launchEnv, CHAT_TESTOPS_LAUNCH_RUN_NUMBER: '' }]) {
|
||||
await assert.rejects(admitLaunchPlan(env, () => assert.fail('counter/refusal precedes all remote reads')));
|
||||
}
|
||||
});
|
||||
|
||||
test('real own-run metadata rejects counter, source, workflow, actor and default-ref contradictions', async () => {
|
||||
for (const changes of [{ run_number: 3 }, { head_sha: 'b'.repeat(40) }, { workflow_id: 100 },
|
||||
{ actor: { id: 1 } }, { triggering_actor: { id: 1 } }, { run_attempt: 2 }, { head_branch: 'feature' }]) {
|
||||
await assert.rejects(fence(launchEnv, changes), /launch_admission/);
|
||||
}
|
||||
await assert.rejects(fence(launchEnv, {}, {}, { default_branch: 'other' }), /launch_admission/);
|
||||
await assert.rejects(fence(launchEnv, {}, { state: 'disabled_manually' }), /launch_admission/);
|
||||
});
|
||||
|
||||
test('no caller-supplied JSON can replace the source-owned GitHub launch admission receipt', async () => {
|
||||
const actual = await fence();
|
||||
await assert.rejects(launchChatPlan(() => assert.fail('no API mutation'), launchEnv, () => assert.fail('no marker'), JSON.parse(JSON.stringify(actual))), /launch_admission/);
|
||||
});
|
||||
|
||||
test('GitHub metadata denial retains only its fixed code and HTTP status', async () => {
|
||||
await assert.rejects(admitLaunchPlan(launchEnv, async () => ({ ok: false, status: 403, text: () => assert.fail('must not read denial body') })), error => {
|
||||
assert.deepEqual(failureProjection(error, 'launch_admission'), { stage: 'launch_admission', code: 'chat_testops_github_http', httpStatus: 403 }); return true;
|
||||
});
|
||||
});
|
||||
|
||||
test('new job receives only the reviewed enabled GitHub integration; disabled/foreign identities stop before writes', async () => {
|
||||
for (const integrationChange of [{ id: 3 }, { projectId: 2 }, { disabled: true }, { disabled: null }, { info: { type: 'gitlab' } }]) {
|
||||
const server = launchServer({ integrationChange });
|
||||
await assert.rejects(launchChatPlan(server.api, launchEnv, () => assert.fail('no marker'), await fence()), /integration_mismatch/);
|
||||
assert.equal(server.calls.some(row => row.method), false);
|
||||
}
|
||||
});
|
||||
|
||||
const uploadContext = { GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/heads/master', GITHUB_EVENT_NAME: 'workflow_dispatch',
|
||||
GITHUB_RUN_ATTEMPT: '1', GITHUB_SHA: 'a'.repeat(40), GITHUB_RUN_ID: '123', ALLURE_JOB_RUN_ID: '9',
|
||||
GITHUB_WORKFLOW_REF: 'paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master' };
|
||||
const uploadSelection = { runId: '123', attempt: '1', workflowSha: 'a'.repeat(40), targetSha: 'a'.repeat(40), allureJobRunId: '9',
|
||||
executionId: 'chat-features.runner-opencode-deepseek-1017.local.create-backlog', canonicalScenario: 'tasks.create-backlog', projectId: 1,
|
||||
planId: 8, jobId: 7, caseId: 91, fullName, launchId: 10 };
|
||||
const uploadReceipt = { status: 'passed', cleanup: 'passed', caseId: 91, campaignId: 'gha-123-1', count: 1,
|
||||
attachmentMetadata: [{ name: 'Synthetic task and source conversation', contentType: 'image/png', contentLength: 400 },
|
||||
{ name: 'Execution provenance and checks', contentType: 'application/json', contentLength: 300 },
|
||||
{ name: 'Sanitized Product E2E record and accounting', contentType: 'application/json', contentLength: 800 }] };
|
||||
const uploadRun = { id: 9, job: { id: 7, type: 'github' }, launchId: 10, url: 'https://github.com/paperclipai/paperclip/actions/runs/123', stage: 'finished' };
|
||||
function uploadApi(change = {}, absent = 0) {
|
||||
let polls = 0;
|
||||
return async (route, method) => {
|
||||
assert.equal(method, 'GET');
|
||||
if (route === '/api/jobrun/9') return uploadRun;
|
||||
if (route === '/api/launch/10') return { id: 10, projectId: 1 };
|
||||
if (route.startsWith('/api/testresult?')) { polls++; return { content: polls <= absent ? [] : [{ id: 45, testCaseId: 91 }], totalElements: polls <= absent ? 0 : 1, totalPages: polls <= absent ? 0 : 1 }; }
|
||||
if (route === '/api/testresult/45') return { id: 45, projectId: 1, launchId: 10, testCaseId: 91, fullName, jobRun: uploadRun, status: 'passed',
|
||||
links: [{ url: uploadRun.url }, { url: `https://github.com/paperclipai/paperclip/commit/${'a'.repeat(40)}` }],
|
||||
get trace() { assert.fail('must not inspect trace'); }, ...change };
|
||||
if (route.startsWith('/api/testresult/attachment?')) return { content: uploadReceipt.attachmentMetadata.map((row, i) => ({ ...row, id: i + 1, missed: false, fromTestCase: false })), totalElements: 3, totalPages: 1 };
|
||||
assert.fail(route);
|
||||
};
|
||||
}
|
||||
|
||||
test('readback verifies exact current launch/case/source/status and safe attachment metadata after processing lag', async () => {
|
||||
const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs');
|
||||
let waits = 0;
|
||||
const result = await verifyChatUpload(uploadApi({}, 1), uploadContext, uploadSelection, uploadReceipt, { wait: async () => { waits++; }, now: () => 0 });
|
||||
assert.equal(waits, 1); assert.equal(result.verification, 'VERIFIED'); assert.equal(result.resultId, 45);
|
||||
assert.equal(result.status, 'passed'); assert.deepEqual(result.attachments, uploadReceipt.attachmentMetadata);
|
||||
assert.ok(!JSON.stringify(result).includes('trace'));
|
||||
});
|
||||
|
||||
test('readback rejects wrong launch, case, current jobrun, source and contradictory result status', async () => {
|
||||
const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs');
|
||||
for (const change of [{ launchId: 11 }, { testCaseId: 92 }, { jobRun: { ...uploadRun, id: 11 } }, { links: [] }, { status: 'failed' }]) {
|
||||
await assert.rejects(verifyChatUpload(uploadApi(change), uploadContext, uploadSelection, uploadReceipt), /upload_mismatch/);
|
||||
}
|
||||
});
|
||||
|
||||
test('readback missing result or screenshot metadata is bounded unverified, never a fabricated pass', async () => {
|
||||
const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs');
|
||||
let waits = 0;
|
||||
await assert.rejects(verifyChatUpload(uploadApi({}, 100), uploadContext, uploadSelection, uploadReceipt,
|
||||
{ wait: async () => { waits++; }, now: () => 0 }), /upload_unverified/);
|
||||
assert.equal(waits, 11);
|
||||
const base = uploadApi();
|
||||
await assert.rejects(verifyChatUpload(async (route, ...args) => route.startsWith('/api/testresult/attachment?')
|
||||
? { content: [], totalElements: 0, totalPages: 0 } : base(route, ...args), uploadContext, uploadSelection, uploadReceipt,
|
||||
{ wait: async () => {}, now: () => 0 }), /upload_unverified/);
|
||||
});
|
||||
|
||||
test('verified failed Product E2E remains failed and cleanup cannot be promoted', async () => {
|
||||
const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs');
|
||||
const result = await verifyChatUpload(uploadApi({ status: 'failed' }), uploadContext, uploadSelection, { ...uploadReceipt, status: 'failed', cleanup: 'failed' });
|
||||
assert.equal(result.status, 'failed'); assert.equal(result.cleanup, 'failed');
|
||||
await assert.rejects(verifyChatUpload(uploadApi(), uploadContext, uploadSelection, { ...uploadReceipt, cleanup: 'failed' }), /upload_mismatch/);
|
||||
});
|
||||
|
||||
|
||||
test('readback stops immediately after a late GET and gives each client only the remaining deadline', async () => {
|
||||
const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs');
|
||||
let time = 0, calls = 0;
|
||||
await assert.rejects(verifyChatUpload(async (route, method, body, timeout) => {
|
||||
calls++; assert.equal(route, '/api/jobrun/9'); assert.equal(timeout, 30_000); time = 120_001; return uploadRun;
|
||||
}, uploadContext, uploadSelection, uploadReceipt, { now: () => time }), /upload_unverified/);
|
||||
assert.equal(calls, 1);
|
||||
time = 0; calls = 0;
|
||||
await assert.rejects(verifyChatUpload(async (route, method, body, timeout) => {
|
||||
calls++;
|
||||
if (calls === 1) { assert.equal(timeout, 30_000); time = 119_990; return uploadRun; }
|
||||
assert.equal(route, '/api/launch/10'); assert.equal(timeout, 10); time = 120_001; return { id: 10, projectId: 1 };
|
||||
}, uploadContext, uploadSelection, uploadReceipt, { now: () => time }), /upload_unverified/);
|
||||
assert.equal(calls, 2);
|
||||
});
|
||||
@@ -0,0 +1,99 @@
|
||||
import { lstat, mkdir, open, readFile, writeFile } from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
import { publicContext, testopsClient, inspectChatMetadata, ensureChatPlan, validateChatSelection, bootstrapJobMetadata, launchChatPlan, admitLaunchPlan, verifyChatUpload, failureProjection } from './chat-feature-testops-api.mjs';
|
||||
import { exportChatAllure, readBoundChatExport, ExportFailure } from './chat-feature-testops-results.mjs';
|
||||
|
||||
export async function run(env = process.env, fetchImpl = fetch) {
|
||||
let stage = 'context', stateRoot;
|
||||
try {
|
||||
const expected = publicContext(env), operation = env.CHAT_TESTOPS_OPERATION;
|
||||
if (!['bootstrap', 'launch-plan', 'validate-selection', 'export', 'verify-upload'].includes(operation)) throw new Error('unknown_operation');
|
||||
if (operation === 'bootstrap' || operation === 'launch-plan') {
|
||||
const eventPath = env.GITHUB_EVENT_PATH;
|
||||
if (!eventPath) throw new Error('bootstrap_original_event_missing');
|
||||
const stat = await lstat(eventPath);
|
||||
if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 65_536) throw new Error('bootstrap_original_event_invalid');
|
||||
const event = JSON.parse(await readFile(eventPath, 'utf8'));
|
||||
if (event.inputs?.operation !== operation || (event.inputs?.ALLURE_JOB_RUN_ID ?? '') !== '') throw new Error('bootstrap_cannot_run_from_testops');
|
||||
}
|
||||
stage = 'output_admission';
|
||||
const requested = env.CHAT_TESTOPS_STATE_ROOT;
|
||||
if (!env.RUNNER_TEMP || !requested || !path.isAbsolute(requested)
|
||||
|| path.dirname(path.resolve(requested)) !== path.resolve(env.RUNNER_TEMP)) throw new Error('invalid_output');
|
||||
await mkdir(requested, { mode: 0o700 }); stateRoot = requested;
|
||||
let result;
|
||||
if (operation === 'export' || operation === 'verify-upload') {
|
||||
stage = 'result_export';
|
||||
const selectionPath = env.CHAT_TESTOPS_SELECTION;
|
||||
if (!selectionPath || !path.isAbsolute(selectionPath) || !selectionPath.startsWith(`${path.resolve(env.RUNNER_TEMP)}${path.sep}`)) throw new Error('invalid_selection_path');
|
||||
const stat = await lstat(selectionPath);
|
||||
if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 16_384) throw new Error('invalid_selection_file');
|
||||
const selected = JSON.parse(await readFile(selectionPath, 'utf8'));
|
||||
if (operation === 'verify-upload') {
|
||||
stage = 'upload_verification';
|
||||
const receiptRoot = path.join(env.RUNNER_TEMP, 'chat-testops-upload-state');
|
||||
const rootStat = await lstat(receiptRoot), file = path.join(receiptRoot, 'export.json');
|
||||
const receiptStat = await lstat(file);
|
||||
if (!rootStat.isDirectory() || rootStat.isSymbolicLink() || !receiptStat.isFile() || receiptStat.isSymbolicLink()
|
||||
|| receiptStat.nlink !== 1 || receiptStat.size > 16_384) throw new Error('invalid_export_receipt');
|
||||
const exported = JSON.parse(await readFile(file, 'utf8'));
|
||||
const bound = await readBoundChatExport(env.CHAT_TESTOPS_EXPORT_ROOT, expected);
|
||||
if (exported.status !== (bound.result.status === 'passed' && bound.result.cleanup === 'passed' ? 'passed' : 'failed')
|
||||
|| exported.cleanup !== bound.result.cleanup) throw new Error('export_verdict_mismatch');
|
||||
const api = await testopsClient(env.ALLURE_TOKEN, fetchImpl);
|
||||
result = await verifyChatUpload(api, env, selected, exported);
|
||||
await writeFile(path.join(stateRoot, 'verification.json'), JSON.stringify(result, null, 2), { flag: 'wx', mode: 0o600 });
|
||||
} else {
|
||||
result = await exportChatAllure({ root: env.CHAT_TESTOPS_EXPORT_ROOT, output: env.ALLURE_RESULTS, expected, selected });
|
||||
await writeFile(path.join(stateRoot, 'export.json'), JSON.stringify(result, null, 2), { flag: 'wx', mode: 0o600 });
|
||||
}
|
||||
} else {
|
||||
let launchAdmission;
|
||||
if (operation === 'launch-plan') { stage = 'launch_admission'; launchAdmission = await admitLaunchPlan(env, fetchImpl); }
|
||||
stage = 'authentication';
|
||||
const api = await testopsClient(env.ALLURE_TOKEN, fetchImpl);
|
||||
if (operation === 'launch-plan') {
|
||||
stage = 'launch_preflight';
|
||||
result = await launchChatPlan(api, env, async identity => {
|
||||
const handle = await open(path.join(stateRoot, 'once.json'), 'wx', 0o600);
|
||||
await handle.writeFile(JSON.stringify({ ...identity, status: 'STARTED_NO_AUTOMATIC_RESEND' }));
|
||||
await handle.close(); stage = 'plan_launch';
|
||||
}, launchAdmission);
|
||||
await writeFile(path.join(stateRoot, 'launch.json'), JSON.stringify(result, null, 2), { flag: 'wx', mode: 0o600 });
|
||||
} else if (operation === 'validate-selection') {
|
||||
stage = 'selection_validation';
|
||||
result = await validateChatSelection(api, env);
|
||||
await writeFile(path.join(stateRoot, 'selection.json'), JSON.stringify(result, null, 2), { flag: 'wx', mode: 0o600 });
|
||||
} else {
|
||||
stage = 'metadata_inspection';
|
||||
const inspected = await inspectChatMetadata(api);
|
||||
await writeFile(path.join(stateRoot, 'metadata.json'), JSON.stringify(inspected, null, 2), { flag: 'wx', mode: 0o600 });
|
||||
let marked = false;
|
||||
const once = async () => {
|
||||
if (marked) return;
|
||||
const handle = await open(path.join(stateRoot, 'once.json'), 'wx', 0o600);
|
||||
await handle.writeFile(JSON.stringify({ ...expected, operation: 'plan_bootstrap', status: 'STARTED_NO_AUTOMATIC_RESEND' }));
|
||||
await handle.close(); marked = true;
|
||||
};
|
||||
stage = 'plan_bootstrap';
|
||||
result = await ensureChatPlan(api, inspected, once);
|
||||
await writeFile(path.join(stateRoot, 'plan.json'), JSON.stringify(result, null, 2), { flag: 'wx', mode: 0o600 });
|
||||
stage = 'bootstrap_job_metadata';
|
||||
const job = await bootstrapJobMetadata(api, env);
|
||||
await writeFile(path.join(stateRoot, 'job.json'), JSON.stringify(job, null, 2), { flag: 'wx', mode: 0o600 });
|
||||
}
|
||||
}
|
||||
return operation === 'export' && result.status !== 'passed' || operation === 'launch-plan' && result.status !== 'REQUESTED' ? 1 : 0;
|
||||
} catch (error) {
|
||||
const safe = failureProjection(error, stage);
|
||||
if (error instanceof ExportFailure) safe.code = 'chat_testops_export_rejected';
|
||||
if (stateRoot) await writeFile(path.join(stateRoot, 'failure.json'), JSON.stringify(safe, null, 2), { flag: 'wx', mode: 0o600 }).catch(() => {});
|
||||
process.stderr.write(`Chat TestOps stopped: ${JSON.stringify(safe)}\n`);
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
run().then(code => { process.exitCode = code; }).catch(() => { process.stderr.write('Chat TestOps stopped: unknown failure\n'); process.exitCode = 1; });
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { mkdtemp, readFile, realpath, rm, writeFile } from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { run } from './chat-feature-testops-cli.mjs';
|
||||
import { fullName, planName } from './chat-feature-testops-api.mjs';
|
||||
const context = { GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/heads/master', GITHUB_EVENT_NAME: 'workflow_dispatch',
|
||||
GITHUB_RUN_ATTEMPT: '1', GITHUB_SHA: 'a'.repeat(40), GITHUB_RUN_ID: '123',
|
||||
GITHUB_WORKFLOW_REF: 'paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master' };
|
||||
|
||||
test('actual metadata bootstrap caller creates no result or agent and reuses exact case/plan', async t => {
|
||||
const temporary = await realpath(await mkdtemp(path.join(tmpdir(), 'testops-bootstrap-')));
|
||||
t.after(() => rm(temporary, { recursive: true }));
|
||||
const state = path.join(temporary, 'state'), requests = [];
|
||||
const eventPath = path.join(temporary, 'event.json'); await writeFile(eventPath, JSON.stringify({ inputs: { operation: 'bootstrap', ALLURE_JOB_RUN_ID: '' } }));
|
||||
const response = value => ({ ok: true, status: 200, json: async () => value, text: async () => JSON.stringify(value) });
|
||||
const fetchImpl = async (url, options) => {
|
||||
requests.push({ url, method: options.method });
|
||||
if (url.endsWith('/api/uaa/oauth/token')) return response({ access_token: 'SYNTHETIC_BEARER' });
|
||||
const route = new URL(url).pathname;
|
||||
if (route === '/api/testcase') return response({ number: 0, totalPages: 1, totalElements: 1, content: [{ id: 91, name: 'Not projected', automated: true }] });
|
||||
if (route === '/api/testcase/91') return response({ id: 91, projectId: 1, fullName, automated: true });
|
||||
if (route === '/api/testplan') return response({ totalPages: 1, content: [{ id: 8, projectId: 1, name: planName, testCasesCount: 1 }] });
|
||||
if (route === '/api/integration/project/1') return response({ totalPages: 1, content: [] });
|
||||
if (route === '/api/jobrun/9') return response({ id: 9, job: { id: 7 }, launchId: 10, url: 'https://github.com/paperclipai/paperclip/actions/runs/123' });
|
||||
if (route === '/api/job/7') return response({ id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml', parameters: [{ id: 17, value: 'bootstrap' }] });
|
||||
if (route === '/api/environment/17') return response({ id: 17, name: 'operation' });
|
||||
assert.equal(route, '/api/testplan/8');
|
||||
return response({ id: 8, projectId: 1, name: planName, testCasesCount: 1, treeSelection: { leafsInclude: [91], inverted: false } });
|
||||
};
|
||||
assert.equal(await run({ ...context, RUNNER_TEMP: temporary, CHAT_TESTOPS_STATE_ROOT: state,
|
||||
CHAT_TESTOPS_OPERATION: 'bootstrap', GITHUB_EVENT_PATH: eventPath, ALLURE_JOB_RUN_ID: '9', ALLURE_TOKEN: 'SYNTHETIC_API_TOKEN' }, fetchImpl), 0);
|
||||
const plan = JSON.parse(await readFile(path.join(state, 'plan.json')));
|
||||
assert.equal(plan.actualResult, null);
|
||||
assert.equal(requests.filter(value => value.method === 'POST').length, 1, 'only auth token exchange; no case/plan write');
|
||||
await assert.rejects(readFile(path.join(state, 'allure-results.json')), { code: 'ENOENT' });
|
||||
});
|
||||
|
||||
test('actual public caller retains only safe auth status on failure before discovery', async t => {
|
||||
const temporary = await realpath(await mkdtemp(path.join(tmpdir(), 'testops-failure-')));
|
||||
t.after(() => rm(temporary, { recursive: true }));
|
||||
const messages = [], original = process.stderr.write;
|
||||
process.stderr.write = value => { messages.push(String(value)); return true; };
|
||||
try {
|
||||
const state = path.join(temporary, 'state');
|
||||
const eventPath = path.join(temporary, 'event.json'); await writeFile(eventPath, JSON.stringify({ inputs: { operation: 'bootstrap' } }));
|
||||
assert.equal(await run({ ...context, RUNNER_TEMP: temporary, CHAT_TESTOPS_STATE_ROOT: state,
|
||||
CHAT_TESTOPS_OPERATION: 'bootstrap', GITHUB_EVENT_PATH: eventPath, ALLURE_TOKEN: 'SYNTHETIC_PRIVATE_TOKEN' },
|
||||
async () => ({ ok: false, status: 403, text: () => assert.fail('must not read denial body') })), 1);
|
||||
assert.deepEqual(JSON.parse(await readFile(path.join(state, 'failure.json'))), { stage: 'authentication', code: 'testops_auth_http', httpStatus: 403 });
|
||||
assert.ok(!messages.join('').includes('SYNTHETIC_PRIVATE_TOKEN'));
|
||||
} finally { process.stderr.write = original; }
|
||||
});
|
||||
|
||||
|
||||
test('original TestOps-triggered bootstrap is rejected even if its official environment is populated', async t => {
|
||||
const temporary = await realpath(await mkdtemp(path.join(tmpdir(), 'testops-wrong-bootstrap-')));
|
||||
t.after(() => rm(temporary, { recursive: true }));
|
||||
const event = path.join(temporary, 'event.json'); await writeFile(event, JSON.stringify({ inputs: { operation: 'bootstrap', ALLURE_JOB_RUN_ID: '9' } }));
|
||||
const original = process.stderr.write;
|
||||
process.stderr.write = () => true;
|
||||
try {
|
||||
assert.equal(await run({ ...context, GITHUB_EVENT_PATH: event, RUNNER_TEMP: temporary, CHAT_TESTOPS_STATE_ROOT: path.join(temporary, 'state'),
|
||||
CHAT_TESTOPS_OPERATION: 'bootstrap', ALLURE_JOB_RUN_ID: '9' }, () => assert.fail('no API access')), 1);
|
||||
} finally { process.stderr.write = original; }
|
||||
});
|
||||
@@ -0,0 +1,134 @@
|
||||
import { createHash, randomUUID } from 'node:crypto';
|
||||
import { lstat, mkdir, readFile, readdir, realpath, writeFile } from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { aliases, executionId } from './chat-feature-testops-api.mjs';
|
||||
|
||||
export class ExportFailure extends Error {}
|
||||
const fail = () => { throw new ExportFailure('chat_testops_export_rejected'); };
|
||||
const hash = bytes => createHash('sha256').update(bytes).digest('hex');
|
||||
const pngMagic = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]);
|
||||
const model = 'openrouter/deepseek/deepseek-v4.1-flash-1017';
|
||||
const pngPath = new RegExp(`^evidence/${executionId.replaceAll('.', '\\.')}/attempt-1/[A-Za-z0-9][A-Za-z0-9._-]{0,199}\\.png$`);
|
||||
|
||||
async function inventory(root, relative = '') {
|
||||
const output = [];
|
||||
const entries = await readdir(path.join(root, relative));
|
||||
if (entries.length > 12) fail();
|
||||
for (const name of entries) {
|
||||
if (!/^[A-Za-z0-9._-]+$/.test(name) || name === '.' || name === '..') fail();
|
||||
const file = relative ? `${relative}/${name}` : name;
|
||||
const stat = await lstat(path.join(root, file));
|
||||
if (stat.isSymbolicLink()) fail();
|
||||
if (stat.isDirectory()) {
|
||||
if (!['evidence', `evidence/${executionId}`, `evidence/${executionId}/attempt-1`].includes(file)) fail();
|
||||
output.push(...await inventory(root, file));
|
||||
}
|
||||
else if (stat.isFile() && stat.nlink === 1 && stat.size <= 12 * 1024 * 1024) output.push(file);
|
||||
else fail();
|
||||
if (output.length > 12) fail();
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
export async function readBoundChatExport(root, expected) {
|
||||
if (!/^[1-9][0-9]{0,19}$/.test(expected.runId ?? '') || expected.attempt !== '1'
|
||||
|| !/^[a-f0-9]{40}$/.test(expected.workflowSha ?? '') || expected.targetSha !== expected.workflowSha) fail();
|
||||
const stat = await lstat(root);
|
||||
if (!stat.isDirectory() || stat.isSymbolicLink() || await realpath(root) !== path.resolve(root)) fail();
|
||||
const files = await inventory(root);
|
||||
if (!files.includes('binding.json')) fail();
|
||||
const bindingBytes = await readFile(path.join(root, 'binding.json'));
|
||||
if (bindingBytes.length > 128_000) fail();
|
||||
const binding = JSON.parse(bindingBytes);
|
||||
if (binding.schema !== 'paperclip.chat-feature-testops-export/v1' || binding.attempt !== '1'
|
||||
|| binding.runId !== expected.runId || binding.workflowSha !== expected.workflowSha || binding.targetSha !== expected.targetSha
|
||||
|| binding.executionId !== executionId || binding.campaignId !== `gha-${expected.runId}-1`
|
||||
|| !Array.isArray(binding.files) || binding.files.length < 2 || binding.files.length > 10) fail();
|
||||
const names = binding.files.map(row => row.path);
|
||||
if (new Set(names).size !== names.length || !names.includes('normalized-results.json') || !names.includes('junit.xml')
|
||||
|| names.some(name => !['normalized-results.json', 'junit.xml'].includes(name) && !pngPath.test(name))
|
||||
|| JSON.stringify([...names, 'binding.json'].sort()) !== JSON.stringify(files.sort())) fail();
|
||||
const retained = new Map();
|
||||
let total = 0;
|
||||
for (const row of binding.files) {
|
||||
if (!/^[a-f0-9]{64}$/.test(row.sha256 ?? '') || !Number.isSafeInteger(row.bytes) || row.bytes < 0) fail();
|
||||
const bytes = await readFile(path.join(root, row.path));
|
||||
total += bytes.length;
|
||||
if (total > 32 * 1024 * 1024 || bytes.length !== row.bytes || hash(bytes) !== row.sha256) fail();
|
||||
retained.set(row.path, bytes);
|
||||
}
|
||||
const bytes = retained.get('normalized-results.json');
|
||||
if (bytes.length > 2_000_000) fail();
|
||||
const campaign = JSON.parse(bytes);
|
||||
if (campaign.schema !== 'paperclip.runner-e2e.campaign/v2' || campaign.campaignId !== binding.campaignId
|
||||
|| !Array.isArray(campaign.results) || campaign.results.length !== 1) fail();
|
||||
const result = campaign.results[0];
|
||||
if (result.executionId !== executionId || result.suiteId !== 'chat-features' || result.caseId !== 'create-backlog'
|
||||
|| result.profileId !== 'runner-opencode-deepseek-1017' || result.provider !== 'opencode' || result.environmentId !== 'local'
|
||||
|| result.model !== model || result.runtimeMode !== 'native' || result.attempt !== 1 || result.source?.sha !== expected.targetSha
|
||||
|| !['passed', 'failed'].includes(result.status) || result.status !== binding.resultStatus
|
||||
|| !['passed', 'failed', 'not_started'].includes(result.cleanup) || result.cleanup !== binding.cleanup
|
||||
|| !Array.isArray(result.matcherResults) || result.matcherResults.length > 200 || !Array.isArray(result.screenshots) || result.screenshots.length > 8) fail();
|
||||
for (const check of result.matcherResults) if (check.matcher?.kind !== 'json_path' || !/^[A-Za-z0-9._-]{1,150}$/.test(check.matcher.path)
|
||||
|| check.matcher.expected !== true || typeof check.passed !== 'boolean') fail();
|
||||
for (const shot of result.screenshots) {
|
||||
const name = `evidence/${executionId}/attempt-1/${shot.file}`;
|
||||
const png = retained.get(name);
|
||||
if (!png || !png.subarray(0, 8).equals(pngMagic) || png.subarray(12, 16).toString('ascii') !== 'IHDR' || hash(png) !== shot.sha256) fail();
|
||||
}
|
||||
if (names.filter(name => pngPath.test(name)).length !== result.screenshots.length) fail();
|
||||
if (result.status === 'passed' && (result.cleanup !== 'passed' || result.matcherResults.length === 0
|
||||
|| result.matcherResults.some(check => !check.passed) || result.runIds?.length !== 1
|
||||
|| !result.screenshots.some(shot => shot.id === 'final-state' && shot.file === 'final-state.png'))) fail();
|
||||
return { binding, result, retained };
|
||||
}
|
||||
|
||||
export async function exportChatAllure({ root, output, expected, selected }) {
|
||||
if (selected.projectId !== 1 || !Number.isSafeInteger(selected.caseId) || selected.caseId < 1 || !aliases.includes(selected.fullName)
|
||||
|| selected.runId !== expected.runId || selected.attempt !== '1' || selected.workflowSha !== expected.workflowSha || selected.targetSha !== expected.targetSha
|
||||
|| selected.executionId !== executionId) fail();
|
||||
const { binding, result, retained } = await readBoundChatExport(root, expected);
|
||||
await mkdir(output, { mode: 0o700 }).catch(error => { if (error.code !== 'EEXIST') throw error; });
|
||||
const outputStat = await lstat(output);
|
||||
if (!outputStat.isDirectory() || outputStat.isSymbolicLink() || (process.getuid && outputStat.uid !== process.getuid()) || (await readdir(output)).length !== 0) fail();
|
||||
const status = result.status === 'passed' && result.cleanup === 'passed' ? 'passed' : 'failed';
|
||||
const attachments = [];
|
||||
const attachmentMetadata = [];
|
||||
for (const shot of result.screenshots) {
|
||||
const source = `${randomUUID()}-attachment.png`;
|
||||
await writeFile(path.join(output, source), retained.get(`evidence/${executionId}/attempt-1/${shot.file}`), { flag: 'wx', mode: 0o600 });
|
||||
attachments.push({ name: 'Synthetic task and source conversation', source, type: 'image/png' });
|
||||
attachmentMetadata.push({ name: 'Synthetic task and source conversation', contentType: 'image/png', contentLength: retained.get(`evidence/${executionId}/attempt-1/${shot.file}`).length });
|
||||
}
|
||||
const projection = { campaignId: binding.campaignId, executionId, runId: expected.runId, attempt: 1,
|
||||
workflowSha: expected.workflowSha, sourceSha: expected.targetSha, status: result.status, cleanup: result.cleanup,
|
||||
profile: result.profileId, model: result.model, environment: result.environmentId, runtime: result.runtimeMode,
|
||||
durationMs: Number.isFinite(result.durationMs) ? result.durationMs : null,
|
||||
failureClass: typeof result.failureClass === 'string' && /^[a-z_]{1,80}$/.test(result.failureClass) ? result.failureClass : null,
|
||||
billingComplete: typeof result.billing?.complete === 'boolean' ? result.billing.complete : null,
|
||||
checks: result.matcherResults.map(check => ({ path: check.matcher.path, passed: check.passed })),
|
||||
note: 'Setup usage and AWS billing are not qualified by this result; missing accounting remains unknown.' };
|
||||
const source = `${randomUUID()}-attachment.json`;
|
||||
const projectionBytes = Buffer.from(JSON.stringify(projection, null, 2));
|
||||
await writeFile(path.join(output, source), projectionBytes, { flag: 'wx', mode: 0o600 });
|
||||
attachmentMetadata.push({ name: 'Execution provenance and checks', contentType: 'application/json', contentLength: projectionBytes.length });
|
||||
attachments.push({ name: 'Execution provenance and checks', source, type: 'application/json' });
|
||||
const normalizedSource = `${randomUUID()}-attachment.json`;
|
||||
await writeFile(path.join(output, normalizedSource), retained.get('normalized-results.json'), { flag: 'wx', mode: 0o600 });
|
||||
attachments.push({ name: 'Sanitized Product E2E record and accounting', source: normalizedSource, type: 'application/json' });
|
||||
attachmentMetadata.push({ name: 'Sanitized Product E2E record and accounting', contentType: 'application/json', contentLength: retained.get('normalized-results.json').length });
|
||||
const value = { uuid: randomUUID(), name: 'Create a task backlog through chat', fullName: selected.fullName,
|
||||
testCaseId: hash(`${JSON.stringify(selected.fullName)}\n`),
|
||||
historyId: hash(JSON.stringify([selected.fullName, result.profileId, result.model, result.environmentId])), status, stage: 'finished',
|
||||
statusDetails: { message: status === 'passed' ? 'Real Product E2E assertions and cleanup passed' : 'Real Product E2E evaluation or cleanup failed; original verdict preserved' },
|
||||
labels: [{ name: 'AS_ID', value: String(selected.caseId) }, { name: 'parentSuite', value: 'Product E2E' },
|
||||
{ name: 'suite', value: 'chat-features' }, { name: 'area', value: 'Chat and collaboration' },
|
||||
{ name: 'framework', value: 'Paperclip Evals' }, { name: 'tag', value: 'chat-feature-testops' }],
|
||||
parameters: [{ name: 'canonicalScenario', value: 'tasks.create-backlog' }, { name: 'model', value: model }, { name: 'environment', value: 'GitHub Actions / AWS' }],
|
||||
links: [{ name: 'GitHub Actions run', type: 'custom', url: `https://github.com/paperclipai/paperclip/actions/runs/${expected.runId}` },
|
||||
{ name: 'Evaluated source', type: 'custom', url: `https://github.com/paperclipai/paperclip/commit/${expected.targetSha}` }],
|
||||
steps: result.matcherResults.map(check => ({ name: check.matcher.path, status: check.passed ? 'passed' : 'failed', stage: 'finished' })), attachments };
|
||||
for (const [key, time] of [['start', result.startedAt], ['stop', result.finishedAt]]) if (Number.isFinite(Date.parse(time))) value[key] = Date.parse(time);
|
||||
await writeFile(path.join(output, `${value.uuid}-result.json`), JSON.stringify(value), { flag: 'wx', mode: 0o600 });
|
||||
return { status, cleanup: result.cleanup, caseId: selected.caseId, campaignId: binding.campaignId, count: 1, attachmentMetadata };
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { mkdtemp, mkdir, readFile, readdir, realpath, rm, symlink, writeFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { exportChatAllure, readBoundChatExport } from './chat-feature-testops-results.mjs';
|
||||
import { executionId, fullName } from './chat-feature-testops-api.mjs';
|
||||
const hash = bytes => createHash('sha256').update(bytes).digest('hex');
|
||||
const expected = { runId: '123', attempt: '1', workflowSha: 'a'.repeat(40), targetSha: 'a'.repeat(40) };
|
||||
const selected = { ...expected, projectId: 1, caseId: 91, fullName, executionId };
|
||||
async function fixture(t, changes = {}) {
|
||||
const directory = await realpath(await mkdtemp(path.join(tmpdir(), 'chat-testops-export-')));
|
||||
t.after(() => rm(directory, { recursive: true }));
|
||||
const root = path.join(directory, 'export'); await mkdir(root);
|
||||
const png = Buffer.from([137,80,78,71,13,10,26,10,0,0,0,13,73,72,68,82]);
|
||||
const shot = { id: 'final-state', file: 'final-state.png', sha256: hash(png) };
|
||||
const result = { executionId, suiteId: 'chat-features', caseId: 'create-backlog', profileId: 'runner-opencode-deepseek-1017', provider: 'opencode',
|
||||
environmentId: 'local', model: 'openrouter/deepseek/deepseek-v4.1-flash-1017', runtimeMode: 'native', attempt: 1,
|
||||
source: { sha: expected.targetSha }, status: 'passed', cleanup: 'passed', runIds: ['synthetic-run'], durationMs: 1234,
|
||||
matcherResults: [{ matcher: { kind: 'json_path', path: 'chatFeature.createdTask', expected: true }, passed: true }], screenshots: [shot], ...changes };
|
||||
const files = new Map([['normalized-results.json', Buffer.from(JSON.stringify({ schema: 'paperclip.runner-e2e.campaign/v2', campaignId: 'gha-123-1', complete: false, results: [result] }))],
|
||||
['junit.xml', Buffer.from('<testsuite tests="1"/>')], [`evidence/${executionId}/attempt-1/final-state.png`, png]]);
|
||||
const binding = { schema: 'paperclip.chat-feature-testops-export/v1', ...expected, executionId, campaignId: 'gha-123-1', resultStatus: result.status, cleanup: result.cleanup,
|
||||
files: [...files].map(([name, bytes]) => ({ path: name, sha256: hash(bytes), bytes: bytes.length })) };
|
||||
for (const [name, bytes] of files) { await mkdir(path.dirname(path.join(root, name)), { recursive: true }); await writeFile(path.join(root, name), bytes); }
|
||||
await writeFile(path.join(root, 'binding.json'), JSON.stringify(binding));
|
||||
return { directory, root, output: path.join(directory, 'allure'), result, binding };
|
||||
}
|
||||
|
||||
test('bound synthetic selected result maps to existing Allure case, independent of whole-suite complete', async t => {
|
||||
const f = await fixture(t);
|
||||
const value = await exportChatAllure({ ...f, expected, selected });
|
||||
assert.equal(value.status, 'passed'); assert.equal(value.count, 1);
|
||||
const name = (await readdir(f.output)).find(name => name.endsWith('-result.json'));
|
||||
const result = JSON.parse(await readFile(path.join(f.output, name)));
|
||||
assert.equal(result.fullName, fullName); assert.deepEqual(result.labels[0], { name: 'AS_ID', value: '91' });
|
||||
assert.equal(result.attachments.filter(row => row.type === 'image/png').length, 1);
|
||||
});
|
||||
|
||||
test('real failed outcome stays failed and no absence can synthesize a pass', async t => {
|
||||
const f = await fixture(t, { status: 'failed', cleanup: 'failed' });
|
||||
assert.equal((await exportChatAllure({ ...f, expected, selected })).status, 'failed');
|
||||
await assert.rejects(exportChatAllure({ root: path.join(f.directory, 'absent'), output: path.join(f.directory, 'missing'), expected, selected }));
|
||||
});
|
||||
|
||||
test('foreign source or selection and tampered file hash reject', async t => {
|
||||
const f = await fixture(t);
|
||||
await assert.rejects(readBoundChatExport(f.root, { ...expected, runId: '456' }));
|
||||
await assert.rejects(exportChatAllure({ ...f, expected, selected: { ...selected, projectId: 2 } }));
|
||||
await writeFile(path.join(f.root, 'junit.xml'), 'tampered');
|
||||
await assert.rejects(readBoundChatExport(f.root, expected));
|
||||
});
|
||||
|
||||
test('an unmanifested file, unknown directory or symlink is rejected before upload', async t => {
|
||||
const f = await fixture(t);
|
||||
await writeFile(path.join(f.root, 'provider-raw.txt'), 'SYNTHETIC_PRIVATE_NEVER_READ');
|
||||
await assert.rejects(readBoundChatExport(f.root, expected));
|
||||
await rm(path.join(f.root, 'provider-raw.txt'));
|
||||
await mkdir(path.join(f.root, 'unknown-empty'));
|
||||
await assert.rejects(readBoundChatExport(f.root, expected));
|
||||
await rm(path.join(f.root, 'unknown-empty'), { recursive: true });
|
||||
await rm(path.join(f.root, 'junit.xml')); await symlink(path.join(f.root, 'normalized-results.json'), path.join(f.root, 'junit.xml'));
|
||||
await assert.rejects(readBoundChatExport(f.root, expected));
|
||||
});
|
||||
|
||||
test('passing result cannot omit independent checks, cleanup or actual feature capture', async t => {
|
||||
for (const changes of [{ matcherResults: [] }, { cleanup: 'failed' }, { runIds: [] }, { screenshots: [] },
|
||||
{ matcherResults: [{ matcher: { kind: 'json_path', path: 'chatFeature.createdTask', expected: true }, passed: false }] }]) {
|
||||
const f = await fixture(t, changes);
|
||||
await assert.rejects(exportChatAllure({ ...f, expected, selected }));
|
||||
}
|
||||
});
|
||||
@@ -101,7 +101,7 @@ export type PrivateCellLayout = "original" | "final" | "final-wrapper";
|
||||
export async function privateCellEvidenceOwner(context: {
|
||||
private: true; root: string; runId: string; attempt: string; executionIds: string[]; cellReportProducer?: CellReportProducerIdentity;
|
||||
}, executionId: string): Promise<PrivateCellEvidenceOwner> {
|
||||
if (context.private !== true || !context.executionIds.includes(executionId) || !/^chat-features\.runner-opencode\.local\.[a-z0-9][a-z0-9.-]{0,159}$/.test(executionId)
|
||||
if (context.private !== true || !context.executionIds.includes(executionId) || (!/^chat-features\.runner-opencode\.local\.[a-z0-9][a-z0-9.-]{0,159}$/.test(executionId) && executionId !== "chat-features.runner-opencode-deepseek-1017.local.create-backlog")
|
||||
|| executionId.includes("..") || !/^[1-9][0-9]{0,19}$/.test(context.runId) || context.attempt !== "1" || !context.cellReportProducer)
|
||||
throw new Error("Invalid private cell report owner");
|
||||
const root = await realpath(context.root), metadata = await lstat(context.root);
|
||||
@@ -116,8 +116,9 @@ export function cellTraceAsset(relative: string, owner: PrivateCellEvidenceOwner
|
||||
const segments = relative.split("/");
|
||||
const reserved = segments.some((part, index) => part === "html-report" && segments[index + 1] === "trace");
|
||||
if (!reserved) return { reserved: false };
|
||||
const scenario = owner.executionId.slice("chat-features.runner-opencode.local.".length);
|
||||
const final = `gha-${owner.runId}-${owner.attempt}-${owner.executionId}/chat-features/runner-opencode/local/${scenario}/attempt-${owner.attempt}/html-report/trace/`;
|
||||
const profile = owner.executionId === "chat-features.runner-opencode-deepseek-1017.local.create-backlog" ? "runner-opencode-deepseek-1017" : "runner-opencode";
|
||||
const scenario = owner.executionId.slice(`chat-features.${profile}.local.`.length);
|
||||
const final = `gha-${owner.runId}-${owner.attempt}-${owner.executionId}/chat-features/${profile}/local/${scenario}/attempt-${owner.attempt}/html-report/trace/`;
|
||||
const prefix = layout === "original" ? "html-report/trace/" : layout === "final" ? final : `html-report/${final}`;
|
||||
const canonical = relative.startsWith(prefix) ? canonicalGeneratedReportAsset(`html/trace/${relative.slice(prefix.length)}`) : undefined;
|
||||
return { reserved: true, canonical };
|
||||
|
||||
@@ -300,7 +300,7 @@ export function manualPrivateSessionPolicy(input: { bucket: string; runId: strin
|
||||
if (!/^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/.test(input.bucket) || !/^[1-9][0-9]{0,19}$/.test(input.runId))
|
||||
throw new Error("Invalid manual private storage identity");
|
||||
const ids = canonicalPrivateManualIds(input.executionIds);
|
||||
if (ids.some(id => !MANUAL_PRIVATE_EXECUTIONS.includes(id)) || input.mode !== "report-write" && ids.length !== 1)
|
||||
if (ids.some(id => !MANUAL_PRIVATE_EXECUTIONS.includes(id) && id !== "chat-features.runner-opencode-deepseek-1017.local.create-backlog") || input.mode !== "report-write" && ids.length !== 1)
|
||||
throw new Error("Private archive sessions require one exact cell or the report writer");
|
||||
const prefix = `arn:aws:s3:::${input.bucket}/runner-e2e-private/bounded-v1/campaigns/gha-${input.runId}-1/`;
|
||||
const cells = ids.map(id => `${prefix}${manualCellKind(id)}.zip`);
|
||||
|
||||
@@ -10,11 +10,12 @@ import { isStandardEvidenceFile } from "./evidence.js";
|
||||
import { BoundedHistoryPublisher } from "./history-budget.js";
|
||||
import { privateArtifactKey, retainPrivateTransfer, transferPrivateArtifact, verifyPrivateHistory, type PrivateEvidenceCompleteness } from "./private-history.js";
|
||||
import { authorizeManualPrivateEvidence, MANUAL_PRIVATE_PROTOCOL, manualCellKind, manualPrivateSessionPolicy } from "./manual-private-retention.js";
|
||||
import { shippedChatEvidenceAuthority } from "./testops-chat-admission.js";
|
||||
|
||||
const hash = (bytes: string | Buffer) => createHash("sha256").update(bytes).digest("hex");
|
||||
function request(env: NodeJS.ProcessEnv) {
|
||||
const ids = JSON.parse(env.MANUAL_PRIVATE_IDS ?? "null") as string[];
|
||||
const approval = authorizeManualPrivateEvidence({ workflowSha: env.GITHUB_SHA ?? "", targetSha: env.TARGET_SHA ?? "",
|
||||
const approval = env.PAPERCLIP_TESTOPS_CHAT_MAIN_ADMISSION !== undefined ? shippedChatEvidenceAuthority(env) : authorizeManualPrivateEvidence({ workflowSha: env.GITHUB_SHA ?? "", targetSha: env.TARGET_SHA ?? "",
|
||||
runId: env.GITHUB_RUN_ID ?? "", attempt: env.GITHUB_RUN_ATTEMPT ?? "", executionIds: ids }, env.MANUAL_PRIVATE_APPROVALS_JSON ?? "");
|
||||
return { ids, approval, bucket: env.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET ?? "" };
|
||||
}
|
||||
|
||||
@@ -21,7 +21,8 @@ delete env.NODE_OPTIONS;
|
||||
for (const [name, command] of [
|
||||
["Vitest", [vitest, "run", "--config", "tests/runner-e2e/vitest.config.ts", ...args]],
|
||||
["Node tests", ["--test", "tests/runner-e2e/artifact-budget.test.mjs", "tests/runner-e2e/native-completion-checks.test.mjs",
|
||||
"tests/runner-e2e/native-completion-git-source.test.mjs", "tests/runner-e2e/native-completion-source-contract.test.mjs"]],
|
||||
"tests/runner-e2e/native-completion-git-source.test.mjs", "tests/runner-e2e/native-completion-source-contract.test.mjs",
|
||||
"tests/runner-e2e/chat-feature-testops-api.test.mjs", "tests/runner-e2e/chat-feature-testops-cli.test.mjs", "tests/runner-e2e/chat-feature-testops-results.test.mjs", "tests/runner-e2e/testops-chat-reporter.test.mjs"]],
|
||||
]) {
|
||||
const result = spawnSync(process.execPath, command, { cwd: root, env, stdio: "inherit" });
|
||||
if (result.error || result.signal) {
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, expect, it, vi } from "vitest";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { authorizeShippedChat, shippedChatEvidenceAuthority } from "./testops-chat-admission.js";
|
||||
|
||||
vi.mock("node:fs", async importOriginal => {
|
||||
const actual = await importOriginal<typeof import("node:fs")>();
|
||||
return { ...actual, readFileSync: vi.fn(actual.readFileSync) };
|
||||
});
|
||||
|
||||
const wrapperDirectories: string[] = [];
|
||||
afterEach(async () => {
|
||||
vi.restoreAllMocks();
|
||||
const actual = await vi.importActual<typeof import("node:fs")>("node:fs");
|
||||
vi.mocked(readFileSync).mockImplementation(actual.readFileSync);
|
||||
for (const directory of wrapperDirectories.splice(0)) rmSync(directory, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
const id = "chat-features.runner-opencode-deepseek-1017.local.create-backlog";
|
||||
function fixture() {
|
||||
const env: NodeJS.ProcessEnv = { PAPERCLIP_TESTOPS_CHAT_MAIN_ADMISSION: "v1", GITHUB_ACTIONS: "true", GITHUB_EVENT_NAME: "workflow_dispatch",
|
||||
GITHUB_REPOSITORY: "paperclipai/paperclip", GITHUB_REF: "refs/heads/master", GITHUB_WORKFLOW_REF: "paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master",
|
||||
GITHUB_JOB: "chat", GITHUB_SHA: "a".repeat(40), TARGET_SHA: "a".repeat(40), TESTOPS_ADMITTED_SHA: "a".repeat(40), GITHUB_RUN_ID: "12345", GITHUB_RUN_ATTEMPT: "1",
|
||||
GITHUB_ACTOR_ID: "123", TESTOPS_TRIGGERING_ACTOR_ID: "123", TESTOPS_ALLOWED_ACTOR_IDS: "[123]", MANUAL_PRIVATE_IDS: JSON.stringify([id]), SELECTED_EXECUTION_ID: id };
|
||||
const event = { ref: "refs/heads/master", sender: { id: 123 }, repository: { full_name: "paperclipai/paperclip", private: false, default_branch: "master" }, inputs: { ALLURE_JOB_RUN_ID: "789", operation: "evaluate" } as Record<string, string> };
|
||||
const selected = runnerMatrix.find(row => row.id === id)!;
|
||||
return { env, event, selected };
|
||||
}
|
||||
it("binds the actual shipped selection to immutable main admission without candidate approval variables", () => {
|
||||
const f = fixture(); const result = authorizeShippedChat(f.env, f.event, f.selected, "b".repeat(64));
|
||||
expect(result).toMatchObject({ authority: "paperclip.shipped-chat-testops/v1", private: true, runId: "12345", attempt: "1", workflowSha: f.env.GITHUB_SHA, targetSha: f.env.GITHUB_SHA, executionIds: [id] });
|
||||
expect(result.authoritySha256).toMatch(/^[a-f0-9]{64}$/);
|
||||
expect(authorizeShippedChat({ ...f.env, LATEST_MASTER_SHA: "c".repeat(40), GITHUB_JOB: "report" }, f.event, f.selected, "b".repeat(64)).authoritySha256).toBe(result.authoritySha256);
|
||||
});
|
||||
it.each([
|
||||
["unknown mode", { PAPERCLIP_TESTOPS_CHAT_MAIN_ADMISSION: "unknown" }],
|
||||
["untrusted repo", { GITHUB_REPOSITORY: "foreign/repo" }],
|
||||
["nondefault ref", { GITHUB_REF: "refs/heads/feature" }],
|
||||
["wrong workflow caller", { GITHUB_WORKFLOW_REF: "paperclipai/paperclip/.github/workflows/runner-full-stack-e2e.yml@refs/heads/master" }],
|
||||
["source substitution", { TARGET_SHA: "c".repeat(40) }],
|
||||
["admitted source mismatch", { TESTOPS_ADMITTED_SHA: "c".repeat(40) }],
|
||||
["replay", { GITHUB_RUN_ATTEMPT: "2" }],
|
||||
["unlisted actor", { TESTOPS_ALLOWED_ACTOR_IDS: "[456]" }],
|
||||
["untrusted job", { GITHUB_JOB: "build" }],
|
||||
["foreign selector", { SELECTED_EXECUTION_ID: "other" }],
|
||||
])("rejects %s", (_name, patch) => { const f = fixture(); expect(() => authorizeShippedChat({ ...f.env, ...patch }, f.event, f.selected, "b".repeat(64))).toThrow(); });
|
||||
it.each(["target_branch", "profile", "model", "retries", "timeout"])("rejects the undeclared %s input", field => {
|
||||
const f = fixture(); f.event.inputs[field] = "override"; expect(() => authorizeShippedChat(f.env, f.event, f.selected, "b".repeat(64))).toThrow();
|
||||
});
|
||||
it("rejects metadata bootstrap as an evaluation admission", () => { const f = fixture(); f.event.inputs.operation = "bootstrap"; expect(() => authorizeShippedChat(f.env, f.event, f.selected, "b".repeat(64))).toThrow(); });
|
||||
it("rejects forged event sender and foreign event source", () => {
|
||||
const f = fixture(); f.event.sender.id = 456; expect(() => authorizeShippedChat(f.env, f.event, f.selected, "b".repeat(64))).toThrow();
|
||||
f.event.sender.id = 123; f.event.repository.full_name = "foreign/repo"; expect(() => authorizeShippedChat(f.env, f.event, f.selected, "b".repeat(64))).toThrow();
|
||||
});
|
||||
it("rejects the existing production OpenCode backlog cell", () => {
|
||||
const f = fixture();
|
||||
const stock = runnerMatrix.find(row => row.id === "chat-features.runner-opencode.local.create-backlog")!;
|
||||
expect(() => authorizeShippedChat(f.env, f.event, stock, "b".repeat(64))).toThrow("Unexpected shipped chat selection or bounds");
|
||||
});
|
||||
it.each(["model", "provider", "work", "clock", "retry", "budget"])("rejects changed selected %s", field => {
|
||||
const f = fixture(); const selected = { ...f.selected, profile: { ...f.selected.profile }, task: { ...f.selected.task }, suite: { ...f.selected.suite, definitionMetadata: { ...f.selected.suite.definitionMetadata } } };
|
||||
if (field === "model") selected.profile.model = "other/model";
|
||||
if (field === "provider") selected.profile.provider = "codex";
|
||||
if (field === "work") selected.task.expectedRunCount = 2;
|
||||
if (field === "clock") selected.task.turnTimeoutMs = 121000;
|
||||
if (field === "retry") selected.task.automaticRetryPolicy = undefined;
|
||||
if (field === "budget") selected.suite.definitionMetadata.companyAndAgentBudgetCents = 1000;
|
||||
expect(() => authorizeShippedChat(f.env, f.event, selected, "b".repeat(64))).toThrow();
|
||||
});
|
||||
|
||||
async function wrapperFixture() {
|
||||
const actual = await vi.importActual<typeof import("node:fs")>("node:fs");
|
||||
const directory = mkdtempSync(path.join(tmpdir(), "testops-chat-authority-"));
|
||||
wrapperDirectories.push(directory);
|
||||
const f = fixture(), eventPath = path.join(directory, "event.json"), launchPath = path.join(directory, "launch.ts");
|
||||
const canonicalLaunch = new URL("./launch.ts", import.meta.url);
|
||||
writeFileSync(eventPath, JSON.stringify(f.event));
|
||||
writeFileSync(launchPath, actual.readFileSync(canonicalLaunch));
|
||||
vi.mocked(readFileSync).mockImplementation(actual.readFileSync).mockClear();
|
||||
const env: NodeJS.ProcessEnv = { ...f.env, GITHUB_EVENT_PATH: eventPath };
|
||||
return { ...f, env, directory, eventPath, launchPath, canonicalLaunch, actual };
|
||||
}
|
||||
|
||||
it("admits the real filesystem wrapper with the actual catalog and canonical launcher without fixture callbacks", async () => {
|
||||
const f = await wrapperFixture();
|
||||
const callbacks = [vi.spyOn(f.selected.profile, "buildAgent"), vi.spyOn(f.selected.task, "buildPrompt")];
|
||||
const result = shippedChatEvidenceAuthority(f.env);
|
||||
expect(result).toMatchObject({ authority: "paperclip.shipped-chat-testops/v1", workflowSha: f.env.GITHUB_SHA,
|
||||
targetSha: f.env.GITHUB_SHA, runId: f.env.GITHUB_RUN_ID, attempt: "1", executionIds: [id], private: true,
|
||||
launchSha256: createHash("sha256").update(f.actual.readFileSync(f.canonicalLaunch)).digest("hex") });
|
||||
expect(Object.isFrozen(result)).toBe(true);
|
||||
expect(callbacks.every(callback => callback.mock.calls.length === 0)).toBe(true);
|
||||
expect(vi.mocked(readFileSync).mock.calls.map(([file]) => String(file))).toEqual([f.eventPath, String(f.canonicalLaunch)]);
|
||||
});
|
||||
|
||||
it.each(["missing", "absent", "directory", "symlink", "oversize", "invalid-json"])("rejects %s event custody through the real wrapper", async kind => {
|
||||
const f = await wrapperFixture();
|
||||
if (kind === "missing") delete f.env.GITHUB_EVENT_PATH;
|
||||
if (kind === "absent") rmSync(f.eventPath);
|
||||
if (kind === "directory") { rmSync(f.eventPath); mkdirSync(f.eventPath); }
|
||||
if (kind === "symlink") {
|
||||
const target = path.join(f.directory, "real-event.json"); writeFileSync(target, JSON.stringify(f.event));
|
||||
rmSync(f.eventPath); symlinkSync(target, f.eventPath);
|
||||
}
|
||||
if (kind === "oversize") writeFileSync(f.eventPath, " ".repeat(65_537));
|
||||
if (kind === "invalid-json") writeFileSync(f.eventPath, "{invalid");
|
||||
expect(() => shippedChatEvidenceAuthority(f.env)).toThrow();
|
||||
expect(vi.mocked(readFileSync).mock.calls.some(([file]) => String(file) === String(f.canonicalLaunch))).toBe(false);
|
||||
});
|
||||
|
||||
it.each(["sender", "bootstrap", "model-input", "source"])("rejects a tampered %s through event parsing and wrapper admission", async kind => {
|
||||
const f = await wrapperFixture();
|
||||
if (kind === "sender") f.event.sender.id = 456;
|
||||
if (kind === "bootstrap") f.event.inputs.operation = "bootstrap";
|
||||
if (kind === "model-input") f.event.inputs.model = "foreign/model";
|
||||
if (kind === "source") f.env.TARGET_SHA = "c".repeat(40);
|
||||
writeFileSync(f.eventPath, JSON.stringify(f.event));
|
||||
expect(() => shippedChatEvidenceAuthority(f.env)).toThrow("Invalid shipped chat authority");
|
||||
});
|
||||
|
||||
it.each(["missing", "model"])("rejects %s selected catalog metadata in the actual wrapper", async kind => {
|
||||
const f = await wrapperFixture();
|
||||
const find = vi.spyOn(runnerMatrix, "find");
|
||||
if (kind === "missing") find.mockReturnValue(undefined);
|
||||
else find.mockReturnValue({ ...f.selected, profile: { ...f.selected.profile, model: "foreign/model" } });
|
||||
expect(() => shippedChatEvidenceAuthority(f.env)).toThrow(kind === "missing" ? "Shipped chat execution is absent" : "Unexpected shipped chat selection or bounds");
|
||||
});
|
||||
|
||||
it.each(["absent", "directory", "denied"])("propagates a %s launcher read from a test-only filesystem port", async kind => {
|
||||
const f = await wrapperFixture();
|
||||
if (kind === "absent") rmSync(f.launchPath);
|
||||
if (kind === "directory") { rmSync(f.launchPath); mkdirSync(f.launchPath); }
|
||||
vi.mocked(readFileSync).mockImplementation((file, options) => {
|
||||
if (String(file) !== String(f.canonicalLaunch)) return f.actual.readFileSync(file, options);
|
||||
if (kind === "denied") throw Object.assign(new Error("Owned launcher unreadable"), { code: "EACCES" });
|
||||
return f.actual.readFileSync(f.launchPath, options);
|
||||
});
|
||||
expect(() => shippedChatEvidenceAuthority(f.env)).toThrow(expect.objectContaining({ code: kind === "absent" ? "ENOENT" : kind === "directory" ? "EISDIR" : "EACCES" }));
|
||||
});
|
||||
|
||||
it("binds changed real launcher bytes to new authority rather than inventing an independent trusted hash", async () => {
|
||||
const f = await wrapperFixture();
|
||||
vi.mocked(readFileSync).mockImplementation((file, options) => f.actual.readFileSync(
|
||||
String(file) === String(f.canonicalLaunch) ? f.launchPath : file, options));
|
||||
const original = shippedChatEvidenceAuthority(f.env);
|
||||
writeFileSync(f.launchPath, "source-only changed launcher bytes\n");
|
||||
const changed = shippedChatEvidenceAuthority(f.env);
|
||||
expect(changed.launchSha256).toBe(createHash("sha256").update(f.actual.readFileSync(f.launchPath)).digest("hex"));
|
||||
expect(changed.launchSha256).not.toBe(original.launchSha256);
|
||||
expect(changed.authoritySha256).not.toBe(original.authoritySha256);
|
||||
});
|
||||
@@ -0,0 +1,48 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { lstatSync, readFileSync } from "node:fs";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { MANUAL_PRIVATE_PROTOCOL } from "./manual-private-retention.js";
|
||||
import type { MatrixExecution } from "./types.js";
|
||||
|
||||
const executionId = "chat-features.runner-opencode-deepseek-1017.local.create-backlog";
|
||||
const authority = "paperclip.shipped-chat-testops/v1";
|
||||
const repository = "paperclipai/paperclip";
|
||||
const workflow = ".github/workflows/chat-feature-testops.yml";
|
||||
|
||||
export function authorizeShippedChat(env: NodeJS.ProcessEnv, event: unknown, selected: MatrixExecution, launchSha256: string) {
|
||||
const payload = event as { ref?: unknown; sender?: { id?: unknown }; repository?: { full_name?: unknown; private?: unknown; default_branch?: unknown }; inputs?: Record<string, unknown> };
|
||||
const branch = payload.repository?.default_branch;
|
||||
const actor = env.GITHUB_ACTOR_ID, triggering = env.TESTOPS_TRIGGERING_ACTOR_ID;
|
||||
const allowed = JSON.parse(env.TESTOPS_ALLOWED_ACTOR_IDS ?? "null");
|
||||
if (env.PAPERCLIP_TESTOPS_CHAT_MAIN_ADMISSION !== "v1" || env.GITHUB_ACTIONS !== "true" || env.GITHUB_EVENT_NAME !== "workflow_dispatch"
|
||||
|| env.GITHUB_REPOSITORY !== repository || payload.repository?.full_name !== repository || payload.repository.private !== false
|
||||
|| typeof branch !== "string" || !/^[A-Za-z0-9._/-]+$/.test(branch) || env.GITHUB_REF !== `refs/heads/${branch}` || payload.ref !== env.GITHUB_REF
|
||||
|| env.GITHUB_WORKFLOW_REF !== `${repository}/${workflow}@refs/heads/${branch}` || !["chat", "report"].includes(env.GITHUB_JOB ?? "")
|
||||
|| !/^[a-f0-9]{40}$/.test(env.GITHUB_SHA ?? "") || env.GITHUB_SHA !== env.TARGET_SHA || env.GITHUB_SHA !== env.TESTOPS_ADMITTED_SHA
|
||||
|| !/^[1-9][0-9]{0,19}$/.test(env.GITHUB_RUN_ID ?? "") || env.GITHUB_RUN_ATTEMPT !== "1"
|
||||
|| !/^[1-9][0-9]{0,19}$/.test(actor ?? "") || !/^[1-9][0-9]{0,19}$/.test(triggering ?? "") || payload.sender?.id !== Number(actor)
|
||||
|| !Array.isArray(allowed) || !allowed.length || allowed.some(v => !Number.isSafeInteger(v) || v <= 0) || !allowed.includes(Number(actor)) || !allowed.includes(Number(triggering))
|
||||
|| !payload.inputs || Object.keys(payload.inputs).some(k => !["ALLURE_JOB_RUN_ID", "ALLURE_USERNAME", "operation"].includes(k)) || (payload.inputs.operation ?? "evaluate") !== "evaluate" || !/^[1-9][0-9]{0,19}$/.test(String(payload.inputs.ALLURE_JOB_RUN_ID ?? ""))
|
||||
|| !/^[a-f0-9]{64}$/.test(launchSha256)) throw new Error("Invalid shipped chat authority");
|
||||
if (JSON.stringify(JSON.parse(env.MANUAL_PRIVATE_IDS ?? "null")) !== JSON.stringify([executionId]) || env.SELECTED_EXECUTION_ID && env.SELECTED_EXECUTION_ID !== executionId
|
||||
|| selected.id !== executionId || selected.suite.id !== "chat-features" || selected.profile.id !== "runner-opencode-deepseek-1017" || selected.environment.id !== "local"
|
||||
|| selected.profile.provider !== "opencode" || selected.profile.model !== "openrouter/deepseek/deepseek-v4.1-flash-1017" || selected.profile.expectedRuntimeMode !== "native"
|
||||
|| selected.requiredCredentials.length !== 1 || selected.requiredCredentials[0] !== "OPENROUTER_API_KEY" || selected.task.id !== "create-backlog"
|
||||
|| selected.task.expectedRunCount !== 1 || selected.task.turnTimeoutMs !== 120_000 || selected.task.attemptTimeoutMs.local !== 420_000 || selected.task.automaticRetryPolicy !== "single_attempt"
|
||||
|| selected.suite.definitionMetadata?.companyAndAgentBudgetCents !== 500 || selected.suite.definitionMetadata?.maximumAttemptsPerCell !== 1) throw new Error("Unexpected shipped chat selection or bounds");
|
||||
const identity = { workflowSha: env.GITHUB_SHA!, targetSha: env.GITHUB_SHA!, runId: env.GITHUB_RUN_ID!, attempt: "1", executionIds: Object.freeze([executionId]) };
|
||||
const authoritySha256 = createHash("sha256").update(JSON.stringify({ authority, ...identity, actor, triggering, jobRunId: payload.inputs.ALLURE_JOB_RUN_ID, launchSha256 })).digest("hex");
|
||||
return Object.freeze({ ...identity, authority, protocol: MANUAL_PRIVATE_PROTOCOL, launchSha256, authoritySha256, approvalSha256: authoritySha256, private: true as const });
|
||||
}
|
||||
|
||||
export function shippedChatEvidenceAuthority(env: NodeJS.ProcessEnv) {
|
||||
const eventPath = env.GITHUB_EVENT_PATH;
|
||||
if (!eventPath) throw new Error("Missing shipped chat event");
|
||||
const stat = lstatSync(eventPath);
|
||||
if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 65536) throw new Error("Invalid shipped chat event custody");
|
||||
const event = JSON.parse(readFileSync(eventPath, "utf8"));
|
||||
const selected = runnerMatrix.find(row => row.id === executionId);
|
||||
if (!selected) throw new Error("Shipped chat execution is absent");
|
||||
const launch = readFileSync(new URL("./launch.ts", import.meta.url));
|
||||
return authorizeShippedChat(env, event, selected, createHash("sha256").update(launch).digest("hex"));
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { appendFileSync, lstatSync, readFileSync } from "node:fs";
|
||||
import { pathToFileURL } from "node:url";
|
||||
|
||||
export function validateChatTestOpsDispatch(env: NodeJS.ProcessEnv, event: unknown, triggeringActorId: number) {
|
||||
const e = event as { ref?: unknown; sender?: { id?: unknown }; repository?: { full_name?: unknown; private?: unknown; default_branch?: unknown }; inputs?: Record<string, unknown> };
|
||||
const branch = e.repository?.default_branch, allowed = JSON.parse(env.TESTOPS_ALLOWED_ACTOR_IDS ?? "null");
|
||||
const operation = e.inputs?.operation ?? "evaluate";
|
||||
if (env.GITHUB_ACTIONS !== "true" || env.GITHUB_EVENT_NAME !== "workflow_dispatch" || env.GITHUB_REPOSITORY !== "paperclipai/paperclip"
|
||||
|| e.repository?.full_name !== "paperclipai/paperclip" || e.repository.private !== false || typeof branch !== "string" || !/^[A-Za-z0-9._/-]+$/.test(branch)
|
||||
|| env.GITHUB_REF !== `refs/heads/${branch}` || e.ref !== env.GITHUB_REF || env.GITHUB_WORKFLOW_REF !== `paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/${branch}`
|
||||
|| !/^[a-f0-9]{40}$/.test(env.GITHUB_SHA ?? "") || env.GITHUB_RUN_ATTEMPT !== "1" || !/^[1-9][0-9]{0,19}$/.test(env.GITHUB_RUN_ID ?? "")
|
||||
|| !/^[1-9][0-9]{0,19}$/.test(env.GITHUB_ACTOR_ID ?? "") || e.sender?.id !== Number(env.GITHUB_ACTOR_ID)
|
||||
|| !Array.isArray(allowed) || !allowed.length || allowed.some(v => !Number.isSafeInteger(v) || v <= 0) || !allowed.includes(Number(env.GITHUB_ACTOR_ID)) || !allowed.includes(triggeringActorId)
|
||||
|| !e.inputs || Object.keys(e.inputs).some(k => !["operation", "ALLURE_JOB_RUN_ID", "ALLURE_USERNAME"].includes(k)) || !["bootstrap", "launch-plan", "evaluate"].includes(String(operation))
|
||||
|| operation === "evaluate" && !/^[1-9][0-9]{0,19}$/.test(String(e.inputs.ALLURE_JOB_RUN_ID ?? ""))
|
||||
|| operation !== "evaluate" && (e.inputs.ALLURE_JOB_RUN_ID ?? "") !== "") throw new Error("Chat TestOps dispatch rejected");
|
||||
return { sha: env.GITHUB_SHA!, operation: String(operation), triggeringActorId: String(triggeringActorId) };
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
try {
|
||||
if (!/^[A-Za-z0-9-]{1,39}$/.test(process.env.GITHUB_TRIGGERING_ACTOR ?? "")) throw new Error("Invalid actor");
|
||||
const file = process.env.GITHUB_EVENT_PATH!;
|
||||
const stat = lstatSync(file);
|
||||
if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 65536) throw new Error("Invalid event");
|
||||
const actor = Number(execFileSync("gh", ["api", `users/${process.env.GITHUB_TRIGGERING_ACTOR}`, "--jq", ".id"], { encoding: "utf8", timeout: 30_000, stdio: ["ignore", "pipe", "pipe"] }).trim());
|
||||
const checked = validateChatTestOpsDispatch(process.env, JSON.parse(readFileSync(file, "utf8")), actor);
|
||||
appendFileSync(process.env.GITHUB_OUTPUT!, `sha=${checked.sha}\noperation=${checked.operation}\ntriggering_actor_id=${checked.triggeringActorId}\n`);
|
||||
} catch { process.stderr.write("Fixed chat TestOps dispatch identity rejected before credential access\n"); process.exitCode = 1; }
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { mkdtemp, mkdir, readFile, rm, symlink, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, expect, it } from "vitest";
|
||||
import { exportTestOpsChat, TESTOPS_CHAT_EXECUTION } from "./testops-chat-export.js";
|
||||
|
||||
const directories: string[] = [];
|
||||
afterEach(async () => { await Promise.all(directories.splice(0).map(d => rm(d, { recursive: true, force: true }))); });
|
||||
const binding = { runId: "123456", attempt: "1", workflowSha: "a".repeat(40), targetSha: "b".repeat(40) };
|
||||
const png = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+/l9sAAAAASUVORK5CYII=", "base64");
|
||||
const digest = (bytes: Buffer) => createHash("sha256").update(bytes).digest("hex");
|
||||
async function fixture() {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), "testops-chat-export-")); directories.push(root);
|
||||
const report = path.join(root, "report"), output = path.join(root, "export");
|
||||
const screenshot = `evidence/${TESTOPS_CHAT_EXECUTION}/attempt-1/final-state.png`;
|
||||
await mkdir(path.dirname(path.join(report, screenshot)), { recursive: true });
|
||||
await writeFile(path.join(report, screenshot), png);
|
||||
const result = {
|
||||
schema: "paperclip.runner-e2e.result/v2", executionId: TESTOPS_CHAT_EXECUTION, suiteId: "chat-features", suiteDefinitionHash: "c".repeat(64),
|
||||
source: { sha: binding.targetSha, ref: "refs/heads/master", workflowRunUrl: `https://github.com/paperclipai/paperclip/actions/runs/${binding.runId}` },
|
||||
attempt: 1, status: "passed", cleanup: "passed", evidenceValid: true, runIds: ["synthetic-run"],
|
||||
profileId: "runner-opencode-deepseek-1017", environmentId: "local", caseId: "create-backlog", provider: "opencode", model: "openrouter/deepseek/deepseek-v4.1-flash-1017", runtimeMode: "native",
|
||||
startedAt: "2026-10-30T00:00:00.000Z", finishedAt: "2026-10-30T00:00:01.000Z", durationMs: 1000,
|
||||
matcherResults: [{ matcher: { kind: "json_path", path: "chat.saved-backlog", expected: true }, passed: true, detail: "Private diagnostic body" }],
|
||||
screenshots: [{ id: "final-state", label: "Fixture", file: "final-state.png", publication: "public-runner-fixture", sha256: digest(png) }],
|
||||
billing: { llm: { runCount: 1, runsWithTokenUsage: 1, runsWithReportedCost: 1, inputTokens: 100, outputTokens: 20, cachedInputTokens: 0, totalTokens: 120, reportedCostUsd: 0.01, costStatus: "reported" },
|
||||
runtime: { provider: "local", agentRunDurationMs: 1000, leaseDurationMs: null, leaseCount: 0, costStatus: "not_metered", costSource: "local_not_metered" },
|
||||
reportedCostUsd: 0.01, estimatedRuntimeCostUsd: 0, observedAndEstimatedCostUsd: null, complete: false },
|
||||
usage: { hiddenReasoning: "Do not export this provider body" }, unknownProviderBody: "Do not export this extra result field",
|
||||
};
|
||||
const campaign = { schema: "paperclip.runner-e2e.campaign/v2", campaignId: `gha-${binding.runId}-1`, expected: [TESTOPS_CHAT_EXECUTION], results: [result], retries: 0, generatedAt: result.finishedAt };
|
||||
const save = () => writeFile(path.join(report, "normalized-results.json"), JSON.stringify(campaign));
|
||||
await save(); return { report, output, result, campaign, save, screenshot };
|
||||
}
|
||||
|
||||
it("exports the actual normalized result, marked PNG and bound hashes without opaque diagnostics", async () => {
|
||||
const f = await fixture(); await writeFile(path.join(f.report, "unapproved.log"), "private original");
|
||||
const receipt = await exportTestOpsChat(f.report, f.output, binding);
|
||||
const text = await readFile(path.join(f.output, "normalized-results.json"), "utf8"), projected = JSON.parse(text);
|
||||
expect(projected.results[0]).toMatchObject({ status: "passed", cleanup: "passed", provider: "opencode", model: f.result.model, runtimeMode: "native" });
|
||||
expect(projected.billing.llm.inputTokens).toBe(100); expect(projected.billing.testsWithCompleteBilling).toBe(0);
|
||||
expect(projected.complete).toBe(false);
|
||||
expect(text).not.toMatch(/Private diagnostic|hiddenReasoning|unknownProviderBody|Do not export/);
|
||||
expect(receipt.files.map(f => f.path).sort()).toEqual([f.screenshot, "junit.xml", "normalized-results.json"].sort());
|
||||
for (const file of receipt.files) expect(digest(await readFile(path.join(f.output, file.path)))).toBe(file.sha256);
|
||||
expect(JSON.parse(await readFile(path.join(f.output, "binding.json"), "utf8"))).toMatchObject({ ...binding, executionId: TESTOPS_CHAT_EXECUTION });
|
||||
expect(await readFile(path.join(f.output, "junit.xml"), "utf8")).toContain('failures="0"');
|
||||
});
|
||||
|
||||
it("preserves a real failed grade and failed cleanup as failed JUnit", async () => {
|
||||
const f = await fixture(); f.result.status = "failed"; f.result.cleanup = "failed"; await f.save();
|
||||
await exportTestOpsChat(f.report, f.output, binding);
|
||||
expect(JSON.parse(await readFile(path.join(f.output, "normalized-results.json"), "utf8")).results[0]).toMatchObject({ status: "failed", cleanup: "failed" });
|
||||
expect(await readFile(path.join(f.output, "junit.xml"), "utf8")).toContain('failures="1"');
|
||||
});
|
||||
|
||||
it.each(["selection", "source", "model", "retry", "evidence", "oracle", "cleanup", "run-count"])("rejects inconsistent %s before exporting a PASS", async kind => {
|
||||
const f = await fixture();
|
||||
if (kind === "selection") f.campaign.expected.push("other");
|
||||
if (kind === "source") f.result.source.sha = "d".repeat(40);
|
||||
if (kind === "model") f.result.model = "other/model";
|
||||
if (kind === "retry") f.result.attempt = 2;
|
||||
if (kind === "evidence") f.result.evidenceValid = false;
|
||||
if (kind === "oracle") f.result.matcherResults[0]!.passed = false;
|
||||
if (kind === "cleanup") f.result.cleanup = "failed";
|
||||
if (kind === "run-count") f.result.runIds.push("extra-run");
|
||||
await f.save(); await expect(exportTestOpsChat(f.report, f.output, binding)).rejects.toThrow();
|
||||
await expect(readFile(path.join(f.output, "binding.json"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
|
||||
it.each(["unmarked", "hash", "symlink"])("rejects %s screenshot evidence", async kind => {
|
||||
const f = await fixture();
|
||||
if (kind === "unmarked") f.result.screenshots[0]!.publication = "private";
|
||||
if (kind === "hash") f.result.screenshots[0]!.sha256 = "d".repeat(64);
|
||||
if (kind === "symlink") { const target = path.join(f.report, "original.png"); await writeFile(target, png); await rm(path.join(f.report, f.screenshot)); await symlink(target, path.join(f.report, f.screenshot)); }
|
||||
await f.save(); await expect(exportTestOpsChat(f.report, f.output, binding)).rejects.toThrow();
|
||||
});
|
||||
|
||||
it("rejects a workflow replay and refuses to overwrite an existing export", async () => {
|
||||
const f = await fixture(); await expect(exportTestOpsChat(f.report, f.output, { ...binding, attempt: "2" })).rejects.toThrow();
|
||||
await exportTestOpsChat(f.report, f.output, binding);
|
||||
await expect(exportTestOpsChat(f.report, f.output, binding)).rejects.toMatchObject({ code: "EEXIST" });
|
||||
});
|
||||
@@ -0,0 +1,104 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { lstat, mkdir, readFile, realpath, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { buildRunnerCampaign } from "./history.js";
|
||||
import { findSecretLeakInJsonValues, assertSecretFree } from "./redaction.js";
|
||||
import { validateRetainedRunnerResult } from "./result-validation.js";
|
||||
import { PUBLIC_RUNNER_SCREENSHOT_MARKER } from "./screenshot-policy.js";
|
||||
import type { RunnerE2EResult } from "./types.js";
|
||||
|
||||
export const TESTOPS_CHAT_EXECUTION = "chat-features.runner-opencode-deepseek-1017.local.create-backlog";
|
||||
const model = "openrouter/deepseek/deepseek-v4.1-flash-1017";
|
||||
const hash = (bytes: Uint8Array | string) => createHash("sha256").update(bytes).digest("hex");
|
||||
const sha = /^[a-f0-9]{40}$/;
|
||||
const pngMagic = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]);
|
||||
export interface TestOpsChatBinding { runId: string; attempt: string; workflowSha: string; targetSha: string }
|
||||
|
||||
function publicBilling(value: NonNullable<RunnerE2EResult["billing"]>) {
|
||||
const pick = (source: object, keys: string[]) => Object.fromEntries(keys.filter(key => Object.hasOwn(source, key)).map(key => [key, (source as Record<string, unknown>)[key]]));
|
||||
const numeric = (source: object, keys: string[]) => {
|
||||
const selected = pick(source, keys);
|
||||
if (Object.values(selected).some(v => v !== null && (typeof v !== "number" || !Number.isFinite(v) || v < 0))) throw new Error("Invalid billing scalar");
|
||||
return selected;
|
||||
};
|
||||
return {
|
||||
llm: { ...numeric(value.llm, ["runCount", "runsWithTokenUsage", "runsWithReportedCost", "runsWithEstimatedCost", "runsWithPartialReportedCost", "inputTokens", "outputTokens", "cachedInputTokens", "totalTokens", "reportedCostUsd", "estimatedCostUsd"]), costStatus: value.llm.costStatus },
|
||||
runtime: { ...numeric(value.runtime, ["agentRunDurationMs", "leaseDurationMs", "leaseCount", "cpuCores", "memoryGiB", "diskGiB", "estimatedListCostUsd"]), provider: value.runtime.provider, costStatus: value.runtime.costStatus, costSource: value.runtime.costSource },
|
||||
...numeric(value, ["reportedCostUsd", "estimatedRuntimeCostUsd", "observedAndEstimatedCostUsd"]), complete: value.complete,
|
||||
} as RunnerE2EResult["billing"];
|
||||
}
|
||||
|
||||
async function ownedRead(root: string, relative: string, limit: number) {
|
||||
const segments = relative.split("/");
|
||||
if (segments.some(s => !/^[A-Za-z0-9._-]+$/.test(s) || s === "." || s === "..")) throw new Error("Unsafe export source");
|
||||
let current = root;
|
||||
const rootStat = await lstat(current);
|
||||
if (!rootStat.isDirectory() || rootStat.isSymbolicLink()) throw new Error("Invalid report root");
|
||||
for (const [index, segment] of segments.entries()) {
|
||||
current = path.join(current, segment);
|
||||
const st = await lstat(current);
|
||||
if (st.isSymbolicLink() || (index < segments.length - 1 ? !st.isDirectory() : !st.isFile() || st.nlink !== 1 || st.size > limit)) throw new Error("Invalid export source");
|
||||
}
|
||||
if (!(await realpath(current)).startsWith(`${await realpath(root)}${path.sep}`)) throw new Error("Export source escapes report");
|
||||
const before = await lstat(current), bytes = await readFile(current), after = await lstat(current);
|
||||
if (bytes.length > limit || before.ino !== after.ino || before.dev !== after.dev || before.size !== after.size || before.mtimeMs !== after.mtimeMs || before.ctimeMs !== after.ctimeMs) throw new Error("Export source changed");
|
||||
return bytes;
|
||||
}
|
||||
|
||||
export async function exportTestOpsChat(reportRoot: string, output: string, binding: TestOpsChatBinding) {
|
||||
if (!/^[1-9][0-9]{0,19}$/.test(binding.runId) || binding.attempt !== "1" || !sha.test(binding.workflowSha) || !sha.test(binding.targetSha)) throw new Error("Invalid TestOps run binding");
|
||||
const raw = JSON.parse((await ownedRead(reportRoot, "normalized-results.json", 8 * 1024 * 1024)).toString("utf8"));
|
||||
const campaignId = `gha-${binding.runId}-${binding.attempt}`;
|
||||
if (raw.schema !== "paperclip.runner-e2e.campaign/v2" || raw.campaignId !== campaignId || !Array.isArray(raw.expected) || raw.expected.length !== 1 || raw.expected[0] !== TESTOPS_CHAT_EXECUTION || !Array.isArray(raw.results) || raw.results.length !== 1 || raw.retries !== 0) throw new Error("Foreign or repeated TestOps selection");
|
||||
const original = raw.results[0];
|
||||
validateRetainedRunnerResult(original);
|
||||
if (original.executionId !== TESTOPS_CHAT_EXECUTION || original.suiteId !== "chat-features" || original.profileId !== "runner-opencode-deepseek-1017" || original.environmentId !== "local" || original.caseId !== "create-backlog" || original.provider !== "opencode" || original.model !== model || original.runtimeMode !== "native" || original.attempt !== 1 || original.source?.sha !== binding.targetSha) throw new Error("Foreign TestOps result");
|
||||
if (!original.billing) throw new Error("Missing retained billing coverage");
|
||||
const checks = original.matcherResults ?? [];
|
||||
if (original.status === "passed" && (original.cleanup !== "passed" || raw.results[0].evidenceValid !== true || checks.length === 0 || checks.some(c => !c.passed) || original.runIds?.length !== 1)) throw new Error("Incomplete passing TestOps evidence");
|
||||
const screenshots = (original.screenshots ?? []).filter(s => s.publication === PUBLIC_RUNNER_SCREENSHOT_MARKER);
|
||||
if (screenshots.length > 8 || original.status === "passed" && !screenshots.some(s => s.id === "final-state" && s.file === "final-state.png")) throw new Error("Missing or excessive fixture screenshots");
|
||||
const files = new Map<string, Buffer>();
|
||||
for (const shot of screenshots) {
|
||||
if (!/^[A-Za-z0-9][A-Za-z0-9._-]{0,199}\.png$/.test(shot.file) || !/^[a-f0-9]{64}$/.test(shot.sha256 ?? "")) throw new Error("Unbound fixture screenshot");
|
||||
const relative = `evidence/${TESTOPS_CHAT_EXECUTION}/attempt-1/${shot.file}`;
|
||||
const bytes = await ownedRead(reportRoot, relative, 12 * 1024 * 1024);
|
||||
if (!bytes.subarray(0, 8).equals(pngMagic) || bytes.subarray(12, 16).toString("ascii") !== "IHDR" || hash(bytes) !== shot.sha256 || files.has(relative)) throw new Error("Invalid fixture PNG");
|
||||
assertSecretFree(bytes, [], "TestOps fixture PNG", { includeShapes: true });
|
||||
files.set(relative, bytes);
|
||||
}
|
||||
const result: RunnerE2EResult = {
|
||||
schema: original.schema, executionId: TESTOPS_CHAT_EXECUTION, suiteId: "chat-features", suiteDefinitionHash: original.suiteDefinitionHash,
|
||||
source: { sha: binding.targetSha, ref: null, workflowRunUrl: `https://github.com/paperclipai/paperclip/actions/runs/${binding.runId}` },
|
||||
attempt: 1, status: original.status, ...(original.failureClass ? { failureClass: original.failureClass } : {}),
|
||||
...(original.status === "failed" ? { error: "Retained chat feature evaluation failed; private originals preserve diagnostics" } : {}),
|
||||
profileId: "runner-opencode-deepseek-1017", environmentId: "local", caseId: "create-backlog", provider: "opencode", model, runtimeMode: "native",
|
||||
runIds: original.runIds?.map(id => { if (!/^[A-Za-z0-9._-]{1,100}$/.test(id)) throw new Error("Invalid task run identity"); return id; }),
|
||||
startedAt: original.startedAt, finishedAt: original.finishedAt, durationMs: original.durationMs, cleanup: original.cleanup,
|
||||
billing: publicBilling(original.billing),
|
||||
matcherResults: checks.map(check => {
|
||||
if (check.matcher.kind !== "json_path" || !/^[A-Za-z0-9._-]{1,150}$/.test(check.matcher.path) || check.matcher.expected !== true) throw new Error("Unsupported chat oracle check");
|
||||
return { matcher: { kind: "json_path", path: check.matcher.path, expected: true }, passed: check.passed, detail: "Retained independent chat-feature oracle check" };
|
||||
}),
|
||||
screenshots: screenshots.map(s => ({ id: s.id, label: "Reviewed synthetic chat-feature fixture", file: s.file, publication: PUBLIC_RUNNER_SCREENSHOT_MARKER, sha256: s.sha256 })),
|
||||
};
|
||||
validateRetainedRunnerResult(result);
|
||||
const campaign = buildRunnerCampaign({ campaignId, generatedAt: raw.generatedAt, expected: [TESTOPS_CHAT_EXECUTION], results: [result] });
|
||||
if (findSecretLeakInJsonValues(campaign, [], { includeShapes: true })) throw new Error("Unsafe TestOps projection");
|
||||
files.set("normalized-results.json", Buffer.from(`${JSON.stringify(campaign, null, 2)}\n`));
|
||||
const failed = result.status !== "passed" || result.cleanup !== "passed";
|
||||
files.set("junit.xml", Buffer.from(`<?xml version="1.0" encoding="UTF-8"?><testsuite name="Chat feature TestOps" tests="1" failures="${failed ? 1 : 0}"><testcase classname="paperclip.Product E2E.chat-features" name="create-backlog" time="${result.durationMs / 1000}">${failed ? '<failure message="Retained evaluation or cleanup failed"/>' : ""}</testcase></testsuite>\n`));
|
||||
const manifest = [...files].sort(([a], [b]) => a.localeCompare(b)).map(([file, bytes]) => ({ path: file, sha256: hash(bytes), bytes: bytes.length }));
|
||||
const receipt = { schema: "paperclip.chat-feature-testops-export/v1", ...binding, executionId: TESTOPS_CHAT_EXECUTION, campaignId, resultStatus: result.status, cleanup: result.cleanup, files: manifest };
|
||||
await mkdir(output, { mode: 0o700 });
|
||||
for (const [relative, bytes] of files) { const target = path.join(output, relative); await mkdir(path.dirname(target), { recursive: true, mode: 0o700 }); await writeFile(target, bytes, { flag: "wx", mode: 0o600 }); }
|
||||
await writeFile(path.join(output, "binding.json"), `${JSON.stringify(receipt, null, 2)}\n`, { flag: "wx", mode: 0o600 });
|
||||
return receipt;
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
exportTestOpsChat(path.resolve(process.env.TESTOPS_CHAT_REPORT_ROOT ?? "runner-e2e-merged-report/normalized"), path.resolve(process.env.TESTOPS_CHAT_EXPORT_ROOT ?? "runner-e2e-testops"), {
|
||||
runId: process.env.GITHUB_RUN_ID ?? "", attempt: process.env.GITHUB_RUN_ATTEMPT ?? "", workflowSha: process.env.GITHUB_SHA ?? "", targetSha: process.env.TESTOPS_CHAT_TARGET_SHA ?? "",
|
||||
}).catch(() => { process.stderr.write("Chat feature TestOps export rejected; private originals retained\n"); process.exitCode = 1; });
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { expect, it } from "vitest";
|
||||
import { prepareChatBuildLock } from "./testops-chat-lock.js";
|
||||
|
||||
const hash = (value: string) => createHash("sha256").update(value).digest("hex");
|
||||
async function fixture(source: "input" | "output" | "unknown") {
|
||||
const root = await mkdtemp(path.join(tmpdir(), "chat-reviewed-lock-"));
|
||||
const input = "reviewed input\n", output = "reviewed output\n";
|
||||
const patch = "diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml\n--- a/pnpm-lock.yaml\n+++ b/pnpm-lock.yaml\n@@ -1 +1 @@\n-reviewed input\n+reviewed output\n";
|
||||
await mkdir(path.join(root, "tests/runner-e2e"), { recursive: true });
|
||||
await writeFile(path.join(root, "package.json"), "{\"name\":\"synthetic-lock-owner\"}\n");
|
||||
await writeFile(path.join(root, "pnpm-lock.yaml"), source === "input" ? input : source === "output" ? output : "unreviewed source\n");
|
||||
await writeFile(path.join(root, "tests/runner-e2e/report-lock.patch"), patch);
|
||||
await writeFile(path.join(root, "tests/runner-e2e/report-lock.json"), JSON.stringify({ inputSha256: hash(input), outputSha256: hash(output), patchSha256: hash(patch) }));
|
||||
execFileSync("git", ["init", "-q"], { cwd: root });
|
||||
execFileSync("git", ["add", "."], { cwd: root });
|
||||
execFileSync("git", ["-c", "user.name=Fixture", "-c", "user.email=fixture@example.invalid", "commit", "-qm", "reviewed fixture"], { cwd: root });
|
||||
return { root, output };
|
||||
}
|
||||
|
||||
it.each(["input", "output"] as const)("uses the existing reviewed %s lock without resolving dependencies", async source => {
|
||||
const f = await fixture(source);
|
||||
try {
|
||||
const receipt = await prepareChatBuildLock(f.root);
|
||||
expect(receipt.runtimeLockSha256).toBe(hash(f.output));
|
||||
expect(receipt.sourceInputsUnchanged).toBe(true);
|
||||
expect(await readFile(path.join(f.root, "pnpm-lock.yaml"), "utf8")).toBe(f.output);
|
||||
expect(await prepareChatBuildLock(f.root)).toEqual(receipt);
|
||||
} finally { await rm(f.root, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
it.each(["unknown-source", "unknown-runtime", "manifest", "patch", "policy"])("rejects %s drift before any installer or artifact", async variant => {
|
||||
const f = await fixture(variant === "unknown-source" ? "unknown" : "output");
|
||||
try {
|
||||
if (variant === "unknown-source") await writeFile(path.join(f.root, "pnpm-lock.yaml"), f.output);
|
||||
if (variant === "unknown-runtime") await writeFile(path.join(f.root, "pnpm-lock.yaml"), "unreviewed runtime\n");
|
||||
if (variant === "manifest") await writeFile(path.join(f.root, "package.json"), "{\"name\":\"unreviewed-graph\"}\n");
|
||||
if (variant === "patch") await writeFile(path.join(f.root, "tests/runner-e2e/report-lock.patch"), "unreviewed patch\n");
|
||||
if (variant === "policy") await writeFile(path.join(f.root, "tests/runner-e2e/report-lock.json"), "{}\n");
|
||||
await expect(prepareChatBuildLock(f.root)).rejects.toThrow();
|
||||
} finally { await rm(f.root, { recursive: true, force: true }); }
|
||||
});
|
||||
@@ -0,0 +1,32 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { prepareReportLock } from "./prepare-report-lock.mjs";
|
||||
|
||||
const hash = (bytes: Buffer) => createHash("sha256").update(bytes).digest("hex");
|
||||
const inputs = ["package.json", ":(glob)**/package.json", "pnpm-workspace.yaml", ".npmrc", ".pnpmfile.cjs",
|
||||
":(glob)**/*.patch", "tests/runner-e2e/report-lock.json"];
|
||||
|
||||
export async function prepareChatBuildLock(root = path.resolve(import.meta.dirname, "../..")) {
|
||||
try {
|
||||
execFileSync("git", ["diff", "--quiet", "HEAD", "--", ...inputs], { cwd: root, stdio: "pipe" });
|
||||
} catch { throw new Error("Source dependency inputs changed before install or packaging"); }
|
||||
const committed = execFileSync("git", ["show", "HEAD:pnpm-lock.yaml"], { cwd: root });
|
||||
const policyBytes = await readFile(path.join(root, "tests/runner-e2e/report-lock.json"));
|
||||
const policy = JSON.parse(policyBytes.toString("utf8"));
|
||||
const sourceLockSha256 = hash(committed);
|
||||
if (![policy.inputSha256, policy.outputSha256].includes(sourceLockSha256))
|
||||
throw new Error("Committed dependency lock is not reviewed");
|
||||
await prepareReportLock(root);
|
||||
const runtimeLockSha256 = hash(await readFile(path.join(root, "pnpm-lock.yaml")));
|
||||
if (runtimeLockSha256 !== policy.outputSha256) throw new Error("Runtime dependency lock is not reviewed");
|
||||
return { sourceSha: execFileSync("git", ["rev-parse", "HEAD"], { cwd: root, encoding: "utf8" }).trim(),
|
||||
sourceLockSha256, runtimeLockSha256, policySha256: hash(policyBytes), sourceInputsUnchanged: true };
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) {
|
||||
prepareChatBuildLock().then(value => console.log(JSON.stringify(value)))
|
||||
.catch(() => { process.stderr.write("Chat TestOps dependency graph is not source-reviewed\n"); process.exitCode = 1; });
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
import { afterEach, expect, it } from "vitest";
|
||||
import { mkdtemp, realpath, rm } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { authorizeManualPrivateEvidence, MANUAL_PRIVATE_EXECUTIONS, MANUAL_PRIVATE_PROTOCOL, manualCellKind, manualPrivateSessionPolicy } from "./manual-private-retention.js";
|
||||
import { canonicalGeneratedReportAsset, cellTraceAsset, inspectCellReportProducer, privateCellEvidenceOwner } from "./manual-private-report-assets.js";
|
||||
|
||||
const id = "chat-features.runner-opencode-deepseek-1017.local.create-backlog";
|
||||
const roots: string[] = [];
|
||||
afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
|
||||
it.each(["cell-read", "cell-write", "report-write"] as const)("keeps the shipped chat %s session scoped to only its existing private archive", mode => {
|
||||
const policy = manualPrivateSessionPolicy({ bucket: "private-evidence", runId: "123", executionIds: [id], mode });
|
||||
const objects = policy.Statement.filter(row => Array.isArray(row.Resource)).flatMap(row => row.Resource);
|
||||
const filename = mode === "report-write" ? "manual-report" : manualCellKind(id);
|
||||
const key = `arn:aws:s3:::private-evidence/runner-e2e-private/bounded-v1/campaigns/gha-123-1/${filename}.zip`;
|
||||
expect(objects).toEqual([key, `${key}.upload-receipt.json`]);
|
||||
expect(JSON.stringify(policy)).not.toContain("*");
|
||||
expect(() => manualPrivateSessionPolicy({ bucket: "private-evidence", runId: "123", executionIds: [id.replace("create-backlog", "delegate-announcement")], mode })).toThrow();
|
||||
});
|
||||
it("preserves the frozen244 candidate approval registry and refuses shipped authority through that old authorizer", () => {
|
||||
expect(MANUAL_PRIVATE_EXECUTIONS).toHaveLength(244);
|
||||
expect(MANUAL_PRIVATE_EXECUTIONS).not.toContain(id);
|
||||
const identity = { workflowSha: "a".repeat(40), targetSha: "b".repeat(40), runId: "123", attempt: "1", executionIds: [id] };
|
||||
expect(() => authorizeManualPrivateEvidence(identity, JSON.stringify({ version: 1, approvals: [{ ...identity, protocol: MANUAL_PRIVATE_PROTOCOL, launchSha256: "c".repeat(64) }] }))).toThrow("Unsupported private manual evidence selection");
|
||||
});
|
||||
it("binds canonical private trace assets to the exact shipped profile and rejects a rewritten profile or sibling case", async () => {
|
||||
const root = await realpath(await mkdtemp(path.join(os.tmpdir(), "testops-private-path-"))); roots.push(root);
|
||||
const context = { root, private: true as const, runId: "123", attempt: "1", executionIds: [id], cellReportProducer: await inspectCellReportProducer() };
|
||||
const owner = await privateCellEvidenceOwner(context, id);
|
||||
const prefix = `gha-123-1-${id}/chat-features/runner-opencode-deepseek-1017/local/create-backlog/attempt-1/html-report/trace/`;
|
||||
expect(cellTraceAsset(`${prefix}playwright-logo.svg`, owner, "final")).toEqual({ reserved: true, canonical: canonicalGeneratedReportAsset("html/trace/playwright-logo.svg") });
|
||||
expect(cellTraceAsset(`${prefix.replace("/runner-opencode-deepseek-1017/", "/runner-opencode/")}playwright-logo.svg`, owner, "final")).toEqual({ reserved: true, canonical: undefined });
|
||||
for (const foreign of [id.replace("create-backlog", "delegate-announcement"), id.replace("deepseek-1017", "unknown")]) {
|
||||
await expect(privateCellEvidenceOwner({ ...context, executionIds: [foreign] }, foreign)).rejects.toThrow("Invalid private cell report owner");
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,70 @@
|
||||
import { createHash } from 'node:crypto';
|
||||
import { appendFile, lstat, readFile, realpath, writeFile } from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
import { publicContext, aliases, executionId, scenario } from './chat-feature-testops-api.mjs';
|
||||
import { run } from './chat-feature-testops-cli.mjs';
|
||||
const hash = bytes => createHash('sha256').update(bytes).digest('hex');
|
||||
const fail = () => { throw new Error('chat_testops_selection_rejected'); };
|
||||
|
||||
async function readVerifiedSelection(file, env) {
|
||||
const expected = publicContext(env);
|
||||
if (!/^[a-f0-9]{64}$/.test(env.EXPECTED_SELECTION_SHA256 ?? '') || !/^[1-9][0-9]{0,15}$/.test(env.ALLURE_JOB_RUN_ID ?? '')) fail();
|
||||
const root = await realpath(env.GITHUB_WORKSPACE ?? process.cwd()), absolute = path.resolve(file);
|
||||
if (!absolute.startsWith(`${root}${path.sep}`)) fail();
|
||||
let current = root;
|
||||
const parts = path.relative(root, absolute).split(path.sep);
|
||||
for (const [index, part] of parts.entries()) {
|
||||
if (!/^[A-Za-z0-9._-]+$/.test(part) || ['.', '..'].includes(part)) fail();
|
||||
current = path.join(current, part);
|
||||
const stat = await lstat(current);
|
||||
if (stat.isSymbolicLink() || (index < parts.length - 1 ? !stat.isDirectory() : !stat.isFile() || stat.nlink !== 1 || stat.size > 16_384)) fail();
|
||||
}
|
||||
const bytes = await readFile(absolute);
|
||||
if (hash(bytes) !== env.EXPECTED_SELECTION_SHA256) fail();
|
||||
const value = JSON.parse(bytes);
|
||||
if (value.schema !== 'paperclip.chat-feature-testops-selection/v1' || value.runId !== expected.runId || value.attempt !== '1'
|
||||
|| value.workflowSha !== expected.workflowSha || value.targetSha !== expected.targetSha || value.allureJobRunId !== env.ALLURE_JOB_RUN_ID
|
||||
|| value.executionId !== executionId || value.canonicalScenario !== scenario || value.projectId !== 1 || !aliases.includes(value.fullName)
|
||||
|| ['planId', 'jobId', 'caseId', 'launchId'].some(key => !Number.isSafeInteger(value[key]) || value[key] <= 0)) fail();
|
||||
return { value, bytes };
|
||||
}
|
||||
|
||||
export async function verifySelectionFile(file, env) {
|
||||
return (await readVerifiedSelection(file, env)).value;
|
||||
}
|
||||
|
||||
export async function reporter(command, argument, env = process.env, fetchImpl = fetch) {
|
||||
publicContext(env);
|
||||
if (command === 'verify-selection') {
|
||||
if (Object.hasOwn(env, 'ALLURE_TOKEN')) fail();
|
||||
await verifySelectionFile(argument, env);
|
||||
return 0;
|
||||
}
|
||||
if (!['bootstrap', 'launch-plan', 'selection', 'upload', 'verify-upload'].includes(command) || !env.RUNNER_TEMP) fail();
|
||||
const stage = command === 'selection' ? 'validate-selection' : command === 'upload' ? 'export' : command;
|
||||
const stateRoot = path.join(env.RUNNER_TEMP, `chat-testops-${command}-state`);
|
||||
const selectionPath = path.join(env.GITHUB_WORKSPACE ?? process.cwd(), 'chat-testops-selection', 'testops-selection.json');
|
||||
let copiedSelectionPath = selectionPath;
|
||||
if (command === 'upload' || command === 'verify-upload') {
|
||||
const { bytes } = await readVerifiedSelection(selectionPath, env);
|
||||
copiedSelectionPath = path.join(env.RUNNER_TEMP, 'chat-testops-authenticated-selection.json');
|
||||
if (command === 'upload') await writeFile(copiedSelectionPath, bytes, { flag: 'wx', mode: 0o600 });
|
||||
else if (!(await readFile(copiedSelectionPath)).equals(bytes)) fail();
|
||||
}
|
||||
const result = await run({ ...env, CHAT_TESTOPS_OPERATION: stage, CHAT_TESTOPS_STATE_ROOT: stateRoot,
|
||||
...(['upload', 'verify-upload'].includes(command) ? { CHAT_TESTOPS_EXPORT_ROOT: path.resolve(argument), CHAT_TESTOPS_SELECTION: copiedSelectionPath,
|
||||
ALLURE_RESULTS: env.ALLURE_RESULTS ?? path.join(env.RUNNER_TEMP, 'chat-testops-allure-results') } : {}) }, fetchImpl);
|
||||
if (command === 'selection' && result === 0) {
|
||||
const bytes = await readFile(path.join(stateRoot, 'selection.json'));
|
||||
await writeFile(path.join(env.GITHUB_WORKSPACE ?? process.cwd(), 'testops-selection.json'), bytes, { flag: 'wx', mode: 0o600 });
|
||||
if (!env.GITHUB_OUTPUT) fail();
|
||||
await appendFile(env.GITHUB_OUTPUT, `sha256=${hash(bytes)}\n`);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
reporter(process.argv[2], process.argv[3]).then(code => { process.exitCode = code; })
|
||||
.catch(() => { process.stderr.write('Chat TestOps reporter stopped: invalid source, custody or selection\n'); process.exitCode = 1; });
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { mkdtemp, mkdir, readFile, realpath, rm, writeFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { verifySelectionFile, reporter } from './testops-chat-reporter.mjs';
|
||||
import { executionId, scenario, fullName } from './chat-feature-testops-api.mjs';
|
||||
const context = { GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/heads/master', GITHUB_EVENT_NAME: 'workflow_dispatch',
|
||||
GITHUB_RUN_ATTEMPT: '1', GITHUB_SHA: 'a'.repeat(40), GITHUB_RUN_ID: '123', ALLURE_JOB_RUN_ID: '9',
|
||||
GITHUB_WORKFLOW_REF: 'paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master' };
|
||||
async function fixture(t, change = {}) {
|
||||
const root = await realpath(await mkdtemp(path.join(tmpdir(), 'testops-selection-')));
|
||||
t.after(() => rm(root, { recursive: true }));
|
||||
const value = { schema: 'paperclip.chat-feature-testops-selection/v1', runId: '123', attempt: '1', workflowSha: 'a'.repeat(40), targetSha: 'a'.repeat(40),
|
||||
allureJobRunId: '9', executionId, canonicalScenario: scenario, projectId: 1, planId: 8, jobId: 7, caseId: 91, fullName, launchId: 10, ...change };
|
||||
const bytes = Buffer.from(JSON.stringify(value));
|
||||
await mkdir(path.join(root, 'chat-testops-selection'));
|
||||
const file = path.join(root, 'chat-testops-selection', 'testops-selection.json'); await writeFile(file, bytes);
|
||||
return { file, env: { ...context, GITHUB_WORKSPACE: root, EXPECTED_SELECTION_SHA256: createHash('sha256').update(bytes).digest('hex') } };
|
||||
}
|
||||
|
||||
test('actual worker verifier requires no token or network and accepts exact source-bound selection hash', async t => {
|
||||
const f = await fixture(t);
|
||||
assert.equal((await verifySelectionFile(f.file, f.env)).caseId, 91);
|
||||
assert.equal(await reporter('verify-selection', f.file, f.env), 0);
|
||||
});
|
||||
|
||||
test('worker verifier rejects token contamination, hash mismatch and correctly-hashed foreign tuples', async t => {
|
||||
const f = await fixture(t);
|
||||
await assert.rejects(reporter('verify-selection', f.file, { ...f.env, ALLURE_TOKEN: 'SYNTHETIC_TOKEN' }));
|
||||
await assert.rejects(verifySelectionFile(f.file, { ...f.env, EXPECTED_SELECTION_SHA256: '0'.repeat(64) }));
|
||||
for (const change of [{ runId: '456' }, { targetSha: 'b'.repeat(40) }, { allureJobRunId: '10' }, { executionId: 'other' }, { projectId: 2 }]) {
|
||||
const foreign = await fixture(t, change);
|
||||
await assert.rejects(verifySelectionFile(foreign.file, foreign.env));
|
||||
}
|
||||
});
|
||||
|
||||
test('actual upload facade copies authenticated workspace bytes to owned temp before successful synthetic export', async t => {
|
||||
const f = await fixture(t), workspace = f.env.GITHUB_WORKSPACE;
|
||||
const temporary = path.join(workspace, 'temp'); await mkdir(temporary);
|
||||
const root = path.join(workspace, 'safe-export'); await mkdir(root);
|
||||
const png = Buffer.from([137,80,78,71,13,10,26,10,0,0,0,13,73,72,68,82]);
|
||||
const hash = bytes => createHash('sha256').update(bytes).digest('hex');
|
||||
const shot = { id: 'final-state', file: 'final-state.png', sha256: hash(png) };
|
||||
const result = { executionId, suiteId: 'chat-features', caseId: 'create-backlog', profileId: 'runner-opencode-deepseek-1017', provider: 'opencode',
|
||||
environmentId: 'local', model: 'openrouter/deepseek/deepseek-v4.1-flash-1017', runtimeMode: 'native', attempt: 1,
|
||||
source: { sha: context.GITHUB_SHA }, status: 'passed', cleanup: 'passed', runIds: ['synthetic'], durationMs: 123,
|
||||
matcherResults: [{ matcher: { kind: 'json_path', path: 'chatFeature.createdTask', expected: true }, passed: true }], screenshots: [shot] };
|
||||
const files = new Map([['normalized-results.json', Buffer.from(JSON.stringify({ schema: 'paperclip.runner-e2e.campaign/v2', campaignId: 'gha-123-1', results: [result] }))],
|
||||
['junit.xml', Buffer.from('<testsuite tests="1"/>')], [`evidence/${executionId}/attempt-1/final-state.png`, png]]);
|
||||
for (const [name, bytes] of files) { await mkdir(path.dirname(path.join(root, name)), { recursive: true }); await writeFile(path.join(root, name), bytes); }
|
||||
await writeFile(path.join(root, 'binding.json'), JSON.stringify({ schema: 'paperclip.chat-feature-testops-export/v1', runId: '123', attempt: '1',
|
||||
workflowSha: context.GITHUB_SHA, targetSha: context.GITHUB_SHA, executionId, campaignId: 'gha-123-1', resultStatus: 'passed', cleanup: 'passed',
|
||||
files: [...files].map(([name, bytes]) => ({ path: name, sha256: hash(bytes), bytes: bytes.length })) }));
|
||||
assert.equal(await reporter('upload', root, { ...f.env, RUNNER_TEMP: temporary }), 0);
|
||||
assert.deepEqual(await readFile(path.join(temporary, 'chat-testops-authenticated-selection.json')), await readFile(f.file));
|
||||
const exported = JSON.parse(await readFile(path.join(temporary, 'chat-testops-upload-state', 'export.json')));
|
||||
const jobRun = { id: 9, job: { id: 7, type: 'github' }, launchId: 10, stage: 'finished', url: 'https://github.com/paperclipai/paperclip/actions/runs/123' };
|
||||
const fetchImpl = async (url, options) => {
|
||||
const parsed = new URL(url), route = parsed.pathname;
|
||||
let value;
|
||||
if (route === '/api/uaa/oauth/token') { assert.equal(options.method, 'POST'); value = { access_token: 'MOCK_ACCESS_TOKEN' }; }
|
||||
else {
|
||||
assert.equal(options.method, 'GET');
|
||||
if (route === '/api/jobrun/9') value = jobRun;
|
||||
else if (route === '/api/launch/10') value = { id: 10, projectId: 1 };
|
||||
else if (route === '/api/testresult') value = { content: [{ id: 45, testCaseId: 91 }], totalElements: 1, totalPages: 1 };
|
||||
else if (route === '/api/testresult/45') value = { id: 45, projectId: 1, launchId: 10, testCaseId: 91, fullName, jobRun, status: 'passed',
|
||||
links: [{ url: jobRun.url }, { url: `https://github.com/paperclipai/paperclip/commit/${context.GITHUB_SHA}` }] };
|
||||
else if (route === '/api/testresult/attachment') value = { content: exported.attachmentMetadata.map((row, i) => ({ ...row, id: i + 1 })), totalElements: exported.attachmentMetadata.length, totalPages: 1 };
|
||||
else assert.fail(route);
|
||||
}
|
||||
return { ok: true, status: 200, json: async () => value, text: async () => JSON.stringify(value) };
|
||||
};
|
||||
assert.equal(await reporter('verify-upload', root, { ...f.env, RUNNER_TEMP: temporary, ALLURE_TOKEN: 'MOCK_API_TOKEN' }, fetchImpl), 0);
|
||||
const verified = JSON.parse(await readFile(path.join(temporary, 'chat-testops-verify-upload-state', 'verification.json')));
|
||||
assert.equal(verified.verification, 'VERIFIED'); assert.equal(verified.caseId, 91); assert.equal(verified.status, 'passed');
|
||||
|
||||
});
|
||||
|
||||
test('actual launch-plan facade performs metadata-only plan launch once and exits without waiting for child', async t => {
|
||||
const f = await fixture(t), workspace = f.env.GITHUB_WORKSPACE;
|
||||
const temporary = path.join(workspace, 'temp'); await mkdir(temporary);
|
||||
const eventPath = path.join(workspace, 'event.json'); await writeFile(eventPath, JSON.stringify({ inputs: { operation: 'launch-plan', ALLURE_JOB_RUN_ID: '' } }));
|
||||
const nonce = '1'.repeat(32), calls = [];
|
||||
const fetchImpl = async (url, options) => {
|
||||
const route = new URL(url).pathname, method = options.method; calls.push({ route, method });
|
||||
const response = value => ({ ok: true, status: 200, json: async () => value, text: async () => JSON.stringify(value) });
|
||||
if (route === '/repos/paperclipai/paperclip') return response({ full_name: 'paperclipai/paperclip', private: false, default_branch: 'master' });
|
||||
if (route === '/repos/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml') return response({ id: 99, path: '.github/workflows/chat-feature-testops.yml', state: 'active' });
|
||||
if (route === '/repos/paperclipai/paperclip/actions/runs/123') return response({ id: 123, workflow_id: 99, run_attempt: 1, run_number: 2, head_sha: context.GITHUB_SHA, head_branch: 'master', event: 'workflow_dispatch', repository: { full_name: 'paperclipai/paperclip', private: false, default_branch: 'master' }, actor: { id: 34892728 }, triggering_actor: { id: 34892728 } });
|
||||
if (route === '/api/uaa/oauth/token') return response({ access_token: 'SYNTHETIC_TOKEN' });
|
||||
if (route === '/api/testcase/91') return response({ id: 91, projectId: 1, fullName, automated: true });
|
||||
if (route === '/api/testplan/8') return response({ id: 8, projectId: 1, name: 'Chat feature evals', testCasesCount: 1, treeSelection: { leafsInclude: [91] } });
|
||||
if (route === '/api/job/7') return response({ id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml', parameters: [{ id: 17, value: 'evaluate' }] });
|
||||
if (route === '/api/integration/2/project/1') return response({ id: 2, projectId: 1, info: { type: 'github' }, disabled: false });
|
||||
if (route === '/api/environment/17') return response({ id: 17, name: 'operation' });
|
||||
if (route === '/api/testplan/8/job') return response([{ job: { id: 7, parameters: [{ id: 17, value: 'evaluate' }] }, testCasesCount: 1 }]);
|
||||
if (route === '/api/launch') return response({ content: [], totalPages: 0 });
|
||||
if (route === '/api/testplan/8/jobparameter') return { ok: true, status: 204 };
|
||||
assert.equal(route, '/api/testplan/8/run');
|
||||
return response({ id: 20, projectId: 1, name: `Chat feature evals · ${nonce}` });
|
||||
};
|
||||
assert.equal(await reporter('launch-plan', undefined, { ...f.env, RUNNER_TEMP: temporary, GITHUB_EVENT_PATH: eventPath,
|
||||
ALLURE_JOB_RUN_ID: '', ALLURE_TOKEN: 'SYNTHETIC_ONLY', CHAT_TESTOPS_CASE_ID: '91', CHAT_TESTOPS_PLAN_ID: '8', CHAT_TESTOPS_JOB_ID: '7',
|
||||
CHAT_TESTOPS_OPERATION_PARAMETER_ID: '17', CHAT_TESTOPS_INTEGRATION_ID: '2', CHAT_TESTOPS_LAUNCH_NONCE: nonce, CHAT_TESTOPS_LAUNCH_RUN_NUMBER: '2', GITHUB_RUN_NUMBER: '2',
|
||||
CHAT_TESTOPS_GITHUB_WORKFLOW_ID: '99', GITHUB_ACTOR_ID: '34892728', TESTOPS_TRIGGERING_ACTOR_ID: '34892728', TESTOPS_ALLOWED_ACTOR_IDS: '[34892728]', GH_TOKEN: 'SYNTHETIC_GH_TOKEN' }, fetchImpl), 0);
|
||||
const state = path.join(temporary, 'chat-testops-launch-plan-state');
|
||||
assert.equal(JSON.parse(await readFile(path.join(state, 'launch.json'))).status, 'REQUESTED');
|
||||
assert.equal(calls.filter(row => row.route === '/api/testplan/8/run').length, 1);
|
||||
assert.equal(calls.at(-1).route, '/api/testplan/8/run');
|
||||
});
|
||||
|
||||
test('actual launch caller rejects five own-run contradictions before TestOps auth or mutation', async t => {
|
||||
const original = process.stderr.write; process.stderr.write = () => true;
|
||||
try {
|
||||
for (const changes of [{ run_number: 3 }, { actor: { id: 1 } }, { head_sha: 'b'.repeat(40) }, { workflow_id: 100 }, { run_attempt: 2 }]) {
|
||||
const f = await fixture(t), workspace = f.env.GITHUB_WORKSPACE;
|
||||
const temporary = path.join(workspace, 'temp'); await mkdir(temporary);
|
||||
const eventPath = path.join(workspace, 'event.json'); await writeFile(eventPath, JSON.stringify({ inputs: { operation: 'launch-plan' } }));
|
||||
let testopsCalls = 0;
|
||||
const fetchImpl = async url => {
|
||||
if (new URL(url).hostname !== 'api.github.com') { testopsCalls++; throw new Error('must not reach TestOps'); }
|
||||
const value = url.endsWith('/repos/paperclipai/paperclip') ? { full_name: 'paperclipai/paperclip', private: false, default_branch: 'master' } : url.includes('/actions/workflows/') ? { id: 99, path: '.github/workflows/chat-feature-testops.yml', state: 'active' }
|
||||
: { id: 123, workflow_id: 99, run_attempt: 1, run_number: 2, head_sha: context.GITHUB_SHA, head_branch: 'master', event: 'workflow_dispatch',
|
||||
repository: { full_name: 'paperclipai/paperclip', private: false, default_branch: 'master' }, actor: { id: 34892728 }, triggering_actor: { id: 34892728 }, ...changes };
|
||||
return { ok: true, text: async () => JSON.stringify(value) };
|
||||
};
|
||||
assert.equal(await reporter('launch-plan', undefined, { ...f.env, RUNNER_TEMP: temporary, GITHUB_EVENT_PATH: eventPath,
|
||||
ALLURE_JOB_RUN_ID: '', ALLURE_TOKEN: 'SYNTHETIC_ONLY', CHAT_TESTOPS_LAUNCH_RUN_NUMBER: '2', GITHUB_RUN_NUMBER: '2',
|
||||
CHAT_TESTOPS_GITHUB_WORKFLOW_ID: '99', GITHUB_ACTOR_ID: '34892728', TESTOPS_TRIGGERING_ACTOR_ID: '34892728', TESTOPS_ALLOWED_ACTOR_IDS: '[34892728]', GH_TOKEN: 'SYNTHETIC_GH_TOKEN' }, fetchImpl), 1);
|
||||
assert.equal(testopsCalls, 0);
|
||||
const error = JSON.parse(await readFile(path.join(temporary, 'chat-testops-launch-plan-state', 'failure.json')));
|
||||
assert.equal(error.stage, 'launch_admission'); assert.equal(error.code, 'chat_testops_launch_admission');
|
||||
}
|
||||
} finally { process.stderr.write = original; }
|
||||
});
|
||||
@@ -0,0 +1,154 @@
|
||||
import { readFileSync } from "node:fs";
|
||||
import { parse } from "yaml";
|
||||
import { expect, it } from "vitest";
|
||||
import { validateChatTestOpsDispatch } from "./testops-chat-dispatch.js";
|
||||
|
||||
const workflow = parse(readFileSync(new URL("../../.github/workflows/chat-feature-testops.yml", import.meta.url), "utf8"));
|
||||
function context() {
|
||||
return { env: { GITHUB_ACTIONS: "true", GITHUB_EVENT_NAME: "workflow_dispatch", GITHUB_REPOSITORY: "paperclipai/paperclip", GITHUB_REF: "refs/heads/master",
|
||||
GITHUB_WORKFLOW_REF: "paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master", GITHUB_SHA: "a".repeat(40), GITHUB_RUN_ID: "12345", GITHUB_RUN_ATTEMPT: "1", GITHUB_ACTOR_ID: "123", TESTOPS_ALLOWED_ACTOR_IDS: "[123]" },
|
||||
event: { repository: { full_name: "paperclipai/paperclip", private: false, default_branch: "master" }, ref: "refs/heads/master", sender: { id: 123 }, inputs: { ALLURE_JOB_RUN_ID: "456" } as Record<string, string> } };
|
||||
}
|
||||
it("admits the actual TestOps service input and only the closed metadata bootstrap alternative", () => {
|
||||
const f = context(); expect(validateChatTestOpsDispatch(f.env, f.event, 123).operation).toBe("evaluate");
|
||||
f.event.inputs = { operation: "bootstrap" }; expect(validateChatTestOpsDispatch(f.env, f.event, 123).operation).toBe("bootstrap");
|
||||
f.event.inputs.operation = "launch-plan"; expect(validateChatTestOpsDispatch(f.env, f.event, 123).operation).toBe("launch-plan");
|
||||
f.event.inputs.ALLURE_JOB_RUN_ID = "456"; expect(() => validateChatTestOpsDispatch(f.env, f.event, 123)).toThrow();
|
||||
delete f.event.inputs.ALLURE_JOB_RUN_ID;
|
||||
f.event.inputs.operation = "other"; expect(() => validateChatTestOpsDispatch(f.env, f.event, 123)).toThrow();
|
||||
});
|
||||
it.each(["unlisted-trigger", "wrong-source", "replay", "extra-input", "missing-jobrun", "foreign-repo"])("rejects %s before any service credential operation", kind => {
|
||||
const f = context(); let trigger = 123;
|
||||
if (kind === "unlisted-trigger") trigger = 456;
|
||||
if (kind === "wrong-source") f.env.GITHUB_REF = "refs/heads/unreviewed";
|
||||
if (kind === "replay") f.env.GITHUB_RUN_ATTEMPT = "2";
|
||||
if (kind === "extra-input") f.event.inputs.target_branch = "other";
|
||||
if (kind === "missing-jobrun") f.event.inputs = {};
|
||||
if (kind === "foreign-repo") f.event.repository.full_name = "foreign/repo";
|
||||
expect(() => validateChatTestOpsDispatch(f.env, f.event, trigger)).toThrow();
|
||||
});
|
||||
it("keeps provider and TestOps service credentials in separate jobs", () => {
|
||||
const jobs = workflow.jobs;
|
||||
for (const [id, job] of Object.entries(jobs) as Array<[string, any]>) {
|
||||
const text = JSON.stringify(job);
|
||||
if (text.includes("secrets.OPENROUTER_API_KEY")) expect(id).toBe("chat");
|
||||
if (text.includes("secrets.ALLURE_TOKEN")) expect(["select", "upload"]).toContain(id);
|
||||
if (id !== "chat") expect(text).not.toContain("pnpm test:e2e:runner --");
|
||||
expect(job.steps.some((step: any) => step.run === "node tests/runner-e2e/testops-chat-dispatch.ts")).toBe(true);
|
||||
}
|
||||
expect(jobs.chat.environment).toBe("runner-e2e-private-paid");
|
||||
expect(jobs.select.environment).toBe("runner-e2e-testops-reporting");
|
||||
expect(jobs.upload.environment).toBe("runner-e2e-testops-reporting");
|
||||
});
|
||||
it("requires token-isolated selection before the fixed AWS run and keeps bootstrap outside all model jobs", () => {
|
||||
expect(workflow.on).toHaveProperty("workflow_dispatch"); expect(Object.keys(workflow.on)).toEqual(["workflow_dispatch"]);
|
||||
expect(Object.keys(workflow.on.workflow_dispatch.inputs)).toEqual(["operation", "ALLURE_JOB_RUN_ID", "ALLURE_USERNAME"]);
|
||||
expect(workflow.on.workflow_dispatch.inputs.operation.default).toBe("evaluate");
|
||||
expect(workflow.jobs.chat.needs).toContain("select"); expect(workflow.jobs.build.needs).toContain("select");
|
||||
expect(workflow.jobs.chat.if).toBe("needs.authorize.outputs.operation == 'evaluate'");
|
||||
expect(workflow.jobs.build.if).toBe("needs.authorize.outputs.operation == 'evaluate'");
|
||||
const paid = workflow.jobs.chat.steps.find((s: any) => s.env?.OPENROUTER_API_KEY);
|
||||
expect(paid.run).toContain("--id chat-features.runner-opencode-deepseek-1017.local.create-backlog --max-parallel 1 --max-automatic-retries 0");
|
||||
expect(workflow.jobs.chat.steps.findIndex((s: any) => s.name === "Verify immutable outputs and token-free TestOps selection")).toBeLessThan(workflow.jobs.chat.steps.indexOf(paid));
|
||||
expect(workflow.jobs.authorize.outputs.runner).toBe("runs-on/fleet=paperclip-public-pr-x64/env=public-ci");
|
||||
expect(workflow.jobs.chat["runs-on"]).toBe("${{ needs.authorize.outputs.runner }}");
|
||||
});
|
||||
it("uploads only the trusted safe projection after successful strict private report retention", () => {
|
||||
const steps = workflow.jobs.report.steps;
|
||||
expect(steps.find((s: any) => s.id === "export").if).toBe("always() && steps.private_report.outcome == 'success'");
|
||||
expect(steps.find((s: any) => s.with?.name?.startsWith("runner-e2e-testops-")).with.path).toBe("runner-e2e-testops/");
|
||||
expect(JSON.stringify(workflow)).not.toContain("MANUAL_PRIVATE_APPROVALS_JSON");
|
||||
expect(Object.values(workflow.jobs).flatMap((j: any) => j.steps).filter((s: any) => s.uses?.startsWith("actions/upload-artifact")).some((s: any) => s.with.path === "tests/runner-e2e/results/")).toBe(false);
|
||||
});
|
||||
|
||||
it("initializes the actual TestOps association and never fabricates setup results", () => {
|
||||
const steps = workflow.jobs.select.steps;
|
||||
const installer = steps.find((s: any) => s.uses?.startsWith("allure-framework/setup-allurectl@"));
|
||||
expect(installer.uses).toBe("allure-framework/setup-allurectl@2d89ee0b7ba2f59dbe9849b6df5763cab78e14db");
|
||||
expect(installer.with["allurectl-version"]).toBe("2.23.1");
|
||||
const bootstrap = steps.find((s: any) => s.run?.includes("testops-chat-reporter.mjs bootstrap"));
|
||||
expect(bootstrap.run).toContain("mkdir -m 700");
|
||||
expect(bootstrap.run).toContain("allurectl watch -- node");
|
||||
expect(bootstrap.env.ALLURE_RESULTS).toBe("${{ runner.temp }}/chat-testops-bootstrap-results");
|
||||
const select = steps.find((s: any) => s.id === "selection");
|
||||
expect(select.run).toContain("allurectl watch -- node");
|
||||
for (const name of ["CHAT_TESTOPS_CASE_ID", "CHAT_TESTOPS_PLAN_ID", "CHAT_TESTOPS_JOB_ID"])
|
||||
expect(select.env[name]).toBe(`\${{ vars.${name} }}`);
|
||||
const metadata = steps.find((s: any) => s.with?.name?.startsWith("chat-testops-setup-"));
|
||||
expect(metadata.if).toBe("always() && needs.authorize.outputs.operation == 'bootstrap'");
|
||||
expect(metadata.with.path).not.toContain("results");
|
||||
});
|
||||
it("authenticates the same-run selection again before converting and uploading", () => {
|
||||
const steps = workflow.jobs.upload.steps;
|
||||
const receipt = steps.find((s: any) => s.with?.path === "chat-testops-selection");
|
||||
expect(receipt.with["artifact-ids"]).toBe("${{ needs.select.outputs.artifact_id }}");
|
||||
const upload = steps.find((s: any) => s.run?.includes("testops-chat-reporter.mjs upload"));
|
||||
expect(upload.run).toContain("allurectl watch -- node");
|
||||
expect(upload.env.EXPECTED_SELECTION_SHA256).toBe("${{ needs.select.outputs.sha256 }}");
|
||||
expect(steps.indexOf(receipt)).toBeLessThan(steps.indexOf(upload));
|
||||
expect(JSON.stringify(workflow.jobs.select)).not.toContain("OPENROUTER_API_KEY");
|
||||
expect(JSON.stringify(workflow.jobs.upload)).not.toContain("OPENROUTER_API_KEY");
|
||||
});
|
||||
|
||||
it("keeps the one plan launch outside Allure watch and every AWS/model job", () => {
|
||||
const steps = workflow.jobs.select.steps;
|
||||
const launch = steps.find((s: any) => s.run?.endsWith("testops-chat-reporter.mjs launch-plan"));
|
||||
expect(launch.if).toBe("needs.authorize.outputs.operation == 'launch-plan'");
|
||||
expect(launch.run).not.toContain("watch");
|
||||
expect(launch.env.CHAT_TESTOPS_OPERATION_PARAMETER_ID).toBe("${{ vars.CHAT_TESTOPS_OPERATION_PARAMETER_ID }}");
|
||||
expect(launch.env.CHAT_TESTOPS_INTEGRATION_ID).toBe("${{ vars.CHAT_TESTOPS_INTEGRATION_ID }}");
|
||||
expect(launch.env.CHAT_TESTOPS_LAUNCH_NONCE).toBe("${{ vars.CHAT_TESTOPS_LAUNCH_NONCE }}");
|
||||
expect(launch.env.CHAT_TESTOPS_LAUNCH_RUN_NUMBER).toBe("${{ vars.CHAT_TESTOPS_LAUNCH_RUN_NUMBER }}");
|
||||
expect(launch.env.TESTOPS_TRIGGERING_ACTOR_ID).toBe("${{ needs.authorize.outputs.triggering_actor_id }}");
|
||||
expect(launch.env.TESTOPS_ALLOWED_ACTOR_IDS).toBe("${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}");
|
||||
expect(launch.env.GH_TOKEN).toBe("${{ github.token }}");
|
||||
expect(launch.run).not.toContain("sleep");
|
||||
expect(steps.find((s: any) => s.uses?.startsWith("allure-framework/setup-allurectl@")).if).toBe("needs.authorize.outputs.operation != 'launch-plan'");
|
||||
for (const name of ["build", "chat"]) expect(workflow.jobs[name].if).toBe("needs.authorize.outputs.operation == 'evaluate'");
|
||||
expect(workflow.concurrency["cancel-in-progress"]).toBe(false);
|
||||
});
|
||||
|
||||
it("accepts the documented TestOps username metadata without treating it as an actor", () => {
|
||||
const f = context(); f.event.inputs.ALLURE_USERNAME = "service-metadata";
|
||||
expect(validateChatTestOpsDispatch(f.env, f.event, 123).operation).toBe("evaluate");
|
||||
expect(() => validateChatTestOpsDispatch(f.env, f.event, 456)).toThrow();
|
||||
expect(JSON.stringify(workflow.jobs)).not.toContain("inputs.ALLURE_USERNAME");
|
||||
});
|
||||
|
||||
it("includes every native Node helper control in the normal unit owner", () => {
|
||||
const launcher = readFileSync(new URL("./run-unit.mjs", import.meta.url), "utf8");
|
||||
const config = readFileSync(new URL("./vitest.config.ts", import.meta.url), "utf8");
|
||||
for (const file of ["chat-feature-testops-api.test.mjs", "chat-feature-testops-cli.test.mjs", "chat-feature-testops-results.test.mjs", "testops-chat-reporter.test.mjs"]) {
|
||||
expect(launcher).toContain(`"tests/runner-e2e/${file}"`);
|
||||
expect(config).toContain(`"${file}"`);
|
||||
}
|
||||
});
|
||||
|
||||
it("verifies the actual server outcome after watch without masking an original failed upload", () => {
|
||||
const steps = workflow.jobs.upload.steps;
|
||||
const watch = steps.find((s: any) => s.id === "allure_upload");
|
||||
const verify = steps.find((s: any) => s.id === "verify_upload");
|
||||
expect(watch.run).toContain("allurectl watch -- node");
|
||||
expect(watch["continue-on-error"]).toBeUndefined();
|
||||
expect(verify.run).toBe("node tests/runner-e2e/testops-chat-reporter.mjs verify-upload runner-e2e-testops");
|
||||
expect(verify.if).toBe("always() && !cancelled() && (steps.allure_upload.outcome == 'success' || steps.allure_upload.outcome == 'failure')");
|
||||
expect(verify.env).toEqual(watch.env);
|
||||
expect(steps.indexOf(verify)).toBeGreaterThan(steps.indexOf(watch));
|
||||
const proof = steps.find((s: any) => s.with?.name?.startsWith("chat-testops-verification-"));
|
||||
expect(proof.with.path.split("\n").filter(Boolean)).toHaveLength(2);
|
||||
expect(proof.with.path).not.toContain("allure-results");
|
||||
});
|
||||
|
||||
it("carries only the existing source-reviewed lock through build and worker install", () => {
|
||||
const build = workflow.jobs.build.steps;
|
||||
const install = build.find((s: any) => s.run?.includes("pnpm install --frozen-lockfile"));
|
||||
expect(install.run).toContain("node tests/runner-e2e/testops-chat-lock.ts > chat-testops-lock.json");
|
||||
expect(install.run).not.toContain("--lockfile-only");
|
||||
const archive = build.find((s: any) => s.run?.includes("tar -czf chat-testops-build"));
|
||||
expect(archive.run).toContain("node tests/runner-e2e/testops-chat-lock.ts > chat-testops-lock.json");
|
||||
expect(archive.run).toContain("chat-testops-lock.json pnpm-lock.yaml");
|
||||
const worker = workflow.jobs.chat.steps.find((s: any) => s.name === "Verify immutable outputs and token-free TestOps selection");
|
||||
expect(worker.run).toContain("node tests/runner-e2e/testops-chat-lock.ts > chat-testops-worker-lock.json");
|
||||
expect(worker.run).toContain("cmp chat-testops-lock.json chat-testops-worker-lock.json");
|
||||
expect(worker.run.indexOf("cmp chat-testops-lock.json")).toBeLessThan(worker.run.indexOf("pnpm install --frozen-lockfile"));
|
||||
});
|
||||
@@ -6,7 +6,7 @@ export default defineConfig({
|
||||
name: "runner-e2e-support",
|
||||
include: ["**/*.test.ts", "**/*.test.mjs"],
|
||||
// These use node:test and run separately in test:e2e:runner:unit.
|
||||
exclude: [...configDefaults.exclude, "native-completion-{checks,git-source,source-contract}.test.mjs", "artifact-budget.test.mjs"],
|
||||
exclude: [...configDefaults.exclude, "native-completion-{checks,git-source,source-contract}.test.mjs", "artifact-budget.test.mjs", "chat-feature-testops-api.test.mjs", "chat-feature-testops-cli.test.mjs", "chat-feature-testops-results.test.mjs", "testops-chat-reporter.test.mjs"],
|
||||
testTimeout: 30_000,
|
||||
},
|
||||
});
|
||||
@@ -1,6 +1,7 @@
|
||||
// @vitest-environment jsdom
|
||||
|
||||
import type { ComponentProps, ReactNode } from "react";
|
||||
import { act as reactAct } from "react";
|
||||
import { flushSync } from "react-dom";
|
||||
import { createRoot } from "react-dom/client";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
@@ -346,55 +347,49 @@ describe("ProjectWorkspaceSummaryCard", () => {
|
||||
cwd: "/Users/dotta/paperclip/.worktrees/PAP-1552-workspace-polish",
|
||||
});
|
||||
|
||||
await act(async () => {
|
||||
root.render(withQueryClient(
|
||||
<ProjectWorkspaceSummaryCard
|
||||
projectRef="paperclip-app"
|
||||
summary={summary}
|
||||
runtimeActionKey={null}
|
||||
runtimeActionPending={false}
|
||||
onRuntimeAction={() => {}}
|
||||
onCloseWorkspace={() => {}}
|
||||
/>,
|
||||
));
|
||||
});
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
await reactAct(async () => {
|
||||
root.render(withQueryClient(
|
||||
<ProjectWorkspaceSummaryCard
|
||||
projectRef="paperclip-app"
|
||||
summary={summary}
|
||||
runtimeActionKey={null}
|
||||
runtimeActionPending={false}
|
||||
onRuntimeAction={() => {}}
|
||||
onCloseWorkspace={() => {}}
|
||||
/>,
|
||||
));
|
||||
});
|
||||
|
||||
const branchTextButton = Array.from(container.querySelectorAll("button"))
|
||||
.find((button) => button.textContent === summary.branchName);
|
||||
const pathTextButton = container.querySelector(`button[title="${summary.cwd}"]`);
|
||||
const branchIconButton = container.querySelector('button[aria-label="Copy branch"]');
|
||||
const pathIconButton = container.querySelector('button[aria-label="Copy path"]');
|
||||
const branchTextButton = Array.from(container.querySelectorAll("button"))
|
||||
.find((button) => button.textContent === summary.branchName);
|
||||
const pathTextButton = container.querySelector(`button[title="${summary.cwd}"]`);
|
||||
const branchIconButton = container.querySelector('button[aria-label="Copy branch"]');
|
||||
const pathIconButton = container.querySelector('button[aria-label="Copy path"]');
|
||||
|
||||
expect(branchTextButton).not.toBeNull();
|
||||
expect(pathTextButton).not.toBeNull();
|
||||
expect(branchIconButton).not.toBeNull();
|
||||
expect(pathIconButton).not.toBeNull();
|
||||
const copyWithFeedback = async (button: Element | null | undefined, text: string) => {
|
||||
expect(button).toBeTruthy();
|
||||
await reactAct(async () => {
|
||||
button!.dispatchEvent(new MouseEvent("click", { bubbles: true }));
|
||||
});
|
||||
expect(writeClipboard).toHaveBeenLastCalledWith(text);
|
||||
expect(button!.nextElementSibling?.className).toContain("opacity-100");
|
||||
await reactAct(async () => { await vi.advanceTimersByTimeAsync(1499); });
|
||||
expect(button!.nextElementSibling?.className).toContain("opacity-100");
|
||||
await reactAct(async () => { await vi.advanceTimersByTimeAsync(1); });
|
||||
expect(button!.nextElementSibling?.className).toContain("opacity-0");
|
||||
};
|
||||
|
||||
await act(async () => {
|
||||
branchTextButton!.click();
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
});
|
||||
expect(writeClipboard).toHaveBeenLastCalledWith(summary.branchName);
|
||||
expect(branchTextButton?.nextElementSibling?.className).toContain("opacity-100");
|
||||
|
||||
await act(async () => {
|
||||
pathTextButton!.dispatchEvent(new MouseEvent("click", { bubbles: true }));
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
});
|
||||
expect(writeClipboard).toHaveBeenLastCalledWith(summary.cwd);
|
||||
expect(pathTextButton?.nextElementSibling?.className).toContain("opacity-100");
|
||||
|
||||
await act(async () => {
|
||||
branchIconButton!.dispatchEvent(new MouseEvent("click", { bubbles: true }));
|
||||
pathIconButton!.dispatchEvent(new MouseEvent("click", { bubbles: true }));
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
});
|
||||
expect(writeClipboard).toHaveBeenCalledWith(summary.branchName);
|
||||
expect(writeClipboard).toHaveBeenCalledWith(summary.cwd);
|
||||
|
||||
act(() => {
|
||||
root.unmount();
|
||||
});
|
||||
await copyWithFeedback(branchTextButton, summary.branchName!);
|
||||
await copyWithFeedback(pathTextButton, summary.cwd!);
|
||||
await copyWithFeedback(branchIconButton, summary.branchName!);
|
||||
await copyWithFeedback(pathIconButton, summary.cwd!);
|
||||
} finally {
|
||||
try {
|
||||
await reactAct(async () => { root.unmount(); });
|
||||
} finally { vi.useRealTimers(); }
|
||||
}
|
||||
});
|
||||
it("colors live service urls green", () => {
|
||||
const root = createRoot(container);
|
||||
|
||||
Reference in new issue
Block a user