test(evals): add a fixed TestOps AWS chat feature route (#18322)

## Thinking Path

> - Paperclip lets people manage AI agents and inspect their saved work.
> - Chat feature evaluations test a real request against saved tasks,
audit records, links and reload behavior.
> - The current suite has Codex and OpenCode cells but no dedicated
runnable DeepSeek 1017 backlog plan in TestOps.
> - TestOps must select the real case before the workflow can access a
model key.
> - This PR adds one fixed, default-branch AWS route for native OpenCode
and DeepSeek 1017.
> - The route keeps original evidence private and sends only a verified
result and approved screenshots to TestOps.

## Linked Issues or Issue Description

**Subsystem affected**

Product E2E evaluation definitions and GitHub Actions delivery.

**Problem or motivation**

A dedicated TestOps plan cannot run the existing backlog chat journey
with native OpenCode. A result must prove saved work and cleanup rather
than report that a model call completed.

**Proposed solution**

Add one OpenCode backlog cell and one fixed public workflow. TestOps
selection validation runs in a token-isolated job before builds or
inference. The workflow retains private originals and reports the bound
outcome to the selected TestOps case.

**Alternatives considered**

A private-repository wrapper would need a new cross-repository GitHub
write credential. The new route uses the existing TestOps GitHub
integration instead. Existing manual candidate approval paths remain
separate.

**Roadmap alignment**

This extends the existing feature evaluation infrastructure. It changes
no production feature, global runtime default or model default.

Current main #18319 supplies the v2 grader, delegation queue accounting
and final screenshot capture; these improvements are preserved. Related
product work: #16814 and #17126 concern backlog handling. This PR
changes neither product implementation and does not claim those changes
are required.

## What Changed

- Add exactly one native OpenCode backlog cell with DeepSeek 1017 under
the separate `runner-opencode-deepseek-1017` profile. Preserve all four
existing Codex and production OpenCode cells and their models.
- Add a default-branch-only workflow with fixed evaluation, zero-result
registration and one plan-launch modes. Require GitHub actor checks and
authenticated singleton selection before inference.
- Add a separate shipped-main evidence authority. Reuse the existing
strict private storage, scanners, inventories and scoped AWS sessions.
- Bind sanitized results, generated JUnit and producer-approved
screenshots to the exact source, run, attempt and case. Keep failed
grades and cleanup failures failed.
- Add TestOps API, CLI and result adapters with pure controls and normal
unit-test wiring. Keep the reporting token out of the evaluated process.
- Verify the current server case result and attachment metadata after
upload, without masking an earlier failed upload or evaluation.
- Apply only the existing reviewed lock correction, verify unchanged
tracked dependency inputs and bind the lock receipt across the
compiled-output handoff before frozen installation.
- Bind the initial administrator launch to one approved workflow run
number and live GitHub source/actor metadata before TestOps
authentication. Use bounded REST case reconciliation and only the
reviewed existing GitHub integration.
- Make two existing CI tests observe their intended state
deterministically: a fake clock for copy-feedback lifetime and a
completed-pass barrier for retry-cancellation batches. Production
behavior and the original assertions remain unchanged.

The two test-only fixes are intentionally included because their
observation boundaries blocked this delivery’s required normal CI. They
preserve every original assertion and change no production behavior. The
original failed runs remain recorded, and the exact causes of those
observations remain unknown.

## Verification

- 157 focused profile, catalog, oracle, filesystem admission, export,
private-path and workflow controls passed on the current-main
composition. The existing stock OpenCode models and the new exact
DeepSeek 1017 selection are checked separately.
- 44 API, CLI, filesystem and post-upload readback controls passed,
including foreign selection, changed hashes, unsafe files, failed
outcomes, expired readback and no-resend behavior.
- The new exact private session and trace layout controls produced four
genuine parent failures and pass with the selected-path extension. The
frozen 244-case candidate approval registry remains unchanged.
- The lock guard has seven filesystem controls plus seven existing
canonical-owner controls. The unrestricted old workflow failed its
actual policy calibration; the corrected frozen-install route passed.
- Eval-owning TypeScript checks, workflow lint and Git whitespace checks
passed. The separate UI test correction was checked with the focused
jsdom unit suite. The server test barrier has pure wrapper and real
scheduler controls; its DB-backed route was not run locally. No
additional local typecheck/build was run for these test-only changes.
- Full native builds, provider execution, app/database behavior and
TestOps registration are reserved for GitHub Actions/AWS. They have not
run locally.
- The initial normal CI run failed the new-workflow policy allowlist and
a heartbeat scheduling assertion. The policy fix has an actual parent
red/current green control and rejects five unsafe route variants.
Heartbeat source is unchanged and its failure cause remains unknown. The
original run is retained.
- The launch-fence actual caller fails against the published parent and
passes the five own-run contradiction controls before authentication.
- The prior `a5aa359d` normal CI completed successfully. The complete
`f9e1f289` run passed all leaf jobs except an existing copy-feedback UI
assertion, plus its dependent verification gate. The four owning UI
files were unchanged from the earlier passing head and actual tested
merge; the exact scheduling cause remains unknown.
- The correction changes only that test to actual React `act` and a
scoped fake clock. All eight component tests pass; early-hide and
never-clear source variants each fail the intended lifetime assertion.
Production clipboard/card behavior is unchanged.
- The `2f27f3fc` run passed the corrected UI shard and every leaf job
except the existing retry-cancellation intermediate-count assertion,
plus its dependent verification gate. The seven relevant owners match
the earlier passing run and historical source comparison; the exact CI
scheduling cause remains unknown.
- The test-only barrier observes a persisted completed worker pass
before asserting the unchanged 20-item cap and two-company fairness,
then releases it for the original 29-item drain and 180-second idle
check. Two extracted-wrapper controls and three actual scheduler
controls pass. The real DB-backed test awaits AWS qualification.
- On `836e80ac`, both repaired test shards passed in AWS. The original
attempt retained a separate E2E cleanup HTTP 500 from a PostgreSQL lock
timeout and one runner shutdown. No production repair or precise
contention/shutdown cause is claimed.
- Exactly one targeted retry of E2E shard 3 and its aggregate gate
passed in attempt 2; exactly one recovery of the interrupted server
shard and its verification gate passed in attempt 3. Other passed jobs
were carried forward. The final normal run is successful. No
whole-workflow retry or evaluation/model retry was launched.
- Both literal required checks (`ci / verify` and `ci / e2e`) are
successful on the exact final source `836e80ac`. Greptile is 5/5 and no
review threads remain open. Actual TestOps registration and the
OpenCode/DeepSeek feature result remain unqualified until their separate
executions.

## Risks

- Actual zero-result registration, tenant API shapes and
public-repository dispatch access must be checked before the first
evaluation. Missing or ambiguous metadata stops the route.
- The real OpenCode/DeepSeek backlog result and Allure outcome remain
unqualified until that run. No passing result is seeded.
- One human query, one accounted work run, 120-second turn, 420-second
attempt, 300-second settled-state bound and zero retries remain
unchanged. Setup usage is outside work accounting and stays unknown.
- The reporting token was configured through the separately reviewed
protected credential bootstrap. It is unavailable to AWS evaluation
jobs.
- Metadata reconciliation is capped at ten pages, 1,000 cases and four
parallel detail reads. Missing or inconsistent metadata stops before
inference. The initial administrator launch counter must be created once
after registration; it does not restrict normal TestOps Run actions.
- Existing manual workflows, historical evaluation grades and candidate
approval records remain unchanged. The original CI failures are
preserved; the test corrections do not establish retrospective causes.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning and tool use. The
runtime did not expose its context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-30 15:08:16 -05:00
1 parent a4308b9791
commit 9e1735ec17
34 files changed
+2802 -74

No files matched your search

+448
View File
@@ -0,0 +1,448 @@
name: Chat feature TestOps
run-name: Chat feature TestOps / ${{ inputs.operation || 'evaluate' }} / ${{ inputs.ALLURE_JOB_RUN_ID || github.run_id }}
on:
workflow_dispatch:
inputs:
operation:
type: choice
options: [evaluate, bootstrap, launch-plan]
default: evaluate
description: Fixed evaluation, zero-result registration, or one TestOps plan launch
ALLURE_JOB_RUN_ID:
type: string
required: false
description: TestOps supplies its job-run identity; no test/source selector is accepted
ALLURE_USERNAME:
type: string
required: false
description: Reserved TestOps service metadata; never used as authorization
permissions:
contents: read
concurrency:
group: chat-feature-testops
cancel-in-progress: false
jobs:
authorize:
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
sha: ${{ steps.guard.outputs.sha }}
operation: ${{ steps.guard.outputs.operation }}
triggering_actor_id: ${{ steps.guard.outputs.triggering_actor_id }}
runner: runs-on/fleet=paperclip-public-pr-x64/env=public-ci
steps:
- &checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ github.sha }}
persist-credentials: false
- &node
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- &guard
name: Verify fixed default-source dispatch and both actor identities
id: guard
env:
GH_TOKEN: ${{ github.token }}
TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
NODE_OPTIONS: ''
NODE_PATH: ''
run: node tests/runner-e2e/testops-chat-dispatch.ts
select:
name: Validate TestOps selection or register zero-result setup
needs: authorize
runs-on: ubuntu-latest
timeout-minutes: 10
environment: runner-e2e-testops-reporting
permissions:
contents: read
actions: read
outputs:
artifact_id: ${{ steps.selection_upload.outputs.artifact-id }}
sha256: ${{ steps.selection.outputs.sha256 }}
steps:
- *checkout
- *node
- *guard
- &allure
if: needs.authorize.outputs.operation != 'launch-plan'
uses: allure-framework/setup-allurectl@2d89ee0b7ba2f59dbe9849b6df5763cab78e14db # v1
with:
allurectl-version: 2.23.1
allure-endpoint: https://paperclip.testops.cloud
allure-project-id: '1'
allure-token: ${{ secrets.ALLURE_TOKEN }}
- name: Register the public job without an evaluation or a synthetic result
if: needs.authorize.outputs.operation == 'bootstrap'
env:
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
ALLURE_ENDPOINT: https://paperclip.testops.cloud
ALLURE_PROJECT_ID: '1'
CHAT_TESTOPS_GITHUB_WORKFLOW_ID: ${{ vars.CHAT_TESTOPS_GITHUB_WORKFLOW_ID }}
ALLURE_RESULTS: ${{ runner.temp }}/chat-testops-bootstrap-results
run: |
mkdir -m 700 "$ALLURE_RESULTS"
allurectl watch -- node tests/runner-e2e/testops-chat-reporter.mjs bootstrap
- name: Validate the one real TestOps selection before any build or provider access
id: selection
if: needs.authorize.outputs.operation == 'evaluate'
env:
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
ALLURE_ENDPOINT: https://paperclip.testops.cloud
ALLURE_PROJECT_ID: '1'
ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
CHAT_TESTOPS_CASE_ID: ${{ vars.CHAT_TESTOPS_CASE_ID }}
CHAT_TESTOPS_PLAN_ID: ${{ vars.CHAT_TESTOPS_PLAN_ID }}
CHAT_TESTOPS_JOB_ID: ${{ vars.CHAT_TESTOPS_JOB_ID }}
CHAT_TESTOPS_GITHUB_WORKFLOW_ID: ${{ vars.CHAT_TESTOPS_GITHUB_WORKFLOW_ID }}
ALLURE_RESULTS: ${{ runner.temp }}/chat-testops-selection-results
run: |
mkdir -m 700 "$ALLURE_RESULTS"
allurectl watch -- node tests/runner-e2e/testops-chat-reporter.mjs selection
- name: Start exactly the validated TestOps plan and exit without waiting for its child
if: needs.authorize.outputs.operation == 'launch-plan'
env:
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
ALLURE_ENDPOINT: https://paperclip.testops.cloud
ALLURE_PROJECT_ID: '1'
CHAT_TESTOPS_CASE_ID: ${{ vars.CHAT_TESTOPS_CASE_ID }}
CHAT_TESTOPS_PLAN_ID: ${{ vars.CHAT_TESTOPS_PLAN_ID }}
CHAT_TESTOPS_JOB_ID: ${{ vars.CHAT_TESTOPS_JOB_ID }}
CHAT_TESTOPS_GITHUB_WORKFLOW_ID: ${{ vars.CHAT_TESTOPS_GITHUB_WORKFLOW_ID }}
CHAT_TESTOPS_OPERATION_PARAMETER_ID: ${{ vars.CHAT_TESTOPS_OPERATION_PARAMETER_ID }}
CHAT_TESTOPS_INTEGRATION_ID: ${{ vars.CHAT_TESTOPS_INTEGRATION_ID }}
CHAT_TESTOPS_LAUNCH_NONCE: ${{ vars.CHAT_TESTOPS_LAUNCH_NONCE }}
CHAT_TESTOPS_LAUNCH_RUN_NUMBER: ${{ vars.CHAT_TESTOPS_LAUNCH_RUN_NUMBER }}
TESTOPS_TRIGGERING_ACTOR_ID: ${{ needs.authorize.outputs.triggering_actor_id }}
TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
GH_TOKEN: ${{ github.token }}
run: node tests/runner-e2e/testops-chat-reporter.mjs launch-plan
- name: Retain only closed metadata from zero-result setup
if: always() && needs.authorize.outputs.operation == 'bootstrap'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: chat-testops-setup-${{ github.run_id }}-1
path: |
${{ runner.temp }}/chat-testops-bootstrap-state/metadata.json
${{ runner.temp }}/chat-testops-bootstrap-state/once.json
${{ runner.temp }}/chat-testops-bootstrap-state/plan.json
${{ runner.temp }}/chat-testops-bootstrap-state/job.json
${{ runner.temp }}/chat-testops-bootstrap-state/failure.json
retention-days: 30
if-no-files-found: error
- name: Retain the one plan-launch receipt and stop state
if: always() && needs.authorize.outputs.operation == 'launch-plan'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: chat-testops-plan-launch-${{ github.run_id }}-1
path: |
${{ runner.temp }}/chat-testops-launch-plan-state/once.json
${{ runner.temp }}/chat-testops-launch-plan-state/launch.json
${{ runner.temp }}/chat-testops-launch-plan-state/failure.json
retention-days: 30
if-no-files-found: error
- name: Retain only the token-free exact selection receipt
id: selection_upload
if: steps.selection.outcome == 'success'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: chat-testops-selection-${{ github.run_id }}-1
path: testops-selection.json
retention-days: 30
if-no-files-found: error
build:
name: Build the immutable shipped native candidate without model credentials
needs: [authorize, select]
if: needs.authorize.outputs.operation == 'evaluate'
runs-on: ${{ needs.authorize.outputs.runner }}
timeout-minutes: 45
outputs:
artifact_id: ${{ steps.bundle.outputs.artifact-id }}
steps:
- *checkout
- *node
- *guard
- &pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: 'false'
NPM_CONFIG_FUND: 'false'
NPM_CONFIG_UPDATE_NOTIFIER: 'false'
with:
version: 9.15.4
- name: Install only the source-reviewed graph without lifecycle scripts
run: |
node tests/runner-e2e/testops-chat-lock.ts > chat-testops-lock.json
pnpm install --frozen-lockfile --ignore-scripts
- name: Build the existing TypeScript and native runner owners
run: |
pnpm --filter @paperclipai/paperclip-runner build:typescript
pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
- name: Package source-bound outputs and resolved lock
run: |
set -euo pipefail
test "$(git rev-parse HEAD)" = "$GITHUB_SHA"
node tests/runner-e2e/testops-chat-lock.ts > chat-testops-lock.json
tar -czf chat-testops-build.tar.gz chat-testops-lock.json pnpm-lock.yaml packages/paperclip-eval-kernel/dist packages/paperclip-runner/dist packages/paperclip-runner/runner/target/debug/paperclip-runnerd packages/paperclip-runner/runner/target/debug/conformance-tracer packages/paperclip-runner/runner/target/debug/fake-harness packages/paperclip-runner/runner/target/debug/fake-codex-app-server packages/paperclip-runner/runner/target/debug/fake-acpx-sidecar
sha256sum chat-testops-build.tar.gz > chat-testops-build.tar.gz.sha256
- name: Retain credential-free immutable outputs
id: bundle
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: chat-testops-build-${{ github.sha }}-${{ github.run_id }}-1
path: |
chat-testops-build.tar.gz
chat-testops-build.tar.gz.sha256
retention-days: 1
if-no-files-found: error
chat:
name: One real native OpenCode backlog request on AWS
needs: [authorize, select, build]
if: needs.authorize.outputs.operation == 'evaluate'
runs-on: ${{ needs.authorize.outputs.runner }}
timeout-minutes: 32
environment: runner-e2e-private-paid
permissions:
contents: read
id-token: write
env: &retention
PAPERCLIP_TESTOPS_CHAT_MAIN_ADMISSION: v1
TESTOPS_ADMITTED_SHA: ${{ needs.authorize.outputs.sha }}
TESTOPS_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
TESTOPS_TRIGGERING_ACTOR_ID: ${{ needs.authorize.outputs.triggering_actor_id }}
TARGET_SHA: ${{ needs.authorize.outputs.sha }}
MANUAL_PRIVATE_IDS: '["chat-features.runner-opencode-deepseek-1017.local.create-backlog"]'
SELECTED_EXECUTION_ID: chat-features.runner-opencode-deepseek-1017.local.create-backlog
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
NODE_OPTIONS: ''
NODE_PATH: ''
AWS_EC2_METADATA_DISABLED: 'true'
steps:
- *checkout
- *node
- *guard
- *pnpm
- name: Check out the same admitted source for trusted retention
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ needs.authorize.outputs.sha }}
path: .testops-trusted
persist-credentials: false
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
artifact-ids: ${{ needs.build.outputs.artifact_id }}
path: chat-testops-build
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
artifact-ids: ${{ needs.select.outputs.artifact_id }}
path: chat-testops-selection
- name: Verify immutable outputs and token-free TestOps selection
env:
EXPECTED_SELECTION_SHA256: ${{ needs.select.outputs.sha256 }}
ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
run: |
set -euo pipefail
(cd chat-testops-build && sha256sum -c chat-testops-build.tar.gz.sha256)
tar -xzf chat-testops-build/chat-testops-build.tar.gz -C "$GITHUB_WORKSPACE"
node tests/runner-e2e/testops-chat-reporter.mjs verify-selection chat-testops-selection/testops-selection.json
node tests/runner-e2e/testops-chat-lock.ts > chat-testops-worker-lock.json
cmp chat-testops-lock.json chat-testops-worker-lock.json
cp pnpm-lock.yaml .testops-trusted/pnpm-lock.yaml
pnpm install --frozen-lockfile --ignore-scripts
(cd .testops-trusted && pnpm install --frozen-lockfile --ignore-scripts --store-dir "$RUNNER_TEMP/chat-retention-store")
- name: Materialize and qualify pinned OpenCode runtime and browser
run: |
node packages/paperclip-runner/scripts/materialize-opencode-binary.mjs
node -e 'const fs=require("node:fs");fs.chmodSync(process.execPath,fs.statSync(process.execPath).mode&~0o022);if(fs.statSync(process.execPath).mode&0o022)throw Error("Writable provider Node")'
google-chrome --version
pnpm exec playwright install ffmpeg
- name: Prepare private evidence with shipped-main authority
id: private
env:
TARGET_ROOT: ${{ github.workspace }}
run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts prepare
- name: Obtain only the exact private cell storage session
id: cell_credentials
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
with:
role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }}
aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
inline-session-policy: ${{ steps.private.outputs.session_policy }}
output-env-credentials: false
output-credentials: true
disable-retry: true
action-timeout-s: 60
- name: Verify strict private custody before the model key
env: &cell_session
PRIVATE_EVIDENCE_ROOT: ${{ steps.private.outputs.root }}
AWS_ACCESS_KEY_ID: ${{ steps.cell_credentials.outputs.aws-access-key-id }}
AWS_SECRET_ACCESS_KEY: ${{ steps.cell_credentials.outputs.aws-secret-access-key }}
AWS_SESSION_TOKEN: ${{ steps.cell_credentials.outputs.aws-session-token }}
AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts verify
- name: Execute only the fixed original request with zero retries
env:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
PAPERCLIP_RUNNER_E2E_PRIVATE_PREFLIGHT: ${{ steps.private.outputs.root }}/preflight.json
PRIVATE_EVIDENCE_ROOT: ${{ steps.private.outputs.root }}
PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-1-chat-features.runner-opencode-deepseek-1017.local.create-backlog
PAPERCLIP_E2E_ACCOUNTING_CAMPAIGN_ID: gha-${{ github.run_id }}-1
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ github.ref }}
PAPERCLIP_PLAYWRIGHT_CHANNEL: chrome
run: pnpm test:e2e:runner -- --id chat-features.runner-opencode-deepseek-1017.local.create-backlog --max-parallel 1 --max-automatic-retries 0 > "$PRIVATE_EVIDENCE_ROOT/launcher-output.log" 2>&1
- name: Retain exact original grade and private evidence, including failures
if: always() && steps.cell_credentials.outcome == 'success'
env: *cell_session
run: node .testops-trusted/cli/node_modules/tsx/dist/cli.mjs .testops-trusted/tests/runner-e2e/manual-private-transfer.ts upload-cell
report:
name: Retain private originals and export only reviewed fixture results
needs: [authorize, select, chat]
if: always() && !cancelled() && needs.authorize.outputs.operation == 'evaluate' && needs.select.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 15
environment: runner-e2e-private-history
permissions:
contents: read
actions: read
id-token: write
env: *retention
steps:
- *checkout
- *node
- *guard
- *pnpm
- name: Install trusted reporting dependencies only
run: |
node tests/runner-e2e/prepare-report-lock.mjs
pnpm install --frozen-lockfile --ignore-scripts --store-dir "$RUNNER_TEMP/chat-report-store"
- name: Prepare exact private report and cell-read policies
id: session
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts report-session
- name: Obtain private report writer
id: report_credentials
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
with:
role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }}
aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
inline-session-policy: ${{ steps.session.outputs.session_policy }}
output-env-credentials: false
output-credentials: true
disable-retry: true
action-timeout-s: 60
- name: Obtain exact private cell reader
id: read_credentials
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
with:
role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }}
aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
inline-session-policy: ${{ steps.session.outputs.read_policy_0 }}
output-env-credentials: false
output-credentials: true
disable-retry: true
action-timeout-s: 60
- name: Download the exact private original and verify all inventories
env:
PRIVATE_REPORT_ROOT: ${{ github.workspace }}
SELECTED_CELL_INDEX: '0'
AWS_ACCESS_KEY_ID: ${{ steps.read_credentials.outputs.aws-access-key-id }}
AWS_SECRET_ACCESS_KEY: ${{ steps.read_credentials.outputs.aws-secret-access-key }}
AWS_SESSION_TOKEN: ${{ steps.read_credentials.outputs.aws-session-token }}
AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
run: |
node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts download-cell
node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts verify-downloads
- name: Build the existing normalized campaign without exposing original logs
env:
PAPERCLIP_RUNNER_E2E_REPORT_ROOT: ${{ github.workspace }}/selected-runner-e2e
PAPERCLIP_RUNNER_E2E_REPORT_OUT: ${{ github.workspace }}/runner-e2e-merged-report/normalized
PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: '["chat-features.runner-opencode-deepseek-1017.local.create-backlog"]'
PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-1
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ github.ref }}
run: |
mkdir -p runner-e2e-merged-report
pnpm test:e2e:runner:report > runner-e2e-merged-report/report.log 2>&1
- name: Retain the complete merged report privately even when its grade failed
id: private_report
if: always() && steps.report_credentials.outcome == 'success'
env:
PRIVATE_REPORT_ROOT: ${{ github.workspace }}
PRIVATE_REPORT_SOURCE: ${{ github.workspace }}/runner-e2e-merged-report
AWS_ACCESS_KEY_ID: ${{ steps.report_credentials.outputs.aws-access-key-id }}
AWS_SECRET_ACCESS_KEY: ${{ steps.report_credentials.outputs.aws-secret-access-key }}
AWS_SESSION_TOKEN: ${{ steps.report_credentials.outputs.aws-session-token }}
AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/manual-private-transfer.ts upload-report
- name: Export only the bound normalized result, JUnit and marked PNGs
id: export
if: always() && steps.private_report.outcome == 'success'
env:
TESTOPS_CHAT_TARGET_SHA: ${{ needs.authorize.outputs.sha }}
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/testops-chat-export.ts
- name: Upload only the sanitized TestOps projection
if: always() && steps.export.outcome == 'success'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: runner-e2e-testops-${{ github.run_id }}-1
path: runner-e2e-testops/
retention-days: 30
if-no-files-found: error
upload:
name: Report the original outcome to its exact TestOps case
needs: [authorize, select, chat, report]
if: always() && !cancelled() && needs.authorize.outputs.operation == 'evaluate' && needs.select.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 10
environment: runner-e2e-testops-reporting
permissions:
contents: read
actions: read
steps:
- *checkout
- *node
- *guard
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: runner-e2e-testops-${{ github.run_id }}-1
path: runner-e2e-testops
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
artifact-ids: ${{ needs.select.outputs.artifact_id }}
path: chat-testops-selection
- *allure
- name: Verify and upload only the bound safe case outcome
id: allure_upload
env: &testops_reporter
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
ALLURE_ENDPOINT: https://paperclip.testops.cloud
ALLURE_PROJECT_ID: '1'
ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
EXPECTED_SELECTION_SHA256: ${{ needs.select.outputs.sha256 }}
ALLURE_RESULTS: ${{ runner.temp }}/chat-testops-allure-results
run: allurectl watch -- node tests/runner-e2e/testops-chat-reporter.mjs upload runner-e2e-testops
- name: Verify the actual TestOps result after upload processing
id: verify_upload
if: always() && !cancelled() && (steps.allure_upload.outcome == 'success' || steps.allure_upload.outcome == 'failure')
env: *testops_reporter
run: node tests/runner-e2e/testops-chat-reporter.mjs verify-upload runner-e2e-testops
- name: Retain only closed actual TestOps verification metadata
if: always() && !cancelled() && (steps.verify_upload.outcome == 'success' || steps.verify_upload.outcome == 'failure')
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: chat-testops-verification-${{ github.run_id }}-1
path: |
${{ runner.temp }}/chat-testops-verify-upload-state/verification.json
${{ runner.temp }}/chat-testops-verify-upload-state/failure.json
retention-days: 30
if-no-files-found: error
+4 -1
View File
@@ -43,7 +43,10 @@ two natural-language task scenarios into local native Codex subscription and
OpenCode Product E2E cells. They
verify saving an unstarted task and delegating a saved announcement back to chat,
using ordinary production instructions, real browser messages and independent
state/provenance checks. The broader YAML catalog remains planning inventory.
state/provenance checks. A separately selected native OpenCode/DeepSeek1017
backlog cell measures TestOps integration acceptance with the same oracle and
original bounds. Its source registration is not an AWS or TestOps pass. The
broader YAML catalog remains planning inventory.
The explicit-only [Daytona restart continuity journey](../tests/runner-e2e/README.md)
extends the existing native restart oracle to a 125-second controller outage,
@@ -0,0 +1,89 @@
# Chat feature TestOps smoke — 2026-10-30
## Finish line
Create one dedicated runnable TestOps plan and obtain one genuine passing
`tasks.create-backlog` result through TestOps → GitHub Actions/AWS → native
OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`. The result must prove
the saved task, assignment, backlog-at-creation state, no worker execution,
agent attribution, rendered link and reload persistence. Confirm cleanup,
safe screenshot/report access and the final TestOps result link.
This is a new integration-acceptance measurement. It preserves the original
244-scenario campaign, attempts, grades and existing Codex measurements.
Delegation and the other catalog scenarios are deferred.
## Source and execution contract
- Public selector: `chat-features.runner-opencode-deepseek-1017.local.create-backlog`.
- Existing YAML query, fixture and independent backlog oracle stay unchanged.
- One human query and one accounted controller work run; 120-second turns,
420-second attempt, 300-second settled-state observation, zero retries.
- Existing 500-cent company/agent policies and unknown setup usage remain
explicit. Morgan is fixture context and must not execute this backlog task.
- Only the existing protected `OPENROUTER_API_KEY` is required. Agent
configuration uses the normal encrypted secret-reference path.
- OpenCode 1017 is selected only for this cell and remains a qualification
candidate until the real run. Other suites retain their existing models.
## Ownership and integration
Campaign owns public catalog/profile calibration and these progress notes.
Core owns the public AWS workflow/execution adapter, one coherent public PR,
and the sole actual operation. Coordinator supplies the public TestOps
selection/reporting helper and the separate private credential sealer.
The public workflow owns the dedicated plan. Source peers precede publication.
The current Cloud TestOps adapter is scoped to `qa-basics` Product journeys;
the new chat plan must have its own exact plan/case mapping. Reuse normalized
result/check/provenance and existing packaged evidence. Do not upload private
traces, provider sessions, raw logs or unreviewed media as TestOps attachments.
## Current progress
The public patch adds only the OpenCode backlog selection while preserving
the two Codex cells. A genuine parent selector control failed; all 68 focused
oracle/catalog controls and the owning type check now pass. The expected
suite-size assertion was updated for the single added cell. Independent source
review passed and the nine-file profile slice is committed as `0a996033`.
No local app, database, native SDK,
provider, model, install or build qualification was run.
The separate protected credential bootstrap seals the existing reporting token
for the source-pinned public reporting environment. Delivery is complete without
exporting plaintext. The public adapter has 53 focused authority/export/workflow
controls and 39 API/CLI/result controls passing, plus owning typechecking and
workflow lint. Unchanged controls are reused after narrow source corrections.
The independent composition source peer passed.
The public infrastructure [PR 18322](https://github.com/paperclipai/paperclip/pull/18322)
is open. Its first normal CI run rejected the new workflow in the existing paid
workflow allowlist and reported a separate heartbeat scheduling assertion; both
failures remain recorded. The policy correction preserves the older workflow
guards and rejects unsafe new-route variants. The reporting token is configured
through the reviewed encrypted bootstrap.
Review also added a one-time initial administrator launch fence. A protected
expected GitHub run number and actual own-run metadata must match before any
TestOps mutation. REST catalog reconciliation preserves exact case aliases;
job wiring uses only the reviewed existing project GitHub integration. These
provider-free helper controls pass. Actual registration IDs, tenant API behavior,
provider execution and the TestOps result remain unqualified. Dedicated plan,
AWS evaluation and final result links remain pending. Source checks alone do not
satisfy the finish line.
The final source consolidation adds actual admission-wrapper filesystem coverage,
the existing reviewed lock normalization with frozen installs and a matched
build/worker receipt, plus same-job server-result readback after Allure upload.
Focused checks cover 41 admission controls, 31 lock/owner/workflow controls and
44 API/reporter controls; unchanged profile/oracle evidence is reused. This is
source verification. The plan and actual model result are still not created.
Current-main composition preserves the four existing Codex and production-qualified
OpenCode cells introduced by #18319. The TestOps cell has its own
`runner-opencode-deepseek-1017` profile, so the existing OpenCode model remains
unchanged. It uses the shipped v2 grader, queue accounting and `final-state.png`
capture. The new profile keeps the original query, one work run, one human message,
120-second turn, 420-second attempt, 300-second observation bound and zero retries.
The existing private session and trace-asset owners admit only this additional
exact selector; the frozen 244-case candidate-approval registry remains unchanged.
Source checks do not establish a TestOps registration or model outcome.
@@ -579,6 +579,7 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
readWorkflow("runner-full-stack-e2e.yml"),
readWorkflow("e2e.yml"),
readWorkflow("runner-protocol-live-evals.yml"),
readWorkflow("chat-feature-testops.yml"),
];
for (const workflow of actionPinWorkflows) {
@@ -618,6 +619,7 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
"runner-full-stack-e2e.yml",
"runner-live-evals.yml",
"runner-protocol-live-evals.yml",
"chat-feature-testops.yml",
];
const paidWorkflowNameSet = new Set(paidWorkflowNames);
const providerSecretReference =
@@ -650,6 +652,38 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
(block) => [...block.matchAll(providerSecretReference)].length > 0,
);
assert.ok(providerJobs.length > 0, `${name} needs a provider-secret job`);
if (name === "chat-feature-testops.yml") {
assert.equal(providerJobs.length, 1);
const assertBoundary = (block) => {
assert.match(block, /^ chat:\n/);
assert.match(block, /\n needs: \[authorize, select, build\]\n/);
assert.match(block, /\n if: needs\.authorize\.outputs\.operation == 'evaluate'\n/);
assert.match(block, /\n environment: runner-e2e-private-paid\n/);
assert.match(block, /\n - \*checkout\n - \*node\n - \*guard\n/);
assert.match(block, /testops-chat-reporter\.mjs verify-selection chat-testops-selection\/testops-selection\.json/);
assert.match(block, /OPENROUTER_API_KEY: \$\{\{ secrets\.OPENROUTER_API_KEY \}\}/);
assert.match(block, /--id chat-features\.runner-opencode-deepseek-1017\.local\.create-backlog --max-parallel 1 --max-automatic-retries 0/);
assert.doesNotMatch(block, /ALLURE_TOKEN/);
assert.equal([...block.matchAll(providerSecretReference)].length, 1);
assert.ok(block.indexOf(" - *guard") < block.search(providerSecretReference));
assert.ok(block.indexOf("testops-chat-reporter.mjs verify-selection") < block.search(providerSecretReference));
};
const block = providerJobs[0];
assertBoundary(block);
assert.equal([...workflow.matchAll(providerSecretReference)].length, 1);
assert.match(workflow, /ref: \$\{\{ github\.sha \}\}/);
assert.match(workflow, /run: node tests\/runner-e2e\/testops-chat-dispatch\.ts/);
assert.match(workflow, /TESTOPS_ALLOWED_ACTOR_IDS: \$\{\{ vars\.RUNNER_E2E_ALLOWED_ACTOR_IDS \}\}/);
for (const mutant of [
block.replace("needs: [authorize, select, build]", "needs: [authorize, build]"),
block.replace("environment: runner-e2e-private-paid", "environment: unprotected"),
block.replace(" - *guard\n", ""),
block.replace("testops-chat-reporter.mjs verify-selection", "testops-chat-reporter.mjs unverified-selection"),
block.replace("OPENROUTER_API_KEY:", "ALLURE_TOKEN:"),
block.replace("--id chat-features.runner-opencode-deepseek-1017.local.create-backlog", "--id chat-features.runner-opencode.local.create-backlog"),
]) assert.throws(() => assertBoundary(mutant));
continue;
}
for (const block of providerJobs) {
if (name === "runner-protocol-live-evals.yml") {
assert.ok(block.includes("name: ${{ needs.authorize.outputs.accounting_mode == 'controlled' && 'runner-e2e-private-paid' || 'runner-e2e-paid' }}"));
@@ -580,16 +580,33 @@ describeEmbeddedPostgres("issue scheduled retry routes", () => {
.where(inArray(heartbeatRuns.id, work.map(row => row.id)));
clock();
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => { throw error; } });
const register = coordinator.register.bind(coordinator);
let initial: Awaited<ReturnType<typeof states>> | undefined;
let releaseFirstPass!: () => void;
const firstPassHeld = new Promise<void>(resolve => { releaseFirstPass = resolve; });
vi.spyOn(coordinator, "register").mockImplementation((queue, task) => register(queue, {
...task,
run: async signal => {
const result = await task.run(signal);
if (initial) return result;
const cancelled = (await states()).filter(row => row.status === "cancelled");
if (cancelled.length === 0) return result;
initial = cancelled;
await firstPassHeld;
return result;
},
}));
try {
await heartbeatService(db, { runtimeEnv: {} }).registerScheduledRetryCancellationWork(coordinator, () => true).ready;
expect((await states()).filter(row => row.status === "cancelled")).toHaveLength(0);
const earlierId = "00000000-0000-4000-8000-000000000000";
await writeIntent({ companyId: a.companyId, retryRunId: earlierId });
work.push({ id: earlierId });
await writeIntent({ companyId: a.companyId, retryRunId: earlierId });
await vi.advanceTimersByTimeAsync(1000);
await expect.poll(async () => (await states()).filter(row => row.status === "cancelled").length).toBe(20);
const initial = (await states()).filter(row => row.status === "cancelled");
expect(new Set(initial.map(row => row.companyId)).size).toBe(2);
await expect.poll(() => initial).toBeDefined();
expect(initial).toHaveLength(20);
expect(new Set(initial!.map(row => row.companyId)).size).toBe(2);
releaseFirstPass();
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect((await states()).filter(row => row.status === "cancelled")).toHaveLength(29);
@@ -600,7 +617,7 @@ describeEmbeddedPostgres("issue scheduled retry routes", () => {
const reads = vi.spyOn(db, "select");
await vi.advanceTimersByTimeAsync(180_000);
expect(reads).not.toHaveBeenCalled(); reads.mockRestore();
} finally { await coordinator.stop(); }
} finally { releaseFirstPass(); await coordinator.stop(); }
});
});
+99 -2
View File
@@ -29,7 +29,8 @@ and removes it with the isolated instance.
The local native OpenCode profile uses the current production-qualified model
and `OPENROUTER_API_KEY` through the existing encrypted company-secret fixture.
Each of the two journeys has a cell for each profile. All four cells are manual.
Each of the two journeys has a cell for each original profile. These four cells
remain unchanged; the TestOps profile below adds one separately named backlog cell.
```sh
pnpm test:e2e:runner:typecheck
@@ -40,13 +41,41 @@ pnpm test:e2e:runner -- --id chat-features.runner-codex-subscription.local.deleg
pnpm test:e2e:runner -- --id chat-features.runner-opencode.local.delegate-announcement --max-automatic-retries 0
```
Run creation first and inspect its packaged report before delegation. For initial
Run creation first and inspect its packaged report before delegation. For the original Codex
acceptance, run each cell three times against fresh fixtures on the same definition
revision. Keep every attempt, including failures; a retry does not erase one. A
fixture or product correction starts a new three-sample measurement. All cells
are explicit-only and excluded from `--all`; this does not implement or qualify
the rest of the YAML inventory.
## OpenCode TestOps integration acceptance
The additional explicit cell
`chat-features.runner-opencode-deepseek-1017.local.create-backlog` runs the same backlog
query, public fixture and saved-state oracle on native OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`. It requires only the selected
`OPENROUTER_API_KEY` profile credential. The launcher stores an encrypted secret
reference for the disposable agents; no key belongs in catalog data or chat.
This selected model is a qualification candidate. Other OpenCode suites keep
their existing model, and delegation is not registered for this profile.
```sh
pnpm test:e2e:runner -- --id chat-features.runner-opencode-deepseek-1017.local.create-backlog --max-automatic-retries 0
```
Use a separately named TestOps integration-acceptance plan and launch for this
single cell. It retains one human request, one accounted controller work run,
120-second turns, a seven-minute attempt, the five-minute stable-state bound
and zero automatic retries. Morgan is created as fixture context but must not
execute the saved backlog task. Existing setup probes and unknown usage remain
reported separately. The original 244-scenario campaign, failed attempts and
Codex measurements are not reset or regraded by this measurement.
The existing normalized result, JUnit and packaged report carry the outcome to
the execution integration. Catalog selection and provider-free oracle checks
do not establish an AWS model pass or a successful TestOps upload. Accept this
integration only after the real AWS execution and its TestOps result agree.
## Bounds and grading
Creation expects one accounted native work run; delegation expects three (CEO
@@ -151,3 +180,71 @@ in the history. Across the original final-definition batch and this recheck,
backlog is 5/5 and delegation is 5/6. The local
`results/chat-feature-recheck/verification.md` records the follow-up evidence;
its standard dashboard is explicitly scoped to the latest campaign.
## TestOps-backed AWS backlog cell
The dedicated `Chat feature TestOps` workflow accepts only the shipped default
branch and the fixed OpenCode backlog cell. `evaluate` is the default operation
and requires TestOps' `ALLURE_JOB_RUN_ID`. A protected reporting job verifies the
project, singleton plan, assigned public workflow, selected case and current
GitHub run before any build or provider access. The evaluated process receives
no TestOps token. It retains the original one-query, one-work-run, 120-second
turn, 420-second attempt, 300-second settled-state and zero-retry limits.
Two metadata operations prepare that route. `bootstrap` registers the actual
public CI job through pinned `allurectl` with an empty results directory and
reconciles only the dedicated case and singleton plan. It produces no passing
test result. After the registered job and protected case/plan/job identifiers
are verified, `launch-plan` sends one TestOps plan-run request and exits.
The TestOps job then dispatches `evaluate`; an operator must not dispatch a
second evaluation manually. Missing or ambiguous registration and selection
metadata stops before inference.
The workflow reuses the existing AWS fleet and strict private evidence storage.
Its shipped-main authority is separate from manual candidate approval. Original
logs, provider records and failure evidence remain private. The reporting adapter
exports only a source/run-bound normalized result, JUnit and screenshots already
marked by the owned fixture screenshot producer with matching file hashes.
A separate protected reporting job converts that projection to the exact TestOps
case. Failed grades and cleanup failures remain failed.
Reporting setup uses the master-only `runner-e2e-testops-reporting` environment,
with the existing reporting token delivered through the reviewed sealed-box
bootstrap. `CHAT_TESTOPS_CASE_ID`, `CHAT_TESTOPS_PLAN_ID` and
`CHAT_TESTOPS_JOB_ID` must name the observed dedicated mapping; the numeric
`CHAT_TESTOPS_GITHUB_WORKFLOW_ID` supports TestOps' registered workflow URL.
No arbitrary source, model, case, retry or budget input is accepted.
The initial `launch-plan` administrator operation also requires a protected
`CHAT_TESTOPS_LAUNCH_RUN_NUMBER`. After registration, the trusted operator creates
this variable once with the reviewed next workflow run number and reads it back.
The launcher compares GitHub's actual run number, attempt, workflow, source and
actors before any TestOps mutation. A later dispatch or replay cannot reuse that
approved number. Missing or uncertain metadata stops; an uncertain request needs
reconciliation rather than a new send. The nonce and launch search provide
additional evidence. Normal TestOps **Run** uses `evaluate` and is independent of
this one-time administrator fence. The operation parameter identifier and launch
nonce are also protected configuration, not dispatch inputs.
`CHAT_TESTOPS_INTEGRATION_ID` names the reviewed existing project GitHub
integration. The launcher verifies its project, type and enabled state, then binds
only the newly registered public job to it. It never changes integration settings
or credentials. Since the job response does not expose that association, only the
real TestOps-to-GitHub dispatch can qualify it.
After `allurectl watch` finishes, the same protected reporter checks the actual
TestOps job-run, launch, selected case, result source/status and attachment
metadata. Processing reads are bounded to 12 polls, five-second gaps and a
120-second processing deadline. Unknown or missing results remain unverified.
The check downloads no attachment content and claims no server-side attachment
byte hash. Its closed receipt is retained separately. An upload success alone
is not a passing feature result, and verifying an original failure cannot turn
that failure into a pass.
Build and worker installs use the existing reviewed report-lock policy. Only its
committed input or output lock is accepted; its authenticated correction is the
only allowed normalization. Dependency manifests, patch inputs and policy must
remain unchanged. Installs are frozen with lifecycle scripts disabled. The
credential-free build rechecks that boundary before packaging, and the worker
matches the source/lock receipt before installing. Arbitrary resolver drift is
rejected before provider access.
+4 -1
View File
@@ -5,7 +5,10 @@ only company, agents and project context through public APIs. Both agents use
production role instructions; evaluator outcomes are never injected. Real browser
submission, task audits, worker document provenance, completion delivery and
reload readback establish the outcome. The existing fixture registry, subscription
credential staging and whole-company cleanup own setup and teardown.
credential staging and whole-company cleanup own setup and teardown. The explicit
OpenCode backlog cell instead uses an encrypted OpenRouter profile secret
reference. It preserves the same fixture and oracle without registering
OpenCode delegation or changing the other suites' model.
[Deployment-targeted journeys](PRODUCT-JOURNEYS.md) resolve only their requested
fixture graph through `FixtureRegistry.setupAll(requested)`. Omitting the
+3 -3
View File
@@ -219,10 +219,10 @@ describe("runner E2E catalog", () => {
expect(localIntegrityTasks).toHaveLength(2);
expect(openRouterBreadthTasks).toHaveLength(3);
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 4, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2,
2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2,
]);
expect(validateRunnerCatalog()).toHaveLength(774);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(774);
expect(validateRunnerCatalog()).toHaveLength(775);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(775);
expect(
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
).toHaveLength(48);
+9 -5
View File
@@ -26,7 +26,7 @@ import { copilotProviderDeathTasks } from "./copilot-active-stop-tasks.js";
import { instructionPersistenceTask } from "./instruction-persistence.js";
import { apiResponseReadingTask } from "./api-response-reading.js";
import { taskTitleTasks, taskTitleDefinitionDigest, TASK_TITLE_BUDGET_CENTS } from "./task-titles.js";
import { CHAT_FEATURE_SUITE, CHAT_FEATURE_BUDGET_CENTS, chatFeatureTasks, chatFeatureProfile, chatFeatureScenarios, chatFeatureDefinitionDigest } from "./chat-feature-cases.js";
import { CHAT_FEATURE_SUITE, CHAT_FEATURE_BUDGET_CENTS, CHAT_FEATURE_OPENCODE_MODEL, chatFeatureTasks, chatFeatureProfile, chatFeatureScenarios, chatFeatureDefinitionDigest } from "./chat-feature-cases.js";
import { blockerTasks, blockerProfile } from "./blocker-cases.js";
import { accountingTasks } from "./accounting-cases.js";
import { continuationTasks, questionResumeTask } from "./continuation-cases.js";
@@ -1609,10 +1609,14 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
id: CHAT_FEATURE_SUITE, label: "Chat feature journeys", manualOnly: true,
description: "Natural catalog requests create backlog work or delegate an announcement and return its saved result.",
groups: ["chat", "native"],
profiles: [nativeProfile({ id: "runner-codex-subscription", label: "Native Codex subscription",
provider: "codex", model: DEFAULT_CODEX_LOCAL_MODEL, credential: "CODEX_AUTH_JSON", supportedEnvironments: ["local"] }),
runnerProfiles.find(profile => profile.id === "runner-opencode")!].map(chatFeatureProfile),
environments: [localEnvironment], tasks: chatFeatureTasks, expectedMatrixSize: 4,
profiles: [chatFeatureProfile(nativeProfile({ id: "runner-codex-subscription", label: "Native Codex subscription",
provider: "codex", model: DEFAULT_CODEX_LOCAL_MODEL, credential: "CODEX_AUTH_JSON", supportedEnvironments: ["local"] })),
chatFeatureProfile(runnerProfiles.find(profile => profile.id === "runner-opencode")!),
chatFeatureProfile(nativeProfile({ id: "runner-opencode-deepseek-1017", label: "Native OpenCode DeepSeek 1017",
provider: "opencode", model: CHAT_FEATURE_OPENCODE_MODEL, credential: "OPENROUTER_API_KEY", supportedEnvironments: ["local"],
modelQualification: { source: "candidate_runner_profile", qualificationId: "chat-feature-testops:deepseek-v4.1-flash-1017" } }))],
environments: [localEnvironment], tasks: chatFeatureTasks, expectedMatrixSize: 5,
excludedExecutionIds: ["chat-features.runner-opencode-deepseek-1017.local.delegate-announcement"],
definitionMetadata: { version: 2, definitionDigest: chatFeatureDefinitionDigest, scheduling: "explicit-only",
instructions: "production-role-defaults", companyAndAgentBudgetCents: CHAT_FEATURE_BUDGET_CENTS,
unpricedUsagePolicy: "allow-through-public-budget-policy", taskRunLimits: { ceo: 3, worker: 1 },
+2 -1
View File
@@ -6,6 +6,7 @@ import type { RunnerProfileFixture, RunnerTaskFixture } from "./types.js";
export const CHAT_FEATURE_SUITE = "chat-features";
export const CHAT_FEATURE_BUDGET_CENTS = 500;
export const CHAT_FEATURE_GRADER = "paperclip.chat-features.v2";
export const CHAT_FEATURE_OPENCODE_MODEL = "openrouter/deepseek/deepseek-v4.1-flash-1017";
export const CHAT_FEATURE_CASES = ["create-backlog", "delegate-announcement"] as const;
export type ChatFeatureCase = typeof CHAT_FEATURE_CASES[number];
export interface ChatScenario {
@@ -48,7 +49,7 @@ export function chatFeatureProfile(profile: RunnerProfileFixture): RunnerProfile
return { ...payload, name: "CEO", title: "CEO", role: "ceo",
capabilities: "Coordinates company work and delegates tasks to the team.",
budgetMonthlyCents: CHAT_FEATURE_BUDGET_CENTS,
adapterConfig: { ...adapterConfig, timeoutSec: 120, modelReasoningEffort: "low" },
adapterConfig: { ...adapterConfig, timeoutSec: 120, ...(profile.id === "runner-opencode-deepseek-1017" ? {} : { modelReasoningEffort: "low" }) },
runtimeConfig: { ...(payload.runtimeConfig as Record<string, unknown>),
heartbeat: { enabled: false, maxConcurrentRuns: 1, maxDailyRuns: 3, maxDailyCostCents: CHAT_FEATURE_BUDGET_CENTS } },
};
+33 -3
View File
@@ -1,7 +1,7 @@
import { describe, expect, it } from "vitest";
import { chatFeatureExecutionRuns, gradeChatFeature, type ChatFeatureObservation } from "./chat-feature-scoring.js";
import { chatFeatureScenarios, readChatFeatureScenarios } from "./chat-feature-cases.js";
import { runnerMatrix } from "./catalog.js";
import { runnerMatrix, runnerProfiles } from "./catalog.js";
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
const announcement = "Welcome to Acorn Notes, launching November 12! Join us with invitation code ACORN123.";
@@ -107,9 +107,15 @@ describe("chat feature saved-state oracle", () => {
});
describe("chat feature selection", () => {
it("registers the two authored queries as explicit local subscription and OpenCode cells", () => {
const cells = runnerMatrix.filter(cell => cell.suite.id === "chat-features");
it("preserves the four authored subscription and production OpenCode cells", () => {
const cells = runnerMatrix.filter(cell => cell.suite.id === "chat-features" && cell.profile.id !== "runner-opencode-deepseek-1017");
expect(cells).toHaveLength(4);
const stock = runnerProfiles.find(profile => profile.id === "runner-opencode")!;
expect(cells.filter(cell => cell.profile.id === stock.id).map(cell => cell.profile.model)).toEqual([stock.model, stock.model]);
for (const cell of cells.filter(cell => cell.profile.id === stock.id)) {
const payload = cell.profile.buildAgent({ environmentId: "env", environmentFixtureId: "local", workspacePath: "/tmp/fixture", executionId: cell.id, secretRefs: { OPENROUTER_API_KEY: { type: "secret_ref", secretId: "fixture-openrouter", version: "latest" } } });
expect(payload.adapterConfig).toMatchObject({ model: stock.model, modelReasoningEffort: "low", timeoutSec: 120 });
}
for (const cell of cells) {
expect(cell.environment.id).toBe("local");
expect(cell.requiredCredentials).toEqual([cell.profile.id === "runner-opencode" ? "OPENROUTER_API_KEY" : "CODEX_AUTH_JSON"]);
@@ -119,6 +125,30 @@ describe("chat feature selection", () => {
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(row => row.id === cell.id)).toBe(false);
}
});
it("selects one bounded OpenCode backlog cell with the requested model and secret reference", () => {
const cells = runnerMatrix.filter(cell => cell.suite.id === "chat-features" && cell.profile.id === "runner-opencode-deepseek-1017");
expect(cells.map(cell => cell.id)).toEqual(["chat-features.runner-opencode-deepseek-1017.local.create-backlog"]);
const cell = cells[0]!;
expect(cell.profile).toMatchObject({ provider: "opencode", model: "openrouter/deepseek/deepseek-v4.1-flash-1017", expectedRuntimeMode: "native" });
expect(cell.profile.modelQualification.source).toBe("candidate_runner_profile");
expect(cell.requiredCredentials).toEqual(["OPENROUTER_API_KEY"]);
expect(cell.task).toMatchObject({ expectedRunCount: 1, turnTimeoutMs: 120_000,
attemptTimeoutMs: { local: 420_000 }, automaticRetryPolicy: "single_attempt" });
expect(cell.task.buildPrompt("nonce")).toBe(chatFeatureScenarios[0]!.query);
const secret = { type: "secret_ref", secretId: "fixture-openrouter", version: "latest" } as const;
const input = { environmentId: "env", environmentFixtureId: "local", workspacePath: "/tmp/fixture", executionId: "backlog" } as const;
const agent = cell.profile.buildAgent({ ...input, secretRefs: { OPENROUTER_API_KEY: secret } });
expect(agent).toMatchObject({ role: "ceo", adapterType: "paperclip_runner", budgetMonthlyCents: 500,
adapterConfig: { provider: "opencode", model: "openrouter/deepseek/deepseek-v4.1-flash-1017", timeoutSec: 120,
env: { OPENROUTER_API_KEY: secret } },
runtimeConfig: { heartbeat: { enabled: false, maxConcurrentRuns: 1, maxDailyRuns: 3, maxDailyCostCents: 500 } } });
expect(agent).not.toHaveProperty("instructionsBundle");
expect(agent.adapterConfig).not.toHaveProperty("modelReasoningEffort");
expect(agent.adapterConfig).not.toHaveProperty("codexPermissionMode");
expect(() => cell.profile.buildAgent({ ...input, secretRefs: {} })).toThrow("Missing fixture secret reference OPENROUTER_API_KEY");
expect(selectRunnerExecutions(parseRunnerSelectors(["--id", cell.id])).map(row => row.id)).toEqual([cell.id]);
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(row => row.id === cell.id)).toBe(false);
});
it("refuses missing, malformed or duplicate catalog entries", () => {
for (const yaml of ["{", "schema_version: 2", "schema_version: 1\nfeature: tasks\nstatus: planned\nscenarios: []"])
expect(() => readChatFeatureScenarios(yaml)).toThrow();
@@ -0,0 +1,342 @@
export const endpoint = 'https://paperclip.testops.cloud';
export const planName = 'Chat feature evals';
export const fullName = 'paperclip.Product E2E.chat-features.create-backlog';
export const aliases = [fullName, 'paperclip.Product E2E.chat-features.tasks.create-backlog'];
class SetupFailure extends Error {
constructor(code, status = null) { super(code); this.code = code; this.status = status; }
}
const fail = code => {
const match = /^(testops_(?:api|auth)|chat_testops_github)_http_([1-5][0-9]{2})$/.exec(code);
throw new SetupFailure(match ? `${match[1]}_http` : code, match ? Number(match[2]) : null);
};
const diagnosticCodes = new Set(['untrusted_chat_testops_context', 'testops_token_missing', 'testops_auth_http',
'testops_api_http', 'testops_auth_invalid', 'testops_route_invalid', 'testops_response_too_large',
'testops_response_invalid', 'testops_metadata_page_incomplete', 'testops_case_identity_invalid',
'testops_case_alias_conflict', 'testops_plan_name_conflict', 'testops_plan_identity_invalid',
'testops_integration_identity_invalid', 'testops_bootstrap_identity_invalid', 'testops_plan_without_matching_case',
'testops_created_case_identity_invalid', 'testops_created_plan_identity_invalid', 'testops_plan_scope_mismatch',
'testops_operation_invalid', 'testops_output_invalid', 'untrusted_public_testops_context', 'chat_testops_identity_missing',
'chat_testops_case_mismatch', 'chat_testops_plan_mismatch', 'chat_testops_plan_job_mismatch', 'chat_testops_job_mismatch',
'chat_testops_jobrun_mismatch', 'chat_testops_launch_mismatch', 'chat_testops_selection_mismatch', 'chat_testops_parameter_mismatch', 'chat_testops_nonce_invalid', 'chat_testops_nonce_conflict', 'chat_testops_catalog_incomplete', 'chat_testops_launch_admission', 'chat_testops_github_http', 'chat_testops_integration_mismatch', 'chat_testops_upload_mismatch', 'chat_testops_upload_unverified']);
export function failureProjection(error, stage) {
const stages = ['context', 'output_admission', 'authentication', 'metadata_inspection', 'plan_bootstrap', 'selection_validation', 'launch_admission', 'launch_preflight', 'plan_launch', 'bootstrap_job_metadata', 'result_export', 'upload_verification', 'receipt'];
return { stage: stages.includes(stage) ? stage : 'unknown',
code: error instanceof SetupFailure && diagnosticCodes.has(error.code) ? error.code : 'unknown_failure',
httpStatus: error instanceof SetupFailure && Number.isInteger(error.status) && error.status >= 100 && error.status < 600 ? error.status : null };
}
const id = value => Number.isSafeInteger(value) && value > 0;
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
export async function testopsClient(token, fetchImpl = fetch) {
if (!token) fail('testops_token_missing');
const auth = await fetchImpl(`${endpoint}/api/uaa/oauth/token`, { method: 'POST', redirect: 'error',
headers: { Accept: 'application/json' }, body: new URLSearchParams({ grant_type: 'apitoken', scope: 'openid', token }),
signal: AbortSignal.timeout(30_000) });
if (!auth.ok) fail(`testops_auth_http_${auth.status}`);
const value = await auth.json();
if (typeof value.access_token !== 'string' || !value.access_token) fail('testops_auth_invalid');
return async (route, method = 'GET', body, timeoutMs = 30_000) => {
if (!Number.isSafeInteger(timeoutMs) || timeoutMs < 1 || timeoutMs > 30_000) fail('chat_testops_upload_unverified');
if (!route.startsWith('/api/') || route.includes('://')) fail('testops_route_invalid');
const response = await fetchImpl(`${endpoint}${route}`, { method, redirect: 'error',
headers: { Accept: 'application/json', Authorization: `Bearer ${value.access_token}`, ...(body ? { 'Content-Type': 'application/json' } : {}) },
...(body ? { body: JSON.stringify(body) } : {}), signal: AbortSignal.timeout(timeoutMs) });
if (!response.ok) fail(`testops_api_http_${response.status}`);
if (response.status === 204) return null;
const text = await response.text();
if (Buffer.byteLength(text) > 2_000_000) fail('testops_response_too_large');
try { return JSON.parse(text); } catch { fail('testops_response_invalid'); }
};
}
const rows = value => {
if (!Array.isArray(value?.content) || value.content.length > 100 || value.totalPages > 1) fail('testops_metadata_page_incomplete');
return value.content;
};
export async function inspectChatMetadata(api) {
const cases = new Map(), catalogIds = new Set();
let totalPages, totalElements;
for (let page = 0; page < 10; page++) {
const catalog = await api(`/api/testcase?${new URLSearchParams({ projectId: '1', page: String(page), size: '100', sort: 'id,asc' })}`);
if (!Array.isArray(catalog?.content) || catalog.content.length > 100 || catalog.number !== page
|| !Number.isSafeInteger(catalog.totalPages) || catalog.totalPages < 0 || catalog.totalPages > 10
|| !Number.isSafeInteger(catalog.totalElements) || catalog.totalElements < 0 || catalog.totalElements > 1000
|| totalPages !== undefined && (totalPages !== catalog.totalPages || totalElements !== catalog.totalElements)) fail('chat_testops_catalog_incomplete');
totalPages = catalog.totalPages; totalElements = catalog.totalElements;
const ids = catalog.content.map(row => {
if (!id(row.id) || catalogIds.has(row.id)) fail('chat_testops_catalog_incomplete');
catalogIds.add(row.id); return row.id;
});
for (let offset = 0; offset < ids.length; offset += 4) {
const batch = ids.slice(offset, offset + 4);
const details = await Promise.all(batch.map(value => api(`/api/testcase/${value}`)));
for (const [index, value] of details.entries()) {
if (value?.id !== batch[index] || value.projectId !== 1 || value.fullName != null && typeof value.fullName !== 'string') fail('testops_case_identity_invalid');
if (!aliases.includes(value.fullName)) continue;
if (value.deleted === true || typeof value.automated !== 'boolean') fail('testops_case_identity_invalid');
cases.set(value.id, { id: value.id, fullName: value.fullName, automated: value.automated });
}
}
if (page + 1 >= totalPages) break;
if (page === 9) fail('chat_testops_catalog_incomplete');
}
if (catalogIds.size !== totalElements) fail('chat_testops_catalog_incomplete');
const plans = rows(await api(`/api/testplan?${new URLSearchParams({ projectId: '1', name: planName, page: '0', size: '100' })}`))
.filter(value => value.name === planName);
if (plans.length > 1) fail('testops_plan_name_conflict');
if (plans.some(value => !id(value.id) || value.projectId !== 1)) fail('testops_plan_identity_invalid');
const integrations = rows(await api('/api/integration/project/1?page=0&size=100'))
.filter(value => value.info?.type === 'github')
.map(value => ({ id: value.id, type: 'github', disabled: typeof value.disabled === 'boolean' ? value.disabled : null }));
if (integrations.some(value => !id(value.id))) fail('testops_integration_identity_invalid');
return { projectId: 1, caseConflict: cases.size > 1, cases: [...cases.values()], plans: plans.map(value => ({ id: value.id, name: planName, count: value.testCasesCount })), integrations };
}
export async function ensureChatPlan(api, inspected, markOnce) {
if (inspected.projectId !== 1 || inspected.plans.length > 1) fail('testops_bootstrap_identity_invalid');
if (inspected.caseConflict === true || inspected.cases.length > 1) fail('testops_case_alias_conflict');
let selected = inspected.cases[0], plan = inspected.plans[0];
if (plan && !selected) fail('testops_plan_without_matching_case');
if (!selected || !plan) await markOnce();
if (!selected) {
const created = await api('/api/testcase', 'POST', { projectId: 1, name: 'Create a task backlog through chat', fullName,
automated: true, description: 'Canonical scenario: tasks.create-backlog. Readiness and recorded outcomes remain separate.',
expectedResult: 'The real agent creates the requested backlog; public Product E2E assertions and cleanup must pass.' });
if (!id(created?.id) || created.projectId !== 1 || created.fullName !== fullName) fail('testops_created_case_identity_invalid');
selected = { id: created.id, fullName, automated: created.automated === true };
}
if (!plan) {
const created = await api('/api/testplan', 'POST', { projectId: 1, name: planName,
treeSelection: { inverted: false, leafsInclude: [selected.id] } });
if (!id(created?.id) || created.projectId !== 1 || created.name !== planName) fail('testops_created_plan_identity_invalid');
plan = { id: created.id, name: planName };
}
const actual = await api(`/api/testplan/${plan.id}`);
if (actual.projectId !== 1 || actual.name !== planName || actual.testCasesCount !== 1
|| actual.treeSelection?.inverted === true || !same(actual.treeSelection?.leafsInclude, [selected.id])
|| (actual.treeSelection?.groupsInclude?.length ?? 0) > 0 || (actual.treeSelection?.leafsExclude?.length ?? 0) > 0) fail('testops_plan_scope_mismatch');
return { projectId: 1, case: selected, plan: { id: plan.id, name: planName, count: 1 },
caseUrl: `${endpoint}/project/1/test-cases/${selected.id}`, planUrl: `${endpoint}/testplan/${plan.id}`,
actualResult: null, note: 'Plan preparation is not an executed measurement or a passing result.' };
}
export const executionId = 'chat-features.runner-opencode-deepseek-1017.local.create-backlog';
export const scenario = 'tasks.create-backlog';
const positive = value => typeof value === 'string' && /^[1-9][0-9]{0,15}$/.test(value) && Number.isSafeInteger(Number(value));
export function publicContext(env) {
if (env.GITHUB_REPOSITORY !== 'paperclipai/paperclip' || env.GITHUB_REF !== 'refs/heads/master'
|| env.GITHUB_EVENT_NAME !== 'workflow_dispatch' || env.GITHUB_RUN_ATTEMPT !== '1'
|| env.GITHUB_WORKFLOW_REF !== 'paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master'
|| !/^[a-f0-9]{40}$/.test(env.GITHUB_SHA ?? '') || !positive(env.GITHUB_RUN_ID)) fail('untrusted_public_testops_context');
return { runId: env.GITHUB_RUN_ID, attempt: '1', workflowSha: env.GITHUB_SHA, targetSha: env.GITHUB_SHA };
}
export async function validateChatSelection(api, env) {
const context = publicContext(env);
const names = ['ALLURE_JOB_RUN_ID', 'CHAT_TESTOPS_CASE_ID', 'CHAT_TESTOPS_PLAN_ID', 'CHAT_TESTOPS_JOB_ID'];
if (names.some(name => !positive(env[name]))) fail('chat_testops_identity_missing');
const [caseId, planId, jobId, jobRunId] = ['CHAT_TESTOPS_CASE_ID', 'CHAT_TESTOPS_PLAN_ID', 'CHAT_TESTOPS_JOB_ID', 'ALLURE_JOB_RUN_ID'].map(name => Number(env[name]));
const selected = await api(`/api/testcase/${caseId}`);
if (selected.id !== caseId || selected.projectId !== 1 || !aliases.includes(selected.fullName) || selected.deleted === true || selected.automated !== true) fail('chat_testops_case_mismatch');
const plan = await api(`/api/testplan/${planId}`);
if (plan.id !== planId || plan.projectId !== 1 || plan.name !== planName || plan.testCasesCount !== 1
|| plan.treeSelection?.inverted === true || !same(plan.treeSelection?.leafsInclude, [caseId])
|| (plan.treeSelection?.groupsInclude?.length ?? 0) > 0 || (plan.treeSelection?.leafsExclude?.length ?? 0) > 0) fail('chat_testops_plan_mismatch');
const assignments = await api(`/api/testplan/${planId}/job`);
if (!Array.isArray(assignments) || assignments.length !== 1 || assignments[0].job?.id !== jobId || assignments[0].testCasesCount !== 1) fail('chat_testops_plan_job_mismatch');
const job = await api(`/api/job/${jobId}`);
const jobUrls = ['https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml'];
if (positive(env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID)) jobUrls.push(`https://github.com/paperclipai/paperclip/actions/workflows/${env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID}`);
if (job.id !== jobId || job.type !== 'github' || !jobUrls.includes(job.url)) fail('chat_testops_job_mismatch');
const run = await api(`/api/jobrun/${jobRunId}`);
if (run.id !== jobRunId || run.job?.id !== jobId || run.job?.type !== 'github'
|| !['scheduled', 'in_progress'].includes(run.stage) || !id(run.launchId)
|| run.url !== `https://github.com/paperclipai/paperclip/actions/runs/${context.runId}`) fail('chat_testops_jobrun_mismatch');
const launch = await api(`/api/launch/${run.launchId}`);
if (launch.id !== run.launchId || launch.projectId !== 1 || launch.closed === true) fail('chat_testops_launch_mismatch');
const selection = await api(`/api/jobrun/${jobRunId}/plan?expected=true`);
if (!Array.isArray(selection) || selection.length !== 1 || selection[0].id !== caseId || selection[0].selector !== selected.fullName) fail('chat_testops_selection_mismatch');
return { schema: 'paperclip.chat-feature-testops-selection/v1', ...context, allureJobRunId: String(jobRunId),
executionId, canonicalScenario: scenario, projectId: 1, planId, jobId, caseId, fullName: selected.fullName, launchId: run.launchId };
}
function configuredIds(env) {
const names = ['CHAT_TESTOPS_CASE_ID', 'CHAT_TESTOPS_PLAN_ID', 'CHAT_TESTOPS_JOB_ID', 'CHAT_TESTOPS_OPERATION_PARAMETER_ID', 'CHAT_TESTOPS_INTEGRATION_ID'];
if (names.some(name => !positive(env[name]))) fail('chat_testops_identity_missing');
return Object.fromEntries(names.map((name, index) => [['caseId', 'planId', 'jobId', 'parameterId', 'integrationId'][index], Number(env[name])]));
}
function publicJobUrl(url, env) {
return url === 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml'
|| positive(env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID) && url === `https://github.com/paperclipai/paperclip/actions/workflows/${env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID}`;
}
async function registeredPlan(api, env) {
const ids = configuredIds(env);
const selected = await api(`/api/testcase/${ids.caseId}`);
if (selected.id !== ids.caseId || selected.projectId !== 1 || !aliases.includes(selected.fullName) || selected.deleted === true || selected.automated !== true) fail('chat_testops_case_mismatch');
const plan = await api(`/api/testplan/${ids.planId}`);
if (plan.id !== ids.planId || plan.projectId !== 1 || plan.name !== planName || plan.testCasesCount !== 1
|| plan.treeSelection?.inverted === true || !same(plan.treeSelection?.leafsInclude, [ids.caseId])
|| (plan.treeSelection?.groupsInclude?.length ?? 0) > 0 || (plan.treeSelection?.leafsExclude?.length ?? 0) > 0) fail('chat_testops_plan_mismatch');
const job = await api(`/api/job/${ids.jobId}`);
if (job.id !== ids.jobId || job.type !== 'github' || !publicJobUrl(job.url, env)) fail('chat_testops_job_mismatch');
const integration = await api(`/api/integration/${ids.integrationId}/project/1`);
if (integration.id !== ids.integrationId || integration.projectId !== 1 || integration.info?.type !== 'github' || integration.disabled !== false) fail('chat_testops_integration_mismatch');
const definition = await api(`/api/environment/${ids.parameterId}`);
if (definition.id !== ids.parameterId || definition.name !== 'operation' || !Array.isArray(job.parameters) || job.parameters.length !== 1
|| job.parameters[0].id !== ids.parameterId || !['bootstrap', 'evaluate'].includes(job.parameters[0].value)) fail('chat_testops_parameter_mismatch');
return { ...ids, selected, plan, job };
}
export async function bootstrapJobMetadata(api, env) {
if (!positive(env.ALLURE_JOB_RUN_ID)) return { status: 'official_job_run_id_missing', actualResult: null };
const run = await api(`/api/jobrun/${env.ALLURE_JOB_RUN_ID}`);
if (!id(run?.job?.id)) fail('chat_testops_jobrun_mismatch');
const job = await api(`/api/job/${run.job.id}`);
if (job.id !== run.job.id || job.type !== 'github' || !Array.isArray(job.parameters) || job.parameters.length > 4) fail('chat_testops_job_mismatch');
const parameters = [];
for (const parameter of job.parameters) {
if (!id(parameter.id)) fail('chat_testops_parameter_mismatch');
const definition = await api(`/api/environment/${parameter.id}`);
const known = ['operation', 'ALLURE_JOB_RUN_ID', 'ALLURE_USERNAME'].includes(definition.name);
parameters.push({ id: parameter.id, name: known ? definition.name : 'UNKNOWN',
value: definition.name === 'operation' && ['bootstrap', 'evaluate', 'launch-plan'].includes(parameter.value) ? parameter.value : 'NOT_PROJECTED' });
}
return { status: 'observed', projectId: 1, jobId: job.id, allureJobRunId: env.ALLURE_JOB_RUN_ID,
publicWorkflowMatches: publicJobUrl(job.url, env), currentRunUrlMatches: run.url === `https://github.com/paperclipai/paperclip/actions/runs/${env.GITHUB_RUN_ID}`,
launchId: id(run.launchId) ? run.launchId : null, parameters, actualResult: null };
}
const launchAdmissions = new WeakSet();
export async function admitLaunchPlan(env, fetchImpl = fetch) {
const source = publicContext(env);
const expected = env.CHAT_TESTOPS_LAUNCH_RUN_NUMBER, actual = env.GITHUB_RUN_NUMBER;
if (!positive(expected) || !positive(actual) || actual !== expected || !positive(env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID)
|| !positive(env.GITHUB_ACTOR_ID) || !positive(env.TESTOPS_TRIGGERING_ACTOR_ID) || !env.GH_TOKEN) fail('chat_testops_launch_admission');
let allowed;
try { allowed = JSON.parse(env.TESTOPS_ALLOWED_ACTOR_IDS ?? 'null'); } catch { fail('chat_testops_launch_admission'); }
if (!Array.isArray(allowed) || !allowed.length || allowed.some(value => !id(value)) || !allowed.includes(Number(env.GITHUB_ACTOR_ID)) || !allowed.includes(Number(env.TESTOPS_TRIGGERING_ACTOR_ID))) fail('chat_testops_launch_admission');
const read = async route => {
const response = await fetchImpl(`https://api.github.com/repos/paperclipai/paperclip${route ? `/${route}` : ''}`, {
method: 'GET', redirect: 'error', headers: { Accept: 'application/vnd.github+json', Authorization: `Bearer ${env.GH_TOKEN}` },
signal: AbortSignal.timeout(30_000) });
if (!response.ok) fail(`chat_testops_github_http_${response.status}`);
const text = await response.text();
if (Buffer.byteLength(text) > 512_000) fail('chat_testops_launch_admission');
try { return JSON.parse(text); } catch { fail('chat_testops_launch_admission'); }
};
const repository = await read('');
const workflow = await read('actions/workflows/chat-feature-testops.yml');
const run = await read(`actions/runs/${source.runId}`);
if (repository.full_name !== 'paperclipai/paperclip' || repository.private !== false || repository.default_branch !== 'master'
|| workflow.id !== Number(env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID) || workflow.path !== '.github/workflows/chat-feature-testops.yml' || workflow.state !== 'active'
|| run.id !== Number(source.runId) || run.workflow_id !== workflow.id || run.run_attempt !== 1 || run.run_number !== Number(expected)
|| run.head_sha !== source.workflowSha || run.head_branch !== 'master' || run.event !== 'workflow_dispatch'
|| run.repository?.full_name !== 'paperclipai/paperclip' || run.repository?.private !== false
|| run.actor?.id !== Number(env.GITHUB_ACTOR_ID) || run.triggering_actor?.id !== Number(env.TESTOPS_TRIGGERING_ACTOR_ID) || !allowed.includes(run.triggering_actor.id)) fail('chat_testops_launch_admission');
const receipt = Object.freeze({ schema: 'paperclip.chat-feature-testops-launch-admission/v1', ...source,
workflowId: workflow.id, runNumber: run.run_number, expectedRunNumber: Number(expected), actorId: run.actor.id, triggeringActorId: run.triggering_actor.id });
launchAdmissions.add(receipt); return receipt;
}
export async function launchChatPlan(api, env, markOnce, admission) {
const source = publicContext(env);
if (!admission || !launchAdmissions.has(admission) || admission.runId !== source.runId || admission.workflowSha !== source.workflowSha
|| admission.targetSha !== source.targetSha || String(admission.expectedRunNumber) !== env.CHAT_TESTOPS_LAUNCH_RUN_NUMBER
|| String(admission.runNumber) !== env.GITHUB_RUN_NUMBER) fail('chat_testops_launch_admission');
const nonce = env.CHAT_TESTOPS_LAUNCH_NONCE;
if (!/^[a-f0-9]{32}$/.test(nonce ?? '')) fail('chat_testops_nonce_invalid');
const known = await registeredPlan(api, env);
const assignments = await api(`/api/testplan/${known.planId}/job`);
if (!Array.isArray(assignments) || assignments.length > 1 || assignments.some(row => row.job?.id !== known.jobId || row.testCasesCount !== 1)) fail('chat_testops_plan_job_mismatch');
const launchName = `${planName} · ${nonce}`;
const existing = rows(await api(`/api/launch?${new URLSearchParams({ projectId: '1', search: launchName, page: '0', size: '100' })}`));
if (existing.length) {
if (existing.length !== 1 || existing[0].name !== launchName || !id(existing[0].id)) fail('chat_testops_nonce_conflict');
return { schema: 'paperclip.chat-feature-testops-launch/v1', ...source, nonce, planId: known.planId,
jobId: known.jobId, caseId: known.caseId, launchId: existing[0].id, status: 'EXISTING_NO_RESEND', actualResult: null };
}
await markOnce({ ...source, launchAdmission: admission, nonce, planId: known.planId, jobId: known.jobId, caseId: known.caseId });
const parameters = [{ id: known.parameterId, value: 'evaluate' }];
await api(`/api/job/${known.jobId}`, 'PATCH', { integrationId: known.integrationId, canRun: true, parameters });
const patched = await api(`/api/job/${known.jobId}`);
if (patched.id !== known.jobId || patched.type !== 'github' || !publicJobUrl(patched.url, env) || !same(patched.parameters, parameters)) fail('chat_testops_parameter_mismatch');
if (!assignments.length) await api(`/api/testplan/${known.planId}/assign`, 'POST', { jobId: known.jobId,
selection: { inverted: false, manual: false, leafsInclude: [known.caseId] } });
await api(`/api/testplan/${known.planId}/jobparameter`, 'POST', { jobId: known.jobId, parameters });
const bound = await api(`/api/testplan/${known.planId}/job`);
if (!Array.isArray(bound) || bound.length !== 1 || bound[0].job?.id !== known.jobId || bound[0].testCasesCount !== 1
|| !same(bound[0].job.parameters, parameters)) fail('chat_testops_plan_job_mismatch');
const launched = await api(`/api/testplan/${known.planId}/run`, 'POST', { launchName });
if (!id(launched?.id) || launched.projectId !== 1 || launched.name !== launchName) fail('chat_testops_launch_mismatch');
return { schema: 'paperclip.chat-feature-testops-launch/v1', ...source, nonce, projectId: 1,
planId: known.planId, jobId: known.jobId, caseId: known.caseId, launchId: launched.id,
status: 'REQUESTED', integrationId: known.integrationId, integrationBindingRequested: true, integrationAssociationReadback: 'not_exposed_by_job_dto', executionId, canonicalScenario: scenario, actualRunId: null, actualResult: null };
}
export async function verifyChatUpload(api, env, selected, exported, { wait = ms => new Promise(resolve => setTimeout(resolve, ms)), now = Date.now } = {}) {
const context = publicContext(env);
if (selected.runId !== context.runId || selected.attempt !== '1' || selected.workflowSha !== context.workflowSha
|| selected.targetSha !== context.targetSha || selected.allureJobRunId !== env.ALLURE_JOB_RUN_ID
|| selected.executionId !== executionId || selected.canonicalScenario !== scenario || selected.projectId !== 1
|| !aliases.includes(selected.fullName) || ['planId', 'jobId', 'caseId', 'launchId'].some(key => !id(selected[key]))
|| exported.caseId !== selected.caseId || exported.campaignId !== `gha-${context.runId}-1` || exported.count !== 1
|| !['passed', 'failed'].includes(exported.status) || !['passed', 'failed', 'not_started'].includes(exported.cleanup)
|| exported.status === 'passed' && exported.cleanup !== 'passed' || !Array.isArray(exported.attachmentMetadata)
|| exported.attachmentMetadata.length < 2 || exported.attachmentMetadata.length > 10) fail('chat_testops_upload_mismatch');
const allowedAttachments = new Map([['Synthetic task and source conversation', 'image/png'],
['Execution provenance and checks', 'application/json'], ['Sanitized Product E2E record and accounting', 'application/json']]);
if (exported.attachmentMetadata.some(row => allowedAttachments.get(row.name) !== row.contentType
|| !Number.isSafeInteger(row.contentLength) || row.contentLength <= 0 || row.contentLength > 12 * 1024 * 1024)
|| exported.attachmentMetadata.filter(row => row.name === 'Execution provenance and checks').length !== 1
|| exported.attachmentMetadata.filter(row => row.name === 'Sanitized Product E2E record and accounting').length !== 1
|| exported.status === 'passed' && !exported.attachmentMetadata.some(row => row.contentType === 'image/png')) fail('chat_testops_upload_mismatch');
const signature = rows => rows.map(row => JSON.stringify([row.name, row.contentType, row.contentLength])).sort();
const runUrl = `https://github.com/paperclipai/paperclip/actions/runs/${context.runId}`;
const sourceUrl = `https://github.com/paperclipai/paperclip/commit/${context.targetSha}`;
const deadline = now() + 120_000;
const get = async route => {
const remaining = deadline - now();
if (remaining < 1) fail('chat_testops_upload_unverified');
const value = await api(route, 'GET', undefined, Math.min(30_000, Math.floor(remaining)));
if (now() >= deadline) fail('chat_testops_upload_unverified');
return value;
};
for (let poll = 0; poll < 12 && now() < deadline; poll++) {
const run = await get(`/api/jobrun/${selected.allureJobRunId}`);
if (run.id !== Number(selected.allureJobRunId) || run.job?.id !== selected.jobId || run.job?.type !== 'github'
|| run.launchId !== selected.launchId || run.url !== runUrl || !['in_progress', 'finished'].includes(run.stage)) fail('chat_testops_jobrun_mismatch');
const launch = await get(`/api/launch/${selected.launchId}`);
if (launch.id !== selected.launchId || launch.projectId !== 1) fail('chat_testops_launch_mismatch');
const listed = await get(`/api/testresult?${new URLSearchParams({ launchId: String(selected.launchId), page: '0', size: '100' })}`);
if (!Array.isArray(listed?.content) || listed.content.length > 1 || !Number.isSafeInteger(listed.totalElements)
|| listed.totalElements !== listed.content.length || !Number.isSafeInteger(listed.totalPages) || listed.totalPages < 0 || listed.totalPages > 1) fail('chat_testops_upload_mismatch');
if (listed.content.length) {
const row = listed.content[0];
if (!id(row.id) || row.testCaseId !== selected.caseId) fail('chat_testops_upload_mismatch');
const result = await get(`/api/testresult/${row.id}`);
if (result.id !== row.id || result.projectId !== 1 || result.launchId !== selected.launchId || result.testCaseId !== selected.caseId
|| result.fullName !== selected.fullName || result.jobRun?.id !== Number(selected.allureJobRunId)
|| result.jobRun?.job?.id !== selected.jobId || result.jobRun?.launchId !== selected.launchId || result.jobRun?.url !== runUrl
|| !Array.isArray(result.links) || !result.links.some(link => link.url === runUrl)
|| !result.links.some(link => link.url === sourceUrl)) fail('chat_testops_upload_mismatch');
if (!['passed', 'failed', 'broken', 'skipped', 'unknown'].includes(result.status)) fail('chat_testops_upload_mismatch');
if (result.status !== 'unknown' && result.status !== exported.status) fail('chat_testops_upload_mismatch');
const attached = await get(`/api/testresult/attachment?${new URLSearchParams({ testResultId: String(row.id), page: '0', size: '100' })}`);
if (!Array.isArray(attached?.content) || attached.content.length > 10 || !Number.isSafeInteger(attached.totalElements)
|| attached.totalElements !== attached.content.length || !Number.isSafeInteger(attached.totalPages) || attached.totalPages < 0 || attached.totalPages > 1
|| new Set(attached.content.map(value => value.id)).size !== attached.content.length
|| attached.content.some(value => !id(value.id) || value.missed === true || value.fromTestCase === true)) fail('chat_testops_upload_mismatch');
if (now() < deadline && result.status === exported.status && same(signature(attached.content), signature(exported.attachmentMetadata))) {
return { schema: 'paperclip.chat-feature-testops-upload-verification/v1', ...context, projectId: 1, planId: selected.planId,
jobId: selected.jobId, allureJobRunId: selected.allureJobRunId, launchId: selected.launchId, caseId: selected.caseId,
resultId: result.id, executionId, status: result.status, cleanup: exported.cleanup, verification: 'VERIFIED',
attachments: exported.attachmentMetadata.map(value => ({ ...value })),
launchUrl: `${endpoint}/launch/${selected.launchId}`, resultUrl: `${endpoint}/testresult/${result.id}` };
}
}
if (poll < 11 && now() + 5000 < deadline) await wait(5000);
}
fail('chat_testops_upload_unverified');
}
@@ -0,0 +1,392 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { mkdtemp, readFile, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { endpoint, planName, fullName, inspectChatMetadata, ensureChatPlan, testopsClient, failureProjection } from './chat-feature-testops-api.mjs';
const caseRow = { id: 91, projectId: 1, fullName, automated: true };
const plan = { id: 8, projectId: 1, name: planName, testCasesCount: 1, treeSelection: { inverted: false, leafsInclude: [91] } };
const page = content => ({ content, totalPages: 1 });
const inspectApi = (cases = [caseRow], plans = [plan]) => async route => {
if (route.startsWith('/api/testcase?')) return { content: cases.map(row => ({ id: row.id, name: 'Not exported', automated: row.automated })), totalElements: cases.length, totalPages: cases.length ? 1 : 0, number: 0 };
if (/^\/api\/testcase\/[0-9]+$/.test(route)) return cases.find(row => row.id === Number(route.split('/').at(-1)));
if (route.startsWith('/api/testplan?')) return page(plans);
if (route.startsWith('/api/integration/project/1')) return page([{ id: 2, info: { type: 'github' }, disabled: false, settings: { secret: 'DO_NOT_EXPORT' } }]);
assert.fail(route);
};
test('inspect reconciles exact historical case identity and discards integration settings', async () => {
const result = await inspectChatMetadata(inspectApi());
assert.deepEqual(result.cases, [{ id: 91, fullName, automated: true }]);
assert.deepEqual(result.integrations, [{ id: 2, type: 'github', disabled: false }]);
assert.ok(!JSON.stringify(result).includes('DO_NOT_EXPORT'));
});
test('multiple alias identities, foreign project and incomplete metadata stop before mutation', async () => {
const conflicts = await inspectChatMetadata(inspectApi([caseRow, { ...caseRow, id: 92, fullName: 'paperclip.Product E2E.chat-features.tasks.create-backlog' }]));
assert.equal(conflicts.caseConflict, true); assert.deepEqual(conflicts.cases.map(row => row.id), [91, 92]);
await assert.rejects(ensureChatPlan(() => assert.fail('no ambiguous write'), conflicts, () => assert.fail('no ambiguous marker')));
await assert.rejects(inspectChatMetadata(inspectApi([{ ...caseRow, projectId: 2 }])), /identity_invalid/);
await assert.rejects(inspectChatMetadata(async () => ({ content: [], totalPages: 11, totalElements: 1100, number: 0 })), /catalog_incomplete/);
});
test('existing exact plan reuses case and plan without a write', async () => {
const result = await ensureChatPlan(async route => { assert.equal(route, '/api/testplan/8'); return plan; },
await inspectChatMetadata(inspectApi()), () => assert.fail('must not mark or write'));
assert.equal(result.case.id, 91);
assert.equal(result.plan.id, 8);
assert.equal(result.actualResult, null);
});
test('bootstrap creates only the missing exact plan after exclusive marker and reads back scope', async () => {
const calls = [];
const result = await ensureChatPlan(async (route, method, body) => {
calls.push({ route, method, body });
if (method === 'POST') { assert.equal(calls[0].route, 'MARK'); assert.deepEqual(body, { projectId: 1, name: planName, treeSelection: { inverted: false, leafsInclude: [91] } }); }
return plan;
}, { projectId: 1, cases: [caseRow], plans: [] }, async () => calls.push({ route: 'MARK' }));
assert.equal(calls.filter(row => row.method === 'POST').length, 1);
assert.equal(result.plan.count, 1);
});
test('empty project bootstrap links the genuinely created case into only the new plan', async () => {
const calls = [];
const createdCase = { ...caseRow, id: 92 };
const createdPlan = { ...plan, treeSelection: { inverted: false, leafsInclude: [92] } };
const result = await ensureChatPlan(async (route, method, body) => {
calls.push({ route, method, body });
assert.equal(calls[0].route, 'MARK');
if (route === '/api/testcase') {
assert.equal(body.projectId, 1); assert.equal(body.fullName, fullName); assert.equal(body.automated, true);
return createdCase;
}
if (route === '/api/testplan') {
assert.deepEqual(body, { projectId: 1, name: planName, treeSelection: { inverted: false, leafsInclude: [92] } });
return createdPlan;
}
assert.equal(route, '/api/testplan/8'); assert.equal(method, undefined); return createdPlan;
}, { projectId: 1, cases: [], plans: [] }, async () => calls.push({ route: 'MARK' }));
assert.equal(result.case.id, 92); assert.equal(result.plan.id, 8);
assert.deepEqual(calls.map(call => call.route), ['MARK', '/api/testcase', '/api/testplan', '/api/testplan/8']);
});
test('known conflicting plan stops before creating an otherwise missing case or marker', async () => {
await assert.rejects(ensureChatPlan(() => assert.fail('no API write or read needed'),
{ projectId: 1, cases: [], plans: [plan] }, () => assert.fail('must not mark')), /plan_without_matching_case/);
});
test('new case is created only after alias reconciliation; an uncertain mutation is never resent', async () => {
let writes = 0, markers = 0;
await assert.rejects(ensureChatPlan(async (_route, method) => {
assert.equal(method, 'POST'); writes++; throw new Error('uncertain transport');
}, { projectId: 1, cases: [], plans: [] }, async () => { markers++; }), /uncertain/);
assert.equal(writes, 1); assert.equal(markers, 1);
});
test('existing broad or conflicting plan never expands or changes its scope', async () => {
for (const actual of [{ ...plan, testCasesCount: 2 }, { ...plan, treeSelection: { leafsInclude: [92] } },
{ ...plan, treeSelection: { leafsInclude: [91], inverted: true } }, { ...plan, projectId: 2 }]) {
await assert.rejects(ensureChatPlan(async (_route, method) => { assert.equal(method, undefined); return actual; },
{ projectId: 1, cases: [caseRow], plans: [plan] }, () => assert.fail('no writes')), /scope_mismatch/);
}
});
test('API credential stays in memory, redirects fail closed and response text is not an error', async () => {
const calls = [];
const api = await testopsClient('PRIVATE_API_TOKEN', async (url, options) => {
calls.push({ url, options });
return calls.length === 1 ? { ok: true, json: async () => ({ access_token: 'PRIVATE_BEARER' }) }
: { ok: false, status: 403, text: () => assert.fail('must not inspect arbitrary error body') };
});
await assert.rejects(api('/api/testplan?projectId=1'), error => { assert.deepEqual(failureProjection(error, 'metadata_inspection'), { stage: 'metadata_inspection', code: 'testops_api_http', httpStatus: 403 }); return true; });
assert.equal(calls[0].url, `${endpoint}/api/uaa/oauth/token`);
assert.equal(calls[1].options.redirect, 'error');
await assert.rejects(api('https://foreign.invalid'), /route_invalid/);
assert.equal(calls.length, 2);
});
test('unknown exception text is never copied into closed diagnostics', () => {
const value = failureProjection(new Error('PRIVATE_RESPONSE_TOKEN_BODY'), 'untrusted raw stage');
assert.deepEqual(value, { stage: 'unknown', code: 'unknown_failure', httpStatus: null });
assert.ok(!JSON.stringify(value).includes('PRIVATE'));
});
import { validateChatSelection } from './chat-feature-testops-api.mjs';
const publicEnv = { GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/heads/master',
GITHUB_EVENT_NAME: 'workflow_dispatch', GITHUB_RUN_ATTEMPT: '1', GITHUB_SHA: 'a'.repeat(40), GITHUB_RUN_ID: '123',
GITHUB_WORKFLOW_REF: 'paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master',
ALLURE_JOB_RUN_ID: '9', CHAT_TESTOPS_CASE_ID: '91', CHAT_TESTOPS_PLAN_ID: '8', CHAT_TESTOPS_JOB_ID: '7' };
const selectedApi = replacements => async route => {
const values = { '/api/testcase/91': caseRow, '/api/testplan/8': plan,
'/api/testplan/8/job': [{ job: { id: 7 }, testCasesCount: 1 }],
'/api/job/7': { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml' },
'/api/jobrun/9': { id: 9, job: { id: 7, type: 'github' }, stage: 'scheduled', launchId: 10, url: 'https://github.com/paperclipai/paperclip/actions/runs/123' },
'/api/launch/10': { id: 10, projectId: 1, closed: false },
'/api/jobrun/9/plan?expected=true': [{ id: 91, selector: fullName }] };
assert.ok(Object.hasOwn(values, route));
return replacements?.[route] ?? values[route];
};
test('real API selection is exact singleton/project/job/launch and current source bound before model access', async () => {
const selected = await validateChatSelection(selectedApi(), publicEnv);
assert.equal(selected.runId, '123'); assert.equal(selected.workflowSha, 'a'.repeat(40));
assert.equal(selected.executionId, 'chat-features.runner-opencode-deepseek-1017.local.create-backlog');
assert.equal(selected.canonicalScenario, 'tasks.create-backlog'); assert.equal(selected.allureJobRunId, '9');
assert.equal(selected.launchId, 10);
});
test('selection refuses missing correlation, a different case, broad plan, foreign job or closed launch', async () => {
await assert.rejects(validateChatSelection(selectedApi(), { ...publicEnv, ALLURE_JOB_RUN_ID: '' }), /identity_missing/);
for (const [route, value] of [
['/api/testcase/91', { ...caseRow, projectId: 2 }],
['/api/testplan/8', { ...plan, testCasesCount: 2 }],
['/api/job/7', { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip-cloud/actions/workflows/qa-staging-smoke.yml' }],
['/api/testplan/8/job', [{ job: { id: 70 }, testCasesCount: 1 }]],
['/api/jobrun/9', { id: 9, job: { id: 70, type: 'github' }, stage: 'scheduled', launchId: 10 }],
['/api/jobrun/9', { id: 9, job: { id: 7, type: 'github' }, stage: 'scheduled', launchId: 10, url: 'https://github.com/paperclipai/paperclip/actions/runs/456' }],
['/api/launch/10', { id: 10, projectId: 1, closed: true }],
['/api/jobrun/9/plan?expected=true', [{ id: 92, selector: fullName }]],
]) await assert.rejects(validateChatSelection(selectedApi({ [route]: value }), publicEnv), /mismatch/);
});
test('selection rejects every non-master, foreign and second-attempt execution before API access', async () => {
for (const change of [{ GITHUB_REF: 'refs/heads/feature' }, { GITHUB_REPOSITORY: 'foreign' },
{ GITHUB_EVENT_NAME: 'pull_request' }, { GITHUB_RUN_ATTEMPT: '2' }, { GITHUB_WORKFLOW_REF: 'other' }]) {
await assert.rejects(validateChatSelection(() => assert.fail('must stop before API'), { ...publicEnv, ...change }), /untrusted/);
}
});
import { launchChatPlan, bootstrapJobMetadata, admitLaunchPlan } from './chat-feature-testops-api.mjs';
const launchEnv = { ...publicEnv, ALLURE_JOB_RUN_ID: '', CHAT_TESTOPS_OPERATION_PARAMETER_ID: '17', CHAT_TESTOPS_INTEGRATION_ID: '2', CHAT_TESTOPS_LAUNCH_NONCE: '1'.repeat(32), CHAT_TESTOPS_LAUNCH_RUN_NUMBER: '2', GITHUB_RUN_NUMBER: '2',
CHAT_TESTOPS_GITHUB_WORKFLOW_ID: '99', GITHUB_ACTOR_ID: '34892728', TESTOPS_TRIGGERING_ACTOR_ID: '34892728', TESTOPS_ALLOWED_ACTOR_IDS: '[34892728]', GH_TOKEN: 'SYNTHETIC_GH_TOKEN' };
const ownRun = { id: 123, workflow_id: 99, run_attempt: 1, run_number: 2, head_sha: 'a'.repeat(40), head_branch: 'master', event: 'workflow_dispatch',
repository: { full_name: 'paperclipai/paperclip', private: false }, actor: { id: 34892728 }, triggering_actor: { id: 34892728 } };
async function fence(env = launchEnv, changes = {}, workflowChanges = {}, repoChanges = {}) {
return admitLaunchPlan(env, async url => ({ ok: true, text: async () => JSON.stringify(url.endsWith('/repos/paperclipai/paperclip') ? { full_name: 'paperclipai/paperclip', private: false, default_branch: 'master', ...repoChanges }
: url.includes('/actions/runs/') ? { ...ownRun, ...changes }
: { id: 99, path: '.github/workflows/chat-feature-testops.yml', state: 'active', ...workflowChanges }) }));
}
function launchServer({ wrongParameter = false, wrongAssignment = false, uncertainRun = false, integrationChange = {} } = {}) {
const calls = [], parameters = [{ id: 17, value: 'bootstrap' }]; let assigned = false, launched = false, requests = 0;
const api = async (route, method, body) => {
calls.push({ route, method, body });
if (route.startsWith('/api/launch?')) return { totalPages: 1, content: launched ? [{ id: 10, name: `Chat feature evals · ${'1'.repeat(32)}` }] : [] };
if (route === '/api/integration/2/project/1') return { id: 2, projectId: 1, info: { type: 'github' }, disabled: false, ...integrationChange };
if (route === '/api/environment/17') return { id: 17, name: wrongParameter ? 'OTHER_PRIVATE_PARAMETER' : 'operation' };
if (route === '/api/job/7') {
if (method === 'PATCH') { assert.deepEqual(body, { integrationId: 2, canRun: true, parameters: [{ id: 17, value: 'evaluate' }] }); parameters[0].value = 'evaluate'; }
return { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml', parameters };
}
if (route === '/api/testplan/8/job') return wrongAssignment ? [{ job: { id: 70 }, testCasesCount: 1 }]
: assigned ? [{ job: { id: 7, parameters }, testCasesCount: 1 }] : [];
if (route === '/api/testplan/8/assign') { assert.equal(method, 'POST'); assert.deepEqual(body, { jobId: 7, selection: { inverted: false, manual: false, leafsInclude: [91] } }); assigned = true; return null; }
if (route === '/api/testplan/8/jobparameter') { assert.deepEqual(body, { jobId: 7, parameters: [{ id: 17, value: 'evaluate' }] }); return null; }
if (route === '/api/testplan/8/run') { requests++; assert.equal(method, 'POST'); if (uncertainRun) throw new Error('synthetic uncertain transport'); launched = true; return { id: 10, projectId: 1, name: body.launchName }; }
return selectedApi()(route);
};
return { api, calls, requests: () => requests };
}
test('launch configures only the exact registered public job/plan then issues one run request and exits', async () => {
const server = launchServer(); let markers = 0;
const result = await launchChatPlan(server.api, launchEnv, async () => { markers++; assert.equal(server.calls.some(row => row.method), false); }, await fence());
assert.equal(markers, 1); assert.equal(server.requests(), 1); assert.equal(result.status, 'REQUESTED');
assert.equal(result.launchId, 10); assert.equal(result.actualRunId, null); assert.equal(result.actualResult, null);
assert.ok(!server.calls.some(row => /jobrun.*rerun|dispatches/.test(row.route)));
});
test('observed launch nonce prevents a second run request across separate local entry attempts', async () => {
const server = launchServer(); await launchChatPlan(server.api, launchEnv, async () => {}, await fence());
const result = await launchChatPlan(server.api, launchEnv, () => assert.fail('existing launch must not mark or write'), await fence());
assert.equal(result.status, 'EXISTING_NO_RESEND'); assert.equal(server.requests(), 1);
});
test('foreign assignment and unobserved operation parameter stop before any write', async () => {
for (const options of [{ wrongAssignment: true }, { wrongParameter: true }]) {
const server = launchServer(options);
await assert.rejects(launchChatPlan(server.api, launchEnv, () => assert.fail('no marker'), await fence()));
assert.equal(server.calls.some(row => row.method), false); assert.equal(server.requests(), 0);
}
});
test('uncertain plan-run transport is sent once and never followed by another mutation', async () => {
const server = launchServer({ uncertainRun: true });
await assert.rejects(launchChatPlan(server.api, launchEnv, async () => {}, await fence()), /uncertain/);
assert.equal(server.requests(), 1);
assert.equal(server.calls.at(-1).route, '/api/testplan/8/run');
});
test('official bootstrap job metadata projects only fixed parameter names and operation enums', async () => {
const job = await bootstrapJobMetadata(async route => {
if (route === '/api/jobrun/9') return { id: 9, job: { id: 7 }, launchId: 10, url: 'https://github.com/paperclipai/paperclip/actions/runs/123' };
if (route === '/api/job/7') return { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml', parameters: [{ id: 17, value: 'bootstrap' }, { id: 18, value: 'SYNTHETIC_PRIVATE_VALUE' }] };
if (route === '/api/environment/17') return { id: 17, name: 'operation' };
return { id: 18, name: 'UNRELATED_NAME' };
}, publicEnv);
assert.equal(job.publicWorkflowMatches, true); assert.equal(job.parameters[0].value, 'bootstrap');
assert.deepEqual(job.parameters[1], { id: 18, name: 'UNKNOWN', value: 'NOT_PROJECTED' });
assert.ok(!JSON.stringify(job).includes('SYNTHETIC_PRIVATE_VALUE'));
});
test('documented stable-ID list pagination is complete before locally matching exact alias detail metadata', async () => {
const calls = [];
const api = async route => {
calls.push(route);
if (route.startsWith('/api/testcase?')) {
const query = new URLSearchParams(route.split('?')[1]);
assert.equal(query.get('projectId'), '1'); assert.equal(query.get('sort'), 'id,asc');
const number = Number(query.get('page'));
return { number, totalPages: 2, totalElements: 2, content: [{ id: number ? 92 : 91 }] };
}
if (route === '/api/testcase/91') return { ...caseRow, description: 'SYNTHETIC_PRIVATE_DESCRIPTION' };
if (route === '/api/testcase/92') return { id: 92, projectId: 1, fullName: 'paperclip.Product E2E.unrelated.case', automated: true, expectedResult: 'DO_NOT_EXPORT' };
return inspectApi([], [])(route);
};
const result = await inspectChatMetadata(api);
assert.deepEqual(result.cases, [{ id: 91, fullName, automated: true }]);
assert.ok(calls.some(route => route.includes('page=1')));
assert.ok(!calls.some(route => route.includes('__search')));
assert.ok(!JSON.stringify(result).includes('SYNTHETIC_PRIVATE_DESCRIPTION'));
assert.ok(!JSON.stringify(result).includes('DO_NOT_EXPORT'));
});
test('changing catalog totals or repeated page IDs cannot become a not-found result', async () => {
for (const duplicate of [false, true]) {
await assert.rejects(inspectChatMetadata(async route => {
if (route.startsWith('/api/testcase?')) {
const number = Number(new URLSearchParams(route.split('?')[1]).get('page'));
return { number, totalPages: 2, totalElements: number && !duplicate ? 3 : 2, content: [{ id: number && !duplicate ? 92 : 91 }] };
}
return caseRow;
}), /catalog_incomplete/);
}
});
test('catalog detail reads have a finite four-request concurrency bound', async () => {
let active = 0, maximum = 0;
const api = async route => {
if (route.startsWith('/api/testcase?')) return { number: 0, totalPages: 1, totalElements: 7, content: Array.from({ length: 7 }, (_, index) => ({ id: index + 1 })) };
if (/^\/api\/testcase\/[0-9]+$/.test(route)) {
active++; maximum = Math.max(maximum, active); await new Promise(resolve => queueMicrotask(resolve)); active--;
return { id: Number(route.split('/').at(-1)), projectId: 1, fullName: null, automated: false };
}
return inspectApi([], [])(route);
};
assert.deepEqual((await inspectChatMetadata(api)).cases, []);
assert.equal(maximum, 4);
});
test('fresh dispatch or replay cannot acquire the fixed initial launch counter even with stale TestOps search', async () => {
for (const env of [{ ...launchEnv, GITHUB_RUN_NUMBER: '3' }, { ...launchEnv, GITHUB_RUN_ATTEMPT: '2' },
{ ...launchEnv, CHAT_TESTOPS_LAUNCH_RUN_NUMBER: '' }]) {
await assert.rejects(admitLaunchPlan(env, () => assert.fail('counter/refusal precedes all remote reads')));
}
});
test('real own-run metadata rejects counter, source, workflow, actor and default-ref contradictions', async () => {
for (const changes of [{ run_number: 3 }, { head_sha: 'b'.repeat(40) }, { workflow_id: 100 },
{ actor: { id: 1 } }, { triggering_actor: { id: 1 } }, { run_attempt: 2 }, { head_branch: 'feature' }]) {
await assert.rejects(fence(launchEnv, changes), /launch_admission/);
}
await assert.rejects(fence(launchEnv, {}, {}, { default_branch: 'other' }), /launch_admission/);
await assert.rejects(fence(launchEnv, {}, { state: 'disabled_manually' }), /launch_admission/);
});
test('no caller-supplied JSON can replace the source-owned GitHub launch admission receipt', async () => {
const actual = await fence();
await assert.rejects(launchChatPlan(() => assert.fail('no API mutation'), launchEnv, () => assert.fail('no marker'), JSON.parse(JSON.stringify(actual))), /launch_admission/);
});
test('GitHub metadata denial retains only its fixed code and HTTP status', async () => {
await assert.rejects(admitLaunchPlan(launchEnv, async () => ({ ok: false, status: 403, text: () => assert.fail('must not read denial body') })), error => {
assert.deepEqual(failureProjection(error, 'launch_admission'), { stage: 'launch_admission', code: 'chat_testops_github_http', httpStatus: 403 }); return true;
});
});
test('new job receives only the reviewed enabled GitHub integration; disabled/foreign identities stop before writes', async () => {
for (const integrationChange of [{ id: 3 }, { projectId: 2 }, { disabled: true }, { disabled: null }, { info: { type: 'gitlab' } }]) {
const server = launchServer({ integrationChange });
await assert.rejects(launchChatPlan(server.api, launchEnv, () => assert.fail('no marker'), await fence()), /integration_mismatch/);
assert.equal(server.calls.some(row => row.method), false);
}
});
const uploadContext = { GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/heads/master', GITHUB_EVENT_NAME: 'workflow_dispatch',
GITHUB_RUN_ATTEMPT: '1', GITHUB_SHA: 'a'.repeat(40), GITHUB_RUN_ID: '123', ALLURE_JOB_RUN_ID: '9',
GITHUB_WORKFLOW_REF: 'paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master' };
const uploadSelection = { runId: '123', attempt: '1', workflowSha: 'a'.repeat(40), targetSha: 'a'.repeat(40), allureJobRunId: '9',
executionId: 'chat-features.runner-opencode-deepseek-1017.local.create-backlog', canonicalScenario: 'tasks.create-backlog', projectId: 1,
planId: 8, jobId: 7, caseId: 91, fullName, launchId: 10 };
const uploadReceipt = { status: 'passed', cleanup: 'passed', caseId: 91, campaignId: 'gha-123-1', count: 1,
attachmentMetadata: [{ name: 'Synthetic task and source conversation', contentType: 'image/png', contentLength: 400 },
{ name: 'Execution provenance and checks', contentType: 'application/json', contentLength: 300 },
{ name: 'Sanitized Product E2E record and accounting', contentType: 'application/json', contentLength: 800 }] };
const uploadRun = { id: 9, job: { id: 7, type: 'github' }, launchId: 10, url: 'https://github.com/paperclipai/paperclip/actions/runs/123', stage: 'finished' };
function uploadApi(change = {}, absent = 0) {
let polls = 0;
return async (route, method) => {
assert.equal(method, 'GET');
if (route === '/api/jobrun/9') return uploadRun;
if (route === '/api/launch/10') return { id: 10, projectId: 1 };
if (route.startsWith('/api/testresult?')) { polls++; return { content: polls <= absent ? [] : [{ id: 45, testCaseId: 91 }], totalElements: polls <= absent ? 0 : 1, totalPages: polls <= absent ? 0 : 1 }; }
if (route === '/api/testresult/45') return { id: 45, projectId: 1, launchId: 10, testCaseId: 91, fullName, jobRun: uploadRun, status: 'passed',
links: [{ url: uploadRun.url }, { url: `https://github.com/paperclipai/paperclip/commit/${'a'.repeat(40)}` }],
get trace() { assert.fail('must not inspect trace'); }, ...change };
if (route.startsWith('/api/testresult/attachment?')) return { content: uploadReceipt.attachmentMetadata.map((row, i) => ({ ...row, id: i + 1, missed: false, fromTestCase: false })), totalElements: 3, totalPages: 1 };
assert.fail(route);
};
}
test('readback verifies exact current launch/case/source/status and safe attachment metadata after processing lag', async () => {
const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs');
let waits = 0;
const result = await verifyChatUpload(uploadApi({}, 1), uploadContext, uploadSelection, uploadReceipt, { wait: async () => { waits++; }, now: () => 0 });
assert.equal(waits, 1); assert.equal(result.verification, 'VERIFIED'); assert.equal(result.resultId, 45);
assert.equal(result.status, 'passed'); assert.deepEqual(result.attachments, uploadReceipt.attachmentMetadata);
assert.ok(!JSON.stringify(result).includes('trace'));
});
test('readback rejects wrong launch, case, current jobrun, source and contradictory result status', async () => {
const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs');
for (const change of [{ launchId: 11 }, { testCaseId: 92 }, { jobRun: { ...uploadRun, id: 11 } }, { links: [] }, { status: 'failed' }]) {
await assert.rejects(verifyChatUpload(uploadApi(change), uploadContext, uploadSelection, uploadReceipt), /upload_mismatch/);
}
});
test('readback missing result or screenshot metadata is bounded unverified, never a fabricated pass', async () => {
const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs');
let waits = 0;
await assert.rejects(verifyChatUpload(uploadApi({}, 100), uploadContext, uploadSelection, uploadReceipt,
{ wait: async () => { waits++; }, now: () => 0 }), /upload_unverified/);
assert.equal(waits, 11);
const base = uploadApi();
await assert.rejects(verifyChatUpload(async (route, ...args) => route.startsWith('/api/testresult/attachment?')
? { content: [], totalElements: 0, totalPages: 0 } : base(route, ...args), uploadContext, uploadSelection, uploadReceipt,
{ wait: async () => {}, now: () => 0 }), /upload_unverified/);
});
test('verified failed Product E2E remains failed and cleanup cannot be promoted', async () => {
const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs');
const result = await verifyChatUpload(uploadApi({ status: 'failed' }), uploadContext, uploadSelection, { ...uploadReceipt, status: 'failed', cleanup: 'failed' });
assert.equal(result.status, 'failed'); assert.equal(result.cleanup, 'failed');
await assert.rejects(verifyChatUpload(uploadApi(), uploadContext, uploadSelection, { ...uploadReceipt, cleanup: 'failed' }), /upload_mismatch/);
});
test('readback stops immediately after a late GET and gives each client only the remaining deadline', async () => {
const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs');
let time = 0, calls = 0;
await assert.rejects(verifyChatUpload(async (route, method, body, timeout) => {
calls++; assert.equal(route, '/api/jobrun/9'); assert.equal(timeout, 30_000); time = 120_001; return uploadRun;
}, uploadContext, uploadSelection, uploadReceipt, { now: () => time }), /upload_unverified/);
assert.equal(calls, 1);
time = 0; calls = 0;
await assert.rejects(verifyChatUpload(async (route, method, body, timeout) => {
calls++;
if (calls === 1) { assert.equal(timeout, 30_000); time = 119_990; return uploadRun; }
assert.equal(route, '/api/launch/10'); assert.equal(timeout, 10); time = 120_001; return { id: 10, projectId: 1 };
}, uploadContext, uploadSelection, uploadReceipt, { now: () => time }), /upload_unverified/);
assert.equal(calls, 2);
});
@@ -0,0 +1,99 @@
import { lstat, mkdir, open, readFile, writeFile } from 'node:fs/promises';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
import { publicContext, testopsClient, inspectChatMetadata, ensureChatPlan, validateChatSelection, bootstrapJobMetadata, launchChatPlan, admitLaunchPlan, verifyChatUpload, failureProjection } from './chat-feature-testops-api.mjs';
import { exportChatAllure, readBoundChatExport, ExportFailure } from './chat-feature-testops-results.mjs';
export async function run(env = process.env, fetchImpl = fetch) {
let stage = 'context', stateRoot;
try {
const expected = publicContext(env), operation = env.CHAT_TESTOPS_OPERATION;
if (!['bootstrap', 'launch-plan', 'validate-selection', 'export', 'verify-upload'].includes(operation)) throw new Error('unknown_operation');
if (operation === 'bootstrap' || operation === 'launch-plan') {
const eventPath = env.GITHUB_EVENT_PATH;
if (!eventPath) throw new Error('bootstrap_original_event_missing');
const stat = await lstat(eventPath);
if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 65_536) throw new Error('bootstrap_original_event_invalid');
const event = JSON.parse(await readFile(eventPath, 'utf8'));
if (event.inputs?.operation !== operation || (event.inputs?.ALLURE_JOB_RUN_ID ?? '') !== '') throw new Error('bootstrap_cannot_run_from_testops');
}
stage = 'output_admission';
const requested = env.CHAT_TESTOPS_STATE_ROOT;
if (!env.RUNNER_TEMP || !requested || !path.isAbsolute(requested)
|| path.dirname(path.resolve(requested)) !== path.resolve(env.RUNNER_TEMP)) throw new Error('invalid_output');
await mkdir(requested, { mode: 0o700 }); stateRoot = requested;
let result;
if (operation === 'export' || operation === 'verify-upload') {
stage = 'result_export';
const selectionPath = env.CHAT_TESTOPS_SELECTION;
if (!selectionPath || !path.isAbsolute(selectionPath) || !selectionPath.startsWith(`${path.resolve(env.RUNNER_TEMP)}${path.sep}`)) throw new Error('invalid_selection_path');
const stat = await lstat(selectionPath);
if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 16_384) throw new Error('invalid_selection_file');
const selected = JSON.parse(await readFile(selectionPath, 'utf8'));
if (operation === 'verify-upload') {
stage = 'upload_verification';
const receiptRoot = path.join(env.RUNNER_TEMP, 'chat-testops-upload-state');
const rootStat = await lstat(receiptRoot), file = path.join(receiptRoot, 'export.json');
const receiptStat = await lstat(file);
if (!rootStat.isDirectory() || rootStat.isSymbolicLink() || !receiptStat.isFile() || receiptStat.isSymbolicLink()
|| receiptStat.nlink !== 1 || receiptStat.size > 16_384) throw new Error('invalid_export_receipt');
const exported = JSON.parse(await readFile(file, 'utf8'));
const bound = await readBoundChatExport(env.CHAT_TESTOPS_EXPORT_ROOT, expected);
if (exported.status !== (bound.result.status === 'passed' && bound.result.cleanup === 'passed' ? 'passed' : 'failed')
|| exported.cleanup !== bound.result.cleanup) throw new Error('export_verdict_mismatch');
const api = await testopsClient(env.ALLURE_TOKEN, fetchImpl);
result = await verifyChatUpload(api, env, selected, exported);
await writeFile(path.join(stateRoot, 'verification.json'), JSON.stringify(result, null, 2), { flag: 'wx', mode: 0o600 });
} else {
result = await exportChatAllure({ root: env.CHAT_TESTOPS_EXPORT_ROOT, output: env.ALLURE_RESULTS, expected, selected });
await writeFile(path.join(stateRoot, 'export.json'), JSON.stringify(result, null, 2), { flag: 'wx', mode: 0o600 });
}
} else {
let launchAdmission;
if (operation === 'launch-plan') { stage = 'launch_admission'; launchAdmission = await admitLaunchPlan(env, fetchImpl); }
stage = 'authentication';
const api = await testopsClient(env.ALLURE_TOKEN, fetchImpl);
if (operation === 'launch-plan') {
stage = 'launch_preflight';
result = await launchChatPlan(api, env, async identity => {
const handle = await open(path.join(stateRoot, 'once.json'), 'wx', 0o600);
await handle.writeFile(JSON.stringify({ ...identity, status: 'STARTED_NO_AUTOMATIC_RESEND' }));
await handle.close(); stage = 'plan_launch';
}, launchAdmission);
await writeFile(path.join(stateRoot, 'launch.json'), JSON.stringify(result, null, 2), { flag: 'wx', mode: 0o600 });
} else if (operation === 'validate-selection') {
stage = 'selection_validation';
result = await validateChatSelection(api, env);
await writeFile(path.join(stateRoot, 'selection.json'), JSON.stringify(result, null, 2), { flag: 'wx', mode: 0o600 });
} else {
stage = 'metadata_inspection';
const inspected = await inspectChatMetadata(api);
await writeFile(path.join(stateRoot, 'metadata.json'), JSON.stringify(inspected, null, 2), { flag: 'wx', mode: 0o600 });
let marked = false;
const once = async () => {
if (marked) return;
const handle = await open(path.join(stateRoot, 'once.json'), 'wx', 0o600);
await handle.writeFile(JSON.stringify({ ...expected, operation: 'plan_bootstrap', status: 'STARTED_NO_AUTOMATIC_RESEND' }));
await handle.close(); marked = true;
};
stage = 'plan_bootstrap';
result = await ensureChatPlan(api, inspected, once);
await writeFile(path.join(stateRoot, 'plan.json'), JSON.stringify(result, null, 2), { flag: 'wx', mode: 0o600 });
stage = 'bootstrap_job_metadata';
const job = await bootstrapJobMetadata(api, env);
await writeFile(path.join(stateRoot, 'job.json'), JSON.stringify(job, null, 2), { flag: 'wx', mode: 0o600 });
}
}
return operation === 'export' && result.status !== 'passed' || operation === 'launch-plan' && result.status !== 'REQUESTED' ? 1 : 0;
} catch (error) {
const safe = failureProjection(error, stage);
if (error instanceof ExportFailure) safe.code = 'chat_testops_export_rejected';
if (stateRoot) await writeFile(path.join(stateRoot, 'failure.json'), JSON.stringify(safe, null, 2), { flag: 'wx', mode: 0o600 }).catch(() => {});
process.stderr.write(`Chat TestOps stopped: ${JSON.stringify(safe)}\n`);
return 1;
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
run().then(code => { process.exitCode = code; }).catch(() => { process.stderr.write('Chat TestOps stopped: unknown failure\n'); process.exitCode = 1; });
}
@@ -0,0 +1,67 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { mkdtemp, readFile, realpath, rm, writeFile } from 'node:fs/promises';
import path from 'node:path';
import { tmpdir } from 'node:os';
import { run } from './chat-feature-testops-cli.mjs';
import { fullName, planName } from './chat-feature-testops-api.mjs';
const context = { GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/heads/master', GITHUB_EVENT_NAME: 'workflow_dispatch',
GITHUB_RUN_ATTEMPT: '1', GITHUB_SHA: 'a'.repeat(40), GITHUB_RUN_ID: '123',
GITHUB_WORKFLOW_REF: 'paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master' };
test('actual metadata bootstrap caller creates no result or agent and reuses exact case/plan', async t => {
const temporary = await realpath(await mkdtemp(path.join(tmpdir(), 'testops-bootstrap-')));
t.after(() => rm(temporary, { recursive: true }));
const state = path.join(temporary, 'state'), requests = [];
const eventPath = path.join(temporary, 'event.json'); await writeFile(eventPath, JSON.stringify({ inputs: { operation: 'bootstrap', ALLURE_JOB_RUN_ID: '' } }));
const response = value => ({ ok: true, status: 200, json: async () => value, text: async () => JSON.stringify(value) });
const fetchImpl = async (url, options) => {
requests.push({ url, method: options.method });
if (url.endsWith('/api/uaa/oauth/token')) return response({ access_token: 'SYNTHETIC_BEARER' });
const route = new URL(url).pathname;
if (route === '/api/testcase') return response({ number: 0, totalPages: 1, totalElements: 1, content: [{ id: 91, name: 'Not projected', automated: true }] });
if (route === '/api/testcase/91') return response({ id: 91, projectId: 1, fullName, automated: true });
if (route === '/api/testplan') return response({ totalPages: 1, content: [{ id: 8, projectId: 1, name: planName, testCasesCount: 1 }] });
if (route === '/api/integration/project/1') return response({ totalPages: 1, content: [] });
if (route === '/api/jobrun/9') return response({ id: 9, job: { id: 7 }, launchId: 10, url: 'https://github.com/paperclipai/paperclip/actions/runs/123' });
if (route === '/api/job/7') return response({ id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml', parameters: [{ id: 17, value: 'bootstrap' }] });
if (route === '/api/environment/17') return response({ id: 17, name: 'operation' });
assert.equal(route, '/api/testplan/8');
return response({ id: 8, projectId: 1, name: planName, testCasesCount: 1, treeSelection: { leafsInclude: [91], inverted: false } });
};
assert.equal(await run({ ...context, RUNNER_TEMP: temporary, CHAT_TESTOPS_STATE_ROOT: state,
CHAT_TESTOPS_OPERATION: 'bootstrap', GITHUB_EVENT_PATH: eventPath, ALLURE_JOB_RUN_ID: '9', ALLURE_TOKEN: 'SYNTHETIC_API_TOKEN' }, fetchImpl), 0);
const plan = JSON.parse(await readFile(path.join(state, 'plan.json')));
assert.equal(plan.actualResult, null);
assert.equal(requests.filter(value => value.method === 'POST').length, 1, 'only auth token exchange; no case/plan write');
await assert.rejects(readFile(path.join(state, 'allure-results.json')), { code: 'ENOENT' });
});
test('actual public caller retains only safe auth status on failure before discovery', async t => {
const temporary = await realpath(await mkdtemp(path.join(tmpdir(), 'testops-failure-')));
t.after(() => rm(temporary, { recursive: true }));
const messages = [], original = process.stderr.write;
process.stderr.write = value => { messages.push(String(value)); return true; };
try {
const state = path.join(temporary, 'state');
const eventPath = path.join(temporary, 'event.json'); await writeFile(eventPath, JSON.stringify({ inputs: { operation: 'bootstrap' } }));
assert.equal(await run({ ...context, RUNNER_TEMP: temporary, CHAT_TESTOPS_STATE_ROOT: state,
CHAT_TESTOPS_OPERATION: 'bootstrap', GITHUB_EVENT_PATH: eventPath, ALLURE_TOKEN: 'SYNTHETIC_PRIVATE_TOKEN' },
async () => ({ ok: false, status: 403, text: () => assert.fail('must not read denial body') })), 1);
assert.deepEqual(JSON.parse(await readFile(path.join(state, 'failure.json'))), { stage: 'authentication', code: 'testops_auth_http', httpStatus: 403 });
assert.ok(!messages.join('').includes('SYNTHETIC_PRIVATE_TOKEN'));
} finally { process.stderr.write = original; }
});
test('original TestOps-triggered bootstrap is rejected even if its official environment is populated', async t => {
const temporary = await realpath(await mkdtemp(path.join(tmpdir(), 'testops-wrong-bootstrap-')));
t.after(() => rm(temporary, { recursive: true }));
const event = path.join(temporary, 'event.json'); await writeFile(event, JSON.stringify({ inputs: { operation: 'bootstrap', ALLURE_JOB_RUN_ID: '9' } }));
const original = process.stderr.write;
process.stderr.write = () => true;
try {
assert.equal(await run({ ...context, GITHUB_EVENT_PATH: event, RUNNER_TEMP: temporary, CHAT_TESTOPS_STATE_ROOT: path.join(temporary, 'state'),
CHAT_TESTOPS_OPERATION: 'bootstrap', ALLURE_JOB_RUN_ID: '9' }, () => assert.fail('no API access')), 1);
} finally { process.stderr.write = original; }
});
@@ -0,0 +1,134 @@
import { createHash, randomUUID } from 'node:crypto';
import { lstat, mkdir, readFile, readdir, realpath, writeFile } from 'node:fs/promises';
import path from 'node:path';
import { aliases, executionId } from './chat-feature-testops-api.mjs';
export class ExportFailure extends Error {}
const fail = () => { throw new ExportFailure('chat_testops_export_rejected'); };
const hash = bytes => createHash('sha256').update(bytes).digest('hex');
const pngMagic = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]);
const model = 'openrouter/deepseek/deepseek-v4.1-flash-1017';
const pngPath = new RegExp(`^evidence/${executionId.replaceAll('.', '\\.')}/attempt-1/[A-Za-z0-9][A-Za-z0-9._-]{0,199}\\.png$`);
async function inventory(root, relative = '') {
const output = [];
const entries = await readdir(path.join(root, relative));
if (entries.length > 12) fail();
for (const name of entries) {
if (!/^[A-Za-z0-9._-]+$/.test(name) || name === '.' || name === '..') fail();
const file = relative ? `${relative}/${name}` : name;
const stat = await lstat(path.join(root, file));
if (stat.isSymbolicLink()) fail();
if (stat.isDirectory()) {
if (!['evidence', `evidence/${executionId}`, `evidence/${executionId}/attempt-1`].includes(file)) fail();
output.push(...await inventory(root, file));
}
else if (stat.isFile() && stat.nlink === 1 && stat.size <= 12 * 1024 * 1024) output.push(file);
else fail();
if (output.length > 12) fail();
}
return output;
}
export async function readBoundChatExport(root, expected) {
if (!/^[1-9][0-9]{0,19}$/.test(expected.runId ?? '') || expected.attempt !== '1'
|| !/^[a-f0-9]{40}$/.test(expected.workflowSha ?? '') || expected.targetSha !== expected.workflowSha) fail();
const stat = await lstat(root);
if (!stat.isDirectory() || stat.isSymbolicLink() || await realpath(root) !== path.resolve(root)) fail();
const files = await inventory(root);
if (!files.includes('binding.json')) fail();
const bindingBytes = await readFile(path.join(root, 'binding.json'));
if (bindingBytes.length > 128_000) fail();
const binding = JSON.parse(bindingBytes);
if (binding.schema !== 'paperclip.chat-feature-testops-export/v1' || binding.attempt !== '1'
|| binding.runId !== expected.runId || binding.workflowSha !== expected.workflowSha || binding.targetSha !== expected.targetSha
|| binding.executionId !== executionId || binding.campaignId !== `gha-${expected.runId}-1`
|| !Array.isArray(binding.files) || binding.files.length < 2 || binding.files.length > 10) fail();
const names = binding.files.map(row => row.path);
if (new Set(names).size !== names.length || !names.includes('normalized-results.json') || !names.includes('junit.xml')
|| names.some(name => !['normalized-results.json', 'junit.xml'].includes(name) && !pngPath.test(name))
|| JSON.stringify([...names, 'binding.json'].sort()) !== JSON.stringify(files.sort())) fail();
const retained = new Map();
let total = 0;
for (const row of binding.files) {
if (!/^[a-f0-9]{64}$/.test(row.sha256 ?? '') || !Number.isSafeInteger(row.bytes) || row.bytes < 0) fail();
const bytes = await readFile(path.join(root, row.path));
total += bytes.length;
if (total > 32 * 1024 * 1024 || bytes.length !== row.bytes || hash(bytes) !== row.sha256) fail();
retained.set(row.path, bytes);
}
const bytes = retained.get('normalized-results.json');
if (bytes.length > 2_000_000) fail();
const campaign = JSON.parse(bytes);
if (campaign.schema !== 'paperclip.runner-e2e.campaign/v2' || campaign.campaignId !== binding.campaignId
|| !Array.isArray(campaign.results) || campaign.results.length !== 1) fail();
const result = campaign.results[0];
if (result.executionId !== executionId || result.suiteId !== 'chat-features' || result.caseId !== 'create-backlog'
|| result.profileId !== 'runner-opencode-deepseek-1017' || result.provider !== 'opencode' || result.environmentId !== 'local'
|| result.model !== model || result.runtimeMode !== 'native' || result.attempt !== 1 || result.source?.sha !== expected.targetSha
|| !['passed', 'failed'].includes(result.status) || result.status !== binding.resultStatus
|| !['passed', 'failed', 'not_started'].includes(result.cleanup) || result.cleanup !== binding.cleanup
|| !Array.isArray(result.matcherResults) || result.matcherResults.length > 200 || !Array.isArray(result.screenshots) || result.screenshots.length > 8) fail();
for (const check of result.matcherResults) if (check.matcher?.kind !== 'json_path' || !/^[A-Za-z0-9._-]{1,150}$/.test(check.matcher.path)
|| check.matcher.expected !== true || typeof check.passed !== 'boolean') fail();
for (const shot of result.screenshots) {
const name = `evidence/${executionId}/attempt-1/${shot.file}`;
const png = retained.get(name);
if (!png || !png.subarray(0, 8).equals(pngMagic) || png.subarray(12, 16).toString('ascii') !== 'IHDR' || hash(png) !== shot.sha256) fail();
}
if (names.filter(name => pngPath.test(name)).length !== result.screenshots.length) fail();
if (result.status === 'passed' && (result.cleanup !== 'passed' || result.matcherResults.length === 0
|| result.matcherResults.some(check => !check.passed) || result.runIds?.length !== 1
|| !result.screenshots.some(shot => shot.id === 'final-state' && shot.file === 'final-state.png'))) fail();
return { binding, result, retained };
}
export async function exportChatAllure({ root, output, expected, selected }) {
if (selected.projectId !== 1 || !Number.isSafeInteger(selected.caseId) || selected.caseId < 1 || !aliases.includes(selected.fullName)
|| selected.runId !== expected.runId || selected.attempt !== '1' || selected.workflowSha !== expected.workflowSha || selected.targetSha !== expected.targetSha
|| selected.executionId !== executionId) fail();
const { binding, result, retained } = await readBoundChatExport(root, expected);
await mkdir(output, { mode: 0o700 }).catch(error => { if (error.code !== 'EEXIST') throw error; });
const outputStat = await lstat(output);
if (!outputStat.isDirectory() || outputStat.isSymbolicLink() || (process.getuid && outputStat.uid !== process.getuid()) || (await readdir(output)).length !== 0) fail();
const status = result.status === 'passed' && result.cleanup === 'passed' ? 'passed' : 'failed';
const attachments = [];
const attachmentMetadata = [];
for (const shot of result.screenshots) {
const source = `${randomUUID()}-attachment.png`;
await writeFile(path.join(output, source), retained.get(`evidence/${executionId}/attempt-1/${shot.file}`), { flag: 'wx', mode: 0o600 });
attachments.push({ name: 'Synthetic task and source conversation', source, type: 'image/png' });
attachmentMetadata.push({ name: 'Synthetic task and source conversation', contentType: 'image/png', contentLength: retained.get(`evidence/${executionId}/attempt-1/${shot.file}`).length });
}
const projection = { campaignId: binding.campaignId, executionId, runId: expected.runId, attempt: 1,
workflowSha: expected.workflowSha, sourceSha: expected.targetSha, status: result.status, cleanup: result.cleanup,
profile: result.profileId, model: result.model, environment: result.environmentId, runtime: result.runtimeMode,
durationMs: Number.isFinite(result.durationMs) ? result.durationMs : null,
failureClass: typeof result.failureClass === 'string' && /^[a-z_]{1,80}$/.test(result.failureClass) ? result.failureClass : null,
billingComplete: typeof result.billing?.complete === 'boolean' ? result.billing.complete : null,
checks: result.matcherResults.map(check => ({ path: check.matcher.path, passed: check.passed })),
note: 'Setup usage and AWS billing are not qualified by this result; missing accounting remains unknown.' };
const source = `${randomUUID()}-attachment.json`;
const projectionBytes = Buffer.from(JSON.stringify(projection, null, 2));
await writeFile(path.join(output, source), projectionBytes, { flag: 'wx', mode: 0o600 });
attachmentMetadata.push({ name: 'Execution provenance and checks', contentType: 'application/json', contentLength: projectionBytes.length });
attachments.push({ name: 'Execution provenance and checks', source, type: 'application/json' });
const normalizedSource = `${randomUUID()}-attachment.json`;
await writeFile(path.join(output, normalizedSource), retained.get('normalized-results.json'), { flag: 'wx', mode: 0o600 });
attachments.push({ name: 'Sanitized Product E2E record and accounting', source: normalizedSource, type: 'application/json' });
attachmentMetadata.push({ name: 'Sanitized Product E2E record and accounting', contentType: 'application/json', contentLength: retained.get('normalized-results.json').length });
const value = { uuid: randomUUID(), name: 'Create a task backlog through chat', fullName: selected.fullName,
testCaseId: hash(`${JSON.stringify(selected.fullName)}\n`),
historyId: hash(JSON.stringify([selected.fullName, result.profileId, result.model, result.environmentId])), status, stage: 'finished',
statusDetails: { message: status === 'passed' ? 'Real Product E2E assertions and cleanup passed' : 'Real Product E2E evaluation or cleanup failed; original verdict preserved' },
labels: [{ name: 'AS_ID', value: String(selected.caseId) }, { name: 'parentSuite', value: 'Product E2E' },
{ name: 'suite', value: 'chat-features' }, { name: 'area', value: 'Chat and collaboration' },
{ name: 'framework', value: 'Paperclip Evals' }, { name: 'tag', value: 'chat-feature-testops' }],
parameters: [{ name: 'canonicalScenario', value: 'tasks.create-backlog' }, { name: 'model', value: model }, { name: 'environment', value: 'GitHub Actions / AWS' }],
links: [{ name: 'GitHub Actions run', type: 'custom', url: `https://github.com/paperclipai/paperclip/actions/runs/${expected.runId}` },
{ name: 'Evaluated source', type: 'custom', url: `https://github.com/paperclipai/paperclip/commit/${expected.targetSha}` }],
steps: result.matcherResults.map(check => ({ name: check.matcher.path, status: check.passed ? 'passed' : 'failed', stage: 'finished' })), attachments };
for (const [key, time] of [['start', result.startedAt], ['stop', result.finishedAt]]) if (Number.isFinite(Date.parse(time))) value[key] = Date.parse(time);
await writeFile(path.join(output, `${value.uuid}-result.json`), JSON.stringify(value), { flag: 'wx', mode: 0o600 });
return { status, cleanup: result.cleanup, caseId: selected.caseId, campaignId: binding.campaignId, count: 1, attachmentMetadata };
}
@@ -0,0 +1,73 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
import { mkdtemp, mkdir, readFile, readdir, realpath, rm, symlink, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { exportChatAllure, readBoundChatExport } from './chat-feature-testops-results.mjs';
import { executionId, fullName } from './chat-feature-testops-api.mjs';
const hash = bytes => createHash('sha256').update(bytes).digest('hex');
const expected = { runId: '123', attempt: '1', workflowSha: 'a'.repeat(40), targetSha: 'a'.repeat(40) };
const selected = { ...expected, projectId: 1, caseId: 91, fullName, executionId };
async function fixture(t, changes = {}) {
const directory = await realpath(await mkdtemp(path.join(tmpdir(), 'chat-testops-export-')));
t.after(() => rm(directory, { recursive: true }));
const root = path.join(directory, 'export'); await mkdir(root);
const png = Buffer.from([137,80,78,71,13,10,26,10,0,0,0,13,73,72,68,82]);
const shot = { id: 'final-state', file: 'final-state.png', sha256: hash(png) };
const result = { executionId, suiteId: 'chat-features', caseId: 'create-backlog', profileId: 'runner-opencode-deepseek-1017', provider: 'opencode',
environmentId: 'local', model: 'openrouter/deepseek/deepseek-v4.1-flash-1017', runtimeMode: 'native', attempt: 1,
source: { sha: expected.targetSha }, status: 'passed', cleanup: 'passed', runIds: ['synthetic-run'], durationMs: 1234,
matcherResults: [{ matcher: { kind: 'json_path', path: 'chatFeature.createdTask', expected: true }, passed: true }], screenshots: [shot], ...changes };
const files = new Map([['normalized-results.json', Buffer.from(JSON.stringify({ schema: 'paperclip.runner-e2e.campaign/v2', campaignId: 'gha-123-1', complete: false, results: [result] }))],
['junit.xml', Buffer.from('<testsuite tests="1"/>')], [`evidence/${executionId}/attempt-1/final-state.png`, png]]);
const binding = { schema: 'paperclip.chat-feature-testops-export/v1', ...expected, executionId, campaignId: 'gha-123-1', resultStatus: result.status, cleanup: result.cleanup,
files: [...files].map(([name, bytes]) => ({ path: name, sha256: hash(bytes), bytes: bytes.length })) };
for (const [name, bytes] of files) { await mkdir(path.dirname(path.join(root, name)), { recursive: true }); await writeFile(path.join(root, name), bytes); }
await writeFile(path.join(root, 'binding.json'), JSON.stringify(binding));
return { directory, root, output: path.join(directory, 'allure'), result, binding };
}
test('bound synthetic selected result maps to existing Allure case, independent of whole-suite complete', async t => {
const f = await fixture(t);
const value = await exportChatAllure({ ...f, expected, selected });
assert.equal(value.status, 'passed'); assert.equal(value.count, 1);
const name = (await readdir(f.output)).find(name => name.endsWith('-result.json'));
const result = JSON.parse(await readFile(path.join(f.output, name)));
assert.equal(result.fullName, fullName); assert.deepEqual(result.labels[0], { name: 'AS_ID', value: '91' });
assert.equal(result.attachments.filter(row => row.type === 'image/png').length, 1);
});
test('real failed outcome stays failed and no absence can synthesize a pass', async t => {
const f = await fixture(t, { status: 'failed', cleanup: 'failed' });
assert.equal((await exportChatAllure({ ...f, expected, selected })).status, 'failed');
await assert.rejects(exportChatAllure({ root: path.join(f.directory, 'absent'), output: path.join(f.directory, 'missing'), expected, selected }));
});
test('foreign source or selection and tampered file hash reject', async t => {
const f = await fixture(t);
await assert.rejects(readBoundChatExport(f.root, { ...expected, runId: '456' }));
await assert.rejects(exportChatAllure({ ...f, expected, selected: { ...selected, projectId: 2 } }));
await writeFile(path.join(f.root, 'junit.xml'), 'tampered');
await assert.rejects(readBoundChatExport(f.root, expected));
});
test('an unmanifested file, unknown directory or symlink is rejected before upload', async t => {
const f = await fixture(t);
await writeFile(path.join(f.root, 'provider-raw.txt'), 'SYNTHETIC_PRIVATE_NEVER_READ');
await assert.rejects(readBoundChatExport(f.root, expected));
await rm(path.join(f.root, 'provider-raw.txt'));
await mkdir(path.join(f.root, 'unknown-empty'));
await assert.rejects(readBoundChatExport(f.root, expected));
await rm(path.join(f.root, 'unknown-empty'), { recursive: true });
await rm(path.join(f.root, 'junit.xml')); await symlink(path.join(f.root, 'normalized-results.json'), path.join(f.root, 'junit.xml'));
await assert.rejects(readBoundChatExport(f.root, expected));
});
test('passing result cannot omit independent checks, cleanup or actual feature capture', async t => {
for (const changes of [{ matcherResults: [] }, { cleanup: 'failed' }, { runIds: [] }, { screenshots: [] },
{ matcherResults: [{ matcher: { kind: 'json_path', path: 'chatFeature.createdTask', expected: true }, passed: false }] }]) {
const f = await fixture(t, changes);
await assert.rejects(exportChatAllure({ ...f, expected, selected }));
}
});
@@ -101,7 +101,7 @@ export type PrivateCellLayout = "original" | "final" | "final-wrapper";
export async function privateCellEvidenceOwner(context: {
private: true; root: string; runId: string; attempt: string; executionIds: string[]; cellReportProducer?: CellReportProducerIdentity;
}, executionId: string): Promise<PrivateCellEvidenceOwner> {
if (context.private !== true || !context.executionIds.includes(executionId) || !/^chat-features\.runner-opencode\.local\.[a-z0-9][a-z0-9.-]{0,159}$/.test(executionId)
if (context.private !== true || !context.executionIds.includes(executionId) || (!/^chat-features\.runner-opencode\.local\.[a-z0-9][a-z0-9.-]{0,159}$/.test(executionId) && executionId !== "chat-features.runner-opencode-deepseek-1017.local.create-backlog")
|| executionId.includes("..") || !/^[1-9][0-9]{0,19}$/.test(context.runId) || context.attempt !== "1" || !context.cellReportProducer)
throw new Error("Invalid private cell report owner");
const root = await realpath(context.root), metadata = await lstat(context.root);
@@ -116,8 +116,9 @@ export function cellTraceAsset(relative: string, owner: PrivateCellEvidenceOwner
const segments = relative.split("/");
const reserved = segments.some((part, index) => part === "html-report" && segments[index + 1] === "trace");
if (!reserved) return { reserved: false };
const scenario = owner.executionId.slice("chat-features.runner-opencode.local.".length);
const final = `gha-${owner.runId}-${owner.attempt}-${owner.executionId}/chat-features/runner-opencode/local/${scenario}/attempt-${owner.attempt}/html-report/trace/`;
const profile = owner.executionId === "chat-features.runner-opencode-deepseek-1017.local.create-backlog" ? "runner-opencode-deepseek-1017" : "runner-opencode";
const scenario = owner.executionId.slice(`chat-features.${profile}.local.`.length);
const final = `gha-${owner.runId}-${owner.attempt}-${owner.executionId}/chat-features/${profile}/local/${scenario}/attempt-${owner.attempt}/html-report/trace/`;
const prefix = layout === "original" ? "html-report/trace/" : layout === "final" ? final : `html-report/${final}`;
const canonical = relative.startsWith(prefix) ? canonicalGeneratedReportAsset(`html/trace/${relative.slice(prefix.length)}`) : undefined;
return { reserved: true, canonical };
+1 -1
View File
@@ -300,7 +300,7 @@ export function manualPrivateSessionPolicy(input: { bucket: string; runId: strin
if (!/^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/.test(input.bucket) || !/^[1-9][0-9]{0,19}$/.test(input.runId))
throw new Error("Invalid manual private storage identity");
const ids = canonicalPrivateManualIds(input.executionIds);
if (ids.some(id => !MANUAL_PRIVATE_EXECUTIONS.includes(id)) || input.mode !== "report-write" && ids.length !== 1)
if (ids.some(id => !MANUAL_PRIVATE_EXECUTIONS.includes(id) && id !== "chat-features.runner-opencode-deepseek-1017.local.create-backlog") || input.mode !== "report-write" && ids.length !== 1)
throw new Error("Private archive sessions require one exact cell or the report writer");
const prefix = `arn:aws:s3:::${input.bucket}/runner-e2e-private/bounded-v1/campaigns/gha-${input.runId}-1/`;
const cells = ids.map(id => `${prefix}${manualCellKind(id)}.zip`);
+2 -1
View File
@@ -10,11 +10,12 @@ import { isStandardEvidenceFile } from "./evidence.js";
import { BoundedHistoryPublisher } from "./history-budget.js";
import { privateArtifactKey, retainPrivateTransfer, transferPrivateArtifact, verifyPrivateHistory, type PrivateEvidenceCompleteness } from "./private-history.js";
import { authorizeManualPrivateEvidence, MANUAL_PRIVATE_PROTOCOL, manualCellKind, manualPrivateSessionPolicy } from "./manual-private-retention.js";
import { shippedChatEvidenceAuthority } from "./testops-chat-admission.js";
const hash = (bytes: string | Buffer) => createHash("sha256").update(bytes).digest("hex");
function request(env: NodeJS.ProcessEnv) {
const ids = JSON.parse(env.MANUAL_PRIVATE_IDS ?? "null") as string[];
const approval = authorizeManualPrivateEvidence({ workflowSha: env.GITHUB_SHA ?? "", targetSha: env.TARGET_SHA ?? "",
const approval = env.PAPERCLIP_TESTOPS_CHAT_MAIN_ADMISSION !== undefined ? shippedChatEvidenceAuthority(env) : authorizeManualPrivateEvidence({ workflowSha: env.GITHUB_SHA ?? "", targetSha: env.TARGET_SHA ?? "",
runId: env.GITHUB_RUN_ID ?? "", attempt: env.GITHUB_RUN_ATTEMPT ?? "", executionIds: ids }, env.MANUAL_PRIVATE_APPROVALS_JSON ?? "");
return { ids, approval, bucket: env.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET ?? "" };
}
+2 -1
View File
@@ -21,7 +21,8 @@ delete env.NODE_OPTIONS;
for (const [name, command] of [
["Vitest", [vitest, "run", "--config", "tests/runner-e2e/vitest.config.ts", ...args]],
["Node tests", ["--test", "tests/runner-e2e/artifact-budget.test.mjs", "tests/runner-e2e/native-completion-checks.test.mjs",
"tests/runner-e2e/native-completion-git-source.test.mjs", "tests/runner-e2e/native-completion-source-contract.test.mjs"]],
"tests/runner-e2e/native-completion-git-source.test.mjs", "tests/runner-e2e/native-completion-source-contract.test.mjs",
"tests/runner-e2e/chat-feature-testops-api.test.mjs", "tests/runner-e2e/chat-feature-testops-cli.test.mjs", "tests/runner-e2e/chat-feature-testops-results.test.mjs", "tests/runner-e2e/testops-chat-reporter.test.mjs"]],
]) {
const result = spawnSync(process.execPath, command, { cwd: root, env, stdio: "inherit" });
if (result.error || result.signal) {
@@ -0,0 +1,154 @@
import { createHash } from "node:crypto";
import { mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, expect, it, vi } from "vitest";
import { runnerMatrix } from "./catalog.js";
import { authorizeShippedChat, shippedChatEvidenceAuthority } from "./testops-chat-admission.js";
vi.mock("node:fs", async importOriginal => {
const actual = await importOriginal<typeof import("node:fs")>();
return { ...actual, readFileSync: vi.fn(actual.readFileSync) };
});
const wrapperDirectories: string[] = [];
afterEach(async () => {
vi.restoreAllMocks();
const actual = await vi.importActual<typeof import("node:fs")>("node:fs");
vi.mocked(readFileSync).mockImplementation(actual.readFileSync);
for (const directory of wrapperDirectories.splice(0)) rmSync(directory, { recursive: true, force: true });
});
const id = "chat-features.runner-opencode-deepseek-1017.local.create-backlog";
function fixture() {
const env: NodeJS.ProcessEnv = { PAPERCLIP_TESTOPS_CHAT_MAIN_ADMISSION: "v1", GITHUB_ACTIONS: "true", GITHUB_EVENT_NAME: "workflow_dispatch",
GITHUB_REPOSITORY: "paperclipai/paperclip", GITHUB_REF: "refs/heads/master", GITHUB_WORKFLOW_REF: "paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master",
GITHUB_JOB: "chat", GITHUB_SHA: "a".repeat(40), TARGET_SHA: "a".repeat(40), TESTOPS_ADMITTED_SHA: "a".repeat(40), GITHUB_RUN_ID: "12345", GITHUB_RUN_ATTEMPT: "1",
GITHUB_ACTOR_ID: "123", TESTOPS_TRIGGERING_ACTOR_ID: "123", TESTOPS_ALLOWED_ACTOR_IDS: "[123]", MANUAL_PRIVATE_IDS: JSON.stringify([id]), SELECTED_EXECUTION_ID: id };
const event = { ref: "refs/heads/master", sender: { id: 123 }, repository: { full_name: "paperclipai/paperclip", private: false, default_branch: "master" }, inputs: { ALLURE_JOB_RUN_ID: "789", operation: "evaluate" } as Record<string, string> };
const selected = runnerMatrix.find(row => row.id === id)!;
return { env, event, selected };
}
it("binds the actual shipped selection to immutable main admission without candidate approval variables", () => {
const f = fixture(); const result = authorizeShippedChat(f.env, f.event, f.selected, "b".repeat(64));
expect(result).toMatchObject({ authority: "paperclip.shipped-chat-testops/v1", private: true, runId: "12345", attempt: "1", workflowSha: f.env.GITHUB_SHA, targetSha: f.env.GITHUB_SHA, executionIds: [id] });
expect(result.authoritySha256).toMatch(/^[a-f0-9]{64}$/);
expect(authorizeShippedChat({ ...f.env, LATEST_MASTER_SHA: "c".repeat(40), GITHUB_JOB: "report" }, f.event, f.selected, "b".repeat(64)).authoritySha256).toBe(result.authoritySha256);
});
it.each([
["unknown mode", { PAPERCLIP_TESTOPS_CHAT_MAIN_ADMISSION: "unknown" }],
["untrusted repo", { GITHUB_REPOSITORY: "foreign/repo" }],
["nondefault ref", { GITHUB_REF: "refs/heads/feature" }],
["wrong workflow caller", { GITHUB_WORKFLOW_REF: "paperclipai/paperclip/.github/workflows/runner-full-stack-e2e.yml@refs/heads/master" }],
["source substitution", { TARGET_SHA: "c".repeat(40) }],
["admitted source mismatch", { TESTOPS_ADMITTED_SHA: "c".repeat(40) }],
["replay", { GITHUB_RUN_ATTEMPT: "2" }],
["unlisted actor", { TESTOPS_ALLOWED_ACTOR_IDS: "[456]" }],
["untrusted job", { GITHUB_JOB: "build" }],
["foreign selector", { SELECTED_EXECUTION_ID: "other" }],
])("rejects %s", (_name, patch) => { const f = fixture(); expect(() => authorizeShippedChat({ ...f.env, ...patch }, f.event, f.selected, "b".repeat(64))).toThrow(); });
it.each(["target_branch", "profile", "model", "retries", "timeout"])("rejects the undeclared %s input", field => {
const f = fixture(); f.event.inputs[field] = "override"; expect(() => authorizeShippedChat(f.env, f.event, f.selected, "b".repeat(64))).toThrow();
});
it("rejects metadata bootstrap as an evaluation admission", () => { const f = fixture(); f.event.inputs.operation = "bootstrap"; expect(() => authorizeShippedChat(f.env, f.event, f.selected, "b".repeat(64))).toThrow(); });
it("rejects forged event sender and foreign event source", () => {
const f = fixture(); f.event.sender.id = 456; expect(() => authorizeShippedChat(f.env, f.event, f.selected, "b".repeat(64))).toThrow();
f.event.sender.id = 123; f.event.repository.full_name = "foreign/repo"; expect(() => authorizeShippedChat(f.env, f.event, f.selected, "b".repeat(64))).toThrow();
});
it("rejects the existing production OpenCode backlog cell", () => {
const f = fixture();
const stock = runnerMatrix.find(row => row.id === "chat-features.runner-opencode.local.create-backlog")!;
expect(() => authorizeShippedChat(f.env, f.event, stock, "b".repeat(64))).toThrow("Unexpected shipped chat selection or bounds");
});
it.each(["model", "provider", "work", "clock", "retry", "budget"])("rejects changed selected %s", field => {
const f = fixture(); const selected = { ...f.selected, profile: { ...f.selected.profile }, task: { ...f.selected.task }, suite: { ...f.selected.suite, definitionMetadata: { ...f.selected.suite.definitionMetadata } } };
if (field === "model") selected.profile.model = "other/model";
if (field === "provider") selected.profile.provider = "codex";
if (field === "work") selected.task.expectedRunCount = 2;
if (field === "clock") selected.task.turnTimeoutMs = 121000;
if (field === "retry") selected.task.automaticRetryPolicy = undefined;
if (field === "budget") selected.suite.definitionMetadata.companyAndAgentBudgetCents = 1000;
expect(() => authorizeShippedChat(f.env, f.event, selected, "b".repeat(64))).toThrow();
});
async function wrapperFixture() {
const actual = await vi.importActual<typeof import("node:fs")>("node:fs");
const directory = mkdtempSync(path.join(tmpdir(), "testops-chat-authority-"));
wrapperDirectories.push(directory);
const f = fixture(), eventPath = path.join(directory, "event.json"), launchPath = path.join(directory, "launch.ts");
const canonicalLaunch = new URL("./launch.ts", import.meta.url);
writeFileSync(eventPath, JSON.stringify(f.event));
writeFileSync(launchPath, actual.readFileSync(canonicalLaunch));
vi.mocked(readFileSync).mockImplementation(actual.readFileSync).mockClear();
const env: NodeJS.ProcessEnv = { ...f.env, GITHUB_EVENT_PATH: eventPath };
return { ...f, env, directory, eventPath, launchPath, canonicalLaunch, actual };
}
it("admits the real filesystem wrapper with the actual catalog and canonical launcher without fixture callbacks", async () => {
const f = await wrapperFixture();
const callbacks = [vi.spyOn(f.selected.profile, "buildAgent"), vi.spyOn(f.selected.task, "buildPrompt")];
const result = shippedChatEvidenceAuthority(f.env);
expect(result).toMatchObject({ authority: "paperclip.shipped-chat-testops/v1", workflowSha: f.env.GITHUB_SHA,
targetSha: f.env.GITHUB_SHA, runId: f.env.GITHUB_RUN_ID, attempt: "1", executionIds: [id], private: true,
launchSha256: createHash("sha256").update(f.actual.readFileSync(f.canonicalLaunch)).digest("hex") });
expect(Object.isFrozen(result)).toBe(true);
expect(callbacks.every(callback => callback.mock.calls.length === 0)).toBe(true);
expect(vi.mocked(readFileSync).mock.calls.map(([file]) => String(file))).toEqual([f.eventPath, String(f.canonicalLaunch)]);
});
it.each(["missing", "absent", "directory", "symlink", "oversize", "invalid-json"])("rejects %s event custody through the real wrapper", async kind => {
const f = await wrapperFixture();
if (kind === "missing") delete f.env.GITHUB_EVENT_PATH;
if (kind === "absent") rmSync(f.eventPath);
if (kind === "directory") { rmSync(f.eventPath); mkdirSync(f.eventPath); }
if (kind === "symlink") {
const target = path.join(f.directory, "real-event.json"); writeFileSync(target, JSON.stringify(f.event));
rmSync(f.eventPath); symlinkSync(target, f.eventPath);
}
if (kind === "oversize") writeFileSync(f.eventPath, " ".repeat(65_537));
if (kind === "invalid-json") writeFileSync(f.eventPath, "{invalid");
expect(() => shippedChatEvidenceAuthority(f.env)).toThrow();
expect(vi.mocked(readFileSync).mock.calls.some(([file]) => String(file) === String(f.canonicalLaunch))).toBe(false);
});
it.each(["sender", "bootstrap", "model-input", "source"])("rejects a tampered %s through event parsing and wrapper admission", async kind => {
const f = await wrapperFixture();
if (kind === "sender") f.event.sender.id = 456;
if (kind === "bootstrap") f.event.inputs.operation = "bootstrap";
if (kind === "model-input") f.event.inputs.model = "foreign/model";
if (kind === "source") f.env.TARGET_SHA = "c".repeat(40);
writeFileSync(f.eventPath, JSON.stringify(f.event));
expect(() => shippedChatEvidenceAuthority(f.env)).toThrow("Invalid shipped chat authority");
});
it.each(["missing", "model"])("rejects %s selected catalog metadata in the actual wrapper", async kind => {
const f = await wrapperFixture();
const find = vi.spyOn(runnerMatrix, "find");
if (kind === "missing") find.mockReturnValue(undefined);
else find.mockReturnValue({ ...f.selected, profile: { ...f.selected.profile, model: "foreign/model" } });
expect(() => shippedChatEvidenceAuthority(f.env)).toThrow(kind === "missing" ? "Shipped chat execution is absent" : "Unexpected shipped chat selection or bounds");
});
it.each(["absent", "directory", "denied"])("propagates a %s launcher read from a test-only filesystem port", async kind => {
const f = await wrapperFixture();
if (kind === "absent") rmSync(f.launchPath);
if (kind === "directory") { rmSync(f.launchPath); mkdirSync(f.launchPath); }
vi.mocked(readFileSync).mockImplementation((file, options) => {
if (String(file) !== String(f.canonicalLaunch)) return f.actual.readFileSync(file, options);
if (kind === "denied") throw Object.assign(new Error("Owned launcher unreadable"), { code: "EACCES" });
return f.actual.readFileSync(f.launchPath, options);
});
expect(() => shippedChatEvidenceAuthority(f.env)).toThrow(expect.objectContaining({ code: kind === "absent" ? "ENOENT" : kind === "directory" ? "EISDIR" : "EACCES" }));
});
it("binds changed real launcher bytes to new authority rather than inventing an independent trusted hash", async () => {
const f = await wrapperFixture();
vi.mocked(readFileSync).mockImplementation((file, options) => f.actual.readFileSync(
String(file) === String(f.canonicalLaunch) ? f.launchPath : file, options));
const original = shippedChatEvidenceAuthority(f.env);
writeFileSync(f.launchPath, "source-only changed launcher bytes\n");
const changed = shippedChatEvidenceAuthority(f.env);
expect(changed.launchSha256).toBe(createHash("sha256").update(f.actual.readFileSync(f.launchPath)).digest("hex"));
expect(changed.launchSha256).not.toBe(original.launchSha256);
expect(changed.authoritySha256).not.toBe(original.authoritySha256);
});
@@ -0,0 +1,48 @@
import { createHash } from "node:crypto";
import { lstatSync, readFileSync } from "node:fs";
import { runnerMatrix } from "./catalog.js";
import { MANUAL_PRIVATE_PROTOCOL } from "./manual-private-retention.js";
import type { MatrixExecution } from "./types.js";
const executionId = "chat-features.runner-opencode-deepseek-1017.local.create-backlog";
const authority = "paperclip.shipped-chat-testops/v1";
const repository = "paperclipai/paperclip";
const workflow = ".github/workflows/chat-feature-testops.yml";
export function authorizeShippedChat(env: NodeJS.ProcessEnv, event: unknown, selected: MatrixExecution, launchSha256: string) {
const payload = event as { ref?: unknown; sender?: { id?: unknown }; repository?: { full_name?: unknown; private?: unknown; default_branch?: unknown }; inputs?: Record<string, unknown> };
const branch = payload.repository?.default_branch;
const actor = env.GITHUB_ACTOR_ID, triggering = env.TESTOPS_TRIGGERING_ACTOR_ID;
const allowed = JSON.parse(env.TESTOPS_ALLOWED_ACTOR_IDS ?? "null");
if (env.PAPERCLIP_TESTOPS_CHAT_MAIN_ADMISSION !== "v1" || env.GITHUB_ACTIONS !== "true" || env.GITHUB_EVENT_NAME !== "workflow_dispatch"
|| env.GITHUB_REPOSITORY !== repository || payload.repository?.full_name !== repository || payload.repository.private !== false
|| typeof branch !== "string" || !/^[A-Za-z0-9._/-]+$/.test(branch) || env.GITHUB_REF !== `refs/heads/${branch}` || payload.ref !== env.GITHUB_REF
|| env.GITHUB_WORKFLOW_REF !== `${repository}/${workflow}@refs/heads/${branch}` || !["chat", "report"].includes(env.GITHUB_JOB ?? "")
|| !/^[a-f0-9]{40}$/.test(env.GITHUB_SHA ?? "") || env.GITHUB_SHA !== env.TARGET_SHA || env.GITHUB_SHA !== env.TESTOPS_ADMITTED_SHA
|| !/^[1-9][0-9]{0,19}$/.test(env.GITHUB_RUN_ID ?? "") || env.GITHUB_RUN_ATTEMPT !== "1"
|| !/^[1-9][0-9]{0,19}$/.test(actor ?? "") || !/^[1-9][0-9]{0,19}$/.test(triggering ?? "") || payload.sender?.id !== Number(actor)
|| !Array.isArray(allowed) || !allowed.length || allowed.some(v => !Number.isSafeInteger(v) || v <= 0) || !allowed.includes(Number(actor)) || !allowed.includes(Number(triggering))
|| !payload.inputs || Object.keys(payload.inputs).some(k => !["ALLURE_JOB_RUN_ID", "ALLURE_USERNAME", "operation"].includes(k)) || (payload.inputs.operation ?? "evaluate") !== "evaluate" || !/^[1-9][0-9]{0,19}$/.test(String(payload.inputs.ALLURE_JOB_RUN_ID ?? ""))
|| !/^[a-f0-9]{64}$/.test(launchSha256)) throw new Error("Invalid shipped chat authority");
if (JSON.stringify(JSON.parse(env.MANUAL_PRIVATE_IDS ?? "null")) !== JSON.stringify([executionId]) || env.SELECTED_EXECUTION_ID && env.SELECTED_EXECUTION_ID !== executionId
|| selected.id !== executionId || selected.suite.id !== "chat-features" || selected.profile.id !== "runner-opencode-deepseek-1017" || selected.environment.id !== "local"
|| selected.profile.provider !== "opencode" || selected.profile.model !== "openrouter/deepseek/deepseek-v4.1-flash-1017" || selected.profile.expectedRuntimeMode !== "native"
|| selected.requiredCredentials.length !== 1 || selected.requiredCredentials[0] !== "OPENROUTER_API_KEY" || selected.task.id !== "create-backlog"
|| selected.task.expectedRunCount !== 1 || selected.task.turnTimeoutMs !== 120_000 || selected.task.attemptTimeoutMs.local !== 420_000 || selected.task.automaticRetryPolicy !== "single_attempt"
|| selected.suite.definitionMetadata?.companyAndAgentBudgetCents !== 500 || selected.suite.definitionMetadata?.maximumAttemptsPerCell !== 1) throw new Error("Unexpected shipped chat selection or bounds");
const identity = { workflowSha: env.GITHUB_SHA!, targetSha: env.GITHUB_SHA!, runId: env.GITHUB_RUN_ID!, attempt: "1", executionIds: Object.freeze([executionId]) };
const authoritySha256 = createHash("sha256").update(JSON.stringify({ authority, ...identity, actor, triggering, jobRunId: payload.inputs.ALLURE_JOB_RUN_ID, launchSha256 })).digest("hex");
return Object.freeze({ ...identity, authority, protocol: MANUAL_PRIVATE_PROTOCOL, launchSha256, authoritySha256, approvalSha256: authoritySha256, private: true as const });
}
export function shippedChatEvidenceAuthority(env: NodeJS.ProcessEnv) {
const eventPath = env.GITHUB_EVENT_PATH;
if (!eventPath) throw new Error("Missing shipped chat event");
const stat = lstatSync(eventPath);
if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 65536) throw new Error("Invalid shipped chat event custody");
const event = JSON.parse(readFileSync(eventPath, "utf8"));
const selected = runnerMatrix.find(row => row.id === executionId);
if (!selected) throw new Error("Shipped chat execution is absent");
const launch = readFileSync(new URL("./launch.ts", import.meta.url));
return authorizeShippedChat(env, event, selected, createHash("sha256").update(launch).digest("hex"));
}
+31
View File
@@ -0,0 +1,31 @@
import { execFileSync } from "node:child_process";
import { appendFileSync, lstatSync, readFileSync } from "node:fs";
import { pathToFileURL } from "node:url";
export function validateChatTestOpsDispatch(env: NodeJS.ProcessEnv, event: unknown, triggeringActorId: number) {
const e = event as { ref?: unknown; sender?: { id?: unknown }; repository?: { full_name?: unknown; private?: unknown; default_branch?: unknown }; inputs?: Record<string, unknown> };
const branch = e.repository?.default_branch, allowed = JSON.parse(env.TESTOPS_ALLOWED_ACTOR_IDS ?? "null");
const operation = e.inputs?.operation ?? "evaluate";
if (env.GITHUB_ACTIONS !== "true" || env.GITHUB_EVENT_NAME !== "workflow_dispatch" || env.GITHUB_REPOSITORY !== "paperclipai/paperclip"
|| e.repository?.full_name !== "paperclipai/paperclip" || e.repository.private !== false || typeof branch !== "string" || !/^[A-Za-z0-9._/-]+$/.test(branch)
|| env.GITHUB_REF !== `refs/heads/${branch}` || e.ref !== env.GITHUB_REF || env.GITHUB_WORKFLOW_REF !== `paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/${branch}`
|| !/^[a-f0-9]{40}$/.test(env.GITHUB_SHA ?? "") || env.GITHUB_RUN_ATTEMPT !== "1" || !/^[1-9][0-9]{0,19}$/.test(env.GITHUB_RUN_ID ?? "")
|| !/^[1-9][0-9]{0,19}$/.test(env.GITHUB_ACTOR_ID ?? "") || e.sender?.id !== Number(env.GITHUB_ACTOR_ID)
|| !Array.isArray(allowed) || !allowed.length || allowed.some(v => !Number.isSafeInteger(v) || v <= 0) || !allowed.includes(Number(env.GITHUB_ACTOR_ID)) || !allowed.includes(triggeringActorId)
|| !e.inputs || Object.keys(e.inputs).some(k => !["operation", "ALLURE_JOB_RUN_ID", "ALLURE_USERNAME"].includes(k)) || !["bootstrap", "launch-plan", "evaluate"].includes(String(operation))
|| operation === "evaluate" && !/^[1-9][0-9]{0,19}$/.test(String(e.inputs.ALLURE_JOB_RUN_ID ?? ""))
|| operation !== "evaluate" && (e.inputs.ALLURE_JOB_RUN_ID ?? "") !== "") throw new Error("Chat TestOps dispatch rejected");
return { sha: env.GITHUB_SHA!, operation: String(operation), triggeringActorId: String(triggeringActorId) };
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
try {
if (!/^[A-Za-z0-9-]{1,39}$/.test(process.env.GITHUB_TRIGGERING_ACTOR ?? "")) throw new Error("Invalid actor");
const file = process.env.GITHUB_EVENT_PATH!;
const stat = lstatSync(file);
if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 65536) throw new Error("Invalid event");
const actor = Number(execFileSync("gh", ["api", `users/${process.env.GITHUB_TRIGGERING_ACTOR}`, "--jq", ".id"], { encoding: "utf8", timeout: 30_000, stdio: ["ignore", "pipe", "pipe"] }).trim());
const checked = validateChatTestOpsDispatch(process.env, JSON.parse(readFileSync(file, "utf8")), actor);
appendFileSync(process.env.GITHUB_OUTPUT!, `sha=${checked.sha}\noperation=${checked.operation}\ntriggering_actor_id=${checked.triggeringActorId}\n`);
} catch { process.stderr.write("Fixed chat TestOps dispatch identity rejected before credential access\n"); process.exitCode = 1; }
}
@@ -0,0 +1,84 @@
import { createHash } from "node:crypto";
import { mkdtemp, mkdir, readFile, rm, symlink, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, expect, it } from "vitest";
import { exportTestOpsChat, TESTOPS_CHAT_EXECUTION } from "./testops-chat-export.js";
const directories: string[] = [];
afterEach(async () => { await Promise.all(directories.splice(0).map(d => rm(d, { recursive: true, force: true }))); });
const binding = { runId: "123456", attempt: "1", workflowSha: "a".repeat(40), targetSha: "b".repeat(40) };
const png = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+/l9sAAAAASUVORK5CYII=", "base64");
const digest = (bytes: Buffer) => createHash("sha256").update(bytes).digest("hex");
async function fixture() {
const root = await mkdtemp(path.join(os.tmpdir(), "testops-chat-export-")); directories.push(root);
const report = path.join(root, "report"), output = path.join(root, "export");
const screenshot = `evidence/${TESTOPS_CHAT_EXECUTION}/attempt-1/final-state.png`;
await mkdir(path.dirname(path.join(report, screenshot)), { recursive: true });
await writeFile(path.join(report, screenshot), png);
const result = {
schema: "paperclip.runner-e2e.result/v2", executionId: TESTOPS_CHAT_EXECUTION, suiteId: "chat-features", suiteDefinitionHash: "c".repeat(64),
source: { sha: binding.targetSha, ref: "refs/heads/master", workflowRunUrl: `https://github.com/paperclipai/paperclip/actions/runs/${binding.runId}` },
attempt: 1, status: "passed", cleanup: "passed", evidenceValid: true, runIds: ["synthetic-run"],
profileId: "runner-opencode-deepseek-1017", environmentId: "local", caseId: "create-backlog", provider: "opencode", model: "openrouter/deepseek/deepseek-v4.1-flash-1017", runtimeMode: "native",
startedAt: "2026-10-30T00:00:00.000Z", finishedAt: "2026-10-30T00:00:01.000Z", durationMs: 1000,
matcherResults: [{ matcher: { kind: "json_path", path: "chat.saved-backlog", expected: true }, passed: true, detail: "Private diagnostic body" }],
screenshots: [{ id: "final-state", label: "Fixture", file: "final-state.png", publication: "public-runner-fixture", sha256: digest(png) }],
billing: { llm: { runCount: 1, runsWithTokenUsage: 1, runsWithReportedCost: 1, inputTokens: 100, outputTokens: 20, cachedInputTokens: 0, totalTokens: 120, reportedCostUsd: 0.01, costStatus: "reported" },
runtime: { provider: "local", agentRunDurationMs: 1000, leaseDurationMs: null, leaseCount: 0, costStatus: "not_metered", costSource: "local_not_metered" },
reportedCostUsd: 0.01, estimatedRuntimeCostUsd: 0, observedAndEstimatedCostUsd: null, complete: false },
usage: { hiddenReasoning: "Do not export this provider body" }, unknownProviderBody: "Do not export this extra result field",
};
const campaign = { schema: "paperclip.runner-e2e.campaign/v2", campaignId: `gha-${binding.runId}-1`, expected: [TESTOPS_CHAT_EXECUTION], results: [result], retries: 0, generatedAt: result.finishedAt };
const save = () => writeFile(path.join(report, "normalized-results.json"), JSON.stringify(campaign));
await save(); return { report, output, result, campaign, save, screenshot };
}
it("exports the actual normalized result, marked PNG and bound hashes without opaque diagnostics", async () => {
const f = await fixture(); await writeFile(path.join(f.report, "unapproved.log"), "private original");
const receipt = await exportTestOpsChat(f.report, f.output, binding);
const text = await readFile(path.join(f.output, "normalized-results.json"), "utf8"), projected = JSON.parse(text);
expect(projected.results[0]).toMatchObject({ status: "passed", cleanup: "passed", provider: "opencode", model: f.result.model, runtimeMode: "native" });
expect(projected.billing.llm.inputTokens).toBe(100); expect(projected.billing.testsWithCompleteBilling).toBe(0);
expect(projected.complete).toBe(false);
expect(text).not.toMatch(/Private diagnostic|hiddenReasoning|unknownProviderBody|Do not export/);
expect(receipt.files.map(f => f.path).sort()).toEqual([f.screenshot, "junit.xml", "normalized-results.json"].sort());
for (const file of receipt.files) expect(digest(await readFile(path.join(f.output, file.path)))).toBe(file.sha256);
expect(JSON.parse(await readFile(path.join(f.output, "binding.json"), "utf8"))).toMatchObject({ ...binding, executionId: TESTOPS_CHAT_EXECUTION });
expect(await readFile(path.join(f.output, "junit.xml"), "utf8")).toContain('failures="0"');
});
it("preserves a real failed grade and failed cleanup as failed JUnit", async () => {
const f = await fixture(); f.result.status = "failed"; f.result.cleanup = "failed"; await f.save();
await exportTestOpsChat(f.report, f.output, binding);
expect(JSON.parse(await readFile(path.join(f.output, "normalized-results.json"), "utf8")).results[0]).toMatchObject({ status: "failed", cleanup: "failed" });
expect(await readFile(path.join(f.output, "junit.xml"), "utf8")).toContain('failures="1"');
});
it.each(["selection", "source", "model", "retry", "evidence", "oracle", "cleanup", "run-count"])("rejects inconsistent %s before exporting a PASS", async kind => {
const f = await fixture();
if (kind === "selection") f.campaign.expected.push("other");
if (kind === "source") f.result.source.sha = "d".repeat(40);
if (kind === "model") f.result.model = "other/model";
if (kind === "retry") f.result.attempt = 2;
if (kind === "evidence") f.result.evidenceValid = false;
if (kind === "oracle") f.result.matcherResults[0]!.passed = false;
if (kind === "cleanup") f.result.cleanup = "failed";
if (kind === "run-count") f.result.runIds.push("extra-run");
await f.save(); await expect(exportTestOpsChat(f.report, f.output, binding)).rejects.toThrow();
await expect(readFile(path.join(f.output, "binding.json"))).rejects.toMatchObject({ code: "ENOENT" });
});
it.each(["unmarked", "hash", "symlink"])("rejects %s screenshot evidence", async kind => {
const f = await fixture();
if (kind === "unmarked") f.result.screenshots[0]!.publication = "private";
if (kind === "hash") f.result.screenshots[0]!.sha256 = "d".repeat(64);
if (kind === "symlink") { const target = path.join(f.report, "original.png"); await writeFile(target, png); await rm(path.join(f.report, f.screenshot)); await symlink(target, path.join(f.report, f.screenshot)); }
await f.save(); await expect(exportTestOpsChat(f.report, f.output, binding)).rejects.toThrow();
});
it("rejects a workflow replay and refuses to overwrite an existing export", async () => {
const f = await fixture(); await expect(exportTestOpsChat(f.report, f.output, { ...binding, attempt: "2" })).rejects.toThrow();
await exportTestOpsChat(f.report, f.output, binding);
await expect(exportTestOpsChat(f.report, f.output, binding)).rejects.toMatchObject({ code: "EEXIST" });
});
+104
View File
@@ -0,0 +1,104 @@
import { createHash } from "node:crypto";
import { lstat, mkdir, readFile, realpath, writeFile } from "node:fs/promises";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { buildRunnerCampaign } from "./history.js";
import { findSecretLeakInJsonValues, assertSecretFree } from "./redaction.js";
import { validateRetainedRunnerResult } from "./result-validation.js";
import { PUBLIC_RUNNER_SCREENSHOT_MARKER } from "./screenshot-policy.js";
import type { RunnerE2EResult } from "./types.js";
export const TESTOPS_CHAT_EXECUTION = "chat-features.runner-opencode-deepseek-1017.local.create-backlog";
const model = "openrouter/deepseek/deepseek-v4.1-flash-1017";
const hash = (bytes: Uint8Array | string) => createHash("sha256").update(bytes).digest("hex");
const sha = /^[a-f0-9]{40}$/;
const pngMagic = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]);
export interface TestOpsChatBinding { runId: string; attempt: string; workflowSha: string; targetSha: string }
function publicBilling(value: NonNullable<RunnerE2EResult["billing"]>) {
const pick = (source: object, keys: string[]) => Object.fromEntries(keys.filter(key => Object.hasOwn(source, key)).map(key => [key, (source as Record<string, unknown>)[key]]));
const numeric = (source: object, keys: string[]) => {
const selected = pick(source, keys);
if (Object.values(selected).some(v => v !== null && (typeof v !== "number" || !Number.isFinite(v) || v < 0))) throw new Error("Invalid billing scalar");
return selected;
};
return {
llm: { ...numeric(value.llm, ["runCount", "runsWithTokenUsage", "runsWithReportedCost", "runsWithEstimatedCost", "runsWithPartialReportedCost", "inputTokens", "outputTokens", "cachedInputTokens", "totalTokens", "reportedCostUsd", "estimatedCostUsd"]), costStatus: value.llm.costStatus },
runtime: { ...numeric(value.runtime, ["agentRunDurationMs", "leaseDurationMs", "leaseCount", "cpuCores", "memoryGiB", "diskGiB", "estimatedListCostUsd"]), provider: value.runtime.provider, costStatus: value.runtime.costStatus, costSource: value.runtime.costSource },
...numeric(value, ["reportedCostUsd", "estimatedRuntimeCostUsd", "observedAndEstimatedCostUsd"]), complete: value.complete,
} as RunnerE2EResult["billing"];
}
async function ownedRead(root: string, relative: string, limit: number) {
const segments = relative.split("/");
if (segments.some(s => !/^[A-Za-z0-9._-]+$/.test(s) || s === "." || s === "..")) throw new Error("Unsafe export source");
let current = root;
const rootStat = await lstat(current);
if (!rootStat.isDirectory() || rootStat.isSymbolicLink()) throw new Error("Invalid report root");
for (const [index, segment] of segments.entries()) {
current = path.join(current, segment);
const st = await lstat(current);
if (st.isSymbolicLink() || (index < segments.length - 1 ? !st.isDirectory() : !st.isFile() || st.nlink !== 1 || st.size > limit)) throw new Error("Invalid export source");
}
if (!(await realpath(current)).startsWith(`${await realpath(root)}${path.sep}`)) throw new Error("Export source escapes report");
const before = await lstat(current), bytes = await readFile(current), after = await lstat(current);
if (bytes.length > limit || before.ino !== after.ino || before.dev !== after.dev || before.size !== after.size || before.mtimeMs !== after.mtimeMs || before.ctimeMs !== after.ctimeMs) throw new Error("Export source changed");
return bytes;
}
export async function exportTestOpsChat(reportRoot: string, output: string, binding: TestOpsChatBinding) {
if (!/^[1-9][0-9]{0,19}$/.test(binding.runId) || binding.attempt !== "1" || !sha.test(binding.workflowSha) || !sha.test(binding.targetSha)) throw new Error("Invalid TestOps run binding");
const raw = JSON.parse((await ownedRead(reportRoot, "normalized-results.json", 8 * 1024 * 1024)).toString("utf8"));
const campaignId = `gha-${binding.runId}-${binding.attempt}`;
if (raw.schema !== "paperclip.runner-e2e.campaign/v2" || raw.campaignId !== campaignId || !Array.isArray(raw.expected) || raw.expected.length !== 1 || raw.expected[0] !== TESTOPS_CHAT_EXECUTION || !Array.isArray(raw.results) || raw.results.length !== 1 || raw.retries !== 0) throw new Error("Foreign or repeated TestOps selection");
const original = raw.results[0];
validateRetainedRunnerResult(original);
if (original.executionId !== TESTOPS_CHAT_EXECUTION || original.suiteId !== "chat-features" || original.profileId !== "runner-opencode-deepseek-1017" || original.environmentId !== "local" || original.caseId !== "create-backlog" || original.provider !== "opencode" || original.model !== model || original.runtimeMode !== "native" || original.attempt !== 1 || original.source?.sha !== binding.targetSha) throw new Error("Foreign TestOps result");
if (!original.billing) throw new Error("Missing retained billing coverage");
const checks = original.matcherResults ?? [];
if (original.status === "passed" && (original.cleanup !== "passed" || raw.results[0].evidenceValid !== true || checks.length === 0 || checks.some(c => !c.passed) || original.runIds?.length !== 1)) throw new Error("Incomplete passing TestOps evidence");
const screenshots = (original.screenshots ?? []).filter(s => s.publication === PUBLIC_RUNNER_SCREENSHOT_MARKER);
if (screenshots.length > 8 || original.status === "passed" && !screenshots.some(s => s.id === "final-state" && s.file === "final-state.png")) throw new Error("Missing or excessive fixture screenshots");
const files = new Map<string, Buffer>();
for (const shot of screenshots) {
if (!/^[A-Za-z0-9][A-Za-z0-9._-]{0,199}\.png$/.test(shot.file) || !/^[a-f0-9]{64}$/.test(shot.sha256 ?? "")) throw new Error("Unbound fixture screenshot");
const relative = `evidence/${TESTOPS_CHAT_EXECUTION}/attempt-1/${shot.file}`;
const bytes = await ownedRead(reportRoot, relative, 12 * 1024 * 1024);
if (!bytes.subarray(0, 8).equals(pngMagic) || bytes.subarray(12, 16).toString("ascii") !== "IHDR" || hash(bytes) !== shot.sha256 || files.has(relative)) throw new Error("Invalid fixture PNG");
assertSecretFree(bytes, [], "TestOps fixture PNG", { includeShapes: true });
files.set(relative, bytes);
}
const result: RunnerE2EResult = {
schema: original.schema, executionId: TESTOPS_CHAT_EXECUTION, suiteId: "chat-features", suiteDefinitionHash: original.suiteDefinitionHash,
source: { sha: binding.targetSha, ref: null, workflowRunUrl: `https://github.com/paperclipai/paperclip/actions/runs/${binding.runId}` },
attempt: 1, status: original.status, ...(original.failureClass ? { failureClass: original.failureClass } : {}),
...(original.status === "failed" ? { error: "Retained chat feature evaluation failed; private originals preserve diagnostics" } : {}),
profileId: "runner-opencode-deepseek-1017", environmentId: "local", caseId: "create-backlog", provider: "opencode", model, runtimeMode: "native",
runIds: original.runIds?.map(id => { if (!/^[A-Za-z0-9._-]{1,100}$/.test(id)) throw new Error("Invalid task run identity"); return id; }),
startedAt: original.startedAt, finishedAt: original.finishedAt, durationMs: original.durationMs, cleanup: original.cleanup,
billing: publicBilling(original.billing),
matcherResults: checks.map(check => {
if (check.matcher.kind !== "json_path" || !/^[A-Za-z0-9._-]{1,150}$/.test(check.matcher.path) || check.matcher.expected !== true) throw new Error("Unsupported chat oracle check");
return { matcher: { kind: "json_path", path: check.matcher.path, expected: true }, passed: check.passed, detail: "Retained independent chat-feature oracle check" };
}),
screenshots: screenshots.map(s => ({ id: s.id, label: "Reviewed synthetic chat-feature fixture", file: s.file, publication: PUBLIC_RUNNER_SCREENSHOT_MARKER, sha256: s.sha256 })),
};
validateRetainedRunnerResult(result);
const campaign = buildRunnerCampaign({ campaignId, generatedAt: raw.generatedAt, expected: [TESTOPS_CHAT_EXECUTION], results: [result] });
if (findSecretLeakInJsonValues(campaign, [], { includeShapes: true })) throw new Error("Unsafe TestOps projection");
files.set("normalized-results.json", Buffer.from(`${JSON.stringify(campaign, null, 2)}\n`));
const failed = result.status !== "passed" || result.cleanup !== "passed";
files.set("junit.xml", Buffer.from(`<?xml version="1.0" encoding="UTF-8"?><testsuite name="Chat feature TestOps" tests="1" failures="${failed ? 1 : 0}"><testcase classname="paperclip.Product E2E.chat-features" name="create-backlog" time="${result.durationMs / 1000}">${failed ? '<failure message="Retained evaluation or cleanup failed"/>' : ""}</testcase></testsuite>\n`));
const manifest = [...files].sort(([a], [b]) => a.localeCompare(b)).map(([file, bytes]) => ({ path: file, sha256: hash(bytes), bytes: bytes.length }));
const receipt = { schema: "paperclip.chat-feature-testops-export/v1", ...binding, executionId: TESTOPS_CHAT_EXECUTION, campaignId, resultStatus: result.status, cleanup: result.cleanup, files: manifest };
await mkdir(output, { mode: 0o700 });
for (const [relative, bytes] of files) { const target = path.join(output, relative); await mkdir(path.dirname(target), { recursive: true, mode: 0o700 }); await writeFile(target, bytes, { flag: "wx", mode: 0o600 }); }
await writeFile(path.join(output, "binding.json"), `${JSON.stringify(receipt, null, 2)}\n`, { flag: "wx", mode: 0o600 });
return receipt;
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
exportTestOpsChat(path.resolve(process.env.TESTOPS_CHAT_REPORT_ROOT ?? "runner-e2e-merged-report/normalized"), path.resolve(process.env.TESTOPS_CHAT_EXPORT_ROOT ?? "runner-e2e-testops"), {
runId: process.env.GITHUB_RUN_ID ?? "", attempt: process.env.GITHUB_RUN_ATTEMPT ?? "", workflowSha: process.env.GITHUB_SHA ?? "", targetSha: process.env.TESTOPS_CHAT_TARGET_SHA ?? "",
}).catch(() => { process.stderr.write("Chat feature TestOps export rejected; private originals retained\n"); process.exitCode = 1; });
}
@@ -0,0 +1,46 @@
import { createHash } from "node:crypto";
import { execFileSync } from "node:child_process";
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { expect, it } from "vitest";
import { prepareChatBuildLock } from "./testops-chat-lock.js";
const hash = (value: string) => createHash("sha256").update(value).digest("hex");
async function fixture(source: "input" | "output" | "unknown") {
const root = await mkdtemp(path.join(tmpdir(), "chat-reviewed-lock-"));
const input = "reviewed input\n", output = "reviewed output\n";
const patch = "diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml\n--- a/pnpm-lock.yaml\n+++ b/pnpm-lock.yaml\n@@ -1 +1 @@\n-reviewed input\n+reviewed output\n";
await mkdir(path.join(root, "tests/runner-e2e"), { recursive: true });
await writeFile(path.join(root, "package.json"), "{\"name\":\"synthetic-lock-owner\"}\n");
await writeFile(path.join(root, "pnpm-lock.yaml"), source === "input" ? input : source === "output" ? output : "unreviewed source\n");
await writeFile(path.join(root, "tests/runner-e2e/report-lock.patch"), patch);
await writeFile(path.join(root, "tests/runner-e2e/report-lock.json"), JSON.stringify({ inputSha256: hash(input), outputSha256: hash(output), patchSha256: hash(patch) }));
execFileSync("git", ["init", "-q"], { cwd: root });
execFileSync("git", ["add", "."], { cwd: root });
execFileSync("git", ["-c", "user.name=Fixture", "-c", "user.email=fixture@example.invalid", "commit", "-qm", "reviewed fixture"], { cwd: root });
return { root, output };
}
it.each(["input", "output"] as const)("uses the existing reviewed %s lock without resolving dependencies", async source => {
const f = await fixture(source);
try {
const receipt = await prepareChatBuildLock(f.root);
expect(receipt.runtimeLockSha256).toBe(hash(f.output));
expect(receipt.sourceInputsUnchanged).toBe(true);
expect(await readFile(path.join(f.root, "pnpm-lock.yaml"), "utf8")).toBe(f.output);
expect(await prepareChatBuildLock(f.root)).toEqual(receipt);
} finally { await rm(f.root, { recursive: true, force: true }); }
});
it.each(["unknown-source", "unknown-runtime", "manifest", "patch", "policy"])("rejects %s drift before any installer or artifact", async variant => {
const f = await fixture(variant === "unknown-source" ? "unknown" : "output");
try {
if (variant === "unknown-source") await writeFile(path.join(f.root, "pnpm-lock.yaml"), f.output);
if (variant === "unknown-runtime") await writeFile(path.join(f.root, "pnpm-lock.yaml"), "unreviewed runtime\n");
if (variant === "manifest") await writeFile(path.join(f.root, "package.json"), "{\"name\":\"unreviewed-graph\"}\n");
if (variant === "patch") await writeFile(path.join(f.root, "tests/runner-e2e/report-lock.patch"), "unreviewed patch\n");
if (variant === "policy") await writeFile(path.join(f.root, "tests/runner-e2e/report-lock.json"), "{}\n");
await expect(prepareChatBuildLock(f.root)).rejects.toThrow();
} finally { await rm(f.root, { recursive: true, force: true }); }
});
+32
View File
@@ -0,0 +1,32 @@
import { createHash } from "node:crypto";
import { execFileSync } from "node:child_process";
import { readFile } from "node:fs/promises";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { prepareReportLock } from "./prepare-report-lock.mjs";
const hash = (bytes: Buffer) => createHash("sha256").update(bytes).digest("hex");
const inputs = ["package.json", ":(glob)**/package.json", "pnpm-workspace.yaml", ".npmrc", ".pnpmfile.cjs",
":(glob)**/*.patch", "tests/runner-e2e/report-lock.json"];
export async function prepareChatBuildLock(root = path.resolve(import.meta.dirname, "../..")) {
try {
execFileSync("git", ["diff", "--quiet", "HEAD", "--", ...inputs], { cwd: root, stdio: "pipe" });
} catch { throw new Error("Source dependency inputs changed before install or packaging"); }
const committed = execFileSync("git", ["show", "HEAD:pnpm-lock.yaml"], { cwd: root });
const policyBytes = await readFile(path.join(root, "tests/runner-e2e/report-lock.json"));
const policy = JSON.parse(policyBytes.toString("utf8"));
const sourceLockSha256 = hash(committed);
if (![policy.inputSha256, policy.outputSha256].includes(sourceLockSha256))
throw new Error("Committed dependency lock is not reviewed");
await prepareReportLock(root);
const runtimeLockSha256 = hash(await readFile(path.join(root, "pnpm-lock.yaml")));
if (runtimeLockSha256 !== policy.outputSha256) throw new Error("Runtime dependency lock is not reviewed");
return { sourceSha: execFileSync("git", ["rev-parse", "HEAD"], { cwd: root, encoding: "utf8" }).trim(),
sourceLockSha256, runtimeLockSha256, policySha256: hash(policyBytes), sourceInputsUnchanged: true };
}
if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) {
prepareChatBuildLock().then(value => console.log(JSON.stringify(value)))
.catch(() => { process.stderr.write("Chat TestOps dependency graph is not source-reviewed\n"); process.exitCode = 1; });
}
@@ -0,0 +1,36 @@
import { afterEach, expect, it } from "vitest";
import { mkdtemp, realpath, rm } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { authorizeManualPrivateEvidence, MANUAL_PRIVATE_EXECUTIONS, MANUAL_PRIVATE_PROTOCOL, manualCellKind, manualPrivateSessionPolicy } from "./manual-private-retention.js";
import { canonicalGeneratedReportAsset, cellTraceAsset, inspectCellReportProducer, privateCellEvidenceOwner } from "./manual-private-report-assets.js";
const id = "chat-features.runner-opencode-deepseek-1017.local.create-backlog";
const roots: string[] = [];
afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
it.each(["cell-read", "cell-write", "report-write"] as const)("keeps the shipped chat %s session scoped to only its existing private archive", mode => {
const policy = manualPrivateSessionPolicy({ bucket: "private-evidence", runId: "123", executionIds: [id], mode });
const objects = policy.Statement.filter(row => Array.isArray(row.Resource)).flatMap(row => row.Resource);
const filename = mode === "report-write" ? "manual-report" : manualCellKind(id);
const key = `arn:aws:s3:::private-evidence/runner-e2e-private/bounded-v1/campaigns/gha-123-1/${filename}.zip`;
expect(objects).toEqual([key, `${key}.upload-receipt.json`]);
expect(JSON.stringify(policy)).not.toContain("*");
expect(() => manualPrivateSessionPolicy({ bucket: "private-evidence", runId: "123", executionIds: [id.replace("create-backlog", "delegate-announcement")], mode })).toThrow();
});
it("preserves the frozen244 candidate approval registry and refuses shipped authority through that old authorizer", () => {
expect(MANUAL_PRIVATE_EXECUTIONS).toHaveLength(244);
expect(MANUAL_PRIVATE_EXECUTIONS).not.toContain(id);
const identity = { workflowSha: "a".repeat(40), targetSha: "b".repeat(40), runId: "123", attempt: "1", executionIds: [id] };
expect(() => authorizeManualPrivateEvidence(identity, JSON.stringify({ version: 1, approvals: [{ ...identity, protocol: MANUAL_PRIVATE_PROTOCOL, launchSha256: "c".repeat(64) }] }))).toThrow("Unsupported private manual evidence selection");
});
it("binds canonical private trace assets to the exact shipped profile and rejects a rewritten profile or sibling case", async () => {
const root = await realpath(await mkdtemp(path.join(os.tmpdir(), "testops-private-path-"))); roots.push(root);
const context = { root, private: true as const, runId: "123", attempt: "1", executionIds: [id], cellReportProducer: await inspectCellReportProducer() };
const owner = await privateCellEvidenceOwner(context, id);
const prefix = `gha-123-1-${id}/chat-features/runner-opencode-deepseek-1017/local/create-backlog/attempt-1/html-report/trace/`;
expect(cellTraceAsset(`${prefix}playwright-logo.svg`, owner, "final")).toEqual({ reserved: true, canonical: canonicalGeneratedReportAsset("html/trace/playwright-logo.svg") });
expect(cellTraceAsset(`${prefix.replace("/runner-opencode-deepseek-1017/", "/runner-opencode/")}playwright-logo.svg`, owner, "final")).toEqual({ reserved: true, canonical: undefined });
for (const foreign of [id.replace("create-backlog", "delegate-announcement"), id.replace("deepseek-1017", "unknown")]) {
await expect(privateCellEvidenceOwner({ ...context, executionIds: [foreign] }, foreign)).rejects.toThrow("Invalid private cell report owner");
}
});
@@ -0,0 +1,70 @@
import { createHash } from 'node:crypto';
import { appendFile, lstat, readFile, realpath, writeFile } from 'node:fs/promises';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
import { publicContext, aliases, executionId, scenario } from './chat-feature-testops-api.mjs';
import { run } from './chat-feature-testops-cli.mjs';
const hash = bytes => createHash('sha256').update(bytes).digest('hex');
const fail = () => { throw new Error('chat_testops_selection_rejected'); };
async function readVerifiedSelection(file, env) {
const expected = publicContext(env);
if (!/^[a-f0-9]{64}$/.test(env.EXPECTED_SELECTION_SHA256 ?? '') || !/^[1-9][0-9]{0,15}$/.test(env.ALLURE_JOB_RUN_ID ?? '')) fail();
const root = await realpath(env.GITHUB_WORKSPACE ?? process.cwd()), absolute = path.resolve(file);
if (!absolute.startsWith(`${root}${path.sep}`)) fail();
let current = root;
const parts = path.relative(root, absolute).split(path.sep);
for (const [index, part] of parts.entries()) {
if (!/^[A-Za-z0-9._-]+$/.test(part) || ['.', '..'].includes(part)) fail();
current = path.join(current, part);
const stat = await lstat(current);
if (stat.isSymbolicLink() || (index < parts.length - 1 ? !stat.isDirectory() : !stat.isFile() || stat.nlink !== 1 || stat.size > 16_384)) fail();
}
const bytes = await readFile(absolute);
if (hash(bytes) !== env.EXPECTED_SELECTION_SHA256) fail();
const value = JSON.parse(bytes);
if (value.schema !== 'paperclip.chat-feature-testops-selection/v1' || value.runId !== expected.runId || value.attempt !== '1'
|| value.workflowSha !== expected.workflowSha || value.targetSha !== expected.targetSha || value.allureJobRunId !== env.ALLURE_JOB_RUN_ID
|| value.executionId !== executionId || value.canonicalScenario !== scenario || value.projectId !== 1 || !aliases.includes(value.fullName)
|| ['planId', 'jobId', 'caseId', 'launchId'].some(key => !Number.isSafeInteger(value[key]) || value[key] <= 0)) fail();
return { value, bytes };
}
export async function verifySelectionFile(file, env) {
return (await readVerifiedSelection(file, env)).value;
}
export async function reporter(command, argument, env = process.env, fetchImpl = fetch) {
publicContext(env);
if (command === 'verify-selection') {
if (Object.hasOwn(env, 'ALLURE_TOKEN')) fail();
await verifySelectionFile(argument, env);
return 0;
}
if (!['bootstrap', 'launch-plan', 'selection', 'upload', 'verify-upload'].includes(command) || !env.RUNNER_TEMP) fail();
const stage = command === 'selection' ? 'validate-selection' : command === 'upload' ? 'export' : command;
const stateRoot = path.join(env.RUNNER_TEMP, `chat-testops-${command}-state`);
const selectionPath = path.join(env.GITHUB_WORKSPACE ?? process.cwd(), 'chat-testops-selection', 'testops-selection.json');
let copiedSelectionPath = selectionPath;
if (command === 'upload' || command === 'verify-upload') {
const { bytes } = await readVerifiedSelection(selectionPath, env);
copiedSelectionPath = path.join(env.RUNNER_TEMP, 'chat-testops-authenticated-selection.json');
if (command === 'upload') await writeFile(copiedSelectionPath, bytes, { flag: 'wx', mode: 0o600 });
else if (!(await readFile(copiedSelectionPath)).equals(bytes)) fail();
}
const result = await run({ ...env, CHAT_TESTOPS_OPERATION: stage, CHAT_TESTOPS_STATE_ROOT: stateRoot,
...(['upload', 'verify-upload'].includes(command) ? { CHAT_TESTOPS_EXPORT_ROOT: path.resolve(argument), CHAT_TESTOPS_SELECTION: copiedSelectionPath,
ALLURE_RESULTS: env.ALLURE_RESULTS ?? path.join(env.RUNNER_TEMP, 'chat-testops-allure-results') } : {}) }, fetchImpl);
if (command === 'selection' && result === 0) {
const bytes = await readFile(path.join(stateRoot, 'selection.json'));
await writeFile(path.join(env.GITHUB_WORKSPACE ?? process.cwd(), 'testops-selection.json'), bytes, { flag: 'wx', mode: 0o600 });
if (!env.GITHUB_OUTPUT) fail();
await appendFile(env.GITHUB_OUTPUT, `sha256=${hash(bytes)}\n`);
}
return result;
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
reporter(process.argv[2], process.argv[3]).then(code => { process.exitCode = code; })
.catch(() => { process.stderr.write('Chat TestOps reporter stopped: invalid source, custody or selection\n'); process.exitCode = 1; });
}
@@ -0,0 +1,138 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
import { mkdtemp, mkdir, readFile, realpath, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { verifySelectionFile, reporter } from './testops-chat-reporter.mjs';
import { executionId, scenario, fullName } from './chat-feature-testops-api.mjs';
const context = { GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/heads/master', GITHUB_EVENT_NAME: 'workflow_dispatch',
GITHUB_RUN_ATTEMPT: '1', GITHUB_SHA: 'a'.repeat(40), GITHUB_RUN_ID: '123', ALLURE_JOB_RUN_ID: '9',
GITHUB_WORKFLOW_REF: 'paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master' };
async function fixture(t, change = {}) {
const root = await realpath(await mkdtemp(path.join(tmpdir(), 'testops-selection-')));
t.after(() => rm(root, { recursive: true }));
const value = { schema: 'paperclip.chat-feature-testops-selection/v1', runId: '123', attempt: '1', workflowSha: 'a'.repeat(40), targetSha: 'a'.repeat(40),
allureJobRunId: '9', executionId, canonicalScenario: scenario, projectId: 1, planId: 8, jobId: 7, caseId: 91, fullName, launchId: 10, ...change };
const bytes = Buffer.from(JSON.stringify(value));
await mkdir(path.join(root, 'chat-testops-selection'));
const file = path.join(root, 'chat-testops-selection', 'testops-selection.json'); await writeFile(file, bytes);
return { file, env: { ...context, GITHUB_WORKSPACE: root, EXPECTED_SELECTION_SHA256: createHash('sha256').update(bytes).digest('hex') } };
}
test('actual worker verifier requires no token or network and accepts exact source-bound selection hash', async t => {
const f = await fixture(t);
assert.equal((await verifySelectionFile(f.file, f.env)).caseId, 91);
assert.equal(await reporter('verify-selection', f.file, f.env), 0);
});
test('worker verifier rejects token contamination, hash mismatch and correctly-hashed foreign tuples', async t => {
const f = await fixture(t);
await assert.rejects(reporter('verify-selection', f.file, { ...f.env, ALLURE_TOKEN: 'SYNTHETIC_TOKEN' }));
await assert.rejects(verifySelectionFile(f.file, { ...f.env, EXPECTED_SELECTION_SHA256: '0'.repeat(64) }));
for (const change of [{ runId: '456' }, { targetSha: 'b'.repeat(40) }, { allureJobRunId: '10' }, { executionId: 'other' }, { projectId: 2 }]) {
const foreign = await fixture(t, change);
await assert.rejects(verifySelectionFile(foreign.file, foreign.env));
}
});
test('actual upload facade copies authenticated workspace bytes to owned temp before successful synthetic export', async t => {
const f = await fixture(t), workspace = f.env.GITHUB_WORKSPACE;
const temporary = path.join(workspace, 'temp'); await mkdir(temporary);
const root = path.join(workspace, 'safe-export'); await mkdir(root);
const png = Buffer.from([137,80,78,71,13,10,26,10,0,0,0,13,73,72,68,82]);
const hash = bytes => createHash('sha256').update(bytes).digest('hex');
const shot = { id: 'final-state', file: 'final-state.png', sha256: hash(png) };
const result = { executionId, suiteId: 'chat-features', caseId: 'create-backlog', profileId: 'runner-opencode-deepseek-1017', provider: 'opencode',
environmentId: 'local', model: 'openrouter/deepseek/deepseek-v4.1-flash-1017', runtimeMode: 'native', attempt: 1,
source: { sha: context.GITHUB_SHA }, status: 'passed', cleanup: 'passed', runIds: ['synthetic'], durationMs: 123,
matcherResults: [{ matcher: { kind: 'json_path', path: 'chatFeature.createdTask', expected: true }, passed: true }], screenshots: [shot] };
const files = new Map([['normalized-results.json', Buffer.from(JSON.stringify({ schema: 'paperclip.runner-e2e.campaign/v2', campaignId: 'gha-123-1', results: [result] }))],
['junit.xml', Buffer.from('<testsuite tests="1"/>')], [`evidence/${executionId}/attempt-1/final-state.png`, png]]);
for (const [name, bytes] of files) { await mkdir(path.dirname(path.join(root, name)), { recursive: true }); await writeFile(path.join(root, name), bytes); }
await writeFile(path.join(root, 'binding.json'), JSON.stringify({ schema: 'paperclip.chat-feature-testops-export/v1', runId: '123', attempt: '1',
workflowSha: context.GITHUB_SHA, targetSha: context.GITHUB_SHA, executionId, campaignId: 'gha-123-1', resultStatus: 'passed', cleanup: 'passed',
files: [...files].map(([name, bytes]) => ({ path: name, sha256: hash(bytes), bytes: bytes.length })) }));
assert.equal(await reporter('upload', root, { ...f.env, RUNNER_TEMP: temporary }), 0);
assert.deepEqual(await readFile(path.join(temporary, 'chat-testops-authenticated-selection.json')), await readFile(f.file));
const exported = JSON.parse(await readFile(path.join(temporary, 'chat-testops-upload-state', 'export.json')));
const jobRun = { id: 9, job: { id: 7, type: 'github' }, launchId: 10, stage: 'finished', url: 'https://github.com/paperclipai/paperclip/actions/runs/123' };
const fetchImpl = async (url, options) => {
const parsed = new URL(url), route = parsed.pathname;
let value;
if (route === '/api/uaa/oauth/token') { assert.equal(options.method, 'POST'); value = { access_token: 'MOCK_ACCESS_TOKEN' }; }
else {
assert.equal(options.method, 'GET');
if (route === '/api/jobrun/9') value = jobRun;
else if (route === '/api/launch/10') value = { id: 10, projectId: 1 };
else if (route === '/api/testresult') value = { content: [{ id: 45, testCaseId: 91 }], totalElements: 1, totalPages: 1 };
else if (route === '/api/testresult/45') value = { id: 45, projectId: 1, launchId: 10, testCaseId: 91, fullName, jobRun, status: 'passed',
links: [{ url: jobRun.url }, { url: `https://github.com/paperclipai/paperclip/commit/${context.GITHUB_SHA}` }] };
else if (route === '/api/testresult/attachment') value = { content: exported.attachmentMetadata.map((row, i) => ({ ...row, id: i + 1 })), totalElements: exported.attachmentMetadata.length, totalPages: 1 };
else assert.fail(route);
}
return { ok: true, status: 200, json: async () => value, text: async () => JSON.stringify(value) };
};
assert.equal(await reporter('verify-upload', root, { ...f.env, RUNNER_TEMP: temporary, ALLURE_TOKEN: 'MOCK_API_TOKEN' }, fetchImpl), 0);
const verified = JSON.parse(await readFile(path.join(temporary, 'chat-testops-verify-upload-state', 'verification.json')));
assert.equal(verified.verification, 'VERIFIED'); assert.equal(verified.caseId, 91); assert.equal(verified.status, 'passed');
});
test('actual launch-plan facade performs metadata-only plan launch once and exits without waiting for child', async t => {
const f = await fixture(t), workspace = f.env.GITHUB_WORKSPACE;
const temporary = path.join(workspace, 'temp'); await mkdir(temporary);
const eventPath = path.join(workspace, 'event.json'); await writeFile(eventPath, JSON.stringify({ inputs: { operation: 'launch-plan', ALLURE_JOB_RUN_ID: '' } }));
const nonce = '1'.repeat(32), calls = [];
const fetchImpl = async (url, options) => {
const route = new URL(url).pathname, method = options.method; calls.push({ route, method });
const response = value => ({ ok: true, status: 200, json: async () => value, text: async () => JSON.stringify(value) });
if (route === '/repos/paperclipai/paperclip') return response({ full_name: 'paperclipai/paperclip', private: false, default_branch: 'master' });
if (route === '/repos/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml') return response({ id: 99, path: '.github/workflows/chat-feature-testops.yml', state: 'active' });
if (route === '/repos/paperclipai/paperclip/actions/runs/123') return response({ id: 123, workflow_id: 99, run_attempt: 1, run_number: 2, head_sha: context.GITHUB_SHA, head_branch: 'master', event: 'workflow_dispatch', repository: { full_name: 'paperclipai/paperclip', private: false, default_branch: 'master' }, actor: { id: 34892728 }, triggering_actor: { id: 34892728 } });
if (route === '/api/uaa/oauth/token') return response({ access_token: 'SYNTHETIC_TOKEN' });
if (route === '/api/testcase/91') return response({ id: 91, projectId: 1, fullName, automated: true });
if (route === '/api/testplan/8') return response({ id: 8, projectId: 1, name: 'Chat feature evals', testCasesCount: 1, treeSelection: { leafsInclude: [91] } });
if (route === '/api/job/7') return response({ id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml', parameters: [{ id: 17, value: 'evaluate' }] });
if (route === '/api/integration/2/project/1') return response({ id: 2, projectId: 1, info: { type: 'github' }, disabled: false });
if (route === '/api/environment/17') return response({ id: 17, name: 'operation' });
if (route === '/api/testplan/8/job') return response([{ job: { id: 7, parameters: [{ id: 17, value: 'evaluate' }] }, testCasesCount: 1 }]);
if (route === '/api/launch') return response({ content: [], totalPages: 0 });
if (route === '/api/testplan/8/jobparameter') return { ok: true, status: 204 };
assert.equal(route, '/api/testplan/8/run');
return response({ id: 20, projectId: 1, name: `Chat feature evals · ${nonce}` });
};
assert.equal(await reporter('launch-plan', undefined, { ...f.env, RUNNER_TEMP: temporary, GITHUB_EVENT_PATH: eventPath,
ALLURE_JOB_RUN_ID: '', ALLURE_TOKEN: 'SYNTHETIC_ONLY', CHAT_TESTOPS_CASE_ID: '91', CHAT_TESTOPS_PLAN_ID: '8', CHAT_TESTOPS_JOB_ID: '7',
CHAT_TESTOPS_OPERATION_PARAMETER_ID: '17', CHAT_TESTOPS_INTEGRATION_ID: '2', CHAT_TESTOPS_LAUNCH_NONCE: nonce, CHAT_TESTOPS_LAUNCH_RUN_NUMBER: '2', GITHUB_RUN_NUMBER: '2',
CHAT_TESTOPS_GITHUB_WORKFLOW_ID: '99', GITHUB_ACTOR_ID: '34892728', TESTOPS_TRIGGERING_ACTOR_ID: '34892728', TESTOPS_ALLOWED_ACTOR_IDS: '[34892728]', GH_TOKEN: 'SYNTHETIC_GH_TOKEN' }, fetchImpl), 0);
const state = path.join(temporary, 'chat-testops-launch-plan-state');
assert.equal(JSON.parse(await readFile(path.join(state, 'launch.json'))).status, 'REQUESTED');
assert.equal(calls.filter(row => row.route === '/api/testplan/8/run').length, 1);
assert.equal(calls.at(-1).route, '/api/testplan/8/run');
});
test('actual launch caller rejects five own-run contradictions before TestOps auth or mutation', async t => {
const original = process.stderr.write; process.stderr.write = () => true;
try {
for (const changes of [{ run_number: 3 }, { actor: { id: 1 } }, { head_sha: 'b'.repeat(40) }, { workflow_id: 100 }, { run_attempt: 2 }]) {
const f = await fixture(t), workspace = f.env.GITHUB_WORKSPACE;
const temporary = path.join(workspace, 'temp'); await mkdir(temporary);
const eventPath = path.join(workspace, 'event.json'); await writeFile(eventPath, JSON.stringify({ inputs: { operation: 'launch-plan' } }));
let testopsCalls = 0;
const fetchImpl = async url => {
if (new URL(url).hostname !== 'api.github.com') { testopsCalls++; throw new Error('must not reach TestOps'); }
const value = url.endsWith('/repos/paperclipai/paperclip') ? { full_name: 'paperclipai/paperclip', private: false, default_branch: 'master' } : url.includes('/actions/workflows/') ? { id: 99, path: '.github/workflows/chat-feature-testops.yml', state: 'active' }
: { id: 123, workflow_id: 99, run_attempt: 1, run_number: 2, head_sha: context.GITHUB_SHA, head_branch: 'master', event: 'workflow_dispatch',
repository: { full_name: 'paperclipai/paperclip', private: false, default_branch: 'master' }, actor: { id: 34892728 }, triggering_actor: { id: 34892728 }, ...changes };
return { ok: true, text: async () => JSON.stringify(value) };
};
assert.equal(await reporter('launch-plan', undefined, { ...f.env, RUNNER_TEMP: temporary, GITHUB_EVENT_PATH: eventPath,
ALLURE_JOB_RUN_ID: '', ALLURE_TOKEN: 'SYNTHETIC_ONLY', CHAT_TESTOPS_LAUNCH_RUN_NUMBER: '2', GITHUB_RUN_NUMBER: '2',
CHAT_TESTOPS_GITHUB_WORKFLOW_ID: '99', GITHUB_ACTOR_ID: '34892728', TESTOPS_TRIGGERING_ACTOR_ID: '34892728', TESTOPS_ALLOWED_ACTOR_IDS: '[34892728]', GH_TOKEN: 'SYNTHETIC_GH_TOKEN' }, fetchImpl), 1);
assert.equal(testopsCalls, 0);
const error = JSON.parse(await readFile(path.join(temporary, 'chat-testops-launch-plan-state', 'failure.json')));
assert.equal(error.stage, 'launch_admission'); assert.equal(error.code, 'chat_testops_launch_admission');
}
} finally { process.stderr.write = original; }
});
@@ -0,0 +1,154 @@
import { readFileSync } from "node:fs";
import { parse } from "yaml";
import { expect, it } from "vitest";
import { validateChatTestOpsDispatch } from "./testops-chat-dispatch.js";
const workflow = parse(readFileSync(new URL("../../.github/workflows/chat-feature-testops.yml", import.meta.url), "utf8"));
function context() {
return { env: { GITHUB_ACTIONS: "true", GITHUB_EVENT_NAME: "workflow_dispatch", GITHUB_REPOSITORY: "paperclipai/paperclip", GITHUB_REF: "refs/heads/master",
GITHUB_WORKFLOW_REF: "paperclipai/paperclip/.github/workflows/chat-feature-testops.yml@refs/heads/master", GITHUB_SHA: "a".repeat(40), GITHUB_RUN_ID: "12345", GITHUB_RUN_ATTEMPT: "1", GITHUB_ACTOR_ID: "123", TESTOPS_ALLOWED_ACTOR_IDS: "[123]" },
event: { repository: { full_name: "paperclipai/paperclip", private: false, default_branch: "master" }, ref: "refs/heads/master", sender: { id: 123 }, inputs: { ALLURE_JOB_RUN_ID: "456" } as Record<string, string> } };
}
it("admits the actual TestOps service input and only the closed metadata bootstrap alternative", () => {
const f = context(); expect(validateChatTestOpsDispatch(f.env, f.event, 123).operation).toBe("evaluate");
f.event.inputs = { operation: "bootstrap" }; expect(validateChatTestOpsDispatch(f.env, f.event, 123).operation).toBe("bootstrap");
f.event.inputs.operation = "launch-plan"; expect(validateChatTestOpsDispatch(f.env, f.event, 123).operation).toBe("launch-plan");
f.event.inputs.ALLURE_JOB_RUN_ID = "456"; expect(() => validateChatTestOpsDispatch(f.env, f.event, 123)).toThrow();
delete f.event.inputs.ALLURE_JOB_RUN_ID;
f.event.inputs.operation = "other"; expect(() => validateChatTestOpsDispatch(f.env, f.event, 123)).toThrow();
});
it.each(["unlisted-trigger", "wrong-source", "replay", "extra-input", "missing-jobrun", "foreign-repo"])("rejects %s before any service credential operation", kind => {
const f = context(); let trigger = 123;
if (kind === "unlisted-trigger") trigger = 456;
if (kind === "wrong-source") f.env.GITHUB_REF = "refs/heads/unreviewed";
if (kind === "replay") f.env.GITHUB_RUN_ATTEMPT = "2";
if (kind === "extra-input") f.event.inputs.target_branch = "other";
if (kind === "missing-jobrun") f.event.inputs = {};
if (kind === "foreign-repo") f.event.repository.full_name = "foreign/repo";
expect(() => validateChatTestOpsDispatch(f.env, f.event, trigger)).toThrow();
});
it("keeps provider and TestOps service credentials in separate jobs", () => {
const jobs = workflow.jobs;
for (const [id, job] of Object.entries(jobs) as Array<[string, any]>) {
const text = JSON.stringify(job);
if (text.includes("secrets.OPENROUTER_API_KEY")) expect(id).toBe("chat");
if (text.includes("secrets.ALLURE_TOKEN")) expect(["select", "upload"]).toContain(id);
if (id !== "chat") expect(text).not.toContain("pnpm test:e2e:runner --");
expect(job.steps.some((step: any) => step.run === "node tests/runner-e2e/testops-chat-dispatch.ts")).toBe(true);
}
expect(jobs.chat.environment).toBe("runner-e2e-private-paid");
expect(jobs.select.environment).toBe("runner-e2e-testops-reporting");
expect(jobs.upload.environment).toBe("runner-e2e-testops-reporting");
});
it("requires token-isolated selection before the fixed AWS run and keeps bootstrap outside all model jobs", () => {
expect(workflow.on).toHaveProperty("workflow_dispatch"); expect(Object.keys(workflow.on)).toEqual(["workflow_dispatch"]);
expect(Object.keys(workflow.on.workflow_dispatch.inputs)).toEqual(["operation", "ALLURE_JOB_RUN_ID", "ALLURE_USERNAME"]);
expect(workflow.on.workflow_dispatch.inputs.operation.default).toBe("evaluate");
expect(workflow.jobs.chat.needs).toContain("select"); expect(workflow.jobs.build.needs).toContain("select");
expect(workflow.jobs.chat.if).toBe("needs.authorize.outputs.operation == 'evaluate'");
expect(workflow.jobs.build.if).toBe("needs.authorize.outputs.operation == 'evaluate'");
const paid = workflow.jobs.chat.steps.find((s: any) => s.env?.OPENROUTER_API_KEY);
expect(paid.run).toContain("--id chat-features.runner-opencode-deepseek-1017.local.create-backlog --max-parallel 1 --max-automatic-retries 0");
expect(workflow.jobs.chat.steps.findIndex((s: any) => s.name === "Verify immutable outputs and token-free TestOps selection")).toBeLessThan(workflow.jobs.chat.steps.indexOf(paid));
expect(workflow.jobs.authorize.outputs.runner).toBe("runs-on/fleet=paperclip-public-pr-x64/env=public-ci");
expect(workflow.jobs.chat["runs-on"]).toBe("${{ needs.authorize.outputs.runner }}");
});
it("uploads only the trusted safe projection after successful strict private report retention", () => {
const steps = workflow.jobs.report.steps;
expect(steps.find((s: any) => s.id === "export").if).toBe("always() && steps.private_report.outcome == 'success'");
expect(steps.find((s: any) => s.with?.name?.startsWith("runner-e2e-testops-")).with.path).toBe("runner-e2e-testops/");
expect(JSON.stringify(workflow)).not.toContain("MANUAL_PRIVATE_APPROVALS_JSON");
expect(Object.values(workflow.jobs).flatMap((j: any) => j.steps).filter((s: any) => s.uses?.startsWith("actions/upload-artifact")).some((s: any) => s.with.path === "tests/runner-e2e/results/")).toBe(false);
});
it("initializes the actual TestOps association and never fabricates setup results", () => {
const steps = workflow.jobs.select.steps;
const installer = steps.find((s: any) => s.uses?.startsWith("allure-framework/setup-allurectl@"));
expect(installer.uses).toBe("allure-framework/setup-allurectl@2d89ee0b7ba2f59dbe9849b6df5763cab78e14db");
expect(installer.with["allurectl-version"]).toBe("2.23.1");
const bootstrap = steps.find((s: any) => s.run?.includes("testops-chat-reporter.mjs bootstrap"));
expect(bootstrap.run).toContain("mkdir -m 700");
expect(bootstrap.run).toContain("allurectl watch -- node");
expect(bootstrap.env.ALLURE_RESULTS).toBe("${{ runner.temp }}/chat-testops-bootstrap-results");
const select = steps.find((s: any) => s.id === "selection");
expect(select.run).toContain("allurectl watch -- node");
for (const name of ["CHAT_TESTOPS_CASE_ID", "CHAT_TESTOPS_PLAN_ID", "CHAT_TESTOPS_JOB_ID"])
expect(select.env[name]).toBe(`\${{ vars.${name} }}`);
const metadata = steps.find((s: any) => s.with?.name?.startsWith("chat-testops-setup-"));
expect(metadata.if).toBe("always() && needs.authorize.outputs.operation == 'bootstrap'");
expect(metadata.with.path).not.toContain("results");
});
it("authenticates the same-run selection again before converting and uploading", () => {
const steps = workflow.jobs.upload.steps;
const receipt = steps.find((s: any) => s.with?.path === "chat-testops-selection");
expect(receipt.with["artifact-ids"]).toBe("${{ needs.select.outputs.artifact_id }}");
const upload = steps.find((s: any) => s.run?.includes("testops-chat-reporter.mjs upload"));
expect(upload.run).toContain("allurectl watch -- node");
expect(upload.env.EXPECTED_SELECTION_SHA256).toBe("${{ needs.select.outputs.sha256 }}");
expect(steps.indexOf(receipt)).toBeLessThan(steps.indexOf(upload));
expect(JSON.stringify(workflow.jobs.select)).not.toContain("OPENROUTER_API_KEY");
expect(JSON.stringify(workflow.jobs.upload)).not.toContain("OPENROUTER_API_KEY");
});
it("keeps the one plan launch outside Allure watch and every AWS/model job", () => {
const steps = workflow.jobs.select.steps;
const launch = steps.find((s: any) => s.run?.endsWith("testops-chat-reporter.mjs launch-plan"));
expect(launch.if).toBe("needs.authorize.outputs.operation == 'launch-plan'");
expect(launch.run).not.toContain("watch");
expect(launch.env.CHAT_TESTOPS_OPERATION_PARAMETER_ID).toBe("${{ vars.CHAT_TESTOPS_OPERATION_PARAMETER_ID }}");
expect(launch.env.CHAT_TESTOPS_INTEGRATION_ID).toBe("${{ vars.CHAT_TESTOPS_INTEGRATION_ID }}");
expect(launch.env.CHAT_TESTOPS_LAUNCH_NONCE).toBe("${{ vars.CHAT_TESTOPS_LAUNCH_NONCE }}");
expect(launch.env.CHAT_TESTOPS_LAUNCH_RUN_NUMBER).toBe("${{ vars.CHAT_TESTOPS_LAUNCH_RUN_NUMBER }}");
expect(launch.env.TESTOPS_TRIGGERING_ACTOR_ID).toBe("${{ needs.authorize.outputs.triggering_actor_id }}");
expect(launch.env.TESTOPS_ALLOWED_ACTOR_IDS).toBe("${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}");
expect(launch.env.GH_TOKEN).toBe("${{ github.token }}");
expect(launch.run).not.toContain("sleep");
expect(steps.find((s: any) => s.uses?.startsWith("allure-framework/setup-allurectl@")).if).toBe("needs.authorize.outputs.operation != 'launch-plan'");
for (const name of ["build", "chat"]) expect(workflow.jobs[name].if).toBe("needs.authorize.outputs.operation == 'evaluate'");
expect(workflow.concurrency["cancel-in-progress"]).toBe(false);
});
it("accepts the documented TestOps username metadata without treating it as an actor", () => {
const f = context(); f.event.inputs.ALLURE_USERNAME = "service-metadata";
expect(validateChatTestOpsDispatch(f.env, f.event, 123).operation).toBe("evaluate");
expect(() => validateChatTestOpsDispatch(f.env, f.event, 456)).toThrow();
expect(JSON.stringify(workflow.jobs)).not.toContain("inputs.ALLURE_USERNAME");
});
it("includes every native Node helper control in the normal unit owner", () => {
const launcher = readFileSync(new URL("./run-unit.mjs", import.meta.url), "utf8");
const config = readFileSync(new URL("./vitest.config.ts", import.meta.url), "utf8");
for (const file of ["chat-feature-testops-api.test.mjs", "chat-feature-testops-cli.test.mjs", "chat-feature-testops-results.test.mjs", "testops-chat-reporter.test.mjs"]) {
expect(launcher).toContain(`"tests/runner-e2e/${file}"`);
expect(config).toContain(`"${file}"`);
}
});
it("verifies the actual server outcome after watch without masking an original failed upload", () => {
const steps = workflow.jobs.upload.steps;
const watch = steps.find((s: any) => s.id === "allure_upload");
const verify = steps.find((s: any) => s.id === "verify_upload");
expect(watch.run).toContain("allurectl watch -- node");
expect(watch["continue-on-error"]).toBeUndefined();
expect(verify.run).toBe("node tests/runner-e2e/testops-chat-reporter.mjs verify-upload runner-e2e-testops");
expect(verify.if).toBe("always() && !cancelled() && (steps.allure_upload.outcome == 'success' || steps.allure_upload.outcome == 'failure')");
expect(verify.env).toEqual(watch.env);
expect(steps.indexOf(verify)).toBeGreaterThan(steps.indexOf(watch));
const proof = steps.find((s: any) => s.with?.name?.startsWith("chat-testops-verification-"));
expect(proof.with.path.split("\n").filter(Boolean)).toHaveLength(2);
expect(proof.with.path).not.toContain("allure-results");
});
it("carries only the existing source-reviewed lock through build and worker install", () => {
const build = workflow.jobs.build.steps;
const install = build.find((s: any) => s.run?.includes("pnpm install --frozen-lockfile"));
expect(install.run).toContain("node tests/runner-e2e/testops-chat-lock.ts > chat-testops-lock.json");
expect(install.run).not.toContain("--lockfile-only");
const archive = build.find((s: any) => s.run?.includes("tar -czf chat-testops-build"));
expect(archive.run).toContain("node tests/runner-e2e/testops-chat-lock.ts > chat-testops-lock.json");
expect(archive.run).toContain("chat-testops-lock.json pnpm-lock.yaml");
const worker = workflow.jobs.chat.steps.find((s: any) => s.name === "Verify immutable outputs and token-free TestOps selection");
expect(worker.run).toContain("node tests/runner-e2e/testops-chat-lock.ts > chat-testops-worker-lock.json");
expect(worker.run).toContain("cmp chat-testops-lock.json chat-testops-worker-lock.json");
expect(worker.run.indexOf("cmp chat-testops-lock.json")).toBeLessThan(worker.run.indexOf("pnpm install --frozen-lockfile"));
});
+1 -1
View File
@@ -6,7 +6,7 @@ export default defineConfig({
name: "runner-e2e-support",
include: ["**/*.test.ts", "**/*.test.mjs"],
// These use node:test and run separately in test:e2e:runner:unit.
exclude: [...configDefaults.exclude, "native-completion-{checks,git-source,source-contract}.test.mjs", "artifact-budget.test.mjs"],
exclude: [...configDefaults.exclude, "native-completion-{checks,git-source,source-contract}.test.mjs", "artifact-budget.test.mjs", "chat-feature-testops-api.test.mjs", "chat-feature-testops-cli.test.mjs", "chat-feature-testops-results.test.mjs", "testops-chat-reporter.test.mjs"],
testTimeout: 30_000,
},
});
@@ -1,6 +1,7 @@
// @vitest-environment jsdom
import type { ComponentProps, ReactNode } from "react";
import { act as reactAct } from "react";
import { flushSync } from "react-dom";
import { createRoot } from "react-dom/client";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
@@ -346,55 +347,49 @@ describe("ProjectWorkspaceSummaryCard", () => {
cwd: "/Users/dotta/paperclip/.worktrees/PAP-1552-workspace-polish",
});
await act(async () => {
root.render(withQueryClient(
<ProjectWorkspaceSummaryCard
projectRef="paperclip-app"
summary={summary}
runtimeActionKey={null}
runtimeActionPending={false}
onRuntimeAction={() => {}}
onCloseWorkspace={() => {}}
/>,
));
});
vi.useFakeTimers();
try {
await reactAct(async () => {
root.render(withQueryClient(
<ProjectWorkspaceSummaryCard
projectRef="paperclip-app"
summary={summary}
runtimeActionKey={null}
runtimeActionPending={false}
onRuntimeAction={() => {}}
onCloseWorkspace={() => {}}
/>,
));
});
const branchTextButton = Array.from(container.querySelectorAll("button"))
.find((button) => button.textContent === summary.branchName);
const pathTextButton = container.querySelector(`button[title="${summary.cwd}"]`);
const branchIconButton = container.querySelector('button[aria-label="Copy branch"]');
const pathIconButton = container.querySelector('button[aria-label="Copy path"]');
const branchTextButton = Array.from(container.querySelectorAll("button"))
.find((button) => button.textContent === summary.branchName);
const pathTextButton = container.querySelector(`button[title="${summary.cwd}"]`);
const branchIconButton = container.querySelector('button[aria-label="Copy branch"]');
const pathIconButton = container.querySelector('button[aria-label="Copy path"]');
expect(branchTextButton).not.toBeNull();
expect(pathTextButton).not.toBeNull();
expect(branchIconButton).not.toBeNull();
expect(pathIconButton).not.toBeNull();
const copyWithFeedback = async (button: Element | null | undefined, text: string) => {
expect(button).toBeTruthy();
await reactAct(async () => {
button!.dispatchEvent(new MouseEvent("click", { bubbles: true }));
});
expect(writeClipboard).toHaveBeenLastCalledWith(text);
expect(button!.nextElementSibling?.className).toContain("opacity-100");
await reactAct(async () => { await vi.advanceTimersByTimeAsync(1499); });
expect(button!.nextElementSibling?.className).toContain("opacity-100");
await reactAct(async () => { await vi.advanceTimersByTimeAsync(1); });
expect(button!.nextElementSibling?.className).toContain("opacity-0");
};
await act(async () => {
branchTextButton!.click();
await new Promise((resolve) => setTimeout(resolve, 0));
});
expect(writeClipboard).toHaveBeenLastCalledWith(summary.branchName);
expect(branchTextButton?.nextElementSibling?.className).toContain("opacity-100");
await act(async () => {
pathTextButton!.dispatchEvent(new MouseEvent("click", { bubbles: true }));
await new Promise((resolve) => setTimeout(resolve, 0));
});
expect(writeClipboard).toHaveBeenLastCalledWith(summary.cwd);
expect(pathTextButton?.nextElementSibling?.className).toContain("opacity-100");
await act(async () => {
branchIconButton!.dispatchEvent(new MouseEvent("click", { bubbles: true }));
pathIconButton!.dispatchEvent(new MouseEvent("click", { bubbles: true }));
await new Promise((resolve) => setTimeout(resolve, 0));
});
expect(writeClipboard).toHaveBeenCalledWith(summary.branchName);
expect(writeClipboard).toHaveBeenCalledWith(summary.cwd);
act(() => {
root.unmount();
});
await copyWithFeedback(branchTextButton, summary.branchName!);
await copyWithFeedback(pathTextButton, summary.cwd!);
await copyWithFeedback(branchIconButton, summary.branchName!);
await copyWithFeedback(pathIconButton, summary.cwd!);
} finally {
try {
await reactAct(async () => { root.unmount(); });
} finally { vi.useRealTimers(); }
}
});
it("colors live service urls green", () => {
const root = createRoot(container);