mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
test(evals): run Product regression from TestOps (#18983)
## Thinking Path > - Paperclip manages agent work through governed product workflows. > - Product E2E tests check those workflows through the real app and runner. > - The saved Product regression plan has 20 imported case identities. > - The plan has no live executor, so historical baseline rows do not qualify its current behavior. > - The existing paid full-stack workflow already owns all 20 required journeys. > - This change binds TestOps selection and reporting to that workflow without changing the product oracle or paid admission. > - Maintainers can run the plan and inspect current case verdicts and retained evidence. ## Linked Issues or Issue Description **What happened?** The standing Product regression plan has no executor. Its dashboard points at a historical 355-result snapshot across several suites and revisions. **Expected behavior** A TestOps plan run must select the intended existing full-stack cells, execute them on GitHub, and report their original case IDs, assertions and source revision. Missing evidence must remain a visible failure. **Steps to reproduce** Open the Product regression plan and inspect Executors. There is no live job. The existing full-stack workflow accepts exact cell IDs but has no TestOps selection or reporting path. **Paperclip version or commit** The branch includes reviewed master correction04a24f4e06. This change does not alter scenario prompts, models, deadlines or assertions. Related public work: #18542 establishes the separate chat-catalog lane. #18666 repairs its workflow URL binding. These changes do not implement this Product plan. ## What Changed - Add optional TestOps service inputs and a manual bootstrap selector to the existing full-stack workflow. - Map the original 20 case IDs and full names to existing native Codex/local cells, with the existing native OpenCode/local task-workspaces case. - Keep reporting tokens in trusted default-branch selection and publication jobs. Keep the existing paid provider authorization and protected environments. - Bind retained results to the workflow, job run, launch, source revision and selection hash. Preserve missing-result failures and earlier attempt verdict receipts. - Verify uploaded identities, coverage and verdicts. Document full-plan and subset reporting limits. - Keep the outstanding-decision reservation live while its existing deletion-fence test asserts the original 409 guard. Release the test provider in finally. Production deletion and accounting code stay unchanged. ## Verification - `node --test tests/runner-e2e/product-regression-testops.test.mjs`: 13 positive and negative controls pass, including the separated CLI handoff. - Existing paid-workflow security regression passes. - `actionlint -shellcheck= .github/workflows/runner-full-stack-e2e.yml`: passes. - The Product E2E catalog compatibility test confirms all 20 exact cells exist: passes. - `pnpm test:e2e:runner:typecheck`: passes. - Fresh Greptile review of final head61ef7a6eis 5/5 (11 files, no new findings); both initial reporting findings are repaired and resolved. - `server/src/__tests__/decision-models.test.ts`: all 37 controls pass on a disposable official PostgreSQL 18 container in 22.02 seconds. The unchanged focused case passed once and all five local samples; the original Linux CI failure remains retained. This is not a claimed deterministic local red reproduction. The new gate explicitly preserves the outstanding-reservation precondition. - The existing master readiness-fixture correction passes all three retained-workspace controls on the same database in 25.79 seconds. It is incorporated from master, not duplicated in this PR's diff. - Final-head Linux CI also passed all 37 decision-model controls in 7.047 seconds; the server shard passed 120 files and 2,004 tests. See [the retained successful shard](https://github.com/paperclipai/paperclip/actions/runs/41870214517/job/126715358759). - Previous CI failures are retained: [three stale readiness-fixture cases](https://github.com/paperclipai/paperclip/actions/runs/41864061890/job/126694812885) and [the reservation fixture race](https://github.com/paperclipai/paperclip/actions/runs/41864061890/job/126694812894). The separate [Canary Dry Run](https://github.com/paperclipai/paperclip/actions/runs/41864061890/job/126694811605) recorded a lost self-hosted runner connection before any step. The fixture repairs are verified on changed source; the separate Spot-interrupted CI attempt and its bounded retry are recorded below. - Actual one-cell GitHub canary on the unchanged product source: https://github.com/paperclipai/paperclip/actions/runs/41860189801 . The native Codex/local message-marker case passed all six assertions and cleanup in 35.24 seconds. The persisted issue is Done and its one run succeeded. This is supporting evidence; it is not TestOps acceptance. - Final-head CI attempt 1 was interrupted: three failed jobs have EC2 `Server.SpotInstanceTermination` receipts; six cancelled jobs show the same simultaneous shutdown. Original logs are retained. One [failed-job retry](https://github.com/paperclipai/paperclip/actions/runs/41870214517/attempts/2) passed all nine interrupted checks while preserving prior successful jobs. - Final-head full repository CI passes, including both required `ci / verify` and `ci / e2e` gates. GitHub still requires a non-author code-owner approval for the workflow change; normal auto-merge is queued. A fresh TestOps-triggered subset/full-plan acceptance remains pending. Local projection checks do not establish a live Product pass. ## Risks - The first bootstrap upload must establish the official TestOps job before the saved plan can use it. - This lane tests one declared local native configuration per case. It does not qualify the historical 355-row matrix. - Upload failures remain visible. A missing retained report produces broken case results and cannot pass. - The existing non-TestOps campaign path must keep its selection and scheduling behavior; paid-workflow regression coverage remains in place. ## Model Used OpenAI Codex with reasoning, code execution and tool use. The host does not expose the exact model ID or context window to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
6364fa7327
commit
b99dd57c82
11 files changed
+764
-7
No files matched your search
@@ -48,6 +48,18 @@ on:
|
||||
description: "Comma-separated full suite.profile.environment.case IDs; exclusive with other selectors"
|
||||
type: string
|
||||
required: false
|
||||
testops_case:
|
||||
description: "Product regression TestOps bootstrap: all, case name or case ID; leave empty for normal campaigns"
|
||||
type: string
|
||||
required: false
|
||||
ALLURE_JOB_RUN_ID:
|
||||
description: "Allure TestOps service input; its requested selection takes precedence"
|
||||
type: string
|
||||
required: false
|
||||
ALLURE_USERNAME:
|
||||
description: "Allure TestOps service input"
|
||||
type: string
|
||||
required: false
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -284,10 +296,67 @@ jobs:
|
||||
retention-days: 30
|
||||
if-no-files-found: error
|
||||
|
||||
testops_selection:
|
||||
name: Bind Product regression TestOps selection
|
||||
needs: authorize
|
||||
if: github.event_name == 'workflow_dispatch' && (inputs.ALLURE_JOB_RUN_ID != '' || inputs.testops_case != '')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
environment: runner-e2e-testops-reporting
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
outputs:
|
||||
ids: ${{ steps.selection.outputs.ids }}
|
||||
job_run_started: ${{ steps.selection.outputs.job_run_started }}
|
||||
job_run_id: ${{ steps.selection.outputs.job_run_id }}
|
||||
selection_sha256: ${{ steps.selection.outputs.selection_sha256 }}
|
||||
launch_url: ${{ steps.selection.outputs.launch_url }}
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
||||
with:
|
||||
ref: ${{ github.sha }}
|
||||
persist-credentials: false
|
||||
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
||||
with:
|
||||
node-version: 24
|
||||
- uses: allure-framework/setup-allurectl@2d89ee0b7ba2f59dbe9849b6df5763cab78e14db # v1
|
||||
with:
|
||||
allurectl-version: '2.23.1'
|
||||
allure-endpoint: https://paperclip.testops.cloud
|
||||
allure-project-id: '1'
|
||||
allure-token: ${{ secrets.ALLURE_TOKEN }}
|
||||
- name: Resolve only admitted plan cases before provider jobs
|
||||
id: selection
|
||||
env:
|
||||
SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
EVAL_CADENCE: ${{ inputs.cadence || 'manual' }}
|
||||
SELECT_ID: ${{ inputs.id }}
|
||||
SELECT_SUITE: ${{ inputs.suite }}
|
||||
SELECT_GROUP: ${{ inputs.group }}
|
||||
SELECT_PROFILE: ${{ inputs.profile }}
|
||||
SELECT_ENVIRONMENT: ${{ inputs.environment }}
|
||||
SELECT_CASE: ${{ inputs.case }}
|
||||
TESTOPS_CASE: ${{ inputs.testops_case }}
|
||||
ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
|
||||
ALLURE_USERNAME: ${{ inputs.ALLURE_USERNAME }}
|
||||
ALLURE_LAUNCH_NAME: Product regression · ${{ github.run_number }}
|
||||
ALLURE_LAUNCH_TAGS: product-regression,github-actions
|
||||
run: node tests/runner-e2e/product-regression-testops-cli.mjs select product-testops/selection.json
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
||||
with:
|
||||
name: product-testops-selection-${{ github.run_id }}-${{ github.run_attempt }}
|
||||
path: product-testops/selection.json
|
||||
if-no-files-found: error
|
||||
retention-days: 30
|
||||
|
||||
catalog:
|
||||
name: Validate catalog and select cells
|
||||
needs: [authorize, target_lock]
|
||||
if: github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true')
|
||||
needs: [authorize, target_lock, testops_selection]
|
||||
if: >-
|
||||
always() && needs.authorize.result == 'success' && needs.target_lock.result == 'success' &&
|
||||
(needs.testops_selection.result == 'success' || needs.testops_selection.result == 'skipped') &&
|
||||
(github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true'))
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
@@ -371,7 +440,7 @@ jobs:
|
||||
SELECT_PROFILE: ${{ inputs.profile }}
|
||||
SELECT_ENVIRONMENT: ${{ inputs.environment }}
|
||||
SELECT_CASE: ${{ inputs.case }}
|
||||
SELECT_ID: ${{ inputs.id }}
|
||||
SELECT_ID: ${{ needs.testops_selection.outputs.ids || inputs.id }}
|
||||
MAX_PARALLEL: ${{ vars.RUNNER_E2E_MAX_PARALLEL || needs.authorize.outputs.max_parallel_default }}
|
||||
MAX_PARALLEL_LIMIT: ${{ needs.authorize.outputs.max_parallel_limit }}
|
||||
REQUESTED_MAX_PARALLEL: ${{ inputs.max_parallel }}
|
||||
@@ -1946,6 +2015,69 @@ jobs:
|
||||
echo "ready=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
publish_testops:
|
||||
name: Publish bound Product regression results to TestOps
|
||||
needs: [authorize, testops_selection, catalog, report]
|
||||
if: always() && needs.testops_selection.outputs.job_run_started == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
environment: runner-e2e-testops-reporting
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
env:
|
||||
SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
SELECTION_SHA256: ${{ needs.testops_selection.outputs.selection_sha256 }}
|
||||
EVAL_CADENCE: manual
|
||||
ALLURE_JOB_RUN_ID: ${{ needs.testops_selection.outputs.job_run_id || inputs.ALLURE_JOB_RUN_ID }}
|
||||
ALLURE_USERNAME: ${{ inputs.ALLURE_USERNAME }}
|
||||
ALLURE_RESULTS: ${{ github.workspace }}/product-regression-allure
|
||||
REPORT_DIR: ${{ github.workspace }}/product-report
|
||||
EXPORT_SUMMARY: ${{ github.workspace }}/product-regression-export.json
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
||||
with:
|
||||
# Only trusted workflow code sees the reporting token. Candidate
|
||||
# reports are parsed as data and never loaded as executable modules.
|
||||
ref: ${{ github.sha }}
|
||||
persist-credentials: false
|
||||
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
||||
with:
|
||||
node-version: 24
|
||||
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
||||
if: needs.testops_selection.result == 'success' && !cancelled()
|
||||
id: download_selection
|
||||
with:
|
||||
name: product-testops-selection-${{ github.run_id }}-${{ github.run_attempt }}
|
||||
path: product-testops
|
||||
- name: Download retained full-stack report
|
||||
if: needs.testops_selection.result == 'success' && !cancelled()
|
||||
continue-on-error: true
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
||||
with:
|
||||
name: runner-e2e-report-${{ github.run_id }}-${{ github.run_attempt }}
|
||||
path: product-report
|
||||
- uses: allure-framework/setup-allurectl@2d89ee0b7ba2f59dbe9849b6df5763cab78e14db # v1
|
||||
if: always()
|
||||
with:
|
||||
allurectl-version: '2.23.1'
|
||||
allure-endpoint: https://paperclip.testops.cloud
|
||||
allure-project-id: '1'
|
||||
allure-token: ${{ secrets.ALLURE_TOKEN }}
|
||||
- name: Project original verdicts and explicitly report missing results
|
||||
if: steps.download_selection.outcome == 'success' && !cancelled()
|
||||
id: project
|
||||
run: node tests/runner-e2e/product-regression-testops-cli.mjs project product-testops/selection.json
|
||||
- name: Upload only the bound result projection
|
||||
if: steps.project.outcome == 'success'
|
||||
run: allurectl upload --validate-json "$ALLURE_RESULTS"
|
||||
- name: Verify server-side case identities, completeness and verdicts
|
||||
if: steps.project.outcome == 'success'
|
||||
run: node tests/runner-e2e/product-regression-testops-cli.mjs verify product-testops/selection.json
|
||||
- name: Finish the original TestOps job run
|
||||
if: always()
|
||||
run: allurectl job-run stop
|
||||
|
||||
publish_history:
|
||||
name: Publish S3 history and Pages bundle with declared screenshots
|
||||
needs: [authorize, catalog, report]
|
||||
|
||||
@@ -1,5 +1,17 @@
|
||||
# Paperclip evaluation guide
|
||||
|
||||
The optional Product regression TestOps lane in `runner-full-stack-e2e.yml`
|
||||
maps the standing 20-case plan to existing isolated Product E2E cells: native
|
||||
Codex/local, with native OpenCode/local for the task-workspaces case. A manual
|
||||
`testops_case` bootstrap accepts `all`, a case name or its TestOps ID. TestOps
|
||||
dispatch supplies `ALLURE_JOB_RUN_ID`; its exact case IDs and full names take
|
||||
precedence. This lane preserves the existing paid admission, fixture, model,
|
||||
timeout, assertion, cleanup and retained-report paths. Reporting credentials
|
||||
stay in trusted default-branch jobs. Missing evidence becomes a broken result.
|
||||
Subset runs do not qualify the full plan or the historical 355-row snapshot.
|
||||
Catalog and projection tests are not live qualification; retain the fresh
|
||||
TestOps launch and GitHub run when reporting acceptance.
|
||||
|
||||
The explicit deployment-targeted [saved agent instructions journey](../tests/runner-e2e/product-journeys/AGENT-SETTINGS.md) checks an owned agent’s ordinary UI save, persisted instruction bytes, one current-run output, and restoration. The task request omits its instruction-only reference. It reuses the existing Product E2E target and report path; local controls do not establish a staging pass.
|
||||
|
||||
The explicit-only [verified API artifact transfer](../tests/runner-e2e/README.md#verified-api-artifact-transfer)
|
||||
|
||||
@@ -415,9 +415,13 @@ describe("company decision service", () => {
|
||||
const f = await fixture(), runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
|
||||
status: "running", invocationSource: "on_demand", responsibleUserId: "alice" });
|
||||
let unlock!: () => void, locked!: (pid: number) => void;
|
||||
let unlock!: () => void, locked!: (pid: number) => void, finishProvider!: () => void;
|
||||
const gate = new Promise<void>(resolve => { unlock = resolve; });
|
||||
const acquired = new Promise<number>(resolve => { locked = resolve; });
|
||||
const providerGate = new Promise<void>(resolve => { finishProvider = resolve; });
|
||||
// Keep the admitted reservation held while deletion checks it. A provider
|
||||
// that already settled may legitimately allow deletion past this guard.
|
||||
f.provider.mockImplementationOnce(async () => { await providerGate; return structuredClone(outcome); });
|
||||
const blocker = db.transaction(async tx => {
|
||||
await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("no key update");
|
||||
const [row] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
|
||||
@@ -440,10 +444,11 @@ describe("company decision service", () => {
|
||||
}, { timeout: 3000 });
|
||||
unlock(); await blocker;
|
||||
expect(await deletion).toMatchObject({ status: 409, details: { code: "agent_decision_accounting_pending" } });
|
||||
finishProvider();
|
||||
expect(await pending).toMatchObject({ status: "succeeded" });
|
||||
expect((await db.select().from(agents).where(eq(agents.id, f.agentId)))[0]).toBeDefined();
|
||||
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId))).toHaveLength(1);
|
||||
} finally { unlock(); await blocker; await deletion; await pending.catch(() => undefined); }
|
||||
} finally { unlock(); finishProvider(); await blocker; await deletion; await pending.catch(() => undefined); }
|
||||
});
|
||||
it("blocks agent deletion while an in-flight or unknown decision retains a budget hold", async () => {
|
||||
const f = await fixture(), runId = randomUUID();
|
||||
|
||||
@@ -1591,6 +1591,28 @@ gh workflow run runner-full-stack-e2e.yml \
|
||||
-f id=core-compatibility.runner-codex.local.message-marker
|
||||
```
|
||||
|
||||
The same trusted workflow can run the standing **Product regression — core
|
||||
workflows** TestOps plan. Bootstrap one actual case with `-f testops_case=66`
|
||||
(message-marker), or select the complete plan with `-f testops_case=all`.
|
||||
Leave the ordinary `id`, `suite`, `group`, `profile`, `environment` and `case`
|
||||
selectors empty in this mode. TestOps dispatch supplies `ALLURE_JOB_RUN_ID` and
|
||||
`ALLURE_USERNAME`; the requested test-plan selection overrides the bootstrap
|
||||
case. All 20 imported case IDs and full names remain stable. The lane uses the
|
||||
existing native Codex/local cells and the existing native OpenCode/local
|
||||
task-workspaces cell. It does not qualify the historical 355-result snapshot.
|
||||
|
||||
The protected `runner-e2e-testops-reporting` environment supplies `ALLURE_TOKEN`.
|
||||
Only default-branch selection and reporting code receives this token. The
|
||||
ordinary paid provider jobs, their authorization and their evidence paths stay
|
||||
the same. Selection binds the job run, launch, source and workflow revisions.
|
||||
Reporting parses the retained normalized campaign, publishes original attempts
|
||||
and assertions, and makes an absent result explicitly broken. It verifies the
|
||||
uploaded case IDs and verdicts before finishing the original TestOps job run.
|
||||
No bootstrap or synthetic control result contributes to the pass count. Tag
|
||||
only a complete 20-case attempt `full-run` and `test-plan-2`; omit `full-run`
|
||||
from focused reruns. Inspect the linked launch and the retained report for
|
||||
acceptance. A local projection test does not establish a live product pass.
|
||||
|
||||
Create a protected `runner-e2e-paid` GitHub environment, restrict it to the
|
||||
default branch, limit environment administration to trusted maintainers, and
|
||||
store the four provider secrets there. This is a second authorization boundary:
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { selectRunnerExecutions, parseRunnerSelectors } from "./selectors.js";
|
||||
import { productRegressionCases } from "./product-regression-testops.mjs";
|
||||
|
||||
it("runs every standing Product regression case through its existing full-stack fixture", () => {
|
||||
const executions = selectRunnerExecutions(parseRunnerSelectors(
|
||||
productRegressionCases.flatMap(row => ["--id", row.executionId]),
|
||||
));
|
||||
expect(executions).toHaveLength(20);
|
||||
for (const row of productRegressionCases) {
|
||||
const execution = executions.find(value => value.id === row.executionId)!;
|
||||
expect(execution.suite.id).toBe(row.suite);
|
||||
expect(execution.task.id).toBe(row.name);
|
||||
expect(execution.profile.id).toBe(row.profile);
|
||||
expect(execution.environment.id).toBe("local");
|
||||
expect(execution.requiredCredentials).not.toContain("CODEX_AUTH_JSON");
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,162 @@
|
||||
import { execFile } from 'node:child_process';
|
||||
import { promisify } from 'node:util';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { appendFile, lstat, mkdir, readFile, writeFile } from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
import { testopsClient } from './chat-feature-testops-api.mjs';
|
||||
import { endpoint, workflowUrls, selectProductRegression, validateSelection, projectProductResults } from './product-regression-testops.mjs';
|
||||
|
||||
const execute = promisify(execFile);
|
||||
const hash = bytes => createHash('sha256').update(bytes).digest('hex');
|
||||
const fail = code => { throw new Error(`product_regression_testops_${code}`); };
|
||||
const positive = value => /^[1-9][0-9]{0,15}$/.test(String(value ?? ''));
|
||||
|
||||
export function admitProductTestOps(env) {
|
||||
if (env.GITHUB_REPOSITORY !== 'paperclipai/paperclip' || env.GITHUB_REF !== 'refs/heads/master'
|
||||
|| env.GITHUB_EVENT_NAME !== 'workflow_dispatch'
|
||||
|| env.GITHUB_WORKFLOW_REF !== 'paperclipai/paperclip/.github/workflows/runner-full-stack-e2e.yml@refs/heads/master'
|
||||
|| !positive(env.GITHUB_RUN_ID) || !positive(env.GITHUB_RUN_ATTEMPT)
|
||||
|| !/^[a-f0-9]{40}$/.test(env.GITHUB_SHA ?? '') || !/^[a-f0-9]{40}$/.test(env.SOURCE_SHA ?? '')
|
||||
|| env.ALLURE_ENDPOINT !== endpoint || env.ALLURE_PROJECT_ID !== '1'
|
||||
|| (env.EVAL_CADENCE ?? 'manual') !== 'manual'
|
||||
|| ['SELECT_ID', 'SELECT_SUITE', 'SELECT_GROUP', 'SELECT_PROFILE', 'SELECT_ENVIRONMENT', 'SELECT_CASE'].some(key => env[key])) fail('untrusted_context');
|
||||
}
|
||||
|
||||
async function readJson(file, limit) {
|
||||
const stat = await lstat(file);
|
||||
if (!stat.isFile() || stat.isSymbolicLink() || stat.nlink !== 1 || stat.size > limit) fail('file_invalid');
|
||||
return JSON.parse(await readFile(file, 'utf8'));
|
||||
}
|
||||
|
||||
export function officialProductContext(stdout) {
|
||||
if (typeof stdout !== 'string' || Buffer.byteLength(stdout) > 16_384) fail('official_context');
|
||||
const found = {};
|
||||
for (const line of stdout.split(/\r?\n/)) for (const key of ['ALLURE_JOB_RUN_ID', 'ALLURE_LAUNCH_ID']) {
|
||||
if (!line.startsWith(`${key}=`)) continue;
|
||||
const value = line.slice(key.length + 1);
|
||||
if (found[key] || !positive(value)) fail('official_context');
|
||||
found[key] = value;
|
||||
}
|
||||
if (Object.keys(found).length !== 2) fail('official_context');
|
||||
return { jobRunId: found.ALLURE_JOB_RUN_ID, launchId: Number(found.ALLURE_LAUNCH_ID) };
|
||||
}
|
||||
|
||||
export async function verifyProductJob(api, context, env, allowUnstarted = false) {
|
||||
const run = await api(`/api/jobrun/${context.jobRunId}`);
|
||||
if (run.id !== Number(context.jobRunId) || run.launchId !== context.launchId || !positive(run.job?.id)
|
||||
|| run.job.type !== 'github' || (run.url !== `https://github.com/paperclipai/paperclip/actions/runs/${env.GITHUB_RUN_ID}`
|
||||
&& !(allowUnstarted && (run.url == null || run.url === '')))) fail('job_run_mismatch');
|
||||
const [job, launch] = await Promise.all([api(`/api/job/${run.job.id}`), api(`/api/launch/${context.launchId}`)]);
|
||||
if (job.id !== run.job.id || job.type !== 'github' || !workflowUrls.has(job.url)
|
||||
|| launch.id !== context.launchId || launch.projectId !== 1) fail('job_mismatch');
|
||||
return { jobId: job.id, launchId: launch.id };
|
||||
}
|
||||
|
||||
export function validateUploadedProductResults(details, selection, summary) {
|
||||
if (!Array.isArray(summary.cases) || summary.cases.length !== selection.cases.length
|
||||
|| new Set(summary.cases.map(row => row.caseId)).size !== selection.cases.length
|
||||
|| summary.cases.some(row => !selection.cases.some(value => value.id === row.caseId)
|
||||
|| !['passed', 'failed', 'broken'].includes(row.status))) fail('expected_status_invalid');
|
||||
if (details.length !== selection.cases.length || new Set(details.map(row => row.testCaseId)).size !== selection.cases.length || details.some(row => {
|
||||
const selected = selection.cases.find(value => value.id === row.testCaseId);
|
||||
return !selected || row.fullName !== selected.fullName || row.projectId !== 1 || row.launchId !== selection.launchId
|
||||
|| row.jobRun?.id !== Number(selection.jobRunId) || !['passed', 'failed', 'broken'].includes(row.status);
|
||||
})) fail('uploaded_identity_mismatch');
|
||||
const counts = Object.fromEntries(['passed', 'failed', 'broken'].map(status => [status, details.filter(row => row.status === status).length]));
|
||||
if (Object.entries(counts).some(([status, count]) => summary[status] !== count)
|
||||
|| details.some(row => summary.cases.find(value => value.caseId === row.testCaseId).status !== row.status)) fail('uploaded_status_mismatch');
|
||||
}
|
||||
|
||||
export async function productTestOps(command, argument, env = process.env, exec = execute, fetchImpl = fetch,
|
||||
wait = ms => new Promise(resolve => setTimeout(resolve, ms))) {
|
||||
admitProductTestOps(env);
|
||||
if (command === 'select') {
|
||||
const requestedJob = env.ALLURE_JOB_RUN_ID || '';
|
||||
if (requestedJob && !positive(requestedJob)) fail('job_run_invalid');
|
||||
// Reject a bad bootstrap selector before opening any run. Before adopting
|
||||
// a TestOps dispatch, independently check its pending job and project.
|
||||
const manualCases = requestedJob ? undefined : selectProductRegression(undefined, env.TESTOPS_CASE || 'all');
|
||||
const api = await testopsClient(env.ALLURE_TOKEN, fetchImpl);
|
||||
if (requestedJob) {
|
||||
const pending = await api(`/api/jobrun/${requestedJob}`);
|
||||
if (!Number.isSafeInteger(pending.launchId) || pending.launchId < 1) fail('job_run_invalid');
|
||||
await verifyProductJob(api, { jobRunId: requestedJob, launchId: pending.launchId }, env, true);
|
||||
}
|
||||
// Start and stop are separated so the official job run remains open while
|
||||
// the existing isolated GitHub matrix runs in other jobs.
|
||||
// A final always() job uses the same official CI identity even if start
|
||||
// times out ambiguously or cancellation interrupts the following reads.
|
||||
await appendFile(env.GITHUB_OUTPUT, 'job_run_started=true\n');
|
||||
await exec('allurectl', ['job-run', 'start'], { env, timeout: 60_000, maxBuffer: 32_768 });
|
||||
const { stdout } = await exec('allurectl', ['job-run', 'env'], { env, timeout: 30_000, maxBuffer: 16_384 });
|
||||
const context = officialProductContext(stdout);
|
||||
if (requestedJob && requestedJob !== context.jobRunId) fail('job_run_mismatch');
|
||||
await appendFile(env.GITHUB_OUTPUT, `job_run_id=${context.jobRunId}\nlaunch_url=${endpoint}/launch/${context.launchId}\n`);
|
||||
const boundEnv = { ...env, ALLURE_JOB_RUN_ID: context.jobRunId };
|
||||
const job = await verifyProductJob(api, context, env);
|
||||
let plan;
|
||||
if (requestedJob) {
|
||||
const planFile = path.join(env.RUNNER_TEMP, 'product-regression-testplan.json');
|
||||
await exec('allurectl', ['job-run', 'plan', '--job-run-id', context.jobRunId, '--output-file', planFile],
|
||||
{ env: boundEnv, timeout: 30_000, maxBuffer: 32_768 });
|
||||
plan = await readJson(planFile, 65_536);
|
||||
}
|
||||
const cases = manualCases ?? selectProductRegression(plan, env.TESTOPS_CASE || 'all', true);
|
||||
const selection = { schema: 'paperclip.product-regression-testops-selection/v1',
|
||||
runId: env.GITHUB_RUN_ID, attempt: env.GITHUB_RUN_ATTEMPT, workflowSha: env.GITHUB_SHA,
|
||||
targetSha: env.SOURCE_SHA, ...context, ...job, cases };
|
||||
validateSelection(selection, env);
|
||||
const bytes = Buffer.from(JSON.stringify(selection, null, 2));
|
||||
await mkdir(path.dirname(argument), { recursive: true });
|
||||
await writeFile(argument, bytes, { flag: 'wx', mode: 0o600 });
|
||||
await appendFile(env.GITHUB_OUTPUT, `ids=${cases.map(row => row.executionId).join(',')}\nselection_sha256=${hash(bytes)}\n`);
|
||||
return 0;
|
||||
}
|
||||
if (!['project', 'verify'].includes(command)) fail('command_invalid');
|
||||
const bytes = await readFile(argument);
|
||||
if (bytes.length > 65_536 || hash(bytes) !== env.SELECTION_SHA256) fail('selection_hash');
|
||||
const selection = validateSelection(JSON.parse(bytes), env);
|
||||
if (env.ALLURE_JOB_RUN_ID !== selection.jobRunId) fail('job_run_mismatch');
|
||||
if (command === 'project') {
|
||||
let campaign;
|
||||
try { campaign = await readJson(path.join(env.REPORT_DIR, 'normalized', 'normalized-results.json'), 20 * 1024 * 1024); }
|
||||
catch (error) { if (error.code !== 'ENOENT') throw error; }
|
||||
const { records, summary } = projectProductResults(campaign, selection, env.SOURCE_SHA);
|
||||
await mkdir(env.ALLURE_RESULTS, { recursive: false, mode: 0o700 });
|
||||
for (const row of records) await writeFile(path.join(env.ALLURE_RESULTS, row.file), JSON.stringify(row.value), { flag: 'wx', mode: 0o600 });
|
||||
await writeFile(env.EXPORT_SUMMARY, JSON.stringify(summary), { flag: 'wx', mode: 0o600 });
|
||||
await appendFile(env.GITHUB_STEP_SUMMARY, `### [Product regression TestOps results](${endpoint}/launch/${selection.launchId})\n\n${summary.passed}/${summary.selected} passed; ${summary.failed} failed; ${summary.broken} broken. ${summary.fullPlan ? 'Complete 20-case plan selection.' : 'Targeted selection; not a full-plan verdict.'}\n`);
|
||||
return 0;
|
||||
}
|
||||
const api = await testopsClient(env.ALLURE_TOKEN, fetchImpl);
|
||||
await verifyProductJob(api, selection, env);
|
||||
const summary = await readJson(env.EXPORT_SUMMARY, 16_384);
|
||||
// Check server-side identity and count. A successful upload is not a pass.
|
||||
let observed, lastVerification;
|
||||
for (let attempt = 0; attempt < 12; attempt++) {
|
||||
observed = await api(`/api/testresult?${new URLSearchParams({ launchId: String(selection.launchId), page: '0', size: '100' })}`);
|
||||
if (!Array.isArray(observed?.content) || observed.totalPages > 1 || observed.totalElements !== observed.content.length) fail('result_page_invalid');
|
||||
const actual = observed.content;
|
||||
if (actual.length === selection.cases.length && actual.every(row => selection.cases.some(value => value.id === row.testCaseId))) {
|
||||
const details = await Promise.all(actual.map(row => api(`/api/testresult/${row.id}`)));
|
||||
try { validateUploadedProductResults(details, selection, summary); return 0; }
|
||||
catch (error) {
|
||||
if (!['product_regression_testops_uploaded_identity_mismatch', 'product_regression_testops_uploaded_status_mismatch'].includes(error.message)) throw error;
|
||||
lastVerification = error;
|
||||
}
|
||||
}
|
||||
// Planned rows can exist before the asynchronous upload processor settles
|
||||
// their run binding and verdict. Observe this one upload, without rerunning
|
||||
// the product or accepting unknown/partial results.
|
||||
if (attempt === 11) { if (lastVerification) throw lastVerification; fail('upload_incomplete'); }
|
||||
await wait(2_000);
|
||||
}
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
productTestOps(process.argv[2], process.argv[3]).then(code => { process.exitCode = code; }).catch(error => {
|
||||
const safe = /^product_regression_testops_[a-z_]+$/.test(error.message) ? error.message : 'product_regression_testops_failed';
|
||||
process.stderr.write(`${safe}\n`); process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
export interface ProductRegressionCase {
|
||||
id: number;
|
||||
suite: string;
|
||||
name: string;
|
||||
profile: string;
|
||||
fullName: string;
|
||||
executionId: string;
|
||||
}
|
||||
|
||||
export const productRegressionCases: readonly ProductRegressionCase[];
|
||||
@@ -0,0 +1,163 @@
|
||||
import { createHash, randomUUID } from 'node:crypto';
|
||||
|
||||
// These are the standing, reviewed TestOps plan-2 identities. A case selects one
|
||||
// existing full-stack cell; it does not change the prompt, matcher or runtime.
|
||||
export const productRegressionCases = [
|
||||
[8, 'core-compatibility', 'plan-revise-accept'],
|
||||
[9, 'configured-repository', 'repository-output'],
|
||||
[30, 'core-compatibility', 'ask-question'],
|
||||
[32, 'first-task', 'reject-no-execution'],
|
||||
[35, 'agent-chat', 'reassign-task'],
|
||||
[45, 'agent-chat', 'multi-repository'],
|
||||
[54, 'task-workspaces', 'task-directory-repository-resume', 'runner-opencode'],
|
||||
[60, 'first-task', 'task-card-accept'],
|
||||
[61, 'continuation', 'completed-action-resume'],
|
||||
[62, 'agent-chat', 'clarify-reuse'],
|
||||
[66, 'core-compatibility', 'message-marker'],
|
||||
[68, 'continuation', 'revision-preserves-approval'],
|
||||
[112, 'agent-chat', 'create-backlog'],
|
||||
[123, 'first-task', 'revise-accept'],
|
||||
[134, 'first-task', 'clarify-propose-accept'],
|
||||
[140, 'agent-chat', 'stop-new-resume'],
|
||||
[145, 'agent-chat', 'continuity-restart'],
|
||||
[159, 'continuation', 'clarification-not-approval'],
|
||||
[164, 'agent-chat', 'plan-handoff'],
|
||||
[168, 'agent-chat', 'new-session'],
|
||||
].map(([id, suite, name, profile = 'runner-codex']) => Object.freeze({
|
||||
id, suite, name, profile,
|
||||
fullName: `paperclip.Product E2E.${suite}.${name}`,
|
||||
executionId: `${suite}.${profile}.local.${name}`,
|
||||
}));
|
||||
|
||||
const fail = code => { throw new Error(`product_regression_testops_${code}`); };
|
||||
const sha = value => createHash('sha256').update(value).digest('hex');
|
||||
const identifier = value => typeof value === 'string' && /^[a-zA-Z0-9._:-]{1,200}$/.test(value);
|
||||
const modelIdentifier = value => typeof value === 'string' && /^[a-zA-Z0-9][a-zA-Z0-9._:/-]{0,199}$/.test(value);
|
||||
export const endpoint = 'https://paperclip.testops.cloud';
|
||||
export const workflowUrl = 'https://github.com/paperclipai/paperclip/actions/workflows/runner-full-stack-e2e.yml';
|
||||
// Official GitHub/Allure job metadata uses these exact aliases for workflow
|
||||
// 343807492, independently read from GitHub's workflow API before registration.
|
||||
export const workflowUrls = new Set([workflowUrl,
|
||||
'https://github.com/paperclipai/paperclip/blob/master/.github/workflows/runner-full-stack-e2e.yml',
|
||||
'https://github.com/paperclipai/paperclip/actions/workflows/343807492',
|
||||
]);
|
||||
|
||||
export function selectProductRegression(plan, manual = 'all', fromTestOps = false) {
|
||||
if (fromTestOps && !plan) fail('plan_required');
|
||||
if (plan) {
|
||||
if (plan.version !== '1.0' || !Array.isArray(plan.tests) || !plan.tests.length || plan.tests.length > productRegressionCases.length) fail('plan_invalid');
|
||||
const selected = plan.tests.map(test => {
|
||||
const row = productRegressionCases.find(value => String(value.id) === String(test.id));
|
||||
if (!row || test.selector !== row.fullName) fail('case_mismatch');
|
||||
return row;
|
||||
});
|
||||
if (new Set(selected.map(row => row.id)).size !== selected.length) fail('duplicate_case');
|
||||
return selected.sort((a, b) => a.id - b.id);
|
||||
}
|
||||
if (manual === 'all') return [...productRegressionCases];
|
||||
const selected = productRegressionCases.find(row => row.name === manual || String(row.id) === manual);
|
||||
if (!selected) fail('manual_selection_invalid');
|
||||
return [selected];
|
||||
}
|
||||
|
||||
export function validateSelection(value, env) {
|
||||
if (value?.schema !== 'paperclip.product-regression-testops-selection/v1'
|
||||
|| value.runId !== env.GITHUB_RUN_ID || value.attempt !== env.GITHUB_RUN_ATTEMPT
|
||||
|| value.workflowSha !== env.GITHUB_SHA || !/^[a-f0-9]{40}$/.test(value.workflowSha)
|
||||
|| value.targetSha !== env.SOURCE_SHA || !/^[a-f0-9]{40}$/.test(value.targetSha)
|
||||
|| !/^[1-9][0-9]{0,15}$/.test(value.jobRunId) || !Array.isArray(value.cases)
|
||||
|| !value.cases.length || value.cases.length > productRegressionCases.length) fail('selection_binding');
|
||||
const selected = selectProductRegression({ version: '1.0', tests: value.cases.map(row => ({ id: row.id, selector: row.fullName })) }, 'all', true);
|
||||
if (JSON.stringify(selected) !== JSON.stringify(value.cases)) fail('selection_binding');
|
||||
return value;
|
||||
}
|
||||
|
||||
export function projectProductResults(campaign, selection, sourceSha) {
|
||||
if (!/^[a-f0-9]{40}$/.test(sourceSha)) fail('source_missing');
|
||||
if (campaign && (campaign.schema !== 'paperclip.runner-e2e.campaign/v2'
|
||||
|| campaign.campaignId !== `gha-${selection.runId}-${selection.attempt}` || campaign.source?.sha !== sourceSha
|
||||
|| !Array.isArray(campaign.results) || !Array.isArray(campaign.expected)
|
||||
|| JSON.stringify([...campaign.expected].sort()) !== JSON.stringify(selection.cases.map(row => row.executionId).sort()))) fail('report_binding');
|
||||
const rows = campaign?.results ?? [];
|
||||
if (rows.length > selection.cases.length || new Set(rows.map(row => row.executionId)).size !== rows.length
|
||||
|| rows.some(row => !selection.cases.some(value => value.executionId === row.executionId))) fail('unexpected_result');
|
||||
const accounting = campaign?.accounting;
|
||||
if (accounting && (accounting.version !== 1 || !['retained_attempts', 'selected_results'].includes(accounting.scope)
|
||||
|| !Array.isArray(accounting.attempts) || accounting.attempts.length > selection.cases.length * 10
|
||||
|| !Number.isSafeInteger(accounting.missingAttempts) || accounting.missingAttempts < 0
|
||||
|| !Number.isSafeInteger(accounting.excludedArtifacts) || accounting.excludedArtifacts < 0
|
||||
|| accounting.attempts.some(row => !selection.cases.some(value => value.executionId === row.executionId)
|
||||
|| !Number.isSafeInteger(row.attempt) || row.attempt < 1 || row.attempt > 10
|
||||
|| !['passed', 'failed'].includes(row.status) || !['passed', 'failed', 'not_started'].includes(row.cleanup)))) fail('accounting_invalid');
|
||||
const records = [], summaries = [];
|
||||
for (const selected of selection.cases) {
|
||||
const attempts = rows.filter(row => row.executionId === selected.executionId).sort((a, b) => a.attempt - b.attempt);
|
||||
if (new Set(attempts.map(row => row.attempt)).size !== attempts.length) fail('duplicate_attempt');
|
||||
if (!attempts.length) attempts.push(null);
|
||||
for (const result of attempts) {
|
||||
const missingArtifact = result?.attempt === 0 && result.status === 'failed'
|
||||
&& result.failureClass === 'permanent_infrastructure' && result.cleanup === 'not_started'
|
||||
&& result.error === 'No result artifact was uploaded' && result.matcherResults === undefined;
|
||||
if (result && ((!missingArtifact && result.source?.sha !== sourceSha)
|
||||
|| (missingArtifact && result.source?.sha != null && result.source.sha !== sourceSha)
|
||||
|| result.suiteId !== selected.suite || result.caseId !== selected.name
|
||||
|| result.profileId !== selected.profile || result.environmentId !== 'local'
|
||||
|| !Number.isSafeInteger(result.attempt) || result.attempt < (missingArtifact ? 0 : 1) || result.attempt > 10
|
||||
|| !['passed', 'failed'].includes(result.status) || (!missingArtifact && !Array.isArray(result.matcherResults)))) fail('result_binding');
|
||||
const checks = result?.matcherResults ?? [];
|
||||
if (checks.length > 300 || checks.some(check => typeof check.passed !== 'boolean')) fail('checks_invalid');
|
||||
const passed = result?.status === 'passed' && result.cleanup === 'passed' && checks.length > 0 && checks.every(check => check.passed);
|
||||
const infrastructure = !result || ['permanent_infrastructure', 'transient_infrastructure', 'setup_blocked'].includes(result.failureClass);
|
||||
const status = passed ? 'passed' : infrastructure ? 'broken' : 'failed';
|
||||
const provenance = {
|
||||
schema: 'paperclip.product-regression-testops-result/v1', caseId: selected.id,
|
||||
executionId: selected.executionId, workflowRunId: selection.runId, workflowAttempt: selection.attempt,
|
||||
workflowSha: selection.workflowSha, sourceSha, status, originalStatus: result?.status ?? null,
|
||||
attempt: result?.attempt ?? null, cleanup: result?.cleanup ?? 'not_started',
|
||||
profile: selected.profile, environment: 'local',
|
||||
model: modelIdentifier(result?.model) ? result.model : null,
|
||||
suiteDefinitionHash: /^[a-f0-9]{64}$/.test(result?.suiteDefinitionHash ?? '') ? result.suiteDefinitionHash : null,
|
||||
failureClass: identifier(result?.failureClass) ? result.failureClass : null,
|
||||
durationMs: Number.isFinite(result?.durationMs) && result.durationMs >= 0 ? result.durationMs : null,
|
||||
checks: checks.map(check => ({
|
||||
name: identifier(check.matcher?.path) ? check.matcher.path : identifier(check.matcher?.kind) ? check.matcher.kind : 'assertion',
|
||||
passed: check.passed,
|
||||
})),
|
||||
retainedAttempts: (accounting?.attempts ?? []).filter(row => row.executionId === selected.executionId).map(row => ({
|
||||
attempt: row.attempt, status: row.status, cleanup: row.cleanup,
|
||||
durationMs: Number.isFinite(row.durationMs) && row.durationMs >= 0 ? row.durationMs : null,
|
||||
startedAt: Number.isFinite(Date.parse(row.startedAt)) ? new Date(row.startedAt).toISOString() : null,
|
||||
finishedAt: Number.isFinite(Date.parse(row.finishedAt)) ? new Date(row.finishedAt).toISOString() : null,
|
||||
})),
|
||||
attemptAccounting: accounting ? { scope: accounting.scope, missingAttempts: accounting.missingAttempts,
|
||||
excludedArtifacts: accounting.excludedArtifacts } : null,
|
||||
note: result && !missingArtifact ? 'Original full-stack assertions, attempts and diagnostics remain in the linked workflow report.'
|
||||
: 'No bound full-stack result was retained. This case is unverified and cannot pass.',
|
||||
};
|
||||
const uuid = randomUUID(), attachment = `${randomUUID()}-attachment.json`;
|
||||
const value = {
|
||||
uuid, name: selected.name, fullName: selected.fullName, testCaseId: sha(selected.fullName),
|
||||
historyId: sha(JSON.stringify([selected.fullName, selected.profile, provenance.model, 'local'])),
|
||||
status, stage: 'finished', statusDetails: { message: provenance.note },
|
||||
labels: [{ name: 'AS_ID', value: String(selected.id) }, { name: 'parentSuite', value: 'Product E2E' },
|
||||
{ name: 'suite', value: selected.suite }, { name: 'framework', value: 'Paperclip Evals' }],
|
||||
parameters: [{ name: 'profile', value: selected.profile }, { name: 'environment', value: 'local / GitHub Actions' },
|
||||
{ name: 'model', value: provenance.model ?? 'Not recorded' }, { name: 'sourceSha', value: sourceSha }],
|
||||
links: [{ name: 'Full-stack execution and retained diagnostics', type: 'custom',
|
||||
url: `https://github.com/paperclipai/paperclip/actions/runs/${selection.runId}` },
|
||||
{ name: 'Evaluated source', type: 'custom', url: `https://github.com/paperclipai/paperclip/commit/${sourceSha}` }],
|
||||
attachments: [{ name: 'Bound execution provenance and assertions', source: attachment, type: 'application/json' }],
|
||||
steps: provenance.checks.map(check => ({ name: check.name, status: check.passed ? 'passed' : 'failed', stage: 'finished' })),
|
||||
};
|
||||
if (result && !missingArtifact) for (const [key, date] of [['start', result.startedAt], ['stop', result.finishedAt]]) {
|
||||
const timestamp = Date.parse(date);
|
||||
if (Number.isFinite(timestamp)) value[key] = timestamp;
|
||||
}
|
||||
records.push({ file: `${uuid}-result.json`, value }, { file: attachment, value: provenance });
|
||||
if (result === attempts.at(-1)) summaries.push({ caseId: selected.id, status });
|
||||
}
|
||||
}
|
||||
return { records, summary: { selected: selection.cases.length, passed: summaries.filter(row => row.status === 'passed').length,
|
||||
failed: summaries.filter(row => row.status === 'failed').length, broken: summaries.filter(row => row.status === 'broken').length,
|
||||
fullPlan: selection.cases.length === productRegressionCases.length, cases: summaries } };
|
||||
}
|
||||
@@ -0,0 +1,233 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { mkdtemp, mkdir, readFile, rm, writeFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { productRegressionCases, selectProductRegression, validateSelection, projectProductResults, workflowUrl, workflowUrls } from './product-regression-testops.mjs';
|
||||
import { admitProductTestOps, officialProductContext, verifyProductJob, validateUploadedProductResults, productTestOps } from './product-regression-testops-cli.mjs';
|
||||
|
||||
const workflowSha = 'a'.repeat(40), targetSha = 'b'.repeat(40);
|
||||
const env = {
|
||||
GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/heads/master', GITHUB_EVENT_NAME: 'workflow_dispatch',
|
||||
GITHUB_WORKFLOW_REF: 'paperclipai/paperclip/.github/workflows/runner-full-stack-e2e.yml@refs/heads/master',
|
||||
GITHUB_RUN_ID: '123', GITHUB_RUN_ATTEMPT: '1', GITHUB_SHA: workflowSha, SOURCE_SHA: targetSha,
|
||||
ALLURE_ENDPOINT: 'https://paperclip.testops.cloud', ALLURE_PROJECT_ID: '1',
|
||||
};
|
||||
const selected = productRegressionCases.find(row => row.id === 66);
|
||||
const selection = cases => ({ schema: 'paperclip.product-regression-testops-selection/v1', runId: '123', attempt: '1',
|
||||
workflowSha, targetSha, jobRunId: '456', launchId: 789, cases });
|
||||
const result = (overrides = {}) => ({
|
||||
executionId: selected.executionId, suiteId: selected.suite, caseId: selected.name,
|
||||
profileId: selected.profile, environmentId: 'local', source: { sha: targetSha }, attempt: 1,
|
||||
status: 'passed', cleanup: 'passed', model: 'gpt-6.1-sol', suiteDefinitionHash: 'c'.repeat(64),
|
||||
matcherResults: [{ matcher: { kind: 'json_path', path: 'observed.outputVerified', expected: true }, passed: true }],
|
||||
...overrides,
|
||||
});
|
||||
const campaign = (results = [result()]) => ({ schema: 'paperclip.runner-e2e.campaign/v2', campaignId: 'gha-123-1',
|
||||
source: { sha: targetSha }, expected: [selected.executionId], results });
|
||||
const allPlan = { version: '1.0', tests: productRegressionCases.map(row => ({ id: String(row.id), selector: row.fullName })) };
|
||||
|
||||
test('standing plan has 20 unique imported identities and one existing local cell per case', () => {
|
||||
assert.equal(productRegressionCases.length, 20);
|
||||
assert.equal(new Set(productRegressionCases.map(row => row.id)).size, 20);
|
||||
assert.equal(new Set(productRegressionCases.map(row => row.executionId)).size, 20);
|
||||
assert.deepEqual(productRegressionCases.filter(row => row.profile !== 'runner-codex').map(row => row.id), [54]);
|
||||
assert.equal(selectProductRegression(allPlan, 'message-marker', true).length, 20);
|
||||
assert.equal(selectProductRegression(undefined, 'message-marker')[0].id, 66);
|
||||
});
|
||||
|
||||
test('a TestOps subset selects only its exact original case; unknown, duplicate and mismatched identities fail', () => {
|
||||
const subset = { version: '1.0', tests: [{ id: '66', selector: selected.fullName }] };
|
||||
assert.deepEqual(selectProductRegression(subset, 'all', true), [selected]);
|
||||
for (const invalid of [undefined, { version: '1.0', tests: [] },
|
||||
{ version: '1.0', tests: [{ id: '66', selector: productRegressionCases[0].fullName }] },
|
||||
{ version: '1.0', tests: [...subset.tests, ...subset.tests] },
|
||||
{ version: '1.0', tests: [{ id: '9999', selector: selected.fullName }] }]) {
|
||||
assert.throws(() => selectProductRegression(invalid, 'all', true));
|
||||
}
|
||||
});
|
||||
|
||||
test('selection authority cannot enter through an untrusted workflow, foreign ref, mixed selector or recurring mode', () => {
|
||||
admitProductTestOps(env);
|
||||
for (const mutation of [{ GITHUB_REF: 'refs/heads/candidate' }, { GITHUB_EVENT_NAME: 'pull_request_target' },
|
||||
{ GITHUB_WORKFLOW_REF: 'paperclipai/paperclip/.github/workflows/other.yml@refs/heads/master' },
|
||||
{ ALLURE_ENDPOINT: 'https://attacker.invalid' }, { SELECT_SUITE: 'core-compatibility' },
|
||||
{ EVAL_CADENCE: 'daily' }, { SOURCE_SHA: '' }]) assert.throws(() => admitProductTestOps({ ...env, ...mutation }));
|
||||
validateSelection(selection([selected]), env);
|
||||
assert.throws(() => validateSelection(selection([selected]), { ...env, GITHUB_RUN_ID: '124' }));
|
||||
assert.throws(() => validateSelection(selection([selected]), { ...env, SOURCE_SHA: workflowSha }));
|
||||
});
|
||||
|
||||
test('official environment parsing keeps only bound IDs and rejects duplicate or malformed context', () => {
|
||||
assert.deepEqual(officialProductContext('ALLURE_TOKEN=private\nALLURE_JOB_RUN_ID=456\nALLURE_LAUNCH_ID=789\n'), { jobRunId: '456', launchId: 789 });
|
||||
for (const stdout of ['ALLURE_JOB_RUN_ID=456\n', 'ALLURE_JOB_RUN_ID=456\nALLURE_JOB_RUN_ID=456\nALLURE_LAUNCH_ID=789',
|
||||
'ALLURE_JOB_RUN_ID=456\nALLURE_LAUNCH_ID=not-an-id']) assert.throws(() => officialProductContext(stdout));
|
||||
});
|
||||
|
||||
test('reported pass requires real assertions, successful cleanup and exact current source', () => {
|
||||
const projected = projectProductResults(campaign(), selection([selected]), targetSha);
|
||||
assert.equal(projected.summary.passed, 1);
|
||||
assert.equal(projected.records[0].value.labels.find(row => row.name === 'AS_ID').value, '66');
|
||||
assert.equal(projected.records[0].value.fullName, selected.fullName);
|
||||
for (const partial of [{ cleanup: 'failed' }, { matcherResults: [] },
|
||||
{ matcherResults: [{ matcher: { kind: 'json_path', path: 'output' }, passed: false }] }]) {
|
||||
assert.equal(projectProductResults(campaign([result(partial)]), selection([selected]), targetSha).summary.passed, 0);
|
||||
}
|
||||
assert.throws(() => projectProductResults(campaign([result({ source: { sha: workflowSha } })]), selection([selected]), targetSha));
|
||||
assert.throws(() => projectProductResults({ ...campaign(), campaignId: 'gha-older-1' }, selection([selected]), targetSha));
|
||||
});
|
||||
|
||||
test('missing results remain explicit broken cases and cannot improve the denominator', () => {
|
||||
const projected = projectProductResults(undefined, selection(productRegressionCases), targetSha);
|
||||
const { cases, ...counts } = projected.summary;
|
||||
assert.deepEqual(counts, { selected: 20, passed: 0, failed: 0, broken: 20, fullPlan: true });
|
||||
assert.equal(cases.length, 20);
|
||||
assert.equal(cases.every(row => row.status === 'broken'), true);
|
||||
assert.equal(projected.records.filter(row => row.file.endsWith('-result.json')).length, 20);
|
||||
const partial = projectProductResults(campaign([]), selection([selected]), targetSha);
|
||||
assert.equal(partial.summary.broken, 1);
|
||||
assert.equal(partial.summary.fullPlan, false);
|
||||
const missing = result({ attempt: 0, status: 'failed', failureClass: 'permanent_infrastructure', cleanup: 'not_started',
|
||||
error: 'No result artifact was uploaded', matcherResults: undefined, source: undefined });
|
||||
assert.equal(projectProductResults(campaign([missing]), selection([selected]), targetSha).summary.broken, 1);
|
||||
assert.throws(() => projectProductResults(campaign([{ ...missing, status: 'passed' }]), selection([selected]), targetSha));
|
||||
});
|
||||
|
||||
test('original functional and infrastructure failures survive', () => {
|
||||
for (const [failureClass, expected] of [['candidate_failure', 'failed'], ['permanent_infrastructure', 'broken'], ['transient_infrastructure', 'broken']]) {
|
||||
const projected = projectProductResults(campaign([result({ status: 'failed', failureClass })]), selection([selected]), targetSha);
|
||||
assert.equal(projected.records[0].value.status, expected);
|
||||
}
|
||||
assert.throws(() => projectProductResults(campaign([result(), result()]), selection([selected]), targetSha));
|
||||
});
|
||||
|
||||
test('authoritative normalized accounting preserves earlier failed attempt verdicts beside the selected result', () => {
|
||||
const measured = { ...campaign([result({ attempt: 2 })]), accounting: { version: 1, scope: 'retained_attempts',
|
||||
missingAttempts: 0, excludedArtifacts: 0, attempts: [
|
||||
{ id: 'retained-failure', executionId: selected.executionId, attempt: 1, status: 'failed', cleanup: 'passed', durationMs: 1000 },
|
||||
{ id: 'selected-success', executionId: selected.executionId, attempt: 2, status: 'passed', cleanup: 'passed', durationMs: 1000 },
|
||||
] } };
|
||||
const projected = projectProductResults(measured, selection([selected]), targetSha);
|
||||
assert.equal(projected.summary.passed, 1);
|
||||
assert.equal(projected.records.filter(row => row.file.endsWith('-result.json')).length, 1);
|
||||
assert.deepEqual(projected.records[1].value.retainedAttempts.map(row => row.status), ['failed', 'passed']);
|
||||
assert.deepEqual(projected.records[1].value.attemptAccounting, { scope: 'retained_attempts', missingAttempts: 0, excludedArtifacts: 0 });
|
||||
assert.throws(() => projectProductResults({ ...measured, accounting: { ...measured.accounting, attempts: [{ ...measured.accounting.attempts[0], status: 'unknown' }] } }, selection([selected]), targetSha));
|
||||
const openCode = productRegressionCases.find(row => row.id === 54);
|
||||
const model = 'openrouter/deepseek/deepseek-v4.1-flash-1017';
|
||||
const record = result({ executionId: openCode.executionId, suiteId: openCode.suite, caseId: openCode.name, profileId: openCode.profile, model });
|
||||
const openCodeCampaign = { ...campaign([record]), expected: [openCode.executionId] };
|
||||
assert.equal(projectProductResults(openCodeCampaign, selection([openCode]), targetSha).records[1].value.model, model);
|
||||
});
|
||||
|
||||
test('the transmission projection excludes raw errors, provider IDs, narratives and unexpected fields', () => {
|
||||
const dirty = result({ error: 'secret-error', runIds: ['private-provider-session'], detail: 'private-context',
|
||||
usage: { credential: 'private-token' }, matcherResults: [{ passed: true, detail: 'secret-detail',
|
||||
matcher: { kind: 'contains', expected: 'fixture-answer-secret', path: 'unsafe narrative with secrets' } }] });
|
||||
const text = JSON.stringify(projectProductResults(campaign([dirty]), selection([selected]), targetSha));
|
||||
for (const secret of ['secret-error', 'private-provider-session', 'private-context', 'private-token', 'secret-detail', 'fixture-answer-secret', 'unsafe narrative']) {
|
||||
assert.equal(text.includes(secret), false, secret);
|
||||
}
|
||||
});
|
||||
|
||||
test('a TestOps job run must belong to this workflow, source launch and GitHub execution', async () => {
|
||||
const values = { '/api/jobrun/456': { id: 456, launchId: 789, job: { id: 50, type: 'github' }, url: 'https://github.com/paperclipai/paperclip/actions/runs/123' },
|
||||
'/api/job/50': { id: 50, type: 'github', url: workflowUrl }, '/api/launch/789': { id: 789, projectId: 1 } };
|
||||
assert.deepEqual(await verifyProductJob(route => values[route], { jobRunId: '456', launchId: 789 }, env), { jobId: 50, launchId: 789 });
|
||||
for (const url of workflowUrls) {
|
||||
assert.deepEqual(await verifyProductJob(route => route === '/api/job/50' ? { ...values[route], url } : values[route],
|
||||
{ jobRunId: '456', launchId: 789 }, env), { jobId: 50, launchId: 789 });
|
||||
}
|
||||
const pending = route => route === '/api/jobrun/456' ? { ...values[route], url: null } : values[route];
|
||||
await verifyProductJob(pending, { jobRunId: '456', launchId: 789 }, env, true);
|
||||
await assert.rejects(verifyProductJob(pending, { jobRunId: '456', launchId: 789 }, env));
|
||||
await assert.rejects(verifyProductJob(route => route === '/api/launch/789' ? { id: 789, projectId: 2 } : values[route], { jobRunId: '456', launchId: 789 }, env));
|
||||
await assert.rejects(verifyProductJob(route => route === '/api/job/50' ? { ...values[route], url: 'https://github.com/other/repo' } : values[route], { jobRunId: '456', launchId: 789 }, env));
|
||||
await assert.rejects(verifyProductJob(route => route === '/api/job/50' ? { ...values[route], url: workflowUrl.replace('runner-full-stack-e2e', 'runner-protocol-live-evals') } : values[route], { jobRunId: '456', launchId: 789 }, env));
|
||||
});
|
||||
|
||||
test('server verification accepts real Allure string statuses and rejects wrong verdicts or identities', () => {
|
||||
const rows = [{ testCaseId: selected.id, fullName: selected.fullName, projectId: 1, launchId: 789, jobRun: { id: 456 }, status: 'passed' }];
|
||||
const summary = { passed: 1, failed: 0, broken: 0, cases: [{ caseId: selected.id, status: 'passed' }] };
|
||||
validateUploadedProductResults(rows, selection([selected]), summary);
|
||||
assert.throws(() => validateUploadedProductResults([{ ...rows[0], status: 'failed' }], selection([selected]), summary));
|
||||
assert.throws(() => validateUploadedProductResults([{ ...rows[0], jobRun: { id: 999 } }], selection([selected]), summary));
|
||||
const other = productRegressionCases[0];
|
||||
const mixed = [rows[0], { ...rows[0], testCaseId: other.id, fullName: other.fullName, status: 'failed' }];
|
||||
const expected = { passed: 1, failed: 1, broken: 0, cases: [{ caseId: selected.id, status: 'passed' }, { caseId: other.id, status: 'failed' }] };
|
||||
validateUploadedProductResults(mixed, selection([selected, other]), expected);
|
||||
assert.throws(() => validateUploadedProductResults([{ ...mixed[0], status: 'failed' }, { ...mixed[1], status: 'passed' }],
|
||||
selection([selected, other]), expected), /uploaded_status_mismatch/);
|
||||
});
|
||||
|
||||
test('the separated selection, retained-report projection and server verification preserve one TestOps subset', async t => {
|
||||
const root = await mkdtemp(path.join(tmpdir(), 'product-testops-control-'));
|
||||
t.after(() => rm(root, { recursive: true, force: true }));
|
||||
const selectionFile = path.join(root, 'selection.json');
|
||||
const runtime = { ...env, RUNNER_TEMP: root, GITHUB_OUTPUT: path.join(root, 'output'), GITHUB_STEP_SUMMARY: path.join(root, 'summary'),
|
||||
ALLURE_JOB_RUN_ID: '456', ALLURE_TOKEN: 'synthetic-test-token', TESTOPS_CASE: 'all',
|
||||
REPORT_DIR: path.join(root, 'report'), ALLURE_RESULTS: path.join(root, 'allure'), EXPORT_SUMMARY: path.join(root, 'export.json') };
|
||||
let uploadedStatus = 'passed';
|
||||
const fetchImpl = async url => {
|
||||
const parsed = new URL(url), route = parsed.pathname;
|
||||
const resources = {
|
||||
'/api/uaa/oauth/token': { access_token: 'synthetic-access-token' },
|
||||
'/api/jobrun/456': { id: 456, launchId: 789, job: { id: 50, type: 'github' }, url: 'https://github.com/paperclipai/paperclip/actions/runs/123' },
|
||||
'/api/job/50': { id: 50, type: 'github', url: [...workflowUrls][1] }, '/api/launch/789': { id: 789, projectId: 1 },
|
||||
'/api/testresult': { content: [{ id: 501, testCaseId: 66 }], totalPages: 1, totalElements: 1 },
|
||||
'/api/testresult/501': { id: 501, testCaseId: 66, fullName: selected.fullName, projectId: 1, launchId: 789, jobRun: { id: 456 }, status: uploadedStatus },
|
||||
};
|
||||
assert.ok(resources[route], route);
|
||||
return new Response(JSON.stringify(resources[route]), { status: 200, headers: { 'Content-Type': 'application/json' } });
|
||||
};
|
||||
const exec = async (command, args) => {
|
||||
assert.equal(command, 'allurectl');
|
||||
if (args[1] === 'env') return { stdout: 'ALLURE_JOB_RUN_ID=456\nALLURE_LAUNCH_ID=789\n' };
|
||||
if (args[1] === 'plan') await writeFile(args.at(-1), JSON.stringify({ version: '1.0', tests: [{ id: '66', selector: selected.fullName }] }));
|
||||
return { stdout: '' };
|
||||
};
|
||||
assert.equal(await productTestOps('select', selectionFile, runtime, exec, fetchImpl), 0);
|
||||
const receipt = JSON.parse(await readFile(selectionFile));
|
||||
assert.deepEqual(receipt.cases, [selected]);
|
||||
runtime.SELECTION_SHA256 = (await readFile(runtime.GITHUB_OUTPUT, 'utf8')).match(/selection_sha256=([a-f0-9]{64})/)[1];
|
||||
await mkdir(path.join(runtime.REPORT_DIR, 'normalized'), { recursive: true });
|
||||
await writeFile(path.join(runtime.REPORT_DIR, 'normalized', 'normalized-results.json'), JSON.stringify(campaign()));
|
||||
assert.equal(await productTestOps('project', selectionFile, runtime, exec, fetchImpl), 0);
|
||||
assert.equal(await productTestOps('verify', selectionFile, runtime, exec, fetchImpl), 0);
|
||||
uploadedStatus = 'unknown';
|
||||
let readinessWaits = 0;
|
||||
assert.equal(await productTestOps('verify', selectionFile, runtime, exec, fetchImpl, async () => { readinessWaits++; uploadedStatus = 'passed'; }), 0);
|
||||
assert.equal(readinessWaits, 1);
|
||||
const failureOutput = path.join(root, 'failed-selection-output');
|
||||
const badPlan = async (command, args) => args[1] === 'plan'
|
||||
? (await writeFile(args.at(-1), JSON.stringify({ version: '1.0', tests: [{ id: '999', selector: selected.fullName }] })), { stdout: '' })
|
||||
: exec(command, args);
|
||||
await assert.rejects(productTestOps('select', path.join(root, 'failed-selection.json'), { ...runtime, GITHUB_OUTPUT: failureOutput }, badPlan, fetchImpl), /case_mismatch/);
|
||||
const cleanupContext = await readFile(failureOutput, 'utf8');
|
||||
assert.match(cleanupContext, /job_run_started=true/);
|
||||
assert.match(cleanupContext, /job_run_id=456/);
|
||||
let opened = false;
|
||||
await assert.rejects(productTestOps('select', path.join(root, 'invalid-manual.json'),
|
||||
{ ...runtime, ALLURE_JOB_RUN_ID: '', TESTOPS_CASE: 'unknown' }, async () => { opened = true; }, fetchImpl), /manual_selection_invalid/);
|
||||
assert.equal(opened, false);
|
||||
uploadedStatus = 'failed';
|
||||
await assert.rejects(productTestOps('verify', selectionFile, runtime, exec, fetchImpl, async () => {}), /uploaded_status_mismatch/);
|
||||
await writeFile(selectionFile, JSON.stringify({ ...receipt, targetSha: workflowSha }));
|
||||
await assert.rejects(productTestOps('project', selectionFile, runtime, exec, fetchImpl), /selection_hash/);
|
||||
});
|
||||
|
||||
test('workflow keeps reporting credentials and trusted code outside candidate/provider jobs', async () => {
|
||||
const workflow = await readFile(new URL('../../.github/workflows/runner-full-stack-e2e.yml', import.meta.url), 'utf8');
|
||||
const selectionJob = workflow.split(' testops_selection:\n')[1].split('\n catalog:\n')[0];
|
||||
const reportingJob = workflow.split(' publish_testops:\n')[1].split('\n publish_history:\n')[0];
|
||||
for (const block of [selectionJob, reportingJob]) {
|
||||
assert.match(block, /environment: runner-e2e-testops-reporting/);
|
||||
assert.match(block, /ref: \$\{\{ github.sha \}\}/);
|
||||
assert.doesNotMatch(block, /secrets\.(OPENAI|OPENROUTER|ANTHROPIC)|ref: \$\{\{ needs.authorize.outputs.target_sha/);
|
||||
}
|
||||
assert.match(workflow, /SELECT_ID: \$\{\{ needs.testops_selection.outputs.ids \|\| inputs.id \}\}/);
|
||||
assert.match(reportingJob.split(' steps:\n')[0], /if: always\(\) && needs.testops_selection.outputs.job_run_started == 'true'/);
|
||||
assert.doesNotMatch(reportingJob.split(' steps:\n')[0], /!cancelled\(\)/);
|
||||
assert.match(reportingJob, /Finish the original TestOps job run\n\s+if: always\(\)/);
|
||||
assert.doesNotMatch(workflow.slice(0, workflow.indexOf('\njobs:')), /workflow_call:/);
|
||||
});
|
||||
@@ -20,7 +20,7 @@ delete env.NODE_OPTIONS;
|
||||
|
||||
for (const [name, command] of [
|
||||
["Vitest", [vitest, "run", "--config", "tests/runner-e2e/vitest.config.ts", ...args]],
|
||||
["Node tests", ["--test", "tests/runner-e2e/artifact-budget.test.mjs", "tests/runner-e2e/native-completion-checks.test.mjs",
|
||||
["Node tests", ["--test", "tests/runner-e2e/product-regression-testops.test.mjs", "tests/runner-e2e/artifact-budget.test.mjs", "tests/runner-e2e/native-completion-checks.test.mjs",
|
||||
"tests/runner-e2e/native-completion-git-source.test.mjs", "tests/runner-e2e/native-completion-source-contract.test.mjs",
|
||||
"tests/runner-e2e/chat-feature-testops-api.test.mjs", "tests/runner-e2e/chat-feature-testops-cli.test.mjs", "tests/runner-e2e/chat-feature-testops-results.test.mjs", "tests/runner-e2e/testops-chat-reporter.test.mjs", "tests/runner-e2e/chat-feature-testops-retained-verification.node-test.mjs",
|
||||
"tests/runner-e2e/chat-completion-testops-api.test.mjs", "tests/runner-e2e/chat-completion-testops-contract.test.mjs",
|
||||
|
||||
@@ -7,7 +7,7 @@ export default defineConfig({
|
||||
globalSetup: ["./verified-provider-bundles.setup.mjs"],
|
||||
include: ["**/*.test.ts", "**/*.test.mjs"],
|
||||
// These use node:test and run separately in test:e2e:runner:unit.
|
||||
exclude: [...configDefaults.exclude, "native-completion-{checks,git-source,source-contract}.test.mjs", "artifact-budget.test.mjs", "chat-feature-testops-api.test.mjs", "chat-feature-testops-cli.test.mjs", "chat-feature-testops-results.test.mjs", "testops-chat-reporter.test.mjs", "chat-completion-testops-{api,contract}.test.mjs", "chat-full-catalog-testops-*.test.mjs", "chat-catalog-testops-streaming.test.mjs"],
|
||||
exclude: [...configDefaults.exclude, "product-regression-testops.test.mjs", "native-completion-{checks,git-source,source-contract}.test.mjs", "artifact-budget.test.mjs", "chat-feature-testops-api.test.mjs", "chat-feature-testops-cli.test.mjs", "chat-feature-testops-results.test.mjs", "testops-chat-reporter.test.mjs", "chat-completion-testops-{api,contract}.test.mjs", "chat-full-catalog-testops-*.test.mjs", "chat-catalog-testops-streaming.test.mjs"],
|
||||
testTimeout: 30_000,
|
||||
},
|
||||
});
|
||||
Reference in new issue
Block a user