From a1845b3b6465c47f50e81ac79ebbc8cb87c84b68 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 30 Oct 2026 16:07:04 -0500 Subject: [PATCH] fix(evals): capture TestOps registration after watch (#18375) ## Thinking Path > - Paperclip manages AI agents and verifies their work through Product E2E tests. > - The fixed Chat feature workflow prepares a TestOps case and plan before any model run. > - Its first zero-result bootstrap completed, but the watched child did not receive the generated job-run ID. > - The official watch log contained a job-run ID, so the missing child variable did not prove that registration was absent. > - This pull request reads official context after watch and validates the linked backend records. > - The workflow retains closed registration metadata and stops on unknown or foreign context. ## Linked Issues or Issue Description Refs: #18322. **What happened?** The successful zero-result bootstrap retained `official_job_run_id_missing` from its watched child. It could not prove the backend job and workflow binding needed for the next setup step. **Expected behavior** After successful watch, read the generated context, validate its current GitHub run and fixed workflow, and retain only closed IDs and parameter metadata. **Steps to reproduce** Run the fixed workflow's `bootstrap` operation on master. Inspect its setup artifact after watch completes. No evaluation or synthetic passing result is needed. **Paperclip version or commit** The original bootstrap used merged #18322. This follow-up starts from `76fde891ccc29283b3f5c8dcdce089c5b3d937d1`. **Deployment mode** GitHub Actions metadata-only setup in the protected reporting environment. ## What Changed - Run an internal context reader after successful bootstrap watch in the same protected step. - Parse only numeric official job-run and launch IDs in memory. Do not log or save raw command output. - Validate current-run, project, launch, job, workflow, and bounded parameter metadata before writing an exclusive registration receipt. - Retain the two known registration receipt paths with the existing setup artifact. - Preserve the original inside-watch receipt, case reconciliation, provider isolation, and evaluation limits. ## Verification - `node --test tests/runner-e2e/chat-feature-testops-api.test.mjs tests/runner-e2e/testops-chat-reporter.test.mjs`: 48 pure/mock/FS controls pass (36 prior and 12 new). The actual local Node version was 26.4.0. Normal Node 24 CI remains required. - Four MJS syntax checks and `git diff --check` pass. No TypeScript noEmit claim is made for this MJS-only helper change. - Actionlint passes for the changed workflow. - Normal [GitHub Actions CI](https://github.com/paperclipai/paperclip/actions/runs/41123894542) passes on exact head `58396de10f0c49294b0bbc0f4709c29990d57d53`: 48 jobs succeeded and one skipped. Both required `ci / e2e` and `ci / verify` pass. - Greptile reviewed that exact head at 5/5 with no actionable findings or unresolved review threads. - No local CLI, provider, model, app, database, or build qualification was run. No replacement bootstrap or paid evaluation was dispatched. ## Risks The official context or backend DTO shape can differ from the supported format. The reader then stops with a closed failure receipt. Pure tests do not qualify the remote CLI or backend. The original zero-result bootstrap and its missing child-context observation remain part of the history. The case's actual outcome remains unmeasured. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, code editing, tool use, and pure test review. The runtime does not expose the exact context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .github/workflows/chat-feature-testops.yml | 3 + tests/runner-e2e/chat-feature-testops-api.mjs | 27 +++- .../chat-feature-testops-api.test.mjs | 42 ++++++ tests/runner-e2e/testops-chat-reporter.mjs | 65 +++++++- .../runner-e2e/testops-chat-reporter.test.mjs | 142 +++++++++++++++++- 5 files changed, 270 insertions(+), 9 deletions(-) diff --git a/.github/workflows/chat-feature-testops.yml b/.github/workflows/chat-feature-testops.yml index 232408401d0..826a340ae91 100644 --- a/.github/workflows/chat-feature-testops.yml +++ b/.github/workflows/chat-feature-testops.yml @@ -89,6 +89,7 @@ jobs: run: | mkdir -m 700 "$ALLURE_RESULTS" allurectl watch -- node tests/runner-e2e/testops-chat-reporter.mjs bootstrap + node tests/runner-e2e/testops-chat-reporter.mjs bootstrap-context - name: Validate the one real TestOps selection before any build or provider access id: selection if: needs.authorize.outputs.operation == 'evaluate' @@ -134,6 +135,8 @@ jobs: ${{ runner.temp }}/chat-testops-bootstrap-state/plan.json ${{ runner.temp }}/chat-testops-bootstrap-state/job.json ${{ runner.temp }}/chat-testops-bootstrap-state/failure.json + ${{ runner.temp }}/chat-testops-bootstrap-state/registration.json + ${{ runner.temp }}/chat-testops-bootstrap-state/registration-failure.json retention-days: 30 if-no-files-found: error - name: Retain the one plan-launch receipt and stop state diff --git a/tests/runner-e2e/chat-feature-testops-api.mjs b/tests/runner-e2e/chat-feature-testops-api.mjs index e77bbc4edac..e6c73b97941 100644 --- a/tests/runner-e2e/chat-feature-testops-api.mjs +++ b/tests/runner-e2e/chat-feature-testops-api.mjs @@ -19,7 +19,7 @@ const diagnosticCodes = new Set(['untrusted_chat_testops_context', 'testops_toke 'chat_testops_case_mismatch', 'chat_testops_plan_mismatch', 'chat_testops_plan_job_mismatch', 'chat_testops_job_mismatch', 'chat_testops_jobrun_mismatch', 'chat_testops_launch_mismatch', 'chat_testops_selection_mismatch', 'chat_testops_parameter_mismatch', 'chat_testops_nonce_invalid', 'chat_testops_nonce_conflict', 'chat_testops_catalog_incomplete', 'chat_testops_launch_admission', 'chat_testops_github_http', 'chat_testops_integration_mismatch', 'chat_testops_upload_mismatch', 'chat_testops_upload_unverified']); export function failureProjection(error, stage) { - const stages = ['context', 'output_admission', 'authentication', 'metadata_inspection', 'plan_bootstrap', 'selection_validation', 'launch_admission', 'launch_preflight', 'plan_launch', 'bootstrap_job_metadata', 'result_export', 'upload_verification', 'receipt']; + const stages = ['context', 'output_admission', 'authentication', 'metadata_inspection', 'plan_bootstrap', 'selection_validation', 'launch_admission', 'launch_preflight', 'plan_launch', 'bootstrap_job_metadata', 'result_export', 'upload_verification', 'bootstrap_context', 'receipt']; return { stage: stages.includes(stage) ? stage : 'unknown', code: error instanceof SetupFailure && diagnosticCodes.has(error.code) ? error.code : 'unknown_failure', httpStatus: error instanceof SetupFailure && Number.isInteger(error.status) && error.status >= 100 && error.status < 600 ? error.status : null }; @@ -188,21 +188,38 @@ async function registeredPlan(api, env) { return { ...ids, selected, plan, job }; } -export async function bootstrapJobMetadata(api, env) { - if (!positive(env.ALLURE_JOB_RUN_ID)) return { status: 'official_job_run_id_missing', actualResult: null }; +export async function bootstrapJobMetadata(api, env, expectedLaunchId) { + const strict = expectedLaunchId !== undefined; + const context = strict ? publicContext(env) : undefined; + if (strict && !id(expectedLaunchId)) fail('chat_testops_launch_mismatch'); + if (!positive(env.ALLURE_JOB_RUN_ID)) { + if (strict) fail('chat_testops_jobrun_mismatch'); + return { status: 'official_job_run_id_missing', actualResult: null }; + } const run = await api(`/api/jobrun/${env.ALLURE_JOB_RUN_ID}`); if (!id(run?.job?.id)) fail('chat_testops_jobrun_mismatch'); + if (strict) { + if (run.id !== Number(env.ALLURE_JOB_RUN_ID) || run.job.type !== 'github' || run.launchId !== expectedLaunchId + || run.url !== `https://github.com/paperclipai/paperclip/actions/runs/${context.runId}`) fail('chat_testops_jobrun_mismatch'); + const launch = await api(`/api/launch/${expectedLaunchId}`); + if (launch.id !== expectedLaunchId || launch.projectId !== 1) fail('chat_testops_launch_mismatch'); + } const job = await api(`/api/job/${run.job.id}`); if (job.id !== run.job.id || job.type !== 'github' || !Array.isArray(job.parameters) || job.parameters.length > 4) fail('chat_testops_job_mismatch'); + if (strict && !publicJobUrl(job.url, env)) fail('chat_testops_job_mismatch'); const parameters = []; + const seen = new Set(); for (const parameter of job.parameters) { - if (!id(parameter.id)) fail('chat_testops_parameter_mismatch'); + if (!id(parameter.id) || strict && seen.has(parameter.id)) fail('chat_testops_parameter_mismatch'); + seen.add(parameter.id); const definition = await api(`/api/environment/${parameter.id}`); + if (strict && definition.id !== parameter.id) fail('chat_testops_parameter_mismatch'); const known = ['operation', 'ALLURE_JOB_RUN_ID', 'ALLURE_USERNAME'].includes(definition.name); parameters.push({ id: parameter.id, name: known ? definition.name : 'UNKNOWN', value: definition.name === 'operation' && ['bootstrap', 'evaluate', 'launch-plan'].includes(parameter.value) ? parameter.value : 'NOT_PROJECTED' }); } - return { status: 'observed', projectId: 1, jobId: job.id, allureJobRunId: env.ALLURE_JOB_RUN_ID, + return { ...(strict ? { schema: 'paperclip.chat-feature-testops-bootstrap-registration/v1', ...context, capturedAfterWatch: true, runnableBindingQualified: false } : {}), + status: 'observed', projectId: 1, jobId: job.id, allureJobRunId: env.ALLURE_JOB_RUN_ID, publicWorkflowMatches: publicJobUrl(job.url, env), currentRunUrlMatches: run.url === `https://github.com/paperclipai/paperclip/actions/runs/${env.GITHUB_RUN_ID}`, launchId: id(run.launchId) ? run.launchId : null, parameters, actualResult: null }; } diff --git a/tests/runner-e2e/chat-feature-testops-api.test.mjs b/tests/runner-e2e/chat-feature-testops-api.test.mjs index a64344d25cb..40ea857bf6d 100644 --- a/tests/runner-e2e/chat-feature-testops-api.test.mjs +++ b/tests/runner-e2e/chat-feature-testops-api.test.mjs @@ -390,3 +390,45 @@ test('readback stops immediately after a late GET and gives each client only the }, uploadContext, uploadSelection, uploadReceipt, { now: () => time }), /upload_unverified/); assert.equal(calls, 2); }); + +function strictBootstrapApi(patch = {}) { + const run = { id: 9, job: { id: 7, type: 'github' }, launchId: 10, url: 'https://github.com/paperclipai/paperclip/actions/runs/123', ...patch.run }; + const launch = { id: 10, projectId: 1, ...patch.launch }; + const job = { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml', + parameters: [{ id: 17, value: 'bootstrap' }, { id: 18, get value() { assert.fail('unrelated parameter value must stay unopened'); } }], ...patch.job }; + return async (route, method) => { + assert.equal(method, undefined); + if (route === '/api/jobrun/9') return run; + if (route === '/api/launch/10') return launch; + if (route === '/api/job/7') return job; + if (route === '/api/environment/17') return { id: 17, name: 'operation', ...patch.definition }; + if (route === '/api/environment/18') return { id: 18, name: 'SYNTHETIC_PRIVATE_NAME' }; + assert.fail(route); + }; +} + +test('post-watch reader binds actual current run source project launch and public workflow without claiming runnable', async () => { + const result = await bootstrapJobMetadata(strictBootstrapApi(), publicEnv, 10); + assert.equal(result.jobId, 7); assert.equal(result.runId, publicEnv.GITHUB_RUN_ID); + assert.equal(result.workflowSha, publicEnv.GITHUB_SHA); assert.equal(result.capturedAfterWatch, true); + assert.equal(result.runnableBindingQualified, false); assert.equal(result.actualResult, null); + assert.deepEqual(result.parameters[1], { id: 18, name: 'UNKNOWN', value: 'NOT_PROJECTED' }); + assert.ok(!JSON.stringify(result).includes('SYNTHETIC_PRIVATE')); +}); + +test('post-watch reader rejects stale foreign project launch source and job identity', async () => { + for (const patch of [{ run: { id: 99 } }, { run: { url: 'https://github.com/paperclipai/paperclip/actions/runs/999' } }, + { run: { launchId: 20 } }, { run: { job: { id: 7, type: 'foreign' } } }, { launch: { projectId: 2 } }, + { launch: { id: 20 } }, { job: { url: 'https://github.com/other/repo/actions/workflows/chat-feature-testops.yml' } }, + { job: { type: 'foreign' } }, { definition: { id: 99 } }]) { + await assert.rejects(bootstrapJobMetadata(strictBootstrapApi(patch), publicEnv, 10)); + } + await assert.rejects(bootstrapJobMetadata(() => assert.fail('no API'), { ...publicEnv, GITHUB_SHA: 'untrusted' }, 10)); + await assert.rejects(bootstrapJobMetadata(() => assert.fail('no API'), { ...publicEnv, ALLURE_JOB_RUN_ID: '' }, 10)); +}); + +test('post-watch reader caps parameter definitions and refuses duplicate IDs', async () => { + for (const parameters of [Array.from({ length: 5 }, (_, i) => ({ id: i + 1 })), [{ id: 17 }, { id: 17 }]]) { + await assert.rejects(bootstrapJobMetadata(strictBootstrapApi({ job: { parameters } }), publicEnv, 10)); + } +}); diff --git a/tests/runner-e2e/testops-chat-reporter.mjs b/tests/runner-e2e/testops-chat-reporter.mjs index 75bd096a6d8..a955e631ccc 100644 --- a/tests/runner-e2e/testops-chat-reporter.mjs +++ b/tests/runner-e2e/testops-chat-reporter.mjs @@ -1,8 +1,10 @@ +import { execFile } from 'node:child_process'; +import { promisify } from 'node:util'; import { createHash } from 'node:crypto'; import { appendFile, lstat, readFile, realpath, writeFile } from 'node:fs/promises'; import path from 'node:path'; import { pathToFileURL } from 'node:url'; -import { publicContext, aliases, executionId, scenario } from './chat-feature-testops-api.mjs'; +import { publicContext, aliases, executionId, scenario, endpoint, testopsClient, bootstrapJobMetadata, failureProjection } from './chat-feature-testops-api.mjs'; import { run } from './chat-feature-testops-cli.mjs'; const hash = bytes => createHash('sha256').update(bytes).digest('hex'); const fail = () => { throw new Error('chat_testops_selection_rejected'); }; @@ -34,8 +36,67 @@ export async function verifySelectionFile(file, env) { return (await readVerifiedSelection(file, env)).value; } -export async function reporter(command, argument, env = process.env, fetchImpl = fetch) { +const officialExec = promisify(execFile); +class BootstrapContextFailure extends Error {} +const rejectBootstrapContext = () => { throw new BootstrapContextFailure('chat_testops_bootstrap_context_invalid'); }; + +export function parseOfficialBootstrapContext(stdout) { + if (typeof stdout !== 'string' || Buffer.byteLength(stdout) > 16_384) rejectBootstrapContext(); + const values = new Map(); + for (const line of stdout.split(/\r?\n/)) { + for (const name of ['ALLURE_JOB_RUN_ID', 'ALLURE_LAUNCH_ID']) { + if (!line.startsWith(`${name}=`)) continue; + const value = line.slice(name.length + 1); + if (values.has(name) || !/^[1-9][0-9]{0,15}$/.test(value) || !Number.isSafeInteger(Number(value))) rejectBootstrapContext(); + values.set(name, value); + } + } + if (values.size !== 2) rejectBootstrapContext(); + return { jobRunId: values.get('ALLURE_JOB_RUN_ID'), launchId: Number(values.get('ALLURE_LAUNCH_ID')) }; +} + +async function captureBootstrapContext(env, fetchImpl, execImpl) { + let stateRoot; + try { + if (env.ALLURE_ENDPOINT !== endpoint || env.ALLURE_PROJECT_ID !== '1' || !env.RUNNER_TEMP + || !env.GITHUB_WORKSPACE || !path.isAbsolute(env.GITHUB_WORKSPACE) || !env.GITHUB_EVENT_PATH) rejectBootstrapContext(); + const eventStat = await lstat(env.GITHUB_EVENT_PATH); + if (!eventStat.isFile() || eventStat.isSymbolicLink() || eventStat.nlink !== 1 || eventStat.size > 65_536) rejectBootstrapContext(); + const event = JSON.parse(await readFile(env.GITHUB_EVENT_PATH, 'utf8')); + if (!event.inputs || Object.keys(event.inputs).some(key => !['operation', 'ALLURE_JOB_RUN_ID', 'ALLURE_USERNAME'].includes(key)) + || event.inputs.operation !== 'bootstrap' || (event.inputs.ALLURE_JOB_RUN_ID ?? '') !== '') rejectBootstrapContext(); + const requested = path.join(await realpath(env.RUNNER_TEMP), 'chat-testops-bootstrap-state'); + const stateStat = await lstat(requested); + if (!stateStat.isDirectory() || stateStat.isSymbolicLink() || await realpath(requested) !== requested + || process.getuid && stateStat.uid !== process.getuid() || stateStat.mode & 0o077) rejectBootstrapContext(); + stateRoot = requested; + for (const name of ['registration.json', 'registration-failure.json']) { + try { await lstat(path.join(stateRoot, name)); rejectBootstrapContext(); } + catch (error) { if (error.code !== 'ENOENT') throw error; } + } + let output; + try { + output = await execImpl('allurectl', ['job-run', 'env'], { cwd: await realpath(env.GITHUB_WORKSPACE), env, + shell: false, encoding: 'utf8', timeout: 30_000, maxBuffer: 16_384, windowsHide: true }); + } catch { throw new BootstrapContextFailure('chat_testops_bootstrap_command_failed'); } + const captured = parseOfficialBootstrapContext(output.stdout); + const api = await testopsClient(env.ALLURE_TOKEN, fetchImpl); + const registration = await bootstrapJobMetadata(api, { ...env, ALLURE_JOB_RUN_ID: captured.jobRunId }, captured.launchId); + await writeFile(path.join(stateRoot, 'registration.json'), JSON.stringify(registration, null, 2), { flag: 'wx', mode: 0o600 }); + return 0; + } catch (error) { + const safe = error instanceof BootstrapContextFailure + ? { stage: 'bootstrap_context', code: error.message, httpStatus: null } + : failureProjection(error, 'bootstrap_context'); + if (stateRoot) await writeFile(path.join(stateRoot, 'registration-failure.json'), JSON.stringify(safe, null, 2), { flag: 'wx', mode: 0o600 }).catch(() => {}); + process.stderr.write(`Chat TestOps bootstrap context stopped: ${JSON.stringify(safe)}\n`); + return 1; + } +} + +export async function reporter(command, argument, env = process.env, fetchImpl = fetch, execImpl = officialExec) { publicContext(env); + if (command === 'bootstrap-context') return captureBootstrapContext(env, fetchImpl, execImpl); if (command === 'verify-selection') { if (Object.hasOwn(env, 'ALLURE_TOKEN')) fail(); await verifySelectionFile(argument, env); diff --git a/tests/runner-e2e/testops-chat-reporter.test.mjs b/tests/runner-e2e/testops-chat-reporter.test.mjs index 3c99400867e..9e982d66f62 100644 --- a/tests/runner-e2e/testops-chat-reporter.test.mjs +++ b/tests/runner-e2e/testops-chat-reporter.test.mjs @@ -1,10 +1,10 @@ import test from 'node:test'; import assert from 'node:assert/strict'; import { createHash } from 'node:crypto'; -import { mkdtemp, mkdir, readFile, realpath, rm, writeFile } from 'node:fs/promises'; +import { mkdtemp, mkdir, readFile, realpath, rm, writeFile, chmod } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import path from 'node:path'; -import { verifySelectionFile, reporter } from './testops-chat-reporter.mjs'; +import { verifySelectionFile, reporter, parseOfficialBootstrapContext } from './testops-chat-reporter.mjs'; import { executionId, scenario, fullName } from './chat-feature-testops-api.mjs'; const context = { GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/heads/master', GITHUB_EVENT_NAME: 'workflow_dispatch', GITHUB_RUN_ATTEMPT: '1', GITHUB_SHA: 'a'.repeat(40), GITHUB_RUN_ID: '123', ALLURE_JOB_RUN_ID: '9', @@ -136,3 +136,141 @@ test('actual launch caller rejects five own-run contradictions before TestOps au } } finally { process.stderr.write = original; } }); + +async function bootstrapContextFixture(t, inputs = { operation: 'bootstrap', ALLURE_JOB_RUN_ID: '' }) { + const f = await fixture(t), workspace = f.env.GITHUB_WORKSPACE; + const temporary = path.join(workspace, 'temp'); await mkdir(temporary); + const state = path.join(temporary, 'chat-testops-bootstrap-state'); await mkdir(state, { mode: 0o700 }); + const originalJob = Buffer.from('{"status":"official_job_run_id_missing","actualResult":null}'); + await writeFile(path.join(state, 'job.json'), originalJob, { mode: 0o600 }); + const event = path.join(workspace, 'event.json'); await writeFile(event, JSON.stringify({ inputs })); + return { ...f, state, originalJob, env: { ...f.env, RUNNER_TEMP: temporary, GITHUB_EVENT_PATH: event, ALLURE_JOB_RUN_ID: '', + ALLURE_ENDPOINT: 'https://paperclip.testops.cloud', ALLURE_PROJECT_ID: '1', ALLURE_TOKEN: 'SYNTHETIC_REPORTER_TOKEN', CHAT_TESTOPS_GITHUB_WORKFLOW_ID: '99' } }; +} +function bootstrapContextFetch(change = {}) { + const calls = []; + const fetchImpl = async (url, options) => { + const route = new URL(url).pathname; calls.push({ route, method: options.method }); + let value; + if (route === '/api/uaa/oauth/token') { assert.equal(options.method, 'POST'); value = { access_token: 'SYNTHETIC_ACCESS' }; } + else { + assert.equal(options.method, 'GET'); + if (route === '/api/jobrun/9') value = { id: 9, job: { id: 7, type: 'github' }, launchId: 10, + url: 'https://github.com/paperclipai/paperclip/actions/runs/123', ...change }; + else if (route === '/api/launch/10') value = { id: 10, projectId: 1 }; + else if (route === '/api/job/7') value = { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/99', + parameters: [{ id: 17, value: 'bootstrap' }] }; + else if (route === '/api/environment/17') value = { id: 17, name: 'operation' }; + else assert.fail(route); + } + return { ok: true, status: 200, json: async () => value, text: async () => JSON.stringify(value) }; + }; + return { fetchImpl, calls }; +} +async function captureSafeErrors(callback) { + const original = process.stderr.write; + let text = ''; + process.stderr.write = chunk => { text += String(chunk); return true; }; + try { return { value: await callback(), text }; } finally { process.stderr.write = original; } +} + +test('official context parser admits only unique positive numeric job-run and launch assignments', () => { + const text = 'ALLURE_TOKEN=SYNTHETIC_SECRET\nALLURE_JOB_RUN_URL=untrusted\nALLURE_JOB_RUN_ID=9\r\nALLURE_LAUNCH_ID=10\n'; + assert.deepEqual(parseOfficialBootstrapContext(text), { jobRunId: '9', launchId: 10 }); + for (const text of ['', 'ALLURE_JOB_RUN_ID=9\n', 'ALLURE_JOB_RUN_ID=0\nALLURE_LAUNCH_ID=10', + 'ALLURE_JOB_RUN_ID=9\nALLURE_LAUNCH_ID=10\nALLURE_JOB_RUN_ID=9', 'ALLURE_JOB_RUN_ID=9\nALLURE_LAUNCH_ID=$(secret)', + 'ALLURE_JOB_RUN_ID=9007199254740992\nALLURE_LAUNCH_ID=10', 'ALLURE_JOB_RUN_ID=9; secret\nALLURE_LAUNCH_ID=10', 'x'.repeat(16_385)]) { + assert.throws(() => parseOfficialBootstrapContext(text), /chat_testops_bootstrap_context_invalid/); + } +}); + +test('post-watch facade runs exact official command with bounded pipes and preserves original missing-child artifact', async t => { + const f = await bootstrapContextFixture(t), server = bootstrapContextFetch(); let executions = 0; + const execImpl = async (command, args, options) => { + executions++; assert.equal(command, 'allurectl'); assert.deepEqual(args, ['job-run', 'env']); + assert.equal(options.shell, false); assert.equal(options.timeout, 30_000); assert.equal(options.maxBuffer, 16_384); + assert.equal(options.encoding, 'utf8'); assert.equal(options.cwd, f.env.GITHUB_WORKSPACE); assert.equal(options.env, f.env); + return { stdout: 'ALLURE_TOKEN=SYNTHETIC_SECRET\nALLURE_JOB_RUN_ID=9\nALLURE_LAUNCH_ID=10\n', stderr: 'SYNTHETIC_SECRET_STDERR' }; + }; + assert.equal(await reporter('bootstrap-context', undefined, f.env, server.fetchImpl, execImpl), 0); + const bytes = await readFile(path.join(f.state, 'registration.json')); + const result = JSON.parse(bytes); + assert.equal(result.schema, 'paperclip.chat-feature-testops-bootstrap-registration/v1'); + assert.equal(result.jobId, 7); assert.equal(result.allureJobRunId, '9'); assert.equal(result.launchId, 10); + assert.equal(result.runId, '123'); assert.equal(result.workflowSha, context.GITHUB_SHA); assert.equal(result.targetSha, context.GITHUB_SHA); + assert.equal(result.currentRunUrlMatches, true); assert.equal(result.publicWorkflowMatches, true); + assert.equal(result.capturedAfterWatch, true); assert.equal(result.runnableBindingQualified, false); assert.equal(result.actualResult, null); + assert.deepEqual(result.parameters, [{ id: 17, name: 'operation', value: 'bootstrap' }]); + assert.ok(!bytes.toString().includes('SYNTHETIC')); assert.equal(f.env.ALLURE_JOB_RUN_ID, ''); + assert.deepEqual(await readFile(path.join(f.state, 'job.json')), f.originalJob); + const second = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, () => assert.fail('no second auth'), () => assert.fail('no second CLI'))); + assert.equal(second.value, 1); assert.equal(executions, 1); + assert.deepEqual(await readFile(path.join(f.state, 'registration.json')), bytes); + assert.equal(server.calls.filter(row => row.method === 'POST').length, 1); + assert.equal(server.calls[0].route, '/api/uaa/oauth/token'); +}); + +test('post-watch missing official IDs fails closed without API auth or invented registration', async t => { + const f = await bootstrapContextFixture(t); + const result = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, () => assert.fail('no API'), + async () => ({ stdout: 'ALLURE_TOKEN=SYNTHETIC_SECRET\n', stderr: 'PRIVATE_STDERR' }))); + assert.equal(result.value, 1); assert.ok(!result.text.includes('SYNTHETIC')); assert.ok(!result.text.includes('PRIVATE')); + assert.equal(JSON.parse(await readFile(path.join(f.state, 'registration-failure.json'))).code, 'chat_testops_bootstrap_context_invalid'); + await assert.rejects(readFile(path.join(f.state, 'registration.json')), { code: 'ENOENT' }); +}); + +test('post-watch command errors discard stdout stderr and raw exception without retry or API', async t => { + const f = await bootstrapContextFixture(t); let calls = 0; + const result = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, () => assert.fail('no API'), async () => { + calls++; throw Object.assign(new Error('SYNTHETIC_PRIVATE_EXCEPTION'), { stdout: 'SYNTHETIC_PRIVATE_STDOUT', stderr: 'SYNTHETIC_PRIVATE_STDERR' }); + })); + assert.equal(result.value, 1); assert.equal(calls, 1); assert.ok(!result.text.includes('SYNTHETIC')); + assert.equal(JSON.parse(await readFile(path.join(f.state, 'registration-failure.json'))).code, 'chat_testops_bootstrap_command_failed'); +}); + +test('post-watch wrong original operation or extra source input stops before official command and auth', async t => { + for (const inputs of [{ operation: 'evaluate' }, { operation: 'launch-plan' }, { operation: 'bootstrap', ALLURE_JOB_RUN_ID: '9' }, { operation: 'bootstrap', model: 'foreign/model' }]) { + const f = await bootstrapContextFixture(t, inputs); + const result = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, () => assert.fail('no API'), () => assert.fail('no CLI'))); + assert.equal(result.value, 1); + } +}); + +test('post-watch foreign backend binding returns safe nonzero and preserves original job evidence', async t => { + const f = await bootstrapContextFixture(t), server = bootstrapContextFetch({ url: 'https://github.com/paperclipai/paperclip/actions/runs/999' }); + const result = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, server.fetchImpl, + async () => ({ stdout: 'ALLURE_JOB_RUN_ID=9\nALLURE_LAUNCH_ID=10\n' }))); + assert.equal(result.value, 1); assert.equal(JSON.parse(await readFile(path.join(f.state, 'registration-failure.json'))).code, 'chat_testops_jobrun_mismatch'); + assert.deepEqual(await readFile(path.join(f.state, 'job.json')), f.originalJob); +}); + +test('bootstrap workflow captures official context only after watch and retains exactly named closed receipts', async () => { + const workflow = await readFile(new URL('../../.github/workflows/chat-feature-testops.yml', import.meta.url), 'utf8'); + const start = workflow.indexOf(' - name: Register the public job without an evaluation or a synthetic result'); + const end = workflow.indexOf(' - name: Validate the one real TestOps selection', start); + const step = workflow.slice(start, end); + assert.ok(step.includes("if: needs.authorize.outputs.operation == 'bootstrap'")); + assert.match(step, /allurectl watch -- node tests\/runner-e2e\/testops-chat-reporter\.mjs bootstrap\n\s+node tests\/runner-e2e\/testops-chat-reporter\.mjs bootstrap-context/); + assert.equal((step.match(/allurectl watch/g) ?? []).length, 1); assert.ok(!step.includes('continue-on-error')); + assert.ok(workflow.includes('${{ runner.temp }}/chat-testops-bootstrap-state/registration.json')); + assert.ok(workflow.includes('${{ runner.temp }}/chat-testops-bootstrap-state/registration-failure.json')); +}); + +test('post-watch refuses nonprivate state custody before any command auth or receipt write', async t => { + const f = await bootstrapContextFixture(t); await chmod(f.state, 0o755); + const result = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, () => assert.fail('no API'), () => assert.fail('no CLI'))); + assert.equal(result.value, 1); + await assert.rejects(readFile(path.join(f.state, 'registration.json')), { code: 'ENOENT' }); + await assert.rejects(readFile(path.join(f.state, 'registration-failure.json')), { code: 'ENOENT' }); + assert.deepEqual(await readFile(path.join(f.state, 'job.json')), f.originalJob); +}); + +test('post-watch authentication failure retains only fixed HTTP diagnostic and never reads body', async t => { + const f = await bootstrapContextFixture(t); let calls = 0; + const result = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, async () => { + calls++; return { ok: false, status: 403, json: () => assert.fail('no auth body read'), text: () => assert.fail('no auth body read') }; + }, async () => ({ stdout: 'ALLURE_JOB_RUN_ID=9\nALLURE_LAUNCH_ID=10\n' }))); + assert.equal(result.value, 1); assert.equal(calls, 1); + const failure = JSON.parse(await readFile(path.join(f.state, 'registration-failure.json'))); + assert.deepEqual(failure, { stage: 'bootstrap_context', code: 'testops_auth_http', httpStatus: 403 }); +});