fix(evals): capture TestOps registration after watch (#18375)

## Thinking Path

> - Paperclip manages AI agents and verifies their work through Product
E2E tests.
> - The fixed Chat feature workflow prepares a TestOps case and plan
before any model run.
> - Its first zero-result bootstrap completed, but the watched child did
not receive the generated job-run ID.
> - The official watch log contained a job-run ID, so the missing child
variable did not prove that registration was absent.
> - This pull request reads official context after watch and validates
the linked backend records.
> - The workflow retains closed registration metadata and stops on
unknown or foreign context.

## Linked Issues or Issue Description

Refs: #18322.

**What happened?**
The successful zero-result bootstrap retained
`official_job_run_id_missing` from its watched child. It could not prove
the backend job and workflow binding needed for the next setup step.

**Expected behavior**
After successful watch, read the generated context, validate its current
GitHub run and fixed workflow, and retain only closed IDs and parameter
metadata.

**Steps to reproduce**
Run the fixed workflow's `bootstrap` operation on master. Inspect its
setup artifact after watch completes. No evaluation or synthetic passing
result is needed.

**Paperclip version or commit**
The original bootstrap used merged #18322. This follow-up starts from
`76fde891ccc29283b3f5c8dcdce089c5b3d937d1`.

**Deployment mode**
GitHub Actions metadata-only setup in the protected reporting
environment.

## What Changed

- Run an internal context reader after successful bootstrap watch in the
same protected step.
- Parse only numeric official job-run and launch IDs in memory. Do not
log or save raw command output.
- Validate current-run, project, launch, job, workflow, and bounded
parameter metadata before writing an exclusive registration receipt.
- Retain the two known registration receipt paths with the existing
setup artifact.
- Preserve the original inside-watch receipt, case reconciliation,
provider isolation, and evaluation limits.

## Verification

- `node --test tests/runner-e2e/chat-feature-testops-api.test.mjs
tests/runner-e2e/testops-chat-reporter.test.mjs`: 48 pure/mock/FS
controls pass (36 prior and 12 new). The actual local Node version was
26.4.0. Normal Node 24 CI remains required.
- Four MJS syntax checks and `git diff --check` pass. No TypeScript
noEmit claim is made for this MJS-only helper change.
- Actionlint passes for the changed workflow.
- Normal [GitHub Actions
CI](https://github.com/paperclipai/paperclip/actions/runs/41123894542)
passes on exact head `58396de10f0c49294b0bbc0f4709c29990d57d53`: 48 jobs
succeeded and one skipped. Both required `ci / e2e` and `ci / verify`
pass.
- Greptile reviewed that exact head at 5/5 with no actionable findings
or unresolved review threads.
- No local CLI, provider, model, app, database, or build qualification
was run. No replacement bootstrap or paid evaluation was dispatched.

## Risks

The official context or backend DTO shape can differ from the supported
format. The reader then stops with a closed failure receipt. Pure tests
do not qualify the remote CLI or backend. The original zero-result
bootstrap and its missing child-context observation remain part of the
history. The case's actual outcome remains unmeasured.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, code editing, tool
use, and pure test review. The runtime does not expose the exact
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-30 16:07:04 -05:00
1 parent 6174b4d7c8
commit a1845b3b64
5 files changed
+270 -9

No files matched your search

@@ -89,6 +89,7 @@ jobs:
run: |
mkdir -m 700 "$ALLURE_RESULTS"
allurectl watch -- node tests/runner-e2e/testops-chat-reporter.mjs bootstrap
node tests/runner-e2e/testops-chat-reporter.mjs bootstrap-context
- name: Validate the one real TestOps selection before any build or provider access
id: selection
if: needs.authorize.outputs.operation == 'evaluate'
@@ -134,6 +135,8 @@ jobs:
${{ runner.temp }}/chat-testops-bootstrap-state/plan.json
${{ runner.temp }}/chat-testops-bootstrap-state/job.json
${{ runner.temp }}/chat-testops-bootstrap-state/failure.json
${{ runner.temp }}/chat-testops-bootstrap-state/registration.json
${{ runner.temp }}/chat-testops-bootstrap-state/registration-failure.json
retention-days: 30
if-no-files-found: error
- name: Retain the one plan-launch receipt and stop state
+22 -5
View File
@@ -19,7 +19,7 @@ const diagnosticCodes = new Set(['untrusted_chat_testops_context', 'testops_toke
'chat_testops_case_mismatch', 'chat_testops_plan_mismatch', 'chat_testops_plan_job_mismatch', 'chat_testops_job_mismatch',
'chat_testops_jobrun_mismatch', 'chat_testops_launch_mismatch', 'chat_testops_selection_mismatch', 'chat_testops_parameter_mismatch', 'chat_testops_nonce_invalid', 'chat_testops_nonce_conflict', 'chat_testops_catalog_incomplete', 'chat_testops_launch_admission', 'chat_testops_github_http', 'chat_testops_integration_mismatch', 'chat_testops_upload_mismatch', 'chat_testops_upload_unverified']);
export function failureProjection(error, stage) {
const stages = ['context', 'output_admission', 'authentication', 'metadata_inspection', 'plan_bootstrap', 'selection_validation', 'launch_admission', 'launch_preflight', 'plan_launch', 'bootstrap_job_metadata', 'result_export', 'upload_verification', 'receipt'];
const stages = ['context', 'output_admission', 'authentication', 'metadata_inspection', 'plan_bootstrap', 'selection_validation', 'launch_admission', 'launch_preflight', 'plan_launch', 'bootstrap_job_metadata', 'result_export', 'upload_verification', 'bootstrap_context', 'receipt'];
return { stage: stages.includes(stage) ? stage : 'unknown',
code: error instanceof SetupFailure && diagnosticCodes.has(error.code) ? error.code : 'unknown_failure',
httpStatus: error instanceof SetupFailure && Number.isInteger(error.status) && error.status >= 100 && error.status < 600 ? error.status : null };
@@ -188,21 +188,38 @@ async function registeredPlan(api, env) {
return { ...ids, selected, plan, job };
}
export async function bootstrapJobMetadata(api, env) {
if (!positive(env.ALLURE_JOB_RUN_ID)) return { status: 'official_job_run_id_missing', actualResult: null };
export async function bootstrapJobMetadata(api, env, expectedLaunchId) {
const strict = expectedLaunchId !== undefined;
const context = strict ? publicContext(env) : undefined;
if (strict && !id(expectedLaunchId)) fail('chat_testops_launch_mismatch');
if (!positive(env.ALLURE_JOB_RUN_ID)) {
if (strict) fail('chat_testops_jobrun_mismatch');
return { status: 'official_job_run_id_missing', actualResult: null };
}
const run = await api(`/api/jobrun/${env.ALLURE_JOB_RUN_ID}`);
if (!id(run?.job?.id)) fail('chat_testops_jobrun_mismatch');
if (strict) {
if (run.id !== Number(env.ALLURE_JOB_RUN_ID) || run.job.type !== 'github' || run.launchId !== expectedLaunchId
|| run.url !== `https://github.com/paperclipai/paperclip/actions/runs/${context.runId}`) fail('chat_testops_jobrun_mismatch');
const launch = await api(`/api/launch/${expectedLaunchId}`);
if (launch.id !== expectedLaunchId || launch.projectId !== 1) fail('chat_testops_launch_mismatch');
}
const job = await api(`/api/job/${run.job.id}`);
if (job.id !== run.job.id || job.type !== 'github' || !Array.isArray(job.parameters) || job.parameters.length > 4) fail('chat_testops_job_mismatch');
if (strict && !publicJobUrl(job.url, env)) fail('chat_testops_job_mismatch');
const parameters = [];
const seen = new Set();
for (const parameter of job.parameters) {
if (!id(parameter.id)) fail('chat_testops_parameter_mismatch');
if (!id(parameter.id) || strict && seen.has(parameter.id)) fail('chat_testops_parameter_mismatch');
seen.add(parameter.id);
const definition = await api(`/api/environment/${parameter.id}`);
if (strict && definition.id !== parameter.id) fail('chat_testops_parameter_mismatch');
const known = ['operation', 'ALLURE_JOB_RUN_ID', 'ALLURE_USERNAME'].includes(definition.name);
parameters.push({ id: parameter.id, name: known ? definition.name : 'UNKNOWN',
value: definition.name === 'operation' && ['bootstrap', 'evaluate', 'launch-plan'].includes(parameter.value) ? parameter.value : 'NOT_PROJECTED' });
}
return { status: 'observed', projectId: 1, jobId: job.id, allureJobRunId: env.ALLURE_JOB_RUN_ID,
return { ...(strict ? { schema: 'paperclip.chat-feature-testops-bootstrap-registration/v1', ...context, capturedAfterWatch: true, runnableBindingQualified: false } : {}),
status: 'observed', projectId: 1, jobId: job.id, allureJobRunId: env.ALLURE_JOB_RUN_ID,
publicWorkflowMatches: publicJobUrl(job.url, env), currentRunUrlMatches: run.url === `https://github.com/paperclipai/paperclip/actions/runs/${env.GITHUB_RUN_ID}`,
launchId: id(run.launchId) ? run.launchId : null, parameters, actualResult: null };
}
@@ -390,3 +390,45 @@ test('readback stops immediately after a late GET and gives each client only the
}, uploadContext, uploadSelection, uploadReceipt, { now: () => time }), /upload_unverified/);
assert.equal(calls, 2);
});
function strictBootstrapApi(patch = {}) {
const run = { id: 9, job: { id: 7, type: 'github' }, launchId: 10, url: 'https://github.com/paperclipai/paperclip/actions/runs/123', ...patch.run };
const launch = { id: 10, projectId: 1, ...patch.launch };
const job = { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml',
parameters: [{ id: 17, value: 'bootstrap' }, { id: 18, get value() { assert.fail('unrelated parameter value must stay unopened'); } }], ...patch.job };
return async (route, method) => {
assert.equal(method, undefined);
if (route === '/api/jobrun/9') return run;
if (route === '/api/launch/10') return launch;
if (route === '/api/job/7') return job;
if (route === '/api/environment/17') return { id: 17, name: 'operation', ...patch.definition };
if (route === '/api/environment/18') return { id: 18, name: 'SYNTHETIC_PRIVATE_NAME' };
assert.fail(route);
};
}
test('post-watch reader binds actual current run source project launch and public workflow without claiming runnable', async () => {
const result = await bootstrapJobMetadata(strictBootstrapApi(), publicEnv, 10);
assert.equal(result.jobId, 7); assert.equal(result.runId, publicEnv.GITHUB_RUN_ID);
assert.equal(result.workflowSha, publicEnv.GITHUB_SHA); assert.equal(result.capturedAfterWatch, true);
assert.equal(result.runnableBindingQualified, false); assert.equal(result.actualResult, null);
assert.deepEqual(result.parameters[1], { id: 18, name: 'UNKNOWN', value: 'NOT_PROJECTED' });
assert.ok(!JSON.stringify(result).includes('SYNTHETIC_PRIVATE'));
});
test('post-watch reader rejects stale foreign project launch source and job identity', async () => {
for (const patch of [{ run: { id: 99 } }, { run: { url: 'https://github.com/paperclipai/paperclip/actions/runs/999' } },
{ run: { launchId: 20 } }, { run: { job: { id: 7, type: 'foreign' } } }, { launch: { projectId: 2 } },
{ launch: { id: 20 } }, { job: { url: 'https://github.com/other/repo/actions/workflows/chat-feature-testops.yml' } },
{ job: { type: 'foreign' } }, { definition: { id: 99 } }]) {
await assert.rejects(bootstrapJobMetadata(strictBootstrapApi(patch), publicEnv, 10));
}
await assert.rejects(bootstrapJobMetadata(() => assert.fail('no API'), { ...publicEnv, GITHUB_SHA: 'untrusted' }, 10));
await assert.rejects(bootstrapJobMetadata(() => assert.fail('no API'), { ...publicEnv, ALLURE_JOB_RUN_ID: '' }, 10));
});
test('post-watch reader caps parameter definitions and refuses duplicate IDs', async () => {
for (const parameters of [Array.from({ length: 5 }, (_, i) => ({ id: i + 1 })), [{ id: 17 }, { id: 17 }]]) {
await assert.rejects(bootstrapJobMetadata(strictBootstrapApi({ job: { parameters } }), publicEnv, 10));
}
});
+63 -2
View File
@@ -1,8 +1,10 @@
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
import { createHash } from 'node:crypto';
import { appendFile, lstat, readFile, realpath, writeFile } from 'node:fs/promises';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
import { publicContext, aliases, executionId, scenario } from './chat-feature-testops-api.mjs';
import { publicContext, aliases, executionId, scenario, endpoint, testopsClient, bootstrapJobMetadata, failureProjection } from './chat-feature-testops-api.mjs';
import { run } from './chat-feature-testops-cli.mjs';
const hash = bytes => createHash('sha256').update(bytes).digest('hex');
const fail = () => { throw new Error('chat_testops_selection_rejected'); };
@@ -34,8 +36,67 @@ export async function verifySelectionFile(file, env) {
return (await readVerifiedSelection(file, env)).value;
}
export async function reporter(command, argument, env = process.env, fetchImpl = fetch) {
const officialExec = promisify(execFile);
class BootstrapContextFailure extends Error {}
const rejectBootstrapContext = () => { throw new BootstrapContextFailure('chat_testops_bootstrap_context_invalid'); };
export function parseOfficialBootstrapContext(stdout) {
if (typeof stdout !== 'string' || Buffer.byteLength(stdout) > 16_384) rejectBootstrapContext();
const values = new Map();
for (const line of stdout.split(/\r?\n/)) {
for (const name of ['ALLURE_JOB_RUN_ID', 'ALLURE_LAUNCH_ID']) {
if (!line.startsWith(`${name}=`)) continue;
const value = line.slice(name.length + 1);
if (values.has(name) || !/^[1-9][0-9]{0,15}$/.test(value) || !Number.isSafeInteger(Number(value))) rejectBootstrapContext();
values.set(name, value);
}
}
if (values.size !== 2) rejectBootstrapContext();
return { jobRunId: values.get('ALLURE_JOB_RUN_ID'), launchId: Number(values.get('ALLURE_LAUNCH_ID')) };
}
async function captureBootstrapContext(env, fetchImpl, execImpl) {
let stateRoot;
try {
if (env.ALLURE_ENDPOINT !== endpoint || env.ALLURE_PROJECT_ID !== '1' || !env.RUNNER_TEMP
|| !env.GITHUB_WORKSPACE || !path.isAbsolute(env.GITHUB_WORKSPACE) || !env.GITHUB_EVENT_PATH) rejectBootstrapContext();
const eventStat = await lstat(env.GITHUB_EVENT_PATH);
if (!eventStat.isFile() || eventStat.isSymbolicLink() || eventStat.nlink !== 1 || eventStat.size > 65_536) rejectBootstrapContext();
const event = JSON.parse(await readFile(env.GITHUB_EVENT_PATH, 'utf8'));
if (!event.inputs || Object.keys(event.inputs).some(key => !['operation', 'ALLURE_JOB_RUN_ID', 'ALLURE_USERNAME'].includes(key))
|| event.inputs.operation !== 'bootstrap' || (event.inputs.ALLURE_JOB_RUN_ID ?? '') !== '') rejectBootstrapContext();
const requested = path.join(await realpath(env.RUNNER_TEMP), 'chat-testops-bootstrap-state');
const stateStat = await lstat(requested);
if (!stateStat.isDirectory() || stateStat.isSymbolicLink() || await realpath(requested) !== requested
|| process.getuid && stateStat.uid !== process.getuid() || stateStat.mode & 0o077) rejectBootstrapContext();
stateRoot = requested;
for (const name of ['registration.json', 'registration-failure.json']) {
try { await lstat(path.join(stateRoot, name)); rejectBootstrapContext(); }
catch (error) { if (error.code !== 'ENOENT') throw error; }
}
let output;
try {
output = await execImpl('allurectl', ['job-run', 'env'], { cwd: await realpath(env.GITHUB_WORKSPACE), env,
shell: false, encoding: 'utf8', timeout: 30_000, maxBuffer: 16_384, windowsHide: true });
} catch { throw new BootstrapContextFailure('chat_testops_bootstrap_command_failed'); }
const captured = parseOfficialBootstrapContext(output.stdout);
const api = await testopsClient(env.ALLURE_TOKEN, fetchImpl);
const registration = await bootstrapJobMetadata(api, { ...env, ALLURE_JOB_RUN_ID: captured.jobRunId }, captured.launchId);
await writeFile(path.join(stateRoot, 'registration.json'), JSON.stringify(registration, null, 2), { flag: 'wx', mode: 0o600 });
return 0;
} catch (error) {
const safe = error instanceof BootstrapContextFailure
? { stage: 'bootstrap_context', code: error.message, httpStatus: null }
: failureProjection(error, 'bootstrap_context');
if (stateRoot) await writeFile(path.join(stateRoot, 'registration-failure.json'), JSON.stringify(safe, null, 2), { flag: 'wx', mode: 0o600 }).catch(() => {});
process.stderr.write(`Chat TestOps bootstrap context stopped: ${JSON.stringify(safe)}\n`);
return 1;
}
}
export async function reporter(command, argument, env = process.env, fetchImpl = fetch, execImpl = officialExec) {
publicContext(env);
if (command === 'bootstrap-context') return captureBootstrapContext(env, fetchImpl, execImpl);
if (command === 'verify-selection') {
if (Object.hasOwn(env, 'ALLURE_TOKEN')) fail();
await verifySelectionFile(argument, env);
+140 -2
View File
@@ -1,10 +1,10 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
import { mkdtemp, mkdir, readFile, realpath, rm, writeFile } from 'node:fs/promises';
import { mkdtemp, mkdir, readFile, realpath, rm, writeFile, chmod } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { verifySelectionFile, reporter } from './testops-chat-reporter.mjs';
import { verifySelectionFile, reporter, parseOfficialBootstrapContext } from './testops-chat-reporter.mjs';
import { executionId, scenario, fullName } from './chat-feature-testops-api.mjs';
const context = { GITHUB_REPOSITORY: 'paperclipai/paperclip', GITHUB_REF: 'refs/heads/master', GITHUB_EVENT_NAME: 'workflow_dispatch',
GITHUB_RUN_ATTEMPT: '1', GITHUB_SHA: 'a'.repeat(40), GITHUB_RUN_ID: '123', ALLURE_JOB_RUN_ID: '9',
@@ -136,3 +136,141 @@ test('actual launch caller rejects five own-run contradictions before TestOps au
}
} finally { process.stderr.write = original; }
});
async function bootstrapContextFixture(t, inputs = { operation: 'bootstrap', ALLURE_JOB_RUN_ID: '' }) {
const f = await fixture(t), workspace = f.env.GITHUB_WORKSPACE;
const temporary = path.join(workspace, 'temp'); await mkdir(temporary);
const state = path.join(temporary, 'chat-testops-bootstrap-state'); await mkdir(state, { mode: 0o700 });
const originalJob = Buffer.from('{"status":"official_job_run_id_missing","actualResult":null}');
await writeFile(path.join(state, 'job.json'), originalJob, { mode: 0o600 });
const event = path.join(workspace, 'event.json'); await writeFile(event, JSON.stringify({ inputs }));
return { ...f, state, originalJob, env: { ...f.env, RUNNER_TEMP: temporary, GITHUB_EVENT_PATH: event, ALLURE_JOB_RUN_ID: '',
ALLURE_ENDPOINT: 'https://paperclip.testops.cloud', ALLURE_PROJECT_ID: '1', ALLURE_TOKEN: 'SYNTHETIC_REPORTER_TOKEN', CHAT_TESTOPS_GITHUB_WORKFLOW_ID: '99' } };
}
function bootstrapContextFetch(change = {}) {
const calls = [];
const fetchImpl = async (url, options) => {
const route = new URL(url).pathname; calls.push({ route, method: options.method });
let value;
if (route === '/api/uaa/oauth/token') { assert.equal(options.method, 'POST'); value = { access_token: 'SYNTHETIC_ACCESS' }; }
else {
assert.equal(options.method, 'GET');
if (route === '/api/jobrun/9') value = { id: 9, job: { id: 7, type: 'github' }, launchId: 10,
url: 'https://github.com/paperclipai/paperclip/actions/runs/123', ...change };
else if (route === '/api/launch/10') value = { id: 10, projectId: 1 };
else if (route === '/api/job/7') value = { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/99',
parameters: [{ id: 17, value: 'bootstrap' }] };
else if (route === '/api/environment/17') value = { id: 17, name: 'operation' };
else assert.fail(route);
}
return { ok: true, status: 200, json: async () => value, text: async () => JSON.stringify(value) };
};
return { fetchImpl, calls };
}
async function captureSafeErrors(callback) {
const original = process.stderr.write;
let text = '';
process.stderr.write = chunk => { text += String(chunk); return true; };
try { return { value: await callback(), text }; } finally { process.stderr.write = original; }
}
test('official context parser admits only unique positive numeric job-run and launch assignments', () => {
const text = 'ALLURE_TOKEN=SYNTHETIC_SECRET\nALLURE_JOB_RUN_URL=untrusted\nALLURE_JOB_RUN_ID=9\r\nALLURE_LAUNCH_ID=10\n';
assert.deepEqual(parseOfficialBootstrapContext(text), { jobRunId: '9', launchId: 10 });
for (const text of ['', 'ALLURE_JOB_RUN_ID=9\n', 'ALLURE_JOB_RUN_ID=0\nALLURE_LAUNCH_ID=10',
'ALLURE_JOB_RUN_ID=9\nALLURE_LAUNCH_ID=10\nALLURE_JOB_RUN_ID=9', 'ALLURE_JOB_RUN_ID=9\nALLURE_LAUNCH_ID=$(secret)',
'ALLURE_JOB_RUN_ID=9007199254740992\nALLURE_LAUNCH_ID=10', 'ALLURE_JOB_RUN_ID=9; secret\nALLURE_LAUNCH_ID=10', 'x'.repeat(16_385)]) {
assert.throws(() => parseOfficialBootstrapContext(text), /chat_testops_bootstrap_context_invalid/);
}
});
test('post-watch facade runs exact official command with bounded pipes and preserves original missing-child artifact', async t => {
const f = await bootstrapContextFixture(t), server = bootstrapContextFetch(); let executions = 0;
const execImpl = async (command, args, options) => {
executions++; assert.equal(command, 'allurectl'); assert.deepEqual(args, ['job-run', 'env']);
assert.equal(options.shell, false); assert.equal(options.timeout, 30_000); assert.equal(options.maxBuffer, 16_384);
assert.equal(options.encoding, 'utf8'); assert.equal(options.cwd, f.env.GITHUB_WORKSPACE); assert.equal(options.env, f.env);
return { stdout: 'ALLURE_TOKEN=SYNTHETIC_SECRET\nALLURE_JOB_RUN_ID=9\nALLURE_LAUNCH_ID=10\n', stderr: 'SYNTHETIC_SECRET_STDERR' };
};
assert.equal(await reporter('bootstrap-context', undefined, f.env, server.fetchImpl, execImpl), 0);
const bytes = await readFile(path.join(f.state, 'registration.json'));
const result = JSON.parse(bytes);
assert.equal(result.schema, 'paperclip.chat-feature-testops-bootstrap-registration/v1');
assert.equal(result.jobId, 7); assert.equal(result.allureJobRunId, '9'); assert.equal(result.launchId, 10);
assert.equal(result.runId, '123'); assert.equal(result.workflowSha, context.GITHUB_SHA); assert.equal(result.targetSha, context.GITHUB_SHA);
assert.equal(result.currentRunUrlMatches, true); assert.equal(result.publicWorkflowMatches, true);
assert.equal(result.capturedAfterWatch, true); assert.equal(result.runnableBindingQualified, false); assert.equal(result.actualResult, null);
assert.deepEqual(result.parameters, [{ id: 17, name: 'operation', value: 'bootstrap' }]);
assert.ok(!bytes.toString().includes('SYNTHETIC')); assert.equal(f.env.ALLURE_JOB_RUN_ID, '');
assert.deepEqual(await readFile(path.join(f.state, 'job.json')), f.originalJob);
const second = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, () => assert.fail('no second auth'), () => assert.fail('no second CLI')));
assert.equal(second.value, 1); assert.equal(executions, 1);
assert.deepEqual(await readFile(path.join(f.state, 'registration.json')), bytes);
assert.equal(server.calls.filter(row => row.method === 'POST').length, 1);
assert.equal(server.calls[0].route, '/api/uaa/oauth/token');
});
test('post-watch missing official IDs fails closed without API auth or invented registration', async t => {
const f = await bootstrapContextFixture(t);
const result = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, () => assert.fail('no API'),
async () => ({ stdout: 'ALLURE_TOKEN=SYNTHETIC_SECRET\n', stderr: 'PRIVATE_STDERR' })));
assert.equal(result.value, 1); assert.ok(!result.text.includes('SYNTHETIC')); assert.ok(!result.text.includes('PRIVATE'));
assert.equal(JSON.parse(await readFile(path.join(f.state, 'registration-failure.json'))).code, 'chat_testops_bootstrap_context_invalid');
await assert.rejects(readFile(path.join(f.state, 'registration.json')), { code: 'ENOENT' });
});
test('post-watch command errors discard stdout stderr and raw exception without retry or API', async t => {
const f = await bootstrapContextFixture(t); let calls = 0;
const result = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, () => assert.fail('no API'), async () => {
calls++; throw Object.assign(new Error('SYNTHETIC_PRIVATE_EXCEPTION'), { stdout: 'SYNTHETIC_PRIVATE_STDOUT', stderr: 'SYNTHETIC_PRIVATE_STDERR' });
}));
assert.equal(result.value, 1); assert.equal(calls, 1); assert.ok(!result.text.includes('SYNTHETIC'));
assert.equal(JSON.parse(await readFile(path.join(f.state, 'registration-failure.json'))).code, 'chat_testops_bootstrap_command_failed');
});
test('post-watch wrong original operation or extra source input stops before official command and auth', async t => {
for (const inputs of [{ operation: 'evaluate' }, { operation: 'launch-plan' }, { operation: 'bootstrap', ALLURE_JOB_RUN_ID: '9' }, { operation: 'bootstrap', model: 'foreign/model' }]) {
const f = await bootstrapContextFixture(t, inputs);
const result = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, () => assert.fail('no API'), () => assert.fail('no CLI')));
assert.equal(result.value, 1);
}
});
test('post-watch foreign backend binding returns safe nonzero and preserves original job evidence', async t => {
const f = await bootstrapContextFixture(t), server = bootstrapContextFetch({ url: 'https://github.com/paperclipai/paperclip/actions/runs/999' });
const result = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, server.fetchImpl,
async () => ({ stdout: 'ALLURE_JOB_RUN_ID=9\nALLURE_LAUNCH_ID=10\n' })));
assert.equal(result.value, 1); assert.equal(JSON.parse(await readFile(path.join(f.state, 'registration-failure.json'))).code, 'chat_testops_jobrun_mismatch');
assert.deepEqual(await readFile(path.join(f.state, 'job.json')), f.originalJob);
});
test('bootstrap workflow captures official context only after watch and retains exactly named closed receipts', async () => {
const workflow = await readFile(new URL('../../.github/workflows/chat-feature-testops.yml', import.meta.url), 'utf8');
const start = workflow.indexOf(' - name: Register the public job without an evaluation or a synthetic result');
const end = workflow.indexOf(' - name: Validate the one real TestOps selection', start);
const step = workflow.slice(start, end);
assert.ok(step.includes("if: needs.authorize.outputs.operation == 'bootstrap'"));
assert.match(step, /allurectl watch -- node tests\/runner-e2e\/testops-chat-reporter\.mjs bootstrap\n\s+node tests\/runner-e2e\/testops-chat-reporter\.mjs bootstrap-context/);
assert.equal((step.match(/allurectl watch/g) ?? []).length, 1); assert.ok(!step.includes('continue-on-error'));
assert.ok(workflow.includes('${{ runner.temp }}/chat-testops-bootstrap-state/registration.json'));
assert.ok(workflow.includes('${{ runner.temp }}/chat-testops-bootstrap-state/registration-failure.json'));
});
test('post-watch refuses nonprivate state custody before any command auth or receipt write', async t => {
const f = await bootstrapContextFixture(t); await chmod(f.state, 0o755);
const result = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, () => assert.fail('no API'), () => assert.fail('no CLI')));
assert.equal(result.value, 1);
await assert.rejects(readFile(path.join(f.state, 'registration.json')), { code: 'ENOENT' });
await assert.rejects(readFile(path.join(f.state, 'registration-failure.json')), { code: 'ENOENT' });
assert.deepEqual(await readFile(path.join(f.state, 'job.json')), f.originalJob);
});
test('post-watch authentication failure retains only fixed HTTP diagnostic and never reads body', async t => {
const f = await bootstrapContextFixture(t); let calls = 0;
const result = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, async () => {
calls++; return { ok: false, status: 403, json: () => assert.fail('no auth body read'), text: () => assert.fail('no auth body read') };
}, async () => ({ stdout: 'ALLURE_JOB_RUN_ID=9\nALLURE_LAUNCH_ID=10\n' })));
assert.equal(result.value, 1); assert.equal(calls, 1);
const failure = JSON.parse(await readFile(path.join(f.state, 'registration-failure.json')));
assert.deepEqual(failure, { stage: 'bootstrap_context', code: 'testops_auth_http', httpStatus: 403 });
});