fix(evals): preserve opaque TestOps parameters (#18394)

## Thinking Path

> - Paperclip verifies real agent work through Product E2E tests.
> - The fixed Chat feature workflow uses TestOps to select and report
one case.
> - TestOps exposes job parameters as ID/value pairs.
> - The schema does not establish that those IDs identify global
environment variables.
> - The reader used that unsupported relationship to infer parameter
names.
> - This pull request keeps IDs opaque, preserves the strict allowed
values, and records closed read diagnostics.

## Linked Issues or Issue Description

Refs: #18322, #18375, #18385.

**What happened?**
A zero-model bootstrap returned HTTP 404 during context readback. The
exact failed GET was not retained and remains unknown. Schema review
also found no established relationship between job-parameter IDs and
global environment-variable IDs, although the reader queried that
namespace to label parameters.

**Expected behavior**
Read only the existing job-run, launch, and job records. Preserve the
job's exact permitted ID/value vector without inventing parameter names.
Record which fixed GET class failed, its positive resource ID, state,
and available HTTP status without raw responses or URLs.

**Steps to reproduce**
Use the actual API helpers with a job-parameter ID that has no global
environment-variable definition. The new pure controls reject an
unexpected definition lookup and verify the opaque vector and
diagnostics without a backend or model call.

**Paperclip version or commit**
This change starts from `8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57` after
#18385.

**Deployment mode**
GitHub Actions metadata setup in the protected TestOps reporting
environment.

## What Changed

- Remove the unsupported global environment-variable lookup from
job-parameter handling.
- Preserve an exact positive ID and `refs/heads/master` value. Use the
fixed workflow's existing `evaluate` default.
- Treat an operation parameter as such only when its ID matches the
protected configured ID and its value is allowed. Keep rejecting empty,
duplicate, conflicting, or unknown vectors before writes.
- Project only an opaque ID with a fixed-default-ref kind and value, or
an unresolved kind with the value withheld.
- Add closed diagnostics for the existing job-run, launch, and job GETs.
A successful call keeps HTTP status null because the API abstraction
does not expose the status.
- Sync one reporter-test assertion and document opaque IDs. Reporter
runtime and workflow source are unchanged.

## Verification

- Four new actual-helper controls fail on the parent.
- `node --test tests/runner-e2e/chat-feature-testops-api.test.mjs
tests/runner-e2e/testops-chat-reporter.test.mjs`: 58 affected
pure/mock/FS controls pass on Node 26.4.0.
- Three MJS syntax checks and `git diff --check` pass. TypeScript noEmit
was not run for this MJS/documentation change.
- Two intermediate exact-projection assertion failures were retained
before the assertions were synced to the new diagnostic contract.
- [Normal Node 24
CI](https://github.com/paperclipai/paperclip/actions/runs/41137009579)
passes on exact head `3c256fb8f7559093c5fb063170e26c01b6e8a0a4`: 48 jobs
passed and one skipped. Both required `ci / e2e` and `ci / verify` pass.
- Greptile reviewed this exact head at 5/5 with no actionable findings
or unresolved threads. At this verification cut, the optional
Contributor trust check was still in progress; it is not an effective
required status context.
- No actual CLI, API, provider, model, app, database, build, install, or
additional bootstrap was run to validate this change locally.

## Risks

The original HTTP 404 route remains unknown. Removing an unsupported
schema assumption does not prove that it caused that failure. The live
job mapping and dispatch remain unqualified. The strict source, actor,
selection, run-number, nonce, once-only send, private custody, case,
model, grader, and clock checks are unchanged. Unknown metadata still
stops. This PR adds no endpoint, workflow mode, permission, credential
path, or passing result.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, code editing, tool
use, and pure test review. The runtime does not expose the exact
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-30 21:41:05 -05:00
1 parent 8ff812cdde
commit d06c5e0eff
4 files changed
+146 -81

No files matched your search

+7 -5
View File
@@ -224,11 +224,13 @@ approved number. Missing or uncertain metadata stops; an uncertain request needs
reconciliation rather than a new send. The nonce and launch search provide
additional evidence. Normal TestOps **Run** uses `evaluate` and is independent of
this one-time administrator fence. The launch nonce is protected configuration,
not a dispatch input. `CHAT_TESTOPS_OPERATION_PARAMETER_ID` is required only when
the registered job defines `operation`; the launcher sets that parameter to
`evaluate`. An official job with only `Branch` keeps its exact
`refs/heads/master` value and uses the fixed workflow's `evaluate` default.
Unknown parameters and other branch values are rejected.
not a dispatch input. Job parameter IDs are opaque. The launcher preserves the
API's exact ID/value pair for `refs/heads/master` and uses the fixed workflow's
`evaluate` default. An operation parameter is used only when the protected
`CHAT_TESTOPS_OPERATION_PARAMETER_ID` matches a returned ID with a `bootstrap`
or `evaluate` value; the launcher then sets it to `evaluate`. Other values,
duplicate IDs and conflicting configuration are rejected. No relationship to
global environment-variable IDs or names is inferred.
`CHAT_TESTOPS_INTEGRATION_ID` names the reviewed existing project GitHub
integration. The launcher verifies its project, type and enabled state, then binds
+70 -47
View File
@@ -9,6 +9,7 @@ const fail = code => {
const match = /^(testops_(?:api|auth)|chat_testops_github)_http_([1-5][0-9]{2})$/.exec(code);
throw new SetupFailure(match ? `${match[1]}_http` : code, match ? Number(match[2]) : null);
};
const bootstrapRequestClasses = new Set(['JOB_RUN_GET', 'LAUNCH_GET', 'JOB_GET']);
const diagnosticCodes = new Set(['untrusted_chat_testops_context', 'testops_token_missing', 'testops_auth_http',
'testops_api_http', 'testops_auth_invalid', 'testops_route_invalid', 'testops_response_too_large',
'testops_response_invalid', 'testops_metadata_page_incomplete', 'testops_case_identity_invalid',
@@ -20,10 +21,14 @@ const diagnosticCodes = new Set(['untrusted_chat_testops_context', 'testops_toke
'chat_testops_job_id_mismatch', 'chat_testops_job_type_mismatch', 'chat_testops_job_parameters_shape_mismatch',
'chat_testops_job_parameters_limit', 'chat_testops_job_url_mismatch', 'chat_testops_jobrun_mismatch', 'chat_testops_launch_mismatch', 'chat_testops_selection_mismatch', 'chat_testops_parameter_mismatch', 'chat_testops_nonce_invalid', 'chat_testops_nonce_conflict', 'chat_testops_catalog_incomplete', 'chat_testops_launch_admission', 'chat_testops_github_http', 'chat_testops_integration_mismatch', 'chat_testops_upload_mismatch', 'chat_testops_upload_unverified']);
export function failureProjection(error, stage) {
const request = error instanceof SetupFailure ? error.request : undefined;
const stages = ['context', 'output_admission', 'authentication', 'metadata_inspection', 'plan_bootstrap', 'selection_validation', 'launch_admission', 'launch_preflight', 'plan_launch', 'bootstrap_job_metadata', 'result_export', 'upload_verification', 'bootstrap_context', 'receipt'];
return { stage: stages.includes(stage) ? stage : 'unknown',
code: error instanceof SetupFailure && diagnosticCodes.has(error.code) ? error.code : 'unknown_failure',
httpStatus: error instanceof SetupFailure && Number.isInteger(error.status) && error.status >= 100 && error.status < 600 ? error.status : null };
httpStatus: error instanceof SetupFailure && Number.isInteger(error.status) && error.status >= 100 && error.status < 600 ? error.status : null,
...(request && bootstrapRequestClasses.has(request.class) && id(request.resourceId) && ['STARTED', 'SUCCEEDED', 'FAILED'].includes(request.state)
? { request: { class: request.class, resourceId: request.resourceId, state: request.state,
httpStatus: Number.isInteger(request.httpStatus) && request.httpStatus >= 100 && request.httpStatus < 600 ? request.httpStatus : null } } : {}) };
}
const id = value => Number.isSafeInteger(value) && value > 0;
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
@@ -186,63 +191,81 @@ async function registeredPlan(api, env) {
const integration = await api(`/api/integration/${ids.integrationId}/project/1`);
if (integration.id !== ids.integrationId || integration.projectId !== 1 || integration.info?.type !== 'github' || integration.disabled !== false) fail('chat_testops_integration_mismatch');
if (!Array.isArray(job.parameters) || job.parameters.length < 1 || job.parameters.length > 2) fail('chat_testops_parameter_mismatch');
const seenIds = new Set(), seenNames = new Set(), parameters = [];
const seenIds = new Set(), parameters = [];
let hasOperation = false, hasRef = false;
for (const parameter of job.parameters) {
if (!id(parameter.id) || seenIds.has(parameter.id)) fail('chat_testops_parameter_mismatch');
seenIds.add(parameter.id);
const definition = await api(`/api/environment/${parameter.id}`);
if (definition.id !== parameter.id || !['Branch', 'operation'].includes(definition.name) || seenNames.has(definition.name)) fail('chat_testops_parameter_mismatch');
seenNames.add(definition.name);
if (definition.name === 'Branch') {
if (parameter.value !== 'refs/heads/master') fail('chat_testops_parameter_mismatch');
parameters.push({ id: parameter.id, value: parameter.value });
} else {
if (ids.parameterId !== parameter.id || !['bootstrap', 'evaluate'].includes(parameter.value)) fail('chat_testops_parameter_mismatch');
if (parameter.id === ids.parameterId) {
if (!['bootstrap', 'evaluate'].includes(parameter.value)) fail('chat_testops_parameter_mismatch');
hasOperation = true;
parameters.push({ id: parameter.id, value: 'evaluate' });
} else {
if (hasRef || parameter.value !== 'refs/heads/master') fail('chat_testops_parameter_mismatch');
hasRef = true;
parameters.push({ id: parameter.id, value: parameter.value });
}
}
if (ids.parameterId !== null && !seenNames.has('operation')) fail('chat_testops_parameter_mismatch');
if (ids.parameterId === null ? parameters.length !== 1 || !hasRef : !hasOperation) fail('chat_testops_parameter_mismatch');
return { ...ids, selected, plan, job, parameters };
}
export async function bootstrapJobMetadata(api, env, expectedLaunchId) {
const strict = expectedLaunchId !== undefined;
const context = strict ? publicContext(env) : undefined;
if (strict && !id(expectedLaunchId)) fail('chat_testops_launch_mismatch');
if (!positive(env.ALLURE_JOB_RUN_ID)) {
if (strict) fail('chat_testops_jobrun_mismatch');
return { status: 'official_job_run_id_missing', actualResult: null };
const requests = [];
const read = async (kind, resourceId, route) => {
const request = { class: kind, resourceId, state: 'STARTED', httpStatus: null };
requests.push(request);
try {
const value = await api(route);
request.state = 'SUCCEEDED';
return value;
} catch (error) {
request.state = 'FAILED';
request.httpStatus = error instanceof SetupFailure ? error.status : null;
throw error;
}
};
try {
const strict = expectedLaunchId !== undefined;
const context = strict ? publicContext(env) : undefined;
if (strict && !id(expectedLaunchId)) fail('chat_testops_launch_mismatch');
if (!positive(env.ALLURE_JOB_RUN_ID)) {
if (strict) fail('chat_testops_jobrun_mismatch');
return { status: 'official_job_run_id_missing', actualResult: null };
}
const run = await read('JOB_RUN_GET', Number(env.ALLURE_JOB_RUN_ID), `/api/jobrun/${env.ALLURE_JOB_RUN_ID}`);
if (!id(run?.job?.id)) fail('chat_testops_jobrun_mismatch');
if (strict) {
if (run.id !== Number(env.ALLURE_JOB_RUN_ID) || run.job.type !== 'github' || run.launchId !== expectedLaunchId
|| run.url !== `https://github.com/paperclipai/paperclip/actions/runs/${context.runId}`) fail('chat_testops_jobrun_mismatch');
const launch = await read('LAUNCH_GET', expectedLaunchId, `/api/launch/${expectedLaunchId}`);
if (launch.id !== expectedLaunchId || launch.projectId !== 1) fail('chat_testops_launch_mismatch');
}
const job = await read('JOB_GET', run.job.id, `/api/job/${run.job.id}`);
if (job.id !== run.job.id) fail('chat_testops_job_id_mismatch');
if (job.type !== 'github') fail('chat_testops_job_type_mismatch');
if (!Array.isArray(job.parameters)) fail('chat_testops_job_parameters_shape_mismatch');
if (job.parameters.length > 4) fail('chat_testops_job_parameters_limit');
if (strict && !publicJobUrl(job.url, env)) fail('chat_testops_job_url_mismatch');
const parameters = [];
const seen = new Set();
for (const parameter of job.parameters) {
if (!id(parameter.id) || strict && seen.has(parameter.id)) fail('chat_testops_parameter_mismatch');
seen.add(parameter.id);
const value = Object.getOwnPropertyDescriptor(parameter, 'value');
const fixedRef = value && Object.hasOwn(value, 'value') && value.value === 'refs/heads/master';
parameters.push({ id: parameter.id, kind: fixedRef ? 'fixed_default_ref' : 'UNRESOLVED',
value: fixedRef ? 'refs/heads/master' : 'NOT_PROJECTED' });
}
return { ...(strict ? { schema: 'paperclip.chat-feature-testops-bootstrap-registration/v1', ...context, capturedAfterWatch: true, runnableBindingQualified: false } : {}),
status: 'observed', projectId: 1, jobId: job.id, allureJobRunId: env.ALLURE_JOB_RUN_ID,
publicWorkflowMatches: publicJobUrl(job.url, env), currentRunUrlMatches: run.url === `https://github.com/paperclipai/paperclip/actions/runs/${env.GITHUB_RUN_ID}`,
launchId: id(run.launchId) ? run.launchId : null, parameters, requests, actualResult: null };
} catch (error) {
const failure = error instanceof SetupFailure ? error : new SetupFailure('unknown_failure');
if (requests.length) failure.request = { ...requests.at(-1) };
throw failure;
}
const run = await api(`/api/jobrun/${env.ALLURE_JOB_RUN_ID}`);
if (!id(run?.job?.id)) fail('chat_testops_jobrun_mismatch');
if (strict) {
if (run.id !== Number(env.ALLURE_JOB_RUN_ID) || run.job.type !== 'github' || run.launchId !== expectedLaunchId
|| run.url !== `https://github.com/paperclipai/paperclip/actions/runs/${context.runId}`) fail('chat_testops_jobrun_mismatch');
const launch = await api(`/api/launch/${expectedLaunchId}`);
if (launch.id !== expectedLaunchId || launch.projectId !== 1) fail('chat_testops_launch_mismatch');
}
const job = await api(`/api/job/${run.job.id}`);
if (job.id !== run.job.id) fail('chat_testops_job_id_mismatch');
if (job.type !== 'github') fail('chat_testops_job_type_mismatch');
if (!Array.isArray(job.parameters)) fail('chat_testops_job_parameters_shape_mismatch');
if (job.parameters.length > 4) fail('chat_testops_job_parameters_limit');
if (strict && !publicJobUrl(job.url, env)) fail('chat_testops_job_url_mismatch');
const parameters = [];
const seen = new Set();
for (const parameter of job.parameters) {
if (!id(parameter.id) || strict && seen.has(parameter.id)) fail('chat_testops_parameter_mismatch');
seen.add(parameter.id);
const definition = await api(`/api/environment/${parameter.id}`);
if (strict && definition.id !== parameter.id) fail('chat_testops_parameter_mismatch');
const known = ['operation', 'ALLURE_JOB_RUN_ID', 'ALLURE_USERNAME', 'Branch'].includes(definition.name);
parameters.push({ id: parameter.id, name: known ? definition.name : 'UNKNOWN',
value: definition.name === 'operation' && ['bootstrap', 'evaluate', 'launch-plan'].includes(parameter.value)
|| definition.name === 'Branch' && parameter.value === 'refs/heads/master' ? parameter.value : 'NOT_PROJECTED' });
}
return { ...(strict ? { schema: 'paperclip.chat-feature-testops-bootstrap-registration/v1', ...context, capturedAfterWatch: true, runnableBindingQualified: false } : {}),
status: 'observed', projectId: 1, jobId: job.id, allureJobRunId: env.ALLURE_JOB_RUN_ID,
publicWorkflowMatches: publicJobUrl(job.url, env), currentRunUrlMatches: run.url === `https://github.com/paperclipai/paperclip/actions/runs/${env.GITHUB_RUN_ID}`,
launchId: id(run.launchId) ? run.launchId : null, parameters, actualResult: null };
}
const launchAdmissions = new WeakSet();
@@ -168,16 +168,12 @@ async function fence(env = launchEnv, changes = {}, workflowChanges = {}, repoCh
: url.includes('/actions/runs/') ? { ...ownRun, ...changes }
: { id: 99, path: '.github/workflows/chat-feature-testops.yml', state: 'active', ...workflowChanges }) }));
}
function launchServer({ wrongParameter = false, wrongAssignment = false, uncertainRun = false, integrationChange = {}, initialParameters = [{ id: 17, value: 'bootstrap' }], expectedParameters = [{ id: 17, value: 'evaluate' }], definitions = {}, jobUrl = 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml' } = {}) {
function launchServer({ wrongAssignment = false, uncertainRun = false, integrationChange = {}, initialParameters = [{ id: 17, value: 'bootstrap' }], expectedParameters = [{ id: 17, value: 'evaluate' }], jobUrl = 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml' } = {}) {
const calls = [], parameters = structuredClone(initialParameters); let assigned = false, launched = false, requests = 0;
const api = async (route, method, body) => {
calls.push({ route, method, body });
if (route.startsWith('/api/launch?')) return { totalPages: 1, content: launched ? [{ id: 10, name: `Chat feature evals · ${'1'.repeat(32)}` }] : [] };
if (route === '/api/integration/2/project/1') return { id: 2, projectId: 1, info: { type: 'github' }, disabled: false, ...integrationChange };
if (route.startsWith('/api/environment/')) {
const id = Number(route.split('/').at(-1));
return definitions[id] ?? { id, name: id === 17 ? wrongParameter ? 'OTHER_PRIVATE_PARAMETER' : 'operation' : 'Branch' };
}
if (route === '/api/job/7') {
if (method === 'PATCH') { assert.deepEqual(body, { integrationId: 2, canRun: true, parameters: expectedParameters }); parameters.splice(0, parameters.length, ...structuredClone(body.parameters)); }
return { id: 7, type: 'github', url: jobUrl, parameters };
@@ -207,7 +203,7 @@ test('observed launch nonce prevents a second run request across separate local
});
test('foreign assignment and unobserved operation parameter stop before any write', async () => {
for (const options of [{ wrongAssignment: true }, { wrongParameter: true }]) {
for (const options of [{ wrongAssignment: true }, { initialParameters: [{ id: 18, value: 'bootstrap' }] }]) {
const server = launchServer(options);
await assert.rejects(launchChatPlan(server.api, launchEnv, () => assert.fail('no marker'), await fence()));
assert.equal(server.calls.some(row => row.method), false); assert.equal(server.requests(), 0);
@@ -221,15 +217,14 @@ test('uncertain plan-run transport is sent once and never followed by another mu
assert.equal(server.calls.at(-1).route, '/api/testplan/8/run');
});
test('official bootstrap job metadata projects only fixed parameter names and operation enums', async () => {
test('official bootstrap job metadata leaves opaque parameter labels unresolved and private values unprojected', async () => {
const job = await bootstrapJobMetadata(async route => {
if (route === '/api/jobrun/9') return { id: 9, job: { id: 7 }, launchId: 10, url: 'https://github.com/paperclipai/paperclip/actions/runs/123' };
if (route === '/api/job/7') return { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml', parameters: [{ id: 17, value: 'bootstrap' }, { id: 18, value: 'SYNTHETIC_PRIVATE_VALUE' }] };
if (route === '/api/environment/17') return { id: 17, name: 'operation' };
return { id: 18, name: 'UNRELATED_NAME' };
assert.fail('no namespace lookup or unrelated request');
}, publicEnv);
assert.equal(job.publicWorkflowMatches, true); assert.equal(job.parameters[0].value, 'bootstrap');
assert.deepEqual(job.parameters[1], { id: 18, name: 'UNKNOWN', value: 'NOT_PROJECTED' });
assert.equal(job.publicWorkflowMatches, true); assert.deepEqual(job.parameters[0], { id: 17, kind: 'UNRESOLVED', value: 'NOT_PROJECTED' });
assert.deepEqual(job.parameters[1], { id: 18, kind: 'UNRESOLVED', value: 'NOT_PROJECTED' });
assert.ok(!JSON.stringify(job).includes('SYNTHETIC_PRIVATE_VALUE'));
});
@@ -404,8 +399,6 @@ function strictBootstrapApi(patch = {}) {
if (route === '/api/jobrun/9') return run;
if (route === '/api/launch/10') return launch;
if (route === '/api/job/7') return job;
if (route === '/api/environment/17') return { id: 17, name: 'operation', ...patch.definition };
if (route === '/api/environment/18') return { id: 18, name: 'SYNTHETIC_PRIVATE_NAME' };
assert.fail(route);
};
}
@@ -415,7 +408,7 @@ test('post-watch reader binds actual current run source project launch and publi
assert.equal(result.jobId, 7); assert.equal(result.runId, publicEnv.GITHUB_RUN_ID);
assert.equal(result.workflowSha, publicEnv.GITHUB_SHA); assert.equal(result.capturedAfterWatch, true);
assert.equal(result.runnableBindingQualified, false); assert.equal(result.actualResult, null);
assert.deepEqual(result.parameters[1], { id: 18, name: 'UNKNOWN', value: 'NOT_PROJECTED' });
assert.deepEqual(result.parameters[1], { id: 18, kind: 'UNRESOLVED', value: 'NOT_PROJECTED' });
assert.ok(!JSON.stringify(result).includes('SYNTHETIC_PRIVATE'));
});
@@ -423,14 +416,14 @@ test('post-watch reader rejects stale foreign project launch source and job iden
for (const patch of [{ run: { id: 99 } }, { run: { url: 'https://github.com/paperclipai/paperclip/actions/runs/999' } },
{ run: { launchId: 20 } }, { run: { job: { id: 7, type: 'foreign' } } }, { launch: { projectId: 2 } },
{ launch: { id: 20 } }, { job: { url: 'https://github.com/other/repo/actions/workflows/chat-feature-testops.yml' } },
{ job: { type: 'foreign' } }, { definition: { id: 99 } }]) {
{ job: { type: 'foreign' } }, { job: { parameters: [{ id: 0 }] } }]) {
await assert.rejects(bootstrapJobMetadata(strictBootstrapApi(patch), publicEnv, 10));
}
await assert.rejects(bootstrapJobMetadata(() => assert.fail('no API'), { ...publicEnv, GITHUB_SHA: 'untrusted' }, 10));
await assert.rejects(bootstrapJobMetadata(() => assert.fail('no API'), { ...publicEnv, ALLURE_JOB_RUN_ID: '' }, 10));
});
test('post-watch reader caps parameter definitions and refuses duplicate IDs', async () => {
test('post-watch reader caps opaque parameters and refuses duplicate IDs', async () => {
for (const parameters of [Array.from({ length: 5 }, (_, i) => ({ id: i + 1 })), [{ id: 17 }, { id: 17 }]]) {
await assert.rejects(bootstrapJobMetadata(strictBootstrapApi({ job: { parameters } }), publicEnv, 10));
}
@@ -460,7 +453,8 @@ test('official legacy URL admission rejects foreign workflow repo query credenti
observedLegacyJobUrl.replace('https://github.com', 'https://user:password@github.com'), `${observedLegacyJobUrl}#extra`]) {
await assert.rejects(validateChatSelection(selectedApi({ '/api/job/7': { id: 7, type: 'github', url } }), publicEnv), /chat_testops_job_mismatch/);
await assert.rejects(bootstrapJobMetadata(strictBootstrapApi({ job: { url } }), publicEnv, 10), error => {
assert.deepEqual(failureProjection(error, 'bootstrap_context'), { stage: 'bootstrap_context', code: 'chat_testops_job_url_mismatch', httpStatus: null });
assert.deepEqual(failureProjection(error, 'bootstrap_context'), { stage: 'bootstrap_context', code: 'chat_testops_job_url_mismatch', httpStatus: null,
request: { class: 'JOB_GET', resourceId: 7, state: 'SUCCEEDED', httpStatus: null } });
return true;
});
const server = launchServer({ jobUrl: url });
@@ -481,7 +475,8 @@ test('bootstrap job predicate failures retain only distinct fixed codes and no o
]) {
await assert.rejects(bootstrapJobMetadata(strictBootstrapApi({ job }), publicEnv, 10), error => {
const projected = failureProjection(error, 'bootstrap_context');
assert.deepEqual(projected, { stage: 'bootstrap_context', code, httpStatus: null });
assert.deepEqual(projected, { stage: 'bootstrap_context', code, httpStatus: null,
request: { class: 'JOB_GET', resourceId: 7, state: 'SUCCEEDED', httpStatus: null } });
assert.ok(!JSON.stringify(projected).includes('SYNTHETIC_PRIVATE'));
return true;
});
@@ -508,33 +503,78 @@ test('launch preserves exact Branch and defaults evaluate only when operation is
test('launch refuses empty unknown duplicate wrong Branch or conflicting operation before any mutation', async () => {
const branch = { id: 19, value: 'refs/heads/master' };
for (const [parameters, opId, definitions] of [
for (const [parameters, opId] of [
[[], undefined, {}],
[[{ id: 19, value: 'refs/heads/other' }], undefined, {}],
[[branch, branch], undefined, {}],
[[branch, { id: 20, value: 'refs/heads/master' }], undefined, {}],
[[{ id: 19, value: 'refs/heads/master' }], undefined, { 19: { id: 99, name: 'Branch' } }],
[[{ id: 0, value: 'refs/heads/master' }], undefined, {}],
[[{ id: 19, value: 'PRIVATE_PARAMETER_VALUE' }], undefined, { 19: { id: 19, name: 'PRIVATE_PARAMETER_NAME' } }],
[[branch], '17', {}],
[[branch], 'bad-id', {}],
[[branch], '0', {}],
[[branch], ' ', {}],
[[{ id: 17, value: 'bootstrap' }], undefined, {}],
[[{ id: 17, value: 'bootstrap' }], '18', {}],
[[{ id: 17, value: 'launch-plan' }], '17', {}],
[[branch, { id: 17, value: 'bootstrap' }, { id: 20, value: 'refs/heads/master' }], '17', {}],
]) {
const env = { ...launchEnv, CHAT_TESTOPS_OPERATION_PARAMETER_ID: opId };
const server = launchServer({ initialParameters: parameters, definitions });
const server = launchServer({ initialParameters: parameters });
await assert.rejects(launchChatPlan(server.api, env, async () => assert.fail('no mutation marker'), await fence(env)), /chat_testops_(?:identity_missing|parameter_mismatch)/);
assert.equal(server.calls.some(call => call.method), false); assert.equal(server.requests(), 0);
}
});
test('bootstrap projects only fixed Branch name and exact master value alongside genuine numeric ID', async () => {
const base = strictBootstrapApi({ job: { parameters: [{ id: 19, value: 'refs/heads/master' }] } });
const result = await bootstrapJobMetadata(route => route === '/api/environment/19' ? { id: 19, name: 'Branch' } : base(route), publicEnv, 10);
assert.deepEqual(result.parameters, [{ id: 19, name: 'Branch', value: 'refs/heads/master' }]);
const hidden = strictBootstrapApi({ job: { parameters: [{ id: 19, value: 'PRIVATE_BRANCH_VALUE' }] } });
const unknown = await bootstrapJobMetadata(route => route === '/api/environment/19' ? { id: 19, name: 'Branch' } : hidden(route), publicEnv, 10);
assert.deepEqual(unknown.parameters, [{ id: 19, name: 'Branch', value: 'NOT_PROJECTED' }]);
test('bootstrap projects an exact default ref without inventing an opaque parameter label', async () => {
const result = await bootstrapJobMetadata(strictBootstrapApi({ job: { parameters: [{ id: 19, value: 'refs/heads/master' }] } }), publicEnv, 10);
assert.deepEqual(result.parameters, [{ id: 19, kind: 'fixed_default_ref', value: 'refs/heads/master' }]);
const unknown = await bootstrapJobMetadata(strictBootstrapApi({ job: { parameters: [{ id: 19, value: 'PRIVATE_BRANCH_VALUE' }] } }), publicEnv, 10);
assert.deepEqual(unknown.parameters, [{ id: 19, kind: 'UNRESOLVED', value: 'NOT_PROJECTED' }]);
assert.ok(!JSON.stringify(unknown).includes('PRIVATE_BRANCH'));
});
test('opaque launch vector uses only returned IDs and never queries any environment namespace', async () => {
const env = { ...launchEnv, CHAT_TESTOPS_OPERATION_PARAMETER_ID: '' };
const server = launchServer({ initialParameters: [{ id: 819, value: 'refs/heads/master' }], expectedParameters: [{ id: 819, value: 'refs/heads/master' }] });
const result = await launchChatPlan(async (...args) => { assert.ok(!args[0].includes('/environment')); return server.api(...args); }, env, async () => {}, await fence(env));
assert.equal(result.status, 'REQUESTED'); assert.equal(server.requests(), 1);
});
test('bootstrap successful reads expose only the three fixed source request classes and numeric IDs', async () => {
const result = await bootstrapJobMetadata(strictBootstrapApi(), publicEnv, 10);
assert.deepEqual(result.requests, [
{ class: 'JOB_RUN_GET', resourceId: 9, state: 'SUCCEEDED', httpStatus: null },
{ class: 'LAUNCH_GET', resourceId: 10, state: 'SUCCEEDED', httpStatus: null },
{ class: 'JOB_GET', resourceId: 7, state: 'SUCCEEDED', httpStatus: null },
]);
});
test('bootstrap actual HTTP client failure identifies each fixed request without reading private bodies', async () => {
for (const [failedRoute, kind, resourceId, count] of [['/api/jobrun/9', 'JOB_RUN_GET', 9, 1], ['/api/launch/10', 'LAUNCH_GET', 10, 2], ['/api/job/7', 'JOB_GET', 7, 3]]) {
const calls = [], base = strictBootstrapApi();
const api = await testopsClient('SYNTHETIC_TOKEN', async (url, options) => {
const route = url.slice(endpoint.length);
if (route === '/api/uaa/oauth/token') return { ok: true, json: async () => ({ access_token: 'SYNTHETIC_ACCESS' }) };
calls.push(route); assert.equal(options.method, 'GET');
if (route === failedRoute) return { ok: false, status: 404, text: () => assert.fail('private error body must stay unopened') };
return { ok: true, status: 200, text: async () => JSON.stringify(await base(route)) };
});
await assert.rejects(bootstrapJobMetadata(api, publicEnv, 10), error => {
const projected = failureProjection(error, 'bootstrap_context');
assert.deepEqual(projected, { stage: 'bootstrap_context', code: 'testops_api_http', httpStatus: 404,
request: { class: kind, resourceId, state: 'FAILED', httpStatus: 404 } });
assert.ok(!JSON.stringify(projected).includes('SYNTHETIC')); return true;
});
assert.equal(calls.length, count);
}
});
test('bootstrap unknown transport errors retain only source-owned request class without private error text', async () => {
await assert.rejects(bootstrapJobMetadata(() => { throw new Error('PRIVATE_URL_TOKEN_RESPONSE'); }, publicEnv, 10), error => {
const projected = failureProjection(error, 'bootstrap_context');
assert.deepEqual(projected, { stage: 'bootstrap_context', code: 'unknown_failure', httpStatus: null,
request: { class: 'JOB_RUN_GET', resourceId: 9, state: 'FAILED', httpStatus: null } });
assert.ok(!JSON.stringify(projected).includes('PRIVATE')); return true;
});
});
@@ -200,7 +200,7 @@ test('post-watch facade runs exact official command with bounded pipes and prese
assert.equal(result.runId, '123'); assert.equal(result.workflowSha, context.GITHUB_SHA); assert.equal(result.targetSha, context.GITHUB_SHA);
assert.equal(result.currentRunUrlMatches, true); assert.equal(result.publicWorkflowMatches, true);
assert.equal(result.capturedAfterWatch, true); assert.equal(result.runnableBindingQualified, false); assert.equal(result.actualResult, null);
assert.deepEqual(result.parameters, [{ id: 17, name: 'operation', value: 'bootstrap' }]);
assert.deepEqual(result.parameters, [{ id: 17, kind: 'UNRESOLVED', value: 'NOT_PROJECTED' }]);
assert.ok(!bytes.toString().includes('SYNTHETIC')); assert.equal(f.env.ALLURE_JOB_RUN_ID, '');
assert.deepEqual(await readFile(path.join(f.state, 'job.json')), f.originalJob);
const second = await captureSafeErrors(() => reporter('bootstrap-context', undefined, f.env, () => assert.fail('no second auth'), () => assert.fail('no second CLI')));