fix(evals): support official TestOps job metadata (#18385)

## Thinking Path

> - Paperclip manages AI agents and verifies real work through Product
E2E tests.
> - The fixed Chat feature workflow uses TestOps to select and report
one backlog case.
> - Official registration exposes a workflow-query URL and a Branch
parameter.
> - The existing reader accepts only workflow-file or numeric URLs, and
the admin launcher requires an operation parameter.
> - These assumptions reject valid official metadata before the feature
can run.
> - This pull request supports the exact observed form while preserving
source, selection, and once-only execution checks.

## Linked Issues or Issue Description

Refs: #18322, #18375.

**What happened?**
The second zero-model bootstrap stopped with a generic job-mismatch
code. Ordinary TestOps UI inspection then showed an official
workflow-query URL outside the accepted literals and a Branch-only job.
The exact first failed predicate in that bootstrap remains unknown.

**Expected behavior**
Accept that exact fixed-workflow URL. Preserve the existing master
Branch value when the job has no operation parameter. Keep rejecting
foreign URLs, unknown parameters, wrong branches, and conflicting
operation IDs before mutation.

**Steps to reproduce**
Pass the official query URL and Branch-only metadata through the
existing bootstrap, selection, and admin-launch owners. The new pure
controls exercise those actual helpers without a CLI, backend, or model
call.

**Paperclip version or commit**
This change starts from `0e15c92073a09e50df99190a37f69a86286b989b` after
#18375.

**Deployment mode**
GitHub Actions and the protected TestOps reporting environment.

## What Changed

- Share one exact URL predicate across bootstrap, evaluation selection,
and admin preparation. Add only the observed official workflow-query
literal.
- Support Branch-only, operation-only, and Branch-plus-operation jobs
with strict definition IDs, names, values, and duplicate checks.
- Preserve the exact master Branch value. Set an existing operation
parameter to `evaluate`; otherwise use the fixed workflow's existing
default.
- Use the same checked parameter vector for job and plan writes and
their readbacks.
- Report distinct fixed bootstrap predicate codes without response
values.
- Document when the protected operation-parameter ID is required.
- Align one existing Cloud-maintenance test with the inherited
managed-Cloud S3 policy. Preserve the same-version local-storage proof
in the parent, use valid synthetic Cloud settings in the child, and
check the expected provider mismatch without S3 requests. Retain the
no-seed and unchanged persisted-row assertions.

## Verification

- URL controls: three failures on the parent; all pass with the
correction.
- Branch controls: two failures and one correct rejection on the
predecessor; all pass with the correction.
- `node --test tests/runner-e2e/chat-feature-testops-api.test.mjs`: 40
owning pure/mock controls pass on Node 26.4.0.
- Two MJS syntax checks and `git diff --check` pass. TypeScript noEmit
was not run for this MJS/documentation change.
- The first [normal CI
run](https://github.com/paperclipai/paperclip/actions/runs/41131094974)
failed only in the managed-skill Cloud fixture and its dependent
verification gate: 46 jobs passed, two failed, and one skipped. The
tested merge included a new upstream policy that rejects the test's
local-disk Cloud configuration. The exact caught runtime exception
remains unknown.
- Three pure controls using the actual policy validator pass: the old
invalid Cloud fixture rejects, the new synthetic S3 fixture admits, and
an invalid master key rejects. An initial tsx wrapper IPC permission
error executed zero controls and remains a separate setup failure.
- The PostgreSQL/maintenance-child test was not run locally. On new-head
AWS CI, `managed-skill-content.test.ts` passed all 17 tests in server
shard 9.
- [Normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41133876636)
is green on exact head `b76d5dd40d7711f1b5f6b6316b1114c0aae02d57`: 48
jobs passed and one skipped. Both required `ci / e2e` and `ci / verify`
pass.
- Greptile reviewed this exact head at 5/5 with no actionable findings
or unresolved review threads.
- No actual CLI, backend, model, provider, app, database, build, or
bootstrap was run to validate this correction locally.

## Risks

The exact original bootstrap predicate is still unknown. This fixes a
demonstrated metadata false negative and preserves the prior failed run.
Actual backend parameter types and integration dispatch still need
verification. Unknown metadata stops before the once marker and writes.
This PR changes no workflow, credential, case, model, grader, clock, or
private-evidence policy. It does not claim a passing feature result.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, code editing, tool
use, and pure test review. The runtime does not expose the exact
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-30 17:04:59 -05:00
1 parent 198218bf55
commit 8595efc20b
4 files changed
+163 -28

No files matched your search

@@ -79,7 +79,9 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
it("loads persisted Cloud configuration for read-only maintenance without seeding it", async () => {
const companyId = await company();
await companySkillService(db).createLocalSkill(companyId, { name: "Cloud CLI", markdown: "# Cloud CLI" });
const skill = await companySkillService(db).createLocalSkill(companyId, { name: "Cloud CLI", markdown: "# Cloud CLI" });
const local = await skillInstallationMaintenance(db).verifyVersions({ companyId, limit: 50 });
expect(local.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: "local_storage" })]);
const key = "cloud-runtime-config:v1";
const run = (phase: string) => promisify(execFile)(process.execPath, [
"--import", fileURLToPath(import.meta.resolve("tsx")),
@@ -87,16 +89,18 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
`--phase=${phase}`, `--company=${companyId}`, "--limit=50",
], { env: { ...process.env, DATABASE_URL: database.connectionString,
PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "disposable-cloud-cli-fixture",
PAPERCLIP_STORAGE_PROVIDER: "local_disk", PAPERCLIP_STORAGE_LOCAL_DIR: path.join(root, "objects") },
PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_STORAGE_S3_BUCKET: "managed-skill-cli-fixture",
PAPERCLIP_STORAGE_S3_REGION: "us-east-1", PAPERCLIP_SECRETS_MASTER_KEY: "12345678901234567890123456789012" },
timeout: 60_000, maxBuffer: 1024 * 1024 });
await expect(run("eligibility")).rejects.toThrow("Initialize Cloud runtime configuration");
expect(await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, key))).toHaveLength(0);
await db.insert(instanceSettings).values({ singletonKey: key, general: { version: 1, revision: 0, config: null } });
const [persisted] = await db.insert(instanceSettings).values({ singletonKey: key, general: { version: 1, revision: 0, config: null } }).returning();
try {
const eligibility = JSON.parse((await run("eligibility")).stdout.trim().split("\n").at(-1)!);
expect(eligibility.results).toEqual([expect.objectContaining({ status: "version_backed" })]);
expect(eligibility.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: "version_backed" })]);
const verification = JSON.parse((await run("verification")).stdout.trim().split("\n").at(-1)!);
expect(verification.results).toEqual([expect.objectContaining({ status: "local_storage" })]);
expect(verification.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: "provider_mismatch" })]);
expect(await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, key))).toEqual([persisted]);
} finally {
await db.delete(instanceSettings).where(eq(instanceSettings.singletonKey, key));
}
+6 -2
View File
@@ -223,8 +223,12 @@ actors before any TestOps mutation. A later dispatch or replay cannot reuse that
approved number. Missing or uncertain metadata stops; an uncertain request needs
reconciliation rather than a new send. The nonce and launch search provide
additional evidence. Normal TestOps **Run** uses `evaluate` and is independent of
this one-time administrator fence. The operation parameter identifier and launch
nonce are also protected configuration, not dispatch inputs.
this one-time administrator fence. The launch nonce is protected configuration,
not a dispatch input. `CHAT_TESTOPS_OPERATION_PARAMETER_ID` is required only when
the registered job defines `operation`; the launcher sets that parameter to
`evaluate`. An official job with only `Branch` keeps its exact
`refs/heads/master` value and uses the fixed workflow's `evaluate` default.
Unknown parameters and other branch values are rejected.
`CHAT_TESTOPS_INTEGRATION_ID` names the reviewed existing project GitHub
integration. The launcher verifies its project, type and enabled state, then binds
+36 -15
View File
@@ -17,7 +17,8 @@ const diagnosticCodes = new Set(['untrusted_chat_testops_context', 'testops_toke
'testops_created_case_identity_invalid', 'testops_created_plan_identity_invalid', 'testops_plan_scope_mismatch',
'testops_operation_invalid', 'testops_output_invalid', 'untrusted_public_testops_context', 'chat_testops_identity_missing',
'chat_testops_case_mismatch', 'chat_testops_plan_mismatch', 'chat_testops_plan_job_mismatch', 'chat_testops_job_mismatch',
'chat_testops_jobrun_mismatch', 'chat_testops_launch_mismatch', 'chat_testops_selection_mismatch', 'chat_testops_parameter_mismatch', 'chat_testops_nonce_invalid', 'chat_testops_nonce_conflict', 'chat_testops_catalog_incomplete', 'chat_testops_launch_admission', 'chat_testops_github_http', 'chat_testops_integration_mismatch', 'chat_testops_upload_mismatch', 'chat_testops_upload_unverified']);
'chat_testops_job_id_mismatch', 'chat_testops_job_type_mismatch', 'chat_testops_job_parameters_shape_mismatch',
'chat_testops_job_parameters_limit', 'chat_testops_job_url_mismatch', 'chat_testops_jobrun_mismatch', 'chat_testops_launch_mismatch', 'chat_testops_selection_mismatch', 'chat_testops_parameter_mismatch', 'chat_testops_nonce_invalid', 'chat_testops_nonce_conflict', 'chat_testops_catalog_incomplete', 'chat_testops_launch_admission', 'chat_testops_github_http', 'chat_testops_integration_mismatch', 'chat_testops_upload_mismatch', 'chat_testops_upload_unverified']);
export function failureProjection(error, stage) {
const stages = ['context', 'output_admission', 'authentication', 'metadata_inspection', 'plan_bootstrap', 'selection_validation', 'launch_admission', 'launch_preflight', 'plan_launch', 'bootstrap_job_metadata', 'result_export', 'upload_verification', 'bootstrap_context', 'receipt'];
return { stage: stages.includes(stage) ? stage : 'unknown',
@@ -146,9 +147,7 @@ export async function validateChatSelection(api, env) {
const assignments = await api(`/api/testplan/${planId}/job`);
if (!Array.isArray(assignments) || assignments.length !== 1 || assignments[0].job?.id !== jobId || assignments[0].testCasesCount !== 1) fail('chat_testops_plan_job_mismatch');
const job = await api(`/api/job/${jobId}`);
const jobUrls = ['https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml'];
if (positive(env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID)) jobUrls.push(`https://github.com/paperclipai/paperclip/actions/workflows/${env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID}`);
if (job.id !== jobId || job.type !== 'github' || !jobUrls.includes(job.url)) fail('chat_testops_job_mismatch');
if (job.id !== jobId || job.type !== 'github' || !publicJobUrl(job.url, env)) fail('chat_testops_job_mismatch');
const run = await api(`/api/jobrun/${jobRunId}`);
if (run.id !== jobRunId || run.job?.id !== jobId || run.job?.type !== 'github'
|| !['scheduled', 'in_progress'].includes(run.stage) || !id(run.launchId)
@@ -162,12 +161,16 @@ export async function validateChatSelection(api, env) {
}
function configuredIds(env) {
const names = ['CHAT_TESTOPS_CASE_ID', 'CHAT_TESTOPS_PLAN_ID', 'CHAT_TESTOPS_JOB_ID', 'CHAT_TESTOPS_OPERATION_PARAMETER_ID', 'CHAT_TESTOPS_INTEGRATION_ID'];
const names = ['CHAT_TESTOPS_CASE_ID', 'CHAT_TESTOPS_PLAN_ID', 'CHAT_TESTOPS_JOB_ID', 'CHAT_TESTOPS_INTEGRATION_ID'];
if (names.some(name => !positive(env[name]))) fail('chat_testops_identity_missing');
return Object.fromEntries(names.map((name, index) => [['caseId', 'planId', 'jobId', 'parameterId', 'integrationId'][index], Number(env[name])]));
const operation = env.CHAT_TESTOPS_OPERATION_PARAMETER_ID;
if (operation !== undefined && operation !== '' && !positive(operation)) fail('chat_testops_identity_missing');
return { ...Object.fromEntries(names.map((name, index) => [['caseId', 'planId', 'jobId', 'integrationId'][index], Number(env[name])])),
parameterId: positive(operation) ? Number(operation) : null };
}
function publicJobUrl(url, env) {
return url === 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml'
|| url === 'https://github.com/paperclipai/paperclip/actions?query=workflow%3A%22Chat+feature+TestOps%22'
|| positive(env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID) && url === `https://github.com/paperclipai/paperclip/actions/workflows/${env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID}`;
}
async function registeredPlan(api, env) {
@@ -182,10 +185,24 @@ async function registeredPlan(api, env) {
if (job.id !== ids.jobId || job.type !== 'github' || !publicJobUrl(job.url, env)) fail('chat_testops_job_mismatch');
const integration = await api(`/api/integration/${ids.integrationId}/project/1`);
if (integration.id !== ids.integrationId || integration.projectId !== 1 || integration.info?.type !== 'github' || integration.disabled !== false) fail('chat_testops_integration_mismatch');
const definition = await api(`/api/environment/${ids.parameterId}`);
if (definition.id !== ids.parameterId || definition.name !== 'operation' || !Array.isArray(job.parameters) || job.parameters.length !== 1
|| job.parameters[0].id !== ids.parameterId || !['bootstrap', 'evaluate'].includes(job.parameters[0].value)) fail('chat_testops_parameter_mismatch');
return { ...ids, selected, plan, job };
if (!Array.isArray(job.parameters) || job.parameters.length < 1 || job.parameters.length > 2) fail('chat_testops_parameter_mismatch');
const seenIds = new Set(), seenNames = new Set(), parameters = [];
for (const parameter of job.parameters) {
if (!id(parameter.id) || seenIds.has(parameter.id)) fail('chat_testops_parameter_mismatch');
seenIds.add(parameter.id);
const definition = await api(`/api/environment/${parameter.id}`);
if (definition.id !== parameter.id || !['Branch', 'operation'].includes(definition.name) || seenNames.has(definition.name)) fail('chat_testops_parameter_mismatch');
seenNames.add(definition.name);
if (definition.name === 'Branch') {
if (parameter.value !== 'refs/heads/master') fail('chat_testops_parameter_mismatch');
parameters.push({ id: parameter.id, value: parameter.value });
} else {
if (ids.parameterId !== parameter.id || !['bootstrap', 'evaluate'].includes(parameter.value)) fail('chat_testops_parameter_mismatch');
parameters.push({ id: parameter.id, value: 'evaluate' });
}
}
if (ids.parameterId !== null && !seenNames.has('operation')) fail('chat_testops_parameter_mismatch');
return { ...ids, selected, plan, job, parameters };
}
export async function bootstrapJobMetadata(api, env, expectedLaunchId) {
@@ -205,8 +222,11 @@ export async function bootstrapJobMetadata(api, env, expectedLaunchId) {
if (launch.id !== expectedLaunchId || launch.projectId !== 1) fail('chat_testops_launch_mismatch');
}
const job = await api(`/api/job/${run.job.id}`);
if (job.id !== run.job.id || job.type !== 'github' || !Array.isArray(job.parameters) || job.parameters.length > 4) fail('chat_testops_job_mismatch');
if (strict && !publicJobUrl(job.url, env)) fail('chat_testops_job_mismatch');
if (job.id !== run.job.id) fail('chat_testops_job_id_mismatch');
if (job.type !== 'github') fail('chat_testops_job_type_mismatch');
if (!Array.isArray(job.parameters)) fail('chat_testops_job_parameters_shape_mismatch');
if (job.parameters.length > 4) fail('chat_testops_job_parameters_limit');
if (strict && !publicJobUrl(job.url, env)) fail('chat_testops_job_url_mismatch');
const parameters = [];
const seen = new Set();
for (const parameter of job.parameters) {
@@ -214,9 +234,10 @@ export async function bootstrapJobMetadata(api, env, expectedLaunchId) {
seen.add(parameter.id);
const definition = await api(`/api/environment/${parameter.id}`);
if (strict && definition.id !== parameter.id) fail('chat_testops_parameter_mismatch');
const known = ['operation', 'ALLURE_JOB_RUN_ID', 'ALLURE_USERNAME'].includes(definition.name);
const known = ['operation', 'ALLURE_JOB_RUN_ID', 'ALLURE_USERNAME', 'Branch'].includes(definition.name);
parameters.push({ id: parameter.id, name: known ? definition.name : 'UNKNOWN',
value: definition.name === 'operation' && ['bootstrap', 'evaluate', 'launch-plan'].includes(parameter.value) ? parameter.value : 'NOT_PROJECTED' });
value: definition.name === 'operation' && ['bootstrap', 'evaluate', 'launch-plan'].includes(parameter.value)
|| definition.name === 'Branch' && parameter.value === 'refs/heads/master' ? parameter.value : 'NOT_PROJECTED' });
}
return { ...(strict ? { schema: 'paperclip.chat-feature-testops-bootstrap-registration/v1', ...context, capturedAfterWatch: true, runnableBindingQualified: false } : {}),
status: 'observed', projectId: 1, jobId: job.id, allureJobRunId: env.ALLURE_JOB_RUN_ID,
@@ -274,7 +295,7 @@ export async function launchChatPlan(api, env, markOnce, admission) {
jobId: known.jobId, caseId: known.caseId, launchId: existing[0].id, status: 'EXISTING_NO_RESEND', actualResult: null };
}
await markOnce({ ...source, launchAdmission: admission, nonce, planId: known.planId, jobId: known.jobId, caseId: known.caseId });
const parameters = [{ id: known.parameterId, value: 'evaluate' }];
const parameters = known.parameters;
await api(`/api/job/${known.jobId}`, 'PATCH', { integrationId: known.integrationId, canRun: true, parameters });
const patched = await api(`/api/job/${known.jobId}`);
if (patched.id !== known.jobId || patched.type !== 'github' || !publicJobUrl(patched.url, env) || !same(patched.parameters, parameters)) fail('chat_testops_parameter_mismatch');
@@ -168,21 +168,24 @@ async function fence(env = launchEnv, changes = {}, workflowChanges = {}, repoCh
: url.includes('/actions/runs/') ? { ...ownRun, ...changes }
: { id: 99, path: '.github/workflows/chat-feature-testops.yml', state: 'active', ...workflowChanges }) }));
}
function launchServer({ wrongParameter = false, wrongAssignment = false, uncertainRun = false, integrationChange = {} } = {}) {
const calls = [], parameters = [{ id: 17, value: 'bootstrap' }]; let assigned = false, launched = false, requests = 0;
function launchServer({ wrongParameter = false, wrongAssignment = false, uncertainRun = false, integrationChange = {}, initialParameters = [{ id: 17, value: 'bootstrap' }], expectedParameters = [{ id: 17, value: 'evaluate' }], definitions = {}, jobUrl = 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml' } = {}) {
const calls = [], parameters = structuredClone(initialParameters); let assigned = false, launched = false, requests = 0;
const api = async (route, method, body) => {
calls.push({ route, method, body });
if (route.startsWith('/api/launch?')) return { totalPages: 1, content: launched ? [{ id: 10, name: `Chat feature evals · ${'1'.repeat(32)}` }] : [] };
if (route === '/api/integration/2/project/1') return { id: 2, projectId: 1, info: { type: 'github' }, disabled: false, ...integrationChange };
if (route === '/api/environment/17') return { id: 17, name: wrongParameter ? 'OTHER_PRIVATE_PARAMETER' : 'operation' };
if (route.startsWith('/api/environment/')) {
const id = Number(route.split('/').at(-1));
return definitions[id] ?? { id, name: id === 17 ? wrongParameter ? 'OTHER_PRIVATE_PARAMETER' : 'operation' : 'Branch' };
}
if (route === '/api/job/7') {
if (method === 'PATCH') { assert.deepEqual(body, { integrationId: 2, canRun: true, parameters: [{ id: 17, value: 'evaluate' }] }); parameters[0].value = 'evaluate'; }
return { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml', parameters };
if (method === 'PATCH') { assert.deepEqual(body, { integrationId: 2, canRun: true, parameters: expectedParameters }); parameters.splice(0, parameters.length, ...structuredClone(body.parameters)); }
return { id: 7, type: 'github', url: jobUrl, parameters };
}
if (route === '/api/testplan/8/job') return wrongAssignment ? [{ job: { id: 70 }, testCasesCount: 1 }]
: assigned ? [{ job: { id: 7, parameters }, testCasesCount: 1 }] : [];
if (route === '/api/testplan/8/assign') { assert.equal(method, 'POST'); assert.deepEqual(body, { jobId: 7, selection: { inverted: false, manual: false, leafsInclude: [91] } }); assigned = true; return null; }
if (route === '/api/testplan/8/jobparameter') { assert.deepEqual(body, { jobId: 7, parameters: [{ id: 17, value: 'evaluate' }] }); return null; }
if (route === '/api/testplan/8/jobparameter') { assert.deepEqual(body, { jobId: 7, parameters: expectedParameters }); return null; }
if (route === '/api/testplan/8/run') { requests++; assert.equal(method, 'POST'); if (uncertainRun) throw new Error('synthetic uncertain transport'); launched = true; return { id: 10, projectId: 1, name: body.launchName }; }
return selectedApi()(route);
};
@@ -432,3 +435,106 @@ test('post-watch reader caps parameter definitions and refuses duplicate IDs', a
await assert.rejects(bootstrapJobMetadata(strictBootstrapApi({ job: { parameters } }), publicEnv, 10));
}
});
const observedLegacyJobUrl = 'https://github.com/paperclipai/paperclip/actions?query=workflow%3A%22Chat+feature+TestOps%22';
test('official observed legacy and existing workflow URLs pass all three actual job consumers', async () => {
for (const url of [observedLegacyJobUrl, 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml',
'https://github.com/paperclipai/paperclip/actions/workflows/99']) {
const env = { ...publicEnv, CHAT_TESTOPS_GITHUB_WORKFLOW_ID: '99' };
const selected = await validateChatSelection(selectedApi({ '/api/job/7': { id: 7, type: 'github', url } }), env);
assert.equal(selected.caseId, 91);
const registered = await bootstrapJobMetadata(strictBootstrapApi({ job: { url } }), env, 10);
assert.equal(registered.publicWorkflowMatches, true);
const server = launchServer({ jobUrl: url });
const launched = await launchChatPlan(server.api, launchEnv, async () => {}, await fence());
assert.equal(launched.status, 'REQUESTED');
assert.equal(server.requests(), 1);
}
});
test('official legacy URL admission rejects foreign workflow repo query credentials and fragment in all consumers', async () => {
for (const url of [observedLegacyJobUrl.replace('/paperclip/actions', '/paperclip-cloud/actions'),
observedLegacyJobUrl.replace('Chat+feature+TestOps', 'Other+workflow'), `${observedLegacyJobUrl}&extra=1`,
observedLegacyJobUrl.replace('https://github.com', 'https://user:password@github.com'), `${observedLegacyJobUrl}#extra`]) {
await assert.rejects(validateChatSelection(selectedApi({ '/api/job/7': { id: 7, type: 'github', url } }), publicEnv), /chat_testops_job_mismatch/);
await assert.rejects(bootstrapJobMetadata(strictBootstrapApi({ job: { url } }), publicEnv, 10), error => {
assert.deepEqual(failureProjection(error, 'bootstrap_context'), { stage: 'bootstrap_context', code: 'chat_testops_job_url_mismatch', httpStatus: null });
return true;
});
const server = launchServer({ jobUrl: url });
await assert.rejects(launchChatPlan(server.api, launchEnv, async () => assert.fail('no mutation marker'), await fence()), /chat_testops_job_mismatch/);
assert.equal(server.calls.some(call => call.method), false);
}
});
test('bootstrap job predicate failures retain only distinct fixed codes and no observed values', async () => {
for (const [job, code] of [
[{ id: 99 }, 'chat_testops_job_id_mismatch'],
[{ type: 'SYNTHETIC_PRIVATE_TYPE' }, 'chat_testops_job_type_mismatch'],
[{ parameters: undefined }, 'chat_testops_job_parameters_shape_mismatch'],
[{ parameters: null }, 'chat_testops_job_parameters_shape_mismatch'],
[{ parameters: 'SYNTHETIC_PRIVATE_PARAMETERS' }, 'chat_testops_job_parameters_shape_mismatch'],
[{ parameters: Array.from({ length: 5 }, (_, i) => ({ id: i + 1 })) }, 'chat_testops_job_parameters_limit'],
[{ url: 'https://SYNTHETIC_PRIVATE_URL.invalid/' }, 'chat_testops_job_url_mismatch'],
]) {
await assert.rejects(bootstrapJobMetadata(strictBootstrapApi({ job }), publicEnv, 10), error => {
const projected = failureProjection(error, 'bootstrap_context');
assert.deepEqual(projected, { stage: 'bootstrap_context', code, httpStatus: null });
assert.ok(!JSON.stringify(projected).includes('SYNTHETIC_PRIVATE'));
return true;
});
}
});
test('launch preserves exact Branch and defaults evaluate only when operation is genuinely absent', async () => {
for (const [parameters, opId, expectedParameters] of [
[[{ id: 19, value: 'refs/heads/master' }], undefined, [{ id: 19, value: 'refs/heads/master' }]],
[[{ id: 19, value: 'refs/heads/master' }], '', [{ id: 19, value: 'refs/heads/master' }]],
[[{ id: 17, value: 'bootstrap' }], '17', [{ id: 17, value: 'evaluate' }]],
[[{ id: 19, value: 'refs/heads/master' }, { id: 17, value: 'bootstrap' }], '17', [{ id: 19, value: 'refs/heads/master' }, { id: 17, value: 'evaluate' }]],
]) {
const env = { ...launchEnv, CHAT_TESTOPS_OPERATION_PARAMETER_ID: opId };
const server = launchServer({ initialParameters: parameters, expectedParameters });
let marked = 0;
const result = await launchChatPlan(server.api, env, async () => { marked++; assert.equal(server.calls.some(call => call.method), false); }, await fence(env));
assert.equal(result.status, 'REQUESTED'); assert.equal(server.requests(), 1); assert.equal(marked, 1);
assert.deepEqual(server.calls.filter(call => call.route === '/api/job/7' && call.method === 'PATCH')[0].body.parameters, expectedParameters);
assert.deepEqual(server.calls.filter(call => call.route === '/api/testplan/8/jobparameter')[0].body.parameters, expectedParameters);
}
});
test('launch refuses empty unknown duplicate wrong Branch or conflicting operation before any mutation', async () => {
const branch = { id: 19, value: 'refs/heads/master' };
for (const [parameters, opId, definitions] of [
[[], undefined, {}],
[[{ id: 19, value: 'refs/heads/other' }], undefined, {}],
[[branch, branch], undefined, {}],
[[branch, { id: 20, value: 'refs/heads/master' }], undefined, {}],
[[{ id: 19, value: 'refs/heads/master' }], undefined, { 19: { id: 99, name: 'Branch' } }],
[[{ id: 19, value: 'PRIVATE_PARAMETER_VALUE' }], undefined, { 19: { id: 19, name: 'PRIVATE_PARAMETER_NAME' } }],
[[branch], '17', {}],
[[branch], 'bad-id', {}],
[[{ id: 17, value: 'bootstrap' }], undefined, {}],
[[{ id: 17, value: 'bootstrap' }], '18', {}],
[[{ id: 17, value: 'launch-plan' }], '17', {}],
[[branch, { id: 17, value: 'bootstrap' }, { id: 20, value: 'refs/heads/master' }], '17', {}],
]) {
const env = { ...launchEnv, CHAT_TESTOPS_OPERATION_PARAMETER_ID: opId };
const server = launchServer({ initialParameters: parameters, definitions });
await assert.rejects(launchChatPlan(server.api, env, async () => assert.fail('no mutation marker'), await fence(env)), /chat_testops_(?:identity_missing|parameter_mismatch)/);
assert.equal(server.calls.some(call => call.method), false); assert.equal(server.requests(), 0);
}
});
test('bootstrap projects only fixed Branch name and exact master value alongside genuine numeric ID', async () => {
const base = strictBootstrapApi({ job: { parameters: [{ id: 19, value: 'refs/heads/master' }] } });
const result = await bootstrapJobMetadata(route => route === '/api/environment/19' ? { id: 19, name: 'Branch' } : base(route), publicEnv, 10);
assert.deepEqual(result.parameters, [{ id: 19, name: 'Branch', value: 'refs/heads/master' }]);
const hidden = strictBootstrapApi({ job: { parameters: [{ id: 19, value: 'PRIVATE_BRANCH_VALUE' }] } });
const unknown = await bootstrapJobMetadata(route => route === '/api/environment/19' ? { id: 19, name: 'Branch' } : hidden(route), publicEnv, 10);
assert.deepEqual(unknown.parameters, [{ id: 19, name: 'Branch', value: 'NOT_PROJECTED' }]);
assert.ok(!JSON.stringify(unknown).includes('PRIVATE_BRANCH'));
});