mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 22:24:16 +01:00
fix(evals): support official TestOps job metadata (#18385)
## Thinking Path > - Paperclip manages AI agents and verifies real work through Product E2E tests. > - The fixed Chat feature workflow uses TestOps to select and report one backlog case. > - Official registration exposes a workflow-query URL and a Branch parameter. > - The existing reader accepts only workflow-file or numeric URLs, and the admin launcher requires an operation parameter. > - These assumptions reject valid official metadata before the feature can run. > - This pull request supports the exact observed form while preserving source, selection, and once-only execution checks. ## Linked Issues or Issue Description Refs: #18322, #18375. **What happened?** The second zero-model bootstrap stopped with a generic job-mismatch code. Ordinary TestOps UI inspection then showed an official workflow-query URL outside the accepted literals and a Branch-only job. The exact first failed predicate in that bootstrap remains unknown. **Expected behavior** Accept that exact fixed-workflow URL. Preserve the existing master Branch value when the job has no operation parameter. Keep rejecting foreign URLs, unknown parameters, wrong branches, and conflicting operation IDs before mutation. **Steps to reproduce** Pass the official query URL and Branch-only metadata through the existing bootstrap, selection, and admin-launch owners. The new pure controls exercise those actual helpers without a CLI, backend, or model call. **Paperclip version or commit** This change starts from `0e15c92073a09e50df99190a37f69a86286b989b` after #18375. **Deployment mode** GitHub Actions and the protected TestOps reporting environment. ## What Changed - Share one exact URL predicate across bootstrap, evaluation selection, and admin preparation. Add only the observed official workflow-query literal. - Support Branch-only, operation-only, and Branch-plus-operation jobs with strict definition IDs, names, values, and duplicate checks. - Preserve the exact master Branch value. Set an existing operation parameter to `evaluate`; otherwise use the fixed workflow's existing default. - Use the same checked parameter vector for job and plan writes and their readbacks. - Report distinct fixed bootstrap predicate codes without response values. - Document when the protected operation-parameter ID is required. - Align one existing Cloud-maintenance test with the inherited managed-Cloud S3 policy. Preserve the same-version local-storage proof in the parent, use valid synthetic Cloud settings in the child, and check the expected provider mismatch without S3 requests. Retain the no-seed and unchanged persisted-row assertions. ## Verification - URL controls: three failures on the parent; all pass with the correction. - Branch controls: two failures and one correct rejection on the predecessor; all pass with the correction. - `node --test tests/runner-e2e/chat-feature-testops-api.test.mjs`: 40 owning pure/mock controls pass on Node 26.4.0. - Two MJS syntax checks and `git diff --check` pass. TypeScript noEmit was not run for this MJS/documentation change. - The first [normal CI run](https://github.com/paperclipai/paperclip/actions/runs/41131094974) failed only in the managed-skill Cloud fixture and its dependent verification gate: 46 jobs passed, two failed, and one skipped. The tested merge included a new upstream policy that rejects the test's local-disk Cloud configuration. The exact caught runtime exception remains unknown. - Three pure controls using the actual policy validator pass: the old invalid Cloud fixture rejects, the new synthetic S3 fixture admits, and an invalid master key rejects. An initial tsx wrapper IPC permission error executed zero controls and remains a separate setup failure. - The PostgreSQL/maintenance-child test was not run locally. On new-head AWS CI, `managed-skill-content.test.ts` passed all 17 tests in server shard 9. - [Normal CI](https://github.com/paperclipai/paperclip/actions/runs/41133876636) is green on exact head `b76d5dd40d7711f1b5f6b6316b1114c0aae02d57`: 48 jobs passed and one skipped. Both required `ci / e2e` and `ci / verify` pass. - Greptile reviewed this exact head at 5/5 with no actionable findings or unresolved review threads. - No actual CLI, backend, model, provider, app, database, build, or bootstrap was run to validate this correction locally. ## Risks The exact original bootstrap predicate is still unknown. This fixes a demonstrated metadata false negative and preserves the prior failed run. Actual backend parameter types and integration dispatch still need verification. Unknown metadata stops before the once marker and writes. This PR changes no workflow, credential, case, model, grader, clock, or private-evidence policy. It does not claim a passing feature result. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, code editing, tool use, and pure test review. The runtime does not expose the exact context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
198218bf55
commit
8595efc20b
4 files changed
+163
-28
No files matched your search
@@ -79,7 +79,9 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
|
||||
|
||||
it("loads persisted Cloud configuration for read-only maintenance without seeding it", async () => {
|
||||
const companyId = await company();
|
||||
await companySkillService(db).createLocalSkill(companyId, { name: "Cloud CLI", markdown: "# Cloud CLI" });
|
||||
const skill = await companySkillService(db).createLocalSkill(companyId, { name: "Cloud CLI", markdown: "# Cloud CLI" });
|
||||
const local = await skillInstallationMaintenance(db).verifyVersions({ companyId, limit: 50 });
|
||||
expect(local.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: "local_storage" })]);
|
||||
const key = "cloud-runtime-config:v1";
|
||||
const run = (phase: string) => promisify(execFile)(process.execPath, [
|
||||
"--import", fileURLToPath(import.meta.resolve("tsx")),
|
||||
@@ -87,16 +89,18 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
|
||||
`--phase=${phase}`, `--company=${companyId}`, "--limit=50",
|
||||
], { env: { ...process.env, DATABASE_URL: database.connectionString,
|
||||
PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "disposable-cloud-cli-fixture",
|
||||
PAPERCLIP_STORAGE_PROVIDER: "local_disk", PAPERCLIP_STORAGE_LOCAL_DIR: path.join(root, "objects") },
|
||||
PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_STORAGE_S3_BUCKET: "managed-skill-cli-fixture",
|
||||
PAPERCLIP_STORAGE_S3_REGION: "us-east-1", PAPERCLIP_SECRETS_MASTER_KEY: "12345678901234567890123456789012" },
|
||||
timeout: 60_000, maxBuffer: 1024 * 1024 });
|
||||
await expect(run("eligibility")).rejects.toThrow("Initialize Cloud runtime configuration");
|
||||
expect(await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, key))).toHaveLength(0);
|
||||
await db.insert(instanceSettings).values({ singletonKey: key, general: { version: 1, revision: 0, config: null } });
|
||||
const [persisted] = await db.insert(instanceSettings).values({ singletonKey: key, general: { version: 1, revision: 0, config: null } }).returning();
|
||||
try {
|
||||
const eligibility = JSON.parse((await run("eligibility")).stdout.trim().split("\n").at(-1)!);
|
||||
expect(eligibility.results).toEqual([expect.objectContaining({ status: "version_backed" })]);
|
||||
expect(eligibility.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: "version_backed" })]);
|
||||
const verification = JSON.parse((await run("verification")).stdout.trim().split("\n").at(-1)!);
|
||||
expect(verification.results).toEqual([expect.objectContaining({ status: "local_storage" })]);
|
||||
expect(verification.results).toEqual([expect.objectContaining({ versionId: skill.currentVersionId, status: "provider_mismatch" })]);
|
||||
expect(await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, key))).toEqual([persisted]);
|
||||
} finally {
|
||||
await db.delete(instanceSettings).where(eq(instanceSettings.singletonKey, key));
|
||||
}
|
||||
|
||||
@@ -223,8 +223,12 @@ actors before any TestOps mutation. A later dispatch or replay cannot reuse that
|
||||
approved number. Missing or uncertain metadata stops; an uncertain request needs
|
||||
reconciliation rather than a new send. The nonce and launch search provide
|
||||
additional evidence. Normal TestOps **Run** uses `evaluate` and is independent of
|
||||
this one-time administrator fence. The operation parameter identifier and launch
|
||||
nonce are also protected configuration, not dispatch inputs.
|
||||
this one-time administrator fence. The launch nonce is protected configuration,
|
||||
not a dispatch input. `CHAT_TESTOPS_OPERATION_PARAMETER_ID` is required only when
|
||||
the registered job defines `operation`; the launcher sets that parameter to
|
||||
`evaluate`. An official job with only `Branch` keeps its exact
|
||||
`refs/heads/master` value and uses the fixed workflow's `evaluate` default.
|
||||
Unknown parameters and other branch values are rejected.
|
||||
|
||||
`CHAT_TESTOPS_INTEGRATION_ID` names the reviewed existing project GitHub
|
||||
integration. The launcher verifies its project, type and enabled state, then binds
|
||||
|
||||
@@ -17,7 +17,8 @@ const diagnosticCodes = new Set(['untrusted_chat_testops_context', 'testops_toke
|
||||
'testops_created_case_identity_invalid', 'testops_created_plan_identity_invalid', 'testops_plan_scope_mismatch',
|
||||
'testops_operation_invalid', 'testops_output_invalid', 'untrusted_public_testops_context', 'chat_testops_identity_missing',
|
||||
'chat_testops_case_mismatch', 'chat_testops_plan_mismatch', 'chat_testops_plan_job_mismatch', 'chat_testops_job_mismatch',
|
||||
'chat_testops_jobrun_mismatch', 'chat_testops_launch_mismatch', 'chat_testops_selection_mismatch', 'chat_testops_parameter_mismatch', 'chat_testops_nonce_invalid', 'chat_testops_nonce_conflict', 'chat_testops_catalog_incomplete', 'chat_testops_launch_admission', 'chat_testops_github_http', 'chat_testops_integration_mismatch', 'chat_testops_upload_mismatch', 'chat_testops_upload_unverified']);
|
||||
'chat_testops_job_id_mismatch', 'chat_testops_job_type_mismatch', 'chat_testops_job_parameters_shape_mismatch',
|
||||
'chat_testops_job_parameters_limit', 'chat_testops_job_url_mismatch', 'chat_testops_jobrun_mismatch', 'chat_testops_launch_mismatch', 'chat_testops_selection_mismatch', 'chat_testops_parameter_mismatch', 'chat_testops_nonce_invalid', 'chat_testops_nonce_conflict', 'chat_testops_catalog_incomplete', 'chat_testops_launch_admission', 'chat_testops_github_http', 'chat_testops_integration_mismatch', 'chat_testops_upload_mismatch', 'chat_testops_upload_unverified']);
|
||||
export function failureProjection(error, stage) {
|
||||
const stages = ['context', 'output_admission', 'authentication', 'metadata_inspection', 'plan_bootstrap', 'selection_validation', 'launch_admission', 'launch_preflight', 'plan_launch', 'bootstrap_job_metadata', 'result_export', 'upload_verification', 'bootstrap_context', 'receipt'];
|
||||
return { stage: stages.includes(stage) ? stage : 'unknown',
|
||||
@@ -146,9 +147,7 @@ export async function validateChatSelection(api, env) {
|
||||
const assignments = await api(`/api/testplan/${planId}/job`);
|
||||
if (!Array.isArray(assignments) || assignments.length !== 1 || assignments[0].job?.id !== jobId || assignments[0].testCasesCount !== 1) fail('chat_testops_plan_job_mismatch');
|
||||
const job = await api(`/api/job/${jobId}`);
|
||||
const jobUrls = ['https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml'];
|
||||
if (positive(env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID)) jobUrls.push(`https://github.com/paperclipai/paperclip/actions/workflows/${env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID}`);
|
||||
if (job.id !== jobId || job.type !== 'github' || !jobUrls.includes(job.url)) fail('chat_testops_job_mismatch');
|
||||
if (job.id !== jobId || job.type !== 'github' || !publicJobUrl(job.url, env)) fail('chat_testops_job_mismatch');
|
||||
const run = await api(`/api/jobrun/${jobRunId}`);
|
||||
if (run.id !== jobRunId || run.job?.id !== jobId || run.job?.type !== 'github'
|
||||
|| !['scheduled', 'in_progress'].includes(run.stage) || !id(run.launchId)
|
||||
@@ -162,12 +161,16 @@ export async function validateChatSelection(api, env) {
|
||||
}
|
||||
|
||||
function configuredIds(env) {
|
||||
const names = ['CHAT_TESTOPS_CASE_ID', 'CHAT_TESTOPS_PLAN_ID', 'CHAT_TESTOPS_JOB_ID', 'CHAT_TESTOPS_OPERATION_PARAMETER_ID', 'CHAT_TESTOPS_INTEGRATION_ID'];
|
||||
const names = ['CHAT_TESTOPS_CASE_ID', 'CHAT_TESTOPS_PLAN_ID', 'CHAT_TESTOPS_JOB_ID', 'CHAT_TESTOPS_INTEGRATION_ID'];
|
||||
if (names.some(name => !positive(env[name]))) fail('chat_testops_identity_missing');
|
||||
return Object.fromEntries(names.map((name, index) => [['caseId', 'planId', 'jobId', 'parameterId', 'integrationId'][index], Number(env[name])]));
|
||||
const operation = env.CHAT_TESTOPS_OPERATION_PARAMETER_ID;
|
||||
if (operation !== undefined && operation !== '' && !positive(operation)) fail('chat_testops_identity_missing');
|
||||
return { ...Object.fromEntries(names.map((name, index) => [['caseId', 'planId', 'jobId', 'integrationId'][index], Number(env[name])])),
|
||||
parameterId: positive(operation) ? Number(operation) : null };
|
||||
}
|
||||
function publicJobUrl(url, env) {
|
||||
return url === 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml'
|
||||
|| url === 'https://github.com/paperclipai/paperclip/actions?query=workflow%3A%22Chat+feature+TestOps%22'
|
||||
|| positive(env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID) && url === `https://github.com/paperclipai/paperclip/actions/workflows/${env.CHAT_TESTOPS_GITHUB_WORKFLOW_ID}`;
|
||||
}
|
||||
async function registeredPlan(api, env) {
|
||||
@@ -182,10 +185,24 @@ async function registeredPlan(api, env) {
|
||||
if (job.id !== ids.jobId || job.type !== 'github' || !publicJobUrl(job.url, env)) fail('chat_testops_job_mismatch');
|
||||
const integration = await api(`/api/integration/${ids.integrationId}/project/1`);
|
||||
if (integration.id !== ids.integrationId || integration.projectId !== 1 || integration.info?.type !== 'github' || integration.disabled !== false) fail('chat_testops_integration_mismatch');
|
||||
const definition = await api(`/api/environment/${ids.parameterId}`);
|
||||
if (definition.id !== ids.parameterId || definition.name !== 'operation' || !Array.isArray(job.parameters) || job.parameters.length !== 1
|
||||
|| job.parameters[0].id !== ids.parameterId || !['bootstrap', 'evaluate'].includes(job.parameters[0].value)) fail('chat_testops_parameter_mismatch');
|
||||
return { ...ids, selected, plan, job };
|
||||
if (!Array.isArray(job.parameters) || job.parameters.length < 1 || job.parameters.length > 2) fail('chat_testops_parameter_mismatch');
|
||||
const seenIds = new Set(), seenNames = new Set(), parameters = [];
|
||||
for (const parameter of job.parameters) {
|
||||
if (!id(parameter.id) || seenIds.has(parameter.id)) fail('chat_testops_parameter_mismatch');
|
||||
seenIds.add(parameter.id);
|
||||
const definition = await api(`/api/environment/${parameter.id}`);
|
||||
if (definition.id !== parameter.id || !['Branch', 'operation'].includes(definition.name) || seenNames.has(definition.name)) fail('chat_testops_parameter_mismatch');
|
||||
seenNames.add(definition.name);
|
||||
if (definition.name === 'Branch') {
|
||||
if (parameter.value !== 'refs/heads/master') fail('chat_testops_parameter_mismatch');
|
||||
parameters.push({ id: parameter.id, value: parameter.value });
|
||||
} else {
|
||||
if (ids.parameterId !== parameter.id || !['bootstrap', 'evaluate'].includes(parameter.value)) fail('chat_testops_parameter_mismatch');
|
||||
parameters.push({ id: parameter.id, value: 'evaluate' });
|
||||
}
|
||||
}
|
||||
if (ids.parameterId !== null && !seenNames.has('operation')) fail('chat_testops_parameter_mismatch');
|
||||
return { ...ids, selected, plan, job, parameters };
|
||||
}
|
||||
|
||||
export async function bootstrapJobMetadata(api, env, expectedLaunchId) {
|
||||
@@ -205,8 +222,11 @@ export async function bootstrapJobMetadata(api, env, expectedLaunchId) {
|
||||
if (launch.id !== expectedLaunchId || launch.projectId !== 1) fail('chat_testops_launch_mismatch');
|
||||
}
|
||||
const job = await api(`/api/job/${run.job.id}`);
|
||||
if (job.id !== run.job.id || job.type !== 'github' || !Array.isArray(job.parameters) || job.parameters.length > 4) fail('chat_testops_job_mismatch');
|
||||
if (strict && !publicJobUrl(job.url, env)) fail('chat_testops_job_mismatch');
|
||||
if (job.id !== run.job.id) fail('chat_testops_job_id_mismatch');
|
||||
if (job.type !== 'github') fail('chat_testops_job_type_mismatch');
|
||||
if (!Array.isArray(job.parameters)) fail('chat_testops_job_parameters_shape_mismatch');
|
||||
if (job.parameters.length > 4) fail('chat_testops_job_parameters_limit');
|
||||
if (strict && !publicJobUrl(job.url, env)) fail('chat_testops_job_url_mismatch');
|
||||
const parameters = [];
|
||||
const seen = new Set();
|
||||
for (const parameter of job.parameters) {
|
||||
@@ -214,9 +234,10 @@ export async function bootstrapJobMetadata(api, env, expectedLaunchId) {
|
||||
seen.add(parameter.id);
|
||||
const definition = await api(`/api/environment/${parameter.id}`);
|
||||
if (strict && definition.id !== parameter.id) fail('chat_testops_parameter_mismatch');
|
||||
const known = ['operation', 'ALLURE_JOB_RUN_ID', 'ALLURE_USERNAME'].includes(definition.name);
|
||||
const known = ['operation', 'ALLURE_JOB_RUN_ID', 'ALLURE_USERNAME', 'Branch'].includes(definition.name);
|
||||
parameters.push({ id: parameter.id, name: known ? definition.name : 'UNKNOWN',
|
||||
value: definition.name === 'operation' && ['bootstrap', 'evaluate', 'launch-plan'].includes(parameter.value) ? parameter.value : 'NOT_PROJECTED' });
|
||||
value: definition.name === 'operation' && ['bootstrap', 'evaluate', 'launch-plan'].includes(parameter.value)
|
||||
|| definition.name === 'Branch' && parameter.value === 'refs/heads/master' ? parameter.value : 'NOT_PROJECTED' });
|
||||
}
|
||||
return { ...(strict ? { schema: 'paperclip.chat-feature-testops-bootstrap-registration/v1', ...context, capturedAfterWatch: true, runnableBindingQualified: false } : {}),
|
||||
status: 'observed', projectId: 1, jobId: job.id, allureJobRunId: env.ALLURE_JOB_RUN_ID,
|
||||
@@ -274,7 +295,7 @@ export async function launchChatPlan(api, env, markOnce, admission) {
|
||||
jobId: known.jobId, caseId: known.caseId, launchId: existing[0].id, status: 'EXISTING_NO_RESEND', actualResult: null };
|
||||
}
|
||||
await markOnce({ ...source, launchAdmission: admission, nonce, planId: known.planId, jobId: known.jobId, caseId: known.caseId });
|
||||
const parameters = [{ id: known.parameterId, value: 'evaluate' }];
|
||||
const parameters = known.parameters;
|
||||
await api(`/api/job/${known.jobId}`, 'PATCH', { integrationId: known.integrationId, canRun: true, parameters });
|
||||
const patched = await api(`/api/job/${known.jobId}`);
|
||||
if (patched.id !== known.jobId || patched.type !== 'github' || !publicJobUrl(patched.url, env) || !same(patched.parameters, parameters)) fail('chat_testops_parameter_mismatch');
|
||||
|
||||
@@ -168,21 +168,24 @@ async function fence(env = launchEnv, changes = {}, workflowChanges = {}, repoCh
|
||||
: url.includes('/actions/runs/') ? { ...ownRun, ...changes }
|
||||
: { id: 99, path: '.github/workflows/chat-feature-testops.yml', state: 'active', ...workflowChanges }) }));
|
||||
}
|
||||
function launchServer({ wrongParameter = false, wrongAssignment = false, uncertainRun = false, integrationChange = {} } = {}) {
|
||||
const calls = [], parameters = [{ id: 17, value: 'bootstrap' }]; let assigned = false, launched = false, requests = 0;
|
||||
function launchServer({ wrongParameter = false, wrongAssignment = false, uncertainRun = false, integrationChange = {}, initialParameters = [{ id: 17, value: 'bootstrap' }], expectedParameters = [{ id: 17, value: 'evaluate' }], definitions = {}, jobUrl = 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml' } = {}) {
|
||||
const calls = [], parameters = structuredClone(initialParameters); let assigned = false, launched = false, requests = 0;
|
||||
const api = async (route, method, body) => {
|
||||
calls.push({ route, method, body });
|
||||
if (route.startsWith('/api/launch?')) return { totalPages: 1, content: launched ? [{ id: 10, name: `Chat feature evals · ${'1'.repeat(32)}` }] : [] };
|
||||
if (route === '/api/integration/2/project/1') return { id: 2, projectId: 1, info: { type: 'github' }, disabled: false, ...integrationChange };
|
||||
if (route === '/api/environment/17') return { id: 17, name: wrongParameter ? 'OTHER_PRIVATE_PARAMETER' : 'operation' };
|
||||
if (route.startsWith('/api/environment/')) {
|
||||
const id = Number(route.split('/').at(-1));
|
||||
return definitions[id] ?? { id, name: id === 17 ? wrongParameter ? 'OTHER_PRIVATE_PARAMETER' : 'operation' : 'Branch' };
|
||||
}
|
||||
if (route === '/api/job/7') {
|
||||
if (method === 'PATCH') { assert.deepEqual(body, { integrationId: 2, canRun: true, parameters: [{ id: 17, value: 'evaluate' }] }); parameters[0].value = 'evaluate'; }
|
||||
return { id: 7, type: 'github', url: 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml', parameters };
|
||||
if (method === 'PATCH') { assert.deepEqual(body, { integrationId: 2, canRun: true, parameters: expectedParameters }); parameters.splice(0, parameters.length, ...structuredClone(body.parameters)); }
|
||||
return { id: 7, type: 'github', url: jobUrl, parameters };
|
||||
}
|
||||
if (route === '/api/testplan/8/job') return wrongAssignment ? [{ job: { id: 70 }, testCasesCount: 1 }]
|
||||
: assigned ? [{ job: { id: 7, parameters }, testCasesCount: 1 }] : [];
|
||||
if (route === '/api/testplan/8/assign') { assert.equal(method, 'POST'); assert.deepEqual(body, { jobId: 7, selection: { inverted: false, manual: false, leafsInclude: [91] } }); assigned = true; return null; }
|
||||
if (route === '/api/testplan/8/jobparameter') { assert.deepEqual(body, { jobId: 7, parameters: [{ id: 17, value: 'evaluate' }] }); return null; }
|
||||
if (route === '/api/testplan/8/jobparameter') { assert.deepEqual(body, { jobId: 7, parameters: expectedParameters }); return null; }
|
||||
if (route === '/api/testplan/8/run') { requests++; assert.equal(method, 'POST'); if (uncertainRun) throw new Error('synthetic uncertain transport'); launched = true; return { id: 10, projectId: 1, name: body.launchName }; }
|
||||
return selectedApi()(route);
|
||||
};
|
||||
@@ -432,3 +435,106 @@ test('post-watch reader caps parameter definitions and refuses duplicate IDs', a
|
||||
await assert.rejects(bootstrapJobMetadata(strictBootstrapApi({ job: { parameters } }), publicEnv, 10));
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
const observedLegacyJobUrl = 'https://github.com/paperclipai/paperclip/actions?query=workflow%3A%22Chat+feature+TestOps%22';
|
||||
|
||||
test('official observed legacy and existing workflow URLs pass all three actual job consumers', async () => {
|
||||
for (const url of [observedLegacyJobUrl, 'https://github.com/paperclipai/paperclip/actions/workflows/chat-feature-testops.yml',
|
||||
'https://github.com/paperclipai/paperclip/actions/workflows/99']) {
|
||||
const env = { ...publicEnv, CHAT_TESTOPS_GITHUB_WORKFLOW_ID: '99' };
|
||||
const selected = await validateChatSelection(selectedApi({ '/api/job/7': { id: 7, type: 'github', url } }), env);
|
||||
assert.equal(selected.caseId, 91);
|
||||
const registered = await bootstrapJobMetadata(strictBootstrapApi({ job: { url } }), env, 10);
|
||||
assert.equal(registered.publicWorkflowMatches, true);
|
||||
const server = launchServer({ jobUrl: url });
|
||||
const launched = await launchChatPlan(server.api, launchEnv, async () => {}, await fence());
|
||||
assert.equal(launched.status, 'REQUESTED');
|
||||
assert.equal(server.requests(), 1);
|
||||
}
|
||||
});
|
||||
|
||||
test('official legacy URL admission rejects foreign workflow repo query credentials and fragment in all consumers', async () => {
|
||||
for (const url of [observedLegacyJobUrl.replace('/paperclip/actions', '/paperclip-cloud/actions'),
|
||||
observedLegacyJobUrl.replace('Chat+feature+TestOps', 'Other+workflow'), `${observedLegacyJobUrl}&extra=1`,
|
||||
observedLegacyJobUrl.replace('https://github.com', 'https://user:password@github.com'), `${observedLegacyJobUrl}#extra`]) {
|
||||
await assert.rejects(validateChatSelection(selectedApi({ '/api/job/7': { id: 7, type: 'github', url } }), publicEnv), /chat_testops_job_mismatch/);
|
||||
await assert.rejects(bootstrapJobMetadata(strictBootstrapApi({ job: { url } }), publicEnv, 10), error => {
|
||||
assert.deepEqual(failureProjection(error, 'bootstrap_context'), { stage: 'bootstrap_context', code: 'chat_testops_job_url_mismatch', httpStatus: null });
|
||||
return true;
|
||||
});
|
||||
const server = launchServer({ jobUrl: url });
|
||||
await assert.rejects(launchChatPlan(server.api, launchEnv, async () => assert.fail('no mutation marker'), await fence()), /chat_testops_job_mismatch/);
|
||||
assert.equal(server.calls.some(call => call.method), false);
|
||||
}
|
||||
});
|
||||
|
||||
test('bootstrap job predicate failures retain only distinct fixed codes and no observed values', async () => {
|
||||
for (const [job, code] of [
|
||||
[{ id: 99 }, 'chat_testops_job_id_mismatch'],
|
||||
[{ type: 'SYNTHETIC_PRIVATE_TYPE' }, 'chat_testops_job_type_mismatch'],
|
||||
[{ parameters: undefined }, 'chat_testops_job_parameters_shape_mismatch'],
|
||||
[{ parameters: null }, 'chat_testops_job_parameters_shape_mismatch'],
|
||||
[{ parameters: 'SYNTHETIC_PRIVATE_PARAMETERS' }, 'chat_testops_job_parameters_shape_mismatch'],
|
||||
[{ parameters: Array.from({ length: 5 }, (_, i) => ({ id: i + 1 })) }, 'chat_testops_job_parameters_limit'],
|
||||
[{ url: 'https://SYNTHETIC_PRIVATE_URL.invalid/' }, 'chat_testops_job_url_mismatch'],
|
||||
]) {
|
||||
await assert.rejects(bootstrapJobMetadata(strictBootstrapApi({ job }), publicEnv, 10), error => {
|
||||
const projected = failureProjection(error, 'bootstrap_context');
|
||||
assert.deepEqual(projected, { stage: 'bootstrap_context', code, httpStatus: null });
|
||||
assert.ok(!JSON.stringify(projected).includes('SYNTHETIC_PRIVATE'));
|
||||
return true;
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
test('launch preserves exact Branch and defaults evaluate only when operation is genuinely absent', async () => {
|
||||
for (const [parameters, opId, expectedParameters] of [
|
||||
[[{ id: 19, value: 'refs/heads/master' }], undefined, [{ id: 19, value: 'refs/heads/master' }]],
|
||||
[[{ id: 19, value: 'refs/heads/master' }], '', [{ id: 19, value: 'refs/heads/master' }]],
|
||||
[[{ id: 17, value: 'bootstrap' }], '17', [{ id: 17, value: 'evaluate' }]],
|
||||
[[{ id: 19, value: 'refs/heads/master' }, { id: 17, value: 'bootstrap' }], '17', [{ id: 19, value: 'refs/heads/master' }, { id: 17, value: 'evaluate' }]],
|
||||
]) {
|
||||
const env = { ...launchEnv, CHAT_TESTOPS_OPERATION_PARAMETER_ID: opId };
|
||||
const server = launchServer({ initialParameters: parameters, expectedParameters });
|
||||
let marked = 0;
|
||||
const result = await launchChatPlan(server.api, env, async () => { marked++; assert.equal(server.calls.some(call => call.method), false); }, await fence(env));
|
||||
assert.equal(result.status, 'REQUESTED'); assert.equal(server.requests(), 1); assert.equal(marked, 1);
|
||||
assert.deepEqual(server.calls.filter(call => call.route === '/api/job/7' && call.method === 'PATCH')[0].body.parameters, expectedParameters);
|
||||
assert.deepEqual(server.calls.filter(call => call.route === '/api/testplan/8/jobparameter')[0].body.parameters, expectedParameters);
|
||||
}
|
||||
});
|
||||
|
||||
test('launch refuses empty unknown duplicate wrong Branch or conflicting operation before any mutation', async () => {
|
||||
const branch = { id: 19, value: 'refs/heads/master' };
|
||||
for (const [parameters, opId, definitions] of [
|
||||
[[], undefined, {}],
|
||||
[[{ id: 19, value: 'refs/heads/other' }], undefined, {}],
|
||||
[[branch, branch], undefined, {}],
|
||||
[[branch, { id: 20, value: 'refs/heads/master' }], undefined, {}],
|
||||
[[{ id: 19, value: 'refs/heads/master' }], undefined, { 19: { id: 99, name: 'Branch' } }],
|
||||
[[{ id: 19, value: 'PRIVATE_PARAMETER_VALUE' }], undefined, { 19: { id: 19, name: 'PRIVATE_PARAMETER_NAME' } }],
|
||||
[[branch], '17', {}],
|
||||
[[branch], 'bad-id', {}],
|
||||
[[{ id: 17, value: 'bootstrap' }], undefined, {}],
|
||||
[[{ id: 17, value: 'bootstrap' }], '18', {}],
|
||||
[[{ id: 17, value: 'launch-plan' }], '17', {}],
|
||||
[[branch, { id: 17, value: 'bootstrap' }, { id: 20, value: 'refs/heads/master' }], '17', {}],
|
||||
]) {
|
||||
const env = { ...launchEnv, CHAT_TESTOPS_OPERATION_PARAMETER_ID: opId };
|
||||
const server = launchServer({ initialParameters: parameters, definitions });
|
||||
await assert.rejects(launchChatPlan(server.api, env, async () => assert.fail('no mutation marker'), await fence(env)), /chat_testops_(?:identity_missing|parameter_mismatch)/);
|
||||
assert.equal(server.calls.some(call => call.method), false); assert.equal(server.requests(), 0);
|
||||
}
|
||||
});
|
||||
|
||||
test('bootstrap projects only fixed Branch name and exact master value alongside genuine numeric ID', async () => {
|
||||
const base = strictBootstrapApi({ job: { parameters: [{ id: 19, value: 'refs/heads/master' }] } });
|
||||
const result = await bootstrapJobMetadata(route => route === '/api/environment/19' ? { id: 19, name: 'Branch' } : base(route), publicEnv, 10);
|
||||
assert.deepEqual(result.parameters, [{ id: 19, name: 'Branch', value: 'refs/heads/master' }]);
|
||||
const hidden = strictBootstrapApi({ job: { parameters: [{ id: 19, value: 'PRIVATE_BRANCH_VALUE' }] } });
|
||||
const unknown = await bootstrapJobMetadata(route => route === '/api/environment/19' ? { id: 19, name: 'Branch' } : hidden(route), publicEnv, 10);
|
||||
assert.deepEqual(unknown.parameters, [{ id: 19, name: 'Branch', value: 'NOT_PROJECTED' }]);
|
||||
assert.ok(!JSON.stringify(unknown).includes('PRIVATE_BRANCH'));
|
||||
});
|
||||
Reference in new issue
Block a user