From bacbab2bc1f5249a5d24f4cac0d13a883a2a3b4f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 31 Oct 2026 11:04:56 -0500 Subject: [PATCH] test(testops): retain safe verification failure predicates (#18470) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat feature evaluations send their original results to TestOps. > - A final reader checks the stored case, run, source, status, and attachments. > - Two real chat cases passed, but the final reader returned a generic mismatch. > - Later server responses passed the same reader, so they do not explain the earlier failure. > - This change records the exact failed predicate without recording field values. > - The next failure can be diagnosed in its actual caller context. ## Linked Issues or Issue Description Refs #18462, #18394, and #18322. The current base includes the independent browser-fixture fix in #18409. **What happened?** The post-upload reader returned `chat_testops_upload_mismatch`. Its closed artifact had no phase or field detail. The original evaluation and TestOps case passed. Later responses from the same five endpoints passed the unchanged reader. Those later interactive responses do not prove what the protected API caller received at failure time. An earlier CI run also exposed a flat-text browser-fixture mismatch. Current master independently fixes that assertion in #18409. This branch adopts that complete upstream test; the final PR diff contains only the two diagnostic files. **Expected behavior** Keep every report-reader acceptance check. Report a fixed phase, failed predicate names, match booleans, and field type names. Do not record response values, credentials, unknown URLs, or traces. **Steps to reproduce** 1. Supply a missing case ID, a conflicting job ID, or a wrong export field to the reader. 2. Project the failure with `failureProjection`. 3. The parent returns only the generic error code. This change also identifies the failed check and distinguishes a missing field from a value of the wrong type. **Paperclip version or commit** The latest retained failure used 47f81e850b14cc209a7cd5681accab860caefe55. The branch now includes current master `3ee4c5b6138da82d0b1f51aa010540561bf43c22`. The upstream API is unchanged. **Deployment mode** GitHub Actions and AWS, with TestOps reporting. ## What Changed - Add private diagnostics to the existing same-code failure throws. - Emit only fixed names, booleans, and closed type names through the existing failure projection. - Copy projected metadata so a caller cannot change the stored diagnostic. - Add failure-path, missing-versus-conflicting-field, and secret-redaction controls. ## Verification - Five new controls fail against the original parent module. A combined early-mismatch/null-entry control also fails against the first diagnostic implementation. Null-safe diagnostic traversal preserves that known failure. - `node --test tests/runner-e2e/chat-feature-testops-api.test.mjs`: 53 passed. - The existing CLI, reporter facade, and exporter tests: 22 passed. - Both changed MJS files pass `node --check`. `git diff --check` passes. - The earlier browser-fixture failure remains recorded. The owning test is now byte-identical to the independently shipped base fix in #18409. No browser-test or production-file delta remains in this PR. - Removing the private diagnostics and restoring the ten original throws reconstructs the entire parent API file byte for byte. Acceptance expressions, API order, deadlines, and successful output are unchanged. - All five retained current server responses still produce VERIFIED. Successful output matches the parent output byte for byte. These responses used interactive authentication at a later time; the original protected-token failure remains unexplained. - Local checks used Node 26.4. Normal CI [41213493964](https://github.com/paperclipai/paperclip/actions/runs/41213493964), attempt 1, passed on 78a40d6b679934b77da2fbaf81f6472d6d98ae19. Both required checks passed, including browser e2e. The adopted upstream browser shard passed. Greptile is 5/5 on this head, and all review threads are resolved. - Full local typecheck, build, app, database, native SDK, provider, and model runs were not run. The diagnostic slice uses pure tests, mocks, and file-system controls. - No public command or user-facing behavior changes. No documentation update is needed. ## Risks - A diagnostic must not expose a sensitive value. A private WeakMap, fixed field names, boolean checks, and closed type names limit the output. Tests include synthetic secret strings and mutation attempts. - Diagnostic checks mirror validation to keep the original guards byte for byte. Future validation edits must update the paired diagnostic and its targeted controls. This is a known maintenance tradeoff. - This patch does not explain or erase either historical reader failure. It changes no Product E2E grade, model, retry, source guard, API route, workflow, or deadline. - Live failure diagnostics remain unqualified until a later authorized run uses this code. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, source inspection, and code execution. The host does not expose the context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- tests/runner-e2e/chat-feature-testops-api.mjs | 113 ++++++++++++++++-- .../chat-feature-testops-api.test.mjs | 104 ++++++++++++++++ 2 files changed, 207 insertions(+), 10 deletions(-) diff --git a/tests/runner-e2e/chat-feature-testops-api.mjs b/tests/runner-e2e/chat-feature-testops-api.mjs index 9b2b1ddfc3f..8d9269ecea9 100644 --- a/tests/runner-e2e/chat-feature-testops-api.mjs +++ b/tests/runner-e2e/chat-feature-testops-api.mjs @@ -10,6 +10,37 @@ const fail = code => { throw new SetupFailure(match ? `${match[1]}_http` : code, match ? Number(match[2]) : null); }; const bootstrapRequestClasses = new Set(['JOB_RUN_GET', 'LAUNCH_GET', 'JOB_GET']); +const uploadFailureChecks = { + context: ['selected_run', 'selected_attempt', 'selected_workflow_sha', 'selected_target_sha', 'selected_jobrun', 'env_jobrun', 'selected_execution', + 'selected_scenario', 'selected_project', 'selected_full_name', 'plan_id', 'job_id', 'case_id', 'launch_id', 'export_case', + 'export_campaign', 'export_count', 'export_status', 'export_cleanup', 'passed_cleanup', 'attachment_array', 'attachment_count'], + attachment_contract: ['attachment_rows', 'provenance_count', 'record_count', 'passed_screenshot'], + jobrun: ['jobrun_id', 'job_id', 'job_type', 'launch_id', 'run_url', 'stage'], + launch: ['launch_id', 'project_id'], + result_page: ['content_array', 'content_limit', 'total_integer', 'total_matches_content', 'pages_integer', 'pages_range'], + result_row: ['result_id', 'case_id'], + result: ['result_id', 'project_id', 'launch_id', 'case_id', 'full_name', 'jobrun_id', 'jobrun_launch', 'jobrun_url', + 'embedded_job', 'links_array', 'known_links'], + result_status: ['status_supported', 'status_matches_export'], + attachment_page: ['content_array', 'content_limit', 'total_integer', 'total_matches_content', 'pages_integer', 'pages_range', + 'unique_ids', 'attachment_rows'], +}; +const uploadFailures = new WeakMap(); +function stopUpload(code, phase, readChecks) { + const error = new SetupFailure(code); + let values; + try { values = readChecks(); } catch {} + const names = uploadFailureChecks[phase].filter(name => Array.isArray(values?.[name]) && typeof values[name][0] === 'boolean'); + const checks = Object.fromEntries(names.map(name => [name, values[name][0]])); + const types = Object.fromEntries(names.filter(name => values[name].length === 2).map(name => { + const value = values[name][1], type = typeof value; + return [name, value === undefined ? 'missing' : value === null ? 'null' : Array.isArray(value) ? 'array' + : ['string', 'number', 'boolean', 'object'].includes(type) ? type : 'other']; + })); + const failedPredicates = Object.keys(checks).filter(name => !checks[name]); + uploadFailures.set(error, { phase, firstFailedPredicate: failedPredicates[0] ?? 'checks_unavailable', failedPredicates, checks, types }); + throw error; +} const diagnosticCodes = new Set(['untrusted_chat_testops_context', 'testops_token_missing', 'testops_auth_http', 'testops_api_http', 'testops_auth_invalid', 'testops_route_invalid', 'testops_response_too_large', 'testops_response_invalid', 'testops_metadata_page_incomplete', 'testops_case_identity_invalid', @@ -22,10 +53,13 @@ const diagnosticCodes = new Set(['untrusted_chat_testops_context', 'testops_toke 'chat_testops_job_parameters_limit', 'chat_testops_job_url_mismatch', 'chat_testops_jobrun_mismatch', 'chat_testops_launch_mismatch', 'chat_testops_selection_mismatch', 'chat_testops_parameter_mismatch', 'chat_testops_nonce_invalid', 'chat_testops_nonce_conflict', 'chat_testops_catalog_incomplete', 'chat_testops_launch_admission', 'chat_testops_github_http', 'chat_testops_integration_mismatch', 'chat_testops_upload_mismatch', 'chat_testops_upload_unverified']); export function failureProjection(error, stage) { const request = error instanceof SetupFailure ? error.request : undefined; + const verification = error instanceof SetupFailure ? uploadFailures.get(error) : undefined; const stages = ['context', 'output_admission', 'authentication', 'metadata_inspection', 'plan_bootstrap', 'selection_validation', 'launch_admission', 'launch_preflight', 'plan_launch', 'bootstrap_job_metadata', 'result_export', 'upload_verification', 'bootstrap_context', 'receipt']; return { stage: stages.includes(stage) ? stage : 'unknown', code: error instanceof SetupFailure && diagnosticCodes.has(error.code) ? error.code : 'unknown_failure', httpStatus: error instanceof SetupFailure && Number.isInteger(error.status) && error.status >= 100 && error.status < 600 ? error.status : null, + ...(verification ? { verification: { ...verification, failedPredicates: [...verification.failedPredicates], + checks: { ...verification.checks }, types: { ...verification.types } } } : {}), ...(request && bootstrapRequestClasses.has(request.class) && id(request.resourceId) && ['STARTED', 'SUCCEEDED', 'FAILED'].includes(request.state) ? { request: { class: request.class, resourceId: request.resourceId, state: request.state, httpStatus: Number.isInteger(request.httpStatus) && request.httpStatus >= 100 && request.httpStatus < 600 ? request.httpStatus : null } } : {}) }; @@ -345,14 +379,36 @@ export async function verifyChatUpload(api, env, selected, exported, { wait = ms || exported.caseId !== selected.caseId || exported.campaignId !== `gha-${context.runId}-1` || exported.count !== 1 || !['passed', 'failed'].includes(exported.status) || !['passed', 'failed', 'not_started'].includes(exported.cleanup) || exported.status === 'passed' && exported.cleanup !== 'passed' || !Array.isArray(exported.attachmentMetadata) - || exported.attachmentMetadata.length < 2 || exported.attachmentMetadata.length > 10) fail('chat_testops_upload_mismatch'); + || exported.attachmentMetadata.length < 2 || exported.attachmentMetadata.length > 10) stopUpload('chat_testops_upload_mismatch', 'context', () => ({ + selected_run: [selected.runId === context.runId, selected.runId], selected_attempt: [selected.attempt === '1', selected.attempt], + selected_workflow_sha: [selected.workflowSha === context.workflowSha, selected.workflowSha], + selected_target_sha: [selected.targetSha === context.targetSha, selected.targetSha], + selected_jobrun: [selected.allureJobRunId === env.ALLURE_JOB_RUN_ID, selected.allureJobRunId], + env_jobrun: [selected.allureJobRunId === env.ALLURE_JOB_RUN_ID, env.ALLURE_JOB_RUN_ID], + selected_execution: [selected.executionId === executionId, selected.executionId], selected_scenario: [selected.canonicalScenario === scenario, selected.canonicalScenario], + selected_project: [selected.projectId === 1, selected.projectId], selected_full_name: [aliases.includes(selected.fullName), selected.fullName], + plan_id: [id(selected.planId), selected.planId], job_id: [id(selected.jobId), selected.jobId], + case_id: [id(selected.caseId), selected.caseId], launch_id: [id(selected.launchId), selected.launchId], + export_case: [exported.caseId === selected.caseId, exported.caseId], export_campaign: [exported.campaignId === `gha-${context.runId}-1`, exported.campaignId], + export_count: [exported.count === 1, exported.count], export_status: [['passed', 'failed'].includes(exported.status), exported.status], + export_cleanup: [['passed', 'failed', 'not_started'].includes(exported.cleanup), exported.cleanup], + passed_cleanup: [!(exported.status === 'passed' && exported.cleanup !== 'passed')], + attachment_array: [Array.isArray(exported.attachmentMetadata), exported.attachmentMetadata], + attachment_count: [Array.isArray(exported.attachmentMetadata) && exported.attachmentMetadata.length >= 2 && exported.attachmentMetadata.length <= 10], + })); const allowedAttachments = new Map([['Synthetic task and source conversation', 'image/png'], ['Execution provenance and checks', 'application/json'], ['Sanitized Product E2E record and accounting', 'application/json']]); if (exported.attachmentMetadata.some(row => allowedAttachments.get(row.name) !== row.contentType || !Number.isSafeInteger(row.contentLength) || row.contentLength <= 0 || row.contentLength > 12 * 1024 * 1024) || exported.attachmentMetadata.filter(row => row.name === 'Execution provenance and checks').length !== 1 || exported.attachmentMetadata.filter(row => row.name === 'Sanitized Product E2E record and accounting').length !== 1 - || exported.status === 'passed' && !exported.attachmentMetadata.some(row => row.contentType === 'image/png')) fail('chat_testops_upload_mismatch'); + || exported.status === 'passed' && !exported.attachmentMetadata.some(row => row.contentType === 'image/png')) stopUpload('chat_testops_upload_mismatch', 'attachment_contract', () => ({ + attachment_rows: [!exported.attachmentMetadata.some(row => allowedAttachments.get(row?.name) !== row?.contentType + || !Number.isSafeInteger(row?.contentLength) || row?.contentLength <= 0 || row?.contentLength > 12 * 1024 * 1024)], + provenance_count: [exported.attachmentMetadata.filter(row => row?.name === 'Execution provenance and checks').length === 1], + record_count: [exported.attachmentMetadata.filter(row => row?.name === 'Sanitized Product E2E record and accounting').length === 1], + passed_screenshot: [!(exported.status === 'passed' && !exported.attachmentMetadata.some(row => row?.contentType === 'image/png'))], + })); const signature = rows => rows.map(row => JSON.stringify([row.name, row.contentType, row.contentLength])).sort(); const runUrl = `https://github.com/paperclipai/paperclip/actions/runs/${context.runId}`; const sourceUrl = `https://github.com/paperclipai/paperclip/commit/${context.targetSha}`; @@ -367,15 +423,29 @@ export async function verifyChatUpload(api, env, selected, exported, { wait = ms for (let poll = 0; poll < 12 && now() < deadline; poll++) { const run = await get(`/api/jobrun/${selected.allureJobRunId}`); if (run.id !== Number(selected.allureJobRunId) || run.job?.id !== selected.jobId || run.job?.type !== 'github' - || run.launchId !== selected.launchId || run.url !== runUrl || !['in_progress', 'finished'].includes(run.stage)) fail('chat_testops_jobrun_mismatch'); + || run.launchId !== selected.launchId || run.url !== runUrl || !['in_progress', 'finished'].includes(run.stage)) stopUpload('chat_testops_jobrun_mismatch', 'jobrun', () => ({ + jobrun_id: [run.id === Number(selected.allureJobRunId), run.id], job_id: [run.job?.id === selected.jobId, run.job?.id], + job_type: [run.job?.type === 'github', run.job?.type], launch_id: [run.launchId === selected.launchId, run.launchId], + run_url: [run.url === runUrl, run.url], stage: [['in_progress', 'finished'].includes(run.stage), run.stage], + })); const launch = await get(`/api/launch/${selected.launchId}`); - if (launch.id !== selected.launchId || launch.projectId !== 1) fail('chat_testops_launch_mismatch'); + if (launch.id !== selected.launchId || launch.projectId !== 1) stopUpload('chat_testops_launch_mismatch', 'launch', () => ({ + launch_id: [launch.id === selected.launchId, launch.id], project_id: [launch.projectId === 1, launch.projectId], + })); const listed = await get(`/api/testresult?${new URLSearchParams({ launchId: String(selected.launchId), page: '0', size: '100' })}`); if (!Array.isArray(listed?.content) || listed.content.length > 1 || !Number.isSafeInteger(listed.totalElements) - || listed.totalElements !== listed.content.length || !Number.isSafeInteger(listed.totalPages) || listed.totalPages < 0 || listed.totalPages > 1) fail('chat_testops_upload_mismatch'); + || listed.totalElements !== listed.content.length || !Number.isSafeInteger(listed.totalPages) || listed.totalPages < 0 || listed.totalPages > 1) stopUpload('chat_testops_upload_mismatch', 'result_page', () => ({ + content_array: [Array.isArray(listed?.content), listed?.content], content_limit: [Array.isArray(listed?.content) && listed.content.length <= 1], + total_integer: [Number.isSafeInteger(listed?.totalElements), listed?.totalElements], + total_matches_content: [Array.isArray(listed?.content) && listed.totalElements === listed.content.length], + pages_integer: [Number.isSafeInteger(listed?.totalPages), listed?.totalPages], + pages_range: [Number.isSafeInteger(listed?.totalPages) && listed.totalPages >= 0 && listed.totalPages <= 1], + })); if (listed.content.length) { const row = listed.content[0]; - if (!id(row.id) || row.testCaseId !== selected.caseId) fail('chat_testops_upload_mismatch'); + if (!id(row.id) || row.testCaseId !== selected.caseId) stopUpload('chat_testops_upload_mismatch', 'result_row', () => ({ + result_id: [id(row.id), row.id], case_id: [row.testCaseId === selected.caseId, row.testCaseId], + })); const result = await get(`/api/testresult/${row.id}`); if (result.id !== row.id || result.projectId !== 1 || result.launchId !== selected.launchId || result.testCaseId !== selected.caseId || result.fullName !== selected.fullName || result.jobRun?.id !== Number(selected.allureJobRunId) @@ -385,14 +455,37 @@ export async function verifyChatUpload(api, env, selected, exported, { wait = ms || result.jobRun.job.url != null && !publicJobUrl(result.jobRun.job.url, env)) || !Array.isArray(result.links) || result.links.some(link => typeof link.url === 'string' && (link.url.startsWith('https://github.com/paperclipai/paperclip/actions/runs/') && link.url !== runUrl - || link.url.startsWith('https://github.com/paperclipai/paperclip/commit/') && link.url !== sourceUrl))) fail('chat_testops_upload_mismatch'); - if (!['passed', 'failed', 'broken', 'skipped', 'unknown'].includes(result.status)) fail('chat_testops_upload_mismatch'); - if (result.status !== 'unknown' && result.status !== exported.status) fail('chat_testops_upload_mismatch'); + || link.url.startsWith('https://github.com/paperclipai/paperclip/commit/') && link.url !== sourceUrl))) stopUpload('chat_testops_upload_mismatch', 'result', () => ({ + result_id: [result.id === row.id, result.id], project_id: [result.projectId === 1, result.projectId], + launch_id: [result.launchId === selected.launchId, result.launchId], case_id: [result.testCaseId === selected.caseId, result.testCaseId], + full_name: [result.fullName === selected.fullName, result.fullName], jobrun_id: [result.jobRun?.id === Number(selected.allureJobRunId), result.jobRun?.id], + jobrun_launch: [result.jobRun?.launchId === selected.launchId, result.jobRun?.launchId], jobrun_url: [result.jobRun?.url === runUrl, result.jobRun?.url], + embedded_job: [!(result.jobRun?.job != null && (result.jobRun.job.id !== run.job.id + || result.jobRun.job.type != null && result.jobRun.job.type !== run.job.type + || result.jobRun.job.url != null && !publicJobUrl(result.jobRun.job.url, env))), result.jobRun?.job], + links_array: [Array.isArray(result.links), result.links], known_links: [Array.isArray(result.links) && !result.links.some(link => typeof link?.url === 'string' + && (link?.url.startsWith('https://github.com/paperclipai/paperclip/actions/runs/') && link?.url !== runUrl + || link?.url.startsWith('https://github.com/paperclipai/paperclip/commit/') && link?.url !== sourceUrl))], + })); + if (!['passed', 'failed', 'broken', 'skipped', 'unknown'].includes(result.status)) stopUpload('chat_testops_upload_mismatch', 'result_status', () => ({ + status_supported: [['passed', 'failed', 'broken', 'skipped', 'unknown'].includes(result.status), result.status], + })); + if (result.status !== 'unknown' && result.status !== exported.status) stopUpload('chat_testops_upload_mismatch', 'result_status', () => ({ + status_matches_export: [result.status === 'unknown' || result.status === exported.status, result.status], + })); const attached = await get(`/api/testresult/attachment?${new URLSearchParams({ testResultId: String(row.id), page: '0', size: '100' })}`); if (!Array.isArray(attached?.content) || attached.content.length > 10 || !Number.isSafeInteger(attached.totalElements) || attached.totalElements !== attached.content.length || !Number.isSafeInteger(attached.totalPages) || attached.totalPages < 0 || attached.totalPages > 1 || new Set(attached.content.map(value => value.id)).size !== attached.content.length - || attached.content.some(value => !id(value.id) || value.missed === true || value.fromTestCase === true)) fail('chat_testops_upload_mismatch'); + || attached.content.some(value => !id(value.id) || value.missed === true || value.fromTestCase === true)) stopUpload('chat_testops_upload_mismatch', 'attachment_page', () => ({ + content_array: [Array.isArray(attached?.content), attached?.content], content_limit: [Array.isArray(attached?.content) && attached.content.length <= 10], + total_integer: [Number.isSafeInteger(attached?.totalElements), attached?.totalElements], + total_matches_content: [Array.isArray(attached?.content) && attached.totalElements === attached.content.length], + pages_integer: [Number.isSafeInteger(attached?.totalPages), attached?.totalPages], + pages_range: [Number.isSafeInteger(attached?.totalPages) && attached.totalPages >= 0 && attached.totalPages <= 1], + unique_ids: [Array.isArray(attached?.content) && new Set(attached.content.map(value => value?.id)).size === attached.content.length], + attachment_rows: [Array.isArray(attached?.content) && !attached.content.some(value => !id(value?.id) || value?.missed === true || value?.fromTestCase === true)], + })); if (now() < deadline && result.status === exported.status && same(signature(attached.content), signature(exported.attachmentMetadata))) { return { schema: 'paperclip.chat-feature-testops-upload-verification/v1', ...context, projectId: 1, planId: selected.planId, jobId: selected.jobId, allureJobRunId: selected.allureJobRunId, launchId: selected.launchId, caseId: selected.caseId, diff --git a/tests/runner-e2e/chat-feature-testops-api.test.mjs b/tests/runner-e2e/chat-feature-testops-api.test.mjs index 9b2bc7c74f7..ef86e1e2c58 100644 --- a/tests/runner-e2e/chat-feature-testops-api.test.mjs +++ b/tests/runner-e2e/chat-feature-testops-api.test.mjs @@ -387,6 +387,110 @@ test('readback rejects authoritative or embedded identity contradictions and a f } }); +async function uploadFailureDiagnostic(api, context = uploadContext, selection = uploadSelection, receipt = uploadReceipt) { + const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs'); + let error; + await assert.rejects(verifyChatUpload(api, context, selection, receipt), value => { error = value; return true; }); + return { error, projected: failureProjection(error, 'upload_verification') }; +} + +test('upload diagnostics identify selected, environment and exported context without an HTTP call', async () => { + const noHttp = () => assert.fail('context rejection must precede HTTP'); + const cases = [ + [{}, { ...uploadSelection, targetSha: 'b'.repeat(40) }, uploadReceipt, 'selected_target_sha', 'string'], + [{ ALLURE_JOB_RUN_ID: undefined }, uploadSelection, uploadReceipt, 'selected_jobrun', 'string'], + [{}, uploadSelection, { ...uploadReceipt, count: 'private-count' }, 'export_count', 'string'], + ]; + for (const [envChange, selected, receipt, first, type] of cases) { + const { projected } = await uploadFailureDiagnostic(noHttp, { ...uploadContext, ...envChange }, selected, receipt); + assert.equal(projected.code, 'chat_testops_upload_mismatch'); + assert.equal(projected.verification.phase, 'context'); + assert.equal(projected.verification.firstFailedPredicate, first); + assert.equal(projected.verification.checks[first], false); + assert.equal(projected.verification.types[first], type); + assert.ok(!JSON.stringify(projected).includes('private-count')); + if (first === 'selected_jobrun') assert.equal(projected.verification.types.env_jobrun, 'missing'); + } +}); + +test('upload diagnostics distinguish absent page and row fields from contradictory numeric identities', async () => { + const base = uploadApi(); + for (const [response, phase, first, type] of [ + [{ content: [], totalPages: 0 }, 'result_page', 'total_integer', 'missing'], + [{ content: [], totalElements: 0, totalPages: 'private-page' }, 'result_page', 'pages_integer', 'string'], + [{ content: [{ id: 45 }], totalElements: 1, totalPages: 1 }, 'result_row', 'case_id', 'missing'], + [{ content: [{ id: 45, testCaseId: 92 }], totalElements: 1, totalPages: 1 }, 'result_row', 'case_id', 'number'], + ]) { + const { projected } = await uploadFailureDiagnostic((route, ...args) => route.startsWith('/api/testresult?') ? response : base(route, ...args)); + assert.equal(projected.verification.phase, phase); assert.equal(projected.verification.firstFailedPredicate, first); + assert.equal(projected.verification.types[first], type); assert.equal(projected.verification.checks[first], false); + assert.ok(!JSON.stringify(projected).includes('private-page')); + } +}); + +test('upload diagnostics retain exact authoritative and result rejection predicates', async () => { + const base = uploadApi(); + const cases = [ + ['/api/jobrun/9', { ...uploadRun, job: { type: 'github' } }, 'jobrun', 'job_id', 'missing'], + ['/api/jobrun/9', { ...uploadRun, job: { id: 8, type: 'github' } }, 'jobrun', 'job_id', 'number'], + ['/api/launch/10', { id: 10, projectId: 2 }, 'launch', 'project_id', 'number'], + ]; + for (const [target, response, phase, first, type] of cases) { + const { projected } = await uploadFailureDiagnostic((route, ...args) => route === target ? response : base(route, ...args)); + assert.equal(projected.verification.phase, phase); assert.equal(projected.verification.firstFailedPredicate, first); + assert.equal(projected.verification.types[first], type); + } + const result = await uploadFailureDiagnostic(uploadApi({ status: 'private-status' })); + assert.equal(result.projected.verification.phase, 'result_status'); + assert.equal(result.projected.verification.firstFailedPredicate, 'status_supported'); + assert.ok(!JSON.stringify(result.projected).includes('private-status')); +}); + +test('upload diagnostics identify local attachment and backend attachment-page failures', async () => { + const local = await uploadFailureDiagnostic(() => assert.fail('attachment contract precedes HTTP'), uploadContext, uploadSelection, + { ...uploadReceipt, attachmentMetadata: uploadReceipt.attachmentMetadata.map((row, index) => index === 0 ? { ...row, contentType: 'private-type' } : row) }); + assert.equal(local.projected.verification.phase, 'attachment_contract'); + assert.equal(local.projected.verification.firstFailedPredicate, 'attachment_rows'); + assert.ok(!JSON.stringify(local.projected).includes('private-type')); + const base = uploadApi(); + const page = await uploadFailureDiagnostic((route, ...args) => route.startsWith('/api/testresult/attachment?') + ? { content: null, totalElements: 0, totalPages: 0 } : base(route, ...args)); + assert.equal(page.projected.verification.phase, 'attachment_page'); + assert.equal(page.projected.verification.firstFailedPredicate, 'content_array'); + assert.equal(page.projected.verification.types.content_array, 'null'); +}); + +test('upload diagnostic projection never retains secret values, unknown fields or mutable aliases', async () => { + const secret = 'DO_NOT_RETAIN_SYNTHETIC_CREDENTIAL_OR_PRIVATE_URL'; + const { error, projected } = await uploadFailureDiagnostic(uploadApi({ fullName: secret, unknown: secret }), + { ...uploadContext, ALLURE_TOKEN: secret }); + assert.equal(projected.verification.phase, 'result'); + assert.equal(projected.verification.firstFailedPredicate, 'full_name'); + assert.equal(projected.verification.types.full_name, 'string'); + assert.ok(!JSON.stringify(projected).includes(secret)); + projected.verification.phase = secret; projected.verification.failedPredicates.push(secret); + projected.verification.checks.full_name = secret; projected.verification.types.full_name = secret; + const again = failureProjection(error, 'upload_verification'); + assert.equal(again.verification.phase, 'result'); assert.equal(again.verification.checks.full_name, false); + assert.ok(!JSON.stringify(again).includes(secret)); + const unknown = failureProjection(Object.assign(new Error(secret), { verification: { phase: secret } }), 'upload_verification'); + assert.equal(unknown.code, 'unknown_failure'); assert.ok(!Object.hasOwn(unknown, 'verification')); +}); + +test('upload diagnostics preserve an early mismatch when later array entries are null', async () => { + const result = await uploadFailureDiagnostic(uploadApi({ id: 46, links: [null] })); + assert.equal(result.projected.verification.phase, 'result'); + assert.equal(result.projected.verification.firstFailedPredicate, 'result_id'); + assert.equal(result.projected.verification.checks.result_id, false); + const base = uploadApi(); + const page = await uploadFailureDiagnostic((route, ...args) => route.startsWith('/api/testresult/attachment?') + ? { content: [null], totalElements: 2, totalPages: 1 } : base(route, ...args)); + assert.equal(page.projected.verification.firstFailedPredicate, 'total_matches_content'); + const local = await uploadFailureDiagnostic(() => assert.fail('no HTTP'), uploadContext, uploadSelection, + { ...uploadReceipt, attachmentMetadata: [{ ...uploadReceipt.attachmentMetadata[0], contentType: 'unexpected' }, null, uploadReceipt.attachmentMetadata[2]] }); + assert.equal(local.projected.verification.firstFailedPredicate, 'attachment_rows'); +}); + test('readback missing result or screenshot metadata is bounded unverified, never a fabricated pass', async () => { const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs'); let waits = 0;