fix(evals): distinguish scheduled TestOps results from uploads (#18676)

## Thinking Path

> - Paperclip lets people manage AI agents and inspect their work.
> - The chat catalog reports each original Product E2E case to TestOps.
> - TestOps creates scheduled result rows before a case runs.
> - Those rows have identity fields but no status property.
> - The publisher currently mistakes any case row for an uploaded
outcome.
> - This PR distinguishes scheduled membership from an outcome after
strict identity checks.
> - Each case can upload once and wait for processing without changing
its grade.

## Linked Issues or Issue Description

Refs #18647, #18666.

**What happened?**

A real TestOps launch had all three scheduled result rows before any
model case started. Each row had the expected case, project, launch and
job-run identity. Its status property was absent. The publisher treated
the row as already uploaded, skipped its only POST and then rejected
readback.

**Expected behavior**

A fully bound scheduled row permits the first immutable case upload. It
does not count as an outcome. Readback waits within the existing bound
until the original status and exact attachments are ready.

**Steps to reproduce**

Run the new scheduled-row control against the parent API. It rejects
before sending the initial result. Fixing only that initial check still
rejects the first scheduled readback after one POST.

## What Changed

- Share the strict result-detail identity check between initial
inspection and readback.
- Recognize an absent status property as scheduled only after that
check. Present valid statuses, including unknown, stay readback-only.
Malformed statuses and foreign identities fail closed.
- Keep scheduled full-catalog readback unready within the same poll and
deadline bounds. Keep the final status, evidence and attachment checks
unchanged.
- Calibrate three planned rows, delayed and permanent processing,
unknown outcomes, wrong job/source and explicit-null status. Retain the
once-only upload and uncertain-response controls. Use the registered
legacy full-name alias in the fixtures.

## Verification

- The real-shape parent control fails with `result_mismatch` after the
first upload is suppressed.
- A controlled initial-only fix sends one POST and still fails on
scheduled readback. The final two-seam fix passes.
- All 77 focused controls pass: `node --test
tests/runner-e2e/chat-catalog-testops-streaming.test.mjs
tests/runner-e2e/chat-completion-testops-api.test.mjs
tests/runner-e2e/chat-full-catalog-testops-adapter.test.mjs`.
- Both changed files pass Node syntax checks. `git diff --check` passes.
- All local controls use synthetic data and mocked HTTP. No local app,
database, native build or provider runs.
- The affected live run was cancelled during build after metadata proved
no original case had started. Its build instance is terminated. No
original grade was changed or inferred.
- Latest-head GitHub CI is green: 48 successful jobs and one skipped.
Typecheck, build, browser and required aggregate checks passed.
- The first CI attempt failed a heartbeat scheduling test’s five-second
idle wait after its 51-admission assertion passed. Its six direct owners
were byte-identical to the prior passing CI. One same-head failed-job
recovery passed the server shard and required aggregate; all 46 earlier
successful jobs were carried unchanged. The original failure is
preserved, and its underlying cause remains unresolved. No heartbeat
source, fixture or timeout was changed.
- Live streaming qualification remains pending. No user command or
documentation contract changes.

## Risks

- The observed JSON distinction is status-property absence, not a
result-stage or UUID field. Those fields are not used.
- A scheduled row must match the current project, launch, case,
registered full name, job run and GitHub source URL. It never qualifies
as passed or complete.
- The same 12-poll and time limits remain. A permanent scheduled state
fails. A present outcome never causes another POST. An uncertain write
still uses GET-only reconciliation.
- Backend handling of uploaded scheduled rows still needs live
qualification. Duplicate, foreign and final-coverage guards remain; this
PR does not assume server deduplication.
- Earlier dispatch and cancelled setup records remain separate. The
first integration HTTP 500 cause is still unknown. No in-flight legacy
workflow is changed or rerun.

## Model Used

OpenAI `gpt-6.1-sol`, with tool use and code execution. The session does
not expose a context-window size. The independent source peer uses the
same model family.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

tests/runner-e2e/chat-catalog-testops-streaming.test.mjs

@@ -9,7 +9,8 @@ import { run } from './chat-completion-testops-cli.mjs';
 import { prepareCaseBundle, readCaseBundle, publishCaseBundle, publicationSet } from './chat-catalog-testops-streaming.mjs';
 const context = { catalog: 'full', runId: '123', attempt: '1', workflowSha: 'a'.repeat(40), targetSha: catalogTargetSha };
 const caseIds = { 'tasks.create-backlog': 388, 'documents-artifacts.write-plan': 26, 'projects.project-defaults': 100 };
-const cases = fullCatalogCases.filter(row => Object.hasOwn(caseIds, row.key)).map(row => ({ ...row, caseId: caseIds[row.key] }));
+const cases = fullCatalogCases.filter(row => Object.hasOwn(caseIds, row.key)).map(row => ({ ...row, caseId: caseIds[row.key],
+  registeredFullName: row.aliases[0] ?? row.fullName }));
 const selection = { schema: 'paperclip.chat-completion-testops-selection/v1', ...context, projectId: 1,
   planId: 102, jobId: 100, launchId: 12, allureJobRunId: '77', cases };
 const digest = hash(JSON.stringify(selection)), current = cases.find(row => row.caseId === 388), runUrl = 'https://github.com/paperclipai/paperclip/actions/runs/123';
@@ -33,8 +34,15 @@ async function fixture(t, failed = false, owned = current, runtimeContext = cont
   const prepared = await prepareCaseBundle(root, output, runtimeContext, selection, owned.executionId, digest);
   return { root, output, prepared };
 }
-function server(prepared, { existing = false, uncertain = false, apply = true } = {}) {
-  let present = existing, posts = 0; const calls = [];
+function server(prepared, { existing = false, uncertain = false, apply = true, scheduled = false, processingReads = 0 } = {}) {
+  let present = existing, posts = 0, pendingReads = processingReads; const calls = [];
+  const resultId = scheduled ? 3347 : 501;
+  const detail = () => {
+    const pending = !present || pendingReads-- > 0;
+    return { id: resultId, testCaseId: 388, projectId: 1, launchId: 12,
+      fullName: current.registeredFullName, jobRun: { id: 77, launchId: 12, url: runUrl }, links: [],
+      ...(!pending ? { status: prepared.exported[0].status } : {}) };
+  };
   const api = async (route, method = 'GET', body) => {
     calls.push({ route, method });
     if (method === 'POST') {
@@ -49,10 +57,11 @@ function server(prepared, { existing = false, uncertain = false, apply = true }
     }
     if (route === '/api/jobrun/77') return { id: 77, job: { id: 100, type: 'github' }, stage: 'finished', launchId: 12, url: runUrl };
     if (route === '/api/launch/12') return { id: 12, projectId: 1, closed: false };
-    if (route.startsWith('/api/testresult?')) return { content: [...(present ? [{ id: 501, testCaseId: 388 }] : []),
-      { id: 502, testCaseId: 26 }], totalElements: present ? 2 : 1, totalPages: 1, number: 0 };
-    if (route === '/api/testresult/501') return { id: 501, testCaseId: 388, projectId: 1, launchId: 12,
-      fullName: current.fullName, status: prepared.exported[0].status, jobRun: { id: 77, launchId: 12, url: runUrl }, links: [] };
+    if (route.startsWith('/api/testresult?')) return { content: scheduled ? cases.map(row => ({
+      id: row.caseId === 388 ? resultId : row.caseId === 100 ? 3346 : 3345, testCaseId: row.caseId }))
+      : [...(present ? [{ id: resultId, testCaseId: 388 }] : []), { id: 502, testCaseId: 26 }],
+      totalElements: scheduled ? 3 : present ? 2 : 1, totalPages: 1, number: 0 };
+    if (route === `/api/testresult/${resultId}`) return detail();
     if (route.startsWith('/api/testresult/attachment?')) return { content: prepared.exported[0].attachments.map((row, index) => ({
       ...row, id: 700 + index, missed: false, fromTestCase: false })), totalElements: 2, totalPages: 1 };
     throw new Error(`Unexpected test route ${route}`);
@@ -60,8 +69,8 @@ function server(prepared, { existing = false, uncertain = false, apply = true }
   return { api, calls, get posts() { return posts; } };
 }
 const immediate = { now: () => 0, wait: async () => {} };
-async function publish(f, s, mark = async () => {}) {
-  return publishCaseBundle(s.api, f.root, f.output, f.prepared, context, selection, current.executionId, digest, mark, immediate);
+async function publish(f, s, mark = async () => {}, options = immediate) {
+  return publishCaseBundle(s.api, f.root, f.output, f.prepared, context, selection, current.executionId, digest, mark, options);
 }
 test('one frozen singleton bundle publishes into the bound shared launch beside siblings', async t => {
   const f = await fixture(t), s = server(f.prepared); let markers = 0;
@@ -76,9 +85,38 @@ test('an existing result is verified without another mutation or a regenerated U
   const value = await publish(f, s, async () => assert.fail('must not mark another upload'));
   assert.equal(value.uploadSent, false); assert.equal(s.posts, 0); assert.equal(value.verification, 'VERIFIED');
 });
+test('an existing unknown outcome remains readback-only and never requests another upload', async t => {
+  const f = await fixture(t), s = server(f.prepared, { existing: true }), original = s.api;
+  s.api = async (...args) => { const row = await original(...args); if (args[0] === '/api/testresult/501') row.status = 'unknown'; return row; };
+  await assert.rejects(publish(f, s, async () => assert.fail('must not mark another upload')), /upload_unverified/);
+  assert.equal(s.posts, 0);
+});
+test('three scheduled launch rows permit one initial upload and remain unready until an outcome is processed', async t => {
+  const f = await fixture(t), s = server(f.prepared, { scheduled: true, processingReads: 1 });
+  let markers = 0, waits = 0;
+  const value = await publish(f, s, async () => { markers++; }, { now: () => 0, wait: async () => { waits++; } });
+  assert.equal(value.verification, 'VERIFIED'); assert.equal(value.uploadSent, true);
+  assert.equal(value.exported.status, 'passed'); assert.equal(value.results[0].resultId, 3347);
+  assert.equal(s.posts, 1); assert.equal(markers, 1); assert.equal(waits, 1);
+});
+test('permanently scheduled readback stops at the original poll bound without another upload', async t => {
+  const f = await fixture(t), s = server(f.prepared, { scheduled: true, processingReads: Infinity });
+  let waits = 0;
+  await assert.rejects(publish(f, s, async () => {}, { now: () => 0, wait: async () => { waits++; } }), /upload_unverified/);
+  assert.equal(s.posts, 1); assert.equal(waits, 11);
+});
+test('scheduled details with a foreign jobrun, contradictory source or malformed status refuse before upload', async t => {
+  for (const change of [row => { row.jobRun.id = 78; }, row => { row.jobRun.url = runUrl + '0'; },
+    row => { row.status = null; }]) {
+    const f = await fixture(t), s = server(f.prepared, { scheduled: true }), original = s.api;
+    s.api = async (...args) => { const row = await original(...args); if (args[0] === '/api/testresult/3347') change(row); return row; };
+    await assert.rejects(publish(f, s, async () => assert.fail('must not mark an invalid result')), /result_mismatch/);
+    assert.equal(s.posts, 0);
+  }
+});
 test('an uncertain applied upload reconciles by GET while an absent upload stops without resend', async t => {
   for (const apply of [true, false]) {
-    const f = await fixture(t), s = server(f.prepared, { uncertain: true, apply });
+    const f = await fixture(t), s = server(f.prepared, { uncertain: true, apply, scheduled: true });
     if (apply) {
       const value = await publish(f, s); assert.equal(value.verification, 'VERIFIED');
       assert.equal(value.uncertainUpload.kind, 'transport_or_response_failure');
@@ -253,7 +291,7 @@ test('the final CLI verifies all authenticated mixed-grade publications without
       const match = u.pathname.match(/^\/api\/testresult\/(\d+)$/);
       if (match) {
         const i = Number(match[1]) - 501; details.push(cases[i].caseId);
-        return Response.json({ id: Number(match[1]), projectId: 1, launchId: 12, testCaseId: cases[i].caseId, fullName: cases[i].fullName,
+        return Response.json({ id: Number(match[1]), projectId: 1, launchId: 12, testCaseId: cases[i].caseId, fullName: cases[i].registeredFullName,
           jobRun: { id: 77, launchId: 12, url: runUrl }, links: [], status: exported[i].status });
       }
       assert.equal(u.pathname, '/api/testresult/attachment');

tests/runner-e2e/chat-completion-testops-api.mjs

@@ -52,6 +52,21 @@ export function isWorkflowUrl(url, workflowId, context = {}) {
     || context.catalog === 'full' && url === `https://github.com/paperclipai/paperclip/blob/master/${workflow}`
     || url === (context.catalog === 'full' ? 'https://github.com/paperclipai/paperclip/actions?query=workflow%3A%22Chat+catalog+TestOps%22' : 'https://github.com/paperclipai/paperclip/actions?query=workflow%3A%22Chat+completion+TestOps%22');
 }
+function validatedResultStatus(result, id, expected, selection, context, runUrl) {
+  const { targetSha } = catalogDefinition(context);
+  if (result?.id !== id || result.projectId !== 1 || result.launchId !== selection.launchId || result.testCaseId !== expected.caseId
+    || result.fullName !== (expected.registeredFullName ?? expected.fullName) || result.jobRun?.id !== Number(selection.allureJobRunId)
+    || result.jobRun?.launchId !== selection.launchId || result.jobRun?.url !== runUrl
+    || result.jobRun.job != null && (result.jobRun.job.id !== selection.jobId
+      || result.jobRun.job.type != null && result.jobRun.job.type !== 'github'
+      || result.jobRun.job.url != null && !isWorkflowUrl(result.jobRun.job.url, context.workflowId, context))
+    || result.links != null && (!Array.isArray(result.links) || result.links.some(link => !link || typeof link.url === 'string'
+      && (link.url.startsWith('https://github.com/paperclipai/paperclip/actions/runs/') && link.url !== runUrl
+        || link.url.startsWith('https://github.com/paperclipai/paperclip/commit/') && link.url !== `https://github.com/paperclipai/paperclip/commit/${targetSha}`)))) reject('result_mismatch');
+  if (!Object.hasOwn(result, 'status')) return undefined;
+  if (!['failed', 'broken', 'passed', 'skipped', 'unknown'].includes(result.status)) reject('result_mismatch');
+  return result.status;
+}
 function assertCase(actual, expected) {
   if (actual.id !== expected.caseId || actual.projectId !== 1 || actual.fullName !== (expected.registeredFullName ?? expected.fullName)
     || actual.automated !== true || actual.deleted === true) reject('case_mismatch');
@@ -269,16 +284,10 @@ async function verifyUploadScope(api, selection, context, exported, executionId,
       if (!expected || !positive(listedRow.id)) reject('foreign_result');
       const wanted = exported.find(value => value.caseId === expected.caseId);
       const result = await get(`/api/testresult/${listedRow.id}`);
-      if (result.id !== listedRow.id || result.projectId !== 1 || result.launchId !== selection.launchId || result.testCaseId !== expected.caseId
-        || result.fullName !== (expected.registeredFullName ?? expected.fullName) || result.jobRun?.id !== Number(selection.allureJobRunId)
-        || result.jobRun?.launchId !== selection.launchId || result.jobRun?.url !== runUrl
-        || result.jobRun.job != null && (result.jobRun.job.id !== selection.jobId
-          || result.jobRun.job.type != null && result.jobRun.job.type !== 'github'
-          || result.jobRun.job.url != null && !isWorkflowUrl(result.jobRun.job.url, context.workflowId, context))
-        || !Array.isArray(result.links) || result.links.some(link => typeof link.url === 'string'
-          && (link.url.startsWith('https://github.com/paperclipai/paperclip/actions/runs/') && link.url !== runUrl
-            || link.url.startsWith('https://github.com/paperclipai/paperclip/commit/') && link.url !== `https://github.com/paperclipai/paperclip/commit/${targetSha}`))
-        || !['unknown', wanted.status].includes(result.status)) reject('result_mismatch');
+      const status = validatedResultStatus(result, listedRow.id, expected, selection, context, runUrl);
+      if (context.catalog === 'full' && status === undefined) return { executionId: expected.executionId, caseId: expected.caseId,
+        resultId: result.id, cleanup: wanted.cleanup, attachmentsVerified: false };
+      if (!Array.isArray(result.links) || !['unknown', wanted.status].includes(status)) reject('result_mismatch');
       const attached = await get(`/api/testresult/attachment?${new URLSearchParams({ testResultId: String(result.id), page: '0', size: '100' })}`);
       const attachments = pageRows(attached);
       if (attached.totalElements !== attachments.length || attachments.length > 10 || new Set(attachments.map(row => row.id)).size !== attachments.length
@@ -325,14 +334,18 @@ export async function inspectPilotCaseUpload(api, selection, context, executionI
     return api(route, 'GET', undefined, Math.min(30_000, remaining));
   };
   for (let poll = 0; poll < 12 && now() < deadline; poll++) {
-    const { run, launch } = await uploadBinding(get, selection, context);
+    const { run, launch, runUrl } = await uploadBinding(get, selection, context);
     if (launch.closed === true) reject('launch_mismatch');
     const rows = await resultRows(get, selection.launchId, authorized.length, true);
     if (rows !== null) {
       if (new Set(rows.map(value => value.id)).size !== rows.length
         || new Set(rows.map(value => value.testCaseId)).size !== rows.length) reject('result_page');
       if (rows.some(value => !positive(value.id) || !authorized.some(expected => expected.caseId === value.testCaseId))) reject('foreign_result');
-      return { present: rows.some(value => value.testCaseId === row.caseId), jobRunStage: run.stage };
+      const existing = rows.find(value => value.testCaseId === row.caseId);
+      if (!existing) return { present: false, jobRunStage: run.stage };
+      const result = await get(`/api/testresult/${existing.id}`);
+      return { present: validatedResultStatus(result, existing.id, row, selection, context, runUrl) !== undefined,
+        jobRunStage: run.stage };
     }
     if (poll < 11 && now() + 5000 < deadline) await wait(5000);
   }