From 3a7349ddc60142283c394121882288f2db07b215 Mon Sep 17 00:00:00 2001 From: Dotta Date: Sun, 4 Oct 2026 21:11:06 -0500 Subject: [PATCH] Verify document navigation in both task interfaces Co-Authored-By: Paperclip --- doc/plans/2026-10-04-native-completion-answer-fix.md | 2 +- tests/runner-e2e/runner.spec.ts | 6 +++++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/doc/plans/2026-10-04-native-completion-answer-fix.md b/doc/plans/2026-10-04-native-completion-answer-fix.md index bc6348833f..ca46c7b43e 100644 --- a/doc/plans/2026-10-04-native-completion-answer-fix.md +++ b/doc/plans/2026-10-04-native-completion-answer-fix.md @@ -6,7 +6,7 @@ The correction keeps the shortened completion procedure while explicitly requiri Only the manual instruction comparison uses the v3 final-answer observation. The existing `native-completion` suite retains its v2 verdict checks and does not acquire the browser-navigation requirement. A calibration demonstrates the same retained input can pass v2 and fail v3; original report files are never rewritten. The harness definition digest changes truthfully for future invocations. Its additional checks read only the actual run-attributed persisted provider final. A blocker label and action do not substitute for an explicit missing-access explanation. Document links must identify the exact task and one revisioned document at the instance origin; wrong origin, task, document, query, credentials or absent evidence fail. Exact call-ID joins, acceptance/termination, ordering, side-effect and budget checks remain enforced. -After saving the strict native snapshot, each completion cell clicks the rendered final-reply link in the browser. It must open the canonical saved document and show the original content marker. Retain the navigation receipt and screenshot independently; a valid-looking Markdown URL alone does not qualify navigation. +After saving the strict native snapshot, each completion cell clicks the rendered final-reply link in the browser. It must open the canonical saved document and show the original content marker in exactly one visible classic document card or document-specific side-panel tab. The navigation check follows the actual configured UI instead of requiring a classic-only element. Retain the navigation receipt and screenshot independently; a valid-looking Markdown URL alone does not qualify navigation. Calibrate with correct paraphrases and plausible wrong answers, including the action-only omission. Replay the retained observations only as labeled additional diagnostics; never overwrite or regrade the original results. Provider-free measurement v2 captures eighteen shared runnerd RPC projections plus six direct OpenCode HTTP projections across v4/v5 start, resume and continuation. The latter exercises the concrete OpenCode backend using a local fake server. Both boundaries measure Paperclip-supplied payloads, not provider stock prompts or model cognition. No synthetic receipt qualifies live model behavior. diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 774c36d57d..2bcfd976f2 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -2717,7 +2717,11 @@ for (const execution of executions) { await expect(link).toBeVisible({ timeout: 30_000 }); await link.click(); await expect(page).toHaveURL(new URL(href, observation.documentLinkContext.appOrigin).href); - const target = page.locator(`[id=${JSON.stringify(`document-${document.key}`)}]`); + const target = page.locator([ + `[id=${JSON.stringify(`document-${document.key}`)}]:visible`, + `[id=${JSON.stringify(`side-panel-content-document:${document.key}`)}]:visible`, + ].join(", ")); + await expect(target).toHaveCount(1); await expect(target).toBeVisible(); await expect(target).toContainText(marker); opened = true;