test(chaos): close a covering modal before clicking Retry, and name it when it blocks (#20715)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) clicks every needed Retry in the real board UI with Playwright. > - In one wave, a task failed and showed Retry. The "You completed 5 tasks!" star modal was open over the page. > - The modal is a Radix dialog. It sets `aria-hidden="true"` on the rest of the page, so `getByRole('button', { name: 'Retry' })` did not find the visible Retry. > - The grade said `ui_retry_button_missing`. That name hides the real cause. > - This pull request makes the harness close a covering modal before it looks for Retry. When a modal does not close and keeps a visible Retry unreachable, the grade says `ui_retry_blocked_by_modal`. > - The benefit is that a chaos grade names the real cause, and a promotional prompt does not fail a wave. ## Linked Issues or Issue Description Refs #20713 (product fix: the star prompt no longer opens over a task that needs Retry) **What happened?** A restart chaos wave saved a "Retry missing" screenshot. The screenshot shows the Retry banner and the star modal over it. The saved DOM shows the Retry button as visible, with a box, inside the banner. The harness still graded the task `ui_retry_button_missing`. **Expected behavior** The harness closes a modal that covers the page, as a user does, and then clicks Retry. If a modal does not close, the grade names the modal as the cause. **Steps to reproduce** 1. Open a task page that shows a failed attempt with Retry. 2. Open a Radix modal over the page (the star prompt did this). 3. Run `BoardUi.clickRetry` on that task. It reports `ui_retry_button_missing`. ## What Changed - `board-ui.ts`: before and while it waits for Retry, `clickRetryOnPage` closes an open modal that does not itself offer Retry. It clicks "Not now" (or "No thanks", "Maybe later", "Dismiss") when present, else it presses Escape. It also closes a modal that opens after Retry appears and swallows the click, then clicks again. At most three dialogs are closed. Each click has a timeout equal to the per-button request wait. - The click result has a new optional `dismissedDialogs` list (first line of each closed dialog). The observer copies it into the task's retry attempt. - New failure reason `ui_retry_blocked_by_modal` (in `types.ts`): a visible Retry exists in the DOM, it is not reachable by role, and a modal stays open. R3 in `grader.ts` reports it with the banner text, like the other UI reasons. - `README.md`: the board UI Retry paragraph names the dismissal and the new reason. - Tests in `board-ui.test.ts` use real Chromium pages with a modal and an `aria-hidden` page: "Not now" closes it and Retry is clicked; Escape closes it and Retry is clicked; a modal that opens on hover over Retry is closed and Retry is clicked; a modal that does not close gives `ui_retry_blocked_by_modal`. `grader.test.ts` covers the new R3 detail. ## Verification - Red first: the three new `board-ui.test.ts` cases failed on master (the covered Retry was reported as missing). - Green: `cd tests/restart-chaos && npx vitest run board-ui.test.ts grader.test.ts triggers.test.ts` (with `observer.test.ts`) gives 99 passed. - `npx tsc --noEmit -p tests/restart-chaos/tsconfig.json` passes. - `pnpm verify` exits 2 because the README is a global CI input. CI supplies the broad coverage. ## Risks - Low risk. This changes only the test harness. - The harness can close a dialog that the product meant to keep open. The closed dialog is recorded in `dismissedDialogs`, so the evidence still shows it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
2026-11-13 07:12:00 +01:00 · 2026-11-10 22:40:49 -06:00
1 parent f77b8a288e
commit 156212cfd5
@@ -511,7 +511,10 @@ windows are written to `storage-faults.json` and removed from the speed grade li
 Every Retry the grader needs is clicked in the real board UI: the harness opens the issue in a headless
 Chromium, waits for the visible Retry button and clicks it. A task whose Retry button is missing fails
 R3 as `ui_retry_button_missing (user saw: …)`; a click that sends no retry request fails as
-`ui_retry_no_request`. Crash-point waves serve no UI and click Retry through the API.
+`ui_retry_no_request`. Before it looks for Retry, the harness closes any open modal that covers the page
+("Not now", else Escape) and records it in the attempt's `dismissedDialogs`. A visible Retry that stays
+unreachable behind a modal that will not close fails as `ui_retry_blocked_by_modal`. Crash-point waves
+serve no UI and click Retry through the API.

 ## Daytona (`--env daytona`)

@@ -4,10 +4,24 @@ import { tmpdir } from "node:os";
 import path from "node:path";
 import { once } from "node:events";
 import { afterAll, beforeAll, describe, expect, it } from "vitest";
-import { BoardUi, UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST, isRetryRequestFor } from "./board-ui.js";
+import { BoardUi, UI_RETRY_BLOCKED_BY_MODAL, UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST, isRetryRequestFor } from "./board-ui.js";

 const retry = (issue: string) => `fetch('/api/issues/${issue}/retry',{method:'POST'})`;
+const modal = (buttons: string) => `<div role="dialog" data-state="open" aria-modal="true" style="position:fixed;inset:30%;background:#fff;z-index:2">
+  <h2>You completed 5 tasks!</h2>${buttons}</div>`;
+const coveredRetry = (issue: string) => `<main aria-hidden="true">
+  <div role="status">Couldn’t finish this attempt. <button onclick="${retry(issue)}">Retry</button></div></main>`;
+const closeModal = "document.querySelector('[role=dialog]').remove();document.querySelector('main').removeAttribute('aria-hidden')";
+const lateModal = `<div role="dialog" data-state="open" aria-modal="true" style="position:fixed;inset:0;background:#fff;z-index:2">
+  <h2>You completed 5 tasks!</h2><button onclick="this.parentElement.remove();document.querySelector('main').removeAttribute('aria-hidden')">Not now</button></div>`;
+const openLateModal = `document.querySelector('main').setAttribute('aria-hidden','true');document.body.insertAdjacentHTML('beforeend', ${JSON.stringify(lateModal).replace(/"/g, "&quot;")})`;
 const pages: Record<string, string> = {
+  "/issues/CHA-8": coveredRetry("i8") + modal(`<button onclick="${closeModal}">Not now</button>`),
+  "/issues/CHA-9": coveredRetry("i9") + modal(`<span>No way out</span>`),
+  "/issues/CHA-11": `<main><div role="status">Couldn’t finish this attempt.
+    <button onmouseover="if (!window.prompted) { window.prompted = true; ${openLateModal} }" onclick="${retry("i11")}">Retry</button></div></main>`,
+  "/issues/CHA-10": coveredRetry("i10") + modal(`<span>Press Escape</span>`)
+    + `<script>document.addEventListener('keydown', event => { if (event.key === 'Escape') { ${closeModal} } })</script>`,
  "/issues/CHA-1": `<div role="status">Couldn’t finish this attempt. <button onclick="${retry("i1")}">Retry</button></div>`,
  "/issues/CHA-2": `<div role="status">Saving files from the previous attempt</div>`,
  "/issues/CHA-7": `<div role="status">Couldn’t finish this attempt. <button style="display:none">Retry</button></div>`,
@@ -89,6 +103,38 @@ describe("BoardUi.clickRetry", () => {
    }
  }, 60_000);

+  it("dismisses a prompt that covers the Retry with Not now, then clicks Retry", async () => {
+    await withBoard(async board => {
+      expect(await board.clickRetry("CHA-8", "i8", 5_000, 1_000)).toMatchObject({ clicked: true, httpStatus: 202, accepted: true, error: null,
+        dismissedDialogs: [expect.stringContaining("You completed 5 tasks!")] });
+    });
+    expect(posts).toContain("/api/issues/i8/retry");
+  }, 60_000);
+
+  it("closes a prompt without a Not now button with Escape", async () => {
+    await withBoard(async board => {
+      expect(await board.clickRetry("CHA-10", "i10", 5_000, 1_000)).toMatchObject({ clicked: true, accepted: true,
+        dismissedDialogs: [expect.stringContaining("You completed 5 tasks!")] });
+    });
+    expect(posts).toContain("/api/issues/i10/retry");
+  }, 60_000);
+
+  it("closes a prompt that opens after Retry appeared, then clicks Retry", async () => {
+    await withBoard(async board => {
+      expect(await board.clickRetry("CHA-11", "i11", 5_000, 1_500)).toMatchObject({ clicked: true, httpStatus: 202, accepted: true, error: null,
+        dismissedDialogs: [expect.stringContaining("You completed 5 tasks!")] });
+    });
+    expect(posts).toContain("/api/issues/i11/retry");
+  }, 60_000);
+
+  it("names a modal that keeps a visible Retry unreachable", async () => {
+    await withBoard(async board => {
+      expect(await board.clickRetry("CHA-9", "i9", 2_000, 500)).toMatchObject({ clicked: false, error: UI_RETRY_BLOCKED_BY_MODAL,
+        bannerText: expect.stringContaining("Couldn’t finish this attempt") });
+    });
+    expect(posts).not.toContain("/api/issues/i9/retry");
+  }, 60_000);
+
  it("reports a Retry button whose click sends no retry request", async () => {
    await withBoard(async board => {
      expect(await board.clickRetry("CHA-3", "i3", 1_500, 500)).toMatchObject({ clicked: true, httpStatus: null, error: UI_RETRY_NO_REQUEST });
@@ -1,9 +1,9 @@
 import { mkdir, writeFile } from "node:fs/promises";
 import path from "node:path";
 import { chromium, type Browser, type ElementHandle, type Page } from "@playwright/test";
-import { UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST } from "./types.js";
+import { UI_RETRY_BLOCKED_BY_MODAL, UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST } from "./types.js";

-export { UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST };
+export { UI_RETRY_BLOCKED_BY_MODAL, UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST };

 export interface UiRetryClick {
  clicked: boolean;
@@ -12,6 +12,7 @@ export interface UiRetryClick {
  error: string | null;
  bannerText: string | null;
  diagnostics?: { screenshot: string | null; dom: string | null } | null;
+  dismissedDialogs?: string[];
 }

 export async function saveRetryMissingDiagnostics(page: Page, directory: string, label: string) {
@@ -37,6 +38,28 @@ export async function saveRetryMissingDiagnostics(page: Page, directory: string,
 }

 const PER_BUTTON_REQUEST_MS = 5_000;
+const DIALOG_SCAN_MS = 1_000;
+const MAX_DISMISSED_DIALOGS = 3;
+const BLOCKING_DIALOG = '[role="dialog"][data-state="open"], [role="alertdialog"][data-state="open"], '
+  + '[aria-modal="true"]:not([data-state="closed"]), dialog[open]';
+const DISMISS_BUTTON = /^(not now|no thanks|maybe later|dismiss)$/i;
+
+const blockingDialog = (page: Page) => page.locator(BLOCKING_DIALOG).filter({ visible: true }).first();
+
+async function dismissBlockingDialog(page: Page): Promise<string | null> {
+  const dialog = blockingDialog(page);
+  if (!await dialog.isVisible().catch(() => false)) return null;
+  if (await dialog.getByRole("button", { name: "Retry", exact: true }).count().catch(() => 0)) return null;
+  const label = (await dialog.innerText().catch(() => "")).trim().split(/\r?\n/)[0]?.slice(0, 120) || "dialog";
+  const handle = await dialog.elementHandle().catch(() => null);
+  const notNow = dialog.getByRole("button", { name: DISMISS_BUTTON }).filter({ visible: true }).first();
+  const clicked = await notNow.count().catch(() => 0) > 0 && await notNow.click({ timeout: 2_000 }).then(() => true, () => false);
+  if (!clicked) await page.keyboard.press("Escape").catch(() => undefined);
+  const closed = handle
+    ? await handle.waitForElementState("hidden", { timeout: 2_000 }).then(() => true, () => false)
+    : !await blockingDialog(page).isVisible().catch(() => false);
+  return closed ? label : null;
+}

 export function isRetryRequestFor(issueRefs: readonly string[], method: string, url: string): boolean {
  if (method !== "POST") return false;
@@ -61,28 +84,55 @@ export async function clickRetryOnPage(page: Page, url: string, issueRefs: reado
    const texts = await page.locator("[role=status], [role=alert]").allInnerTexts().catch(() => []);
    return texts.map(text => text.trim()).filter(Boolean).join(" | ").slice(0, 500) || null;
  };
-  try {
-    await page.getByRole("button", { name: "Retry", exact: true }).filter({ visible: true }).first()
-      .waitFor({ state: "visible", timeout: timeoutMs });
-  } catch {
+  const reachable = page.getByRole("button", { name: "Retry", exact: true }).filter({ visible: true }).first();
+  const dismissedDialogs: string[] = [];
+  const dismissed = () => (dismissedDialogs.length ? { dismissedDialogs } : {});
+  const dismissDialogs = async () => {
+    const before = dismissedDialogs.length;
+    while (dismissedDialogs.length < MAX_DISMISSED_DIALOGS) {
+      const closed = await dismissBlockingDialog(page);
+      if (closed === null) break;
+      dismissedDialogs.push(closed);
+    }
+    return dismissedDialogs.length > before;
+  };
+  const blockedByModal = async () => await blockingDialog(page).isVisible().catch(() => false)
+    && await page.getByRole("button", { name: "Retry", exact: true, includeHidden: true }).filter({ visible: true }).count().catch(() => 0) > 0;
+  const deadline = Date.now() + timeoutMs;
+  for (;;) {
+    await dismissDialogs();
+    const remaining = deadline - Date.now();
+    if (remaining <= 0) break;
+    if (await reachable.waitFor({ state: "visible", timeout: Math.min(remaining, DIALOG_SCAN_MS) }).then(() => true, () => false)) break;
+  }
+  if (!await reachable.isVisible().catch(() => false)) {
    const bannerText = await banner();
    const diagnostics = diagnosticsDirectory
      ? await saveRetryMissingDiagnostics(page, diagnosticsDirectory, issueRefs[0] ?? "issue").catch(() => null) : undefined;
-    return { clicked: false, httpStatus: null, accepted: null, error: UI_RETRY_BUTTON_MISSING, bannerText,
-      ...(diagnostics !== undefined ? { diagnostics } : {}) };
+    const blocked = await blockedByModal();
+    return { clicked: false, httpStatus: null, accepted: null, error: blocked ? UI_RETRY_BLOCKED_BY_MODAL : UI_RETRY_BUTTON_MISSING, bannerText,
+      ...(diagnostics !== undefined ? { diagnostics } : {}), ...dismissed() };
  }
  const bannerText = await banner();
  for (const button of await retryButtonsBannerFirst(page)) {
-    const response = page.waitForResponse(candidate => isRetryRequestFor(issueRefs, candidate.request().method(), candidate.url()),
-      { timeout: perButtonMs }).catch(() => null);
-    await button.click().catch(() => undefined);
-    const answered = await response;
-    if (!answered) continue;
-    const body = await answered.json().catch(() => null) as { accepted?: unknown } | null;
-    return { clicked: true, httpStatus: answered.status(), accepted: answered.ok() ? body?.accepted === true : false,
-      error: answered.ok() ? null : `retry request failed with ${answered.status()}`, bannerText };
+    for (let attempt = 0; attempt < 2; attempt++) {
+      await dismissDialogs();
+      const response = page.waitForResponse(candidate => isRetryRequestFor(issueRefs, candidate.request().method(), candidate.url()),
+        { timeout: perButtonMs }).catch(() => null);
+      await button.click({ timeout: perButtonMs }).catch(() => undefined);
+      const answered = await response;
+      if (answered) {
+        const body = await answered.json().catch(() => null) as { accepted?: unknown } | null;
+        return { clicked: true, httpStatus: answered.status(), accepted: answered.ok() ? body?.accepted === true : false,
+          error: answered.ok() ? null : `retry request failed with ${answered.status()}`, bannerText, ...dismissed() };
+      }
+      if (!await dismissDialogs()) break;
+    }
  }
-  return { clicked: true, httpStatus: null, accepted: null, error: UI_RETRY_NO_REQUEST, bannerText };
+  if (await blockedByModal()) {
+    return { clicked: false, httpStatus: null, accepted: null, error: UI_RETRY_BLOCKED_BY_MODAL, bannerText, ...dismissed() };
+  }
+  return { clicked: true, httpStatus: null, accepted: null, error: UI_RETRY_NO_REQUEST, bannerText, ...dismissed() };
 }

 export class BoardUi {
@@ -244,6 +244,8 @@ describe("T-oracles", () => {
      retry: { at: 2, observedRunId: "r1", httpStatus: null, accepted: null, error: "ui_retry_button_missing", via: "ui", bannerText: "Saving files" } } as unknown as TaskRecord;
    expect(gradeR3(record)).toEqual({ verdict: "fail", detail: "ui_retry_button_missing (user saw: Saving files)" });
    expect(gradeR3({ ...record, retry: { ...record.retry!, clicked: true, error: "ui_retry_no_request" } } as TaskRecord).detail).toMatch(/^ui_retry_no_request/);
+    expect(gradeR3({ ...record, retry: { ...record.retry!, error: "ui_retry_blocked_by_modal" } } as TaskRecord).detail)
+      .toBe("ui_retry_blocked_by_modal (user saw: Saving files)");
  });

  it("reports R2 as not checked for a sandbox-only workspace instead of passing it", () => {
@@ -1,4 +1,4 @@
-import { UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST, type FaultReceipt, type OracleResult, type RuleId, type RuleResult, type TaskGrade, type TaskRecord, type TaskSample } from "./types.js";
+import { UI_RETRY_BLOCKED_BY_MODAL, UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST, type FaultReceipt, type OracleResult, type RuleId, type RuleResult, type TaskGrade, type TaskRecord, type TaskSample } from "./types.js";
 import { g1Decision } from "./workloads.js";
 import { enforcedUniversalFailures, gradeUniversal, type UniversalDbEvidence } from "./universal.js";

@@ -112,7 +112,7 @@ function gradeOwnRetry(record: TaskRecord): RuleResult {
    }
    return { verdict: "n/a", detail: "no Retry needed" };
  }
-  if (record.retry.via === "ui" && (record.retry.error === UI_RETRY_BUTTON_MISSING || record.retry.error === UI_RETRY_NO_REQUEST)) {
+  if (record.retry.via === "ui" && [UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST, UI_RETRY_BLOCKED_BY_MODAL].includes(record.retry.error ?? "")) {
    return { verdict: "fail", detail: `${record.retry.error}${record.retry.bannerText ? ` (user saw: ${record.retry.bannerText})` : ""}` };
  }
  if (record.retry.error || record.retry.accepted !== true) {
@@ -718,7 +718,8 @@ export class TaskObserver {
    if (this.board) {
      const click = await this.board.clickRetry(issueRef, issueId);
      Object.assign(attempt, { via: "ui", httpStatus: click.httpStatus, accepted: click.accepted, error: click.error, bannerText: click.bannerText,
-        ...(click.diagnostics !== undefined ? { diagnostics: click.diagnostics } : {}) });
+        ...(click.diagnostics !== undefined ? { diagnostics: click.diagnostics } : {}),
+        ...(click.dismissedDialogs?.length ? { dismissedDialogs: click.dismissedDialogs } : {}) });
      if (click.accepted) attempt.acceptedAt = Math.max(now, Date.now());
      return attempt;
    }
@@ -256,10 +256,12 @@ export interface RetryAttempt {
  via?: "ui" | "api";
  bannerText?: string | null;
  diagnostics?: { screenshot: string | null; dom: string | null } | null;
+  dismissedDialogs?: string[];
 }

 export const UI_RETRY_BUTTON_MISSING = "ui_retry_button_missing";
 export const UI_RETRY_NO_REQUEST = "ui_retry_no_request";
+export const UI_RETRY_BLOCKED_BY_MODAL = "ui_retry_blocked_by_modal";

 export interface FaultReceipt {
  faultId: string;