Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/runtime-conference.ts
T
DottaandClaude Opus 5.5 cbd636d652 fix(evals): wait for rendered replies, bound runs by role and grade chat outcomes structurally (#20685)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat-feature eval catalog runs real chat journeys and grades
saved state, rendered links and run ledgers.
> - The second full-cohort run on master `9296c74e8d` failed 15 cases
where the product did the right thing and the harness did not.
> - The causes were three harness classes: links read before the reply
rendered, raw run counts that rejected product-caused follow-up runs,
and keyword graders on free-form replies. Four browser locators also
targeted retired controls.
> - This pull request fixes those harness layers. It does not change the
product.
> - The benefit is that these cases measure the feature again: correct
behaviour passes, and the relevant broken behaviour still fails.

## Linked Issues or Issue Description

**What happened**
In the second full-cohort chat-eval run on master `9296c74e8d`, these
cases failed although the reply and the saved state were correct:
`agents.choose-primary`, `cases.case-history`,
`budgets-costs.set-budget` (links read about 2s before the reply
rendered); `task-coordination.independent-review`,
`cases.update-case-facts`, `routines.manage-other-owner`,
`questions-and-approvals.review-plan` (a legitimate continuation or the
product's completion follow-up exceeded a raw run count or was
cancelled); `cases.cases-disabled`, `inbox-search.find-design`,
`navigation-preferences.broken-link`, `cases.browse-cases` (keyword or
clause graders rejected correct replies); `agent-chat.conference-room`,
`workspaces.inspect-git`, `task-coordination.block-on-design`,
`skills.author-skill` (browser steps used stale settings or controls).

**Expected behavior**
A correct reply and correct saved state pass. A wrong outcome, extra or
duplicate agent work, a substitution or a missing route still fails.

**Steps to reproduce**
Run `runner-full-stack-e2e.yml` with `cadence=manual`, `all=false` and
an `id` list of the cases above against master `9296c74e8d`. Read
`result.json` and `snapshots/` in each cell's evidence.

Refs #20600

## What Changed

- `chat-rendered-links.ts`: `collectConversationLinks` waits (at most
10s, once per reply per page) for the latest saved lead reply to render
when its conversation is open. A page that does not show the
conversation does not wait. `observeInternal` and the flow conversation
read use it.
- `budgets-costs.set-budget`: waits for the rendered reply before it
reads anchors. It fills the budget only after the form shows the saved
cap, and checks that the typed value stays.
- New `chat-catalog-domains/run-roles.ts`: identifies a run by role (a
worker woken by a comment that a named run saved on the task; the CEO's
chat completion follow-up).
- `task-coordination.independent-review`: admits one Morgan continuation
woken by the CEO's authorization comment and one Riley review run. The
review request may come from that continuation's native status decision.
Other runs still fail.
- `cases.update-case-facts`: a phase settles when its CEO turn and at
most one Morgan continuation woken by that turn's task comment finish. A
new `role-bound-runs` check rejects any other run. `terminal-counts`
gets an optional `maximumTerminalStatusCounts`.
- `routines.manage-other-owner`: admits one Morgan run on a task that
the CEO created for Morgan. A schedule change must be Morgan's own write
from that run. The ownership check is structural (a linked owner task,
or a link to the routine or Morgan's page). The keyword checks on the
reply are removed.
- `questions-and-approvals.review-plan`: after the stale-review check,
lets the product's completion follow-up settle before final run custody.
The existing grader already validates it.
- `cases.cases-disabled`: the negation-blind false-claim regex is
removed. The case passes with a reply link to the Experimental settings
page (the scripted person follows it and sees the Cases control) or a
pending choice card. Saved state still proves that nothing was created.
- `inbox-search.find-design`: the per-clause regex is removed. The first
task link in the reply must be the task with the approved revision.
- `navigation-preferences.broken-link`: a link to the similar task fails
only when its label names the original (identifier or exact title). A
separate mention passes.
- `cases.browse-cases`: an excluded case must not be a result row. When
the reply has no result list, every case link is a result, so an
excluded case must not be linked.
- Browser steps: `conference-room` reloads after the fixture enables the
flag. `inspect-git` and `block-on-design` open the panel with "Show
properties" and pick the tab from the side panel launcher.
`author-skill` does not repeat the full inspection in `verifyResult`
when `act` already inspected the same saved version.
- Each changed case has a new `oracleVersion`.
`chat-catalog-descriptors.json` is in sync.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts <119
affected test files> --testTimeout=180000`: 1673 passed, 3 skipped. One
file (`chat-factual-query-input.test.ts`) did not load in the local
worktree because `@paperclipai/plugin-sdk` is not built there. It does
not import changed code. CI runs it.
- `chat-catalog-consistency.test.ts` passes (242 descriptors).
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no pin change.
- `node --test` on the three `chat-full-catalog-testops-*.test.mjs`
files: 40 passed.
- Calibration against retained evidence: the retained replies and run
shapes of cases-disabled, browse-cases, find-design, broken-link,
independent-review, update-case-facts and manage-other-owner now pass.
Broken variants fail (closed case as a result row, earlier design linked
first, original label on the similar task's link, unprompted or
duplicate worker runs, a second review run, a routine change outside the
owner's run, no route link).
- Paid runs on this branch: see the PR comments.

## Risks

- Low risk. Only eval harness code changes.
- The structural graders do not read reply wording. A reply that words
the outcome wrongly but links the right records can pass. A person reads
the reply.
- `navigation-preferences.sidebar-order` is not changed. The shipped
sidebar has no per-project order or collapsible Agents section, so the
case premise needs an owner decision.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 20:24:41 -06:00

69 lines
5.5 KiB
TypeScript

import { chatCatalogOriginalPhaseDeadline } from "../chat-catalog-contract.js";
import { expect } from "@playwright/test";
import { checks } from "./internal-support.js";
import { waitForReply } from "./identity-support.js";
import { waitForRenderedReply } from "./internal-comment-reference.js";
import { runtimeDefinition, runtimeObservation, observeRuntime } from "./runtime-support.js";
/**
* The reply must send the person to the real Conference Room: the scripted human clicks the
* link the reply rendered (no typed-URL fallback) and must land on the board composer in the
* same company. How the reply words the surface's scope is free-form and is not graded by
* keywords; read the passing reply to confirm it describes a separate board surface without
* promising this chat's history or execution.
*/
export function gradeConference(value: unknown) {
const c = checks(); const o = runtimeObservation(value);
c.add("independent-conference-evidence", !!o, "Actual deployment, gate, company route and rendered board surface are required."); if (!o) return c.result();
c.add("supported-local-enabled-context", o.state.deploymentMode === "local_trusted" && o.state.conferenceEnabled === true
&& o.state.agentChatEnabled === true, "Both actual gates are enabled in the supported local-trusted deployment.");
c.add("reply-links-conference-room", o.state.conferenceLinkClicked === true,
"The reply rendered a link to this company's Conference Room and the person followed it.");
c.add("real-company-conference-surface", o.state.conferenceVisible === true && o.state.openedCompanyPrefix === o.state.companyPrefix
&& o.state.noBrokenRoute === true, "The reply's link opens the real Conference Room composer within the original company.");
c.add("navigation-has-no-provider-action", o.state.runsBeforeOpen === 1 && o.state.runsAfterOpen === 1
&& o.state.standingIssueBefore === o.state.standingIssueAfter, "Opening the board surface preserves the ordinary source and starts no hidden execution.");
return c.result();
}
export const runtimeConferenceCases = [runtimeDefinition("agent-chat.conference-room", async ctx => {
const health = await ctx.input.api.get<{ deploymentMode: string }>("/api/health");
if (health.deploymentMode !== "local_trusted") throw new Error("Conference Room scenario requires a genuinely local-trusted target");
await ctx.input.api.patch("/api/instance/settings/experimental", { enableConferenceRoomChat: true });
const settings = await ctx.input.api.get<{ enableConferenceRoomChat: boolean; enableAgentChat: boolean }>("/api/instance/settings/experimental");
// The open page cached the experimental settings before this fixture enabled Conference Room; a person
// arriving after the instance enabled it loads the current settings, so reload before the query is sent.
await ctx.input.page.reload({ waitUntil: "domcontentloaded" });
return { deploymentMode: health.deploymentMode, conferenceEnabled: settings.enableConferenceRoomChat,
agentChatEnabled: settings.enableAgentChat, companyPrefix: ctx.input.fixtures.company.issuePrefix };
}, gradeConference, {
oracleVersion: "paperclip.chat-catalog.runtime-conference.v5",
sourceFiles: ["chat-catalog-domains/runtime-conference.ts", "chat-catalog-domains/runtime-support.ts", "chat-catalog-domains/internal-comment-reference.ts", "chat-scenarios/agent-chat.yaml"],
async act(ctx, state) {
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx)); state.runsBeforeOpen = (await ctx.allRuns()).length; state.standingIssueBefore = ctx.issue()?.id;
// gha-42939717023: the saved reply linked /COM/board-chat, but the anchors were read before it rendered.
await waitForRenderedReply(ctx, { leadId: ctx.input.fixtures.agent.id, runs: await ctx.allRuns() });
const o = await observeRuntime(ctx, state);
// Find the link as rendered in the thread: the UI may rewrite a reply's href.
const anchors = ctx.input.page.locator('[id^="comment-"] a[href]');
const rendered = await anchors.evaluateAll(nodes => nodes.map(node => (node as HTMLAnchorElement).href));
const origin = new URL(ctx.input.page.url()).origin;
const index = rendered.findIndex(href => { try { const url = new URL(href);
return url.origin === origin && url.pathname === `/${state.companyPrefix}/board-chat`; } catch { return false; } });
state.conferenceLinkClicked = index >= 0;
await ctx.input.evidence("runtime-conference-links.json", { companyId: o.companyId, rendered, replyLinks: o.links });
if (index < 0) return;
const source = ctx.input.page.url();
await anchors.nth(index).click();
await expect(ctx.input.page.getByPlaceholder("Ask anything about your organization...")).toBeVisible();
await expect(ctx.input.page.getByText("Conference Room", { exact: true }).first()).toBeVisible();
state.conferenceVisible = true;
const parts = new URL(ctx.input.page.url()).pathname.split("/"); state.openedCompanyPrefix = parts[1];
state.noBrokenRoute = parts.at(-1) === "board-chat";
await ctx.input.evidence("runtime-conference-opened.json", { companyId: o.companyId, url: ctx.input.page.url(), composerVisible: true });
state.runsAfterOpen = (await ctx.allRuns()).length;
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
await expect(ctx.input.page.getByTestId("task-chat-composer-input")).toBeVisible();
await ctx.refreshIssue(); state.standingIssueAfter = ctx.issue()?.id;
},
async verifyResult(ctx) { await expect(ctx.input.page.getByTestId("task-chat-composer-input")).toBeVisible(); },
})];