Commit Graph
5 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 e879a24293 test(evals): check the wrong-resolver reply links Alex's request and task-link labels match their targets (#21028)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys against real agents
and grades saved product state and rendered links, never reply wording
> - Full chat-eval cohort 4 had two cases whose graders passed replies
with a structural flaw that the grader could see but did not check
> - `connection-setup.wrong-person-resolves` promises that chat explains
whose action is required, but its grader had no reply check at all
> - `navigation-preferences.find-project-view` passed a reply with a
link labelled COM-2 that opened COM-1
> - This pull request adds two structural reply checks, with calibration
tests for valid, reworded-valid and broken replies
> - The benefit is that a reply that does not point to the request, or
that sends a person to the wrong task, fails the eval

## Linked Issues or Issue Description

**What happened?**
- `connection-setup.wrong-person-resolves`: another member said "I will
finish Alex's connection request for them". The product refused that
member (403 on setup options, phase, complete and decline) and kept
Alex's card pending. The grader checked only that. The reply's content
was not checked in any form.
- `navigation-preferences.find-project-view`: the saved reply contained
`[COM-2 Launch open website](/COM/issues/COM-1)`. The grader passed it.

**Expected behavior**
- The wrong-resolver reply points the person at Alex's actual pending
request. The card on that page says "Waiting for Alex", which is whose
action is required. The grader checks the link from saved state, and the
browser step follows the reply's own link to that card.
- Every task link in the find-project-view reply opens the task that its
visible identifier names.

**Steps to reproduce**
1. Run the two cases (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`) on master at f1aef0cb58.
2. Read the saved replies and the grader checks in the evidence.

## What Changed

- `external-wrong-resolver.ts`: new check
`reply-links-alex-pending-request`. The reply must render a same-origin
link to the related task that holds Alex's card (`/<company>/issues/<id
or identifier>`). `verifyResult` now follows that rendered link (not a
constructed URL) and confirms the "Waiting for Alex" card. Setup records
the company prefix. Oracle `paperclip.chat-catalog.external.v2` -> `v3`.
Reply wording is still not graded.
- `chat-rendered-links.ts`: new shared helper
`mislabeledIssueLinks(text, identifierFor?)`. It returns inline Markdown
links whose label names a task identifier (same prefix as the target)
other than the task the link opens. Labels without such a token, and
non-task targets, are not judged. Task-id targets are compared through
an optional resolver.
- `identity-navigation-preferences.ts`: find-project-view adds
`task-links-match-their-labels` with that helper. Oracle
`paperclip.chat-catalog.identity.v1` ->
`paperclip.chat-catalog.identity.find-project-view.v2`;
`chat-rendered-links.ts` joins its source files.
- `chat-catalog-descriptors.json`: the two descriptors carry the new
oracle versions and source files.
- The scenario YAML does not change, so the testops manifest has no
`changedFields` entry.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/external-wrong-resolver.test.ts
tests/runner-e2e/chat-rendered-links.test.ts
tests/runner-e2e/chat-catalog-domains/identity.test.ts`: 92 tests pass.
- Calibration, wrong resolver: valid (links CMP-1), reworded valid
(links by task id, absolute same-origin link) pass; no link, another
task, another company prefix, another origin, and a sub-path fail.
Red/green: with the master grader, the "does not point at Alex's
request" test fails.
- Calibration, labels: valid, reworded (label without an identifier,
document anchor, task-id target resolved), and a different-prefix token
pass; the recorded cohort-4 link `[COM-2 Launch open
website](/COM/issues/COM-1)` fails. The helper flags exactly that link
in the retained cohort-4 reply.
- Retained cohort-4 wrong-resolver reply links `/COM/issues/COM-1` (the
related task), so it passes the new link check. Its wording ("completing
it there is the right move") is a guidance problem, fixed in #21027, not
something saved state can grade.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-campaign chat-catalog`: 3856 tests pass. `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
`tsc -p tests/runner-e2e/tsconfig.json` passes.
- A paid run of both cases with the product fixes is linked in a comment
below.

## Risks

- A valid reply that names Alex but links nothing now fails
wrong-person-resolves. The case contract says chat explains whose action
is required; the card is where Paperclip shows it, so pointing to it is
the structural form of that promise.
- The label check could flag a label that intentionally names two tasks
of the same company and links only one. That reply would still be
misleading, so failing it is acceptable.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:44:15 -06:00
DottaandClaude Opus 5.5 bf18b348c5 fix(evals): wait for saved, rendered replies and settled forms, and step through multi-question cards (#21009)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog drives real chat journeys in a browser and
grades saved product state
> - In full cohort 3, four cases acted on the page before the page was
ready, or did not step through a card
> - The product was correct in each case, so these failures hide real
product failures
> - This pull request makes the harness wait for the saved, rendered
reply and for a settled form, and step through multi-question cards
> - The benefit is fewer false failures, with the same product
requirements

## Linked Issues or Issue Description

Refs #21000
Refs #21008

**What happened?**
- `companies.fast-responses`, `agents.choose-primary`,
`budgets-costs.set-budget`: the server recorded the chat run as finished
0.2 to 1.3 seconds before it saved the reply comment. The harness read
the reply links right after the run finished. It found no reply, so it
found no links.
- `budgets-costs.set-budget`: the budget card remounted after the
harness typed the new amount. The remount reset the draft, so the Update
budget button stayed disabled until the click timed out.
- `questions-and-approvals.necessary-question`: the agent asked the
required product-name question and one optional question. The card
showed Next on its first page. The harness only clicked Submit answers.
The grade also required exactly one question on the card.

**Expected behavior**
The harness reads a turn's links after its reply is saved and rendered.
It saves the budget only on a settled form. It pages through a card,
answers its one required question and skips optional ones.

**Steps to reproduce**
Run the three link cases when the reply is saved after the run finishes.
Run set-budget when the budget card reloads after typing. Run
necessary-question when the card has an optional second question.

## What Changed

- `chat-rendered-links.ts`: new shared `waitForSavedTurnReplyRender`. It
waits (bounded, 15 s) until each finished lead run's reply is saved,
then uses the existing render wait for the latest reply. It returns
`unsaved` when no reply arrives. The caller then fails with its own
error.
- `identity-support.ts`: new `waitForTurnReplyLinks` uses that wait for
the source conversation. `companies.fast-responses` (v3),
`agents.choose-primary` (v4) and `budgets-costs.set-budget` (v3) call it
before they read reply links.
- `budgets-costs.set-budget`: `settledInput` resolves when the same
input node stays on the page for one second. The harness types and saves
on that node. If the card still remounts and drops the amount, it types
it again, at most three times.
- `questions-and-approvals.necessary-question` (v2): the harness pages
through a multi-question card. It answers the one required question and
clicks Skip on each optional question. The grade now requires exactly
one required question, and that question must ask for the product name
as free text. Other questions must be optional. An optional necessary
question, a second required question, or a required question about
something else still fails.
- Descriptors are updated. No scenario yaml changed, so the TestOps
manifest does not change.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-rendered-links.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts
tests/runner-e2e/chat-catalog-fingerprint.test.ts
tests/runner-e2e/chat-catalog-domains/identity.test.ts`: 100 tests pass.
- New tests: a reply saved after the run finished is waited for and then
rendered; a run that saves no reply returns `unsaved` within its bound.
The necessary-question grade accepts the recorded card (required name
plus optional follow-up) and rejects the broken cards above.
- `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
reports no change.
- One paid run on a combined branch with #21008. The result is in a
comment on this pull request.

## Risks

- A turn whose run saves no reply now waits up to 15 seconds before the
case continues. Only the three named cases call the wait.
- `settledInput` marks the input node with a `data-e2e-settle-mark`
attribute to detect a remount. It does not change product behavior.
- #21000 changes the render-wait expiry in the same file. Whichever
merges second rebases.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 21:33:32 -06:00
DottaandClaude Opus 5.5 65ff80a4bb fix(evals): grade chat acknowledgements, saved replies and links on saved state (#21000)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys against real agents
and grades saved product state
> - A review of full cohort 3 found three cases whose oracle passed
wrong behavior, and eight passes whose final evidence saved no rendered
links
> - `companies.agent-observability` passed although the agent denied the
saved collector. `inbox-search.show-my-work` counted links that existed
only in run-stream prose. `instance-operations.restart-service` accepted
a reply whose only link was the `/<prefix>/` placeholder
> - The final evidence read ran on a page that had not rendered the
conversation yet, so it saved no links
> - This pull request grades those cases on saved-state reads, the saved
reply and resolved links, and reads final evidence from the rendered
conversation
> - The benefit is that these cases fail when the product is wrong and
keep their links when it is right

## Linked Issues or Issue Description

Refs #20998
Refs #20999

**What happened?**
- `companies.agent-observability`: the board saved and tested a
collector. Asked to acknowledge it, the agent said there was no endpoint
or credentials. The oracle graded only human-side state, so it passed.
- `inbox-search.show-my-work`: the saved reply said "Answered above"
over nothing. The links the oracle counted came from run-stream prose
that the bubble also shows.
- `instance-operations.restart-service`: the reply's only action link
was `/<prefix>/company/settings/instance`. The oracle did not check
links.
- Eight cohort-3 passes (`access.other-company`, `agents.agent-api-key`,
`assistant-connections.revoke-personal-grant`,
`budgets-costs.resolve-budget`, `recovery.retry-exhausted`,
`steering.stop-versus-pause`, `task-coordination.pause-tree`,
`teams.install-team`) saved a final observation with no links and a
failed grade. The graded reload had the links. The final evidence read
ran after `verifyResult` left the page mid-load, and an expired render
wait from an earlier document made the read instant.

**Expected behavior**
Each oracle fails the observed wrong behavior and passes valid, reworded
behavior. Final evidence holds the links that the conversation renders.

**Steps to reproduce**
Grade the retained cohort-3 observations with the old oracles. They
pass. Grade them with these oracles. The three wrong cases fail.

## What Changed

- `companies.agent-observability` (oracle v4): each acknowledgement turn
must read the saved trace export status successfully. The read is bound
to the company by its redaction-safe `call_api` receipt. A reply must
also render a link to `/<prefix>/company/settings` for the fixture
company. The reply's wording is not matched.
- `inbox-search.show-my-work` (oracle v5): actionable links must be
carried by the saved reply: a Markdown link target in the saved body, or
a task identifier written in it, which the conversation links. New
shared helper `savedReplyLinks`.
- `instance-operations.restart-service` (oracle v3): new check
`reply-links-resolve` rejects any rendered or saved link target that
keeps a template placeholder (`<...>` or `{...}`). New helpers
`markdownLinkTargets` and `unresolvedLinkTarget` in
`chat-rendered-links.ts`.
- Link capture: the final success-path evidence read reopens the
rendered conversation first. An expired render wait now belongs to its
loaded document, not to the Playwright page.
- Updated `chat-catalog-descriptors.json` for the three oracle versions
and source files. No scenario yaml changed, so the TestOps manifest does
not change.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/identity-observability.test.ts
tests/runner-e2e/chat-catalog-domains/internal.test.ts
tests/runner-e2e/chat-catalog-domains/privileged-oracles.test.ts
tests/runner-e2e/chat-catalog-domains/privileged-restart-history.test.ts
tests/runner-e2e/chat-rendered-links.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts
tests/runner-e2e/chat-catalog-fingerprint.test.ts`
- Calibration tests for each case: a valid result, a reworded valid
result, and the broken cohort-3 behavior.
- Red/green: the new render-wait test fails on the old wait logic and
passes with this change.
- `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes. `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no
change.
- A paid run on a combined branch with the two product fixes is recorded
in a comment on this pull request.

## Risks

- `companies.agent-observability` needs the agent-readable status route
from #20998. Without it, the case fails, which is correct.
- The show-my-work check needs links in the saved reply. Before #20999,
an OpenCode reply that left its answer in the run stream fails this
case.
- The final evidence read can wait up to 30 seconds to reopen the
conversation. The read is still bounded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:49:38 -06:00
DottaandClaude Opus 5.5 cbd636d652 fix(evals): wait for rendered replies, bound runs by role and grade chat outcomes structurally (#20685)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat-feature eval catalog runs real chat journeys and grades
saved state, rendered links and run ledgers.
> - The second full-cohort run on master `9296c74e8d` failed 15 cases
where the product did the right thing and the harness did not.
> - The causes were three harness classes: links read before the reply
rendered, raw run counts that rejected product-caused follow-up runs,
and keyword graders on free-form replies. Four browser locators also
targeted retired controls.
> - This pull request fixes those harness layers. It does not change the
product.
> - The benefit is that these cases measure the feature again: correct
behaviour passes, and the relevant broken behaviour still fails.

## Linked Issues or Issue Description

**What happened**
In the second full-cohort chat-eval run on master `9296c74e8d`, these
cases failed although the reply and the saved state were correct:
`agents.choose-primary`, `cases.case-history`,
`budgets-costs.set-budget` (links read about 2s before the reply
rendered); `task-coordination.independent-review`,
`cases.update-case-facts`, `routines.manage-other-owner`,
`questions-and-approvals.review-plan` (a legitimate continuation or the
product's completion follow-up exceeded a raw run count or was
cancelled); `cases.cases-disabled`, `inbox-search.find-design`,
`navigation-preferences.broken-link`, `cases.browse-cases` (keyword or
clause graders rejected correct replies); `agent-chat.conference-room`,
`workspaces.inspect-git`, `task-coordination.block-on-design`,
`skills.author-skill` (browser steps used stale settings or controls).

**Expected behavior**
A correct reply and correct saved state pass. A wrong outcome, extra or
duplicate agent work, a substitution or a missing route still fails.

**Steps to reproduce**
Run `runner-full-stack-e2e.yml` with `cadence=manual`, `all=false` and
an `id` list of the cases above against master `9296c74e8d`. Read
`result.json` and `snapshots/` in each cell's evidence.

Refs #20600

## What Changed

- `chat-rendered-links.ts`: `collectConversationLinks` waits (at most
10s, once per reply per page) for the latest saved lead reply to render
when its conversation is open. A page that does not show the
conversation does not wait. `observeInternal` and the flow conversation
read use it.
- `budgets-costs.set-budget`: waits for the rendered reply before it
reads anchors. It fills the budget only after the form shows the saved
cap, and checks that the typed value stays.
- New `chat-catalog-domains/run-roles.ts`: identifies a run by role (a
worker woken by a comment that a named run saved on the task; the CEO's
chat completion follow-up).
- `task-coordination.independent-review`: admits one Morgan continuation
woken by the CEO's authorization comment and one Riley review run. The
review request may come from that continuation's native status decision.
Other runs still fail.
- `cases.update-case-facts`: a phase settles when its CEO turn and at
most one Morgan continuation woken by that turn's task comment finish. A
new `role-bound-runs` check rejects any other run. `terminal-counts`
gets an optional `maximumTerminalStatusCounts`.
- `routines.manage-other-owner`: admits one Morgan run on a task that
the CEO created for Morgan. A schedule change must be Morgan's own write
from that run. The ownership check is structural (a linked owner task,
or a link to the routine or Morgan's page). The keyword checks on the
reply are removed.
- `questions-and-approvals.review-plan`: after the stale-review check,
lets the product's completion follow-up settle before final run custody.
The existing grader already validates it.
- `cases.cases-disabled`: the negation-blind false-claim regex is
removed. The case passes with a reply link to the Experimental settings
page (the scripted person follows it and sees the Cases control) or a
pending choice card. Saved state still proves that nothing was created.
- `inbox-search.find-design`: the per-clause regex is removed. The first
task link in the reply must be the task with the approved revision.
- `navigation-preferences.broken-link`: a link to the similar task fails
only when its label names the original (identifier or exact title). A
separate mention passes.
- `cases.browse-cases`: an excluded case must not be a result row. When
the reply has no result list, every case link is a result, so an
excluded case must not be linked.
- Browser steps: `conference-room` reloads after the fixture enables the
flag. `inspect-git` and `block-on-design` open the panel with "Show
properties" and pick the tab from the side panel launcher.
`author-skill` does not repeat the full inspection in `verifyResult`
when `act` already inspected the same saved version.
- Each changed case has a new `oracleVersion`.
`chat-catalog-descriptors.json` is in sync.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts <119
affected test files> --testTimeout=180000`: 1673 passed, 3 skipped. One
file (`chat-factual-query-input.test.ts`) did not load in the local
worktree because `@paperclipai/plugin-sdk` is not built there. It does
not import changed code. CI runs it.
- `chat-catalog-consistency.test.ts` passes (242 descriptors).
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no pin change.
- `node --test` on the three `chat-full-catalog-testops-*.test.mjs`
files: 40 passed.
- Calibration against retained evidence: the retained replies and run
shapes of cases-disabled, browse-cases, find-design, broken-link,
independent-review, update-case-facts and manage-other-owner now pass.
Broken variants fail (closed case as a result row, earlier design linked
first, original label on the similar task's link, unprompted or
duplicate worker runs, a second review run, a routine change outside the
owner's run, no route link).
- Paid runs on this branch: see the PR comments.

## Risks

- Low risk. Only eval harness code changes.
- The structural graders do not read reply wording. A reply that words
the outcome wrongly but links the right records can pass. A person reads
the reply.
- `navigation-preferences.sidebar-order` is not changed. The shipped
sidebar has no per-project order or collapsible Agents section, so the
case premise needs an owner decision.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 20:24:41 -06:00
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00