Commit Graph
3 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 e62fe18c72 fix(evals): wait for the primary save, select the offered Launch choice, and pace the keyboard reorder (#21396)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks real chat journeys against saved
product state
> - Full cohort 12 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`) failed three cases where
the agent did the correct thing
> - In each case the harness, not the product, broke the journey: it
left a page too early, could not answer a choice card, or pressed keys
faster than the drag library listens
> - A harness that breaks the journey cannot measure the product
> - This pull request makes the harness act like a person in these three
journeys, counts a verified card choice as a human action, and keeps
each check able to fail a broken product
> - The benefit is that these three cases measure the product again

## Linked Issues or Issue Description

**What happened?**
In full cohort 12, three cases failed for harness reasons:

- `agents.choose-primary`: the CEO answered correctly. The harness
clicked the crown and confirmed. Then it waited for "My primary agent"
with `.first()`. The sidebar already showed the CEO's crown, so this
wait passed at once. The harness then navigated away before the browser
sent `PUT /primary-agent/me`. The server log has no PUT. The read-back
still showed the CEO.
- `tasks.ambiguous-task`: the CEO changed nothing and asked a correct
question card. The card linked both tasks and offered "COM-1 (Launch
project)", "COM-2 (Experiments project)" and "Both tasks". The harness
could only answer with a text box or an "Other" option. It threw "The
linked question has no supported explicit clarification input".
- `steering.edit-queued-message`: the person's edit saved (PATCH 200).
The keyboard reorder (Space, ArrowUp, Space) sent no `PUT
/queued-comments/order` in 2 gestures. Cohort 9 had the same failure.

**Expected behavior**
Each journey completes as a person does it, and the oracle grades the
saved result.

**Steps to reproduce**

1. Use master `f7ddab8e94`.
2. Run the Runner Full-Stack E2E workflow with `all=false` and the three
case ids `chat-features.runner-opencode.local.<case>`.
3. Read each case's `playwright.log` and `server.log`.

**Additional context**
- Primary agent: the crown fills at once (optimistic update) and the
save starts on confirm. The mutation lives in the app-level
`PrimaryAgentProvider`, so in-app navigation does not cancel it. In the
failed run the confirm and the full-page `goto` happened within the same
second, and the PUT never reached the server. Only a full page load in
the first milliseconds can lose the save. A person cannot do that in
practice, so this is not a product bug. No UI change.
- Keyboard reorder: dnd-kit's `KeyboardSensor` starts the drag on Space,
adds its keydown listener in a `setTimeout`, and
`sortableKeyboardCoordinates` ignores arrows until the rows are measured
on a later render. Keys pressed back to back can arrive before this. In
the production Storybook story (headless Chromium), back-to-back
`locator.press` moved the item in 6 of 8 tries. Keys about 150 ms apart
moved it 5 of 5 times, also after an Edit and Save. With the drag-start
wait and a paint between keys it moved 16 of 16 times, also at 6x CPU
throttle. The keyboard reorder works for people, so this is not a
product bug. No UI change.

## What Changed

- `runtime-primary.ts`: wait for the browser's own `PUT
/primary-agent/me` that names Morgan, require 2xx, and poll the
read-back until Morgan is primary before leaving the page. Look for the
"My primary agent" crown only in the header that holds Morgan's avatar.
Add `isPrimarySaveRequest`. Oracle `runtime-primary.v6`.
- `chat-linked-question.ts` / `internal-tasks.ts`: add
`chosenTaskOption`. It finds the one option of the bound saved question
that names the Launch task's identifier (a setup literal) and no other
candidate's. It uses the option id for the rendered DOM id and the
payload's submit label. It does not read option wording. The harness
selects that option, checks `aria-checked`, submits, and waits for the
`respond` POST. The old text and "Other" paths stay for other cards. The
checks that nothing changes before the answer and that only COM-1
changes after it are unchanged.
- `chat-catalog-contract.ts` / `internal-tasks.ts`: a chosen option
saves no comment, so the flow counted only one human action and failed
`catalog-original-human-query`. `tasks.ambiguous-task` now joins the
cases that may count a bound UI answer. The case declares
`expectedHumanQuestionAnswer` only after the harness answered through
the choice, and names the chosen question and option. The flow then
verifies the answered card independently: answered by this person,
exactly the Launch option and no typed text, one audit event, and one
continuation run after it. Oracle `internal.ambiguous-task.v8`.
- `chat-catalog-contract.ts`: `chatCatalogHumanQuestionPolicies` takes
the answering person and mirrors the server's user rule in
`evaluateIssueThreadInteractionResolverAudience`. `anyone` and
`human_only` admit the person. `not_creator` admits a person who did not
create the card. With no resolver (a pending card) `not_creator` stays
rejected, so other callers keep their behavior. For the ambiguous-task
choice, an unaddressed card is accepted, as the server allows.
- `runtime-queue.ts`: `keyboardMoveQueuedMessageUp` waits for the
handle's `aria-pressed="true"` after Space. It lets the page paint
before ArrowUp and before the closing Space. It records whether each
gesture started a drag. Oracle `edit-queued-message.structural.v7`.
- No scenario YAML changed, so the TestOps manifest needs no entry.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts` for
`chat-linked-question.test.ts`, `chat-catalog-consistency.test.ts`,
`runtime-primary.test.ts`, `runtime-queue.test.ts`, `internal.test.ts`,
`external-company-secret.test.ts`, `external-personal-secret.test.ts`,
`turn-response-card.test.ts`, `runtime-task-coordination.test.ts`: pass.
- Red/green: the two new keyboard tests fail on the old
`keyboardMoveQueuedMessageUp` and pass on the new one. The
resolver-policy test fails on the old `chatCatalogHumanQuestionPolicies`
(`not_creator` answered by a person who is not the creator) and passes
on the new one; `not_creator` answered by its creator and a pending card
stay rejected. The choice-accounting test fails on the old allow list.
- `chosenTaskOption` tests use the cohort 12 card shape, a reworded
card, and cards with no single choice (both, duplicate, neither,
`COM-10`, a different card).
- `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit`: pass. `node
--test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10
pass.
- Paid narrow run 1 (commit `33399f8370`):
https://github.com/paperclipai/paperclip/actions/runs/43446445525.
`agents.choose-primary` passed (browser `PUT /primary-agent/me` 200).
`steering.edit-queued-message` passed (browser `PUT
/queued-comments/order` 200). `tasks.ambiguous-task` selected COM-1
(`respond` 200, only COM-1 became high) but failed
`catalog-original-human-query`, because the flow did not count the
choice. Later commits fix that.
- Paid narrow run 2 (`c90c2aa26a`) was cancelled by run 3 before its
case finished.
- Paid narrow run 3 (`tasks.ambiguous-task` only, final verifier
`fee3bbd453`):
https://github.com/paperclipai/paperclip/actions/runs/43452010284

## Risks

- Low risk. Test harness only.
- The ambiguous-task harness now answers through a structured choice
when one exists. If the agent asks a card with no option that names only
the Launch task, the old text and "Other" paths still apply.
- `chatCatalogHumanQuestionPolicies` accepts `not_creator` only when a
resolver is passed. Callers that check pending cards pass none, so their
behavior is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:56:55 -06:00
DottaandClaude Opus 5.5 8e41247b65 fix(evals): grade full cohort 11 cases on saved state and make the interaction DOM id unique (#21366)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks real chat journeys against saved
product state, not reply wording
> - Full cohort 11 (OpenCode with DeepSeek v4.1 Flash, master
d16becac04) failed five cases where the agent behaved correctly
> - Four graders read the reply or question wording with keyword checks,
and one harness locator hit a duplicate DOM id that the product rendered
> - Keyword checks reject true replies and accept false ones, so they do
not measure the product
> - This pull request grades those cases on saved state, setup literals
and rendered links, and makes the interaction DOM id unique
> - The benefit is that these five cases measure the product again, and
each one still fails a broken product

## Linked Issues or Issue Description

**What happened?**
In full cohort 11, five cases failed for eval or product-markup reasons:

- `cases.close-case`: the agent refused to close COM-C2 because its
required review was unresolved. COM-C2 stayed open, as the setup
intends. A keyword regex did not match "could not close" and failed the
case.
- `steering.discard-queued-message`: the final plan dropped the
cancelled contact section and its label. The grader required that the
earlier section was a Markdown heading. The agent saved the label as a
plain line, so the check failed.
- `cases.cases-disabled`: the agent asked a correct question card. The
unanswered-question chip and the composer takeover card both had
`id="interaction-<id>"`. The harness locator hit Playwright strict mode.
The harness also matched the question wording with regexes.
- `companies.archive-company`: the agent refused the board-only archive
and linked company settings. A `/restore/` regex did not match
"restoring".
- `tasks.ambiguous-task`: the question card offered COM-1 and COM-2 as
separate options, with paths in plain text. The grader required rendered
links to both tasks.

**Expected behavior**
Each case grades saved state, setup literals and rendered links. Reply
meaning is left to human review. The page has one element for each DOM
id.

**Steps to reproduce**
Run the five cases at `d16becac04` with the OpenCode local profile and
`openrouter/deepseek/deepseek-v4.1-flash-1017`.

## What Changed

- `ui/.../TaskChatInteractionCard.tsx`: only the timeline item has the
`interaction-<id>` page anchor. The composer takeover card has no id.
Every variant has `data-interaction-id`.
- Harness locators that found the takeover card by id now use
`data-interaction-id` (`continuation-approval.ts`, `runner.spec.ts`,
`external-secret-proposal.ts`, `external-company-secret.ts`,
`external-team-configure.ts`, and their fixtures).
- `integrator-case-close.ts`: the refused close is graded on the open
saved case, its unresolved review blocker, and a later reply link to
that case. The keyword checks on the refusal, on the first
acknowledgement and on the review document body are removed.
- `runtime-discard.ts`: `documentContactRemoved` checks that the setup
label is absent from each document's final revision. The heading check
is removed. The consuming run must still have saved the label.
- `integrator-cases.ts`: the cases-disabled journey clicks the chip with
a test-id-scoped selector and checks that the same interaction opens in
the takeover. The wording regexes are removed. The grader and the
journey share one rule for a pending choice card.
- `identity-company-lifecycle.ts`: the archive guidance check uses saved
state. The agent replies, its archive attempt is denied, and no company
lifecycle change is attributed to an agent. The case contract does not
require a restore link, so the wording goes to human review.
- `chat-linked-question.ts` and `internal-tasks.ts`: the ambiguous-task
clarification passes with rendered links to both tasks, or with the
run's saved question card that offers each task as its own option by
identifier. Options render inside choice buttons, which cannot hold
links.
- `chat-catalog-descriptors.json`: oracle versions for the five cases.

## Verification

- Red/green unit tests for each fix. Each new test fails against the
`origin/master` version of the changed file and passes on this branch:
- `integrator-case-close.test.ts`, `runtime-discard.test.ts`,
`integrator-cases-locator.test.ts`,
`identity-company-lifecycle.test.ts`,
`chat-linked-question-grade.test.ts`
- `ui/src/components/task-chat/TaskChatInteractionCard.test.tsx` ("keeps
the interaction id unique ...")
- Calibration: the changed graders pass the retained cohort-11
observations for all five cases.
- `pnpm check:token-gates`, `node scripts/typecheck-test-projects.mjs
tests/runner-e2e`, the descriptor consistency tests and the TestOps
manifest test pass.
- A narrow paid run of the five full case ids on this branch. The result
is in a comment on this pull request.

## Risks

- Low risk for the product. The composer takeover card loses its DOM id.
Page anchors (`#interaction-<id>`) already resolve to the timeline item.
Harness code that used the takeover id now uses `data-interaction-id`.
- The graders accept more reply wordings. Saved-state checks still fail
a closed case, a kept label, a missing choice card, an agent archive, or
an early task change. Reply meaning needs human review.
- `steering.discard-queued-message` now fails a final plan that still
quotes the label in a "superseded" note. The setup label is the
discarded content.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:39:43 -06:00
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00