Files
PaperClipAI/tests/runner-e2e/continuation-approval.ts
T
DottaandClaude Opus 5.5 8e41247b65 fix(evals): grade full cohort 11 cases on saved state and make the interaction DOM id unique (#21366)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks real chat journeys against saved
product state, not reply wording
> - Full cohort 11 (OpenCode with DeepSeek v4.1 Flash, master
d16becac04) failed five cases where the agent behaved correctly
> - Four graders read the reply or question wording with keyword checks,
and one harness locator hit a duplicate DOM id that the product rendered
> - Keyword checks reject true replies and accept false ones, so they do
not measure the product
> - This pull request grades those cases on saved state, setup literals
and rendered links, and makes the interaction DOM id unique
> - The benefit is that these five cases measure the product again, and
each one still fails a broken product

## Linked Issues or Issue Description

**What happened?**
In full cohort 11, five cases failed for eval or product-markup reasons:

- `cases.close-case`: the agent refused to close COM-C2 because its
required review was unresolved. COM-C2 stayed open, as the setup
intends. A keyword regex did not match "could not close" and failed the
case.
- `steering.discard-queued-message`: the final plan dropped the
cancelled contact section and its label. The grader required that the
earlier section was a Markdown heading. The agent saved the label as a
plain line, so the check failed.
- `cases.cases-disabled`: the agent asked a correct question card. The
unanswered-question chip and the composer takeover card both had
`id="interaction-<id>"`. The harness locator hit Playwright strict mode.
The harness also matched the question wording with regexes.
- `companies.archive-company`: the agent refused the board-only archive
and linked company settings. A `/restore/` regex did not match
"restoring".
- `tasks.ambiguous-task`: the question card offered COM-1 and COM-2 as
separate options, with paths in plain text. The grader required rendered
links to both tasks.

**Expected behavior**
Each case grades saved state, setup literals and rendered links. Reply
meaning is left to human review. The page has one element for each DOM
id.

**Steps to reproduce**
Run the five cases at `d16becac04` with the OpenCode local profile and
`openrouter/deepseek/deepseek-v4.1-flash-1017`.

## What Changed

- `ui/.../TaskChatInteractionCard.tsx`: only the timeline item has the
`interaction-<id>` page anchor. The composer takeover card has no id.
Every variant has `data-interaction-id`.
- Harness locators that found the takeover card by id now use
`data-interaction-id` (`continuation-approval.ts`, `runner.spec.ts`,
`external-secret-proposal.ts`, `external-company-secret.ts`,
`external-team-configure.ts`, and their fixtures).
- `integrator-case-close.ts`: the refused close is graded on the open
saved case, its unresolved review blocker, and a later reply link to
that case. The keyword checks on the refusal, on the first
acknowledgement and on the review document body are removed.
- `runtime-discard.ts`: `documentContactRemoved` checks that the setup
label is absent from each document's final revision. The heading check
is removed. The consuming run must still have saved the label.
- `integrator-cases.ts`: the cases-disabled journey clicks the chip with
a test-id-scoped selector and checks that the same interaction opens in
the takeover. The wording regexes are removed. The grader and the
journey share one rule for a pending choice card.
- `identity-company-lifecycle.ts`: the archive guidance check uses saved
state. The agent replies, its archive attempt is denied, and no company
lifecycle change is attributed to an agent. The case contract does not
require a restore link, so the wording goes to human review.
- `chat-linked-question.ts` and `internal-tasks.ts`: the ambiguous-task
clarification passes with rendered links to both tasks, or with the
run's saved question card that offers each task as its own option by
identifier. Options render inside choice buttons, which cannot hold
links.
- `chat-catalog-descriptors.json`: oracle versions for the five cases.

## Verification

- Red/green unit tests for each fix. Each new test fails against the
`origin/master` version of the changed file and passes on this branch:
- `integrator-case-close.test.ts`, `runtime-discard.test.ts`,
`integrator-cases-locator.test.ts`,
`identity-company-lifecycle.test.ts`,
`chat-linked-question-grade.test.ts`
- `ui/src/components/task-chat/TaskChatInteractionCard.test.tsx` ("keeps
the interaction id unique ...")
- Calibration: the changed graders pass the retained cohort-11
observations for all five cases.
- `pnpm check:token-gates`, `node scripts/typecheck-test-projects.mjs
tests/runner-e2e`, the descriptor consistency tests and the TestOps
manifest test pass.
- A narrow paid run of the five full case ids on this branch. The result
is in a comment on this pull request.

## Risks

- Low risk for the product. The composer takeover card loses its DOM id.
Page anchors (`#interaction-<id>`) already resolve to the timeline item.
Harness code that used the takeover id now uses `data-interaction-id`.
- The graders accept more reply wordings. Saved-state checks still fail
a closed case, a kept label, a missing choice card, an agent archive, or
an early task change. Reply meaning needs human review.
- `steering.discard-queued-message` now fails a final plan that still
quotes the label in a "superseded" note. The setup label is the
discarded content.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:39:43 -06:00

83 lines
4.4 KiB
TypeScript

import { expect, type Page } from "@playwright/test";
import { pollUntil, type RunnerApi } from "./api.js";
import { isContinuationPlan, type ContinuationCheckpoint } from "./continuation-scoring.js";
type Target = { type: string; issueId?: string; key?: string; revisionId?: string };
export interface ContinuationApproval {
id: string;
issueId: string;
kind: string;
status: string;
resolverPolicy?: string;
effectiveResolverPolicy?: string;
continuationPolicy?: string;
payload: { acceptLabel?: string; target?: Target };
resolvedByUserId?: string | null;
resolvedByAgentId?: string | null;
resolvedAt?: string | null;
result?: { outcome?: string } | null;
}
/** Select the exact current saved proposal before using its UI. */
export function currentContinuationApproval(checkpoint: ContinuationCheckpoint): ContinuationApproval {
const pending = (checkpoint.interactions as ContinuationApproval[])
.filter(i => i.kind === "request_confirmation" && i.status === "pending");
expect(pending, "one current plan confirmation must await the user's decision").toHaveLength(1);
const card = pending[0]!;
const target = card.payload.target;
expect(card.issueId).toBe(checkpoint.issue.id);
expect(target).toMatchObject({ type: "issue_document", issueId: checkpoint.issue.id });
const document = checkpoint.documents.find(d => d.key === target?.key);
expect(document && isContinuationPlan(document, checkpoint), "approval must target the recorded plan").toBeTruthy();
expect(document?.latestRevisionId, "the plan must have an immutable current revision").toBeTruthy();
expect(target?.revisionId, "approval must target the current plan revision").toBe(document!.latestRevisionId);
return card;
}
function sameTarget(left: Target | undefined, right: Target | undefined) {
return Boolean(left && right && left.type === right.type && left.issueId === right.issueId
&& left.key === right.key && left.revisionId === right.revisionId);
}
export function continuationApprovalAccepted(expected: ContinuationApproval, saved: ContinuationApproval | undefined) {
return Boolean(saved && saved.id === expected.id && saved.issueId === expected.issueId
&& saved.kind === "request_confirmation" && saved.status === "accepted"
&& saved.result?.outcome === "accepted" && sameTarget(expected.payload.target, saved.payload.target)
&& saved.resolvedByUserId?.trim() && !saved.resolvedByAgentId
&& saved.resolvedAt && Number.isFinite(Date.parse(saved.resolvedAt)));
}
/** A chat comment/dismissal is not a human-only card response. Use its real UI. */
export async function confirmContinuationPlan(input: {
page: Page;
api: Pick<RunnerApi, "get">;
checkpoint: ContinuationCheckpoint;
deadlineAt: number;
}) {
const expected = currentContinuationApproval(input.checkpoint);
const path = `/api/issues/${input.checkpoint.issue.id}`;
const [interactions, documents] = await Promise.all([
input.api.get<ContinuationApproval[]>(`${path}/interactions`),
input.api.get<ContinuationCheckpoint["documents"]>(`${path}/documents`),
]);
const current = currentContinuationApproval({ ...input.checkpoint, interactions, documents });
expect(current.id, "the reviewed confirmation must still be current").toBe(expected.id);
expect(sameTarget(current.payload.target, expected.payload.target), "the reviewed plan must not change before approval").toBe(true);
const card = input.page.getByTestId("task-chat-composer-takeover")
.locator(`[data-interaction-id=${JSON.stringify(current.id)}]`);
await expect(card, "the exact current confirmation must be visible").toBeVisible();
await card.getByRole("button", { name: current.payload.acceptLabel ?? "Approve", exact: true }).click();
const saved = await pollUntil({
label: "continuation plan human acceptance",
deadlineAt: Math.min(input.deadlineAt, Date.now() + 30_000),
intervalMs: 250,
load: async () => (await input.api.get<ContinuationApproval[]>(`${path}/interactions`)).find(i => i.id === current.id),
accept: result => continuationApprovalAccepted(current, result),
reject: result => result && (!sameTarget(current.payload.target, result.payload.target)
|| !["pending", "accepted"].includes(result.status)
|| result.status === "accepted" && !continuationApprovalAccepted(current, result))
? "the reviewed confirmation changed or did not record human acceptance" : undefined,
});
return saved!;
}