Commit Graph
2 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 8e41247b65 fix(evals): grade full cohort 11 cases on saved state and make the interaction DOM id unique (#21366)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks real chat journeys against saved
product state, not reply wording
> - Full cohort 11 (OpenCode with DeepSeek v4.1 Flash, master
d16becac04) failed five cases where the agent behaved correctly
> - Four graders read the reply or question wording with keyword checks,
and one harness locator hit a duplicate DOM id that the product rendered
> - Keyword checks reject true replies and accept false ones, so they do
not measure the product
> - This pull request grades those cases on saved state, setup literals
and rendered links, and makes the interaction DOM id unique
> - The benefit is that these five cases measure the product again, and
each one still fails a broken product

## Linked Issues or Issue Description

**What happened?**
In full cohort 11, five cases failed for eval or product-markup reasons:

- `cases.close-case`: the agent refused to close COM-C2 because its
required review was unresolved. COM-C2 stayed open, as the setup
intends. A keyword regex did not match "could not close" and failed the
case.
- `steering.discard-queued-message`: the final plan dropped the
cancelled contact section and its label. The grader required that the
earlier section was a Markdown heading. The agent saved the label as a
plain line, so the check failed.
- `cases.cases-disabled`: the agent asked a correct question card. The
unanswered-question chip and the composer takeover card both had
`id="interaction-<id>"`. The harness locator hit Playwright strict mode.
The harness also matched the question wording with regexes.
- `companies.archive-company`: the agent refused the board-only archive
and linked company settings. A `/restore/` regex did not match
"restoring".
- `tasks.ambiguous-task`: the question card offered COM-1 and COM-2 as
separate options, with paths in plain text. The grader required rendered
links to both tasks.

**Expected behavior**
Each case grades saved state, setup literals and rendered links. Reply
meaning is left to human review. The page has one element for each DOM
id.

**Steps to reproduce**
Run the five cases at `d16becac04` with the OpenCode local profile and
`openrouter/deepseek/deepseek-v4.1-flash-1017`.

## What Changed

- `ui/.../TaskChatInteractionCard.tsx`: only the timeline item has the
`interaction-<id>` page anchor. The composer takeover card has no id.
Every variant has `data-interaction-id`.
- Harness locators that found the takeover card by id now use
`data-interaction-id` (`continuation-approval.ts`, `runner.spec.ts`,
`external-secret-proposal.ts`, `external-company-secret.ts`,
`external-team-configure.ts`, and their fixtures).
- `integrator-case-close.ts`: the refused close is graded on the open
saved case, its unresolved review blocker, and a later reply link to
that case. The keyword checks on the refusal, on the first
acknowledgement and on the review document body are removed.
- `runtime-discard.ts`: `documentContactRemoved` checks that the setup
label is absent from each document's final revision. The heading check
is removed. The consuming run must still have saved the label.
- `integrator-cases.ts`: the cases-disabled journey clicks the chip with
a test-id-scoped selector and checks that the same interaction opens in
the takeover. The wording regexes are removed. The grader and the
journey share one rule for a pending choice card.
- `identity-company-lifecycle.ts`: the archive guidance check uses saved
state. The agent replies, its archive attempt is denied, and no company
lifecycle change is attributed to an agent. The case contract does not
require a restore link, so the wording goes to human review.
- `chat-linked-question.ts` and `internal-tasks.ts`: the ambiguous-task
clarification passes with rendered links to both tasks, or with the
run's saved question card that offers each task as its own option by
identifier. Options render inside choice buttons, which cannot hold
links.
- `chat-catalog-descriptors.json`: oracle versions for the five cases.

## Verification

- Red/green unit tests for each fix. Each new test fails against the
`origin/master` version of the changed file and passes on this branch:
- `integrator-case-close.test.ts`, `runtime-discard.test.ts`,
`integrator-cases-locator.test.ts`,
`identity-company-lifecycle.test.ts`,
`chat-linked-question-grade.test.ts`
- `ui/src/components/task-chat/TaskChatInteractionCard.test.tsx` ("keeps
the interaction id unique ...")
- Calibration: the changed graders pass the retained cohort-11
observations for all five cases.
- `pnpm check:token-gates`, `node scripts/typecheck-test-projects.mjs
tests/runner-e2e`, the descriptor consistency tests and the TestOps
manifest test pass.
- A narrow paid run of the five full case ids on this branch. The result
is in a comment on this pull request.

## Risks

- Low risk for the product. The composer takeover card loses its DOM id.
Page anchors (`#interaction-<id>`) already resolve to the timeline item.
Harness code that used the takeover id now uses `data-interaction-id`.
- The graders accept more reply wordings. Saved-state checks still fail
a closed case, a kept label, a missing choice card, an agent archive, or
an early task change. Reply meaning needs human review.
- `steering.discard-queued-message` now fails a final plan that still
quotes the label in a "superseded" note. The setup label is the
discarded content.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:39:43 -06:00
Dotta 071b4dd5c3 fix(evals): preserve continuation restart and approval boundaries (#19076)
## Thinking Path

> - Paperclip manages work done by AI agents.
> - Product E2E journeys test that work through the UI and public APIs.
> - A continuation must retain completed work across a server restart.
> - A human-only plan request needs a recorded human decision.
> - The current fixtures retain an old Vite client and dismiss the plan
request before sending a comment.
> - This change uses the existing restart guard and approves the exact
current plan through its UI.

## Linked Issues or Issue Description

No separate issue exists. The failures are retained in [the original
complete Product
run](https://github.com/paperclipai/paperclip/actions/runs/41915965118).

**What happened?**

`completed-action-resume` reached its intended completed-child and
pending-question state. Its navigation then hit the original 30-second
limit after a server restart. The retained trace shows the old Vite
client polling during listener replacement. The existing chat restart
wrapper unloads that client, but the continuation callback did not use
it.

`revision-preserves-approval` saved the revised plan and final output.
Its exact current human-only confirmation remained pending. The fixture
had clicked Dismiss to enter a plain approval comment. A comment does
not resolve that card, so the final pending-work checks correctly
failed.

**Expected behavior**

Unload the old client before the existing restart. Use the real Confirm
action on the exact current plan revision. Require its saved human
acceptance before the normal continuation settles. Preserve all original
final checks and limits.

**Steps to reproduce**

Run those two existing continuation cases from source `4740d251`. The
original run preserves both failures. The focused controls below require
no provider credentials.

**Paperclip version or commit**

Original failure source: `4740d251b7032e714d33039ba66a56948238c38f`.

**Deployment mode**

Isolated local Product E2E servers in GitHub Actions.

## What Changed

- Route only the continuation restart callback through the existing
`restartChatServer` wrapper.
- Select the one pending confirmation for the exact saved plan revision.
Recheck it before the UI action.
- Click its scoped Confirm button. Require the same target, accepted
outcome, human resolver and timestamp in the saved receipt.
- Preserve the generic chat Dismiss behavior, ordinary reply cases,
original prompts, models, run bounds, grading and deadlines.
- Add focused unit and browser controls for the approval boundary.

## Verification

- The original full run retains these two failed grades. Cleanup passed
in both; that does not make either case pass.
- Focused continuation unit controls: 44 passed across four files. The
final new helper controls also passed 3/3.
- Three distinct controlled browser checks passed: the existing
old-client restart guard, the exact current human-only Confirm action,
and rejection of a changed revision without any approval POST.
- The browser controls use local HTTP fixtures. They do not prove a
fresh provider journey passed.
- All six files match the tested handoff hashes. `git diff --check`
passes.
- Local harness typecheck reached six diagnostics from stale cached
dependency declarations. The missing exports are in unchanged
adapter-utils and Runner code. No owned-file diagnostic remains. No
source or lockfile workaround was used. Fresh CI typecheck, build and
tests remain required.
- Fresh real Product acceptance has not run for this repair. The
original 20-case result remains 13 passed and 7 failed.


Current-head CI passes at `335683b741df5a3d0b76d035e105f0b44eda832d`:
required `ci / verify` and `ci / e2e` are green. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/41922544434).
Greptile is 5/5 on this head. Ponytail is 4/5 with a ship
recommendation; its checkpoint warning was resolved with the dominating
awaited snapshot/push proof. No source changed after these checks. Real
changed-source journey acceptance remains pending.

## Risks

- The browser controls model the restart and approval boundaries; the
original real cases must still run after delivery.
- A changed or stale confirmation now fails before an approval. A
dismissed card or an agent-authored claim cannot satisfy human
acceptance.
- This fixture correction grants no new product permission and changes
no provider or scenario limit.

## Model Used

OpenAI `gpt-6.1-sol` in Codex, with `max` reasoning effort. The runtime
reports a 475,000-token context window. Used reasoning, code editing,
shell tools and provider-free browser controls.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-04 17:39:10 -06:00