mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
DottaandClaude Opus 5.5
e879a24293
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys against real agents
and grades saved product state and rendered links, never reply wording
> - Full chat-eval cohort 4 had two cases whose graders passed replies
with a structural flaw that the grader could see but did not check
> - `connection-setup.wrong-person-resolves` promises that chat explains
whose action is required, but its grader had no reply check at all
> - `navigation-preferences.find-project-view` passed a reply with a
link labelled COM-2 that opened COM-1
> - This pull request adds two structural reply checks, with calibration
tests for valid, reworded-valid and broken replies
> - The benefit is that a reply that does not point to the request, or
that sends a person to the wrong task, fails the eval
## Linked Issues or Issue Description
**What happened?**
- `connection-setup.wrong-person-resolves`: another member said "I will
finish Alex's connection request for them". The product refused that
member (403 on setup options, phase, complete and decline) and kept
Alex's card pending. The grader checked only that. The reply's content
was not checked in any form.
- `navigation-preferences.find-project-view`: the saved reply contained
`[COM-2 Launch open website](/COM/issues/COM-1)`. The grader passed it.
**Expected behavior**
- The wrong-resolver reply points the person at Alex's actual pending
request. The card on that page says "Waiting for Alex", which is whose
action is required. The grader checks the link from saved state, and the
browser step follows the reply's own link to that card.
- Every task link in the find-project-view reply opens the task that its
visible identifier names.
**Steps to reproduce**
1. Run the two cases (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`) on master at f1aef0cb58.
2. Read the saved replies and the grader checks in the evidence.
## What Changed
- `external-wrong-resolver.ts`: new check
`reply-links-alex-pending-request`. The reply must render a same-origin
link to the related task that holds Alex's card (`/<company>/issues/<id
or identifier>`). `verifyResult` now follows that rendered link (not a
constructed URL) and confirms the "Waiting for Alex" card. Setup records
the company prefix. Oracle `paperclip.chat-catalog.external.v2` -> `v3`.
Reply wording is still not graded.
- `chat-rendered-links.ts`: new shared helper
`mislabeledIssueLinks(text, identifierFor?)`. It returns inline Markdown
links whose label names a task identifier (same prefix as the target)
other than the task the link opens. Labels without such a token, and
non-task targets, are not judged. Task-id targets are compared through
an optional resolver.
- `identity-navigation-preferences.ts`: find-project-view adds
`task-links-match-their-labels` with that helper. Oracle
`paperclip.chat-catalog.identity.v1` ->
`paperclip.chat-catalog.identity.find-project-view.v2`;
`chat-rendered-links.ts` joins its source files.
- `chat-catalog-descriptors.json`: the two descriptors carry the new
oracle versions and source files.
- The scenario YAML does not change, so the testops manifest has no
`changedFields` entry.
## Verification
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/external-wrong-resolver.test.ts
tests/runner-e2e/chat-rendered-links.test.ts
tests/runner-e2e/chat-catalog-domains/identity.test.ts`: 92 tests pass.
- Calibration, wrong resolver: valid (links CMP-1), reworded valid
(links by task id, absolute same-origin link) pass; no link, another
task, another company prefix, another origin, and a sub-path fail.
Red/green: with the master grader, the "does not point at Alex's
request" test fails.
- Calibration, labels: valid, reworded (label without an identifier,
document anchor, task-id target resolved), and a different-prefix token
pass; the recorded cohort-4 link `[COM-2 Launch open
website](/COM/issues/COM-1)` fails. The helper flags exactly that link
in the retained cohort-4 reply.
- Retained cohort-4 wrong-resolver reply links `/COM/issues/COM-1` (the
related task), so it passes the new link check. Its wording ("completing
it there is the right move") is a guidance problem, fixed in #21027, not
something saved state can grade.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-campaign chat-catalog`: 3856 tests pass. `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
`tsc -p tests/runner-e2e/tsconfig.json` passes.
- A paid run of both cases with the product fixes is linked in a comment
below.
## Risks
- A valid reply that names Alex but links nothing now fails
wrong-person-resolves. The case contract says chat explains whose action
is required; the card is where Paperclip shows it, so pointing to it is
the structural form of that promise.
- The label check could flag a label that intentionally names two tasks
of the same company and links only one. That reply would still be
misleading, so failing it is acceptable.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>