mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
98c18784c6
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 4 (OpenCode with DeepSeek v4.1 flash, master f1aef0cb58)
failed 14 cases for eval reasons, not product reasons
> - Some phase captures read links before the reply rendered. Some
graders matched keywords in free-form replies. One locator was stale.
One fixture answer looked like a placeholder. Some checks could not say
what failed
> - This pull request grades those cases on saved state, setup literals
and rendered links, and it fixes the observers and fixtures
> - The benefit is that these cases fail only when the product fails,
and a failure names the part that broke
## Linked Issues or Issue Description
**What happened?**
In full chat cohort 4, valid agent behavior failed these cases:
case-history, goals.explain-purpose and workspaces.preview-service
(phase captures read before the reply rendered); routines.pause-owned
("If you want that one stopped too, use Pause on its task page" matched
a cancellation regex); budgets-costs.spend-breakdown ("unknown, not
$0.00" matched a stated-zero regex); steering.edit-queued-message
("change ... and drag it above" missed a keyword list);
connection-setup.choose-provider (the resumed page says "Finish with
Notion"); questions-and-approvals.necessary-question (the answer "Acorn
2a4f72ec4ae2-1" was saved as "Acorn"). Some checks could not say what
failed: onboarding.new-company compared the whole members response as
JSON and saved no after-state; agents.choose-primary threw one combined
error.
**Expected behavior**
Each case grades the feature contract on saved state, setup literals and
rendered or clicked links. A valid reworded reply passes. A broken
product still fails. A failure names the clause and its values.
**Steps to reproduce**
Grade the retained cohort 4 evidence for the cases above with the
earlier oracles.
Refs #21009, #21002, #21007
## What Changed
- Phase captures (case-history first phase, goals.explain-purpose
phases, preview-service query phase, ambiguous-task clarification,
edit-queued-message guidance) now wait for the saved reply to render
(`waitForTurnReplyLinks`) before they observe. When a saved reply yields
no rendered links, the observation records thread and anchor diagnostics
(`linkDiagnostics`) and logs them.
- routines.pause-owned: remove the reply keyword clause. Saved state at
the reply proves the run kept running, and the final state proves it
completed normally.
- budgets-costs.spend-breakdown: the unresolved record is graded
structurally. The reply must identify it (rendered link, identifier or
run id), and a chat run must have read its saved accounting (cost
summary, run record or unranged company report, bound by input digest).
"$0.00" in the reply is not matched.
- steering.edit-queued-message: grade the rendered task link in the
reply, the refused agent edit and reorder (403), and the unchanged queue
at the reply. Keyword lists are removed.
- connection-setup.choose-provider: the sign-in locator also accepts
"Finish with Notion" (the resumed-draft label). The button has no test
id or other stable name.
- questions-and-approvals.necessary-question: the answer is now
"Quillmere". The case also accepts the exact answer in a saved
deliverable (a document revision or text attachment on the linked task,
saved by an agent of the company after the answer). The yaml expected
outcome records this decision, with a `changedFields` entry in the
testops manifest.
- onboarding.new-company: save the members after-state and compare
identity, role, status and grants, and name what changed. Guidance is
graded on a rendered link to `/onboarding` (the new-organization
wizard), the business literal from the query, and the saved company
count. The source company and person get realistic names for this case
and are restored at cleanup, because the CEO reasoned from the generated
names that the request was a graded test.
- agents.choose-primary: the settled-Chat check is split into four named
clauses with values, saved as evidence on failure.
- A question card the turn ended on counts as its reply (decision after
cohort 4) in cli-operations.owned-dev-instance, onboarding.new-company
and status-cards.other-authors-card. The card's Markdown-rendered fields
(question form description; confirmation prompt and details) count as
reply links. Link requirements still apply.
- Every changed case bumps its `oracleVersion`; the descriptor snapshot
and inventory digest are updated.
Why each old check was wrong and why the new one still catches a broken
product:
| Case | Old check was wrong because | A broken product still fails
because |
|---|---|---|
| phase captures | runs succeeded and the chat waited before the reply
rendered, so links were empty | the same link checks run on the rendered
reply |
| routines.pause-owned | a correct offer to pause the run read as a
cancellation | the checkpoint run must be running and the run must later
succeed |
| spend-breakdown | "unknown, not $0.00" read as "$0" | the reply must
identify the record and a chat run must read its saved accounting;
totals and per-agent amounts stay graded |
| edit-queued-message | correct wording missed a keyword list | the
reply must link the task, the agent's edit and reorder must be refused,
and the queue must be unchanged at the reply |
| choose-provider | the resumed page label was not accepted | the button
must still be visible on the selected Notion surface |
| necessary-question | the answer read as a placeholder; deliverables
did not count | the exact answer must be saved in the linked work after
the answer, by an agent of the company |
| new-company | timestamps and profiles made the JSON compare fail
without saying why | membership identity, role, status and grants must
be unchanged |
| choose-primary | one combined error | the same four conditions, each
named |
| card as reply | a card answer read as an empty reply | card text must
exist and link requirements still apply |
## Verification
- Calibration tests (valid, reworded valid, broken) for each changed
grader: `integrator-routine-pause.test.ts`,
`identity-accounting.test.ts` (includes the cohort 4 reply),
`runtime-queue-controls.test.ts` (includes the cohort 4 reply),
`external-provider-choice.test.ts`, `identity.test.ts`,
`identity-onboarding-work.test.ts`,
`runtime-primary-ready-chat.test.ts`, `integrator-status-cards.test.ts`,
`turn-response-card.test.ts`, `chat-rendered-links.test.ts`.
- The corrected spend-breakdown grader passes on the retained cohort 4
evidence (its cost-summary read binds to the unresolved task) and fails
with the reads removed.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog chat-rendered chat-linked`: 284 files passed.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
passed.
- `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
unchanged.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) on this branch
combined with the guidance PR: see the PR comment for results.
## Risks
- The new-company guidance check now needs a rendered link to
`/onboarding`. A reply that only names the company switcher fails. The
companion guidance PR tells agents this route.
- spend-breakdown does not grade wording, so a reply that reads the
record and still calls it "$0" is not caught. The amounts, total and
identification are still graded.
- The new-company case renames the source company and the person for the
case and restores them at cleanup.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>