mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
f1aef0cb58
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog runs real chat journeys against real agents and grades saved product state > - Full cohort 3 failed six cases where the product behaved correctly and the eval was wrong > - A wrong failure hides real product failures in the noise, so each check must accept valid behavior > - Each check must still fail the broken product behavior it exists to catch > - This pull request replaces those checks with checks of saved state, read receipts and rendered links > - The benefit is fewer false failures, with no case made easier ## Linked Issues or Issue Description Refs #21000 **What happened?** - `projects.project-work`, `tasks.private-task`, `status-cards.other-authors-card` and part of `task-coordination.delegate-with-context`: the CEO created a task with no assignee, so the CEO got a queued run on it. The CEO then reassigned the task. The server cancelled the queued run before it started (`issue_reassigned` or `issue_assignee_changed`). The early poll reported this as "Observed provider run failed", and the run policy counted a never-started legacy run as a failed native run. - `budgets-costs.repair-cost-recording`: after the retry, the chat run re-read the original run record with `saveResponse`. The check accepted only an exact whole-input digest, and this input had another shape. - `secrets.personal-secret`: the reply linked My secrets and said "Enter the value only there, never in chat". `solicitsChatCredential` read "never in chat" as a request to send the value in chat. - `companies.import-preview`: the reply linked `/COM/agents/morgan-package`. The check accepted only `/agents/<uuid>`. - `cases.close-case`: the closure was correct. The result turn said "Case COM-C1 is saved as closed". A keyword pattern did not match that sentence. **Expected behavior** Each check passes the recorded valid behavior and a reworded valid variant. Each check fails the broken behavior. **Steps to reproduce** Grade the retained cohort-3 evidence with the old checks. They fail. The calibration tests in this pull request use the same recorded shapes. ## What Changed - Run policy: a new rule admits a queued run that the server retired before it started because its task changed assignee. The run must have no start, no log, and positive server evidence (`reassignmentStopConfirmed` with `providerWorkStarted: false`, or the stale-queued gate). The task must not be the source chat. A later run by another agent on the same task must exist. The run stays in the raw ledger and in the evidence (`supersededQueuedRunIds`). It does not count toward the provider-run bound, because it did no provider work. A case can have at most one such run (`catalog-superseded-queued-runs`). The early poll defers it to the grade instead of failing at once. Cases read the ledger without it. Other cancellations stay failures. - `budgets-costs.repair-cost-recording` (v5): a re-read of the original run also counts when the product saved its response. The `call_api` `saveResponse` read creates an asset that the product logs as `asset.created` by that run. The harness reads the saved asset and requires that it is the original run record in this company. - `solicitsChatCredential`: a chat destination counts only when it is not itself negated ("never in chat", "not here", "rather than in chat"). This is a scope rule, not a new phrase exception. "Paste it here" and mixed requests still fail. `secrets.personal-secret` is v18. - `companies.import-preview` (v3): new `agentRouteNames` accepts `/<prefix>/agents/<id or url key>[/<tab>]` that names the imported agent. It rejects the existing Morgan, other companies and other origins. - `cases.close-case` (v2): the result turn is graded on its own saved replies. Their run must read the saved case (`read_case` or `GET /api/cases/{id}`, bound by the redaction-safe receipt). The reply must render a link to that case. The case must be done. The wording is not matched. `api-read-receipts` now binds the dedicated `read_case` tool. - The four reassignment cases get new oracle versions. Descriptors are updated. No scenario yaml changed, so the TestOps manifest does not change. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts` on the 17 affected test files: 516 tests pass. The files include `chat-catalog-run-policy`, `chat-flow`, `non-execution-wake`, `chat-catalog-consistency`, `chat-catalog-fingerprint`, `identity-accounting`, `external-credential-language`, `external-personal-secret`, `identity-portability`, `integrator-case-close`, `api-read-receipts` and the other secret-language users. - Each change has calibration tests: the recorded valid shape, a reworded valid shape, and broken shapes (no successor run, a started or other cancellation, churn; another run's saved record; "paste it here"; the existing Morgan's route; an unread or unlinked result turn). - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - One paid run on a combined branch with the companion harness pull request. The result is in a comment on this pull request. ## Risks - The superseded-run rule trusts the successor run as proof of reassignment. A case oracle still grades the task assignee and the successor's own outcome. - The close-case result turn no longer checks the reply's words. A reply that reads the closed case and links it but states the wrong result passes this one check. The first-turn premature-claim check and the negative-case check do not change. - The credential grader change also affects other secret cases. It only removes the negated-destination false positive. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>