mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Cases let one primary task move between agents through handoffs > - The chat eval `cases.handoff-same-task` asks the case owner, Morgan, in chat: "Hand this case to Riley for review with the findings you saved." > - The eval failed in every cohort. Morgan's chat run handed the case to Riley directly. The grader required a separate Morgan run on the primary task to do the handoff. > - The product allows the direct route. The assignee may hand off its own task from any of its runs while no other run owns the task. The case's expected outcome does not ask for a separate run. > - This pull request grades the run that recorded the handoff, by provenance, and accepts both the direct and the routed shape. > - The benefit is that the eval measures the feature (same task, new owner, saved findings, real review) and not one tool sequence. ## Linked Issues or Issue Description **What happened?** Paid run 43413553210 on masterd16becac04failed `chat-features.runner-opencode.local.cases.handoff-same-task` with `normal-primary-execution-authority`, `catalog-bounded-native-runs` and `catalog-exact-run-outcomes`. **Expected behavior** The handoff was correct: same primary task, Riley assigned, phase `review`, instructions point at the saved `findings` document, Riley reviewed and finished the task, the case stayed open. The eval should pass this. **Steps to reproduce** Run the case. The ledger has 3 runs: the stopped Morgan investigation (cancelled), Morgan's chat answer on the source conversation (succeeded, it called the handoff), and Riley's review (succeeded). The handoff comment's `createdByRunId` is the chat run. The handoff `sourceRunId` is null because no run owned the released primary task. **Paperclip version or commit** masterd16becac04(failed run). This branch is based on master5504151eae. **Additional context** `server/src/services/case-handoffs.ts` accepts an agent handoff on an existing task when the agent is the assignee and no other run owns the task ("Agents may hand off only their own task", "Another run owns this task"). Ordinary task writes have the same rule: the assignee adopts an unowned in-progress checkout (`assertCheckoutOwner`). The grader added a stricter rule that the product and `cases.yaml` do not define. Retained runs from 9 cohorts all show the direct shape. ## What Changed - Replace `normal-primary-execution-authority` with `owner-execution-authority`. It finds the run that saved the handoff comment. That run must be a Morgan run that started after the person's request, not the stopped run. It must be the single answering chat run, or a Morgan primary-task run woken by a comment that the chat run saved. If the handoff records an owning execution, it must be that same run. - Add `every-run-has-a-handoff-role`: one stopped run, one chat answer, the recording run, and one Riley review. Any other run fails. - Run bounds: 3 to 4 runs, `cancelled: 1`, `succeeded: 2..3`. Oracle version `case-handoff.structural.v3`. The descriptor JSON matches. - Tests: the retained direct shape passes. Stale, foreign, concurrent, pre-request and unwoken recording runs fail. Extra Morgan or Riley runs fail. The run policy accepts 3 and 4 runs and rejects 2 and 5. ## Verification - Red: the new tests fail on the old grader (3 failed, 4 passed). Green: 7 passed. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on 21 related runner-e2e support files: 187 passed. - `pnpm test:e2e:runner:typecheck`: passed. - Regrade of retained evidence (full, full2-5, full7-9, full11 reclaim): 7 of 9 pass. full7 and full8 still fail `review-through-normal-admission-not-case-closure`, because Riley left the task blocked or in review. So the grader still rejects a real review failure. - Paid run 43424861825 of this case on this branch (a7cf944f9c, OpenCode + DeepSeek): PASS, oracle v3, all checks passed. Runs: stopped Morgan investigation (cancelled), Morgan chat answer (succeeded, recorded the handoff comment), Riley review (succeeded). Primary task done, case still open. ## Risks - Low. Eval-only change. No product code changes. - The grader now accepts a direct handoff from the owner's chat run. It still requires the product's own owner checks, recorded by run provenance. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>