mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat request reaches the correct saved outcome > - The `source-settled-with-active-work` run policy checks that the CEO chat is settled while delegated work can continue > - After a delegated task finishes, the product starts its normal `chat_task_completed` follow-up on the source chat > - The policy required every source run to be `succeeded` and the source to be `waiting`, so a re-grade during that follow-up failed > - This pull request admits that one product follow-up while it is in flight, and keeps every other source run strict > - The benefit is that the eval grades the saved private-delegation outcome, not the timing of the reload ## Linked Issues or Issue Description **What happened?** The OpenCode cell of `chat-features.runner-opencode.local.tasks.private-task` failed with "Independent oracle survives browser reload: expected true, received false". The saved state was correct. The CEO answered, Morgan saved the private draft in the private project, and the unrelated member could not read it. The grade passed for its full 5-second stability window. Morgan's run then finished, and 73 ms later the product started the CEO `chat_task_completed` follow-up on the source chat. The harness reloaded the page and re-graded while that follow-up was running. The only failed check was `catalog-settled-source-separate-work`, because it required all source runs to be `succeeded`. **Expected behavior** The case already counts this follow-up in its run bound (`expectedRunCount: 3`, `minimumRunCount: 2`). A queued or running product follow-up on an answered source must not fail the case. **Steps to reproduce** Run the private-task case. Let the worker finish within about 5 seconds of the first passing grade. The post-reload re-grade sees the follow-up as `running` and fails. ## What Changed - Add `chatCatalogSourceCompletionFollowUp` to `tests/runner-e2e/chat-catalog-run-policy.ts`. It identifies the lead's `chat_task_completed` run on the source whose every completion update names separate work in the run ledger. - In the `source-settled-with-active-work` check, admit at most one such follow-up while it is `queued`, `running` or `succeeded`. Accept a source `conversationState` of `active` only while that follow-up is in flight. - Keep the other rules strict. The original answer must exist and must have succeeded. Any other active, failed or extra source run still fails. The declared run-count bound still applies. - Also admit the follow-up before its turn is prepared. The server claims the run first and saves the completion updates after that, so a queued run, or a running run in preparation, has only its delivery IDs. It qualifies only while queued or running, when automation woke it, and after a separate worker run in the ledger succeeded. - Add a red/green test that reproduces the observed race and covers the negative cases. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-run-policy.test.ts`: the new test fails before the change (1 failed, 13 passed) and passes after it (14 passed). The queued-snapshot cases were added after review. - The related runner-e2e tests (`chat-catalog-run-policy`, `chat-catalog-readiness`, `chat-catalog-failure-retirement`, `runtime-dependency`, `internal`) pass: 200 tests. - `pnpm test:e2e:runner:typecheck` passes. - Paid run [43374419951](https://github.com/paperclipai/paperclip/actions/runs/43374419951) on the first commit of this branch: `chat-features.runner-opencode.local.tasks.private-task` passed. ## Risks - Low risk. The change is in eval grading only. It does not change product code. - The check accepts a follow-up that is still running at grade time. A later failure of that follow-up is not observed by this case. Completion delivery is covered by its own completion cases. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>