mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
DottaandClaude Opus 5.5
ba5ba54439
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `tasks.completion-with-gate` checks that the CEO does not force Done while an independent human review is still required > - Its setup waits for a real Morgan implementation that awaits the configured review stage > - The setup timed out and did not say which state field was wrong > - The retained evidence shows a product defect: the native completion path never started the configured review stage (fixed in #20882) > - This pull request makes the readiness timeout name each unmet field, accepts valid product continuation runs, and replaces reply keyword grading with a structural check > - The benefit is that the eval measures the gate, and its setup failures are self-explaining ## Linked Issues or Issue Description **What happened** The eval failed with "Timed out waiting for Real prior Morgan implementation awaiting independent review". Morgan uploaded `website.html` and reported needs_review. The task became in_review through a generic review card, but `executionState` stayed null, so the poll (pending stage for Riley) never matched. The run finished at 229 s of the 240 s setup window. The grader also matched reply words with regular expressions. **Expected behavior** Setup reaches the configured review stage through supported product paths. A timeout names each unmet field. The grader proves the outcome from saved state and run receipts, not reply wording. **Steps to reproduce** 1. Dispatch `chat-features.runner-opencode.local.tasks.completion-with-gate` on master. 2. Read `server.log` and `snapshots/chat-final-run-ledger.json`: `authoritativeDecision: in_review`, `finalizationReasonCode: actionable_attention_pending`, no executionState. ## What Changed - `internal-task-completion.ts`: `unmetPriorDeliveryReview` lists each unmet readiness field. The poll uses it for `accept` and `timeoutDetail`. - Setup and grader accept one or more succeeded Morgan runs (a product continuation is valid). Any failed, cancelled or timed-out worker run is an unmet readiness condition, so setup stops before the paid CEO turn. The delivery must come from one of these runs. - The `truthful-outstanding-review` reply regex check is replaced by `checked-actual-completion-gate`: the CEO's chat run read the task (`GET /api/issues/{id}` or `get_task`, matched by record-bound receipt) and replied. The saved pending stage, unchanged policy and absence of a forced Done still prove the outcome. - Run bound 2 → 3, setup window 240 s → 330 s, attempt 540 s → 660 s. `oracleVersion` is `paperclip.chat-catalog.completion-with-gate.v2`. The descriptor is updated. - Cherry-picks both #20882 commits so the paid run exercises the product fix. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on `internal.test.ts`, `internal-comment-reference.test.ts`, `internal-edit-ownership.test.ts`, `chat-catalog-contract.test.ts`, `chat-catalog-consistency.test.ts` (184 tests) and 12 other suites that reference the case (133 tests). - Calibration: two reworded valid replies pass. A missing gate read, an empty reply, a forced Done and a failed worker run fail. The retained product state (in_review, no executionState) produces named unmet fields. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no new errors in the changed test files. The local worktree has unbuilt packages, so CI supplies the full typecheck. ## Risks - The reply's meaning is no longer machine-checked. A person must read the reply in the paid run. - The larger run bound and windows accept slower valid runs. Readiness still fails on any failed worker run. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
32 lines
2.4 KiB
TypeScript
32 lines
2.4 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import { unmetPriorDeliveryReview } from "./internal-task-completion.js";
|
|
|
|
describe("completion gate setup readiness", () => {
|
|
const review = { status: "pending", currentStageId: "stage", currentStageType: "review", currentParticipant: { type: "user", userId: "riley" } };
|
|
const ready = () => ({ task: { id: "task", status: "in_review", executionState: structuredClone(review) },
|
|
runs: [{ id: "run", agentId: "morgan", status: "succeeded" }],
|
|
attachments: [{ id: "html", sha256: "a", createdByAgentId: "morgan", originatingRunId: "run" }] });
|
|
it("accepts the configured review pending after a delivered implementation", () => {
|
|
expect(unmetPriorDeliveryReview(ready(), "morgan", "riley")).toEqual([]);
|
|
const continued = ready(); continued.runs.push({ id: "next", agentId: "morgan", status: "succeeded" });
|
|
expect(unmetPriorDeliveryReview(continued, "morgan", "riley")).toEqual([]);
|
|
});
|
|
it("names each unmet field of a review requested outside the configured stage", () => {
|
|
const generic = ready(); Object.assign(generic.task, { executionState: null });
|
|
expect(unmetPriorDeliveryReview(generic, "morgan", "riley")).toEqual([
|
|
"executionState.status=null (want pending)", "executionState.currentParticipant=null (want the configured reviewer)"]);
|
|
});
|
|
it("names a bypassed review, an active run and a missing delivery", () => {
|
|
const bypassed = ready(); Object.assign(bypassed.task, { status: "done", executionState: null });
|
|
bypassed.runs.push({ id: "live", agentId: "morgan", status: "running" }); bypassed.attachments = [];
|
|
expect(unmetPriorDeliveryReview(bypassed, "morgan", "riley")).toEqual([
|
|
"task.status=done (want in_review)", "executionState.status=null (want pending)",
|
|
"executionState.currentParticipant=null (want the configured reviewer)", "active worker runs=1 (want 0)",
|
|
"worker deliveries from a succeeded worker run=0 of 0 attachments (want at least 1)"]);
|
|
const failedBeside = ready(); failedBeside.runs.push({ id: "lost", agentId: "morgan", status: "failed" });
|
|
expect(unmetPriorDeliveryReview(failedBeside, "morgan", "riley")).toEqual(["failed worker runs=1 (want 0)"]);
|
|
const other = ready(); other.task.executionState.currentParticipant.userId = "someone";
|
|
expect(unmetPriorDeliveryReview(other, "morgan", "riley")).toEqual(["executionState.currentParticipant=user:other (want the configured reviewer)"]);
|
|
});
|
|
});
|