Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts
T
DottaandClaude Opus 5.5 42fd790e80 fix(evals): grade approval delivery on the saved answer, not on which queued wake is listed (#20981)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys against real agents
and grades the saved product state
> - The `steering.approval-delivery` case checks that a person's answer
to an agent's question stays separate from editable queued messages and
reaches the agent once
> - The task composer lists one pending wake at a time. It shows
editable ordinary messages before a saved card response, by design since
#19331
> - When the chat lead relays the person's answer as its own task
comment, the composer lists that relay. The saved answer waits on its
own wake and is not listed
> - The oracle required the answer and the relay in one listing. The
product never produces that shape, so the case timed out every time the
lead relayed
> - This pull request grades the case on the saved answer and on its
delivery, and accepts either valid listing
> - The benefit is that the case measures the promise in its expected
outcome and still fails on a lost, replayed, or editable answer

## Linked Issues or Issue Description

**What happened?**
The paid chat eval `steering.approval-delivery` (OpenCode, DeepSeek
flash) did the answer, edit, reorder, and discard steps. It then timed
out while it waited for the answer to show in the queue.

**Steps to reproduce**
Run the case. Let the chat lead relay the answer as its own task
comment. After the person's two messages are discarded, the queue lists
only the relay. The answer is saved as `answered` and keeps its own
pending wake.

**Expected behavior**
The case passes when the answer stays saved and unchanged, cannot be
edited through the queue, and reaches exactly one later agent turn.

Refs #19331, #20899

## What Changed

- `runtime-approval-queue.ts`: add `approvalResponseListing`. After the
discards, accept the answer listed alone and locked, or only the chat
lead's relay listed. Reject any person message or an editable answer.
- Keep the edit attempt through the queue route for both listings.
Require it to be refused and the saved answer to stay unchanged. Check
the locked card in the UI when the answer is listed.
- Add `classifyApprovalSuccessors`. Grade delivery on saved run state:
exactly one later worker turn bound to the answer, with the question ID
and Acorn in its native input. Allow at most one other turn that carries
only the lead's relay. Reject any other turn.
- Accept the Acorn document from the bound turn or the relay turn. Still
require the earlier work to be kept, all runs succeeded, the queue
empty, and the task done.
- Oracle `approval-delivery.v6`. Run bound 5 with a minimum of 4, to
allow the one relay turn.
- Calibration tests: the retained relay shape passes, the original
listed shape passes, and these fail: no bound turn, two bound turns, a
person message left behind the relay, an unexpected worker turn, two
relay turns, a recorded listing that disagrees with the queue, and a
listed answer whose controls were not seen as locked.

No YAML changed. The expected outcome already promises retention and
once-only delivery, not a queue listing.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts` (46 passed)
- `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` (clean)
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check` (no
change)
- One paid single-case run of this branch. The result is added below in
a comment.

## Risks

- Low risk. The change is in eval code only. No product behavior
changes.
- The grader now allows one extra worker turn for the lead's relay. A
turn that carries any other input still fails the case.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:13:43 -06:00

103 lines
13 KiB
TypeScript

import { expect, it } from "vitest";
import { approvalResponseListing, classifyApprovalSuccessors, gradeApprovalQueue, leadRelayEntries, ordinaryApprovalEntries, pendingInputYield } from "./runtime-approval-queue.js";
function valid() {
const payload = { questionSet: { questions: [{ id: "name", prompt: "What is the product name?" }] } };
return { scenarioId: "steering.approval-delivery", companyId: "company", leadId: "ceo", agents: [], links: [], humanQueries: [], reply: "", tasks: [{ id: "website" }], activity: [],
runs: [{ id: "asked", status: "succeeded" }, { id: "prior", status: "succeeded" }, { id: "canonical", status: "succeeded" }, { id: "consumer", status: "succeeded" }],
state: { task: { id: "website" }, worker: { id: "morgan" }, humanId: "human", questionRunId: "asked", questionRunYielded: true, workerRunId: "prior", workerWasReallyWaiting: true, ordinaryQueueOwned: true, ordinaryQueuedBeforeAnswer: true,
questionBefore: { id: "question", kind: "ask_user_questions", status: "pending", issueId: "website", createdByAgentId: "morgan", sourceRunId: "asked", payload },
questionAfter: { id: "question", status: "answered", resolvedByUserId: "human" as string | null, resolvedByAgentId: null as string | null, payload, result: { answers: [{ questionId: "name", otherText: "Acorn" }] } },
responseRetainedAcrossOrdinaryMutations: true, responseUneditable: true, queryReply: "Use Acorn for the existing naming question; submit that bound answer through its supported response control.",
actualOrdinaryEdit: true, actualOrdinaryReorder: true, ordinaryReorderRestricted: false, actualOrdinaryDiscard: true, humanAnswerSubmitted: true, canonicalAnswerAttributed: false,
responseQueue: { entries: [{ source: { kind: "interaction", interactionId: "question" }, canEdit: false, canDiscard: false }] } as { entries: unknown[] }, responseListing: "listed", responseListedImmutable: true,
consumingRunCount: 1, relayRunCount: 0, unexpectedSuccessorRunCount: 0, responseNativeInputBound: true, queueEmptyAfterDelivery: true, actualAcornResult: true, initialWorkRetained: true } };
}
it("accepts real human response retained separately from mutated ordinary comments", () => expect(gradeApprovalQueue(valid()).passed).toBe(true));
it("accepts a permitted actual canonical agent response with native source attribution", () => { const o = valid(); Object.assign(o.state, { canonicalAnswerAttributed: true, humanAnswerSubmitted: false, queryReply: "I answered Morgan's naming question with Acorn; the response is saved for normal continuation." }); Object.assign(o.state.questionAfter, { resolvedByUserId: null, resolvedByAgentId: "ceo" }); expect(gradeApprovalQueue(o).passed).toBe(true); });
it("rejects a fabricated question without actual prior native provenance", () => { const o = valid(); o.state.questionBefore.createdByAgentId = "other"; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects an ordinary comment masquerading as bound response", () => { const o = valid(); (o.state.responseQueue.entries[0] as { source: { kind: string } }).source.kind = "comment"; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects a canonical response recording a different name", () => { const o = valid(); Object.assign(o.state, { canonicalAnswerAttributed: true, humanAnswerSubmitted: false }); Object.assign(o.state.questionAfter, { resolvedByUserId: null, resolvedByAgentId: "ceo" }); o.state.questionAfter.result.answers[0]!.otherText = "Birch"; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects wrong accepted answer", () => { const o = valid(); o.state.questionAfter.result.answers[0]!.otherText = "Other"; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects revoking the decision by discarding ordinary text", () => { const o = valid(); o.state.responseRetainedAcrossOrdinaryMutations = false; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects an editable interaction response", () => { const o = valid(); (o.state.responseQueue.entries[0] as { canEdit: boolean }).canEdit = true; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects unsupported actor answering for the human", () => { const o = valid(); o.state.questionAfter.resolvedByUserId = "other"; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects ordinary message operations performed on a different queue", () => { const o = valid(); o.state.actualOrdinaryReorder = false; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects duplicate decision consumers", () => { const o = valid(); o.state.consumingRunCount = 2; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects native input lacking the bound question ID and answer", () => { const o = valid(); o.state.responseNativeInputBound = false; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects loss of earlier settled work", () => { const o = valid(); o.state.initialWorkRetained = false; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects missing actual response evidence", () => expect(gradeApprovalQueue(undefined).passed).toBe(false));
it("restricts every mutating ID to ordinary entries in a mixed two-comment plus response queue", () => {
const ordinary = { comment: { id: "ordinary" } }, response = { source: { kind: "interaction" }, comment: { id: "decision" } };
const ids = ordinaryApprovalEntries({ entries: [ordinary, response, { comment: { id: "other" } }] } as Parameters<typeof ordinaryApprovalEntries>[0]).map(row => row.comment.id);
expect(ids).toEqual(["ordinary", "other"]); expect(ids).not.toContain("decision");
});
it("accepts the real mixed-queue UI reorder refusal while preserving immutable decision placement", () => { const o = valid(); o.state.actualOrdinaryReorder = false; o.state.ordinaryReorderRestricted = true; expect(gradeApprovalQueue(o).passed).toBe(true); });
// The canonical reply is read by a person, not matched by words: the verdict rests on the saved answer path.
it.each(["Use Acorn for the existing naming question; submit that bound answer through its supported response control.", "Open Morgan's naming card and send Acorn there so the decision stays attached to it.", "Got it."])("verdict does not depend on reply wording: %s", reply => { const o = valid(); o.state.queryReply = reply; expect(gradeApprovalQueue(o).passed).toBe(true); });
it("rejects a question whose run never parked on it", () => { const o = valid(); o.state.questionRunYielded = false; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects a question attributed to the later active turn instead of the turn that asked", () => { const o = valid(); o.state.questionBefore.sourceRunId = "prior"; o.state.questionRunId = "prior"; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects an answer with no actual recording path", () => { const o = valid(); o.state.humanAnswerSubmitted = false; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects a missing parked-question turn in the run ledger", () => { const o = valid(); o.runs.shift(); expect(gradeApprovalQueue(o).passed).toBe(false); });
// Shape recorded by the retained runner-opencode attempt (gha-42926276473): request_human_input parks the turn.
const parked = { id: "asked", status: "succeeded", resultJson: { nativeResult: { reportedWorkDisposition: "yielded", evidence: [{ ref: "interaction:question" }],
artifacts: [{ ref: "interaction:question", kind: "issue_thread_interaction" }], continuation: { kind: "response_wake", idempotencyKey: "interaction-response:question" } } } };
it("recognizes the product's parked pending-input turn", () => expect(pendingInputYield(parked, "question")).toBe(true));
it.each([
["still running", { ...parked, status: "running" }],
["completed instead of yielded", { ...parked, resultJson: { nativeResult: { ...parked.resultJson.nativeResult, reportedWorkDisposition: "completed" } } }],
["parked on another interaction", { ...parked, resultJson: { nativeResult: { ...parked.resultJson.nativeResult, continuation: { kind: "response_wake", idempotencyKey: "interaction-response:other" } } } }],
["no result", { id: "asked", status: "succeeded", resultJson: null }],
])("rejects a run that is not parked on the question (%s)", (_label, run) => expect(pendingInputYield(run, "question")).toBe(false));
// Retained runner-opencode attempt (gha-43120714598): the chat lead relayed the person's answer as its own comment on the task.
// The composer lists one wake at a time and prefers that editable comment, so the answer's own wake waited unlisted behind it.
// After the gate, the lead's relay may take one extra worker turn; the answer must still reach exactly one bound turn.
function behindRelay() { const o = valid();
Object.assign(o.state, { responseQueue: { entries: [{ comment: { id: "relay", authorAgentId: "ceo", authorUserId: null } }] }, responseListing: "behind_lead_relay", responseListedImmutable: undefined, relayRunCount: 1 });
o.runs.push({ id: "relay-turn", status: "succeeded" }); return o; }
it("accepts the answer waiting unlisted behind the chat lead's own relay when it is then delivered once", () => expect(gradeApprovalQueue(behindRelay()).passed).toBe(true));
it("accepts the answer behind the relay when the relay never needed its own turn", () => { const o = behindRelay(); o.state.relayRunCount = 0; o.runs.pop(); expect(gradeApprovalQueue(o).passed).toBe(true); });
it("rejects an answer that waited behind the relay but was never delivered to a bound turn", () => { const o = behindRelay(); o.state.consumingRunCount = 0; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects an answer behind the relay delivered to two turns", () => { const o = behindRelay(); o.state.consumingRunCount = 2; o.runs.push({ id: "replay", status: "succeeded" }); expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects a person's message still queued behind the relay", () => { const o = behindRelay();
o.state.responseQueue.entries.push({ comment: { id: "left", authorUserId: "human", authorAgentId: null } }); expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects a worker turn that is neither the bound answer nor the lead's relay", () => { const o = behindRelay(); o.state.unexpectedSuccessorRunCount = 1; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects more relay-only turns than one relay wake can produce", () => { const o = behindRelay(); o.state.relayRunCount = 2; o.runs.push({ id: "relay-2", status: "succeeded" }); expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects a recorded listing that disagrees with the queue actually observed", () => { const o = behindRelay(); o.state.responseListing = "listed"; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects a listed answer whose card controls were never shown as locked", () => { const o = valid(); o.state.responseListedImmutable = undefined as never; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects a person's ordinary message left beside the bound response", () => { const o = valid();
o.state.responseQueue.entries.push({ comment: { id: "left", authorUserId: "human", authorAgentId: null } }); expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects a second bound response entry", () => { const o = valid();
o.state.responseQueue.entries.push({ source: { kind: "interaction", interactionId: "question" }, canEdit: false, canDiscard: false }); expect(gradeApprovalQueue(o).passed).toBe(false); });
it("classifies the queue listing from the observed entries", () => {
const answer = { source: { kind: "interaction", interactionId: "question" }, canEdit: false, canDiscard: false, comment: { id: "question" } };
const relay = { comment: { id: "relay", authorAgentId: "ceo", authorUserId: null } }, mine = { comment: { id: "mine", authorUserId: "human", authorAgentId: null } };
const listing = (entries: unknown[]) => approvalResponseListing({ entries } as Parameters<typeof approvalResponseListing>[0], "question", "ceo");
expect(listing([answer])).toBe("listed");
expect(listing([relay])).toBe("behind_lead_relay");
expect(listing([relay, mine])).toBeNull();
expect(listing([{ ...answer, canEdit: true }])).toBeNull();
expect(listing([{ ...answer, source: { kind: "interaction", interactionId: "other" } }])).toBeNull();
expect(listing([])).toBeNull();
});
it("splits later worker turns into the bound answer, the lead's relay and anything else", () => {
const split = classifyApprovalSuccessors([
{ id: "bound", contextSnapshot: { interactionId: "question" } },
{ id: "relay", contextSnapshot: { wakeCommentIds: ["relay-comment"] } },
{ id: "mixed", contextSnapshot: { wakeCommentIds: ["relay-comment", "person-comment"] } },
{ id: "other-answer", contextSnapshot: { interactionId: "other" } },
{ id: "bare", contextSnapshot: {} },
], "question", new Set(["relay-comment"]));
expect(split).toEqual({ bound: ["bound"], relay: ["relay"], other: ["mixed", "other-answer", "bare"] });
});
it("keeps only the person's own messages as editable ordinary entries", () => {
const queue = { entries: [{ comment: { id: "mine", authorUserId: "human", authorAgentId: null } }, { comment: { id: "relay", authorUserId: null, authorAgentId: "ceo" } },
{ source: { kind: "interaction" }, comment: { id: "decision" } }] } as unknown as Parameters<typeof ordinaryApprovalEntries>[0];
expect(ordinaryApprovalEntries(queue, "human").map(row => row.comment.id)).toEqual(["mine"]);
expect(leadRelayEntries(queue, "ceo").map(row => row.comment.id)).toEqual(["relay"]);
});