Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts
T
DottaandClaude Opus 5.5 a75d7df9c7 fix(evals): gate steering approval-delivery and steer-active on real product state (#20899)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat feature catalog runs real native agents to prove the chat
and task-steering promises.
> - Two steering cases failed in setup before they could measure the
product.
> - `steering.approval-delivery` waited for a worker run to stay running
while its question was pending. The product parks a native turn as soon
as a durable question is saved, so that state never occurs.
> - `steering.steer-active` waited for the queue to say steering is
"available". The OpenCode harness reports that it cannot steer, so the
queue says "unsupported".
> - This pull request gates each case on the real product state and
grades the behavior that state promises.
> - The benefit is that both cases now measure their user promise on the
evaluated profile, without faked runs or faked steering.

## Linked Issues or Issue Description

**What happened**
Both cases failed during setup on `runner-opencode`.
`steering.approval-delivery` stopped with "The actual question fixture
did not reach its running pending-input boundary": the worker's
`request_human_input` call parked its turn (run succeeded,
`reportedWorkDisposition: yielded`, continuation `response_wake` for the
question). `steering.steer-active` timed out waiting for
`steeringDisposition: "available"`; the OpenCode driver declares
`steering: false`, so the queue reports `"unsupported"` and the page
shows Interrupt instead of Steer.

**Expected behavior**
`steering.approval-delivery` should treat the saved yield and the
pending question as the pending-input state, queue ordinary comments and
the bound answer behind a real active worker turn, and prove one
continuation receives the answer. `steering.steer-active` should grade
same-turn steering when the harness can steer, and grade the honest
fallback when it cannot: the message stays visibly queued, nothing is
recorded as steered, and one normal turn consumes it.

**Steps to reproduce**
1. Dispatch
`chat-features.runner-opencode.local.steering.approval-delivery` on
master.
2. See the setup stop after the worker run succeeds with a yielded
result.
3. Dispatch `chat-features.runner-opencode.local.steering.steer-active`
on master.
4. See setup time out while `GET /api/issues/:id/queued-comments`
reports `steeringDisposition: "unsupported"`.

## What Changed

- `runtime-approval-queue.ts`: gate the question on the parked turn
(`pendingInputYield`: succeeded, yielded, `response_wake` keyed to the
question). An ordinary human message then wakes a real second Morgan
turn that waits at the task gate. Ordinary comments and the bound answer
queue behind that turn. The run count goes from 3 to 4.
- `runtime-approval-queue.ts`: remove the reply keyword checks
(`acornNamingDirection` and the "I have answered" regex). Grade exactly
one saved answer path instead: the human response control, or an
attributed canonical response.
- `runtime-active-steering.ts`: accept either definitive capability.
Unsupported branch: no enabled Steer control, a visible Interrupt
control, a steer attempt with the person's session refused with `409
steering_unsupported`, no `issue.queued_comment_steered` activity, the
exact message still queued, and one continuation that consumes it and
registers a rendered blue `website.html`.
- `runtime-active-steering.ts`: remove the reply keyword checks and the
`/blue/` match on the delivery document. The rendered page proves the
color.
- Update calibration tests in both directions, the workspace setup test,
and the descriptors (`oracleVersion` bumps, `expectedRunCount` 4 for
approval-delivery). The scenario YAML is unchanged.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/runtime-active-steering.test.ts
tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts
tests/runner-e2e/chat-catalog-domains/runtime-approval-workspace.test.ts
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts`: 87 passed.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed
files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged.
- The live proof is a paid run of
`chat-features.runner-opencode.local.steering.approval-delivery` and
`chat-features.runner-opencode.local.steering.steer-active`.

## Risks

- approval-delivery now depends on a human message waking the worker
while its question is pending. The product does this
(`supersedeOnUserComment: false` keeps the card open, and comments wake
the assignee unless the task is done or cancelled). If the worker does
not run the gate on that turn, setup fails with a named label.
- steer-active on OpenCode now measures the honest unsupported fallback,
not same-turn delivery. A steering-capable profile still runs the
same-turn branch.
- Low risk to the product: only eval code changes.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:28:44 -06:00

69 lines
9.6 KiB
TypeScript

import { expect, it } from "vitest";
import { gradeApprovalQueue, leadRelayEntries, ordinaryApprovalEntries, pendingInputYield } from "./runtime-approval-queue.js";
function valid() {
const payload = { questionSet: { questions: [{ id: "name", prompt: "What is the product name?" }] } };
return { scenarioId: "steering.approval-delivery", companyId: "company", leadId: "ceo", agents: [], links: [], humanQueries: [], reply: "", tasks: [{ id: "website" }], activity: [],
runs: [{ id: "asked", status: "succeeded" }, { id: "prior", status: "succeeded" }, { id: "canonical", status: "succeeded" }, { id: "consumer", status: "succeeded" }],
state: { task: { id: "website" }, worker: { id: "morgan" }, humanId: "human", questionRunId: "asked", questionRunYielded: true, workerRunId: "prior", workerWasReallyWaiting: true, ordinaryQueueOwned: true, ordinaryQueuedBeforeAnswer: true,
questionBefore: { id: "question", kind: "ask_user_questions", status: "pending", issueId: "website", createdByAgentId: "morgan", sourceRunId: "asked", payload },
questionAfter: { id: "question", status: "answered", resolvedByUserId: "human" as string | null, resolvedByAgentId: null as string | null, payload, result: { answers: [{ questionId: "name", otherText: "Acorn" }] } },
responseRetainedAcrossOrdinaryMutations: true, responseUneditable: true, queryReply: "Use Acorn for the existing naming question; submit that bound answer through its supported response control.",
actualOrdinaryEdit: true, actualOrdinaryReorder: true, ordinaryReorderRestricted: false, actualOrdinaryDiscard: true, humanAnswerSubmitted: true, canonicalAnswerAttributed: false,
responseQueue: { entries: [{ source: { kind: "interaction", interactionId: "question" }, canEdit: false, canDiscard: false }] }, consumingRunCount: 1, responseNativeInputBound: true, queueEmptyAfterDelivery: true, actualAcornResult: true, initialWorkRetained: true } };
}
it("accepts real human response retained separately from mutated ordinary comments", () => expect(gradeApprovalQueue(valid()).passed).toBe(true));
it("accepts a permitted actual canonical agent response with native source attribution", () => { const o = valid(); Object.assign(o.state, { canonicalAnswerAttributed: true, humanAnswerSubmitted: false, queryReply: "I answered Morgan's naming question with Acorn; the response is saved for normal continuation." }); Object.assign(o.state.questionAfter, { resolvedByUserId: null, resolvedByAgentId: "ceo" }); expect(gradeApprovalQueue(o).passed).toBe(true); });
it("rejects a fabricated question without actual prior native provenance", () => { const o = valid(); o.state.questionBefore.createdByAgentId = "other"; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects an ordinary comment masquerading as bound response", () => { const o = valid(); o.state.responseQueue.entries[0]!.source.kind = "comment"; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects a canonical response recording a different name", () => { const o = valid(); Object.assign(o.state, { canonicalAnswerAttributed: true, humanAnswerSubmitted: false }); Object.assign(o.state.questionAfter, { resolvedByUserId: null, resolvedByAgentId: "ceo" }); o.state.questionAfter.result.answers[0]!.otherText = "Birch"; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects wrong accepted answer", () => { const o = valid(); o.state.questionAfter.result.answers[0]!.otherText = "Other"; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects revoking the decision by discarding ordinary text", () => { const o = valid(); o.state.responseRetainedAcrossOrdinaryMutations = false; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects an editable interaction response", () => { const o = valid(); o.state.responseQueue.entries[0]!.canEdit = true; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects unsupported actor answering for the human", () => { const o = valid(); o.state.questionAfter.resolvedByUserId = "other"; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects ordinary message operations performed on a different queue", () => { const o = valid(); o.state.actualOrdinaryReorder = false; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects duplicate decision consumers", () => { const o = valid(); o.state.consumingRunCount = 2; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects native input lacking the bound question ID and answer", () => { const o = valid(); o.state.responseNativeInputBound = false; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects loss of earlier settled work", () => { const o = valid(); o.state.initialWorkRetained = false; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects missing actual response evidence", () => expect(gradeApprovalQueue(undefined).passed).toBe(false));
it("restricts every mutating ID to ordinary entries in a mixed two-comment plus response queue", () => {
const ordinary = { comment: { id: "ordinary" } }, response = { source: { kind: "interaction" }, comment: { id: "decision" } };
const ids = ordinaryApprovalEntries({ entries: [ordinary, response, { comment: { id: "other" } }] } as Parameters<typeof ordinaryApprovalEntries>[0]).map(row => row.comment.id);
expect(ids).toEqual(["ordinary", "other"]); expect(ids).not.toContain("decision");
});
it("accepts the real mixed-queue UI reorder refusal while preserving immutable decision placement", () => { const o = valid(); o.state.actualOrdinaryReorder = false; o.state.ordinaryReorderRestricted = true; expect(gradeApprovalQueue(o).passed).toBe(true); });
// The canonical reply is read by a person, not matched by words: the verdict rests on the saved answer path.
it.each(["Use Acorn for the existing naming question; submit that bound answer through its supported response control.", "Open Morgan's naming card and send Acorn there so the decision stays attached to it.", "Got it."])("verdict does not depend on reply wording: %s", reply => { const o = valid(); o.state.queryReply = reply; expect(gradeApprovalQueue(o).passed).toBe(true); });
it("rejects a question whose run never parked on it", () => { const o = valid(); o.state.questionRunYielded = false; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects a question attributed to the later active turn instead of the turn that asked", () => { const o = valid(); o.state.questionBefore.sourceRunId = "prior"; o.state.questionRunId = "prior"; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects an answer with no actual recording path", () => { const o = valid(); o.state.humanAnswerSubmitted = false; expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects a missing parked-question turn in the run ledger", () => { const o = valid(); o.runs.shift(); expect(gradeApprovalQueue(o).passed).toBe(false); });
// Shape recorded by the retained runner-opencode attempt (gha-42926276473): request_human_input parks the turn.
const parked = { id: "asked", status: "succeeded", resultJson: { nativeResult: { reportedWorkDisposition: "yielded", evidence: [{ ref: "interaction:question" }],
artifacts: [{ ref: "interaction:question", kind: "issue_thread_interaction" }], continuation: { kind: "response_wake", idempotencyKey: "interaction-response:question" } } } };
it("recognizes the product's parked pending-input turn", () => expect(pendingInputYield(parked, "question")).toBe(true));
it.each([
["still running", { ...parked, status: "running" }],
["completed instead of yielded", { ...parked, resultJson: { nativeResult: { ...parked.resultJson.nativeResult, reportedWorkDisposition: "completed" } } }],
["parked on another interaction", { ...parked, resultJson: { nativeResult: { ...parked.resultJson.nativeResult, continuation: { kind: "response_wake", idempotencyKey: "interaction-response:other" } } } }],
["no result", { id: "asked", status: "succeeded", resultJson: null }],
])("rejects a run that is not parked on the question (%s)", (_label, run) => expect(pendingInputYield(run, "question")).toBe(false));
// Retained runner-opencode attempt (gha-43077703607): the CEO cannot answer a board-only question form, so it relayed the
// person's answer as its own comment on the task. That comment queues beside the person's messages and the bound response.
it("accepts the chat lead's own relayed comment beside the bound response", () => { const o = valid();
o.state.responseQueue.entries.push({ comment: { id: "relay", authorAgentId: "ceo", authorUserId: null } } as never); expect(gradeApprovalQueue(o).passed).toBe(true); });
it("rejects a person's ordinary message left beside the bound response", () => { const o = valid();
o.state.responseQueue.entries.push({ comment: { id: "left", authorUserId: "human", authorAgentId: null } } as never); expect(gradeApprovalQueue(o).passed).toBe(false); });
it("rejects a second bound response entry", () => { const o = valid();
o.state.responseQueue.entries.push({ source: { kind: "interaction", interactionId: "question" }, canEdit: false, canDiscard: false } as never); expect(gradeApprovalQueue(o).passed).toBe(false); });
it("keeps only the person's own messages as editable ordinary entries", () => {
const queue = { entries: [{ comment: { id: "mine", authorUserId: "human", authorAgentId: null } }, { comment: { id: "relay", authorUserId: null, authorAgentId: "ceo" } },
{ source: { kind: "interaction" }, comment: { id: "decision" } }] } as unknown as Parameters<typeof ordinaryApprovalEntries>[0];
expect(ordinaryApprovalEntries(queue, "human").map(row => row.comment.id)).toEqual(["mine"]);
expect(leadRelayEntries(queue, "ceo").map(row => row.comment.id)).toEqual(["relay"]);
});