Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/api-read-receipts.test.ts
T
DottaandClaude Opus 5.5 f1aef0cb58 fix(evals): grade full cohort 3 false failures on saved state, read receipts and rendered links (#21008)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys against real agents
and grades saved product state
> - Full cohort 3 failed six cases where the product behaved correctly
and the eval was wrong
> - A wrong failure hides real product failures in the noise, so each
check must accept valid behavior
> - Each check must still fail the broken product behavior it exists to
catch
> - This pull request replaces those checks with checks of saved state,
read receipts and rendered links
> - The benefit is fewer false failures, with no case made easier

## Linked Issues or Issue Description

Refs #21000

**What happened?**
- `projects.project-work`, `tasks.private-task`,
`status-cards.other-authors-card` and part of
`task-coordination.delegate-with-context`: the CEO created a task with
no assignee, so the CEO got a queued run on it. The CEO then reassigned
the task. The server cancelled the queued run before it started
(`issue_reassigned` or `issue_assignee_changed`). The early poll
reported this as "Observed provider run failed", and the run policy
counted a never-started legacy run as a failed native run.
- `budgets-costs.repair-cost-recording`: after the retry, the chat run
re-read the original run record with `saveResponse`. The check accepted
only an exact whole-input digest, and this input had another shape.
- `secrets.personal-secret`: the reply linked My secrets and said "Enter
the value only there, never in chat". `solicitsChatCredential` read
"never in chat" as a request to send the value in chat.
- `companies.import-preview`: the reply linked
`/COM/agents/morgan-package`. The check accepted only `/agents/<uuid>`.
- `cases.close-case`: the closure was correct. The result turn said
"Case COM-C1 is saved as closed". A keyword pattern did not match that
sentence.

**Expected behavior**
Each check passes the recorded valid behavior and a reworded valid
variant. Each check fails the broken behavior.

**Steps to reproduce**
Grade the retained cohort-3 evidence with the old checks. They fail. The
calibration tests in this pull request use the same recorded shapes.

## What Changed

- Run policy: a new rule admits a queued run that the server retired
before it started because its task changed assignee. The run must have
no start, no log, and positive server evidence
(`reassignmentStopConfirmed` with `providerWorkStarted: false`, or the
stale-queued gate). The task must not be the source chat. A later run by
another agent on the same task must exist. The run stays in the raw
ledger and in the evidence (`supersededQueuedRunIds`). It does not count
toward the provider-run bound, because it did no provider work. A case
can have at most one such run (`catalog-superseded-queued-runs`). The
early poll defers it to the grade instead of failing at once. Cases read
the ledger without it. Other cancellations stay failures.
- `budgets-costs.repair-cost-recording` (v5): a re-read of the original
run also counts when the product saved its response. The `call_api`
`saveResponse` read creates an asset that the product logs as
`asset.created` by that run. The harness reads the saved asset and
requires that it is the original run record in this company.
- `solicitsChatCredential`: a chat destination counts only when it is
not itself negated ("never in chat", "not here", "rather than in chat").
This is a scope rule, not a new phrase exception. "Paste it here" and
mixed requests still fail. `secrets.personal-secret` is v18.
- `companies.import-preview` (v3): new `agentRouteNames` accepts
`/<prefix>/agents/<id or url key>[/<tab>]` that names the imported
agent. It rejects the existing Morgan, other companies and other
origins.
- `cases.close-case` (v2): the result turn is graded on its own saved
replies. Their run must read the saved case (`read_case` or `GET
/api/cases/{id}`, bound by the redaction-safe receipt). The reply must
render a link to that case. The case must be done. The wording is not
matched. `api-read-receipts` now binds the dedicated `read_case` tool.
- The four reassignment cases get new oracle versions. Descriptors are
updated. No scenario yaml changed, so the TestOps manifest does not
change.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts` on the 17
affected test files: 516 tests pass. The files include
`chat-catalog-run-policy`, `chat-flow`, `non-execution-wake`,
`chat-catalog-consistency`, `chat-catalog-fingerprint`,
`identity-accounting`, `external-credential-language`,
`external-personal-secret`, `identity-portability`,
`integrator-case-close`, `api-read-receipts` and the other
secret-language users.
- Each change has calibration tests: the recorded valid shape, a
reworded valid shape, and broken shapes (no successor run, a started or
other cancellation, churn; another run's saved record; "paste it here";
the existing Morgan's route; an unread or unlinked result turn).
- `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
reports no change.
- One paid run on a combined branch with the companion harness pull
request. The result is in a comment on this pull request.

## Risks

- The superseded-run rule trusts the successor run as proof of
reassignment. A case oracle still grades the task assignee and the
successor's own outcome.
- The close-case result turn no longer checks the reply's words. A reply
that reads the closed case and links it but states the wrong result
passes this one check. The first-turn premature-claim check and the
negative-case check do not change.
- The credential grader change also affects other secret cases. It only
removes the negated-destination false positive.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:07:06 -06:00

79 lines
6.1 KiB
TypeScript

import { expect, it } from "vitest";
import { apiReadStatuses, apiReadDigests, readRecord, recordParams } from "./api-read-receipts.js";
import { savedCall } from "./api-read-receipts.test-fixtures.js";
const activity = (pathParams: Record<string, string>, query?: Record<string, unknown>) => ({ operationId: "GET /api/issues/{id}/activity", pathParams, ...(query ? { query } : {}) });
const reads = [{ operationId: "GET /api/issues/{id}/activity", pathParams: recordParams("id", "task-uuid", "COM-1"), paged: true }];
it("the redactor removes the response body and the task id, keeping only the input digest", () => {
const saved = JSON.stringify(savedCall(activity({ id: "task-uuid" }), 200, "1", "run-1", [{ entityId: "task-uuid" }]));
expect(saved).not.toContain("task-uuid");
expect(saved).toContain("apiPayloadSha256");
});
it("binds a successful saved read to the record by id, identifier and paging query", () => {
expect(readRecord(savedCall(activity({ id: "task-uuid" }), 200), reads)).toBe(true);
expect(readRecord(savedCall(activity({ id: "COM-1" }), 200), reads)).toBe(true);
expect(readRecord(savedCall(activity({ id: "COM-1" }, { limit: 50 }), 200), reads)).toBe(true);
expect(readRecord(savedCall(activity({ id: "COM-1" }, { limit: "50" }), 200), reads)).toBe(true);
});
it("rejects other records, other operations, failures and unpaired results", () => {
expect(readRecord(savedCall(activity({ id: "other-task" }), 200), reads)).toBe(false);
expect(readRecord(savedCall({ operationId: "GET /api/issues/{id}", pathParams: { id: "task-uuid" } }, 200), reads)).toBe(false);
expect(readRecord(savedCall(activity({ id: "task-uuid" }), 403), reads)).toBe(false);
expect(readRecord(savedCall(activity({ id: "task-uuid" }), 200).slice(0, 1), reads)).toBe(false);
expect(readRecord([...savedCall(activity({ id: "task-uuid" }), 200, "1", "run-a").slice(0, 1), ...savedCall(activity({ id: "other" }), 200, "1", "run-b").slice(1)], reads)).toBe(false);
});
it("reports the status of a bound read so a 404 can prove an unavailable record", () => {
expect(apiReadStatuses(savedCall(activity({ id: "task-uuid" }), 404), apiReadDigests(reads))).toEqual([{ operationId: "GET /api/issues/{id}/activity", ok: false, status: 404 }]);
});
const getTask = (taskId: string, status: number, tool = "get_task", runId = "run-1") => savedCall({ taskId }, status, "2", runId, { id: taskId }, tool, "GET /api/issues/{id}");
const issueRead = [{ operationId: "GET /api/issues/{id}", pathParams: recordParams("id", "task-uuid", "COM-1") }];
it("the redactor keeps a dedicated get_task input in plaintext and its result as a receipt", () => {
const [use, result] = getTask("task-uuid", 404);
expect(JSON.stringify(use)).toContain("task-uuid");
expect(JSON.stringify(result)).not.toContain("task-uuid");
expect(JSON.stringify(result)).toContain("apiPayloadSha256");
});
it("binds dedicated task read tools to the record like the call_api input the server builds", () => {
expect(apiReadStatuses(getTask("task-uuid", 404), apiReadDigests(issueRead))).toEqual([{ operationId: "GET /api/issues/{id}", ok: false, status: 404 }]);
expect(readRecord(getTask("COM-1", 200), issueRead)).toBe(true);
for (const tool of ["paperclip_get_task", "mcp__paperclip__get_task"]) expect(readRecord(getTask("task-uuid", 200, tool), issueRead)).toBe(true);
const docs = [{ operationId: "GET /api/issues/{id}/documents/{key}", pathParams: [{ id: "task-uuid", key: "plan" }] }];
expect(readRecord(savedCall({ taskId: "task-uuid", key: "plan" }, 200, "3", "run-1", {}, "read_task_document", "GET /api/issues/{id}/documents/{key}"), docs)).toBe(true);
});
it("rejects dedicated reads of other records, other tools, mismatched receipts and unpaired results", () => {
expect(readRecord(getTask("other-task", 200), issueRead)).toBe(false);
expect(readRecord(getTask("task-uuid", 200, "comment_on_task"), issueRead)).toBe(false);
expect(readRecord(getTask("task-uuid", 200, "get_task_context"), issueRead)).toBe(false);
expect(readRecord(savedCall({ taskId: "task-uuid" }, 200, "2", "run-1", {}, "get_task", "GET /api/issues/{id}/comments"), issueRead)).toBe(false);
expect(readRecord(getTask("task-uuid", 200).slice(0, 1), issueRead)).toBe(false);
expect(readRecord([...getTask("task-uuid", 200, "get_task", "run-a").slice(0, 1), ...getTask("task-uuid", 200, "get_task", "run-b").slice(1)], issueRead)).toBe(false);
});
// Full cohort 3 (cases.close-case): the chat read the closed case with the dedicated read_case tool,
// whose saved input is redacted to the digest of `{ caseId }`.
it("binds a dedicated read_case to the case by id, and rejects other cases and failures", () => {
const id = "c80a5b89-41a8-4e97-b812-e4081d92d53b", other = "d80a5b89-41a8-4e97-b812-e4081d92d53b";
const reads = [{ operationId: "GET /api/cases/{id}", pathParams: recordParams("id", id, "COM-C1") }];
const read = (caseId: string, status = 200, tool = "read_case") => savedCall({ caseId }, status, "2", "run-1", { id: caseId }, tool, "GET /api/cases/{id}");
expect(readRecord(read(id), reads)).toBe(true);
// Paid run 43143392360: the result turn read the case record page, `{ caseId, section: "case" }`.
const page = (input: Record<string, unknown>) => savedCall(input, 200, "2", "run-1", { id }, "read_case", "GET /api/cases/{id}");
for (const input of [{ caseId: id, section: "case" }, { caseId: id, section: "case", offsetBytes: 0 }, { caseId: id, section: "case", limitBytes: 8192 }])
expect(readRecord(page(input), reads), JSON.stringify(input)).toBe(true);
// Pages that may omit the case's status and completion do not count.
for (const input of [{ caseId: other, section: "case" }, { caseId: id, section: "document", documentKey: "review" },
{ caseId: id, section: "fields" }, { caseId: id, section: "case", limitBytes: 4 }])
expect(readRecord(page(input), reads), JSON.stringify(input)).toBe(false);
expect(readRecord(read(id, 200, "mcp__paperclip__read_case"), reads)).toBe(true);
expect(readRecord(read(other), reads)).toBe(false);
expect(readRecord(read(id, 404), reads)).toBe(false);
expect(readRecord(read(id, 200, "update_case"), reads)).toBe(false);
});