Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/api-read-receipts.ts
T
DottaandClaude Opus 5.5 f1aef0cb58 fix(evals): grade full cohort 3 false failures on saved state, read receipts and rendered links (#21008)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys against real agents
and grades saved product state
> - Full cohort 3 failed six cases where the product behaved correctly
and the eval was wrong
> - A wrong failure hides real product failures in the noise, so each
check must accept valid behavior
> - Each check must still fail the broken product behavior it exists to
catch
> - This pull request replaces those checks with checks of saved state,
read receipts and rendered links
> - The benefit is fewer false failures, with no case made easier

## Linked Issues or Issue Description

Refs #21000

**What happened?**
- `projects.project-work`, `tasks.private-task`,
`status-cards.other-authors-card` and part of
`task-coordination.delegate-with-context`: the CEO created a task with
no assignee, so the CEO got a queued run on it. The CEO then reassigned
the task. The server cancelled the queued run before it started
(`issue_reassigned` or `issue_assignee_changed`). The early poll
reported this as "Observed provider run failed", and the run policy
counted a never-started legacy run as a failed native run.
- `budgets-costs.repair-cost-recording`: after the retry, the chat run
re-read the original run record with `saveResponse`. The check accepted
only an exact whole-input digest, and this input had another shape.
- `secrets.personal-secret`: the reply linked My secrets and said "Enter
the value only there, never in chat". `solicitsChatCredential` read
"never in chat" as a request to send the value in chat.
- `companies.import-preview`: the reply linked
`/COM/agents/morgan-package`. The check accepted only `/agents/<uuid>`.
- `cases.close-case`: the closure was correct. The result turn said
"Case COM-C1 is saved as closed". A keyword pattern did not match that
sentence.

**Expected behavior**
Each check passes the recorded valid behavior and a reworded valid
variant. Each check fails the broken behavior.

**Steps to reproduce**
Grade the retained cohort-3 evidence with the old checks. They fail. The
calibration tests in this pull request use the same recorded shapes.

## What Changed

- Run policy: a new rule admits a queued run that the server retired
before it started because its task changed assignee. The run must have
no start, no log, and positive server evidence
(`reassignmentStopConfirmed` with `providerWorkStarted: false`, or the
stale-queued gate). The task must not be the source chat. A later run by
another agent on the same task must exist. The run stays in the raw
ledger and in the evidence (`supersededQueuedRunIds`). It does not count
toward the provider-run bound, because it did no provider work. A case
can have at most one such run (`catalog-superseded-queued-runs`). The
early poll defers it to the grade instead of failing at once. Cases read
the ledger without it. Other cancellations stay failures.
- `budgets-costs.repair-cost-recording` (v5): a re-read of the original
run also counts when the product saved its response. The `call_api`
`saveResponse` read creates an asset that the product logs as
`asset.created` by that run. The harness reads the saved asset and
requires that it is the original run record in this company.
- `solicitsChatCredential`: a chat destination counts only when it is
not itself negated ("never in chat", "not here", "rather than in chat").
This is a scope rule, not a new phrase exception. "Paste it here" and
mixed requests still fail. `secrets.personal-secret` is v18.
- `companies.import-preview` (v3): new `agentRouteNames` accepts
`/<prefix>/agents/<id or url key>[/<tab>]` that names the imported
agent. It rejects the existing Morgan, other companies and other
origins.
- `cases.close-case` (v2): the result turn is graded on its own saved
replies. Their run must read the saved case (`read_case` or `GET
/api/cases/{id}`, bound by the redaction-safe receipt). The reply must
render a link to that case. The case must be done. The wording is not
matched. `api-read-receipts` now binds the dedicated `read_case` tool.
- The four reassignment cases get new oracle versions. Descriptors are
updated. No scenario yaml changed, so the TestOps manifest does not
change.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts` on the 17
affected test files: 516 tests pass. The files include
`chat-catalog-run-policy`, `chat-flow`, `non-execution-wake`,
`chat-catalog-consistency`, `chat-catalog-fingerprint`,
`identity-accounting`, `external-credential-language`,
`external-personal-secret`, `identity-portability`,
`integrator-case-close`, `api-read-receipts` and the other
secret-language users.
- Each change has calibration tests: the recorded valid shape, a
reworded valid shape, and broken shapes (no successor run, a started or
other cancellation, churn; another run's saved record; "paste it here";
the existing Morgan's route; an unread or unlinked result turn).
- `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
reports no change.
- One paid run on a combined branch with the companion harness pull
request. The result is in a comment on this pull request.

## Risks

- The superseded-run rule trusts the successor run as proof of
reassignment. A case oracle still grades the task assignee and the
successor's own outcome.
- The close-case result turn no longer checks the reply's words. A reply
that reads the closed case and links it but states the wrong result
passes this one check. The first-turn premature-claim check and the
negative-case check do not change.
- The credential grader change also affects other secret cases. It only
removes the negated-destination false positive.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:07:06 -06:00

129 lines
8.0 KiB
TypeScript

import { nativeSha256 } from "../../../server/src/services/native-runtime/canonical.js";
/** The digest the port saves for a redacted call_api input (the server's own canonical SHA-256). */
export const callApiInputDigest = (input: Record<string, unknown>) => nativeSha256(input);
/**
* Record-bound read receipts that survive run-event redaction.
*
* PaperclipControlPlanePort saves call_api inputs and results with their
* payloads replaced by a SHA-256 digest; only the operationId, ok and status
* remain. A read is therefore tied to a record by recomputing the digest of
* every input that names it (path parameter by id or identifier, with no query
* or a small paging query) and requiring the paired result to report the
* expected status. The agent's own wording is never matched.
*/
const objectRecord = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
const PAGE_LIMITS = Array.from({ length: 1001 }, (_, index) => index).flatMap(limit => [{ limit }, { limit: String(limit) }]);
const cache = new Map<string, ReadonlySet<string>>();
const CASE_READ_OPERATION = "GET /api/cases/{id}";
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
// The first page of the case record section (the default), which carries the case's status and completion.
// Fields-only pages and byte-limited pages may omit them, so they do not count as reading the case.
// An explicit limit equal to the default page size (8 KiB, also the maximum) reads the same page.
const CASE_FIRST_PAGES: ReadonlyArray<Record<string, unknown>> = [undefined, "case"].flatMap(section => [undefined, 0].flatMap(offsetBytes =>
[undefined, 8 * 1024].map(limitBytes => ({ ...(section ? { section } : {}), ...(offsetBytes !== undefined ? { offsetBytes } : {}),
...(limitBytes !== undefined ? { limitBytes } : {}) }))));
const caseReadTool = (name: unknown) => typeof name === "string" && /^(?:[A-Za-z0-9_-]*__)?(?:paperclip_)?read_case$/.test(name);
export interface ApiRead {
operationId: string;
/** Each alternative names the record, e.g. [{ id: uuid }, { id: "COM-1" }]; `undefined` means the server fills the path. */
pathParams: ReadonlyArray<Record<string, string> | undefined>;
/** Accept a `limit` paging query (number or string, 0..1000) in addition to none. */
paged?: boolean;
/** Other accepted queries, e.g. a project filter. */
queries?: ReadonlyArray<Record<string, unknown>>;
}
/** Digests of every accepted call_api input for these reads. */
export function apiReadDigests(reads: readonly ApiRead[]): ReadonlySet<string> {
const key = JSON.stringify(reads);
const cached = cache.get(key); if (cached) return cached;
const digests = new Set<string>();
for (const read of reads) {
const queries: Array<Record<string, unknown> | undefined> = [undefined, {}, ...(read.paged ? PAGE_LIMITS : []), ...(read.queries ?? [])];
// The dedicated read_case tool saves its own redacted input and reports GET /api/cases/{id}. Its first page of
// the case record is `{ caseId, section?: "case", offsetBytes?: 0 }` (caseReadPageShape).
if (read.operationId === CASE_READ_OPERATION) for (const pathParams of read.pathParams)
if (pathParams?.id && UUID.test(pathParams.id)) for (const page of CASE_FIRST_PAGES) digests.add(callApiInputDigest({ caseId: pathParams.id, ...page }));
for (const pathParams of read.pathParams) for (const query of queries) {
digests.add(callApiInputDigest({ operationId: read.operationId, ...(pathParams ? { pathParams } : {}), ...(query ? { query } : {}) }));
if (!pathParams) digests.add(callApiInputDigest({ operationId: read.operationId, pathParams: {}, ...(query ? { query } : {}) }));
}
}
cache.set(key, digests);
return digests;
}
/** Path parameter alternatives naming one record by id and, when known, its human identifier. */
export function recordParams(name: string, ...values: Array<string | null | undefined>): Array<Record<string, string>> {
return [...new Set(values.filter((value): value is string => typeof value === "string" && value.length > 0))].map(value => ({ [name]: value }));
}
/**
* Dedicated task read tools and the call_api input the server builds for them
* (taskBridgeApiRequest in paperclip-runner-tool-authority.ts). Their saved
* tool_use keeps the plaintext taskId, while the result carries the same
* redacted receipt (apiOperationId, ok, status) as call_api. Harnesses may
* namespace the name (paperclip_get_task, mcp__paperclip__get_task).
*/
const TASK_READ_TOOLS: Readonly<Record<string, string>> = {
get_task: "GET /api/issues/{id}",
list_task_documents: "GET /api/issues/{id}/documents",
read_task_document: "GET /api/issues/{id}/documents/{key}",
};
function taskReadTool(name: unknown): string | undefined {
const match = typeof name === "string" ? /^(?:[A-Za-z0-9_-]*__)?(?:paperclip_)?([a-z_]+)$/.exec(name) : null;
return match && Object.hasOwn(TASK_READ_TOOLS, match[1]!) ? match[1] : undefined;
}
function taskReadInput(tool: string, input: Record<string, unknown>): Record<string, unknown> | undefined {
if (typeof input.taskId !== "string" || !input.taskId) return;
if (tool === "read_task_document" && (typeof input.key !== "string" || !input.key)) return;
return { operationId: TASK_READ_TOOLS[tool], pathParams: { id: input.taskId, ...(tool === "read_task_document" ? { key: input.key } : {}) } };
}
/**
* Statuses of this run's record reads whose input digest is one of `digests`,
* paired to their tool_use by run and item id: call_api uses by their saved
* digest, and dedicated task read tools by the digest of the call_api input
* the server builds from their plaintext input.
*/
export function apiReadStatuses(events: readonly Record<string, unknown>[], digests: ReadonlySet<string>): Array<{ operationId: string; ok: boolean; status: number }> {
const uses = new Map<string, string>();
const statuses: Array<{ operationId: string; ok: boolean; status: number }> = [];
for (const row of events) {
const envelope = objectRecord(objectRecord(row.payload).prpEvent), item = objectRecord(objectRecord(envelope.payload).item);
const run = String(row.runId ?? envelope.runId ?? "");
if (item.type === "tool_use" && item.name === "call_api") {
const input = objectRecord(item.input);
if (input.apiPayloadRedacted === true && typeof input.operationId === "string" && digests.has(String(input.apiPayloadSha256))) uses.set(`${run}:${String(item.id)}`, input.operationId);
continue;
}
if (item.type === "tool_use" && caseReadTool(item.name)) {
const input = objectRecord(item.input);
// Saved redacted as `read_case`; a namespaced name may keep its plaintext `{ caseId }`.
const digest = input.apiPayloadRedacted === true ? String(input.apiPayloadSha256)
: typeof input.caseId === "string" && Object.keys(input).length === 1 ? callApiInputDigest({ caseId: input.caseId }) : undefined;
if (digest && digests.has(digest)) uses.set(`${run}:${String(item.id)}`, CASE_READ_OPERATION);
continue;
}
const tool = item.type === "tool_use" ? taskReadTool(item.name) : undefined;
if (tool) {
const request = taskReadInput(tool, objectRecord(item.input));
if (request && digests.has(callApiInputDigest(request))) uses.set(`${run}:${String(item.id)}`, String(request.operationId));
continue;
}
if (item.type !== "tool_result") continue;
const operationId = uses.get(`${run}:${String(item.tool_use_id)}`), result = objectRecord(item.result);
if (!operationId || result.apiOperationId !== operationId || typeof result.status !== "number") continue;
statuses.push({ operationId, ok: result.ok === true, status: result.status });
}
return statuses;
}
/** True when the run successfully completed one of these record-bound reads. */
export function readRecord(events: readonly Record<string, unknown>[], reads: readonly ApiRead[]) {
return apiReadStatuses(events, apiReadDigests(reads)).some(read => read.ok && read.status >= 200 && read.status < 300);
}