Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/api-read-receipts.ts
T
DottaandClaude Opus 5.5 8e332ccb5e fix(evals): name stale SDK pins and fix four chat-eval harness defects (#20579)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat feature eval catalog (`tests/runner-e2e`) runs real chat
journeys against a real server and grades the saved state, the run's
tool events and the rendered reply
> - The full-cohort run on master `f4337bb5fb` had 12 failures that come
from the harness, not from the product or the model
> - Six plugin and secrets cells stopped in 7–9 s, before the server
started, because the SDK public source pins were stale
> - Four more cells failed on harness timing, a first-visit prompt, or a
read receipt that ignored a dedicated tool
> - This pull request fixes those four harness defects and adds a cheap
guard that names stale SDK pins
> - The benefit is that these cells now report on the product and the
model again

## Linked Issues or Issue Description

Refs #19614, #20550, #20453, #20417, #20483

**What happened**

On master `f4337bb5fb`, these chat eval cells failed for harness
reasons:

- `plugins.configure-plugin`, `plugins.disable-plugin`,
`plugins.install-plugin`, `plugins.use-plugin-tool`,
`routines.plugin-managed-routine` and `secrets.least-scope-delivery`
stopped before the server started. The source SDK privacy scan refused,
because `tests/runner-e2e/source-sdk-public-inputs.json` did not match
`packages/shared/src`. #20550 and #20362 already resynced the pins on
master. The pins are current at `ce1ee9e89f`.
- `agent-chat.track-delegated-work` and
`navigation-preferences.edit-profile`: the reply was correct and had the
right link. The case read the rendered links immediately after the runs
settled, before the reply rendered, so it recorded no links.
- `inbox-search.archive-done` and `inbox-search.restore-archive`: setup
completes five tasks, so the GitHub star prompt from #20453 opened over
the chat and blocked the composer.
- `status-cards.explain-card`: the follow-up run read the deleted source
with `get_task` and received a 404. The reply said so. `apiReadStatuses`
counted only `call_api` reads, so the grader recorded no unavailable
read.

Why the existing pin check did not stop the drift: `ci / Runner support
suite` runs the source SDK tests and is part of the required `ci /
verify`. On #19614 it passed at 03:43Z on a merge ref from before #20417
(06:12Z) and #20483 (08:14Z). #19614 merged at 14:02Z without a new CI
run on current master. After that, the suite failed on every PR until
#20550. So the check exists and fires; the gap was a stale green result,
and the failure message was opaque (`SDK public source proof
unavailable`).

**Expected behavior**

- A change to a pinned SDK input without a pin refresh fails with the
changed files and the command that fixes it.
- Cases read rendered links only after the saved reply renders.
- The eval browser is a returning user's browser that has already seen
the star prompt. The product does not change.
- A dedicated task read tool (`get_task`, `list_task_documents`,
`read_task_document`, also when namespaced) counts as a read of the
record it names, with the status its receipt reports.

**Steps to reproduce**

1. Run `chat-features.runner-opencode.local.<case>` for one of the cases
above on master `f4337bb5fb`.
2. Read `result.json` and `snapshots/chat-catalog-observation.json` in
the evidence bundle.
3. See the failures described above.

## What Changed

- `tests/runner-e2e/source-sdk-public-inputs.mjs`: add
`staleSdkPublicInputs(root)`. It compares the registry with `git ls-tree
HEAD` for every pinned input (the roots and build inputs that the pins
file lists), and returns each changed, unpinned or removed file.
- `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: a test fails
when the committed pins do not match the committed tree. The message
names the files and gives the `--write` command. The Runner support
suite runs it for every `packages/**` change. A synthetic test covers
changed, unpinned and removed inputs.
- `tests/runner-e2e/chat-catalog-domains/runtime-handoffs.ts` and
`identity-profile.ts`: call the existing `waitForRenderedReply` before
they read rendered links.
- `tests/runner-e2e/chat-returning-browser.ts` (new) and
`chat-catalog-flow.ts`: inside the flow's failure boundary, before
setup, mark `paperclip.github-star-prompt.seen` in the eval browser. An
init script covers later documents of the app origin, and the open page
is marked directly. A test checks that the key equals
`STAR_PROMPT_SEEN_KEY` in `ui/src/components/GitHubStarPrompt.tsx`.
- `tests/runner-e2e/chat-catalog-domains/api-read-receipts.ts`: bind
dedicated task read tools by the digest of the `call_api` input that the
server builds for them (`taskBridgeApiRequest`). The result must carry
the same `apiOperationId`. Other tools, other records, mismatched
receipts and unpaired results do not count.
- Bump `oracleVersion` in code and descriptor for every case that uses
`api-read-receipts.ts` (shared grader change) and for
`navigation-preferences.edit-profile`. Add
`internal-comment-reference.ts` to the source files of the two cases
that now import it.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts`
(whole config): 11301 passed; 1 local-only failure in
`controller-replacement-smoke.test.ts` (server spawn, no changed
imports).
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
--testTimeout=180000 tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts
tests/runner-e2e/chat-returning-browser.test.ts`
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
source-sdk`
- `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse
HEAD) --check` reports `changed: false`.
- Red checks: with the `f4337bb5fb` registry, the new pin test fails and
names 15 inputs. Without the new waits, the two rendered-reply tests
fail. A regrade of the retained `explain-card` run events now gives the
follow-up run `unavailable: true, live: false` and the first run `live:
true`.
- Paid runs of the affected cells
(`chat-features.runner-opencode.local.*`):
- Run 42852698879 (head `d54c04d5f0`): `track-delegated-work`,
`restore-archive`, `explain-card`, `configure-plugin` and
`plugin-managed-routine` pass. I read each reply and saved state.
`track-delegated-work` now records the rendered `/COM/issues/COM-2`
link. `explain-card` records `laterUnavailable: true, laterLive: false`
from the follow-up `get_task` 404.
- `plugins.install-plugin` now starts its server (the pin refusal is
gone). It fails a later grading check, `truthful-installation-handoff`:
the reply sends the person to the plugin settings page but does not name
the package. This is outside this PR.
- `inbox-search.archive-done`: the star prompt no longer blocks the
composer, and the query is sent. The agent then archives the 501 Done
tasks one at a time (about 2 s each, 57 in the 240 s window), because
only the per-task archive route exists. This is outside this PR.
- `navigation-preferences.edit-profile` did not start in that run (the
AWS credentials step timed out). Run 42857020103 (head `2f0b80aa84`):
pass. The agent write gets 401, the reply links
`/COM/company/settings/instance/profile`, and the person follows the
rendered link and saves the name.

## Risks

- Low risk. Test harness only; no product code changes.
- The receipt change counts more reads as record reads. It only adds
dedicated reads bound to the exact record id and operation, so a grader
that needs a read can pass on a genuine `get_task` read. The affected
cases have new oracle versions.
- The star prompt is not tested by any chat catalog case. Product e2e
tests for the prompt do not use this flow.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 11:01:44 -06:00

108 lines
6.1 KiB
TypeScript

import { nativeSha256 } from "../../../server/src/services/native-runtime/canonical.js";
/** The digest the port saves for a redacted call_api input (the server's own canonical SHA-256). */
export const callApiInputDigest = (input: Record<string, unknown>) => nativeSha256(input);
/**
* Record-bound read receipts that survive run-event redaction.
*
* PaperclipControlPlanePort saves call_api inputs and results with their
* payloads replaced by a SHA-256 digest; only the operationId, ok and status
* remain. A read is therefore tied to a record by recomputing the digest of
* every input that names it (path parameter by id or identifier, with no query
* or a small paging query) and requiring the paired result to report the
* expected status. The agent's own wording is never matched.
*/
const objectRecord = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
const PAGE_LIMITS = Array.from({ length: 1001 }, (_, index) => index).flatMap(limit => [{ limit }, { limit: String(limit) }]);
const cache = new Map<string, ReadonlySet<string>>();
export interface ApiRead {
operationId: string;
/** Each alternative names the record, e.g. [{ id: uuid }, { id: "COM-1" }]; `undefined` means the server fills the path. */
pathParams: ReadonlyArray<Record<string, string> | undefined>;
/** Accept a `limit` paging query (number or string, 0..1000) in addition to none. */
paged?: boolean;
/** Other accepted queries, e.g. a project filter. */
queries?: ReadonlyArray<Record<string, unknown>>;
}
/** Digests of every accepted call_api input for these reads. */
export function apiReadDigests(reads: readonly ApiRead[]): ReadonlySet<string> {
const key = JSON.stringify(reads);
const cached = cache.get(key); if (cached) return cached;
const digests = new Set<string>();
for (const read of reads) {
const queries: Array<Record<string, unknown> | undefined> = [undefined, {}, ...(read.paged ? PAGE_LIMITS : []), ...(read.queries ?? [])];
for (const pathParams of read.pathParams) for (const query of queries) {
digests.add(callApiInputDigest({ operationId: read.operationId, ...(pathParams ? { pathParams } : {}), ...(query ? { query } : {}) }));
if (!pathParams) digests.add(callApiInputDigest({ operationId: read.operationId, pathParams: {}, ...(query ? { query } : {}) }));
}
}
cache.set(key, digests);
return digests;
}
/** Path parameter alternatives naming one record by id and, when known, its human identifier. */
export function recordParams(name: string, ...values: Array<string | null | undefined>): Array<Record<string, string>> {
return [...new Set(values.filter((value): value is string => typeof value === "string" && value.length > 0))].map(value => ({ [name]: value }));
}
/**
* Dedicated task read tools and the call_api input the server builds for them
* (taskBridgeApiRequest in paperclip-runner-tool-authority.ts). Their saved
* tool_use keeps the plaintext taskId, while the result carries the same
* redacted receipt (apiOperationId, ok, status) as call_api. Harnesses may
* namespace the name (paperclip_get_task, mcp__paperclip__get_task).
*/
const TASK_READ_TOOLS: Readonly<Record<string, string>> = {
get_task: "GET /api/issues/{id}",
list_task_documents: "GET /api/issues/{id}/documents",
read_task_document: "GET /api/issues/{id}/documents/{key}",
};
function taskReadTool(name: unknown): string | undefined {
const match = typeof name === "string" ? /^(?:[A-Za-z0-9_-]*__)?(?:paperclip_)?([a-z_]+)$/.exec(name) : null;
return match && Object.hasOwn(TASK_READ_TOOLS, match[1]!) ? match[1] : undefined;
}
function taskReadInput(tool: string, input: Record<string, unknown>): Record<string, unknown> | undefined {
if (typeof input.taskId !== "string" || !input.taskId) return;
if (tool === "read_task_document" && (typeof input.key !== "string" || !input.key)) return;
return { operationId: TASK_READ_TOOLS[tool], pathParams: { id: input.taskId, ...(tool === "read_task_document" ? { key: input.key } : {}) } };
}
/**
* Statuses of this run's record reads whose input digest is one of `digests`,
* paired to their tool_use by run and item id: call_api uses by their saved
* digest, and dedicated task read tools by the digest of the call_api input
* the server builds from their plaintext input.
*/
export function apiReadStatuses(events: readonly Record<string, unknown>[], digests: ReadonlySet<string>): Array<{ operationId: string; ok: boolean; status: number }> {
const uses = new Map<string, string>();
const statuses: Array<{ operationId: string; ok: boolean; status: number }> = [];
for (const row of events) {
const envelope = objectRecord(objectRecord(row.payload).prpEvent), item = objectRecord(objectRecord(envelope.payload).item);
const run = String(row.runId ?? envelope.runId ?? "");
if (item.type === "tool_use" && item.name === "call_api") {
const input = objectRecord(item.input);
if (input.apiPayloadRedacted === true && typeof input.operationId === "string" && digests.has(String(input.apiPayloadSha256))) uses.set(`${run}:${String(item.id)}`, input.operationId);
continue;
}
const tool = item.type === "tool_use" ? taskReadTool(item.name) : undefined;
if (tool) {
const request = taskReadInput(tool, objectRecord(item.input));
if (request && digests.has(callApiInputDigest(request))) uses.set(`${run}:${String(item.id)}`, String(request.operationId));
continue;
}
if (item.type !== "tool_result") continue;
const operationId = uses.get(`${run}:${String(item.tool_use_id)}`), result = objectRecord(item.result);
if (!operationId || result.apiOperationId !== operationId || typeof result.status !== "number") continue;
statuses.push({ operationId, ok: result.ok === true, status: result.status });
}
return statuses;
}
/** True when the run successfully completed one of these record-bound reads. */
export function readRecord(events: readonly Record<string, unknown>[], reads: readonly ApiRead[]) {
return apiReadStatuses(events, apiReadDigests(reads)).some(read => read.ok && read.status >= 200 && read.status < 300);
}