mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog runs real chat journeys against real agents and grades saved product state > - Full cohort 3 failed six cases where the product behaved correctly and the eval was wrong > - A wrong failure hides real product failures in the noise, so each check must accept valid behavior > - Each check must still fail the broken product behavior it exists to catch > - This pull request replaces those checks with checks of saved state, read receipts and rendered links > - The benefit is fewer false failures, with no case made easier ## Linked Issues or Issue Description Refs #21000 **What happened?** - `projects.project-work`, `tasks.private-task`, `status-cards.other-authors-card` and part of `task-coordination.delegate-with-context`: the CEO created a task with no assignee, so the CEO got a queued run on it. The CEO then reassigned the task. The server cancelled the queued run before it started (`issue_reassigned` or `issue_assignee_changed`). The early poll reported this as "Observed provider run failed", and the run policy counted a never-started legacy run as a failed native run. - `budgets-costs.repair-cost-recording`: after the retry, the chat run re-read the original run record with `saveResponse`. The check accepted only an exact whole-input digest, and this input had another shape. - `secrets.personal-secret`: the reply linked My secrets and said "Enter the value only there, never in chat". `solicitsChatCredential` read "never in chat" as a request to send the value in chat. - `companies.import-preview`: the reply linked `/COM/agents/morgan-package`. The check accepted only `/agents/<uuid>`. - `cases.close-case`: the closure was correct. The result turn said "Case COM-C1 is saved as closed". A keyword pattern did not match that sentence. **Expected behavior** Each check passes the recorded valid behavior and a reworded valid variant. Each check fails the broken behavior. **Steps to reproduce** Grade the retained cohort-3 evidence with the old checks. They fail. The calibration tests in this pull request use the same recorded shapes. ## What Changed - Run policy: a new rule admits a queued run that the server retired before it started because its task changed assignee. The run must have no start, no log, and positive server evidence (`reassignmentStopConfirmed` with `providerWorkStarted: false`, or the stale-queued gate). The task must not be the source chat. A later run by another agent on the same task must exist. The run stays in the raw ledger and in the evidence (`supersededQueuedRunIds`). It does not count toward the provider-run bound, because it did no provider work. A case can have at most one such run (`catalog-superseded-queued-runs`). The early poll defers it to the grade instead of failing at once. Cases read the ledger without it. Other cancellations stay failures. - `budgets-costs.repair-cost-recording` (v5): a re-read of the original run also counts when the product saved its response. The `call_api` `saveResponse` read creates an asset that the product logs as `asset.created` by that run. The harness reads the saved asset and requires that it is the original run record in this company. - `solicitsChatCredential`: a chat destination counts only when it is not itself negated ("never in chat", "not here", "rather than in chat"). This is a scope rule, not a new phrase exception. "Paste it here" and mixed requests still fail. `secrets.personal-secret` is v18. - `companies.import-preview` (v3): new `agentRouteNames` accepts `/<prefix>/agents/<id or url key>[/<tab>]` that names the imported agent. It rejects the existing Morgan, other companies and other origins. - `cases.close-case` (v2): the result turn is graded on its own saved replies. Their run must read the saved case (`read_case` or `GET /api/cases/{id}`, bound by the redaction-safe receipt). The reply must render a link to that case. The case must be done. The wording is not matched. `api-read-receipts` now binds the dedicated `read_case` tool. - The four reassignment cases get new oracle versions. Descriptors are updated. No scenario yaml changed, so the TestOps manifest does not change. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts` on the 17 affected test files: 516 tests pass. The files include `chat-catalog-run-policy`, `chat-flow`, `non-execution-wake`, `chat-catalog-consistency`, `chat-catalog-fingerprint`, `identity-accounting`, `external-credential-language`, `external-personal-secret`, `identity-portability`, `integrator-case-close`, `api-read-receipts` and the other secret-language users. - Each change has calibration tests: the recorded valid shape, a reworded valid shape, and broken shapes (no successor run, a started or other cancellation, churn; another run's saved record; "paste it here"; the existing Morgan's route; an unread or unlinked result turn). - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - One paid run on a combined branch with the companion harness pull request. The result is in a comment on this pull request. ## Risks - The superseded-run rule trusts the successor run as proof of reassignment. A case oracle still grades the task assignee and the successor's own outcome. - The close-case result turn no longer checks the reply's words. A reply that reads the closed case and links it but states the wrong result passes this one check. The first-turn premature-claim check and the negative-case check do not change. - The credential grader change also affects other secret cases. It only removes the negated-destination false positive. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
129 lines
8.0 KiB
TypeScript
129 lines
8.0 KiB
TypeScript
import { nativeSha256 } from "../../../server/src/services/native-runtime/canonical.js";
|
|
|
|
/** The digest the port saves for a redacted call_api input (the server's own canonical SHA-256). */
|
|
export const callApiInputDigest = (input: Record<string, unknown>) => nativeSha256(input);
|
|
|
|
/**
|
|
* Record-bound read receipts that survive run-event redaction.
|
|
*
|
|
* PaperclipControlPlanePort saves call_api inputs and results with their
|
|
* payloads replaced by a SHA-256 digest; only the operationId, ok and status
|
|
* remain. A read is therefore tied to a record by recomputing the digest of
|
|
* every input that names it (path parameter by id or identifier, with no query
|
|
* or a small paging query) and requiring the paired result to report the
|
|
* expected status. The agent's own wording is never matched.
|
|
*/
|
|
const objectRecord = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
|
const PAGE_LIMITS = Array.from({ length: 1001 }, (_, index) => index).flatMap(limit => [{ limit }, { limit: String(limit) }]);
|
|
const cache = new Map<string, ReadonlySet<string>>();
|
|
const CASE_READ_OPERATION = "GET /api/cases/{id}";
|
|
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
|
|
// The first page of the case record section (the default), which carries the case's status and completion.
|
|
// Fields-only pages and byte-limited pages may omit them, so they do not count as reading the case.
|
|
// An explicit limit equal to the default page size (8 KiB, also the maximum) reads the same page.
|
|
const CASE_FIRST_PAGES: ReadonlyArray<Record<string, unknown>> = [undefined, "case"].flatMap(section => [undefined, 0].flatMap(offsetBytes =>
|
|
[undefined, 8 * 1024].map(limitBytes => ({ ...(section ? { section } : {}), ...(offsetBytes !== undefined ? { offsetBytes } : {}),
|
|
...(limitBytes !== undefined ? { limitBytes } : {}) }))));
|
|
const caseReadTool = (name: unknown) => typeof name === "string" && /^(?:[A-Za-z0-9_-]*__)?(?:paperclip_)?read_case$/.test(name);
|
|
|
|
export interface ApiRead {
|
|
operationId: string;
|
|
/** Each alternative names the record, e.g. [{ id: uuid }, { id: "COM-1" }]; `undefined` means the server fills the path. */
|
|
pathParams: ReadonlyArray<Record<string, string> | undefined>;
|
|
/** Accept a `limit` paging query (number or string, 0..1000) in addition to none. */
|
|
paged?: boolean;
|
|
/** Other accepted queries, e.g. a project filter. */
|
|
queries?: ReadonlyArray<Record<string, unknown>>;
|
|
}
|
|
|
|
/** Digests of every accepted call_api input for these reads. */
|
|
export function apiReadDigests(reads: readonly ApiRead[]): ReadonlySet<string> {
|
|
const key = JSON.stringify(reads);
|
|
const cached = cache.get(key); if (cached) return cached;
|
|
const digests = new Set<string>();
|
|
for (const read of reads) {
|
|
const queries: Array<Record<string, unknown> | undefined> = [undefined, {}, ...(read.paged ? PAGE_LIMITS : []), ...(read.queries ?? [])];
|
|
// The dedicated read_case tool saves its own redacted input and reports GET /api/cases/{id}. Its first page of
|
|
// the case record is `{ caseId, section?: "case", offsetBytes?: 0 }` (caseReadPageShape).
|
|
if (read.operationId === CASE_READ_OPERATION) for (const pathParams of read.pathParams)
|
|
if (pathParams?.id && UUID.test(pathParams.id)) for (const page of CASE_FIRST_PAGES) digests.add(callApiInputDigest({ caseId: pathParams.id, ...page }));
|
|
for (const pathParams of read.pathParams) for (const query of queries) {
|
|
digests.add(callApiInputDigest({ operationId: read.operationId, ...(pathParams ? { pathParams } : {}), ...(query ? { query } : {}) }));
|
|
if (!pathParams) digests.add(callApiInputDigest({ operationId: read.operationId, pathParams: {}, ...(query ? { query } : {}) }));
|
|
}
|
|
}
|
|
cache.set(key, digests);
|
|
return digests;
|
|
}
|
|
|
|
/** Path parameter alternatives naming one record by id and, when known, its human identifier. */
|
|
export function recordParams(name: string, ...values: Array<string | null | undefined>): Array<Record<string, string>> {
|
|
return [...new Set(values.filter((value): value is string => typeof value === "string" && value.length > 0))].map(value => ({ [name]: value }));
|
|
}
|
|
|
|
/**
|
|
* Dedicated task read tools and the call_api input the server builds for them
|
|
* (taskBridgeApiRequest in paperclip-runner-tool-authority.ts). Their saved
|
|
* tool_use keeps the plaintext taskId, while the result carries the same
|
|
* redacted receipt (apiOperationId, ok, status) as call_api. Harnesses may
|
|
* namespace the name (paperclip_get_task, mcp__paperclip__get_task).
|
|
*/
|
|
const TASK_READ_TOOLS: Readonly<Record<string, string>> = {
|
|
get_task: "GET /api/issues/{id}",
|
|
list_task_documents: "GET /api/issues/{id}/documents",
|
|
read_task_document: "GET /api/issues/{id}/documents/{key}",
|
|
};
|
|
function taskReadTool(name: unknown): string | undefined {
|
|
const match = typeof name === "string" ? /^(?:[A-Za-z0-9_-]*__)?(?:paperclip_)?([a-z_]+)$/.exec(name) : null;
|
|
return match && Object.hasOwn(TASK_READ_TOOLS, match[1]!) ? match[1] : undefined;
|
|
}
|
|
function taskReadInput(tool: string, input: Record<string, unknown>): Record<string, unknown> | undefined {
|
|
if (typeof input.taskId !== "string" || !input.taskId) return;
|
|
if (tool === "read_task_document" && (typeof input.key !== "string" || !input.key)) return;
|
|
return { operationId: TASK_READ_TOOLS[tool], pathParams: { id: input.taskId, ...(tool === "read_task_document" ? { key: input.key } : {}) } };
|
|
}
|
|
|
|
/**
|
|
* Statuses of this run's record reads whose input digest is one of `digests`,
|
|
* paired to their tool_use by run and item id: call_api uses by their saved
|
|
* digest, and dedicated task read tools by the digest of the call_api input
|
|
* the server builds from their plaintext input.
|
|
*/
|
|
export function apiReadStatuses(events: readonly Record<string, unknown>[], digests: ReadonlySet<string>): Array<{ operationId: string; ok: boolean; status: number }> {
|
|
const uses = new Map<string, string>();
|
|
const statuses: Array<{ operationId: string; ok: boolean; status: number }> = [];
|
|
for (const row of events) {
|
|
const envelope = objectRecord(objectRecord(row.payload).prpEvent), item = objectRecord(objectRecord(envelope.payload).item);
|
|
const run = String(row.runId ?? envelope.runId ?? "");
|
|
if (item.type === "tool_use" && item.name === "call_api") {
|
|
const input = objectRecord(item.input);
|
|
if (input.apiPayloadRedacted === true && typeof input.operationId === "string" && digests.has(String(input.apiPayloadSha256))) uses.set(`${run}:${String(item.id)}`, input.operationId);
|
|
continue;
|
|
}
|
|
if (item.type === "tool_use" && caseReadTool(item.name)) {
|
|
const input = objectRecord(item.input);
|
|
// Saved redacted as `read_case`; a namespaced name may keep its plaintext `{ caseId }`.
|
|
const digest = input.apiPayloadRedacted === true ? String(input.apiPayloadSha256)
|
|
: typeof input.caseId === "string" && Object.keys(input).length === 1 ? callApiInputDigest({ caseId: input.caseId }) : undefined;
|
|
if (digest && digests.has(digest)) uses.set(`${run}:${String(item.id)}`, CASE_READ_OPERATION);
|
|
continue;
|
|
}
|
|
const tool = item.type === "tool_use" ? taskReadTool(item.name) : undefined;
|
|
if (tool) {
|
|
const request = taskReadInput(tool, objectRecord(item.input));
|
|
if (request && digests.has(callApiInputDigest(request))) uses.set(`${run}:${String(item.id)}`, String(request.operationId));
|
|
continue;
|
|
}
|
|
if (item.type !== "tool_result") continue;
|
|
const operationId = uses.get(`${run}:${String(item.tool_use_id)}`), result = objectRecord(item.result);
|
|
if (!operationId || result.apiOperationId !== operationId || typeof result.status !== "number") continue;
|
|
statuses.push({ operationId, ok: result.ok === true, status: result.status });
|
|
}
|
|
return statuses;
|
|
}
|
|
|
|
/** True when the run successfully completed one of these record-bound reads. */
|
|
export function readRecord(events: readonly Record<string, unknown>[], reads: readonly ApiRead[]) {
|
|
return apiReadStatuses(events, apiReadDigests(reads)).some(read => read.ok && read.status >= 200 && read.status < 300);
|
|
}
|