Files
PaperClipAI/tests/runner-e2e/private-refusal-code.ts
T
DottaandClaude Opus 5.5 1de794cb80 fix(evals): name runtime-built evidence refusals by their fixed template (#19920)
## Thinking Path

- #19908 made a refused private evidence transfer print a closed refusal
code instead of one generic line.
- Its first real use printed `refusal: dynamic`: the trusted modules
build some refusals at runtime (secret-leak findings,
publication-boundary limits), and those were deliberately not named
because their text can carry paths or counts.
- The fixed part of each template is safe and is exactly what an
operator needs to know which check refused.

## Linked Issues or Issue Description

**What happened**

After #19908, a refused `runs-adapters.inspect-failed-run` attempt
(workflow run 42422571919) printed `Restricted manual evidence operation
failed (refusal: dynamic)`, so the reason was still unknown.

**Expected behavior**

A refused transfer still keeps message text private, but names which
check refused it with a closed code, so the attempt can be diagnosed.

**Steps to reproduce**

1. Dispatch `runner-full-stack-e2e.yml` manually for a chat cell whose
private evidence fails transfer validation.
2. Open the "Retain complete approved manual originals" step log: it
shows the single fixed failure line with no reason.

## What Changed

- `private-refusal-code.ts`: runtime-built refusals from the trusted
modules map to fixed template codes (`template-secret-leak`,
`template-publication-boundary-exceeded`,
`template-invalid-file-metadata`,
`template-history-expiration-unprovable`). Variable parts are never
printed.
- Tests: each template is named without its variable text, and a source
scan asserts every runtime-built `new Error` in the trusted modules maps
to a named code (not `dynamic`).

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/private-refusal-code.test.ts
tests/runner-e2e/manual-private-transfer.test.ts`: 20 passed.
- `npx tsc --noEmit -p tests/runner-e2e`: clean.

## Risks

- Template codes reveal which kind of check refused (for example that a
secret-shaped value was found), not where or what. That is the intended
operator signal; the finding itself stays private.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with
tool use (shell, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 18:41:39 -06:00

47 lines
2.7 KiB
TypeScript

import { readFileSync } from "node:fs";
/** Trusted evidence modules whose fixed refusal messages may be named publicly. */
export const PRIVATE_REFUSAL_MODULES = [
"manual-private-transfer.ts", "manual-private-handoff.ts", "manual-private-retention.ts", "manual-private-report-assets.ts",
"private-history.ts", "private-framework-report.ts", "history-budget.ts", "evidence.ts", "redaction.ts",
"manual-dispatch-authority.ts", "testops-chat-admission.ts",
] as const;
/** Fixed templates of runtime-built refusals in the trusted modules (variable parts are not reported). */
const TEMPLATE_REFUSALS: ReadonlyArray<readonly [RegExp, string]> = [
[/^Secret leak in /, "template-secret-leak"],
[/ exceeds the publication boundary \(\d+ files, \d+ bytes\)$/, "template-publication-boundary-exceeded"],
[/ contains invalid or duplicate file metadata$/, "template-invalid-file-metadata"],
[/^Bounded history cannot prove expiration with /, "template-history-expiration-unprovable"],
];
let known: ReadonlySet<string> | undefined;
/** Literal `new Error("...")` messages in the trusted modules; dynamic messages are never included. */
export function fixedRefusalMessages(): ReadonlySet<string> {
known ??= new Set(PRIVATE_REFUSAL_MODULES.flatMap(file => {
const source = readFileSync(new URL(`./${file}`, import.meta.url), "utf8");
return [...source.matchAll(/new Error\("([^"$`\\]+)"\)/g)].map(match => match[1]!);
}));
return known;
}
/**
* A closed, public-safe name for why a private evidence operation refused. Only a fixed literal
* message from the trusted modules is named (as a slug); otherwise a Node error code, an error type,
* or "dynamic". The message text of anything else is never returned.
*/
export function privateRefusalCode(error: unknown, fixed?: ReadonlySet<string>): string {
// Naming must never prevent the fixed failure line: a module read failure degrades to "unnamed".
try { return refusalCode(error, fixed ?? fixedRefusalMessages()); } catch { return "unnamed"; }
}
function refusalCode(error: unknown, fixed: ReadonlySet<string>): string {
if (!(error instanceof Error)) return "non-error-thrown";
if (fixed.has(error.message)) return error.message.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "").slice(0, 96);
// Runtime-built messages from the trusted modules: name the fixed template, never its variable parts.
for (const [pattern, name] of TEMPLATE_REFUSALS) if (pattern.test(error.message)) return name;
const code = (error as NodeJS.ErrnoException).code;
if (typeof code === "string" && /^(?:E[A-Z]{2,15}|ERR_[A-Z0-9_]{2,48})$/.test(code)) return `fs-${code}`;
if (error.name !== "Error" && /^[A-Z][A-Za-z]{2,40}$/.test(error.name)) return `error-type-${error.name}`;
return "dynamic";
}