Files
PaperClipAI/tests
DottaandPaperclip 106d89314f fix(evals): keep raw diagnostics out of public history (#13850)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Product E2E campaigns retain evidence from live provider runs.
> - The public publisher copied per-attempt files based on their
extension.
> - Credential redaction does not remove hidden reasoning or provider
session IDs.
> - This pull request keeps those raw diagnostics out of public evidence
bundles.
> - Public reports retain normalized grades and declared fixture
screenshots.

## Linked Issues or Issue Description

**What happened?**

The Product history publisher accepted all JSON, log, Markdown, and text
files under an attempt directory. API snapshots and process logs can
contain provider reasoning even after credential redaction.

**Expected behavior**

Keep raw attempt diagnostics in retained Actions artifacts. Publish
normalized results and declared fixture screenshots. Preserve the
original grades and attempt evidence.

**Steps to reproduce**

Place a credential-redacted API snapshot with a reasoning event under an
attempt's snapshots directory. The previous public path check admitted
it because it ended in .json.

## What Changed

- Remove extension-based admission for per-attempt text files in both S3
and Pages staging.
- Require PNG paths to appear in the existing screenshot declaration
allowlist.
- Preserve root normalized results, grading, billing, provenance, and
reviewed screenshot behavior.
- Add seven denial cases for raw diagnostics, renamed files, malformed
JSON, and false screenshot declarations.
- Update report copy and the publication security contract.

## Verification

- Focused history and report tests: 47 passed.
- Replayed the filter against a copy of retained Grok evidence: six
diagnostic files removed, one declared screenshot retained, zero
provider calls. Original artifacts remain unchanged.
- git diff --check passed.
- Product typecheck in the reused local dependency tree reports an
unrelated plugin type mismatch for organizationSwitcher in
server/src/services/plugin-capability-validator.ts. CI will verify a
fresh install.
- No local browser or Docker execution.

## Risks

Public reports no longer link raw per-attempt logs or API snapshots.
Those files remain in the original Actions artifact. This change affects
future publication only; it does not withdraw or rewrite existing
immutable public campaigns. Normalized result content and marked
screenshot review retain their existing boundaries. No workflow
authorization, provider credentials, application behavior, or database
changes.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code editing, and tool
execution. The exact deployment model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-22 21:31:39 -05:00
..