mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 17:51:58 +01:00
DottaandPaperclip
ca3f99449e
## Thinking Path
> - Paperclip manages AI agents and their work.
> - Product E2E reports preserve the original evaluation and its
evidence.
> - The catalog report joins one safe projection for each selected case.
> - One unavailable projection currently blocks every retained sibling
result.
> - A reporting gap must not become a fabricated evaluation result.
> - This PR adds an explicit offline retained-reporting mode to the
existing aggregate owner.
> - It preserves authentic grades and reports unavailable originals as
separate operational gaps.
## Linked Issues or Issue Description
**What happened?**
A completed 196-case catalog run produced 153 safe projections. The
strict aggregate rejected the incomplete set, so no outcomes reached
TestOps. One additional retained original can be recovered through the
unchanged strict reader. The expected retained archive keys for the
remaining 42 are absent.
**Expected behavior**
Publish authentic retained outcomes. Mark unavailable original evidence
as a reporting gap with an UNKNOWN original grade. Keep exact
selected-case coverage and all original evidence checks.
**Steps to reproduce**
1. Prepare two valid singleton projections for a three-case selection.
2. Supply a terminal, source-bound custody disposition for the third
case.
3. Observe that the existing aggregate rejects the incomplete projection
set.
Refs: #18647, #18676, #18680. This PR changes evaluation infrastructure
only. It does not merge product fixes or change the candidate
application.
## What Changed
- Add an explicit `retained-reporting` mode to the existing catalog
aggregate CLI.
- Validate each authentic singleton with the unchanged strict reader,
merger and Allure exporter.
- Require an exact, disjoint partition of authentic originals and
explicitly recorded missing retained archives.
- Emit separate BROKEN reporting-gap records. Their original status,
native-run count, checks and cleanup are null. They contain no
fabricated execution time or test steps.
- Freeze one Allure payload and a file-hash manifest. Keep recovery
executor identity separate from original measurement identity.
- Document the operator data-preparation and once-only publication
procedure.
## Verification
- Meaningful parent RED: the existing aggregate rejects the valid
partial original set with `Incomplete projection set`.
- Pure Vitest: 30 tests pass, including four new recovery controls and
26 unchanged strict exporter controls.
- Existing API and streaming Node controls pass alongside the new
partition controls.
- Negative controls reject foreign origin, duplicate or unexplained
membership, active upstream jobs, corrupt authentic data, foreign
archive keys, injected private text and a supplied gap PASS.
- Actual retained data validates as 154 authentic originals: 29 passed
and 125 failed. Seven failed originals also have failed cleanup. There
are 42 separate UNKNOWN reporting gaps. This is not a new model run or
regrade.
- `git diff --check` passes.
- Local runner noEmit encountered stale workspace package output/type
errors. No changed-file type errors remain. The current-head AWS/GHA
typecheck, build and test gates pass. Full app/database/provider setup
was not run locally under the user's execution constraint.
- CI run 41547666905 is successful at head
b20ee216f1: 48 successful jobs and 1
allowed skip. One unchanged server fixture failed before its test body
with PostgreSQL23505 on its short company prefix. One failed-only
recovery passed. All 46 prior successful executions were carried forward
with identical execution/step timestamps. The original failure is
retained; its conflicting row identity was not captured. No server code
or timeout changed.
- Greptile is 5/5 on the exact current head. All known review threads
are resolved. The nonrequired Paperclip Review check is queued; it is
not a required CI gate.
## Risks
The disposition is an operator data record, not an authentication
mechanism. The operator must authenticate the original selection,
terminal job state and exact-key custody observations before using it. A
missing retained archive does not establish that an evaluation passed or
failed. Upstream producer byte/privacy limits remain unchanged. The
command does not upload, dispatch agents, acquire credentials or grant
privileges. An ambiguous later upload must be reconciled with GET
requests, without another export or blind resend.
## Model Used
OpenAI GPT-6.1 (`gpt-6.1-sol`), with reasoning, repository tools and
code execution. The host does not expose a context-window value. No
model or provider evaluation was run for this change.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>