mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-02 08:10:40 +01:00
DottaandPaperclip
987e0932bf
## Thinking Path > - Paperclip manages AI agents and their work. > - The chat feature evaluation checks a real task request on AWS. > - The case passed, but its final TestOps reader rejected attachment metadata. > - The later browser view passes the same check. It uses a different identity and time. > - Another model run cannot explain that difference. > - This pull request adds a fixed read-only check of the retained measurement with the reporting API identity. > - The check separates its own executor from the original measurement and preserves the original grade. ## Linked Issues or Issue Description Refs #18470 and #18462. **What happened?** The real chat case passed all ten checks and cleanup in [run 41217990335](https://github.com/paperclipai/paperclip/actions/runs/41217990335). Final verification failed at `attachment_page / attachment_rows`. The reader had already accepted the source, run, case, status, page counts, and unique attachment IDs. Later interactive metadata had valid IDs and false attachment flags. That later view does not explain the original API caller failure. **Expected behavior** A read-only operation can inspect the same retained result with the existing protected reporting identity. It must report the exact current attachment flags without running a model, uploading results, or changing the original grade. **Steps to reproduce** 1. Read the closed failure from the linked evaluation run. 2. After this workflow is reviewed and deployed, dispatch its fixed no-input operation on master. 3. Inspect its closed receipt. It records both the new executor and the original measurement. **Paperclip version/commit** The retained measurement used `e40de0c9e88210dc476dcf651461c0a8d1b98a59`. **Deployment mode** GitHub Actions metadata job with the existing protected reporting environment. No app, database, provider, or AWS worker starts. ## What Changed - Add one manual no-input workflow for the fixed retained measurement. - Admit the live executor and the original run separately. Verify exact selection and sanitized export artifact hashes before TestOps authentication. - Read only the five existing TestOps metadata routes with the existing client. Its normal API-token session exchange is the only authentication POST. - Record closed per-row attachment types and flags. Replay the unchanged verifier against the captured responses using historical context as data. Keep the actual process environment unchanged. - Add nine mock and archive-safety tests to the existing Node unit roster. ## Verification - `node --test tests/runner-e2e/chat-feature-testops-retained-verification.node-test.mjs`: 9 passed on Node 26.4.0. - Three Node syntax checks, actionlint, the relevant workflow policy check, and `git diff --check` passed. - The real pinned sanitized export ZIP passes the new safe-export extractor and existing bound reader. The first local attempt used a noncanonical `/var` temporary path and hit the existing canonical-path guard; the canonical `/private/tmp` attempt passed without a source change. - Corrected a review finding in failure attribution. Only actual authentication or protected GET errors now receive `apiFailure`. Historical replay failures retain their original phase and predicate diagnostics. - Initial test run: eight passed and one test expected the generic upload error instead of the existing specific job-run error. Corrected that test expectation. No verifier behavior changed. - Tests exercise separate identities, foreign source and actor rejection, origin and artifact integrity, exact five GETs, all three attachment-row refusals, secret redaction, wrong job/run/source, and real archive traversal/symlink rejection. - The unchanged API and export readers are source-pinned. Existing evaluation and acceptance code is unchanged. - [Normal CI 41220476243](https://github.com/paperclipai/paperclip/actions/runs/41220476243) passed on exact head `b79e85f71567f662d6f3d680ec66ec49a132589a`: 55 successes, three skipped checks, and two neutral reviews. Both required checks passed. Greptile is 5/5 and all review threads are resolved. - The nine new controls are registered in the family unit command. The separate normal PR-CI runner suites do not execute that roster, so their green status is not proof these nine ran on Node 24. The manual workflow explicitly runs them on Node 24 before any protected API read. - The real protected API operation has not run. A separately approved merge and metadata dispatch remain required. No model, app, database, or native build ran locally. ## Risks - The fixed artifacts can expire. Any absent or changed artifact fails before TestOps authentication. - The current protected response snapshot may differ from the original failure-time state. The receipt states that limit and makes no historical cause claim. - This is a fixed diagnostic operation. It does not repair the original workflow or grant authority to run another evaluation. - The workflow uses the existing protected reporting token. It logs no response body, unknown field, credential, or unknown URL. Only closed receipts are retained. ## Model Used OpenAI `gpt-6.1-sol`, through Codex, with reasoning, code editing, shell tools, and source review. The host does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>