2 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	f0ff466aaf	fix(testops): wait for complete attachment evidence (#18509) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Product E2E evaluations check real user requests and their saved results. > - The TestOps reporter also verifies the uploaded result and attachment evidence. > - One failed attachment-row snapshot stopped the reader before its existing observation allowance. > - A later read with the same protected credential reference verified the original result. The original failed subfield remains unknown. > - This pull request handles attachment readiness within the existing bound while preserving the complete final evidence predicate. > - The reporter can wait for complete evidence without accepting missing, inherited, or incorrect files. ## Linked Issues or Issue Description Refs #18477 and #18470. **What happened?** The chat case passed its ten checks and cleanup in [run 41217990335](https://github.com/paperclipai/paperclip/actions/runs/41217990335), but the final reader failed at `attachment_page / attachment_rows`. The unchanged reader later verified that same retained result through the protected reporting identity in [read-only run 41268693639](https://github.com/paperclipai/paperclip/actions/runs/41268693639). This proves the observed snapshot differs. It does not identify which original row subfield failed or prove an upload race. The reader already permits twelve polls and 120 seconds for complete result and attachment metadata. It nevertheless rejects `missed === true` immediately. That flag cannot satisfy complete evidence, but it can be observed again within the existing allowance. **Expected behavior** Wait within the original limit for attachment evidence to become complete. Accept only the same full source, case, status, cleanup, attachment-name, media-type, byte-count, and non-missed predicate. Reject malformed IDs and inherited rows immediately. Permanent missing or incorrect evidence must still fail. **Steps to reproduce** 1. Call the real verifier with otherwise valid responses and one `missed: true` attachment. 2. Return the complete attachment on the second poll. 3. The old reader fails on the first snapshot. The regression now reaches the original complete acceptance predicate on the second snapshot. **Paperclip version/commit** Base `987e0932bf66e703750cd04698b3c7e8993580a7`. **Deployment mode** TestOps reporting observer after a Product E2E evaluation. This change starts no model or app and changes no feature oracle. ## What Changed - Move the strict `missed === true` condition from immediate structural rejection into bounded readiness. - Keep page shape, counts, unique positive IDs, inherited-row rejection, and every source and case guard hard. - Require the same complete final signature and status before success. Keep twelve polls, five-second intervals, the 120-second deadline, and request limits. - Retain closed per-poll counts and readiness booleans on verification and unverified failure. Clone failure observations so callers cannot change retained diagnostics. - Add seven focused regression and negative controls. Sync one existing consumer test with the precise new failure phase and predicate. ## Verification - Genuine parent RED: the real missed-then-complete verifier test failed at the original attachment-row guard. - API owner tests: 60 passed. Direct CLI, exporter, reporter, and retained-reader consumer tests: 31 passed. Node 26.4.0. - Permanent missing evidence reaches twelve polls and eleven waits, then fails. An elapsed 120-second deadline stops after the first page. Malformed IDs and inherited rows still fail immediately. Incorrect byte counts never pass. Closed diagnostics exclude private fields and return detached observation copies. - Three Node syntax checks and `git diff --check` passed. - Two review reproductions failed on the first PR head: a null row lost its controlled invalid-ID error, and a final-GET deadline crossing lost its exact readiness predicate. The corrected head preserves both diagnostics. Null rows still fail immediately; no deadline or acceptance condition changed. - No local app, database, provider, native build, or new evaluation ran. The MJS change needs no TypeScript emission. - [Normal CI 41269715187](https://github.com/paperclipai/paperclip/actions/runs/41269715187) passed on exact head `4e77c1220b0ba81e971591c1c7d28f9064e992cc`: 55 successes, three skipped checks, and one neutral review. Both required checks passed. Greptile is 5/5 and review threads are resolved. - The controlled normal TestOps-launched confirmation is pending merge. The family unit roster runs the local controls; normal PR CI uses separate runner suites, so CI green alone does not claim that roster ran there. ## Risks - A missed file may remain missing. The observer fails if complete evidence does not arrive inside the original allowance. - This is readiness handling, not a claim that `missed` caused the historical failure. Prior passed case grades and failed reporting workflows remain retained. - The previous fixed retained diagnostic intentionally pins the original helper. Its completed receipt remains valid; after this helper changes, that old diagnostic fails closed instead of silently replaying a different observer. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with reasoning, code editing, shell tools, and independent source review. The host does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 22:54:28 -05:00
DottaandPaperclip	987e0932bf	test(testops): inspect a fixed retained chat result (#18477) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - The chat feature evaluation checks a real task request on AWS. > - The case passed, but its final TestOps reader rejected attachment metadata. > - The later browser view passes the same check. It uses a different identity and time. > - Another model run cannot explain that difference. > - This pull request adds a fixed read-only check of the retained measurement with the reporting API identity. > - The check separates its own executor from the original measurement and preserves the original grade. ## Linked Issues or Issue Description Refs #18470 and #18462. **What happened?** The real chat case passed all ten checks and cleanup in [run 41217990335](https://github.com/paperclipai/paperclip/actions/runs/41217990335). Final verification failed at `attachment_page / attachment_rows`. The reader had already accepted the source, run, case, status, page counts, and unique attachment IDs. Later interactive metadata had valid IDs and false attachment flags. That later view does not explain the original API caller failure. **Expected behavior** A read-only operation can inspect the same retained result with the existing protected reporting identity. It must report the exact current attachment flags without running a model, uploading results, or changing the original grade. **Steps to reproduce** 1. Read the closed failure from the linked evaluation run. 2. After this workflow is reviewed and deployed, dispatch its fixed no-input operation on master. 3. Inspect its closed receipt. It records both the new executor and the original measurement. **Paperclip version/commit** The retained measurement used `e40de0c9e88210dc476dcf651461c0a8d1b98a59`. **Deployment mode** GitHub Actions metadata job with the existing protected reporting environment. No app, database, provider, or AWS worker starts. ## What Changed - Add one manual no-input workflow for the fixed retained measurement. - Admit the live executor and the original run separately. Verify exact selection and sanitized export artifact hashes before TestOps authentication. - Read only the five existing TestOps metadata routes with the existing client. Its normal API-token session exchange is the only authentication POST. - Record closed per-row attachment types and flags. Replay the unchanged verifier against the captured responses using historical context as data. Keep the actual process environment unchanged. - Add nine mock and archive-safety tests to the existing Node unit roster. ## Verification - `node --test tests/runner-e2e/chat-feature-testops-retained-verification.node-test.mjs`: 9 passed on Node 26.4.0. - Three Node syntax checks, actionlint, the relevant workflow policy check, and `git diff --check` passed. - The real pinned sanitized export ZIP passes the new safe-export extractor and existing bound reader. The first local attempt used a noncanonical `/var` temporary path and hit the existing canonical-path guard; the canonical `/private/tmp` attempt passed without a source change. - Corrected a review finding in failure attribution. Only actual authentication or protected GET errors now receive `apiFailure`. Historical replay failures retain their original phase and predicate diagnostics. - Initial test run: eight passed and one test expected the generic upload error instead of the existing specific job-run error. Corrected that test expectation. No verifier behavior changed. - Tests exercise separate identities, foreign source and actor rejection, origin and artifact integrity, exact five GETs, all three attachment-row refusals, secret redaction, wrong job/run/source, and real archive traversal/symlink rejection. - The unchanged API and export readers are source-pinned. Existing evaluation and acceptance code is unchanged. - [Normal CI 41220476243](https://github.com/paperclipai/paperclip/actions/runs/41220476243) passed on exact head `b79e85f71567f662d6f3d680ec66ec49a132589a`: 55 successes, three skipped checks, and two neutral reviews. Both required checks passed. Greptile is 5/5 and all review threads are resolved. - The nine new controls are registered in the family unit command. The separate normal PR-CI runner suites do not execute that roster, so their green status is not proof these nine ran on Node 24. The manual workflow explicitly runs them on Node 24 before any protected API read. - The real protected API operation has not run. A separately approved merge and metadata dispatch remain required. No model, app, database, or native build ran locally. ## Risks - The fixed artifacts can expire. Any absent or changed artifact fails before TestOps authentication. - The current protected response snapshot may differ from the original failure-time state. The receipt states that limit and makes no historical cause claim. - This is a fixed diagnostic operation. It does not repair the original workflow or grant authority to run another evaluation. - The workflow uses the existing protected reporting token. It logs no response body, unknown field, credential, or unknown URL. Only closed receipts are retained. ## Model Used OpenAI `gpt-6.1-sol`, through Codex, with reasoning, code editing, shell tools, and source review. The host does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 22:16:20 -05:00