3 Commits

Include renames

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	d157ce223c	fix(evals): keep large failed private evidence and the merged report within the private checks (#19932) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat evaluation lane runs each chat case in a paid runner and keeps the private evidence of each attempt in a bounded private archive > - A failed attempt keeps its Playwright trace and video, and the bundle has a 256 MB allowance > - Playwright's HTML reporter copies each attachment into `html-report/data/<sha1>.<ext>`, and the same bytes also stay in `playwright-output/` > - The bundle holds the original tree and the packaged tree, so a 30 MB trace used about 125 MB of the allowance before the blob report copies > - Large failed attempts went over the allowance, and then all of their private evidence was lost > - This pull request removes the `playwright-output/` copy only when the HTML report holds the identical bytes > - A second gap lost the whole merged campaign report of failed cells: the report copied trace viewer font, SVG and manifest files to a place where the trusted seal refuses them > - This pull request stops that copy, and it names a refused report carrier with a closed category > - The benefit is that large failed attempts keep their evidence, the merged report is kept, and the HTML report and its trace viewer still open the trace ## Linked Issues or Issue Description **What happened** Failed chat evaluation attempts with large Playwright traces lost all of their private evidence. The bundle went over the 256 MB private evidence allowance. For example, the failed `tasks.organize-open-work` cell made a 254 MB bundle. That bundle held the same 30.7 MB trace four times (`playwright-output/.../trace.zip` and `html-report/data/<sha1>.zip`, in both the original tree and the packaged tree), plus two blob report copies. Larger traces went over the limit, and then the transfer refused the whole bundle. Also, the merged campaign report of a failed cell was refused with `refusal: unsupported-packaged-evidence-carrier` (for example `runs-adapters.inspect-failed-run`, run 42431162923). The cell's evidence manifest lists its trace viewer files `html-report/trace/codicon.*.ttf`, `playwright-logo.svg` and `manifest.webmanifest`. `report.ts` copied them to `normalized/evidence/<id>/attempt-1/html-report/trace/`. The generated-report seal accepts trace viewer files only under `html/trace/`, so it refused these three files and lost the whole report. I reproduced this from the retained cell archive: a merge of its blob report passes the seal rule, and the three normalized copies are the only files that fail it. **Expected behavior** A failed attempt keeps its trace, and the trace uses as little of the allowance as possible. The HTML report and the trace viewer still open the trace for a person who downloads the bundle. The merged report of a failed cell is kept. If the seal refuses a carrier, the refusal names its category, not its path. No privacy, integrity or size check changes. **Steps to reproduce** 1. Dispatch `runner-full-stack-e2e.yml` with `cadence=manual` for a private chat case that fails with a large trace, for example `chat-features.runner-opencode.local.tasks.organize-open-work`. 2. Download `manual-cell-<sha256(executionId)>.zip` from the private history bucket. 3. List the archive. The trace bytes are in `playwright-output/` and in `html-report/data/` in both `originals/` and `sealed-results/`. 4. For the report refusal: look at the "Retain complete manual merged evidence only in restricted storage" step of the same run. It prints `refusal: unsupported-packaged-evidence-carrier` and no `manual-report.zip` is retained. ## What Changed - Add `tests/runner-e2e/private-report-copies.ts` with `dropDuplicatePlaywrightOutput`. It removes a `playwright-output/` file only when `html-report/data/<sha1 of content><ext>` is a regular file (not a symlink) with the same size and the same SHA-256. It writes the removed paths, their kept paths and digests to `snapshots/playwright-output-copies.json`. - The function does nothing when the cell has a framework JSON report (`framework-report.json` or `snapshots/playwright-framework.json`), because that report registers `playwright-output/` attachments by path. It also does nothing when `playwright-output`, `html-report` or `html-report/data` is missing or is a symlink. - Call it in the private manual branch of `tests/runner-e2e/launch.ts`, before packaging and before both seals. Both evidence trees then keep one HTML report copy of each trace and video. - The HTML report, its trace viewer assets and the blob report do not change. - `tests/runner-e2e/report.ts` does not stage a cell file into `normalized/evidence/` when its type is one the generated-report seal refuses (the same `isStandardEvidenceFile("html-report/...")` rule). The merged report has its own authenticated trace viewer under `html/trace/`. The cell archive still keeps the files. - `tests/runner-e2e/manual-private-transfer.ts` adds `assertGeneratedReportCarriers`, which runs before the report seal. It throws a fixed message per closed category (`font`, `svg`, `webmanifest`, `image`, `source map`, `no extension`, `other`, `trace viewer asset`, `trace viewer bytes`), so the refusal line names, for example, `unsupported-generated-report-carrier-font`. It never names a path, and the seal stays the authority. - No change to the target-checked trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`), so candidate branches do not need an update. No change to the limits or the secret scans. ## Verification - `vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/private-report-copies.test.ts tests/runner-e2e/manual-private-cell-report-assets.test.ts tests/runner-e2e/private-framework-report.test.ts tests/runner-e2e/manual-private-handoff.test.ts`: 79 passed. - The new tests use a real failed Playwright run with the HTML, JUnit and blob reporters and a real retained trace. They show these things: - Only the duplicate `playwright-output/` files are removed. The HTML report and blob report bytes do not change. - The deduplicated cell passes `packageEvidence`, `sealStandardOriginalEvidence`, `sealManualResultEvidence`, the trusted `validateSealedOriginals` and `validateSealedInventory`, the strict archive pack and unpack, and `stageSealedPrivateCell`. - A tampered or symlinked HTML report copy keeps the original file. - Framework JSON cells and symlinked data directories are not changed. - `vitest run --config tests/runner-e2e/vitest.config.ts manual-private private-report private-refusal private-framework generated-report report`: 25 files, 311 passed. - `generated-report-carriers.test.ts` shows each refused carrier gets its closed category code, the code has no path, and the real seal refuses the same tree. `report.test.ts` shows the font, SVG and manifest are not staged and the other files are. - A new test runs the real launcher private branch from `launch.ts`. It shows that the sealed originals keep one trace copy, in `html-report/data/`. - Live run: I applied the commit to a branch from `chat-eval/master-candidate` (`chat-eval/case-evidence-dedupe`) and dispatched `chat-features.runner-opencode.local.tasks.organize-open-work` (run 42432263920). I will add the bundle size result in a comment. ## Risks - Low risk. The change runs only in the private manual evidence branch. It removes a file only after a full content comparison with the kept copy. If a check fails, the file stays. - A person who wants the trace must open `html-report/data/<sha1>.zip` (the HTML report links to it), not `playwright-output/<test>/trace.zip`. The receipt in `snapshots/playwright-output-copies.json` names the new location. - `report.ts` and `manual-private-transfer.ts` run from the master checkout in the report job, so the report fix takes effect only after merge. The normalized per-cell HTML report copy no longer has the viewer font, logo and manifest. The merged report under `html/` and the cell archive keep a complete viewer. - The blob report still has a compressed copy of the trace, and the bundle still holds the original and the packaged tree. A trace over about 60 MB can still go over the allowance. This change does not fix that case. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), run in Claude Code with tool use (shell, file editing, GitHub CLI and AWS CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 19:37:12 -06:00
DottaandPaperclip	8c9dd1e444	test(runner): qualify Kimi Code lifecycle and recovery (#18029) ... Add the Kimi lifecycle and recovery qualification harness. Preserve original grades and cleanup evidence, strict native ownership audits, bounded startup and cleanup, and early rejection of unsupported local fault execution before credentials or resources. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-30 22:54:37 -05:00
DottaandPaperclip	cd0c86e5e1	fix(evals): support private evidence for the frozen chat catalog (#18170) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E tests need to retain evidence from both successful and failed work. > - Manual private runs use the existing protected AWS evidence route. > - Its fixed eight-case registry blocks other scenarios in the frozen chat catalog. > - Failed Playwright cells also produce static trace files that need exact owner and content checks. > - This pull request admits the fixed catalog, keeps eight cells per run, and retains the existing report identities. > - Exact source and run approval still controls each private operation. ## Linked Issues or Issue Description Refs: #18134, #17888. **What happened?** The manual private route accepts only its first eight execution IDs. It rejects other IDs from the frozen 244-scenario chat catalog before a run can start. The dotted `area.case` report identity also fails the retained report parser. A failed Playwright cell can produce canonical trace assets that the private evidence owner does not yet recognize. **Expected behavior** The trusted route must recognize exactly the frozen 244 OpenCode/local execution IDs. Each original run must still select at most eight cells and have exact protected approval. The evidence owners must retain complete scanned originals from the actual pinned report producer. Unknown carriers and credential contamination must still fail closed. **Steps to reproduce** Use a known frozen chat ID outside the first eight with the current manual classifier. The classifier rejects it. Parse a dotted chat execution ID with the current report parser. The parser rejects it. Generate a synthetic failed Node-only Playwright test and pass its actual trace/report tree through the private evidence owners. This change includes the narrow parser repair from the existing open #17888. It does not merge or close that PR. The related source work was checked for overlap before this branch was created. ## What Changed - Replace the small private registry with the exact frozen 244 execution IDs. Preserve the two existing undotted standalone task IDs. - Keep the private run limit at eight through classification, approval rows, cell binding and transfer sessions. Preserve nonprivate selector behavior. - Bind private cell report assets to the actual installed pinned producer, selected cell, canonical path and exact bytes before the existing scans and archive checks. Pass that verified owner into both real launcher packaging calls. - Exclude installed build staging from the private evidence walk. Retain the real standard-file cap and reject unknown nested carriers. - Add the existing dotted chat report grammar. Preserve other report families and legacy task IDs. - Use the existing public announcement dismissal control in one external-link/upload E2E test so a real announcement cannot cover its composer button. Preserve the open menu and existing timeout. - Document source compatibility and operator gates. No workflow, product, dependency, lockfile, case callback, query, clock or budget changes. ## Verification - 132 distinct focused controls are closed. The current affected owner suite passes 39 controls under a nondefault temporary directory. The other 93 unchanged controls retain their previous passes. The initial registry, map-order and test-tooling failures remain recorded. - Two actual launcher call-expression controls fail on the prior source because canonical trace files are dropped. Both pass after the private-owner wiring. These controls do not claim full launcher lifecycle or actual E2E qualification. - Temporary paths use the real operating-system temporary directory, including the handoff check. - The controls check all 244 IDs through real private routing, cell binding and original-run approval. They reject unknown IDs, aliases, wrong profiles, traversal, private duplicates and excess private cells. - The controls preserve actual nonprivate 33-ID and duplicate selector behavior. - A real synthetic Node-only failed Playwright test produces the report and automatic trace. The control uses actual package, seal, archive, receipt and restored validation paths with mock AWS transport only. - Owning TypeScript `--noEmit` check passes. No local product server, browser, model, database, SDK build or evaluation was run. - Prior-head CI run 40906120388 failed one E2E test because the real announcement footer intercepted its Add to composer click. The other 28 tests in that shard passed. The narrow fixture uses the existing UI dismissal pattern. The added handler passes an isolated noEmit check against the real Playwright Page type. An additional strict check of the whole browser test fails on five unchanged shared-fixture type errors; checking that unchanged shared file alone gives identical diagnostics. The initial shell exit was misreported and corrected in the validation record. Browser verification is deferred to the next normal CI run; no local browser or server was started. - Repository-wide local build and test commands were not run because this is a scoped source change. Normal required PR checks will provide the repository CI result. ## Risks A registry entry is not proof that a target or feature is ready. A future target must use the exact matching private helper and producer contract, retain any stronger target scanners, and produce fresh source fingerprints. Existing running jobs keep their pinned workflow and target sources. All original attempt limits, declared inputs, private storage checks, fatal scanners and exact administrator-controlled run approval remain in force. The new cell report contract accepts only canonical assets from the actual pinned Playwright installation. Other assets and layouts still fail closed. No private evidence or generated report is included in this PR. ## Model Used OpenAI GPT-6 through Codex, with tool use and code execution. The runtime does not expose a finer model deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 11:42:35 -05:00