mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
DottaandClaude Opus 5.5
d157ce223c
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat evaluation lane runs each chat case in a paid runner and
keeps the private evidence of each attempt in a bounded private archive
> - A failed attempt keeps its Playwright trace and video, and the
bundle has a 256 MB allowance
> - Playwright's HTML reporter copies each attachment into
`html-report/data/<sha1>.<ext>`, and the same bytes also stay in
`playwright-output/`
> - The bundle holds the original tree and the packaged tree, so a 30 MB
trace used about 125 MB of the allowance before the blob report copies
> - Large failed attempts went over the allowance, and then all of their
private evidence was lost
> - This pull request removes the `playwright-output/` copy only when
the HTML report holds the identical bytes
> - A second gap lost the whole merged campaign report of failed cells:
the report copied trace viewer font, SVG and manifest files to a place
where the trusted seal refuses them
> - This pull request stops that copy, and it names a refused report
carrier with a closed category
> - The benefit is that large failed attempts keep their evidence, the
merged report is kept, and the HTML report and its trace viewer still
open the trace
## Linked Issues or Issue Description
**What happened**
Failed chat evaluation attempts with large Playwright traces lost all of
their private evidence. The bundle went over the 256 MB private evidence
allowance. For example, the failed `tasks.organize-open-work` cell made
a 254 MB bundle. That bundle held the same 30.7 MB trace four times
(`playwright-output/.../trace.zip` and `html-report/data/<sha1>.zip`, in
both the original tree and the packaged tree), plus two blob report
copies. Larger traces went over the limit, and then the transfer refused
the whole bundle.
Also, the merged campaign report of a failed cell was refused with
`refusal: unsupported-packaged-evidence-carrier` (for example
`runs-adapters.inspect-failed-run`, run 42431162923). The cell's
evidence manifest lists its trace viewer files
`html-report/trace/codicon.*.ttf`, `playwright-logo.svg` and
`manifest.webmanifest`. `report.ts` copied them to
`normalized/evidence/<id>/attempt-1/html-report/trace/`. The
generated-report seal accepts trace viewer files only under
`html/trace/`, so it refused these three files and lost the whole
report. I reproduced this from the retained cell archive: a merge of its
blob report passes the seal rule, and the three normalized copies are
the only files that fail it.
**Expected behavior**
A failed attempt keeps its trace, and the trace uses as little of the
allowance as possible. The HTML report and the trace viewer still open
the trace for a person who downloads the bundle. The merged report of a
failed cell is kept. If the seal refuses a carrier, the refusal names
its category, not its path. No privacy, integrity or size check changes.
**Steps to reproduce**
1. Dispatch `runner-full-stack-e2e.yml` with `cadence=manual` for a
private chat case that fails with a large trace, for example
`chat-features.runner-opencode.local.tasks.organize-open-work`.
2. Download `manual-cell-<sha256(executionId)>.zip` from the private
history bucket.
3. List the archive. The trace bytes are in `playwright-output/` and in
`html-report/data/` in both `originals/` and `sealed-results/`.
4. For the report refusal: look at the "Retain complete manual merged
evidence only in restricted storage" step of the same run. It prints
`refusal: unsupported-packaged-evidence-carrier` and no
`manual-report.zip` is retained.
## What Changed
- Add `tests/runner-e2e/private-report-copies.ts` with
`dropDuplicatePlaywrightOutput`. It removes a `playwright-output/` file
only when `html-report/data/<sha1 of content><ext>` is a regular file
(not a symlink) with the same size and the same SHA-256. It writes the
removed paths, their kept paths and digests to
`snapshots/playwright-output-copies.json`.
- The function does nothing when the cell has a framework JSON report
(`framework-report.json` or `snapshots/playwright-framework.json`),
because that report registers `playwright-output/` attachments by path.
It also does nothing when `playwright-output`, `html-report` or
`html-report/data` is missing or is a symlink.
- Call it in the private manual branch of `tests/runner-e2e/launch.ts`,
before packaging and before both seals. Both evidence trees then keep
one HTML report copy of each trace and video.
- The HTML report, its trace viewer assets and the blob report do not
change.
- `tests/runner-e2e/report.ts` does not stage a cell file into
`normalized/evidence/` when its type is one the generated-report seal
refuses (the same `isStandardEvidenceFile("html-report/...")` rule). The
merged report has its own authenticated trace viewer under
`html/trace/`. The cell archive still keeps the files.
- `tests/runner-e2e/manual-private-transfer.ts` adds
`assertGeneratedReportCarriers`, which runs before the report seal. It
throws a fixed message per closed category (`font`, `svg`,
`webmanifest`, `image`, `source map`, `no extension`, `other`, `trace
viewer asset`, `trace viewer bytes`), so the refusal line names, for
example, `unsupported-generated-report-carrier-font`. It never names a
path, and the seal stays the authority.
- No change to the target-checked trusted files
(`manual-private-handoff.ts`, `manual-private-retention.ts`,
`manual-private-report-assets.ts`), so candidate branches do not need an
update. No change to the limits or the secret scans.
## Verification
- `vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/private-report-copies.test.ts
tests/runner-e2e/manual-private-cell-report-assets.test.ts
tests/runner-e2e/private-framework-report.test.ts
tests/runner-e2e/manual-private-handoff.test.ts`: 79 passed.
- The new tests use a real failed Playwright run with the HTML, JUnit
and blob reporters and a real retained trace. They show these things:
- Only the duplicate `playwright-output/` files are removed. The HTML
report and blob report bytes do not change.
- The deduplicated cell passes `packageEvidence`,
`sealStandardOriginalEvidence`, `sealManualResultEvidence`, the trusted
`validateSealedOriginals` and `validateSealedInventory`, the strict
archive pack and unpack, and `stageSealedPrivateCell`.
- A tampered or symlinked HTML report copy keeps the original file.
- Framework JSON cells and symlinked data directories are not changed.
- `vitest run --config tests/runner-e2e/vitest.config.ts manual-private
private-report private-refusal private-framework generated-report
report`: 25 files, 311 passed.
- `generated-report-carriers.test.ts` shows each refused carrier gets
its closed category code, the code has no path, and the real seal
refuses the same tree. `report.test.ts` shows the font, SVG and manifest
are not staged and the other files are.
- A new test runs the real launcher private branch from `launch.ts`. It
shows that the sealed originals keep one trace copy, in
`html-report/data/`.
- Live run: I applied the commit to a branch from
`chat-eval/master-candidate` (`chat-eval/case-evidence-dedupe`) and
dispatched
`chat-features.runner-opencode.local.tasks.organize-open-work` (run
42432263920). I will add the bundle size result in a comment.
## Risks
- Low risk. The change runs only in the private manual evidence branch.
It removes a file only after a full content comparison with the kept
copy. If a check fails, the file stays.
- A person who wants the trace must open `html-report/data/<sha1>.zip`
(the HTML report links to it), not `playwright-output/<test>/trace.zip`.
The receipt in `snapshots/playwright-output-copies.json` names the new
location.
- `report.ts` and `manual-private-transfer.ts` run from the master
checkout in the report job, so the report fix takes effect only after
merge. The normalized per-cell HTML report copy no longer has the viewer
font, logo and manifest. The merged report under `html/` and the cell
archive keep a complete viewer.
- The blob report still has a compressed copy of the trace, and the
bundle still holds the original and the packaged tree. A trace over
about 60 MB can still go over the allowance. This change does not fix
that case.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), run in Claude Code with
tool use (shell, file editing, GitHub CLI and AWS CLI).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
51 lines
3.1 KiB
TypeScript
51 lines
3.1 KiB
TypeScript
import { afterEach, expect, it } from "vitest";
|
|
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { assertGeneratedReportCarriers } from "./manual-private-transfer.js";
|
|
import { sealManualResultEvidence, type ManualPrivateHandoff } from "./manual-private-handoff.js";
|
|
import { canonicalGeneratedReportAsset } from "./manual-private-report-assets.js";
|
|
import { MANUAL_PRIVATE_EXECUTIONS, MANUAL_PRIVATE_PROTOCOL } from "./manual-private-retention.js";
|
|
import { privateRefusalCode } from "./private-refusal-code.js";
|
|
|
|
const roots: string[] = [];
|
|
afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
|
|
async function report(files: Record<string, string | Buffer>) {
|
|
const root = await mkdtemp(path.join(os.tmpdir(), "generated-report-carriers-")); roots.push(root);
|
|
const source = path.join(root, "runner-e2e-merged-report");
|
|
for (const [name, data] of Object.entries({ "html/index.html": "<html></html>", "normalized/summary.md": "summary", ...files })) {
|
|
await mkdir(path.dirname(path.join(source, name)), { recursive: true });
|
|
await writeFile(path.join(source, name), data);
|
|
}
|
|
return { root, source };
|
|
}
|
|
const cell = "normalized/evidence/chat-features.runner-opencode.local.case/attempt-1/html-report/trace";
|
|
|
|
it("accepts standard carriers and authenticated trace viewer assets", async () => {
|
|
const logo = await readFile(canonicalGeneratedReportAsset("html/trace/playwright-logo.svg")!);
|
|
const { source } = await report({ "html/trace/playwright-logo.svg": logo, "html/data/0123456789abcdef0123456789abcdef01234567.zip": "zip" });
|
|
await expect(assertGeneratedReportCarriers(source)).resolves.toBeUndefined();
|
|
});
|
|
|
|
it.each([
|
|
[`${cell}/codicon.DCmgc-ay.ttf`, "font"],
|
|
[`${cell}/playwright-logo.svg`, "svg"],
|
|
[`${cell}/manifest.webmanifest`, "webmanifest"],
|
|
["html/data/0123456789abcdef0123456789abcdef01234567.jpeg", "image"],
|
|
["normalized/report.js.map", "source-map"],
|
|
["normalized/LICENSE", "no-extension"],
|
|
["normalized/archive.tar", "other"],
|
|
["html/trace/index.js.map", "trace-viewer-asset"],
|
|
["html/trace/playwright-logo.svg", "trace-viewer-bytes"],
|
|
])("names a refused %s carrier by its closed category, never its path", async (name, category) => {
|
|
const { root, source } = await report({ [name]: "synthetic carrier" });
|
|
const refusal = await assertGeneratedReportCarriers(source).then(() => undefined, (error: unknown) => error);
|
|
const code = privateRefusalCode(refusal);
|
|
expect(code).toBe(`unsupported-generated-report-carrier-${category}`);
|
|
expect(code).not.toContain("chat-features");
|
|
// The authoritative seal refuses the same tree, so the preflight never hides an admissible report.
|
|
const context = { root, private: true, protocol: MANUAL_PRIVATE_PROTOCOL, workflowSha: "a".repeat(40), targetSha: "b".repeat(40), runId: "123",
|
|
attempt: "1", executionIds: [MANUAL_PRIVATE_EXECUTIONS[0]!] } as ManualPrivateHandoff;
|
|
await expect(sealManualResultEvidence(context, [], source, "generated-report")).rejects.toThrow("Unsupported packaged evidence carrier");
|
|
});
|