Files
PaperClipAI/tests/runner-e2e/private-reporter-artifacts.ts
T
DottaandPaperclip 192d83e828 Fix chat catalog prerequisite clocks and retained evidence (#18377)
## Thinking Path

> - Paperclip evals must prove actual work and retain inspectable
evidence.
> - Project defaults must reach new work while prior work remains
active.
> - A hidden HTTP deadline invalidated that prerequisite before the case
clock.
> - Report rewriting and missing evidence files obscured real outcomes.
> - Observer assumptions also rejected valid history, reads and lease
bookkeeping.
> - This PR repairs those layers and keeps the original feature
requirements and clocks.

## Linked Issues or Issue Description

Refs #18364 for product repairs. Refs #18170 for protected catalog
evidence.

The held worker's native fetch had a five-minute header timeout.
Successful cases lacked required files. Restart compared stored Markdown
to rendered text. Diagnostic redaction changed trusted Playwright
program files, causing the existing producer guard to reject
failed-report retention. Valid dedicated project reads and automatic
environment leases were also misclassified.

This PR targets **`codex/chat-catalog-master-integration`**, not master.
Its base combines master `51ea624344`, preserved catalog `e2b35be76f`
and all product/test changes from #18364, including the native chat
response-stage contract repair. The frozen preservation branch remains
unchanged. The full catalog stays off master. Shipped subscription
aliases retain their 120-second turn limit.

## What Changed

- Hold the exact worker with native HTTP, a one-use agent and the
original absolute deadline. Keep its capability in an owned 0700
directory / 0600 script outside evidence trees. Put only its path in
task/tool inputs. Register the token with the existing secret scanner.
Remove file/listener after normal execution and failed setup.
- Use the already-selected original native case deadline for
prerequisite readiness. Consume only public lifecycle state and fixed
recovery diagnostics, preserve manual-repair/terminal failures, and
record bounded state transitions and recovery flags. Other lanes retain
their authored 45-second setup cap. Nine tests pass actual server
diagnostics through the public agent normalizer.
- Poll canonical source state with the existing settlement predicate;
collect full domain/browser evidence after settlement.
- Capture required `final-state.png` and actual `api-state.json`. Keep
restart logs inside owned roots and recover abnormal evidence from the
actual capture root.
- Compare attributed rendered history before/after restart.
Independently require exact stored bytes, identity and no duplicate
admissions.
- Authenticate and scan canonical Playwright program files with the
existing validator before preserving their exact bytes. Keep ordinary
redaction and every original path, producer, secret, size and transfer
guard. Trusted protocol files remain unchanged.
- Expose only fixed diagnostics, recognized tool names and bounded
counts. Keep raw provider output private.
- Accept correlated `list_projects` reads only when the same native call
delivered the exact full saved target-project defaults. Cover real
OpenCode and server event producers; reject stale, truncated, foreign
and unpaired reads.
- Version project-defaults as **v4**. Accept natural “newly admitted”
future-scope wording; still reject replies missing the future/existing
distinction. Exclude only automatic lease acquisition/release with the
exact `environment_lease` entity type from settings-change rejection.
Retain rejection of actual budget/environment edits, mismatched types
and unknown events. Keep active-work, inheritance and reply checks
strict.

## Verification

**Qualified: [pilot
41256592937](https://github.com/paperclipai/paperclip/actions/runs/41256592937)
passes 3/3, with all 35 independent checks passing.**

| Case | Checks | Full case duration | Evidence and cleanup |
| --- | --- | --- | --- |
| Instance restart | 9/9 | 140.1 s | complete |
| Project defaults | 11/11 | 204.2 s | complete |
| Project work | 15/15 | 204.0 s | complete |

All three are original-attempt results from source
`c51e079f7f2364305b131ffd549e263ff78c5335`, containing the exact product
files from `13c2b61f31d873a475ea0c3c026b19926ff42301`. Actual saved
conversation contracts were inspected, as were ordinary native worker
contracts where present. Each protected archive includes the actual API
state and final PNG; cleanup completes. Original queries,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, finite case clocks,
concurrency one and zero eval retries are preserved. Earlier passes do
not transfer into this qualification.

Both current heads pass all checks: product **62** (56 success, one
neutral, five skipped), catalog **56** (54 success, one neutral, one
skipped). Both have exact-head Greptile **5/5**, zero unresolved review
threads and clean mergeability. The three trusted private-evidence
protocol files remain byte-identical to the dispatch workflow. Raw
evidence stays protected.

- **202 focused catalog/evidence tests and catalog typecheck pass.**
Nine readiness controls consume actual lifecycle diagnostics through the
public agent normalizer, which omits private operation state. Manual
repair, terminal states and unknown errors still fail.
- Real Node 24 control: old fetch fails at 301,464 ms; native HTTP
completes after release at 310,054 ms under the same 340-second control
deadline. Actual native OpenCode/Runner also completes a separate source
while the held worker stays blocked. Failed-setup regressions verify
resource cleanup.
- All **45 privacy/report controls pass**. The old failed-report trace
rewriting reproduces the producer-guard rejection; authenticated
canonical program bytes now survive unchanged. Contaminated assets still
fail. No secret, producer, path, transfer, expansion or size guard is
weakened.
- Oracle **`paperclip.chat-catalog.project-defaults.v4`** retains
active-work, inheritance, actual mutation/read and reply requirements.
Positive and negative controls cover correlated full saved-default
reads, automatic lease bookkeeping and natural “newly admitted” wording.
Original failed grades remain retained alongside explicit versioned
regrades.
- Product validation in #18364 includes 2,044 discovery tests, 112
real-database/HTTP/native-runner/OpenAPI consumer tests, and passing
typecheck/build. Task-owned Docker PostgreSQL unblocked local DB checks
and was removed afterward.

Retained attempt ledger; no earlier result substitutes for the qualified
run:

| Attempt | Recorded outcome / finding |
| --- | --- |
| 41114636294 / 41119215255 / 41203017450 | Initial report/setup and
unavailable-archive failures. Functional partial success did not qualify
the reports. |
|
[41207155025](https://github.com/paperclipai/paperclip/actions/runs/41207155025)
| 0/3. Rendered-Markdown mismatch; other archives unavailable. |
|
[41211541655](https://github.com/paperclipai/paperclip/actions/runs/41211541655)
| Work-only diagnostic: 15/15 with required files and cleanup. |
|
[41212946325](https://github.com/paperclipai/paperclip/actions/runs/41212946325)
| 2/3. Defaults did not settle; transfer failed. |
|
[41216880087](https://github.com/paperclipai/paperclip/actions/runs/41216880087)
| Defaults failed; counters did not identify finish outcomes. |
|
[41219519639](https://github.com/paperclipai/paperclip/actions/runs/41219519639)
| Native work succeeded but grader did not; report assets and expansion
limits rejected transfer. |
|
[41222196525](https://github.com/paperclipai/paperclip/actions/runs/41222196525)
| 1/3. Restart exposed fabricated empty-response success; defaults
exposed backfill and missing dedicated-read recognition. All archives
retained. |
| 41226224666 / 41226710271 | Cancelled before paid cells to repair
generated outputs, credential custody and native event-shape coverage. |
|
[41227965230](https://github.com/paperclipai/paperclip/actions/runs/41227965230)
| 2/3. Defaults 9/11 correctly rejected existing-task backfill and reply
scope. |
|
[41233723348](https://github.com/paperclipai/paperclip/actions/runs/41233723348)
| Restart 9/9. Defaults 10/11: v2 misclassified automatic lease events.
Work was cancelled because replacement was dispatched prematurely; no
work pass inferred. |
|
[41236679567](https://github.com/paperclipai/paperclip/actions/runs/41236679567)
| 1/3. Defaults 11/11. Restart 4/9: no answer. Work failed and the
unchanged ZIP expansion guard rejected full retention; missing outcome
remains inconclusive. |
|
[41240010122](https://github.com/paperclipai/paperclip/actions/runs/41240010122)
| Restart 4/9 under authority guidance: its literal native criterion
still required the unavailable action. Defaults 10/11: v3 rejected valid
“newly admitted” wording; v4 explicitly regrades 11/11 while retaining
the original grade and privacy findings. Work 10/15: created/repaired a
task without final chat reply. All three archives retained. |
|
[41243833425](https://github.com/paperclipai/paperclip/actions/runs/41243833425)
| Source `6fa0c83674` / product `dddd2d5206`: 2/3. Restart 9/9 and work
15/15; defaults failed prerequisite setup before any canonical
query/native run because its 45-second helper cap cut short a normal
60-second probe. |
| 41247798414 | Cancelled before paid cells after review caught a
private-field assumption in the readiness helper. The repair uses public
lifecycle state and fixed recovery diagnostics. |
| 41248428190 | Cancelled before paid cells after the unresolved-thread
sweep caught a conversation identity edge case. The repair uses
canonical identity and carries the user ID through heartbeat. |
|
[41250502608](https://github.com/paperclipai/paperclip/actions/runs/41250502608)
| Source `7b1de24ce1` / product `c0bf249807`: 1/3. Work 15/15. Restart
4/9 despite the repaired contract actually being admitted; it continued
discovery after board-only 403 and had no reply. Nearby HTTP 200 reads
are not call-correlated to the MCP timeouts. Defaults timed out before
settlement (zero finish proposals), then exceeded the unchanged 256 MiB
original-evidence bound; detailed domain behavior remains inconclusive.
|

## Risks

The prerequisite needs native Node in the worker environment. Trusted
program bytes remain subject to the existing authentication and
scanners. The audit exception rejects unknown events. Oversized ZIPs can
still be rejected, as in the retained work attempt; this PR does not
weaken that guard or claim an unobserved outcome. A passing pilot
qualifies the selected attempts, not full-catalog reliability.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, tool use, code
execution and live eval tooling. Exact hosted model revision and
context-window size are not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

- [x] Final same-source three-cell pilot passes 3/3 with required
evidence and cleanup

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-31 19:52:57 -05:00

54 lines
3.9 KiB
TypeScript

import { lstat, readFile, readdir, rm, writeFile } from "node:fs/promises";
import path from "node:path";
import { inspectEmbeddedReport, inspectZip } from "./evidence.js";
import { diagnosticSecrets, type DiagnosticSecretExpansion } from "./private-diagnostics.js";
import { findSecretLeak, redactText, sanitizeJson } from "./redaction.js";
import { cellTraceAsset, readPrivateCellTraceAsset, type PrivateCellEvidenceOwner } from "./manual-private-report-assets.js";
export interface ReporterPrivacyIssue { file: string; disposition: "redacted" | "omitted"; reason: string }
const sources = ["server.log", "playwright.log", "junit.xml", "html-report", "blob-report", "playwright-output"];
const text = new Set([".log", ".txt", ".md", ".xml", ".html", ".json", ".js", ".css"]);
/** Only reporter/diagnostic sinks are touched, after independent product oracles
* finish. Product API snapshots and work-product bytes are never pre-sanitized.
* Uninspectable/leaking compressed reports are omitted rather than fabricated. */
export async function sanitizePrivateReporterArtifacts(root: string, values: readonly string[] | DiagnosticSecretExpansion,
privateCell?: { owner: PrivateCellEvidenceOwner }): Promise<ReporterPrivacyIssue[]> {
const secrets = diagnosticSecrets(values).values; const issues: ReporterPrivacyIssue[] = [];
const visit = async (relative: string): Promise<void> => {
const filename = path.join(root, relative); const info = await lstat(filename).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return null; throw error; });
if (!info) return;
const safeName = redactText(relative, secrets);
if (info.isDirectory()) { for (const name of await readdir(filename)) await visit(path.join(relative, name)); return; }
if (privateCell && cellTraceAsset(relative, privateCell.owner, "original").reserved) {
// These are immutable Playwright program files. Scan and authenticate
// them with the same guard used by packaging; diagnostic text rewriting
// can change clean JavaScript and invalidate its producer identity.
await readPrivateCellTraceAsset({ owner: privateCell.owner, layout: "original", relative,
file: filename, remaining: info.size, secrets });
return;
}
if (info.isSymbolicLink()) { await rm(filename); issues.push({ file: safeName, disposition: "omitted", reason: "reporter symlink cannot be inspected" }); return; }
if (!info.isFile()) return;
if (safeName !== relative) { await rm(filename); issues.push({ file: safeName, disposition: "omitted", reason: "credential in reporter filename" }); return; }
const extension = path.extname(filename).toLowerCase();
if (extension === ".zip") {
const reason = await inspectZip(filename, secrets);
if (reason) { await rm(filename); issues.push({ file: safeName, disposition: "omitted", reason }); }
} else if (text.has(extension)) {
if (info.size > 128 * 1024 * 1024) { await rm(filename); issues.push({ file: safeName, disposition: "omitted", reason: "reporter text exceeds inspection bound" }); return; }
const raw = await readFile(filename, "utf8");
const opaque = extension === ".html" ? await inspectEmbeddedReport(raw, secrets) : null;
if (opaque) { await rm(filename); issues.push({ file: safeName, disposition: "omitted", reason: opaque }); return; }
const leak = findSecretLeak(raw, secrets);
let safe: string;
if (extension === ".json") {
try { safe = `${JSON.stringify(sanitizeJson(JSON.parse(raw), secrets), null, 2)}\n`; }
catch { await rm(filename); issues.push({ file: safeName, disposition: "omitted", reason: "reporter JSON could not be safely parsed" }); return; }
} else safe = redactText(raw, secrets);
if (safe !== raw) await writeFile(filename, safe, { mode: 0o600 });
if (leak) issues.push({ file: safeName, disposition: "redacted", reason: leak });
}
};
for (const source of sources) await visit(source);
return issues;
}