Files
PaperClipAI/tests/runner-e2e/prepare-report-lock.mjs
T
DottaandPaperclip 287b04d72f fix(evals): keep report dependency proof with lockfile refresh (#18063)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/192/changes/paperclip/192
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages AI agents and runs paid Product E2E evaluations.
> - Paid work needs a report that can install its trusted dependencies.
> - The report uses a reviewed offline lock correction.
> - New UI dependencies can make that correction stale while its lock
hash still matches.
> - The lockfile bot must refresh the correction with its lock changes.
> - This PR also checks manifest consistency offline before model
admission.

## Linked Issues or Issue Description

**What happened?**

Current master declares new CodeMirror and YAML UI dependencies. Its
committed lock omits those UI importer entries. The existing report
correction accepts that lock, but a frozen install then fails with
`ERR_PNPM_OUTDATED_LOCKFILE`. This can waste completed paid work before
the report fails.

Refs #18059. That bot PR supplies the matching lock. This PR keeps the
root lock bot-owned and supplies its exact offline report correction.
The correction changes 21 importer lines and leaves all package records
unchanged.

**Expected behavior**

The existing bot includes matching report proof in its dependency PR.
Paid authorization rejects stale manifests before model work. Report
installation still accepts only reviewed hashes and applies no registry
resolution.

**Steps to reproduce**

1. Run a frozen pnpm 9.15.4 install on master at
`7f10cc058568e3c1ac133cf24f3b4000cbdfca0e`.
2. Observe the missing UI importer entries.
3. Apply the reviewed correction in this PR. The frozen install passes.

## What Changed

- Add an explicit refresh mode that derives the offline patch and hashes
from a known committed lock and the bot's resolved lock.
- Keep the existing unknown-lock, patch hash and output hash guards.
- Include the proof files in the existing lockfile bot PR. New bot PRs
use complete descriptions; existing descriptions stay intact.
- Validate the frozen lock against current manifests offline before paid
admission, using the existing pinned pnpm version.
- Refresh the current correction and document this process. The root
lockfile is not committed here.

## Verification

- 24 focused report-lock and workflow-security tests pass. They cover
offline replay, tampered input/patch/output rejection, real Git refresh,
unknown-base refusal and unchanged proof.
- A full frozen install passes after the reviewed correction, with
lifecycle scripts disabled.
- A fresh source archive validates offline with an empty package store.
- A deliberate manifest change fails that same offline check with
`ERR_PNPM_OUTDATED_LOCKFILE`.
- Both workflow files parse as YAML. All 95 shell steps pass Bash syntax
checks.
- `git diff --check` passes. These checks make no model requests.
- Full repository checks and fresh reviews must pass on the final PR
head before merge.

## Risks

- Dependency changes must still pass review. Refresh refuses an unknown
committed baseline.
- The bot now commits two additional proof files. Its write permissions
and target branch stay unchanged. Existing PR descriptions are
preserved.
- The paid authorization job adds an offline manifest check. It runs no
package lifecycle scripts.
- No budget, provider credential, environment permission, schedule or
paid enable gate changes.

## Model Used

- OpenAI Codex, GPT-6, with reasoning and code execution tools. The
exact serving variant and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 15:30:58 -05:00

55 lines
3.6 KiB
JavaScript

import { createHash } from 'node:crypto';
import { execFileSync } from 'node:child_process';
import { readFile, writeFile, mkdtemp, rm } from 'node:fs/promises';
import path from 'node:path';
import { tmpdir } from 'node:os';
import { pathToFileURL } from 'node:url';
const sha256 = value => createHash('sha256').update(value).digest('hex');
export async function prepareReportLock(root = path.resolve(import.meta.dirname, '../..')) {
const policy = JSON.parse(await readFile(path.join(root, 'tests/runner-e2e/report-lock.json'), 'utf8'));
const lockPath = path.join(root, 'pnpm-lock.yaml');
const bytes = await readFile(lockPath), current = sha256(bytes);
if (current === policy.outputSha256) return 'already-reviewed';
if (current !== policy.inputSha256) throw new Error('Reporting lock changed; review the dependency correction before install');
const patch = await readFile(path.join(root, 'tests/runner-e2e/report-lock.patch'));
if (sha256(patch) !== policy.patchSha256) throw new Error('Reviewed reporting lock patch changed');
const result = execFileSync('git', ['apply', '--numstat', '-'], { cwd: root, input: patch, encoding: 'utf8' });
if (!/^\d+\s+\d+\s+pnpm-lock\.yaml\n$/.test(result)) throw new Error('Reporting correction must change only the lockfile');
const temporary = await mkdtemp(path.join(tmpdir(), 'eval-reviewed-lock-'));
try {
await writeFile(path.join(temporary, 'pnpm-lock.yaml'), bytes);
execFileSync('git', ['apply', '-'], { cwd: temporary, input: patch });
const corrected = await readFile(path.join(temporary, 'pnpm-lock.yaml'));
if (sha256(corrected) !== policy.outputSha256) throw new Error('Reviewed reporting lock result changed');
await writeFile(lockPath, corrected);
return 'reviewed-correction';
} finally { await rm(temporary, { recursive: true, force: true }); }
}
export async function refreshReportLock(root = path.resolve(import.meta.dirname, '../..')) {
const directory = path.join(root, 'tests/runner-e2e');
const policyPath = path.join(directory, 'report-lock.json');
const policy = JSON.parse(await readFile(policyPath, 'utf8'));
const committed = execFileSync('git', ['show', 'HEAD:pnpm-lock.yaml'], { cwd: root });
const inputSha256 = sha256(committed);
if (![policy.inputSha256, policy.outputSha256].includes(inputSha256)) {
throw new Error('Committed report lock is not reviewed; refusing to refresh its proof');
}
const resolved = await readFile(path.join(root, 'pnpm-lock.yaml'));
if (resolved.equals(committed)) return 'unchanged';
const patch = execFileSync('git', ['diff', '--no-ext-diff', '--no-textconv', '--no-color', 'HEAD', '--', 'pnpm-lock.yaml'], { cwd: root });
const changed = execFileSync('git', ['apply', '--numstat', '-'], { cwd: root, input: patch, encoding: 'utf8' });
if (!/^\d+\s+\d+\s+pnpm-lock\.yaml\n$/.test(changed)) throw new Error('Report correction must change only the lockfile');
await writeFile(path.join(directory, 'report-lock.patch'), patch);
await writeFile(policyPath, JSON.stringify({ inputSha256, outputSha256: sha256(resolved), patchSha256: sha256(patch),
basis: 'Generated with the trusted master lockfile refresh. Review this offline correction and its hashes with the dependency changes.' }, null, 2) + '\n');
return 'refreshed';
}
if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) {
const args = process.argv.slice(2);
if (args.length === 0) console.log(await prepareReportLock());
else if (args.length === 1 && args[0] === '--refresh') console.log(await refreshReportLock());
else throw new Error('Usage: prepare-report-lock.mjs [--refresh]');
}