Files
PaperClipAI/tests/runner-e2e/prepare-report-lock.test.ts
T
DottaandPaperclip a8bd012635 fix(evals): apply reviewed lock corrections to reports and qualification images (#17937)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/98/changes/paperclip/98
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages agents and records their work.
> - Product E2E campaigns measure real agent behavior and model spend.
> - A paid hosted run completed both cases and all ten model receipts.
> - Its report job could not install dependencies from the committed
lock.
> - The trusted checkout needs a reviewed dependency correction.
> - This change applies a reviewed lock correction before the frozen
report install.

## Linked Issues or Issue Description

Refs: #17928

**Description**
The report in [run
40576181788](https://github.com/paperclipai/paperclip/actions/runs/40576181788)
failed with `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH`. Both paid cases had
passed. The worker and build jobs already restored the resolved source
lock, but the report did not.

**Steps to reproduce**
Dispatch the bounded daily campaign on the default branch with a stale
committed lock configuration. Workers use the resolved lock. The report
uses the stale lock and fails before report generation.

**Expected behavior**
The report installs frozen dependencies from a verified lock for the
exact trusted source. Evidence can be reported without repeating paid
work.

## What Changed

- Apply a reviewed lock correction from the trusted report checkout
before its frozen install. Use the same correction for the qualification
image build, which also requires a compatible frozen lock.
- Pin the original lock, correction and output SHA-256. The trusted
authorization job rejects unknown inputs before any paid work.
- Apply the same correction to trusted manual history publication.
Qualification image builds apply it only when their target equals the
workflow revision; other targets keep their own lock.
- Update existing Claude ACP and postgres patch identities and link the
declared yaml dependency. Existing package tarball integrity entries
stay unchanged; the declared yaml dependency adds exactly yaml@2.9.1
with pinned integrity.
- Keep runtime registry resolution and worker target locks out of the
credential-bearing report.
- Test valid correction, idempotence and changed-lock, changed-patch and
wrong-output rejection.

## Verification

- Focused workflow and lock-correction tests: 36 passed.
- `actionlint .github/workflows/runner-full-stack-e2e.yml` passed.
- Original worker evidence: 2/2 passed, ten final model receipts, zero
unresolved model requests.
- The original report was regenerated from retained evidence with zero
provider calls and saved privately with replay provenance.
- Broader repository checks run in CI. Prior full Product checks passed
for #17928.

## Risks

The correction is tied to the reviewed committed lock. Dependency input
changes need a reviewed correction update. No new registry version or
lifecycle script can enter the privileged report through this fix. The
original failed workflow stays failed. No root lockfile is committed.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 20:55:13 -05:00

41 lines
2.6 KiB
TypeScript

import { mkdtemp, mkdir, readFile, writeFile, copyFile, rm } from 'node:fs/promises';
import { createHash } from 'node:crypto';
import { execFileSync } from 'node:child_process';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { expect, it } from 'vitest';
import { parse } from 'yaml';
import { prepareReportLock } from './prepare-report-lock.mjs';
it.each(['valid', 'unknown-lock', 'changed-patch', 'wrong-output'])(
'installs only the reviewed dependency correction: %s', async variant => {
const root = await mkdtemp(path.join(tmpdir(), 'eval-report-lock-test-'));
const here = import.meta.dirname, relative = path.join(root, 'tests/runner-e2e');
try {
await mkdir(relative, { recursive: true });
for (const file of ['report-lock.json', 'report-lock.patch']) await copyFile(path.join(here, file), path.join(relative, file));
const input = execFileSync('git', ['show', 'HEAD:pnpm-lock.yaml'], { cwd: path.resolve(here, '../..') });
const policy = JSON.parse(await readFile(path.join(relative, 'report-lock.json'), 'utf8'));
expect(createHash('sha256').update(input).digest('hex')).toBe(policy.inputSha256);
const lock = path.join(root, 'pnpm-lock.yaml');
await writeFile(lock, variant === 'unknown-lock' ? 'unknown\n' : input);
if (variant === 'changed-patch') await writeFile(path.join(relative, 'report-lock.patch'), 'changed');
if (variant === 'wrong-output') { policy.outputSha256 = '0'.repeat(64); await writeFile(path.join(relative, 'report-lock.json'), JSON.stringify(policy)); }
if (variant === 'valid') {
expect(await prepareReportLock(root)).toBe('reviewed-correction');
expect(createHash('sha256').update(await readFile(lock)).digest('hex')).toBe(policy.outputSha256);
expect(await prepareReportLock(root)).toBe('already-reviewed');
const before = parse(input.toString()).packages;
const after = parse(await readFile(lock, 'utf8')).packages;
expect(Object.keys(after).filter(key => !Object.hasOwn(before, key))).toEqual(['yaml@2.9.1']);
for (const [key, value] of Object.entries(before)) expect(after[key]).toEqual(value);
expect(after['yaml@2.9.1'].resolution.integrity).toBe('sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw==');
} else {
await expect(prepareReportLock(root)).rejects.toThrow();
expect(await readFile(lock)).toEqual(variant === 'unknown-lock' ? Buffer.from('unknown\n') : input);
}
} finally { await rm(root, { recursive: true, force: true }); }
},
);