Files
PaperClipAI/server/scripts/reporting-lock-policy.test.mjs
T
DottaandPaperclip 3fca8a1500 test(evals): renew the reporting lock after iMessage update (#19283)
## Thinking Path

> - Paperclip verifies the selected source and reporting dependencies
before it obtains model credentials.
> - The reporting helper admits only an exact reviewed lock or its exact
reviewed correction.
> - Master updated the iMessage dependency and protobuf after the last
reporting-policy review.
> - The controller lock therefore stopped matching either admitted hash.
> - A real five-case launch failed at this guard before any case
execution.
> - This PR renews the exact hashes and keeps the existing patch and
rejection mechanism unchanged.
> - A normal-CI repository check now exercises the real helper against
the committed lock before paid execution.

## Linked Issues or Issue Description

Refs: #19261, #18914.

**What happened?**
The five selected execution jobs in the retained launch 3002 stopped
during “Verify source-bound build and authenticated selection before
credentials.” The first observed error was `Reporting lock changed;
review the dependency correction before install` in the trusted
controller helper. Model-key and original case execution steps were
skipped. The failed attempt remains infrastructure evidence; this PR
makes no native product-grade claim.

**Expected behavior**
Admit the exact reviewed current controller dependency graph. Keep
rejecting unknown locks, changed correction patches, and wrong output
bytes before credentials or provider execution.

**Steps to reproduce**
Run the reporting helper against the committed current-master lock. The
parent policy recognizes 754… and 60f…, while the actual controller lock
is 6f252b…. The parent stops at the same pre-credential boundary as AWS.
The existing correction patch reverses and reapplies cleanly to that
lock.

## What Changed

- Renew only the reporting policy input and output SHA256 values. The
output is the exact committed current-master lock after
`@photon-ai/advanced-imessage` 2.1.0→2.2.0 and `@bufbuild/protobuf`
2.15.0→2.16.0.
- Derive input
`ea91060774e3289d5d07bc58f72d8953f5a6dcb9db98aa9a1adf202476bf89ac` by
reversing the existing correction patch from the current lock.
Reapplying that byte-identical patch yields output
`6f252b0c8099ed837a297f97f1fc5e14e96bc7530b0239065d0789b33b2ee477`
exactly.
- Keep patch SHA256
`2adced9a66052ba5752140ec6cc6670f679552a9a8559bd1faea10036afeeb97`,
helper code, and guard behavior unchanged. Update the policy basis to
describe the review truthfully.
- Add one normal server-suite check. It copies the committed lock and
reviewed policy into a temporary directory and runs the real helper. The
prior focused suite already covered this contract, but its runner-e2e
location is outside normal root Vitest projects.
`server/vitest.config.ts` already includes `scripts/**/*.test.mjs`; no
workflow change is needed.

No `pnpm-lock.yaml`, workflow, source pin, descriptor, oracle, model,
permission, credential, scanner, or case bound changes. Immutable source
6773 and the failed launch remain unchanged.

## Verification

- Parent existing reporting suite: 4 fail and 3 pass because the
committed lock is outside the admitted pair. Renewed policy: all 7 pass,
including unknown-lock, changed-patch, wrong-output, and refresh-path
rejection controls.
- New normal-CI repository control fails with the exact paid-step
reporting-lock error on the parent policy. The corrected combined run
passes 8 controls across 2 suites.
- Exact patch reverse/forward roundtrip preserves the full current lock
bytes. The actual CLI returns `already-reviewed`; the working lock hash
stays 6f252b… and is not modified.
- Focused command: `node node_modules/vitest/vitest.mjs run --config
../report-lock-imessage-vitest.config.mts`. The external local config
selects only the two affected suites with the existing 30-second test
limit.
- `git diff --check` passes. No SDK/native/source build, model call,
evaluation retry, or private archive download was needed for local
proof. Full workspace typecheck/build were not run for this
policy-and-test change. Latest-head required CI and genuine review
remain pending.

## Risks

This policy deliberately rejects the next unreviewed lock change. The
normal-CI check should expose that drift before paid fanout. The input
is the current lock with the already approved correction reversed; it is
not a claim that a new dependency graph was resolved or installed
locally. Local guard controls do not prove a repaired AWS execution or
any of the five product journeys. The next measured attempt still
requires normal protected merge and fresh actual-controller admission.

## Model Used

OpenAI `gpt-6.1-sol`, with reasoning, repository tools, and local code
execution. The host did not expose a context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 11:32:46 -06:00

27 lines
1.4 KiB
JavaScript

import { execFileSync } from 'node:child_process';
import { createHash } from 'node:crypto';
import { copyFile, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { expect, it } from 'vitest';
import { prepareReportLock } from '../../tests/runner-e2e/prepare-report-lock.mjs';
it('admits the committed controller lock through the reviewed reporting policy before paid execution', async () => {
const repository = fileURLToPath(new URL('../../', import.meta.url));
const root = await mkdtemp(path.join(tmpdir(), 'controller-report-policy-'));
const relative = 'tests/runner-e2e';
try {
await mkdir(path.join(root, relative), { recursive: true });
for (const file of ['report-lock.json', 'report-lock.patch']) {
await copyFile(path.join(repository, relative, file), path.join(root, relative, file));
}
await writeFile(path.join(root, 'pnpm-lock.yaml'), execFileSync('git', ['show', 'HEAD:pnpm-lock.yaml'], { cwd: repository }));
await prepareReportLock(root);
const policy = JSON.parse(await readFile(path.join(root, relative, 'report-lock.json'), 'utf8'));
expect(createHash('sha256').update(await readFile(path.join(root, 'pnpm-lock.yaml'))).digest('hex')).toBe(policy.outputSha256);
} finally {
await rm(root, { recursive: true, force: true });
}
});