feat(evals): retain portable Runner controller evidence (#17469)

Retain private Runner request evidence for offline cost reconciliation. Bound post-child reads, reject hidden paid tools, exclude configured credentials, bind incomplete results, and replace redacted proof atomically.

Verified at f5c1791623: full hosted CI, Greptile 5/5, secondary review summary 5/5, no unresolved threads. No paid evals or schedule changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

packages/paperclip-runner/docs/runner-protocol-live-evals.md

@@ -276,6 +276,139 @@ operating allowance, or enable a paid campaign. The existing Product E2E relay
 and launch generator remain the integration owners; the hosted Runner workflow
 still needs its own qualified controller integration.
 
+
+## Controlled request budgets
+
+The source-checkout controller is
+`packages/paperclip-runner/scripts/controlled-evals/run.mjs`. Runner/evals owns
+this orchestration. It reuses the existing eval request proxy and ledger from
+`tests/runner-e2e` through one import graph; it does not copy their reservation
+or provider-usage rules. It requires the workspace's installed `tsx` runtime.
+It is not a published Runner package entry point.
+
+The controller accepts the existing `--request` and `--output` contract. Set
+`PAPERCLIP_RUNNER_EVAL_CONTROL_MANIFEST` to an explicit reviewed manifest and
+supply the selected upstream credential only to this trusted controller. For
+Runner Evals, pass this controller path as `eval_program.py run --runner-cli`;
+the manifest identifies the actual built `eval-session.js`. The provider child
+receives only its temporary proxy credential and private home.
+
+The manifest schema is `paperclip.runner-eval.controller/v1`. It contains:
+
+- `campaignId`, `maxUsd`, `deadlineAt`, `pricingVerifiedAt`, and
+  `pricingExpiresAt`; admission must fit a current four-hour window.
+- `sourceRevision` and `definitionsRevision`, each an exact commit ID.
+  These are the trusted preparer's provenance declarations.
+- `runnerCli`, `runnerPackage`, and `runnerd`, each an absolute `path` and
+  content `sha256`. The controller checks the three files before launch.
+- An absolute durable `outputDirectory`, outside the disposable case workspace.
+- `route` (`direct` or `braintrust`), plus `projectId` for Braintrust.
+- `models`, using the existing exact-model price, request-limit, and reviewed
+  full-lifecycle `provisionalBound` contract.
+- `overhead`, using the existing four-component operating allowance with a
+  current full-lifecycle bound for every component. This is reserved once.
+- `attempts`, each with a unique `executionId` and `attemptId`,
+  `requestSha256`, `provider`, optional `acpxAgent`, exact `model`, `maxUsd`,
+  `maxRequests`, `maxConcurrent`, and `timeoutMs`. Native Codex and ACPX
+  Claude are supported. Each execution has one explicit attempt and no retry.
+
+Use `requestBindingSha256` from `controlled-evals/manifest.mjs` to bind the
+complete compiled request. It excludes only the temporary working directory
+and native executable path; the executable digest remains bound and its path
+is checked separately. The controller gives the child a retained copy of the
+validated request. Changing prompts, fixtures, models, limits, or attempts
+requires a new reviewed binding.
+
+The sum of every cell's allocation and the once-only operating allowance must
+fit `maxUsd` before any cell starts. A request reserves against its cell's
+ledger before forwarding. Cells cannot borrow unused allocations. Unknown
+requests keep their reservations, and failed or missing cell evidence keeps
+that cell's full allocation in the campaign report. An existing cell directory
+or output is never reset for a retry. Cancellation closes admission, drains
+upstream receipts within the proxy limit, and stops the owned CLI process
+group. If the CLI exits while its owned group survives, the controller closes
+admission, stops that group, bounds inherited output pipes, and records an
+incomplete attempt. Unverified provider cleanup remains explicit and retains
+the allocation.
+
+Each result retains `controllerAccounting` alongside the original provider
+outcome and usage. Retain the durable controller directory as private evidence:
+its admission, request, gateway descriptor, request receipts, ledger, original
+provider artifact, and final result are required for reconciliation. The
+private `controller-evidence.json` bundle can travel with the immutable attempt.
+Set `PAPERCLIP_RUNNER_EVAL_ARCHIVE_CLI` to the sibling `archive.mjs` when using a
+private eval program with controller retention support. That program validates
+the request before launch, captures the bundle before finalizing the attempt,
+and marks missing or incomplete capture as a non-retryable accounting failure.
+It must use zero infrastructure retries. Older private programs do not capture
+this bundle; result metadata alone is not proof of retention.
+
+```sh
+node packages/paperclip-runner/scripts/controlled-evals/report.mjs \
+  /private/path/controller-manifest.json > /private/path/controller-report.json
+```
+
+To retain or inspect the evidence independently of the original worker:
+
+```sh
+node packages/paperclip-runner/scripts/controlled-evals/archive.mjs pack \
+  /private/path/controller-manifest.json attempt-id \
+  /private/path/runs/attempt-id/controller-evidence.json
+node packages/paperclip-runner/scripts/controlled-evals/archive.mjs report-attempts \
+  /private/path/runs/attempt-id
+```
+
+`pack` is exclusive and never overwrites evidence. The private bundle contains
+the full manifest and exact bytes of the admission, request, gateway descriptor,
+ledger, receipts, provider artifact, result, and bounded stdout/stderr. It omits
+executables, private homes, and caches. Files carry byte lengths and SHA-256
+hashes. These detect corruption; they are not signatures or proof against a
+host that can rewrite all evidence. Known configured credential values are
+checked before encoding and after decoding. Keep bundles private.
+
+Controller post-child reads, archive creation, and durable reports use the same
+nonblocking, no-follow, byte-capped regular-file reader. Configured private/provider
+keys and credentials inside JSON configuration contents are checked in both raw
+and JSON-escaped form. Attempt IDs contain only letters, digits, `_`, `.`, and `-`
+so each retained attempt has one directory name.
+
+The reader accepts only bounded regular files and fixed evidence paths. A
+bundle is limited to 64 MiB of decoded content plus encoding overhead; this
+storage-format limit does not qualify a vendor charge or operating-cost bound.
+`report` accepts standalone bundle files; `report-attempts` additionally binds
+them to the surrounding immutable request and reconstructed controller output,
+including the provider outcome, usage, and complete accounting result. The
+private program may add provenance or a stricter non-retryable retention
+failure. Missing source proof remains incomplete. Both work after the
+original worker directory is gone, using the same hash-chain ledger verifier.
+They group attempts by manifest, count operating holds once, deduplicate exact
+copies, reject conflicting copies, and keep full allocations for missing
+selected attempts. They never consult old worker paths to fill archive gaps.
+
+The campaign collector preserves the bundle in private runs and returns its
+independently replayed totals as `controlledCosts`. For this collector, each
+manifest `executionId` must equal its catalog `cellId`, and the complete
+manifest selection, model, provider, agent profile, and driver must match the
+catalog, including cells with no retained result. Extra attempts are rejected.
+Existing `costs` remain
+legacy provider-usage observations; do not add them to `controlledCosts`.
+The public attempt projection omits bundles and raw `controllerAccounting`.
+Private reports label ledger-known amounts and conservative holds as provisional
+or incomplete; a missing verifier or invalid evidence cannot become a zero cost.
+
+The report recomputes ledger costs, checks their attempt/manifest bindings, and
+preserves known partial charges. It reports operating costs as held and total
+costs as provisional when evidence is intact, or incomplete otherwise. It does
+not release historical holds or claim invoice
+settlement. Existing provider usage is retained separately; do not add the
+provider and controller cost observations as if they were separate charges.
+
+This local controller is not yet wired into the hosted matrix. Its durable
+attempt-directory guard applies to one shared filesystem, not independent
+GitHub workers. Hosted integration must bind unique cells to the prepared
+allocation, reject workflow/job reruns, and preserve the controller evidence
+before enabling paid work. Native OTLP routing, report integration, vendor
+metering bounds, and actual paid qualification remain separate gates.
 ## Hosted campaign
 
 Use the `Runner Direct Live Protocol Evals` workflow. Dispatch the workflow

packages/paperclip-runner/scripts/controlled-evals/archive.mjs

@@ -0,0 +1,186 @@
+import { lstat, readdir, writeFile } from 'node:fs/promises';
+import { basename, join, resolve } from 'node:path';
+import { pathToFileURL } from 'node:url';
+import { bindRequest, canonical, fields, readJson, requestBindingSha256, sha256, units, usd, validateManifest } from './manifest.mjs';
+import { attemptDirectory, controlledResultArtifact } from './run.mjs';
+import { summarizeControlledCampaign } from './report.mjs';
+import { FILE_LIMITS, RECEIPT, limitFor, readRegularFile } from './evidence-files.mjs';
+
+const MiB = 1_048_576;
+const MAX_RAW_BYTES = 64 * MiB;
+const MAX_ARCHIVE_BYTES = 96 * MiB;
+
+function secretValues({ environment = process.env, secrets = [] } = {}) {
+  const configured = [...secrets, ...Object.entries(environment)
+    .filter(([name]) => /(?:KEY|TOKEN|SECRET|PASSWORD|ACCESS_KEY_ID|CONFIG_CONTENT)$/i.test(name))
+    .map(([, value]) => value)].filter(value => typeof value === 'string' && value.length > 0);
+  for (const [name, value] of Object.entries(environment)) {
+    if (!/CONFIG_CONTENT$/i.test(name) || typeof value !== 'string') continue;
+    try {
+      const visit = item => {
+        if (!item || typeof item !== 'object') return;
+        for (const [key, entry] of Object.entries(item)) {
+          if (/(?:key|token|secret|password)$/i.test(key) && typeof entry === 'string' && entry) configured.push(entry);
+          else visit(entry);
+        }
+      };
+      visit(JSON.parse(value));
+    } catch {}
+  }
+  return [...new Set(configured.flatMap(value => [value, JSON.stringify(value).slice(1, -1)]))];
+}
+function excludeSecrets(bytes, secrets) {
+  if (secrets.some(secret => bytes.includes(Buffer.from(secret)))) throw new Error('Controller evidence contains a configured credential');
+}
+
+async function realDirectory(path) {
+  try {
+    const stat = await lstat(path);
+    if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error('Controller evidence directory is not a real directory');
+    return true;
+  } catch (error) { if (error.code === 'ENOENT') return false; throw error; }
+}
+
+export async function packControlledAttempt({ manifest: value, attemptId, ...options }) {
+  const { manifest, manifestSha256 } = validateManifest(value, Date.now(), { admission: false });
+  const allocation = manifest.attempts.find(attempt => attempt.attemptId === attemptId);
+  if (!allocation) throw new Error('Attempt is absent from the controller manifest');
+  const secrets = secretValues(options);
+  const manifestBytes = Buffer.from(canonical(manifest));
+  if (manifestBytes.length > MiB) throw new Error('Controller manifest exceeds its archive limit');
+  excludeSecrets(manifestBytes, secrets);
+  const root = attemptDirectory(manifest, attemptId), files = {};
+  let total = manifestBytes.length;
+  if (await realDirectory(root)) {
+    const names = [...FILE_LIMITS.keys()];
+    if (await realDirectory(join(root, 'receipts'))) {
+      for (const name of await readdir(join(root, 'receipts'))) {
+        if (RECEIPT.test(`receipts/${name.replace(/\.tmp$/, '')}`) && name.endsWith('.tmp')) continue;
+        if (!RECEIPT.test(`receipts/${name}`)) throw new Error('Unexpected controller receipt path');
+        names.push(`receipts/${name}`);
+      }
+    }
+    if (names.length > allocation.maxRequests + FILE_LIMITS.size) throw new Error('Too many controller receipts');
+    for (const name of names.sort()) {
+      let bytes;
+      try { bytes = await readRegularFile(join(root, name), limitFor(name)); }
+      catch (error) { if (error.code === 'ENOENT') continue; throw error; }
+      total += bytes.length;
+      if (total > MAX_RAW_BYTES) throw new Error('Controller evidence exceeds its archive limit');
+      excludeSecrets(bytes, secrets);
+      files[name] = { sizeBytes: bytes.length, sha256: sha256(bytes), base64: bytes.toString('base64') };
+    }
+  }
+  return { schema: 'paperclip.runner-eval.control-evidence/v1', manifest, manifestSha256, attemptId, files };
+}
+
+export function decodeControlledEvidence(value, options) {
+  fields(value, ['schema', 'manifest', 'manifestSha256', 'attemptId', 'files']);
+  const { manifest, manifestSha256 } = validateManifest(value.manifest, Date.now(), { admission: false });
+  const allocation = manifest.attempts.find(attempt => attempt.attemptId === value.attemptId);
+  if (value.schema !== 'paperclip.runner-eval.control-evidence/v1' || value.manifestSha256 !== manifestSha256 || !allocation) throw new Error('Controller archive identity mismatch');
+  fields(value.files, Object.keys(value.files ?? {}));
+  const entries = Object.entries(value.files);
+  if (entries.length > allocation.maxRequests + FILE_LIMITS.size) throw new Error('Too many controller evidence files');
+  const secrets = secretValues(options), manifestBytes = Buffer.from(canonical(manifest));
+  if (manifestBytes.length > MiB) throw new Error('Controller manifest exceeds its archive limit');
+  excludeSecrets(manifestBytes, secrets);
+  let total = manifestBytes.length;
+  const files = new Map();
+  for (const [name, file] of entries) {
+    const limit = limitFor(name);
+    fields(file, ['sizeBytes', 'sha256', 'base64']);
+    if (!limit || !Number.isSafeInteger(file.sizeBytes) || file.sizeBytes < 0 || file.sizeBytes > limit
+        || typeof file.base64 !== 'string' || file.base64.length !== 4 * Math.ceil(file.sizeBytes / 3)
+        || typeof file.sha256 !== 'string' || !/^[a-f0-9]{64}$/.test(file.sha256)) throw new Error('Invalid controller evidence file');
+    total += file.sizeBytes;
+    if (total > MAX_RAW_BYTES) throw new Error('Controller evidence exceeds its archive limit');
+    const bytes = Buffer.from(file.base64, 'base64');
+    if (bytes.length !== file.sizeBytes || bytes.toString('base64') !== file.base64 || sha256(bytes) !== file.sha256) throw new Error('Controller evidence digest mismatch');
+    excludeSecrets(bytes, secrets);
+    files.set(name, bytes);
+  }
+  return { manifest, manifestSha256, attemptId: value.attemptId, files };
+}
+
+export async function readControlledEvidence(path, options) {
+  const value = JSON.parse((await readRegularFile(path, MAX_ARCHIVE_BYTES)).toString('utf8'));
+  decodeControlledEvidence(value, options);
+  return value;
+}
+
+export async function readAttemptControlledEvidence(directory, options) {
+  const bundle = await readControlledEvidence(join(directory, 'controller-evidence.json'), options);
+  const artifact = JSON.parse((await readRegularFile(join(directory, 'artifact.json'), 16 * MiB)).toString('utf8'));
+  const request = JSON.parse((await readRegularFile(join(directory, 'request.json'), 8 * MiB)).toString('utf8'));
+  const allocation = bundle.manifest.attempts.find(attempt => attempt.attemptId === bundle.attemptId);
+  if (basename(resolve(directory)) !== bundle.attemptId || artifact.attemptId !== bundle.attemptId
+      || requestBindingSha256(request) !== allocation.requestSha256
+      || artifact.controllerAccounting && artifact.controllerAccounting.manifestSha256 !== bundle.manifestSha256) {
+    throw new Error('Immutable attempt does not match its controller archive');
+  }
+  const { files } = decodeControlledEvidence(bundle, options);
+  const reportBytes = files.get('result.json'), providerBytes = files.get('provider-artifact.json');
+  if (reportBytes) {
+    let report, provider;
+    try { report = JSON.parse(reportBytes.toString('utf8')); provider = providerBytes ? JSON.parse(providerBytes.toString('utf8')) : undefined; }
+    catch { throw new Error('Malformed immutable controller result proof'); }
+    const expected = controlledResultArtifact(provider, report, allocation), actual = { ...artifact };
+    delete expected.provenance; delete actual.provenance;
+    if (actual.controllerEvidenceError !== undefined) {
+      if (typeof actual.controllerEvidenceError !== 'string' || !actual.controllerEvidenceError.trim()
+          || typeof actual.accountingError !== 'string' || !actual.accountingError.trim()
+          || canonical(actual.accountingFailure) !== canonical({ class: 'provider_budget_coverage_unknown', category: 'provider_budget', retryable: false })) {
+        throw new Error('Invalid immutable controller retention failure');
+      }
+      delete actual.controllerEvidenceError;
+      if (expected.accountingFailure) { actual.accountingFailure = expected.accountingFailure; actual.accountingError = expected.accountingError; }
+      else { delete actual.accountingFailure; delete actual.accountingError; }
+    }
+    if (canonical(actual) !== canonical(expected)) throw new Error('Immutable result disagrees with its controller proof');
+  }
+
+  return bundle;
+}
+
+export async function summarizeControlledEvidence(bundles, options) {
+  if (!Array.isArray(bundles) || !bundles.length) throw new Error('No controller evidence supplied');
+  const groups = new Map();
+  for (const bundle of bundles) {
+    const decoded = decodeControlledEvidence(bundle, options);
+    let group = groups.get(decoded.manifest.campaignId);
+    if (group && group.manifestSha256 !== decoded.manifestSha256) throw new Error('Conflicting controller campaign manifests');
+    if (!group) {
+      group = { ...decoded, filesByAttempt: new Map(), digests: new Map() };
+      groups.set(decoded.manifest.campaignId, group);
+    }
+    const digest = sha256(canonical(bundle));
+    if (group.digests.has(decoded.attemptId) && group.digests.get(decoded.attemptId) !== digest) throw new Error('Conflicting controller attempt archives');
+    group.digests.set(decoded.attemptId, digest);
+    group.filesByAttempt.set(decoded.attemptId, decoded.files);
+  }
+  const campaigns = [];
+  for (const group of [...groups.values()].sort((a, b) => a.manifest.campaignId.localeCompare(b.manifest.campaignId))) {
+    campaigns.push(await summarizeControlledCampaign(group.manifest, { filesByAttempt: group.filesByAttempt }));
+  }
+  const sum = field => usd(campaigns.reduce((total, campaign) => total + units(campaign[field]), 0n));
+  return { schema: 'paperclip.runner-eval.control-history/v1', costStatus: campaigns.some(c => c.costStatus === 'incomplete') ? 'incomplete' : 'provisional',
+    knownUsd: sum('knownUsd'), heldUsd: sum('heldUsd'), committedUsd: sum('committedUsd'), operatingHeldUsd: sum('operatingHeldUsd'),
+    incompleteAttempts: campaigns.reduce((total, c) => total + c.incompleteAttempts, 0), overBudget: campaigns.some(c => c.overBudget), campaigns };
+}
+
+if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) {
+  const [command, ...args] = process.argv.slice(2);
+  if (command === 'verify' && args.length === 2) {
+    const { manifest } = validateManifest(await readJson(args[0]));
+    bindRequest(manifest, await readJson(args[1], 8 * MiB));
+  } else if (command === 'pack' && args.length === 3) {
+    const bundle = await packControlledAttempt({ manifest: await readJson(args[0]), attemptId: args[1] });
+    await writeFile(args[2], JSON.stringify(bundle) + '\n', { flag: 'wx', mode: 0o600 });
+  } else if (['report', 'report-attempts'].includes(command) && args.length) {
+    const read = command === 'report' ? readControlledEvidence : readAttemptControlledEvidence;
+    const result = await summarizeControlledEvidence(await Promise.all(args.map(path => read(path))));
+    process.stdout.write(JSON.stringify(result, null, 2) + '\n');
+    if (result.costStatus === 'incomplete') process.exitCode = 2;
+  } else throw new Error('Usage: archive.mjs verify <manifest.json> <request.json> | pack <manifest.json> <attempt-id> <output.json> | report <evidence.json>... | report-attempts <attempt-directory>...');
+}

packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs

@@ -1,24 +1,31 @@
 import assert from 'node:assert/strict';
 import { test } from 'node:test';
-import { mkdtemp, mkdir, readFile, readdir, rm, symlink, writeFile } from 'node:fs/promises';
+import { execFileSync } from 'node:child_process';
+import { mkdtemp, mkdir, readFile, readdir, rm, symlink, truncate, writeFile } from 'node:fs/promises';
 import { writeFileSync } from 'node:fs';
 import { tmpdir } from 'node:os';
 import { join } from 'node:path';
 import { canonical, costRuntime, requestBindingSha256, sha256, validateManifest } from './manifest.mjs';
 import { attemptDirectory, runControlledEval } from './run.mjs';
 import { summarizeControlledCampaign } from './report.mjs';
+import { readRegularFile, replaceEvidenceFile } from './evidence-files.mjs';
+import { decodeControlledEvidence, packControlledAttempt, readControlledEvidence, readAttemptControlledEvidence, summarizeControlledEvidence } from './archive.mjs';
+import { aggregateProtocolEvalCampaign, sanitizeProtocolEvalRuns } from '../runner-protocol-eval-campaign.mjs';
 
 const childScript = options => `
 import assert from 'node:assert/strict';
 import {createHash} from 'node:crypto';
 import {spawn} from 'node:child_process';
 import {once} from 'node:events';
-import {readFile,writeFile} from 'node:fs/promises';
+import {readFile,writeFile,rm,readdir,symlink,truncate} from 'node:fs/promises';
+import {dirname,join} from 'node:path';
+import {execFileSync} from 'node:child_process';
 const args=new Map();for(let i=2;i<process.argv.length;i+=2)args.set(process.argv[i],process.argv[i+1]);
 const request=JSON.parse(await readFile(args.get('--request'),'utf8'));
 if(${Boolean(options.resistantDescendant)}) {
- const descendant=spawn(process.execPath,['-e',"process.on('SIGTERM',()=>{});process.stdout.write('ready');setInterval(()=>{},1000)"],{stdio:['ignore','pipe','ignore']});
+ const orphan=${Boolean(options.orphanExit)};
- await once(descendant.stdout,'data');
+ const descendant=spawn(process.execPath,['-e',"process.on('SIGTERM',()=>{});"+(orphan?"process.send('ready');":"process.stdout.write('ready');")+"setInterval(()=>{},1000)"],{stdio:orphan?['ignore','inherit','inherit','ipc']:['ignore','pipe','ignore']});
+ if(orphan)await once(descendant,'message');else await once(descendant.stdout,'data');
  await writeFile(request.session.workingDirectory+'/descendant.pid',String(descendant.pid));
 }
 
@@ -34,6 +41,17 @@ if(!${Boolean(options.missingArtifact)})await writeFile(args.get('--output'),JSO
  requestRouting:${options.missingRoute ? 'null' : "{mode:'controller_gateway',provider:descriptor.provider,declaredUpstreamRoute:descriptor.upstreamRoute,campaignId:descriptor.campaignId,executionId:descriptor.executionId,attemptId:descriptor.attemptId,descriptorSha256:createHash('sha256').update(descriptorRaw).digest('hex')}"},
  snapshot:{status:'closed',process:{runnerExited:true,runnerPid:process.pid,providerPid:process.pid}},statuses,
  leaked:${options.leakToken ? 'process.env.PAPERCLIP_RUNNER_EVAL_GATEWAY_TOKEN' : 'null'}}));
+if(${Boolean(options.invalidEvidence)}) {
+ const root=dirname(args.get('--output'));
+ const name=${JSON.stringify(options.invalidEvidence ?? '')};
+ const file=name==='receipt'?join(root,'receipts',(await readdir(join(root,'receipts')))[0]):join(root,name);
+ await rm(file);
+ const kind=${JSON.stringify(options.invalidFileType ?? '')};
+ if(kind==='fifo')execFileSync('mkfifo',[file]);
+ else if(kind==='symlink')await symlink('/dev/zero',file);
+ else {await writeFile(file,'');await truncate(file,16*1024*1024+1);}
+}
+if(${Boolean(options.orphanExit)})process.exit(0);
 `;
 
 async function fixture(t, { count = 1, allocation = '0.01', providerProfile = 'codex', route = 'direct', ...options } = {}) {
@@ -117,6 +135,7 @@ test('disjoint concurrent cells cannot borrow an allowance and unknown requests
 test('rejects oversized or unqualified allocations, changed requests and changed executables before dispatch', async t => {
   const f = await fixture(t, { count: 2 });
   for (const mutate of [
+    value => { value.attempts[0].attemptId = 'nested/attempt'; },
     value => { value.maxUsd = '0.01'; }, value => { value.attempts[1].executionId = value.attempts[0].executionId; },
     value => { value.attempts[0].provider = 'acpx'; value.attempts[0].acpxAgent = 'codex'; },
     value => { delete value.overhead.components[0].provisionalBound; }, value => { value.models[0].secret = 'unexpected'; },
@@ -141,6 +160,23 @@ for (const fault of ['missingArtifact', 'missingRoute', 'leakToken']) test(`${fa
   if (fault === 'leakToken') { assert.equal(result.report.secretExcluded, false); assert.equal(artifact.leaked, '[redacted]'); }
 });
 
+for (const missingArtifact of [false, true]) test(`CLI exit stops surviving owned processes, missing artifact=${missingArtifact}`, async t => {
+  const f = await fixture(t, { requests: 0, resistantDescendant: true, orphanExit: true, missingArtifact });
+  const pidFile = join(f.requests[0].request.session.workingDirectory, 'descendant.pid');
+  let pid;
+  t.after(() => { if (pid) { try { process.kill(pid, 'SIGKILL'); } catch {} } });
+  const result = await f.run();
+  pid = Number(await readFile(pidFile, 'utf8'));
+  assert.equal(result.report.child.stopped, 'provider_group_survived_cli_exit');
+  for (let i = 0; i < 50; i++) {
+    try { process.kill(pid, 0); } catch (error) { if (error.code === 'ESRCH') break; throw error; }
+    await new Promise(resolve => setTimeout(resolve, 20));
+  }
+  assert.throws(() => process.kill(pid, 0), { code: 'ESRCH' });
+  assert.equal(result.report.status, 'incomplete');
+  assert.equal((await summarizeControlledCampaign(f.manifest)).committedUsd, '0.014000000');
+});
+
 test('cancellation closes the proxy, bounds the owned child, and preserves unknown request holds', async t => {
   const f = await fixture(t, { requests: 1, waitForever: true });
   const abort = new AbortController(); let calls = 0;
@@ -277,3 +313,264 @@ test('cancellation still kills an owned resistant descendant after the CLI exits
   assert.equal(result.report.cleanupVerified, false);
   assert.equal((await summarizeControlledCampaign(f.manifest)).committedUsd, '0.014000000');
 });
+
+
+test('portable archives replay after worker deletion, deduplicate attempts, and hold absent cells', async t => {
+  const f = await fixture(t, { count: 2, requests: 1 });
+  await f.run(0);
+  const first = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0', environment: f.environment });
+  const partial = await summarizeControlledEvidence([first, first]);
+  assert.equal(partial.incompleteAttempts, 1);
+  assert.equal(partial.knownUsd, '0.000068000');
+  assert.equal(partial.committedUsd, '0.014068000');
+  await f.run(1);
+  const second = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-1' });
+  const expected = await summarizeControlledCampaign(f.manifest);
+  await rm(f.manifest.outputDirectory, { recursive: true });
+  const restored = await summarizeControlledEvidence([first, second, first]);
+  assert.deepEqual(restored.campaigns, [expected]);
+  assert.equal(restored.operatingHeldUsd, '0.004000000');
+  assert.equal(restored.knownUsd, '0.000136000');
+  assert.equal(restored.costStatus, 'provisional');
+});
+
+test('archived partial proof preserves known costs without consulting intact worker files', async t => {
+  const f = await fixture(t, { requests: 1 }); await f.run();
+  const bundle = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0' });
+  for (const name of ['admission.json', 'request.json', 'result.json', Object.keys(bundle.files).find(name => name.startsWith('receipts/'))]) {
+    const partial = structuredClone(bundle); delete partial.files[name];
+    const summary = await summarizeControlledEvidence([partial]);
+    assert.equal(summary.incompleteAttempts, 1, name);
+    assert.equal(summary.knownUsd, '0.000068000', name);
+    assert.equal(summary.committedUsd, '0.014000000', name);
+  }
+  const absent = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0' });
+  absent.files = {};
+  assert.equal((await summarizeControlledEvidence([absent])).committedUsd, '0.014000000');
+});
+
+test('archive creation retains never-started selections and refuses nonregular or oversized proof', async t => {
+  const f = await fixture(t);
+  const empty = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0' });
+  assert.deepEqual(empty.files, {});
+  assert.equal((await summarizeControlledEvidence([empty])).heldUsd, '0.014000000');
+  await f.run();
+  const root = attemptDirectory(f.manifest, 'attempt-0');
+  await rm(join(root, 'stderr.log'));
+  await symlink(f.manifest.runnerCli.path, join(root, 'stderr.log'));
+  await assert.rejects(packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0' }));
+  await rm(join(root, 'stderr.log'));
+  await writeFile(join(root, 'stderr.log'), Buffer.alloc(1_048_577));
+  await assert.rejects(packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0' }), /bounded regular file/);
+});
+
+test('archive validation rejects corruption, traversal, and conflicting campaign or attempt evidence', async t => {
+  const f = await fixture(t, { requests: 1 }); await f.run();
+  const bundle = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0' });
+  for (const mutate of [
+    b => { b.manifest.campaignId = 'other'; },
+    b => { b.files['ledger.jsonl'].base64 = 'x' + b.files['ledger.jsonl'].base64.slice(1); },
+    b => { b.files['../outside'] = b.files['ledger.jsonl']; },
+    b => { b.files['stderr.log'].sizeBytes = -1; },
+  ]) { const b = structuredClone(bundle); mutate(b); assert.throws(() => decodeControlledEvidence(b)); }
+  const conflict = structuredClone(bundle); delete conflict.files['stderr.log'];
+  await assert.rejects(summarizeControlledEvidence([bundle, conflict]), /Conflicting controller attempt/);
+  const other = structuredClone(bundle); other.manifest.maxUsd = '2'; other.manifestSha256 = sha256(canonical(other.manifest));
+  await assert.rejects(summarizeControlledEvidence([bundle, other]), /Conflicting controller campaign/);
+});
+
+test('configured credentials are excluded before encoding and after archive decoding', async t => {
+  const f = await fixture(t); await f.run();
+  const root = attemptDirectory(f.manifest, 'attempt-0');
+  const secret = 'synthetic-braintrust-archive-secret';
+  const environment = { BRAINTRUST_API_KEY: secret };
+  await writeFile(join(root, 'stderr.log'), secret);
+  await assert.rejects(packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0', environment }), /configured credential/);
+  const unchecked = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0', environment: {} });
+  assert(!JSON.stringify(unchecked).includes(secret));
+  assert.throws(() => decodeControlledEvidence(unchecked, { environment }), /configured credential/);
+  const file = join(f.root, 'archive.json'); await writeFile(file, JSON.stringify(unchecked));
+  await assert.rejects(readControlledEvidence(file, { environment }), /configured credential/);
+});
+
+for (const name of ['provider-artifact.json', 'receipt', 'ledger.jsonl']) for (const kind of ['fifo', 'symlink', 'oversized']) {
+  test(`post-child ${name} rejects ${kind} without blocking or releasing its allocation`, { timeout: 20_000 }, async t => {
+    const f = await fixture(t, { requests: 1, invalidEvidence: name, invalidFileType: kind });
+    const result = await f.run();
+    assert.equal(result.report.status, 'incomplete');
+    assert.equal(result.report.accounting.knownUsd, '0.000068000');
+    const summary = await summarizeControlledCampaign(f.manifest);
+    assert.equal(summary.incompleteAttempts, 1);
+    assert.equal(summary.committedUsd, '0.014000000');
+  });
+}
+
+test('durable reports bound every evidence file and retain full holds for invalid proof', { timeout: 20_000 }, async t => {
+  const f = await fixture(t, { requests: 1 }); await f.run();
+  const root = attemptDirectory(f.manifest, 'attempt-0');
+  const receipt = `receipts/${(await readdir(join(root, 'receipts')))[0]}`;
+  for (const name of ['admission.json', 'request.json', 'result.json', 'provider-artifact.json', 'gateway.json', 'ledger.jsonl', receipt]) {
+    const path = join(root, name), original = await readFile(path);
+    for (const kind of ['fifo', 'symlink', 'oversized']) {
+      await rm(path);
+      if (kind === 'fifo') execFileSync('mkfifo', [path]);
+      else if (kind === 'symlink') await symlink('/dev/zero', path);
+      else { await writeFile(path, ''); await truncate(path, 16 * 1024 * 1024 + 1); }
+      const summary = await summarizeControlledCampaign(f.manifest);
+      assert.equal(summary.incompleteAttempts, 1, `${name}:${kind}`);
+      assert.equal(summary.committedUsd, '0.014000000', `${name}:${kind}`);
+    }
+    await rm(path); await writeFile(path, original);
+  }
+  assert.equal((await summarizeControlledCampaign(f.manifest)).incompleteAttempts, 0);
+});
+
+for (const name of ['PAPERCLIP_AGENT_PRIVATE_KEY', 'PAPERCLIP_AI_PROVIDER_KEY', 'AWS_SECRET_ACCESS_KEY', 'OPENCODE_CONFIG_CONTENT']) {
+  test(`archive excludes configured ${name} before and after encoding`, async t => {
+    const f = await fixture(t); await f.run();
+    const secret = name === 'PAPERCLIP_AGENT_PRIVATE_KEY' ? '-----BEGIN PRIVATE KEY-----\nsynthetic-private-material\n-----END PRIVATE KEY-----' : 'synthetic-extra-credential';
+    const environment = { [name]: name === 'OPENCODE_CONFIG_CONTENT' ? JSON.stringify({ provider: { apiKey: secret } }) : secret };
+    const root = attemptDirectory(f.manifest, 'attempt-0');
+    await writeFile(join(root, 'stderr.log'), JSON.stringify({ retained: secret }));
+    await assert.rejects(packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0', environment }), /configured credential/);
+    const unchecked = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0', environment: {} });
+    assert.throws(() => decodeControlledEvidence(unchecked, { environment }), /configured credential/);
+  });
+}
+
+test('retention annotations preserve an already-failed controller result and its known costs', async t => {
+  const f = await fixture(t, { requests: 1, missingRoute: true }); await f.run();
+  const bundle = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0' });
+  const artifact = JSON.parse(await readFile(f.requests[0].outputPath, 'utf8'));
+  assert.equal(artifact.controllerAccounting.status, 'incomplete');
+  artifact.controllerEvidenceError = 'A later retention check failed';
+  artifact.accountingError = artifact.controllerEvidenceError;
+  const directory = join(f.root, 'runs', 'attempt-0'); await mkdir(directory, { recursive: true });
+  for (const [name, value] of Object.entries({ 'request.json': f.requests[0].request, 'artifact.json': artifact, 'controller-evidence.json': bundle })) {
+    await writeFile(join(directory, name), JSON.stringify(value));
+  }
+  const retained = await readAttemptControlledEvidence(directory);
+  const report = await summarizeControlledEvidence([retained]);
+  assert.equal(report.costStatus, 'incomplete');
+  assert.equal(report.knownUsd, '0.000068000');
+  assert.equal(report.committedUsd, '0.014000000');
+  assert.equal(JSON.parse(await readFile(join(directory, 'artifact.json'), 'utf8')).accountingError, artifact.controllerEvidenceError);
+});
+
+test('campaign collection retains and audits private archives, while public projection omits them', async t => {
+  const f = await fixture(t, { count: 2, requests: 1 }); await f.run();
+  const bundle = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0' });
+  const download = join(f.root, 'downloads', 'cell'), attempt = join(download, 'runs', 'attempt-0');
+  await mkdir(attempt, { recursive: true });
+  const evalCase = { id: 'case-0', checks: [] }, config = { model: 'fixture', provider: 'codex', driver: 'codex_app_server' };
+  for (const [name, value] of Object.entries({
+    'request.json': f.requests[0].request,
+    'artifact.json': JSON.parse(await readFile(f.requests[0].outputPath, 'utf8')),
+    'score.json': { attemptId: 'attempt-0', caseId: evalCase.id, passed: true, disposition: 'passed', checks: [] },
+    'case.json': evalCase, 'config.json': config, 'controller-evidence.json': bundle,
+  })) await writeFile(join(attempt, name), JSON.stringify(value));
+  const cell = { cellId: 'execution-0', caseId: evalCase.id, rosterFile: 'roster.json', rosterId: 'roster-0', ...config };
+  const missingCell = { ...cell, cellId: 'execution-1', caseId: 'case-1' };
+  const program = join(f.root, 'evals/paperclip-runner');
+  for (const folder of ['rosters', 'cases', 'configs']) await mkdir(join(program, folder), { recursive: true });
+  await writeFile(join(program, 'rosters/roster.json'), JSON.stringify({ config: '../configs/config.json' }));
+  await writeFile(join(program, 'cases/case-1.json'), JSON.stringify({ ...evalCase, id: 'case-1' }));
+  await writeFile(join(program, 'configs/config.json'), JSON.stringify(config));
+  const catalogPath = join(f.root, 'catalog.json');
+  await writeFile(catalogPath, JSON.stringify({ schema: 'paperclip.runner-protocol-eval.catalog/v1', campaignId: f.manifest.campaignId,
+    selection: { kind: 'subset' }, cells: [cell, missingCell], rosters: [{ ...cell, cases: [evalCase.id, missingCell.caseId] }] }));
+  await writeFile(join(download, 'cell.json'), JSON.stringify({ ...cell, exitCode: 0 }));
+  await rm(f.manifest.outputDirectory, { recursive: true });
+  const input = { catalogPath, downloadsRoot: join(f.root, 'downloads'), evalsRoot: f.root,
+    runsOut: join(f.root, 'collected'), campaignOut: join(f.root, 'campaign.json'),
+    source: { paperclip: { sha: f.manifest.sourceRevision }, evals: { sha: f.manifest.definitionsRevision } } };
+  const result = await aggregateProtocolEvalCampaign(input);
+  assert.equal(result.controlledCosts.knownUsd, '0.000068000');
+  assert.equal(result.controlledCosts.costStatus, 'incomplete');
+  assert.equal(result.controlledCosts.committedUsd, '0.014068000');
+  assert.equal(result.controlledCosts.incompleteAttempts, 1);
+  assert.deepEqual(JSON.parse(await readFile(join(input.runsOut, 'attempt-0', 'controller-evidence.json'), 'utf8')), bundle);
+  const publicRoot = join(f.root, 'public');
+  await sanitizeProtocolEvalRuns({ runsRoot: input.runsOut, publicRunsRoot: publicRoot });
+  assert.deepEqual((await readdir(join(publicRoot, 'attempt-0'))).sort(), ['artifact.json', 'case.json', 'config.json', 'score.json']);
+  assert(!(await readFile(join(publicRoot, 'attempt-0', 'artifact.json'), 'utf8')).includes('controllerAccounting'));
+  const changedRequest = { ...f.requests[0].request, prompt: 'copied from another execution' };
+  await writeFile(join(attempt, 'request.json'), JSON.stringify(changedRequest));
+  await assert.rejects(aggregateProtocolEvalCampaign(input), /Immutable attempt does not match/);
+  await writeFile(join(attempt, 'request.json'), JSON.stringify(f.requests[0].request));
+  const originalArtifact = JSON.parse(await readFile(f.requests[0].outputPath, 'utf8'));
+  for (const mutate of [
+    value => { value.usage = { estimatedCostNanodollars: 0 }; },
+    value => { value.turn = { status: 'completed', assistantText: 'changed outcome' }; },
+    value => { value.controllerAccounting.cleanupVerified = false; },
+    value => { delete value.controllerAccounting; },
+  ]) {
+    const changed = structuredClone(originalArtifact); mutate(changed);
+    await writeFile(join(attempt, 'artifact.json'), JSON.stringify(changed));
+    await assert.rejects(aggregateProtocolEvalCampaign(input), /Immutable result disagrees/);
+  }
+  await writeFile(join(attempt, 'artifact.json'), JSON.stringify({ ...originalArtifact, provenance: { runnerPackage: 'private-program-provenance' } }));
+  await aggregateProtocolEvalCampaign(input);
+  const retainedFailure = { ...originalArtifact, controllerEvidenceError: 'Retained verification was interrupted', accountingError: 'Retained verification was interrupted',
+    accountingFailure: { class: 'provider_budget_coverage_unknown', category: 'provider_budget', retryable: false } };
+  await writeFile(join(attempt, 'artifact.json'), JSON.stringify(retainedFailure));
+  await aggregateProtocolEvalCampaign(input);
+  await writeFile(join(attempt, 'artifact.json'), JSON.stringify(originalArtifact));
+  const catalog = JSON.parse(await readFile(catalogPath, 'utf8'));
+  for (const change of [{ model: 'wrong-model' }, { provider: 'acpx', acpxAgent: 'claude' }, { driver: 'other-driver' }]) {
+    const driftedCatalog = structuredClone(catalog); Object.assign(driftedCatalog.cells[0], change);
+    await writeFile(catalogPath, JSON.stringify(driftedCatalog));
+    await writeFile(join(attempt, 'config.json'), JSON.stringify({ ...config, ...change }));
+    await assert.rejects(aggregateProtocolEvalCampaign(input), /Controller archive identity drifted/);
+  }
+  await writeFile(join(attempt, 'config.json'), JSON.stringify(config));
+  for (const change of [{ model: 'wrong-unexecuted-model' }, { provider: 'acpx', acpxAgent: 'claude', driver: 'acpx_runtime' }]) {
+    const driftedCatalog = structuredClone(catalog); Object.assign(driftedCatalog.cells[1], change);
+    await writeFile(catalogPath, JSON.stringify(driftedCatalog));
+    await assert.rejects(aggregateProtocolEvalCampaign(input), /Controller archive identity drifted/);
+  }
+  await writeFile(catalogPath, JSON.stringify(catalog));
+  await rm(join(attempt, 'controller-evidence.json'));
+  await assert.rejects(aggregateProtocolEvalCampaign(input), /Missing private controller evidence/);
+});
+
+
+test('credential redaction replaces a swapped symlink without changing its target', async t => {
+  const f = await fixture(t, { leakToken: true, requests: 0 });
+  const result = await f.run();
+  assert.equal(result.report.secretExcluded, false);
+  const path = join(attemptDirectory(f.manifest, 'attempt-0'), 'provider-artifact.json');
+  const bytes = await readRegularFile(path, 16_777_216);
+  const target = join(f.root, 'untouched-target'); await writeFile(target, 'target must remain unchanged');
+  await rm(path); await symlink(target, path);
+  await replaceEvidenceFile(path, bytes);
+  assert.equal(await readFile(target, 'utf8'), 'target must remain unchanged');
+  assert.deepEqual(await readRegularFile(path, 16_777_216), bytes);
+  await symlink(target, `${path}.tmp`);
+  await assert.rejects(replaceEvidenceFile(path, bytes), { code: 'EEXIST' });
+  assert.equal(await readFile(target, 'utf8'), 'target must remain unchanged');
+});
+
+test('missing provider artifacts cannot bypass immutable result binding', async t => {
+  const f = await fixture(t, { missingArtifact: true, requests: 0 }); await f.run();
+  const bundle = await packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0' });
+  assert.equal(bundle.files['provider-artifact.json'], undefined);
+  assert.equal(JSON.parse(Buffer.from(bundle.files['result.json'].base64, 'base64')).providerArtifactSha256, undefined);
+  const directory = join(f.root, 'runs', 'attempt-0'); await mkdir(directory, { recursive: true });
+  const artifact = JSON.parse(await readFile(f.requests[0].outputPath, 'utf8'));
+  for (const [name, value] of Object.entries({ 'request.json': f.requests[0].request, 'artifact.json': artifact, 'controller-evidence.json': bundle })) await writeFile(join(directory, name), JSON.stringify(value));
+  await readAttemptControlledEvidence(directory);
+  for (const mutate of [
+    value => { value.usage = { estimatedCostNanodollars: 0 }; },
+    value => { value.turn = { status: 'completed', assistantText: 'forged success' }; },
+    value => { value.controllerAccounting.cleanupVerified = true; },
+  ]) {
+    const changed = structuredClone(artifact); mutate(changed);
+    await writeFile(join(directory, 'artifact.json'), JSON.stringify(changed));
+    await assert.rejects(readAttemptControlledEvidence(directory), /Immutable result disagrees/);
+  }
+  const malformed = structuredClone(bundle), bytes = Buffer.from('{');
+  malformed.files['result.json'] = { sizeBytes: bytes.length, sha256: sha256(bytes), base64: bytes.toString('base64') };
+  await writeFile(join(directory, 'controller-evidence.json'), JSON.stringify(malformed));
+  await assert.rejects(readAttemptControlledEvidence(directory), /Malformed immutable controller result proof/);
+});

packages/paperclip-runner/scripts/controlled-evals/evidence-files.mjs

@@ -0,0 +1,36 @@
+import { constants } from 'node:fs';
+import { open, rename } from 'node:fs/promises';
+
+const MiB = 1_048_576;
+export const FILE_LIMITS = new Map([
+  ['admission.json', MiB], ['request.json', 8 * MiB], ['gateway.json', 8192],
+  ['ledger.jsonl', 16 * MiB], ['result.json', MiB], ['provider-artifact.json', 16 * MiB],
+  ['stdout.log', MiB], ['stderr.log', MiB],
+]);
+export const RECEIPT = /^receipts\/[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}\.json$/;
+export const limitFor = name => FILE_LIMITS.get(name) ?? (RECEIPT.test(name) ? MiB : 0);
+
+export async function readRegularFile(path, maximum) {
+  const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
+  try {
+    const stat = await handle.stat();
+    if (!stat.isFile() || stat.size > maximum) throw new Error('Controller evidence is not a bounded regular file');
+    const bytes = Buffer.alloc(stat.size + 1);
+    let length = 0;
+    while (length < bytes.length) {
+      const { bytesRead } = await handle.read(bytes, length, bytes.length - length, null);
+      if (!bytesRead) break;
+      length += bytesRead;
+    }
+    if (length !== stat.size) throw new Error('Controller evidence changed while being read');
+    return bytes.subarray(0, length);
+  } finally { await handle.close(); }
+}
+
+export async function replaceEvidenceFile(path, bytes) {
+  const temporary = `${path}.tmp`;
+  const handle = await open(temporary, 'wx', 0o600);
+  try { await handle.writeFile(bytes); await handle.sync(); }
+  finally { await handle.close(); }
+  await rename(temporary, path);
+}

packages/paperclip-runner/scripts/controlled-evals/manifest.mjs

@@ -72,7 +72,7 @@ export function validateManifest(value, now = Date.now(), { admission = true } =
     const provider = attempt.provider === 'codex' && attempt.acpxAgent === undefined ? 'openai'
       : attempt.provider === 'acpx' && attempt.acpxAgent === 'claude' ? 'anthropic' : null;
     const reservation = prices.get(`${provider}:${attempt.model}`);
-    if (!provider || !id(attempt.executionId) || !id(attempt.attemptId) || !hash(attempt.requestSha256)
+    if (!provider || !id(attempt.executionId) || !id(attempt.attemptId) || !/^[A-Za-z0-9_.-]+$/.test(attempt.attemptId) || !hash(attempt.requestSha256)
         || attempts.has(attempt.attemptId) || executions.has(attempt.executionId) || reservation === undefined
         || !Number.isSafeInteger(attempt.timeoutMs) || attempt.timeoutMs < 1 || attempt.timeoutMs > 4 * 3_600_000
         || !Number.isSafeInteger(attempt.maxRequests) || attempt.maxRequests < 1 || attempt.maxRequests > 1000

packages/paperclip-runner/scripts/controlled-evals/report.mjs

@@ -1,19 +1,32 @@
-import { readFile } from 'node:fs/promises';
 import { join, resolve } from 'node:path';
 import { pathToFileURL } from 'node:url';
 import { canonical, costRuntime, readJson, requestBindingSha256, sha256, units, usd, validateManifest } from './manifest.mjs';
 import { attemptDirectory } from './run.mjs';
+import { limitFor, readRegularFile } from './evidence-files.mjs';
 
-export async function summarizeControlledCampaign(value) {
+export async function summarizeControlledCampaign(value, { filesByAttempt } = {}) {
   const { manifest, manifestSha256, allocatedUsd, overheadUsd } = validateManifest(value, Date.now(), { admission: false });
   let known = 0n, committed = units(overheadUsd);
   const attempts = [];
   for (const allocation of manifest.attempts) {
     const root = attemptDirectory(manifest, allocation.attemptId);
+    const readEvidence = async name => {
+      const limit = limitFor(name);
+      if (!limit) throw new Error('Unexpected controller evidence path');
+      if (filesByAttempt === undefined) return readRegularFile(join(root, name), limit);
+      const bytes = filesByAttempt.get(allocation.attemptId)?.get(name);
+      if (!Buffer.isBuffer(bytes) || bytes.length > limit) throw new Error('Missing archived controller evidence');
+      return bytes;
+    };
+    const readDocument = async (name, maximum = 1_048_576) => {
+      const bytes = await readEvidence(name);
+      if (bytes.length > maximum) throw new Error('Controller document exceeds its byte limit');
+      return JSON.parse(bytes.toString('utf8'));
+    };
     let snapshot, report, verified = false;
     try {
-      const ledgerBytes = await readFile(join(root, 'ledger.jsonl'));
+      const ledgerBytes = await readEvidence('ledger.jsonl');
-      const ledger = await costRuntime.readEvalCostLedger(join(root, 'ledger.jsonl'));
+      const ledger = costRuntime.parseEvalCostLedger(ledgerBytes.toString('utf8'));
       const candidateSnapshot = ledger.state.snapshot();
       const ledgerDeadline = Date.parse(ledger.state.limits.deadlineAt);
       const fixedLimits = { campaignId: manifest.campaignId, maxUsd: allocation.maxUsd, deadlineAt: ledger.state.limits.deadlineAt,
@@ -23,7 +36,7 @@ export async function summarizeControlledCampaign(value) {
           || canonical(candidateSnapshot.admittedAttempts) !== canonical([{ executionId: allocation.executionId, attemptId: allocation.attemptId }])
           || candidateSnapshot.requests.some(request => request.executionId !== allocation.executionId || request.attemptId !== allocation.attemptId || request.phase !== 'agent')) throw new Error('Ledger identity does not match its allocation');
       snapshot = candidateSnapshot;
-      const admission = await readJson(join(root, 'admission.json'));
+      const admission = await readDocument('admission.json');
       const started = Date.parse(admission.startedAt);
       const expectedLimits = { campaignId: manifest.campaignId, maxUsd: allocation.maxUsd,
         deadlineAt: new Date(Math.min(Date.parse(manifest.deadlineAt), started + allocation.timeoutMs)).toISOString(),
@@ -31,20 +44,20 @@ export async function summarizeControlledCampaign(value) {
       if (admission.schema !== 'paperclip.runner-eval.admission/v1' || admission.manifestSha256 !== manifestSha256
           || admission.campaignId !== manifest.campaignId || canonical(admission.attempt) !== canonical(allocation)
           || canonical(admission.limits) !== canonical(expectedLimits) || canonical(ledger.state.limits) !== canonical(expectedLimits)) throw new Error('Ledger does not match its admission');
-      report = await readJson(join(root, 'result.json'));
+      report = await readDocument('result.json');
-      const request = await readJson(join(root, 'request.json'), 8_388_608);
+      const request = await readDocument('request.json', 8_388_608);
       if (requestBindingSha256(request) !== allocation.requestSha256 || request.runnerd.sha256.replace(/^sha256:/, '') !== manifest.runnerd.sha256
           || canonical(Object.keys(report.receiptHashes ?? {}).sort()) !== canonical(snapshot.requests.map(item => item.requestId).sort())) throw new Error('Retained request evidence does not match');
       for (const item of snapshot.requests) {
         if (!/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/.test(item.requestId)) throw new Error('Invalid retained receipt identity');
-        const bytes = await readFile(join(root, 'receipts', `${item.requestId}.json`));
+        const bytes = await readEvidence(`receipts/${item.requestId}.json`);
         const receipt = JSON.parse(bytes.toString('utf8'));
         if (sha256(bytes) !== report.receiptHashes[item.requestId] || receipt.requestId !== item.requestId
             || receipt.route !== manifest.route || receipt.costUsd !== item.costUsd) throw new Error('Retained receipt does not match its ledger');
       }
-      const artifactBytes = await readFile(join(root, 'provider-artifact.json'));
+      const artifactBytes = await readEvidence('provider-artifact.json');
       const artifact = JSON.parse(artifactBytes.toString('utf8'));
-      const descriptorBytes = await readFile(join(root, 'gateway.json'));
+      const descriptorBytes = await readEvidence('gateway.json');
       const descriptor = JSON.parse(descriptorBytes.toString('utf8'));
       if (report.schema !== 'paperclip.runner-eval.control-result/v1' || report.manifestSha256 !== manifestSha256
           || report.campaignId !== manifest.campaignId || report.executionId !== allocation.executionId || report.attemptId !== allocation.attemptId

packages/paperclip-runner/scripts/controlled-evals/run.mjs

@@ -4,6 +4,7 @@ import { lstat, mkdir, open, readFile, realpath, rename, rm, writeFile } from 'n
 import { dirname, isAbsolute, join, relative, resolve, sep } from 'node:path';
 import { pathToFileURL } from 'node:url';
 import { parseArgs } from 'node:util';
+import { limitFor, readRegularFile, replaceEvidenceFile } from './evidence-files.mjs';
 import { bindRequest, canonical, costRuntime, readJson, sha256, validateManifest } from './manifest.mjs';
 
 async function atomicJson(file, value) {
@@ -42,17 +43,24 @@ async function runChild(input) {
   if (input.signal?.aborted) onAbort();
   try {
     const [code, signal] = await once(child, 'exit');
-    if (stopped) {
+    const groupPresent = !absent(-child.pid);
-      let groupPresent = true;
+    if (groupPresent && !stopped) stop('provider_group_survived_cli_exit');
-      try { process.kill(-child.pid, 0); } catch (error) { if (error.code === 'ESRCH') groupPresent = false; }
+    if (groupPresent) await escalation;
-      if (groupPresent) await escalation;
-    }
     return { code, signal, pid: child.pid, stopped, stdout: Buffer.concat(output.stdout).toString(), stderr: Buffer.concat(output.stderr).toString() };
   } finally {
     clearTimeout(timeout); clearTimeout(force); input.signal?.removeEventListener('abort', onAbort);
+    child.stdout.destroy(); child.stderr.destroy();
   }
 }
 
+export function controlledResultArtifact(artifact, report, attempt) {
+  return { ...(artifact ?? { schema: 'paperclip-runner/eval-session-artifact/v1', attemptId: attempt.attemptId,
+    requestedModel: attempt.model, infrastructureError: 'Controlled provider did not retain an artifact' }),
+    ...(report.failure ? { accountingError: 'Controlled request accounting or cleanup is incomplete',
+      accountingFailure: { class: 'provider_budget_coverage_unknown', category: 'provider_budget', retryable: false } } : {}),
+    controllerAccounting: report };
+}
+
 export async function runControlledEval({ manifestPath, requestPath, outputPath, environment = process.env, signal, transport }) {
   if (!['linux', 'darwin'].includes(process.platform)) throw new Error('Controlled eval process ownership requires Linux or macOS');
   signal?.throwIfAborted();
@@ -130,10 +138,9 @@ registerHooks({ resolve(specifier, context, nextResolve) {
     child = await runChild({ cli: stagedCli, loader, requestPath: retainedRequestPath, artifactPath, descriptorPath,
       cwd: request.session.workingDirectory ?? root, environment: childEnvironment, deadlineAt, signal, closeGateway: () => gateway.close() });
     for (const stream of ['stdout', 'stderr']) await writeFile(join(root, `${stream}.log`), redact(child[stream]), { flag: 'wx', mode: 0o600 });
-    const raw = await readFile(artifactPath, 'utf8');
+    const raw = (await readRegularFile(artifactPath, limitFor('provider-artifact.json'))).toString('utf8');
-    if (Buffer.byteLength(raw) > 16_777_216) throw new Error('Provider artifact exceeds its byte limit');
     secretExcluded = secrets.every(secret => !raw.includes(secret));
-    if (!secretExcluded) await writeFile(artifactPath, redact(raw), { mode: 0o600 });
+    if (!secretExcluded) await replaceEvidenceFile(artifactPath, redact(raw));
     providerArtifactSha256 = sha256(redact(raw));
     artifact = JSON.parse(redact(raw));
     const expected = { mode: 'controller_gateway', provider, declaredUpstreamRoute: manifest.route, campaignId: manifest.campaignId,
@@ -159,20 +166,18 @@ registerHooks({ resolve(specifier, context, nextResolve) {
   if (snapshot.overBudget || snapshot.conflictingRequests || snapshot.inflight || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests) failure ??= 'request_accounting_unbounded';
   const receiptHashes = {};
   try {
-    for (const request of snapshot.requests) receiptHashes[request.requestId] = sha256(await readFile(join(root, 'receipts', `${request.requestId}.json`)));
+    for (const request of snapshot.requests) receiptHashes[request.requestId] = sha256(await readRegularFile(join(root, 'receipts', `${request.requestId}.json`), limitFor(`receipts/${request.requestId}.json`)));
   } catch { failure ??= 'request_receipt_missing'; }
-  const ledgerSha256 = sha256(await readFile(ledgerPath));
+  let ledgerSha256;
+  try { ledgerSha256 = sha256(await readRegularFile(ledgerPath, limitFor('ledger.jsonl'))); }
+  catch { failure ??= 'ledger_evidence_invalid'; }
   const report = { schema: 'paperclip.runner-eval.control-result/v1', campaignId: manifest.campaignId, manifestSha256,
     executionId: attempt.executionId, attemptId: attempt.attemptId, requestSha256: attempt.requestSha256,
     allocationUsd: attempt.maxUsd, limits, startedAt, finishedAt: new Date().toISOString(), status: failure ? 'incomplete' : 'closed', failure,
     child: child ? { code: child.code, signal: child.signal, stopped: child.stopped } : null,
     cleanupVerified, secretExcluded, providerArtifactSha256, ledgerSha256, receiptHashes, accounting: snapshot };
   await atomicJson(join(root, 'result.json'), report);
-  const resultArtifact = artifact ?? { schema: 'paperclip-runner/eval-session-artifact/v1', attemptId: attempt.attemptId,
+  const resultArtifact = controlledResultArtifact(artifact, report, attempt);
-    requestedModel: attempt.model, infrastructureError: 'Controlled provider did not retain an artifact' };
-  if (failure) Object.assign(resultArtifact, { accountingError: 'Controlled request accounting or cleanup is incomplete',
-    accountingFailure: { class: 'provider_budget_coverage_unknown', category: 'provider_budget', retryable: false } });
-  resultArtifact.controllerAccounting = report;
   await mkdir(dirname(resolve(outputPath)), { recursive: true, mode: 0o700 });
   await writeFile(outputPath, JSON.stringify(resultArtifact, null, 2) + '\n', { flag: 'wx', mode: 0o600 });
   return { code: failure ? 2 : child?.code ?? 2, report };

packages/paperclip-runner/scripts/controlled-evals/shared-cost-runtime.ts

@@ -1,3 +1,3 @@
-export { EvalSpendState, openEvalCostLedger, readEvalCostLedger, validEvalLedgerId } from '../../../../tests/runner-e2e/cost-ledger.js';
+export { EvalSpendState, openEvalCostLedger, readEvalCostLedger, parseEvalCostLedger, validEvalLedgerId } from '../../../../tests/runner-e2e/cost-ledger.js';
 export { evalRequestReservation, startEvalGateway } from '../../../../tests/runner-e2e/eval-gateway.js';
 export { EVAL_OVERHEAD_COMPONENTS, validateOperatingOverhead } from '../../../../tests/runner-e2e/operating-overhead.js';

packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs

@@ -26,6 +26,7 @@ const ATTEMPT_FILES = new Set([
   "compiled-fixture.json",
   "case.json",
   "config.json",
+  "controller-evidence.json",
 ]);
 
 function json(value) {
@@ -183,6 +184,7 @@ export async function buildProtocolEvalCatalog({
       configFile: relative(programRoot, configPath).split(sep).join("/"),
       model: String(roster.model ?? config.model ?? "unknown"),
       provider: String(config.provider ?? "codex"),
+      ...(config.provider === "acpx" ? { acpxAgent: config.acpxAgent } : {}),
       driver: String(config.driver ?? "codex_app_server"),
       credentialName,
       ...(config.provider === "acpx" && config.acpxAgent === "grok"
@@ -213,6 +215,7 @@ export async function buildProtocolEvalCatalog({
       caseId,
       model: roster.model,
       provider: roster.provider,
+      ...(roster.acpxAgent ? { acpxAgent: roster.acpxAgent } : {}),
       driver: roster.driver,
       credentialName: roster.credentialName,
       ...(roster.authenticationMode ? { authenticationMode: roster.authenticationMode } : {}),
@@ -405,6 +408,7 @@ export async function aggregateProtocolEvalCampaign({
     catalog.cells.map((cell) => [cell.cellId, cell]),
   );
   const copiedAttemptIds = new Set();
+  const controllerBundles = [];
   for (const statusPath of await findFiles(downloadsRoot, "cell.json")) {
     const status = await loadObject(statusPath);
     const cellId = safeId(status.cellId, "recorded cell ID");
@@ -453,6 +457,31 @@ export async function aggregateProtocolEvalCampaign({
         ) {
           throw new Error(`Downloaded attempt identity drifted for ${cellId}`);
         }
+        const bundlePath = join(attemptRoot, entry.name, "controller-evidence.json");
+        const bundleStat = await lstat(bundlePath).catch(error => {
+          if (error.code === "ENOENT") return null;
+          throw error;
+        });
+        if (bundleStat) {
+          const { readAttemptControlledEvidence } = await import("./controlled-evals/archive.mjs");
+          const bundle = await readAttemptControlledEvidence(join(attemptRoot, entry.name));
+          const allocation = bundle.manifest.attempts.find(item => item.attemptId === attemptId);
+          const selectedExecutions = new Set(bundle.manifest.attempts.map(item => item.executionId));
+          const allocationsMatchCatalog = bundle.manifest.attempts.every(item => {
+            const cell = expectedByCell.get(item.executionId);
+            return cell && item.model === cell.model && item.provider === cell.provider
+              && item.acpxAgent === cell.acpxAgent
+              && cell.driver === (item.provider === "codex" ? "codex_app_server" : "acpx_runtime");
+          });
+          if (!allocationsMatchCatalog || bundle.manifest.campaignId !== catalog.campaignId || allocation.executionId !== cellId ||
+              selectedExecutions.size !== expectedByCell.size || [...expectedByCell.keys()].some(id => !selectedExecutions.has(id)) ||
+              bundle.manifest.sourceRevision !== source.paperclip.sha || bundle.manifest.definitionsRevision !== source.evals.sha) {
+            throw new Error(`Controller archive identity drifted for ${cellId}`);
+          }
+          controllerBundles.push(bundle);
+        } else if ("controllerAccounting" in artifact || "controllerEvidenceError" in artifact) {
+          throw new Error(`Missing private controller evidence for ${attemptId}; cost metadata alone is insufficient`);
+        }
         await copyAttempt(
           join(attemptRoot, entry.name),
           join(runsOut, attemptId),
@@ -518,6 +547,15 @@ export async function aggregateProtocolEvalCampaign({
       (result) => result.disposition === "infrastructure_failure",
     ).length,
   };
+  let controlledCosts;
+  if (controllerBundles.length) {
+    const { summarizeControlledEvidence } = await import("./controlled-evals/archive.mjs");
+    controlledCosts = await summarizeControlledEvidence(controllerBundles);
+    const allocatedAttempts = new Set(controlledCosts.campaigns.flatMap(campaign => campaign.attempts.map(attempt => attempt.attemptId)));
+    if ([...copiedAttemptIds].some(id => !allocatedAttempts.has(id))) {
+      throw new Error("Retained attempt is outside the controlled campaign allocation");
+    }
+  }
   const generatedAt = new Date().toISOString();
   const campaign = {
     schema: "paperclip.runner-protocol-eval.campaign/v1",
@@ -525,6 +563,7 @@ export async function aggregateProtocolEvalCampaign({
     generatedAt,
     selection: catalog.selection,
     costs: sumAttemptCosts(attemptUsages),
+    ...(controlledCosts ? { controlledCosts } : {}),
     source: {
       paperclip: source.paperclip,
       evals: source.evals,

tests/runner-e2e/cost-ledger.ts

@@ -222,7 +222,10 @@ export class EvalSpendState {
 }
 
 export async function readEvalCostLedger(file: string) {
-  const raw = await readFile(file, "utf8");
+  return parseEvalCostLedger(await readFile(file, "utf8"));
+}
+
+export function parseEvalCostLedger(raw: string) {
   if (!raw.endsWith("\n")) throw new Error("Eval ledger has an incomplete durable record");
   const lines = raw.trimEnd().split("\n").map(value => JSON.parse(value) as Line);
   let previous = "", state: EvalSpendState | undefined;

tests/runner-e2e/eval-gateway.test.ts

@@ -139,6 +139,9 @@ it.each([
   { type: "namespace", name: "local", tools: [{ type: "web_search" }] },
   { type: "namespace", name: "local", tools: [{ type: "namespace", name: "nested", tools: [{ type: "web_search" }] }] },
   { type: "namespace", name: "local" },
+  { type: "namespace", name: "local", tools: [{ type: "custom", name: "x", tools: [{ type: "web_search" }] }] },
+  { type: "namespace", name: "local", tools: [{ type: "function", name: "x", tools: [{ type: "web_search" }] }] },
+  { type: "custom", name: "x", tools: [{ type: "web_search" }] },
 ])("rejects unqualified tools inside namespaces before reservation", async tool => {
   const f = await fixture(), transport = vi.fn<typeof fetch>();
   await expect(forwardEvalRequest({ ...f, payload: { ...f.payload, tools: [tool] }, transport })).rejects.toThrow();

tests/runner-e2e/eval-gateway.ts

@@ -72,11 +72,11 @@ function boundedPayload(value: unknown, route: EvalGatewayRoute) {
   if (body.speed !== undefined && body.speed !== "standard" || body.inference_geo !== undefined) throw new EvalRequestError("unqualified_billing_mode");
   if (body.tools !== undefined && (!Array.isArray(body.tools) || body.tools.some(tool => {
     const definition = row(tool), type = definition.type;
-    if (route.provider !== "openai") return type !== undefined && type !== "custom";
+    if (route.provider !== "openai") return type !== undefined && type !== "custom" || Object.hasOwn(definition, "tools");
     if (type === "namespace") return typeof definition.name !== "string" || !definition.name
       || !Array.isArray(definition.tools) || !definition.tools.length
-      || definition.tools.some(nested => !["function", "custom"].includes(String(row(nested).type)));
+      || definition.tools.some(nested => !["function", "custom"].includes(String(row(nested).type)) || Object.hasOwn(row(nested), "tools"));
-    return !["function", "custom"].includes(String(type));
+    return !["function", "custom"].includes(String(type)) || Object.hasOwn(definition, "tools");
   }))) throw new EvalRequestError("unqualified_hosted_tool");
   const inspect = (item: unknown): void => {
     if (Array.isArray(item)) { item.forEach(inspect); return; }