diff --git a/packages/paperclip-runner/docs/runner-protocol-live-evals.md b/packages/paperclip-runner/docs/runner-protocol-live-evals.md index 6d3cf586e0b..0232ef8ef37 100644 --- a/packages/paperclip-runner/docs/runner-protocol-live-evals.md +++ b/packages/paperclip-runner/docs/runner-protocol-live-evals.md @@ -159,6 +159,84 @@ change the hosted matrix. The controller must enforce request admission, deadlines, and limits before forwarding traffic. A local fake-API probe only proves routing; it does not qualify live provider or Braintrust behavior. + +## Controlled request budgets + +The source-checkout controller is +`packages/paperclip-runner/scripts/controlled-evals/run.mjs`. Runner/evals owns +this orchestration. It reuses the existing eval request proxy and ledger from +`tests/runner-e2e` through one import graph; it does not copy their reservation +or provider-usage rules. It requires the workspace's installed `tsx` runtime. +It is not a published Runner package entry point. + +The controller accepts the existing `--request` and `--output` contract. Set +`PAPERCLIP_RUNNER_EVAL_CONTROL_MANIFEST` to an explicit reviewed manifest and +supply the selected upstream credential only to this trusted controller. For +Runner Evals, pass this controller path as `eval_program.py run --runner-cli`; +the manifest identifies the actual built `eval-session.js`. The provider child +receives only its temporary proxy credential and private home. + +The manifest schema is `paperclip.runner-eval.controller/v1`. It contains: + +- `campaignId`, `maxUsd`, `deadlineAt`, `pricingVerifiedAt`, and + `pricingExpiresAt`; admission must fit a current four-hour window. +- `sourceRevision` and `definitionsRevision`, each an exact commit ID. + These are the trusted preparer's provenance declarations. +- `runnerCli`, `runnerPackage`, and `runnerd`, each an absolute `path` and + content `sha256`. The controller checks the three files before launch. +- An absolute durable `outputDirectory`, outside the disposable case workspace. +- `route` (`direct` or `braintrust`), plus `projectId` for Braintrust. +- `models`, using the existing exact-model price, request-limit, and reviewed + full-lifecycle `provisionalBound` contract. +- `overhead`, using the existing four-component operating allowance with a + current full-lifecycle bound for every component. This is reserved once. +- `attempts`, each with a unique `executionId` and `attemptId`, + `requestSha256`, `provider`, optional `acpxAgent`, exact `model`, `maxUsd`, + `maxRequests`, `maxConcurrent`, and `timeoutMs`. Native Codex and ACPX + Claude are supported. Each execution has one explicit attempt and no retry. + +Use `requestBindingSha256` from `controlled-evals/manifest.mjs` to bind the +complete compiled request. It excludes only the temporary working directory +and native executable path; the executable digest remains bound and its path +is checked separately. The controller gives the child a retained copy of the +validated request. Changing prompts, fixtures, models, limits, or attempts +requires a new reviewed binding. + +The sum of every cell's allocation and the once-only operating allowance must +fit `maxUsd` before any cell starts. A request reserves against its cell's +ledger before forwarding. Cells cannot borrow unused allocations. Unknown +requests keep their reservations, and failed or missing cell evidence keeps +that cell's full allocation in the campaign report. An existing cell directory +or output is never reset for a retry. Cancellation closes admission, drains +upstream receipts within the proxy limit, and stops the owned CLI process +group. Unverified provider cleanup remains explicit and retains the allocation. + +Each result retains `controllerAccounting` alongside the original provider +outcome and usage. Retain the durable controller directory as private evidence: +its admission, request, gateway descriptor, request receipts, ledger, original +provider artifact, and final result are required for reconciliation. The +private program's existing immutable attempt copy includes the result metadata; +its current default history publisher does not archive this separate directory. +Do not infer receipt retention from that metadata alone. + +```sh +node packages/paperclip-runner/scripts/controlled-evals/report.mjs \ + /private/path/controller-manifest.json > /private/path/controller-report.json +``` + +The report recomputes ledger costs, checks their attempt/manifest bindings, and +preserves known partial charges. It reports operating costs as held and total +costs as provisional when evidence is intact, or incomplete otherwise. It does +not release historical holds or claim invoice +settlement. Existing provider usage is retained separately; do not add the +provider and controller cost observations as if they were separate charges. + +This local controller is not yet wired into the hosted matrix. Its durable +attempt-directory guard applies to one shared filesystem, not independent +GitHub workers. Hosted integration must bind unique cells to the prepared +allocation, reject workflow/job reruns, and preserve the controller evidence +before enabling paid work. Native OTLP routing, report integration, vendor +metering bounds, and actual paid qualification remain separate gates. ## Controller-assigned native observability The same CLI accepts `--observability-config /absolute/path/observability.json` @@ -455,3 +533,23 @@ checksum-verified Grok binary before packaging the portable runtime. Targets without the Grok package keep their existing build behavior. This protocol workflow uses the explicitly selected API key; subscription credentials are not delivered by it. + +The controlled Codex profile disables web search. The shared proxy accepts local +function/custom tool namespaces and still rejects hosted tools, including any +hidden inside a namespace. These routes do not qualify web-search charges. + +The controller stages the exact verified CLI and daemon bytes in its private +attempt directory. A Node module-resolution hook keeps the CLI's relative +imports pointed at the trusted source checkout; it does not reload the original +entry file. Installed dependencies still belong to that trusted checkout. The +staged copies are removed after verified process cleanup and kept if cleanup is +incomplete. This path requires the repository's Node 24.11+ runtime. + +Campaign reports verify the retained request binding and the digest of every +request receipt before reducing a cell's allocation. Missing admission, request, +or receipt evidence preserves known ledger charges and holds the rest of the +full allocation. + +On cancellation, the controller keeps the scheduled process-group SIGKILL until +any remaining owned descendants receive it, even if the CLI exits first. A +forced stop still leaves cleanup unverified and retains the full allocation. diff --git a/packages/paperclip-runner/package.json b/packages/paperclip-runner/package.json index 8080d157740..f42661d79c4 100644 --- a/packages/paperclip-runner/package.json +++ b/packages/paperclip-runner/package.json @@ -85,7 +85,7 @@ "typecheck:browser": "tsc -p tsconfig.browser.json --noEmit", "test": "pnpm run test:typescript && pnpm run test:rust", "test:typescript": "pnpm run test:typescript:prep && vitest run", - "test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node ../../scripts/ci-profile/node-test.mjs test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/provider-pack-runtime-tree.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-runtime-dependency-roots.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs scripts/provider-pack-executable-shims.test.mjs && node --test scripts/build-copilot-distribution.test.mjs scripts/copilot-inner-distribution.test.mjs scripts/copilot-smoke-resources.test.mjs scripts/discover-copilot-acp.test.mjs scripts/materialize-copilot-binary.test.mjs scripts/install-copilot-assets.test.mjs scripts/probe-copilot-acp.test.mjs scripts/qualify-copilot-acp.test.mjs test/acpx-message-boundaries-package-contract.test.mjs", + "test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node ../../scripts/ci-profile/node-test.mjs test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/provider-pack-runtime-tree.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-runtime-dependency-roots.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs scripts/provider-pack-executable-shims.test.mjs scripts/controlled-evals/controller.test.mjs && node --test scripts/build-copilot-distribution.test.mjs scripts/copilot-inner-distribution.test.mjs scripts/copilot-smoke-resources.test.mjs scripts/discover-copilot-acp.test.mjs scripts/materialize-copilot-binary.test.mjs scripts/install-copilot-assets.test.mjs scripts/probe-copilot-acp.test.mjs scripts/qualify-copilot-acp.test.mjs test/acpx-message-boundaries-package-contract.test.mjs", "test:typescript:vitest": "node ./scripts/run-pr-vitest-lane.mjs", "test:rust": "cargo test --release --manifest-path runner/Cargo.toml --locked --workspace", "test:codex": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test codex_provider", diff --git a/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs b/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs new file mode 100644 index 00000000000..a4af19aa000 --- /dev/null +++ b/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs @@ -0,0 +1,279 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; +import { mkdtemp, mkdir, readFile, readdir, rm, symlink, writeFile } from 'node:fs/promises'; +import { writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { canonical, costRuntime, requestBindingSha256, sha256, validateManifest } from './manifest.mjs'; +import { attemptDirectory, runControlledEval } from './run.mjs'; +import { summarizeControlledCampaign } from './report.mjs'; + +const childScript = options => ` +import assert from 'node:assert/strict'; +import {createHash} from 'node:crypto'; +import {spawn} from 'node:child_process'; +import {once} from 'node:events'; +import {readFile,writeFile} from 'node:fs/promises'; +const args=new Map();for(let i=2;i{});process.stdout.write('ready');setInterval(()=>{},1000)"],{stdio:['ignore','pipe','ignore']}); + await once(descendant.stdout,'data'); + await writeFile(request.session.workingDirectory+'/descendant.pid',String(descendant.pid)); +} + +const descriptorRaw=await readFile(args.get('--gateway-config'),'utf8'),descriptor=JSON.parse(descriptorRaw); +for(const key of ['OPENAI_API_KEY','ANTHROPIC_API_KEY','BRAINTRUST_API_KEY','GH_TOKEN','NODE_OPTIONS','PYTHONPATH'])assert.equal(process.env[key],undefined); +const statuses=[]; +for(let i=0;i<${options.requests ?? 3};i++){ + const response=await fetch(descriptor.baseUrl+(descriptor.provider==='openai'?'/v1/responses':'/v1/messages'),{method:'POST',headers:{authorization:'Bearer '+process.env.PAPERCLIP_RUNNER_EVAL_GATEWAY_TOKEN,'content-type':'application/json'},body:JSON.stringify({model:request.model,input:'synthetic fixture'})}); + statuses.push(response.status);await response.text(); +} +if(${Boolean(options.waitForever)})await new Promise(()=>setInterval(()=>{},1000)); +if(!${Boolean(options.missingArtifact)})await writeFile(args.get('--output'),JSON.stringify({schema:'paperclip-runner/eval-session-artifact/v1',attemptId:request.attemptId,requestedModel:request.model, + requestRouting:${options.missingRoute ? 'null' : "{mode:'controller_gateway',provider:descriptor.provider,declaredUpstreamRoute:descriptor.upstreamRoute,campaignId:descriptor.campaignId,executionId:descriptor.executionId,attemptId:descriptor.attemptId,descriptorSha256:createHash('sha256').update(descriptorRaw).digest('hex')}"}, + snapshot:{status:'closed',process:{runnerExited:true,runnerPid:process.pid,providerPid:process.pid}},statuses, + leaked:${options.leakToken ? 'process.env.PAPERCLIP_RUNNER_EVAL_GATEWAY_TOKEN' : 'null'}})); +`; + +async function fixture(t, { count = 1, allocation = '0.01', providerProfile = 'codex', route = 'direct', ...options } = {}) { + const root = await mkdtemp(join(tmpdir(), 'runner-controller-test-')); + t.after(() => rm(root, { recursive: true, force: true })); + const cli = join(root, 'fake-cli.mjs'), binary = join(root, 'runnerd'), archive = join(root, 'runner.tgz'); + await writeFile(cli, childScript(options)); await writeFile(binary, 'synthetic daemon'); await writeFile(archive, 'synthetic package'); + const provenance = async path => ({ path, sha256: sha256(await readFile(path)) }); + const now = Date.now(), verifiedAt = new Date(now - 1000).toISOString(), expiresAt = new Date(now + 86_400_000).toISOString(); + const overhead = { version: 1, reference: 'synthetic-allowance', runner: 'github-hosted', verifiedAt, expiresAt, + components: costRuntime.EVAL_OVERHEAD_COMPONENTS.map(component => ({ component, maxUsd: '0.001', reference: 'synthetic-proof' })) }; + const bound = configuration => ({ version: 1, scope: 'full_lifecycle', evidenceSha256: 'a'.repeat(64), configurationSha256: sha256(canonical(configuration)) }); + for (const component of overhead.components) component.provisionalBound = bound({ reference: overhead.reference, runner: overhead.runner, verifiedAt, expiresAt, component: { ...component } }); + const biller = providerProfile === 'claude' ? 'anthropic' : 'openai'; + const price = { provider: biller, model: 'fixture', version: 'synthetic-pricing', sourceUrl: 'https://provider.invalid/pricing', inputUsdPerMillion: '2', cachedReadUsdPerMillion: '1', outputUsdPerMillion: '10', ...(biller === 'openai' ? { cacheWrite30mUsdPerMillion: '2', cacheWrite12hUsdPerMillion: '2' } : { cacheWrite5mUsdPerMillion: '2', cacheWrite1hUsdPerMillion: '2' }) }; + const model = { provider: biller, price, maximumInputTokens: 1000, maximumOutputTokens: 100, timeoutMs: 1000 }; + model.provisionalBound = bound({ provider: model.provider, mode: route, price, maximumInputTokens: 1000, maximumOutputTokens: 100, reservation: '0.003000000' }); + const manifest = { schema: 'paperclip.runner-eval.controller/v1', campaignId: 'synthetic-controller', maxUsd: '1', + deadlineAt: new Date(now + 60_000).toISOString(), pricingVerifiedAt: verifiedAt, pricingExpiresAt: expiresAt, overhead, + sourceRevision: 'a'.repeat(40), definitionsRevision: 'b'.repeat(40), runnerCli: await provenance(cli), runnerPackage: await provenance(archive), runnerd: await provenance(binary), + outputDirectory: join(root, 'controller'), route, ...(route === 'braintrust' ? { projectId: '11111111-1111-4111-8111-111111111111' } : {}), models: [model], attempts: [] }; + const requests = []; + for (let index = 0; index < count; index++) { + const workspace = join(root, `workspace-${index}`); await mkdir(workspace); + const request = { schema: 'paperclip-runner/eval-session-request/v1', attemptId: `attempt-${index}`, prompt: 'synthetic', provider: providerProfile === 'claude' ? 'acpx' : 'codex', ...(providerProfile === 'claude' ? { acpxAgent: 'claude' } : {}), model: 'fixture', + runnerd: { path: binary, sha256: manifest.runnerd.sha256 }, limits: { turnTimeoutMs: 5000, maxAgentTurns: 1, maxEstimatedCostNanodollars: 1_000_000_000 }, session: { workingDirectory: workspace } }; + const requestPath = join(root, `request-${index}.json`), outputPath = join(root, `output-${index}.json`); + await writeFile(requestPath, JSON.stringify(request)); requests.push({ request, requestPath, outputPath }); + manifest.attempts.push({ executionId: `execution-${index}`, attemptId: request.attemptId, requestSha256: requestBindingSha256(request), provider: request.provider, ...(request.acpxAgent ? { acpxAgent: request.acpxAgent } : {}), model: 'fixture', maxUsd: allocation, maxRequests: 2, maxConcurrent: 1, timeoutMs: 5000 }); + } + const manifestPath = join(root, 'manifest.json'); + const save = () => writeFile(manifestPath, JSON.stringify(manifest)); await save(); + const environment = { PATH: process.env.PATH, OPENAI_API_KEY: 'synthetic-upstream-private', ANTHROPIC_API_KEY: 'other-private', BRAINTRUST_API_KEY: 'other-braintrust', GH_TOKEN: 'other-github', NODE_OPTIONS: '--require must-not-inherit', PYTHONPATH: '/must-not-inherit' }; + return { root, manifest, manifestPath, requests, environment, save, + run: (index = 0, transport = () => success(biller), signal) => runControlledEval({ manifestPath, ...requests[index], environment, transport, signal }) }; +} +let sequence = 0; +function success(provider = 'openai') { + if (provider === 'anthropic') return Response.json({ type: 'message', id: `synthetic-${++sequence}`, model: 'fixture', usage: { input_tokens: 8, cache_read_input_tokens: 2, cache_creation_input_tokens: 0, output_tokens: 5, cache_creation: { ephemeral_5m_input_tokens: 0, ephemeral_1h_input_tokens: 0 } } }, { headers: { 'x-bt-used-endpoint': 'ANTHROPIC_API_KEY', 'x-bt-cached': 'MISS' } }); + return Response.json({ object: 'response', id: `synthetic-${++sequence}`, status: 'completed', model: 'fixture', usage: { + input_tokens: 10, input_tokens_details: { cache_write_tokens: 0, cached_tokens: 2 }, output_tokens: 5, output_tokens_details: { reasoning_tokens: 3 } } }); +} + +test('real child requests reserve before dispatch, stop at the cap and retain only controller credentials', async t => { + const f = await fixture(t); + let calls = 0; + const result = await f.run(0, async (_url, init) => { + calls++; + assert.equal(new Headers(init.headers).get('authorization'), 'Bearer synthetic-upstream-private'); + const ledger = await costRuntime.readEvalCostLedger(join(attemptDirectory(f.manifest, 'attempt-0'), 'ledger.jsonl')); + assert.equal(ledger.state.snapshot().inflight, 1); + return success(); + }); + assert.equal(calls, 2); assert.equal(result.code, 0); assert.equal(result.report.cleanupVerified, true); + const artifact = JSON.parse(await readFile(f.requests[0].outputPath, 'utf8')); + assert.deepEqual(artifact.statuses, [200, 200, 429]); + assert.equal(artifact.controllerAccounting.accounting.knownUsd, '0.000136000'); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 0); assert.equal(summary.heldUsd, '0.004000000'); assert.equal(summary.committedUsd, '0.004136000'); + for (const name of ['admission.json', 'result.json', 'ledger.jsonl', 'provider-artifact.json', 'stdout.log', 'stderr.log']) { + assert(!(await readFile(join(attemptDirectory(f.manifest, 'attempt-0'), name), 'utf8')).includes(f.environment.OPENAI_API_KEY)); + } + await assert.rejects(f.run(), /output already exists/); + await rm(f.requests[0].outputPath); + await assert.rejects(f.run(), { code: 'EEXIST' }); + assert.equal(calls, 2); +}); + +test('disjoint concurrent cells cannot borrow an allowance and unknown requests retain their reservations', async t => { + const f = await fixture(t, { count: 2, allocation: '0.003' }); f.manifest.maxUsd = '0.01'; await f.save(); + let calls = 0; + const transport = async () => { calls++; return Response.json({ error: 'synthetic refusal' }, { status: 429 }); }; + const results = await Promise.all([f.run(0, transport), f.run(1, transport)]); + assert.equal(calls, 2); assert(results.every(result => result.report.status === 'closed')); + assert(results.every(result => result.report.accounting.heldUsd === '0.003000000')); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.committedUsd, '0.010000000'); assert.equal(summary.knownUsd, '0.000000000'); assert.equal(summary.overBudget, false); + assert(summary.attempts.every(attempt => attempt.boundedUnresolvedRequests === 1)); +}); + +test('rejects oversized or unqualified allocations, changed requests and changed executables before dispatch', async t => { + const f = await fixture(t, { count: 2 }); + for (const mutate of [ + value => { value.maxUsd = '0.01'; }, value => { value.attempts[1].executionId = value.attempts[0].executionId; }, + value => { value.attempts[0].provider = 'acpx'; value.attempts[0].acpxAgent = 'codex'; }, + value => { delete value.overhead.components[0].provisionalBound; }, value => { value.models[0].secret = 'unexpected'; }, + ]) { const value = structuredClone(f.manifest); mutate(value); assert.throws(() => validateManifest(value)); } + f.requests[0].request.prompt = 'drift'; await writeFile(f.requests[0].requestPath, JSON.stringify(f.requests[0].request)); + await assert.rejects(f.run(), /allocated identity/); + await writeFile(f.manifest.runnerCli.path, 'changed CLI'); + await assert.rejects(f.run(1), /provenance changed/); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 2); assert.equal(summary.heldUsd, '0.024000000'); +}); + +for (const fault of ['missingArtifact', 'missingRoute', 'leakToken']) test(`${fault} keeps the allocation held and preserves the attempt failure`, async t => { + const f = await fixture(t, { [fault]: true, requests: 1 }); + const result = await f.run(); + assert.equal(result.code, 2); assert.equal(result.report.status, 'incomplete'); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 1); assert.equal(summary.committedUsd, '0.014000000'); assert.equal(summary.costStatus, 'incomplete'); + assert.equal(summary.knownUsd, '0.000068000'); + const artifact = JSON.parse(await readFile(f.requests[0].outputPath, 'utf8')); + assert.equal(artifact.accountingFailure.retryable, false); + if (fault === 'leakToken') { assert.equal(result.report.secretExcluded, false); assert.equal(artifact.leaked, '[redacted]'); } +}); + +test('cancellation closes the proxy, bounds the owned child, and preserves unknown request holds', async t => { + const f = await fixture(t, { requests: 1, waitForever: true }); + const abort = new AbortController(); let calls = 0; + const result = await f.run(0, async (_url, init) => { + calls++; abort.abort(); + return new Promise((_resolve, reject) => init.signal.addEventListener('abort', () => reject(new Error('synthetic interrupted request')), { once: true })); + }, abort.signal); + assert.equal(calls, 1); assert.equal(result.code, 2); assert.equal(result.report.cleanupVerified, false); + assert.equal(result.report.accounting.unknownRequests, 1); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.committedUsd, '0.014000000'); assert.equal(summary.incompleteAttempts, 1); +}); + +test('changed receipts cannot release allocations and expired campaigns remain inspectable', async t => { + const f = await fixture(t, { requests: 1 }); await f.run(); + const root = attemptDirectory(f.manifest, 'attempt-0'); + const resultPath = join(root, 'result.json'), result = JSON.parse(await readFile(resultPath, 'utf8')); + result.accounting.knownUsd = '0.000000000'; await writeFile(resultPath, JSON.stringify(result)); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 1); assert.equal(summary.committedUsd, '0.014000000'); assert.equal(summary.knownUsd, '0.000068000'); + assert.throws(() => validateManifest(f.manifest, Date.now() + 2 * 86_400_000)); + assert.doesNotThrow(() => validateManifest(f.manifest, Date.now() + 2 * 86_400_000, { admission: false })); +}); + + +test('a lost final report preserves known ledger charges while holding the remaining cell allowance', async t => { + const f = await fixture(t, { requests: 1 }); await f.run(); + await rm(join(attemptDirectory(f.manifest, 'attempt-0'), 'result.json')); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.knownUsd, '0.000068000'); assert.equal(summary.committedUsd, '0.014000000'); + assert.equal(summary.incompleteAttempts, 1); +}); + + +test('Claude cells use the assigned Braintrust provider and keep its credential on the controller', async t => { + const f = await fixture(t, { providerProfile: 'claude', route: 'braintrust', requests: 1 }); + let calls = 0; + const result = await f.run(0, async (url, init) => { + calls++; assert.equal(url, 'https://gateway.braintrust.dev/messages'); + const headers = new Headers(init.headers); + assert.equal(headers.get('authorization'), 'Bearer other-braintrust'); + assert.equal(headers.get('x-bt-endpoint-name'), 'ANTHROPIC_API_KEY'); + assert.equal(headers.get('x-bt-project-id'), f.manifest.projectId); + return success('anthropic'); + }); + assert.equal(calls, 1); assert.equal(result.code, 0); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 0); assert.equal(summary.knownUsd, '0.000068000'); +}); + + +test('existing outputs and physical workspace aliases are rejected before provider work', async t => { + const f = await fixture(t); + let calls = 0; + const transport = async () => { calls++; return success(); }; + await writeFile(f.requests[0].outputPath, 'existing evidence'); + await assert.rejects(f.run(0, transport), /already exists/); + assert.equal(await readFile(f.requests[0].outputPath, 'utf8'), 'existing evidence'); + await rm(f.requests[0].outputPath); + f.manifest.outputDirectory = join(f.requests[0].request.session.workingDirectory, '..evidence'); await f.save(); + await assert.rejects(f.run(0, transport), /outside the disposable/); + f.manifest.outputDirectory = join(f.root, 'aliased-evidence'); + await symlink(f.requests[0].request.session.workingDirectory, f.manifest.outputDirectory); await f.save(); + await assert.rejects(f.run(0, transport), /resolves inside/); + assert.equal(calls, 0); +}); + +for (const missing of ['request.json', 'admission.json', 'receipt']) test(`lost ${missing} keeps known charges and the full allocation`, async t => { + const f = await fixture(t, { requests: 1 }); await f.run(); + const root = attemptDirectory(f.manifest, 'attempt-0'); + const file = missing === 'receipt' ? join(root, 'receipts', (await readdir(join(root, 'receipts')))[0]) : join(root, missing); + await rm(file); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.knownUsd, '0.000068000'); + assert.equal(summary.committedUsd, '0.014000000'); + assert.equal(summary.incompleteAttempts, 1); +}); + +for (const changed of ['request', 'receipt']) test(`changed ${changed} evidence cannot release the allocation`, async t => { + const f = await fixture(t, { requests: 1 }); await f.run(); + const root = attemptDirectory(f.manifest, 'attempt-0'); + const file = changed === 'receipt' ? join(root, 'receipts', (await readdir(join(root, 'receipts')))[0]) : join(root, 'request.json'); + const value = JSON.parse(await readFile(file, 'utf8')); + if (changed === 'request') value.prompt = 'different request'; else value.costUsd = '0.000000000'; + await writeFile(file, JSON.stringify(value)); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.knownUsd, '0.000068000'); + assert.equal(summary.committedUsd, '0.014000000'); + assert.equal(summary.incompleteAttempts, 1); +}); + +for (const replaced of ['runnerCli', 'runnerd']) test(`launches verified bytes after the original ${replaced} path is replaced`, async t => { + const f = await fixture(t, { requests: 1 }); + await writeFile(f.manifest.runnerCli.path, childScript({ requests: 1 }) + "\nassert.equal(await readFile(request.runnerd.path, 'utf8'), 'synthetic daemon');\n"); + f.manifest.runnerCli.sha256 = sha256(await readFile(f.manifest.runnerCli.path)); await f.save(); + Object.defineProperty(f.environment, 'OPENAI_API_KEY', { get() { + writeFileSync(f.manifest[replaced].path, replaced === 'runnerCli' ? "throw new Error('replacement must not execute');" : 'replaced daemon'); + return 'synthetic-upstream-private'; + }}); + const result = await f.run(); + assert.equal(result.code, 0); assert.equal(result.report.cleanupVerified, true); + assert.equal((await summarizeControlledCampaign(f.manifest)).incompleteAttempts, 0); + await assert.rejects(readdir(join(attemptDirectory(f.manifest, 'attempt-0'), 'executables')), { code: 'ENOENT' }); +}); + +test('staged CLI relative imports survive a symlinked evidence directory', async t => { + const f = await fixture(t, { requests: 1 }); + await writeFile(join(f.root, 'helper.mjs'), "export const marker = 'trusted sibling';\n"); + await writeFile(f.manifest.runnerCli.path, "import { marker } from './helper.mjs';\n" + childScript({ requests: 1 }) + "\nassert.equal(marker, 'trusted sibling');\n"); + f.manifest.runnerCli.sha256 = sha256(await readFile(f.manifest.runnerCli.path)); + const alias = join(f.root, 'evidence-alias'); + await mkdir(join(f.root, 'evidence-real')); await symlink(join(f.root, 'evidence-real'), alias); + f.manifest.outputDirectory = join(alias, 'controller'); await f.save(); + const result = await f.run(); + assert.equal(result.code, 0); + assert.equal((await summarizeControlledCampaign(f.manifest)).incompleteAttempts, 0); +}); + +test('cancellation still kills an owned resistant descendant after the CLI exits', async t => { + const f = await fixture(t, { requests: 1, waitForever: true, resistantDescendant: true }); + f.manifest.attempts[0].timeoutMs = 15000; await f.save(); + const abort = new AbortController(); let ready; + const started = new Promise(resolve => { ready = resolve; }); + const running = f.run(0, async () => { ready(); return success(); }, abort.signal); + await started; + const pid = Number(await readFile(join(f.requests[0].request.session.workingDirectory, 'descendant.pid'), 'utf8')); + t.after(() => { try { process.kill(pid, 'SIGKILL'); } catch {} }); + abort.abort(); + const result = await running; + assert.equal(result.report.child.stopped, 'controller_cancelled'); + const absent = () => { try { process.kill(pid, 0); return false; } catch (error) { return error.code === 'ESRCH'; } }; + for (let retry = 0; retry < 100 && !absent(); retry++) await new Promise(resolve => setTimeout(resolve, 10)); + assert.equal(absent(), true); + assert.equal(result.report.cleanupVerified, false); + assert.equal((await summarizeControlledCampaign(f.manifest)).committedUsd, '0.014000000'); +}); diff --git a/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs b/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs new file mode 100644 index 00000000000..0ca1bb655d0 --- /dev/null +++ b/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs @@ -0,0 +1,93 @@ +import { createHash } from 'node:crypto'; +import { createRequire } from 'node:module'; +import { readFile } from 'node:fs/promises'; +import { isAbsolute } from 'node:path'; + +const require = createRequire(new URL('../../../../server/package.json', import.meta.url)); +const { tsImport } = await import(require.resolve('tsx/esm/api')); +export const costRuntime = await tsImport('./shared-cost-runtime.ts', import.meta.url); +export const sha256 = bytes => createHash('sha256').update(bytes).digest('hex'); +export const canonical = value => JSON.stringify(value, (_key, item) => item && typeof item === 'object' && !Array.isArray(item) + ? Object.fromEntries(Object.keys(item).sort().map(key => [key, item[key]])) : item); +export const usd = value => `${value / 1_000_000_000n}.${String(value % 1_000_000_000n).padStart(9, '0')}`; +export function units(value) { + if (typeof value !== 'string' || !/^(?:0|[1-9][0-9]*)(?:\.[0-9]{1,9})?$/.test(value)) throw new Error('Invalid controller USD amount'); + const [whole, fraction = ''] = value.split('.'); + return BigInt(whole) * 1_000_000_000n + BigInt(fraction.padEnd(9, '0')); +} +export function fields(value, names) { + if (!value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).some(key => !names.includes(key))) throw new Error('Invalid controller fields'); +} +const hash = value => typeof value === 'string' && /^[a-f0-9]{64}$/.test(value); +const id = value => costRuntime.validEvalLedgerId(value); +export function requestBindingSha256(request) { + const copy = structuredClone(request); + if (!copy.session || !copy.runnerd) throw new Error('Invalid controlled request'); + delete copy.session.workingDirectory; + delete copy.runnerd.path; + return sha256(canonical(copy)); +} +export async function readJson(file, maximum = 1_048_576) { + const bytes = await readFile(file); + if (bytes.length > maximum) throw new Error('Controller document exceeds its byte limit'); + return JSON.parse(bytes.toString('utf8')); +} +export function validateManifest(value, now = Date.now(), { admission = true } = {}) { + fields(value, ['schema', 'campaignId', 'maxUsd', 'deadlineAt', 'pricingVerifiedAt', 'pricingExpiresAt', 'overhead', + 'sourceRevision', 'definitionsRevision', 'runnerCli', 'runnerPackage', 'runnerd', 'outputDirectory', 'route', 'projectId', 'models', 'attempts']); + const deadline = Date.parse(value.deadlineAt); + const verified = Date.parse(value.pricingVerifiedAt), expires = Date.parse(value.pricingExpiresAt); + if (value.schema !== 'paperclip.runner-eval.controller/v1' || !id(value.campaignId) + || !Number.isFinite(now) || !Number.isFinite(deadline) || admission && (deadline <= now || deadline - now > 4 * 3_600_000) + || !Number.isFinite(verified) || !Number.isFinite(expires) || verified >= deadline || expires <= verified || admission && verified > now || expires <= deadline || expires - verified > 31 * 86_400_000 + || !/^[a-f0-9]{40}$/.test(value.sourceRevision ?? '') || !/^[a-f0-9]{40}$/.test(value.definitionsRevision ?? '') + || !isAbsolute(value.outputDirectory ?? '') || !['direct', 'braintrust'].includes(value.route) + || value.route === 'braintrust' && !/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(value.projectId ?? '') + || value.route === 'direct' && value.projectId !== undefined) throw new Error('Invalid controller campaign identity or validity window'); + for (const artifact of [value.runnerCli, value.runnerPackage, value.runnerd]) { + fields(artifact, ['path', 'sha256']); + if (!isAbsolute(artifact.path ?? '') || !hash(artifact.sha256)) throw new Error('Invalid controller executable provenance'); + } + const overhead = costRuntime.validateOperatingOverhead(value.overhead, value.maxUsd); + if (admission && Date.parse(value.overhead.verifiedAt) > now || Date.parse(value.overhead.expiresAt) <= deadline + || value.overhead.components.some(component => !component.provisionalBound)) throw new Error('Controller requires current full-lifecycle operating bounds'); + if (!Array.isArray(value.models) || !value.models.length || value.models.length > 16 + || !Array.isArray(value.attempts) || !value.attempts.length || value.attempts.length > 1000) throw new Error('Invalid controlled campaign selection'); + const prices = new Map(); + const rateFields = ['inputUsdPerMillion', 'cachedReadUsdPerMillion', 'outputUsdPerMillion', 'cacheWrite5mUsdPerMillion', 'cacheWrite1hUsdPerMillion', 'cacheWrite30mUsdPerMillion', 'cacheWrite12hUsdPerMillion']; + for (const model of value.models) { + fields(model, ['provider', 'price', 'maximumInputTokens', 'maximumOutputTokens', 'timeoutMs', 'provisionalBound']); + fields(model.price, ['provider', 'model', 'version', 'sourceUrl', 'longContext', ...rateFields]); + if (model.price.longContext) { fields(model.price.longContext, ['aboveInputTokens', 'rates']); fields(model.price.longContext.rates, rateFields); } + const url = new URL(model.price.sourceUrl); + const key = `${model.provider}:${model.price.model}`; + if (!['openai', 'anthropic'].includes(model.provider) || !model.provisionalBound || url.protocol !== 'https:' + || url.username || url.password || url.search || url.hash || prices.has(key)) throw new Error('Invalid controller model pricing'); + prices.set(key, units(costRuntime.evalRequestReservation({ ...model, mode: value.route, apiKey: 'validation' }))); + } + const attempts = new Set(), executions = new Set(); + let allocated = overhead; + for (const attempt of value.attempts) { + fields(attempt, ['executionId', 'attemptId', 'requestSha256', 'provider', 'acpxAgent', 'model', 'maxUsd', 'maxRequests', 'maxConcurrent', 'timeoutMs']); + const provider = attempt.provider === 'codex' && attempt.acpxAgent === undefined ? 'openai' + : attempt.provider === 'acpx' && attempt.acpxAgent === 'claude' ? 'anthropic' : null; + const reservation = prices.get(`${provider}:${attempt.model}`); + if (!provider || !id(attempt.executionId) || !id(attempt.attemptId) || !hash(attempt.requestSha256) + || attempts.has(attempt.attemptId) || executions.has(attempt.executionId) || reservation === undefined + || !Number.isSafeInteger(attempt.timeoutMs) || attempt.timeoutMs < 1 || attempt.timeoutMs > 4 * 3_600_000 + || !Number.isSafeInteger(attempt.maxRequests) || attempt.maxRequests < 1 || attempt.maxRequests > 1000 + || !Number.isSafeInteger(attempt.maxConcurrent) || attempt.maxConcurrent < 1 || attempt.maxConcurrent > 16 + || units(attempt.maxUsd) < reservation) throw new Error('Invalid controlled attempt allocation'); + attempts.add(attempt.attemptId); executions.add(attempt.executionId); allocated += units(attempt.maxUsd); + } + if (allocated > units(value.maxUsd)) throw new Error('Controlled allocations exceed the campaign budget'); + return { manifest: structuredClone(value), manifestSha256: sha256(canonical(value)), allocatedUsd: usd(allocated), overheadUsd: usd(overhead) }; +} +export function bindRequest(manifest, request) { + const attempt = manifest.attempts.find(item => item.attemptId === request.attemptId); + if (!attempt || requestBindingSha256(request) !== attempt.requestSha256 || request.provider !== attempt.provider + || request.acpxAgent !== (attempt.acpxAgent ?? null) && request.acpxAgent !== attempt.acpxAgent + || request.model !== attempt.model || request.runnerd.sha256.replace(/^sha256:/, '') !== manifest.runnerd.sha256 + || request.runnerd.path !== manifest.runnerd.path) throw new Error('Controlled request does not match its allocated identity'); + return attempt; +} diff --git a/packages/paperclip-runner/scripts/controlled-evals/report.mjs b/packages/paperclip-runner/scripts/controlled-evals/report.mjs new file mode 100644 index 00000000000..23e4f996036 --- /dev/null +++ b/packages/paperclip-runner/scripts/controlled-evals/report.mjs @@ -0,0 +1,84 @@ +import { readFile } from 'node:fs/promises'; +import { join, resolve } from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { canonical, costRuntime, readJson, requestBindingSha256, sha256, units, usd, validateManifest } from './manifest.mjs'; +import { attemptDirectory } from './run.mjs'; + +export async function summarizeControlledCampaign(value) { + const { manifest, manifestSha256, allocatedUsd, overheadUsd } = validateManifest(value, Date.now(), { admission: false }); + let known = 0n, committed = units(overheadUsd); + const attempts = []; + for (const allocation of manifest.attempts) { + const root = attemptDirectory(manifest, allocation.attemptId); + let snapshot, report, verified = false; + try { + const ledgerBytes = await readFile(join(root, 'ledger.jsonl')); + const ledger = await costRuntime.readEvalCostLedger(join(root, 'ledger.jsonl')); + const candidateSnapshot = ledger.state.snapshot(); + const ledgerDeadline = Date.parse(ledger.state.limits.deadlineAt); + const fixedLimits = { campaignId: manifest.campaignId, maxUsd: allocation.maxUsd, deadlineAt: ledger.state.limits.deadlineAt, + maxRequests: allocation.maxRequests, maxConcurrent: allocation.maxConcurrent, maxAttemptsPerExecution: 1 }; + if (canonical(ledger.state.limits) !== canonical(fixedLimits) || !Number.isFinite(ledgerDeadline) + || ledgerDeadline <= Date.parse(manifest.pricingVerifiedAt) || ledgerDeadline > Date.parse(manifest.deadlineAt) + || canonical(candidateSnapshot.admittedAttempts) !== canonical([{ executionId: allocation.executionId, attemptId: allocation.attemptId }]) + || candidateSnapshot.requests.some(request => request.executionId !== allocation.executionId || request.attemptId !== allocation.attemptId || request.phase !== 'agent')) throw new Error('Ledger identity does not match its allocation'); + snapshot = candidateSnapshot; + const admission = await readJson(join(root, 'admission.json')); + const started = Date.parse(admission.startedAt); + const expectedLimits = { campaignId: manifest.campaignId, maxUsd: allocation.maxUsd, + deadlineAt: new Date(Math.min(Date.parse(manifest.deadlineAt), started + allocation.timeoutMs)).toISOString(), + maxRequests: allocation.maxRequests, maxConcurrent: allocation.maxConcurrent, maxAttemptsPerExecution: 1 }; + if (admission.schema !== 'paperclip.runner-eval.admission/v1' || admission.manifestSha256 !== manifestSha256 + || admission.campaignId !== manifest.campaignId || canonical(admission.attempt) !== canonical(allocation) + || canonical(admission.limits) !== canonical(expectedLimits) || canonical(ledger.state.limits) !== canonical(expectedLimits)) throw new Error('Ledger does not match its admission'); + report = await readJson(join(root, 'result.json')); + const request = await readJson(join(root, 'request.json'), 8_388_608); + if (requestBindingSha256(request) !== allocation.requestSha256 || request.runnerd.sha256.replace(/^sha256:/, '') !== manifest.runnerd.sha256 + || canonical(Object.keys(report.receiptHashes ?? {}).sort()) !== canonical(snapshot.requests.map(item => item.requestId).sort())) throw new Error('Retained request evidence does not match'); + for (const item of snapshot.requests) { + if (!/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/.test(item.requestId)) throw new Error('Invalid retained receipt identity'); + const bytes = await readFile(join(root, 'receipts', `${item.requestId}.json`)); + const receipt = JSON.parse(bytes.toString('utf8')); + if (sha256(bytes) !== report.receiptHashes[item.requestId] || receipt.requestId !== item.requestId + || receipt.route !== manifest.route || receipt.costUsd !== item.costUsd) throw new Error('Retained receipt does not match its ledger'); + } + const artifactBytes = await readFile(join(root, 'provider-artifact.json')); + const artifact = JSON.parse(artifactBytes.toString('utf8')); + const descriptorBytes = await readFile(join(root, 'gateway.json')); + const descriptor = JSON.parse(descriptorBytes.toString('utf8')); + if (report.schema !== 'paperclip.runner-eval.control-result/v1' || report.manifestSha256 !== manifestSha256 + || report.campaignId !== manifest.campaignId || report.executionId !== allocation.executionId || report.attemptId !== allocation.attemptId + || report.requestSha256 !== allocation.requestSha256 || report.ledgerSha256 !== sha256(ledgerBytes) + || report.providerArtifactSha256 !== sha256(artifactBytes) || canonical(report.accounting) !== canonical(snapshot) + || report.status !== 'closed' || report.cleanupVerified !== true || report.secretExcluded !== true || report.failure !== null + || report.startedAt !== admission.startedAt || !Number.isFinite(Date.parse(report.finishedAt)) || Date.parse(report.finishedAt) < started + || !report.child || report.child.stopped !== null || artifact.snapshot?.status !== 'closed' || artifact.snapshot?.process?.runnerExited !== true + || descriptor.schema !== 'paperclip.runner-eval.gateway/v1' || descriptor.provider !== (allocation.provider === 'codex' ? 'openai' : 'anthropic') + || descriptor.model !== allocation.model || descriptor.upstreamRoute !== manifest.route + || descriptor.campaignId !== manifest.campaignId || descriptor.executionId !== allocation.executionId || descriptor.attemptId !== allocation.attemptId + || canonical(artifact.requestRouting) !== canonical({ mode: 'controller_gateway', provider: descriptor.provider, declaredUpstreamRoute: manifest.route, + campaignId: manifest.campaignId, executionId: allocation.executionId, attemptId: allocation.attemptId, descriptorSha256: sha256(descriptorBytes) }) + || artifact.attemptId !== allocation.attemptId || artifact.requestedModel !== allocation.model + || snapshot.inflight || snapshot.conflictingRequests || snapshot.overBudget + || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests) throw new Error('Incomplete controller evidence'); + verified = true; + } catch {} + const cost = snapshot ? units(snapshot.knownUsd) : 0n; + const observedCommitment = snapshot ? units(snapshot.committedUsd) : 0n; + const commitment = verified ? observedCommitment : observedCommitment > units(allocation.maxUsd) ? observedCommitment : units(allocation.maxUsd); + known += cost; committed += commitment; + attempts.push({ executionId: allocation.executionId, attemptId: allocation.attemptId, status: verified ? 'retained' : 'incomplete', + allocationUsd: allocation.maxUsd, knownUsd: usd(cost), heldUsd: usd(commitment - cost), committedUsd: usd(commitment), + unknownRequests: snapshot?.unknownRequests ?? null, boundedUnresolvedRequests: snapshot?.boundedUnresolvedRequests ?? null }); + } + return { schema: 'paperclip.runner-eval.control-campaign/v1', campaignId: manifest.campaignId, manifestSha256, + limitUsd: manifest.maxUsd, allocatedUsd, operatingHeldUsd: overheadUsd, knownUsd: usd(known), heldUsd: usd(committed - known), + committedUsd: usd(committed), overBudget: committed > units(manifest.maxUsd), incompleteAttempts: attempts.filter(attempt => attempt.status === 'incomplete').length, + costStatus: attempts.some(attempt => attempt.status === 'incomplete') || committed > units(manifest.maxUsd) ? 'incomplete' : 'provisional', attempts }; +} +if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) { + if (process.argv.length !== 3) throw new Error('Usage: report.mjs '); + const result = await summarizeControlledCampaign(await readJson(process.argv[2])); + process.stdout.write(JSON.stringify(result, null, 2) + '\n'); + if (result.incompleteAttempts || result.overBudget) process.exitCode = 2; +} diff --git a/packages/paperclip-runner/scripts/controlled-evals/run.mjs b/packages/paperclip-runner/scripts/controlled-evals/run.mjs new file mode 100644 index 00000000000..42fda084873 --- /dev/null +++ b/packages/paperclip-runner/scripts/controlled-evals/run.mjs @@ -0,0 +1,193 @@ +import { spawn } from 'node:child_process'; +import { once } from 'node:events'; +import { lstat, mkdir, open, readFile, realpath, rename, rm, writeFile } from 'node:fs/promises'; +import { dirname, isAbsolute, join, relative, resolve, sep } from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { parseArgs } from 'node:util'; +import { bindRequest, canonical, costRuntime, readJson, sha256, validateManifest } from './manifest.mjs'; + +async function atomicJson(file, value) { + const temporary = `${file}.tmp`; + const handle = await open(temporary, 'wx', 0o600); + try { await handle.writeFile(JSON.stringify(value, null, 2) + '\n'); await handle.sync(); } + finally { await handle.close(); } + await rename(temporary, file); +} +const absent = pid => { try { process.kill(pid, 0); return false; } catch (error) { return error.code === 'ESRCH'; } }; +export const attemptDirectory = (manifest, attemptId) => join(manifest.outputDirectory, sha256(attemptId)); + +async function runChild(input) { + const child = spawn(process.execPath, ['--import', input.loader, input.cli, '--request', input.requestPath, '--output', input.artifactPath, '--gateway-config', input.descriptorPath], { + cwd: input.cwd, env: input.environment, detached: true, stdio: ['ignore', 'pipe', 'pipe'], + }); + let stopped = null, force, escalation; + const output = { stdout: [], stderr: [] }, sizes = { stdout: 0, stderr: 0 }; + const stop = reason => { + if (stopped) return; + stopped = reason; + void input.closeGateway().catch(() => {}); + try { process.kill(-child.pid, 'SIGTERM'); } catch {} + escalation = new Promise(resolve => { + force = setTimeout(() => { try { process.kill(-child.pid, 'SIGKILL'); } catch {} finally { resolve(); } }, 5000); + }); + }; + for (const stream of ['stdout', 'stderr']) child[stream].on('data', bytes => { + const available = Math.max(0, 1_048_576 - sizes[stream]); + output[stream].push(bytes.subarray(0, available)); sizes[stream] += bytes.length; + if (sizes[stream] > 1_048_576) stop('output_limit'); + }); + const onAbort = () => stop('controller_cancelled'); + input.signal?.addEventListener('abort', onAbort, { once: true }); + const timeout = setTimeout(() => stop('attempt_deadline'), Math.max(1, input.deadlineAt - Date.now())); + if (input.signal?.aborted) onAbort(); + try { + const [code, signal] = await once(child, 'exit'); + if (stopped) { + let groupPresent = true; + try { process.kill(-child.pid, 0); } catch (error) { if (error.code === 'ESRCH') groupPresent = false; } + if (groupPresent) await escalation; + } + return { code, signal, pid: child.pid, stopped, stdout: Buffer.concat(output.stdout).toString(), stderr: Buffer.concat(output.stderr).toString() }; + } finally { + clearTimeout(timeout); clearTimeout(force); input.signal?.removeEventListener('abort', onAbort); + } +} + +export async function runControlledEval({ manifestPath, requestPath, outputPath, environment = process.env, signal, transport }) { + if (!['linux', 'darwin'].includes(process.platform)) throw new Error('Controlled eval process ownership requires Linux or macOS'); + signal?.throwIfAborted(); + const { manifest, manifestSha256 } = validateManifest(await readJson(manifestPath)); + const request = await readJson(requestPath, 8_388_608), attempt = bindRequest(manifest, request); + try { await lstat(outputPath); throw new Error('Controlled output already exists'); } + catch (error) { if (error.code !== 'ENOENT') throw error; } + if (request.session.workingDirectory) { + const suffix = relative(resolve(request.session.workingDirectory), resolve(manifest.outputDirectory)); + if (!suffix || suffix !== '..' && !suffix.startsWith(`..${sep}`) && !isAbsolute(suffix)) throw new Error('Controller evidence must be outside the disposable workspace'); + } + const originalCliUrl = pathToFileURL(await realpath(manifest.runnerCli.path)).href; + const verified = {}; + for (const name of ['runnerCli', 'runnerPackage', 'runnerd']) { + const artifact = manifest[name], bytes = await readFile(artifact.path); + if (sha256(bytes) !== artifact.sha256) throw new Error('Controlled eval executable provenance changed'); + if (name !== 'runnerPackage') verified[name] = bytes; + } + const provider = attempt.provider === 'codex' ? 'openai' : 'anthropic'; + const keyName = manifest.route === 'braintrust' ? 'BRAINTRUST_API_KEY' : provider === 'openai' ? 'OPENAI_API_KEY' : 'ANTHROPIC_API_KEY'; + const key = environment[keyName]; + if (!key || /[\r\n]/.test(key)) throw new Error('Missing controlled eval upstream credential'); + const model = manifest.models.find(item => item.provider === provider && item.price.model === attempt.model); + const root = attemptDirectory(manifest, attempt.attemptId); + await mkdir(manifest.outputDirectory, { recursive: true, mode: 0o700 }); + if (request.session.workingDirectory) { + const suffix = relative(await realpath(request.session.workingDirectory), await realpath(manifest.outputDirectory)); + if (!suffix || suffix !== '..' && !suffix.startsWith(`..${sep}`) && !isAbsolute(suffix)) throw new Error('Controller evidence resolves inside the disposable workspace'); + } + await mkdir(root, { mode: 0o700 }); + const started = Date.now(), startedAt = new Date(started).toISOString(); + const deadlineAt = Math.min(Date.parse(manifest.deadlineAt), started + attempt.timeoutMs); + const limits = { campaignId: manifest.campaignId, maxUsd: attempt.maxUsd, deadlineAt: new Date(deadlineAt).toISOString(), + maxRequests: attempt.maxRequests, maxConcurrent: attempt.maxConcurrent, maxAttemptsPerExecution: 1 }; + await atomicJson(join(root, 'admission.json'), { schema: 'paperclip.runner-eval.admission/v1', campaignId: manifest.campaignId, + manifestSha256, attempt, limits, startedAt }); + const executables = join(root, 'executables'); + await mkdir(executables, { mode: 0o700 }); + const stagedCli = join(executables, 'entry.mjs'), stagedRunnerd = join(executables, 'runnerd'), loader = join(executables, 'imports.mjs'); + await writeFile(stagedCli, verified.runnerCli, { flag: 'wx', mode: 0o400 }); + await writeFile(stagedRunnerd, verified.runnerd, { flag: 'wx', mode: 0o500 }); + const stagedUrl = JSON.stringify(pathToFileURL(await realpath(stagedCli)).href), originalUrl = JSON.stringify(originalCliUrl); + await writeFile(loader, `import { registerHooks } from 'node:module'; +registerHooks({ resolve(specifier, context, nextResolve) { + return nextResolve(specifier, context.parentURL === ${stagedUrl} ? { ...context, parentURL: ${originalUrl} } : context); +} });\n`, { flag: 'wx', mode: 0o400 }); + const retainedRequestPath = join(root, 'request.json'); + const childRequest = structuredClone(request); childRequest.runnerd.path = stagedRunnerd; + await writeFile(retainedRequestPath, JSON.stringify(childRequest) + '\n', { flag: 'wx', mode: 0o600 }); + await mkdir(join(root, 'receipts'), { mode: 0o700 }); + await mkdir(join(root, 'home'), { mode: 0o700 }); + await mkdir(join(root, 'tmp'), { mode: 0o700 }); + const ledgerPath = join(root, 'ledger.jsonl'); + const ledger = await costRuntime.openEvalCostLedger(ledgerPath, limits); + let gateway, child, artifact, failure = null, cleanupVerified = false, secretExcluded = true; + let providerArtifactSha256; + const secrets = [key]; + const redact = raw => secrets.reduce((text, secret) => text.split(secret).join('[redacted]'), raw); + try { + await ledger.admitAttempt(attempt.executionId, attempt.attemptId); + gateway = await costRuntime.startEvalGateway({ ledger, identity: { executionId: attempt.executionId, attemptId: attempt.attemptId, phase: 'agent' }, + route: { ...model, mode: manifest.route, projectId: manifest.projectId, endpointName: provider === 'openai' ? 'OPENAI_API_KEY' : 'ANTHROPIC_API_KEY', apiKey: key }, + deadlineAt, transport, persist: receipt => atomicJson(join(root, 'receipts', `${receipt.requestId}.json`), receipt), + }); + secrets.push(gateway.token); + const descriptor = { schema: 'paperclip.runner-eval.gateway/v1', provider, model: attempt.model, baseUrl: gateway.baseUrl, + campaignId: manifest.campaignId, executionId: attempt.executionId, attemptId: attempt.attemptId, upstreamRoute: manifest.route }; + const descriptorPath = join(root, 'gateway.json'), descriptorRaw = JSON.stringify(descriptor) + '\n'; + await writeFile(descriptorPath, descriptorRaw, { flag: 'wx', mode: 0o600 }); + const childEnvironment = Object.fromEntries(['PATH', 'LANG', 'LANGUAGE', 'LC_ALL', 'TZ'].flatMap(name => environment[name] === undefined ? [] : [[name, environment[name]]])); + Object.assign(childEnvironment, { HOME: join(root, 'home'), TMPDIR: join(root, 'tmp'), TMP: join(root, 'tmp'), TEMP: join(root, 'tmp'), PAPERCLIP_RUNNER_EVAL_GATEWAY_TOKEN: gateway.token }); + const artifactPath = join(root, 'provider-artifact.json'); + signal?.throwIfAborted(); + if (Date.now() >= deadlineAt) throw new Error('Controlled attempt deadline reached before spawn'); + child = await runChild({ cli: stagedCli, loader, requestPath: retainedRequestPath, artifactPath, descriptorPath, + cwd: request.session.workingDirectory ?? root, environment: childEnvironment, deadlineAt, signal, closeGateway: () => gateway.close() }); + for (const stream of ['stdout', 'stderr']) await writeFile(join(root, `${stream}.log`), redact(child[stream]), { flag: 'wx', mode: 0o600 }); + const raw = await readFile(artifactPath, 'utf8'); + if (Buffer.byteLength(raw) > 16_777_216) throw new Error('Provider artifact exceeds its byte limit'); + secretExcluded = secrets.every(secret => !raw.includes(secret)); + if (!secretExcluded) await writeFile(artifactPath, redact(raw), { mode: 0o600 }); + providerArtifactSha256 = sha256(redact(raw)); + artifact = JSON.parse(redact(raw)); + const expected = { mode: 'controller_gateway', provider, declaredUpstreamRoute: manifest.route, campaignId: manifest.campaignId, + executionId: attempt.executionId, attemptId: attempt.attemptId, descriptorSha256: sha256(descriptorRaw) }; + if (canonical(artifact.requestRouting) !== canonical(expected) || artifact.attemptId !== attempt.attemptId || artifact.requestedModel !== attempt.model) throw new Error('Provider did not retain the assigned route'); + const processInfo = artifact.snapshot?.process; + const pids = [...new Set([child.pid, processInfo?.runnerPid, processInfo?.providerPid, processInfo?.sidecarPid, processInfo?.agentPid].filter(pid => Number.isSafeInteger(pid) && pid > 0))]; + cleanupVerified = !child.stopped && artifact.snapshot?.status === 'closed' && processInfo?.runnerExited === true + && Number.isSafeInteger(processInfo.runnerPid) && processInfo.runnerPid > 0 + && Number.isSafeInteger(processInfo.providerPid) && processInfo.providerPid > 0 && pids.every(absent); + if (!cleanupVerified || !secretExcluded) throw new Error('Controlled provider cleanup or credential exclusion is unverified'); + } catch { + failure = 'controlled_attempt_incomplete'; + } finally { + try { await gateway?.close(); } catch { failure = 'gateway_cleanup_unverified'; } + try { await ledger.close(); } catch { failure = 'ledger_close_failed'; } + } + if (cleanupVerified) { + try { await rm(executables, { recursive: true }); } + catch { cleanupVerified = false; failure ??= 'executable_cleanup_unverified'; } + } + const snapshot = ledger.snapshot(); + if (snapshot.overBudget || snapshot.conflictingRequests || snapshot.inflight || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests) failure ??= 'request_accounting_unbounded'; + const receiptHashes = {}; + try { + for (const request of snapshot.requests) receiptHashes[request.requestId] = sha256(await readFile(join(root, 'receipts', `${request.requestId}.json`))); + } catch { failure ??= 'request_receipt_missing'; } + const ledgerSha256 = sha256(await readFile(ledgerPath)); + const report = { schema: 'paperclip.runner-eval.control-result/v1', campaignId: manifest.campaignId, manifestSha256, + executionId: attempt.executionId, attemptId: attempt.attemptId, requestSha256: attempt.requestSha256, + allocationUsd: attempt.maxUsd, limits, startedAt, finishedAt: new Date().toISOString(), status: failure ? 'incomplete' : 'closed', failure, + child: child ? { code: child.code, signal: child.signal, stopped: child.stopped } : null, + cleanupVerified, secretExcluded, providerArtifactSha256, ledgerSha256, receiptHashes, accounting: snapshot }; + await atomicJson(join(root, 'result.json'), report); + const resultArtifact = artifact ?? { schema: 'paperclip-runner/eval-session-artifact/v1', attemptId: attempt.attemptId, + requestedModel: attempt.model, infrastructureError: 'Controlled provider did not retain an artifact' }; + if (failure) Object.assign(resultArtifact, { accountingError: 'Controlled request accounting or cleanup is incomplete', + accountingFailure: { class: 'provider_budget_coverage_unknown', category: 'provider_budget', retryable: false } }); + resultArtifact.controllerAccounting = report; + await mkdir(dirname(resolve(outputPath)), { recursive: true, mode: 0o700 }); + await writeFile(outputPath, JSON.stringify(resultArtifact, null, 2) + '\n', { flag: 'wx', mode: 0o600 }); + return { code: failure ? 2 : child?.code ?? 2, report }; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) { + const controller = new AbortController(); + const cancel = () => controller.abort(); + process.once('SIGINT', cancel); process.once('SIGTERM', cancel); + try { + const { values } = parseArgs({ options: { request: { type: 'string' }, output: { type: 'string' } } }); + if (!values.request || !values.output || !process.env.PAPERCLIP_RUNNER_EVAL_CONTROL_MANIFEST) throw new Error('A controlled manifest, request and output are required'); + const result = await runControlledEval({ manifestPath: process.env.PAPERCLIP_RUNNER_EVAL_CONTROL_MANIFEST, + requestPath: values.request, outputPath: values.output, signal: controller.signal }); + process.exitCode = result.code; + } catch { process.stderr.write('Controlled Runner eval admission or retention failed; inspect retained controller evidence.\n'); process.exitCode = 2; } + finally { process.removeListener('SIGINT', cancel); process.removeListener('SIGTERM', cancel); } +} diff --git a/packages/paperclip-runner/scripts/controlled-evals/shared-cost-runtime.ts b/packages/paperclip-runner/scripts/controlled-evals/shared-cost-runtime.ts new file mode 100644 index 00000000000..38f45e0839a --- /dev/null +++ b/packages/paperclip-runner/scripts/controlled-evals/shared-cost-runtime.ts @@ -0,0 +1,3 @@ +export { EvalSpendState, openEvalCostLedger, readEvalCostLedger, validEvalLedgerId } from '../../../../tests/runner-e2e/cost-ledger.js'; +export { evalRequestReservation, startEvalGateway } from '../../../../tests/runner-e2e/eval-gateway.js'; +export { EVAL_OVERHEAD_COMPONENTS, validateOperatingOverhead } from '../../../../tests/runner-e2e/operating-overhead.js'; diff --git a/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts b/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts index cd1cb1e03e1..0ef09abd57f 100644 --- a/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts +++ b/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts @@ -73,6 +73,7 @@ it.each([["codex", undefined], ["acpx", "claude"]])("routes %s/%s with only its expect(config).toContain('env_key = "PAPERCLIP_AI_PROVIDER_KEY"'); expect(config).toContain("requires_openai_auth = false"); expect(config).toContain("shell_snapshot = false"); + expect(config).toContain('web_search = "disabled"'); expect(config).not.toContain(token); expect((await stat(join(env.CODEX_HOME!, "config.toml"))).mode & 0o777).toBe(0o600); } diff --git a/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts b/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts index 0d32670c3a6..a4e27f9c180 100644 --- a/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts +++ b/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts @@ -70,6 +70,7 @@ export async function prepareEvalGatewayRuntime( environment.PAPERCLIP_AI_PROVIDER_KEY = token; await writeFile(join(codexHome, "config.toml"), [ 'model_provider = "paperclip_eval"', + 'web_search = "disabled"', `model = ${JSON.stringify(request.model)}`, '[model_providers.paperclip_eval]', 'name = "Paperclip eval controller"', diff --git a/tests/runner-e2e/eval-gateway.test.ts b/tests/runner-e2e/eval-gateway.test.ts index 6e38d356bea..aefca15a9e3 100644 --- a/tests/runner-e2e/eval-gateway.test.ts +++ b/tests/runner-e2e/eval-gateway.test.ts @@ -121,6 +121,29 @@ it.each([ await expect(forwardEvalRequest({ ...f, payload, transport })).rejects.toThrow(); expect(transport).not.toHaveBeenCalled(); expect(f.ledger.snapshot().requests).toHaveLength(0); }); +it("admits local function and custom tools inside a Codex namespace", async () => { + const f = await fixture(); + const tools = [{ type: "namespace", name: "local", tools: [ + { type: "function", name: "read_task", parameters: { type: "object" } }, + { type: "custom", name: "apply_patch" }, + ] }]; + const transport = vi.fn(async (_url, init) => { + expect(JSON.parse(init!.body as string).tools).toEqual(tools); + return response(); + }); + const result = await forwardEvalRequest({ ...f, payload: { ...f.payload, tools }, transport }); + expect(result.outcome).toBe("succeeded"); + expect(transport).toHaveBeenCalledOnce(); +}); +it.each([ + { type: "namespace", name: "local", tools: [{ type: "web_search" }] }, + { type: "namespace", name: "local", tools: [{ type: "namespace", name: "nested", tools: [{ type: "web_search" }] }] }, + { type: "namespace", name: "local" }, +])("rejects unqualified tools inside namespaces before reservation", async tool => { + const f = await fixture(), transport = vi.fn(); + await expect(forwardEvalRequest({ ...f, payload: { ...f.payload, tools: [tool] }, transport })).rejects.toThrow(); + expect(transport).not.toHaveBeenCalled(); expect(f.ledger.snapshot().requests).toHaveLength(0); +}); it("does not dispatch when the private pre-request receipt cannot be persisted", async () => { const f = await fixture(), transport = vi.fn(); await expect(forwardEvalRequest({ ...f, persist: async () => { throw new Error("disk unavailable"); }, transport })).rejects.toThrow("disk unavailable"); diff --git a/tests/runner-e2e/eval-gateway.ts b/tests/runner-e2e/eval-gateway.ts index ecd9b55ede8..d7cc20ab0db 100644 --- a/tests/runner-e2e/eval-gateway.ts +++ b/tests/runner-e2e/eval-gateway.ts @@ -71,8 +71,12 @@ function boundedPayload(value: unknown, route: EvalGatewayRoute) { if (body.background === true || body.mcp_servers !== undefined || body.service_tier !== undefined && !["auto", "default", "standard_only"].includes(String(body.service_tier))) throw new EvalRequestError("unqualified_billing_mode"); if (body.speed !== undefined && body.speed !== "standard" || body.inference_geo !== undefined) throw new EvalRequestError("unqualified_billing_mode"); if (body.tools !== undefined && (!Array.isArray(body.tools) || body.tools.some(tool => { - const type = row(tool).type; - return route.provider === "openai" ? !["function", "custom"].includes(String(type)) : type !== undefined && type !== "custom"; + const definition = row(tool), type = definition.type; + if (route.provider !== "openai") return type !== undefined && type !== "custom"; + if (type === "namespace") return typeof definition.name !== "string" || !definition.name + || !Array.isArray(definition.tools) || !definition.tools.length + || definition.tools.some(nested => !["function", "custom"].includes(String(row(nested).type))); + return !["function", "custom"].includes(String(type)); }))) throw new EvalRequestError("unqualified_hosted_tool"); const inspect = (item: unknown): void => { if (Array.isArray(item)) { item.forEach(inspect); return; }