mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-27 11:20:42 +01:00
feat(evals): enforce fixed Runner request allocations (#17452)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner evals execute real provider harnesses against maintained fixtures. > - Turn-level usage arrives after requests have already spent money. > - Parallel cells need distinct allowances before requests start. > - Missing output must preserve the remaining allowance and any known charges. > - This pull request adds a controller that reuses the existing request ledger and gateway. > - It makes a bounded local Runner attempt reviewable before hosted scheduling is enabled. ## Linked Issues or Issue Description Refs #17440. That PR added the explicit Runner gateway route. This PR owns request admission and retained campaign accounting for that route. The existing hosted matrix is not enabled or changed here. Its global allocation, rerun rejection, and private evidence archive remain separate integration work. ## What Changed - Validate a trusted manifest with exact request, CLI, package, daemon, model, and source identities. Assign each selected cell a distinct allowance. Count operating overhead once. Reject allocations above the campaign limit before launch. - Run native Codex or ACPX Claude through the shared request proxy and append-only ledger. Pass only the proxy credential to the child. Refuse local attempt reuse and preserve existing output files. - Close gateway admission on cancellation. Retain the ledger, receipts, route descriptor, child result, and cleanup evidence outside the disposable workspace. - Recompute campaign accounting from retained ledgers. Missing or invalid final evidence holds the full cell allowance while preserving known charges. Reports distinguish provisional and incomplete costs. - Accept namespaces containing only local function/custom tools. Disable web search in the controlled Codex profile. Continue to reject hosted tools, including tools hidden inside a namespace. - Add controller tests to the existing Runner static test path and document this source-checkout entry point. ## Verification - 20 controller tests pass. They use real child processes, local HTTP, and the actual ledger. Cases cover reservations before dispatch, concurrent cell isolation, caps, cancellation, replay rejection, lost artifacts, changed receipts, and Braintrust Claude routing. Upstream responses are synthetic. - 52 shared gateway tests and 55 Runner CLI tests pass. The namespace acceptance and web-search configuration regressions failed before the repair and pass afterward. - Runner TypeScript/Rust checks, Runner build, shared E2E typecheck, and `git diff --check` pass. - The first actual-harness probe found that Codex advertised both a local tool namespace and web search. It reached no upstream transport. That failure is retained; the compatibility repair does not allow paid hosted tools. - An earlier test failure showed that separate dynamic imports created different admission-error classes. The shared runtime bridge now loads one module graph. Another regression showed that a missing final report hid known ledger charges; the collector now preserves those charges. - The repaired actual-harness probe passed its control checks for native Codex and ACPX Claude: each reached exactly one synthetic upstream request after the ledger reservation; both retained the unresolved hold and route evidence; both recorded clean process exit. A scan of 41 retained files found no synthetic upstream credential. Deliberate provider rejection preserved the failed Codex turn and Claude timeout; these are not semantic passes. Prices, operating bounds, and the package archive identity in this probe were synthetic. - Review regressions reproduced lost request/receipt evidence, lost admission metadata, executable replacement after verification, and a SIGTERM-resistant descendant surviving the CLI. The repairs now pass. Executable bytes are staged before launch; a trusted Node resolution hook preserves relative imports. A symlink-path regression also failed before its repair and now passes. The final staged native probe repeated both one-request checks successfully. - Full current-head [CI](https://github.com/paperclipai/paperclip/actions/runs/40115932485) passed on `4781487215116afd10a415d4db53b1ec8e004dd5`: all 58 check runs and the legacy status are successful, neutral, or skipped. Greptile and Paperclip both reviewed this commit at 5/5. All five finding threads are fixed and resolved. No paid provider request ran during this change. ## Risks - This is a trusted source-checkout controller, not a published package command. It needs the repository's installed shared cost dependencies. - The manifest declares reviewed prices and full-lifecycle bounds. Synthetic tests do not qualify real prices, invoices, storage, or Braintrust metering. - The attempt-directory guard is local. Hosted fanout still needs unique global allocations, rerun rejection, and durable private retention. Do not enable the existing matrix on this change alone. - Forced child termination can leave cleanup unverified. That attempt remains incomplete and holds its full allocation. These checks do not claim a host security boundary against an adversarial target. - The current private publisher does not archive the separate controller directory. Native trace relay, private report integration, and live qualification remain open. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. This runtime does not expose the exact model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
4b89eec644
commit
41c42895fa
11 files changed
+782
-3
No files matched your search
@@ -159,6 +159,84 @@ change the hosted matrix. The controller must enforce request admission,
|
||||
deadlines, and limits before forwarding traffic. A local fake-API probe only
|
||||
proves routing; it does not qualify live provider or Braintrust behavior.
|
||||
|
||||
|
||||
## Controlled request budgets
|
||||
|
||||
The source-checkout controller is
|
||||
`packages/paperclip-runner/scripts/controlled-evals/run.mjs`. Runner/evals owns
|
||||
this orchestration. It reuses the existing eval request proxy and ledger from
|
||||
`tests/runner-e2e` through one import graph; it does not copy their reservation
|
||||
or provider-usage rules. It requires the workspace's installed `tsx` runtime.
|
||||
It is not a published Runner package entry point.
|
||||
|
||||
The controller accepts the existing `--request` and `--output` contract. Set
|
||||
`PAPERCLIP_RUNNER_EVAL_CONTROL_MANIFEST` to an explicit reviewed manifest and
|
||||
supply the selected upstream credential only to this trusted controller. For
|
||||
Runner Evals, pass this controller path as `eval_program.py run --runner-cli`;
|
||||
the manifest identifies the actual built `eval-session.js`. The provider child
|
||||
receives only its temporary proxy credential and private home.
|
||||
|
||||
The manifest schema is `paperclip.runner-eval.controller/v1`. It contains:
|
||||
|
||||
- `campaignId`, `maxUsd`, `deadlineAt`, `pricingVerifiedAt`, and
|
||||
`pricingExpiresAt`; admission must fit a current four-hour window.
|
||||
- `sourceRevision` and `definitionsRevision`, each an exact commit ID.
|
||||
These are the trusted preparer's provenance declarations.
|
||||
- `runnerCli`, `runnerPackage`, and `runnerd`, each an absolute `path` and
|
||||
content `sha256`. The controller checks the three files before launch.
|
||||
- An absolute durable `outputDirectory`, outside the disposable case workspace.
|
||||
- `route` (`direct` or `braintrust`), plus `projectId` for Braintrust.
|
||||
- `models`, using the existing exact-model price, request-limit, and reviewed
|
||||
full-lifecycle `provisionalBound` contract.
|
||||
- `overhead`, using the existing four-component operating allowance with a
|
||||
current full-lifecycle bound for every component. This is reserved once.
|
||||
- `attempts`, each with a unique `executionId` and `attemptId`,
|
||||
`requestSha256`, `provider`, optional `acpxAgent`, exact `model`, `maxUsd`,
|
||||
`maxRequests`, `maxConcurrent`, and `timeoutMs`. Native Codex and ACPX
|
||||
Claude are supported. Each execution has one explicit attempt and no retry.
|
||||
|
||||
Use `requestBindingSha256` from `controlled-evals/manifest.mjs` to bind the
|
||||
complete compiled request. It excludes only the temporary working directory
|
||||
and native executable path; the executable digest remains bound and its path
|
||||
is checked separately. The controller gives the child a retained copy of the
|
||||
validated request. Changing prompts, fixtures, models, limits, or attempts
|
||||
requires a new reviewed binding.
|
||||
|
||||
The sum of every cell's allocation and the once-only operating allowance must
|
||||
fit `maxUsd` before any cell starts. A request reserves against its cell's
|
||||
ledger before forwarding. Cells cannot borrow unused allocations. Unknown
|
||||
requests keep their reservations, and failed or missing cell evidence keeps
|
||||
that cell's full allocation in the campaign report. An existing cell directory
|
||||
or output is never reset for a retry. Cancellation closes admission, drains
|
||||
upstream receipts within the proxy limit, and stops the owned CLI process
|
||||
group. Unverified provider cleanup remains explicit and retains the allocation.
|
||||
|
||||
Each result retains `controllerAccounting` alongside the original provider
|
||||
outcome and usage. Retain the durable controller directory as private evidence:
|
||||
its admission, request, gateway descriptor, request receipts, ledger, original
|
||||
provider artifact, and final result are required for reconciliation. The
|
||||
private program's existing immutable attempt copy includes the result metadata;
|
||||
its current default history publisher does not archive this separate directory.
|
||||
Do not infer receipt retention from that metadata alone.
|
||||
|
||||
```sh
|
||||
node packages/paperclip-runner/scripts/controlled-evals/report.mjs \
|
||||
/private/path/controller-manifest.json > /private/path/controller-report.json
|
||||
```
|
||||
|
||||
The report recomputes ledger costs, checks their attempt/manifest bindings, and
|
||||
preserves known partial charges. It reports operating costs as held and total
|
||||
costs as provisional when evidence is intact, or incomplete otherwise. It does
|
||||
not release historical holds or claim invoice
|
||||
settlement. Existing provider usage is retained separately; do not add the
|
||||
provider and controller cost observations as if they were separate charges.
|
||||
|
||||
This local controller is not yet wired into the hosted matrix. Its durable
|
||||
attempt-directory guard applies to one shared filesystem, not independent
|
||||
GitHub workers. Hosted integration must bind unique cells to the prepared
|
||||
allocation, reject workflow/job reruns, and preserve the controller evidence
|
||||
before enabling paid work. Native OTLP routing, report integration, vendor
|
||||
metering bounds, and actual paid qualification remain separate gates.
|
||||
## Controller-assigned native observability
|
||||
|
||||
The same CLI accepts `--observability-config /absolute/path/observability.json`
|
||||
@@ -455,3 +533,23 @@ checksum-verified Grok binary before packaging the portable runtime. Targets
|
||||
without the Grok package keep their existing build behavior. This protocol
|
||||
workflow uses the explicitly selected API key; subscription credentials are
|
||||
not delivered by it.
|
||||
|
||||
The controlled Codex profile disables web search. The shared proxy accepts local
|
||||
function/custom tool namespaces and still rejects hosted tools, including any
|
||||
hidden inside a namespace. These routes do not qualify web-search charges.
|
||||
|
||||
The controller stages the exact verified CLI and daemon bytes in its private
|
||||
attempt directory. A Node module-resolution hook keeps the CLI's relative
|
||||
imports pointed at the trusted source checkout; it does not reload the original
|
||||
entry file. Installed dependencies still belong to that trusted checkout. The
|
||||
staged copies are removed after verified process cleanup and kept if cleanup is
|
||||
incomplete. This path requires the repository's Node 24.11+ runtime.
|
||||
|
||||
Campaign reports verify the retained request binding and the digest of every
|
||||
request receipt before reducing a cell's allocation. Missing admission, request,
|
||||
or receipt evidence preserves known ledger charges and holds the rest of the
|
||||
full allocation.
|
||||
|
||||
On cancellation, the controller keeps the scheduled process-group SIGKILL until
|
||||
any remaining owned descendants receive it, even if the CLI exits first. A
|
||||
forced stop still leaves cleanup unverified and retains the full allocation.
|
||||
@@ -85,7 +85,7 @@
|
||||
"typecheck:browser": "tsc -p tsconfig.browser.json --noEmit",
|
||||
"test": "pnpm run test:typescript && pnpm run test:rust",
|
||||
"test:typescript": "pnpm run test:typescript:prep && vitest run",
|
||||
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node ../../scripts/ci-profile/node-test.mjs test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/provider-pack-runtime-tree.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-runtime-dependency-roots.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs scripts/provider-pack-executable-shims.test.mjs && node --test scripts/build-copilot-distribution.test.mjs scripts/copilot-inner-distribution.test.mjs scripts/copilot-smoke-resources.test.mjs scripts/discover-copilot-acp.test.mjs scripts/materialize-copilot-binary.test.mjs scripts/install-copilot-assets.test.mjs scripts/probe-copilot-acp.test.mjs scripts/qualify-copilot-acp.test.mjs test/acpx-message-boundaries-package-contract.test.mjs",
|
||||
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node ../../scripts/ci-profile/node-test.mjs test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/provider-pack-runtime-tree.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-runtime-dependency-roots.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs scripts/provider-pack-executable-shims.test.mjs scripts/controlled-evals/controller.test.mjs && node --test scripts/build-copilot-distribution.test.mjs scripts/copilot-inner-distribution.test.mjs scripts/copilot-smoke-resources.test.mjs scripts/discover-copilot-acp.test.mjs scripts/materialize-copilot-binary.test.mjs scripts/install-copilot-assets.test.mjs scripts/probe-copilot-acp.test.mjs scripts/qualify-copilot-acp.test.mjs test/acpx-message-boundaries-package-contract.test.mjs",
|
||||
"test:typescript:vitest": "node ./scripts/run-pr-vitest-lane.mjs",
|
||||
"test:rust": "cargo test --release --manifest-path runner/Cargo.toml --locked --workspace",
|
||||
"test:codex": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test codex_provider",
|
||||
|
||||
@@ -0,0 +1,279 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { test } from 'node:test';
|
||||
import { mkdtemp, mkdir, readFile, readdir, rm, symlink, writeFile } from 'node:fs/promises';
|
||||
import { writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { canonical, costRuntime, requestBindingSha256, sha256, validateManifest } from './manifest.mjs';
|
||||
import { attemptDirectory, runControlledEval } from './run.mjs';
|
||||
import { summarizeControlledCampaign } from './report.mjs';
|
||||
|
||||
const childScript = options => `
|
||||
import assert from 'node:assert/strict';
|
||||
import {createHash} from 'node:crypto';
|
||||
import {spawn} from 'node:child_process';
|
||||
import {once} from 'node:events';
|
||||
import {readFile,writeFile} from 'node:fs/promises';
|
||||
const args=new Map();for(let i=2;i<process.argv.length;i+=2)args.set(process.argv[i],process.argv[i+1]);
|
||||
const request=JSON.parse(await readFile(args.get('--request'),'utf8'));
|
||||
if(${Boolean(options.resistantDescendant)}) {
|
||||
const descendant=spawn(process.execPath,['-e',"process.on('SIGTERM',()=>{});process.stdout.write('ready');setInterval(()=>{},1000)"],{stdio:['ignore','pipe','ignore']});
|
||||
await once(descendant.stdout,'data');
|
||||
await writeFile(request.session.workingDirectory+'/descendant.pid',String(descendant.pid));
|
||||
}
|
||||
|
||||
const descriptorRaw=await readFile(args.get('--gateway-config'),'utf8'),descriptor=JSON.parse(descriptorRaw);
|
||||
for(const key of ['OPENAI_API_KEY','ANTHROPIC_API_KEY','BRAINTRUST_API_KEY','GH_TOKEN','NODE_OPTIONS','PYTHONPATH'])assert.equal(process.env[key],undefined);
|
||||
const statuses=[];
|
||||
for(let i=0;i<${options.requests ?? 3};i++){
|
||||
const response=await fetch(descriptor.baseUrl+(descriptor.provider==='openai'?'/v1/responses':'/v1/messages'),{method:'POST',headers:{authorization:'Bearer '+process.env.PAPERCLIP_RUNNER_EVAL_GATEWAY_TOKEN,'content-type':'application/json'},body:JSON.stringify({model:request.model,input:'synthetic fixture'})});
|
||||
statuses.push(response.status);await response.text();
|
||||
}
|
||||
if(${Boolean(options.waitForever)})await new Promise(()=>setInterval(()=>{},1000));
|
||||
if(!${Boolean(options.missingArtifact)})await writeFile(args.get('--output'),JSON.stringify({schema:'paperclip-runner/eval-session-artifact/v1',attemptId:request.attemptId,requestedModel:request.model,
|
||||
requestRouting:${options.missingRoute ? 'null' : "{mode:'controller_gateway',provider:descriptor.provider,declaredUpstreamRoute:descriptor.upstreamRoute,campaignId:descriptor.campaignId,executionId:descriptor.executionId,attemptId:descriptor.attemptId,descriptorSha256:createHash('sha256').update(descriptorRaw).digest('hex')}"},
|
||||
snapshot:{status:'closed',process:{runnerExited:true,runnerPid:process.pid,providerPid:process.pid}},statuses,
|
||||
leaked:${options.leakToken ? 'process.env.PAPERCLIP_RUNNER_EVAL_GATEWAY_TOKEN' : 'null'}}));
|
||||
`;
|
||||
|
||||
async function fixture(t, { count = 1, allocation = '0.01', providerProfile = 'codex', route = 'direct', ...options } = {}) {
|
||||
const root = await mkdtemp(join(tmpdir(), 'runner-controller-test-'));
|
||||
t.after(() => rm(root, { recursive: true, force: true }));
|
||||
const cli = join(root, 'fake-cli.mjs'), binary = join(root, 'runnerd'), archive = join(root, 'runner.tgz');
|
||||
await writeFile(cli, childScript(options)); await writeFile(binary, 'synthetic daemon'); await writeFile(archive, 'synthetic package');
|
||||
const provenance = async path => ({ path, sha256: sha256(await readFile(path)) });
|
||||
const now = Date.now(), verifiedAt = new Date(now - 1000).toISOString(), expiresAt = new Date(now + 86_400_000).toISOString();
|
||||
const overhead = { version: 1, reference: 'synthetic-allowance', runner: 'github-hosted', verifiedAt, expiresAt,
|
||||
components: costRuntime.EVAL_OVERHEAD_COMPONENTS.map(component => ({ component, maxUsd: '0.001', reference: 'synthetic-proof' })) };
|
||||
const bound = configuration => ({ version: 1, scope: 'full_lifecycle', evidenceSha256: 'a'.repeat(64), configurationSha256: sha256(canonical(configuration)) });
|
||||
for (const component of overhead.components) component.provisionalBound = bound({ reference: overhead.reference, runner: overhead.runner, verifiedAt, expiresAt, component: { ...component } });
|
||||
const biller = providerProfile === 'claude' ? 'anthropic' : 'openai';
|
||||
const price = { provider: biller, model: 'fixture', version: 'synthetic-pricing', sourceUrl: 'https://provider.invalid/pricing', inputUsdPerMillion: '2', cachedReadUsdPerMillion: '1', outputUsdPerMillion: '10', ...(biller === 'openai' ? { cacheWrite30mUsdPerMillion: '2', cacheWrite12hUsdPerMillion: '2' } : { cacheWrite5mUsdPerMillion: '2', cacheWrite1hUsdPerMillion: '2' }) };
|
||||
const model = { provider: biller, price, maximumInputTokens: 1000, maximumOutputTokens: 100, timeoutMs: 1000 };
|
||||
model.provisionalBound = bound({ provider: model.provider, mode: route, price, maximumInputTokens: 1000, maximumOutputTokens: 100, reservation: '0.003000000' });
|
||||
const manifest = { schema: 'paperclip.runner-eval.controller/v1', campaignId: 'synthetic-controller', maxUsd: '1',
|
||||
deadlineAt: new Date(now + 60_000).toISOString(), pricingVerifiedAt: verifiedAt, pricingExpiresAt: expiresAt, overhead,
|
||||
sourceRevision: 'a'.repeat(40), definitionsRevision: 'b'.repeat(40), runnerCli: await provenance(cli), runnerPackage: await provenance(archive), runnerd: await provenance(binary),
|
||||
outputDirectory: join(root, 'controller'), route, ...(route === 'braintrust' ? { projectId: '11111111-1111-4111-8111-111111111111' } : {}), models: [model], attempts: [] };
|
||||
const requests = [];
|
||||
for (let index = 0; index < count; index++) {
|
||||
const workspace = join(root, `workspace-${index}`); await mkdir(workspace);
|
||||
const request = { schema: 'paperclip-runner/eval-session-request/v1', attemptId: `attempt-${index}`, prompt: 'synthetic', provider: providerProfile === 'claude' ? 'acpx' : 'codex', ...(providerProfile === 'claude' ? { acpxAgent: 'claude' } : {}), model: 'fixture',
|
||||
runnerd: { path: binary, sha256: manifest.runnerd.sha256 }, limits: { turnTimeoutMs: 5000, maxAgentTurns: 1, maxEstimatedCostNanodollars: 1_000_000_000 }, session: { workingDirectory: workspace } };
|
||||
const requestPath = join(root, `request-${index}.json`), outputPath = join(root, `output-${index}.json`);
|
||||
await writeFile(requestPath, JSON.stringify(request)); requests.push({ request, requestPath, outputPath });
|
||||
manifest.attempts.push({ executionId: `execution-${index}`, attemptId: request.attemptId, requestSha256: requestBindingSha256(request), provider: request.provider, ...(request.acpxAgent ? { acpxAgent: request.acpxAgent } : {}), model: 'fixture', maxUsd: allocation, maxRequests: 2, maxConcurrent: 1, timeoutMs: 5000 });
|
||||
}
|
||||
const manifestPath = join(root, 'manifest.json');
|
||||
const save = () => writeFile(manifestPath, JSON.stringify(manifest)); await save();
|
||||
const environment = { PATH: process.env.PATH, OPENAI_API_KEY: 'synthetic-upstream-private', ANTHROPIC_API_KEY: 'other-private', BRAINTRUST_API_KEY: 'other-braintrust', GH_TOKEN: 'other-github', NODE_OPTIONS: '--require must-not-inherit', PYTHONPATH: '/must-not-inherit' };
|
||||
return { root, manifest, manifestPath, requests, environment, save,
|
||||
run: (index = 0, transport = () => success(biller), signal) => runControlledEval({ manifestPath, ...requests[index], environment, transport, signal }) };
|
||||
}
|
||||
let sequence = 0;
|
||||
function success(provider = 'openai') {
|
||||
if (provider === 'anthropic') return Response.json({ type: 'message', id: `synthetic-${++sequence}`, model: 'fixture', usage: { input_tokens: 8, cache_read_input_tokens: 2, cache_creation_input_tokens: 0, output_tokens: 5, cache_creation: { ephemeral_5m_input_tokens: 0, ephemeral_1h_input_tokens: 0 } } }, { headers: { 'x-bt-used-endpoint': 'ANTHROPIC_API_KEY', 'x-bt-cached': 'MISS' } });
|
||||
return Response.json({ object: 'response', id: `synthetic-${++sequence}`, status: 'completed', model: 'fixture', usage: {
|
||||
input_tokens: 10, input_tokens_details: { cache_write_tokens: 0, cached_tokens: 2 }, output_tokens: 5, output_tokens_details: { reasoning_tokens: 3 } } });
|
||||
}
|
||||
|
||||
test('real child requests reserve before dispatch, stop at the cap and retain only controller credentials', async t => {
|
||||
const f = await fixture(t);
|
||||
let calls = 0;
|
||||
const result = await f.run(0, async (_url, init) => {
|
||||
calls++;
|
||||
assert.equal(new Headers(init.headers).get('authorization'), 'Bearer synthetic-upstream-private');
|
||||
const ledger = await costRuntime.readEvalCostLedger(join(attemptDirectory(f.manifest, 'attempt-0'), 'ledger.jsonl'));
|
||||
assert.equal(ledger.state.snapshot().inflight, 1);
|
||||
return success();
|
||||
});
|
||||
assert.equal(calls, 2); assert.equal(result.code, 0); assert.equal(result.report.cleanupVerified, true);
|
||||
const artifact = JSON.parse(await readFile(f.requests[0].outputPath, 'utf8'));
|
||||
assert.deepEqual(artifact.statuses, [200, 200, 429]);
|
||||
assert.equal(artifact.controllerAccounting.accounting.knownUsd, '0.000136000');
|
||||
const summary = await summarizeControlledCampaign(f.manifest);
|
||||
assert.equal(summary.incompleteAttempts, 0); assert.equal(summary.heldUsd, '0.004000000'); assert.equal(summary.committedUsd, '0.004136000');
|
||||
for (const name of ['admission.json', 'result.json', 'ledger.jsonl', 'provider-artifact.json', 'stdout.log', 'stderr.log']) {
|
||||
assert(!(await readFile(join(attemptDirectory(f.manifest, 'attempt-0'), name), 'utf8')).includes(f.environment.OPENAI_API_KEY));
|
||||
}
|
||||
await assert.rejects(f.run(), /output already exists/);
|
||||
await rm(f.requests[0].outputPath);
|
||||
await assert.rejects(f.run(), { code: 'EEXIST' });
|
||||
assert.equal(calls, 2);
|
||||
});
|
||||
|
||||
test('disjoint concurrent cells cannot borrow an allowance and unknown requests retain their reservations', async t => {
|
||||
const f = await fixture(t, { count: 2, allocation: '0.003' }); f.manifest.maxUsd = '0.01'; await f.save();
|
||||
let calls = 0;
|
||||
const transport = async () => { calls++; return Response.json({ error: 'synthetic refusal' }, { status: 429 }); };
|
||||
const results = await Promise.all([f.run(0, transport), f.run(1, transport)]);
|
||||
assert.equal(calls, 2); assert(results.every(result => result.report.status === 'closed'));
|
||||
assert(results.every(result => result.report.accounting.heldUsd === '0.003000000'));
|
||||
const summary = await summarizeControlledCampaign(f.manifest);
|
||||
assert.equal(summary.committedUsd, '0.010000000'); assert.equal(summary.knownUsd, '0.000000000'); assert.equal(summary.overBudget, false);
|
||||
assert(summary.attempts.every(attempt => attempt.boundedUnresolvedRequests === 1));
|
||||
});
|
||||
|
||||
test('rejects oversized or unqualified allocations, changed requests and changed executables before dispatch', async t => {
|
||||
const f = await fixture(t, { count: 2 });
|
||||
for (const mutate of [
|
||||
value => { value.maxUsd = '0.01'; }, value => { value.attempts[1].executionId = value.attempts[0].executionId; },
|
||||
value => { value.attempts[0].provider = 'acpx'; value.attempts[0].acpxAgent = 'codex'; },
|
||||
value => { delete value.overhead.components[0].provisionalBound; }, value => { value.models[0].secret = 'unexpected'; },
|
||||
]) { const value = structuredClone(f.manifest); mutate(value); assert.throws(() => validateManifest(value)); }
|
||||
f.requests[0].request.prompt = 'drift'; await writeFile(f.requests[0].requestPath, JSON.stringify(f.requests[0].request));
|
||||
await assert.rejects(f.run(), /allocated identity/);
|
||||
await writeFile(f.manifest.runnerCli.path, 'changed CLI');
|
||||
await assert.rejects(f.run(1), /provenance changed/);
|
||||
const summary = await summarizeControlledCampaign(f.manifest);
|
||||
assert.equal(summary.incompleteAttempts, 2); assert.equal(summary.heldUsd, '0.024000000');
|
||||
});
|
||||
|
||||
for (const fault of ['missingArtifact', 'missingRoute', 'leakToken']) test(`${fault} keeps the allocation held and preserves the attempt failure`, async t => {
|
||||
const f = await fixture(t, { [fault]: true, requests: 1 });
|
||||
const result = await f.run();
|
||||
assert.equal(result.code, 2); assert.equal(result.report.status, 'incomplete');
|
||||
const summary = await summarizeControlledCampaign(f.manifest);
|
||||
assert.equal(summary.incompleteAttempts, 1); assert.equal(summary.committedUsd, '0.014000000'); assert.equal(summary.costStatus, 'incomplete');
|
||||
assert.equal(summary.knownUsd, '0.000068000');
|
||||
const artifact = JSON.parse(await readFile(f.requests[0].outputPath, 'utf8'));
|
||||
assert.equal(artifact.accountingFailure.retryable, false);
|
||||
if (fault === 'leakToken') { assert.equal(result.report.secretExcluded, false); assert.equal(artifact.leaked, '[redacted]'); }
|
||||
});
|
||||
|
||||
test('cancellation closes the proxy, bounds the owned child, and preserves unknown request holds', async t => {
|
||||
const f = await fixture(t, { requests: 1, waitForever: true });
|
||||
const abort = new AbortController(); let calls = 0;
|
||||
const result = await f.run(0, async (_url, init) => {
|
||||
calls++; abort.abort();
|
||||
return new Promise((_resolve, reject) => init.signal.addEventListener('abort', () => reject(new Error('synthetic interrupted request')), { once: true }));
|
||||
}, abort.signal);
|
||||
assert.equal(calls, 1); assert.equal(result.code, 2); assert.equal(result.report.cleanupVerified, false);
|
||||
assert.equal(result.report.accounting.unknownRequests, 1);
|
||||
const summary = await summarizeControlledCampaign(f.manifest);
|
||||
assert.equal(summary.committedUsd, '0.014000000'); assert.equal(summary.incompleteAttempts, 1);
|
||||
});
|
||||
|
||||
test('changed receipts cannot release allocations and expired campaigns remain inspectable', async t => {
|
||||
const f = await fixture(t, { requests: 1 }); await f.run();
|
||||
const root = attemptDirectory(f.manifest, 'attempt-0');
|
||||
const resultPath = join(root, 'result.json'), result = JSON.parse(await readFile(resultPath, 'utf8'));
|
||||
result.accounting.knownUsd = '0.000000000'; await writeFile(resultPath, JSON.stringify(result));
|
||||
const summary = await summarizeControlledCampaign(f.manifest);
|
||||
assert.equal(summary.incompleteAttempts, 1); assert.equal(summary.committedUsd, '0.014000000'); assert.equal(summary.knownUsd, '0.000068000');
|
||||
assert.throws(() => validateManifest(f.manifest, Date.now() + 2 * 86_400_000));
|
||||
assert.doesNotThrow(() => validateManifest(f.manifest, Date.now() + 2 * 86_400_000, { admission: false }));
|
||||
});
|
||||
|
||||
|
||||
test('a lost final report preserves known ledger charges while holding the remaining cell allowance', async t => {
|
||||
const f = await fixture(t, { requests: 1 }); await f.run();
|
||||
await rm(join(attemptDirectory(f.manifest, 'attempt-0'), 'result.json'));
|
||||
const summary = await summarizeControlledCampaign(f.manifest);
|
||||
assert.equal(summary.knownUsd, '0.000068000'); assert.equal(summary.committedUsd, '0.014000000');
|
||||
assert.equal(summary.incompleteAttempts, 1);
|
||||
});
|
||||
|
||||
|
||||
test('Claude cells use the assigned Braintrust provider and keep its credential on the controller', async t => {
|
||||
const f = await fixture(t, { providerProfile: 'claude', route: 'braintrust', requests: 1 });
|
||||
let calls = 0;
|
||||
const result = await f.run(0, async (url, init) => {
|
||||
calls++; assert.equal(url, 'https://gateway.braintrust.dev/messages');
|
||||
const headers = new Headers(init.headers);
|
||||
assert.equal(headers.get('authorization'), 'Bearer other-braintrust');
|
||||
assert.equal(headers.get('x-bt-endpoint-name'), 'ANTHROPIC_API_KEY');
|
||||
assert.equal(headers.get('x-bt-project-id'), f.manifest.projectId);
|
||||
return success('anthropic');
|
||||
});
|
||||
assert.equal(calls, 1); assert.equal(result.code, 0);
|
||||
const summary = await summarizeControlledCampaign(f.manifest);
|
||||
assert.equal(summary.incompleteAttempts, 0); assert.equal(summary.knownUsd, '0.000068000');
|
||||
});
|
||||
|
||||
|
||||
test('existing outputs and physical workspace aliases are rejected before provider work', async t => {
|
||||
const f = await fixture(t);
|
||||
let calls = 0;
|
||||
const transport = async () => { calls++; return success(); };
|
||||
await writeFile(f.requests[0].outputPath, 'existing evidence');
|
||||
await assert.rejects(f.run(0, transport), /already exists/);
|
||||
assert.equal(await readFile(f.requests[0].outputPath, 'utf8'), 'existing evidence');
|
||||
await rm(f.requests[0].outputPath);
|
||||
f.manifest.outputDirectory = join(f.requests[0].request.session.workingDirectory, '..evidence'); await f.save();
|
||||
await assert.rejects(f.run(0, transport), /outside the disposable/);
|
||||
f.manifest.outputDirectory = join(f.root, 'aliased-evidence');
|
||||
await symlink(f.requests[0].request.session.workingDirectory, f.manifest.outputDirectory); await f.save();
|
||||
await assert.rejects(f.run(0, transport), /resolves inside/);
|
||||
assert.equal(calls, 0);
|
||||
});
|
||||
|
||||
for (const missing of ['request.json', 'admission.json', 'receipt']) test(`lost ${missing} keeps known charges and the full allocation`, async t => {
|
||||
const f = await fixture(t, { requests: 1 }); await f.run();
|
||||
const root = attemptDirectory(f.manifest, 'attempt-0');
|
||||
const file = missing === 'receipt' ? join(root, 'receipts', (await readdir(join(root, 'receipts')))[0]) : join(root, missing);
|
||||
await rm(file);
|
||||
const summary = await summarizeControlledCampaign(f.manifest);
|
||||
assert.equal(summary.knownUsd, '0.000068000');
|
||||
assert.equal(summary.committedUsd, '0.014000000');
|
||||
assert.equal(summary.incompleteAttempts, 1);
|
||||
});
|
||||
|
||||
for (const changed of ['request', 'receipt']) test(`changed ${changed} evidence cannot release the allocation`, async t => {
|
||||
const f = await fixture(t, { requests: 1 }); await f.run();
|
||||
const root = attemptDirectory(f.manifest, 'attempt-0');
|
||||
const file = changed === 'receipt' ? join(root, 'receipts', (await readdir(join(root, 'receipts')))[0]) : join(root, 'request.json');
|
||||
const value = JSON.parse(await readFile(file, 'utf8'));
|
||||
if (changed === 'request') value.prompt = 'different request'; else value.costUsd = '0.000000000';
|
||||
await writeFile(file, JSON.stringify(value));
|
||||
const summary = await summarizeControlledCampaign(f.manifest);
|
||||
assert.equal(summary.knownUsd, '0.000068000');
|
||||
assert.equal(summary.committedUsd, '0.014000000');
|
||||
assert.equal(summary.incompleteAttempts, 1);
|
||||
});
|
||||
|
||||
for (const replaced of ['runnerCli', 'runnerd']) test(`launches verified bytes after the original ${replaced} path is replaced`, async t => {
|
||||
const f = await fixture(t, { requests: 1 });
|
||||
await writeFile(f.manifest.runnerCli.path, childScript({ requests: 1 }) + "\nassert.equal(await readFile(request.runnerd.path, 'utf8'), 'synthetic daemon');\n");
|
||||
f.manifest.runnerCli.sha256 = sha256(await readFile(f.manifest.runnerCli.path)); await f.save();
|
||||
Object.defineProperty(f.environment, 'OPENAI_API_KEY', { get() {
|
||||
writeFileSync(f.manifest[replaced].path, replaced === 'runnerCli' ? "throw new Error('replacement must not execute');" : 'replaced daemon');
|
||||
return 'synthetic-upstream-private';
|
||||
}});
|
||||
const result = await f.run();
|
||||
assert.equal(result.code, 0); assert.equal(result.report.cleanupVerified, true);
|
||||
assert.equal((await summarizeControlledCampaign(f.manifest)).incompleteAttempts, 0);
|
||||
await assert.rejects(readdir(join(attemptDirectory(f.manifest, 'attempt-0'), 'executables')), { code: 'ENOENT' });
|
||||
});
|
||||
|
||||
test('staged CLI relative imports survive a symlinked evidence directory', async t => {
|
||||
const f = await fixture(t, { requests: 1 });
|
||||
await writeFile(join(f.root, 'helper.mjs'), "export const marker = 'trusted sibling';\n");
|
||||
await writeFile(f.manifest.runnerCli.path, "import { marker } from './helper.mjs';\n" + childScript({ requests: 1 }) + "\nassert.equal(marker, 'trusted sibling');\n");
|
||||
f.manifest.runnerCli.sha256 = sha256(await readFile(f.manifest.runnerCli.path));
|
||||
const alias = join(f.root, 'evidence-alias');
|
||||
await mkdir(join(f.root, 'evidence-real')); await symlink(join(f.root, 'evidence-real'), alias);
|
||||
f.manifest.outputDirectory = join(alias, 'controller'); await f.save();
|
||||
const result = await f.run();
|
||||
assert.equal(result.code, 0);
|
||||
assert.equal((await summarizeControlledCampaign(f.manifest)).incompleteAttempts, 0);
|
||||
});
|
||||
|
||||
test('cancellation still kills an owned resistant descendant after the CLI exits', async t => {
|
||||
const f = await fixture(t, { requests: 1, waitForever: true, resistantDescendant: true });
|
||||
f.manifest.attempts[0].timeoutMs = 15000; await f.save();
|
||||
const abort = new AbortController(); let ready;
|
||||
const started = new Promise(resolve => { ready = resolve; });
|
||||
const running = f.run(0, async () => { ready(); return success(); }, abort.signal);
|
||||
await started;
|
||||
const pid = Number(await readFile(join(f.requests[0].request.session.workingDirectory, 'descendant.pid'), 'utf8'));
|
||||
t.after(() => { try { process.kill(pid, 'SIGKILL'); } catch {} });
|
||||
abort.abort();
|
||||
const result = await running;
|
||||
assert.equal(result.report.child.stopped, 'controller_cancelled');
|
||||
const absent = () => { try { process.kill(pid, 0); return false; } catch (error) { return error.code === 'ESRCH'; } };
|
||||
for (let retry = 0; retry < 100 && !absent(); retry++) await new Promise(resolve => setTimeout(resolve, 10));
|
||||
assert.equal(absent(), true);
|
||||
assert.equal(result.report.cleanupVerified, false);
|
||||
assert.equal((await summarizeControlledCampaign(f.manifest)).committedUsd, '0.014000000');
|
||||
});
|
||||
@@ -0,0 +1,93 @@
|
||||
import { createHash } from 'node:crypto';
|
||||
import { createRequire } from 'node:module';
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import { isAbsolute } from 'node:path';
|
||||
|
||||
const require = createRequire(new URL('../../../../server/package.json', import.meta.url));
|
||||
const { tsImport } = await import(require.resolve('tsx/esm/api'));
|
||||
export const costRuntime = await tsImport('./shared-cost-runtime.ts', import.meta.url);
|
||||
export const sha256 = bytes => createHash('sha256').update(bytes).digest('hex');
|
||||
export const canonical = value => JSON.stringify(value, (_key, item) => item && typeof item === 'object' && !Array.isArray(item)
|
||||
? Object.fromEntries(Object.keys(item).sort().map(key => [key, item[key]])) : item);
|
||||
export const usd = value => `${value / 1_000_000_000n}.${String(value % 1_000_000_000n).padStart(9, '0')}`;
|
||||
export function units(value) {
|
||||
if (typeof value !== 'string' || !/^(?:0|[1-9][0-9]*)(?:\.[0-9]{1,9})?$/.test(value)) throw new Error('Invalid controller USD amount');
|
||||
const [whole, fraction = ''] = value.split('.');
|
||||
return BigInt(whole) * 1_000_000_000n + BigInt(fraction.padEnd(9, '0'));
|
||||
}
|
||||
export function fields(value, names) {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).some(key => !names.includes(key))) throw new Error('Invalid controller fields');
|
||||
}
|
||||
const hash = value => typeof value === 'string' && /^[a-f0-9]{64}$/.test(value);
|
||||
const id = value => costRuntime.validEvalLedgerId(value);
|
||||
export function requestBindingSha256(request) {
|
||||
const copy = structuredClone(request);
|
||||
if (!copy.session || !copy.runnerd) throw new Error('Invalid controlled request');
|
||||
delete copy.session.workingDirectory;
|
||||
delete copy.runnerd.path;
|
||||
return sha256(canonical(copy));
|
||||
}
|
||||
export async function readJson(file, maximum = 1_048_576) {
|
||||
const bytes = await readFile(file);
|
||||
if (bytes.length > maximum) throw new Error('Controller document exceeds its byte limit');
|
||||
return JSON.parse(bytes.toString('utf8'));
|
||||
}
|
||||
export function validateManifest(value, now = Date.now(), { admission = true } = {}) {
|
||||
fields(value, ['schema', 'campaignId', 'maxUsd', 'deadlineAt', 'pricingVerifiedAt', 'pricingExpiresAt', 'overhead',
|
||||
'sourceRevision', 'definitionsRevision', 'runnerCli', 'runnerPackage', 'runnerd', 'outputDirectory', 'route', 'projectId', 'models', 'attempts']);
|
||||
const deadline = Date.parse(value.deadlineAt);
|
||||
const verified = Date.parse(value.pricingVerifiedAt), expires = Date.parse(value.pricingExpiresAt);
|
||||
if (value.schema !== 'paperclip.runner-eval.controller/v1' || !id(value.campaignId)
|
||||
|| !Number.isFinite(now) || !Number.isFinite(deadline) || admission && (deadline <= now || deadline - now > 4 * 3_600_000)
|
||||
|| !Number.isFinite(verified) || !Number.isFinite(expires) || verified >= deadline || expires <= verified || admission && verified > now || expires <= deadline || expires - verified > 31 * 86_400_000
|
||||
|| !/^[a-f0-9]{40}$/.test(value.sourceRevision ?? '') || !/^[a-f0-9]{40}$/.test(value.definitionsRevision ?? '')
|
||||
|| !isAbsolute(value.outputDirectory ?? '') || !['direct', 'braintrust'].includes(value.route)
|
||||
|| value.route === 'braintrust' && !/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(value.projectId ?? '')
|
||||
|| value.route === 'direct' && value.projectId !== undefined) throw new Error('Invalid controller campaign identity or validity window');
|
||||
for (const artifact of [value.runnerCli, value.runnerPackage, value.runnerd]) {
|
||||
fields(artifact, ['path', 'sha256']);
|
||||
if (!isAbsolute(artifact.path ?? '') || !hash(artifact.sha256)) throw new Error('Invalid controller executable provenance');
|
||||
}
|
||||
const overhead = costRuntime.validateOperatingOverhead(value.overhead, value.maxUsd);
|
||||
if (admission && Date.parse(value.overhead.verifiedAt) > now || Date.parse(value.overhead.expiresAt) <= deadline
|
||||
|| value.overhead.components.some(component => !component.provisionalBound)) throw new Error('Controller requires current full-lifecycle operating bounds');
|
||||
if (!Array.isArray(value.models) || !value.models.length || value.models.length > 16
|
||||
|| !Array.isArray(value.attempts) || !value.attempts.length || value.attempts.length > 1000) throw new Error('Invalid controlled campaign selection');
|
||||
const prices = new Map();
|
||||
const rateFields = ['inputUsdPerMillion', 'cachedReadUsdPerMillion', 'outputUsdPerMillion', 'cacheWrite5mUsdPerMillion', 'cacheWrite1hUsdPerMillion', 'cacheWrite30mUsdPerMillion', 'cacheWrite12hUsdPerMillion'];
|
||||
for (const model of value.models) {
|
||||
fields(model, ['provider', 'price', 'maximumInputTokens', 'maximumOutputTokens', 'timeoutMs', 'provisionalBound']);
|
||||
fields(model.price, ['provider', 'model', 'version', 'sourceUrl', 'longContext', ...rateFields]);
|
||||
if (model.price.longContext) { fields(model.price.longContext, ['aboveInputTokens', 'rates']); fields(model.price.longContext.rates, rateFields); }
|
||||
const url = new URL(model.price.sourceUrl);
|
||||
const key = `${model.provider}:${model.price.model}`;
|
||||
if (!['openai', 'anthropic'].includes(model.provider) || !model.provisionalBound || url.protocol !== 'https:'
|
||||
|| url.username || url.password || url.search || url.hash || prices.has(key)) throw new Error('Invalid controller model pricing');
|
||||
prices.set(key, units(costRuntime.evalRequestReservation({ ...model, mode: value.route, apiKey: 'validation' })));
|
||||
}
|
||||
const attempts = new Set(), executions = new Set();
|
||||
let allocated = overhead;
|
||||
for (const attempt of value.attempts) {
|
||||
fields(attempt, ['executionId', 'attemptId', 'requestSha256', 'provider', 'acpxAgent', 'model', 'maxUsd', 'maxRequests', 'maxConcurrent', 'timeoutMs']);
|
||||
const provider = attempt.provider === 'codex' && attempt.acpxAgent === undefined ? 'openai'
|
||||
: attempt.provider === 'acpx' && attempt.acpxAgent === 'claude' ? 'anthropic' : null;
|
||||
const reservation = prices.get(`${provider}:${attempt.model}`);
|
||||
if (!provider || !id(attempt.executionId) || !id(attempt.attemptId) || !hash(attempt.requestSha256)
|
||||
|| attempts.has(attempt.attemptId) || executions.has(attempt.executionId) || reservation === undefined
|
||||
|| !Number.isSafeInteger(attempt.timeoutMs) || attempt.timeoutMs < 1 || attempt.timeoutMs > 4 * 3_600_000
|
||||
|| !Number.isSafeInteger(attempt.maxRequests) || attempt.maxRequests < 1 || attempt.maxRequests > 1000
|
||||
|| !Number.isSafeInteger(attempt.maxConcurrent) || attempt.maxConcurrent < 1 || attempt.maxConcurrent > 16
|
||||
|| units(attempt.maxUsd) < reservation) throw new Error('Invalid controlled attempt allocation');
|
||||
attempts.add(attempt.attemptId); executions.add(attempt.executionId); allocated += units(attempt.maxUsd);
|
||||
}
|
||||
if (allocated > units(value.maxUsd)) throw new Error('Controlled allocations exceed the campaign budget');
|
||||
return { manifest: structuredClone(value), manifestSha256: sha256(canonical(value)), allocatedUsd: usd(allocated), overheadUsd: usd(overhead) };
|
||||
}
|
||||
export function bindRequest(manifest, request) {
|
||||
const attempt = manifest.attempts.find(item => item.attemptId === request.attemptId);
|
||||
if (!attempt || requestBindingSha256(request) !== attempt.requestSha256 || request.provider !== attempt.provider
|
||||
|| request.acpxAgent !== (attempt.acpxAgent ?? null) && request.acpxAgent !== attempt.acpxAgent
|
||||
|| request.model !== attempt.model || request.runnerd.sha256.replace(/^sha256:/, '') !== manifest.runnerd.sha256
|
||||
|| request.runnerd.path !== manifest.runnerd.path) throw new Error('Controlled request does not match its allocated identity');
|
||||
return attempt;
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
import { canonical, costRuntime, readJson, requestBindingSha256, sha256, units, usd, validateManifest } from './manifest.mjs';
|
||||
import { attemptDirectory } from './run.mjs';
|
||||
|
||||
export async function summarizeControlledCampaign(value) {
|
||||
const { manifest, manifestSha256, allocatedUsd, overheadUsd } = validateManifest(value, Date.now(), { admission: false });
|
||||
let known = 0n, committed = units(overheadUsd);
|
||||
const attempts = [];
|
||||
for (const allocation of manifest.attempts) {
|
||||
const root = attemptDirectory(manifest, allocation.attemptId);
|
||||
let snapshot, report, verified = false;
|
||||
try {
|
||||
const ledgerBytes = await readFile(join(root, 'ledger.jsonl'));
|
||||
const ledger = await costRuntime.readEvalCostLedger(join(root, 'ledger.jsonl'));
|
||||
const candidateSnapshot = ledger.state.snapshot();
|
||||
const ledgerDeadline = Date.parse(ledger.state.limits.deadlineAt);
|
||||
const fixedLimits = { campaignId: manifest.campaignId, maxUsd: allocation.maxUsd, deadlineAt: ledger.state.limits.deadlineAt,
|
||||
maxRequests: allocation.maxRequests, maxConcurrent: allocation.maxConcurrent, maxAttemptsPerExecution: 1 };
|
||||
if (canonical(ledger.state.limits) !== canonical(fixedLimits) || !Number.isFinite(ledgerDeadline)
|
||||
|| ledgerDeadline <= Date.parse(manifest.pricingVerifiedAt) || ledgerDeadline > Date.parse(manifest.deadlineAt)
|
||||
|| canonical(candidateSnapshot.admittedAttempts) !== canonical([{ executionId: allocation.executionId, attemptId: allocation.attemptId }])
|
||||
|| candidateSnapshot.requests.some(request => request.executionId !== allocation.executionId || request.attemptId !== allocation.attemptId || request.phase !== 'agent')) throw new Error('Ledger identity does not match its allocation');
|
||||
snapshot = candidateSnapshot;
|
||||
const admission = await readJson(join(root, 'admission.json'));
|
||||
const started = Date.parse(admission.startedAt);
|
||||
const expectedLimits = { campaignId: manifest.campaignId, maxUsd: allocation.maxUsd,
|
||||
deadlineAt: new Date(Math.min(Date.parse(manifest.deadlineAt), started + allocation.timeoutMs)).toISOString(),
|
||||
maxRequests: allocation.maxRequests, maxConcurrent: allocation.maxConcurrent, maxAttemptsPerExecution: 1 };
|
||||
if (admission.schema !== 'paperclip.runner-eval.admission/v1' || admission.manifestSha256 !== manifestSha256
|
||||
|| admission.campaignId !== manifest.campaignId || canonical(admission.attempt) !== canonical(allocation)
|
||||
|| canonical(admission.limits) !== canonical(expectedLimits) || canonical(ledger.state.limits) !== canonical(expectedLimits)) throw new Error('Ledger does not match its admission');
|
||||
report = await readJson(join(root, 'result.json'));
|
||||
const request = await readJson(join(root, 'request.json'), 8_388_608);
|
||||
if (requestBindingSha256(request) !== allocation.requestSha256 || request.runnerd.sha256.replace(/^sha256:/, '') !== manifest.runnerd.sha256
|
||||
|| canonical(Object.keys(report.receiptHashes ?? {}).sort()) !== canonical(snapshot.requests.map(item => item.requestId).sort())) throw new Error('Retained request evidence does not match');
|
||||
for (const item of snapshot.requests) {
|
||||
if (!/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/.test(item.requestId)) throw new Error('Invalid retained receipt identity');
|
||||
const bytes = await readFile(join(root, 'receipts', `${item.requestId}.json`));
|
||||
const receipt = JSON.parse(bytes.toString('utf8'));
|
||||
if (sha256(bytes) !== report.receiptHashes[item.requestId] || receipt.requestId !== item.requestId
|
||||
|| receipt.route !== manifest.route || receipt.costUsd !== item.costUsd) throw new Error('Retained receipt does not match its ledger');
|
||||
}
|
||||
const artifactBytes = await readFile(join(root, 'provider-artifact.json'));
|
||||
const artifact = JSON.parse(artifactBytes.toString('utf8'));
|
||||
const descriptorBytes = await readFile(join(root, 'gateway.json'));
|
||||
const descriptor = JSON.parse(descriptorBytes.toString('utf8'));
|
||||
if (report.schema !== 'paperclip.runner-eval.control-result/v1' || report.manifestSha256 !== manifestSha256
|
||||
|| report.campaignId !== manifest.campaignId || report.executionId !== allocation.executionId || report.attemptId !== allocation.attemptId
|
||||
|| report.requestSha256 !== allocation.requestSha256 || report.ledgerSha256 !== sha256(ledgerBytes)
|
||||
|| report.providerArtifactSha256 !== sha256(artifactBytes) || canonical(report.accounting) !== canonical(snapshot)
|
||||
|| report.status !== 'closed' || report.cleanupVerified !== true || report.secretExcluded !== true || report.failure !== null
|
||||
|| report.startedAt !== admission.startedAt || !Number.isFinite(Date.parse(report.finishedAt)) || Date.parse(report.finishedAt) < started
|
||||
|| !report.child || report.child.stopped !== null || artifact.snapshot?.status !== 'closed' || artifact.snapshot?.process?.runnerExited !== true
|
||||
|| descriptor.schema !== 'paperclip.runner-eval.gateway/v1' || descriptor.provider !== (allocation.provider === 'codex' ? 'openai' : 'anthropic')
|
||||
|| descriptor.model !== allocation.model || descriptor.upstreamRoute !== manifest.route
|
||||
|| descriptor.campaignId !== manifest.campaignId || descriptor.executionId !== allocation.executionId || descriptor.attemptId !== allocation.attemptId
|
||||
|| canonical(artifact.requestRouting) !== canonical({ mode: 'controller_gateway', provider: descriptor.provider, declaredUpstreamRoute: manifest.route,
|
||||
campaignId: manifest.campaignId, executionId: allocation.executionId, attemptId: allocation.attemptId, descriptorSha256: sha256(descriptorBytes) })
|
||||
|| artifact.attemptId !== allocation.attemptId || artifact.requestedModel !== allocation.model
|
||||
|| snapshot.inflight || snapshot.conflictingRequests || snapshot.overBudget
|
||||
|| snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests) throw new Error('Incomplete controller evidence');
|
||||
verified = true;
|
||||
} catch {}
|
||||
const cost = snapshot ? units(snapshot.knownUsd) : 0n;
|
||||
const observedCommitment = snapshot ? units(snapshot.committedUsd) : 0n;
|
||||
const commitment = verified ? observedCommitment : observedCommitment > units(allocation.maxUsd) ? observedCommitment : units(allocation.maxUsd);
|
||||
known += cost; committed += commitment;
|
||||
attempts.push({ executionId: allocation.executionId, attemptId: allocation.attemptId, status: verified ? 'retained' : 'incomplete',
|
||||
allocationUsd: allocation.maxUsd, knownUsd: usd(cost), heldUsd: usd(commitment - cost), committedUsd: usd(commitment),
|
||||
unknownRequests: snapshot?.unknownRequests ?? null, boundedUnresolvedRequests: snapshot?.boundedUnresolvedRequests ?? null });
|
||||
}
|
||||
return { schema: 'paperclip.runner-eval.control-campaign/v1', campaignId: manifest.campaignId, manifestSha256,
|
||||
limitUsd: manifest.maxUsd, allocatedUsd, operatingHeldUsd: overheadUsd, knownUsd: usd(known), heldUsd: usd(committed - known),
|
||||
committedUsd: usd(committed), overBudget: committed > units(manifest.maxUsd), incompleteAttempts: attempts.filter(attempt => attempt.status === 'incomplete').length,
|
||||
costStatus: attempts.some(attempt => attempt.status === 'incomplete') || committed > units(manifest.maxUsd) ? 'incomplete' : 'provisional', attempts };
|
||||
}
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) {
|
||||
if (process.argv.length !== 3) throw new Error('Usage: report.mjs <controller-manifest.json>');
|
||||
const result = await summarizeControlledCampaign(await readJson(process.argv[2]));
|
||||
process.stdout.write(JSON.stringify(result, null, 2) + '\n');
|
||||
if (result.incompleteAttempts || result.overBudget) process.exitCode = 2;
|
||||
}
|
||||
@@ -0,0 +1,193 @@
|
||||
import { spawn } from 'node:child_process';
|
||||
import { once } from 'node:events';
|
||||
import { lstat, mkdir, open, readFile, realpath, rename, rm, writeFile } from 'node:fs/promises';
|
||||
import { dirname, isAbsolute, join, relative, resolve, sep } from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
import { parseArgs } from 'node:util';
|
||||
import { bindRequest, canonical, costRuntime, readJson, sha256, validateManifest } from './manifest.mjs';
|
||||
|
||||
async function atomicJson(file, value) {
|
||||
const temporary = `${file}.tmp`;
|
||||
const handle = await open(temporary, 'wx', 0o600);
|
||||
try { await handle.writeFile(JSON.stringify(value, null, 2) + '\n'); await handle.sync(); }
|
||||
finally { await handle.close(); }
|
||||
await rename(temporary, file);
|
||||
}
|
||||
const absent = pid => { try { process.kill(pid, 0); return false; } catch (error) { return error.code === 'ESRCH'; } };
|
||||
export const attemptDirectory = (manifest, attemptId) => join(manifest.outputDirectory, sha256(attemptId));
|
||||
|
||||
async function runChild(input) {
|
||||
const child = spawn(process.execPath, ['--import', input.loader, input.cli, '--request', input.requestPath, '--output', input.artifactPath, '--gateway-config', input.descriptorPath], {
|
||||
cwd: input.cwd, env: input.environment, detached: true, stdio: ['ignore', 'pipe', 'pipe'],
|
||||
});
|
||||
let stopped = null, force, escalation;
|
||||
const output = { stdout: [], stderr: [] }, sizes = { stdout: 0, stderr: 0 };
|
||||
const stop = reason => {
|
||||
if (stopped) return;
|
||||
stopped = reason;
|
||||
void input.closeGateway().catch(() => {});
|
||||
try { process.kill(-child.pid, 'SIGTERM'); } catch {}
|
||||
escalation = new Promise(resolve => {
|
||||
force = setTimeout(() => { try { process.kill(-child.pid, 'SIGKILL'); } catch {} finally { resolve(); } }, 5000);
|
||||
});
|
||||
};
|
||||
for (const stream of ['stdout', 'stderr']) child[stream].on('data', bytes => {
|
||||
const available = Math.max(0, 1_048_576 - sizes[stream]);
|
||||
output[stream].push(bytes.subarray(0, available)); sizes[stream] += bytes.length;
|
||||
if (sizes[stream] > 1_048_576) stop('output_limit');
|
||||
});
|
||||
const onAbort = () => stop('controller_cancelled');
|
||||
input.signal?.addEventListener('abort', onAbort, { once: true });
|
||||
const timeout = setTimeout(() => stop('attempt_deadline'), Math.max(1, input.deadlineAt - Date.now()));
|
||||
if (input.signal?.aborted) onAbort();
|
||||
try {
|
||||
const [code, signal] = await once(child, 'exit');
|
||||
if (stopped) {
|
||||
let groupPresent = true;
|
||||
try { process.kill(-child.pid, 0); } catch (error) { if (error.code === 'ESRCH') groupPresent = false; }
|
||||
if (groupPresent) await escalation;
|
||||
}
|
||||
return { code, signal, pid: child.pid, stopped, stdout: Buffer.concat(output.stdout).toString(), stderr: Buffer.concat(output.stderr).toString() };
|
||||
} finally {
|
||||
clearTimeout(timeout); clearTimeout(force); input.signal?.removeEventListener('abort', onAbort);
|
||||
}
|
||||
}
|
||||
|
||||
export async function runControlledEval({ manifestPath, requestPath, outputPath, environment = process.env, signal, transport }) {
|
||||
if (!['linux', 'darwin'].includes(process.platform)) throw new Error('Controlled eval process ownership requires Linux or macOS');
|
||||
signal?.throwIfAborted();
|
||||
const { manifest, manifestSha256 } = validateManifest(await readJson(manifestPath));
|
||||
const request = await readJson(requestPath, 8_388_608), attempt = bindRequest(manifest, request);
|
||||
try { await lstat(outputPath); throw new Error('Controlled output already exists'); }
|
||||
catch (error) { if (error.code !== 'ENOENT') throw error; }
|
||||
if (request.session.workingDirectory) {
|
||||
const suffix = relative(resolve(request.session.workingDirectory), resolve(manifest.outputDirectory));
|
||||
if (!suffix || suffix !== '..' && !suffix.startsWith(`..${sep}`) && !isAbsolute(suffix)) throw new Error('Controller evidence must be outside the disposable workspace');
|
||||
}
|
||||
const originalCliUrl = pathToFileURL(await realpath(manifest.runnerCli.path)).href;
|
||||
const verified = {};
|
||||
for (const name of ['runnerCli', 'runnerPackage', 'runnerd']) {
|
||||
const artifact = manifest[name], bytes = await readFile(artifact.path);
|
||||
if (sha256(bytes) !== artifact.sha256) throw new Error('Controlled eval executable provenance changed');
|
||||
if (name !== 'runnerPackage') verified[name] = bytes;
|
||||
}
|
||||
const provider = attempt.provider === 'codex' ? 'openai' : 'anthropic';
|
||||
const keyName = manifest.route === 'braintrust' ? 'BRAINTRUST_API_KEY' : provider === 'openai' ? 'OPENAI_API_KEY' : 'ANTHROPIC_API_KEY';
|
||||
const key = environment[keyName];
|
||||
if (!key || /[\r\n]/.test(key)) throw new Error('Missing controlled eval upstream credential');
|
||||
const model = manifest.models.find(item => item.provider === provider && item.price.model === attempt.model);
|
||||
const root = attemptDirectory(manifest, attempt.attemptId);
|
||||
await mkdir(manifest.outputDirectory, { recursive: true, mode: 0o700 });
|
||||
if (request.session.workingDirectory) {
|
||||
const suffix = relative(await realpath(request.session.workingDirectory), await realpath(manifest.outputDirectory));
|
||||
if (!suffix || suffix !== '..' && !suffix.startsWith(`..${sep}`) && !isAbsolute(suffix)) throw new Error('Controller evidence resolves inside the disposable workspace');
|
||||
}
|
||||
await mkdir(root, { mode: 0o700 });
|
||||
const started = Date.now(), startedAt = new Date(started).toISOString();
|
||||
const deadlineAt = Math.min(Date.parse(manifest.deadlineAt), started + attempt.timeoutMs);
|
||||
const limits = { campaignId: manifest.campaignId, maxUsd: attempt.maxUsd, deadlineAt: new Date(deadlineAt).toISOString(),
|
||||
maxRequests: attempt.maxRequests, maxConcurrent: attempt.maxConcurrent, maxAttemptsPerExecution: 1 };
|
||||
await atomicJson(join(root, 'admission.json'), { schema: 'paperclip.runner-eval.admission/v1', campaignId: manifest.campaignId,
|
||||
manifestSha256, attempt, limits, startedAt });
|
||||
const executables = join(root, 'executables');
|
||||
await mkdir(executables, { mode: 0o700 });
|
||||
const stagedCli = join(executables, 'entry.mjs'), stagedRunnerd = join(executables, 'runnerd'), loader = join(executables, 'imports.mjs');
|
||||
await writeFile(stagedCli, verified.runnerCli, { flag: 'wx', mode: 0o400 });
|
||||
await writeFile(stagedRunnerd, verified.runnerd, { flag: 'wx', mode: 0o500 });
|
||||
const stagedUrl = JSON.stringify(pathToFileURL(await realpath(stagedCli)).href), originalUrl = JSON.stringify(originalCliUrl);
|
||||
await writeFile(loader, `import { registerHooks } from 'node:module';
|
||||
registerHooks({ resolve(specifier, context, nextResolve) {
|
||||
return nextResolve(specifier, context.parentURL === ${stagedUrl} ? { ...context, parentURL: ${originalUrl} } : context);
|
||||
} });\n`, { flag: 'wx', mode: 0o400 });
|
||||
const retainedRequestPath = join(root, 'request.json');
|
||||
const childRequest = structuredClone(request); childRequest.runnerd.path = stagedRunnerd;
|
||||
await writeFile(retainedRequestPath, JSON.stringify(childRequest) + '\n', { flag: 'wx', mode: 0o600 });
|
||||
await mkdir(join(root, 'receipts'), { mode: 0o700 });
|
||||
await mkdir(join(root, 'home'), { mode: 0o700 });
|
||||
await mkdir(join(root, 'tmp'), { mode: 0o700 });
|
||||
const ledgerPath = join(root, 'ledger.jsonl');
|
||||
const ledger = await costRuntime.openEvalCostLedger(ledgerPath, limits);
|
||||
let gateway, child, artifact, failure = null, cleanupVerified = false, secretExcluded = true;
|
||||
let providerArtifactSha256;
|
||||
const secrets = [key];
|
||||
const redact = raw => secrets.reduce((text, secret) => text.split(secret).join('[redacted]'), raw);
|
||||
try {
|
||||
await ledger.admitAttempt(attempt.executionId, attempt.attemptId);
|
||||
gateway = await costRuntime.startEvalGateway({ ledger, identity: { executionId: attempt.executionId, attemptId: attempt.attemptId, phase: 'agent' },
|
||||
route: { ...model, mode: manifest.route, projectId: manifest.projectId, endpointName: provider === 'openai' ? 'OPENAI_API_KEY' : 'ANTHROPIC_API_KEY', apiKey: key },
|
||||
deadlineAt, transport, persist: receipt => atomicJson(join(root, 'receipts', `${receipt.requestId}.json`), receipt),
|
||||
});
|
||||
secrets.push(gateway.token);
|
||||
const descriptor = { schema: 'paperclip.runner-eval.gateway/v1', provider, model: attempt.model, baseUrl: gateway.baseUrl,
|
||||
campaignId: manifest.campaignId, executionId: attempt.executionId, attemptId: attempt.attemptId, upstreamRoute: manifest.route };
|
||||
const descriptorPath = join(root, 'gateway.json'), descriptorRaw = JSON.stringify(descriptor) + '\n';
|
||||
await writeFile(descriptorPath, descriptorRaw, { flag: 'wx', mode: 0o600 });
|
||||
const childEnvironment = Object.fromEntries(['PATH', 'LANG', 'LANGUAGE', 'LC_ALL', 'TZ'].flatMap(name => environment[name] === undefined ? [] : [[name, environment[name]]]));
|
||||
Object.assign(childEnvironment, { HOME: join(root, 'home'), TMPDIR: join(root, 'tmp'), TMP: join(root, 'tmp'), TEMP: join(root, 'tmp'), PAPERCLIP_RUNNER_EVAL_GATEWAY_TOKEN: gateway.token });
|
||||
const artifactPath = join(root, 'provider-artifact.json');
|
||||
signal?.throwIfAborted();
|
||||
if (Date.now() >= deadlineAt) throw new Error('Controlled attempt deadline reached before spawn');
|
||||
child = await runChild({ cli: stagedCli, loader, requestPath: retainedRequestPath, artifactPath, descriptorPath,
|
||||
cwd: request.session.workingDirectory ?? root, environment: childEnvironment, deadlineAt, signal, closeGateway: () => gateway.close() });
|
||||
for (const stream of ['stdout', 'stderr']) await writeFile(join(root, `${stream}.log`), redact(child[stream]), { flag: 'wx', mode: 0o600 });
|
||||
const raw = await readFile(artifactPath, 'utf8');
|
||||
if (Buffer.byteLength(raw) > 16_777_216) throw new Error('Provider artifact exceeds its byte limit');
|
||||
secretExcluded = secrets.every(secret => !raw.includes(secret));
|
||||
if (!secretExcluded) await writeFile(artifactPath, redact(raw), { mode: 0o600 });
|
||||
providerArtifactSha256 = sha256(redact(raw));
|
||||
artifact = JSON.parse(redact(raw));
|
||||
const expected = { mode: 'controller_gateway', provider, declaredUpstreamRoute: manifest.route, campaignId: manifest.campaignId,
|
||||
executionId: attempt.executionId, attemptId: attempt.attemptId, descriptorSha256: sha256(descriptorRaw) };
|
||||
if (canonical(artifact.requestRouting) !== canonical(expected) || artifact.attemptId !== attempt.attemptId || artifact.requestedModel !== attempt.model) throw new Error('Provider did not retain the assigned route');
|
||||
const processInfo = artifact.snapshot?.process;
|
||||
const pids = [...new Set([child.pid, processInfo?.runnerPid, processInfo?.providerPid, processInfo?.sidecarPid, processInfo?.agentPid].filter(pid => Number.isSafeInteger(pid) && pid > 0))];
|
||||
cleanupVerified = !child.stopped && artifact.snapshot?.status === 'closed' && processInfo?.runnerExited === true
|
||||
&& Number.isSafeInteger(processInfo.runnerPid) && processInfo.runnerPid > 0
|
||||
&& Number.isSafeInteger(processInfo.providerPid) && processInfo.providerPid > 0 && pids.every(absent);
|
||||
if (!cleanupVerified || !secretExcluded) throw new Error('Controlled provider cleanup or credential exclusion is unverified');
|
||||
} catch {
|
||||
failure = 'controlled_attempt_incomplete';
|
||||
} finally {
|
||||
try { await gateway?.close(); } catch { failure = 'gateway_cleanup_unverified'; }
|
||||
try { await ledger.close(); } catch { failure = 'ledger_close_failed'; }
|
||||
}
|
||||
if (cleanupVerified) {
|
||||
try { await rm(executables, { recursive: true }); }
|
||||
catch { cleanupVerified = false; failure ??= 'executable_cleanup_unverified'; }
|
||||
}
|
||||
const snapshot = ledger.snapshot();
|
||||
if (snapshot.overBudget || snapshot.conflictingRequests || snapshot.inflight || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests) failure ??= 'request_accounting_unbounded';
|
||||
const receiptHashes = {};
|
||||
try {
|
||||
for (const request of snapshot.requests) receiptHashes[request.requestId] = sha256(await readFile(join(root, 'receipts', `${request.requestId}.json`)));
|
||||
} catch { failure ??= 'request_receipt_missing'; }
|
||||
const ledgerSha256 = sha256(await readFile(ledgerPath));
|
||||
const report = { schema: 'paperclip.runner-eval.control-result/v1', campaignId: manifest.campaignId, manifestSha256,
|
||||
executionId: attempt.executionId, attemptId: attempt.attemptId, requestSha256: attempt.requestSha256,
|
||||
allocationUsd: attempt.maxUsd, limits, startedAt, finishedAt: new Date().toISOString(), status: failure ? 'incomplete' : 'closed', failure,
|
||||
child: child ? { code: child.code, signal: child.signal, stopped: child.stopped } : null,
|
||||
cleanupVerified, secretExcluded, providerArtifactSha256, ledgerSha256, receiptHashes, accounting: snapshot };
|
||||
await atomicJson(join(root, 'result.json'), report);
|
||||
const resultArtifact = artifact ?? { schema: 'paperclip-runner/eval-session-artifact/v1', attemptId: attempt.attemptId,
|
||||
requestedModel: attempt.model, infrastructureError: 'Controlled provider did not retain an artifact' };
|
||||
if (failure) Object.assign(resultArtifact, { accountingError: 'Controlled request accounting or cleanup is incomplete',
|
||||
accountingFailure: { class: 'provider_budget_coverage_unknown', category: 'provider_budget', retryable: false } });
|
||||
resultArtifact.controllerAccounting = report;
|
||||
await mkdir(dirname(resolve(outputPath)), { recursive: true, mode: 0o700 });
|
||||
await writeFile(outputPath, JSON.stringify(resultArtifact, null, 2) + '\n', { flag: 'wx', mode: 0o600 });
|
||||
return { code: failure ? 2 : child?.code ?? 2, report };
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) {
|
||||
const controller = new AbortController();
|
||||
const cancel = () => controller.abort();
|
||||
process.once('SIGINT', cancel); process.once('SIGTERM', cancel);
|
||||
try {
|
||||
const { values } = parseArgs({ options: { request: { type: 'string' }, output: { type: 'string' } } });
|
||||
if (!values.request || !values.output || !process.env.PAPERCLIP_RUNNER_EVAL_CONTROL_MANIFEST) throw new Error('A controlled manifest, request and output are required');
|
||||
const result = await runControlledEval({ manifestPath: process.env.PAPERCLIP_RUNNER_EVAL_CONTROL_MANIFEST,
|
||||
requestPath: values.request, outputPath: values.output, signal: controller.signal });
|
||||
process.exitCode = result.code;
|
||||
} catch { process.stderr.write('Controlled Runner eval admission or retention failed; inspect retained controller evidence.\n'); process.exitCode = 2; }
|
||||
finally { process.removeListener('SIGINT', cancel); process.removeListener('SIGTERM', cancel); }
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
export { EvalSpendState, openEvalCostLedger, readEvalCostLedger, validEvalLedgerId } from '../../../../tests/runner-e2e/cost-ledger.js';
|
||||
export { evalRequestReservation, startEvalGateway } from '../../../../tests/runner-e2e/eval-gateway.js';
|
||||
export { EVAL_OVERHEAD_COMPONENTS, validateOperatingOverhead } from '../../../../tests/runner-e2e/operating-overhead.js';
|
||||
@@ -73,6 +73,7 @@ it.each([["codex", undefined], ["acpx", "claude"]])("routes %s/%s with only its
|
||||
expect(config).toContain('env_key = "PAPERCLIP_AI_PROVIDER_KEY"');
|
||||
expect(config).toContain("requires_openai_auth = false");
|
||||
expect(config).toContain("shell_snapshot = false");
|
||||
expect(config).toContain('web_search = "disabled"');
|
||||
expect(config).not.toContain(token);
|
||||
expect((await stat(join(env.CODEX_HOME!, "config.toml"))).mode & 0o777).toBe(0o600);
|
||||
}
|
||||
|
||||
@@ -70,6 +70,7 @@ export async function prepareEvalGatewayRuntime(
|
||||
environment.PAPERCLIP_AI_PROVIDER_KEY = token;
|
||||
await writeFile(join(codexHome, "config.toml"), [
|
||||
'model_provider = "paperclip_eval"',
|
||||
'web_search = "disabled"',
|
||||
`model = ${JSON.stringify(request.model)}`,
|
||||
'[model_providers.paperclip_eval]',
|
||||
'name = "Paperclip eval controller"',
|
||||
|
||||
@@ -121,6 +121,29 @@ it.each([
|
||||
await expect(forwardEvalRequest({ ...f, payload, transport })).rejects.toThrow();
|
||||
expect(transport).not.toHaveBeenCalled(); expect(f.ledger.snapshot().requests).toHaveLength(0);
|
||||
});
|
||||
it("admits local function and custom tools inside a Codex namespace", async () => {
|
||||
const f = await fixture();
|
||||
const tools = [{ type: "namespace", name: "local", tools: [
|
||||
{ type: "function", name: "read_task", parameters: { type: "object" } },
|
||||
{ type: "custom", name: "apply_patch" },
|
||||
] }];
|
||||
const transport = vi.fn<typeof fetch>(async (_url, init) => {
|
||||
expect(JSON.parse(init!.body as string).tools).toEqual(tools);
|
||||
return response();
|
||||
});
|
||||
const result = await forwardEvalRequest({ ...f, payload: { ...f.payload, tools }, transport });
|
||||
expect(result.outcome).toBe("succeeded");
|
||||
expect(transport).toHaveBeenCalledOnce();
|
||||
});
|
||||
it.each([
|
||||
{ type: "namespace", name: "local", tools: [{ type: "web_search" }] },
|
||||
{ type: "namespace", name: "local", tools: [{ type: "namespace", name: "nested", tools: [{ type: "web_search" }] }] },
|
||||
{ type: "namespace", name: "local" },
|
||||
])("rejects unqualified tools inside namespaces before reservation", async tool => {
|
||||
const f = await fixture(), transport = vi.fn<typeof fetch>();
|
||||
await expect(forwardEvalRequest({ ...f, payload: { ...f.payload, tools: [tool] }, transport })).rejects.toThrow();
|
||||
expect(transport).not.toHaveBeenCalled(); expect(f.ledger.snapshot().requests).toHaveLength(0);
|
||||
});
|
||||
it("does not dispatch when the private pre-request receipt cannot be persisted", async () => {
|
||||
const f = await fixture(), transport = vi.fn<typeof fetch>();
|
||||
await expect(forwardEvalRequest({ ...f, persist: async () => { throw new Error("disk unavailable"); }, transport })).rejects.toThrow("disk unavailable");
|
||||
|
||||
@@ -71,8 +71,12 @@ function boundedPayload(value: unknown, route: EvalGatewayRoute) {
|
||||
if (body.background === true || body.mcp_servers !== undefined || body.service_tier !== undefined && !["auto", "default", "standard_only"].includes(String(body.service_tier))) throw new EvalRequestError("unqualified_billing_mode");
|
||||
if (body.speed !== undefined && body.speed !== "standard" || body.inference_geo !== undefined) throw new EvalRequestError("unqualified_billing_mode");
|
||||
if (body.tools !== undefined && (!Array.isArray(body.tools) || body.tools.some(tool => {
|
||||
const type = row(tool).type;
|
||||
return route.provider === "openai" ? !["function", "custom"].includes(String(type)) : type !== undefined && type !== "custom";
|
||||
const definition = row(tool), type = definition.type;
|
||||
if (route.provider !== "openai") return type !== undefined && type !== "custom";
|
||||
if (type === "namespace") return typeof definition.name !== "string" || !definition.name
|
||||
|| !Array.isArray(definition.tools) || !definition.tools.length
|
||||
|| definition.tools.some(nested => !["function", "custom"].includes(String(row(nested).type)));
|
||||
return !["function", "custom"].includes(String(type));
|
||||
}))) throw new EvalRequestError("unqualified_hosted_tool");
|
||||
const inspect = (item: unknown): void => {
|
||||
if (Array.isArray(item)) { item.forEach(inspect); return; }
|
||||
|
||||
Reference in new issue
Block a user