mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 22:13:59 +01:00
fix(evals): derive the controlled Runner eval policy from committed defaults (#19253)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner protocol qualification runs real providers against 41 cases and reports them in TestOps. > - Controlled mode (#17889) adds a spend cap, private evidence storage, and cost and trace accounting. > - Operators must hand-maintain one large JSON variable for each campaign size, price review, and lockfile change. > - A one-cell policy blocked the first 41-case run, and its prices expired after 24 hours. > - This pull request derives that policy from committed defaults and the selected cases. > - The benefit is that a full run needs no policy edit, and all three safety properties stay. ## Linked Issues or Issue Description Refs #17889, #19011, #19070. **What existing behavior does this improve?** Controlled Runner protocol campaigns read `RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON`. That variable holds the cell limit, a price table with a review expiry, a pinned lockfile digest, and self-hashed bound proofs. **Current behavior** A campaign with more cells than `maxCells` fails before provider work. Expired prices also fail before provider work. Any change to the lockfile needs a new pin. A one-cell policy made all 41 TestOps cases fail as infrastructure, so an operator had to edit the variable by hand before the full run. **Proposed behavior** The workflow derives the policy at preparation time. The cell count comes from the selected catalog. Prices come from a committed table with a version and a source URL. The controller installs the committed frozen lockfile. One optional variable, `RUNNER_PROTOCOL_EVAL_MAX_USD`, can override the committed $400 spend cap. **Reason and benefit** Each piece that was hand-maintained duplicated a check that already exists. The manifest digest binds every allocation. The frozen install binds dependencies to the reviewed commit. Under model-request readiness, the self-hashed bounds never changed admission or accounting. **Breaking changes** The old variable is no longer read. It can stay set or be deleted. Retained manifests that include the old proofs still replay. ## What Changed - Add `packages/paperclip-runner/scripts/controlled-evals/hosted-defaults.json`. It holds the spend cap, duration, route, Claude Sonnet 5 prices, allocation, overhead reservation and trace limits. - Add `hostedPolicy()` in `hosted.mjs`. It derives `maxCells` from the selection. It gives each campaign a validity window from preparation to one hour after its deadline. It applies the optional spend cap override. - Remove `capacityBound` and the policy self-hash from hosted preparation. `manifest.mjs` needs model `provisionalBound` only when readiness is not `model_requests`. - Workflow: stop reading the policy variable. The controller build installs the committed `pnpm-lock.yaml` with `--frozen-lockfile` and fails on any lockfile drift. Preparation checks that the runtime receipt lockfile digest equals the committed lockfile. The catalog limit is now 1 to 512 selected cells. - Keep everything else the same: the spend-cap check before provider work, private S3 retention of the manifest and derived `hosted-policy.json`, receipts, tracing, first-attempt and zero-retry gates, and paid-environment approvals. - Update `hosted.test.mjs`, `portable.test.mjs`, `runner-protocol-eval-workflow-security.test.mjs`, and `docs/runner-protocol-live-evals.md`. ## Verification - `node --test scripts/controlled-evals/hosted.test.mjs`: 8 pass. The new tests show that `maxCells` equals the selection and that the validity window covers the deadline with no committed expiry. They also show that the spend cap override applies. With the committed defaults, 41 cells validate with $339 reserved, and 49 cells fail on the $400 budget. - Negative controls still fail: drift in selection, source or attempt; a bad lock digest; missing tracing; a duration over 4 hours; a malformed or too-low spend cap; and zero cells. - `node --test scripts/runner-protocol-eval-workflow-security.test.mjs`: 20 pass. Controlled admission now passes whether the old variable is absent, still set, or malformed. A real git checkout with a modified `pnpm-lock.yaml` fails the controller guard before install. - `node --test scripts/controlled-evals/controller.test.mjs scripts/controlled-evals/portable.test.mjs`: all pass. - `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check` pass. - `tests/runner-e2e/workflow-security.test.ts` has 5 failures that this change does not cause. They concern `pr-trusted.yml`, `chat-catalog-testops-case.yml` and `runner-full-stack-e2e.yml`. This PR does not touch those files. - I have not run a live controlled campaign on this branch. Controlled mode runs only from the default branch. The current 41-case run uses master and the existing variable. ## Risks - The committed price table has no expiry. A provider price change needs a reviewed edit to the table. The spend cap still limits the maximum spend. - If the committed lockfile does not match the package manifests, the controller build fails before provider work. CI already requires a frozen lockfile. - The spend cap is now $400 by default. It was $339 in the last manual policy. The cap is a reservation limit, not expected spend. The expected spend for 41 cases is about $5–20. - Only ACPX Claude Sonnet 5 has a committed allocation, as in the old variable. A new controlled model needs a committed price entry and allocation. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
45073d97ae
commit
b1d00eeca0
9 files changed
+288
-170
No files matched your search
@@ -71,7 +71,6 @@ jobs:
|
||||
testops_enabled: ${{ steps.mode.outputs.testops_enabled }}
|
||||
accounting_mode: ${{ steps.mode.outputs.mode }}
|
||||
retry_count: ${{ steps.mode.outputs.retries }}
|
||||
controller_lock_sha256: ${{ steps.mode.outputs.controller_lock_sha256 }}
|
||||
test_runner: ${{ steps.runner.outputs.runner }}
|
||||
max_parallel_default: ${{ steps.runner.outputs.max_parallel_default }}
|
||||
max_parallel_limit: ${{ steps.runner.outputs.max_parallel_limit }}
|
||||
@@ -168,7 +167,6 @@ jobs:
|
||||
TESTOPS_REQUESTED: ${{ inputs.report_to_testops || inputs.ALLURE_JOB_RUN_ID != '' }}
|
||||
REQUESTED_ROSTERS: ${{ inputs.rosters }}
|
||||
RETRIES: ${{ inputs.max_infrastructure_retries || 0 }}
|
||||
CONTROLLER_POLICY_JSON: ${{ vars.RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON }}
|
||||
TARGET_SHA: ${{ steps.target.outputs.sha }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
@@ -179,12 +177,6 @@ jobs:
|
||||
test "$TARGET_SHA" = "$GITHUB_SHA"
|
||||
test "$GITHUB_RUN_ATTEMPT" = 1
|
||||
test "$RETRIES" = 0
|
||||
if ! jq -e '.schema == "paperclip.runner-eval.hosted-policy/v1"' <<< "${CONTROLLER_POLICY_JSON:-}" >/dev/null; then
|
||||
echo 'Controlled runs require a reviewed RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON; no provider work started.' >&2
|
||||
exit 1
|
||||
fi
|
||||
lock_sha="$(jq -er '.controllerLockSha256 | select(type == "string" and test("^[a-f0-9]{64}$"))' <<< "$CONTROLLER_POLICY_JSON")"
|
||||
echo "controller_lock_sha256=$lock_sha" >> "$GITHUB_OUTPUT"
|
||||
;;
|
||||
legacy) test "$EVENT_NAME" = workflow_dispatch ;;
|
||||
*) exit 1 ;;
|
||||
@@ -415,15 +407,19 @@ jobs:
|
||||
|
||||
- name: Enforce controlled limits against the selected catalog
|
||||
if: needs.authorize.outputs.accounting_mode == 'controlled'
|
||||
env:
|
||||
CONTROLLER_POLICY_JSON: ${{ vars.RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
jq -e 'all(.cells[]; (.provider == "codex" and (.acpxAgent // "") == "" and .driver == "codex_app_server") or (.provider == "acpx" and .acpxAgent == "claude" and .driver == "acpx_runtime"))' runner-protocol-eval-catalog.json >/dev/null || {
|
||||
echo 'Controlled selection supports native Codex and ACPX Claude only; select their rosters explicitly.' >&2; exit 1;
|
||||
}
|
||||
maximum="$(jq -er '.maxCells | select(type == "number" and floor == . and . > 0 and . <= 512)' <<< "$CONTROLLER_POLICY_JSON")"
|
||||
jq -e --argjson maximum "$maximum" '.cells | length <= $maximum' runner-protocol-eval-catalog.json >/dev/null
|
||||
# Every selected profile needs a committed allocation. No evals code has run in this job yet.
|
||||
jq -e --slurpfile defaults packages/paperclip-runner/scripts/controlled-evals/hosted-defaults.json \
|
||||
'($defaults[0].allocations | map([.provider, (.acpxAgent // ""), .model])) as $allowed | all(.cells[]; [.provider, (.acpxAgent // ""), .model] as $profile | any($allowed[]; . == $profile))' \
|
||||
runner-protocol-eval-catalog.json >/dev/null || {
|
||||
echo 'Controlled selection includes a profile without a committed allocation in hosted-defaults.json.' >&2; exit 1;
|
||||
}
|
||||
# The campaign size is the selected catalog; the spend cap is enforced when the manifest is prepared.
|
||||
jq -e '(.cells | length) >= 1 and (.cells | length) <= 512' runner-protocol-eval-catalog.json >/dev/null
|
||||
|
||||
- name: Require the chat-report renderer before paid execution
|
||||
run: |
|
||||
@@ -549,7 +545,7 @@ jobs:
|
||||
|
||||
build_controller:
|
||||
name: Build trusted controller without private archive access
|
||||
needs: [authorize, target_lock]
|
||||
needs: [authorize]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
@@ -579,26 +575,14 @@ jobs:
|
||||
with:
|
||||
version: 9.15.4
|
||||
|
||||
- name: Download resolved lockfile for review binding
|
||||
if: needs.authorize.outputs.accounting_mode == 'controlled'
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
||||
with:
|
||||
artifact-ids: ${{ needs.target_lock.outputs.artifact_id }}
|
||||
path: ${{ runner.temp }}/runner-controller-lock
|
||||
|
||||
- name: Build portable controller in an unprivileged job
|
||||
if: needs.authorize.outputs.accounting_mode == 'controlled'
|
||||
id: build
|
||||
env:
|
||||
EXPECTED_LOCK_SHA256: ${{ needs.authorize.outputs.controller_lock_sha256 }}
|
||||
RESOLVED_LOCK_SHA256: ${{ needs.target_lock.outputs.lock_sha256 }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
test "$(git rev-parse HEAD)" = "$GITHUB_SHA"
|
||||
test "$RESOLVED_LOCK_SHA256" = "$EXPECTED_LOCK_SHA256"
|
||||
lock="$RUNNER_TEMP/runner-controller-lock/pnpm-lock.yaml"
|
||||
test "$(sha256sum "$lock" | cut -d ' ' -f 1)" = "$EXPECTED_LOCK_SHA256"
|
||||
cp "$lock" pnpm-lock.yaml
|
||||
# The controller installs exactly the committed lockfile of the trusted revision.
|
||||
git diff --quiet HEAD -- pnpm-lock.yaml
|
||||
pnpm install --frozen-lockfile --ignore-scripts --store-dir "$RUNNER_TEMP/runner-controller-pnpm-store"
|
||||
node packages/paperclip-runner/scripts/controlled-evals/build-runtime.mjs \
|
||||
"$RUNNER_TEMP/runner-protocol-runtime" "$GITHUB_SHA" > "$RUNNER_TEMP/controller-runtime-receipt.json"
|
||||
@@ -700,21 +684,27 @@ jobs:
|
||||
|
||||
- name: Verify copied controller before private preparation
|
||||
if: needs.authorize.outputs.accounting_mode == 'controlled'
|
||||
id: controller_lock
|
||||
env:
|
||||
EXPECTED_LOCK_SHA256: ${{ needs.authorize.outputs.controller_lock_sha256 }}
|
||||
RUNTIME_RECEIPT_SHA256: ${{ needs.build_controller.outputs.runtime_receipt_sha256 }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
node packages/paperclip-runner/scripts/controlled-evals/verify-runtime.mjs "$RUNNER_TEMP/runner-protocol-runtime" "$GITHUB_SHA" "$RUNTIME_RECEIPT_SHA256"
|
||||
test "$(jq -r '.inputs["pnpm-lock.yaml"]' "$RUNNER_TEMP/runner-protocol-runtime/runtime.json")" = "$EXPECTED_LOCK_SHA256"
|
||||
test "$(git rev-parse HEAD)" = "$GITHUB_SHA"
|
||||
committed="$(sha256sum pnpm-lock.yaml | cut -d ' ' -f 1)"
|
||||
test "$(jq -r '.inputs["pnpm-lock.yaml"]' "$RUNNER_TEMP/runner-protocol-runtime/runtime.json")" = "$committed"
|
||||
echo "sha256=$committed" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Compile requests and bind the reviewed operating policy
|
||||
- name: Compile requests and derive the committed operating policy
|
||||
if: needs.authorize.outputs.accounting_mode == 'controlled'
|
||||
id: prepare
|
||||
env:
|
||||
PAPERCLIP_PROTOCOL_EVAL_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
PAPERCLIP_PROTOCOL_EVALS_SHA: ${{ needs.authorize.outputs.evals_sha }}
|
||||
CONTROLLER_POLICY_JSON: ${{ vars.RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON }}
|
||||
PAPERCLIP_RUNNER_EVAL_CONTROLLER_LOCK_SHA256: ${{ steps.controller_lock.outputs.sha256 }}
|
||||
PAPERCLIP_RUNNER_EVAL_BRAINTRUST_PROJECT_ID: ${{ vars.RUNNER_E2E_BRAINTRUST_PROJECT_ID }}
|
||||
# Optional campaign spend cap; the committed default applies when unset.
|
||||
RUNNER_PROTOCOL_EVAL_MAX_USD: ${{ vars.RUNNER_PROTOCOL_EVAL_MAX_USD }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
test "$(git rev-parse HEAD)" = "$GITHUB_SHA"
|
||||
@@ -723,12 +713,6 @@ jobs:
|
||||
(cd runner-protocol-build && sha256sum --check runner-protocol-build.tar.gz.sha256)
|
||||
mkdir runner-protocol-build/extracted
|
||||
tar -xzf runner-protocol-build/runner-protocol-build.tar.gz -C runner-protocol-build/extracted
|
||||
policy="$RUNNER_TEMP/runner-controller-policy.json"
|
||||
POLICY_PATH="$policy" python3 - <<'PYTHON'
|
||||
import os
|
||||
with os.fdopen(os.open(os.environ["POLICY_PATH"], os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "w") as target:
|
||||
target.write(os.environ["CONTROLLER_POLICY_JSON"])
|
||||
PYTHON
|
||||
worker="/tmp/paperclip-runner-controlled-${GITHUB_RUN_ID}"
|
||||
python3 .paperclip-evals/evals/paperclip-runner/tools/prepare_controlled_campaign.py \
|
||||
--catalog runner-protocol-catalog/runner-protocol-eval-catalog.json \
|
||||
@@ -739,8 +723,7 @@ jobs:
|
||||
control="$RUNNER_TEMP/runner-protocol-control"
|
||||
node "$RUNNER_TEMP/runner-protocol-runtime/controlled-evals/hosted.mjs" prepare \
|
||||
runner-protocol-catalog/runner-protocol-eval-catalog.json "$RUNNER_TEMP/prepared-requests.json" \
|
||||
"$policy" runner-protocol-build/extracted "$control" > "$RUNNER_TEMP/control-preparation.json"
|
||||
cp "$policy" "$control/hosted-policy.json"
|
||||
runner-protocol-build/extracted "$control" > "$RUNNER_TEMP/control-preparation.json"
|
||||
echo "manifest_sha256=$(jq -r .manifestSha256 "$RUNNER_TEMP/control-preparation.json")" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Obtain private archive credentials for transfer steps only
|
||||
|
||||
Reference in new issue
Block a user