From e4b30163bba629914904b068511b45bafa21d93b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 20:55:18 -0500 Subject: [PATCH] fix(evals): preserve the complete trusted controller artifact inventory (#17943) https://entire.com/gh/paperclipai/-/trails/103/changes/paperclip/103 ## Thinking Path > - Paperclip manages real agent work. > - Controlled Runner evaluations record model spend and traces. > - A trusted build creates one portable controller with a complete file inventory. > - GitHub artifact upload omitted one hidden dependency metadata file. > - The next job correctly refused the incomplete inventory before any paid worker started. > - This change preserves all inventoried bytes without relaxing verification. ## Linked Issues or Issue Description Refs #17889, #17929. **What happened?** [Run 40585138437](https://github.com/paperclipai/paperclip/actions/runs/40585138437) failed at copied controller verification. Its artifact contains 294 files; its signed inventory expects 295. The only missing file is `server/node_modules/protobufjs/node_modules/@protobufjs/pool/.npmignore`. No extra files exist. Every paid worker was skipped and no model request ran. **Expected behavior** The trusted controller artifact preserves its full inventory, including hidden package metadata. The strict verifier rejects any actual change or missing file. **Steps to reproduce** Build the portable runtime, upload the directory with the action's default hidden-file exclusion, then verify the downloaded artifact against the original receipt. ## What Changed - Enable hidden files only on the fresh trusted controller code upload. - Download that exact upload and run a real artifact-service round-trip test before any private preparation. The test fails if the upload drops hidden files again; it tests the actual transport instead of matching a YAML flag. - Keep the strict byte inventory and trusted source/receipt checks. - Document why this code artifact differs from private attempt evidence, which still uses the protected archive. ## Verification - Exact retained artifact comparison: one missing `.npmignore`, zero extra files. - `actionlint .github/workflows/runner-protocol-live-evals.yml` passed. - Protocol workflow security tests: 14 passed. - The artifact-service integration test is explicit-only locally. The controlled workflow supplies the real downloaded artifact, trusted source and receipt digest. - No product runtime, provider routing or grading code changed. CI runs broader checks. Fresh controlled workflow qualification follows merge. ## Risks Hidden files are allowed only in the code-only output of the provider-credential-free build. Raw private evidence and credentials do not enter that upload. The artifact verifier still requires every file and SHA-256 from the trusted receipt. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../runner-controller-artifact-transport.test.mjs | 12 ++++++++++++ .github/workflows/runner-protocol-live-evals.yml | 15 +++++++++++++++ .../docs/runner-protocol-live-evals.md | 6 ++++++ tests/runner-e2e/workflow-security.test.ts | 6 ++++++ 4 files changed, 39 insertions(+) create mode 100644 .github/tests/runner-controller-artifact-transport.test.mjs diff --git a/.github/tests/runner-controller-artifact-transport.test.mjs b/.github/tests/runner-controller-artifact-transport.test.mjs new file mode 100644 index 00000000000..2c144cae80a --- /dev/null +++ b/.github/tests/runner-controller-artifact-transport.test.mjs @@ -0,0 +1,12 @@ +import test from 'node:test'; +import { verifyControllerRuntime } from '../../packages/paperclip-runner/scripts/controlled-evals/verify-runtime.mjs'; + +test('the uploaded controller preserves every inventoried runtime file', { + skip: !process.env.PAPERCLIP_CONTROLLER_TRANSPORT_DIRECTORY, +}, async () => { + await verifyControllerRuntime({ + directory: process.env.PAPERCLIP_CONTROLLER_TRANSPORT_DIRECTORY, + sourceRevision: process.env.GITHUB_SHA, + receiptSha256: process.env.PAPERCLIP_CONTROLLER_RECEIPT_SHA256, + }); +}); diff --git a/.github/workflows/runner-protocol-live-evals.yml b/.github/workflows/runner-protocol-live-evals.yml index 3fd5f41d0d2..ba3bd0d14c0 100644 --- a/.github/workflows/runner-protocol-live-evals.yml +++ b/.github/workflows/runner-protocol-live-evals.yml @@ -513,10 +513,25 @@ jobs: uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: runner-protocol-runtime-${{ github.run_id }}-${{ github.run_attempt }} + include-hidden-files: true path: ${{ runner.temp }}/runner-protocol-runtime/ retention-days: 30 if-no-files-found: error + - name: Download controller artifact for a credential-free transport test + if: needs.authorize.outputs.accounting_mode == 'controlled' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + artifact-ids: ${{ steps.upload.outputs.artifact-id }} + path: ${{ runner.temp }}/runner-controller-roundtrip + + - name: Verify the real uploaded controller inventory before private preparation + if: needs.authorize.outputs.accounting_mode == 'controlled' + env: + PAPERCLIP_CONTROLLER_TRANSPORT_DIRECTORY: ${{ runner.temp }}/runner-controller-roundtrip + PAPERCLIP_CONTROLLER_RECEIPT_SHA256: ${{ steps.build.outputs.runtime_receipt_sha256 }} + run: node --test .github/tests/runner-controller-artifact-transport.test.mjs + prepare_controller: name: Prepare one shared cost and trace manifest needs: [authorize, catalog, build_runner, build_controller] diff --git a/packages/paperclip-runner/docs/runner-protocol-live-evals.md b/packages/paperclip-runner/docs/runner-protocol-live-evals.md index 478eaf869cc..94801d17540 100644 --- a/packages/paperclip-runner/docs/runner-protocol-live-evals.md +++ b/packages/paperclip-runner/docs/runner-protocol-live-evals.md @@ -952,3 +952,9 @@ For this policy version, `operatingReconciliation` is always `pending`; the immutable controller summary has no operating invoice settlement step. Mixed histories retain that warning when any campaign uses the policy. A shared `accountingReadiness` label appears only when every campaign uses it. + + +The trusted controller artifact preserves hidden dependency metadata as well as +code. Its inventory verifier requires every recorded byte, including package +`.npmignore` files. Only the fresh provider-credential-free controller build uses +this upload setting; private attempts still use the protected archive path. diff --git a/tests/runner-e2e/workflow-security.test.ts b/tests/runner-e2e/workflow-security.test.ts index d68df923c6a..698f7e71575 100644 --- a/tests/runner-e2e/workflow-security.test.ts +++ b/tests/runner-e2e/workflow-security.test.ts @@ -981,6 +981,12 @@ it("keeps dependency resolution and shared caches out of Runner protocol archive } const builder = jobs.find(job => job.startsWith(" build_controller:"))!; expect(builder).not.toMatch(/id-token: write|environment:|secrets\./u); + const upload = builder.split(" - name: Retain the trusted controller code for every worker\n")[1]!.split("\n - ")[0]!; + expect(upload).toContain("include-hidden-files: true"); + expect(upload).toContain("id: upload"); + expect(builder).toContain("artifact-ids: ${{ steps.upload.outputs.artifact-id }}"); + expect(builder).toContain("run: node --test .github/tests/runner-controller-artifact-transport.test.mjs"); + expect(builder).toContain("PAPERCLIP_CONTROLLER_RECEIPT_SHA256: ${{ steps.build.outputs.runtime_receipt_sha256 }}"); const preparer = jobs.find(job => job.startsWith(" prepare_controller:"))!; expect(preparer).not.toMatch(/pnpm install/u); expect(preparer.indexOf("verify-runtime.mjs")).toBeLessThan(preparer.indexOf('hosted.mjs" prepare'));