From 3f91d58dd5d8666078efa465b754f5baced46ef9 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 26 Oct 2026 10:17:14 -0500 Subject: [PATCH] fix(ci): provide Copilot auth only to protected paid eval cells (#17743) ## Thinking Path > - Paperclip manages AI agents and their work. > - Paid runner evals test real provider integrations. > - Copilot cells use the saved-token connection path. > - The two trusted workflows did not pass the Copilot credential to those cells. > - This PR supplies it only when the selected cell needs it. > - Existing environment and actor gates protect access in this public repository. ## Linked Issues or Issue Description Related: #15705 and #15706. A search found no open PR that adds this workflow credential binding. **Describe the bug** Copilot Product and direct-protocol eval cells require `COPILOT_GITHUB_TOKEN`. The protected paid workflows do not currently pass that environment secret to the selected execution step. **Expected behavior** Only a selected Copilot cell receives this credential. Ordinary PR, build, catalog, image and reporting jobs receive none. **Steps to reproduce** Dispatch the trusted full-stack workflow for one Copilot cell. Credential validation fails before the task starts, although the protected environment can hold a valid token. **Environment** GitHub Actions in the public `paperclipai/paperclip` repository, with the reviewed workflow on `master` and the `runner-e2e-paid` environment. ## What Changed - Pass `COPILOT_GITHUB_TOKEN` in the paid Product and direct-protocol execution steps, only when `matrix.credentialName` selects it. - Add the missing Copilot protocol-catalog credential mapping and verify it reaches the selected execution matrix. - Add security checks for the selected-cell binding and bounded protected-job scope. Extend the scan that rejects provider credentials in unrelated workflows. - Document the Copilot credential scope. No secret values, account data or private test evidence are in this PR. ## Verification - Product workflow security: 16 tests passed on this head. - Protocol catalog and workflow security: 21 tests passed, including the Copilot matrix regression reproduced red before its fix. - Current-head required `ci / verify` and `ci / e2e` are green. Greptile is 5/5 with zero unresolved review threads. - The first CI attempt retained runner-shutdown failures; one failed-job retry passed on the unchanged head. Both attempts remain in [the CI run](https://github.com/paperclipai/paperclip/actions/runs/40342429106). - `git diff --check` passed. - A selected-credential scan found zero matching secret values in the seven changed files. - The environment setting was separately checked: `master` only, required approval from existing allowlisted code owners, and administrator bypass disabled. The new secret exists only at environment scope; no repository-wide duplicate was created. - The packaged Copilot runtime verified the selected token and the explicit model without a model prompt. A fresh local UI journey then created an agent, completed its first task and verified the downloaded artifact and cleanup. ## Risks The selected paid cells gain access to one additional environment credential. Missing credentials still fail closed. Default-branch checks, initiating/rerunning actor checks, protected environment access and per-cell credential selection remain in place. This PR adds no trigger and gives no credential to builds or ordinary PR checks. ## Model Used OpenAI Codex, based on GPT-6, with repository tools and browser regression tests. The exact runtime model identifier is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .github/workflows/runner-full-stack-e2e.yml | 1 + .../workflows/runner-protocol-live-evals.yml | 1 + .../scripts/runner-protocol-eval-campaign.mjs | 1 + .../runner-protocol-eval-campaign.test.mjs | 24 +++++++++++++++++ ...r-protocol-eval-workflow-security.test.mjs | 8 ++++++ tests/runner-e2e/SECURITY.md | 8 ++++-- tests/runner-e2e/workflow-security.test.ts | 27 ++++++++++++++++++- 7 files changed, 67 insertions(+), 3 deletions(-) diff --git a/.github/workflows/runner-full-stack-e2e.yml b/.github/workflows/runner-full-stack-e2e.yml index 99cdaf52326..bc285f4b695 100644 --- a/.github/workflows/runner-full-stack-e2e.yml +++ b/.github/workflows/runner-full-stack-e2e.yml @@ -1184,6 +1184,7 @@ jobs: OPENROUTER_API_KEY: ${{ matrix.credentialName == 'OPENROUTER_API_KEY' && secrets.OPENROUTER_API_KEY || '' }} XAI_API_KEY: ${{ matrix.credentialName == 'XAI_API_KEY' && secrets.XAI_API_KEY || '' }} GROK_AUTH_JSON: ${{ matrix.credentialName == 'GROK_AUTH_JSON' && secrets.GROK_AUTH_JSON || '' }} + COPILOT_GITHUB_TOKEN: ${{ matrix.credentialName == 'COPILOT_GITHUB_TOKEN' && secrets.COPILOT_GITHUB_TOKEN || '' }} DAYTONA_API_KEY: ${{ matrix.environmentId == 'daytona' && secrets.DAYTONA_API_KEY || '' }} PAPERCLIP_E2E_DAYTONA_IMAGE: ${{ needs.daytona_image.outputs.image }} PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH: ${{ github.workspace }}/packages/paperclip-runner/provider-pack diff --git a/.github/workflows/runner-protocol-live-evals.yml b/.github/workflows/runner-protocol-live-evals.yml index 850d30ba5c7..41c997c2939 100644 --- a/.github/workflows/runner-protocol-live-evals.yml +++ b/.github/workflows/runner-protocol-live-evals.yml @@ -553,6 +553,7 @@ jobs: OPENROUTER_API_KEY: ${{ matrix.credentialName == 'OPENROUTER_API_KEY' && secrets.OPENROUTER_API_KEY || '' }} XAI_API_KEY: ${{ matrix.credentialName == 'XAI_API_KEY' && secrets.XAI_API_KEY || '' }} PAPERCLIP_ACPX_GROK_AUTH_JSON_SECRET: ${{ matrix.credentialName == 'PAPERCLIP_ACPX_GROK_AUTH_JSON_SECRET' && secrets.GROK_AUTH_JSON || '' }} + COPILOT_GITHUB_TOKEN: ${{ matrix.credentialName == 'COPILOT_GITHUB_TOKEN' && secrets.COPILOT_GITHUB_TOKEN || '' }} PAPERCLIP_CLAUDE_MANAGED_PROFILE_ID: ${{ vars.PAPERCLIP_CLAUDE_MANAGED_PROFILE_ID }} PAPERCLIP_CLAUDE_MANAGED_AGENT_ID: ${{ vars.PAPERCLIP_CLAUDE_MANAGED_AGENT_ID }} PAPERCLIP_CLAUDE_MANAGED_AGENT_VERSION: ${{ vars.PAPERCLIP_CLAUDE_MANAGED_AGENT_VERSION }} diff --git a/packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs b/packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs index f410814d878..7d4015e7a21 100644 --- a/packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs +++ b/packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs @@ -72,6 +72,7 @@ export function credentialForConfig(config, grokAuthenticationMode = "api_key") if (config.provider === "acpx") { if (config.acpxAgent === "pi") return "OPENROUTER_API_KEY"; if (config.acpxAgent === "claude") return "ANTHROPIC_API_KEY"; + if (config.acpxAgent === "copilot") return "COPILOT_GITHUB_TOKEN"; if (config.acpxAgent === "grok") { return validateGrokAuthenticationMode(grokAuthenticationMode) === "subscription" ? "PAPERCLIP_ACPX_GROK_AUTH_JSON_SECRET" diff --git a/packages/paperclip-runner/scripts/runner-protocol-eval-campaign.test.mjs b/packages/paperclip-runner/scripts/runner-protocol-eval-campaign.test.mjs index 459f1fa3aa0..154d22a26af 100644 --- a/packages/paperclip-runner/scripts/runner-protocol-eval-campaign.test.mjs +++ b/packages/paperclip-runner/scripts/runner-protocol-eval-campaign.test.mjs @@ -104,6 +104,30 @@ test("maps every qualified driver to one explicit credential boundary", () => { ); }); +test("selects the Copilot token for the exact protocol roster and execution matrix", async () => { + const { root, program, config } = await fixture(); + await writeFile(join(program, "configs/live-opencode-model.json"), JSON.stringify({ + ...config, + provider: "acpx", + driver: "acpx_runtime", + acpxAgent: "copilot", + model: "gpt-5.6-luna", + })); + const catalog = await buildProtocolEvalCatalog({ + evalsRoot: root, + campaignId: "copilot-credential-01", + rosterSelection: "live-opencode-model.json", + }); + assert.equal(catalog.rosters.length, 1); + assert.equal(catalog.rosters[0].credentialName, "COPILOT_GITHUB_TOKEN"); + assert.equal(catalog.cells.length, 1); + assert.equal(catalog.cells[0].credentialName, "COPILOT_GITHUB_TOKEN"); + assert.deepEqual( + catalog.matrices.flatMap(matrix => matrix.include).map(cell => cell.credentialName), + ["COPILOT_GITHUB_TOKEN"], + ); +}); + async function grokFixture() { const value = await fixture(); value.config = { diff --git a/packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs b/packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs index 210b7954b53..6bf6e0f9e75 100644 --- a/packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs +++ b/packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs @@ -38,6 +38,14 @@ test("direct live eval workflow keeps paid execution behind stable actor authori assert.match(workflow, /github\.triggering_actor/u); assert.match(workflow, /refs\/heads\/\$DEFAULT_BRANCH/u); assert.match(workflow, /Reauthorize paid execution before provider access/u); + const copilotCredential = "COPILOT_GITHUB_TOKEN: ${{ matrix.credentialName == 'COPILOT_GITHUB_TOKEN' && secrets.COPILOT_GITHUB_TOKEN || '' }}"; + assert.equal(workflow.split(copilotCredential).length - 1, 1); + const paid = workflow.slice(workflow.indexOf(" eval_shard_0:"), workflow.indexOf(" eval_shard_1:")); + assert.ok(paid.includes("name: runner-e2e-paid")); + assert.ok(paid.includes(copilotCredential)); + const reauthorizationIndex = paid.indexOf("Reauthorize paid execution before provider access"); + assert.ok(reauthorizationIndex >= 0); + assert.ok(reauthorizationIndex < paid.indexOf(copilotCredential)); assert.match( workflow, /Reauthorize paid execution before provider access[\s\S]*actions\/checkout@[0-9a-f]{40}[\s\S]*Run one immutable direct protocol cell/u, diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index 555a6673b20..c9ad6c6d91d 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -129,8 +129,12 @@ and account identifiers inside its JSON join the evidence redaction set. The subscription suite remains manual-only and never falls back to `XAI_API_KEY`. Create `runner-e2e-paid`, restrict deployments to the default branch, and put -only `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `OPENROUTER_API_KEY`, `XAI_API_KEY`, and -`DAYTONA_API_KEY` in it. `KIMI_MODEL_API_KEY` remains a local-only catalog name +only provider credentials needed by the selected paid suites, including +`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `OPENROUTER_API_KEY`, `XAI_API_KEY`, +`COPILOT_GITHUB_TOKEN` and `DAYTONA_API_KEY` in it. Copilot is passed only to +cells selecting `credentialName: COPILOT_GITHUB_TOKEN`, in the paid Product or +direct-protocol execution step. Never expose it to ordinary PR, build, catalog, +image or report jobs. `KIMI_MODEL_API_KEY` remains a local-only catalog name for pending explicit profiles and is intentionally absent from the workflow environment. Product E2E pending-profile admission remains gated separately. Do not duplicate these credentials as repository- or diff --git a/tests/runner-e2e/workflow-security.test.ts b/tests/runner-e2e/workflow-security.test.ts index e6c8b1c16e4..91bcd45830f 100644 --- a/tests/runner-e2e/workflow-security.test.ts +++ b/tests/runner-e2e/workflow-security.test.ts @@ -15,6 +15,30 @@ const everydayOracleImage = "python@sha256:9d2e5553305c7c7b0097999bb17187c69b921ccd6bc9d40e4bb5ebe652c00285"; describe("public repository paid workflow security", () => { + it("exposes Copilot only to selected cells in the protected, reauthorized paid jobs", async () => { + for (const [filename, jobName, nextJobName, stepName] of [ + ["runner-full-stack-e2e.yml", "test", "report", "Run paid cell"], + ["runner-protocol-live-evals.yml", "eval_shard_0", "eval_shard_1", "Run one immutable direct protocol cell"], + ] as const) { + const workflow = await readFile(path.join(repositoryRoot, ".github/workflows", filename), "utf8"); + const credentialSteps = workflow.split(/\n(?= {6}- )/u).filter(step => step.includes("secrets.COPILOT_GITHUB_TOKEN")); + expect(credentialSteps, filename).toHaveLength(1); + const credentialStep = credentialSteps[0]!; + expect(credentialStep, filename).toContain(`name: ${stepName}`); + expect(credentialStep, filename).toContain("COPILOT_GITHUB_TOKEN: ${{ matrix.credentialName == 'COPILOT_GITHUB_TOKEN' && secrets.COPILOT_GITHUB_TOKEN || '' }}"); + const jobStart = workflow.indexOf(`\n ${jobName}:`); + const jobEnd = workflow.indexOf(`\n ${nextJobName}:`, jobStart + 1); + expect(jobStart, filename).toBeGreaterThanOrEqual(0); + expect(jobEnd, filename).toBeGreaterThan(jobStart); + const paidJob = workflow.slice(jobStart, jobEnd); + expect(paidJob, filename).toContain("name: runner-e2e-paid"); + expect(paidJob, filename).toContain(credentialStep); + const reauthorizationIndex = paidJob.indexOf("Reauthorize paid execution before provider access"); + expect(reauthorizationIndex, filename).toBeGreaterThanOrEqual(0); + expect(reauthorizationIndex, filename).toBeLessThan(paidJob.indexOf("secrets.COPILOT_GITHUB_TOKEN")); + } + }); + it("keeps the manual hosted Linux image build credential-free and pins the authorized target", async () => { const workflow = await readFile(path.join(repositoryRoot, ".github/workflows/docker-runner-check.yml"), "utf8"); const manual = workflow.slice(workflow.indexOf(" authorize_manual:")); @@ -549,6 +573,7 @@ describe("public repository paid workflow security", () => { OPENROUTER_API_KEY: "matrix.credentialName == 'OPENROUTER_API_KEY'", XAI_API_KEY: "matrix.credentialName == 'XAI_API_KEY'", GROK_AUTH_JSON: "matrix.credentialName == 'GROK_AUTH_JSON'", + COPILOT_GITHUB_TOKEN: "matrix.credentialName == 'COPILOT_GITHUB_TOKEN'", DAYTONA_API_KEY: "matrix.environmentId == 'daytona'", })) { expect(fullStack).toContain( @@ -576,7 +601,7 @@ describe("public repository paid workflow security", () => { ); const providerSecretReferences = [ ...contents.matchAll( - /secrets(?:\.(?:OPENAI_API_KEY|ANTHROPIC_API_KEY|OPENROUTER_API_KEY|XAI_API_KEY|GROK_AUTH_JSON|DAYTONA_API_KEY)\b|\[['"](?:OPENAI_API_KEY|ANTHROPIC_API_KEY|OPENROUTER_API_KEY|XAI_API_KEY|GROK_AUTH_JSON|DAYTONA_API_KEY)['"]\])/g, + /secrets(?:\.(?:OPENAI_API_KEY|ANTHROPIC_API_KEY|OPENROUTER_API_KEY|XAI_API_KEY|GROK_AUTH_JSON|DAYTONA_API_KEY|COPILOT_GITHUB_TOKEN)\b|\[['"](?:OPENAI_API_KEY|ANTHROPIC_API_KEY|OPENROUTER_API_KEY|XAI_API_KEY|GROK_AUTH_JSON|DAYTONA_API_KEY|COPILOT_GITHUB_TOKEN)['"]\])/g, ), ]; if (providerSecretReferences.length > 0) {