fix(evals): select Grok subscription protocol credentials explicitly (#13901)

## Thinking Path

> - Paperclip manages agent work through shared runner contracts.
> - Direct protocol evals qualify provider behavior against a mock
control plane.
> - Grok supports API keys and company subscription credentials.
> - The hosted protocol workflow selected an API key for every Grok
cell.
> - Product subscription support did not enable subscription protocol
runs.
> - This change adds explicit subscription selection and checks the
recorded authentication mode.

## Linked Issues or Issue Description

Refs #13878, #13882, #12618.

The direct Grok protocol roster cannot run with subscription
authentication through the trusted default-branch workflow. Add an
explicit selector while keeping API-key dispatches compatible. Keep the
actor allowlist, protected environment, immutable source revisions, and
publication gates.

## What Changed

- Add `grok_authentication` with `api_key` and `subscription` choices.
Keep `api_key` as the compatibility default.
- Deliver the protected `GROK_AUTH_JSON` secret only to a
subscription-selected Grok cell. Do not provide an API key to that cell.
- Read authentication mode from the pinned eval program's actual roster
summary. Retain it in the cell, catalog, campaign roster, and result.
- Reject missing or mismatched authentication evidence during
aggregation. Preserve cell metadata and an allowlisted failure reason
before failing a cell, so malformed evidence cannot hide the retained
attempt.
- Document credential setup, source separation, and temporary-secret
cleanup.

## Verification

- `node --test
packages/paperclip-runner/scripts/runner-protocol-eval-campaign.test.mjs
packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: 34 tests passed.
- Validated all 39 Grok cells at eval revision
`3213dbec7e8ca1865ea95e6db7e7d34b095eb47a`; every selected cell requests
only the subscription credential. Validation made zero provider calls.
- Negative coverage rejects invalid selectors and missing or API
authentication evidence in an otherwise passing subscription attempt.
- `git diff --check` passed. All 53 current-head checks passed; the
unchanged callback-drain timing test passed its bounded rerun, and the
failed attempt is retained. Greptile reviewed
`ecd3dcc0e998f07cf56fcb1f087946f50f388bec` at 5/5 with no remaining
findings.
- No Docker or broad builds ran on the developer machine. CI performs
repository checks on the configured fleet.

## Risks

Grok runs require an eval revision that records `authenticationMode` in
the roster summary. Missing evidence fails closed. The credential
contains account access and refresh tokens; an owner must approve its
delivery to the protected environment before a live run. The change adds
no PR trigger or authorization bypass. Live subscription protocol
qualification remains pending this workflow reaching master.

## Model Used

OpenAI GPT-6 through Codex, with tool use and code execution. The exact
serving model identifier and context-window size are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-09-23 18:43:21 -05:00
1 parent 794b09f834
commit db8f8fe5b7
6 files changed
+198 -8

No files matched your search

@@ -22,6 +22,14 @@ on:
description: "Optional lower concurrency (at least 2, no higher than the configured campaign limit)"
type: string
required: false
grok_authentication:
description: "Explicit Grok credential source; subscription uses protected GROK_AUTH_JSON"
type: choice
options:
- api_key
- subscription
default: api_key
required: false
max_infrastructure_retries:
description: "Automatic retries only for explicitly retryable infrastructure failures (0-3)"
type: number
@@ -256,6 +264,7 @@ jobs:
MAX_PARALLEL_LIMIT: ${{ needs.authorize.outputs.max_parallel_limit }}
REQUESTED_MAX_PARALLEL: ${{ inputs.max_parallel }}
ROSTERS: ${{ inputs.rosters || 'all' }}
GROK_AUTHENTICATION: ${{ inputs.grok_authentication || 'api_key' }}
run: |
set -euo pipefail
if ! [[ "$MAX_PARALLEL" =~ ^[1-9][0-9]*$ ]] || [ "$MAX_PARALLEL" -lt 2 ] || [ "$MAX_PARALLEL" -gt "$MAX_PARALLEL_LIMIT" ]; then
@@ -272,6 +281,7 @@ jobs:
node packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs catalog \
--evals-root .paperclip-evals \
--rosters "$ROSTERS" \
--grok-authentication "$GROK_AUTHENTICATION" \
--campaign-id "gha-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" \
--max-parallel "$MAX_PARALLEL" \
--output runner-protocol-eval-catalog.json
@@ -542,6 +552,7 @@ jobs:
ANTHROPIC_API_KEY: ${{ matrix.credentialName == 'ANTHROPIC_API_KEY' && secrets.ANTHROPIC_API_KEY || '' }}
OPENROUTER_API_KEY: ${{ matrix.credentialName == 'OPENROUTER_API_KEY' && secrets.OPENROUTER_API_KEY || '' }}
XAI_API_KEY: ${{ matrix.credentialName == 'XAI_API_KEY' && secrets.XAI_API_KEY || '' }}
PAPERCLIP_ACPX_GROK_AUTH_JSON_SECRET: ${{ matrix.credentialName == 'PAPERCLIP_ACPX_GROK_AUTH_JSON_SECRET' && secrets.GROK_AUTH_JSON || '' }}
PAPERCLIP_CLAUDE_MANAGED_PROFILE_ID: ${{ vars.PAPERCLIP_CLAUDE_MANAGED_PROFILE_ID }}
PAPERCLIP_CLAUDE_MANAGED_AGENT_ID: ${{ vars.PAPERCLIP_CLAUDE_MANAGED_AGENT_ID }}
PAPERCLIP_CLAUDE_MANAGED_AGENT_VERSION: ${{ vars.PAPERCLIP_CLAUDE_MANAGED_AGENT_VERSION }}
@@ -582,19 +593,36 @@ jobs:
--runner-package runner-protocol-build/extracted/paperclip-runner.tgz \
--runnerd runner-protocol-build/extracted/paperclip-runnerd \
--runs-root cell-output/runs \
--summary-path cell-output/roster-summary.json \
--max-infrastructure-retries "$MAX_INFRASTRUCTURE_RETRIES" \
--run-id "gha-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${CELL_ID}"
status=$?
set -e
CELL_EXIT_CODE="$status" node --input-type=module <<'NODE'
import { writeFileSync } from "node:fs";
import { readFileSync, writeFileSync } from "node:fs";
let authenticationMode;
let authenticationEvidenceFailure;
if (["XAI_API_KEY", "PAPERCLIP_ACPX_GROK_AUTH_JSON_SECRET"].includes(process.env.CREDENTIAL_NAME)) {
const expected = process.env.CREDENTIAL_NAME === "XAI_API_KEY" ? "api_key" : "subscription";
try {
const observed = JSON.parse(readFileSync("cell-output/roster-summary.json", "utf8")).authenticationMode;
// Never copy arbitrary provider output into cell metadata or errors.
if (["api_key", "subscription"].includes(observed)) authenticationMode = observed;
if (authenticationMode !== expected) authenticationEvidenceFailure = "grok_authentication_evidence_mismatch";
} catch {
authenticationEvidenceFailure = "grok_authentication_evidence_unreadable";
}
}
writeFileSync("cell-output/cell.json", `${JSON.stringify({
schema: "paperclip.runner-protocol-eval.cell/v1",
cellId: process.env.CELL_ID,
rosterFile: process.env.ROSTER_FILE,
caseId: process.env.CASE_ID,
...(authenticationMode ? { authenticationMode } : {}),
...(authenticationEvidenceFailure ? { authenticationEvidenceFailure } : {}),
exitCode: Number(process.env.CELL_EXIT_CODE),
}, null, 2)}\n`, { mode: 0o600 });
if (authenticationEvidenceFailure) throw new Error(authenticationEvidenceFailure);
NODE
exit "$status"