mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-06 10:48:12 +02:00
fix(evals): select Grok subscription protocol credentials explicitly (#13901)
## Thinking Path > - Paperclip manages agent work through shared runner contracts. > - Direct protocol evals qualify provider behavior against a mock control plane. > - Grok supports API keys and company subscription credentials. > - The hosted protocol workflow selected an API key for every Grok cell. > - Product subscription support did not enable subscription protocol runs. > - This change adds explicit subscription selection and checks the recorded authentication mode. ## Linked Issues or Issue Description Refs #13878, #13882, #12618. The direct Grok protocol roster cannot run with subscription authentication through the trusted default-branch workflow. Add an explicit selector while keeping API-key dispatches compatible. Keep the actor allowlist, protected environment, immutable source revisions, and publication gates. ## What Changed - Add `grok_authentication` with `api_key` and `subscription` choices. Keep `api_key` as the compatibility default. - Deliver the protected `GROK_AUTH_JSON` secret only to a subscription-selected Grok cell. Do not provide an API key to that cell. - Read authentication mode from the pinned eval program's actual roster summary. Retain it in the cell, catalog, campaign roster, and result. - Reject missing or mismatched authentication evidence during aggregation. Preserve cell metadata and an allowlisted failure reason before failing a cell, so malformed evidence cannot hide the retained attempt. - Document credential setup, source separation, and temporary-secret cleanup. ## Verification - `node --test packages/paperclip-runner/scripts/runner-protocol-eval-campaign.test.mjs packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`: 34 tests passed. - Validated all 39 Grok cells at eval revision `3213dbec7e8ca1865ea95e6db7e7d34b095eb47a`; every selected cell requests only the subscription credential. Validation made zero provider calls. - Negative coverage rejects invalid selectors and missing or API authentication evidence in an otherwise passing subscription attempt. - `git diff --check` passed. All 53 current-head checks passed; the unchanged callback-drain timing test passed its bounded rerun, and the failed attempt is retained. Greptile reviewed `ecd3dcc0e998f07cf56fcb1f087946f50f388bec` at 5/5 with no remaining findings. - No Docker or broad builds ran on the developer machine. CI performs repository checks on the configured fleet. ## Risks Grok runs require an eval revision that records `authenticationMode` in the roster summary. Missing evidence fails closed. The credential contains account access and refresh tokens; an owner must approve its delivery to the protected environment before a live run. The change adds no PR trigger or authorization bypass. Live subscription protocol qualification remains pending this workflow reaching master. ## Model Used OpenAI GPT-6 through Codex, with tool use and code execution. The exact serving model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
794b09f834
commit
db8f8fe5b7
6 files changed
+198
-8
No files matched your search
@@ -22,6 +22,14 @@ on:
|
||||
description: "Optional lower concurrency (at least 2, no higher than the configured campaign limit)"
|
||||
type: string
|
||||
required: false
|
||||
grok_authentication:
|
||||
description: "Explicit Grok credential source; subscription uses protected GROK_AUTH_JSON"
|
||||
type: choice
|
||||
options:
|
||||
- api_key
|
||||
- subscription
|
||||
default: api_key
|
||||
required: false
|
||||
max_infrastructure_retries:
|
||||
description: "Automatic retries only for explicitly retryable infrastructure failures (0-3)"
|
||||
type: number
|
||||
@@ -256,6 +264,7 @@ jobs:
|
||||
MAX_PARALLEL_LIMIT: ${{ needs.authorize.outputs.max_parallel_limit }}
|
||||
REQUESTED_MAX_PARALLEL: ${{ inputs.max_parallel }}
|
||||
ROSTERS: ${{ inputs.rosters || 'all' }}
|
||||
GROK_AUTHENTICATION: ${{ inputs.grok_authentication || 'api_key' }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if ! [[ "$MAX_PARALLEL" =~ ^[1-9][0-9]*$ ]] || [ "$MAX_PARALLEL" -lt 2 ] || [ "$MAX_PARALLEL" -gt "$MAX_PARALLEL_LIMIT" ]; then
|
||||
@@ -272,6 +281,7 @@ jobs:
|
||||
node packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs catalog \
|
||||
--evals-root .paperclip-evals \
|
||||
--rosters "$ROSTERS" \
|
||||
--grok-authentication "$GROK_AUTHENTICATION" \
|
||||
--campaign-id "gha-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" \
|
||||
--max-parallel "$MAX_PARALLEL" \
|
||||
--output runner-protocol-eval-catalog.json
|
||||
@@ -542,6 +552,7 @@ jobs:
|
||||
ANTHROPIC_API_KEY: ${{ matrix.credentialName == 'ANTHROPIC_API_KEY' && secrets.ANTHROPIC_API_KEY || '' }}
|
||||
OPENROUTER_API_KEY: ${{ matrix.credentialName == 'OPENROUTER_API_KEY' && secrets.OPENROUTER_API_KEY || '' }}
|
||||
XAI_API_KEY: ${{ matrix.credentialName == 'XAI_API_KEY' && secrets.XAI_API_KEY || '' }}
|
||||
PAPERCLIP_ACPX_GROK_AUTH_JSON_SECRET: ${{ matrix.credentialName == 'PAPERCLIP_ACPX_GROK_AUTH_JSON_SECRET' && secrets.GROK_AUTH_JSON || '' }}
|
||||
PAPERCLIP_CLAUDE_MANAGED_PROFILE_ID: ${{ vars.PAPERCLIP_CLAUDE_MANAGED_PROFILE_ID }}
|
||||
PAPERCLIP_CLAUDE_MANAGED_AGENT_ID: ${{ vars.PAPERCLIP_CLAUDE_MANAGED_AGENT_ID }}
|
||||
PAPERCLIP_CLAUDE_MANAGED_AGENT_VERSION: ${{ vars.PAPERCLIP_CLAUDE_MANAGED_AGENT_VERSION }}
|
||||
@@ -582,19 +593,36 @@ jobs:
|
||||
--runner-package runner-protocol-build/extracted/paperclip-runner.tgz \
|
||||
--runnerd runner-protocol-build/extracted/paperclip-runnerd \
|
||||
--runs-root cell-output/runs \
|
||||
--summary-path cell-output/roster-summary.json \
|
||||
--max-infrastructure-retries "$MAX_INFRASTRUCTURE_RETRIES" \
|
||||
--run-id "gha-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${CELL_ID}"
|
||||
status=$?
|
||||
set -e
|
||||
CELL_EXIT_CODE="$status" node --input-type=module <<'NODE'
|
||||
import { writeFileSync } from "node:fs";
|
||||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
let authenticationMode;
|
||||
let authenticationEvidenceFailure;
|
||||
if (["XAI_API_KEY", "PAPERCLIP_ACPX_GROK_AUTH_JSON_SECRET"].includes(process.env.CREDENTIAL_NAME)) {
|
||||
const expected = process.env.CREDENTIAL_NAME === "XAI_API_KEY" ? "api_key" : "subscription";
|
||||
try {
|
||||
const observed = JSON.parse(readFileSync("cell-output/roster-summary.json", "utf8")).authenticationMode;
|
||||
// Never copy arbitrary provider output into cell metadata or errors.
|
||||
if (["api_key", "subscription"].includes(observed)) authenticationMode = observed;
|
||||
if (authenticationMode !== expected) authenticationEvidenceFailure = "grok_authentication_evidence_mismatch";
|
||||
} catch {
|
||||
authenticationEvidenceFailure = "grok_authentication_evidence_unreadable";
|
||||
}
|
||||
}
|
||||
writeFileSync("cell-output/cell.json", `${JSON.stringify({
|
||||
schema: "paperclip.runner-protocol-eval.cell/v1",
|
||||
cellId: process.env.CELL_ID,
|
||||
rosterFile: process.env.ROSTER_FILE,
|
||||
caseId: process.env.CASE_ID,
|
||||
...(authenticationMode ? { authenticationMode } : {}),
|
||||
...(authenticationEvidenceFailure ? { authenticationEvidenceFailure } : {}),
|
||||
exitCode: Number(process.env.CELL_EXIT_CODE),
|
||||
}, null, 2)}\n`, { mode: 0o600 });
|
||||
if (authenticationEvidenceFailure) throw new Error(authenticationEvidenceFailure);
|
||||
NODE
|
||||
exit "$status"
|
||||
|
||||
|
||||
Reference in new issue
Block a user