diff --git a/doc/evals.md b/doc/evals.md index ed0484532d8..409003efeca 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -10,6 +10,12 @@ and Daytona, with independent account/model attribution. Its registered cells remain pending until their live evidence passes; its API-only scope leaves the subscription and lifecycle gates separate. +The same guide registers the separate explicit-only `hermes-bedrock-connections` +suite for one exact `us-east-1` inference profile on local and Daytona. Its two +cells remain pending. They verify the explicitly selected personal grant and +public region/protocol/model routing before and after native completion, using +a short-lived AWS bearer credential and the existing budget/evidence pipeline. + Paperclip has two live eval families with different questions, owners, and evidence. Choose the family before selecting a model, profile, or case. diff --git a/doc/plans/2026-10-06-hermes-native-runner.md b/doc/plans/2026-10-06-hermes-native-runner.md index af06d6a1c49..d7e6f73a318 100644 --- a/doc/plans/2026-10-06-hermes-native-runner.md +++ b/doc/plans/2026-10-06-hermes-native-runner.md @@ -709,3 +709,31 @@ because GitHub returned HTTP 429 for the pinned source archive. No native fixtur or provider execution occurred. This is retained as a download infrastructure failure; it is not evidence of a closure mismatch. The separate credential-free AWS build continues against its recorded immutable source. + +### Corrected cloud image and Bedrock fixture preparation, 2026-10-07 + +The corrected credential-free AWS build completed successfully in 11.73 minutes +at source `d389750b90787ea5240d0f8f4e92396a1d9d20a1`. Its immutable image digest is +`sha256:5f457b7aed4dc224c77125edb7b17fd1aedc9e7bad0a9ce92d454cecc660e767`. +The independent collected proof verifies fresh Linux closure +`6166fadd24dae41b9fdfd994e6c7bc129e9317ed252b7dabdd777b691ab47771` +and the original frozen dependency-lock SHA +`f5ee14ee77b1dc7771fe455d619c880fc64b1e62e40a15704addc9f7430e5c50`. +No provider credentials or local Docker were used. This proves the corrected +Linux distribution; it does not prove execution-host namespace compatibility, +browser behavior or Daytona. The ledger reserves $19 of the $25 cap after the +additional $2 cloud-build reservation; actual billing remains incomplete. + +The new explicit-only `hermes-bedrock-connections` suite registers two pending +completion cells on local and Daytona. Read-only AWS discovery confirmed the +exact `us.anthropic.claude-haiku-4-5-20251001-v1:0` inference profile is active +in `us-east-1`. The fixture uses an ephemeral region-bound bearer, a public +personal Connections account and its explicitly selected grant. It verifies +the public account's owner, grant, region, protocol, authentication and model +catalog before task creation and after completion, and grades the actual native +run's selected account, grant and exact inference profile. The server inherits +no ambient AWS environment settings. It retains the existing 200-cent company +and agent budget admission and one-attempt policy. All 129 affected fixture and +catalog tests, Product E2E TypeScript, two-cell discovery and diff checks passed. +No Bedrock inference, credential refresh or Daytona execution is claimed from +this fixture preparation. Hermes remains pending qualification. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 8c799882d14..99fe8cce4cb 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -246,6 +246,8 @@ Shell variables take precedence over the local file. The recognized names are: - `KIMI_MODEL_API_KEY` (local-only pending Kimi CLI/ACP profiles) - `XAI_API_KEY` (local Grok API-key profile) - `GROK_AUTH_JSON` (local native Grok subscription profile) +- `GEMINI_API_KEY` (Hermes Google API profile) +- `AWS_BEARER_TOKEN_BEDROCK` (explicit Hermes Bedrock profile; use a short-lived token for the declared region) - `DAYTONA_API_KEY` - `CURSOR_AUTH_TOKEN` (extended Cursor candidate) - `COPILOT_GITHUB_TOKEN` (extended Copilot candidate) @@ -1541,6 +1543,26 @@ protocols, Bedrock, refresh/revocation, images and live controls need their own qualification. The selected candidate metadata drives pre-credential Hermes provisioning and image/provider-pack selection for every Hermes profile. +The separate explicit-only `hermes-bedrock-connections` suite declares two +pending one-turn cells, on local and Daytona, for +`us.anthropic.claude-haiku-4-5-20251001-v1:0` in `us-east-1`. +Supply `AWS_BEARER_TOKEN_BEDROCK` ephemerally to the launcher after verifying +the runtime and AWS identity. This suite creates a personal managed connection +with Bedrock routing and selects its returned grant explicitly; routed accounts +do not become responsible-user provider defaults. Public readback before and +after the task must match the region, protocol, authentication, model catalog +and account owner. The native run must match the selected connection, grant, +owner, Hermes harness and exact inference profile. Both 200-cent budget checks, +the single-attempt rule and normal evidence/cleanup contracts also apply. +This scope proves Bedrock completion only; it does not qualify credential +refresh or make a local AWS run equivalent to Daytona. The paid GitHub workflow +does not provision this short-lived credential; use the explicit CLI path. + +```sh +pnpm test:e2e:runner -- --list --suite hermes-bedrock-connections +pnpm test:e2e:runner -- --id hermes-bedrock-connections.runner-acpx-hermes-bedrock.local.hello-complete --max-automatic-retries 0 +``` + The separate Runner Evals `extended-harnesses` campaign lives in the private `paperclip-evals` repository and grades semantic protocol behavior against the mock control plane. Neither suite substitutes for the other. diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index 2bec73c7782..0f22e2ed362 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -30,6 +30,18 @@ public structured evidence. Treat changes to the workflow, harness, fixture prompts, evidence packager, and publisher as security-sensitive production changes. +## Short-lived Bedrock qualification + +The explicit-only `hermes-bedrock-connections` suite accepts a short-lived, +region-bound `AWS_BEARER_TOKEN_BEDROCK` from the launcher. The browser fixture +posts it through the sensitive public Connections API into encrypted storage; +the agent selects that exact personal grant. The isolated server inherits no +ambient `AWS_*` values, including profile, credential-file paths, static keys, +session tokens or container-role endpoints. The selected bearer joins the +attempt redaction set. Do not persist it in catalog data or evidence, copy an +operator's AWS files, create a long-lived key for this fixture, or widen IAM +permissions. The paid GitHub workflow does not supply this credential. + ## GitHub authorization Set `RUNNER_E2E_ALLOWED_ACTOR_IDS` to a non-empty JSON array of numeric GitHub diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 0940d3ed036..464e575b80a 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -146,10 +146,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 63, 12, 6, 8, 2, 2, 2, 40, 10, 3, 16, 16, 2, 6, 8, 46, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116, + 63, 12, 6, 8, 2, 2, 2, 40, 10, 2, 3, 16, 16, 2, 6, 8, 46, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116, ]); - expect(validateRunnerCatalog()).toHaveLength(718); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(718); + expect(validateRunnerCatalog()).toHaveLength(720); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(720); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index d94a6b276c8..0bb8a714a3a 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -5,7 +5,7 @@ import { nativeCompletionProfile, NATIVE_COMPLETION_BUDGET_CENTS } from "./nativ import { chatConfirmationTasks } from "./chat-cases.js"; import { buildConnectionSuite } from "./connection-cases.js"; import { hiringTemplateTasks, hiringTemplateProfile, hiringTemplateDefinitionDigest } from "./hiring-template-cases.js"; -import { HERMES_API_CONNECTION_BUDGET_CENTS, hermesApiConnectionChoices, hermesApiConnectionDefinitionDigest } from "./hermes-api-connections.js"; +import { HERMES_API_CONNECTION_BUDGET_CENTS, hermesApiConnectionChoices, hermesApiConnectionDefinitionDigest, hermesBedrockConnectionChoice } from "./hermes-api-connections.js"; import { nativeActiveStopTasks } from "./native-active-stop-tasks.js"; import { cursorNativeTasks } from "./cursor-native-cases.js"; import { instructionPersistenceTask } from "./instruction-persistence.js"; @@ -180,6 +180,7 @@ function nativeProfile(input: { supportedEnvironments?: readonly (typeof ENVIRONMENT_IDS)[number][]; modelQualification?: RunnerProfileFixture["modelQualification"]; ranking?: RunnerProfileFixture["ranking"]; + managedConnectionRouting?: RunnerProfileFixture["managedConnectionRouting"]; }): RunnerProfileFixture { return { ...input, @@ -375,6 +376,15 @@ export const hermesApiConnectionProfiles: readonly RunnerProfileFixture[] = [ })), ]; +/** Region-bound Bedrock credentials enter only through a managed connection. */ +export const hermesBedrockConnectionProfile = nativeProfile({ + id: "runner-acpx-hermes-bedrock", label: "Hermes Bedrock (candidate)", + provider: "acpx", acpxAgent: "hermes", qualificationCandidate: "hermes", + credential: hermesBedrockConnectionChoice.credential, model: hermesBedrockConnectionChoice.model, + managedConnectionRouting: structuredClone(hermesBedrockConnectionChoice.routing), + modelQualification: { source: "candidate_runner_profile", qualificationId: "hermes:v2026.9.24:bedrock:us-east-1:discovery-2026-10-07:pending" }, +}); + /** Narrow legacy ACP lanes used only by the explicit context-integrity matrix. */ export const legacyAcpxProfiles: readonly RunnerProfileFixture[] = [ legacyProfile({ @@ -1215,6 +1225,20 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ coverage: "api-account-native-completion-only", sourceDigest: hermesApiConnectionDefinitionDigest, }, }, + { + id: "hermes-bedrock-connections", label: "Hermes managed Bedrock connection", manualOnly: true, + description: "One native browser task using an explicitly selected personal Bedrock account, exact inference profile and independently verified region.", + groups: ["native"], profiles: [hermesBedrockConnectionProfile], environments: runnerEnvironments, + tasks: [{ ...openRouterBreadthTasks.find(task => task.id === "hello-complete")!, automaticRetryPolicy: "single_attempt" }], + expectedMatrixSize: 2, + definitionMetadata: { + version: 1, qualification: "pending", scheduling: "explicit-only", authenticatedDiscoveryDate: "2026-10-07", + accountMethod: "api_key", accountMode: "delegated", providerTurns: 1, expectedUserSource: "public-account-owner-before-task", + routingSource: "public-selected-account-before-and-after-task", credentialScope: "ephemeral-region-bound-bearer", + budgetMonthlyCents: HERMES_API_CONNECTION_BUDGET_CENTS, maximumAttemptsPerCell: 1, + coverage: "bedrock-account-native-completion-only", sourceDigest: hermesApiConnectionDefinitionDigest, + }, + }, { id: "instruction-persistence", label: "Instruction Persistence", description: "Agent-owned text and binary files round trip through the editor, survive a server restart and fresh task, and synchronize concurrent edits per file with last-sync-wins.", @@ -1595,6 +1619,7 @@ export function suiteDefinitionHash(suite: RunnerSuiteFixture) { id: profile.id, model: profile.model, qualification: profile.modelQualification, + ...(profile.managedConnectionRouting ? { managedConnectionRouting: profile.managedConnectionRouting } : {}), })), environments: suite.environments.map((environment) => ({ id: environment.id, @@ -1691,7 +1716,7 @@ function assertNoRawSecretValues(value: unknown, label: string) { export function validateRunnerCatalog(): MatrixExecution[] { const connectionSuite = runnerSuites.find(suite => suite.id === "provider-connections")!; - const allProfiles = [...connectionSuite.profiles, ...extendedHarnessProfiles, ...hermesApiConnectionProfiles.filter(p => !extendedHarnessProfiles.some(existing => existing.id === p.id)), ...runnerProfiles, ...legacyAcpxProfiles, ...pendingContextIntegrityProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))]; + const allProfiles = [...connectionSuite.profiles, ...extendedHarnessProfiles, hermesBedrockConnectionProfile, ...hermesApiConnectionProfiles.filter(p => !extendedHarnessProfiles.some(existing => existing.id === p.id)), ...runnerProfiles, ...legacyAcpxProfiles, ...pendingContextIntegrityProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))]; const allTasks = [ ...connectionSuite.tasks, extendedHarnessFileTask, diff --git a/tests/runner-e2e/harness-env.ts b/tests/runner-e2e/harness-env.ts index c2e53b6a870..d47c82cd2da 100644 --- a/tests/runner-e2e/harness-env.ts +++ b/tests/runner-e2e/harness-env.ts @@ -25,7 +25,7 @@ const AMBIENT_EXTERNAL_STATE_KEYS = [ "PAPERCLIP_STORAGE_S3_PREFIX", "PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE", ] as const; -const PROVIDER_SECRET_KEY = /^(?:OPENAI|ANTHROPIC|OPENROUTER|DAYTONA|XAI|GROK|CURSOR|COPILOT|GITHUB|GH)(?:_|$)/; +const PROVIDER_SECRET_KEY = /^(?:AWS|OPENAI|ANTHROPIC|OPENROUTER|DAYTONA|XAI|GROK|CURSOR|COPILOT|GITHUB|GH)(?:_|$)/; export function runnerE2EServerControlPaths(temporaryRoot: string) { const controlDirectory = path.join(temporaryRoot, "control"); @@ -116,7 +116,7 @@ export function buildRunnerE2EProcessEnvironment( const agent = execution.profile.qualificationCandidate; if (!agent || agent === "cursor") continue; const admittedSuite = execution.suite.id === "extended-harnesses" - || (execution.suite.id === "hermes-api-connections" && agent === "hermes") + || (["hermes-api-connections", "hermes-bedrock-connections"].includes(execution.suite.id) && agent === "hermes") || execution.suite.id === "rich-acp-warm-continuity"; if (!admittedSuite || !execution.suite.manualOnly) { throw new Error("Candidate qualification requires an explicit provider qualification suite"); diff --git a/tests/runner-e2e/hermes-api-connections.ts b/tests/runner-e2e/hermes-api-connections.ts index 1518e358b92..6c628c9a694 100644 --- a/tests/runner-e2e/hermes-api-connections.ts +++ b/tests/runner-e2e/hermes-api-connections.ts @@ -1,25 +1,41 @@ import { createHash } from "node:crypto"; import { readFileSync } from "node:fs"; +import { isDeepStrictEqual } from "node:util"; +import type { AiProviderRouting } from "../../packages/shared/src/ai-provider-routing.js"; import type { RunnerProfileFixture } from "./types.js"; const record = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; const present = (value: unknown) => typeof value === "string" && value.trim().length > 0; export const HERMES_API_CONNECTION_BUDGET_CENTS = 200; +export const isHermesConnectionSuite = (suiteId: string) => + suiteId === "hermes-api-connections" || suiteId === "hermes-bedrock-connections"; +export const hermesBedrockConnectionChoice = { + credential: "AWS_BEARER_TOKEN_BEDROCK", + model: "us.anthropic.claude-haiku-4-5-20251001-v1:0", + routing: { kind: "bedrock", protocol: "bedrock", auth: "bearer", region: "us-east-1", + models: [{ id: "us.anthropic.claude-haiku-4-5-20251001-v1:0" }] }, +} as const satisfies { credential: RunnerProfileFixture["credential"]; model: string; routing: AiProviderRouting }; /** Establish the intended personal-account user independently, before a paid task. */ export async function captureHermesApiAccountOwner(input: { api: { get(path: string): Promise }; companyId: string; connectionId: string; provider: string; + expectedRouting?: AiProviderRouting; expectedGrantId?: string; }) { const listed = await input.api.get<{ currentUserId: string; connections: { id: string; companyId: string; provider: string; method: string; ownership: string; ownerUserId?: string; status: string; + routing?: AiProviderRouting; grantId?: string; }[] }>(`/api/companies/${input.companyId}/ai-connections`); const matches = listed.connections.filter(connection => connection.id === input.connectionId); const selected = matches[0]; - return { expectedResponsibleUserId: selected?.ownerUserId ?? null, checks: [ + return { expectedResponsibleUserId: selected?.ownerUserId ?? null, + ...(input.expectedRouting ? { routing: { expected: input.expectedRouting, observed: selected?.routing ?? null } } : {}), + checks: [ { id: "selected-account-public-readback", passed: matches.length === 1 && selected?.companyId === input.companyId && selected.provider === input.provider && selected.method === "api_key" && selected.ownership === "personal" && selected.status === "connected" }, { id: "authenticated-caller-known", passed: present(listed.currentUserId) }, { id: "selected-account-owned-by-caller", passed: present(selected?.ownerUserId) && selected?.ownerUserId === listed.currentUserId }, + ...(input.expectedRouting ? [{ id: "selected-account-routing", passed: isDeepStrictEqual(selected?.routing, input.expectedRouting) }] : []), + ...(input.expectedGrantId !== undefined ? [{ id: "selected-account-grant", passed: present(input.expectedGrantId) && selected?.grantId === input.expectedGrantId }] : []), ] }; } @@ -40,6 +56,7 @@ export async function captureHermesApiBudgets(input: { /** Grade public run/account/model metadata; a model's completion claim cannot supply it. */ export function gradeHermesApiConnection(input: { companyId: string; agentId: string; issueId: string; connectionId: string; provider: string; model: string; expectedResponsibleUserId: string; + accountMode?: "responsible_user" | "delegated"; expectedGrantId?: string; runs: readonly { companyId: string; agentId: string; status: string; runtimeMode?: string; issueId?: string | null; responsibleUserId?: string | null; @@ -53,7 +70,8 @@ export function gradeHermesApiConnection(input: { return [ { id: "one-successful-native-run", passed: input.runs.length === 1 && run?.status === "succeeded" && run.runtimeMode === "native" }, { id: "company-agent-task-scope", passed: run?.companyId === input.companyId && run.agentId === input.agentId && run.issueId === input.issueId }, - { id: "selected-managed-api-account", passed: account.connectionId === input.connectionId && account.provider === input.provider && account.method === "api_key" && account.mode === "responsible_user" }, + { id: "selected-managed-api-account", passed: account.connectionId === input.connectionId && account.provider === input.provider && account.method === "api_key" && account.mode === (input.accountMode ?? "responsible_user") }, + ...(input.expectedGrantId !== undefined ? [{ id: "selected-managed-account-grant", passed: present(input.expectedGrantId) && account.grantId === input.expectedGrantId }] : []), { id: "responsible-user-attribution", passed: present(input.expectedResponsibleUserId) && run?.responsibleUserId === input.expectedResponsibleUserId && account.responsibleUserId === input.expectedResponsibleUserId }, { id: "native-hermes-provider", passed: provider.kind === "acpx" && provider.agent === "hermes" && provider.model === input.model }, { id: "exact-native-model", passed: identity.kind === "acpx" && identity.requestedModel === input.model && identity.effectiveModel === input.model }, diff --git a/tests/runner-e2e/hermes-bedrock-connections.test.ts b/tests/runner-e2e/hermes-bedrock-connections.test.ts new file mode 100644 index 00000000000..b46d69bab96 --- /dev/null +++ b/tests/runner-e2e/hermes-bedrock-connections.test.ts @@ -0,0 +1,61 @@ +import { describe, expect, it } from "vitest"; +import { runnerMatrix, runnerSuites, suiteDefinitionHash } from "./catalog.js"; +import { captureHermesApiAccountOwner, gradeHermesApiConnection, hermesBedrockConnectionChoice } from "./hermes-api-connections.js"; +import { buildPaperclipServerEnvironment, buildRunnerE2EProcessEnvironment } from "./harness-env.js"; +import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js"; + +describe("Hermes managed Bedrock qualification", () => { + const suite = runnerSuites.find(value => value.id === "hermes-bedrock-connections")!; + const cells = runnerMatrix.filter(value => value.suite.id === suite.id); + const routing = structuredClone(hermesBedrockConnectionChoice.routing); + const model = hermesBedrockConnectionChoice.model; + + it("registers only two explicit pending single-attempt cells with exact routing", () => { + expect(cells).toHaveLength(2); + expect(new Set(cells.map(value => value.environment.id))).toEqual(new Set(["local", "daytona"])); + expect(suite).toMatchObject({ manualOnly: true, definitionMetadata: { qualification: "pending", accountMode: "delegated", providerTurns: 1, maximumAttemptsPerCell: 1, budgetMonthlyCents: 200 } }); + for (const cell of cells) { + expect(cell.profile).toMatchObject({ credential: "AWS_BEARER_TOKEN_BEDROCK", model, managedConnectionRouting: routing, modelQualification: { source: "candidate_runner_profile" } }); + expect(cell.requiredCredentials).toContain("AWS_BEARER_TOKEN_BEDROCK"); + expect(cell.task).toMatchObject({ id: "hello-complete", expectedRunCount: 1, automaticRetryPolicy: "single_attempt" }); + expect(JSON.parse(buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION!)).toEqual([{ agent: "hermes", model }]); + } + expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(value => value.suite.id === suite.id)).toBe(false); + expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...cells[0]!, suite: { ...suite, manualOnly: false } }])).toThrow("explicit"); + }); + + it("records routing changes in the historical definition digest", () => { + const changed = { ...suite, profiles: suite.profiles.map(profile => ({ ...profile, managedConnectionRouting: { ...routing, region: "us-west-2" } })) }; + expect(suiteDefinitionHash(changed)).not.toBe(suiteDefinitionHash(suite)); + }); + + it("keeps ambient AWS identities and the selected bearer out of the server environment", () => { + const source = { PATH: "/fixture/bin", AWS_PROFILE: "operator", AWS_ACCESS_KEY_ID: "fixture-access", AWS_SECRET_ACCESS_KEY: "fixture-secret", AWS_SESSION_TOKEN: "fixture-session", AWS_BEARER_TOKEN_BEDROCK: "fixture-bearer", AWS_SHARED_CREDENTIALS_FILE: "/operator/credentials", AWS_CONFIG_FILE: "/operator/config", AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/operator/role" }; + const server = buildPaperclipServerEnvironment(buildRunnerE2EProcessEnvironment(source, [cells[0]!])); + expect(server.PATH).toBe(source.PATH); + for (const key of Object.keys(source).filter(key => key.startsWith("AWS_"))) expect(server[key]).toBeUndefined(); + }); + + it.each(["valid", "region", "protocol", "auth", "model", "missing-routing", "grant", "missing-grant"])("rejects incorrect public routing before a paid task: %s", async fault => { + const observed: Record = structuredClone(routing); + if (fault === "region") observed.region = "us-west-2"; + if (fault === "protocol") observed.protocol = "messages"; + if (fault === "auth") observed.auth = "api_key"; + if (fault === "model") observed.models = [{ id: "wrong-model" }]; + const receipt = await captureHermesApiAccountOwner({ companyId: "company", connectionId: "account", provider: "anthropic", expectedRouting: routing, expectedGrantId: "grant", api: { + async get() { return { currentUserId: "user", connections: [{ id: "account", companyId: "company", provider: "anthropic", method: "api_key", ownership: "personal", ownerUserId: "user", status: "connected", routing: fault === "missing-routing" ? undefined : observed, grantId: fault === "missing-grant" ? undefined : fault === "grant" ? "foreign" : "grant" }] } as T; }, + } }); + expect(receipt.checks.every(check => check.passed)).toBe(fault === "valid"); + expect(receipt.routing?.observed).toEqual(fault === "missing-routing" ? null : observed); + }); + + it.each(["valid", "grant", "missing-grant", "mode", "owner", "model"])("grades the actual explicitly selected native account and inference profile: %s", fault => { + const account = { connectionId: "account", grantId: "grant", provider: "anthropic", method: "api_key", mode: "delegated", responsibleUserId: "user" }; + if (fault === "grant") account.grantId = "foreign"; + if (fault === "missing-grant") account.grantId = ""; + if (fault === "mode") account.mode = "responsible_user"; + if (fault === "owner") account.responsibleUserId = "foreign"; + const checks = gradeHermesApiConnection({ companyId: "company", agentId: "agent", issueId: "task", connectionId: "account", expectedGrantId: "grant", provider: "anthropic", model, expectedResponsibleUserId: "user", accountMode: "delegated", runs: [{ companyId: "company", agentId: "agent", issueId: "task", responsibleUserId: "user", status: "succeeded", runtimeMode: "native", contextSnapshot: { aiConnection: account }, runnerProfileJson: { nativeExecutionInput: { provider: { kind: "acpx", agent: "hermes", model } }, sessionCheckpoint: { providerIdentity: { kind: "acpx", requestedModel: model, effectiveModel: fault === "model" ? "wrong-model" : model } } } }] }); + expect(checks.every(check => check.passed)).toBe(fault === "valid"); + }); +}); diff --git a/tests/runner-e2e/live-fixtures.test.ts b/tests/runner-e2e/live-fixtures.test.ts index 604f9d1806e..827d781cdd0 100644 --- a/tests/runner-e2e/live-fixtures.test.ts +++ b/tests/runner-e2e/live-fixtures.test.ts @@ -97,6 +97,7 @@ describe("live runner fixtures", () => { ["runner-acpx-hermes-api-openai", "hermes-api-connections", "hello-complete"], ["runner-acpx-hermes-api-xai", "hermes-api-connections", "hello-complete"], ["runner-acpx-hermes-api-google", "hermes-api-connections", "hello-complete"], + ["runner-acpx-hermes-bedrock", "hermes-bedrock-connections", "hello-complete"], ["runner-codex", "hiring-templates", "hire-coder-template-reuse"], ["runner-acpx-claude", "hiring-templates", "hire-coder-template-reuse"], ["runner-codex", "everyday-workflows", "hire-reuse"], @@ -116,7 +117,7 @@ describe("live runner fixtures", () => { )!; const provider = profile === "runner-acpx-hermes-api-xai" ? "xai" : profile === "runner-acpx-hermes-api-google" ? "google" - : ["runner-acpx-claude", "runner-acpx-hermes-api-anthropic"].includes(profile) ? "anthropic" + : ["runner-acpx-claude", "runner-acpx-hermes-api-anthropic", "runner-acpx-hermes-bedrock"].includes(profile) ? "anthropic" : ["runner-opencode", "runner-acpx-hermes"].includes(profile) ? "openrouter" : "openai"; let connected = false; let companyBody: any; @@ -141,7 +142,8 @@ describe("live runner fixtures", () => { allAgents: false, }); connected = true; - return { connectionId: "managed-account" }; + expect(data.routing).toEqual(execution.profile.managedConnectionRouting); + return { connectionId: "managed-account", grantId: "managed-grant" }; } return { id: "secret" }; }, @@ -159,7 +161,7 @@ describe("live runner fixtures", () => { }, }); expect(connected).toBe(true); - if (suite === "hermes-api-connections") { + if (["hermes-api-connections", "hermes-bedrock-connections"].includes(suite)) { expect(companyBody.budgetMonthlyCents).toBe(200); expect(agentBody.budgetMonthlyCents).toBe(200); } @@ -167,7 +169,9 @@ describe("live runner fixtures", () => { expect(agentBody.runtimeConfig.aiConnection).toEqual({ provider, method: "api_key", - mode: "responsible_user", + ...(execution.profile.managedConnectionRouting + ? { mode: "delegated", connectionId: "managed-account", grantId: "managed-grant" } + : { mode: "responsible_user" }), }); expect((fixtures as any).aiConnection.connectionId).toBe( "managed-account", diff --git a/tests/runner-e2e/live-fixtures.ts b/tests/runner-e2e/live-fixtures.ts index b24b0a47e45..0118e5bb455 100644 --- a/tests/runner-e2e/live-fixtures.ts +++ b/tests/runner-e2e/live-fixtures.ts @@ -1,5 +1,6 @@ import { NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js"; -import { HERMES_API_CONNECTION_BUDGET_CENTS } from "./hermes-api-connections.js"; +import { HERMES_API_CONNECTION_BUDGET_CENTS, isHermesConnectionSuite } from "./hermes-api-connections.js"; +import type { AiConnectionBinding } from "../../packages/shared/src/ai-connections.js"; import path from "node:path"; import { installedReleaseDaytonaPlugin } from "./installed-release.js"; import { isManagedHiringCase } from "./chat-cases.js"; @@ -40,11 +41,7 @@ interface AgentRecord { } interface ManagedAccountFixture { connectionId: string; - binding: { - provider: "openai" | "anthropic" | "openrouter" | "xai" | "google"; - method: "api_key"; - mode: "responsible_user"; - }; + binding: Extract; } interface ProjectRecord { @@ -140,7 +137,7 @@ export async function setupLiveFixtures(input: { return api.post("/api/companies", { name: `Runner E2E ${execution.id} ${input.executionNonce}`, description: "Ephemeral paid full-stack runner acceptance fixture", - budgetMonthlyCents: execution.suite.id === "hermes-api-connections" ? HERMES_API_CONNECTION_BUDGET_CENTS + budgetMonthlyCents: isHermesConnectionSuite(execution.suite.id) ? HERMES_API_CONNECTION_BUDGET_CENTS : ["native-completion", "native-instruction-consolidation", "native-connection-guidance"].includes(execution.suite.id) || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id)) ? NATIVE_COMPLETION_BUDGET_CENTS : execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS @@ -252,7 +249,7 @@ export async function setupLiveFixtures(input: { async setup(resolved) { const company = value(resolved, "company"); const key = execution.profile.credential; - const provider = key === "ANTHROPIC_API_KEY" ? "anthropic" + const provider = key === "ANTHROPIC_API_KEY" || key === "AWS_BEARER_TOKEN_BEDROCK" ? "anthropic" : key === "OPENROUTER_API_KEY" ? "openrouter" : key === "OPENAI_API_KEY" ? "openai" : key === "XAI_API_KEY" ? "xai" @@ -260,21 +257,26 @@ export async function setupLiveFixtures(input: { if (!provider) throw new Error(`Unsupported managed hiring credential ${key}`); const apiKey = input.credentials[key]; if (!apiKey) throw new Error(`Missing credential ${key}`); - const account = await api.postSensitive<{ connectionId: string }>( + const routing = execution.profile.managedConnectionRouting; + const account = await api.postSensitive<{ connectionId: string; grantId: string }>( `/api/companies/${company.id}/ai-connections`, { provider, method: "api_key", name: `Runner E2E account ${input.executionNonce}`, ownership: "personal", + ...(routing ? { routing } : {}), apiKey, agentIds: [], allAgents: false, }, ); + if (routing && !account.grantId) throw new Error("Routed managed connection did not return its personal grant"); return { connectionId: account.connectionId, - binding: { provider, method: "api_key", mode: "responsible_user" }, + binding: routing + ? { provider, method: "api_key", mode: "delegated", connectionId: account.connectionId, grantId: account.grantId } + : { provider, method: "api_key", mode: "responsible_user" }, }; }, }); @@ -304,7 +306,7 @@ export async function setupLiveFixtures(input: { || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id))) { agent.budgetMonthlyCents = 1_000; } - if (execution.suite.id === "hermes-api-connections") agent.budgetMonthlyCents = HERMES_API_CONNECTION_BUDGET_CENTS; + if (isHermesConnectionSuite(execution.suite.id)) agent.budgetMonthlyCents = HERMES_API_CONNECTION_BUDGET_CENTS; if (managedHiring) { const account = value(resolved, "ai-connection"); const config = agent.adapterConfig as Record; diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index bad6b48143c..3c706f1765d 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1,5 +1,5 @@ import { runPlanTaskFlow } from "./plan-task-flow.js"; -import { captureHermesApiAccountOwner, captureHermesApiBudgets, gradeHermesApiConnection } from "./hermes-api-connections.js"; +import { captureHermesApiAccountOwner, captureHermesApiBudgets, gradeHermesApiConnection, isHermesConnectionSuite } from "./hermes-api-connections.js"; import { assertNativeCompletionSelection, NATIVE_COMPLETION_PREFLIGHT_ENV, verifyNativeCompletionPreflight } from "./native-completion-admission.js"; import { assertNativeInstructionSelection, verifyNativeInstructionPreflight, NATIVE_INSTRUCTION_PREFLIGHT_ENV, NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_DEFAULT_SHA256 } from "./native-instruction-consolidation.js"; import { captureNativeDefault, gradeNativeDefault, nativeCompletionWorkspaceDigest } from "./native-completion-defaults.js"; @@ -895,10 +895,12 @@ for (const execution of executions) { daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE, }); - if (execution.suite.id === "hermes-api-connections") { + if (isHermesConnectionSuite(execution.suite.id)) { if (!fixtures.aiConnection) throw new Error("Hermes API account is missing before task creation"); hermesApiAccountOwner = await captureHermesApiAccountOwner({ api, companyId: fixtures.company.id, - connectionId: fixtures.aiConnection.connectionId, provider: fixtures.aiConnection.binding.provider }); + connectionId: fixtures.aiConnection.connectionId, provider: fixtures.aiConnection.binding.provider, + ...(execution.profile.managedConnectionRouting ? { expectedRouting: execution.profile.managedConnectionRouting, + expectedGrantId: fixtures.aiConnection.binding.mode === "delegated" ? fixtures.aiConnection.binding.grantId : "" } : {}) }); await writeSanitizedJson(snapshotsDir, "hermes-api-owner-before-execution.json", hermesApiAccountOwner, secrets); if (!hermesApiAccountOwner.checks.every(check => check.passed)) throw new Error("Hermes API owner failed admission before task creation"); const receipt = await captureHermesApiBudgets({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id }); @@ -2877,11 +2879,22 @@ for (const execution of executions) { } } } - if (execution.suite.id === "hermes-api-connections") { + if (isHermesConnectionSuite(execution.suite.id)) { if (!fixtures?.aiConnection || !issue?.id || !hermesApiAccountOwner?.expectedResponsibleUserId) throw new Error("Hermes connection qualification is missing its selected account, expected user or task"); const checks = gradeHermesApiConnection({ companyId: fixtures.company.id, agentId: fixtures.agent.id, issueId: issue.id, connectionId: fixtures.aiConnection.connectionId, provider: fixtures.aiConnection.binding.provider, - expectedResponsibleUserId: hermesApiAccountOwner.expectedResponsibleUserId, model: execution.profile.model, runs: selectedRuns }); + expectedResponsibleUserId: hermesApiAccountOwner.expectedResponsibleUserId, model: execution.profile.model, runs: selectedRuns, + accountMode: fixtures.aiConnection.binding.mode, + ...(execution.profile.managedConnectionRouting ? { expectedGrantId: fixtures.aiConnection.binding.mode === "delegated" ? fixtures.aiConnection.binding.grantId : "" } : {}) }); + if (execution.profile.managedConnectionRouting) { + const after = await captureHermesApiAccountOwner({ api, companyId: fixtures.company.id, + connectionId: fixtures.aiConnection.connectionId, provider: fixtures.aiConnection.binding.provider, + expectedRouting: execution.profile.managedConnectionRouting, + expectedGrantId: fixtures.aiConnection.binding.mode === "delegated" ? fixtures.aiConnection.binding.grantId : "" }); + checks.push(...after.checks.map(check => ({ ...check, id: `after-task-${check.id}` })), + { id: "account-owner-preserved", passed: after.expectedResponsibleUserId === hermesApiAccountOwner.expectedResponsibleUserId }); + await writeSanitizedJson(snapshotsDir, "hermes-api-owner-after-execution.json", after, secrets); + } matcherResults.push(...checks.map(check => ({ matcher: { kind: "json_path" as const, path: `hermesConnection.${check.id}`, expected: true }, passed: check.passed, detail: "Public native run metadata must match the selected managed API account and exact model." }))); await writeSanitizedJson(snapshotsDir, "hermes-api-connection.json", { checks }, secrets); expect(checks.every(check => check.passed), "Hermes managed account and native model attribution").toBe(true); diff --git a/tests/runner-e2e/types.ts b/tests/runner-e2e/types.ts index ea2ce460e1d..ce299a0ecb9 100644 --- a/tests/runner-e2e/types.ts +++ b/tests/runner-e2e/types.ts @@ -1,3 +1,5 @@ +import type { AiProviderRouting } from "../../packages/shared/src/ai-provider-routing.js"; + export const CREDENTIAL_NAMES = [ "OPENAI_API_KEY", "ANTHROPIC_API_KEY", @@ -6,6 +8,7 @@ export const CREDENTIAL_NAMES = [ "XAI_API_KEY", "GROK_AUTH_JSON", "GEMINI_API_KEY", + "AWS_BEARER_TOKEN_BEDROCK", "DAYTONA_API_KEY", "CURSOR_AUTH_TOKEN", "COPILOT_GITHUB_TOKEN", @@ -87,6 +90,8 @@ export interface RunnerProfileFixture { sourceUrl: string; }; credential: Exclude; + /** Value-free routing saved on an explicitly selected managed connection. */ + managedConnectionRouting?: AiProviderRouting; supportedEnvironments: readonly RunnerEnvironmentId[]; expectedRuntimeMode: RunnerGeneration; expectedRuntimeMetadata: {