From e60a921656fc10319c2f850883202b33ea8a8ca5 Mon Sep 17 00:00:00 2001 From: Dotta Date: Wed, 7 Oct 2026 16:31:16 -0500 Subject: [PATCH] test(hermes): bound API qualification before task dispatch Require one attempt and public readback of scoped 200-cent company and agent budgets before the paid API task is created. Reject unlimited or foreign budget records. Preserve unknown usage as unknown. Record all five successful paid Linux browser workflows and verified temporary AWS host/access cleanup. The complete support suite passes 1,833 Vitest tests and 128 Node assertions, with one skipped Vitest test. Product E2E TypeScript and diff checks pass. Co-Authored-By: Paperclip --- doc/plans/2026-10-06-hermes-native-runner.md | 23 +++++++++++++++---- tests/runner-e2e/FIXTURES.md | 2 ++ tests/runner-e2e/README.md | 3 +++ tests/runner-e2e/catalog.ts | 5 ++-- .../runner-e2e/hermes-api-connections.test.ts | 19 ++++++++++++--- tests/runner-e2e/hermes-api-connections.ts | 15 ++++++++++++ tests/runner-e2e/live-fixtures.test.ts | 7 +++++- tests/runner-e2e/live-fixtures.ts | 5 +++- tests/runner-e2e/runner.spec.ts | 7 +++++- 9 files changed, 74 insertions(+), 12 deletions(-) diff --git a/doc/plans/2026-10-06-hermes-native-runner.md b/doc/plans/2026-10-06-hermes-native-runner.md index cdfc5450ec4..61d35488328 100644 --- a/doc/plans/2026-10-06-hermes-native-runner.md +++ b/doc/plans/2026-10-06-hermes-native-runner.md @@ -537,7 +537,7 @@ explicitly privileged cloud container: protected files remained hidden, assigned skills remained read-only, and allowed workspace writes succeeded. This establishes this EC2 host's compatibility; it does not qualify Daytona. -Three real Linux Product E2E cells passed through Chromium, an isolated Paperclip +All five core Linux Product E2E cells passed through Chromium, an isolated Paperclip server/database, Runnerd, ACPX, native Hermes and managed OpenRouter model `deepseek/deepseek-v4-flash-0731`: @@ -546,16 +546,21 @@ server/database, Runnerd, ACPX, native Hermes and managed OpenRouter model | `hello-complete` | `71743c4f328f` | 72.351 seconds | 1 | Passed | | `question-resume-complete` | `703f08761be6` | 133.251 seconds | 2 | Passed | | `plan-approve-complete` | `8662152371fd` | 140.890 seconds | 2 | Passed | +| `structured-question-restart-resume` | `87eaa1dd7deb` | 137.277 seconds | 2 | Passed | +| `file-edit-validate` | `758fe67e2613` | 132.001 seconds | 1 | Passed | -All three results bind the exact image-source commit above. The catalog environment +All five results bind the exact image-source commit above. The catalog environment is `local`, with a separately recorded AWS EC2 Linux execution host; these are not Daytona/remote-target passes. The first case independently verified Done, one final answer and a successful native run, with a reviewed final screenshot. The second retained the question and final screenshots and verified the answer and continuation. The third verified plan approval, continuation and completion, -with two retained screenshots. Runtime cost is not metered by that local catalog; external +with two reviewed screenshots. The fourth verified retained question state and +conversation continuation after a real controller restart, with three screenshots. +The file case checked the actual final file bytes and command validation, with +one screenshot. Runtime cost is not metered by that local catalog; external EC2 cost is separate. Native token usage was available for the completion run -and one of the two runs in each question/approval workflow. Model cost remains unpriced/incomplete, and +and one of the two runs in each question/approval/restart workflow. Model cost remains unpriced/incomplete, and shared-key aggregate budget readings do not establish exact per-run cost. The initial cloud browser attempt failed before server bootstrap because the @@ -570,8 +575,18 @@ no credentials or raw provider traces. The new explicit-only `hermes-api-connections` matrix declares ten API-account completion cells with independently graded native account/model attribution. +Each permits one attempt and configures 200-cent company and agent budgets. +Public readback must verify both budgets and their scope before task creation; +unpriced usage stays unknown and does not become an exact billing receipt. Authenticated OpenAI, Anthropic, xAI and Google catalog reads succeeded without inference. Those discoveries and the new fixture calibration do not constitute live provider qualification. The complete original release gates remain open: the rest of the live controls, attachments, state/routines, credential lifecycle, subscriptions/custom protocols/Bedrock and actual Daytona proof are still required. + +The final API fixture support suite passed 1,833 Vitest tests (one skipped) and +all 128 companion Node assertions. Product E2E TypeScript and `git diff --check` +passed. The five verified encrypted cloud proofs are preserved; the disposable +EC2 instance was terminated and its unused role, instance profile and security +group removed. No local Docker was used. These fixture refinements require +fresh current-head CI and review; they do not complete release qualification. diff --git a/tests/runner-e2e/FIXTURES.md b/tests/runner-e2e/FIXTURES.md index 3fdd12780fd..6510fe90244 100644 --- a/tests/runner-e2e/FIXTURES.md +++ b/tests/runner-e2e/FIXTURES.md @@ -301,6 +301,8 @@ responsible user. Calibration rejects foreign accounts/companies/tasks/users, wrong providers/methods/models/harnesses, missing evidence and extra runs. The suite's definition includes the account/runner harness source digest. Its API-only scope does not substitute for subscription or lifecycle qualification. +The task permits one attempt. Both public 200-cent budgets must match the fixture +scope before task creation; unlimited or foreign budget records fail admission. ## Persistent agent files diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index a16c223b220..fc9d26edc6a 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1522,6 +1522,9 @@ the native run's company/task/agent scope, responsible user, selected account, provider/method, native Hermes harness and requested/effective model. Missing or mismatched metadata fails even when the answer is correct. The report retains the checks alongside the ordinary completion, screenshot, token/cost and cleanup evidence. +Each cell admits one attempt and configures 200-cent company and agent budgets. +Public budget readback must pass before task creation. Unpriced model usage +remains unknown; this configured limit is not an exact billing receipt. ```sh pnpm test:e2e:runner -- --list --suite hermes-api-connections diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 02b92cdea42..46253b96c9d 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -5,7 +5,7 @@ import { nativeCompletionProfile, NATIVE_COMPLETION_BUDGET_CENTS } from "./nativ import { chatConfirmationTasks } from "./chat-cases.js"; import { buildConnectionSuite } from "./connection-cases.js"; import { hiringTemplateTasks, hiringTemplateProfile, hiringTemplateDefinitionDigest } from "./hiring-template-cases.js"; -import { hermesApiConnectionChoices, hermesApiConnectionDefinitionDigest } from "./hermes-api-connections.js"; +import { HERMES_API_CONNECTION_BUDGET_CENTS, hermesApiConnectionChoices, hermesApiConnectionDefinitionDigest } from "./hermes-api-connections.js"; import { nativeActiveStopTasks } from "./native-active-stop-tasks.js"; import { cursorNativeTasks } from "./cursor-native-cases.js"; import { instructionPersistenceTask } from "./instruction-persistence.js"; @@ -1206,11 +1206,12 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ id: "hermes-api-connections", label: "Hermes managed API connections", manualOnly: true, description: "One native browser task per exact model using an independently selected managed API account; catalog discovery is not inference proof.", groups: ["native"], profiles: hermesApiConnectionProfiles, environments: runnerEnvironments, - tasks: [openRouterBreadthTasks.find(task => task.id === "hello-complete")!], + tasks: [{ ...openRouterBreadthTasks.find(task => task.id === "hello-complete")!, automaticRetryPolicy: "single_attempt" }], expectedMatrixSize: 10, definitionMetadata: { version: 1, qualification: "pending", scheduling: "explicit-only", authenticatedDiscoveryDate: "2026-10-07", accountMethod: "api_key", accountMode: "responsible_user", providerTurns: 1, + budgetMonthlyCents: HERMES_API_CONNECTION_BUDGET_CENTS, maximumAttemptsPerCell: 1, coverage: "api-account-native-completion-only", sourceDigest: hermesApiConnectionDefinitionDigest, }, }, diff --git a/tests/runner-e2e/hermes-api-connections.test.ts b/tests/runner-e2e/hermes-api-connections.test.ts index 95e74c97cb7..98b85787572 100644 --- a/tests/runner-e2e/hermes-api-connections.test.ts +++ b/tests/runner-e2e/hermes-api-connections.test.ts @@ -2,7 +2,7 @@ import { describe, expect, it } from "vitest"; import { runnerMatrix, runnerSuites, suiteDefinitionHash } from "./catalog.js"; import { buildMatrixJobs, parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js"; import { buildRunnerE2EProcessEnvironment } from "./harness-env.js"; -import { gradeHermesApiConnection } from "./hermes-api-connections.js"; +import { captureHermesApiBudgets, gradeHermesApiConnection } from "./hermes-api-connections.js"; describe("Hermes managed API connection qualification", () => { const suite = runnerSuites.find(s => s.id === "hermes-api-connections")!; @@ -11,8 +11,8 @@ describe("Hermes managed API connection qualification", () => { expect(cells).toHaveLength(10); expect(suite.manualOnly).toBe(true); expect(new Set(cells.map(e => e.environment.id))).toEqual(new Set(["local", "daytona"])); - expect(cells.every(e => e.task.id === "hello-complete" && e.task.expectedRunCount === 1)).toBe(true); - expect(suite.definitionMetadata).toMatchObject({ qualification: "pending", accountMethod: "api_key", accountMode: "responsible_user", coverage: "api-account-native-completion-only" }); + expect(cells.every(e => e.task.id === "hello-complete" && e.task.expectedRunCount === 1 && e.task.automaticRetryPolicy === "single_attempt")).toBe(true); + expect(suite.definitionMetadata).toMatchObject({ qualification: "pending", accountMethod: "api_key", accountMode: "responsible_user", coverage: "api-account-native-completion-only", budgetMonthlyCents: 200, maximumAttemptsPerCell: 1 }); expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(e => e.suite.id === suite.id)).toBe(false); }); it("pins the exact selected candidate and model in the operator admission", () => { @@ -38,6 +38,19 @@ describe("Hermes managed API connection qualification", () => { it("accepts independently observed account/model metadata", () => { expect(gradeHermesApiConnection(valid).every(check => check.passed)).toBe(true); }); + it.each(["valid", "company-budget", "agent-budget", "company-scope", "agent-scope", "agent-company"])("checks %s through public budget readback before a paid task", async fault => { + const company = { id: "company", budgetMonthlyCents: fault === "company-budget" ? 0 : 200 }; + const agent = { id: "agent", companyId: "company", budgetMonthlyCents: fault === "agent-budget" ? 0 : 200 }; + if (fault === "company-scope") company.id = "foreign"; + if (fault === "agent-scope") agent.id = "foreign"; + if (fault === "agent-company") agent.companyId = "foreign"; + const paths: string[] = []; + const receipt = await captureHermesApiBudgets({ companyId: "company", agentId: "agent", api: { + async get(url: string) { paths.push(url); return (url === "/api/companies/company" ? company : agent) as T; }, + } }); + expect(new Set(paths)).toEqual(new Set(["/api/companies/company", "/api/agents/agent"])); + expect(receipt.checks.every(check => check.passed)).toBe(fault === "valid"); + }); it.each(["company", "task", "account", "provider", "method", "user", "model", "harness", "missing", "extra-run"])("rejects %s evidence even with a successful answer", fault => { const wrong = structuredClone(valid), run = wrong.runs[0]!; if (fault === "company") run.companyId = "foreign"; diff --git a/tests/runner-e2e/hermes-api-connections.ts b/tests/runner-e2e/hermes-api-connections.ts index 60eb7cb57da..5a8a63cf7a7 100644 --- a/tests/runner-e2e/hermes-api-connections.ts +++ b/tests/runner-e2e/hermes-api-connections.ts @@ -5,6 +5,21 @@ import type { RunnerProfileFixture } from "./types.js"; const record = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; const present = (value: unknown) => typeof value === "string" && value.trim().length > 0; +export const HERMES_API_CONNECTION_BUDGET_CENTS = 200; + +/** Read both public budgets before creating the paid task. Unknown usage stays unknown. */ +export async function captureHermesApiBudgets(input: { + api: { get(path: string): Promise }; companyId: string; agentId: string; +}) { + const [company, agent] = await Promise.all([ + input.api.get<{ id: string; budgetMonthlyCents: unknown }>(`/api/companies/${input.companyId}`), + input.api.get<{ id: string; companyId: string; budgetMonthlyCents: unknown }>(`/api/agents/${input.agentId}`), + ]); + return { budgetMonthlyCents: HERMES_API_CONNECTION_BUDGET_CENTS, checks: [ + { id: "company-budget", passed: company.id === input.companyId && company.budgetMonthlyCents === HERMES_API_CONNECTION_BUDGET_CENTS }, + { id: "agent-budget", passed: agent.id === input.agentId && agent.companyId === input.companyId && agent.budgetMonthlyCents === HERMES_API_CONNECTION_BUDGET_CENTS }, + ] }; +} /** Grade public run/account/model metadata; a model's completion claim cannot supply it. */ export function gradeHermesApiConnection(input: { diff --git a/tests/runner-e2e/live-fixtures.test.ts b/tests/runner-e2e/live-fixtures.test.ts index d977a0310ba..604f9d1806e 100644 --- a/tests/runner-e2e/live-fixtures.test.ts +++ b/tests/runner-e2e/live-fixtures.test.ts @@ -119,10 +119,11 @@ describe("live runner fixtures", () => { : ["runner-acpx-claude", "runner-acpx-hermes-api-anthropic"].includes(profile) ? "anthropic" : ["runner-opencode", "runner-acpx-hermes"].includes(profile) ? "openrouter" : "openai"; let connected = false; + let companyBody: any; let agentBody: any; const api = { async post(url: string, data: any) { - if (url === "/api/companies") return { id: "company", name: "Test" }; + if (url === "/api/companies") { companyBody = data; return { id: "company", name: "Test" }; } if (url.endsWith("/agents")) { agentBody = data; return { id: "lead", ...data }; @@ -158,6 +159,10 @@ describe("live runner fixtures", () => { }, }); expect(connected).toBe(true); + if (suite === "hermes-api-connections") { + expect(companyBody.budgetMonthlyCents).toBe(200); + expect(agentBody.budgetMonthlyCents).toBe(200); + } expect(agentBody.adapterConfig.env).toBeUndefined(); expect(agentBody.runtimeConfig.aiConnection).toEqual({ provider, diff --git a/tests/runner-e2e/live-fixtures.ts b/tests/runner-e2e/live-fixtures.ts index 6e92451bebe..b24b0a47e45 100644 --- a/tests/runner-e2e/live-fixtures.ts +++ b/tests/runner-e2e/live-fixtures.ts @@ -1,4 +1,5 @@ import { NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js"; +import { HERMES_API_CONNECTION_BUDGET_CENTS } from "./hermes-api-connections.js"; import path from "node:path"; import { installedReleaseDaytonaPlugin } from "./installed-release.js"; import { isManagedHiringCase } from "./chat-cases.js"; @@ -139,7 +140,8 @@ export async function setupLiveFixtures(input: { return api.post("/api/companies", { name: `Runner E2E ${execution.id} ${input.executionNonce}`, description: "Ephemeral paid full-stack runner acceptance fixture", - budgetMonthlyCents: ["native-completion", "native-instruction-consolidation", "native-connection-guidance"].includes(execution.suite.id) + budgetMonthlyCents: execution.suite.id === "hermes-api-connections" ? HERMES_API_CONNECTION_BUDGET_CENTS + : ["native-completion", "native-instruction-consolidation", "native-connection-guidance"].includes(execution.suite.id) || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id)) ? NATIVE_COMPLETION_BUDGET_CENTS : execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS : execution.suite.id === "stock-harness" ? 1_000 : 0, @@ -302,6 +304,7 @@ export async function setupLiveFixtures(input: { || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id))) { agent.budgetMonthlyCents = 1_000; } + if (execution.suite.id === "hermes-api-connections") agent.budgetMonthlyCents = HERMES_API_CONNECTION_BUDGET_CENTS; if (managedHiring) { const account = value(resolved, "ai-connection"); const config = agent.adapterConfig as Record; diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 5ab50a50e38..4a67aad8e74 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1,5 +1,5 @@ import { runPlanTaskFlow } from "./plan-task-flow.js"; -import { gradeHermesApiConnection } from "./hermes-api-connections.js"; +import { captureHermesApiBudgets, gradeHermesApiConnection } from "./hermes-api-connections.js"; import { assertNativeCompletionSelection, NATIVE_COMPLETION_PREFLIGHT_ENV, verifyNativeCompletionPreflight } from "./native-completion-admission.js"; import { assertNativeInstructionSelection, verifyNativeInstructionPreflight, NATIVE_INSTRUCTION_PREFLIGHT_ENV, NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_DEFAULT_SHA256 } from "./native-instruction-consolidation.js"; import { captureNativeDefault, gradeNativeDefault, nativeCompletionWorkspaceDigest } from "./native-completion-defaults.js"; @@ -894,6 +894,11 @@ for (const execution of executions) { daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE, }); + if (execution.suite.id === "hermes-api-connections") { + const receipt = await captureHermesApiBudgets({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id }); + await writeSanitizedJson(snapshotsDir, "hermes-api-budgets-before-execution.json", receipt, secrets); + if (!receipt.checks.every(check => check.passed)) throw new Error("Hermes API budgets failed admission before task creation"); + } if (["native-completion", NATIVE_INSTRUCTION_SUITE].includes(execution.suite.id)) { const receipt = await captureNativeDefault({ api, agentId: fixtures.agent.id, companyId: fixtures.company.id }); const grade = gradeNativeDefault(receipt, execution.suite.id === NATIVE_INSTRUCTION_SUITE ? NATIVE_INSTRUCTION_DEFAULT_SHA256 : undefined);